Podcasts about Sidekiq

Play Episode Listen Later May 9, 2024 43:43

In Elixir Wizards Office Hours Episode 8, hosts Sundi Myint and Owen Bickford lead an engaging Q&A session with co-host Dan Ivovich, diving deep into the nuances of DevOps. Drawing from his extensive experience, Dan navigates topics from the early days before Docker to managing diverse polyglot environments and optimizing observability. This episode offers insights for developers of all levels looking to sharpen their DevOps skills. Explore the realms of Docker, containerization, DevOps workflows, and the deployment intricacies of Elixir applications. Key topics discussed in this episode: Understanding DevOps and starting points for beginners Best practices for deploying applications to the cloud Using Docker for containerization Managing multiple programming environments with microservices Strategies for geographic distribution and ensuring redundancy Localization considerations involving latency and device specs Using Prometheus and OpenTelemetry for observability Adjusting scaling based on application metrics Approaching failure scenarios, including database migrations and managing dependencies Tackling challenges in monitoring setups and alert configurations Implementing incremental, zero-downtime deployment strategies The intricacies of hot code upgrades and effective state management Recommended learning paths, including Linux and CI/CD workflows Tools for visualizing system health and monitoring Identifying actionable metrics and setting effective alerts Links mentioned: Ansible open source IT automation engine https://www.ansible.com/ Wikimedia engine https://doc.wikimedia.org/ Drupal content management software https://www.drupal.org/ Capistrano remote server automation and deployment https://capistranorb.com/ Docker https://www.docker.com/ Circle CI CI/CD Tool https://circleci.com/ DNS Cluster https://hex.pm/packages/dnscluster ElixirConf 2023 Chris McCord Phoenix Field Notes https://youtu.be/Ckgl9KO4E4M Nerves https://nerves-project.org/ Oban job processing in Elixir https://getoban.pro/ Sidekiq background jobs for Ruby https://sidekiq.org/ Prometheus https://prometheus.io/ PromEx https://hexdocs.pm/promex/PromEx.html GitHub Actions - Setup BEAM: https://github.com/erlef/setup-beam Jenkins open source automation server https://www.jenkins.io/ DataDog Cloud Monitoring https://www.datadoghq.com/

"You've Got a Job to Do" with Joel Meador

Play Episode Listen Later Apr 4, 2024 44:26

Today on Elixir Wizards Office Hours, SmartLogic Engineer Joel Meador joins Dan Ivovich to discuss all things background jobs. The behind-the-scenes heroes of app performance and scalability, background jobs take center stage as we dissect their role in optimizing user experience and managing heavy-lifting tasks away from the main application flow. From syncing with external systems to processing large datasets, background jobs are pivotal to successful application management. Dan and Joel share their perspectives on monitoring, debugging, and securing background jobs, emphasizing the need for a strategic approach to these hidden workflows. Key topics discussed in this episode: The vital role of background jobs in app performance Optimizing user experience through background processing Common pitfalls: resource starvation and latency issues Strategies for effective monitoring and debugging of task runners and job schedulers Data integrity and system security in open source software Background job tools like Oban, Sidekiq, Resque, Cron jobs, Redis pub sub CPU utilization and processing speed Best practices for implementing background jobs Keeping jobs small, focused, and well-monitored Navigating job uniqueness, locking, and deployment orchestration Leveraging asynctask for asynchronous operations The art of continuous improvement in background job management Links mentioned in this episode: https://redis.io/ Oban job processing library https://hexdocs.pm/oban/Oban.html Resque Ruby library for background jobs https://github.com/resque Sidekiq background processing for Ruby https://github.com/sidekiq Delayed Job priority queue system https://github.com/collectiveidea/delayed_job RabbitMQ messaging and streaming broker https://www.rabbitmq.com/ Mnesia distributed telecommunications DBMS https://www.erlang.org/doc/man/mnesia.html Task for Elixir https://hexdocs.pm/elixir/1.12/Task.html ETS in-memory store for Elixir and Erlang objects https://hexdocs.pm/ets/ETS.html Cron - https://en.wikipedia.org/wiki/Cron Donate to Miami Indians of Indiana https://www.miamiindians.org/take-action Joel Meador on Tumblr https://joelmeador.tumblr.com/ Special Guest: Joel Meador.

Web Development Frameworks: Elixir and Phoenix vs. Ruby on Rails with Owen Bickford & Dan Ivovich

Play Episode Listen Later Dec 7, 2023 41:41

On today's episode, Elixir Wizards Owen Bickford and Dan Ivovich compare notes on building web applications with Elixir and the Phoenix Framework versus Ruby on Rails. They discuss the history of both frameworks, key differences in architecture and approach, and deciding which programming language to use when starting a project. Both Phoenix and Rails are robust frameworks that enable developers to build high-quality web apps—Phoenix leverages functional programming in Elixir and Erlang's networking for real-time communication. Rails follows object-oriented principles and has a vast ecosystem of plug-ins. For data-heavy CRUD apps, Phoenix's immutable data pipelines provide some advantages. Developers can build great web apps with either Phoenix or Rails. Phoenix may have a slight edge for new projects based on its functional approach, built-in real-time features like LiveView, and ability to scale efficiently. But, choosing the right tech stack depends heavily on the app's specific requirements and the team's existing skills. Topics discussed in this episode: History and evolution of Phoenix Framework and Ruby on Rails Default project structure and code organization preferences in each framework Comparing object-oriented vs functional programming paradigms CRUD app development and interaction with databases Live reloading capabilities in Phoenix LiveView vs Rails Turbolinks Leveraging WebSockets for real-time UI updates Testing frameworks like RSpec, Cucumber, Wallaby, and Capybara Dependency management and size of standard libraries Scalability and distribution across nodes Readability and approachability of object-oriented code Immutability and data pipelines in functional programming Types, specs, and static analysis with Dialyzer Monkey patching in Ruby vs extensible core language in Elixir Factors to consider when choosing between frameworks Experience training new developers on Phoenix and Rails Community influences on coding styles Real-world project examples and refactoring approaches Deployment and dev ops differences Popularity and adoption curves of both frameworks Ongoing research into improving Phoenix and Rails Links Mentioned in this Episode: SmartLogic.io (https://smartlogic.io/) Dan's LinkedIn (https://www.linkedin.com/in/divovich/) Owen's LinkedIn (https://www.linkedin.com/in/owen-bickford-8b6b1523a/) Ruby https://www.ruby-lang.org/en/ Rails https://rubyonrails.org/ Sams Teach Yourself Ruby in 21 Days (https://www.overdrive.com/media/56304/sams-teach-yourself-ruby-in-21-days) Learn Ruby in 7 Days (https://www.thriftbooks.com/w/learn-ruby-in-7-days---color-print---ruby-tutorial-for-guaranteed-quick-learning-ruby-guide-with-many-practical-examples-this-ruby-programming-book--to-build-real-life-software-projects/18539364/#edition=19727339&idiq=25678249) Build Your Own Ruby on Rails Web Applications (https://www.thriftbooks.com/w/build-your-own-ruby-on-rails-web-applications_patrick-lenz/725256/item/2315989/?utm_source=google&utm_medium=cpc&utm_campaign=low_vol_backlist_standard_shopping_customer_acquisition&utm_adgroup=&utm_term=&utm_content=593118743925&gad_source=1&gclid=CjwKCAiA1MCrBhAoEiwAC2d64aQyFawuU3znN0VFgGyjR0I-0vrXlseIvht0QPOqx4DjKjdpgjCMZhoC6PcQAvD_BwE#idiq=2315989&edition=3380836) Django https://github.com/django Sidekiq https://github.com/sidekiq Kafka https://kafka.apache.org/ Phoenix Framework https://www.phoenixframework.org/ Phoenix LiveView https://hexdocs.pm/phoenixliveview/Phoenix.LiveView.html#content Flask https://flask.palletsprojects.com/en/3.0.x/ WebSockets API https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API WebSocket connection for Phoenix https://github.com/phoenixframework/websock Morph Dom https://github.com/patrick-steele-idem/morphdom Turbolinks https://github.com/turbolinks Ecto https://github.com/elixir-ecto Capybara Testing Framework https://teamcapybara.github.io/capybara/ Wallaby Testing Framework https://wallabyjs.com/ Cucumber Testing Framework https://cucumber.io/ RSpec https://rspec.info/

history startups mvp engineers hackers automation saas developers programming algorithms react computer science ux agile makers coding ui builders github libraries information technology devops rails scrum frameworks deployment scripts kafka cucumbers mobile apps django elixir git technologists software engineering scalability web development flutter ci cd bootcamps paas ruby on rails web apps prototypes wallaby programming languages flask iaas technical debt ecto rest apis crud design patterns computer programming functional programming erlang sdlc user stories bickford pull requests distributed systems test driven development open source community clean code unit testing websockets liveview code reviews version control compilers state management front end development data structures developer productivity object oriented programming code quality activerecord sidekiq rspec dependency management integration testing phoenix framework multiprocessing immutable data monkey patching

Mike Perham on Keeping it solo (RubyConf 2023)

Software Sessions

Play Episode Listen Later Nov 21, 2023 51:26

Mike Perham is the creator of Sidekiq, a background job processor for Ruby. He's also the creator of Faktory a similar product for multiple language environments. We talk about the RubyConf keynote and Ruby's limitations, supporting products as a solo developer, and some ideas for funding open source like a public utility. Recorded at RubyConf 2023 in San Diego. -- A few topics covered: Sidekiq (Ruby) vs Faktory (Polyglot) Why background job solutions are so common in Ruby Global Interpreter Lock (GIL) Ractors (Actor concurrency) Downsides of Multiprocess applications When to use other languages Getting people to pay for Sidekiq Keeping a solo business Being selective about customers Ways to keep support needs low Open source as a public utility Mike Mike's blog mastodon Sidekiq faktory From Employment to Independence Ruby Ractor The Practical Effects of the GVL on Scaling in Ruby Transcript You can help correct transcripts on GitHub. Introduction [00:00:00] Jeremy: I'm here at RubyConf San Diego with Mike Perham. He's the creator of Sidekiq and Faktory. [00:00:07] Mike: Thank you, Jeremy, for having me here. It's a pleasure. Sidekiq [00:00:11] Jeremy: So for people who aren't familiar with, I guess we'll start with Sidekiq because I think that's what you're most known for. If people don't know what it is, maybe you can give like a small little explanation. [00:00:22] Mike: Ruby apps generally have two major pieces of infrastructure powering them. You've got your app server, which serves your webpages and the browser. And then you generally have something off on the side that... It processes, you know, data for a million different reasons, and that's generally called a background job framework, and that's what Sidekiq is. [00:00:41] It, Rails is usually the thing that, that handles your web stuff, and then Sidekiq is the Sidekiq to Rails, so to speak. [00:00:50] Jeremy: And so this would fit the same role as, I think in Python, there's celery. and then in the Ruby world, I guess there is, uh, Resque is another kind of job. [00:01:02] Mike: Yeah, background job frameworks are quite prolific in Ruby. the Ruby community's kind of settled on that as the, the standard pattern for application development. So yeah, we've got, a half a dozen to a dozen different, different examples throughout history, but the major ones today are, Sidekiq, Resque, DelayedJob, GoodJob, and, and, and others down the line, yeah. Why background jobs are so common in Ruby [00:01:25] Jeremy: I think working in other languages, you mentioned how in Ruby, there's this very clear, preference to use these job scheduling systems, these job queuing systems, and I'm not. I'm not sure if that's as true in, say, if somebody's working in Java, or C sharp, or whatnot. And I wonder if there's something specific about Ruby that makes people kind of gravitate towards this as the default thing they would use. [00:01:52] Mike: That's a good question. What makes Ruby... The one that so needs a background job system. I think Ruby, has historically been very single threaded. And so, every Ruby process can only do so much work. And so Ruby oftentimes does, uh, spin up a lot of different processes, and so having processes that are more focused on one thing is, is, is more standard. [00:02:24] So you'll have your application server processes, which focus on just serving HTTP responses. And then you have some other sort of focused process and that just became background job processes. but yeah, I haven't really thought of it all that much. But, uh, you know, something like Java, for instance, heavily multi threaded. [00:02:45] And so, and extremely heavyweight in terms of memory and startup time. So it's much more frequent in Java that you just start up one process and that's it. Right, you just do everything in that one process. And so you may have dozens and dozens of threads, both serving HTTP and doing work on the side too. Um, whereas in Ruby that just kind of naturally, there was a natural split there. Global Interpreter Lock [00:03:10] Jeremy: So that's actually a really good insight, because... in the keynote at RubyConf, Mats, the creator of Ruby, you know, he mentioned the, how the fact that there is this global, interpreter lock, [00:03:23] or, or global VM lock in Ruby, and so you can't, really do multiple things in parallel and make use of all the different cores. And so it makes a lot of sense why you would say like, okay, I need to spin up separate processes so that I can actually take advantage of, of my, system. [00:03:43] Mike: Right. Yeah. And the, um, the GVL. is the acronym we use in the Ruby community, or GIL. Uh, that global lock really kind of is a forcing function for much of the application architecture in Ruby. Ruby, uh, applications because it does limit how much processing a single Ruby process can do. So, uh, even though Sidekiq is heavily multi threaded, you can only have so many threads executing. [00:04:14] Because they all have to share one core because of that global lock. So unfortunately, that's, that's been, um, one of the limiter, limiting factors to Sidekiq scalability is that, that lock and boy, I would pay a lot of money to just have that lock go away, but. You know, Python is going through a very long term experiment about trying to remove that lock and I'm very curious to see how well that goes because I would love to see Ruby do the same and we'll see what happens in the future, but, it's always frustrating when I come to another RubyConf and I hear another Matt's keynote where he's asked about the GIL and he continues to say, well, the GIL is going to be around, as long as I can tell. [00:04:57] so it's a little bit frustrating, but. It's, it's just what you have to deal with. Ractors [00:05:02] Jeremy: I'm not too familiar with them, but they, they did mention during the keynote I think there Ractors or something like that. There, there, there's some way of being able to get around the GIL but there are these constraints on them. And in the context of Sidekiq and, and maybe Ruby in general, how do you feel about those options or those solutions? [00:05:22] Mike: Yeah, so, I think it was Ruby 3. 2 that introduced this concept of what they call a Ractor, which is like a thread, except it does not have the global lock. It can run independent to the global lock. The problem is, is because it doesn't use the global lock, it has pretty severe constraints on what it can do. [00:05:47] And the, and more specifically, the data it can access. So, Ruby apps and Rails apps throughout history have traditionally accessed a lot of global data, a lot of class level data, and accessed all this data in a, in a read only fashion. so there's no race conditions because no one's changing any of it, but it's still, lots of threads all accessing the same variables. [00:06:19] Well, Ractors can't do that at all. The only data Ractors can access is data that they own. And so that is completely foreign to Ruby application, traditional Ruby applications. So essentially, Ractors aren't compatible with the vast majority of existing Ruby code. So I, I, I toyed with the idea of prototyping Sidekiq and Ractors, and within about a minute or two, I just ran into these, these, uh... [00:06:51] These very severe constraints, and so that's why you don't see a lot of people using Ractors, even still, even though they've been out for a year or two now, you just don't see a lot of people using them, because they're, they're really limited, limited in what they can do. But, on the other hand, they're unlimited in how well they can scale. [00:07:12] So, we'll see, we'll see. Hopefully in the future, they'll make a lot of improvements and, uh, maybe they'll become more usable over time. Downsides of multiprocess (Memory usage) [00:07:19] Jeremy: And with the existence of a job queue or job scheduler like Sidekiq, you're able to create additional processes to get around that global lock, I suppose. What are the... downsides of doing so versus another language like we mentioned Java earlier, which is capable of having true parallelism in the same process. [00:07:47] Mike: Yeah, so you can start up multiple Ruby processes to process things truly in parallel. The issue is that you do get some duplication in terms of memory. So your Ruby app maybe take a gigabyte per process. And, you can do copy on write forking. You can fork and get some memory sharing with copy on write semantics on Unix operating systems. [00:08:21] But you may only get, let's say, 30 percent memory savings. So, there's still a significant memory overhead to forking, you know, let's say, eight processes versus having eight threads. You know, you, you, you may have, uh, eight threads can operate in a gigabyte process, but if you want to have eight processes, that may take, let's say, four gigabytes of RAM. [00:08:48] So you, you still, it's not going to cost you eight gigabytes of RAM, you know, it's not like just one times eight, but, there's still a overhead of having those separate processes. [00:08:58] Jeremy: would you say it's more of a cost restriction, like it costs you more to run these applications, or are there actual problems that you can't solve because of this restriction. [00:09:13] Mike: Help me understand, what do you mean by restriction? Do you mean just the GVL in general, or the fact that forking processes still costs memory? [00:09:22] Jeremy: I think, well, it would be both, right? So you're, you have two restrictions right now. You have the, the GVL, which means you can't have parallelism within the same process. And then your other option is to spin up a bunch of processes, which you have said is the downside there is that you're using a lot more RAM. [00:09:43] I suppose my question is that Does that actually stop you from doing anything? Like, if you throw more money at the problem, you go like, we're going to have more instances, I'll pay for the RAM, it's fine, can that basically get you out of these situations or are these limitations actually stopping you from, from doing things you could do in other languages? [00:10:04] Mike: Well, you certainly have to manage the multiple processes, right? So you've gotta, you know, if one child process crashes, you've gotta have a parent or supervisor process watching all that and monitoring and restarting the process. I don't think it restricts you. Necessarily, it just, it adds complexity to your deployment. [00:10:24] and, and it's just a question of efficiency, right? Instead of being able to deploy on a, on a one gigabyte droplet, I've got to deploy to a four gigabyte droplet, right? Because I just, I need the RAM to run the eight processes. So it, it, it's more of just a purely a function of how much money am I going to have to throw at this problem. [00:10:45] And what's it going to cost me in operational costs to operate this application in production? When to use other languages? [00:10:53] Jeremy: So during the. Keynote, uh, Matz had mentioned that Rails, is really suitable as this one person framework, like you can have a very small team or maybe even yourself and, and build this product. And so I guess from... Your perspective, once you cross a certain threshold, is like, what Ruby and what Sidekiq provides not enough, and that's why you need to start looking into other languages? [00:11:24] Or like, where's the, turning point, or the, if you [00:11:29] Mike: Right, right. The, it's all about the problem you're trying to solve, right? At the end of the day, uh, the, the question is just what are we trying to solve and how are we trying to solve it? So at a higher level, you got to think about the architecture. if the problem you're trying to solve, if the service you're trying to build, if the app you're trying to operate. [00:11:51] If that doesn't really fall into the traditional Ruby application architecture, then you, you might look at it in another language or another ecosystem. something like Go, for instance, can compile down to a single binary, which makes deployment really easy. It makes shipping up a product. on to a user's machine, much simpler than deploying a Ruby application onto a user's desktop machine, for instance, right? [00:12:22] Um, Ruby does have this, this problem of how do you package everything together and deploy it somewhere? Whereas Go, when you can just compile to a single binary, now you've just got a single thing. And it's just... Drop it on the file system and execute it. It's easy. So, um, different, different ecosystems have different application architectures, which empower different ways of solving the same problems. [00:12:48] But, you know, Rails as a, as a one man framework, or sorry, one person framework, It, it, I don't, I don't necessarily, that's a, that's sort of a catchy marketing slogan, but I just think of Rails as the most productive framework you can use. So you, as a single person, you can maximize what you ship and the, the, the value that you can create because Rails is so productive. [00:13:13] Jeremy: So it, seems like it's maybe the, the domain or the type of application you're making. Like you mentioned the command line application, because you want to be able to deliver it to your user easily. Just give them a binary, something like Go or perhaps Rust makes a lot more sense. and then I could see people saying that if you're doing something with machine learning, like the community behind Python, it's, they're just, they're all there. [00:13:41] So Room for more domains in Ruby [00:13:41] Mike: That was exactly the example I was going to use also. Yeah, if you're doing something with data or AI, Python is going to be a more, a more traditional, natural choice. that doesn't mean Ruby can't do it. That doesn't mean, you wouldn't be able to solve the problem with Ruby. And, and there's, that just also means that there's more space for someone who wants to come in and make an impact in the Ruby community. [00:14:03] Find a problem that Ruby's not really well suited to solving right now and build the tooling out there to, to try and solve it. You know, I, I saw a talk, from the fellow who makes the Glimmer gem, which is a native UI toolkit. Uh, a gem for building native UIs in Ruby, which Ruby traditionally can't do, but he's, he's done an amazing job at sort of surfacing APIs to build these, um, these native, uh, native applications, which I think is great. [00:14:32] It's awesome. It's, it's so invigorating to see Ruby in a new space like that. Um, I talked to someone else who's doing the Polars gem, which is focused on data processing. So it kind of takes, um, Python and Pandas and brings that to Ruby, which is, is awesome because if you're a Ruby developer, now you've got all these additional tools which can allow you to solve new sets of problems out there. [00:14:57] So that's, that's kind of what's exciting in the Ruby community right now is just bring it into new spaces. Faktory [00:15:03] Jeremy: In addition to Sidekiq, you have, uh, another product called Faktory, I believe. And so does that serve a, a similar purpose? Is that another job scheduling, job queueing system? [00:15:16] Mike: It is, yes. And it's, it's, it's similar in a way to Sidekiq. It looks similar. It's got similar concepts at the core of it. At the end of the day, Sidekiq is limited to Ruby. Because Sidekiq executes in a Ruby VM, it executes the jobs, and the jobs are, have to be written in Ruby because you're running in the Ruby VM. [00:15:38] Faktory was my attempt to bring, Sidekiq functionality to every other language. I wanted, I wanted Sidekiq for JavaScript. I wanted Sidekiq for Go. I wanted Sidekiq for Python because A, a lot of these other languages also could use a system, a background job system. And the problem though is that. [00:16:04] As a single man, I can't port Sidekiq to every other language. I don't know all the languages, right? So, Faktory kind of changes the architecture and, um, allows you to execute jobs in any language. it, it replaces Redis and provides a server where you just fetch jobs, and you can use it from it. [00:16:26] You can use that protocol from any language to, to build your own worker processes that execute jobs in whatever language you want. [00:16:35] Jeremy: When you say it replaces Redis, so it doesn't use Redis, um, internally, it has its own. [00:16:41] Mike: It does use Redis under the covers. Yeah, it starts Redis as a child process and, connects to it over a Unix socket. And so it's really stable. It's really fast. from the outside, the, the worker processes, they just talk to Faktory. They don't know anything about Redis at all. [00:16:59] Jeremy: I see. And for someone who, like we mentioned earlier in the Python community, for example, there is, um, Celery. For someone who is using a task scheduler like that, what's the incentive to switch or use something different? [00:17:17] Mike: Well, I, I always say if you're using something right now, I'm not going to try and convince you to switch necessarily. It's when you have pain that you want to switch and move away. Maybe you have Maybe there's capabilities in the newer system that you really need that the old system doesn't provide, but Celery is such a widely known system that I'm not necessarily going to try and convince people to move away from it, but if people are looking for a new system, one of the things that Celery does that Faktory does not do is Celery provides like data adapters for using store, lots of different storage systems, right? [00:17:55] Faktory doesn't do that. Faktory is more, has more of the Rails mantra of, you know, Omakase where we choose, I choose to use Redis and that's it. You don't, you don't have a choice for what to use because who cares, you know, at the end of the day, let Faktory deal with it. it's, it's not something that, You should even necessarily be concerned about it. [00:18:17] Just, just try Faktory out and see how it works for you. Um, so I, I try to take those operational concerns off the table and just have the user focus on, you know, usability, performance, and that sort of thing. but it is, it's, it's another background job system out there for people to try out and see if they like that. [00:18:36] And, and if they want to, um, if they know Celery and they want to use Celery, more power to Faktory them. Sidekiq (Ruby) or Faktory (Polyglot) [00:18:43] Jeremy: And Sidekiq and Faktory, they serve a very similar purpose. For someone who they have a new project, they haven't chosen a job. scheduling system, if they were using Ruby, would it ever make sense for them to use Faktory versus use Sidekiq? [00:19:05] Mike: Uh Faktory is excellent in a polyglot situation. So if you're using multiple languages, if you're creating jobs in Ruby, but you're executing them in Python, for instance, um, you know, if you've, I have people who are, Creating jobs in PHP and executing them in Python, for instance. That kind of polyglot scenario, Sidekiq can't do that at all. [00:19:31] So, Faktory is useful there. In terms of Ruby, Ruby is just another language to Faktory. So, there is a Ruby API for using Faktory, and you can create and execute Ruby jobs with Faktory. But, you'll find that in the Ruby community, Sidekiq is much widely... much more widely used and understood and known. So if you're just using Ruby, I think, I think Sidekiq is the right choice. [00:19:59] I wouldn't look at Faktory. But if you do need, find yourself needing that polyglot tool, then Faktory is there. Temporal [00:20:07] Jeremy: And this is maybe one, maybe one layer of abstraction higher, but there's a product called Temporal that has some of this job scheduling, but also this workflow component. I wonder if you've tried that out and how you think about that product? [00:20:25] Mike: I've heard of them. I don't know a lot about the product. I do have a workflow API, the Sidekiq batches, which allow you to fan out jobs and then, and then execute callbacks when all the jobs in that, in that batch are done. But I don't, provide sort of a, a high level. Graphical Workflow Editor or anything like that. [00:20:50] Those to me are more marketing tools that you use to sell the tool for six figures. And I don't think they're usable. And I don't think they're actually used day to day. I provide an API for developers to use. And developers don't like moving blocks of code around in a GUI. They want to write code. And, um, so yeah, temporal, I, like I said, I don't know much about them. [00:21:19] I also, are they a venture capital backed startup? [00:21:22] Jeremy: They are, is my understanding, [00:21:24] Mike: Yeah, that, uh, any, any sort of venture capital backed startup, um, who's building technical infrastructure. I, I would look long and hard at, I'm, I think open source is the right core to build on. Of course I sell commercial software, but. I'm bootstrapped. I'm profitable. [00:21:46] I'm going to be around forever. A VC backed startup, they tend to go bankrupt, because they either get big or they go out of business. So that would be my only comment is, is, be a little bit leery about relying on commercial venture capital based infrastructure for, for companies, uh, long term. Getting people to pay for Sidekiq [00:22:05] Jeremy: So I think that's a really interesting part about your business is that I think a lot of open source maintainers have a really big challenge figuring out how to make it as a living. The, there are so many projects that they all have a very permissive license and you can use them freely one example I can think of is, I, I talked with, uh, David Kramer, who's the CTO at Sentry, and he, I don't think they use it anymore, but they, they were using Nginx, right? [00:22:39] And he's like, well, Nginx, they have a paid product, like Nginx. Plus that or something. I don't know what the name is, but he was like, but I'm not going to pay for it. Right. I'm just going to use the free one. Why would I, you know, pay for the, um, the paid thing? So I, I, I'm kind of curious from your perspective when you were coming up with Sidekiq both as an open source product, but also as a commercial one, how did you make that determination of like to make a product where it's going to be useful in its open source form? [00:23:15] I can still convince people to pay money for it. [00:23:19] Mike: Yeah, the, I was terrified, to be blunt, when I first started out. when I started the Sidekiq project, I knew it was going to take a lot of time. I knew if it was successful, I was going to be doing it for the next decade. Right? So I started in 2012, and here I am in 2023, over a decade, and I'm still doing it. [00:23:38] So my expectation was met in that regard. And I knew I was not going to be able to last that long. If I was making zero dollars, right? You just, you burn out. Nobody can last that long. Well, I guess there are a few exceptions to that rule, but yeah, money, I tend to think makes things a little more sustainable for sure. [00:23:58] Especially if you can turn it into a full time job solving and supporting a project that you, you love and, and is, is, you know, your, your, your baby, your child, so to speak, your software, uh, uh, creation that you've given to the world. but I was terrified. but one thing I did was at the time I was blogging a lot. [00:24:22] And so I was telling people about Sidekiq. I was telling people what was to come. I was talking about ideas and. The one thing that I blogged about was financial experiments. I said bluntly to the, to, to the Ruby community, I'm going to be experimenting with financial stability and sustainability with this project. [00:24:42] So not only did I create this open source project, but I was also publicly saying I I need to figure out how to make this work for the next decade. And so eventually that led to Sidekiq Pro. And I had to figure out how to build a closed source Ruby gem, which, uh, There's not a lot of, so I was kind of in the wild there. [00:25:11] But, you know, thankfully all the pieces came together and it was actually possible. I couldn't have done it if it wasn't possible. Like, we would not be talking if I couldn't make a private gem. So, um, but it happened to work out. Uh, and it allowed me to, to gate features behind a paywall effectively. And, and yeah, you're right. [00:25:33] It can be tough to make people pay for software. but I'm a developer who's selling to other developers, not, not just developers, open source developers, and they know that they have this financial problem, right? They know that there's this sustainability problem. And I was blunt in saying, this is my solution to my sustainability. [00:25:56] So, I charge what I think is a very fair price. It's only a thousand dollars a year to a hobbyist. That may seem like a lot of money to a business. It's a drop in the bucket. So it was easy for developers to say, Hey, listen, we want to buy this tool for a thousand bucks. It'll ensure our infrastructure is maintained for the next decade. [00:26:18] And it's, and it's. And it's relatively cheap. It's way less than, uh, you know, a salary or even a laptop. So, so that's, that's what I did. And, um, it's, it worked out great. People, people really understood. Even today, I talk to people and they say, we, we signed up for Sidekiq Pro to support you. So it's, it's, it's really, um, invigorating to hear people, uh, thank me and, and they're, they're actively happy that they're paying me and our customers. [00:26:49] Jeremy: it's sort of, uh, maybe a not super common story, right, in terms of what you went through. Because when I think of open core businesses, I think of companies like, uh, GitLab, which are venture funded, uh, very different scenario there. I wonder, like, in your case, so you started in 2012, and there were probably no venture backed competitors, right? [00:27:19] People saying that we're going to make this job scheduling system and some VC is going to give me five million dollars and build a team to work on this. It was probably at the time, maybe it was Rescue, which was... [00:27:35] Mike: There was a venture backed system called IronMQ, [00:27:40] Jeremy: Hmm. [00:27:41] Mike: And I'm not sure if they're still around or not, but they... They took, uh, one or more funding rounds. I'm not sure exactly, but they were VC backed. They were doing, background jobs, scheduled jobs, uh, you know, running container, running container jobs. They, they eventually, I think, wound up sort of settling on Docker containers. [00:28:06] They'll basically spin up a Docker container. And that container can do whatever it wants. It can execute for a second and then shut down, or it can run for, for however long, but they would, um, yeah, I, yeah, I'll, I'll stop there because I don't know the actual details of exactly their system, but I'm not sure if they're still around, but that's the only one that I remember offhand that was around, you know, years ago. [00:28:32] Yeah, it's, it's mostly, you know, low level open source infrastructure. And so, anytime you have funded startups, they're generally using that open source infrastructure to build their own SaaS. And so SaaS's are the vast majority of where you see sort of, uh, commercial software. [00:28:51] Jeremy: so I guess in that way it, it, it gave you this, this window or this area where you could come in and there wasn't, other than that iron, product, there wasn't this big money that you were fighting against. It was sort of, it was you telling people openly, I'm, I'm working on this thing. [00:29:11] I need to make money so that I can sustain it. And, if you, yeah. like the work I do, then, you know, basically support me. Right. And, and so I think that, I'm wondering how we can reproduce that more often because when you see new products, a lot of times it is VC backed, right? [00:29:35] Because people say, I need to work on this. I need to be paid. and I can't ask a team to do this. For nothing, right? So [00:29:44] Mike: Yeah. It's. It's a wicked problem. Uh, it's a really, really hard problem to solve if you take vc you there, that that really kind of means that you need to be making tens if not hundreds of millions of dollars in sales. If you are building a small or relatively small. You know, put small in quotes there because I don't really know what that means, but if you have a small open source project, you can't charge huge amounts for it, right? [00:30:18] I mean, Sidekiq is a, I would call a medium sized open source project, and I'm charging a thousand bucks for it. So if you're building, you know, I don't know, I don't even want to necessarily give example, but if you're building some open source project, and It's one of 300 libraries that people's applications will depend on. [00:30:40] You can't necessarily charge a thousand dollars for that library. depending on the size and the capabilities, maybe you can, maybe you can't. But there's going to be a long tail of open source projects that just, they can't, they can't charge much, if anything, for them. So, unfortunately, we have, you know, these You kind of have two pathways. [00:31:07] Venture capital, where you've got to sell a ton, or free. And I've kind of walked that fine line where I'm a small business, I can charge a small amount because I'm bootstrapped. And, and I don't need huge amounts of money, and I, and I have a project that is of the right size to where I can charge a decent amount of money. [00:31:32] That means that I can survive with 500 or a thousand customers. I don't need to have a hundred million dollars worth of customers. Because I, you know, when I started the business, one of the constraints I said is I don't want to hire anybody. I'm just going to be solo. And part of the, part of my ability to keep a low price and, and keep running sustainably, even with just You know, only a few hundred customers is because I'm solo. [00:32:03] I don't have the overhead of investors. I don't have the overhead of other employees. I don't have an office space. You know, my overhead is very small. So that is, um, you know, I just kind of have a unique business in that way, I guess you might say. Keeping the business solo [00:32:21] Jeremy: I think that's that's interesting about your business as well But the fact that you've kept it you've kept it solo which I would imagine in most businesses, they need support people. they need, developers outside of maybe just one. Um, there's all sorts of other, I don't think overhead is the right word, but you just need more people, right? [00:32:45] And, and what do you think it is about Sidekiq that's made it possible for it to just be a one person operation? [00:32:52] Mike: There's so much administrative overhead in a business. I explicitly create business policies so that I can run solo. you know, my support policy is officially you get one email ticket or issue per quarter. And, and anything more than that, I can bounce back and say, well, you're, you're requiring too much support. [00:33:23] In reality, I don't enforce that at all. And people email me all the time, but, but things like. Things like dealing with accounting and bookkeeping and taxes and legal stuff, licensing, all that is, yeah, a little bit of overhead, but I've kept it as minimal as I can. And part of that is I don't want to hire another employee because then that increases the administrative overhead that I have. [00:33:53] And Sidekiq is so tied to me and my knowledge that if I hire somebody, they're probably not going to know Ruby and threading and all the intricate technical detail necessary to build and maintain and support the system. And so really you'll kind of regress a little bit. We won't be able to give as good support because I'm busy helping that other employee. Being selective about customers [00:34:23] Mike: So, yeah, it's, it's a tightrope act where you've got to really figure out how can I scale myself as far as possible without overwhelming myself. The, the overwhelming thing that I have that I've never been able to solve. It's just dealing with billing inquiries, customers, companies, emailing me saying, how do we buy this thing? [00:34:46] Can I get an invoice? Every company out there, it seems wants an invoice. And the problem with invoicing is it takes a lot more. manual labor and administrative overhead to issue that invoice to collect payment on the invoice. So that's one of the reasons why I have a very strict policy about credit card only for, for the vast majority of my customers. [00:35:11] And I demand that companies pay a lot more. You have to have a pretty big enterprise license if you want an invoice. And if the company, if the company comes back and complains and says, well, you know, that's ridiculous. We don't, we don't want to pay that much. We don't need it that much. Uh, you know, I, I say, okay, well then you have two, two things, two, uh, two things. [00:35:36] You can either pay with a credit card or you can not use Sidekiq. Like, that's, that's it. I'm, I don't need your money. I don't want the administrative overhead of dealing with your accounting department. I just want to support my, my customers and build my software. And, and so, yeah, I don't want to turn into a billing clerk. [00:35:55] So sometimes, sometimes the, the, the best thing in business that you can do is just say no. [00:36:01] Jeremy: That's very interesting because I think being a solo... Person is what probably makes that possible, right? Because if you had the additional staff, then you might say like, Well, I need to pay my staff, so we should be getting, you know, as much business as [00:36:19] Mike: Yeah. Chasing every customer you can, right. But yeah. [00:36:22] Every customer is different. I mean, I have some customers that just, they never contact me. They pay their bill really fast or right on time. And they're paying me, you know, five figures, 20, a year. And they just, it's a, God bless them because those are, are the. [00:36:40] Best customers to have and the worst customers are the ones who are paying 99 bucks a month and everything that they don't understand or whatever is a complaint. So sometimes, sometimes you, you want to, vet your customers from that perspective and say, which one of these customers are going to be good? [00:36:58] Which ones are going to be problematic? [00:37:01] Jeremy: And you're only only person... And I'm not sure how many customers you have, but [00:37:08] Mike: I have 2000 [00:37:09] Jeremy: 2000 customers. [00:37:10] Okay. [00:37:11] Mike: Yeah. [00:37:11] Jeremy: And has that been relatively stable or has there been growth [00:37:16] Mike: It's been relatively stable the last couple of years. Ruby has, has sort of plateaued. Um, it's, you don't see a lot of growth. I'm getting probably, um, 15, 20 percent growth maybe. Uh, so I'm not growing like a weed, like, you know, venture capital would want to see, but steady incremental growth is, is, uh, wonderful, especially since I do very little. [00:37:42] Sales and marketing. you know, I come to RubyConf I, I I tweet out, you know, or I, I toot out funny Mastodon Toots occasionally and, and, um, and, and put out new releases of the software. And, and that's, that's essentially my, my marketing. My marketing is just staying in front of developers and, and, and being a presence in the Ruby community. [00:38:06] But yeah, it, it's, uh. I, I, I see not a, not a huge amount of churn, but I see enough sales to, to, to stay up and keep my head above water and to keep growing, um, slowly but surely. Support needs haven't grown [00:38:20] Jeremy: And as you've had that steady growth, has the support burden not grown with it? [00:38:27] Mike: Not as much because once customers are on Sidekiq and they've got it working, then by and large, you don't hear from them all that much. There's always GitHub issues, you know, customers open GitHub issues. I love that. but yeah, by and large, the community finds bugs. and opens up issues. And so things remain relatively stable. [00:38:51] I don't get a lot of the complete newbie who has no idea what they're doing and wants me to, to tell them how to use Sidekiq that I just don't see much of that at all. Um, I have seen it before, but in that case, generally, I, I, I politely tell that person that, listen, I'm not here to educate you on the product. [00:39:14] It's there's documentation in the wiki. Uh, and there's tons of, of more Ruby, generic Ruby, uh, educational material out there. That's just not, not what I do. So, so yeah, by and large, the support burden is, is not too bad because once people are, are up and running, it's stable and, and they don't, they don't need to contact me. [00:39:36] Jeremy: I wonder too, if that's perhaps a function of the price, because if you're a. new developer or someone who's not too familiar with how to do job processing or what they want to do when you, there is the open source product, of course. but then the next step up, I believe is about a hundred dollars a month. [00:39:58] And if you're somebody who is kind of just getting started and learning how things work, you're probably not going to pay that, is my guess. And so you'll never hear from them. [00:40:11] Mike: Right, yeah, that's a good point too, is the open source version, which is what people inevitably are going to use and integrate into their app at first. Because it's open source, you're not going to email me directly, um, and when people do email me directly, Sidekiq support questions, I do, I reply literally, I'm sorry I don't respond to private email, unless you're a customer. [00:40:35] Please open a GitHub issue and, um, that I try to educate both my open source users and my commercial customers to try and stay in GitHub issues because private email is a silo, right? Private email doesn't help anybody else but them. If I can get people to go into GitHub issues, then that's a public record. [00:40:58] that people can search. Because if one person has that problem, there's probably a dozen other people that have that same problem. And then that other, those other 11 people can search and find the solution to their problem at four in the morning when I'm asleep. Right? So that's, that's what I'm trying to do is, is keep, uh, keep everything out in the open so that people can self service as much as possible. Sidekiq open source [00:41:24] Jeremy: And on the open source side, are you still primarily the main contributor? Or do you have other people that are [00:41:35] Mike: I mean, I'd say I do 90 percent of the work, which is why I don't feel guilty about keeping 100 percent of the money. A lot of open source projects, when they look for financial sustainability, they also look for how can we split this money amongst the team. And that's, that's a completely different topic that I've. [00:41:55] is another reason why I've stayed solo is if I hire an employee and I pay them 200, 000 a year as a developer, I'm meanwhile keeping all the rest of the profits of the company. And so that almost seems a little bit unfair. because we're both still working 40 hours a week, right? Why am I the one making the vast majority of the, of the profit and the money? [00:42:19] Um, so, uh, I've always, uh, that's another reason why I've stayed solo, but, but yeah, having a team of people working on something, I do get, regular commits, regular pull requests from people, fixing a bug that they found or just making a tweak that. that they saw, that they thought they could improve. [00:42:42] A little more rarely I get a significant improvement or feature, as a pull request. but Sidekiq is so stable these days that it really doesn't need a team of people maintaining it. The volume of changes necessary, I can easily keep up with that. So, I'm still doing 90 95 percent of the work. Are there other Sidekiq-like opportunities out there? [00:43:07] Jeremy: Yeah, so I think Sidekiq has sort of a unique positioning where it's the code base itself is small enough where you can maintain it yourself and you have some help, but primarily you're the main maintainer. And then you have enough customers who are willing to, to pay for the benefit it gives them on top of what the open source product provides. [00:43:36] cause it's, it's, you were talking about how. Every project people work on, they have, they could have hundreds of dependencies, right? And to ask somebody to, to pay for each of them is, is probably not ever going to happen. And so it's interesting to think about how you have things like, say, you know, OpenSSL, you know, it's a library that a whole bunch of people rely on, but nobody is going to pay a monthly fee to use it. [00:44:06] You have things like, uh, recently there was HashiCorp with Terraform, right? They, they decided to change their license because they, they wanted to get, you know, some of that value back, some of the money back, and the community basically revolted. Right? And did a fork. And so I'm kind of curious, like, yeah, where people can find these sweet spots like, like Sidekiq, where they can find this space where it's just small enough where you can work on it on your own and still get people to pay for it. [00:44:43] It's, I'm trying to picture, like, where are the spaces? Open source as a public utility [00:44:48] Mike: We need to look at other forms of financing beyond pure capitalism. If this is truly public infrastructure that needs to be maintained for the long term, then why are we, why is it that we depend on capitalism to do that? Our roads, our water, our sewer, those are not Capitalist, right? Those are utilities, that's public infrastructure that we maintain, that the government helps us maintain. [00:45:27] And in a sense, tech infrastructure is similar or could be thought of in a similar fashion. So things like Open Collective, things like, uh, there's a, there's a organization in Europe called NLNet, I think, out of the Netherlands. And they do a lot of grants to various open source projects to help them improve the state of digital infrastructure. [00:45:57] They support, for instance, Mastodon as a open source project that doesn't have any sort of corporate backing. They see that as necessary social media infrastructure, uh, for the long term. And, and I, and I think that's wonderful. I like to see those new directions being explored where you don't have to turn everything into a product, right? [00:46:27] And, and try and market and sale, um, and, and run ads and, and do all this stuff. If you can just make the case that, hey, this is, this is useful public infrastructure that so many different, um, Technical, uh, you know, applications and businesses could rely on, much like FedEx and DHL use our roads to the benefit of their own, their own corporate profits. [00:46:53] Um, why, why, why shouldn't we think of tech infrastructure sort of in a similar way? So, yeah, I would like to see us explore more. in that direction. I understand that in America that may not happen for quite a while because we are very, capitalist focused, but it's encouraging to see, um, places like Europe, uh, a little more open to, to trialing things like, cooperatives and, and grants and large long term grants to, to projects to see if they can, uh, provide sustainability in, in, you know, in a new way. [00:47:29] Jeremy: Yeah, that's a good point because I think right now, a lot of the open source infrastructure that we all rely on, either it's being paid for by large companies and at the whim of those large companies, if Google decides we don't want to pay for you to work on this project anymore, where does the money come from? [00:47:53] Right? And on the other hand, there's the thousands, tens of thousands of people who are doing it. just for free out of the, you know, the goodness of their, their heart. And that's where a lot of the burnout comes from. Right. So I think what you're saying is that perhaps a lot of these pieces that we all rely on, that our, our governments, you know, here in the United States, but also around the world should perhaps recognize as this is, like you said, this is infrastructure, and we should be. [00:48:29] Paying these people to keep the equivalent of the roads and, and, uh, all that working. [00:48:37] Mike: Yeah, I mean, I'm not, I'm not claiming that it's a perfect analogy. There's, there's, there's lots of questions that are unanswered in that, right? How do you, how do you ensure that a project is well maintained? What does that even look like? What does that mean? you know, you can look at a road and say, is it full of potholes or is it smooth as glass, right? [00:48:59] It's just perfectly obvious, but to a, to a digital project, it's, it's not as clear. So, yeah, but, but, but exploring those new ways because turning everybody into a businessman so that they can, they can keep their project going, it, it, it itself is not sustainable, right? so yeah, and that's why everything turns into a SaaS because a SaaS is easy to control. [00:49:24] It's easy to gatekeep behind a paywall and it's easy to charge for, whereas a library on GitHub. Yeah. You know, what do you do there? You know, obviously GitHub has sponsors, the sponsors feature. You've got Patreon, you've got Open Collective, you've got Tidelift. There's, there's other, you know, experiments that have been run, but nothing has risen to the top yet. [00:49:47] and it's still, it's still a bit of a grind. but yeah, we'll see, we'll see what happens, but hopefully people will keep experimenting and, and maybe, maybe governments will start. Thinking in the direction of, you know, what does it mean to have a budget for digital infrastructure maintenance? [00:50:04] Jeremy: Yeah, it's interesting because we, we started thinking about like, okay, where can we find spaces for other Sidekiqs? But it sounds like maybe, maybe that's just not realistic, right? Like maybe we need more of a... Yeah, a rethinking of, I guess the, the structure of how people get funded. Yeah. [00:50:23] Mike: Yeah, sometimes the best way to solve a problem is to think at a higher level. You know, we, the, the sustainability problem in American Silicon Valley based open source developers is naturally going to tend toward venture capital and, and capitalism. And I, you know, I think, I think that's, uh, extremely problematic on a, on a lot of different, in a lot of different ways. [00:50:47] And, and so sometimes you need to step back and say, well, maybe we're, maybe we just don't have the right tool set to solve this problem. But, you know, I, I. More than that, I'm not going to speculate on because it is a wicked problem to solve. [00:51:04] Jeremy: Is there anything else you wanted to, to mention or thought we should have talked about? [00:51:08] Mike: No, I, I, I loved the talk, of sustainability and, and open source. And I, it's, it's a, it's a topic really dear to my heart, obviously. So I, I am happy to talk about it at length with anybody, anytime. So thank you for having me. [00:51:25] Jeremy: All right. Thank you very much, Mike.

David Copeland on Medium Sized Decisions (RubyConf 2023)

Software Sessions

Play Episode Listen Later Nov 17, 2023 48:33

David was the chief software architect and director of engineering at Stitch Fix. He's also the author of a number of books including Sustainable Web Development with Ruby on Rails and most recently Ruby on Rails Background Jobs with Sidekiq. He talks about how he made decisions while working with a medium sized team (~200 developers) at Stitch Fix. The audio quality for the first 19 minutes is not great but the correct microphones turn on right after that. Recorded at RubyConf 2023 in San Diego. A few topics covered: Ruby's origins at Stitch Fix Thoughts on Go Choosing technology and cloud services Moving off heroku Building a platform team Where Ruby and Rails fit in today The role of books and how different people learn Large Language Model's effects on technical content Related Links David's Blog Mastodon Transcript You can help correct transcripts on GitHub. Intro [00:00:00] Jeremy: Today. I want to share another conversation from RubyConf San Diego. This time it's with David Copeland. He was a chief software architect and director of engineering at stitch fix. And at the start of the conversation, you're going to hear about why he decided to write the book, sustainable web development with Ruby on rails. Unfortunately, you're also going to notice the sound quality isn't too good. We had some technical difficulties. But once you hit the 20 minute mark of the recording, the mics are going to kick in. It's going to sound way better. So I hope you stick with it. Enjoy. Ruby at Stitch Fix [00:00:35] David: Stitch Fix was a Rails shop. I had done a lot of Rails and learned a lot of things that worked and didn't work, at least in that situation. And so I started writing them down and I was like, I should probably make this more than just a document that I keep, you know, privately on my computer. Uh, so that's, you know, kind of, kind of where the genesis of that came from and just tried to, write everything down that I thought what worked, what didn't work. Uh, if you're in a situation like me. Working on a product, with a medium sized, uh, team, then I think the lessons in there will be useful, at least some of them. Um, and I've been trying to keep it up over, over the years. I think the first version came out a couple years ago, so I've been trying to make sure it's always up to date with the latest stuff and, and Rails and based on my experience and all that. [00:01:20] Jeremy: So it's interesting that you mention, medium sized team because, during the, the keynote, just a few moments ago, Matz the creator of Ruby was talking about how like, Oh, Rails is really suitable for this, this one person team, right? Small, small team. And, uh, he was like, you're not Google. So like, don't worry about, right. Can you scale to that level? Yeah. Um, and, and I wonder like when you talk about medium size or medium scale, like what are, what are we talking? [00:01:49] David: I think probably under 200 developers, I would say. because when I left Stitch Fix, it was closing in on that number of developers. And so it becomes, you know, hard to... You can kind of know who everybody is, or at least the names sound familiar of everybody. But beyond that, it's just, it's just really hard. But a lot of it was like, I don't have experience at like a thousand developer company. I have no idea what that's like, but I definitely know that Rails can work for like... 200 ish people how you can make it work basically. yeah. [00:02:21] Jeremy: The decision to use Rails, I'm assuming that was made before you joined? [00:02:26] David: Yeah, the, um, the CTO of Stitch Fix, he had come in to clean up a mess made by contractors, as often happens. They had used Django, which is like the Python version of Rails. And he, the CTO, he was more familiar with Rails. So the first two developers he hired, also familiar with Rails. There wasn't a lot to maintain with the Django app, so they were like, let's just start fresh, fresh with Rails. yeah, but it's funny because a lot of the code in that Rails app was, like, transliterated from Python. So you could, it would, it looked like the strangest Ruby code in the world because it was basically, there was no test. So they were like, let's just write the Ruby version of this Python just so we know it works. but obviously that didn't, didn't last forever, so. [00:03:07] Jeremy: So, so what's an example of a, of a tell? Where you're looking at the code and you're like, oh, this is clearly, it came from Python. [00:03:15] David: You'd see like, very, very explicit, right? Like Python, there's a lot of like single line things. very like, this sounds like a dig, but it's very simple looking code. Like, like I don't know Python, but I was able to change this Django app. And I had to, I could look at it and you can figure out immediately how it works. Cause there's. Not much to it. There's nothing fancy. So, like, this, this Ruby code, there was nothing fancy. You'd be like, well, maybe they should have memoized that, or maybe they should have taken that into another class, or you could have done this with a hash or something like that. So there was, like, none of that. It was just, like, really basic, plain code like you would see in any beginning programming language kind of thing. Which is at least nice. You can understand it. but you probably wouldn't have written it that way at first in Ruby. Thoughts on Go [00:04:05] Jeremy: Yeah, that's, that's interesting because, uh, people sometimes talk about the Go programming language and how it looks, I don't know if simple is the right word, but it's something where you look at the code and even if you don't necessarily understand Go, it's relatively straightforward. Yeah. I wonder what your thoughts are on that being a strength versus that being, like, [00:04:25] David: Yeah, so at Stitch Fix at one point we had a pro, we were moving off of Heroku and we were going to, basically build a deployment platform using ECS on AWS. And so the deployment platform was a Rails app and we built a command line tool using Ruby. And it was fine, but it was a very complicated command line tool and it was very slow. And so one of the developers was like, I'm going to rewrite it in Go. I was like, ugh, you know, because I just was not a big fan. So he rewrote it in Go. It was a bazillion times faster. And then I was like, okay, I'm going to add, I'll add a feature to it. It was extremely easy. Like, it's just like what you said. I looked at it, like, I don't know anything about Go. I know what is happening here. I can copy and paste this and change things and make it work for what I want to do. And it did work. And it was, it was pretty easy. so there's that, I mean, aesthetically it's pretty ugly and it's, I, I. I can't really defend that as a real reason to not use it, but it is kind of gross. I did do Go, I did a small project in Go after Stitch Fix, and there's this vibe in Go about like, don't create abstractions. I don't know where I got that from, but every Go I look at, I'm like we should make an abstraction for this, but it's just not the vibe. They just don't like doing that. They like it all written out. And I see the value because you can look at the code and know what it does and you don't have to chase abstractions anywhere. But. I felt like I was copying and pasting a lot of, a lot of things. Um, so I don't know. I mean, the, the team at Stitch Fix that did this like command line app in go, they're the platform team. And so their job isn't to write like web apps all day, every day. There's kind of in and out of all kinds of things. They have to try to figure out something that they don't understand quickly to debug a problem. And so I can see the value of something like go if that's your job, right? You want to go in and see what the issue is. Figure it out and be done and you're not going to necessarily develop deep expertise and whatever that thing is that you're kind of jumping into. Day to day though, I don't know. I think it would make me kind of sad. (laughs) [00:06:18] Jeremy: So, so when you say it would make you kind of sad, I mean, what, what about it? Is it, I mean, you mentioned that there's a lot of copy and pasting, so maybe there's code duplication, but are there specific things where you're like, oh, I just don't? [00:06:31] David: Yeah, so I had done a lot of Java in my past life and it felt very much like that. Where like, like the Go library for making an HTTP call for like, I want to call some web service. It's got every feature you could ever want. Everything is tweakable. You can really, you can see why it's designed that way. To dial in some performance issue or solve some really esoteric thing. It's there. But the problem is if you just want to get an JSON, it's just like huge production. And I felt like that's all I really want to do and it's just not making it very easy. And it just felt very, very cumbersome. I think that having to declare types also is a little bit of a weird mindset because, I mean, I like to make types in Ruby, I like to make classes, but I also like to just use hashes and stuff to figure it out. And then maybe I'll make a class if I figure it out, but Go, you can't. You have to have a class, you have to have a type, you have to think all that ahead of time, and it just, I'm not used to working that way, so it felt, I mean, I guess I could get used to it, but I just didn't warm up to that sort of style of working, so it just felt like I was just kind of fighting with the vibe of the language, kind of. Yeah, [00:07:40] Jeremy: so it's more of the vibe or the feel where you're writing it and you're like this seems a little too... Explicit. I feel like I have to be too verbose. It just doesn't feel natural for me to write this. [00:07:53] David: Right, it's not optimized for what in my mind is the obvious case. And maybe that's not the obvious case for the people that write Go programs. But for me, like, I just want to like get this endpoint and get the JSON back as a map. Not any easier than any other case, right? Whereas like in Ruby, right? And you can, I think if you include net HTTP, you can just type get. And it will just return whatever that is. Like, that's amazing. It's optimized for what I think is a very common use case. So it makes me feel really productive. It makes me feel pretty good. And if that doesn't work out long term, I can always use something more complicated. But I'm not required to dig into the NetHttp library just to do what in my mind is something very simple. [00:08:37] Jeremy: Yeah, I think that's something I've noticed myself in working with Ruby. I mean, you have the standard library that's very... Comprehensive and the API surface is such that, like you said there, when you're trying to do common tasks, a lot of times they have a call you make and it kind of does the thing you expected or hoped for. [00:08:56] David: Yeah, yeah. It's kind of, I mean, it's that whole optimized for programmer happiness thing. Like it does. That is the vibe of Ruby and it seems like that is still the way things are. And, you know, I, I suppose if I had a different mindset, I mean, because I work with developers who did not like using Ruby or Rails. They loved using Go or Java. And I, I guess there's probably some psychological analysis we could do about their background and history and mindset that makes that make sense. But, to me, I don't know. It's, it's nice when it's pleasant. And Ruby seems pleasant. (laughs) Choosing Technology [00:09:27] Jeremy: as a... Software Architect, or as a CTO, when, when you're choosing technology, what are some of the things you look at in terms of, you know? [00:09:38] David: Yeah, I mean, I think, like, it's a weird criteria, but I think what is something that the team is capable of executing with? Because, like, most, right, most programming languages all kind of do the same thing. Like, you can kind of get most stuff done in most common popular programming languages. So, it's probably not... It's not true that if you pick the wrong language, you can't build the app. Like, that's probably not really the case. At least for like a web app or something. so it's more like, what is the team that's here to do it? What are they comfortable and capable of doing? I worked on a project with... It was a mix of like junior engineers who knew JavaScript, and then some senior engineers from Google. And for whatever reason someone had chosen a Rails app and none of them were comfortable or really yet competent with doing Ruby on Rails and they just all hated it and like it didn't work very well. Um, and so even though, yes, Rails is a good choice for doing stuff for that team at that moment. Not a good choice. Right. So I think you have to go in and like, what, what are we going to be able to execute on so that when the business wants us to do something, we just do it. And we don't complain and we don't say, Oh, well we can't because this technology that we chose, blah, blah, blah. Like you don't ever want to say that if possible. So I think that's. That's kind of the, the top thing. I think second would be how widely supported is it? Like you don't want to be the cutting edge user that's finding all the bugs in something really. Like you want to use something that's stable. Postgres, MySQL, like those work, those are fine. The bugs have been sorted out for most common use cases. Some super fancy edge database, I don't know if I'd want to be doing, doing that you know? Choosing cloud services [00:11:15] Jeremy: How do you feel about the cloud specific services and databases? Like are you comfortable saying like, oh, I'm going to use... Google Cloud, BigQuery. Yeah. [00:11:27] David: That sort of thing. I think it would kind of fall under the same criteria that I was just, just saying like, so with AWS it's interesting 'cause when we moved from Heroku to AWS by EC2 RDS, their database thing, uh, S3, those have been around for years, probably those are gonna work, but they always introduce new things. Like we, we use RabbitMQ and AWS came out with. Some, I forget what it was, it was a queuing service similar to Rabbit. We were like, Oh, maybe we should switch to that. But it was clear that they weren't really ready to support it. So. Yeah, so we didn't, we didn't switch to that. So I, you gotta try to read the tea leaves of the provider to see are they committed to, to supporting this thing or is this there to get some enterprise client to move into the cloud. And then the idea is to move off of that transitional thing into what they do support. And it's hard to get a clear answer from them too. So it takes a little bit of research to figure out, Are they going to support this or not? Because that's what you don't want. To move everything into some very proprietary cloud system and have them sunset it and say, Oh yeah, now you've got to switch again. Uh, that kind of sucks. So, it's a little trickier. [00:12:41] Jeremy: And what kind of questions or research do you do? Is it purely a function of this thing has existed for X number of years so I feel okay? [00:12:52] David: I mean, it's kind of similar to looking at like some gem you're going to add to your project, right? So you'll, you'll look at how often does it change? Is it being updated? Uh, what is the documentation? Does it look like someone really cared about the documentation? Does the documentation look updated? Are there issues with it that are being addressed or, or not? Um, so those are good signals. I think, talking to other practitioners too can be good. Like if you've got someone who's experienced. You can say, hey, do you know anybody back channeling through, like, everybody knows somebody that works at AWS, you can probably try to get something there. at Stitch Fix, we had an enterprise support contract, and so your account manager will sometimes give you good information if you ask. Again, it's a, they're not going to come out and say, don't use this product that we have, but they might communicate that in a subtle way. So you have to triangulate from all these sources to try to. to try to figure out what, what you want to do. [00:13:50] Jeremy: Yeah, it kind of makes me wish that there was a, a site like, maybe not quite like, can I use, right? Can I use, you can see like, oh, can I use this in my browser? Is there, uh, like an AWS or a Google Cloud? Can I trust this? Can I trust this? Yeah. Is this, is this solid or not? [00:14:04] David: Right, totally. It's like, there's that, that site where you, it has all the Apple products and it says whether or not you should buy it because one may or may not be coming out or they may be getting rid of it. Like, yeah, that would... For cloud services, that would be, that would be nice. [00:14:16] Jeremy: Yeah, yeah. That's like the Mac Buyer's Guide. And then we, we need the, uh, the technology. Yeah. Maybe not buyers. Cloud Provider Buyer's Guide, yeah. I guess we are buyers. [00:14:25] David: Yeah, yeah, totally, totally. [00:14:27] Jeremy: it's interesting that you, you mentioned how you want to see that, okay, this thing is mature. I think it's going to stick around because, I, interviewed, someone who worked on, I believe it was the CloudWatch team. Okay. Daniel Vassalo, yeah. so he left AWS, uh, after I think about 10 years, and then he wrote a book called, uh, The Good Parts of AWS. Oh! And, if you read his book, most of the services he says to use are the ones that are, like, old. Yeah. He's, he's basically saying, like, S3, you know you're good. Yeah. Right? but then all these, if you look at the AWS webpage, they have who knows, I don't know how many hundreds of services. Yeah. He's, he's kind of like I worked there and I would not use, you know, all these new services. 'cause I myself, I don't trust [00:15:14] David: it yet. Right. And so, and they're working there? Yeah, they're working there. Yeah. No. One of the VPs at Stitch Fix had worked on Google Cloud and so when we were doing this transition from Heroku, he was like, we are not using Google Cloud. I was like, really? He's like AWS is far ahead of the game. Do not use Google Cloud. I was like, all right, I don't need any more info. You work there. You said don't. I'm gonna believe you. So [00:15:36] Jeremy: what, what was his did he have like a core point? [00:15:39] David: Um, so he never really had anything bad to say about Google per se. Like I think he enjoyed his time there and I think he thought highly of who he worked with and what he worked on and that sort of thing. But his, where he was coming from was like AWS was so far ahead. of Google on anything that we would use, he was like, there's, there's really no advantage to, to doing it. AWS is a known quantity, right? it's probably still the case. It's like, you know, you've heard the nobody ever got fired for using IBM or using Microsoft or whatever the thing is. Like, I think that's, that was kind of the vibe. And he was like, moving all of our infrastructure right before we're going to go public. This is a serious business. We should just use something that we know will work. And he was like, I know this will work. I'm not confident about. Google, uh, for our use case. So we shouldn't, we shouldn't risk it. So I was like, okay, I trust you because I didn't know anything about any of that stuff at the time. I knew Heroku and that was it. So, yeah. [00:16:34] Jeremy: I don't know if it's good or bad, but like you said, AWS seems to be the default choice. Yeah. And I mean, there's people who use Azure. I assume it's mostly primarily Microsoft. Yeah. And then there's Google Cloud. It's not really clear why you would pick it, unless there was a specific service or something that only they had. [00:16:55] David: Yeah, yeah. Or you're invested in Google, you know, you want to keep everything there. I mean, I don't know. I haven't really been at that level to make that kind of decision, and I would probably choose AWS for the reasons discussed, but, yeah. Moving off Heroku [00:17:10] Jeremy: And then, so at Stitch Fix, you said you moved off of Heroku [00:17:16] David: yeah. Yeah, so we were heavy into Heroku. I think that we were told that at one point we had the biggest Heroku Postgres database on their platform. Not a good place to be, right? You never want to be the biggest customer person, usually. but the problem we were facing was essentially we were going to go public. And to do that, you're under all the scrutiny. about many things, including the IT systems and the security around there. So, like, by default, a Postgres, a Heroku Postgres database is, like, on the internet. It's only secured by the password. all their services are on the internet. So, not, not ideal. they were developing their private cloud service at that time. And so that would have given us, in theory, on paper, it would have solved all of our problems. And we liked Heroku and we liked the developer experience. It was great. but... Heroku private spaces, it was still early. There's a lot of limitations that when they explained why those limitations, they were reasonable. And if we had. started from scratch on Heroku Private Spaces. It probably would have worked great, but we hadn't. So we just couldn't make it work. So we were like, okay, we're going to have to move to AWS so that everything can be basically off the internet. Like our public website needs to be on the internet and that's kind of it. So we need to, so that's basically was the, was the impetus for that. but it's too bad because I love Heroku. It was great. I mean, they were, they were a great partner. They were great. I think if Stitch Fix had started life a year later, Private Spaces. Now it's, it's, it's way different than it was then. Cause it's been, it's a mature product now, so we could have easily done that, but you know, the timing didn't work out, unfortunately. [00:18:50] Jeremy: And that was a compliance thing to, [00:18:53] David: Yeah. And compliance is weird cause they don't tell you what to do, but they give you some parameters that you need to meet. And so one of them is like how you control access. So, so going public, the compliance is around the financial data and. Ensuring that the financial data is accurate. So a lot of the systems at Stichfix were storing the financial data. We, you know, the warehouse management system was custom made. Uh, all the credit card processing was all done, like it was all in some databases that we had running in Heroku. And so those needed to be subject to stricter security than we could achieve with just a single password that we just had to remember to rotate when someone like left the team. So that was, you know, the kind of, the kind of impetus for, for all of that. [00:19:35] Jeremy: when you were using Heroku, Salesforce would have already owned it then. Did you, did you get any sense that you weren't really sure about the future of the platform while you're on it or, [00:19:45] David: At that time, no, it seemed like they were still innovating. So like, Heroku has a Redis product now. They didn't at the time we wish that they did. They told us they're working on it, but it wasn't ready. We didn't like using the third parties. Kafka was not a thing. We very much were interested in that. We would have totally used it if it was there. So they were still. Like doing bigger innovations then, then it seems like they are now. I don't know. It's weird. Like they're still there. They still make money, I assume for Salesforce. So it doesn't feel like they're going away, but they're not innovating at the pace that they were kind of back in the day. [00:20:20] Jeremy: it used to feel like when somebody's asking, I want to host a Rails app. Then you would say like, well, use Heroku because it's basically the easiest to get started. It's a known quantity and it's, it's expensive, but, it seemed for, for most people, it was worth it. and then now if I talk to people, it's like. Not what people suggest anymore. [00:20:40] David: Yeah, because there's, there's actual competitors. It's crazy to me that there was no competitors for years, and now there's like, Render and Fly. io seem to be the two popular alternatives. Um, I doubt they're any cheaper, honestly, but... You get a sense, right, that they're still innovating, still building those platforms, and they can build with, you know, all of the knowledge of what has come before them, and do things differently that might, that might help. So, I still use Heroku for personal things just because I know it, and I, you know, sometimes you don't feel like learning a new thing when you just want to get something done, but, yeah, I, I don't know if we were starting again, I don't know, maybe I'd look into those things. They, they seem like they're getting pretty mature and. Heroku's resting on its laurels, still. [00:21:26] Jeremy: I guess I never quite the mindset, right? Where you You have a platform that's doing really well and people really like it and you acquire it and then it just It seems like you would want to keep it rolling, right? (laughs) [00:21:38] David: Yeah, it's, it is wild, I mean, I guess... Why did you, what was Salesforce thinking they were going to get? Uh, who knows maybe the person at Salesforce that really wanted to purchase it isn't there. And so no one at Salesforce cares about it. I mean, there's all these weird company politics that like, who knows what's going on and you could speculate. all day. What's interesting is like, there's definitely some people in the Ruby community who work there and still are working there. And that's like a little bit of a canary for me. I'm like, all right, well, if that person's still working there, that person seems like they're on the level and, and, and, and seems pretty good. They're still working there. It, it's gotta be still a cool place to be or still doing something, something good. But, yeah, I don't know. I would, I would love to know what was going on in all the Salesforce meetings about acquiring that, how to manage it. What are their plans for it? I would love to know that stuff. [00:22:29] Jeremy: maybe you had some experience with this at Stitch Fix But I've heard with Heroku some of their support staff at least in the past they would, to some extent, actually help you troubleshoot, like, what's going on with your app. Like, if your app is, like, using a whole bunch of memory, and you're out of memory, um, they would actually kind of look into that, for you, which is interesting, because it's like, that's almost like a services thing than it is just a platform. [00:22:50] David: Yeah. I mean, they, their support, you would get, you would get escalated to like an engineer sometimes, like who worked on that stuff and they would help figure out what the problem was. Like you got the sense that everybody there really wanted the platform to be good and that they were all sort of motivated to make sure that everybody. You know, did well and used the platform. And they also were good at, like a thing that trips everybody up about Heroku is that your app restarts every day. And if you don't know anything about anything, you might think that is stupid. Why, why would I want that? That's annoying. And I definitely went through that and I complained to them a lot. And I'm like, if you only could not restart. And they very patiently and politely explained to me why that it needed to do that, they weren't going to remove that, and how to think about my app given that reality, right? Which is great because like, what company does that, right? From the engineers that are working on it, like No, nobody does that. So, yeah, no, I haven't escalated anything to support at Heroku in quite some time, so I don't know if it's still like that. I hope it is, but I'm not really, not really sure. Building a platform team [00:23:55] Jeremy: Yeah, that, uh, that reminds me a little bit of, I think it's Rackspace? There's, there's, like, another hosting provider that was pretty popular before, and they... Used to be famous for that type of support, where like your, your app's having issues and somebody's actually, uh, SSHing into your box and trying to figure out like, okay, what's going on? which if, if that's happening, then I, I can totally see where the, the price is justified. But if the support is kind of like dropping off to where it's just, they don't do that kind of thing, then yeah, I can see why it's not so much of a, yeah, [00:24:27] David: We used to think of Heroku as like they were the platform team before we had our own platform team and they, they acted like it, which was great. [00:24:35] Jeremy: Yeah, I don't have, um, experience with, render, but I, I, I did, talk to someone from there, and it does seem like they're, they're trying to fill that role, um, so, yeah, hopefully, they and, and other companies, I guess like Vercel and things like that, um, they're, they're all trying to fill that space, [00:24:55] David: Yeah, cause, cause building our own internal platform, I mean it was the right thing to do, but it's, it's a, you can't just, you have to have a team on it, it's complicated, getting all the stuff in AWS to work the way you want it to work, to have it be kind of like Heroku, like it's not trivial. if I'm a one person company, I don't want to be messing around with that particularly. I want to just have it, you know, push it up and have it go and I'm willing to pay for that. So it seems logical that there would be competitors in that space. I'm glad there are. Hopefully that'll light a fire under, under everybody. [00:25:26] Jeremy: so in your case, it sounds like you moved to having your own platform team and stuff like that, uh, partly because of the compliance thing where you're like, we need our, we need to be isolated from the internet. We're going to go to AWS. If you didn't have that requirement, do you still think like that would have been the time to, to have your own platform team and manage that all yourself? [00:25:46] David: I don't know. We, we were thinking an issue that we were running into when we got bigger, um, was that, I mean, Heroku, it, It's obviously not as flexible as AWS, but it is still very flexible. And so we had a lot of internal documentation about this is how you use Heroku to do X, Y, and Z. This is how you set up a Stitch Fix app for Heroku. Like there was just the way that we wanted it to be used to sort of. Just make it all manageable. And so we were considering having a team spun up to sort of add some tooling around that to sort of make that a little bit easier for everybody. So I think there may have been something around there. I don't know if it would have been called a platform team. Maybe we call, we thought about calling it like developer happiness or because you got developer experience or something. We, we probably would have had something there, but. I do wonder how easy it would have been to fund that team with developers if we hadn't had these sort of business constraints around there. yeah, um, I don't know. You get to a certain size, you need some kind of manageability and consistency no matter what you're using underneath. So you've got to have, somebody has to own it to make sure that it's, that it's happening. [00:26:50] Jeremy: So even at your, your architect level, you still think it would have been a challenge to, to. Come to the executive team and go like, I need funding to build this team. [00:27:00] David: You know, certainly it's a challenge because everybody, you know, right? Nobody wants to put developers in anything, right? There are, there are a commodity and I mean, that is kind of the job of like, you know, the staff engineer or the architect at a company is you don't have, you don't have the power to put anybody on anything you, you have the power to Schedule a meeting with a VP or the CTO and they will listen to you. And that's basically, you've got to use that power to convince them of what you want done. And they're all reasonable people, but they're balancing 20 other priorities. So it would, I would have had to, it would have been a harder case to make that, Hey, I want to take three engineers. And have them write tooling to make Heroku easier to use. What? Heroku is not easy to use. Why aren't, you know, so you really, I would, it would be a little bit more of a stretch to walk them through it. I think a case could be made, but, definitely would take some more, more convincing than, than what was needed in our case. [00:27:53] Jeremy: Yeah. And I guess if you're able to contrast that with, you were saying, Oh, I need three people to help me make Heroku easier. Your actual platform team on AWS, I imagine was much larger, right? [00:28:03] David: Initially it was, there was, it was three people did the initial move over. And so by the time we went public, we'd been on this new system for, I don't know, six to nine months. I can't remember exactly. And so at that time the platform team was four or five people, and I, I mean, so percentage wise, right, the engineering team was maybe almost 200, 150, 200. So percentage wise, maybe a little small, I don't know. but it kind of gets back to the power of like the rails and the one person framework. Like everything we did was very much the same And so the Rails app that managed the deployment was very simple. The, the command line app, even the Go one with all of its verbosity was very, very simple. so it was pretty easy for that small team to manage. but, Yeah, so it was sort of like for redundancy, we probably needed more than three or four people because you know, somebody goes out sick or takes a vacation. That's a significant part of the team. But in terms of like just managing the complexity and building it and maintaining it, like it worked pretty well with, you know, four or five people. Where Rails fits in vs other technology [00:29:09] Jeremy: So during the Keynote today, they were talking about how companies like GitHub and Shopify and so on, they're, they're using Rails and they're, they're successful and they're fairly large. but I think the thing that was sort of unsaid was the fact that. These companies, while they use Rails, they use a lot of other, technology as well. And, and, and kind of increasing amounts as well. So, I wonder from your perspective, either from your experience at StitchFix or maybe going forward, what is the role that, that Ruby and Rails plays? Like, where does it make sense for that to be used versus like, Okay, we need to go and build something in Java or, you know, or Go, that sort of thing? [00:29:51] David: right. I mean, I think for like your standard database backed web app, it's obviously great. especially if your sort of mindset bought into server side rendering, it's going to be great at that. so like internal tools, like the customer service dashboard or... You know, something for like somebody who works at a company to use. Like, it's really great because you can go super fast. You're not going to be under a lot of performance constraints. So you kind of don't even have to think about it. Don't even have to solve it. You can, but you don't have to, where it wouldn't work, I guess, you know, if you have really strict performance. Requirements, you know, like a, a Go version of some API server is going to use like percentages of what, of what Rails would use. If that's meaningful, if what you're spending on memory or compute is, is meaningful, then, then yeah. That, that becomes worthy of consideration. I guess if you're, you know, if you're making a mobile app, you probably need to make a mobile app and use those platforms. I mean, I guess you can wrap a Rails app sort of, but you're still making, you still need to make a mobile app, that does something. yeah. And then, you know, interestingly, the data science part of Stitch Fix was not part of the engineering team. They were kind of a separate org. I think Ruby and Rails was probably the only thing they didn't use over there. Like all the ML stuff, everything is either Java or Scala or Python. They use all that stuff. And so, yeah, if you want to do AI and ML with Ruby, you, it's, it's hard cause there's just not a lot there. You really probably should use Python. It'll make your life easier. so yeah, those would be some of the considerations, I guess. [00:31:31] Jeremy: Yeah, so I guess in the case of, ML, Python, certainly, just because of the, the ecosystem, for maybe making a command line application, maybe Go, um, Go or Rust, perhaps, [00:31:44] David: Right. Cause you just get a single binary. Like the problem, I mean, I wrote this book on Ruby command line apps and the biggest problem is like, how do I get the Ruby VM to be anywhere so that it can then run my like awesome scripts? Like that's kind of a huge pain. (laughs) So [00:31:59] Jeremy: and then you said, like, if it's Very performance sensitive, which I am kind of curious in, in your experience with the companies you've worked at, when you're taking on a project like that, do you know up front where you're like, Oh, the CPU and memory usage is going to be a problem, or is it's like you build it and you're like, Oh, this isn't working. So now I know. [00:32:18] David: yeah, I mean, I, I don't have a ton of great experience there at Stitch Fix. The biggest expense the company had was the inventory. So like the, the cost of AWS was just de minimis compared to all that. So nobody ever came and said, Hey, you've got to like really save costs on, on that stuff. Cause it just didn't really matter. at the, the mental health startup I was at, it was too early. But again, the labor costs were just far, far exceeded the amount of money I was spending on, on, um, you know, compute and infrastructure and stuff like that. So, Not knowing anything, I would probably just sort of wait and see if it's a problem. But I suppose you always take into account, like, what am I actually building? And like, what does this business have to scale to, to make it worthwhile? And therefore you can kind of do a little bit of planning ahead there. But, I dunno, I think it would kind of have to depend. [00:33:07] Jeremy: There's a sort of, I guess you could call it a meme, where people say like, Oh, it's, it's not, it's not Rails that's slow, it's the, the database that's slow. And, uh, I wonder, is that, is that accurate in your experience, or, [00:33:20] David: I mean, most of the stuff that we had that was slow was the database, because like, it's really easy to write a crappy query in Rails if you're not, if you're not careful, and then it's really easy to design a database that doesn't have any indexes if you're not careful. Like, you, you kind of need to know that, But of course, those are easy to fix too, because you just add the index, especially if it's before the database gets too big where we're adding indexes is problematic. But, I think those are just easy performance mistakes to make. Uh, especially with Rails because you're not, I mean, a lot of the Rails developers at Citrix did not know SQL at all. I mean, they had to learn it eventually, but they didn't know it at all. So they're not even knowing that what they're writing could possibly be problematic. It's just, you're writing it the Rails way and it just kind of works. And at a small scale, it does. And it doesn't matter until, until one day it does. [00:34:06] Jeremy: And then in, in the context of, let's say, using ActiveRecord and instantiating the objects, or, uh, the time it takes to render templates, that kinds of things, to, at least in your experience, that wasn't such of an issue. [00:34:20] David: No, and it was always, I mean, whenever we looked at why something was slow, it was always the database and like, you know, you're iterating over some active records and then, and then, you know, you're going into there and you're just following this object graph. I've got a lot of the, a lot of the software at Stitch Fix was like internal stuff and it was visualizing complicated data out of the database. And so if you didn't think about it, you would just start dereferencing and following those relationships and you have this just massive view and like the HTML is fine. It's just that to render this div, you're. Digging into some active record super deep. and so, you know, that was usually the, the, the problems that we would see and they're usually easy enough to fix by making an index or. Sometimes you do some caching or something like that. and that solved most of the, most of the issues [00:35:09] Jeremy: The different ways people learn [00:35:09] Jeremy: so you're also the author of the book, Sustainable Web Development with Ruby on Rails. And when you talk to people about like how they learn things, a lot of them are going on YouTube, they're going on, uh, you know, looking for blogs and things like that. And so as an author, what do you think the role is of, of books now? Yeah, [00:35:29] David: I have thought about this a lot, because I, when I first got started, I'm pretty old, so books were all you had, really. Um, so they seem very normal and natural to me, but... does someone want to sit down and read a 400 page technical book? I don't know. so Dave Thomas who runs Pragmatic Bookshelf, he was on a podcast and was asked the same question and basically his answer, which is my answer, is like a long form book is where you can really lay out your thinking, really clarify what you mean, really take the time to develop sometimes nuanced, examples or nuanced takes on something that are Pretty hard to do in a short form video or in a blog post. Because the expectation is, you know, someone sends you an hour long YouTube video, you're probably not going to watch that. Two minute YouTube video is sure, but you can't, you can't get into so much, kind of nuanced detail. And so I thought that was, was right. And that was kind of my motivation for writing. I've got some thoughts. They're too detailed. It's, it's too much set up for a blog post. There's too much of a nuanced element to like, really get across. So I need to like, write more. And that means that someone's going to have to read more to kind of get to it. But hopefully it'll be, it'll be valuable. one of the sessions that we're doing later today is Ruby content creators, where it's going to be me and Noel Rappin and Dave Thomas representing the old school dudes that write books and probably a bunch of other people that do, you know, podcasts videos. It'd be interesting to see, I really want to know how do people learn stuff? Because if no one reads books to learn things, then there's not a lot of point in doing it. But if there is value, then, you know. It should be good and should be accessible to people. So, that's why I do it. But I definitely recognize maybe I'm too old and, uh, I'm not hip with the kids or, or whatever, whatever the case is. I don't know. [00:37:20] Jeremy: it's tricky because, I think it depends on where you are in the process of learning that thing. Because, let's say, you know a fair amount about the technology already. And you look at a book, in a lot of cases it's, it's sort of like taking you from nothing to something. And so you're like, well, maybe half of this isn't relevant to me, but then if I don't read it, then I'm probably missing a lot still. And so you're in this weird in be in between zone. Another thing is that a lot of times when people are trying to learn something, they have a specific problem. And, um, I guess with, with books, it's, you kind of don't know for sure if the thing you're looking for is going to be in the book. [00:38:13] David: I mean, so my, so my book, I would not say as a beginner, it's not a book to learn how to do Rails. It's like you already kind of know Rails and you want to like learn some comprehensive practices. That's what my book is for. And so sometimes people will ask me, I don't know Rails, should I get your book? And I'm like, no, you should not. but then you have the opposite thing where like the agile web development with Rails is like the beginner version. And some people are like, Oh, it's being updated for Rails 7. Should I get it? I'm like, probably not because How to go from zero to rails hasn't changed a lot in years. There's not that much that's going to be new. but, how do you know that, right? Hopefully the Table of Contents tells you. I mean, the first book I wrote with Pragmatic, they basically were like, The Table of Contents is the only thing the reader, potential reader is going to have to have any idea what's in the book. So, You need to write the table of contents with that in mind, which may not be how you'd write the subsections of a book, but since you know that it's going to serve these dual purposes of organizing the book, but also being promotional material that people can read, you've got to keep that in mind, because otherwise, how does anybody, like you said, how does anybody know what's, what's going to be in there? And they're not cheap, I mean, these books are 50 bucks sometimes, and That's a lot of money for people in the U. S. People outside the U. S. That's a ton of money. So you want to make sure that they know what they're getting and don't feel ripped off. [00:39:33] Jeremy: Yeah, I think the other challenge is, at least what I've heard, is that... When people see a video course, for whatever reason, they, they set, like, a higher value to it. They go, like, oh, this video course is, 200 dollars and it's, like, seems like a lot of money, but for some people it's, like, okay, I can do that. But then if you say, like, oh, this, this book I've been researching for five years, uh, I want to sell it for a hundred bucks, people are going to be, like no. No way., [00:40:00] David: Yeah. Right. A hundred bucks for a book. There's no way. That's a, that's a lot. Yeah. I mean, producing video, I've thought about doing video content, but it seems so labor intensive. Um, and it's kind of like, It's sort of like a performance. Like I was mentioning before we started that I used to play in bands and like, there's a lot to go into making an even mediocre performance. And so I feel like, you know, video content is the same way. So I get that it like, it does cost more to produce, but, are you getting more information out of it? I, that, I don't know, like maybe not, but who knows? I mean, people learn things in different ways. So, [00:40:35] Jeremy: It's just like this perception thing, I think. And, uh, I'm not sure why that is. Um, [00:40:40] David: Yeah, maybe it's newer, right? Maybe books feel older so they're easier to make and video seems newer. I mean, I don't know. I would love to talk to engineers who are like... young out of college, a few years into their career to see what their perception of this stuff is. Cause I mean, there was no, I mean, like I said, I read books cause that's all there was. There was no, no videos. You, you go to a conference and you read a book and that was, that was all you had. so I get it. It seems a whole video. It's fancier. It's newer. yeah, I don't know. I would love to hear a wide variety of takes on it to see what's actually the, the future, you know? [00:41:15] Jeremy: sure, yeah. I mean, I think it probably can't just be one or the other, right? Like, I think there are... Benefits of each way. Like, if you have the book, you can read it at your own pace without having to, like, scroll through the video, and you can easily copy and paste the, the code segments, [00:41:35] David: Search it. Go back and forth. [00:41:36] Jeremy: yeah, search it. So, I think there's a place for it, but yeah, I think it would be very interesting, like you said, to, to see, like, how are people learning, [00:41:45] David: Right. Right. Yeah. Well, it's the same with blogs and podcasts. Like I, a lot of podcasters I think used to be bloggers and they realized that like they can get out what they need by doing a podcast. And it's way easier because it's more conversational. You don't have to do a bunch of research. You don't have to do a bunch of editing. As long as you're semi coherent, you can just have a conversation with somebody and sort of get at some sort of thing that you want to talk about or have an opinion about. And. So you, you, you see a lot more podcasts and a lot less blogs out there because of that. So it's, that's kind of like the creators I think are kind of driving that a little bit. yeah. So I don't know. [00:42:22] Jeremy: Yeah, I mean, I can, I can say for myself, the thing about podcasts is that it's something that I can listen to while I'm doing something else. And so you sort of passively can hopefully pick something up out of that conversation, but... Like, I think it's maybe not so good at the details, right? Like, if you're talking code, you can talk about it over voice, but can you really visualize it? Yeah, yeah, yeah. I think if you sit down and you try to implement something somebody talked about, you're gonna be like, I don't know what's happening. [00:42:51] David: Yeah. [00:42:52] Jeremy: So, uh, so, so I think there's like these, these different roles I think almost for so like maybe you know the podcast is for you to Maybe get some ideas or get some familiarity with a thing and then when you're ready to go deeper You can go look at a blog post or read a book I think video kind of straddles those two where sometimes video is good if you want to just see, the general concept of a thing, and have somebody explain it to you, maybe do some visuals. that's really good. but then it can also be kind of detailed, where, especially like the people who stream their process, right, you can see them, Oh, let's, let's build this thing together. You can ask me questions, you can see how I think. I think that can be really powerful. at the same time, like you said, it can be hard to say, like, you know, I look at some of the streams and it's like, oh, this is a three hour stream and like, well, I mean, I'm interested. I'm interested, but yeah, it's hard enough for me to sit through a, uh, a three hour movie, [00:43:52] David: Well, then that, and that gets into like, I mean, we're, you know, we're at a conference and they, they're doing something a little, like, there are conference talks at this conference, but there's also like. sort of less defined activities that aren't a conference talk. And I think that could be a reaction to some of this too. It's like I could watch a conference talk on, on video. How different is that going to be than being there in person? maybe it's not that different. Maybe, maybe I don't need to like travel across the country to go. Do something that I could see on video. So there's gotta be something here that, that, that meets that need that I can't meet any other way. So it's all these different, like, I would like to think that's how it is, right? All this media all is a part to play and it's all going to kind of continue and thrive and it's not going to be like, Oh, remember books? Like maybe, but hopefully not. Hopefully it's like, like what you're saying. Like it's all kind of serving different purposes that all kind of work together. Yeah. [00:44:43] Jeremy: I hope that's the case, because, um, I don't want to have to scroll through too many videos. [00:44:48] David: Yeah. The video's not for me. Large Language Models [00:44:50] Jeremy: I, I like, I actually do find it helpful, like, like I said, for the high level thing, or just to see someone's thought process, but it's like, if you want to know a thing, and you have a short amount of time, maybe not the best, um, of course, now you have all the large language model stuff where you like, you feed the video in like, Hey, tell, tell, tell me, uh, what this video is about and give me the code snippets and all that stuff. I don't know how well it works, but it seems [00:45:14] David: It's gotta get better. Cause you go to a support site and they're like, here's how to fix your problem, and it's a video. And I'm like, can you just tell me? But I'd never thought about asking the AI to just look at the video and tell me. So yeah, it's not bad. [00:45:25] Jeremy: I think, that's probably where we're going. So it's, uh, it's a little weird to think about, but, [00:45:29] David: yeah, yeah. I was just updating, uh, you know, like I said, I try to keep the book updated when new versions of Rails come out, so I'm getting ready to update it for Rails 7. 1 and in Amazon's, Kindle Direct Publishing as their sort of backend for where you, you know, publish like a Kindle book and stuff, and so they added a new question, was AI used in the production of this thing or not? And if you answer yes, they want you to say how much, And I don't know what they're gonna do with that exactly, but I thought it was pretty interesting, cause I would be very disappointed to pay 50 for a book that the AI wrote, right? So it's good that they're asking that? Yeah. [00:46:02] Jeremy: I think the problem Amazon is facing is where people wholesale have the AI write the book, and the person either doesn't review it at all, or maybe looks at a little, a little bit. And, I mean, the, the large language model stuff is very impressive, but If you have it generate a technical book for you, it's not going to be good. [00:46:22] David: yeah. And I guess, cause cause like Amazon, I mean, think about like Amazon scale, like they're not looking at the book at all. Like I, I can go click a button and have my book available and no person's going to look at it. they might scan it or something maybe with looking for bad words. I don't know, but there's no curation process there. So I could, yeah. I could see where they could have that, that kind of problem. And like you as the, as the buyer, you don't necessarily, if you want to book on something really esoteric, there are a lot of topics I wish there was a book on that there isn't. And as someone generally want to put it on Amazon, I could see a lot of people buying it, not realizing what they're getting and feeling ripped off when it was not good. [00:47:00] Jeremy: Yeah, I mean, I, I don't know, if it's an issue with the, the technical stuff. It probably is. But I, I know they've definitely had problems where, fiction, they have people just generating hundreds, thousands of books, submitting them all, just flooding it. [00:47:13] David: Seeing what happens. [00:47:14] Jeremy: And, um, I think that's probably... That's probably the main reason why they ask you, cause they want you to say like, uh, yeah, you said it wasn't. And so now we can remove your book. [00:47:24] David: right. Right. Yeah. Yeah. [00:47:26] Jeremy: I mean, it's, it's not quite the same, but it's similar to, I don't know what Stack Overflow's policy is now, but, when the large language model stuff started getting big, they had a lot of people answering the questions that were just. Pasting the question into the model [00:47:41] David: Which because they got it from [00:47:42] Jeremy: and then [00:47:43] David: The Got model got it from Stack Overflow. [00:47:45] Jeremy: and then pasting the answer into Stack Overflow and the person is not checking it. Right. So it's like, could be right, could not be right. Um, cause, cause to me, it's like, if, if you generate it, if you generate the answer and the answer is right, and you checked it, I'm okay with that. [00:48:00] David: Yeah. Yeah. [00:48:01] Jeremy: but if you're just like, I, I need some karma, so I'm gonna, I'm gonna answer these questions with, with this bot, I mean, then maybe [00:48:08] David: I could have done that. You're not adding anything. Yeah, yeah. [00:48:11] Jeremy: it's gonna be a weird, weird world, I think. [00:48:12] David: Yeah, no kidding. No kidding. [00:48:15] Jeremy: that's a, a good place to end it on, but is there anything else you want to mention, [00:48:19] David: No, I think we covered it all just yeah, you could find me online. I'm Davetron5000 on Ruby. social Mastodon, I occasionally post on Twitter, but not that much anymore. So Mastodon's a place to go. [00:48:31] Jeremy: David, thank you so much [00:48:32] David: All right. Well, thanks for having me.

Mike Perham @Sidekiq. Bootstrapping a SaaS to $7M solo

saas.unbound

Play Episode Listen Later Nov 16, 2023 42:15

saas.unbound is a podcast for and about founders who are working on scaling inspiring products that people love brought to you by https://saas.group/ . I'm your host Anna Nadeina, Head of Growth for saas.group. In this episode #2a we talk with Mike Perham, founder and CEO at Sidekiq (https://sidekiq.org/), a full-featured background processing framework for Ruby. Mike unveils the concept behind Sidekiq explaining how this tool allows developers to build and scale Ruby applications. He explains how Sidekiq works by creating jobs to process data and distribute them across many machines. Listen as Mike discusses the synergy between the open-source project and the commercial aspect of Sidekiq, contributing to its ongoing development and success. Get a look into the process of building dependable software and how to monetize an open-source project as Mike shares his journey. Discover the benefits of an open-core model and learn about Mike's strategies for building the project, finding contributors, and pricing his product.Subscribe to our channel to be the first to see the interviews that we publish twice a week - https://www.youtube.com/@saas-group?sub_confirmation=1 Stay up to date: LinkedIn - https://www.linkedin.com/company/14790796 Twitter - https://twitter.com/SaaS_group Website - https://saas.group/

ceo head growth discover solo saas bootstrapping sidekiq mike perham

Season 11 Kickoff: The Hosts Discuss Branching Out from Elixir to Compare Notes

code literal way down abstraction jd davis sidekiq

Play Episode Listen Later Oct 19, 2023 32:21

Hosts Dan Ivovich, Owen Bickford, and Sundi Myint kick off the 11th season of the Elixir Wizards podcast. This season's theme is “Branching Out from Elixir,” which expands the conversation to compare notes with experts from other communities; they discuss their experiences with other languages like JavaScript, PHP, Python, Ruby, C#, Go, and Dart before and after learning Elixir. This season's conversations will illuminate how problems are solved in different languages vs. Elixir; upcoming episode topics teased include education, data processing, deployment strategies, and garbage collection; the hosts express excitement for conversations analyzing similarities and differences between communities. Topics Discussed in this Episode Season 11 branches out from Elixir to compare notes with other programming communities Sundi, Owen, and Dan introduce the season theme and their interest in exploring these conversations The hosts compare their experiences with PHP, JavaScript, Python, Ruby, C#, Go, Dart and Elixir The Wizards compare and contrast differences in their personal experience building similar things with different languages Dan dreams in Ruby and uses it for quick prototypes Comparing problem-solving approaches across languages will reframe perspectives Upcoming episodes explore data processing workflows, machine learning, and game development Pop Quiz: Who's that Pokémon... or language, or framework? Links Mentioned https://smartlogic.io/ https://codepen.io/ https://i.redd.it/0lg7979qtr511.jpg

development model budget testing comparison hosts views integration kickoff models pok programming react releases compare vault python qa dart devops javascript rails timelines deployment documentation php estimates pipelines elixir kubernetes nix controllers vue hiring process ux design angular branching celery bootcamps routing terraform mvc app development jquery flask user research etl redis ansible .net wireframes data processing liveview mqtt tailwind css mockups rabbitmq c sharp code quality product ownership branching out sidekiq event storming technical product management release planning

22: Abstraction Goes All the Way Down

Rooftop Ruby Podcast

Play Episode Listen Later Aug 9, 2023 26:14 Transcription Available

We talk about CODE by Charles Petzold, which is great and everyone should read.Collin is done ranking his hacks and is doing some comfort C programming. Meanwhile Joel has finished implementing Literal in a real project and is figuring out how to use Sidekiq across processes.Follow us on Mastodon: Rooftop Ruby Collin Joel Show art created by JD Davis.

385: The Boring Parts of Tech

Startups For the Rest of Us

Play Episode Listen Later May 23, 2023 24:41

Joël is joined by thoughtbot Software Developer and Dirt Jumper Daniel Nolan. Dirt jumping is BMX-style riding

head google pr tech boring upgrade dirt optimizing github vw bmx rails prs sdks software developers mandy moore honey badger sentry apm debugging refactoring new relic coverband quenneville sidekiq daniel for daniel it daniel you daniel so daniel well daniel there

Episode 661 | Millions in Revenue As a One-Person Software Company

Play Episode Listen Later May 16, 2023 27:57

In episode 661, Rob Walling chats with Mike Perham, the founder of Sidekiq, who is a solo founder doing millions in revenue as a one-person business.If this isn't unique enough, Sidekiq originally started as an open-source project before he later monetized it by selling features that aren't available in the core product. You'll also hear how it took him ten years to become "an overnight success" because of all the things Mike tried before launching Sidekiq. Episode Sponsor: Find your perfect developer or a team at Lemon.io/startupsThe competition for incredible engineers and developers has never been more fierce. Lemon.io helps you cut through the noise and find great talent through its network of engineers in Europe and Latin America.They take care of the vetting, interviewing, and testing of candidates to make sure that you are working with someone who can hit the ground running.When it comes to hiring, the time it takes to write your job description, list...Read more... »Click the icon below to listen.

europe millions latin america revenue one person software company rob walling sidekiq startupsthe mike perham

Victor Adossi on Yak Shaving

Software Sessions

Play Episode Listen Later Jan 2, 2023 110:47

Victor is a software consultant in Tokyo who describes himself as a yak shaver. He writes on his blog at vadosware and curates Awesome F/OSS, a mailing list of open source products. He's also a contributor to the Open Core Ventures blog. Before our conversation Victor wrote a structured summary of how he works on projects. I recommend checking that out in addition to the episode. Topics covered: Most people should use Dokku or CapRover But he uses Kubernetes anyways Hosting a Database in Kubernetes Learning technology You don't really know a thing until something goes wrong History of Frontend Development Context from lower layers of the stack and historical projects Good project pages have comparisons to other products Choosing technologies Language choice affects maintainability Knowing an ecosystem Victor's preferred stack Technology bake offs Posting findings means you get free corrections Why people use medium instead of personal sites Victor VADOSWARE - Blog How Victor works on Projects - Companion post for this episode Awesome FOSS - Curated list of OSS projects NimbusWS - Hosted OSS built on top of budget cloud providers Unvalidated Ideas - Startup ideas for side project inspiration PodcastSaver - Podcast index that allows you to choose Postgres or MeiliSearch and compare performance and results of each Victor's preferred stack Docker - Containers Kubernetes - Container provisioning (Though at the beginning of the episode he suggests Dokku for single server or CapRover for multiple) TypeScript - JavaScript with syntax for types. Victor's default choice. Rust - Language he uses if doing embedded work, performance is critical, or more correctness is desired Haskell - Language he uses if correctness and type system is the most important for the project Postgresql - General purpose database that's good enough for most use cases including full text search. KeyDB - Redis compatible database for caching. Acquired by Snap and then made open source. Victor uses it over Redis because it is multi threaded and supports flash storage without a Redis Enterprise license. Pulumi - Provision infrastructure with the languages you're already using instead of a specialized one or YAML Svelte and SvelteKit - Preferred frontend stack. Previously used Nuxt. Search engines Postgres Full Text Search vs the rest Optimizing Postgres Text Search with Trigrams OpenSearch - Amazon's fork of Elasticsearch typesense meilisearch sonic Quickwit JavaScript build tools Babel SWC Webpack esbuild parcel Vite Turbopack JavaScript frameworks React Vue Svelte Ember Frameworks built on top of frameworks Next - React Nuxt - Vue SvelteKit - Svelte Astro - Multiple Historical JavaScript tools and frameworks Underscore jQuery MooTools Backbone AngularJS Knockout Aurelia GWT Bower - Frontend package manager Grunt - Task runner Gulp - Task runner Related Links Dokku - Open source single-host alternative to Heroku Cloud Native Buildpacks - Buildpacks created by Heroku and Pivotal and used by Dokku CapRover - An open source PaaS-like abstraction built on top of Docker Swarm Kelsey Hightower's tweet about being cautious about running databases on Kubernetes Settling the Myth of Transparent HugePages for Databases Kubernetes Container Storage Interface (CSI) Kubernetes Local Persistent Volumes Longhorn - Distributed block storage for Kubernetes Postgres docs Postgres TOAST Everything I've seen on optimizing Postgres on ZFS Kubernetes Workload Resources Kubernetes Network Plugins Kubernetes Ingress Traefik Kubernetes the Hard Way (Setting up a cluster in a way that optimizes for learning) How does TLS work Let's Encrypt Cert manager for Kubernetes Choose Boring Technology A Linux user's guide to Logical Volume Management Docker networking overview Kubernetes Scheduler Tauri - Build desktop applications with web technology and Rust ripgrep - CLI tool to recursively search directory for a regex pattern (Meant to be a rust replacement for grep) angle-grinder / ag - CLI tool to parse and process log files written in rust Object.observe ECMAScript Proposal to be Withdrawn Ruby on Rails - Ruby web framework Django - Python web framework Laravel - PHP web framework Adonis - JavaScript NestJS - JavaScript What is a NullPointerException, and how do I fix it? Mastodon Clap - CLI argument parser for Rust AWS CDK - Provision AWS infrastructure using programming languages Terraform - Provision infrastructure with terraform language URL canonicalization of duplicate pages and the use of the canonical tag - Used by dev.to to send google traffic to the original blogpost instead of dev.to Transcript You can help edit this transcript on GitHub. [00:00:00] Jeremy: This episode, I talk to Victor Adossi who describes himself as a yak shaver. Someone who likes trying a whole bunch of different technologies, seeing the different options. We talk about what he uses, the evolution of front end development, and his various projects. Talking to just different people it's always good to get where they're coming from because something that works for Google at their scale is going to be different than what you're doing with one of your smaller projects. [00:00:31] Victor: Yeah, the context. Of course in direct conflict with that statement, I definitely use Google technology despite not needing to at all right? Like, you know, 99% of people who are doing like people like to call it indiehacking or building small products could probably get by with just Dokku. If you know Dokku or like CapRover. Are two projects that'll be like, Oh, you can just push your code here, we'll build it up like a little mini Heroku PaaS thing and just go on one big server, right? Like 99% of the people could just use that. But of course I'm not doing that. So I'm a bit of a hypocrite in that sense. I know what I should be doing, but I'm not doing that. I am writing a Kubernetes cluster with like five nodes for no reason. Uh, yeah, I dunno, people don't normally count the controllers. [00:01:24] Jeremy: Dokku and CapRover, I think those are where it's supposed to create a heroku like experience I think it's based off of the heroku buildpacks right? At least Dokku is? [00:01:36] Victor: Yeah Buildpacks has actually been spun out into like a community thing so like pivotal and heroku, it's like buildpacks.io, they're trying to build a wider standard around it so that more people can get involved. And buildpacks are actually obviously fantastic as a technology and as a a process piece. There's not much else like them and you know, that's obvious from like Heroku's success and everything. I know Dokku uses that. I don't know that Caprover does, but I haven't, I haven't really run Caprover that much. They, they probably do. Like at this point if you're going to support building from code, it seems silly to try and build your own buildpacks. Cause that's what you will do, eventually. So you might as well use what's there. Anyway, this is like just getting to like my personal opinions at this point, but like, if you think containers are a bad idea in 2022, You're wrong, you should, you should stop. Like you should, you should stop. Think about it. I mean, obviously there's not, um, I got a really great question at an interview once, which is, where are containers a bad idea? That's probably one of the best like recent interview questions I've ever gotten cause I was like, Oh yeah, I mean, like, you can't, it can't be perfect everywhere, right? Nothing's perfect everywhere. So it's like, where is it? Uh, and of course the answer was networking, right? (unintelligible) So if you need absolute performance, but like for just about everything else. Containers are kind of it at this point. Like, time has born it out, I think. So yeah, I always just like bias at taking containers at this point. So I'm probably more of a CapRover person than a Dokku person, even though I have not used, I don't use CapRover. [00:03:09] Jeremy: Well, like something that I've heard with containers, and maybe it's changed recently, but, but something that was kind of holdout was when people would host a database sometimes they would oh we just don't wanna put this in a container and I wonder if like that matches with your thinking or if things have changed. [00:03:27] Victor: I am not a database administrator right like I read postgres docs and I read the, uh, the Postgres documentation, and I think I know a bit about postgres but I don't commit right like so and I also haven't, like, oh, managed X terabytes on one server that you are making sure never goes down kind of deal. But the stickiness for me, at least from when I've run, So I've done a lot of tests with like ZFS and Postgres and like, um, and also like just trying to figure out, and I run Postgres in Kubernetes of course, like on my cluster and a lot of the stuff I found around is, is like fiddly kernel things like sort of base kernel settings that you need to have set. Like, you know, stuff like should you be using transparent huge pages, like stuff like that. But once you have that settled. Containers are just processes with name spacing and resource control, right? Like, that's it. there are some other ins and outs, but for the most part, if you're fine running a process, so people ran processes, right? And they were just completely like unprotected. Then people made users for the processes and they limited the users and ran the processes, right? Then the next step is now you can run a process and then do the limiting the name spaces in cgroups dynamically. Like there, there's, there's sort of not a humongous difference, unless you're hitting something very specific. Uh, but yeah, databases have been a point of contention, but I think, Kelsey Hightower had that tweet yeah. That was like, um, don't run databases in Kubernetes. And I think he called it back. [00:04:56] Victor: I don't know, but I, I know that was uh, was one of those things that people were really unsure about at first, but then after people sort of like felt it out, they were like, Oh, it's actually fine. Yeah. [00:05:06] Jeremy: Yeah I vaguely remember one of the concerns having to do with persistent storage. Like there were challenges with Kubernetes and needing to keep that storage around and I don't know if that's changed yeah or if that's still a concern. [00:05:18] Victor: Uh, I'd say that definitely has changed. Uh, and it was, it was a concern, depending on where you were. Mostly people who are running AKS or EKS or you know, all those other managed Kubernetes, they're just using EBS or like whatever storage provider is like offering for storage. Most of those people don't actually have that much of a problem with, storage in general. Now, high performance storage is obviously different, right? So like, so you'll, you're gonna have to start doing manual, like local volume management and stuff like that. it was a problem, because obviously CSI (Kubernetes Container Storage Interface) didn't exist for some period of time, and like there was, it was hard to know what to do for if you were just running a Kubernetes cluster. I think a lot of people were just using local, first of all, local didn't even exist for a bit. Um, they were just using host path, right? And just like, Oh, it's on the disk somewhere. Where do we, we have to go get it right? Or we have to like, sort of manage that. So that was something most people weren't ready for, especially if you were just, if you weren't like sort of a, a, a traditional sysadmin and used to doing that stuff. And then of course local volumes came out, but I think they still had to be, um, pre-provisioned. So that's sysadmin stuff that most people, you know, maybe aren't, aren't necessarily ready for. Uh, and then most of the general solutions were slow. So like, I used Longhorn (https://longhorn.io) for a long time and Longhorn, Longhorn's great. And super easy to set up, but it can be slower and you can have some, like, delays in mount time. it wasn't ideal for, for most people. So yeah, I, overall it's true. Databases, Databases in Kubernetes were kind of fraught with peril for a while, but it wasn't for the reason that, it wasn't for the fundamental reason that Kubernetes was just wrong or like, it wasn't the reason most people think of, which is just like, Oh, you're gonna break your database. It's more like, running a database is hard and Kubernetes hasn't solved all the hard problems. Like, cuz that's what Kubernetes does. It basically solves a lot of problems in a very generic way. Right. So it just hadn't solved all those problems yet at this point. I think it's got decent answers on a lot of them. So I, I mean, I don't know. I I do it. Don't, don't take what I'm saying to your, you know, PM meeting or your standup meeting, uh, anyone who's listening. But it's more like if you could solve the problems with databases in the sense before. You could probably solve 'em on Kubernetes now with a good understanding of Kubernetes. Cause at the end of the day, it's all the same stuff. Just Kubernetes makes it a little easier to, uh, do it dynamically. [00:07:50] Jeremy: It sounds like you could do it before, but some of the, I guess the tools or the ways of doing persistent storage were not quite there yet, or they were difficult to use. And so that was why people at the start were like, Okay, maybe it's not a good idea, but, now maybe there's some established practices for how you should run a database in Kubernetes. And I, I suppose the other aspect too is that, like you were saying, Kubernetes is its own thing. You gotta learn Kubernetes and all its intricacies. And then running a database is also its own challenge. So if you stack the two of them together and, and the path was not really clear then maybe at the start it wasn't the best idea. Um, uh, if somebody was going to try it out now, was there like a specific resource you looked at or a specific path to where like okay this is is how I'm going to do it. [00:08:55] Victor: I'll just say what I normally recommend to everybody. Cause it depends on which path you wanna go right? If you wanna go down like running a database path first and figure that out, fill out that skill tree. Like go read the Postgres docs. Well, first of all, use Postgres. That's the first tip there. But like, read those documents. And obviously you don't have to understand everything. You won't understand everything. But knowing the big pieces and sort of letting your brain see the mention of like a whole bunch of things, like what is toast? Oh, you can do compression on columns. Like, you can do some, some things concurrently. Um, you know, what ALTER TABLE looks like. You get all that stuff kind of in your head. Um, and then I personally really believe in sort of learning by building and just like iterating. you won't get it right the first time. It's just like, it's not gonna happen. You're get, you can, you can get better the first time, right? By being really prepared and like, and leave yourself lots of outs, but you kind of have to like, get it out there. Do do your best to make sure that you can't fail, uh, catastrophically, right? So this is like, goes back to that decision to like use ZFS as the bottom of this I'm just like, All right, well, I, I'm not a file systems expert, but if I. I could delegate some of that, you know, some of that, I can get some of that knowledge from someone else. Um, and I can make it easier for me to not fail catastrophically. For the database side, actually read documentation on Postgres or the whatever database you're going to use, make sure you at least understand that. Then start running it like locally or whatever. Again, Docker use, use Docker locally. It's, it's, it's fine. and then, you know, sort of graduate to running sort of more progressively, more complicated versions. what I would say for the Kubernetes side is actually similar. the Kubernetes docs are really good. they're very large. but they're good. So you can actually go through and know all the, like, workload, workload resources, know, like what a config map is, what a secret is, right? Like what etcd is doing in this whole situation. you know, what a kublet is versus an API server, right? Like the, the general stuff, like if you go through all that, you should have like a whole bunch of ideas at least floating around in your head. And then once you try and start setting up a server, they will all start to pop up again, right? And they'll all start to like, you, like, Oh, okay, I need a CNI (Container Networking) plugin because something needs to make the services available, right? Or something needs to power the ingress, right? Like, if I wanna be able to get traffic, I need an ingress object. But what listens, what does that, what makes that ingress object do anything? Oh, it's an ingress controller. nginx, you know, almost everyone's heard of nginx, so they're like, okay. Um, nginx, has an ingress control. Actually there's, there used to be two, I assume there's still two, but there's like one that's maintained by Kubernetes, one that's maintained by nginx, the company or whatever. I use traefik, it's fantastic. but yeah, so I think those things kind of fall out and that is almost always my first way to explain it and to start building. And tinkering iteratively. So like, read the documentation, get a good first grasp of it, and then start building yourself because you'll, you'll get way more questions that way. Like, you'll ask way more questions, you won't be able to make progress. Uh, and then of course you can, you know, hop into slacks or like start looking around and, and searching on the internet. oh, one of the things that really helped me out early learning Kubernetes was, Kelsey Hightower's, um, learn Kubernetes the hard way. I'm also a big believer in doing things the hard way, at least knowing what you're choosing to not know, right? distributing file system, Deltas, right? Or like changes to a file system over the network is not a new problem. Other people have solved it. There's a lot of complexity there. but if you at least know the sort of surface level of what the thing does and what it's supposed to do and how it's supposed to do it, you can make a decision on, Oh, how deep am I going to go? Right? To prevent yourself from like, making a mistake or going too deep in the rabbit hole. If you have an idea of the sort of ecosystem and especially like, Oh, here, like the basics of how I can use this thing, that's generally very good. And doing things the hard way is a great way to get a, a feel for that, right? Cause if you take some chunk and like, you know, the first level of doing things the hard way, uh, or, you know, Kelsey Hightower's guide is like, get a machine, right? Like, so, like, if you somehow were like, Oh, I wanna run a Kubernetes cluster. but, you know, I don't want use necessarily EKS and you wanna learn it the hard way. You have to go get a machine, right? If you, if you're not familiar, if you run on Heroku the whole time, like you didn't manage your own machines, you gotta go like, figure out EC2, right? Or, I personally use, hetzner I love hetzner, so you have to go figure out hetzner, digital ocean, whatever. Right. And then the next thing's like, you know, the guide's changed a lot, and I haven't, I haven't looked at it in like, in years, actually a while since I, since I've sort of been, I guess living it, but it's, it's like generate certificates, right? So if you've never dealt with SSL and like, sort of like, or I should say TLS uh, and generating certificates and how that whole dance works, right? Which is fascinating because it's like, oh, right, nothing's secure on the internet, except that we distribute root certificates on computers that are deployed in every OS, right? Like, that's a sort of fundamental understanding you may not go deep enough to realize, but if you are fascinated by it, trying to do it manually would lead you down that path. You'd be like, Oh, what, like what is this thing? What is a CSR? Like, why, who is signing my request? Right? And it's like, why do we trust those people? Right? And it's like, you know, that kind of thing comes out and I feel like you can only get there from trying to do it, you know, answering the questions you can. Right. And again, it takes some judgment to know when you should not go down a rabbit hole. uh, and then iterating. of course there are people who are excellent at explaining. you can find some resources that are shortcuts. But, uh, I think particularly my bread and butter has been just to try and do it the hard way. Avoid pitfalls or like rabbit holes when you can. But know that the rabbit hole is there, and then keep going. And sometimes if something's just too hard, you're not gonna get it the first time. Like maybe you'll have to wait like another three months, you'll try again and you'll know more sort of ambiently about everything else. You get a little further that time. that's how I feel about that. Anyway. [00:15:06] Jeremy: That makes sense to me. I think sometimes when people take on a project, they try to learn too many things at the same time. I, I think the example of Kubernetes and Postgres is pretty good example, where if you're not familiar with how do I install Postgres on bare metal or a vm, trying to make sense of that while you're trying to into is probably gonna be pretty difficult. So, so splitting them up and learning them individually, that makes a lot of sense to me. And the whole deciding how deep you wanna go. That's interesting too, because I think that's very specific to the person right because sometimes you wanna go a little deeper because otherwise you don't understand how the two things connect together. But other times it's just like with the example with certificates, some people they may go like, I just put in let's encrypt it gives me my cert I don't care right then, and then, and some people they wanna know like okay how does the whole certificate infrastructure work which I think is interesting, depending on who you are, maybe you go ahh maybe it doesn't really matter right. [00:16:23] Victor: Yeah, and, you know, shout out to Let's Encrypt . It's, it's amazing, right? think Singlehandedly the most, most of the deployment of HTTPS that happens these days, right? so many so many of like internet providers and uh, sort of service providers will use it right? Under the covers. Like, Hey, we've got you free SSL through Let's Encrypt, right? Like, kind of like under the, under the covers. which is awesome. And they, and they do it. So if you're listening to this, donate to them. I've done it. So now that, now the pressure is on whoever's listening, but yeah, and, and I, I wanna say I am that person as well, right? Like, I use, Cert Manager on my cluster, right? So I'm just like, I don't wanna think about it, but I, you know, but I, I feel like I thought about it one time. I have a decent grasp. If something changes, then I guess I have to dive back in. I think it, you've heard the, um, innovation tokens idea, right? I can't remember the site. It's like, um, do, like do boring tech or something.com (https://boringtechnology.club/) . Like it shows up on sort of hacker news from time to time, essentially. But it's like, you know, you have a certain amount of tokens and sort of, uh, we'll call them tokens, but tolerance for complexity or tolerance for new, new ideas or new ways of doing things, new processes. Uh, and you spend those as you build any project, right? you can be devastatingly effective by just sticking to the stack, you know, and not introducing anything new, even if it's bad, right? and there's nothing wrong with LAMP stack, I don't wanna annoy anybody, but like if you, if you're running LAMP or if you run on a hostgator, right? Like, if you run on so, you know, some, some service that's really old but really works for you isn't, you know, too terribly insecure or like, has the features you need, don't learn Kubernetes then, right? Especially if you wanna go fast. cuz you, you're spending tokens, right? You're spending, essentially brain power, right? On learning whatever other thing. So, but yeah, like going back to that, databases versus databases on Kubernetes thing, you should probably know one of those before you, like, if you're gonna do that, do that thing. You either know Kubernetes and you like, at least feel comfortable, you know, knowing Kubernetes extremely difficult obviously, but you feel comfortable and you feel like you can debug. Little bit of a tangent, but maybe that's even a better, sort of watermark if you know how to debug a thing. If, if it's gone wrong, maybe one or five or 10 or 20 times and you've gotten out. Not without documentation, of course, cuz well, if you did, you're superhuman. But, um, but you've been able to sort of feel your way out, right? Like, Oh, this has gone wrong and you have enough of a model of the system in your head to be like, these are the three places that maybe have something wrong with them. Uh, and then like, oh, and then of course it's just like, you know, a mad dash to kind of like, find, find the thing that's wrong. You should have confidence about probably one of those things before you try and do both when it's like, you know, complex things like databases and distributed systems management, uh, and orchestration. [00:19:18] Jeremy: That's, that's so true in, in terms of you are comfortable enough being able to debug a problem because it's, I think when you are learning about something, a lot of times you start with some kind of guide or some kind of tutorial and you follow the steps. And if it all works, then great. Right? But I think it's such a large leap from that to something went wrong and I have to figure it out. Right. Whether it's something's not right in my Dockerfile or my postgres instance uh, the queries are timing out. so many things that could go wrong, that is the moment where you're forced to figure out, okay, what do I really know about this not thing? [00:20:10] Victor: Exactly. Yeah. Like the, the rubber's hitting the road it's uh you know the car's about to crash or has already crashed like if I open the bonnet, do I know what's happening right or am I just looking at (unintelligible). And that's, it's, I feel sort a little sorry or sad for, for devs that start today because there's so much. Complexity that's been built up. And a lot of it has a point, but you need to kind of have seen the before to understand the point, right? So I like, I like to use front end as an example, right? Like the front end ecosystem is crazy, and it has been crazy for a very long time, but the steps are actually usually logical, right? Like, so like you start with, you know, HTML, CSS and JavaScript, just plain, right? And like, and you can actually go in lots of directions. Like HTML has its own thing. CSS has its own sort of evolution sort of thing. But if we look at JavaScript, you're like, you're just writing JavaScript on every page, right? And like, just like putting in script tags and putting in whatever, and it's, you get spaghetti, you get spaghetti, you start like writing, copying the same function on multiple pages, right? You just, it, it's not good. So then people, people make jquery, right? And now, now you've got like a, a bundled set of like good, good defaults that you can, you can go for, right? And then like, you know, libraries like underscore come out for like, sort of like not dom related stuff that you do want, you do want everywhere. and then people go from there and they go to like backbone or whatever. it's because Jquery sort of also becomes spaghetti at some point and it becomes hard to manage and people are like, Okay, we need to sort of like encapsulate this stuff somehow, right? And like the new tools or whatever is around at the same timeframe. And you, you, you like backbone views for example. and you have people who are kind of like, ah, but that's not really good. It's getting kind of slow. Uh, and then you have, MVC stuff comes out, right? Like Angular comes out and it's like, okay, we're, we're gonna do this thing called dirty checking, and it's gonna be, it's gonna be faster and it's gonna be like, it's gonna be less sort of spaghetti and it's like a little bit more structured. And now you have sort of like the rails paradigm, but on the front end, and it takes people to get a while to get adjusted to that, but then that gets too heavy, right? And then dirty checking is realized to be a mistake. And then, you get stuff like MVVM, right? So you get knockout, like knockout js and you got like Durandal, and like some, some other like sort of front end technologies that come up to address that problem. Uh, and then after that, like, you know, it just keeps going, right? Like, and if you come in at the very end, you're just like, What is happening? Right? Like if it, if it, if someone doesn't sort of boil down the complexity and reduce it a little bit, you, you're just like, why, why do we do this like this? Right? and sometimes there's no good reason. Sometimes the complexity is just like, is unnecessary, but having the steps helps you explain it, uh, or helps you understand how you got there. and, and so I feel like that is something younger people or, or newer devs don't necessarily get a chance to see. Cause it just, it would take, it would take very long right? And if you're like a new dev, let's say you jumped into like a coding bootcamp. I mean, I've got opinions on coding boot camps, but you know, it's just like, let's say you jumped into one and you, you came out, you, you made it. It's just, there's too much to know. sure, you could probably do like HTML in one month. Well, okay, let's say like two weeks or whatever, right? If you were, if you're literally brand new, two weeks of like concerted effort almost, you know, class level, you know, work days right on, on html, you're probably decently comfortable with it. Very comfortable. CSS, a little harder because this is where things get hard. Cause if you, if you give two weeks for, for HTML, CSS is harder than HTML kind of, right? Because the interactions are way more varied. Right? Like, and, and maybe it's one of those things where you just, like, you, you get somewhat comfortable and then just like know that in the future you're gonna see something you don't understand and have to figure it out. Uh, but then JavaScript, like, how many months do you give JavaScript? Because if you go through that first like, sort of progression that I, I I, I, I mentioned everyone would have a perfect sort of, not perfect but good understanding of the pieces, right? Like, why did we start transpiling at all? Right? Like, uh, or why did you know, why did we adopt libraries? Like why did Bower exist? No one talks about Bower anymore, obviously, but like, Bower was like a way to distribute front end only packages, right? Um, what is it? Um, Uh, yes, there's grunt. There's like the whole build system thing, right? Once, once we decide we're gonna, we're gonna do stuff to files before we, before we push. So there's grunt, there's, uh, gulp, which is like grunt, but like, Oh, we're gonna do it all in memory. We're gonna pipe, we're gonna use this pipes thing to make sure everything goes fast. then there's like, of course that leads like the insanity that's webpack. And then there's like parcel, which did better. There's vite there's like, there's all this, there's this progression, but how many months would it take to know that progression? It, it's too long. So they end up just like, Hey, you're gonna learn react. Which is the right thing because it's like, that's what people hire for, right? But then you're gonna be in react and be like, What's webpack, right? And it's like, but you can't go down. You can't, you don't have the time. You, you can't sort of approach that problem from the other direction where you, which would give you better understanding cause you just don't have the time. I think it's hard for newer devs to overcome this. Um, but I think there are some, there's some hope on the horizon cuz some things are simpler, right? Like some projects do reduce complexity, like, by watching another project sort of innovate so like react. Wasn't the first component, first framework, right? Like technically, I, I think, I think you, you might have to give that to like, to maybe backbone because like they had views and like marionette also went with that. Like maybe, I don't know, someone, someone I'm sure will get in like, send me an angry email, uh, cuz I forgot you Moo tools or like, you know, Ember Ember. They've also, they've also been around, I used to be a huge Ember fan, still, still kind of am, but I don't use it. but if you have these, if you have these tools, right? Like people aren't gonna know how to use them and Vue was able to realize that React had some inefficiencies, right? So React innovates the sort of component. So Reintroduces the component based model component first, uh, front end development model. Vue sees that and it's like, wait a second, if we just export this like data object, and of course that's not the only innovation of Vue, but if we just export this data object, you don't have to do this fine grained tracking yourself anymore, right? You don't have to tell React or tell your the system which things change when other things change, right? Like you, you don't have to set up this watching and stuff, right? Um, and that's one of the reasons, like Vue is just, I, I, I remember picking up Vue and being like, Oh, I'm done. I'm done with React now. Because it just doesn't make sense to use React because they Vue essentially either, you know, you could just say they learned from them or they, they realize a better way to do things that is simpler and it's much easier to write. Uh, and you know, functionally similar, right? Um, similar enough that it's just like, oh they boil down some of that complexity and we're a step forward and, you know, in other ways, I think. Uh, so that's, that's awesome. Every once in a while you get like a compression in the complexity and then it starts to ramp up again and you get maybe another compression. So like joining the projects that do a compression. Or like starting to adopting those is really, can be really awesome. So there's, there's like, there's some hope, right? Cause sometimes there is a compression in that complexity and you you might be lucky enough to, to use that instead of, the thing that's really complex after years of building on it. [00:27:53] Jeremy: I think you're talking about newer developers having a tough time making sense of the current frameworks but the example you gave of somebody starting from HTML and JavaScript going to jquery backbone through the whole chain, that that's just by nature of you've put in a lot of time right you've done a lot of work working with each of these technologies you see the progression as if someone is starting new just by nature of you being new you won't have been able to spend that time [00:28:28] Victor: Do you think it could work? again, the, the, the time aspect is like really hard to get like how can you just avoid spending time um to to learn things that's like a general problem I think that problem is called education in the general sense. But like, does it make sense for a, let's say a bootcamp or, or any, you know, school right? To attempt to guide people through the previous solutions that didn't work, right? Like in math, you don't start with calculus, right? It just wouldn't, it doesn't make sense, right? But we try and start with calculus in software, right? We're just like, okay, here's the complexity. You've got all of it. Don't worry. Just look at this little bit. If, you know, if the compiler ever spits out a weird error uh oh, like, you're, you're, you're in for trouble cuz you, you just didn't get the. get the basics. And I think that's maybe some of what is missing. And the thing is, it is like the constraints are hard, right? No one has infinite time, right? Or like, you know, even like, just tons of time to devote to learning, learning just front end, right? That's not even all of computing, That's not even the algorithm stuff that some companies love to throw at you, right? Uh, or the computer sciencey stuff. I wonder if it makes more sense to spend some time taking people through the progression, right? Because discovering that we should do things via components, let's say, or, or at least encapsulate our functionality to components and compose that way, is something we, we not everyone knew, right? Or, you know, we didn't know wild widely. And so it feels like it might make sense to touch on that sort of realization and sort of guide the student through, you know, maybe it's like make five projects in a week and you just get progressively more complex. But then again, that's also hard cause effort, right? It's just like, it's a hard problem. But, but I think right now, uh, people who come in at the end and sort of like see a bunch of complexity and just don't know why it's there, right? Like, if you've like, sort of like, this is, this applies also very, this applies to general, but it applies very well to the Kubernetes problem as well. Like if you've never managed nginx on more than one machine, or if you've never tried to set up a, like a, to format your file system on the machine you just rented because it just, you know, comes with nothing, right? Or like, maybe, maybe some stuff was installed, but, you know, if you had to like install LVM (Logical Volume Manager) yourself, if you've never done any of that, Kubernetes would be harder to understand. It's just like, it's gonna be hard to understand. overlay networks are hard for everyone to understand, uh, except for network people who like really know networking stuff. I think it would be better. But unfortunately, it takes a lot of time for people to take a sort of more iterative approach to, to learning. I try and write blog posts in this way sometimes, but it's really hard. And so like, I'll often have like an idea, like, so I call these, or I think of these as like onion, onion style posts, right? Where you either build up an onion sort of from the inside and kind of like go out and like add more and more layers or whatever. Or you can, you can go from the outside and sort of take off like layers. Like, oh, uh, Kubernetes has a scheduler. Why do they need a scheduler? Like, and like, you know, kind of like, go, go down. but I think that might be one of the best ways to learn, but it just takes time. Or geniuses and geniuses who are good at two things, right? Good at the actual technology and good at teaching. Cuz teaching is a skill and it's very hard. and, you know, shout out to teachers cuz that's, it's, it's very difficult, extremely frustrating. it's hard to find determinism in, in like methods and solutions. And there's research of course, but it's like, yeah, that's, that's a lot harder than the computer being like, Nope, that doesn't work. Right? Like, if you can't, if you can't, like if you, if the function call doesn't work, it doesn't work. Right. If the person learned suboptimally, you won't know Right. Until like 10 years down the road when, when they can't answer some question or like, you know, when they, they don't understand. It's a missing fundamental piece anyway. [00:32:24] Jeremy: I think with the example of front end, maybe you don't have time to walk through the whole history of every single library and framework that came but I think at the very least, if you show someone, or you teach someone how to work with css, and you have them, like you were talking about components before you have them build a site where there's a lot of stuff that gets reused, right? Maybe you have five pages and they all have the same nav bar. [00:33:02] Victor: Yeah, you kind of like make them do it. [00:33:04] Jeremy: Yeah. You make 'em do it and they make all the HTML files, they copy and paste it, and probably your students are thinking like, ah, this, this kind of sucks [00:33:16] Victor: Yeah [00:33:18] Jeremy: And yeah, so then you, you come to that realization, and then after you've done that, then you can bring in, okay, this is why we have components. And similarly you brought up, manual dom manipulation with jQuery and things like that. I, I'm sure you could come up with an example of you don't even necessarily need to use jQuery. I think people can probably skip that step and just use the the, the API that comes with the browser. But you can have them go in like, Oh, you gotta find this element by the id and you gotta change this based on this, and let them experience the. I don't know if I would call it pain, but let them experience like how it was. Right. And, and give them a complex enough task where they feel like something is wrong right. Or, or like, there, should be something better. And then you can go to you could go straight to vue or react. I'm not sure if we need to go like, Here's backbone, here's knockout. [00:34:22] Victor: Yeah. That's like historical. Interesting. [00:34:27] Jeremy: I, I think that would be an interesting college course or something that. Like, I remember when, I went through school, one of the classes was programming languages. So we would learn things like, Fortran and stuff like that. And I, I think for a more frontend centered or modern equivalent you could go through, Hey, here's the history of frontend development here's what we used to do and here's how we got to where we are today. I think that could be actually a pretty interesting class yeah [00:35:10] Victor: I'm a bit interested to know you learned fortran in your PL class. I, think when I went, I was like, lisp and then some, some other, like, higher classes taught haskell but, um, but I wasn't ready for haskell, not many people but fortran is interesting, I kinda wanna hear about that. [00:35:25] Jeremy: I think it was more in terms of just getting you exposed to historically this is how things were. Right. And it wasn't so much of like, You can take strategies you used in Fortran into programming as a whole. I think it was just more of like a, a survey of like, Hey, here's, you know, here's Fortran and like you were saying, here's Lisp and all, all these different languages nd like at least you, you get to see them and go like, yeah, this is kind of a pain. [00:35:54] Victor: Yeah [00:35:55] Jeremy: And like, I understand why people don't choose to use this anymore but I couldn't take away like a broad like, Oh, I, I really wish we had this feature from, I think we were, I think we were using Fortran 77 or something like that. I think there's Fortran 77, a Fortran 90, and then there's, um, I think, [00:36:16] Victor: Like old fortran, deprecated [00:36:18] Jeremy: Yeah, yeah, yeah. So, so I think, I think, uh, I actually don't know if they're, they're continuing to, um, you know, add new things or maintain it or it's just static. But, it's, it's more, uh, interesting in terms of, like we were talking front end where it's, as somebody who's learning frontend development who is new and you get to see how, backbone worked or how Knockout worked how grunt and gulp worked. It, it's like the kind of thing where it's like, Oh, okay, like, this is interesting, but let us not use this again. Right? [00:36:53] Victor: Yeah. Yeah. Right. But I also don't need this, and I will never again [00:36:58] Jeremy: yeah, yeah. It's, um, but you do definitely see the, the parallels, right? Like you were saying where you had your, your Bower and now you have NPM and you had Grunt and Gulp and now you have many choices [00:37:14] Victor: Yeah. [00:37:15] Jeremy: yeah. I, I think having he history context, you know, it's interesting and it can be helpful, but if somebody was. Came to me and said hey I want to learn how to build websites. I get into front end development. I would not be like, Okay, first you gotta start moo tools or GWT. I don't think I would do that but it I think at a academic level or just in terms of seeing how things became the way they are sure, for sure it's interesting. [00:37:59] Victor: Yeah. And I, I, think another thing I don't remember who asked or why, why I had to think of this lately. um but it was, knowing the differentiators between other technologies is also extremely helpful right? So, What's the difference between ES build and SWC, right? Again, we're, we're, we're leaning heavy front end, but you know, just like these, uh, sorry for context, of course, it's not everyone a front end developer, but these are two different, uh, build tools, right? For, for JavaScript, right? Essentially you can think of 'em as transpilers, but they, I think, you know, I think they also bundle like, uh, generally I'm not exactly sure if, if ESbuild will bundle as well. Um, but it's like one is written in go, the other one's written in Rust, right? And sort of there's, um, there's, in addition, there's vite which is like vite does bundle and vite does a lot of things. Like, like there's a lot of innovation in vite that has to have to do with like, making local development as fast as possible and also getting like, you're sort of making sure as many things as possible are strippable, right? Or, or, or tree shakeable. Sorry, is is is the better, is the better term. Um, but yeah, knowing, knowing the, um, the differences between projects is often enough to sort of make it less confusing for me. Um, as far as like, Oh, which one of these things should I use? You know, outside of just going with what people are recommending. Cause generally there is some people with wisdom sometimes lead the crowd sometimes, right? So, so sometimes it's okay to be, you know, a crowd member as long as you're listening to the, to, to someone worth listening to. Um, and, and so yeah, I, I think that's another thing that is like the mark of a good project or, or it's not exclusive, right? It's not, the condition's not necessarily sufficient, but it's like a good projects have the why use this versus x right section in the Readme, right? They're like, Hey, we know you could use Y but here's why you should use us instead. Or we know you could use X, but here's what we do better than X. That might, you might care about, right? That's, um, a, a really strong indicator of a project. That's good cuz that means the person who's writing the project is like, they've done this, the survey. And like, this is kind of like, um, how good research happens, right? It's like most of research is reading what's happening, right? To knowing, knowing the boundary you're about to push, right? Or try and sort of like push one, make one step forward in, um, so that's something that I think the, the rigor isn't in necessarily software development everywhere, right? Which is good and bad. but someone who's sort of done that sort of rigor or, and like, and, and has, and or I should say, has been rigorous about knowing the boundary, and then they can explain that to you. They can be like, Oh, here's where the boundary was. These people were doing this, these people were doing this, these people were doing this, but I wanna do this. So you just learned now whether it's right for you and sort of the other points in the space, which is awesome. Yeah. Going to your point, I feel like that's, that's also important, it's probably not a good idea to try and get everyone to go through historical artifacts, but if just a, a quick explainer and sort of, uh, note on the differentiation, Could help for sure. Yeah. I feel like we've skewed too much frontend. No, no more frontend discussion this point. [00:41:20] Jeremy: It's just like, I, I think there's so many more choices where the, the mental thought that has to go into, Okay, what do I use next I feel is bigger on frontend. I guess it depends on the project you're working on but if you're going to work on anything front end if you haven't done it before or you don't have a lot of experience there's so many build tools so many frameworks, so many libraries that yeah, but we [00:41:51] Victor: Iterate yeah, in every direction, like the, it's good and bad, but frontend just goes in every direction at the same time Like, there's so many people who are so enthusiastic and so committed and and it's so approachable that like everyone just goes in every direction at the same time and like a lot of people make progress and then unfortunately you have try and pick which, which branch makes sense. [00:42:20] Jeremy: We've been kind of talking about, some of your experiences with a few things and I wonder if you could explain the the context you're thinking of in terms of the types of projects you typically work on like what are they what's the scale of them that sort of thing. [00:42:32] Victor: So I guess I've, I've gone through a lot of phases, right? In sort of what I use in in my tooling and what I thought was cool. I wrote enterprise java like everybody else. Like, like it really doesn't talk about it, but like, it's like almost at some point it was like, you're either a rail shop or a Java shop, for so many people. And I wrote enterprise Java for a, a long time, and I was lucky enough to have friends who were really into, other kinds of computing and other kinds of programming. a lot of my projects were wrapped around, were, were ideas that I was expressing via some new technology, let's say. Right? So, I wrote a lot of haskell for, for, for a while, right? But what did I end up building with that was actually a job board that honestly didn't go very far because I was spending much more time sort of doing, haskell things, right? And so I learned a lot about sort of what I think is like the pinnacle of sort of like type development in, in the non-research world, right? Like, like right on the edge of research and actual usability. But a lot of my ideas, sort of getting back to the, the ideas question are just things I want to build for myself. Um, or things I think could be commercially viable or like do, like, be, be well used, uh, and, and sort of, and profitable things, things that I think should be built. Or like if, if I see some, some projects as like, Oh, I wish they were doing this in this way, Right? Like, I, I often consider like, Oh, I want, I think I could build something that would be separate and maybe do like, inspired from other projects, I should say, Right? Um, and sort of making me understand a sort of a different, a different ecosystem. but a lot of times I have to say like, the stuff I build is mostly to scratch an itch I have. Um, and or something I think would be profitable or utilizing technology that I've seen that I don't think anyone's done in the same way. Right? So like learning Kubernetes for example, or like investing the time to learn Kubernetes opened up an entire world of sort of like infrastructure ideas, right? Because like the leverage you get is so high, right? So you're just like, Oh, I could run an aws, right? Like now that I, now that I know this cuz it's like, it's actually not bad, it's kind of usable. Like, couldn't I do that? Right? That kind of thing. Right? Or um, I feel like a lot of the times I'll learn a technology and it'll, it'll make me feel like certain things are possible that they, that weren't before. Uh, like Rust is another one of those, right? Like, cuz like Rust will go from like embedded all the way to WASM, which is like a crazy vertical stack. Right? It's, that's a lot, That's a wide range of computing that you can, you can touch, right? And, and there's, it's, it's hard to learn, right? The, the, the, the, uh, the, the ramp to learning it is quite steep, but, it opens up a lot of things you can write, right? It, it opens up a lot of areas you can go into, right? Like, if you ever had an idea for like a desktop app, right? You could actually write it in Rust. There's like, there's, there's ways, there's like is and there's like, um, Tauri is one of my personal favorites, which uses web technology, but it's either I'm inspired by some technology and I'm just like, Oh, what can I use this on? And like, what would this really be good at doing? or it's, you know, it's one of those other things, like either I think it's gonna be, Oh, this would be cool to build and it would be profitable. Uh, or like, I'm scratching my own itch. Yeah. I think, I think those are basically the three sources. [00:46:10] Jeremy: It's, it's interesting about Rust where it seems so trendy, I guess, in lots of people wanna do something with rust, but then in a lot of they also are not sure does it make sense to write in rust? Um, I, I think the, the embedded stuff, of course, that makes a lot of sense. And, uh, you, you've seen a sort of surge in command line apps, stuff ripgrep and ag, stuff like that, and places like that. It's, I think the benefits are pretty clear in terms of you've got the performance and you have the strong typing and whatnot and I think where there's sort of the inbetween section that's kind of unclear to me at least would I build a web application in rust I'm not sure that sort of thing [00:47:12] Victor: Yeah. I would, I characterize it as kind of like, it's a tool toolkit, so it really depends on the problem. And think we have many tools that there's no, almost never a real reason to pick one in particular right? Like there's, Cause it seems like just most of, a lot of the work, like, unless you're, you're really doing something interesting, right? Like, uh, something that like, oh, I need to, I need to, like, I'm gonna run, you know, billions and billions of processes. Like, yeah, maybe you want erlang at that point, right? Like, maybe, maybe you should, that should be, you know, your, your thing. Um, but computers are so fast these days, and most languages have, have sort of borrowed, not borrowed, but like adopted features from others that there's, it's really hard to find a, a specific use case, for one particular tool. Uh, so I often just categorize it by what I want out of the project, right? Or like, either my goals or project goals, right? Depending on, and, or like business goals, if you're, you know, doing this for a business, right? Um, so like, uh, I, I basically, if I want to go fast and I want to like, you know, reduce time to market, I use type script, right? Oh, and also I'm a, I'm a, like a type zealot. I, I'd say so. Like, I don't believe in not having types, right? Like, it's just like there's, I think it's crazy that you would like have a function but not know what the inputs could be. And they could actually be anything, right? , you're just like, and then you have to kind of just keep that in your head. I think that's silly. Now that we have good, we, we have, uh, ways to avoid the, uh, ceremony, right? You've got like hindley Milner type systems, like you have a way to avoid the, you can, you know, predict what types of things will be, and you can, you don't have to write everything everywhere. So like, it's not that. But anyway, so if I wanna go fast, the, the point is that going back to that early, like the JS ecosystem goes everywhere at the same time. Typescript is excellent because the ecosystem goes everywhere at the same time. And so you've got really good ecosystem support for just about everything you could do. Um, uh, you could write TypeScript that's very loose on the types and go even faster, but in general it's not very hard. There's not too much ceremony and just like, you know, putting some stuff that shows you what you're using and like, you know, the objects you're working with. and then generally if I wanna like, get it really right, I I'll like reach for haskell, right? Cause it's just like the sort of contortions, and again, this takes time, this not fast, but, right. the contortions you can do in the type system will make it really hard to write incorrect code or code that doesn't, that isn't logical with itself. Of course interfacing with the outside world. Like if you do a web request, it's gonna fail sometimes, right? Like the network might be down, right? So you have to, you basically pull that, you sort of wrap that uncertainty in your system to whatever degree you're okay with. And then, but I know it'll be correct, right? But and correctness is just not important. Most of like, Oh, I should , that's a bad quote. Uh, it's not that correct is not important. It's like if you need to get to market, you do not necessarily need every single piece of your code to be correct, Right? If someone calls some, some function with like, negative one and it's not an important, it's not tied to money or it's like, you know, whatever, then maybe it's fine. They just see an error and then like you get an error in your back and you're like, Oh, I better fix that. Right? Um, and then generally if I want to be correct and fast, I choose rust these days. Right? Um, these days. and going back to your point, a lot of times that means that I'm going to write in Typescript for a lot of projects. So that's what I'll do for a lot of projects is cuz I'll just be like, ah, do I need like absolute correctness or like some really, you know, fancy sort of type stuff. No. So I don't pick haskell. Right. And it's like, do I need to be like mega fast? No, probably not. Cuz like, cuz so I don't necessarily don't necessarily need rust. Um, maybe it's interesting to me in terms of like a long, long term thing, right? Like if I, if I'm think, oh, but I want x like for example, tight, tight, uh, integration with WASM, for example, if I'm just like, oh, I could see myself like, but that's more of like, you know, for a fun thing that I'm doing, right? Like, it's just like, it's, it's, you don't need it. You don't, that's premature, like, you know, that's a premature optimization thing. But if I'm just like, ah, I really want the ability to like maybe consider refactoring some of this out into like a WebAssembly thing later, then I'm like, Okay, maybe, maybe I'll, I'll pick Rust. Or like, if I, if I like, I do want, you know, really, really fast, then I'll like, then I'll go Rust. But most of the time it's just like, I want a good ecosystem so I don't have to build stuff myself most of the time. Uh, and you know, type script is good enough. So my stack ends up being a lot of the time just in type script, right? Yeah. [00:52:05] Jeremy: Yeah, I think you've encapsulated the reason why there's so many packages on NPM and why there's so much usage of JavaScript and TypeScript in general is that it, it, it fits the, it's good enough. Right? And in terms of, in terms of speed, like you said, most of the time you don't need of rust. Um, and so typescript I think is a lot more approachable a lot of people have to use it because they do front end work anyways. And so that kinda just becomes the I don't know if I should say the default but I would say it's probably the most common in terms of when somebody's building a backend today certainly there's other languages but JavaScript and TypeScript is everywhere. [00:52:57] Victor: Yeah. Uh, I, I, I, another thing is like, I mean, I'm, of ignored the, like, unreasonable effectiveness of like rails Cause there's just a, there's tons of just like rails warriors out there, and that's great. They're they're fantastic. I'm not a, I'm not personally a huge fan of rails but that's, uh, that's to my own detriment, right? In, in some, in some ways. But like, Rails and Django sort of just like, people who, like, I'm gonna learn this framework it's gonna be excellent. It most, they have a, they have carved out a great ecosystem for themselves. Um, or like, you know, even php right? PHP and like Laravel, or whatever. Uh, and so I'm ignoring those, like, those pockets of productivity, right? Those pockets of like intense productivity that people like, have all their needs met in that same way. Um, but as far as like general, general sort of ecosystem size and speed for me, um, like what you said, like applies to me. Like if I, if I'm just like, especially if I'm just like, Oh, I just wanna build a backend, Like, I wanna build something that's like super small and just does like, you know, maybe a few, a couple, you know, endpoints or whatever and just, I just wanna throw it out there. Right? Uh, I, I will pick, yeah. Typescript. It just like, it makes sense to me. I also think note is a better. VM or platform to build on than any of the others as well. So like, like I, by any of the others, I mean, Python, Perl, Ruby, right? Like sort of in the same class of, of tool. So I I am kind of convinced that, um, Node is better, than those as far as core abilities, right? Like threading Right. Versus the just multi-processing and like, you know, other, other, other solutions and like, stuff like that. So, if you want a boring stack, if I don't wanna use any tokens, right? Any innovation tokens I reach for TypeScript. [00:54:46] Jeremy: I think it's good that you brought up. Rails and, and Django because, uh, personally I've done, I've done work with Rails, and you're right in that Rails has so many built in, and the ways to do them are so well established that your ability to be productive and build something really fast hard to compete with, at least in my experience with available in the Node ecosystem. Um, on the other hand, like I, I also see what you mean by the runtimes. Like with Node, you're, you're built on top of V8 and there's so many resources being poured into it to making it fast and making it run pretty much everywhere. I think you probably don't do too much work with managed services, but if you go to a managed service to run your code, like a platform as a service, they're gonna support Node. Will they support your other preferred language? Maybe, maybe not, You know that they will, they'll be able to run node apps so but yeah I don't know if it will ever happen or maybe I'm just not familiar with it, but feel like there isn't a real rails of javascript. [00:56:14] Victor: Yeah, you're, totally right. There are, there are. It's, it's weird. It's actually weird that there, like Uh, but, but, I kind of agree with you. There's projects that are trying it recently. There's like Adonis, um, there is, there are backends that also do, like, will do basic templating, like Nest, NestJS is like really excellent. It's like one of the best sort of backend, projects out there. I I, I but like back in the day, there were projects like Sails, which was like very much trying to do exactly what Rails did, but it just didn't seem to take off and reach that critical mass possibly because of the size of the ecosystem, right? Like, how many alternatives to Rails are there? Not many, right? And, and now, anyway, maybe let's say the rest of 'em sort of like died out over the years, but there's also like, um, hapi HAPI, uh, which is like also, you know, similarly, it was like angling themselves to be that, but they just never, they never found the traction they needed. I think, um, or at least to be as wide, widely known as Rails is for, for, for the, for the Ruby ecosystem, um, but also for people to kind of know the magic, cause. Like I feel like you're productive in Rails only when you imbibe the magic, right? You, you, know all the magic context and you know the incantations and they're comforting to you, right? Like you've, you've, you have the, you have the sort of like, uh, convention. You're like, if you're living and breathing the convention, everything's amazing, right? Like, like you can't beat that. You're just like, you're in the zone but you need people to get in that zone. And I don't think node has, people are just too, they're too frazzled. They're going like, there's too much options. They can't, it's hard to commit, right? Like, imagine if you'd committed to backbone. Like you got, you can't, It's, it's over. Oh, it's not over. I mean, I don't, no, I don't wanna, you know, disparage the backbone project. I don't use it, but, you know, maybe they're still doing stuff and you know, I'm sure people are still working on it, but you can't, you, it's hard to commit and sort of really imbibe that sort of convention or, or, or sort of like, make yourself sort of breathe that product when there's like 10 products that are kind of similar and could be useful as well. Yeah, I think that's, that's that's kind of big. It's weird that there isn't a rails, for NodeJS, but, but people are working on it obviously. Like I mentioned Adonis, there's, there's more. I'm leaving a bunch of them out, but that's part of the problem. [00:58:52] Jeremy: On, on one hand, it's really cool that people are trying so many different things because hopefully maybe they can find something that like other people wouldn't have thought of if they all stick same framework. but on the other hand, it's ... how much time have we spent jumping between all these different frameworks when what we could have if we had a rails. [00:59:23] Victor: Yeah the, the sort of wasted time is, is crazy to think about it uh, I do think about that from time to time. And you know, and personally I waste a lot of my own time. Like, just, just rec

amazon google running myth tokyo os speed medium seo pl windows workers ebay sonic hackers saas snap personally depending nest meant react wordpress assurance platforms ux rust api complexity languages python astro aws java github databases lamp assuming vm mastodon maus javascript html pivotal rails clap knockout acquired versus csr css php ssd containers longhorns django docker node kubernetes js cuz sails v8 foss pelican vue ssl haskell gitlab elastic yc bower bake off milner moo grunt paas tls gulp terraform v1 typescript cli mvc jquery nodejs npm redis readme encrypt heroku webassembly lisp hacker news ansible postgres laravel eks wasm ebs aks cdk nginx sqlite fortran ec2 deltas swc kelsey hightower hapi activitypub zfs mvvm nuxt dsls lvm gwt redis labs dockerfile lucene sidekiq yak shaving durandal nestjs jeremy it jeremy you jeremy so victor it victor there

Episode 149: Naytri Sramek on the GitHub Accelerator and M12 GitHub Fund

Sustain

Play Episode Listen Later Dec 16, 2022 40:10

Guest Naytri Sramek Panelists Richard Littauer | Justin Dorfman Show Notes Hello and welcome to Sustain! The podcast where we talk about sustaining open source for the long haul. Today, we're super excited to have joining us as our guest, Naytri Sramek, who's the Senior Director of Strategy at GitHub. Have you heard of the GitHub Accelerator and M12 GitHub Fund? Well, this is a great day to be joining us because Naytri is here to talk about these programs that they've been launching to help support and sustain OSS over the long haul. Naytri shares GitHub's journey which began with the GitHub Sponsors launch in 2019, bringing on enterprise sponsors, and how it led into launching the GitHub Accelerator program and the M12 GitHub Fund. Go ahead and download this episode now to learn more. [00:01:23] Naytri reveals the two things they've been launching which are the GitHub Accelerator and the M12 GitHub Fund. She also tells us about bringing on enterprise sponsors since they've benefited from open source. [00:06:25] Peter Thomas, who worked at Intuit and is creator of Karate Labs, is brought up and Justin wonders if he's involved in this venture or if there are others. [00:09:37] A question comes up regarding if the growth of the projects has been tracked with the money that GitHub has given to developers, if they've been able to quit their jobs since the money was given to them, and if those projects have improved. [00:15:35] We hear the focus of the GitHub sponsors, the Accelerator, and the M12 Fund. [00:19:57] Justin brings up the difficult issue of how to deal with developers that build these critical pieces of software, but they don't want to deal with the responsibility and wonders how Naytri and her team deal with this issue. [00:23:18] There's a 10-week course for the accelerator program and we hear how it works, and if it will be available to everyone in the future on GitHub. [00:29:28] Naytri explains how the communities are being funded. [00:32:47] A point is brought up about how long can these strategies and programs live on so maintainers and open source developers can make a good living, and Naytri goes in depth about the need for more sources of funding and funding models. [00:36:34] Find out where you can learn more about the GitHub Accelerator and the M12 Fund. Quotes [00:17:40] “The M12 GitHub Fund is all about how we do invest in the tools that are built on GitHub's platform.” [00:24:33] “I want 20 people making $200,000 a year.” [00:24:58] “The GitHub Accelerator course itself will be open source.” [00:28:08] “As we've expanded the program into more countries, we've doubled the number of countries that sponsors cover right now.” [00:30:10] “Commits aren't universal. You shouldn't just be rewarded for the code.” [00:33:07] “The way we're thinking about the accelerator and the fund is we need so many more sources of funding and funding models to be able to support open source creators as well as communities.” Spotlight [00:37:44] Justin's spotlight is Jessica Lord, who's the GitHub Sponsors Product Lead. [00:38:14] Richard's spotlight is Bill Watterson, author of Calvin and Hobbes. [00:38:23] Naytri's spotlight is Mike Perham and a 10 year anniversary post he wrote to Sidekiq. Links SustainOSS (https://sustainoss.org/) SustainOSS Twitter (https://twitter.com/SustainOSS?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) SustainOSS Discourse (https://discourse.sustainoss.org/) podcast@sustainoss.org (mailto:podcast@sustainoss.org) Richard Littauer Twitter (https://twitter.com/richlitt?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) Justin Dorfman Twitter (https://twitter.com/jdorfman?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) Naytri Sramek LinkedIn (https://www.linkedin.com/in/naytri-sramek-a6872516?trk=people-guest_people_search-card) naytri@github.com (mailto:naytri@github.com) fund@github.com (mailto:fund@github.com) GitHub Accelerator (https://accelerator.github.com/) GitHub Blog- An open source economy-built by developers, for developers by Naytri Sramek (https://github.blog/2022-11-09-an-open-source-economy-built-by-developers-for-developers/) Sustain Podcast-Episode 56: Dominic Tarr on Coding What You Want, Living on A Boat, and the Early Days of Node.js (https://podcast.sustainoss.org/56) Karate Labs (https://www.karatelabs.io/) Hopscotch (https://www.gethopscotch.com/) Justin Dorfman Tweet: The hard decisions popular open source project maintainers need to make…daily. (https://twitter.com/jdorfman/status/1597991465673596935) Jessica Lord-GitHub (https://github.com/jlord) Bill Watterson-Wikipedia (https://en.wikipedia.org/wiki/Bill_Watterson) Happy 10th Birthday, Sidekiq! -by Mike Perham (https://www.mikeperham.com/2022/01/17/happy-10th-birthday-sidekiq/) Credits Produced by Richard Littauer (https://www.burntfen.com/) Edited by Paul M. Bahr at Peachtree Sound (https://www.peachtreesound.com/) Show notes by DeAnn Bahr Peachtree Sound (https://www.peachtreesound.com/) Special Guest: Naytri Sramek.

strategy living boats fund senior director quotes edited spotlight github sustain accelerator commits early days intuit oss node hobbes hopscotch bill watterson peter thomas sidekiq mike perham jessica lord dominic tarr

LCC 288 - L'épisode marathon mastodonien

Les Cast Codeurs Podcast

Play Episode Listen Later Nov 21, 2022 101:47

Dans ce long épisode, retrouvez Emmanuel, Guillaume, Antonio et Arnaud qui reviennent sur les dernières sorties de GraalVM, GoLang, JBanking, Spring, Spring Modulith, Quarkus, Apache Maven. Vous retrouverez aussi de nombreux sujets infrastructure, cloud, méthodologie le tout accompagné d’un pachyderme très à la mode en ce moment: Mastodon. Enregistré le 18 novembre 2022 Téléchargement de l'épisode LesCastCodeurs-Episode–288.mp3 News Langages Alina Yurenko annonce la sortie de GraalVM 22.3 https://medium.com/graalvm/graalvm–22–3-is-here-jdk–19-builds-jlink-support-new-monitoring-features-and-more-f6e2b2eeff95 l'article mentionne l'annonce faite à JavaOne qu'Oracle contribue GraalVM CE à la communauté Open JDK https://www.graalvm.org/2022/openjdk-announcement/ support du JDK 19 possibilité de télécharger facilement (dans un script) la distribution avec un one-line (bash/curl) possibilité de compiler jWebserver en un exécutable natif diverses améliorations sur le monitoring et l'expérience développeur de native image (JFR, jvmstat, head dump…) nouvelles versions des reachability metadata nouvelle API native image et diverses autres updates sur le support de Python, de Ruby, des contributions de la communauté Go fête ses 13 ans https://go.dev/blog/13years avec la grosse release de 1.18, avec le support des workspaces, du fuzzing, mais surtout des generics aussi une commande govuln qui fait analyse statique - intéressant la notion d'outil dans le langage les build go sont vérouillés vu qu'ils reconstruisent tout et qu'ils dépendent d'un sha1 pour les dependences git et beaucoup plus de choses ici https://go.dev/blog/supply-chain workspace qui permet de travailler sur plusieurs modules en parallèle sans avoir a changer tous les go.mod à la main Librairies Sortie de JBanking 4 par Marc Wrobel https://www.marcwrobel.fr/sortie-de-jbanking–4–0–0 Une librairie utilitaire pour assister dans le développement d'applications bancaires Support des codes ISO des pays, des monnaies, des codes BIC, des IBAN, et aussi du calendrier des jours fériés des banques internationales Spring Modulith, un projet expérimental d'Oliver Drotbohm, qui permet de s'assurer de la structure et architecture de ses projets Spring, par exemple pour vérifier les dépendances propres entre modules, pour bien structurer ses applications Spring Boot https://spring.io/blog/2022/10/21/introducing-spring-modulith Une version alpha de Quarkus 3 arrive ! https://quarkus.io/blog/road-to-quarkus–3/ Plein d'upgrades : Hibernate ORM 6, Jakarta EE 10, Eclipse MicroProfile 6, HTTP/3, io_uring, Virtual Threads de Loom et Structured Concurrency, java.util.concurrent.Flow pour s'affranchir de Reactive Streams Version cible Java 11, mais recommendation d'utiliser Java 17 les versions 3 seront en parallèle des versions 2 le temps que l’écosystème passe à la 3, notamment les dependences jakartaee peut essayer facilement depuis la CLI quarkus create app --stream=3.0 quelques casse de compatibilités attendues mais minimisées, spécialement dans le core garde java 11 car demande de la communauté Spring 6.0 est sorti https://spring.io/blog/2022/11/16/spring-framework–6–0-goes-ga Java 17+ de base Jakarta EE 9+ Hibernate 6+ foundations pour Ahead of Time transformations pour GraalVM Exploration des threads virtuels https://spring.io/blog/2022/10/11/embracing-virtual-threads tester sur les threads servlets et autre SpringBoot arrive plus tard Détail des changements https://github.com/spring-projects/spring-framework/wiki/What%27s-New-in-Spring-Framework–6.x/ Infrastructure Stop using CPU limits on Kubernetes https://home.robusta.dev/blog/stop-using-cpu-limits L'auteur fait une comparaison amusante avec le besoin de boire de l'eau ! Il vaut mieux définir des requêtes (des besoins en eau / CPU), plutôt que des limites (pas le droit de boire plus / d'utiliser plus de CPU) c'est plus nuancé que ca, parce que aux cas limites des choses peuvent mal se passer cas 1: on atteind 100% d'usage. Le process avait définit un request mais en fait a besoin de plus en pratique, et là paf il se met à mal fonctionner, donc dès que votre systeme stresse, vous avez des erreurs en cascade cas 2: un ou plusieurs noeuds sont recyclés, ce qui veut dire que vous avez beaucoup de redémarrages de pods et du coup ca met la pression sur le CPU, tester ces cas là, certaines applis qui démarrent trop lentement ont tendance à tomber en cascade Comment faire des attaques d'injection sur les intelligences artificielles qui recoivent du contenu utilisateur https://hackaday.com/2022/09/16/whats-old-is-new-again-gpt–3-prompt-injection-attack-affects-ai/ le jeu est de donner des phrases ambigues qui font faire à l'IA des choses qu'elle n'est pas sensé faire un des outils c'est ignore les instructions au dessus et fait un truc que je veux que tu fasses et qui n'est pas dans ta programmation initiale Voir toucher l'intention initiale de l'AI (lui faire dire) et donc d'atteindre des sphères non publiques du service Mastodon et la scalabilité https://framablog.org/2022/11/13/de-la-friture-sur-le-fediverse/ la decentralisatione et le protocole Mastodon est couteux en job donc une personne moderement populaire 27k personnes, devrait bouger vers son instance dédiée ce qui amènerait à couter assez cher par mois (en tous cas plus que 8$/mois) L’auteur explique que les devs devraient favorier un protocol fortement decentraliser plutot qu’optimiser pour les grosses instances un article qui couvre la configuration aux petits oignons de Sidekiq, qui traite les queues de tâches, pour scaler une instance Mastodon https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/ Rollouts de release a l’échelle avec Argo (rollback options) https://monzo.com/blog/2022/11/02/argo-rollouts-at-scale/ gros investissement sur ArgoCD Mais encore release à la main par les ingenieurs et tout ou rien pour une application idealement: push dans git et oublie, prometheus metriques dirigent le rollout basé sur des alertes génériques, garder le sisteme ouvert pour des stratégies de rollout alternatives dans le futur basé sur Argo Rollouts et sur des erreurs generiques (20% de calls en erreur, beaucoup d’erreurs de base de donnees, crashs notifie dans slack en async du success ou de l’echec interessant de voir qu’ils s’appuient sur des metriques simples Lessons apprises migration est un gros job automatiser la migration au maximum meme si c’est un one shot change le moteurt avant de changer l’UX (progressive rollout) ca simplifie les chosez Cloud Google adopte progressivement Adoptium Temurin comme version officielle de JDK dans ses produits https://glaforge.appspot.com/article/building-and-deploying-java–17-apps-on-cloud-run-with-cloud-native-buildpacks-on-temurin nous avions mentionné l'annonce de ce support dans l'épisode précédent https://blog.adoptium.net/2022/10/adoptium-welcomes-google/ dans l'article de Guillaume, il utilise les Cloud Native Buildpacks, configuré pour utiliser Java 17, et par défaut, c'est bien Temurin qui est utilisé quand on build à partir des sources dans l'exemple, une application Micronaut, développée avec Java 17, est déployée sur Google Cloud Run Pourquoi on quitte le Cloud https://world.hey.com/dhh/why-we-re-leaving-the-cloud–654b47e0 témoignage de DHH de 37Signal (basecamp et hey) Les 30% de marges d'Amazon viennent de quelque part. On dépense 500k en RDS et ES. On peut acheter beaucoup de machines pour ce prix La réduction des ops est un mythe. On a autant de personnes gérant les services AWS ou Google Cloud Le gain pourrait être la micro startup qui ne sait pas si elle aura des clients ou les volumes de demandes très variables et imprédictibles Mais on a une croissance planifiée Donc on rapatrie Présentation de Mickaël Roger de Thales, enregistrée à Cloud Nord, qui explique le fonctionnement de l'offre S3NS de Thales et Google Cloud pour le “cloud de confiance” https://www.youtube.com/watch?v=OBwBeqd-YFs Web Est-ce que le Web3 peut battre le cloud ? https://blog.scottlogic.com/2022/10/31/can-web3-beat-the-cloud.html Le Web3 est une autre approche pour des applications décentralisées, ce n'est pas un successeur du Web 2.0 classique, et il a généralement besoin du Web 2.0 pour offrir une interface à ses utilisateurs Ce n'est pas que pour faire des cryptomonnaies qui gâchent de l'électricité, ou des NFTs qui ne donnent pas vraiment de titre de propriété d'une oeuvre d'art Dans cet article, l'auteur essaie d'implémenter une fonctionnalité (le fait de pouvoir rajouter des “applaudissements” à un article, un peu comme sur Medium), en implémentant un smart contract en Web3. Mais il se heurte à plein d'écueils le long de sa route, à la dépendance à plein d'autres services, au fait que ce n'est pas la personne qui “vote” qui devrait payer l'action mais celui qui héberge. Au final, il est obligé d'ajouter plein d'adhérences qui font qu'au lieu d'être décentralisée, l'application dépends de trop d'autres services, et a finalement besoin du Web 2.0 pour fonctionner, et du Cloud L'autre déconvenue est sur le prix de chacune des transactions, qui est finalement exorbitant par rapport à une approche Web 2.0 classique Décentralisation amène de la lenteur (latence) Objectifs du Web3 c'est d'etre le propriétaire de ses processes et ses data et de mettre des agents qui interagissent avec des données Outillage Comment debugguer les images Docker slim ou distroless https://iximiuz.com/en/posts/docker-debug-slim-containers/ Les images slim / distroless sont sympas car elles permettent d'avoir des petits conteneurs qui ne prennent pas trop de place, qui parfois sont plus rapides à charger, mais également qui exposent une surface d'attaque beaucoup plus faible Par contre, comme il n'y a pas tous les outils (parfois pas de shell, par exemple), c'est plus compliqué de comprendre ce qu'il se passe à l'intérieur quand quelque chose ne fonctionne pas L'article propose quelques approches pour pallier à cela : Installer des outils à la demande dans un conteneur qui tourne (à coup de apt-get) Passer temporairement à une image plus grosse et plus complète (par ex, distroless a des images avec un tag debug) Utiliser docker run avec un shared namespace Utiliser docker exec et un mount Podman Desktop, une alternative à Docker Desktop, mais utilisant podman https://podman-desktop.io/ Docker annonce une technical preview de conteneurs WASM https://www.docker.com/blog/docker-wasm-technical-preview/ nouveau packaging qui wrap un exécutable WASM et le fait tourner avec le runtime wasmEdge c'est un nouveau type de conteneur il y a beaucoup d'activité autour de WASM, et il y a eu de nombreuses annonces et démonstration lors de la conférence CloudNativeCon et le jour spécial sur WASM, lors de KubeCon https://www.infoq.com/news/2022/11/cloud-native-wasm-day/ docker utilise Docker Desktop et docker engine pour demarrer des “shim" Ses shim (processeS) lancent soit runc (donc pour faire tourner un containeur) soit wasmedge pour faire tourner des modules wasm Donc docker s'éloigne des container et essaie de toucher l'orchestration Un petit tutoriel utilisant Docker et YouTube-dl pour récupérer / consulter les stats (views, likes) de vos vidéos (ou d'autres) sur YouTube https://glaforge.appspot.com/article/retrieve-youtube-views-count-with-youtube-dl-jq-and-a-docker-container Apache Maven propose une extension de “build cache” (qui devrait accélérer les builds, sans tout tout le temps recompiler) https://maven.apache.org/extensions/maven-build-cache-extension/ basé sur une clé construite des sources, des plugins etc par module permet paralelisation et de deploiement sur des agents genre dans le cloud on controle les regles de contournement des invarients (genre changement de compile, timestamp dans les manifests etc) Le guide complet pour publier une librairie Java sur Maven Central https://maciejwalkowiak.com/blog/guide-java-publish-to-maven-central/ Y compris l'intégration avec Github Actions et l'utilisation de Github Secrets pour les clés PGP Et enfin la configuration de JReleaser pour encore faciliter la tâche lorsque l'on pousse une nouvelle version Apache Maven 4.0.0-alpha–2 is out https://maven.apache.org/docs/4.0.0-alpha–2/release-notes.html améliorations cli: --also-make , --resume (plus besoin de pré ciser d'où le build doit recommencer), --non-recursive, --fail-on-severity Utilisation du même timestamps dans tous les modules build/consumer POMs (versioning automatique du parent, versioning automatique des dépendances dans le réacteur, détection automatique des sous modules) new maven 4 api et beaucoup d'autres choses: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351403&projectId=12316922 Data Faker le nouveau générateur de données de test https://github.com/datafaker-net/datafaker C'est un fork de Java Faker https://github.com/DiUS/java-faker Tout ça inspiré de Ruby Faker https://github.com/faker-ruby/faker La boite australienne qui l'avait créé ne maintenait plus le projet, ne le publiait plus dans Maven Central et il y avait des centaines de PRs Vous pouvez générer des données de centaines de provider (ex. adresse, compte bancaire, livres, films, etc) https://github.com/datafaker-net/datafaker#providers en plusieurs langues Exécuter facilement des programmes Java avec dépendances, sans build, avec JBang https://maciejwalkowiak.com/blog/single-file-java-with-jbang/ Dans la même veine que ce que Groovy propose depuis de nombreuses années avec sons système @Grapes qui récupère les dépendances nécessaires L'article montre un exemple simple, puis avec Spring Boot, comment faire un JAR aussi, voire comment conteneurisé sa petite appli Architecture Amélie Benoit continue ses fabuleux sketchnotes sur le thème des design patterns https://twitter.com/AmelieBenoit33/status/1587397290251149312 celui ci est sur le pattern Adapter il y a eu aussi le pattern Builder https://twitter.com/AmelieBenoit33/status/1584778615610228737 l'Observer https://twitter.com/AmelieBenoit33/status/1579706242318360576 ou le Singleton https://twitter.com/AmelieBenoit33/status/1570313646605234177 https://twitter.com/AmelieBenoit33/status/1589869904404316162 Un petit coup de décorateur https://twitter.com/AmelieBenoit33/status/1592468635599372289 35 misconceptions sur les dates et les heures https://www.meziantou.net/misconceptions-about-date-and-time.htm y a t'il toujours 24 heures par jour, 60 secondes dans une minute? ou 365 jours par an ? est-ce que les jours sont toujours consécutifs ? tout le monde a t'il le même calendrier ? lundi est il le premier jour de la semaine ? Méthodologies Interview d'un designer sur comment enlever la friction https://review.firstround.com/amazons-friction-killing-tactics-to-make-products-more-seamless?ct=t designer a Amazon (Music, Alexa), IMDB, Skype for Business types de fictions (choses non familières, friction inhérente - produit avancé - et chemin de friction important, friction par desalignement avec le comportement humain) la troisième catégorie difficile à anticiper en construisant des produits: on ajoute, enlève ou marque des frictions C'est sur le chemin du client Avant le premier contact Signature et premiere tâche transactionnelle (bien choisir la tache pour etre assez simple et ce que l'utilisateur répète) Premier moment de plaisir (regarder les points contre intuitifs dans les données, ou les cas d'utilisation en echec) l'indifférence genre la friction la plus importante pour les nouveaux produits Comment écouter son utilisateur? habitat naturel: sondes dans l'appli, tests chez l'utilisateur en milieu reel en utilisation du produit mentions et revues: aussi métriques d'usage (choses inhabituelles ou inattendues) standard de l'industrie: attentes des clients façonné par ça (barre de recherche en haut) Toujours migrer son audience vers le chemin de moindre resistance Comment éliminer la friction? réduire l'anxiété: décision et perte amènent de l'anxiété. supprimer les étapes non nécessaires: définir la liste des decisions du client et les questionner. (Heuristiques par defaut?) mitiger le changement de contexte: naviguer hors de l'appli pour faire quelque chose, risque d'abandon. Arrêter un livre pour lire un mot dans le dictionnaire, faciliter le retour et le rappel du contexte quand ils reviennent. Comment masquer la friction? temps d'attente: trouver de la valeur (message d'information) bouger la friction au début dans les services (carte credit tout de suite) s’ils investissent dans leur experience (vote), ils sont plus engagés et loyaux: friction positive : sense d'appartenance Glossaire et aide-mémoire sur l'approche de l'Event Storming https://github.com/ddd-crew/eventstorming-glossary-cheat-sheet Si vous ne connaissez pas event storming, ça ne va pas vous éclairer assez plutôt un outil pour rafraichir votre mémoire voir aussi episode sur event storming https://lescastcodeurs.com/2020/06/05/lcc–233-interview-sur-l-event-storming-avec-thomas-pierrain-et-bruno-boucard/ Sécurité Sigstore passe en General Availability, en version 1 https://opensource.googleblog.com/2022/10/sigstore-project-announces-general-availability-and-v1-releases.html Sujet également couvert par InfoQ https://www.infoq.com/news/2022/11/sigstore-stability-ga/ Sigstore est la pour aider au niveau de la sécurisation de la supply chain de code Notamment au niveau des signatures Ca addresse ce que fait PGP amis le rend plus utilisable et permet un usage supplémentaire par un log lisible par tous Plus d'infos dans une interview on espère Loi, société et organisation La proposition de loi sur la sécurisation de l'open source aux Etats-Unis https://blog.tidelift.com/tidelift-advisory-us-senators-introduce-the-securing-open-source-software-act-of–2022 (edited) Holly Cummins sur le sujet du code vestimentaire des femmes dans la tech https://hollycummins.com/fashion-and-programming-ii/ Pourquoi en 2023 on a encore autant d'abrutis qui font des remarques sur les vêtements que portent les femmes qui font des présentations à des conférences, et pire, sur le fait de savoir si elles sont à leur goût ou pas La tenue vestimentaire n'a rien à voir avec le talent, les connaissances, le professionnalisme, l'expertise des personnes Les femmes ont le droit de porter les vêtements qu'elles veulent sans être jugées par des idiots qui feraient mieux de retourner dans leur caverne Avec le rachat de Twitter par Elon Musk, beaucoup de gens commencent à s'intéresser de plus près à Mastodon. On trouve de nombreux articles sur Mastodon ces jours ci https://gorillasun.de/blog/getting-started-with-mastodon et vous, avez vous un compte sur Mastodon ? quelle instance avez-vous choisie ? quels outils (client, mobile, web, etc) utilisez vous ? Pour ma part je n’ai pas de compte Mastodon (je n’utilise pas twitter non plus). J’ai rapidement regardé ce matin ça n’est pas facile de trouver une instance : celles que j’ai regardé ont fermé les inscriptions (d’après ce que j’ai pu lire à cause de problèmes pour gérer l’afflux de nouveaux utilisateurs, à cause de l’augmentation de la création de comptes spam, ou dans l’objectif de répartir les utilisateurs sur d’autres instances moins connues). Du coup j’ai pour le moment abandonné l’idée de me créer un compte. Le site JavaBubble liste plein de développeurs Java qui ont maintenant un compte sur Mastodon https://javabubble.org/ Les Cast Codeurs sur Mastodon : @agoncal@fosstodon.org @aheritier@mastodon.social @glaforge@uwyn.net @emmanuelbernard@mamot.fr Conférences La liste des conférences provenant de Developers Conferences Agenda/List par Aurélie Vache et contributeurs : 23–25 novembre 2022 : Agile Grenoble 2022 - Grenoble (France) 25 novembre 2022 : HACK-IT-N 2022 - Bordeaux (France) 1 décembre 2022 : Devops DDay #7 - Marseille (France) 2 décembre 2022 : BDX I/O - Bordeaux (France) 2 décembre 2022 : DevFest Dijon 2022 - Dijon (France) 14–16 décembre 2022 : API Days Paris - Paris (France) & Online 15–16 décembre 2022 : Agile Tour Rennes - Rennes (France) 19–20 janvier 2023 : Touraine Tech - Tours (France) 25–28 janvier 2023 : SnowCamp - Grenoble (France) 2 février 2023 : Very Tech Trip - Paris (France) 2 février 2023 : AgiLeMans - Le Mans (France) 9–11 février 2023 : World AI Cannes - Cannes (France) 16–19 février 2023 : PyConFR - Bordeaux (France) 7 mars 2023 : Kubernetes Community Days France - Paris (France) 23–24 mars 2023 : SymfonyLive Paris - Paris (France) 5–7 avril 2023 : FIC - Lille Grand Palais (France) 12–14 avril 2023 : Devoxx France - Paris (France) 10–12 mai 2023 : Devoxx UK - London (UK) 12 mai 2023 : AFUP Day Lille & Lyon (France) 12–13 octobre 2023 : Volcamp 2023 - Clermont Ferrand (France) Nous contacter Pour réagir à cet épisode, venez discuter sur le groupe Google https://groups.google.com/group/lescastcodeurs Contactez-nous via twitter https://twitter.com/lescastcodeurs Faire un crowdcast ou une crowdquestion Soutenez Les Cast Codeurs sur Patreon https://www.patreon.com/LesCastCodeurs Tous les épisodes et toutes les infos sur https://lescastcodeurs.com/

Episode 437: It's Not Only Sauerkraut and Pickles (Brittany + Nick)

crossfit monitor devops pickles rails honey badger sauerkraut rubyconf sidekiq rubykaigi

Play Episode Listen Later Sep 28, 2022 30:52

After diving into CrossFit and their favorite comfort foods, Nick and Brittany talk all things conferences: RubyKaigi 2022, Rails SaaS and Rubyconf 2022. Nick ends the episode by sharing some cool Rails upgrade tooling he is working on. Show Notes & Links: CrossFit (https://www.crossfit.com/) RubyKaigi 2022 (https://rubykaigi.org/2022/) Rails SaaS Conference (https://railssaas.com/) Rubyconf 2022 (https://rubyconf.org/) Rubyconf Mini 2022 (https://www.rubyconfmini.com/) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

Episode 435: My Boss is Here with Joel Schlundt

boss shop cto badass established monitor tails devops rails honey badger ruby on rails textus from wednesday sidekiq steptember

Play Episode Listen Later Sep 14, 2022 24:39

Joel Schlundt is the CTO of TextUs and Brittany's boss. After diving into two origin stories, the pair discuss the Senior Backend Engineer role they are hiring for, applications they never want to build again and the importance of off-sites for remote teams. They cap the conversation off with a horror story. Announcements The Ruby on Rails Podcast Store (https://the-ruby-on-rails-podcast.myspreadshop.com/all?listModeOverride=DESIGN) Fun original designs like “Tell Me Your Developer Origin Story”, “Ruby is Badass”, “Ruby on Tails” and “the Established in Rails” series in shirts, hoodies, magnets, stickers and bandanas. Net proceeds of store purchases benefit Steptember, to raise funds for research and innovation for cerebral palsy. From Wednesday, September 14th through Saturday, September 17th 2022, all customers get free standard shipping on all orders. This does include international standard shipping. Shop here (https://the-ruby-on-rails-podcast.myspreadshop.com/all?listModeOverride=DESIGN). Show Notes & Links: Senior Software Engineer, Backend @ TextUs (https://boards.greenhouse.io/textus/jobs/4645583004) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

Episode 433: The One About Layoffs (Brittany + Brian)

monitor layoffs devops tracker honey badger sidekiq

Play Episode Listen Later Aug 31, 2022 28:23

The tech industry is in unprecedented times with the amount of layoffs happening. Brittany and Brian share their personal layoff stories and offer some advice to laid off employees, retained employees and companies. While empathizing with the shock layoffs cause, the duo offer some silver linings. Show Notes & Links: Tech Layoffs In 2022: The U.S. Companies That Have Cut Jobs (https://news.crunchbase.com/startups/tech-layoffs-2022/) Layoffs.fyi Tracker (http://layoffs.fyi/) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

Episode 431: For the Love of Consulting and the Cloud with Mina Slater

Play Episode Listen Later Aug 17, 2022 28:01

Mina Slater is a developer on Mission Control, thoughtbot's DevOps and SRE team and serves as a Scholarship program organizer with Ruby Central. Mina details why it is humbling to embrace the cloud and what SRE means. She and Brittany then share their hot takes consulting versus working on a product team. Show Notes & Links: thoughtbot Mission Control (https://thoughtbot.com/mission-control) Site Reliability Engineering: Google (https://sre.google/) Mina Slater (@minar528) on Twitter (https://twitter.com/minar528) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

cloud consulting scholarships monitor devops slater honey badger sre mission control sidekiq ruby central

Episode 429: Rubber Duck Dev Show with Chris Booth and Creston Jamison

booth monitor devops honey badger rubber ducks creston sidekiq

Play Episode Listen Later Aug 3, 2022 53:11

Brittany guested on the Rubber Duck Dev Show, a livestream with two Rubyists (Chris & Creston) who chat live about all the aspects of development. They talk over the origins of how Brittany became the host of The Ruby on Rails Podcast and their thoughts and struggles on project management. Show Notes & Links: Rubber Duck Dev Show | Wednesdays at 8p EST (https://www.rubberduckdevshow.com/) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

Episode 427: Nick is an Influencer (Brittany + Nick)

japanese influencers monitor devops rails honey badger naming things sidekiq rubyists

Play Episode Listen Later Jul 20, 2022 29:15

Nick is officially an influencer! After Nick accidentally launched a co-host of Rubyists learning Japanese (including Brittany!), the duo chat about learning styles and recommended podcasts. Oh, and upgrade your Rails applications, listeners! Show Notes & Links: Duolingo (https://www.duolingo.com/) akr / all-ruby (https://github.com/akr/all-ruby) Leetcode (https://leetcode.com/) Naming Things (https://www.namingthings.org/) Rubber Duck Dev Show (https://www.rubberduckdevshow.com/) Rails Versions 7.0.3.1, 6.1.6.1, 6.0.5.1, and 5.2.8.1 have been released! (https://rubyonrails.org/2022/7/12/Rails-Versions-7-0-3-1-6-1-6-1-6-0-5-1-and-5-2-8-1-have-been-released) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast).

Episode 426: Neurodiversity and Starting a Podcast with Julie J

code atlantis monitor neurodiversity starting a podcast codecademy code academy skillsoft sidekiq

Play Episode Listen Later Jul 13, 2022 30:32

Making her podcast debut, Julie J is a junior engineer at Codecademy from Skillsoft. Julie answers Brittany's questions about how she learned to code, her role at Codeacademy and her involvement in the Neurodivergent ERG at Codecademy. Then, Julie reveals her plans for a new podcast in the Ruby community! Show Notes & Links: Codeacademy (https://www.codecademy.com) Code in Place (https://codeinplace.stanford.edu/) All Aboard Bootcamp (https://allaboardbootcamp.com/) Julie (@codewithjulie) / Twitter (https://twitter.com/codewithjulie) Ruby for All (@rubyforall) / Twitter (https://twitter.com/rubyforall) Sponsored By: AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast). Atlantis Technology (https://www.atlantistech.com/careers) Atlantis is looking for great engineers! Why work at Atlantis? You'll work with great people. You'll work on projects that change the world. No matter where you are in your career, they're prepared to help you advance it. Find out more here (https://www.atlantistech.com/careers).

343: Opt-In To Oversharing

Play Episode Listen Later Jun 21, 2022 30:31

Chris is weathering through a slight lull, a holding period, where his team waits for new features to become available with some of the platforms they integrate with, and as they think out new facets of the platform they're building. Steph has been thinking recently about working in isolation. It's a topic that Joël Quenneville pointed out to her and mentioned. Can engineers work in isolation and be successful? Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: CHRIS: Always be singing. STEPH: I can't remember if I've shared the story with you. But I had a beautiful little human moment with someone at airport security. Because when I travel with my mic, I always get stopped because there's the middle long, thin piece that looks like what you would screw on to a gun like for a silencer. And so [laughs] as he was going through, the person was looking at it, and then he called over a buddy. And then they called over another buddy, and there's like three TSA agents all looking at the X-ray screen. And finally, they're like, "Yeah, we need to flag it." So they moved it over. And then he was digging through, and he pulled out the big metal piece. And I said, "It's for a microphone." And he's like, "Okay," and he kept looking, and then he finally found the microphone. And he lit up because I guess he wasn't really sure to believe me at first when I said it. But he lit up, and he was like, "Karaoke?" [laughs] I was like, "No, it's for podcasting." CHRIS: But not 100% no because we do sing plenty on this show, so... STEPH: I think that's what made me think of it. It was your singing. [laughs] CHRIS: Yep. My wonderful, wonderful singing. STEPH: Hello and welcome to another episode of The Bike Shed, a weekly Podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So, hey, Chris? What's new in your world? CHRIS: What's new in my world? We are in sort of a...what's the word? There's a bit of a lull right now, not like a big lull, but we had a bunch of clear work that came into the team, did a bunch of iterations, some testing, built some new features, et cetera. And now there's a small holding period basically where we wait for some new features to become available with some of the platforms that we integrate with and also as we think out some new facets of the platform that we're building. So we've got this little bit of time here where we're not necessarily building out as many new novel features. But instead, as a dev team, we're taking this moment to be like, oh, cool, let's tie down. I want to make a sailing analogy here, but I don't know sailing. It's like tie down the somethings and batten the hatches, maybe. That sounds like a thing. [chuckles] But so we have a couple of projects going right now. We want to really accept the truth and lean into Sidekiq. So right now, we have a mix of ActiveJob and Sidekiq jobs. And they're confusing, and et cetera, et cetera. So we want to kind of lean into that, upgrade dependencies, that sort of thing. We are, again, doing a little bit of work on the observability foundation of our system. so how do we know what's going on at runtime? Also, working on just some core features and functionality. We have done a little bit of an exploration into the event processing stuff, some of that that I've been talking about. It's actually been very interesting. So we're working with Customer.io as a platform, which is omnichannel communication behavior-based messaging sort of thing. So when a user does X, send them an email and then wait three days. And if they haven't responded, then do this other thing. And I think I've said this in previous episodes; I'm so wildly impressed with that platform. They have done such a good job. And I know that good software doesn't happen in a vacuum. In fact, if we're being honest, a lot of the software out there is not very good. And not only do they do a good job, but it's across...there's a ton of functionality in Customer.io. And it's interesting because we're finding ourselves leaning into it even more because it is such a solid platform and because it connects into our event system. Like, it's a segment destination, so all of our analytics events get piped into Customer.io, and then we can action on any of them. And the actions can be quite complicated. And this is where we're getting into good idea, terrible idea space. And to be clear, this is still just an exploration. But we basically wanted a way to do more. There are a bunch of different actions that you can take so, like send an email, send an SMS, or there are a couple of other slightly fancier ones. You can trigger an event within the Customer.io system. You can actually do an arbitrary HTTP POST, PUT, PATCH, whatever, any web requests you want to make. So if you want to integrate with essentially anything else out there, you can do that. You can send some structured data over the wire. And so we've now been like, okay, what if, and stay with me here, what if we use our analytic system and we send events whenever a user does something, and then that event eventually trickles down to Customer.io? Within that, we allow ourselves to respond to that event by emitting a different event within the system, within Customer.io. And then, via the webhook functionality, we fire that back to the Rails application. And then there we can do whatever we want. And in a way, that sounds absurd because we're starting from our app, and then we're sending some events down, processing them in certain ways, sending it back to the app, and then maybe doing something. In particular, one of the things we want to do is richly formatted Slack alerts. And Customer.io has a Slack alert functionality, but they can't have any of the fancy stuff. They can't link to our customer in the admin dashboard. So we found that that functionality is particularly useful for our admin team. And so we're like, ah, this feels weird. But if we were to do this loop out and back, then ideally, we get the power of Customer.io for non-technical users or non-engineering team users to configure workflows and to say, "When a user does this, I actually want to alert the admin team via Slack." And we want it to be rich and have buttons that you can click and all that kind of stuff. And although the thing that I just described seems complicated, is a word that I'll use for it, confusing at times, it isn't...like, I don't want to do all of that in the app. I don't want the app to have to think about how do I wait three days? We technically can do that with Sidekiq, but it gets us in trouble and whatnot, whereas Customer.io that's a core concept for them. And so, again, very much exploration. This will probably be a future good idea, terrible idea segment. But that's been an interesting one to explore. STEPH: You have quite a talent for you preface something as a bad idea, and you do a very good job of making it sound reasonable and good. [laughs] So it's interesting to be on that side of like, good idea, bad idea. It's like, I'm looking for the bad. And I have questions, but overall, [chuckles] you do a very good job of being very thoughtful and walking through why it makes sense or what are the benefits of it. So you answered some of my questions around why still send it to Customer.io versus just having it all in-house. So the fact that the admin team has access to it makes a lot of sense. I want to clarify one point. So when you send it to Customer.io, Customer.io then needs to send a message back to your application. And then that's when you customize the Slack message. Do you need Customer.io to send that message, or could you just fire off an event to Customer.io to say, "Hey, capture this, but don't do anything with this. And then we're going to send the Slack message because we want to customize it."? CHRIS: I think the key is that we want to leverage the fact that Customer.io is the platform that our operations team really is now becoming comfortable with and using for this behavioral automation workflow type logic. So that idea of when this event, you know, when this triggering event happens, if this condition is true, then respond in this way. And so because Customer.io is the platform that A, is quite good at that and B, is where our admin team is now thinking about doing that, one thing that we might do let's say a user completes some action within the application. So they fill out a form to submit their interest in some new platform feature. Initially, what we might want to do there is alert ourselves to say, "Hey, this happened. Take some action." And then eventually, we may want to instead switch that over and send an email to the customer with the next steps that they need to do. And the ability to gradually transition across that spectrum is really interesting to me, and again, Customer.io being the platform, sort of the hub for how we respond to these events. At the same time, I know that this feels like a generic message processing system that might be a Kafka queue somewhere else. And so I've got that in the back of my head of like, is this weird? I think it's a little weird. But it also, thus far as we're exploring it, is very approachable for the admin team, very familiar for them, and reasonably powerful. And also, there's a drag and drop editor for the events and the payloads. And it knows for this event, here's the stuff that's available to you. And so the ability for our admin team to interact with that interface is really great. And we don't have to build it. We don't have to think about it. But I will say I've worked at so many different companies that have their ad hoc system that makes it easy to do generic X, Y, and Z. And it's bad, and it falls down. And it's impossible to know when anything happens. And so, I've got a lot of concerns in the back of my head, which I will want to at least think through and understand the trade-offs that we're making if we pursue this path, but it is very interesting to me. So right now, a lot of this logic does live in the app. But it means that it requires a code change for anything that we want to do like this. We want to have a Slack alert whenever X happens. Now, the developers are in the loop for all of that. And really, it's the operations team that owns the decisioning on that. And so if they can also self-serve and instrument the action, the alert, the follow-up, the whatever it is, if we can give them those primitives in a platform that they already understand, that sounds nice. I'm intrigued, is what I'll say. So anyway, while we're in this lull period, we are trying out some fun stuff like that and exploring those sorts of things. STEPH: I like that perspective that you're putting on it, or at least the one that's standing out to me is the concept of ownership is like who gets to own these actions. But then beyond that, that's the part where I feel a little squirmy is, so we are using this third-party tool because it makes life easier. But then, at what point when we start building software around this third-party tool to then customize it back on our own side. Then if someone is in Customer.io, so if an admin user is in there and then they trigger an event, is there going to be confusion as to what's going to happen? And can they retry an event? Because I'm realizing my initial suggestion where it was like, hey, notify Customer.io that this is there but then also manage sending the Slack message that would prevent them from being able to have that retry capability. And that may be very much worth preserving. So then it's understood that hey, if you want to manage this, we are giving you full access to manage this work. We may customize it, but this is still the interface in which you go through to have three tries or to manage that workflow or these actions that get sent to users. CHRIS: Yeah. I think you've perfectly highlighted the why this might not be a great idea or at least the concerns to explore before adopting this more thoroughly. And even just the idea of adopting it more thoroughly, like, how tied into the system are we? How business-critical does this new external piece of software become? Because I've seen that to be really problematic where there are organizations that I've worked with that are like, "Oh God, we would love to move off of system X. But unfortunately, it's basically the one thing holding this business up." And I'm like, yeah, I get that. And that happens. So yeah, being really intentional with that. And that's why we're very much in an exploration place. But we have a bunch of stuff that we've done that required engineering work. And we're now seeing like, actually, could we map this into this other tool? And can we build the set of primitives in that space that now this team can own that whole experience? And then critically, can they debug it? Will we know when something goes wrong, et cetera? Those are always parts. At this point, I don't think I can just imagine a happy path. And I hope this isn't true for the rest of my life. But the work as a software developer, especially after having done a couple of rounds of it and as a consultant, I just imagine failure modes. It's all I do. I'll be like, okay, we just need to wire X up to Z, and then we need to fire off a request. And then, once we get the message back, then we can process them. I'm like, right. You just described 13 things that can go wrong. Now let's imagine each of the different failure states because that's all I'm going to do. Who cares about the happy path? Those are easy. Those write themselves. It's all of the failure modes that I need to think about. And someday, when I retire, and I go to a log cabin in the woods, and I don't talk to people for a while, maybe I'll go back to a place of only happy paths. But that is not my truth right now. STEPH: I can't tell you how many people in my personal life I have annoyed so much [laughs] because all I see are failure modes. And one, that's a delightful t-shirt. [laughs] I'd love to have that. And then yeah, I feel you because there are so many times where someone is...like, I'm with someone who's like a big idea person. And so they're just launching into what-ifs, and we did this. And I can't help it, and I have learned to help it. But it has been a struggle with some strong feedback from family and friends to reel it in. Because then I will start to think through okay, well, what's the details? And I have some questions. What happens when this happens? And yeah, all I see are failure modes. [laughs] It is very true for me too, and not always...not so great. So I, too, shall get a log cabin one day and try to forget all of that. CHRIS: I will say I painted that as a particularly glib version of myself. But some of what I'm doing right now, particularly joining an early-stage startup and taking the role of CTO, was very much to try and intentionally resist that. Because right now, I have to be really careful with how much of the potential edge cases and whatnot. I'm considering exactly how robust of a platform are we building? Very is the answer. But what about extremely? Because extremely is an option but extremely costs four times as much. Mostly in time being the critical element there. And so part of the work that I'm doing now is just trying to push on those edges, push on those boundaries, find the places where we can move quickly, and still build a robust platform because frankly, we're building...Sagewell is a financial platform under the hood, and I can't be flippant with that. We as a team have to be really careful with the thing that we're building. But we also have to move quickly. We have to be able to iterate. We have to be able to build something and try it out and see if it works. And then, if it doesn't, maybe shelve it and pull it out of the codebase. And it has been a real challenge, but it was the challenge that I wanted here. And so I've been enjoying that work, but it has been a stretch, a growth moment, let's call it. STEPH: I don't know if you've shared that particular goal with me in transitioning to a CTO role, but I really, really like it. One, it's very aligned with who you are. You're very thoughtful, and you look for areas to push and ways to do that. And then I also struggle in those areas, and thoughtbot specifically and consulting has helped push me in directions, push me out of my comfort zones but still in a safe space where I have other people to talk to as I'm making those decisions and pushing past the comfort areas that I have. But one of them is that I will initially think things have to be perfect or really planned. And I had a really nice conversation with Chad Pytel, who is one of the Founders of thoughtbot and also COO and host of the Giant Robots Smashing Into Other Giant Robots Podcast. And we were chatting about a new offering that thoughtbot is bringing to the market. And it's one that I've been involved with. And I started getting really in the weeds of like, but we really have to plan out how this is going to look and all the actions that need to take place before then we can really sell this type of engagement to a new client. And as I was going through this list of worries, when I was done, he mentioned he's like, "All of those are valid and something to consider." He's like, "But we don't have any customers yet." So the first part is we feel that we are in a space that we have enough of information to get started. And it's something that we've done before. And then, we'd like to see where customers align with us on this need because we're going to end up shaping this work in response to what their needs are. And so, we can't really begin that shaping until we understand more of what people are looking for. I was like, oh yeah, that's such a nice point. It just reminded me in regard to pushing those boundaries of yes, we need planning upfront, and we look for failure modes. But then there's also an important aspect of then finding ways to keep moving forward and getting more feedback and then balancing those two. CHRIS: Yeah, I think that's definitely right the as always, anchoring it to the customer. What is it that they need? How do we connect with them and hear from them? And ideally, keep those feedback loops as short as possible. That's the game, and everything else fits around that. But yeah, so we're trying some stuff. We'll see how it goes. I will certainly report back, depending on how it plays out. But that's a little bit of what's up in my world. What's up in your world? STEPH: I have been thinking recently about working in isolation. It's a topic that Joël Quenneville, who's another thoughtboter and has been on the show a number of times, it was a topic that he'd actually pointed out to me and mentioned. And so, I wanted to bring that here and share it with you because I'd love to get some of your thoughts on this as well. But I've typically had the viewpoint that when developers are sent off to work on a large, nebulous task, that it's a recipe for disaster, and almost everyone's going to lose in that scenario. And it tends to be a combination of isolation, very distant due dates, and loosely defined scope that leads to those really poor results. However, as developers, it's not inconceivable for us to land in that position. And it's very similar to my current project, who I'm working with Joël on, where we were given a very fuzzy project with some really aggressive goals, and the engagement is going really well. So that led Joël and I to wonder why is this working? This is the thing that we said that people should never do, but it's actually going quite well for us. So reflecting upon some of the things that are working well for us, even though we are in more of an isolated state than we would typically work, some of the things that I've been reflecting on or some of the strategies I should say that we've applied to this situation is number one, we did work hard to plug into an existing team. So when we joined, we joined more of an ad hoc volunteer team. And in everybody's spare time, those individuals were then contributing to the CI process in terms of trying to speed things up and improve things for the rest of the team. But otherwise, there wasn't really a team. There wasn't much structure to it. So it felt like everybody was very much off in their own world doing their own thing, occasionally putting up some code changes for review. And then you had to gain a lot of context to understand what it was that they were doing. So one of the things that I advocated for early on that I thought was more of just my personal preference but I think has actually worked well in regards to the success of the project as well is to plug into an existing team. So even if you are not working with that team on their day-to-day tasks, but you want to have more people to interact with and more people to share your context with. So you are essentially reducing the isolation of you're no longer these two people who are off in a corner working on something, and nobody has any idea what you're doing, and only one person is getting a status update. There is now a whole channel or team of people that have some insight as to what's going on. And they can also really unblock you for when you get stuck because then if you do have a question, but there's that one person who has been like your go-to person for this whole project, if they're out on vacation, or if they leave, or just something happens, you're suddenly blocked. And you don't know who to go to because you've been part of this larger company, but you haven't interacted with anybody outside of that one person. So at least if you're plugged into another team, you've immediately got some friends or some other people to go to and say, "Hey, I'm not sure who can help me with this, but I have this problem." And then, from there, you can get more help. CHRIS: This is super interesting. To start, I really like that you're framing this in terms of this is a thing that we often recommend against or see as an anti-pattern, and yet in this particular case, it's working. Let's look at that. Because I think the things that you're like, huh, that's interesting. That phrase "Huh, that's interesting" is very interesting. It often highlights like oh, something is behaving counter to how we would expect it to, so let's dig in and explore that. And so I love that that was the reaction and then sort of the conversation that spilled out of that. I'm also not super surprised that the combination of you and Joël were able to find a way to make this successful because you are two of the most capable developers that I've worked with but also particularly excellent communicators and advocates for the work that you're doing and the way that one should do the work. So the idea that there's a situation that may not be the ideal mode of working and that you're able to take that and say, "What if we shift it just a little bit and make it a little bit more manageable and whatnot?" So unsurprised, frankly, that you found a way collectively to make this a little bit better. And then I think yeah, it sounds like you're doing the things...so just like, we're in isolation, hmm, that doesn't seem great. Let's unisolate and connect to some people, and that just feels so true. I'm very interested to hear, though. I'm guessing there's more to this story or other things that you've done. Are there other tactics or ways that you've shifted this around? STEPH: Yeah, there's a couple more. So this is one that (And thank you for the kind words.) this was one that I think Joël is really exceptional at. So Joël is really good at building diagrams and graphs and then sharing that with the team as sort of like we've spent a couple of days understanding this big, messy concept. Here's a nice condensed graph that shows how we went about understanding this. And then here's the big overall picture of what we've learned from this, which has been wonderful for so many reasons. And every time that we share something with the team, one, it just helps build camaraderie, especially in remote days, it just builds camaraderie on hey, we're all online. And we're working. And here's the thing that I'm working through or struggling with or something that I learned. I often do that, especially when I get frustrated and something goes wrong. I love to share the I did this today. It went terribly. [laughs] Let me tell you about it, so you're aware of it in case it helps you. And specifically, the diagrams are really nice because then other people can just see and appreciate it, or they can point something out that we didn't know. Or they'll see a different angle because they're more familiar with the system. So they can say, "Oh yeah, that totally makes sense," or "I had no idea that was happening." So that's been a really nice way to engage with the team. And so, essentially, the little title for that strategy is just overshare. Just share all the things that you're doing and find ways to make it digestible for the team so then they can go along on this big, nebulous journey with you. And you can also put it in threads so that way, you're not flooding a channel, but then people can opt-in to that oversharing if they would like more insight into the work that you're doing. CHRIS: Opt-in to that oversharing. [laughs] STEPH: Exactly. I mean, it's not forced oversharing; it's just it's here if people would like it. That was a really nice compliment that some other thoughtboters received from their client team is someone had mentioned that there's so much information that's getting shared from the thoughtboters that they had trouble keeping up. And they really liked that. They really appreciated that they could then go check out this channel or these threads and see exactly the type of work that was happening and the outcomes of it. And then they could just check it maybe beginning of the day, end of the day and get that knowledge dump. Some of the other strategies that we've used are giving ourselves mini-goals to accomplish as part of the larger, more nebulous task. So as we have this very large goal in mind, it's like, where's the small piece? Where's an entry point? What's a task or a goal that we can define? And then we want to break that down into what questions do we need to ask? How can we start moving in this direction? And we want to find something that has an answer. So each time that we start researching once we've gotten to that point...and this is hard. I feel like people may know that, but I should just say that this is hard to take something nebulous and then find the entry point and break down some goals. And that has been one of the wonderful parts of then having a buddy for this type of project because then we can bounce ideas off of each other. And we can also help the other person not go too deep into an area. Because I have definitely had moments where I've been very passionate about like, "We need to do this," and Joël is just like, do we? And I'm like, "Yeah." And he's like, "Do we though?" And I'm like, "I guess not. I just really, really want to." [laughs] It's been very helpful to have a partner balance some of those feelings. And once you can break down some of that amorphous problem into those smaller goals, then you can also create tickets, which is also a really nice way to then surface the work that you're doing. You can document how you're researching, document the question. And then once you have that question of what you're in search of, it's so nice because then once you find the answer, that's immediately a good moment to pause and reflect. So I think in a recent episode, we were chatting about this where Joël and I were trying to understand why the tests weren't being balanced properly across each process that was available. And we found the answer, and we started immediately digging into fixing it or solutions. And then it took us a moment to go back and say, "Actually, this ticket is really just about understanding the problem, not fixing the problem." And so that was a nice; now that we understand the problem, let's go back high-level to define our next goal from this big, nebulous task because maybe fixing that balancing is the right thing to do, but maybe not, and we just need to reconsider. So for that portion of breaking down a big, nebulous task and then identifying smaller tasks that you can achieve, time-boxing has been huge for us in regards of what's something that we can accomplish this week, or what's something we can accomplish today that will then move us forward? And then making sure that we are setting deadlines for ourselves. So normally, this is another area where it's like, huh, that's interesting. I'm a big believer in deadlines. But I do think self-imposed deadlines are really helpful. CHRIS: I'm intrigued to hear you say that you're not a big fan of deadlines because I assume we're actually more aligned on this. But deadlines that are arbitrary and also come with fixed scope and other immovable things, yes, those are the worst in the world. But deadlines that we set for ourselves, and then we use that as a mechanism to hone and refine the scope that we're going to get out the door by that deadline, I find those incredibly useful. And that sounds like that's the same sort of thing you have going on here is like by saying we're willing to expend this much to get a result, that defines the work going into it. STEPH: Yeah, that's fair. Everything that you said is true, too; in regards to, I'm realizing I default that when I hear the word deadline, I'm so used to teams having deadlines that are defined by other individuals that are not part of the work. And as you said, the scope has already been defined, and it can't be changed. And it's all of the bad things that then go with it. So when I think of deadlines, I immediately think of that type of deadline versus the more self-imposed, yes, we can revisit, yes, the team has bought in and understands why this is important. Those types of deadlines are very helpful. It's that first part that I default to that I think of immediately, and I need some reassurance that that is not the type of deadline that I'm looking at or being forced to meet. I have a very similar feeling for estimates. Like, those both fall in the same category for me is; as soon as I hear estimation and deadline, I get nervous. And then I just need to understand the purpose of both and who is setting both of those and the communication around them. And then what does that failure mode look like, the one that we're always looking for? So yeah, deadlines and estimations fit into that. Initially, I'm very hesitant and cautious, but I think they're both very good tools. CHRIS: Yeah, I feel like those are very closely related. And they're definitely tools that can be used for great good or for great evil. And so, ideally, we advocate for the great good usage. But more generally, I love, again, the sharing around the process and what's worked for you in this less typical or often somewhat problematic workflow. I will say, again, so I gave you the series of compliments earlier, and I stand by those compliments for you and Joël. But I think also the sort of related aspect is that you two are both quite senior, very capable, very comfortable suggesting changes, suggesting workflows. So I think the potential dangers of isolation are still very much there. And the fact that the two of you have been able to find a way to work more effectively and perhaps change the terms of things just a little bit to make this effective is A, unsurprising but B, not something that I would expect of every team. I think you've described a wonderful list of the specifics as to how you did that. And ideally, if folks that are perhaps a little earlier on in their career are sent out for a month with a wild project, and they're sent to do it in isolation, hopefully, they can borrow from that list. But again, I do think this is a thing that, from an organizational perspective, we should be very careful with when we're imposing this isolation on it because it takes two fantastic folks like you and Joël to break out of the shackles of it. STEPH: The more we're talking about this, the more apparent it's also becoming that I started with this; how do you manage isolation? And my answer is you get out of it. [laughs] Get out of isolation as quickly as possible. Someone thought it was a good idea to put you there or a good idea to structure it that way. Or maybe they didn't mean it intentionally, but that's how things then shook out. So that's really what a lot of those strategies are about is, then how do I get myself out of this corner that you put me in? Because nobody put Stephanie in a corner. So it's essentially that's all the strategies are looking for ways to say, hey, I'm isolated, but I really don't want to be, and it's dangerous for me to be isolated in this way. Even as a more senior capable developer, it's more likely that things could go wrong, and miscommunications, misaligned expectations. So I need to find ways to then bring the work that I'm doing to make it more relevant to other people on the team. So then we can have more overlap, or at least I can share a lot of the work that's being done. CHRIS: Yeah, absolutely. I think with that wonderful summary and, frankly, utterly fantastic movie reference, what do you think? Should we wrap up? STEPH: Let's do it. Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeee!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

founders coo cto slack sms karaoke patch tsa rails kafka oh god mandy moore oversharing quenneville chris yeah sidekiq chad pytel chris thanks chris toomey

342: Sky Icing

Play Episode Listen Later Jun 14, 2022 43:42

Another toaster strudel debate?! Plus, the results are in for the most listened-to podcast in the RoR community! :: drum roll :: Steph has a "Dear Gerrit" message to share. Chris has a follow-up on mobile app strategy. The Bike Shed: 328: Terrible Simplicity (https://www.bikeshed.fm/328) When To Fetch: Remixing React Router - Ryan Florence (https://www.youtube.com/watch?v=95B8mnhzoCM) Virtual Event - Save Time & Money with Discovery Sprints (https://thoughtbot.com/events/save-time-money-with-discovery) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: STEPH: thoughtbot's next virtual event "Save Time & Money with Discovery Sprints" is coming up on June 17th, from 2 - 3 PM Eastern. It's a discussion with team members from product management, design and development. From a developer perspective, topics will include how to plan a product's architecture, both the MVP and future version, how to lead a tech spikes into integrations and conduct a build vs buy reviews of third party providers. Head to thoughtbot.com/events to register, the event is June 17th 2 - 3 PM ET. Even if you can't make it, registering will get you on the list for the recording. CHRIS: We're the second-best. We're the second-best. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, what's new in your world? STEPH: I'm very happy to report that I picked up a treat from the store recently. So while I was in Boston and we were hanging out in person, we talked about Pop-Tarts because that always comes up as a debate, as it should. And then also Toaster Strudels came up, so I now have a package of Toaster Strudels, and those are legit. Pop-Tart or Toaster Strudel, I am team Toaster Strudel, which I know you're going to ask me about icing and if I put it on there, so go ahead. I'm going to pause. [laughs] CHRIS: It sounds like I don't even need to say anything. But yes, inquiring minds want to know. STEPH: I think that's also my very defensive response because yes, I put icing on my Toaster Strudel. CHRIS: How interesting. [laughs] STEPH: But it feels like a whole different class of pastry. So I'm very defensive about my stance on Pop-Tarts with no icing put Strudel with icing. CHRIS: A whole different class of pastry. Got it. Noted. Understood. So did you travel? Like, were these in your luggage that you flew back with? STEPH: [laughs] Oh no. They would be all gooey and melty. No, we bought them when we got back to North Carolina. Oh, that'd be a pro move; just pack little individual Strudels as your airplane snack. Ooh, I might start doing that now. That sounds like a great airplane snack. CHRIS: You got to be careful though if the icing, you know, if it's pressurized from ground level and then you get up there, and it explodes. And you gotta be careful. Or is it the reverse? It's lower pressure up in the plane. So it might explode. STEPH: [laughs] Either way, it might explode. CHRIS: Well, yeah. If you somehow buy a packet of icing that is sky icing that is at that pressure, and you bring it down, then...but if you take it up and down, I think it's fine. If you open it at the top, you might be in danger. If you open icing under the ocean, I think nothing's going to happen. So these are the ranges that we're playing with. STEPH: I will be very careful sky icing and probably pack two so that way I have a backup just in case. So if one explodes, we'll be like, all right, now I know what I'm working with and be more prepared for the next one. CHRIS: That's just smart. STEPH: I try to make smart travel decisions, Toaster Strudels on the go. Aside from travel treats and sky icing, I have some news regarding Planet Argon, who is a Ruby on Rails consultancy regarding their latest published this year's Ruby on Rails community survey results. And so they list a lot of fabulous different topics in there. And one of them includes a learning section that highlights most listened to podcasts in the Ruby on Rails community as well as blogs and some other resources. And Bike Shed is listed as the second most listened to podcast in the Ruby on Rails community, so whoo, golf clap. CHRIS: Fantastic. STEPH: And in addition to that, the thoughtbot blog got a really nice shout-out. So the thoughtbot blog is in the number two spot for the most visited blogs in the community. In the first spot is Ruby Weekly, which is like, you know, okay, that feels fair, that feels good. So it's really exciting for the thoughtbot blog because a lot of people work really hard on curating and creating that content. So that's wonderful that so many people are enjoying it. And then I should also highlight that for the podcast in first place is Remote Ruby, so congrats to Chris, Jason, and Andrew for grabbing that number one spot. And Brittany Martin, host of the Ruby on Rails Podcast, along with Brian Mariani, Jemma Issroff, and Nick Schwaderer, are in the number three spot. And some people say that Ruby is losing steam but look at all that content and all those highly ranked podcasts. I mean, we like Ruby so much we're spending time recording ourselves talking about it. So I say long live Ruby, long live Rails. CHRIS: Yes. Long live Ruby indeed. And yeah, it's definitely an honor to be on the list and to be amongst such other wonderful shows. Certainly big fans of the work of those other podcasts. We even did a joint adventure with them at one point, and that was a really wonderful experience, so yeah, honored to be on the list alongside them. And to have folks out there in the world listening to our tech talk and nonsense always nice to hear. STEPH: Yeah. You and I show up and say lots of silly things and technical things into the podcast. The true heroes are the ones that went and voted. So thank you to everybody who voted. That's greatly appreciated. It's really nice feedback. Because we get listener responses and questions, and those are wonderful because it lets us know that people are listening. But I have to say that having the survey results is also really nice. It lets us know people like the show. Oh, but I did go back and look at some of the previous stats because then I was like, huh, so I'm paying attention. I looked at this year's, and I was like, I wonder what last year's was or the year before that. And I think this survey comes out every two years because I didn't see one for 2021. But I did find the survey results for 2020, which we were in the number one spot for 2020, and Remote Ruby was in the second spot. So I feel like now we've got a really nice, healthy podcasting war situation going on to see who can grab the first spot. We've got two years, everybody, to see who [laughs] grabs the number one spot. That's a lot of prep time for a competition. CHRIS: Yeah, I feel like we should be like, I don't know, planning elaborate pranks on them or something like that now. Is that where this is at? It's something like that, I think. STEPH: I think so. I think this is where you put like sky frosting inside someone's suitcase, and that's the type of prank that you play. [laughs] CHRIS: The best of pranks. STEPH: We'll definitely put together a little task force. And we'll start thinking of pranks that we all need to start playing on each other for the podcasting wars that we're entering for the next few years. But anywho, what's going on in your world? CHRIS: Let's see, what's going on in my world? A fun thing happened recently. I had a chance to reflect back on some architectural choices that we've made in the Sagewell platform. And one of those specific choices is how we've approached building our native mobile apps. We made what some listeners may remember is an interesting set of choices. In particular, in Episode 328, which we'll include a link to in the show notes, I shared with you the approach that we're doing, which is basically like, Inertia is great, web user great. We like the web as a platform. What if we were to wrap it in a native shell and find this interesting and somewhat unique hybrid trade-off point? And so, at that point, we were building it. We had most of it built out, and things were going quite well. I think we maybe had the iOS app in the store and the Android app approaching the store or something like that. At this point, both apps have been released to the store, so they are live. Production users are signing in. It's wonderful. But I had a moment in the past couple of weeks to reassess or look at that set of choices and evaluate it. And thankfully, I'm happy with the choices that we've made. So that's good. But to get into the specifics, there were two things that happened that really, really framed the choice that we made, so one was we introduced a major new feature. We basically overhauled the first-run experience, the onboarding that users experience, and added a new, pretty fundamental facet to the platform. It's a bunch of new screens, and flows, and error states, and all of this complexity. And in the process, we iterated on it a bunch. Like, first, it looked like this, and then we changed the order of the screens and switched out the error messages, and et cetera, et cetera. And I'll be honest, we never even thought about the mobile apps. It just wasn't even a consideration. And interestingly, we did as a final check before going fully live and releasing this out to the full production audience; we did spot check it in the mobile apps, and it didn't work. But it didn't work for a very specific, boring, technical reason that we were able to resolve. It has to do with iframes and WebViews and embedded something, something. And we had to set a flag. Thankfully, it was solvable without a deploy of the native mobile apps. And otherwise, we never thought about the native apps. Specifically, we were able to add this fundamental set of features to our platform. And they just worked in native mobile. And they were the same as they roughly are if you're on a mobile WebView or if you're on a desktop web, you know, slightly different in terms of form factor. But the functionality was all the same. And critically, the error states and the edge cases and the flow, there's so much to think about when you're adding a nontrivial feature to an app. And the fact that we didn't have to consider it really spoke to the choice that we made here. And again, to name it, the choice that we made is we're basically just reusing the same WebViews, the same Rails controllers, and the same what are Svelte components under the hood but the same essentially view layer as well. And we are wrapping that in a native iOS. It's a Swift application shell, and on Android, it's a Kotlin application shell. But under the hood, it's the same web stuff. And that was really great. We just got these new features. And you know what? If we have to rip that whole set of functionality out, again, we won't need to deploy. We won't need to rethink it. Or, if we want to subtly tweak it, we can do that. If we want to think about feature flags or analytics, or error states or error reporting, all of this just naturally falls out of the approach that we took. And that was really wonderful. STEPH: That's super nice. I also love this saga of like, you made a choice, and then you're coming back to revisit and share how it's going. So as someone who's never done this before, in regards of wrapping an application in the manner that you have and then publishing it and distributing it that way, what does that process look like? Is this one of those like you run a command, and literally, it's going to wrap the application and then make it hostable on the different mobile app stores? Or what's that? Am I oversimplifying the process? What does that look like? CHRIS: I think there are a lot of platforms or frameworks I think would probably be the better word like Capacitor is something that comes to mind or Ionic or Expo. There are a handful of them that are a little more fully featured in what they provide. So you just point us at your React Views and whatnot, and we'll wrap that up, and it'll be great. But those are for, I may be overgeneralizing here, but my understanding is those are for more heavy client-side bundles that are talking to a common API. And so you're basically taking your same rich client-side application and bundling that up for reuse on the native app, the native app platforms. And so I think those do have some release to the store sort of thing. In our case, we went a little bit further with that integration wrapper thing that we built. So that is a thing that we maintain. We have a Sagewell iOS repo and a Sagewell Android repo. There's a bunch of Swift and Kotlin code, respectively, in each of them, and we deploy to the stores manually. We're doing that whole process. But critically, the code that is in each of those repositories is just the bridge glue code that says, oh, when this Inertia navigation event happens, I'm going to push a WebView to the navigation stack. And that's what that is. I'm going to render the tab bar of buttons at the bottom with the navigation elements that I get from the server. But it's very much server-driven UI, is the way that I would describe it. And it's wrapping WebViews versus actually having the whole client bundle wrapped up in the thing. It's unfortunately subtle to try and talk through on the radio, but yeah. [laughs] STEPH: You're doing great; this is helping. So if there's a change that you want to make, you go to the Rails application, and you make that change. And then do you need to update anything on that iOS repo? It sounds like you don't, which then you don't have to push a new update to the store. CHRIS: Correct. For the vast majority of things, we do not need to make any changes. It's very rare for us to deploy the iOS or the Android app is a different way to put it or to push new releases to the store. It happens we may want to add a new feature to the sort of bridge layer that we built, but increasingly, those are rare. And now it's basically like, yeah, we're just wrapping those WebViews, and it's going great. And again, to name it, it's a trade-off. It's an intentional trade-off that we've made. We're never going to have the richest, most deep platform integration, smooth experience. We are making a small trade-off on that front. But given where we're at as an organization, given how early we are, how much iteration and change, we chose an architecture that optimizes for that change. And so again, like what you just said, yeah, I can...you know how it's really nice to be able to deploy six times a day on a web app, and that's a very straightforward thing to do? It is not so straightforward in the native mobile world. And so, we now have afforded ourselves the ability to do that. But critically, and this is the fun part in my mind, have the trade-offs in the controls. So if we were just like, it's just a WebView, and that's it, and we put it in the stores, and we're done, that is too far of an extreme in my mind. I think the performance trade-offs, the experience trade-offs, it wouldn't feel like a native app like in a deep way, in a problematic way. And so as an example, we have a navigation bar at the top of our app, particularly on iOS, that is native iOS navigation. And we have a tab bar at the bottom, which is native tab UI element. I forget actually what it's called, but it's those elements. And we hide the web application navigation when we're in the mobile context. So we actually swap those out and say, like, let's actually promote these to formal native functionality. We also, within our UI on the web, have a persistent button in the top right corner of your screen that says, "Need help? Reach out to your retirement advocate." who is the person that you get to work with. You can send questions, et cetera, et cetera. It's this little help sidebar drawer thing that pops out. And we have that as a persistent HTML button in the top corner of the web frame. But when we're on native, we push that up as a distinct element in the native UI section. And then again, the bridge that I'm talking about allows for bi-directional communication between the JavaScript side and the native side or the native side and the JavaScript side. And so it's those sorts of pieces that have now afforded us all of the freedom to tinker, and we don't need to re-release where we're like, oh, we want to add a new weird button that does a thing in the WebView when you click on a button outside the WebView. We now just have that built-in. STEPH: Yeah, I really like the flexibility that you're describing. When you promoted those elements to be more native-friendly so, like the navigation or the footer or the little get help chat, is that something that then your team implemented in like the iOS or the Kotlin repo? Okay, I see you nodding, but other people can't see that, so...[laughs] CHRIS: Yeah. I was going to also say the words, but yes, those are now implemented as native parts. So the thing that we built isn't purely agnostic decoupled. It is Sagewell-specific; a lot of it is low-level. Like, let's say we want to wrap an Inertia app in a native mobile wrapper. Like, 90% of the code in it is that, but then there are little bits that are like, and put a button up there. And that button is the Sagewell button. And so it's not entirely decoupled from us. But it mostly is this agnostic bridge to connect things together. STEPH: Yeah, the way you're describing it sounds really nice in terms of you're able to get out the app quickly and have a mobile app quickly that works on both platforms, and then you're still able to deploy changes without having to push that. That was always my biggest mental, or emotional hurdle with the idea of mobile development was the concept of that you really had to batch everything together and then submit it for review and approval and then get it released. And then you got to hope people then upgrade and get the newest version. And it just felt like such a process, not that I ever did much of it. This was all just even watching like the mobile team and all the work that they had to do. And I had sympathy pains for them. But the fact that this approach allows you to avoid a lot of that but still have some nice, customized, more native elements. Yeah, I'm basically just recapping everything you said because I like all of it. CHRIS: Well, thank you, friend. Like I said, I've really enjoyed it, and similar to you, I'm addicted to the feedback loop of the web. It's beautiful. I can deploy ten times or however many I want. Anytime I want, I can push out a new version. And that ability to iterate, to test, to explore, to tweak, to not have to do as much formal testing upfront because I'm terrified that if a bug sneaks out, then, it'll take me two weeks to address it; it just is so, so freeing. And so to give that up moving into a native context. Perhaps I'm fighting too hard to hold on to my dream of the ability to rapidly iterate. But I really do believe in that and especially for where we're at as an organization right now. But, and a critical but here, again, it's a trade-off like anything else. And recently, I happened to be out about in the town, and I decided, oh, you know what? Let me open up the app. Let me see what it's like. And I wasn't on great internet. And so I open the app, and it loads because, you know, it's a native app, so it pops up. But then the thing that actually happened is a loading spinner in the middle of the screen and sort of a gray nothing for a little while until the server request to fetch the necessary UI elements to render the login screen appeared. And that experience was not great. In particular, that experience is core to the experience of using the app every single time. Every time you use it, you're going to have a bad time because we're re-downloading that UI element. And there's caching, and there's things that could happen there to help with that. But fundamentally, that experience is going to be a pretty common one. It's the first thing that you experience when you're opening the app. And so I noticed that and I chatted with the team, and I was like, hey, I feel like this is actually something that fixing this I think would really fundamentally move us along that spectrum of like, we've definitely made some trade-offs here. But overall, it feels snappy and like a native app. And so, we opted to prioritize work on a native login screen for both platforms. This also allows us to more deeply integrate. So particularly, we're going to get biometric logins like fingerprints or face scans, or whatever it is. But critically, it's that experience of like, I open the Sagewell native app on my iOS phone, and then it loads immediately. And then I show it my face like we do these days, and then it opens up and shows me everything that I want to see inside of it. And it's that first-run experience that feels worth the extra effort and the constraints. Because now that it's native mobile, that means in order to change it, we have to do a deploy, not a deploy, release; that's what they call it in the native world. [laughs] You can tell I'm well-versed in this ecosystem. But yeah, we're now choosing that trade-off. And what I really liked about this sort of set of things like the feature that we were able to just accidentally get for free on native because that's how this thing is built. And then likewise, the choice to opt into a fully native login screen like having that lever, having that control over I'm going to optimize for iteration generally, but where it's important, we want to optimize for performance and experience. And now we have this little slider that we can go back and forth. And frankly, we could choose to screen by screen just slowly replace everything in the app with true native WebViews backed by APIs. And we could Ship of Theseus style replace every element of the app with true native mobile things until none of the old bridge code exists. And our users, in theory, would never know. Having that flexibility is really nice given the trade-off and the choice that we've made. STEPH: You said a word there that I missed. You said ship something style. CHRIS: Ship of Theseus. STEPH: What is that? CHRIS: It's like an old biblical story, I want to say, but it's basically the idea of, like, you have the ship. And then some boards start to rot out, so replace those boards. And then the mast breaks, you replace the mast. And slowly, you've replaced every element on the ship. Is it still the same ship at that point? And so it's sort of a philosophical question. So if we replace every single view in this app with a native view, is it still the same map? Philosophers will philosophize about it forever, but whatever. As long as we get to keep iterating and shipping software, then I'm happy. STEPH: [laughs] Y'all philosophize. That's that word, right? CHRIS: Yeah. STEPH: And do your philosopher thing. We'll just keep building and shipping. CHRIS: I don't know if I pronounced it right. It's like either Theseus or Theseus, and I'm sure I said the wrong one. And now that I've said the other, I'm sure both of them are wrong somehow. It's like a USB where there's up and down, and yet somehow it takes three tries. So anyway, I may have mispronounced it, and I may be misattributing it, but that's the idea I was going for. STEPH: Well, given I wasn't even familiar with the word until just now, I'm going to give both pronunciations a thumbs up. I also really like how you decided that for the login screen, that's the area that you don't want people to wait because I agree if you're opening an application or opening...maybe it's the first time, maybe it's the 100th time. Who knows? But that feels important. Like, that needs to be snappy. I need to know it's responsive. And it builds trust from the minute that I clicked on that application. And if it takes a long time, I just immediately I'm like, what are y'all doing? Are y'all real? Do you know what you're doing over there? So I like how you focused on that experience. But then once I log in, like if something is slow to log me in, I will make up excuses for the application all day where I'm like, well, you know, maybe it's my connection. It's fine. I can wait for the next screen to load. That feels more reasonable. And it doesn't undermine my trust nearly as much as when I first click on the app. So that feels like a really nice trade-off as well, or at least a nice area that you've improved while still having those other trade-offs and benefits that you mentioned. CHRIS: To highlight it, you used a phrase there which I really liked. Like, it's building trust. If something's a little bit off in that first run experience every single time, then it kind of puts a question in the back of your head, maybe not even consciously. But you're just kind of looking at it, and you're like, what are you doing there? What are you up to, friend? Humans say to the apps they use on their phone. That's normal, right? When you talk... But to name it, we've also done a round of performance work throughout the app. And so there are a couple of layers to it. But it was work that we had planned for a while, but we kept deferring. But now that we're seeing more usage of the native apps, the native apps experience the same surface area of performance stuff but all the more so because they may be on degraded network connections, et cetera. And so this is another example where this whole thing kind of pays off. The performance work that we did affects everything. It affects the web. It's the same under the hood. It's let's reduce the network requests that we're making in the payloads that we're sending, particularly the network requests to upstream things, so like the banking partner that we're using and those APIs, like, collating all the data to then render the screen. Because of Inertia, we only have a single sort of back and forth conversation via the API as opposed to I think it's pretty common to have like seven different APIs and four different spinners on the screen. We're not doing that, none of that on my watch. [chuckles] But we minimize the background calls to the other parties that we're integrating with. And then, we reduce the payload of data that we're sending on each request. And each of those were like, we had to think about things and tweak and poke, but again it's uniform. So mobile web has that now, desktop web has that now. Android, iOS, they all just inherited it sort of that just happened one day without a deploy or release, without a release of either of the native mobile apps. We did deploy to the web to make that happen, but that's easy. I can do that a bunch of times a day. One last thing I want to share as we're on this topic of trade-offs and levers, there was a really great conference talk that I watched recently, which was Ryan Florence of remix.run also React Router fame if you're familiar with him from that. But he was talking about the most recent version of Remix, which is their meta framework on top of React. But they've done some really interesting stuff around processing data, fetching data, when and how to sequence that. And again, that thing that I talked about of nine different loading spinners on the screen, Remix is taking a very different approach but is targeting that same thing of like, that's not great for user experience. Cumulative layout shift being the actual number that you can monitor for this. But in that talk, there are features that they've added to Remix as a framework where you can just decide, like, do we wait for this or do we not? Do we make sure we have all of the data, or do we say, you know what? Actually, this is going to be below the fold. So it's okay to defer loading this until after we send down the first payload. And then we'll kick in, and we'll do it from the client-side. But it's this wonderful feature of the framework that they're adding in where there's basically just a keyword that you can add to sort of toggle that behavior. And again, it's this idea of like trade-offs. Are we okay with more layout shift, or are we okay with more waiting? Which is it that we're going to optimize for? And I really love that idea of putting that power very simply in the hands of the developers to make those trade-off decisions and optimize over time for what's important. So we'll share a link to that talk in the show notes as well. But it was very much in the same space of like, how do I have the power to decide and to change my mind over time? That's what I want. But yeah, with that, I think that's enough of me updating on the mobile app. I'll continue to share as new things happen. But again, I'm at this point very happy with where we're at. So yeah, it's been fun. But yeah, what else is up in your world? STEPH: I have a dear Gerrit message that I wrote earlier, so I want to share that with you. Gerrit is the system that we're using for when we push up code changes that then manages very similar in the competitive space of like GitHub and GitLab, and Bitbucket. And so the team that I'm working with we are using Gerrit. And Gerrit and I, you know, we get along for the most part. We've managed to have a working relationship. [chuckles] But this week, I wrote my dear Gerrit letter is that I really miss being able to tell a story with my commit messages. That is the biggest pain that I'm feeling right now. So for anyone that's less familiar or if you already are familiar with Gerrit, each change that Gerrit shows represents a single commit that's under review. And each change is identified by a Change-Id. So the basic concept of Gerrit is that you only have one commit per review. So if you were to translate that to GitHub terminology, every pull request is only going to have one commit, and so you really can't push up multiple. And so, where that has been causing me the most pain is I miss being able to tell a story. So like even simple stories that are like, hey, I removed something that's not used. I love separating that type of stuff into its own commit just so then people can see that as they're going through review. Now, before I merge, I'm likely to squash, and that doesn't feel important that it needs to be its own commit. That's really just for the reviewer so they can follow along for the changes. But the other one, I can slowly get over that one. Because essentially, the way I get around that is then when I do push up my code for review, is I then go through my change request, and then I just add comments. So I will highlight that line and say, "Hey, I'm removing this because it's not in use." And so, I found a workaround for that one. But the one I haven't found a workaround for is that I don't push up my local work very often because I love having lots of local, tiny, green commits so that way I can know the progress that I'm at. I know where I'm headed. Also, I have a safe space to roll back to, but then that means that I may have five or six commits that I have locally, but I haven't pushed up somewhere. And that is bothering me more and more hour by hour the more I think about it that I can't push stuff up because it makes me nervous. Because, I mean, usually, at least by the end of the day, I push everything up, so it's stored somewhere. And I don't have to worry about that work disappearing. Now I am working on a dev machine. So there is that aspect of it's technically...it's not even on my local machine. It is stored somewhere that I should still be able to access. CHRIS: What's a dev machine? The way you're saying it, it sounds like it's a virtual machine, not like a laptop. But what's a dev machine? STEPH: Good question. So the dev machine is a remote server or remote machine that then I am accessing, and then that's where I'm performing. That's where I'm writing all of my work. And then that's also kind of the benefit is everything is not local; it's controlled by the team. So then that also means that other teams, other individuals can help set up these environments for future developers. So then you have that consistency across everyone's working with the same Rails version, or gems, or has access to the same tools. So in that sense, my work isn't just on my laptop because then that would really worry me because then I've got nowhere...it's not backed up anywhere. So at least it is somewhere it's being stored that then could be accessed by someone. So actually, now, as I'm talking this through, that does help alleviate my concern about this a bit. [laughs] But I still miss it; I still miss being able to just push up my work and then have multiple commits. And I looked into it because I was like, well, maybe I'm misunderstanding something about Gerrit, and there's a way around this. And that's still always a chance. But from the research that I've done, it doesn't seem to be. And there are actually two very fiery takes that I saw that I have to share because they made me laugh. When I was Googling, the question of like, "Can I push up multiple commits to one single Gerrit CR? Or is there just a way to, like, can I have this concept of like a branch and then I have many commits, but then I turn it into one CR? Whatever the world would give me. What do they have? [laughs] I'm laughing just looking at this now. One of the responses was, have you tried squashing your commits into one commit? And I was like, [laughs] "Yeah, that's not what I had in mind, but sure." And then the other one, this is the more fiery take. They were very defensive about Gerrit, and they wrote that "People who don't like Gerrit usually just hack shit together. They cut corners and love squashing commits or throwing away history. And those people hate Gerrit. Developers who care love it. It's definitely possible and easy to produce agile software." And I just...that made me laugh. I was like, cool, I'm a developer that cuts corners and loves squashing commits. [laughs] CHRIS: So you don't care is what that take says. STEPH: I'm a developer who does not care. CHRIS: You know, Steph, I've worked with you for a while. And I've been looking for the opportunity to have this hard conversation with you. But I just wish you cared a little more about the software that you're writing, about the people that you're working with, about the commits that you're authoring. I just see it in every facet of your work. You just don't care. To be very clear for anyone listening at home, that is the deepest of sarcasm that I can make. Steph cares so very much. It's one of the things that I really enjoy about you. STEPH: I mean, we had the episode about toxic traits. This would have been the perfect time to confront me about my lack of caring about software and the processes that we have. So winding down on that saga, it seems to be the answer is no, friend; I cannot push up multiple commits. Oh, I tried to hack it. I am someone that tries to hack shit together because I tried to get around it just to see what would happen. [laughs] Because the docs had suggested that each change is identified by a Change-Id. And I was like, hmm, so what if there were two commits that had the same Change-Id, would Gerrit treat those as patch sets? Because right now, when you push up a change, you can see all the different patch sets, so that's nice. So that's a nice feature of Gerrit as you can see the history of, like, someone pushed up this change. They took in some feedback. They pushed up a new change. And so that history is there for each push that someone has provided. And I wondered maybe if they had the same Change-Id that then the patch sets would show the first commit and then the second commit. And so I manually altered the commits two of them to reference the same Change-Id. And I have to say, Gerrit was on to me because they gave me a very nice error message that said, "Same Change-Id and multiple changes. Squash the commits with the same Change-Ids or ensure Change-Ids are unique for each commit. And I thought, dang, Gerrit, you saw me coming. [laughs] So that didn't work either. I'm still in a world of where I now wait. I wait until I'm ready for someone to review stuff, and I have to squash everything, and then I go comment on my CRs to help out reviewers. CHRIS: I really like the emotional backdrop that you provided here where you're spending a minute; you're like, you know what? Maybe it's me. And there's the classic Seymour Skinner principle from The Simpsons. Am I out of touch? No, it's the children who are wrong. [laughs] And I liked that you took us on a whole tour of that. You're like, maybe it's me. I'll maybe read up. Nope, nope. So yeah, that's rough. There's a really interesting thing of tools constraining you. And then sometimes being like, I'm just going to yield control and back away and accept this thing that doesn't feel right to me. Like, Prettier does a bunch of stuff that I really don't like. It shapes code in a way, and I'm just like, no, that's not...nope, you know what? I've chosen to never care about this again. And there's so much utility in that choice. And so I've had that work out really well. Like with Prettier, that's a great example whereby yielding control over to this tool and just saying, you know what? Whatever you produce, that is our format; I don't care. And we're not going to talk about it, and that's that. That's been really useful for myself and for the teams that I'm on to just all kind of adopt that mindset and be like, yeah, no, it may not be what I would choose but whatever. And then we have nice formatted code; it's great. It happens automatically, love it. But then there are those times where I'm like; I tried to do that because I've had success with that mindset of being like, I know my natural thing is to try and micromanage and control every little bit of this code. But remember that time where it worked out really well for me to be like, I don't care, I'm just going to not care about this thing? And I try to not care about some stuff, which it sounds like that's what you're doing right here. [laughs] And you're like, I tried to not care, but I care. I care so much. And now you're in that [chuckles] complicated space. So I feel for you, Steph. I'm sorry you're in that complicated space of caring so much and not being able to turn that off [laughs] nor configure the software to do the thing you want. STEPH: I appreciate it. I should also share that the team that I'm working with they also don't love this. Like, they don't love Gerrit. So when I shared in the Slack channel my dear Gerrit message, they're both like, "Yeah, we feel you. [laughs] Like, we're in the same spot," which was also helpful because I just wanted to validate like, this is the pain I'm feeling. Is someone else doing something clever or different that I just don't know about? And so that was very helpful for them to say, "Nope, we feel you. We're in the same spot. And this is just the state that we're in." I think they have started transitioning some other repos over to GitLab and have several repos in Gitlab, but this one is still currently using Gerrit. So they very much commiserate with some of the things that I'm feeling and understand. And this does feel like one of those areas where I do care deeply. And frankly, this is one of those spaces that I do care about, but it's also like, I can work around it. There are some reasonable things that I can do, and it's fine as we just talked through. Like, the fact that my commits are not just locally on my machine already makes me feel better now that I've really processed that. So there are lower risks. It is more of just like a workflow. It's just, you know, it's crushing my work vibe. CHRIS: Harshing your buzz. STEPH: In the great words of Queen Elsa, I gotta let it go. This is the thing I'm letting go. So that's kind of what's going on in my world. What else is going on in your world? CHRIS: Well, first and foremost, fantastic reference and segue. I really liked that. But yeah, let's see, [laughs] what else is going on in my world? We had an interesting thing happen last week. So we had an outage on the platform last week. And then we had an incident review today, so a formal sort of post-mortem incident review. There are a couple of different names that folks have given to these. But this is a practice that we want to build within our engineering culture is when stuff goes wrong, we want to make sure that we have meaningful conversations around to try to address the root causes. Ideally, blameless is a word that gets used often in this context. And I've heard folks sort of take either side of that. Like, it's critical that it's blameless so that it doesn't feel like it's an attack. But also, like, I don't know, if one person did something, we should say that. So finding that gentle middle ground of having honest, real conversations but in a context of safety. Like, we're all going to make mistakes. We're all going to ship bugs; let's be clear about that. And so it's okay to sort of...anyway, that's about the process. We had an outage. The specific outage was that we have introduced a new process. This is a Sidekiq process to work off a specific queue. So we wanted that to have discrete treatment. That had been running, and then it stopped running; we still don't know why. So we never got to the root-root cause. Well, we know what the mechanism was, which was the dyno count for that process was at zero. And so, eventually, we found a bunch of jobs backed up in the Sidekiq admin. We're like, that's weird. And then, we went over to Heroku's configuration dashboard. And we saw, huh, that's weird. There are zero dynos processing this. That wasn't true yesterday. But unfortunately, Heroku doesn't log or have an audit trail around changes to those process counts. It's just not available. So that's unfortunate. And then the actual question of like, how did this happen? It probably had to be someone on the team. So there is like, someone did a thing. But that is almost immaterial because, again, people are going to do things, bugs will get shipped, et cetera. So the conversation very quickly turned to observability and understanding. I think we've done a pretty good job of instrumenting error reporting and being quite responsive to that, making sure the signal-to-noise ratio is very actionable. So if we see a bug or a Sentry alert come through, we're able to triage that pretty quickly, act on it where it is a real bug, understand where it's a bit of noise in the system, that sort of thing. But in this case, there were no errors. There was no Sentry. There was nothing; there was the absence of something. And so it was this really interesting case of that's where observability, I think, can really come in and help. So the idea of what can we do here? Well, we can monitor the count of jobs backed up in Sidekiq queue. That's one option. We could do some threshold alerting around the throughput of processed events coming from this other backend. There are a bunch of different ways, but it basically pushed us in the direction of doubling down and reinforcing the foundation of our observability within the platform. So we're just kicking that mini-project off now, but it is something we're like, yeah, we feel like we could add some here. In particular, we recently added Datadog to the stack. So we now have Datadog to aggregate our logs and ideally do some metric analysis, those sort of things, build some dashboards, et cetera. I haven't explored Datadog much thus far. But my sense is they've got the whiz-bang things that we need here. But yeah, it was an interesting outage. That wasn't fun. The incident conversation was actually a good conversation as a team. And then the outcome of like, how do we double down on observability? I'm actually quite excited for. STEPH: This is a fun moment for me because I have either joined teams that didn't have Datadog or have any of that sort of observability built into their system or that sort of dashboard that people go to. Or I've joined teams, and they already have it, and then nobody or people rarely look at it. And so I'm always intrigued between like what's that catalyst that then sparked a team to then go ahead and add this? And so I'm excited to hear you're in that moment of like, we need more observability. How do we go about this? And as soon as you said Datadog, I was like, yeah, that sounds nice because then it sounds like a place that you can check on to make sure that everything is still running. But then there's still also that manual process where I'm presuming unless there's something else you have in mind. There's still that manual process of someone has to check the dashboard; someone then has to understand if there's no count, no squiggly lines, that's a bad thing and to raise a concern. So I'm intrigued with my own initial reaction of, like, yeah, that sounds great. But now I'm also thinking about it still adds a lot of...the responsibility is still on a human to think of this thing and to go check it. Versus if there's something that gets sent to someone to alert you and say like, "Hey, this queue hasn't been processed in 48 hours. There may be a concern that actually feels nicer." It feels safer. CHRIS: Oh yeah, definitely. I think observability is this category of tools and workflows and whatnot. But I think what you're describing of proactive alerting that's the ideal. And so it would be wonderful if I never had to look at any of these tools ever. And I just knew if I got, let's say, it's PagerDuty connected up whatever, and I got a push notification from PagerDuty saying, "Hey, go look at this thing." That's all I ever need to think about. It's like, well, I haven't gotten a PagerDuty in a while, so everything must be fine, and having a deep trust in that. Similar to like, if we have a great test suite and it's green, I feel confident deploying the sort of absence of an alert being the thing that I can trust. But right now, we're early enough in this journey that I think what we need to do is stand up a bunch of these different graphs and charts and metric analysis and aggregations and whatnot, and then start to squint at it for a while and be like, which of these would I be really concerned if it started to wibble? And then you can figure the alerting around said wibble rate. And that's the dream. That's where we want to get to, but I think we've got to crawl, walk, run on this. So it'll be an adventure. This is very much the like; we're starting a thing. I'll tell you about it more when we've done it. But what you're describing is exactly what we want to get to. STEPH: I love wibble rate. That's my new measurement I'm going to start using for everything. It's funny, as you're bringing this up, it's making me think about the past week that Joël Quenneville and I have had with our client work. Because a somewhat similar situation came up in regards where something happened, and something was broken. And it seemed it was hard to define exactly what moment caused that to break and what was going on. But it had a big impact on the team because it essentially meant none of the bills were going through. And so that's a big situation when you got 100-plus people that are pushing up code and expecting some of the build processes to run. But it was one of those that the more we dug into it, the more it seemed very rare that it would happen. So, in this case, as a sort of a juxtaposition to your scenario, we actually took the opposite approach of where we're like; this is rare. But we did load up a lot of contexts. Actually, I was thinking back to the advice that you gave me in a previous episode where I was talking about at what point do you dig in versus try to stay at surface level? And this was one of those, like, we've spent a couple of days on getting context for this and understanding. So it felt really important and worthwhile to then invest a little bit more time to then document it. But then we still went with the simplest approach of like, this is weird. It shouldn't happen again. We think we understand it but then let's add a little bit of documentation or wiki page around like, hey, if you do run into this, here are some steps that will fix everything. And then, if you need to use this, let somebody know because this is so odd it shouldn't happen. So we took that approach in this case where we didn't increase the observability. It was more like we provided a fire extinguisher very close to the location in case it happens. And so that way, it's there should the need arise, but we're hoping it just never gets used. We're also in the process of changing how a lot of that logic works. So we didn't really want to optimize for observability into a system that is actively being changed because it should look very different in upcoming months. But overall, I love the conversations that you bring about observability, and I'm excited to hear about what wibble rates you decide to add to your Datadog dashboard. CHRIS: There's a delicate art and science to the selection of the wibble rates. So I will certainly report back as we get into that work. But with that, shall we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

Episode 421: I've Been Here For Awhile with David Hill

code react monitor devops honey badger awhile david hill hotwire staff software engineer railsconf sidekiq

Play Episode Listen Later Jun 8, 2022 30:31

David Hill is a Staff Software Engineer with CareRev. He just gave his first ever conference presentation at RailsConf 2022, titled “React-ing to Hotwire”. He is thinking about starting a podcast (yes!) so he asks Brittany her thoughts on co-hosts and topics. Show Notes & Links: React-ing to Hotwire | Railsconf 2022 (https://railsconf.org/program/sessions#session-1303) The Rails SaaS Conference (https://railssaas.com/) Framework Friends (https://www.frameworkfriends.com/) Code and the Coding Coders who Code it (https://podcast.drbragg.dev/) Railsconf Guide (https://railsconf.org/scholarships) David Hill on Twitter (@DavidEsmale) (https://twitter.com/DavidEsmale) Sponsored By: AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast). Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution.

341: Fundamentals and Weird Stuff

Play Episode Listen Later Jun 7, 2022 35:27

Steph and Chris are recording together! Like, in the same room, physically together. Chris talks about slowly evolving the architecture in an app they're working on and settling on directory structure. Steph's still working on migrating unit tests over to RSpec. They answer a listener question: "As senior-level developers, how do you set goals to ensure that you keep growing?" This episode is brought to you by BuildPulse (https://buildpulse.io/bikeshed). Start your 14-day free trial of BuildPulse today. Faking External Services In Tests With Adapters (https://thoughtbot.com/blog/faking-external-services-in-tests-with-adapters) Testing Third-Party Interactions (https://thoughtbot.com/blog/testing-third-party-interactions) Jen Dary - On Future Goals (https://www.beplucky.com/on-future-goals/) Charity Majors - The Engineer Manager Pedulum (https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/) Charity Majors Bike Shed Episode (https://www.bikeshed.fm/302) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: STEPH: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So, hey, Chris, what's new in your world? CHRIS: What is new in my world? Actually, this episode feels different. There's something different about it. I can't quite put my finger on it. I think it may be that we're actually physically in the same room recording for the first time in two years and a little bit more, which is wild. STEPH: I can't believe it's been that long. I feel like it wasn't that long ago that we were in The Bike Shed...oh, I said The Bike Shed studio. I'm being very biased. Our recording studio [laughs] is the more proper description for it. Yeah, two and a half years. And we tried to make this happen a couple of months ago when I was visiting Boston, and then it just didn't work out. But today, we made it. CHRIS: Today, we made it. Here we are. So hopefully, the audio sounds great, and we get all that more richness in conversation because of the physical in-person manner. We're trying it out. It'll be fun. But let's see, to the normal tech talk and nonsense, what's new in my world? So we've been slowly evolving the architecture in the app that we're working on. And we settled on something that I kind of like, so I wanted to talk about it, directory structure, probably the most interesting topic in the world. I think there's some good stuff in here. So we have the normal stuff. There are app models, app controllers, all those; those make sense. We have app jobs which right now, I would say, is in a state of flux. We're in the sad place where some things are application records, and some things are Sidekiq workers. We have made the decision to consolidate everything onto Sidekiq workers, which is just strictly more powerful as to the direction we're going to go. But for right now, I'm not super happy with the state of app jobs, but whatever, we have that. But the things that I like so we have app commands; I've talked about app commands before. Those are command objects. They use dry-rb do notation, and they allow us to sequence a bunch of things that all may fail, and we can process them all in a much more reasonable way. It's been really interesting exploring that, building on it, introducing it to new developers who haven't worked in that mode before. And everyone who's come into the project has both picked it up very quickly and enjoyed it, and found it to be a nice expressive mode. So app commands very happy with that. App queries is another one that we have. We've talked about this before, query objects. I know we're a big fan. [laughs] I got a golf clap across the room here, which I could see live in person. It was amazing. I could feel the wind wafting across the room from the golf clap. [chuckles] But yeah, query objects, they're fantastic. They take a relation, they return a relation, but they allow us to build more complex queries outside of our models. The new one, here we go. So this stuff would all normally fall into app services, which services don't mean anything. So we do not have an app services directory in our application. But the new one that we have is app clients. So these are all of our HTTP clients wrapping external third parties that we're interacting with. But with each of them, we've taken a particular structure, a particular approach. So for each of them, we're using the adapter pattern. There's a blog post on the Giant Robots blog that I can point to that sort of speaks to the adapter pattern that we're using here. But basically, in production mode, there is an HTTP backend that actually makes the real requests and does all that stuff. And in test mode, there is a test backend for each of these clients that allows us to build up a pretty representative fake, and so we're faking it up before the HTTP layer. But we found that that's a good trade-off for us. And then we can say, like, if this fake backend gets a request to /users, then we can respond in whatever way that we want. And overall, we found that pattern to be really fantastic. We've been very happy with it. So it's one more thing. All of them were just gathering in-app models. And so it was only very recently that we said no, no, these deserve their own name. They are a pattern. We've repeated this pattern a bunch. We like this pattern. We want to even embrace this pattern more, so long live app clients. STEPH: I love it. I love app clients. It's been a while since I've been on a project that had that directory. But there was a greenfield project that I was working on. I think it might have been I was working with Boston.rb and working on giving them a new site or something like that and introduced app clients. And what you just said is perfect in terms of you've identified a pattern, and then you captured that and gave it its own directory to say, "Hey, this is our pattern. We've established it, and we really like it." That sounds awesome. It's also really nice as someone who's new to a codebase; if I jump in and if I look at app clients, I can immediately see what are the third parties that we're working with? And that feels really nice. So yeah, that sounds great. I'm into it. CHRIS: Yeah, I think it really was the question of like, is this a pattern we want to embrace and highlight within the codebase, or is this sort of a duplication but irrelevant like not really that important? And we decided no, this is a thing that matters. We currently have 17 of these clients, so 17 different third-party external things that we're integrating with. So for someone who doesn't really like service-oriented architecture, I do seem to have found myself in a place. But here we are, you know, we do what we have to with what we're given. But yes, 17 and growing our app clients. STEPH: That is a lot. [laughs] My eyes widened a bit when you said 17. I'm curious because you highlighted that app services that's not really a thing. Like, it doesn't mean anything. It doesn't have the same meaning of the app queries directory or app commands or app clients where it's like, this is a pattern we've identified, and named, and want to propagate. For app services, I agree; it's that junk drawer. But I guess in some ways...well, I'm going to say something, and then I'm going to decide how I feel about it. That feels useful because then, if you have something but you haven't established a pattern for it, you need a place for it to go. It still needs to live somewhere. And you don't necessarily want to put it in app models. So I'm curious, where do you put stuff that doesn't have an established pattern yet? CHRIS: It's a good question. I think it's probably app models is our current answer. Like, these are things that model stuff. And I'm a big believer in the it doesn't need to be an application record-backed object to go on app models. But slowly, we've been taking stuff out. I think it'd be very common for what we talk about as query objects to just be methods in the respective application record. So the user record, as a great example, has all of these methods for doing any sort of query that you might want to do. And I'm a fan of extracting that out into this very specific place called app queries. Commands are now another thing that I think very typically would fall into the app services place. Jobs naming that is something different. Clients we've got serializers is another one that we have at the top level, so those are four. We use Blueprinter within the app. And again, it's sort of weird. We don't really have an API. We're using Inertia. So we are still serializing to JSON across the boundary. And we found it was useful to encapsulate that. And so we have serializers as a directory, but they just do that. We do have policies. We're using Pundit for authorization, so that's another one that we have. But yeah, I think the junk drawerness probably most goes to app models. But at this point, more and more, I feel like we have a place to put things. It's relatively clear should this be in a controller, or should this be in a query object, or should this be in a command? I think I'm finding a place of happiness that, frankly, I've been searching for for a long time. You could say my whole life I've been searching for this contented state of I think I know where stuff goes in the app, mostly, most of the time. I'm just going to say this, and now that you've asked the excellent question of like, yeah, but no, where are you hiding some stuff? I'm going to open up models. Next week I'm going to be like, oh, I forgot about all of that nonsense. But the things that we have defined I'm very happy with. STEPH: That feels really fair for app models. Because like you said, I agree that it doesn't need to be ActiveRecord-backed to go on app models. And so, if it needs to live somewhere, do you add a junk drawer, or do you just create app models and reuse that? And I think it makes a lot of sense to repurpose app models or to let things slide in there until you can extract them and let them live there until there's a pattern that you see. CHRIS: We do. There's one more that I find hilarious, which is app lib, which my understanding...I remember at one point having one of those afternoons where I'm just like, I thought stuff works, but stuff doesn't seem to work. I thought lib was a directory in Rails apps. And it was like, oh no, now we autoload only under the app. So you should put lib under app. And I was just like, okay, whatever. So we have app lib with very little in it. [laughs] But that isn't so much a junk drawer as it is stuff that's like, this doesn't feel specific to us. This goes somewhere else. This could be extracted from the app. But I just find it funny that we have an app lib. It just seems wrong. STEPH: That feels like one of those directories that I've just accepted. Like, it's everywhere. It's like in all the apps that I work in. And so I've become very accustomed to it, and I haven't given it the same thoughtfulness that I think you have. I'm just like, yeah, it's another place to look. It's another place to go find some stuff. And then if I'm adding to it, yeah, I don't think I've been as thoughtful about it. But that makes sense that it's kind of silly that we have it, and that becomes like the junk drawer. If you're not careful with it, that's where you stick things. CHRIS: I appreciate you're describing my point of view as thoughtfulness. I feel like I may actually be burdened with historical knowledge here because I worked on Rails apps long, long ago when lib didn't go in-app, and now it does. And I'm like, wait a minute, but like, no, no, it's fine. These are the libraries within your app. I can tell that story. So, again, thank you for saying that I was being thoughtful. I think I was just being persnickety, and get off my lawn is probably where I was at. STEPH: Oh, full persnicketiness. Ooh, that's tough to say. [laughs] CHRIS: But yeah, I just wanted to share that little summary, particularly the app clients is an interesting one. And again, I'll share the adapter pattern blog post because I think it's worked really well for us. And it's allowed us to slowly build up a more robust test suite. And so now our feature specs do a very good job of simulating the reality of the world while also dealing with the fact that we have these 17 external situations that we have to interact with. And so, how do you balance that VCR versus other things? We've talked about this a bunch of times on different episodes. But app clients has worked great with the adapter pattern, so once more, rounding out our organizational approach. But yeah, that's what's up in my world. What's up in your world? STEPH: So I have a small update to give. But before I do, you just made me think of something in regards to that article that talks about the adapter pattern. And there's also another article that's by Joël Quenneville that's testing third-party interactions. And he made me reflect on a time where I was giving the RSpec course, and we were talking about different ways to test third-party interactions. And there are a couple of different ways that are mentioned in this article. There are stub methods on adapter, stub HTTP request, stub request to fake adapter, and stub HTTP request to fake service. All that sounds like a lot. But if you read through the article, then it gives an example of each one. But I've found it really helpful that if you're in a space that you still don't feel great about testing third-party interactions and you're not sure which approach to take; if you work with one API and apply all four different strategies, it really helps cement how to work through that process and the different benefits of each approach and the trade-offs. And we did that during the RSpec course, and I found it really helpful just from the teacher perspective to go through each one. And there were some great questions and discussions that came out of it. So I wanted to put that plug out there in the world that if you're struggling testing third-party interactions, we'll include a link to this article. But I think that's a really solid way to build a great foundation of, like, I know how to test a third-party app. Let me choose which strategy that I want to use. Circling back to what's going on in my world, I am still working on migrating unit tests over to RSpec. It's a thing. It's part of the work that I do. [laughs] I can't say it's particularly enjoyable, but it will have a positive payoff. And along that journey, I've learned some things or rediscovered some things. One of them is read expectations very carefully. So when I was migrating a test over to RSpec, I read it as where we expected a record to exist. The test was actually testing that a record did not exist. And so I probably spent an hour understanding, going through the code being like, why isn't this record getting created like I expect it to? And I finally went back, and I took a break, and I went back. And I was like, oh, crap, I read the expectation wrong. So read expectations very carefully. The other one...this one's not learned, but it is reinforced. Mystery Guests are awful. So as I've been porting over the behavior over to RSpec, the other tests are using fixtures, and I'm moving that over to use factorybot instead. And at first, I was trying to be minimal with the data that I was bringing over. That failed pretty spectacularly. So I have learned now that I have to go and copy everything that's in the fixtures, and then I move it over to factorybot. And it's painful, but at least then I'm doing that thing that we talked about before where I'm trying to load as little context as possible for each test. But then once I do have a green test, I'm going back through it, and I'm like, okay, we probably don't care about when you were created. We probably don't care when you're updated because every field is set for every single record. So I am going back and then playing a game of if I remove this line, does the test still pass? And if I remove that line, does the test still pass? And so far, that has been painful, but it does have the benefit of then I'm removing some of the setup. So Mystery Guests are very painful. I've also discovered that custom error messages can be tricky because I brought over some tests. And some of these, I'm realizing, are more user error than anything else. Anywho, I added one of the custom error messages that you can add, and I added it over to RSpec. But I had written an incorrect, invalid statement in RSpec where I was looking...I was expecting for a record to exist. But I was using the find by instead of where. So you can call exist on the ActiveRecord relation but not on the actual record that gets returned. But I had the custom error message that was popping up and saying, "Oh, your record wasn't found," and I just kept getting that. So I was then diving through the code to understand again why my record wasn't found. And once I removed that custom error message, I realized that it was actually because of how I'd written the RSpec assertion that was wrong because then RSpec gave me a wonderful message that was like, hey, you're trying to call exist on this record, and you can't do that. Instead, you need to call it on a relation. So I've also learned don't bring over custom error messages until you have a green test, and even then, consider if it's helpful because, frankly, the custom error message wasn't that helpful. It was very similar to what RSpec was going to tell us in general. So there was really no need to add that custom step to it. For the final bit that I've learned or the hurdle that I've been facing is that migrating tests descriptions are hard unless they map over. So RSpec has the context and then a description for it that goes with the test. Test::Unit has methods like method names instead. So it may be something like test redemption codes, and then it runs through the code. Well, as I'm trying to migrate that knowledge over to RSpec, it's not clear to me what we're testing. Okay, we're testing redemption codes. What about them? Should they pass? Should they fail? Should they change? What are they redeeming? There's very little context. So a lot of my tests are copying that method name, so I know which tests I'm focused on, and I'm bringing over. And then in the description, it's like, Steph needs help adding a test description, and then I'm pushing that up and then going to the team for help. So they can help me look through to understand, like, what is it that this test is doing? What's important about this domain? What sort of terminology should I include? And that has been working, but I didn't see that coming as part of this whole migrating stuff over. I really thought this might be a little bit more of a copypasta job. And I have learned some trickery is afoot. And it's been more complicated than I thought it was going to be. CHRIS: Well, at a minimum, I can say thank you for sharing all of your hard-learned lessons throughout this process. This does sound arduous, but hopefully, at the end of it, there will be a lot of value and a cleaned-up test suite and all of those sorts of things. But yeah, it's been an adventure you've been on. So on behalf of the people who you are sharing all of these things with, thank you. STEPH: Well, thank you. Yeah, I'm hoping this is very niche knowledge that there aren't many people in the world that are doing this exact work that this happens to be what this team needs. So yeah, it's been an adventure. I've certainly learned some things from it, and I still have more to go. So not there yet, but I'm also excited for when we can actually then delete this portion of the build process. And then also, I think, get rid of fixtures because I didn't think about that from the beginning either. But now that I'm realizing that's how those tests are working, I suspect we'll be able to delete those. And that'll be really nice because now we also have another single source of truth in factory_bot as to how valid records are being built. Mid-Roll Ad: Flaky tests take the joy out of programming. You push up some code, wait for the tests to run, and the build fails because of a test that has nothing to do with your change. So you click rebuild, and you wait. Again. And you hope you're lucky enough to get a passing build this time. Flaky tests slow everyone down, break your flow and make things downright miserable. In a perfect world, tests would only break if there's a legitimate problem that would impact production. They'd fail immediately and consistently, not intermittently. But the world's not perfect, and flaky tests will happen, and you don't have time to fix them all today. So how do you know where to start? BuildPulse automatically detects and tracks your team's flaky tests. Better still, it pinpoints the ones that are disrupting your team the most. With this list of top offenders, you'll know exactly where to focus your effort for maximum impact on making your builds more stable. In fact, the team at Codecademy was able to identify their flakiest tests with BuildPulse in just a few days. By focusing on those tests first, they reduced their flaky builds by more than 68% in less than a month! And you can do the same because BuildPulse integrates with the tools you're already using. It supports all the major CI systems, including CircleCI, GitHub Actions, Jenkins, and others. And it analyzes test results for all popular test frameworks and programming languages, like RSpec, Jest, Go, pytest, PHPUnit, and more. So stop letting flaky tests slow you down. Start your 14-day free trial of BuildPulse today. To learn more, visit buildpulse.io/ bikeshed. That's buildpulse.io/bikeshed. Pivoting just a bit, there's a listener question that I'm really excited for us to dive into. And this listener question comes from Joël Quenneville. Hey, Joël. All right, so Joël writes in, "As senior-level developers, how do you set goals to ensure that you keep growing? How do you know what are high-value areas for you to improve? How do you stay sharp? Do you just keep adding new languages to your tool belt? Do you pull back and try to dig into more theoretical concepts? Do you feel like you have enough tech skills and pivot to other things like communication or management skills? At the start of a dev career, there's an overwhelming list of things that it feels like you need to know all at once. Eventually, there comes a point where you no longer feel like you're drowning under the list of things that you need to learn. You're at least moderately competent in all the core concepts. So what's next?" This is a big, fun, scary question. I really like this question. Thank you, Joël, for sending it in because I think there's so much here that can be discussed. I can kick us off with a few thoughts. I want to first highlight one of the things that...or one of the things that resonates with me from this question is how Joël describes going through and reaching senior status how it can really feel like working through a backlog of features. So as a developer, I want to understand this particular framework, or as a developer, I want to be able to write clear and fast tests, or as a developer, I want to contribute to an open-source project. But now that that backlog is empty, you're wondering what's next on your roadmap, which is where I think that sort of big, fun scariness comes into play. So the first idea is to take a moment and embrace that success. You have probably worked really hard to get where you're at in your career. And there's nothing wrong with taking a pause and enjoying the view and just being appreciative of the fact that you are able to get your work done quickly or that you feel very confident in the work that you're doing. Growth is often very important to our careers, but I also think it's important to recognize when you've achieved certain growth and then, if you want to, just enjoy that and pause. And you're not constantly pushing yourself to the next level. I think that is a totally reasonable and healthy thing to do. The second thing that comes to mind is that you're on a Choose Your Own Adventure mode now, so you get to...I would encourage folks, once you've reached this stage, to reflect on where you're at and consider what is your dream? What are your aspirations? Maybe they're related to tech; maybe they're not. And consider where is it that you want to go next? And then, what are the concrete steps that will help you achieve those goals? So there's a really great article by Jen Dary, who's a career coach and owner of Plucky Manager Training, that describes this process. And there's a really great blog post that I'll be sure to include a link to in the show notes. But she has a couple of great questions that will then help you identify, like, what are my goals? Some of those questions are, "If I could do anything and money wasn't an object, I'd spend my time doing dot dot dot." And that doesn't necessarily mean sitting on a beach with your toes in the sand all day. I mean, it could, but then that probably just means you need a vacation. So take the vacation. And then, once you start to get bored, where does your mind start to wander? What are the things that you want to do? Where are you interested in spending your time? And then, once you have an idea of how you'd like to spend your time, you can consider what actions you could take next that will point you in that direction. There's also the benefit that by this point, you probably have an idea of the type of things that you like to do and where you like to spend your time. And so you can figure out which areas of expertise you want to invest in. So do you like more greenfield projects? Do you like architecture discussions? Do you like giving talks? Do you like teaching? Or maybe you're interested in management. I think there's also a more concrete approach that you can take that. You can just talk to your managers in your team and say, "Hey, what big, hard problems are you looking to solve? And then, you can get some inspiration from them and see if their problems align with your interests. Maybe it's not even your own team, but you can talk to other companies and see what other problems industries are trying to solve. That might be an area that then spurs some curiosity or some interest. And then, where do you feel underutilized? So with your current day-to-day, are there areas where you feel that you wish you had more responsibilities or more opportunities, but you feel like you don't have access to those opportunities? Maybe that's an area to explore as well. This feels like a wonderful coaching question in terms of you have done it; congratulations. You've reached a really great spot in your career. And so now you're figuring out that big next step. This is going to be highly customized to each person to then figuring out what it is that's going to help you feel fulfilled over the next five years, ten years, however long you want to project out. Those are some of my thoughts. How about you? What do you think? CHRIS: Well, first, those are some great thoughts. I appreciate that I get to follow them now. It's going to be a hard act to follow. But yeah, I think Joël has asked a fantastic question. And coming from Joël, I know how intentional and thoughtful a learner, and sharer, and teacher and all of these things are. So it's all the more sort of framed in that for me knowing Joël personally. I think to start, the kernel of the question is as senior developers, that's the like...or senior level developers is the way Joël phrased it, but it treats it as sort of this discrete moment in time, which I think there's maybe even something to unpack. And I think we've probably talked about this in previous episodes, but like, what does that even mean? And I think part of the story here is going from reactive where it's like, I don't know how anything works. I know a little bit. I can code some. And every day, I'm presented with new problems that I just don't understand. And I'm trying to build up that base of knowledge. Slowly, you know, you start, and it's like 95% of the time you feel like that. And slowly, the dial switches over, and maybe it's only 25% of the time you feel like that. Somewhere along that spectrum is the line of senior developer. I don't actually know where it is, but it's somewhere in there. And so I think it's that space where you can move from reactive learning things as necessitated by the work that's coming at you to I want to proactively choose the things that I want to be learning to try and expand the stuff that I know, and the ways that I can think about the work without being in direct response to a piece of work coming at me. So with that in mind, what do you do with this proactive space? And I think the way Joël frames the question, again, to what I know of Joël, he's such an intentional person. And I wouldn't be surprised if Joël is very purposeful and thinks about this and has approached it as a specific thing that he's doing. I have certainly been in more of “I'll figure it out when I get there.” I'll explore. Or actually, probably the most pointed thing that I did was I joined a consultancy. And that was a very purposeful choice early on in my career because I'm like, I think I know a little bit. I don't think I know a lot. I would like to know a lot. That seems fun. So what do I think is the best way to do that? My guess, and it turned out to be very much true, is if I join a consultancy, I'm going to see a bunch of different projects, different types of technologies, organizations, communication structure, stuff that works, stuff that doesn't work. And to be honest, I actually thought I would try out the consultancy thing for a little while, like a year or two, and then go on to my next adventure. Spoiler alert: I stayed for seven years. It was one of the best periods of my professional life. And I found it to be a much deeper well than I expected it to be. But for anyone that's looking for, like, how can I structure my career in a way that will just automatically provide the sort of novelty and space to grow? I would highly recommend a consultancy like thoughtbot. I wonder if they're hiring. STEPH: Well, yes, we are hiring. That was a perfect plug that I wasn't expecting for that to come. But yes, thoughtbot is totally hiring. We'll include a link in the show notes to all the jobs. [laughs] CHRIS: Sounds fantastic. But very sincerely, that was the best choice that I could have made and was a way to flip the situation around such that I don't have to be thinking about what I want to be learning. The learning will come to me. But even within that, I still tried to be intentional from time to time. And I would say, again, I don't have a holistic theory of how to improve. I just have some stuff that's kind of worked out well. One thing is focusing on fundamentals wherever I can, or a different way to put it is giving myself permission to spend a little bit more time whenever my work brushes up against what I would consider fundamentals. So things that are in that space are like SQL. Every time I'm working on something, I'm like, ah, I could use like a CROSS JOIN here, but I don't know what that is. Maybe I'll spend an extra 30 minutes Googling around and trying to figure out what a CROSS JOIN is. Is that a thing? Is a CROSS JOIN a real thing? I may be making it up. [laughs] A window function, I know that those are real. Maybe I'll learn what a window function is. I think a CROSS JOIN is a real thing. A LEFT OUTER JOIN that's a cool thing. And so, each time I've had that, SQL has been something that expanding my knowledge; I've continually felt like that was a good investment. Or fundamentals of HTTP, that's another one that really has served me well. With Ruby being the primary language that I program in, deeply understanding the language and the fundamentals and the semantics of it that's another place that has been a good investment. But by contrast, I would say I probably haven't gone as deep on the frameworks that I work with. So Rails is maybe a little bit different just because, like many people, I came to Ruby through Rails, and I've learned a lot of Rails. But like in JavaScript, I've worked with many different JavaScript frameworks. And I have been a little more intentional with how much time I invest into furthering my skills in them because I've seen them change and evolve enough times. And if anything, I'm trying to look for ones that are like, what if it's less about the framework and more about JavaScript and web fundamentals underneath? Thus, I've found myself in Svelte land. But I think it's that choice of trying to anchor to fundamentals wherever possible. And then I would say the other thing that's been really beneficial for me is what can I do that's wildly outside the stuff that I already know? And so probably the most pointed example I have of this is learning Elm. So I previously spent most of my career working in Ruby and JavaScript, so primarily object-oriented languages without a strong type system. And then, I was able to go over and experience this whole different paradigm way of working, way of structuring programs, feedback loop. There was so much about it that was really, really interesting. And even though I don't get to work in Elm, frankly, as much as I would want at this point or really at all, it informs everything that I do moving forward. And I think that falls out of the fact that it was so different than what I was doing. So if I were to do that again, probably the next type of language I would learn is Lisp because those are like, well, that's a whole other category of thing that I've heard about. People say some fun stuff about them, but I don't really know. So it's that fundamentals and weird stuff is how I would describe it. And by weird, I mean outside of the core base of knowledge that I have. STEPH: I love that categorization, and I'll stick with it, fundamentals and weird stuff, to stretch and grow and find some other areas. I also really like your framing, the reactive versus proactive. I think that's a really nice way to put it because so much of your career is you are just learning what your company needs you to learn, or you're learning what you need to keep progressing and to feel more competent with the types of features or the work that you're handling. And I think that's why Jen Dary's blog post resonates with me so much because it's probably...up until now, a lot of someone's career, maybe not Joël's particularly, but I know probably for my career, a lot of it has been reactive in terms of what are the things that I need to learn? And so then once you reach that point of like, okay, I feel competent and reasonably good at all the things that I needed to know, where do I want to go next? And rather than focus on necessarily the plans that are laid out in front of me, I can then go wide and think about what are some of the bigger things that I want to tackle? What are the things that are meaningful to me? Because then I can now push forward to this bigger goal versus achieving a particular salary band or title or things like that. But I can focus on something else that I really want to contribute to. And there do seem to be two common paths. So once you reach that level, either you typically go into management, or you become that more like principal and then onward and upward, whatever is after principal. I don't even know what's after that, [laughs] but the titles that come after principal. So there's management, and then I've seen the other very strong contributors, so Aaron Patterson comes to mind. And I feel like those people then typically will migrate to places where they get to contribute to a language or to a framework. And I think it comes down to the idea of impact because both of those provide a greater impact. So if you go into management, you can influence and affect a team of individuals, and you can increase the value created by that team. Then you've likely exceeded the value that you would have created as your own individual contributor. Or, if you contribute to a language or a framework, then your technical decisions impact a larger community. So I think that would be another good thing to reflect on is what type of impact am I looking for? What type of communities do I want to have a positive impact on? And that may spur some inspiration around where you want to go next, the things that you want to focus on. CHRIS: Yeah, I think one of the things we're picking up in that that Joël mentioned in his question is the idea of there is the individual contributor path. But then there's also the management path, which is the typical sort of that's the progression. And I think, for one, naming that the individual contributor path and the idea of going to principal dev and those sorts of outcomes is a fantastic path in and of itself. I think often it's like, well, you know, you go along for a certain amount of time, and then you become a manager. It's like, those are actually distinctly different things. And people have different levels of interest and aptitude in them, and I think recognizing that is critically important. And so, not expecting that management just comes after individual contributor is an important thing. The other thing I'll say is Charity Majors, who we had on the podcast a bit back, has a wonderful blog post about the pendulum swing called The Engineer/Manager Pendulum. And so in it, she talks about folks that have taken an exploration over into manager land and then come back to the individual contributor path or vice versa sort of being able to move between them, treating them as two potentially parallel career tracks but ones that we can move between. And her assertion is that often folks that are particularly strong have spent some time in both camps because then you gain this empathy, this understanding of what's the whole picture? How are we doing all of this? How do we think about communication, et cetera? So, again, to name it, like, I think it's totally fine to stay on one of those tracks to really know which of those tracks speaks to you or to even move between them a little bit and to explore it and to find out what's true. So we'll include a link to that in the show notes. And we'll also include a link to the previous episode a while back when we had Charity on. But yeah, those I think are some critical thoughts as well because those are different areas that we can grow as developers and expand on our impact within the team. And so, we want to make sure we have those options on the table as well. STEPH: Absolutely. I love where teams will support individuals to feel comfortable shifting between experiences like that because it does make you a more well-rounded contributor to that product team, not just as an engineer, but then you will also understand what everybody else is working on and be able to have more meaningful conversations with them about the company goals and then the type of work that's being done. So yeah, I love it. If you're in a place that you can maybe fail a little bit, hopefully not in a too painful way, but you can take a risk and say, "Hey, I want to try something and see if I like it," then I think that's wonderful. And take the risk and see how it goes. And just know that you have an exit strategy should you decide that you don't like that work or that type of work isn't for you. But at least now you know a little bit more about yourself. Overall, I want to respond directly to something that Joël highlighted around how do you know what are high-value areas for you to improve? And I think there are two definitions there because you can either let the people around you and your team define that high value for you, and maybe that really resonates with you, and it's something that you enjoy. And so you can go to your manager and say, "Hey, what are some high-value areas where I can make an impact for the team?" Or it could be very personal. And what are the high-value areas for you? Maybe there's a particular industry that you want to work on. Maybe you want to hit the public speaking circuit. And so you define more intrinsically what are those high-value areas for you? And I think that's a good place to start collecting feedback and start looking at what's high value for you personally and then what's high value to the company and see if there's any overlap there. With that, I think we've covered a good variety of things to explore and then highlighted some of the different ways that you and I have also considered this question. I think it's a fabulous question. Also, I think it's one, even if you're not at that senior level, to ask this question. Like, go ahead and start asking it early and often and revisit your answers and see how they change. I think that would be a really powerful habit to establish early in your career and then could help guide you along, and then you can reflect on some of your earlier choices. So, Joël, thank you so much for that question. STEPH: On that note, shall we wrap up? CHRIS: Let's wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

Episode 418: Flash Forward to Railsconf Day 2 with Andrea Fomera

building portland forum monitor devops honey badger podia flash forward senior software developer railsconf personal site sidekiq

Play Episode Listen Later May 18, 2022 31:23

Andrea Fomera is a Senior Software Developer at Podia who finds enjoyment in updating dependencies and crafting high quality, robust and maintainable code. She and Brittany flash forward to Railsconf Day 2 to discuss Andrea's talk, goals for the conference and an epic quest to find the best donut in Portland. Show Notes & Links: Andrea's Personal Site (https://afomera.dev) Andrea Fomera (@afomera) · Twitter (https://twitter.com/afomera) Learn Hotwire by Building a Forum (https://store.afomera.dev/learn-hotwire) Podia (https://www.podia.com/) WNB.rb (https://www.wnb-rb.dev/) Sponsored By: AppSignal (https://www.appsignal.com/rorpodcast) Monitor your apps from A to Z: error tracking, performance insights, server metrics, uptime pages, custom dashboards, and more. AppSignal works for all popular Ruby frameworks and automatically instruments and creates beautiful dashboards for Sidekiq, Active Job, and other integrations. As a listener of The Ruby on Rails Podcast, you get a 10% discount and a box of sweet treats. Start your 30-day free trial at https://www.appsignal.com/rorpodcast (https://www.appsignal.com/rorpodcast). Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution.

Isolator with Sidekiq - RUBY 543

All Ruby Podcasts by Devchat.tv

Play Episode Listen Later Apr 28, 2022 41:02

Anton Ivanopoulos joins the show today to share his approach with using Isolator and Sidekiq to ensure simple, efficient background jobs for Ruby. Discover how Isolator and Sidekiq integrate and how you can have more reliable message processing, group jobs into a set to follow their progress, and ultimately stop worrying about queues and focus on your app. Anton shares his story how he moved from delayed jobs to Sidekiq and why he replaced his backend and why Sidekiq is more effective in the long run. In this Episode… What is Isolator and how does it integrate with Sidekiq? A new way to catch the errors and add confidence and reliability message processing. How to build good habits, moving deploys from being atomic things to things where there is continuous deployment for efficient background jobs. Sponsors Top End Devs (https://topenddevs.com/) Coaching | Top End Devs (https://topenddevs.com/coaching) Links Do you want to take your career to the next level? Access a free coaching session with Charles Max Wood today (https://topenddevs.com/coaching) Listen to these podcasts, ad free, with Top End Devs Premium! (https://topenddevs.com/premium) Delivering Video with Less than 1s Latency with CacheFly (https://www.cachefly.com/) Picks Anton – Spirit Island Board Game (https://boardgamegeek.com/boardgame/162886/spirit-island) Charles – The Lost Ruins Of Arnak Board Game (https://boardgamegeek.com/boardgame/312484/lost-ruins-arnak) Charles – Rails Remote Conf (http://railsremoteconf.com/) John – GitHub: importmap-rails (https://github.com/rails/importmap-rails) John – Turbo (https://codecurious.dev/introduction-to-hotwire-and-turbo/) John – Stimulus (https://dev.to/bhumi/stimulus-rails-7-tutorial-5a6a) Valentino – Julia Evans (https://twitter.com/b0rk?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) Special Guest: Anton Ivanopoulos.

discover anton latency cachefly charles max wood isolator sidekiq coaching top end devs

Isolator with Sidekiq - RUBY 543

Ruby Rogues

Play Episode Listen Later Apr 28, 2022 41:02

Anton Ivanopoulos joins the show today to share his approach with using Isolator and Sidekiq to ensure simple, efficient background jobs for Ruby. Discover how Isolator and Sidekiq integrate and how you can have more reliable message processing, group jobs into a set to follow their progress, and ultimately stop worrying about queues and focus on your app. Anton shares his story how he moved from delayed jobs to Sidekiq and why he replaced his backend and why Sidekiq is more effective in the long run. In this Episode…What is Isolator and how does it integrate with Sidekiq? A new way to catch the errors and add confidence and reliability message processing. How to build good habits, moving deploys from being atomic things to things where there is continuous deployment for efficient background jobs.Sponsors Top End Devs Coaching | Top End Devs Links Do you want to take your career to the next level? Access a free coaching session with Charles Max Wood today Listen to these podcasts, ad free, with Top End Devs Premium! Delivering Video with Less than 1s Latency with CacheFly Picks Anton – Spirit Island Board Game Charles – The Lost Ruins Of Arnak Board Game Charles – Rails Remote Conf John – GitHub: importmap-rails John – Turbo John – Stimulus Valentino – Julia Evans Special Guest: Anton Ivanopoulos.Sponsored By: Coaching | Top End Devs: Do you want to level up your career? or go freelance? or start a podcast or youtube channel? Let Charles Max Wood Help You Achieve Your Dreams Top End Devs: Learn to Become a Top 5% Developer. Join our community of ambitious and engaged programmers to learn how.

discover developers anton latency isolator sidekiq

#49 - Mike Perham // Creator of Sidekiq

alphalist.CTO Podcast - For CTOs and Technical Leaders

Play Episode Listen Later Apr 14, 2022 53:48

Mike Perham, the creator of Sidekiq

discover creator os vc open source rails yoy sidekiq mike perham

326: Dongle Life

gusto software engineers sidekiq kelly sutton

Play Episode Listen Later Feb 15, 2022 41:02

Chris is making hiring progress and loves asdf and M1 laptops. Steph is anticipating the arrival of one dongle to rule them all and talks about moving away from having a lot of Bluetooth connections. Two other big things on Steph's mind are education around factories because they're v important and shared examples and how they can be overused. She and Chris agree that it is better to tell stories in tests instead. This episode is brought to you by ScoutAPM (https://scoutapm.com/bikeshed). Give Scout a try for free today and Scout will donate $5 to the open source project of your choice when you deploy. Services down? New Relic (https://newrelic.com/bikeshed) offers full stack visibility with 16 different monitoring products in a single platform. GitHub - asdf-vm/asdf: Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more (https://github.com/asdf-vm/asdf) Factories Should be the Bare Minimum (https://thoughtbot.com/blog/factories-should-be-the-bare-minimum) Mystery Guest (https://thoughtbot.com/blog/mystery-guest) GitHub - varvet/pundit: Minimal authorization through OO design and pure Ruby classes (https://github.com/varvet/pundit) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: STEPH: Hello and welcome to another episode of The Bike Shed. [laughs] CHRIS: Hello, and I'm singing, and I love singing. STEPH: It's Buddy the Elf; what's your favorite color? [laughter] For reals, here we go. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So hey, Chris. What's new in your world? CHRIS: My world continues to be focused on hiring as a pretty core aspect of things. We have happily had one offer extended and accepted, so that's great. We've got a person who will be joining the team in a couple of weeks. That's very exciting. And we're continuing in conversations with some other folks. So I look forward to the place where I can be on the other side of this and have that team and be growing the team and not having to focus because hiring takes a lot of effort. It is something that I believe should be done as well as possible and intentionally as possible and then just outreach and all that. So yeah, I'll be fine with being on the other side of that. But it's going well, so that is nice. STEPH: That's awesome that you're making progress. Once you have hired your team, will you then add to the agenda to hire someone to help with hiring? CHRIS: I don't actually know if the organization, if the whole company has someone who's focused on hiring. I think that can make sense. Working through recruiters and things like that is something that I've seen in the past. I've seen it work for certain organizations. I've also been on the receiving end of plenty of obviously copy and pasted very generic "Hey, person, I saw that you do lots of Java and other enterprise code software. Would you like to come work with us?" I'm like, none of those are true, and I do not want to go work with you. But thanks, I still appreciate the outreach. [laughs] So I am intrigued to see how we think about it. More generally, this is something that you and I have talked about offline but the idea that you kind of always want to be hiring. We do have specific roles that we've identified that the budget has space for. But more generally, ideally, we're going to need to hire more people down the road, and that will happen at a particular point. But having those conversations, starting to talk to people, now planting the idea of like, hey, you're great, and I would love to work with you someday and just keeping those lines of communication open. Networking is perhaps what the people call it. I don't know; I've never felt super comfortable with that word, but I think it's that and being friendly and staying connected with people whose work I respect and would love to work with more. So that's part of what I will come out of this with is yeah, let's always be hiring in a certain sense. STEPH: I'm glad you expanded on it because I was just thinking I have specific ideas as to what always be hiring means to me and what those activities would include. So I was curious what it means to you. And I agree, I think it's a lot of networking. It's a lot of taking chats and social chats with folks and just talking about the company and finding out where they're at. And then one day, if it works out that then they want to make a shift, then you've already got that relationship that started, and they're already potentially interested in your team. I guess some of the other big stuff that comes to mind, too, is like thoughtbot we have the blog. I feel like that's always really helpful too. Like when you help somebody, when you publish information that then helps them in their career, I feel like that will then draw people towards you as well. CHRIS: Yeah, the thoughtbot blog and basically everything that thoughtbot does, the podcast here, or Upcase, or all those things were so incredibly helpful in the hiring. But I also know they're hard to spin up, is what I would say. The thoughtbot blog has I don't even know how many hundreds of thousands of hours maybe. It's weird to try and put a number to it. But I've written a handful of posts for it, and I'm not great at writing them. They take me way longer than they should, but they took many hours. And then I had wonderful peer review by other developers at thoughtbot. And so, the amount of effort that goes into the thoughtbot blog absolutely produces wonderful benefits. But it's not free by any means, and similarly, the podcasts or Upcase or any of those sort of things. Similarly, the one that's actually most interesting that I see a lot of organizations go for initially and then often walk back is open source. Like, oh, we have this internal library that we built to do something. What we'll do is we'll just package it up and share it with the world, and then it'll be great. And the maintenance burden and support necessity of an open-source project is so high. I've actually historically gotten into the mode of suggesting...when I was working with clients, they would start to mention this and be like, "Oh yeah, we think we'll open source this thing, and it'll be great." I'm like, "Are you sure, though? Do you definitely want to?" There's definitely a difference between open sourcing and just putting an idea out there is one thing that I would say. Can you just write a blog post that has code snippets but not reusable code that you have to maintain that people, unfortunately, I think unfairly expect responsiveness and maintenance over time? And what if you stopped using that technology? What if you stop using this thing, but your name is still attached to it? And people have expectations of what that looks like. Or people come in and say, "Hey, this is great, but I want to change it in this way." And you're like, "Yeah, but that actually doesn't work for us. That's not how we use it. But we would be on the hook to maintain that code if we accept your pull request." And so, as wonderful as open source is, I tend to be on the more conservative end of the spectrum of like, are you definitely sure you want to open source this? Is there another way that you can share this with the world? Can it be a conference talk, or a blog post, or something like that? But it is an interesting one. STEPH: Yeah, I've been a part of several teams that have started with that; let's start an engineering blog. And their hearts are totally in the right place, and I understand why they want to do it. But like you just said, there's a cost to that. And if you don't have something like thoughtbot has like an investment day or a time for engineers to then be able to contribute to that blog, then either they're just not because they're not going to have their downtime to be able to do that. And it is hard to write and publish and be happy with what you're going to publish with the world. I really like what you're talking about in terms of the maintenance burden because I can't remember if it was an Upcase conversation or if there was something...but I was early on at thoughtbot and had a similar thought of why can't we just open source it? Why can't we make it public? And there was a very big thoughtful discussion around well, we have to have all these considerations in place. Who's going to maintain it? Just like FactoryBot is a really big internal project at thoughtbot. And there's typically a rotation of folks who will then take ownership and then onboard other people who are interested in it and curate the issues. And it's very important work, but you have to allocate time for it. All of that to say, I totally agree. There's a big burden that goes with it. CHRIS: Yeah, it's interesting that this has been an evolving thought in my head, and it makes me sad is another thing I'll say about it. I wish it were easier to just put code out there in the world and have the expectations properly calibrated for like, hey, I did this thing. Here's a code sample. It worked for me. Actually, I found dropping something in a Gist...a Gist just has a point-in-time connotation to it that I like. Like, if I see a code sample in a Gist, I'm like, I have no expectation that that person is going to do anything or respond to anything I have to say. But this is great because I now have this sample code that helps me get a little bit further. And I may have to vendor that code or take it on myself, and I now own it. It's not this person's responsibility. But the minute you have a repo with a README that says stuff and like, here are the installation instructions, the expectations just flip in a way that I don't think is...at least I become cautious around. And that does make me sad, though. STEPH: Yeah, it feels like you went from offering an example to I'm offering a product. And so then as soon as people feel like, oh, you're giving me something as a product that you maintain, then I'm going to have higher expectations of it should work how I expected it to work. I'm going to ask questions. And yeah, you make a lot of good points. CHRIS: Would you like to pay me $0 for me to build software for you? That sounds fun. STEPH: [laughs] CHRIS: And open source is such a wonderful thing. And so I'm interested in...like, I follow a lot of folks who are in the open-source world and have found ways to make it make sense financially or otherwise or organizationally. Open Collective and things like that is one option or OpenCore and then paid pro models and things like that like Sidekiq as an example. Sidekiq just celebrated ten years with some wild numbers in terms of the revenue, and it's like, yeah, that's fantastic. This is a cornerstone piece of software in the Ruby and Rails community. And also, Mike Perham had a great outcome from it. I think that's a win. So maybe blogging, maybe, but not sure. Probably not open source is my suggestion, at least for me. But one thing that I am interested in that hasn't been an option in my mind for a long time, but I'd love to get back to is conferences and going there, especially with a small team from an organization. The three developers we go, and we hang out at a conference and the company has a space there. And there's room to have conversations and meet people. That is one that I would love to continue in a way of making sure that our name is in people's minds as a place that they could work in this world. It is interesting, though, that it gets scoped a little bit like we are definitely a Rails shop. But that's not all that we are, or that's not the complete totality of our technical identity, so it becomes interesting. But I think it's probably the most representative. And I definitely see the Ruby and Rails community is having a good product-centric mindset that is definitely the sort of thing that I want in the teams that I'm building. STEPH: Yeah, I think that's an awesome idea because it's a way that you could focus on creating content. It'll likely have a big impact. But then you can also replay that content, but it's not the commitment of a blog or a commitment of open source. CHRIS: But yeah, so hiring has been, I would say, most of what I've been doing. One other thing that was fun this week, so I have my new laptop that I've had now for a couple of months, I'd say. And just this week, we had a very frustrating issue where Heroku stopped deploying our application. Just one day, it was like, nah, it doesn't work anymore. And I was like, well, that's less cool than I want it to be. And so one of the developers on the team dug into it, and it turned out Node-sass was the answer, which we're not even using is extra unpleasant. It's just part of Sprockets and Webpack or something like that. There's some downstream dependency sequence. We're using Tailwind and PostCSS. So we don't even need Node-sass. I think maybe PostCSS does. But anyway, turns out Heroku had switched to using version 16 of Node just without telling us. We were previously on 14, and then Node-sass didn't build on that. There was just this weird dependency chain that stopped working one day. And we weren't pinning the Node version within our application. So one of the developers figured this out, pinned us back to version 14 something of Node, and that was fine. But then my computer got confused because the versions were out of sync. Anyway, asdf is great. That's the first thing I'm going to say. So I use asdf to manage the versions of Ruby, and Node, and Yarn, and Elm, and basically everything else that I use. And I love that it's all under one hood, so asdf, wonderful. Also, my laptop, wonderful. I really love the M1 fancy laptop. But what was fun was I had to install the new Node version. And this was the first time in the three months I've had this computer that I've heard the fans come on. Finally, I asked it to do something hard enough that it was like, whoa, whoa, whoa, I'm going to need some backup here. And so the fans finally kicked in. So I don't know what's going on installing Node, but good for everyone involved, [laughter] impressive to make such aggressive use of all of the hardware in my computer. STEPH: Yeah, I love asdf. I miss it right now because I'm on my client machine, and we're not using asdf. Instead, we are using Chruby, C-H Ruby to manage Ruby versions. asdf is awesome. That's fun. It's the first time that the fans kicked on. I'm intrigued with my machine. I haven't really paid attention to it when the fans kick on except the one time where I had like a Ruby process that was running away, and I had to figure out what was going on there. Because then all the CPU was just being dedicated to Ruby even when I wasn't using Ruby. But since then, I haven't heard the fans. It's been very, very quiet. It's lovely. I like when it's quiet. CHRIS: Oh, it's been great. It was interesting because it was this weird noise that I'd forgotten about. STEPH: [laughs] CHRIS: My previous computer was so old that this was happening regularly whenever my backup process would run. Apparently, that is a very computationally intensive activity. So I would hear the fans kick in, immediately go find the backup process and say pause for 60 minutes or whatever it was. Just like, leave me alone. Stop it. The computer is getting too hot. You need to calm down. But now, with the new computer, there was nothing I could do to make it happen. And then finally it happened, and I was like, oh yeah, I guess this computer has fans. That's neat. But yeah, so things that are great, asdf and the M1 laptops. STEPH: Nice. Yeah, you're one of the few individuals I know that's using one of the M1 chip. So it's been reassuring to hear how well it's going because I did not opt in to that new-new. I opted in to the give me something stable and steady that I know so that way I don't have to fuss with it because I can then fuss with all the other things that I need to fuss about. CHRIS: So much fussing to do. STEPH: Lots of fussing. Fussing and cussing is what I do over here. CHRIS: [laughs] Mid-roll ad Hey, friends, let's take a quick break to hear from today's sponsor, New Relic. All right, so you've probably experienced this before where you're just starting to fall asleep, and it's a calm, code-free peaceful sleep, and then you're jolted awake by an emergency page. It's your night on call, and something is wrong. But I have some good news because you have New Relic, which means you can quickly run down the incident checklist and find that problem. So let's see, our real user monitoring metrics look good. And that's where New Relic measures the speed and performance of your end-users as they navigate the site. But it looks like there's an error in application performance monitoring. If we click on the error, we can find the deployment marker where it all began, roll back the change, and, ooh, problem is solved. We can go back to bed, back to sleep, and back to happy. That's the power of combining 16 different monitoring products into one platform. You can pinpoint issues down to the line of code so you know exactly why the problem happened and can resolve it quickly. That's why more than 14,000 other companies, including GitHub and Epic Games, use New Relic to improve their software. So you know that next late-night call is just waiting to happen, so get New Relic before it does. And you can get access to the whole New Relic platform and 100 gigabytes of data free forever. No credit card required. Sign up at newrelic.com/bikeshed. That's New Relic N-E-W-R-E-L-I-C .com/bikeshed, newrelic.com/bikeshed. CHRIS: Well, speaking of, what have you been fussing and cussing about this week, Steph? STEPH: So this is more in the pranting area, which is our portmanteau for praise and rant, where I'm super excited. I have a delivery coming from Amazon today. So I'm that person that keeps checking and waiting for it to show up. But I'm finally going to have one dongle to rule them all. I have a very messy approach right now [laughs] where I have all the dongles and have to plug everything in. And you know what? Normally it's fine. It's fine because I do it once, and I don't have to mess with it that much. But because I now have my thoughtbot laptop and I have a client laptop, and I needed to be able to switch back and forth, it is just too much. And I was realizing how many dongles I'm having to use. So I have one dongle to rule them all. It's showing up today. It's a very exciting day. CHRIS: I'm very excited for you. I recently made a similar switch when I got this new laptop. I was like, you know what? I'm going to look into it because power can come over USB-C and whatnot. And I was like, all right, it's finally time. I want to be able to just click in. And it's one of those things that feels trivial, or at least in my mind, I'm like, this doesn't feel like it'll make that big of a difference. But it makes it so much easier to disconnect my laptop, go somewhere else, and then come back. And I noticed myself doing that more, which I think is a positive thing. Otherwise, I'm just anchored to my desk. I'm like, I don't want to unplug everything and then have to replug it. That's like a whole thing. But now that it's not, I am more mobile, more flexible in where I'm working from, and I found benefits from that. So I'm a fan. I'm very happy that this is going to show up for you [laughs] and really change the way you're working. STEPH: Well, I've started moving away from a lot of Bluetooth connections as well because my keyboard will support Bluetooth, my headphones support Bluetooth. And I liked the idea of being wireless. But then, especially from switching laptops back and forth and then having to reconnect and all of it, it was just too tedious to go back and forth. And yeah, I'm with you where I didn't want to have to leave my desk and unplug everything and then bring it back where I'm playing, you know, like the game Operation where you had to reach in and then you had to grab different little bones? If you don't know the game Operation, that sounds really weird. But it felt like a game of Operation where then I was having to find all the dongles and connect them and plug them all in. And yeah, so it's going to be wonderful. CHRIS: Even knowing the game Operation, that still sounds kind of weird. STEPH: [laughs] CHRIS: But I really love that there are people out there listening that are like, what are they talking about? STEPH: What weird childhood did you have? CHRIS: Yeah, I'm definitely Team Wired-Almost-Everything. The only thing that I have that's wireless is my headphones. And it only works kind of, and I have to trick them sometimes. And the worst thing is occasionally my computer will have control, whatever, they're connected. So I'm listening to music on my computer and then suddenly, my phone will just steal it. It's like, what are you doing? No. Or, randomly, my headphones will be sitting away from me, and they'll just connect. And I'll be in the middle of a call on something else. Like, I'm here talking to you, and suddenly my headphones are like, hey, we wanted to join the party. It's like no, absolutely not, [laughs] not at this moment, under no circumstances. So I don't really believe in Bluetooth as a technology. I'm very much a fan, particularly with things like keyboards and whatnot. Bluetooth I've yet to be convinced that it is a sound technology. STEPH: I have the headphones where they try to be very smart, and they are pretty smart where they will block out sound. But then, if I am talking, then it will put me in more of an auditory space where then I can more easily hear, and it won't filter out sound as aggressively. But I've noticed a problem. And it's when I'm watching anything that's funny that then I'm laughing. So every time I laugh, my headphones think I'm talking to someone, and then it will switch over to where it's trying to let me hear more sounds out in the universe. And then it kicks back on because it's like, okay, she's done talking. It's a very jarring experience. [laughs] And I haven't figured out how to turn that setting off. It's like, oh, I just can't watch funny stuff with my headphones right now, which is also problematic with pairing because I tend to laugh a lot with pairing. It's a thing. I'm working on it. The struggles of Shteph. CHRIS: Well, at a minimum, it sounds like your dongle life is going to be improving very soon, and that's exciting. STEPH: Dongle life, it'll be single dongle life. That's it. [singing] All the single dongles, all the single dongles. Put your adapters up. [laughter] On a different note, talking about some of the work that I've been doing this past week with Joël Quenneville on our client work, is that we have been looking for ways that we still want to build up CI time. We've talked about the fact that we're working on some of that horizontal scaling. And I don't have an update there. But the other update I have is where we want to be very strategic about where we invest our time because improving the test is not trivial work. A lot of the low-hanging fruit has already been done, so triaging a flaky test can be very difficult, and it can take us a while. So we just want to make sure and verify that before we invest a lot of time into a portion of the test that then we know what the outcome is going to be. Are we improving developers' lives by this much? And how do we measure that? Are we reducing the CI build time, and how do we know that? And one of the areas that I really wanted to focus on is FactoryBot because there are a lot of factories. The factories tend to do a ton. So they are calling out to the database and building a lot of associations. And that's something that the team knows about as well is that there are just so many SQL queries that get executed in tests. And it would be great if we could reduce the number of SQL queries that are going out. And FactoryBot includes some ActiveSupport notifications, which means you can subscribe to factories being run which then gives you access to details like which factories are being used? What build strategy is used? Are you calling build build_stubbed or create? And the factory's execution time. So then the idea of this is that if we can harness a lot of the data that we can collect from FactoryBot, then can we ask questions around what's our slowest factory? How long does it take, and how many places is it being called? Because then ideally, we can calculate to say, okay, if this factory takes this long and it's used in this number of places, then we can have a formula to figure out how many minutes of our test suite is spent just on executing that factory. And then if we can reduce the time of that factory, let's say by half, then we know how much time we're shaving off of our CI build. And then we have this more concrete verified okay; this is worth our investment. We want to pursue this, even if the factory may take us a full day to improve because it does so much. And it's just gnarly. So it's going to take some time to really refactor it into a more simplified state. So, in theory, this sounds really, really great, and it was a lot of thanks to Josh Clayton, who was the one that advocated saying that we could use the ActiveSupport notifications to find a lot of this data. And so Josh and I paired on this for a bit to look into can we answer some of those other questions as well? And we were testing it on a small side project that he had, which was great because the other codebase is very big, and feedback is just a lot slower. So we wanted to first prototype it and have a proof of concept in a very quick space and just to be able to look through the data and make sure the assumptions that we had and the value would be there. So we applied that first, and that was going really well. And then Joël Quenneville took that strategy and then applied it to all the specs in the spec models directory and ran it for the much larger client codebase and got some really great results. And we also used a low fidelity approach where we wanted to be able to see which factories were the most popular. So how often are they getting called? And the average execution time. So that way, we could then quickly look at this scatter plot, and then we could see, okay, who's in the far upper right quadrant? Because those are the factories that are causing the most pain. But we started looking into a graphing library and what are we going to pull in. And Josh had the great idea. He's like, "I wonder if Google Sheets has a scatter plot. Can we export this to CSV data and then copy it from the terminal and import it into Google Sheets?" And it turns out that you can. So then we grabbed it and put it in Google Sheets and then just converted it into a scatter plot, which was really nice because then we didn't have to incorporate any chart library or any graphics or anything. We could just plop it into Google Sheets and then easily share it. So we now have this list thanks to Joël because he ran it through the spec models directory of all the factories that are getting called. And it's really interesting. And there's one, in particular, that is high on the list. And it was actually one of the first ones that we worked with when we were troubleshooting a test that took us a while when we first joined the project. And the average time for this factory is four seconds, and it gets called over 500 times. It's like 527 times. So then if we multiply that, so if we say, all right, it takes about 4 seconds times 527 and then divide it for 60 for minutes, that's 35 minutes, 35 minutes for that factory. Now, granted, these are getting parallelized across different processes. But still, if you divide that up across four processes, that's a non-trivial amount of time. So I think this is going to be really helpful and really interesting data that we can then use to drive our decisions to say, okay, we want to take this factory and let's say even if we can cut it into half, let's say if we bring it from 4 seconds to 2 seconds, it'll go from 35 minutes to 17 and a half. CHRIS: Oh wow, I love the methodical approach. I love that actually having a number you're like; this is how much pain or the cost of this right now. And so we've identified that this is this high-level thing. I love the intentional starting with, like, let's measure it. Let's understand where the most bang for the buck is. In particular, the graph that you're describing reminds me of one I haven't actually worked with it much. But Code Climate has a graph that they use, which is it's churn versus complexity. So it's like, you may have a very complex piece of your code, but someone wrote it once, and it just sits in the corner. And you know what? It quietly does its job. And yes, it's very complex, but nobody needs to touch it. So it's not a big deal. And then you have stuff that changes constantly, but it's super simple, so that's fine too. Your UsersController is probably going to change a bunch; that's fine. But the stuff that is constantly changing and very complex that's the magic quadrant that, like, pay a lot of attention to that. And similarly, which are the ones that are being used a lot and take a while? That's the magic quadrant. I'm intrigued now. I want to search for more magic quadrants that deserve attention. But for now, that sounds like a lot of fun. So now, what's the approach that you're going to take? I imagine you need to alias that factory and have it exist because some tests will rely on certain details of it. This is my guess. So let me see if this is the way that you're thinking about it, alias the factory, so you have a representation that does all the stuff that the current one does. But then you have a new one that is much more pared down. And then, on a test by test basis, you start switching it over and trying to move things to the lower weight, the slimmer version of the factory. But I would think you would want to do a gradual process if there are 520-ish usages. Because otherwise, just changing that factory out from under all the tests, I imagine you'd break some tests if you just were like, what if it did less? STEPH: Yeah, I like that idea of the incremental approach. And that all sounds great, especially the alias because you're right; we want to change it incrementally and not all of them at once. But then essentially implement, one, because I want to see what does the pared-down factory look like? What is the basic factory that we can get away with? And then how long does it take for that factory to execute? Because then that will help confirm, can we really get it down to two seconds? Or is this just a factory that's always going to take three and a half seconds, and then it's not really that much of a payoff? Maybe we should look for a different factory to investigate. And then also understanding from the test are people reaching for this factory all the time because it builds up the world and all of these tests need the full world? Or are people just reaching for this because it does the one or two things that they need, and we can get away with a much slimmer factory? So right now, it's in the space of understanding why are people reaching for this? What are the tests they actually need? And yeah, how can we do it incrementally? At one point, we may even be able to try to programmatically switch it out. Maybe we just find 50 tests that are using this once we have the slimmed-down version and we replace...50 is probably too big. But if we replace X number of tests with this factory, how many of them fail? Maybe 10% of them passed. Cool, let's just take those 10% as a win and issue those as a PR. So that could be a strategy as well just to find if there's any that are super easy to change. All we had to do is literally change the name of the factory. The other big part that's on my mind is education around factories. I think a lot of people on the team understand that factories are very important. They can be very helpful. They can also be very cumbersome. But it feels like a good opportunity to say, "Hey, we are specifically working on these factories. Here's the reasoning that led us to work on these factories. When you're in the space of factories, please be mindful about what are you reaching for? Is there a slimmed-down factory that you can reach for? Maybe you can implement your own slimmed-down factory if one doesn't already exist." So I like the idea of coupling it with also just broader awareness because we are but two people. So I would love for more people to be part of the changes. CHRIS: Unsurprisingly, there are some wonderful blog posts on the thoughtbot blog that speak to this topic. One that I'm a fan of is Factories Should be the Bare Minimum. This was written by Matt Sumner. And it describes basically that idea of factories shouldn't build the worlds. They should give you the pieces that you can use to build the world but not build the world entirely. And so I'm a big believer in that, having your factories be as minimal as possible. They should be valid, but that's about it. And then I will often reach for extracted helper methods and keeping those as locally scoped as possible often in the spec file, or if not, maybe they're sharing spec support. But being intentional with where we reach for them and not having everyone use the same thing that just slowly gets added to. And it's like, do I actually need everything that's in there? The other thing that's interesting is the idea of having a factory that does a ton is, in my mind, sort of in direct contradiction to what I believe factories exist for which is when I think of factories, they're useful to fill in the rest of the details such that you don't have mystery guests in your test. But you can explicitly say build me a user who has an email that looks like this because, in this test, I care about the email, but I don't care about the rest of the details. I don't care about their name. I don't care about their password, or the roles, or any of the other details. Just let the factory deal with that because it's not important to the test. But I want to make sure that the relevant detail is present and specified within the spec. If you have a factory that builds everything in the world, that's like build a user and then grabs the first action from the project that that user has, because I know that they do because they use the big factory, that is just in direct contradiction to what we want factories to do. We want tests to tell their story. We want to avoid mystery guests. Factories are a great way to do that while still remaining concise. But if your factories just build the world, then there are some mystery guests in the world, I can assure you. STEPH: Yeah, I agree where factories have served as an abstraction for what I think is important to the test. But then there becomes this moment of where someone thinks, well, I need to build up these records, but I don't really need to reference them directly. I just have some coupled code that's going to rely on these. And so I don't explicitly need them, but they need to be there. So I'm going to extract it away, and a factory feels like a good place for me to extract that too. And I would take the very hard opposite approach where if you have coupled code and you have these dependencies that aren't necessarily explicitly used in the test, but they are required for the test, I'd rather see a painful test setup than have all of that extracted away from me. Because then if I do need to triage or troubleshoot that test, it's going to take a lot of just mental overhead to work through what do I actually need here and why? So I'd rather see that painful test set up then have it moved somewhere else. But I think a lot of people take the opposite approach of where if I abstracted away, my test looks prettier. And I'm like, yeah, but maybe to you in the moment, but it's going to cause me a lot of pain further down the road when I have to work with this. So show me all the crap that you had to do upfront. Just let me know. [laughs] I'd rather the test be honest with me. And then it's a really nice jumping-off point because you can see a test that does all of this. And instead of blaming the test and thinking it's the test's fault, you recognize this test has a lot of complicated setup, and it's probably because of the code and how the code was written. And we should look at refactoring the code, not at how can we make our tests look prettier? CHRIS: Unsurprisingly, I agree with 100% of that. Someday we'll find things other than Pop-Tarts and IPAs that we disagree on. But today is not that day. [laughs] Once again, today is not that day. Mid-roll Ad Hi, friends. And now a quick break to hear from today's sponsor, Scout APM. Scout APM is an application performance monitoring tool that's designed to help developers find and fix performance issues quickly. With an intuitive interface, Scout will tie bottlenecks to source code, so you can quickly pinpoint and resolve those performance abnormalities like N+1 queries, slow database queries, and memory bloat. Scout also recently implemented external service monitoring, adding even more granularity when it comes to HTTP requests and API calls. So give Scout a try today with a free 14-day trial and experience first-hand why developers worldwide call Scout their best friend. And as an added bonus for Bike Shed listeners, Scout will donate $5 to the open-source project of your choice when you deploy. To learn more, visit scoutapm.com/bikeshed. That's scoutapm.com/bikeshed. STEPH: Well, here's one more that maybe you'll agree with, maybe you won't. We'll see. I'll try not to lead you in either direction, but shared examples. If I'm going to rant for a little bit, shared examples are in that space of where they just get used so heavily, and they abstract away important information about the test. And it makes the test so succinct that I don't actually know what the test is doing. And I've seen a number of places where a shared example has been extracted, and it is only used within that test file once, maybe twice. And I'm just like, friends, too much abstraction. Please keep it close. [laughs] We don't need to move it away. We want our test to be friendly and just full of context, which is what I mean when I say friendly. I want full of context is what I'm looking for, well-named variables. And I won't be able to read the test and see what's happening. So my little complaint for today would also be about shared examples and how they can be overused. And they do have a really neat purpose. They can be helpful for if you're testing maybe a controller action and you want to say you're extracting that authentication, making sure that a controller always has authentication and then that is getting included. Sure, that feels very helpful. But that's really one of the few cases I can think of where a shared example comes into play. And if you are testing code over and over throughout different parts of your codebase, there's probably a part of your codebase there that needs to then be pulled out into a class and test that class in isolation. And then you don't need to retest it throughout all of your other classes. Have I already ranted about shared examples? I can't recall at this point if I have or not. [laughs] CHRIS: I don't think we have talked about shared examples before. And I appreciate you not leading the witness here. But I think I'm in agreement with you, particularly the way you refined it there at the end because that controller example is the one rare case where I might reach for it. But in general, I think this is one of those things that I saw early on in my career. I was like, oh, cool; this is a way to clean stuff up and DRY and all those wonderful things. And then I've definitely felt the pain of just overuse of shared examples and ways to pull details out of tests. But it's like, I want to see the details. And I think broadly, that's the theme that you and I are very aligned on is like, no, no, no, tell me the story in tests. I am much less interested in having these concise tests that have a single line, and it's like, expect foo to have bar. And it's like, why? Because...oh, there's a let and then a subject, and it's a shared...oh okay. Now that I can put it together, I can tell the story, but I cannot look at this test and see a story. I want to see a story, friends. So yes, I'm totally in alignment, especially with the slight caveat at the end of like, there are cases where it's useful. Similarly, I've used let. I definitely have not even that long ago. And I stand by the usage, but it was very rare. It's very rare, and it is something that I'll look at and be like, am I sure? Definitely, is this the right thing, or did I do something wrong? Because if I find myself leaning towards let, it's like there's something that I don't think is important to the story of this test that still needs to happen. Why is that? What's going on here? Something feels off about that. And similarly, with the shared examples, it's like, is there not a different way to extract this such that I can test it in a way that I have confidence in, and then we're good? I occasionally will talk myself into using shared examples or something like it where I'm like, oh, but it's really important that everything in the app has that authentication layer put in. And so, I should definitely have this very easily reusable test that can ensure that I have it. But there's a tautology there of well, if I write the test, then I'm definitely thinking about the implementation. But if I forget the implementation, I might also forget the test. And so, it actually doesn't provide any real safety. And in those cases, that's a rare case where I would reach for some weird metaprogramming thing that's like, controllers must do the thing. And we say that in application control and then everything inherited from that will raise if it doesn't implement the authentication layer. Something like weird code that says, "You shall not pass. You must, in fact, implement the authentication layer." Rather than saying, "Oh, we'll just make it really easy to test it so that we always test and, therefore, always use the necessary authentication layer." But yeah, that's a hard one to describe in the radio. So I don't know if that came through clearly. But that's sort of my headspace on this. STEPH: Yeah, and all of that makes sense. I'm trying to think of a good example. And it's been a while since I've used Pundit, but I feel like Pundit may have a really good example of this where it's very easy to document to say, hey, all of these controllers need to make sure that they call out to this class or that there's authentication. I can't remember the exact code and how that works. But I feel like Pundit has a really good example of that behavior. CHRIS: I think they do. It's something where I think it's a configuration level thing, but you say, "Hey, Pundit, we should definitely authorize any access to models." And so Pundit then has a before action, or it's an around filter one of those. But it will raise an unauthorized error, I want to say. Like, you did not do the authorization dance in this. And that's a great example of like, I like that it is loud and annoying and in your face. And it is not possible for me to forget it because we configured it throughout all controllers. And so it's that sort of thing that I would probably reach for even though that code gets complicated and messy, and actions at a distance. But it's worth that trade-off in my mind to have, like, I don't want to forget to do the authorization stuff. Permissions matter. STEPH: That was a really nice pre-emptive approach as well. Because in most cases that we're describing, it's the I'm going to write a controller, and then I need to add this test to verify and prove that yes, I didn't forget the authentication stuff versus upfront, you're setting in a configuration to say, "Hey, please remind me to do the configuration or the authentication step that I don't miss this." So that's also a really, really nice approach. CHRIS: Yeah, the same version of me that's going to forget to write the test is going to forget to write the implementation. So I don't want to trust that version of me to save that version of me. I'm equally untrustworthy in those situations. STEPH: You want to trust the version of you that's going to get yelled at by the code if you don't do it. CHRIS: Yep, I'm going to trust the version of me that was like, I don't trust any future version of me. I will yell at myself if I have not done the necessary things. STEPH: [laughs] CHRIS: To be clear, this is like a life philosophy of mine. I don't try to remember things because I forget stuff a lot. It just happens. And so if I need to take something out the door with me, it goes in front of the door but extra critically, and this is the subtle line. Because plenty of people do that trick where you put a thing in front of the door because then you can't leave without it. There's no way to forget it. But by virtue of that, you cannot put something in front of the door until it is time to use it. Like, if ever you have to go and be like, oh, I don't need it now, though, so I'm going to move it out of the way, open the door, and then leave. No, no, no, because then you've broken the magic of the thing in front of the door must leave with you. So it's a very subtle line. I will play games with myself. I'll be like, I am forgetful. I will not remember this. I do not trust future me, so I'm going to play a trick on them. But you got to calibrate it just right. STEPH: That's really funny because I totally [laughs] didn't think about it until now how you described it. But I have definitely done that where I set a rule for myself, but then I'll break it. And then, of course, everything all of it collapses. There is a time when Tim, my husband, was going through a developer bootcamp. And as he was learning the whole world and everything that's out there, he would ask me all these questions. And he's like, "Do you know this?" And I'd be like, "No." He's like, "Do you know that?" I was like, "No." He was like, "I thought you knew this stuff." He's like, "I thought this was your job." And I was like, "Yeah, I'm really good at finding it and Googling it. But I work really hard to not store this in short-term memory because I'm filling it up with other stuff. So I work really hard to be able to find this stuff and track it and Google it." But now, there's a lot of stuff that I try very hard to not hold on to until I need it. But that was a funny moment where he seemed very upset that I didn't know stuff. And I was like, "Well, welcome to web development. There is too much to know. You're going to have to have a really good catalog system." CHRIS: Also, just so we're clear, it's going to change by next Thursday, so don't hang on to anything like it's just true forever. STEPH: [laughs] CHRIS: SQL will probably be around. That's about it. That's the one thing that I'm really confident in. STEPH: Yeah, that feels fair. Get really good at understanding HTTP forms, SQL, all that feels like some really good groundwork. CHRIS: There are some foundations. We should have a foundations episode where we talk about what we think the foundations are, the stuff that we bet won't be different in 10 years. But everything else is going to change by next Thursday, specifically. STEPH: Yeah, I like the idea of foundations. I'd be intrigued to see what we talk about and what happens there because I feel like that's going to be very representative of already what we talk about. We often will sprinkle in some new-new, especially thanks to a lot of the adventures that you go on. But I feel like a lot of the stuff that we talk about we always bring it back to the foundation because we do want the experiences that we're having to be applicable to everyone else as well. So yeah, that would be interesting to see what comes out of that. On that note, shall we wrap up? CHRIS: Let's wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeee!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

amazon google pr services networking operation buddy api elf mid bluetooth java github minimal dry epic games someday rails googling pop tarts cpu m1 usb c elixir pundits sql node elm yarn ipas christo factories mandy moore gist tailwinds google sheets oo bare minimum permissions csv readme heroku new relic sprockets dongle erlang mystery guest webpack quenneville open collective fussing chris yeah christ here chris so chris oh chris well sidekiq postcss chris thanks mike perham chris let josh clayton upcase chris toomey

133 - Kelly Sutton, Software Engineer at Gusto

Rails with Jason

Play Episode Listen Later Feb 14, 2022 52:54

In this episode, Kelly Sutton and I discuss Sidekiq, structuring large codebases with Packwerk, namespaces, the real purpose of private methods, and the upcoming Sin City Ruby conference.KellySutton.comKelly Sutton on Twitter

324: Coding Time!

Play Episode Listen Later Feb 1, 2022 41:23

Chris updates us on his new window manager of choice, Moom, and tells us what's good with it. He's also giving yet another task manager a go: OmniFocus. (Sorry Things.) Steph talks about defining test classes in RSpec and readdresses flaky tests to improve CI build time. Chris is worried about productivity. He's still not coding as much as he'd like to be. Steph lends an ear, and together, they discuss potential ways Chris could gain back a little bit of coding time at work. This episode is brought to you by ScoutAPM (https://scoutapm.com/bikeshed). Give Scout a try for free today and Scout will donate $5 to the open source project of your choice when you deploy. Moom (https://apps.apple.com/us/app/moom/id419330170?mt=12) OmniFocus (https://apps.apple.com/us/app/omnifocus-3/id1346190318) Is It Worth the Time? (https://xkcd.com/1205/) Knapsack Pro (https://knapsackpro.com/) Shopify Monolith (https://shopify.engineering/shopify-monolith) Sacrificial Test Classes (https://blog.bitwrangler.com/2016/11/10/sacrificial-test-classes.html) rspecq (https://github.com/skroutz/rspecq) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: STEPH: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So hey, Chris, what's new in your world? CHRIS: What's new in my world? Well, hey, Steph. Oh, I have an update on a thing that I think I talked about a while back or at least asked on Twitter. But I've been looking for a window manager for forever. And in that way that I sort of overcorrected a while back, I think where I'm no longer allowed to do anything related to productivity or dev tools. I was just forbidden because it was a time sink. I'm slowly trying to correct back and be like, you know what? I regularly think about how it would be nice to have a better window manager. So previously, I had used Divvy, D-I-V-V-Y, which is fine. It did an okay job, but it just didn't have quite the level of control that I wanted, or maybe I didn't investigate it enough. But it felt like it was lacking. So I did a little bit of research. A bunch of people recommended different things. There was Spectacle; there was Rectangle. There was a whole bunch of other things that I'm forgetting now because I have settled on Moom, M-O-O-M. Those are fun words. STEPH: I feel like you keep bringing interesting words [laughs] because last time, it was Things where you're tracking all the things. And now we have Moom to track the space. All right. CHRIS: If this is my legacy as a podcaster, then I feel like I will have done well just, you know, weird sounds mostly that's what he's going for. But yes, I've been using Moom now for…[laughs] God, it's just ridiculous to say, but here we are. STEPH: [laughs] CHRIS: I've been using it. I've been enjoying it. In particular, the thing that I liked about it...a bunch of the other ones that I looked at were like, oh, we've got all these different configurations. And you can move things any which way, and you can have any number of hotkeys. And I was like, wait, wait, wait, say more right now. You want to take over my global namespace of hotkeys and just clutter it with 19 different things? You know that that is a limited space that I'm working with here. And so Moom, somewhat uniquely, at least in the ones that I experienced, was what I would describe as a modal window manager. So much like Vim is modal where you start out in normal mode, and you're moving around and you kind of bounce and search and all of that, and then you enter insert mode. And in insert mode, keys do different things. And then in command mode...it's got all these different modes. And so there are lots of different namespaces for hotkeys. It's one of the things that makes Vim so powerful. Moom is similar in that there's one global activation hotkey. And then, within that, I can have a whole namespace of hotkeys. So like M will put something in the middle of my screen now. F will put something full-screen. And I don't need to remember weird multikey combinations for that. There's just the one to get started, and then I've configured it such that the tab will bounce to a secondary display and sort of rotate through them. M and F and Q and P I've got it physically laid out on the keyboard. So it looks like my screen. Q being on the left side will push something to the left side, P to the right side. And I'm very happy with that. I don't need a lot out of this tool. I don't need very complex management or scripting or any of that, which are very nice features that exist in the other ones. But that combination, the one hotkey to rule them all, and then the sub hotkeys within it, and the ability to mostly move between the screens and then put stuff where I want it is great. I'm very happy. STEPH: I think I've figured it out. So Moom, I think it's a combination of move and zoom, and that's how they got Moom. CHRIS: You're probably right. STEPH: That does sound really nice. I'm a Spectacle fan. And I have enjoyed it and just stuck with it because I haven't felt a need to change from it. And it's really nice where I use my arrow keys for which direction I want to go. So that has been easy for me to recall. But that sounds really nice, all the things that you're describing with Moom. CHRIS: Does spectacle have the like, is it some Command Option Control and then left or right or up or down? Or is it you type something, and then you type left, right, up, down? STEPH: I have to actually touch my keyboard to answer that question because I have the muscle memory, which is an interesting thing that my muscles knows it, but my brain has to really think about it. So I think it's like the Option Command, and then yeah, then use the arrow keys. CHRIS: Gotcha. That's roughly what I had when I was using Divvy previously, but I found just enough of a limitation there. And so Moom has been great as another tool. But I think Spectacle has a lot more features in terms of scripting and other fancier stuff that you can do, which is both super intriguing and, again, sort of the thing that I'm not allowed to do. [laughs] So I went with, like, this tool seems fine and has the one feature that I really want. That said, you brought up Things, which is the to-do list app that I've been looking at. I've been using it for a week now. It's great. I'm enjoying having a more structured way to say, like, here's what I'm doing today. Here's what I'm doing tomorrow. It's been wonderful. But I'm already looking at OmniFocus as a better version. STEPH: [laughs] CHRIS: Because I think there's some stuff that I don't love, and yes, I can hear my own voice in the back of my head that's like, always chasing that next thing. But I haven't actually made the effort to switch over or even tried. I've used OmniFocus in the past. But anyway, I'll let you know if I do make additional moves there. STEPH: Yeah, I'm enjoying this journey. Keep me up to date on it. I've heard of OmniFocus, but I know nothing about it. But I feel like I've heard good things. So I like this journey you're going on where you just keep switching and trying new things. That's fun for me [laughs], and there's chasing productivity. So I'm into it; I'm here for it. CHRIS: If I just invest enough hours to save a handful of minutes down the road, then I will have...oh no, wait, that's not how this goes. There's, of course, an xkcd about this which we can include in the show notes. But I'm trying to be very intentional with it. I waited for many years before I allowed myself to reinvestigate the world of to-do lists. And I'm hopefully going to keep it to just a couple of weeks of nonsense and then back to a few years of stable. That's the dream. But yeah, that's some of the smaller things that are up in my world. I have another topic that I want to chat about. But I'd love to hear what's new in your world? STEPH: Yeah, I have some interesting bits that I can talk about with the project that I'm working on. But more concretely, I have something that's been on my mind that I don't think that I've talked about here on the show, but I think would be fun to talk about because I just happened to run into it this week while working on some code. And it's the idea of defining test classes in RSpec so as you are testing part of your code, but then you want to create just like a fake class, something that you can use as a substitute for real application code. And so it's a really nice way that then you can have this replica behavior, but then maybe it's just one particular method or some behavior that you need to use in the class but then doesn't actually go to the real code. That's wonderful. That's great. One thing that I've learned is that with RSpec is when you are introducing a test class, so let's say if you have your RSpec describe and then either a string or it's the name of a class, and then you have a block so do, and then within that block is where you write your test. If you create a temporary class, say, like I have my class test class, and then I have some behavior, that gets defined in the global namespace. It's not scoped to that particular RSpec example. And the reason for that it's not specific to RSpec. RSpec is not the one that's doing this; it's actually Ruby behavior. So for Ruby, when you're defining within a block like that, if you're defining a constant, if you're defining another class inside of a block, it's going to use the outer namespace as its namespace. So if you had a top-level class that you were defining, but if you define a class as a block, and then inside of that block you define a constant, that constant is then defined in the object namespace instead of within that particular class that you have written. And so that's why RSpec has this behavior. Because someone brought up a really great question about this on RSpec::Core asking about it, and they're like, yeah, that's actually how Ruby works. And so we're not going to change RSpec's behavior since that is how Ruby has decided to handle this. And the part where this becomes important is when you define a test class within an RSpec example. While it may be unlikely that someone is going to use that exact same name for their test class that they're going to create in their RSpec example, if they were to use that same name, then you're going to have a collision between the two. One of them's going to win, and you're probably going to end up with some really weird test failures because it's going to get confusing as to which class is being used, and they may not match up with each other. So one way around this, and this is going to be one of the rare times that I suggest this, but let. Let is scoped to an RSpec example. And so you could define a class inside of a let, and then that will scope it to the example. There are probably some other approaches as well, but that's the one that I'm most familiar with to ensure that when you define that class or constant, it's not getting defined in the global namespace and ensuring that none of the other tests have access to it. CHRIS: Well, this is certainly interesting. I'm pretty sure I've been operating under the opposite assumption for the entirety of my career. This is good to know. I feel like I probably have had tests that failed because of this. And then I learned this truth, and then I subsequently forgot it. I don't know if you know this, but if you define a method within just a helper method that you extract in RSpec, are those also on the global namespace? I don't define classes in RSpec blocks that often. It's pretty rare. Like if I have a controller concern sort of thing that I want to test, I might say random controller and inject the thing there or some other abstracted piece. That is the only case I can think of where I have a fake model or a fake controller or something like that for test purposes. But it doesn't come up that often. I do extract a heck ton of local helper methods. And I'm wondering now, are those all in the shared global namespace? STEPH: I'm pretty sure they're not. And I'm getting on the edges of my knowledge here, but I think it has to do with the fact of when you're defining a constant. So if you're defining a class versus an actual constant, that will get into the global namespace because it's using the outer scoping. But in my experience, I'm pretty sure that's not true for the method just because I remember one time I did some funky stuff with RSpec. And I remember seeing that I couldn't access those methods from another example. CHRIS: I like the honesty. And you're like, to be clear, I was doing something weird, but I learned that day. Okay, that's good because at least that part maps to my understanding. So methods may be safe, but classes get shared. Very interesting. STEPH: And it's something that I rarely think about or had worried about just because if I'm defining a fake test class, I often will put it somewhere that's intended to be more global. So I'll stuff it somewhere in like spec support. So then other people can see, hey, I've already mimicked this behavior. So if you need to use the same thing, just go ahead and use this. It's not often that I am adding that class directly to the RSpec example group. So I think I've been fortunate where I haven't actually run into that conflict for that reason. But this came up while giving an RSpec course. And while we were just in a very small, tiny codebase and replicating some examples, someone in the class was like, "Hey, by the way, do you know that that's in the global namespace?" And I was like, "No, friend. Tell me more." So thanks to that person, they're the ones that actually enlightened me about how it's going into that namespace and how it can actually pollute your testing namespace. There's a really good article that's written by Ken Mayer. And we'll be sure to include a link in the show notes that talks about it and also provides the let example as a way to work around this. And also links to the GitHub discussion on RSpec::Core, where they talk about this behavior and why things are the way that they are. Circling back to some of the more general project-y things that I alluded to earlier, I've shared a bit about the project that I'm working on. But just to recap it, it is focused on helping a very large team that has a large number of tests, around 85,000. And they are looking to address flaky tests that they have and overall really improve their CI build time. So right now, it takes about 30 minutes for the build to take place. But they also have flaky tests, and then that slows things down. And so, the re-verify rate has been painful for them. There's been some really great work that has improved that, particularly there is a, I think we've talked about this before, but where they're re-verifying certain flaky tests, which isn't great because they're still flaky tests, but at least they're not preventing people from moving forward and shipping code. But some of the bigger stuff that is just on my mind is when you have a very large team and a very large application, by large team, I'm talking about 100 developers, and they are all contributing to this codebase. And there are around 85,000 tests, and that has grown substantially in the last 12 months. And so, if you think about the trajectory of the addition of those tests, it is just going to continue to grow. So there's a concern there of even if we address flaky tests and we improve things, there's an architecture concern of how do we really reduce the CI build time? And so there's that aspect, and then there's also the aspect of then well, how do we still work to improve the tests and the codebase as well as we go across all of these disparate teams? And right now, there is a bit of a culture where engineers don't feel empowered where they can necessarily address all of the flaky tests or things that they run into. And so there is a bit of a mindset of I'm stuck on this, or this test failed, or it's flaky, or I don't understand it. So I'm just going mute it, or I'm going to hand it off to someone else to work on it. So there are three big areas that are on my mind. The first one is architecture. You can throw architecture at it. There's also the code quality that's a concern. And then how do you improve the code quality in a way that you're improving it fast enough that then you've got 100 other developers that are also contributing to it at the same time? And then individual IC empowerment where then people feel like, hey, I ran into a slow test or a flaky test, and I feel like I can triage this, and I can make changes. For the architecture piece, we're still in the infancy stages of how to approach this and the strategy that we're using. But one of the ideas that has come up is how do we reduce tentpoles? And the tentpole is like when you're running your test and, let's say that it's parallelized, all of the various tests. But there is one process that takes like 20 minutes, and then the other process is completed in 5 minutes as a drastic example. And overall, you could have reduced your time if you had managed to split that 120-minute process across all the other workers who are then available for that work. So there are some tentpoles that are taking place. And that could be one first step in reducing the CI build time. There are also discussions around how to scale horizontally. Right now, we don't think that's something we can do with the service that we're using to run the test. But it's something that maybe we need to manually look into is then how do we build a queue of all these tests and not where we just split test by a file, which is typically how the Parallelize gem does it. But you could actually split up tests within a file. So if you had a particularly large file, that doesn't necessarily matter. But then building a queue of all these tests so then as each test finishes, a worker can just grab that next test. And then also you can easily scale up and scale down workers. As I'm saying that, that feels big, that's a lot to invest in. But that as an idea is how can we essentially then scale the architecture? So even as we continue to invest in the tests, in the system, and they continue to grow, our architecture can keep up with it. CHRIS: That last bit there is super interesting to me. It's something that I've looked into and haven't pursued yet. We're currently running on CircleCI with our test suite. And I don't even know that we pushed on parallelization because we're early enough on that. And we turned off bcrypt recently, which super-duper helps with the speed up. But overall, the test suite time is fine, is where I would put it. It had crept up, though, to a place where it was starting to be painful, is how I would describe it. And I think it's very easy for that to just continue growing and suddenly, it's 20 and 25 minutes. And then, depending on your merge strategy and all of that, it can be all the more complicated, and this gets in the way of deploys. And so, I think it is a super important thing to keep an eye on. I know Charity Majors pushes really hard for 15 minutes from merge to deploy to production. And so if your CI suite takes 25 minutes, then already you're stuck. As an aside, I just once more want to say out into the ether, CircleCI or any other CI platform, if you would allow me to say yes, we've already tested this Git hash, this Git SHA, or the working tree, ideally, because that's also deterministic, I would love that feature. I would love to not have to rebuild the same code when it gets merged into main, just saying once more out into the world. Also, GitHub, if you want to put me on the merge queue beta, I would love that if anybody out there is listening. [laughs] STEPH: I like how this has become a special requests hotline for all the things [laughs] that you're hoping to get a part of or features you'd like to see added. CHRIS: Hello, internet. I have some requests. STEPH: [laughs] CHRIS: I would love to see those things, but in the world where those don't exist. The particular thing that you're talking sort of a test queue, is something that I've seen. So Knapsack is a...what's the word? It's a tool; it's a service. It's a combination of things. But it does that essentially where it starts up a local build agent. And then it basically says like, all right, give me all of the tests that you need to run, and then I will feed them back to each of the individual agents that there's one agent running per parallelized process. And so say you've got five of them. The first one says, "Hey, give me a test," and runs it. And the second one says, "Give me a test," and et cetera. And so, the queue manager on the other side is in charge of that orchestration. And it means that they basically all finish in identical time, with one being an outlier, whichever one happens to be the longest. But it's only going to be however long your longest test is is basically that outlier versus what you're describing of like, well, if we split it by file, we can end up with more naive things where there's a bunch of feature specs on one of them, and it skews by two minutes. We obviously don't want that. So Knapsack, in particular, is a tool that I've looked at, but generally, I'm very interested in that as a solution to how do we maximally take advantage of parallelization there? STEPH: Interesting. I have not heard of Knapsack. There is one that sounds similar. It's called RSpec Queue. And it does some really interesting work where it will split the individual test, so it won't do it by file. It will also look at historical data to then try to be intelligent about how it's going to split it and find the longer running test. And I believe it uses Redis to then keep track of the test set up in run and things that still need to be run. That is a gem that the team is looking into using as well. I don't know how that works if that can integrate with the current platform as we're using TeamCity to run tests. I don't know if that's something that can integrate with TeamCity, if it's a replacement. I don't have all of the knowledge about RSpec Queue yet. But it seems to do a number of the things that we're interested in. So even if we can't use the gem, then maybe it's something that we can still imitate. CHRIS: The other thing that I'm surprised we haven't said yet is this is one of the places where people would often reach for microservices. I feel like we have to have the microservice conversation at this moment. Microservices can actually be a great solution to organizational problems. As a team scales, it does become really hard to manage a large group of developers. And so microservices introduces a very fixed boundary that then draws nice lines that you can have around things. And so, the individual build time for a portion of your application can be much more manageable by virtue of that. But it has this huge cost of technical complexity and overhead and et cetera, et cetera, all of the reasons that we may not want to go that route. And so interestingly, I was just looking at Shopify's Deconstructing the Monolith blog post, which I think at this point, they've skewed a little bit more into the microservices. Shopify is huge, one of the largest Rails apps out there. And so looking at them and being like, oh, what are they doing? It's an interesting sort of plot a course and to see how long they waited before they even started thinking about the much deeper things and even exploring microservices. But in this blog post, they talk about a different approach where they stuck with sort of a monolith. But then they started to introduce Rails engines and clear encapsulation within the large codebase such that then you can actually start to say, well, we don't need to run all the tests every time because if we're making a change within this section of the application, then we just need to run those tests. I've also heard of organizations having some logic that can determine based on the code change; we know the associated test files that we should run. I'm scared of that is how I would describe it. I want to trust my test suite. I want to be able to deploy on a Friday and say if tests are green, it's going out to production. That's great. And I worry about that sort of thing. That's hard to get right. That feels like caching, right? And that's one of those things that we historically get wrong a lot. But nonetheless, that is an approach that large organizations I've heard having good success with. So some way to determine what's the affected code and what tests do we need to rerun and et cetera. And that can really drastically reduce down the scope of each CI build. But those are some larger things that I have not had to reach for on any of the applications I've worked with. I've taken different approaches, different ways to reduce the time or otherwise Parallelizer et cetera. But it's interesting for when you get to a certain scale. STEPH: Yeah, it's funny that you bring up that idea because that came up in conversation with some of the other developers as well, was the idea of, like, what if we could just not run all the tests? You changed one file, and you don't need to run everything. And I immediately was like, that sounds very cool and super hard to be able to get right. And a lot of this code is extremely coupled, which then moves to the code quality area. So I suspect a lot of the test times could be improved by creating smaller objects because right now, a lot of the tests will load the entire world because they have to. They have to test everything. And so that is creating a ton of data, and then taking a long time to run versus if we were able to split out that code into smaller objects and test in unit tests, then that would also help speed up. But that's also hard to do. Where do you look first? We do have some great data, thanks to RSpec. RSpec is letting us know how long each test file takes to run, and then we are capturing that data. So I can go look at which files and say, oh, this file takes 10 minutes to run. Let's look at that file first versus some of the other ones that are performing better. But that is a battle that will take a long time to win. And it's something that takes consistency and then also encouraging others to join that battle. So while it's very important, it doesn't address the concern of tests growing rapidly and then being able to support that. Something that you said in a previous episode also was on my mind in talking about building processes in a way that encouraged people that they can make small, quick changes. And I think that's really important. So if we can build out the architecture to help scale this so then the tests were running in say 15 minutes, then if someone saw a test and they wanted to make a small refactor, they saw a factory.create, and they're like, oh, that could be a FactoryBot.build_stubbed instead and issue that into a pull request or change request and get that merged. I don't know if people feel as comfortable doing that right now because it takes them 30 minutes or longer to run the test. But that idea of how do we get a structure in place where people can make tiny, little improvements and do that as a whole, as a team, to then work on the code quality concerns? CHRIS: That last little bit is so interesting where you're saying, like, oh, we have a FactoryBot.create, FactoryBot.Build, but it has the overhead of having to go through the 30-minute test suite. But coming back to the thing we were talking about before, what if we didn't have to run all the tests? Although I find it very hard to tell, given a code change in actual production code, what tests do I need to run? When I'm just changing a test, I'm pretty sure I know which test I need to run in order to determine if that test still runs correctly. So that feels is there an optimization that can happen there? Which is I've only made test changes; therefore only run the changed tests. And then that's an encouragement to say, like; this is a part of our codebase that we are trying to improve on. Let's optimize the iteration speed there. You'd have to figure out how to write that. And so it's probably much like my productivity adventures, maybe not a good investment. Although given that this is such an organizational concern, maybe that is the thing that's worth spending an afternoon on and seeing if it could happen. Because if you can speed that process up, get more [inaudible 23:46] and more iteration in fixing the tests, that feels like it could be a win. STEPH: I think that's a really good idea. I think we could certainly tell that if a file's changed, that it's only a test file that has changed. And then I've heard very good things from the other developers that TeamCity has a wonderful API to work with. And so there's a way that we could then tell TeamCity to say, hey,...or it may not even be a TeamCity command. It may just be somewhere in the universe we have to say, "Hey, RSpec, only run this test," or "TeamCity, we're only going to feed you this one RSpec test to run, so user agent but only run this particular test." So I really like that idea. I think that's really intriguing. And I'll bring it up with the team because that would be a huge win, especially as Joël and I are really focused more on tests. That would just improve our lives. So selfishly, I'm excited about that idea because we are touching less of the application code and more focused on improving the test at this point. CHRIS: I mean, if right now you're getting, say, 5 or 10 pull requests through a day which frankly feels like a high bar on this, if suddenly that's 10 to 20, that's material right there. STEPH: Yeah, I don't know how large of an impact it would have for the rest of the team because I don't know how often they're only making changes to a test file, but it still feels like a nice optimization to have. Cool. Well, thanks. I appreciate that idea. CHRIS: My pleasure. Mid-roll Ad And now a quick break to hear from today's sponsor, Scout APM. Scout APM is leading-edge application performance monitoring that's designed to help Rails developers quickly find and fix performance issues without having to deal with the headache or overhead of enterprise platform feature bloat. With a developer-centric UI and tracing logic that ties bottlenecks to source code, you can quickly pinpoint and resolve those performance abnormalities like N+1 queries, slow database queries, memory bloat, and much more. Scout's real-time alerting and weekly digest emails let you rest easy knowing Scout's on watch and resolving performance issues before your customers ever see them. Scout has also launched its new error monitoring feature add-on for Python applications. Now you can connect your error reporting and application monitoring data on one platform. See for yourself why developers call Scout their best friend and try our error monitoring and APM free for 14 days; no credit card needed. And as an added-on bonus for Bike Shed listeners, Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at scoutapm.com/bikeshed. That's scoutapm.com/bikeshed. CHRIS: What else is going on in my world? I continue to not code a ton which is interesting and probably makes sense for right now. But to share a small anecdote from this week, we had retro, and I ended up attending retro ever so slightly late. I was doing a hiring interview, which is super exciting. Again, for anyone that's out there, we are hiring at Sagewell Financial. And I would love to chat with you if that sounds interesting. But so I was having a wonderful hiring conversation that ran a little bit long. So I was a little bit late to retro, and I arrived, say like eight minutes in, and someone was expressing a concern. And the concern was, I very sincerely know this to be true, but they were saying in the most positive way. But they were like, "It'd be great if Chris could code more," and not in the judgmental like, Chris, why are you not getting as much done? Not in that way at all, very much in the it would be great if Chris had more time, if there wasn't as much pulling my attention in different directions. But then it kind of went into this interesting direction. So we then go back through and address the concerns and talk as a group about how we resolve them. But this one was like, my name was in the concern, again, in a very positive way, in a very supportive way. And we had a wonderful conversation, and there were really great ideas that were passed around. But man, did I feel weird having my name in a retro item. [laughs] STEPH: So one thing I've learned is that you do a really good job when you are giving presentations and being in the spotlight. But I don't think you actually love it. You love sharing content and things that you have learned. But I could see how being a focal point, especially if there's a concern or something that could have a negative connotation, that would feel squeamish. It would make me feel squeamish. CHRIS: I hadn't thought about it in that way. But as you say it, also, this conversation is a meta version of that. Like, let's talk about me talking about me. I don't want to be the center of attention. But I love technology or process. I love talking about the work. That's great. And so I'm happy to do that. I'm happy to stand in front of a room and talk about it. But yeah, when it's about me, that's weird. And so now I'm going to move...well, no, I'm not going to move on [laughs] because this is the topic right now. But so there's a bunch of things that we have been trying to introduce. And I think this is a useful part of the conversation more broadly and less about me. So one of the things that I think I mentioned in a previous episode was the introduction of point-dev, which is each week, we rotate through a person. And that person is in charge of triaging the errors, making sure that nothing is stuck in Sidekiq, responding to any support requests, et cetera, et cetera. But they're meant to be the frontline such that everyone else can be heads down and really focus on the work. And what was interesting of the three developers that are working on the project, I am point-dev this week. So I was like, yes, that's awesome this week because I'm the person on the frontline. That has not helped me, but in the future, it will. And then one of the other developers mentioned that they feel like it's really useful but also feel like it's been noisy. And we realized the previous week was their week on point-dev. But the other developer was like, "Yeah, it's been great. I haven't had to think about anything." And so they have been off of that rotation for two weeks now. They'll be taking it over next week. But it is doing exactly its job of providing that attention coverage so that they can keep their focus on the code, and that's really wonderful. So I'll be honest, when we started talking about it, there was a tiny voice in my head that was like, is this a failure mode? Should we be dealing with the noise rather than having a process to address it in the moment? Should we be dealing with the root cause rather than the symptoms? And I still think that's a good point of view. But we found so much value from this. And as I've mentioned it, many people are like, oh yeah, we have that. It's great. I've heard enough positive things. So I've backed away from that. But there was a voice in my head that was like, are we failing right now? But yeah, so point-dev has been really wonderful. And next week, I will have to...well, frankly, the next two weeks, I'm off of point-dev appointments, so I'm very excited about that. I've been doing some of the product management or sort of the tech side of the product management and helping to triage cards and make sure that there's very clear work lined up for the engineering team when they're ready to do that. I'm trying to back away from that just a little bit. And one of the things that we did there was introduced an inbox column in our Trello board. You know how I love a good inbox. You know how I love to get to inbox zero. But that is a good way for me, for anyone now in the organization, which I don't want everyone to have to learn our processes, but just saying, "This is the place that you put requests, and we will deal with them. I assure you of that." It has been great because that means I don't need to be quite as responsive in Slack. I can just gently redirect people, "Hey, if you don't mind, please put this in Slack in the inbox column, and that'll be great." That thing, though, that gentle pushback in Slack is one of the things that I've struggled with. And this was one of the more personal aspects of the conversation that happened in retro was me being, like, if we're being honest, I tried to do that. But it's not my favorite thing to do in the world. Whenever someone asks me something, I want to be helpful. I don't want to seem rude or brisk or like I'm too busy for you, et cetera, et cetera. So I will often respond to the question or do the thing that they're asking and then say, "In the future, if you could go to this other place." And ideally, I'm slowly moving forward and being like, "No, no, no, please go to the other place. We've talked about this a few times." But it is an interesting example of one of the specific aspects of my personality coming through in this. But that introduction of an inbox has been great. Love me a good inbox, as I said. And then, more generally, we just tried to talk through what are the things that I'm doing? Do I need to own all of those uniquely? And some of them the answer we decided was yes but some of them we decided no. And we started to sort of distribute the work there or some of the meetings or different aspects of it. And so overall, it was a really great conversation but also very weird for me. STEPH: Yeah, because then you wonder, am I not doing the right thing? Am I not spending my time the right way? But then hopefully, that meeting helped reinforce that yes, you are spending your time the right way and that you're doing a lot of productive things. There are just too many productive things for you to do, and so you have to prioritize those aggressively. I like all the things that you just highlighted. There's one in particular, the last one that you mentioned about finding things that you can hand off to others. And I love that for a couple of reasons. It came up in a recent conversation that I was having with some other thoughtbot developers around when someone's on a project, typically someone just falls into being the point person. They just happen to be the person that the client talks to and ask questions and goes through the most. And that's something that is okay. But we want to make sure that that's not a bad thing, that everybody is treated equally, that everybody is given equal opportunities and room to grow. And so, in my mind, whenever someone is that point person, or you have fallen into that role, it is your job to then pull other people up. So if you have been given the responsibility of running a particular meeting each week, then go ahead and do it once or twice, so you can demo it and show it to someone else as to how you do this. But then tag somebody else and say, "Hey, I'm going to let you or ask you to run this next time." So then that person can experience it. They can demo their style, and then it continues on to have more people. So I really like that you are highlighting it's not just beneficial for you to then distribute those tasks, but it's empowering for everybody else on the team as well. I'm curious, so what was the final outcome? It sounds like there are some really good things in place, and you're transitioning, handing some things off. But I can't imagine that things have gotten...all of your priorities are still there. So do you think you'll actually code more, or what's the outcome for next week? CHRIS: Short term, maybe probably not, if we're being honest, but trending in that direction. So one of the things that's going on right now is hiring. That is just an activity that takes a lot of time. And I care a lot about doing that well, both for the organization and then for individuals on the other side. I want to be respectful of their time and communicate in reasonable timelines and not leave people without an answer or follow up or those sorts of things. It probably makes sense for that to sit with me as the starting contact. And then from there, folks that are continuing on in our hiring process they're going to talk to many other members of the team, and that won't just be me. But there are a lot of first conversations that I'm having. And so right now, my schedule has a bunch of that, which is fine and good. And that will hopefully, at some point, we'll hire some great people. And then we'll be on the other side of that. And that piece of the work that I have right now goes away. Some of the other outcomes that we named there were a couple of action items. And so I think those will help, but they're sort of we got to work towards that. One is transitioning a meeting, but it's a biweekly meeting. And I'm not going to just not attend the next one. So it'll be me and one of the other developers attending to transition ownership of that meeting moving forward. And then from there, so like, two weeks from now, I will not have that consideration on my calendar. And that's like one 30-minute block that I get back or, depending on how you think about it, one block that that 30-minute broke up. I do want to touch back just on something that you're saying there. I think you're being very kind to me in saying like, no, but you've got so many things, and so it's hard to do that. I think that's true, but that's kind of the work overall, and my version of that is one thing. But everyone sort of has, as a team, we have a version of like, how are we being most productive? Are we making sure we're doing the most important things? And so it was interesting in the moment, but I think it was a very good conversation. And I want to make sure that both we as a team and then me as an individual, wherever that happens to be the case, are open to these sort of constructive things. Like, frankly, to do the work to figure out how to get work off my plate that hasn't felt like the most important thing. It felt like close to the most important thing, but then there were all the other things that I had to do. So I wasn't doing the work to figure out how to not do the work. It is a complicated sentence that I just said. But this was a case where retro, I think, very usefully highlighted that this was a good thing for us collectively to put effort into such that we can be more productive moving forward. It happened to be slightly more focused on me rather than the entirety of the team. But broadly, that kind of thinking is why I'm a huge fan of retro. I think it's a great place to take a step back think about how we're doing the work rather than just being in the work day-to-day. STEPH: So if I'm internalizing what you said correctly, let me know if not, but it sounds like you're in one of those places, and I've witnessed this with other people and myself where someone is overwhelmed. They have a lot to do, and they're very focused in that grind and in that moment of doing all the things that they have to do. And it's very hard to then say, "I'm in the weeds right now. And then I also have to figure out how to get out of the weeds." And that's a very different skill and mental space to be able to do that. Because often, when you're just in that mode, all you can focus on is a bit on survival at that time. And then it may take other people to notice to say, "Hey, you're in the weeds. We need to figure out a way to help you not live there and to find ways to distribute some of the work." Does that sound like a fair assessment? Because I think I say all that because I've just seen people in that position. And then they think back, like, oh, I should have offloaded stuff earlier. And it's like, yeah, true, totally. And it often takes a retro or someone else coming to you and saying, "Hey, I've noticed...I looked at your calendar today; how can I help?" [laughs] CHRIS: I think that's probably the right calibration. And mostly, my emphasis was just I want to make sure that broadly, any team that I'm on has the space for this sort of conversation. And that thing that you're saying exactly that phrasing of like, "Hey, I saw your calendar. How are you doing? How's that going, though? Are you feeling okay? [laughs] You can't sleep and whatnot." That can be a really useful thing to have and to have organizational norms about what are our expectations of how many meetings someone should have in a week. And where do we start to think about different things? You did use the phrase overwhelmed. I want to say that I'm like 101% whelmed. So I'm just ever so slightly overwhelmed, but it is like I'm in the weeds. I need to figure out how to clear some of the weeds so that then I can get out of it. And it was a great conversation that came from that. STEPH: That's awesome. I'm glad you got a good team that, frankly, felt comfortable bringing it up, and then that you could lean on them for ways to talk about how you could code some more and talk about priorities and where you want to focus your time. CHRIS: It will be an interesting thing. As the team grows, I don't expect this to get easier. We talked about this a number of weeks back. And I think for a while; hopefully, we clear a little bit of dust here, and then I get back to being a little bit more on the code, and that's going to happen for a while. But as I think about the longer sort of the future of the company, this is something I'm going to have to revisit a handful of times. And it's a really interesting question that I'm still struggling with internally. And where do I want to be versus what will be needed and whatnot? So it'll be interesting to see how it evolves. But for now, I think I can gain back a little bit of coding time, a little bit of maker time versus manager time, as Paul Graham's essay goes. And yeah, I think that'll be good. STEPH: Yeah, I like how you're already looking forward to the fact that it will probably fluctuate because, yeah, right now, you are sort of paying a tax. You are building up to then where you can have more people on the team. And then that may give you back some of your time where then you can code because you can outsource some of the work to them. But then, as the team grows, so are other responsibilities. And traditionally, being in a CTO role and most CTOs I know will code here and there because they want to, and they enjoy it, but it is not their full-time job. So I think you're really wise to have already noticed that and start thinking about how that's going to trend in the future. And it sounds like you might need to figure out how to throw some architecture at it. So then you can scale horizontally, and then you can just have more time to do all the things. Yeah, that's right. [laughs] CHRIS: You're suggesting microservices, right? That's how my job becomes easy? STEPH: Yeah. Well, I'm thinking more like RSpec Queue, but we'll have RSpec Chris or some version of that. CHRIS: Chris Queue. STEPH: Chris Queue. [laughs] CHRIS: And then I just paralyze my human, and then it'll be great. STEPH: Yeah, that's always worked out well in the movies. Whenever somebody clones themselves, that goes super well. CHRIS: Multiplicity is a fantastic piece of cinema, and I stand by that. STEPH: I haven't seen it, but I feel like it doesn't end well for the main character. CHRIS: I feel like every time I mention a movie, you haven't seen it. I feel like we need to do a movie marathon at some point just to catch up so that we've got shared analogies. But yeah, it's a fun movie. It's fine. It turns out fine in the end. But there are some humorous adventures that happen in the middle. Cloning maybe [laughs] isn't the most direct option to solve productivity problems. STEPH: [laughs] Yeah, I think I've got Labyrinth, Hackers, and Multiplicity now on the watch list. And I appreciate the fact that you know that I'm not likely to watch them, although out of the three, Hackers will probably happen. CHRIS: All right, what if I were to get a bunch of Pop-Tarts, non-frosted? STEPH: Ooh. CHRIS: Does that change -- STEPH: Wait, are you going to send them to me? Because if you just have them, that's no good. [laughter] CHRIS: Eat Pop-Tarts on a video call and be like, "Look at this movie. It's great." [laughter] STEPH: All right, bribery definitely works for me. [laughs] CHRIS: Okay, so got it, noted. And based on the nature of the conversation that we have devolved into here, I think we've probably reached a good point. What do you think? Should we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Byeeeeeeeeeee!!!!! Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

god love time hackers cto slack shopify api mid labyrinth coding python ui github deconstructing spectacle rails ic trello pop tarts monolith cloning git circling mandy moore microservices apm vim paul graham multiplicity redis omnifocus divvy rectangle circleci charity majors knapsack moom chris you chris well chris short sidekiq rspec is it worth teamcity chris thanks chris okay chris all chris toomey

03 выпуск 10 сезона. Porting YJIT to Rust, Webpacker has been retired, Remix vs Next.js, Vanilla List и прочее

Play Episode Listen Later Jan 24, 2022 42:01

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Porting YJIT to Rust Webpacker has been retired Happy 10th Birthday, Sidekiq! Bad Ruby: Hash Value Omission Reduce your method calls by 99.9% by replacing Thread#pass with Queue#pop Lecter - show executable code by request OnlineMigrations - catch unsafe PostgreSQL migrations in development and run them easier in production PostgreSQL бесплатные книги Web Remix vs Next.js TypeScript Features to Avoid Replacing jQuery (110kb) With Umbrella JS (8kb) Vanilla List - a Directory of Vanilla JavaScript Plugins Open sourcing Chirpy CSS Fingerprint

remix retired rust thread vanilla queue directories postgresql lecter porting sidekiq webpacker

321: Leaving Breadcrumbs

Play Episode Listen Later Jan 11, 2022 40:20

Steph tells a cute story about escape artist huskies, and on a technical note, shares a journey in regards to class variables and modules inheritance. Chris talks about how he's starting to pursue analytics and one of the things that he's struggling with that he's always historically struggled with is the idea of historical data. He's also noticed a lack of formalization of certain things and is working with his team to remedy that. This episode is brought to you by ScoutAPM (https://scoutapm.com/bikeshed). Give Scout a try for free today and Scout will donate $5 to the open source project of your choice when you deploy. Mike Burns: How to Skim a Pull Request (https://thoughtbot.com/blog/a-smelly-list) RSpec Documentation (https://rspec.info/documentation/) Don't Let the Internet Dupe You, Event Sourcing is Hard (https://chriskiehl.com/article/event-sourcing-is-hard) Datomic (https://www.datomic.com/) timefora_boolean (https://github.com/calebhearth/time_for_a_boolean) Sentry (https://sentry.io/) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: CHRIS: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, it's an entirely new year. What is new in your new year? STEPH: Well, the year is off to an interesting start because we helped rescue a husky. CHRIS: Rescue as in now this is your dog or rescue as in the dog was trapped in a well, and another dog told you about the dog being trapped in a well, and then you helped the trapped? [laughs] Which of those situations are we working with? STEPH: [laughs] I'm really wishing it was the second version [laughs] where there's a dog that tells me about another dog trapped in a well. No, this is a third version where there was a husky that was wandering around the gym that we go to. And so Tim, my husband, called and said that "There's this husky, and he's super sweet, but he seems very lost." And our gym is located near a major road, and so we were worried that he was going to wander about and get hit. So I hopped into our car and took a crate and a leash, and he hopped right in. Clearly, he belonged to somebody; he'd just escaped. So he hops right in, and then we bring him home. But I put him in the backyard because I want to keep him separate from our dog, Utah, just because I don't know this dog, and I want to keep him safe. And I go back inside to grab a few things. I come back out, and the husky is gone. And I'm like, well, shit. [laughs] Now I'm starting to understand why this husky is missing or why this husky seemed lost. So then I started looking for the husky, and Tim comes home. He's helping me look for the husky. And it was one of those awful moments where we live near...it's not a major road, but people tend to speed on it. And the husky and I happen to see each other across the road. And so the husky was like, oh, human friend and starts coming across the road towards me. And there's this large SUV that's also coming from the other direction. I'm like, oh, this is it. This is my nightmare. This is becoming real. This dog is about to get hit. Thankfully, the driver saw the husky and stopped in time, so everything was fine. And the husky just finished trotting across the road to me, brought him in, kept him in the kennel in the garage. We didn't have any backyard adventures after that. The husky then thanked us by howling most of the night. [laughs] So this poor husky has had an adventure. We've had an adventure. And then, around 4:30 in the morning, I go out because I'm checking on the husky and going to let him out. And I'm scrolling on the app called Nextdoor. And I see that someone posted a picture of this exact husky that's like, "Please help me find my dog." And I was like, yes. Because we were going to have to take him to a county shelter or at least go see if he had a chip so then we could return him. But thankfully, we found the owner. I found out the husky's name is Sebastian. And then we had him for a few more hours, and then we had a wonderful husky and human reunion. CHRIS: That story had everything. It had ups; it had downs; it had huskies. It had escape artist huskies, in fact. I have...this is only through Reddit because that's how people learn about things in the world, but huskies are a rather vocal dog breed. So when you say the dog was howling, huskies have a particular way of almost singing, and it kind of sounds like yelling rather than more traditional dog sounds. Was that the experience you had? STEPH: Luckily, it wasn't too bad. His howling was more just; he didn't want to be in the crate. He seems like an indoor dog. So he's like, what am I doing outside in the garage? I should be indoors. And so he wasn't too loud. It was more he was just bemoaning his situation. But our dog Utah could hear him upset in the garage. And so that was also getting Utah upset because he didn't understand why there was a dog so close. And that was what led to the sleepless night because we couldn't get both of them to calm down. Because then, as soon as one of them calm down, the other one would get the other one riled. CHRIS: As it so often happens. STEPH: I'm so grateful that it turned out to be a happy story, though. That part was wonderful. And if we see the husky again, now we know his name is Sebastian and that he'll just come home with us. [chuckles] And we'll know how to return him since he seems to be an escape artist. CHRIS: And we were best friends forever. STEPH: On a more technical note, I have quite the journey to share in regards to class variables and modules inheritance. But before I dive in, I'm curious, what's new in your world? CHRIS: Oh. Well, I'm excited to dig into that story. But I've got two smaller things in my world this week that are top of mind. I don't really have answers on them. I have more questions. One is we're starting to pursue analytics. We want to try and understand our system a little bit better. What is the experience of our users? How are they coming into the system? What are they doing? How long does it take them to do the things that we want them to do? All those sorts of questions you want to be able to answer about your application. And one of the things that I'm struggling with that I've always historically struggled with is the idea of historical data. So data changes over time, and often we actually want to know about those transition points. We want to know about the different states that a user or any record in the system has been in. And I'm finding myself feeling the same pain that I felt many times and starting to think again about the relevant options out there in the world. To give a slightly more pointed example of what we're dealing with, users come in, and then there are a few steps for them to actually sign up for the application. And so their user record or their application, if you will, will go through a couple of different states. So they can be basically approved directly, and now they're an active user of the system, that's one option. But they can also end up in a state where they're pending review. And then eventually, depending on the outcome of that review, whether it's manual or someone intervenes or what have you, then eventually they can transition to either being denied or being accepted. And then they'll again be an active user. And so there's a question now of how many of the users that end up in that pending state end up transitioning into active. And as I looked at the database, I was like, I do not have this information right now. I know their current state. And the logs could tell me all of this. We don't have proper log archiving right now. And I also don't have a system for, like, let me pull down gigabytes of logs and try and sift through that to understand the answer, especially for something domain level like this. But this is one specific example that represents a category of things in my mind. The stuff that I've looked at in this space otherwise is Event Sourcing. So the idea that rather than having a discrete representation of the state of your application, you store every event as an individual log, essentially of like user did X, thing happened, Y occurred. And then, at any given point, you need to know about the state of your system; you just reduce all of those events through some magical reducer that produces the current state. I also very recently read an article called Event sourcing is Hard. So I have that in my head as a counterpoint. This seems like a thing that is non-trivial to do, makes sense for a certain scale. But of course, like anything else, it has its trade-offs. Another thing that I've looked at and never really pursued mostly because it's in a different ecosystem, is Datomic, D-A-T-O-M-I-C, which I think I've mentioned before. But it's a database that actually stores data in this historical format. And so you can ask for the current value, but then you can also ask for what are all the states that this user has been in? And what are the timestamps of those changes? One small thing that we do have that I really like...so this is one example of us; I think leaning into wanting to have more information, higher fidelity information, is often we want to know something like was this ticket paid? Did someone pay for this ticket? And so paid is a BooleanProperty on this ticket record within our system. So the ticket can be held for a little while and eventually gets paid. And now, yes, it has been paid for. It is good. You can use it. But often, we want to know not just that it's paid but when it was paid. And so there's a gem that we are using on the project called timeforaboolean by former thoughtboter, Caleb Hearth. And it does a wonderful job of basically instead of storing a Boolean value in the database, you store a timestamp. But then the Boolean can be inferred. If there is a value, if there's a timestamp for that record in the database, then there are a bunch of helper methods that get introduced that say, like, paid? That's now a method that I can ask, and it will tell us that. But we can also find the paidat, paid_at value. And so we have this higher fidelity data when we need it, but we can also collapse it down to the simpler representation. Because most often, all we need to know is, have they paid for it? Cool, then they're good. They can come into the concert, that sort of thing. But yeah, this is a broader question that I don't have a great answer to. I think Postgres and Rails and just the nature of how we approach these applications pushes us in a certain direction. Another thing I'm exploring is downstream analytic systems. What if I send a bunch of events to them, and they act as a half-event sourcing type thing? But yeah, this is going to be, I think, an open question for me for a while. STEPH: Yeah, you said a lot of really good options. When you're talking about in our ecosystem, we get pushed in one direction or the other that makes me think of the projects that I've been on. Typically, what they'll reach for first is something like a Papertrail. So then, that way, they can check for the historical versions of an object and how it was changed and see who changed it. That's one way to track the logs. I like the idea that if you can outsource it and send all of those events to a logging system and then essentially ask for that data back as you need it. You made me think of a recent project as well where we needed to track the state. So it was a patient matching system. And we really needed to know when a patient match was created or disconnected and then who did that and perhaps for what reason. And to ensure that we had as much information as possible, we took that opportunity to just create a record for it. So we had a patient match record or...I forgot the name of the other one where we created where a patient did not have a match. But we were creating a record every time someone did that. Granted, probably that's not going to happen nearly as often as someone paying for an event or the situations that you're describing. This was ideally infrequently that someone was going to unmatch a patient because it meant that our system had matched people that shouldn't be matched, and then a human had intervened. But yeah, it's interesting the space that you're in. And you listed all the good things that I would have thought of. CHRIS: I think you listed Papertrail, which is one that I hadn't actually thought of yet for this particular instance. This only came up earlier today also. So this is new in my head that I'm really being pushed in this direction. But I think Papertrail could be a good solution for where we're at. But it is one of those where you often don't know the thing you want to know. And I'm terrified of losing data of like; I had the data. I knew it at one point in time, but now I can't reknow it in the future because I didn't write it down. That's one of the things that I just don't want to happen in the world. And so finding those ways of like, how can we architect a system so that we can do the normal, straightforward, boring things most of the time but then when we need to expand out the analytics dimension of the system that we're working on...and trying to thread that needle and find the ideal optimization on both sides is a tricky one. But yeah, I'll definitely take another look at Papertrail and see if that...at a minimum, I think that's a good solution for where we're at now. And then this is going to be a thought that's going to roll around in the back of my head for a while. So if I come up with anything else, perhaps a grander solution, I'll certainly bring that back to The Bike Shed. But yeah, what else is up in your world? I want to hear the story of the class variables. STEPH: Well, it is quite a journey. So I hope you're ready. Specifically, I was pairing with Joël, who was working on fixing a test that had been marked as being skipped for a while. We weren't really sure why. We figured maybe because it's flaky. But then, as Joël had restored that test, he realized it was actually failing consistently. So it was a test that was failing for a reason folks maybe didn't understand, but they decided to cancel or to skip that test. But they didn't actually want to get rid of it because it seemed like a pretty important test based on the description. So Joël saw it and got excited because it seemed very relevant to some of the work he was already doing. So then, he is now investigating why this test is failing consistently. So in this story, we have four main characters: we have a class, two modules, and a class variable. So enter the class stage left. All right, so this class defines a class variable which I have to say is not something I work with very much in Ruby. So class variables kind of felt a bit novel and diving back into like, oh yeah, these are a thing. So the class defines a class variable that's called cache and assigns this variable to an instance of a cache. So then this class includes two modules who we'll call Module A and Module B. And we'll enter them stage right. And both of these models look to see if cache is already set. And if it's not, they also set the cache class variable. So with that information, in our test, we don't want to exercise the real cache just because then if other tests are reading from that cache, which is proving to be a source of flakiness for these tests, then they are overriding each other's expectations, and it's causing some of the tests to flake. So instead, we want to use a fake cache, just like an in-memory cache. So the test and its setup is already overriding. It's setting that class variable to say, hey, I want you to be a fake cache, just be in-memory. However, while executing that test, one of the modules is checking to see if that cache is set, which is being set in our test setup. So test setup sets the value. We're running the test but then in the module, the model checks to see if it's set, and it's suddenly nil instead of using the cache that we had set. So now it's defaulting back to say, "Oh, it's unset. So let me go back and set it to the real cache," which is exactly what we're trying to avoid. So then the question became, if we're setting the class variable in our class, why is it being populated in one of the modules but it's not being populated in the other module? So one of them has it set to the in-memory cache, but the other one does not. So I'm going to gloss over some of the details because this stuff is pretty tangling. But essentially, when the test is running, and it's loading the class, and we are overriding that class variable, it's getting shared with one of the modules because as soon as one of the models does set that class variable, there's a bidirectional link that gets set between the parent class which is the module in this case, and the class itself. And as soon as that module sets the class variable, then they're going to talk to each other, and they're going to reference the same value. However, this only seems to happen for one of the parents. You can't do this for both. So if you have two parents that are trying to share a class variable with the same class, that doesn't work. So that's a particular bug that we were running into. I do have some good news because if anybody is very nervous about the situation that I'm describing, I feel you. The good news is that in Ruby 3, they actually warn when this is happening and have introduced an error. So you don't have this inheritance confusion that can come out of the fact that these parent classes are also trying to share a class variable with this child class. So in Ruby 3, if you are writing a class variable in that class but then you try to overwrite that class variable in the parent of that class or by the module that's being included, then an error is going to be raised. So it's going to warn you if you're creating this bidirectional link between those two class variables and that you shouldn't be overriding the child's ownership of that class variable. Instead, if you're going to use class variables, which, one, is not my cup of tea, but if you're going to use class variables, it should be defined in the parent class, and then it can be shared downstream in the inheritance versus trying to go upstream and then having your ancestors essentially override some of those class variables. So all of that is to say we were on a very interesting journey of understanding how class variables work, how the inheritance works, how that bidirectional link is getting established, and then how Ruby 3 comes in to warn us if something funky is happening. CHRIS: Oh, that is interesting. And I'm now going to catalog that as a piece of information that my brain will retain for roughly the amount of time that we are recording this podcast and then immediately forget. STEPH: As you should. [laughs] CHRIS: It's one of the reasons that I try to avoid inheritance. And I try to avoid class variables as much as possible because of this category of problem, a very subtle bug that you have to try and really hone in. And you have to be very smart to debug this sort of thing. I don't want to be that smart. I want to code in a way that I can be less smart on any given Thursday. That's my goal in life. I will ask one other question, though. So there's just a cache that this class and pair of modules are hanging around with, and then you want to swap it out for in-memory. This sounds remarkably like the Rails cache. Is this cache distinct special? Could it not just be backed by rails.cache, THE cache within the rails context, which can be backed by Memcached, or Redis, or in-memory when you're in tests, or the NullStore, which I think is the default in development is probably how that goes? Is there a particular reason? Is this a special cache? Is there additional behavior that this cache has beyond the normal thing? Or is it just like, at some point, someone's like, oh, I need a cache. I'm just going to use a class variable, that'll be easy, which it definitely is, but then you run into complexities. And caches are one of those hard things to get right. So it's one where I would immediately be like, whoa, whoa, I would love to not make up our own cache here. So I'm wondering, is there a distinct reason, or is it just this happened, and here we are? STEPH: So I think we are using a custom cache that we are pointing to. So it is another service. It's not a Rails cache or an abstraction that we can point to and use. It is a different cache that we are using. And I'm trying to think back to the exact code. But there is a method that essentially checks to say, hey, should I use the real cache? Should I use the in-memory cache? And that is something that we've explored to find a way to make this more global for the test suite because we really want to control this for all the tests. Because it's very easy to not realize in the test that you should avoid using that shared global cache. And so that way, the tests don't interact with each other but instead always use an individualized cache for each test to make sure that it is self-sufficient and independent. But we haven't gotten that far yet in figuring out how we can take a more global approach with this. CHRIS: Gotcha. So I don't know the details. I assume there are reasons here. But just to play this out, if we find ourselves saying we have a reason to have a distinct cache, to have a special cache over here, but it's a cache...and caches fundamentally, that word always will raise my attention. It will be like, okay, this is a place that bugs will come and aggregate. And we need a distinct one that has special behavior as an external service, or that is just something like in... There's a wonderful blog post that Mike Burns wrote at one point that was about...I think it was something like things that will make me look at your pull request in more detail. And I really loved it because it did capsulate all of these like, yeah, there are good reasons to do everything on this list. But if you do any of them, I will look at your pull requests and be like, oh, that's interesting. Why are we doing that, though? Do we have to do that? Are you sure? Are you triple sure we have to do that? And this is definitely one of those things where caches automatically catch my attention. Even if we're using the built-in cache, I'm like, do we need to? Is that a definite thing? And then all the more so when we're using a custom bespoke one. Again, I assume that there are reasons that there's something special that's going on here. Perhaps the caching behavior is distinct from just it's Redis, and we throw data. And if it falls out the backside, that's fine. Maybe you need entirely different behavior here. But it is something that I would poke at a bunch. STEPH: Yeah, you're asking a lot of good questions. I will have to go back and look at some of the code because we spent enough time in Ruby specifics that I didn't pay as much attention to the cache. Because right now, as we are working on these tests, we're trying to fix just the test without changing the application code, one, because that feels like a safer space. And if the test is flaky, we're just trying to change the test first. But some of these tests we're starting to realize I'm not sure we can fix the test without also changing some of the application code, or the way that we do have to fix the test is really an incentive to back up and say maybe now's the time that we look at some of the application code. Because another question that comes to mind is why use a class variable, and does this need to be shared by the class and the modules? And there's a part of me that suspects that maybe some of this logic was extracted to a module, but then it wasn't cleaned up in the other places. And so that's why we still have a reference. And it's essentially then being shared and set and unset and reset in those different places. So I think you ask some good questions, and I have some more questions of my own when we have time to revisit that portion of the test and application. As another example of some of the tests that I've been working on, one of the tests that I...because we have a list, we can usually tell some of the tests that are flaky. So one of the ones that I was investigating was a similar issue where there was a shared resource, and someone had tried to mock it out. So they had taken the time to say, hey, I don't actually want to use that real resource that's over there; instead, I want to just return the scanned value. But instead, they'd accidentally stubbed out a class-level method instead of the instance-level method. And so it was running, but it wasn't actually stubbing anything else since that's the method that's not getting called. So that was just an oversight for that test. So I fixed that test. But I noticed that we were using allow any instance of, so then I did take the time to go through that file and change and move away from the use of allow any instance of. And for folks that are less familiar with allow any instance of, RSpec has some really great docs that talk about how it's very helpful for dealing with legacy code. But essentially, it is a code smell that you're using; allow any instance of because you are saying that my test is or my code is so complex that I can't really mock out the specific instances that I want to and then return specific behavior. So instead, I'm having to use this more global approach to say, hey, any instance of this method, I want you to mock it out versus this very specific instance that I know that I'm working with. But we can include a link in the show notes because there's a nice write-up that talks about some of the reasons that allow any instance of is not recommended. So that's been kind of fun. There's been a little bit of joy to get to refactor away from that and actually stub out a specific instance. Part of the work, too, that I'm noticing as Joël and I are going through these tests is leaving breadcrumbs for other developers as well because they have a very large team. And they're very junior friendly, which is just incredible. I love that so much about this company. And because they do hire a lot of juniors, then it is a tough codebase. It's a fairly old codebase. So as these juniors are coming in, they're seeing a lot of these patterns. And they're propagating these old patterns that aren't necessarily the best patterns to propagate. But they're doing their best, and then they are reusing what they're seeing. So part of the work as we are revising these tests, my hope is that people will see some of these newer patterns and use those instead of following some of the older patterns. CHRIS: I can only imagine that you're writing borderline novels in your pull request descriptions and commit messages there. I do wonder, is there an index of those that you're collecting? So there's like, here's the test remediation examples list, and you're slowly adding to them. This was a weird one with a class variable. And this was a weird one that had flakiness due to waiting or asynchronous behavior. And gathering examples of those, but specifically from the codebase. I could see that being a really useful artifact because I happily traverse through git blame all the time. But I don't know that that's always a thing. And frankly, I have to work for it sometimes. So if there is that list of here are pull requests that specifically did X, Y, and Z, I think that could be super useful. STEPH: Yeah, that's a great idea. And yes, they have some shared team documentation that speaks to specifically flaky tests because they're aware that this is a problem. They are working together to address this. And they have documentation that states ways to avoid flaky tests. If you encounter a flaky test, here are some of the ways that you can triage to find out what's wrong. So as Joël and I have been finding good examples, then we've been contributing to that document. And they also have team meetings. So our plan is to attend some of those meetings and be like, "Hey, this is just some of the stuff that we've seen this week, some of the things that we improved and changed," and share the progress that we're making. Since everyone is aware that there are these developers that are working hard to improve the test suite, but then share that information with the rest of the team so they too can feel...one, they can just see the changes that are taking place. But they too can also benefit and apply those strategies themselves when they see a flaky test. Oh, but you did just remind me of a thing. So one of the tests that I was going through...I'm very intentionally going through and making the smallest change possible. So I will do the gross, ugly fix whatever it is to get something to pass, and then I will commit it. And then I'll think about okay, well, how can I make this better? So essentially, I have the fix, whether it's pretty or not. And then, after that, I start to have other commits that make it prettier. And so, I had a pull request that had four commits that told the story that I was very happy about and progressed along in a more positive direction. And I issued that, and I discovered that Gerrit, when it sees four commits, it split all of them into their own change request. And so, instead of having what I thought would be this nice story, now got split across these four change requests. And I thought, well, that's less helpful. So I ended up squashing two of them, but I still kept three of them because they stood alone, and each told a story. But that's something that I've learned about Gerrit. CHRIS: Always so interesting how our tools shape our work. STEPH: And it made me think back to the listener who asked the question about ensuring that CI runs for each commit. Well, here you go, Gerrit. [chuckles] Gerrit does it for you. It ensures that every commit gets split into its own change request. CHRIS: I mean, as you said earlier, not my cup of tea but... [laughs] STEPH: Yeah, I'm still lukewarm. I'm still discovering Gerrit and how we get along. Mid-roll Ad And now a quick break to hear from today's sponsor, Scout APM. Scout APM is leading-edge application performance monitoring that's designed to help Rails developers quickly find and fix performance issues without having to deal with the headache or overhead of enterprise platform feature bloat. With a developer-centric UI and tracing logic that ties bottlenecks to source code, you can quickly pinpoint and resolve those performance abnormalities like N+1 queries, slow database queries, memory bloat, and much more. Scout's real-time alerting and weekly digest emails let you rest easy knowing Scout's on watch and resolving performance issues before your customers ever see them. Scout has also launched its new error monitoring feature add-on for Python applications. Now you can connect your error reporting and application monitoring data on one platform. See for yourself why developers call Scout their best friend and try our error monitoring and APM free for 14 days; no credit card needed. And as an added-on bonus for Bike Shed listeners, Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at scoutapm.com/bikeshed. That's scoutapm.com/bikeshed. What else is going on in your world? CHRIS: In my world, we keep adding new users to the system. We keep doing more stuff. These are all wonderful things, the direction you certainly want to be heading. But as we're doing that, I've recognized that we had a lack of process and a lack of formalization of certain things. And a lot of the noise of the work was just coming to me because I was the person that everybody knew. I can ask a question; Chris will know the answer, et cetera. And then there were things that we needed to keep an eye on. But because it was everyone's job, it was no one's job. So we've introduced the idea of a point person on the engineering team. So this is a role that will rotate each week. I think you and I have worked on a handful of projects that had something similar to this. There was a team that we worked with that had an ad hoc list, which were just little tasks that needed to be done by developers. So there was one person who would run with that. I've heard it called captain before, the sprint captain. We're not really doing sprints. So for various reasons, that title didn't work for me. But point person is what I went with here. And so the idea is rather than having product management or anyone else in the organization just individually reaching out to developers, we want to try and choke that off, have a single point of communication. And so just today, I introduced into Slack, a group, but it's a group of one person. So @pointdev is technically the handle for this person. It's a group in Slack. And each week, we'll rotate who the members of that team are. And technically, you could add multiple, but the idea is this is just one person. So we'll rotate the person. And what ends up happening is if anyone...say the product manager says, "@pointdev, what's the status on..." blah, blah, blah, that will notify the person who is the point person that week. So that's a nice feature in Slack so that we can condense it down and say rather than asking individuals, ask this alias. We're introducing one layer of abstraction in our communication tools, much like we do in our software. So I'm drafting now the list of like, here's all the stuff that I think this person...because we're trying to push all of the quote, unquote, "other work" the non-product feature development work into this person's purview for a given week. So it's monitor Sentry for any new errors as they come up, triage them, and figure out what we want to do. Ideally, and this is perhaps aspirational, I would like to keep inbox zero in Sentry. I know how you feel about that more generally and perhaps even more specifically within the world of errors, but that's my dream. We're going to see how it goes. STEPH: I don't know if people know I am the opposite of inbox zero. This is the life that I'm living. CHRIS: What about with errors, though? What about something like Sentry? STEPH: I want to say that I would be a better human with my email. But I'm going, to be honest [laughs] and say that I would probably have the same approach where I am not an inbox zero person. I've come to terms with it. I used to really strive and think I needed to change. But I have reached a point of comfort with this is who I am. There are many like us, so shout out to all y'all. CHRIS: Oh yeah, by far the more common approach, I think. So specifically with the errors, I struggle a bit with it because what ends up happening is we are implicitly ignoring the errors. And if we're doing that, I would rather just sit around and have a conversation and be like, let's just explicitly ignore them. There's a button in the UI. We can ignore them. If this is not a real error, we can add it to the list of things that we do not report on. We can ignore that error. We can ignore it for a week and add a card to Trello that has a due date that says, "Hey, we got to work on this." But let's take that implicit indifference to that particular error mode of our application and make it explicit. Let's draw that line in the sand such that when I see a new error pop up, I'm like, oh, that seems like something I should do something about. I really want high signal-to-noise when I'm seeing errors coming. And so I'm willing to work for that. But it is a trade-off, and it does take effort. And it's noisy, especially browser extensions, and whatnot, just fighting the page. Facebook showed up one day. I don't know how Facebook got in there. Someone was browsing our website from within Facebook's browser, which I didn't know was a thing, but they had their own thing. And it fires a bunch of events, and Sentry was just like, let me slurp all of those up. Those seem fun. That was noisy. So we had to turn those off, but we explicitly turned them off. STEPH: I do like the approach that you're taking where it's one person, and then it's a rotating shift because I think that makes it more reasonable for someone's who's like, hey, this is going to be noisy for a week. And then you're going to look through these emails and check all these errors, and then either silence them because you don't think that they're interesting or mute them for now. Or if you're going to convert it into a ticket, set a due date, whatever the triage approach is going to be. But that feels more achievable versus inbox zero for life is just exhausting. But I feel like if you're doing it rotating week by week, that seems like a nice approach and also easier to keep it at inbox zero because that way, you are keeping up with all the errors. Because I agree; otherwise, what's the point of tracking all the errors if you're just going to ignore them? CHRIS: Yeah, definitely the rotating, I think, is critical. I think the other thing that's been critical specifically on the error front is we've had now a handful of meetings where we triage the backlog together, the backlog of errors. So like, what all is coming into Sentry? What's going on? And we go through the process of determining is this a real thing? Should we fix this? Should we ignore it? And we do that together so that it becomes not just one person's intuition about whether or not this is important or not or what the source of it might be but a shared intuition such that now any one of us, when it's our week, can ideally represent the team in that way and be like, never mind, never tell us about this again because it's very easy to silence things in Sentry that you would actually like to know about when they become real. But right now, we have this edge case that is an ignorable version. So trying to get there that's been fun. But yeah, once again, Sentry, that's one of the things on this person's list. There are ad hoc support tickets for our operations team. So anything that needs to happen on a user's behalf that currently needs a developer to console, let's funnel all of those to this one individual, respond to any new questions. So this is where that Slack handle will be useful. Check for any stuck jobs in Sidekiq. So is there anything that's been retrying for a while? Because it probably shouldn't. Maybe one or two retries is cool, but past that, something has gone wrong. And we should either get in there and fix it or just kill that job because it's never going to succeed, which is quite often the case but go in there and keep an eye on those and then look for anything. We're starting to use due dates within Trello, which is currently our project management system. We'll see. Someday we're definitely going to grow out of that. But for now, it's good enough and checking for anything that's overdue or coming up in the next week in terms of due dates and just making sure that we're being responsive to that. And so, I really like the idea of having this be a named set of things and a singular focus for one individual. Because again, that idea of like, if it's everybody's job, it's nobody's job. Or if it's nobody's job, then it's my job, and I don't want it to exclusively be my job. [chuckles] So I'm trying to make it not exclusively my job and to share the knowledge about it and make sure that these are skills that we all have and ideas and et cetera. But also, I would be fine to answer fewer questions in Slack each day. STEPH: I have to admit, as soon as you were telling me that you had established this role, I was quietly congratulating you on helping delegate some of these responsibilities to the team. Because like you said, you are then the person that takes on all these tasks. CHRIS: There's a laziness to that. Like, it's easy for me to just answer the questions. It's harder for me to put up a wall and say, "No, no, we have a process for this." And quite possibly, what's going to happen behind the scenes is that questions are going to come in to whoever is this point person. They're not going to know the answer. They're going to reach out to me, and then that conversation is still going to happen. But even by doing that now, now that person will see that answer, will understand the thinking or the background, the context that I have. And so it's that weird thing of like, it would be so much easier for me to just answer one question. But to answer all the questions, well, I can't do that. And so I'm working to try and do more of the delegation to try and hand things off when they're in a known state and to identify this sort of stuff so that the team broadly can be stronger and better able to support everyone else in the organization. So that's the dream. We'll see how it goes. STEPH: Yeah, I love that approach. I'm also thinking how interesting this role is because I'm imagining a mix between someone who is like the front point person at like an ER. So like, things are coming in, and they're in a tragic state and need help and need to be diagnosed. But at the same time, you mentioned they're going around. They're checking Sidekiq. They're looking at some email errors. So they're also that night shift guard that's walking around with a flashlight just poking in each room. So it seems like a very stressful and low-key role all at the same time, all mixed up into one week. That person probably needs a beer at the end of the week. CHRIS: There is a version of the story in my head that is...I wouldn't say this feels like a failure mode, but I would rather this not have to exist at all. I would rather things to be calmly humming along and not require a dedicated person each week to deal with the noise. I don't think that's realistic, certainly not as early on as we are in our organization. But I do wonder, is this a crutch? Is this something that we should be paying more attention to? And I know in teams that you and I have worked with in the past that has been a recognition of like, this is a crutch. But it's a costly crutch. Like, we're taking an entire...in our case, it's not requiring the entirety of a developer's week. They're able to do this pretty easily and then still get a bunch...like, 75% of their time is still feature work. But we're just choking down who's the person that will be responding to questions when they pop up so that fewer individuals are interrupted? But I have seen organizations where this definitely filled an entire week and spilled out more than. And then there was the recognition of that and the addition of another person that comes along and tries to fix stuff along the way as opposed to just responding. And so I want to make sure this isn't a band-aid but is, in fact, a necessary layer that we then try and shore up, you know, we should have fewer errors. That feels true. Okay, cool. Let's fix the bugs in the app. And these ad hoc things that an admin needs to have done can that be a button in the UI? Can they actually self-serve in those cases? And we're slowly moving towards those. Ideally, fewer jobs get stuck in Sidekiq. And so, my hope is that this isn't a job that gets harder and harder over time. It's a job that potentially, if we're being honest, probably stays about this hard. I don't think it's ever going to be just like, nope, nobody needs to do anything. The app just runs, and it's great. And it never has bugs. But that is a question in my mind as I start to embrace this thing of like one person is dedicated for a week to this. And if right now it's only 25% of their time, okay, that's probably fine. But if suddenly it's 50% of their time or 75% or 100% of their time for that whole week, that becomes too high of a bar in my mind. And I want to keep a close eye on it and make sure it's not trending in that direction. And I will be one of the people on the rotation. So I'll get to be in the trenches. STEPH: I appreciate all the thoughtfulness that you're putting into it. And I'm thinking back on a project where we had a similar rotation because we had an issue Slack channel. And so anytime there was an issue, then it would get posted in there. And before, it was going out to everyone, or there was one particular person that was always picking it up and then trying to delegate it to others as they needed to. But then we started a similar rotation. And one of the key benefits that I found from that is it signaled to the team, hey, this person might get pulled away. They can pick another ticket or two, but we need to give them lower priority tickets because there's a chance that they're going to get pulled away to work on something else. And that's okay, and we're going to plan for it. Versus without this role in mind, then you had people all taking on high priority tickets, but then someone had to be the one that's like, well, I'm going to punt on my high priority and feel stressed about the fact that I've got this other thing to deal with. But then, I didn't actually do the work that I planned for. So I feel like you're helping introduce calmness into the week, even if it is a stressful role. But then there's the goal that this becomes less of a stressful role, and if you see it trending in the opposite direction, then that's something to investigate. But I also feel like triage and communication is such an important part of being a developer that it also feels very relevant upskilling for the whole team to go through. So there's also that benefit of where this approach also empowers the rest of the team to also experiences, build empathy, look for additional fixes, and then also build these important skills. Overall, I really applaud your thoughtfulness. And I think it's a really good idea. And it will be interesting to see which direction that this role trends if it gets easier or if it's getting harder over time. CHRIS: Well, thanks. I appreciate that. And I'll certainly report back as we develop this but hopefully, it stays about where it is. That feels right. And I think I'll probably...that's one of those things that I will monitor. And if I feel it moving in the wrong direction, then step in and try and get it back to this space because this feels like a maintainable reasonable amount. And we shouldn't be fixing every bug and adding every button to the UI. That's just actually not how it works, unfortunately, would love to. That's not true. You shouldn't have every button in the UI. That's so many buttons. But broadly, I hope we can maintain roughly this, and I think identifying it and laying it out now I'm feeling good about having that structure. So yeah, we'll see how it goes. Will report back. But again, thank you for the kind words. With that tour of a bunch of different things, should we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes as it really helps other people find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Byeeeeeeeee!!!!! Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

320: Remember The Fun: 2021 Recap

Play Episode Listen Later Dec 21, 2021 34:02

Steph and Chris recap their favorite things of 2019 and 2020 and share their 2021 list. Happy Holidays, y'all! Steph: * Feature flags and calm deploys * Creating observable systems * Debugging * Working in seasons * Don't forget the fun “The longer I'm in the software game, the more I want things to be calm” - Steph Chris: * Pushing logic back to the server * Svelte (https://svelte.dev/) * Remote work (but maybe hybrid!) * Vim * Joining a startup as CTO This episode is brought to you by ScoutAPM (https://scoutapm.com/bikeshed). Give Scout a try for free today and Scout will donate $5 to the open source project of your choice when you deploy. Listen to episodes from 2020 and 2019

44 выпуск 09 сезона. Ruby 3.1.0 Preview 1, no-go fantasy, Hydrogen, Invisible JavaScript Backdoor, React Freeze и прочее

Play Episode Listen Later Nov 14, 2021 52:00

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Ruby 3.1.0 Preview 1 Released What's New in Sidekiq 6.3 A no-go fantasy: writing Go in Ruby with Ruby Next How to store timestamps in Rails Supercharge tests with circle-ci & crystalball [Part 1] Run Your Rails App On Kubernetes: A Step-by-Step Tutorial Web Meet Hydrogen: Shopify's React Framework for Dynamic, Contextual and Personalized E-Commerce The Invisible JavaScript Backdoor Rust Is The Future of JavaScript Infrastructure Bree - the best job scheduler for Node.js and JavaScript Teaful: tiny, easy and powerful React state management Recoil - an experimental set of utilities for state management with React React Freeze - prevent React component subtrees from rendering MiniMasonry.js - minimalist dependancy free Masonry layout librar RWpod Cafe 27 (04.12.2021) Сбор и голосование за темы новостей

fantasy invisible preview dynamic react freeze hydrogen javascript backdoor no go node contextual masonry recoil sidekiq ruby ruby

Sidekiq Race Conditions [Chris Toomey]

The Swyx Mixtape

Play Episode Listen Later Nov 11, 2021 11:19

Listen to The Bikeshed (24mins in) https://www.bikeshed.fm/313TranscriptSo we had a bug that occurred in the application where something was supposed to have happened. And then there was an email that needed to go out to tell the user that this thing had happened. And the bug popped up within AppSignal and said something was nil that shouldn't have been nil.Particularly, we're using a gem called Time For a Boolean, which is by Caleb Hearth. And he's a former thoughtboter and maintains this wonderful gem that instead of having a Boolean for like, is this thing approved, or is it paid? Or is it processed? You use a timestamp. And then this gem gives you nice Boolean-like methods on top of that timestamp. Because it turns out, very often just having the Boolean of like, this was paid, it turns out you really want to know when it was paid. That would be a really useful piece of information. And so, while you're still in Postgres land, it's nice to be able to reach for this and have the affordances of the Boolean-like interface but also have the timestamp where available.So anyway, the email was trying to process but that timestamp...let's pretend that it was paid as the one that matters here so paid at was nil, which was very concerning. Because this was the email that's like, hey, that thing was processed. Or let's say it was processed, actually, because that's closer to what it was. Hey, this thing was processed, and here's an email notification to tell you that. But the process timestamp was nil. I was like, oh no. Oh no. And so when I saw this pop up, I was like, this is very bad. Everything is very bad. Oh goodness.Turns out what had happened was...because I very quickly chased after this, looked in the background job queue, looked in Sidekiq's UI, and the job was gone. So it had been processed. I was like, wait a minute, how? How did this fix itself? Like, that's not the kind of bug that resolves itself, except, in this case, it was. This was an interaction that I'd run into many times before. Sidekiq was immediately processing the job. But the job was being enqueued from within the context of a database transaction. And the database transaction had not been committed yet. But Sidekiq was already off to the races trying to process.So the record that was being worked on, the database record, had local changes within the context of that transaction, but that hadn't been committed. Sidekiq then reads that record from the database, but it's now out of sync because that tiny bit of Sidekiq is apparently very fast off to the races immediately. And so there's just this tiny little bit of time that can occur. And this is also a fun one where this isn't going to happen every time. It's only going to happen sometimes. Like, if the queue had a couple of other things in it, Sidekiq probably would have not gotten to this until the database transaction had fully closed.So the failure mode here is super annoying. But the solution is pretty easy. You just have to make sure that you enqueue outside of the database transaction. But I'm going to be honest, that's difficult to always do right.STEPH: That's a gnarly bug or something to investigate that I don't think I have run into before. Could you talk a little bit more about enqueueing the job outside the database transaction?CHRIS: Sure. And I think I've talked about this on a previous episode a while back because I have run into this one a few times. But I think it is sufficiently rare; like, you need almost a perfect storm because the database transaction is going to close very quickly. Sidekiq needs to be all that much more speedy in picking up the job in order for this to happen.But basically, the idea is within some processing logic that we have in our system; we find a record, we do some work. And then we need to update that record to assign this timestamp or whatever it is. And then we also want to inform the user, so we're going to enqueue a job to send the email notification. But for all of the database work, we are wrapping it in a transaction because we want it to either succeed or fail atomically. So there are three different records that we need to update. We want all of them to be updated or none of them to be updated. So, therefore, we wrap it in a transaction.And the way we had written, this was to also enqueue the job from within the transaction. That wasn't something we were actively intentionally doing because those are different systems. It doesn't really mean anything. But we were still within the block of ApplicationRecord.transaction do. We're now inside of that block. We're doing all of the record updates. And then the last piece of work that we want to think about is enqueueing the job to send the email.The problem is if we're still within that database transaction if it's yet to be committed, then when Sidekiq picks up that job to run it, it will see the prior state of the world. And it's only if the Sidekiq job waits a little bit that then the database transaction will have been committed. The record is now updated and available to be read by Sidekiq in the correct updated state.And so there's this tiny little bit of inconsistency that can happen. It's basically because Sidekiq is going out to Redis, which is a distinct system. It doesn't have any knowledge of the database transaction at play. That's why I sometimes consider using a Postgres-backed background job system because then actually the job can be as part of the database transaction.STEPH: Cool. That's helpful. That makes a lot of sense the way you explained the whole you're actually enqueueing the job from inside that transaction. I'm curious, that prompts another question. In the case where you mentioned you're using a transaction because you want to make sure that if something fails to update so, everything gets updated together, in the event that something does fail to update because you were previously enqueueing that job from the transaction, does that mean that the update could have failed but that email would still have gone out?CHRIS: That does not. And the reason for that is because we're within dry-monad world. And so dry-monad will implicitly capture the ActiveRecord rollback, which I think is an exception that gets raised or somehow...But basically, if that database transaction fails for any reason and ends up getting rolled back, then dry-monads will not continue processing through the rest of the sequential operation. And so, therefore, even if we move the enqueuing of the email outside of the database transaction, the sequential nature of that processing and the dry-monad stuff that we have in play will handle that. And I think that would more generally be true because I think Rails raises an exception on rollback. Not certain there. But I know in our case, we're fine on that. And we have actually explicitly checked7 for that sort of thing.STEPH: So I meant a slightly different question because that makes sense to me everything that you just said where if it's outside of the transaction, then that sequential order won't fire because of that ActiveRecord migration error. But when you have the enqueuing inside of the transaction because then that's going to be inside of the sequential order, maybe before the rollback error gets raised. Does that make sense?CHRIS: Yes. I think what you're asking is basically like, do we make sure to not send the job if the rest of the stuff didn't succeed?STEPH: I'm just wondering from a transaction perspective, actually. If you have a transaction wrapped block and then you have in there, like, update this record, send email, end block, let's say update...well, I guess it's going raise because you've got probably like an update bank. Okay, so then yeah, you won't get to the next line. Got it. Got it. Got it. I just had to walk myself through that because I forgot that you probably...I have to visualize [laughs] as to what that code probably looks like. All right, that answered my question.CHRIS: Okay. So back up to the top level then, this is the problem that we have. And looking through the codebase, we actually have it in a bunch of different places. So the solution in any one of those cases is to just take the line of code where we're saying enqueue UserMailer.deliver_later take that line of code, move it outside of the database transaction, and make sure it only happens if the database transaction succeeds. That's very easy to do in one case.But my concern was this is a very easy failure mode to end up in. And this is a very easy incorrect version of the code to write. As far as I can tell, we never want to write the code where this is happening inside of the transaction because it has this failure mode. But how do we enforce that? That was the thing that came to mind. So I immediately did a quick look of like, is there a RuboCop thing I can do here or something?And I actually found something even more specific, which was so exciting to find. It's a gem called Isolator. And its job is to detect non-atomic interactions within database transactions. And so it's fantastic. I was like, wait, really? Is this going to do the thing? And so I just installed the gem, configured it where I wanted, and then ran the test suite. And it showed me every place throughout the app right now where we were doing this pattern of behavior like enqueueing work from within a database transaction, which was great.STEPH: Ooh, that's really nifty. I kind of want to install that and just run it on my current client's codebase and see what I find.CHRIS: This feels like something like strong migrations where it's like, yeah, this is great. I kind of want to have this as part of my core toolset now. This one feels even perhaps slightly more so because sometimes I look at strong migrations, and I'm like, no, no, no, strong migrations, I get why you would say that, but for reasons, this is actually fine. And they have configurations within it to say, like, no, this is okay. Isolator feels like it's always telling me something I want to know. So this, very quickly, I'm like, I think this might be part of my toolset moving forward on every single app forever.

learning business technology race conditions ui rails redis postgres boolean activerecord isolator sidekiq chris yes chris okay chris toomey

313: Forty-Seven Percent

Play Episode Listen Later Oct 19, 2021 42:05

Steph talks about binging a few Things Worth Learning podcast episodes and particularly enjoyed an episode that featured one of thoughtbot's design directors, Sameera Kapila. Sam shared her expertise about management and inclusion, and Steph shares her favorite parts. Chris shares the story of a surprising error and the resulting journey through database transactions and Sidekiq that eventually resolved the issue. He also shares some follow up on the broken build and the merging process changes they introduced (spoiler, the process changes have been rolled back). Leading Inclusively, with Sameera Kapila - Things Worth Learning Podcast (https://www.youtube.com/watch?v=eiV6_3pZFc0) How to Skim a Pull Request (https://thoughtbot.com/blog/a-smelly-list) Isolator (https://github.com/palkan/isolator) aftercommiteverywhere (https://github.com/Envek/after_commit_everywhere) timefora_boolean (https://github.com/calebhearth/time_for_a_boolean) Transcript: STEPH: Oh man, I'm about to stop eating my pop-tart. I'll put it away. It's within distance. I'm going to eat it. CHRIS: Your high-fat content unfrosted pop-tart. STEPH: You know, surprise Sunday twist: it has icing on it. CHRIS: Steph, who even are you? STEPH: [laughs] CHRIS: There are a few canonical anchor facts that one knows about other people, and when one of those... STEPH: I like to keep everyone, including myself, on their toes. CHRIS: Or you've just secretly accepted that the icing adds another textural flavor adventure component. It's just better with icing. STEPH: All right, all right, all right. There's a complicated answer to this. And the complicated [chuckles] answer to this is that the more organic ingredients that I recognize when reading about pop-tarts are by a particular company, and they all have frosting on them. And the more generic pop-tarts that don't have frosting on them, I don't know how to pronounce a lot of those ingredients. So I'm like, no, but okay, I still eat them. But I prefer the ingredients I can pronounce. So I either go with the ingredients I can't pronounce or have a little bit of frosting on my pop-tart. And I'm going with the non-cancer route for today. CHRIS: For today, in this moment, and accepting the frosting. Okay, all right. Well, that is complicated. [laughs] It's tricky out there. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together we're here to share a bit of what we've learned along the way. So, Steph, what's new in your world? STEPH: Hey, Chris. So the weather, I'm going to talk about the weather for a little bit. [chuckles] It's been almost non-stop rain for the past several days, which is fine. I'm sure it's great for plant life. But it's really hard on my dog Utah because then we can't go outside for our normal walks and playtime. Although he is my four-legged water baby because he absolutely loves water, and puddles, and playing in the rain. So he's very fine with going outside and playing for a long time. But then I have to essentially give him a full-on bath before I want to bring him back in. So not wanting to have to give him a bath each time, in the spirit of improvising, we started finding more indoor games to play. And I've started teaching him to play hide and seek. And he's not great at it mainly because he will only stay until I'm out of eyesight, and then he will come and find me. And so I have to be really, really fast at finding a hiding spot to like dash around a corner or hide behind the door. But I think he enjoys it because he will find me and then he seems very excited. And we go back, and we play again. And so I just have to work on teaching him to wait a bit longer so I can find better hiding spots. CHRIS: When you said that, at first, I was like, how did you teach him to hide? But I realize he's only playing the seek part of the game, and you're only playing the hide part of the game. STEPH: [laughs] CHRIS: I'm just so used to you exchange roles back and forth. First, you hide, then you seek, and then you switch it up. That would be a lot to get your dog to be like, now I'm going to secretly hide. STEPH: [laughs] I'd be very impressed. Yes, we have very distinct roles in this game. I am the one that always counts and hides. But he's a very good seeker. So that's been fun. We just got to work on getting a little better at it. But on a more tech-related note, one of the design directors at thoughtbot, Sameera Kapila, who also goes by Sam, was a guest on the podcast Things Worth Learning, which is hosted by Matt Stauffer. And Matt is also the host of The Five-Minute Geek Show and The Laravel Podcast. And in the show Things Worth Learning, Matt meets with individuals that are excited to share something that they're deeply passionate about; maybe it's tech, maybe it's not. And I've binged a couple of those episodes. And I really like how you can choose between the podcast format or the YouTube format. So then you can really watch the conversation unfold, which I know you and I a couple of times have thought it would be fun if people could see us because there are so many facial emotions and gestures that go along with conversations. So it was really delightful. And speaking of delightful, Sam shared her expertise about management and inclusion. And I definitely recommend listening to the episode because I can't share everything that Sam shared. But a couple of the topics that Sam mentioned that I really enjoyed and would love to chat about, so the first one is about helping someone, in this case, someone that you manage that comes to you with a concern. So there's often a presumption that just because someone comes to you with a concern or an issue that they've experienced at work, that they're the ones that will also want to work to address that concern, and that's often not true. It can be true; maybe that person wants to be involved. But they're often coming to you in the leadership or management role to say, "Hey, I've had this issue," and they really want help with that instead of walking away with homework for it. Because then that trains people to essentially be in this mindset of well, if I bring up this concern, then I'm going to be the one that has to address it, even if I'm the one that's most negatively impacted by this. And addressing this concern could be actively harmful to me. And she shared a really great real-world example from her own experience where her and another co-worker had noticed a concern about the hiring process. And her and that co-worker got together, and they talked about the concerns. They even rehearsed for the meeting because they were trained by the tech industry to say, "Hey, if you bring up a concern, you're going to be responsible for addressing and then resolving that concern." And so they had that meeting with the person in leadership. And they were pretty nervous about how it was going to go. And that person in leadership said to them, "Thank you both so much for sharing that. That must have been such a burden. And this is my responsibility to fix. And here are what my next steps are." And that was amazing because it allowed Sam and the other person to go back to client work. And they also received follow-up conversations about how that issue was being addressed. So there was even that feedback loop as to how things were going to change. And I have a personal example that...I really resonated with the example that Sam provided because I remember there are different teams that I've been a part of, where often I was one of the few women engineers on the team. And so we often have conversations about how do we get more women engineers into the company? And they're wonderful conversations. But there's a part of me that always felt resentful about, like, why am I here? Why am I the one fixing this? I understand I have some more insight and expertise, and experience in this area. But I was also frustrated by the fact that I was the one that was in that meeting often with other women, and it felt like our responsibility to fix this. And I used to feel bad about feeling resentful towards that. Because I was like, shouldn't I want to help other people? And I do. But Sam's example really helped remind me and clarify that yes, just because there's a concern doesn't necessarily mean you should be the one to address it. And it really takes everybody involved, or it takes leadership to step up and address that concern. CHRIS: Oh, that's really interesting the way Sam is framing that and describing the situation of not having any problem that you bring in be now your work to solve. Like, oh, I found the issue, and now we've got to go do this. But the idea that you can bring something to light and then be able to walk away from it. And the particular thing that you were saying that if your interaction is always that when you reference something when you bring in a concern that then your manager works with you to figure out how you can solve it, then you get this mental block of like, well, do I even want to say anything? Because I don't want to try and deal with big, amorphous unclear issues. So maybe I just won't even say anything. And so this as a way to make sure that there's room for all of the conversation is a really interesting framing that I hadn't really thought about, frankly, but it's very interesting. I haven't seen this interview either. So I'm definitely excited to give this a look because Sam is wonderful. And the topic that you're describing here sounds fantastic as well. STEPH: Yeah. There was an important moment for me where...one of my managers is Matt Sumner, who's been on the show. And when Matt was my manager, at one point, we were having a one on one, and we would often go for walks for our one on one. And I mentioned something about "I have this concern, or I have this problem, but I don't really know how to fix it. So I'm not sure I'm ready to talk about it." And Matt, in his delightful way, was like, "We can still talk about it. You don't have to have an answer or a solution." I'm like, "Yeah, but I feel like I should be able to fix it. Like, if you have a concern, or if you have something that you want to gripe about, then you should come to the table with solutions for it." And Matt was like, "No, you don't need to do that at all. We can totally gripe about stuff or talk about concerns and then either figure out the solutions together or go to other people for ideas." And that was really important to me because, like you'd mentioned, otherwise, it felt like this mental block where then it feels like you can't air out some of the things that you're worried about or have concerns about because then you think you're the only one responsible. And you may not be able to come up with the best solution. You may need other people to then help you strategize and come up with ideas. And I just love, love, love that part of Sam's discussion. And oh, there was one other part about the conversation. Well, there are lots of parts that were amazing. But another one in particular that blew my mind is about Comic Sans, the font, the font that everyone loves to hate. [chuckles] And I learned that it's one of the most legible fonts for kids. And it's one of the more accessible fonts for people with dyslexia. And it's actually recommended...I think there are still more academic studies that need to be done to really classify fonts that are best for people that have dyslexia. But Comic Sans is recommended by The British Dyslexia Association and the Dyslexia Association of Ireland. And there are some other really great posts that talk about the benefits of using a font like Comic Sans because the typeface has long ascenders and descenders and generous letter spacing and asymmetrical lowercase b and d to then help distinguish those letters. And I just thought that was so cool. This font that everybody wants to rip apart because it seems whimsical, unprofessional gets overused. There are lots of reasons, I suppose. [laughs] But there's a really big benefit to it, and it can help others. And I just found that very whimsical in itself. CHRIS: I love the idea that there are multiple levels of knowing about Comic Sans. First, you're just like, I don't even know the name, but it's that comic book-looking font. And then obviously, the next step is to be like Comic Sans? How could you ever use that? It's an atrocity. And then it's like, but actually, Comic Sans has some things going for it. And it is a really interesting consideration and something that you wouldn't necessarily think of. But then once you learn it, you're like, okay. Man, I wonder how many other things in the world have this interesting shape to them? Hmm. STEPH: Do you know the history behind Comic Sans? CHRIS: I do not. STEPH: I read about it fairly recently, but I'm probably going to botch some of the details. But I believe it was designed or created by Vincent Connare. And it was created for Microsoft. And Vincent was working on a project where I think there was a dog that was essentially going to have these bubbles that would then show you different parts of the application and walk you through the different features. And the dog had a very comic book feel to the character. And so then Vincent designed a font to go along with that comic book character, this dog and came up with Comic Sans. I don't think the dog actually launched with that particular font. But since the font was still developed, it was released as part of the available fonts. And there we go, there is the birth of Comic Sans. And then it just received so much love and ire all throughout history. [chuckles] CHRIS: There's something that you said there that I want to loop back on when you were talking about chatting with Matt Sumner and saying, "Here's this thing, but I don't know how to solve it. So I don't even want to bring it up." I really liked the framing that you gave and the fact that Matt was like, "No, no, we can still talk about it. We can at least explore this thing, have a conversation." I think that's really wonderful. There's a very similar thing that I experience a lot when doing code review, particularly when I'm in more of a leadership role within a team, which is I often want to highlight something that feels a little bit off to me in the code, but I may not have a specific solution. Like, I may see a variable name, or I may see a controller action that feels like it's the wrong shape or something. And I'll often name it but explicitly say, "I actually don't have a better idea here. So feel free to continue on with this, but I want to name it. So in case that sparks something in you, if you were also feeling some incongruousness, maybe it's worth you spending another minute to think about it, but I want to make sure my comment isn't blocking or otherwise making you feel uncomfortable." If I just come to you and I'm like, "This feels wrong," and that's all I say, that to me is unacceptable code review. Because now I want all of my code review feedback to be very actionable, it's either here's the thing that I feel strongly I think we should definitely change this. If you disagree, let's have a conversation. But yeah, this one definitely needs to change. Here's the thing that, like, I don't know, maybe we could break this into two lines and split it up. But if you don't like that, that's fine. Do whatever. And so then it's I've given the person my thoughts but given them clarity and a free rein to do whatever they want with that information. And then there are ones where I'm like, I don't even know what I think we should do here, but I think something. But if you don't have any ideas...like, I don't have any ideas specifically. If you don't have any ideas, it's fine. We'll continue on with this and maybe revisit it down the road. But I want to make sure each of those different tiers is actionable for the other person, and I'm not just giving them homework or something to be sad about because that would be bad code review. STEPH: I'm just imagining a PR comment that says, "I don't know what we should do here. But I don't think this is it," [laughs] and that just creating sadness. That's so interesting to me because I have flip-flopped with that opinion in regards to there are times that I very much resonate and do what you just said where I will point out to someone where I'm like, "I'm not sure why, but I just have concerns about this. And I don't know if you also ran into anything that was weird about this and would like to talk about it. I don't have any really great ideas, so I think this is good for now. And we should keep moving forward, so we're not blocked on it," but just wanted to, as you mentioned, highlight it in case it sparks something for the other person or for someone else that's reviewing the code. And then there are other times where I'll look at something, and I'm like, "Yeah, it's not great. There's something that feels brittle or potentially maybe hard to maintain or things like that. But I don't have a better idea." And I don't comment on it because I'm like, I don't want to distract that person or block them. And I do think it's good enough, and I don't have anything to add to the conversation, so I just leave it out. So it's interesting to me where is that line of when I feel like it's important enough to comment to then potentially spark some conversation versus just letting it go so then I don't add any distraction to their work? CHRIS: I think it's when the spidey-sense gets past 47%. It's a very specific number. I do the same thing where there's something, and I'm like, you know what? I can't even clearly express what about this makes me feel something off, and so I won't even comment on it, and I agree. And then there are things that trip past some magical line in the sand. And I'm like, you know what? I think I'm going to say something here, but I don't even have a recommendation. And then there's a whole spectrum of the nature of code review and, again, 47% being the specific number. STEPH: There's actually a thoughtbot blog post that correlates nicely to that concept of spidey sense. It's written by Mike Burns, and it's titled How to Skim a Pull Request. But essentially, grabbing from one of the lines here is where Mike presents an unexplained, incomplete, and arbitrarily grouped list of keywords that will cause us thoughtboters to read your code with more care and suspicion. [laughs] That feels perfectly aligned with that idea of spidey sense, spidey-sense 101. I'll be sure to include a link in the show notes. Or, you know, 40%. CHRIS: I think it was 47%. It's a very precise number. [chuckles] STEPH: Very precise nonsensical number. Got it. [laughs] CHRIS: If I'm making up fake statistics, I'm not going to have them round to an even 10. [laughter] STEPH: Makes it seem more legit somehow. CHRIS: Exactly. STEPH: But that's really the novelties that I wanted to chat about. Mid-roll Ad And now a quick break to hear from today's sponsor, Scout APM. Scout APM is leading-edge application performance monitoring that's designed to help Rails developers quickly find and fix performance issues without having to deal with the headache or overhead of enterprise platform feature bloat. With a developer-centric UI and tracing logic that ties bottlenecks to source code, you can quickly pinpoint and resolve those performance abnormalities like N+1 queries, slow database queries, memory bloat, and much more. Scout's real-time alerting and weekly digest emails let you rest easy knowing Scout's on watch and resolving performance issues before your customers ever see them. Scout has also launched its new error monitoring feature add-on for Python applications. Now you can connect your error reporting and application monitoring data on one platform. See for yourself why developers call Scout their best friend and try our error monitoring and APM free for 14 days; no credit card needed. And as an added-on bonus for Bike Shed listeners, Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at scoutapm.com/bikeshed. That's scoutapm.com/bikeshed. STEPH: What's new in your world? CHRIS: I have some follow up on a recent topic that we talked about. So we had a kerfuffle which I described where we had a branch that got merged and the rebase some stuff got out of hand. And so we introduced some process, the protected branch configuration within GitHub that required the branches to be up-to-date before they can be merged and CI to be passing. And everybody was happy. It was like, this is great. Turns out it was never turned on. That's actually the day I was like, man; this is really straightforward. There's been no annoyance here. And then I got to the point where it was like; this seems weird because we just merged a lot of things in rapid succession. I went and checked, and it turns out what I thought was the name of the branch protection rule in GitHub's UI is, in fact, a regular expression pattern. It might not be a full regular expression but like a wildcard pattern for the branch name to match to, and so it's specific. I created this rule, and in small, gray text underneath, it said, "This applies to zero branches." I missed that the first time but then the second time going back, I was like, oh, I actually wanted it to apply to more than zero branches. So I went back in and changed that. It's a great example of very subtle UI that just slipped past me. STEPH: I was going to say in your defense, the very subtle gray font to say, "This applies to zero," feels tricky. CHRIS: That...also, going through the work of creating this thing and if that results in zero branches that would match, maybe that's the thing to emphasize on creation. I would love that. Because in my case, I was trying very specifically to target an existing branch. There is the ability to say, "Oh, any bugfix-* named branch," if you're using branch naming strategies like that, you can use this for that sort of thing. So it may be that currently, there are no branches with that name. But in my case, I was just like, please, main, anytime anything is happening on main, that is what we want to do. I just needed to put the word main there. But anyway, once I actually turned it on, insufferable, absolutely not, cannot survive in this world. We have a relatively small team. There are three of us, and not everyone is even full-time, and my time is pulled in a lot of different directions. So I'm actually not pushing as much code as I might otherwise. Even with that, nope, absolutely not. Our CI is like; I don't know, five-ish minutes per run. Turns out, especially Monday mornings, we have a volley of things that will have been reviewed and trickled in through Friday afternoon. And then there's a bunch of work we want to land Monday morning. And then, just at any point, it turns out, yes, this was untenable. So we have turned it off. I would like to revisit this down the road and introduce the MergeQueue functionality, so the idea of being able to say, "Yeah, you just name when you want something to go in, and then the system will manage the annoying finicky work there." But for now, I had to give up on my dream of everything running on CI, on a feature branch, before it gets merged. STEPH: Ooph, that phrase, "I had to give up on my dream," that breaks my heart for you. [laughs] CHRIS: I may be going a little bit fanciful with my language but, like, a little. STEPH: [laughs] CHRIS: I liked this thing. I want to exist in that world. But it is not feasible given the current state of the world. And that will only get worse over time, is my expectation. So I get to revisit this when I have the time to more thoroughly figure a thing out. But for now, I don't know, merge whatever; it will be fun. STEPH: There's a small part of me that feels a little reassured that it was a terrible time, although I hate that it was a terrible time. But I have felt that pain on so many other projects where I am constantly waiting, and I'm constantly checking to be like, can I merge? Can I merge? Can I merge? And then I can merge, but then someone beats me to it. And I'm like, oh, then I got to restart. And I got to wait, and I'm constantly checking. So that feels like it helps validate my experience. [chuckles] I am excited for that MergeQueue. I would be super excited to try that out and hear about how it goes just because that seems more like the dream where you can just say, hey, I want this PR to go whenever it can go. Just take care of it. I want it to be rebased, whatever the flow is, and have it be merged, so I don't ever have to check on it again. CHRIS: But once we configured this, there was a new thing that appeared in the GitHub UI, which was auto-merge. And so that was a button where I could say like, "Hey, merge this whenever CI passes," which was a nice upgrade, but it didn't have the additional logic of and rebase as necessary. Or the more subtle logic of like, you don't actually want to rebase where you have five different branches that are all trying to merge, and they keep rebasing. You want to have the idea of a queue, and so you get in line. And you rebase when it's your turn, and then you run the CI. And you try and be as smart as possible about that. If anyone at GitHub is listening, I would love if you all threw this into your platform, and then you could ping Slack if anything went wrong. But otherwise, there are, like I said, existing tools. At some point, I will probably, I don't know, over a long weekend or something like that, sit down with a large cup of coffee and explore these. But today is not that day. STEPH: I'm excited to hear about that day. CHRIS: So that is a tale of woe and sadness. But luckily, I get to balance it out with a tale of happiness and good outcomes. So that's good. The happiness and good outcome story does start with trouble, as they always do. So we had a bug that occurred in the application where something was supposed to have happened. And then there was an email that needed to go out to tell the user that this thing had happened. And the bug popped up within AppSignal and said something was nil that shouldn't have been nil. Particularly, we're using a gem called Time For a Boolean, which is by Caleb Hearth. And he's a former thoughtboter and maintains this wonderful gem that instead of having a Boolean for like, is this thing approved, or is it paid? Or is it processed? You use a timestamp. And then this gem gives you nice Boolean-like methods on top of that timestamp. Because it turns out, very often just having the Boolean of like, this was paid, it turns out you really want to know when it was paid. That would be a really useful piece of information. And so, while you're still in Postgres land, it's nice to be able to reach for this and have the affordances of the Boolean-like interface but also have the timestamp where available. So anyway, the email was trying to process but that timestamp...let's pretend that it was paid as the one that matters here so paid at was nil, which was very concerning. Because this was the email that's like, hey, that thing was processed. Or let's say it was processed, actually, because that's closer to what it was. Hey, this thing was processed, and here's an email notification to tell you that. But the process timestamp was nil. I was like, oh no. Oh no. And so when I saw this pop up, I was like, this is very bad. Everything is very bad. Oh goodness. Turns out what had happened was...because I very quickly chased after this, looked in the background job queue, looked in Sidekiq's UI, and the job was gone. So it had been processed. I was like, wait a minute, how? How did this fix itself? Like, that's not the kind of bug that resolves itself, except, in this case, it was. This was an interaction that I'd run into many times before. Sidekiq was immediately processing the job. But the job was being enqueued from within the context of a database transaction. And the database transaction had not been committed yet. But Sidekiq was already off to the races trying to process. So the record that was being worked on, the database record, had local changes within the context of that transaction, but that hadn't been committed. Sidekiq then reads that record from the database, but it's now out of sync because that tiny bit of Sidekiq is apparently very fast off to the races immediately. And so there's just this tiny little bit of time that can occur. And this is also a fun one where this isn't going to happen every time. It's only going to happen sometimes. Like, if the queue had a couple of other things in it, Sidekiq probably would have not gotten to this until the database transaction had fully closed. So the failure mode here is super annoying. But the solution is pretty easy. You just have to make sure that you enqueue outside of the database transaction. But I'm going to be honest, that's difficult to always do right. STEPH: That's a gnarly bug or something to investigate that I don't think I have run into before. Could you talk a little bit more about enqueueing the job outside the database transaction? CHRIS: Sure. And I think I've talked about this on a previous episode a while back because I have run into this one a few times. But I think it is sufficiently rare; like, you need almost a perfect storm because the database transaction is going to close very quickly. Sidekiq needs to be all that much more speedy in picking up the job in order for this to happen. But basically, the idea is within some processing logic that we have in our system; we find a record, we do some work. And then we need to update that record to assign this timestamp or whatever it is. And then we also want to inform the user, so we're going to enqueue a job to send the email notification. But for all of the database work, we are wrapping it in a transaction because we want it to either succeed or fail atomically. So there are three different records that we need to update. We want all of them to be updated or none of them to be updated. So, therefore, we wrap it in a transaction. And the way we had written, this was to also enqueue the job from within the transaction. That wasn't something we were actively intentionally doing because those are different systems. It doesn't really mean anything. But we were still within the block of ApplicationRecord.transaction do. We're now inside of that block. We're doing all of the record updates. And then the last piece of work that we want to think about is enqueueing the job to send the email. The problem is if we're still within that database transaction if it's yet to be committed, then when Sidekiq picks up that job to run it, it will see the prior state of the world. And it's only if the Sidekiq job waits a little bit that then the database transaction will have been committed. The record is now updated and available to be read by Sidekiq in the correct updated state. And so there's this tiny little bit of inconsistency that can happen. It's basically because Sidekiq is going out to Redis, which is a distinct system. It doesn't have any knowledge of the database transaction at play. That's why I sometimes consider using a Postgres-backed background job system because then actually the job can be as part of the database transaction. STEPH: Cool. That's helpful. That makes a lot of sense the way you explained the whole you're actually enqueueing the job from inside that transaction. I'm curious, that prompts another question. In the case where you mentioned you're using a transaction because you want to make sure that if something fails to update so, everything gets updated together, in the event that something does fail to update because you were previously enqueueing that job from the transaction, does that mean that the update could have failed but that email would still have gone out? CHRIS: That does not. And the reason for that is because we're within dry-monad world. And so dry-monad will implicitly capture the ActiveRecord rollback, which I think is an exception that gets raised or somehow...But basically, if that database transaction fails for any reason and ends up getting rolled back, then dry-monads will not continue processing through the rest of the sequential operation. And so, therefore, even if we move the enqueuing of the email outside of the database transaction, the sequential nature of that processing and the dry-monad stuff that we have in play will handle that. And I think that would more generally be true because I think Rails raises an exception on rollback. Not certain there. But I know in our case, we're fine on that. And we have actually explicitly checked7 for that sort of thing. STEPH: So I meant a slightly different question because that makes sense to me everything that you just said where if it's outside of the transaction, then that sequential order won't fire because of that ActiveRecord migration error. But when you have the enqueuing inside of the transaction because then that's going to be inside of the sequential order, maybe before the rollback error gets raised. Does that make sense? CHRIS: Yes. I think what you're asking is basically like, do we make sure to not send the job if the rest of the stuff didn't succeed? STEPH: I'm just wondering from a transaction perspective, actually. If you have a transaction wrapped block and then you have in there, like, update this record, send email, end block, let's say update...well, I guess it's going raise because you've got probably like an update bank. Okay, so then yeah, you won't get to the next line. Got it. Got it. Got it. I just had to walk myself through that because I forgot that you probably...I have to visualize [laughs] as to what that code probably looks like. All right, that answered my question. CHRIS: Okay. So back up to the top level then, this is the problem that we have. And looking through the codebase, we actually have it in a bunch of different places. So the solution in any one of those cases is to just take the line of code where we're saying enqueue UserMailer.deliver_later take that line of code, move it outside of the database transaction, and make sure it only happens if the database transaction succeeds. That's very easy to do in one case. But my concern was this is a very easy failure mode to end up in. And this is a very easy incorrect version of the code to write. As far as I can tell, we never want to write the code where this is happening inside of the transaction because it has this failure mode. But how do we enforce that? That was the thing that came to mind. So I immediately did a quick look of like, is there a RuboCop thing I can do here or something? And I actually found something even more specific, which was so exciting to find. It's a gem called Isolator. And its job is to detect non-atomic interactions within database transactions. And so it's fantastic. I was like, wait, really? Is this going to do the thing? And so I just installed the gem, configured it where I wanted, and then ran the test suite. And it showed me every place throughout the app right now where we were doing this pattern of behavior like enqueueing work from within a database transaction, which was great. STEPH: Ooh, that's really nifty. I kind of want to install that and just run it on my current client's codebase and see what I find. CHRIS: This feels like something like strong migrations where it's like, yeah, this is great. I kind of want to have this as part of my core toolset now. This one feels even perhaps slightly more so because sometimes I look at strong migrations, and I'm like, no, no, no, strong migrations, I get why you would say that, but for reasons, this is actually fine. And they have configurations within it to say, like, no, this is okay. Isolator feels like it's always telling me something I want to know. So this, very quickly, I'm like, I think this might be part of my toolset moving forward on every single app forever. And actually, there's another gem that I used. It's made by the same team. So this is from the folks over at Evil Martians, which is another Rails consultancy out there in the world. And the Isolator gem is one thing that they've produced. And then I think the same author of it who is an Evil Martian's employee created the aftercommiteverywhere gem. So aftercommit is one of Rails' ActiveRecord callbacks. But in this case, it allows you to use it everywhere, as the name implies. And so rather than actually having to take that line of code out of the database transaction block, which is naturally where we would write it because that's how we think about the code and how we want to express it, you can just use this aftercommit method, wrap the call in that, so it's after_commit, and then a block. So either braces or do..end. That enqueueing of the email now just gets wrapped in that. And so what that does is it says, "Defer this until after the transaction commits. If the transaction does not commit, if we roll it back, then don't run it." And what was nice is the actual code change when I finally submitted all of this was add the gem to the gem file. And then everywhere that we're doing the wrong thing, which running the test suite told me, I just went in, and I wrapped that line in after_commit and a block. And it was such a nice, clean...like, I didn't have to move the code around or actually shift the lines, which was my first attempt at this. I was able to just annotate each of those lines and say, "You're special, you're special, you're special," And then I'm done. And again, the first gem told me every case where I needed to do that. It's like, well, this is a wonderful little outcome here. STEPH: That's really nice, yeah, how you can make the changes and then, like you said, re-run the test or re-run that gem, and it lets you know what else still needs to be updated. I'm intrigued where you mentioned you didn't have to move any lines, though. Maybe I just need to look at the gem and see it, but I'm still envisioning that you have your transaction do block. And then you're doing some things; you're updating records, and then you have your end. And then after that, it's when you want to enqueue the email. And with this after_commit, you actually added that method call inside of the transaction but then wrapped the call to Sidekiq to send the email inside of that block. CHRIS: Correct. Yeah. So it's basically like saying, "Here's almost an anonymous function." If you think about a Ruby block in that nomenclature, you're saying, like, here's some work to do when and if the transaction succeeds. And so it meant that I was able to keep the code in the way that we as humans would talk about it but deal with the murky details, and edge cases of database transactions, and Sidekiq, and whatnot. Sort of just handle it by saying like...it almost feels like an annotation or a decoration or something like that. But it was this, in my mind, almost like a perfect melding of I don't want to think about this. Oh, cool. Okay, here's a quick, easy way to deal with it but to not have to fundamentally change how I write the code. STEPH: Interesting. So I like all the things you're saying. I'll be honest, I'm not totally sold, and I'm trying to think of why. I think the benefits...one, as you mentioned, it's something you don't have to think about or at least signals to others that hey, maybe you should think about this to the extent that you use after_commit. And so that way, you don't have these asynchronous events taking place inside the transaction. So I like that visibility and communication to the rest of the team. Putting it inside of the transaction feels interesting. I don't know why; I feel a little weird about this. [laughs] I'm bringing my true self. CHRIS: That's fair. So if we're being honest, I solved this first by finding the Isolator gem. Well, I solved it first by just doing it manually. I went through the app, and I found all the places. And I was like, you know what? I'm worried that the next person authoring code like this, it's so easy to fall into this trap. Like, this is such a subtle little thing that our brains are not thinking about. And so I had first fixed it, and so I had a diff that involved moving lots of lines of code, every instance of this moved from being in the database transaction out of it. And that was fine. I was fine with that as a solution. But it was a little bit noisy because I was moving a bunch of lines. So then I brought in the Isolator gem. I actually reset that, and I went back to before I had made the fix, ran the test just to make sure Isolator was actually finding every instance. They did; that was great. So I was like, all right, cool. This is better because now I have this thing that will tell anyone when this happens. So I'm very happy about that. Because frankly, this is some hard-earned knowledge that I had to read Sidekiq and remember how database transactions work and convince myself of what was going on here and finally come to what I believe the solution is. And now Isolator is just like, cool, that's encapsulated. And it gives a very nice failure message in the test suite. So it's like, excellent. I really like this. But still looking at it, the diff, the amount of code that I had to change, it's like, well, naturally, this is how we want to write this code, but for reasons, we can't. And it's appeasing the computer more than it's appeasing the reader or the author of the code. And so then I happen to be reading through the Isolator gem's README, and they mention the aftercommiteverywhere gem. And I was like, oh, that's interesting. So one more time, I reset. And then I really tried fixing it with after_commit. And the look of the diff there felt nice to me because the lines got a little more on them, but they didn't move. And so it's like, this is how we naturally would have authored it, and now it works correctly. And I liked that. But I understand your hesitation because you're like, but the thing is, it's wrong. And so you've made the wrong not wrong anymore, but you didn't...and so I get your hesitation. I still like the fancy version. STEPH: Yeah, I think you just helped me figure out my grumpiness with it or why I'm not totally sold on it. And it was in regards to adding a dependency to avoid a noisy diff is the oversimplified version that I was processing or the reason that I was a bit grumpy about adding this other gem for that. But then you also just brought a lot of other really good reasons. One thing that you said that I do really like is adding tools that help us author code in a more natural style, the way that we want to highlight this process, and how this application does work, and how this business logic flows. So given in that light, that makes me feel better about it. But yeah, I think that was my initial grumpiness. I was like, it'll be a noisy diff. It's okay. CHRIS: I think I definitely share your hesitation, or you're like, hmm, that's an interesting reason to bring more code into the application. But at the same time, I think the counterpoint that comes to mind for me is we're using Ruby because of its expressiveness; at least, that's why I'm using Ruby. I really want the code that I write to be as close as possible to the thing that I would say to another human about like, oh okay, when a user signs up for the application, we need to create a record in our system, and then we need to send them an email. And then we need to do this other thing. And so, the closer that our code is to those words that I would use to describe to another human, the happier I am. And I will put in some pretty significant effort to hold that line as long as the code can also be correct. And so, the Isolator gem here does a great job of enforcing that correctness. And then after_commit allows me to still maintain that expressiveness and not have to think about the murky details as much or not have to reshape my code to match the murky realities of different persistence engines. But I do agree. I think it's a good thing to look at and ask, like, is it worth it? Are you sure? And in this case, I will say, "Yeah, I think so," but with that amount of certainty in my voice, [chuckles] which is not a ton. STEPH: I think this is going back to my days of working with dependency bot PRs where every time there was an upgrade for a gem, I always ask, what do you do here? [chuckles] Do we need to upgrade you? Can we just remove you from the codebase? So I'm fairly...I don't know, resistant is a strong word. I'm skeptical of when we're adding stuff in, and I just want to question the value that it's adding. But I want to circle back to something that you said, and that is hard-earned knowledge. And that part I understand so much where when you have gone through a fair amount of work to uncover an issue, and then you want to make sure that others don't have to go through that. This is a really nice way to highlight; hey, there's something that's tricky about computers and software here, and we need to watch out for that. And I want to help you lookout for that. Versus this is just inherit information where this needs to happen outside or after that transaction. And so that makes a really nice entry point where someone can look to say, "Why did we add this gem?" And then there's a commit message that goes with it that explains this is why we use this after_commit gem because we're specifically looking to avoid this type of bug. And I love that. CHRIS: Yeah, I think more lines of git commit message than diff on this one. So yeah, I wrote a short novel describing all of the features, describing the different pieces that are coming together. And then it's actually a +28 -6 diff. So it's a very small code change. But yeah, lots of story captured there. STEPH: And if you had just moved the lines, you could still have that commit message. But it's not likely that someone's going to look up that git commit change or that message that went along with it because they're not going to know to blame that one. But if they look at that particular edition of after_commit, they're more likely to find that historical context. So long story short, I think you have walked me through my initial grumpiness and provided some really good ways to avoid that really tricky failure mode for other developers. CHRIS: Well, thank you. I'm getting Steph's seal of approval starting from grumpy places. [laughs] I feel good. All right. STEPH: I'll have some special Stephanie's approval stickers designed and printed for you. CHRIS: I hope you're not joking because I very much want a yellow heart that says, "Steph-approved." STEPH: [laughs] CHRIS: And I can put it on PRs, and I can put it on the wall. [laughs] STEPH: Well, now I have to find a sticker designer and make a...well, it's just a yellow heart. I can probably handle this. I'm going to use Comic Sans. That will be the approved part. [laughs] Yellow hearts and Comic Sans for everybody. CHRIS: Well, with that absolutely fantastic call back to earlier parts of the episode, shall we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Byeeeeeeee! Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

man pr microsoft ireland putting utah sort yellow slack forty mid python ui github rails versus prs mandy moore skim apm defer comic sans redis readme postgres boolean pull requests mike burns chris yeah christ here activerecord chris so chris oh chris well isolator sidekiq chris yes chris thanks matt stauffer british dyslexia association chris okay evil martians chris exactly chris toomey

312: Spooky Stories

Play Episode Listen Later Oct 12, 2021 38:50

Chris evaluates the pros and cons between using Sidekiq or Active Job with Sidekiq. He sees exceptions everywhere. Steph talks about an SSL error that she encountered recently. It's officially spooky season, y'all! sidekiq-symbols (https://github.com/aprescott/sidekiq-symbols) Transcript: CHRIS: Additional radiation just makes Spider-Man more powerful. STEPH: [laughs] Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. Hey, Chris, what's new in your world? CHRIS: Fall is in the air. It's one of those, like, came out of nowhere. I knew it was coming. I knew it was going to happen. But now it's time for pumpkin beer and pumpkin spice lattes, and exclusively watching the movie Hocus Pocus for the next month or so or some variation of those themes. But unrelated to that, I did a thing that I do once, let's call it every year or so, where I had to make the evaluation between Sidekiq or Active Job with Sidekiq, as the actual implementation as the background job engine that is running. And I just keep running through this same cycle. To highlight it, Active Job is the background job system within Rails. It is a nice abstraction that allows you to connect to any of a number of them, so I think Delayed Job is one. Sidekiq is one. Resque is probably another. I'm sure there's a bunch of others. But historically, I've almost always used Sidekiq. Every project I've worked on has used Sidekiq. But the question is do you use Active Job with the adapter set to Sidekiq and then you're sort of living in both worlds, or do you lean in entirely and you use Sidekiq? And so that would mean that your jobs are defined to include Sidekiq::Worker because that's the actual thing that provides the magic as opposed to inheriting from Application Job. And then do you accept all of the trade-offs therein? And every time I go back and forth. And I'm like, well, but I want this feature, but I don't want that feature. But I want these things. So I've made a decision, but I want to talk ever so briefly through the decision points that were part of it. Have you done this back and forth? Are you familiar with the annoying choice that exists here? STEPH: It's been a while since I've had the opportunity to make that choice. I'm usually joining projects where that decision has already been made. So I can't think of a recent time that I've thought through it. And my current project is using that combination of where we are using Active Job and Sidekiq. CHRIS: So I think there's even a middle ground there where that was the configuration that I'd set up on the project that I'm working on. But you can exist in both worlds. And you can selectively opt for certain background jobs to be fully Sidekiq. And if you do that, then instead of saying, "Performlater," You say, "Performasync." And there are a couple of other configurations. It gives you access to the full Sidekiq API. And you can do things like hey, Sidekiq, here's the maximum number of retries or a handful of other things. But then you have to trade away a bunch of the niceties that Active Job gives. So as an example, one thing that Active Job provides that's really nice is the use of GlobalID. So GlobalID is a feature that they added to Rails a while back. And it's a way to uniquely identify a given record within your system such that when you say performlater, you can say, InvitationMailer.performlater and then pass it a user record so like an instance of a user model. And what will happen in the background is that gets serialized, but instead of serializing the whole user object because we don't actually want that, it will do the GlobalID magic. And so it'll turn into, I think it's GID:// so almost like a URL. But then it'll be, I think, your application name/model name down the road. And the Perform method actually gets invoked via the background system. Then you will just get handed that user record back, but it's not the same instance of the user record. It sort of freezes and thaws it. It's really nice. It's a wonderful little feature. Sidekiq wants nothing to do with that. STEPH: I'm so glad that you highlighted that feature because that was on my mind; I think this week where I was reviewing...somebody had made the comment where they were concerned about passing a record to a job and saying how that wouldn't play nicely with Sidekiq. And in the back of my mind, I'm like, yeah, that's right. But then I was also I'm pretty sure this got addressed, though. And I couldn't recall specifically if it was a Sidekiq enhancement or if it was a Rails enhancement. So you just cleared something up for me that I had not had time to confirm myself. So thanks. CHRIS: Well, to be clear, this works if you are using Active Job with Sidekiq as the adapter, but not if you are using a true Sidekiq worker. So if you opt-out of the Active Job flow, then you have to say, "Perform_async," and if you pass it a record, that's not going to work out particularly nicely. The other similar thing is that Sidekiq does not allow the use of keyword args, which, I'm going, to be honest, I really like keyword arguments, especially for background jobs or shuttling data through your system. And there's almost a lazy evaluation. I want some nicety to make sure that when I am putting something into a background job that I'm actually using the correct call signature, essentially passing the correct data in the correct shape. Am I passing a record, or am I passing the ID? Am I passing a list of options or a single option? Those sort of trade-offs that are really easy to subtly get wrong. I came around on this one because I realized although Active Job does support keyword arguments, the way it does that is it just has a JSON serialization format for them. So a keyword argument turns into a positional array with an associated hash that allows for the lookup or whatever. Basically, again, they handle the details. You get to use keyword args, which is great, with the exception that when you're actually calling performlater, that method performlater is a method missing type magic method. So it does not actually check the keyword arguments at that point. You're basically just passing an options hash as opposed to true keyword arguments that would error because they don't match up. And so when I figured that out, I was like, oh, never mind. This doesn't actually do the thing that I care about. It's a little bit nicer in terms of the signature of the method when you're defining your background job itself, but it doesn't actually do any logical checking. It doesn't give me any safety or robustness within my system. So I don't care about that. I did find a project called sidekiq-symbols, which does some things under the hood to how Sidekiq serializes and deserializes jobs, which I think gives largely the same behavior as Active Job. So I can now define my Sidekiq jobs with keyword arguments. Things will work. I can't use GlobalID. That's still out. But that's fine. I can do a little helper method that basically does the same thing as GlobalID or at least close approximation. But sidekiq-symbols lets me have keyword arg-like signatures in my methods; basically, it is. But again, it doesn't actually do any check-in when I'm enqueueing a job, and I am sad about that. STEPH: Yeah, that's another interesting distinction. And I'm unsurprisingly with you that I would favor having keyword args and having that additional safety in place. Okay, so I've been keeping track. And so far, it sounds like we have two points because I'm doing a little scorecard here between Active Job and Sidekiq. And we have two points in favor of Active Job because they offer a GlobalID, which then allows us to pass in a record, and then it takes care of the serialization for us. And then also, keyword args, which I agree with you that's a really nice feature to have in place as well. So I'm curious, so it sounded like you're leaning towards Active Job, but I don't want to spoil the ending. CHRIS: Yes, I could see why that's what you would be taking away from the conversation thus far. So again, just to reiterate, Active Job and Sidekiq with this sidekiq-symbols extension they both support keyword args, kind of. They support defining your job with keyword args and then enqueueing a job passing something that looks like keyword args. But it ends up...nobody's actually checking anything, so it's mostly like a syntactic nicety as opposed to any sort of correctness, which is still nicer, but it's not the thing that I actually want. Either way, nobody supports it, so it is not available to me. Therefore, it is not a consideration point. The GlobalID thing is nice, but it is really, again, it's a nicety more than anything. I have gone, and I'm leaning in the direction of full Sidekiq and Sidekiq everywhere as opposed to Active Job in most cases, but then Sidekiq when we need it. And that's because Sidekiq just has a lot more power and a lot more functionality. So, in particular, Sidekiq has a feature which allows you to say...it's a block that you put at the top of your Sidekiq job that says retries exhausted or something. I think Sidekiq retries exhausted is the actual full name of that at that point, which is really unfortunate in my mind, but anyway, I'll deal. At that point, you know that Sidekiq has exhausted all of the retries, and you can treat it as failed. I'm going, to be honest, I went on a quest to find a way to say, hey, I'm going to put some work into the background. It's really important for me to know if this work succeeds or if it fails. It's very easy to know if it succeeds because that just happens in-line in the method. But we can have an exception raised at basically any point; Sidekiq does a great job of catching those, of retrying, of having fundamental mechanisms there. But this is the best that I can get for this job failed. And so Active Job, as far as I can tell, does not have anything for this in order to say, yep, we are done. We are not going to keep working on this. This work has failed. It is dead. Dead is; actually, I think the more correct term for where we're at because failed is a temporary state, and then you retry after a failure. Whereas dead is, this has gone through all of its retries, and it will never be run again. Therefore, we should treat this as not having run. And in my case, the thing that I want to do is inform the user that this operation that we were trying to do on their behalf has not succeeded, will not succeed. And please reach out or otherwise deal with the fact that we were unable to do the thing that they asked us to do. That feels like a really important thing for me to be able to do, to be able to communicate back to my users. This is one of those situations where I'm looking at the available options, and I'm like, I feel like I can't be the only one who wants to know when something goes wrong. This feels like a thing that's important. But this is the best example that I've found, the Sidekiq retries exhausted block. And unfortunately, when I'm using it, it gets yielded the Sidekiq JSON blob deserialized, so it's like Ruby hash. But it's still like this blob of data. It's not the same data that gets passed into perform. And so, as a result, when I want to look up the record that was associated with it, I have to do this nested dig into the available hash of data. And it just feels like this is not a well-paved path. This is not something that is a deeply thought about or recommended use case. But again, I don't feel like I'm doing something weird here. Am I doing something weird, Steph, wanting to tell my users when I was unable to do the thing they asked me to do? [chuckles] STEPH: That feels like a very rhetorical question. [laughs] CHRIS: It does. I apologize. I'm leading the witness. But in your sincere heart of hearts, what do you think? STEPH: No, that certainly doesn't sound weird. I'm actually thinking back to some of the jobs that cause me stress in regards to knowing when they failed and then having that communication of knowing that we've exhausted all the retries. And, of course, knowing when those retries are exhausted is incredibly helpful. I am intrigued, though,, because you're highlighting that Active Job doesn't have the same option around setting the retry. And I'm trying to recall exactly how it's set. But I feel like I have set the retry count for Active Job. And maybe, as you mentioned before, that's because it's an abstraction, or I'm not sure if Active Job actually has that native support. So I feel a little confused there where I think my default instinct would have been Active Job does have that retry capability. But it sounds like you've discovered otherwise. CHRIS: I'm not actually sure what Active Jobs core retry logic or option looks like. So fundamentally, as far as I understand it, Active Job is an abstraction. And under the hood, you're always connecting an adapter. So it's either going to be Sidekiq, or Resque, or Delayed Job, or other. And each of those systems, whichever system you have as the adapter, is the one that's actually going to be managing retries. And so I know Sidekiq happens to have as a default 25 retries. And that spans, I think it's a two-week exponential back off. And Sidekiq has some very robust logic that they have implemented as the way retries exist within Sidekiq. I'm not sure what that would look like if you're trying to express it abstractly because it is slightly different. I know there was some good work that was done on Sidekiq to allow the Sidekiq options that's a method at the top level of the job, even if it's an Active Job job to express the retries. So that may be what you've seen, or there may be truly an abstraction that exists within Active Job, and then each adapter needs to know how to handle retries. But frankly, the what can Sidekiq do that Active Job can't? There's a whole bunch of stuff around limiting when you would retry limiting, enqueuing a job if there already exists one, when and how do those records get locked. There's a whole bunch of stuff. Sidekiq has a lot of power under the hood. And so if we want to be leaning into that, that's why I'm leaning towards let's just be Sidekiq all the time. Let's become Sidekiq experts. Let's accept that as a deep architectural decision within the app as opposed to just relying on the abstraction. Because fundamentally, if we're just using Active Job, we're not going to have access to the full power of Sidekiq or whatever the underlying system is, so sort of that decision that I'm making, but I don't know specifically around the retries. STEPH: Okay, thanks. That's really helpful. It's been a while since I've had to make this decision. I'm really enjoying you sharing your adventure because I'm trying to think what's the risk? If you don't use Active Job, what are the trade-offs? And you'd mentioned some of them around the GlobalID and keyword args, which are some niceties. But overall, if you don't go with the abstraction, if you lean into Sidekiq, the risk is then you want to migrate to a different enqueuing service. And something that we talk about is mitigating that risk, so then you can swap it out. That's also something I have never done or encountered where we've had to make that change. And it feels like a very low risk in my mind. CHRIS: Sidekiq feels like the thing you would migrate to, not a thing you would migrate from. It feels like it is the most powerful. And if anything, I expect at some point we'll be upgrading to Sidekiq pro or enterprise or whatever the higher versions that you pay for, but you get more features there. So in that sense, that is the calculation. That's the risk trade-off in my mind is that we're leaning into this technology and coupling ourselves more closely to it. But I don't see that as one that will reassess in the same way that people talk about Active Record and it being an ORM. And it's like, oh, we're abstracting the database underneath, and I'm like, no, I'm not. I'm always using Postgres. Please do not take Postgres. I'm not going to switch over to MySQL next week. That's totally fine if you start on MySQL. It's unlikely you're going to port over to Postgres. We may port to an entirely…like it's a Cassandra column store with a Kafka queue, I don't know, something weird down the road. But it's not going to be swapping out Postgres for MySQL or vice versa. Like you said, that's probably not a change that's going to happen. But that I think is the consideration. The other consideration I have in my mind is Active Job is the abstraction that exists within Rails. And so I can treat it as the lowest common denominator, and folks joining the project, it's nice to have that familiarity. So perform_later is the method on the Active Job jobs, and it has a certain shape to it. People may be familiar with that. Mailers will automatically use Active Job just implicitly under the hood. And so there's a familiarity, a discoverability. It's just kind of up the middle choice. And so if I can stick with that, I think there's a nicety there. But in this case, I think I'm choosing I would like the power and consistency on the Sidekiq side, and so I'm leaning into that. STEPH: Yeah, that makes a lot of sense to me. And I liked the other example you provided around things that were not likely to swap out and Postgres, MySQL, your database being one of them. And in favor of an example that I do have for something that...I do enjoy wrapping. It's not something that I adhere to strictly, but I do enjoy it when I have the space to make this choice. So I do enjoy wrapping HTTPClients, not just because then I can swap it out for a different HTTPClient, which frankly, that's also rare that I do that. Once I choose an HTTPClient, I'm probably pretty happy, and I don't need to swap it out. But I really like being able to extend to the API specifically if they don't handle error responses in a way that I would like to or if they raise, and then I want to change the API to have a more thoughtful interface and where I don't have to rescue those errors. But instead, I can interact with this object that then represents an error state. So that was just one example that came to mind for things that I do enjoy having an abstraction around and not just so I can swap it out because that feels like a very low risk, but more frankly, so I can extend the API. CHRIS: I definitely share the I almost always wrap APIs, or I try and hide whatever the implementation detail whether it be HTTPParty, or Faraday or whatever it is that I'm using and trying to hide that deeply within the system. And then I have whatever API client that we define. And that's what we're interacting with. It's interesting that you bring up errors and exceptions there because that's the one other thing that has caused me this...what I'm describing now seems perhaps like, oh, here's just a list of pros and cons, a simple decision was made, and there we are. This represents some real soul searching on my part, if we will. And one of the last things that I ran into that was just so frustrating is that Sidekiq is explicitly built around the idea of exceptions; Sidekiq retries if there is an exception raised in the job, otherwise, it treats it as success, and that's it. That is the entirety of it. That is the story. But if you raise an exception in a job, then you can't test that job because now it's raising an exception. You can't test retries or this retry exhausted block that I'm trying to lean into. I'm like, I want to put that in a feature spec and say, oh, this job goes in the background, but it's in a failure state, and therefore, the user sees the failure message. Sorry, I can't do that because the only way to actually fail a job is via an exception. And I've actually gone to some links in this application to try to introduce more structured data flow. I've talked a bunch about the command objects and the dry-monads and all those things. And I've really loved them where I've gotten to use them. But then I run into one of these edge cases where Sidekiq is like, no, no, no, you can't do that. And so now I have parts of my system that very purposefully return data as opposed to raising an exception. And I just have to turn around and directly raise that failure as an exception, and it just feels less expressive. I actually just ran into the identical thing with Pundit. They have a little bit better control over it; I can choose whether or not I want the raising version or not. But I see exceptions everywhere, and I want a little more discrete data flow. [chuckles] That is my dream. So anyway, I chose Sidekiq is the summary here. And slowly, we're going to migrate entirely to Sidekiq. And I'm going to be totally fine with it. And I'm done griping now. STEPH: This is your own little October Halloween movie, that I see exceptions everywhere. CHRIS: They're so spooky. STEPH: [laughs] That's cool about Pundit. I'm not sure I knew that, that you get to essentially turn on or off that exception flow behavior. On one hand, I'm like, that's nice. You get the option. On the other hand, I'm like, well, let's just not do it. Let's just never raise on people. But at least they give people options; that seems really cool. CHRIS: They do give the option. I think you can choose different strategies there. And also, if we're being honest, I'm newer to Pundit. And I used a different thing, which was to get the Policy Object and ask it a question. I wanted to ask, is this enabled or not? Can a user do this or not? That should not raise an exception. I'm just asking a question. We're just being real chill about this. I just want to know some information. Let's flow some data through our system. We don't need exceptions for that. STEPH: Why are you yelling at me? I just have a question. [laughs] CHRIS: Yeah. I figured out how to be easy on that front. Sidekiq apparently has no be easy mode, but that's fine. You know what? We're going to make it work, and it's going to be fine. But it is interesting deciding which of these facets of the system that I'm building do I really care about? Which are the ones where I'm like, whatever, just pick something, and we'll move forward, it's not a big deal? Versus, we're actually going to be doing a lot of work in the background. This is the thing that I care about deeply. I want to know about failure and success. I want to really understand that and have a robust answer to what our architecture looks like there. Similarly, Pundit for authorization. I believe that authorization will be a critical aspect of our system. It's typically a pretty important thing. But for us, I think we're going to have different types of users who can log in and see different subsets of data and having a consistent and concrete way that we have chosen to implement that we are able to test, that we're able to verify. I think that's another core competency within the app. But you only get to have so many of those. You can only be really good at a couple of things. And so I'm in that place where I'm like, which are our top five when I say are the things that I care a lot about? And then which are the things where I'm like, I don't know, whatever, just run with it? STEPH: Just a little bit ago, I came so close to singing because you said the I want to know phrase again. And that, I'm realizing, [laughs] is a trigger for me and a song where I want to sing. I held it back this time. CHRIS: It's smart. You got to learn anytime you sing on mic that is part of the permanent record. STEPH: Edward Loveall at thoughtbot, since I sang in a recent episode, did the delightful thing where then he grabbed that clip of where you talk a little bit, and then I sing and then encouraged everyone to go listen to it. And in which I responded, like, I would highly recommend that you save your ears and don't listen to it. But yes, singing on the mic is a thing. I do it from time to time. I can't hold it back. CHRIS: We all do. But since it doesn't seem that you're going to sing in this moment, I think I can probably wrap up my Odyssey of choosing between Sidekiq and Active Job. I hope those details were useful to anyone other than me. It was an adventure, so I figured I'd share it. But yeah, that about wraps it up on my side. Mid-roll Ad And now a quick break to hear from today's sponsor, Scout APM. Scout APM is leading-edge application performance monitoring that's designed to help Rails developers quickly find and fix performance issues without having to deal with the headache or overhead of enterprise platform feature bloat. With a developer-centric UI and tracing logic that ties bottlenecks to source code, you can quickly pinpoint and resolve those performance abnormalities like N+1 queries, slow database queries, memory bloat, and much more. Scout's real-time alerting and weekly digest emails let you rest easy knowing Scout's on watch and resolving performance issues before your customers ever see them. Scout has also launched its new error monitoring feature add-on for Python applications. Now you can connect your error reporting and application monitoring data on one platform. See for yourself why developers call Scout their best friend and try our error monitoring and APM free for 14 days; no credit card needed. And as an added-on bonus for Bike Shed listeners, Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at scoutapm.com/bikeshed. That's scoutapm.com/bikeshed. STEPH: So, I would love to talk about an SSL error that I encountered recently. So one of the important processes in our application is sending data to another system. And while sending data to that other system, we started seeing the following error that the read "Certificate verify failed." And then in parens, it states, "Unable to get local issuer certificate." So upon seeing that error, I initially thought, okay, something is wrong with their SSL certificate or their SSL configuration. And that's not something that I have control over and can fix. So we should reach out and let them know to take a look at their SSL config. But it turns out that their team already knew about the issue. They had recently updated or renewed their SSL cert, and they saw our messages were no longer being processed, and they were reaching out to us for help. So at that point, I'm still pretty sure that it's related to something on their end, and it's not something that I can really fix on our end. But we can help them troubleshoot. Maybe there's a workaround that we can add to still get messages processing while they're looking into their SSL config. It seemed like they still just needed help. So it was something that was still worth diving into. So going back to the first error, I want to talk a little bit about it because I realized that I understand SSL just enough, just the surface to get by as a developer. But then, every time that I run into a specific error with it, then I really have to refresh my understanding as to what could be wrong, so then I can troubleshoot more effectively. So for anyone that could use a refresher on that certificate verification process, when your browser or your server is connecting to a site that uses SSL, then your browser server, whichever one you're using, is going to download that site certificate and verify a couple of things. So it's going to check does the certificate contain the domain name of the website? So essentially, you gave us a certificate. Is this your certificate? Does it match the site that we're connecting to? Is this cert issued by a trusted certificate authority? So did someone that we trust give you this certificate? And is the cert still valid, or has it expired? So that part is pretty straightforward. The second part, "Unable to get local issuer certificate," so that's the part I was less certain about. And I took this to mean that they had passed two of those three checks that their cert included the site's name, and it had not expired. But for some reason, we aren't able to determine if their cert was issued by someone that we should trust. So following that journey, my next question was, so what are they giving us? So this is a tool that I don't get to use very often, but I reached for OpenSSL and, specifically, the s_client command, which connects to a specified domain and prints all certificates in the certificate chain. You may already know this, but the certificate chain is basically a fancy way of saying, show me all the certificates necessary to prove your site certificate was authorized by a trusted certificate authority. CHRIS: I did not know that. STEPH: Okay, I honestly didn't either. [laughs] CHRIS: I liked that you thought I would, though. So thank you, but no. [chuckles] STEPH: Yeah, it's one of those areas of SSL where I know just enough. But that was something that was new to me. I thought there was a site certificate, and I didn't realize that there is this chain of certificates that has to be honored. So going back and looking through that output of the certificate chain, that's what highlighted to me that their server was giving us their certificate and saying, hey, you should trust our site certificate. It's legit because it was authorized by, let's say, XYZ certificate. And so if it were a proper certificate chain, then they would give us that XYZ cert. And essentially, we can use this chain of certificates to get back to a trusted authority that then everybody knows that we can trust. However, they weren't actually giving us a reference certificate; they were giving us something else. So essentially, they were saying, "Hey, look at our certificate and look at this very trustworthy reference that we have." But they're actually failing to give us that reference. So to bring it all home, we can download that intermediate certificate that they reference; that is something that is publicly accessible. That's why we're able to then verify each certificate that's provided in that chain. We could go and download that intermediate certificate from that certificate authority. We could combine that with their site-specific certificate, include that in our request to their system, and then complete the certificate chain. And boom, we're back in business. But it was quite a journey. CHRIS: That is quite the journey. And yeah, I definitely knew very little of that, although everything you're saying makes sense. And I have a bunch of cubbyholes in my brain for SSL knowledge. And the words you said all fit into the spaces that I have in my brain, but I didn't know a bunch of those pieces. So thank you for sharing that. SSL and cryptography, more generally or password hashing or things like that, occupy this special place in my brain where I'm both really interested in them. And I will occasionally research them. If I see a blog article, I'll be like, oh yeah, I want to read more about this password hashing. And what's a Salt? And what's a Pepper? And what are we doing there? And what is BCrypt versus SCrypt? What are all these things? This is cool. And almost the arms race on the two sides of how do we demonstrate trust in a secure manner on the internet? But at the same time, I am not allowed to do anything with this information. I outsource this as much as humanly possible because it's one of those things that you just should not do yourself and SSL perhaps even more so. So I have configured aspects of my password hashing. But I 100% just lean on the fact that Let's Encrypt exists in the world. And prior to that, it was a little more work. But frankly, earlier on in my career, I wasn't dealing with the SSL parts of things. But I'm so grateful to Let's Encrypt as a project that exists. And now, on almost every platform that I work with, there's just a checkbox for please do the SSL work for me, make it good, make it work, and then I will be happy. And I'm so glad that that organization exists and really pushed the envelope also. I forget what it was, but it was only like three years ago where SSL was not actually nearly as common as it is now. And now it is pervasive and everywhere. And all of the sites have it, and so that is a wonderful thing. But I don't actually know much. I know that I should have it. I must have it. I should force it. That's true. So I push that out… STEPH: Hello. CHRIS: Are you trying to get me to sing? [chuckles] STEPH: [laughs] No, but I did want to know if you get the reference, the Salt-N-Pepa. CHRIS: Push It Real Good the song? Yeah, okay. STEPH: Yeah, you got it. [chuckles] CHRIS: I will just say the lyrics. I shall not sing the lyrics. I would say that, though, that yes, yes, they do that. STEPH: Thank you for acknowledging my very terrible reference. Circling back just a little bit too in regards to...I'm with you; this is a world that is not one that I am very deeply technical in and something that I learned a fair amount while troubleshooting this particular SSL error. And it was very interesting. But there's also that concern where it's like, that was interesting. And we worked around the issue, but this also feels very fragile. So we still haven't fixed it on their end where they are sending the wrong certificate. So then that's why we had to do more investigative work, and then download the certificate that they meant to send us, and then send back a complete certificate chain so that we don't have this error anymore. But should they change anything about their certificate, should they renew anything like that, then suddenly, we're going to break again. And then, the next developer is going to have to go through the same journey. And this wasn't a light journey. This was a good half-day journey to figure out what was going on and to spend the time, and then to also get that fix out to production. So it's a meaningful task that I don't want anyone else to have to go through. But we are relying on someone else updating their configuration. So, on one hand, we're in a good spot until they are able to update. But on the other hand, I wrote a heck of a commit message for the next person just describing like, friend, just grab some coffee if we're going to chat. It's a very small code change, but you need to know the scoop. So should you need to replicate this because they've changed something, or if this happens…because we work with a number of systems that we send data to. So if someone else should run into a similar issue, they will understand some of the troubleshooting techniques that I used and be able to look up that chain and find out if there's a missing cert or something else they need to provide. So it feels like a win, but I'm also nervous for future selves, future developers. So there's another approach that I haven't mentioned yet, but it was often a top recommendation for when dealing with SSL errors. And specifically, it was turning off SSL verification. And I saw that, and I was like, well, that won't work. I'm definitely sending sensitive, important data. And I need to verify that who I'm sending this to is really the person that I want to send this data to. So that was not an option for me. But it made me very nervous how often that was an approach that people would recommend and be like, oh, it's okay, just turn off SSL. You'll be fine. Like, don't worry about it. CHRIS: I feel like this so perfectly fits into the...some of our work is finding the information and connecting the pieces together and making it work. But some of it is that heuristic sense, that voice in the back of your head that is like, wait, I'm sorry, what? You want me to just turn off the security perimeter and hope that the velociraptors won't come in? That doesn't seem like it's going to end well. I get that that's an easy option that we have available to us right now and will solve the immediate problem but then let's play this out. There are four or five Jurassic Park movies now that tell the story of that. So let's be careful. STEPH: It always ends super well, though, right? Like, it's totally fine. [laughs] CHRIS: [laughs] Exclusively. Although it's funny that you mentioned OpenSSL no verify because just this past week, I used that very same configuration. I think it was okay in my case; I'm pretty sure. But it is interesting because when I saw it, I was like, oh no, can't do that. Certainly not that. Don't turn off the security feature. That's the wrong way to deal with the issue. But in the particular case that I'm working with, I'm using Redis, Heroku Redis, in particular, in a Heroku configuration. And the nature of how Heroku configures the Redis instances and the connectivity to our app into our dyno...I forget why. I read an article. They wrote it; Heroku wrote it. I trust them; they're good. I've outsourced my trust to people that I do trust. The trust chain actually maps really well to the certificate trust chain. I trust that Heroku has taken security deeply seriously. And for some reason, their configuration of Redis requires that I turn on OpenSSL no verify mode. So I'm using this now both in Sidekiq, and then we're using our Redis instance for our Rails cache as well. So in both cases, I said, "It's fine. Don't worry about it." I used the Don't worry about it configuration. And I didn't love it but I think it's okay. And partly, I'm trying to say this into the internet radio right now just in case anyone's listening who's like, no, no, no, you can't do that. That's bad. So I'm willing to be deeply wrong on the internet in favor of someone telling me and then I get to get out in front of it. But I think it's fine. Pretty sure it's fine. It should be fine. STEPH: I love love love that you gave a very visual example of velociraptors, and then you're like, oh, but I turned it off. [laughs] So I'm going to start sending you a velociraptor gif each day. CHRIS: I hope you do. I hope the internet holds you accountable to that. STEPH: [laughs] CHRIS: And I really look forward to [laughs] moving forward because that's a great way to start the day. Well, it doesn't need to start the day, but I look forward to them. STEPH: [laughs] I am really intrigued because I'm with you. Like you said, there are certain entities that are in our trust chain where it's like, hey, you are running this for us, and so I do have faith and trust in you that you wouldn't steer me wrong and provide a bad recommendation. Someone on Stack Overflow telling me to turn off SSL verify uh; that's not my trust chain. Heroku or someone else telling me I'm going to take it a little more seriously. And so I'm also interested in hearing from...what'd you say? You're speaking into the internet phone. [laughs] What'd you say? CHRIS: I think I said internet radio. But yeah, in a way. I mean, we're recording over Skype right now. So in a manner of speaking, we're on the internet phone to make our internet radio show. STEPH: [laughs] Oh goodness, the internet radio. I'm also intrigued to hear if other people are like, oh, no, no, no. Yeah, that sounds like an interesting scenario. Because I would think you'd still want your connection to...you said it's for Redis. So you still want that connection to be verified. But then if Redis itself can't have a specific...yeah, we're testing the boundaries of my SSL knowledge here as to how the heck you would even establish that SSL connection or the verification process. CHRIS: Me too. And it also exists in an interesting space where Heroku is rather clear in their documentation about this. And it was a surprising claim when I saw it. And so, I don't expect them to be flippant about a thing that is important. Like, if they're like, "No, no, no, it is okay. You can turn off the security thing, don't worry." I trust that they're not just like, oh, we didn't think about it too much. But we figured why not? It's not a big deal. I'm sure that they have thought about it deeply because it is an important thing. And so in a weird way, my trust of them and the severity of what this thing represents, I'm like, oh yeah, I super trust that because you're not going to get a major thing wrong. You might get a minor, small, subtle thing wrong. But this is a pretty major configuration change. As I say it, I'm now getting more worried. I'm now like, I feel fine about this. This doesn't seem like a problem at all. But then I keep saying stuff, and I'm like, oh no. That's why I love having a podcast; I find out things about myself as I talk into a microphone to you. STEPH: We come here to share our deep, dark developer secrets. Chris: Spooky developer therapy. STEPH: But just to clarify, even though you've turned off the SSL verify, you're still connecting over SSL. CHRIS: Yes, I believe that's the case. And if I'm remembering, I think the nature of how this works is they're using a self-signed certificate because of shared infrastructure or something, something that made sense when I read it. But it was the idea that they are doing a self-signed certificate. Therefore, to what you were talking about earlier, there isn't the certificate authority in the chain of those because it's self-signed. And so, they are not a trusted certificate authority. Therefore, that certificate that they have generated would not be trusted. But it does still allow for the SSL handshake and then communication to happen over SSL. It's just that fundamental question of trust. I'm saying, in this case, for reasons, it's okay. Trust me that I trust them. We're good. Which, again, I don't feel great about, but I think yes, it is still SSL, but it is a self-signed certificate. So we have to make this configuration change. STEPH: Yeah, all of that makes sense. And it certainly sounds like you have been very thoughtful about that change and put in some investigative work. So on that note, I have a very unrelated bad joke for you. CHRIS: I'm very excited. STEPH: All right, here we go. All right, so what do you call an alligator wearing a vest? CHRIS: I don't know. What do you call an alligator wearing a vest? STEPH: An investigator. [laughter] On that note, shall we wrap up? CHRIS: Oh, let's wrap up. We should also include a link in the show notes to the episode where you told the joke about the elephant hiding in the trees because that's one of my favorite jokes. You slayed me with that one. [laughs] But on that note, yes, let us wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes,,as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Byeeeeeeeeee!!! Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

296: Speedy Performance with Nate Berkopec

Play Episode Listen Later Jun 15, 2021 63:33

Nate Berkopec is the author of the Complete Guide to Rails Performance, the creator of the Rails Performance Workshop, and the co-maintainer of Puma. He talks with Steph about being known as "The Rails Speed Guy," and how he ended up with that title, publishing content, working on workshops, and also contributing to open source projects. (You could say he's kind of a busy guy!) Speedshop (https://www.speedshop.co/) Puma (https://github.com/puma/puma/commits/master?author=nateberkopec) The Rails Performance Workshop (https://www.speedshop.co/rails-performance-workshop.html) The Complete Guide to Rails Performance (https://www.railsspeed.com/) How To Use Turbolinks to Make Fast Rails Apps (https://www.speedshop.co/2015/05/27/100-ms-to-glass-with-rails-and-turbolinks.html) Sidekiq (https://sidekiq.org/) Follow Nate Berkopec on Twitter (https://twitter.com/nateberkopec) Visit Nate's Website (https://www.nateberkopec.com/) Sign up for Nate's Speedshop Ruby Performance Newsletter (https://speedshop.us11.list-manage.com/subscribe?u=1aa0f43522f6d9ef96d1c5d6f&id=840412962b) Transcript: STEPH: All right. I'll kick us off with our fancy intro. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. And this week, Chris is taking a break. But while he's away, I'm joined by Nate Berkopec, who is the owner of Speedshop, a Ruby on Rails performance consultancy. And, Nate, in addition to running a consultancy, you're the co-maintainer of Puma. You're also an author as you wrote a book called The Complete Guide to Rails Performance. And you run the workshop called The Rails Performance Workshop. So, Nate, I'm sensing a theme here. NATE: Yeah, make code go fast. STEPH: And you've been doing that for quite a while, haven't you? NATE: Yeah. It's pretty much been since 2015, or so I think. It all started when I actually wrote a blog post about Turbolinks that got a lot of pick up. My hot take at the time was that Turbolinks is actually a good thing. That take has since become uncontroversial, but it was quite controversial in 2015. So I got a lot of pick up on that, and I realized I liked working on performance, and people seem to want to hear about it. So I've been in that groove ever since. STEPH: When you started down the path of really focusing on performance, were you running your own consultancy at that point, or were you working for someone else? NATE: I would say it didn't really kick off until I actually published The Complete Guide to Rails Performance. So after that came out, which was, I think, March of 2016…I hope I'm getting that right. It wasn't until after that point when it was like, oh, I'm the Rails performance guy now. And I started getting emails inbound about that. I didn't really have any time when I was actually working on the CGRP to do that sort of thing. I just made that my full-time job to actually write, and market, and publish that. So it wasn't until after that that I was like, oh, I'm a performance consultant now. This is the lane I've driven myself into. I don't think I really had that as a strategy when I was writing the book. I wasn't like, okay, this is what I'm going to do. I'm going to build some reputation around this, and then that'll help me be a better consultant with this. But that's what ended up happening. STEPH: I see. So it sounds like it really started more as a passion and something that you wanted to share. And it has manifested to this point where you are the speed guy. NATE: Yeah, I think you could say that. I think when I started writing about it, I just knew...I liked it. I liked the work of performance. In a lot of ways, performance is a much more concrete discipline than a lot of other sub-disciplines of programming where I joke my job is number go down. It's very measurable, and it's very clear when you've made a difference. You can say, “Hey, this number was this, and now it's this. Look what I did.” And I always loved that concreteness of performance work. It makes it actually a lot more like a real kind of engineering discipline where I think of performance engineering as clarifying requirements and the limitations and then building a project that meets the requirements while staying within those limitations and constraints. And that's often not quite as clear for other disciplines like general feature work. It's kind of hard to say sometimes, like, did you actually make the user's life better by implementing such and such? That's more of a guess. That's more of a less clear relationship. And with performance, nobody's going to wake up ten years from today and wish that their app was slower. So we can argue about the relative importance of performance in an application, but we don't really argue about whether or not we made it faster because we can prove that. STEPH: Yeah. That's one area that working with different teams (as I tend to shift the clients that I'm working with every six months) where we often push hard around feature work to say, “How can we measure this? How can we know that we are delivering something valuable to users?” But as you said, that's really tricky. It's hard to evaluate. And then also, when you add on the fact that if I am leaving that project in six months, then I don't have the same insights to understand how something went for that team. So I can certainly appreciate the satisfaction that comes from knowing that, yes, you are delivering a faster app. And it's very measurable, given the time that you're there, whether it's a short time or if it's a long time that you're with that team. NATE: Yeah, totally. My consulting engagements are often really short. I don't really do a lot of super long-term stuff, and that's usually fine because I can point to stuff and say, “Yep. This thing was at A, and now it's at B. And that's what you hired me to do, so now it's done.” STEPH: I am curious; given that you have so many different facets where you are running your consultancy, you are also often publishing a lot of content and working on workshops and then also contributing to open source projects. What does a typical week look like for you? NATE: Well, right now is actually a decent example. I have client work two or three days a week. And I'm actually working on a new product right now that I'm calling Sidekiq in Practice, which is a course/workshop about scaling Sidekiq from zero to 1000 jobs per second. And I'll spend the other days of the week working on that. My content is...I always struggle with how much time to spend on blogging specifically because it takes so much time for me to come up with a post and publish that. But the newsletter that I write, which I try to write two once a week, I haven't been doing so well with it lately. But I think I got 50 newsletters done in 2020 or something like that. STEPH: Wow. NATE: And so I do okay on the per-week basis. And it's all content I've never published anywhere else. So that actually is like 45 minutes of me sitting down on a Monday and being like rant, [chuckles] slam keyboard and rant and then hit send. And my open source work is mostly 15 minutes a day while I'm drinking morning coffee kind of stuff. So I try to spread myself around and do a lot of different stuff. And a lot of that means, I think, pulling back in terms of thinking how much you need to spend on something, especially with newsletters, email newsletters, it was very easy to overthink that and spend a lot of time revising and whatever. But some newsletter is better than no newsletter. And especially when it comes to content and marketing, I've learned that frequency and regularity is more important than each and every post is the greatest thing that's ever come out since sliced bread. So trying to build a discipline and a practice around doing that regularly is more important for me. STEPH: I like that, some newsletter is better than no newsletter. I was listening to your chat with Brittany Martin on the Ruby on Rails podcast. And you said something very honest that I appreciated where you said, “Writing is really hard, and writing sucks.” And that made me laugh in the moment because even though I do enjoy writing, I still find it very hard to be disciplined, to sit down and make it happen. And then you go into that editor mode where you critique everything, and then you never really get it published because you are constantly fixing it. It sounds like...you've mentioned you set aside about 45 minutes on a Monday, and you crank out some work. How do you work through that inner critic? How do you get past it to the point where then you just publish? NATE: You have to separate the steps. You have to not do editing and first drafting at the same time. And the reason why I say it sucks and it's hard is because I think a lot of people don't do a lot of regular writing, maybe get intimidated when they try to start. And they're like, “Wow, this is really hard. This is not fun.” And I'm just trying to say that's everybody's experience and if it doesn't get any better, because it doesn't, [chuckles] there's nothing wrong with you, that's just writing, it's hard. For me, especially with the newsletter, I just have to give myself permission not to edit and to just hit send when I'm done. I try to do some spell checking,, and that's it. I just let it go. I'm not going back and reading it through again and making sure that I was very clear and cogent in all my points and that there's a really good flow through that newsletter. I think it comes with a little bit of confidence in your own ideas and your own experience and knowledge, believing that that's worth sharing and that's worth somebody's time, even if it's not a perfect expression of what's in your head. Like, a 75% expression is good enough, especially in a newsletter format where it's like 500 to 700 words. And it's something that comes once a week. And maybe not everyone's amazing, but some of them are, enough of them are that people stay subscribed. So I think a combination of separating editing and first drafting and just having enough confidence and the basis where you have to say, “It doesn't have to be perfect every single time.” STEPH: Yeah, I think that's something that I learned a while back to apply to my coding process where I had to separate those two steps of where I have to let the creator in me just create and write some code and make it work, and then come back to the editing process, and taking a similar approach with writing. As you may be familiar with thoughtbot, we're big advocates when it comes to sharing content and sharing things that we have learned throughout the week and different projects that we're working on. And often when people join thoughtbot, they're very excited to contribute to the blog. But it is daunting for that first post because you think it has to be this really grand novel. And it has to be something that is really going to appeal to everybody, and it's going to help everyone. And then over time, you learn it's like, oh well, actually it can be this very just small thing that I learned that maybe only helps 20 people, but it still helped those 20 people. And learning to publish more frequently versus going for those grand pieces is more favorable and often more helpful for people. NATE: Yeah, totally. That's something that is difficult for people at first. But everything in my experience has led me to believe that frequency and regularity is just as, if not more important than the quality of any individual piece of content that I put out. So that's not to say that...I guess it's weird advice to give because people will take it too far the other way and think that means he's saying quality doesn't matter. No, of course, it does, but I think just everyone's internal biases are just way too tuned towards this thing must be perfect. I've also learned we're just really bad judges internally of what is useful and good for people. Stuff that I think is amazing and really interesting sometimes I'll put that out, and nobody cares. [chuckles] And the other stuff I put out that's just like the 45-minute banging out newsletter, people email me back and say, “This is the most helpful thing anyone's ever read.” So that quality bias also assumes that you know what is good and actually we're not really good at that, knowing every time what our audience needs is actually really difficult. STEPH: That's totally fair. And I have definitely run into that too, where I have something that I'm very proud of and excited to share, and I realize it relates to a very small group of people. But then there's something small that I do every day, and then I just happen to tweet about it or talk about it, and suddenly that's the thing that everybody's really excited about. So yeah, you never know. So share it all. NATE: Yeah. And it's important to listen. I pay attention to what people get interested in from what I put out, and I will do more of that in the future. STEPH: You mentioned earlier that you are working on another workshop focused on Sidekiq. What can you tell me about that? NATE: So it's meant to be a guide to scaling Sidekiq from zero to 1000 requests per second. And it's meant to be a missing guide to all the things that happen, like the situations that can crop up operationally when you're working on an application that does a lot of work with Sidekiq. Whereas Mike Sidekiq, Wiki, or the docs are great about how do, you do this? What does this setting mean? And the basics of getting it just running, Sidekiq in practice, is meant to be the last half of that. How do you get it to run 1,000 jobs per second in a day-to-day application? So it's the collected wisdom and collected battle scars from five years of getting called in to fix people's Sidekiq installations and very much a product of what are the actual problems that people experience, and how do you fix and deal with those? So stuff about memory and managing Sidekiq memory usage, how to think about queues. Like, what should your queue structure be? How many should you have? Like, how do you organize jobs into queues, and how do you deal with problems like some client is dropping 10,000, 20,000 jobs into a queue. And now the other jobs I put in that queue have 20,000 jobs in front of them. And now this other job I've got will take three hours to get through that queue. How do you deal with problems like that? All the stuff that people have come to me over the years and that I've had to help them fix. STEPH: That sounds really great. Because yeah, I find that teams who are often in this space with Sidekiq we just let it run until there's a fire. And then suddenly, we start to care as to how it's processing, and we care about our queue structure and how many workers that we have that are pulling from that queue. So that sounds really helpful. When you're building a workshop, do you often go back to any of those customers and pull more ideas from them, or do you find that you just have enough examples from your collective work with clients that that itself creates a course? NATE: Usually, pretty much every chapter in the workshop I've probably implemented like three-plus times, so I don't really have to go back to any individual customer. I have had some interesting stuff with my current client, Gusto. And Gusto is going through some background job reorganization right now and actually started to implement a lot of the things that I'm advocating in the workshop actually without talking to me. It was a good validation of hey, we all actually think the same here. And a lot of the solutions that they were implementing were things that I was ready to put down into those workshops. So I'd like to see those solutions implemented and succeed. So I think a lot of the stuff in here has been pretty battle-tested. STEPH: For the Rails Performance Workshop, you started off doing those live and in-person with teams, and then you have since switched to now it is a CLI course, correct? NATE: That's correct. Yep. STEPH: I love that very much. When you've talked about it, it does feel very appropriate in terms of developers and how we like to consume content and learn. So that is really novel and also, it seems like a really nice win for you. So then other people can take this course, but you are no longer the individual that has to deliver it to their team, that they can independently take the course and go through it on their own. Are you thinking about doing the same thing for the Sidekiq course, or what are your plans for that one? NATE: Yeah, it's the exact same structure. So it's going to be delivered via the command line. Although I would say Sidekiq in practice has more text components. So it's going to be a combination of a very short manual or book, and some video, and some hands-on exercises. So, an equal blend between all three of those components. And it's a lot of stuff that I've learned over having to teach; I guess intermediate to advanced programming concepts for the last five years now that people learn at different paces. And one of the great things about this kind of format is you can pick it up, drop it off, and move at your own speed. Whereas a lot of times when I would do this in person, I think I would lose people halfway through because they would get stuck on something that I couldn't go back to because we only had four hours of the day. And if you deliver it in a class format, you're one person, and I've got 24 other people in this room. So it's infinitely pausable and replayable, and you can go back, or you can just skip ahead. If you've got a particular problem and you're like, hey, I just want to figure out how to fix such and such; you can do that. You can just come in and do a particular thing and then leave, and that's fine. So it's a good format that way. And I've definitely learned a lot from switching to pre-recorded and pre-prepared stuff rather than trying to do this all live in person. STEPH: That is one of the lessons that I've learned as well from the couple of workshops that I've led is that doing them in person, there's a lot of energy. And I really enjoy that part where I get to see people respond to the content. And then I get a lot of great feedback from people about what type of questions they have, where they are getting stuck. And that part is so important to me that I always love doing them live first. But then you get to the point, as you'd mentioned, where if you have a room full of 20 people and you have two people that are stuck, how do you help them but then still keep the class going forward? And then, if you are trying to tailor this content for a wide audience…so maybe beginners could take the Rails Performance Workshop, or they could also take the Sidekiq course. But you also want the more senior engineers to get something out of it as well. It's a very challenging task to make that content scale for everyone. NATE: Yeah. What you said there about getting feedback and learning was definitely something that I got out of doing the Rails Performance Workshop in person like three dozen times, was the ability to look over people's shoulders and see where they got stuck. Because people won't email me and say, “Hey, this thing is really confusing.” Or “It doesn't work the way you said it does for me.” But when I'm in the same room with them, I can look over their shoulder and be like, “Hey, you're stuck here.” People will not ask questions. And you can get past that in an in-person environment. Or there are even certain questions people will ask in person, but they won't take the time to sit down and email me about. So I definitely don't regret doing it in person for so long because I think I learned a lot about how to teach the material and what was important and how people...what were the problems that people would encounter and stuff like that. So that was useful. And definitely, the Rails Performance Workshop would not be in the place that it is today if I hadn't done that. STEPH: Yeah, helping people feel comfortable asking questions is incredibly hard and something I've gone so far in the past where I've created an anonymous way for people to submit questions. So during class, even if you didn't want to ask a question in front of everybody, you could submit a question to this forum, and I would get notified. I could bring it up, and we could answer it together. And even taking that strategy, I found that people wouldn't ask questions. And I guess it circles back to that inner critic that we have that's also preventing us from sharing knowledge that we have with the world because we're always judging what we're going to share and what we're going to ask in front of our peers who we respect. So I can certainly relate to being able to look over someone's shoulder and say, “Hey, I think you're stuck. We should talk. Let me walk you through this or help you out.” NATE: There are also weird dynamics around in-person, not necessarily in a small group setting. But I think one thing I really picked up on and learned from RailsConf2021 which was done online, was that in-person question asking requires a certain amount of confidence and bravado that you're not...People are worried about looking stupid, and they won't ask things in a public or semi-public setting that they think might make them look dumb. And so then the people that do end up asking questions are sometimes overconfident. They don't even ask a question. They just want to show off how smart they are about a particular issue. This is more of an issue at conferences. But the quality of questions that I got in the Q&A after RailsConf this year (They did it as Discord chats.) was way better. The quality of questions and discussion after my RailsConf talk was miles better than I've ever had at a conference before. Like, not even close. So I think experimenting with different formats around interaction is really good and interesting. Because it's clear there's no perfect format for everybody and experimenting with these different settings and different methods of delivery has been very useful to me. STEPH: Yeah, that makes a ton of sense. And I'm really glad then for those opportunities where we're discovering that certain forums will help us get more feedback and questions from people because then we can incorporate that and to future conferences where people can speak up and ask questions, and not necessarily be the one that's very confident and enjoys hearing their own voice. For the Rails Performance Workshop, what are some of the general things that you dive into for that workshop? I'm curious, what is it like to attend that workshop? Although I guess one can't attend it anymore. But what is it like to take that workshop? NATE: Well, you still can attend it in some sense because I do corporate bookings for it. So if you want to buy 20 seats, then I can come in and basically do a Q&A every week while everybody takes the workshop. Anyway, I still do that. I have one coming up in July, actually. But my overall approach to performance is to always start with monitoring. So the course starts with goals and monitoring and understanding where you want to go and where you are when it comes to performance. So the first module of the Rails Performance Workshop is actually really a group exercise that's about what are our performance requirements and how can we set those? Both high-level and low-levels. So what is our goal for page load time? How are we going to measure that? How are we going to use that to back into lower-level metrics? What is our goal for back-end response times? What is our goal for JavaScript bundle sizes? That all flows from a higher-level metric of how fast you want the page to load or how fast you want a route to change in a React app or something, and it talks about those goals. And then where should you even start with where those numbers should be? And then how are you going to measure it? What are the browser events that matter here? What tools are available to help you to get that data? Because without measurement, you don't really have a performance practice. You just have people guessing at what stuff is faster and what is not. And I teach performance as a scientific process as science and engineering. And so, in the scientific method, we have hypotheses. We test those hypotheses, and then we learn based on those tests of our hypotheses. So that requires us to A, have a hypothesis, so like, I think that doing X makes this faster. And I talk about how you generate hypotheses using profiling, using tools that will show you where all the time goes when you do this particular operation of your software—and then measuring what happens when you do that? And that's benchmarking. So if you think that getting rid of method X or changing method X will speed up the app, benchmarking tells you did you actually speed it up or not? And there are all sorts of little finer points to making sure that that hypothesis and that experiment is tested in a valid way. I spend a lot of time in the workshop yapping about the differences between development/local environments and production environments and which ones matter. Because what differences matter, it's not often the ones that we think about, but instead it's differences like actually in Rails apps the asset packaging and asset pipeline performs very differently in production than it does in development, works very differently. And it makes it one of the primary reasons development is slower than production, so making sure that we understand how to change those settings to more production-like settings. I talk a lot about data. It's the other primary difference between development and production is production has a million users, and development has 10. So when you call things like User.all, that behavior is very different in production than it is locally. So having decent production-like data is another big one that I like to harp on in the workshops. So it's a process in the workshop of you just go lesson by lesson. And it's a lot of video followed up by hands-on exercises that half of them are pre-baked problems where I'm like, hey, take a look at this Turbolinks app that I've given you and look at it in DevTools. And here's what you should see. And then the other half is like, go work on your application. And here are some pull requests I think you should probably go try on your app. So it's a combination of hands-on and videos of the actual experience going through it. STEPH: I love how you start with a smaller application that everyone can look at and then start to learn how performant is this particular application that I'm looking at? Versus trying to assess, let's say, their own application where there may be a number of other variables that they have to consider. That sounds really nice. You'd mentioned one of the first exercises is talking about setting some of those goals and perhaps some of those benchmarks that you want to meet in terms of how fast should this page load, or how quickly should a response from the API be? Do you have a certain set of numbers for those benchmarks, or is it something that is different for each product? NATE: Well, to some extent, Google has suddenly given us numbers to work with. So as of this month, I think, June 2021, Google has started to use what they're calling Core Web Vitals in their ranking of search results. They've always tried to say it's not a huge ranking factor, et cetera, et cetera, but it does exist. It is being used. And that data is based on Chrome user telemetry. So every time you go to a website in Chrome, it measures three metrics and sends those back to Google. And those three metrics are Largest Contentful Paint (LCP), First Input Delay (FID), and Cumulative Layout Shift (CLS). And First Input Delay and Cumulative Layout shift are more important for your single-page apps kind of stuff. It's hard to screw those up with a Golden Path Rails app that just does Turbolinks or Hotwire or whatever. But Largest Contentful Paint is an easy one to screw up. So Google's line in the sand that they've drawn is 2.5 seconds for Largest Contentful Paint. So that's saying that from clicking on your website in a Google search result, it should take 2.5 seconds for the page to paint the largest component of that new page. That's often an image or a video or a large H1 tag or something like that. And that process then will help you to...to get to 2.5 seconds in Largest Contentful Paint; there are things that have to happen along the way. We have to download and execute all JavaScript. We have to download CSS. We have to send and receive back-end responses. In the case of a simple Hotwire app, it's one back-end response. But in the case of a single-page app, you got to download the document and then maybe download several XHR fetches or whatever. So there's a chain of events that has to happen there. And you have to walk that back now from 2.5 seconds in Largest Contentful Paint. So that's the line that I'm seeing getting drawn in the sand right now with Google's Core Web Vitals. So pretty much any meaningful web application performance metric can be walked back from that. STEPH: Okay. That's super helpful. I wasn't aware of the Core Web Vitals and that particular stat that Google is using to then rank the sites. I was going to ask, this kind of blends in nicely into when do you start caring about performance? So if you have a new application that you are just starting to get to market, based on the fact that Google is going to start ranking you right away, you do have to care some right out of the gate. But I am curious, when do you start caring more about performance, and are there certain tools and benchmarking that you want to have in place from day one versus other things that you'll say, “Well, we can wait until we have X numbers of users or other conditions before we add more profiling?” NATE: I'd say as an approach, I teach people not to have a performance strategy of monitoring. So if your strategy is to have dashboards and look at them regularly, you're going to lose. Eventually, you're not going to look at that dashboard, or more often, you just don't understand what you're looking at. You just install New Relic or Datadog or whatever, and you don't know how to turn a dashboard into actual action. Also, it seems to just wear teams out, and there's no clear mechanism when you just have a dashboard of turning that into oh, well, this has to now be something that somebody on our team has to go work on. Contrast that with bugs, so teams usually have very defined processes around bugs. So usually, what happens is you'll get an Exception Notification through Sentry or Bugsnag or whatever your preferred Exception Notification service is. That gets read by a developer. And then you turn that into a Jira ticket or a Kanban board or whatever. And then that is where work is done and prioritized. Contrast that with performance; there's often no clear mechanism for turning metrics into stuff that people actually work on. So understanding at your organization how that's going to work and setting up a process that automatically will turn performance issues into actual work that people get done is important. The way that I generally teach people to do this is to focus instead of dashboards and monitoring, on alerts, on automated thresholds that get tripped and then sends somebody's an email or put something in the Kanban board or whatever. It just has to be something that automatically gets fired. Different tools have different ways of doing this. Datadog has pretty much built their entire product around monitoring and what they call monitors. That's a perfectly fine way to do it, whatever your chosen performance monitoring tool, which I would say is a required thing. I don't think there's really any good excuse in 2021 for not having a performance monitoring tool. There are a million different ways to slice it. You can do it yourself with OpenTelemetry and then like statsD, I don't know, or pay someone else like everyone else does for Datadog or New Relic or AppSignal or whatever. But you got to have one installed. And then I would say you have to have some sort of automated alerting. Now that alerting means that you've also decided on thresholds. And that's the hard work that doesn't get done when your strategy is just monitoring. So it's very easy to just install a dashboard and say, “Hey, I have this average page time load dashboard. That means I'm paying attention to performance.” But if you don't have a clear answer to what number is good and what number is bad, then that dashboard cannot be turned into real action. So that's why I push monitoring so hard is because it allows people to ignore performance is all that matters, and it forces you to make the decision upfront as to what number matters. So that is what I would say, install some kind of performance monitoring. I don't really care what kind. Nowadays, I also think there's probably no excuse to not have Real User Monitoring. So there's enough GDPR compliance Real User Monitoring now that I think everyone should be using it. So for industry terms, Real User Monitoring is just performance monitoring in the browser. So it's just users' browser APIs and sends those back to you or your third-party provider, so having that so you actually are collecting back-end and front-end performance metrics. And then making decisions around what is bad and what is good. Probably everybody should just start with a page load time monitor, Largest Contentful Paint monitor. And if you've got a single-page app, probably hooking up some stuff around route changes or whatever your app...because you don't actually have page loads on every single time you navigate. You have to instrument whatever those interactions are. So having those up and then just drawing some lines that say, “Hey, we want our React route changes to always be one second or less.” So I will set an alert that if the 95th percentile is one second or more, I'm going to get alerted. There's a lot of different ways to do that, and everybody will have different needs there. But having a handful of automated monitors is probably a place to start. STEPH: I like how you also focus on once you have decided those thresholds and have that monitoring in place, but then how do you make it actionable? Because I have certainly been part of teams where we get those alerts, but we don't necessarily...what you just mentioned, prioritize that work to get done until we have perhaps a user complaint about it. Or we start actually having pages that are timing out and not loading, and then they get bumped up in the priority queue. So I really like that idea that if we agree upon those thresholds and then we get alerted, we treat that alert as if it is a user that is letting us know that a page is too slow and that they are unable to use our application, so then we can prioritize that work. NATE: And it's not all that dissimilar to bugs, really. And I think most teams have processes around correctness issues. And so, all that my strategy is really advocating for is to make performance fail loudly in the same way that most exceptions do. [chuckles] Once you get to that point, I think a lot of teams have processes around prioritization for bugs versus features and all that. And just getting performance into that conversation at least tends to make that solve itself. STEPH: I'm curious, as you're joining teams and helping them with their performance issues, are there particular buckets or categories of performance issues that are the most common in terms of, let's say, 50% of issues are SQL-related N+1 issues? What tends to be the breakdown that you see? NATE: So, when it comes to why something is slow in a Ruby application, I teach a method that I call DRM. And that doesn't have anything to do with actual DRM. It's just memorable because it reminds me of things I don't like. DRM stands for Database Ruby and Memory and in that order. So the most common issue is database, the second most common issue is issues with your Ruby code. The least common issue is memory. Specifically, I'm talking about allocation of objects, creating lots of objects. So probably 80% of your issues are in some way database-related. In Rails, it's 50% of those are probably N+1. And then 30% of database issues are probably what I would call unnecessary SQL. So it's not necessarily N+1, but it's a SQL query for information that you already had, or you could do in a more efficient way. So a common thing for unnecessary SQL would be people will filter an ActiveRecord::Collection like ten different ways when they could have just loaded the whole collection, filtered it with Ruby in the ten different ways afterwards, and that works really well if the collection that you're loading is like 10, 20. Turning that into one database query, plus a bunch of calls to innumerable methods is often way faster than doing that as ten separate database queries. Also, that tends to be a more robust approach. This doesn't happen in most companies, but what could happen is the database is like a shared resource. It's a resource that everybody is affected by. So a performance degradation to the database is the worst possible scenario because everything is affected. But if you screw up what's happening at an individual Rails process, then only that Rails process is affected. The blast radius is tiny. It's just that one request. So doing less stuff in the database while it can actually seem like, oh, that doesn't feel right. I'm supposed to do a lot of stuff in the database. It actually can reduce blast radiuses of performance issues because you're not doing it on this database that everyone has to have access to. There are a lot of areas of gray here. And I talk a lot in all my other material like why -- There's a lot of nuance here. So database is the main stuff. Issues in how you write your Ruby code is probably the other one. Usually, that's just what I would call code that goes bump in the night. It's code that you don't know is running but actually is. Profilers are what help us figure that out. So oftentimes, I'll have someone open up a profiler on their controller action for the first time. And they're like, wait a minute, I had no idea that such and such was running during this controller action, and actually, we don't need to do that at all. So why is it here? So that's the second most common issue. And then the third issue that really doesn't actually come up all that often is object allocation, numbers of objects that get created. So primarily, this is a problem in index actions or actions transactions that deal with big collections. So in Ruby, we often get overly focused on garbage collection, but garbage collection doesn't take any time if you just don't create objects. And object creation itself takes time. So looking at code through the lens of what object does this code create? And trying to get rid of those object allocations can often be a pretty productive way to make stuff faster. STEPH: You said a lot of amazing things there. So I'm debating on which one to follow up on. I think the one that stuck out to me the most where I have felt pain around this is you mentioned identifying code that goes bump in the night or code that is running, but it doesn't need to be run. And that is something that I've run into with applications where we have a code path that seems important, but yet I can't prove that it's being executed and exactly why it's there and what flow it's supporting. And I'm curious, do you have any tips or tricks in how you've helped teams identify that this code path isn't used and it's something that we can remove and then that itself will help speed up the performance of that particular endpoint? NATE: Like, there's no performance cost to like 100 models in an application that never actually get used. There's really no performance downside to code in an app that doesn't actually ever get run. But instead, what happens is code gets added into callbacks that usually is probably the biggest offender that's like, always do this thing after you do X. But then, two years later, you don't always need to do that thing after you do X. So the callbacks always run, but sometimes requirements change, and they don't always need to be run. So usually, it's enough to just pop the profiler now on something. And I have people look at it, and they're like, “I don't know why any of this is happening.” Like, it's usually a pretty big Eureka moment once we look at a flame graph for the first time and people understand how to read those, and they understand what they're looking at. But sometimes there's a bit of a process where especially in a bigger app where it's like, “Such and such is running, and this was an entire other team that's working on this. I have no idea what this even does.” So on bigger apps, there's going to be more learning that has to get done there. You have to learn about other parts of the application that maybe you've never learned about before. But profiling helps us to not only see what code is running but also what that relative importance is. Like, okay, maybe this one callback runs, and you don't know what it does, and it's probably unnecessary. But if it only takes 1% of the total time to run this action, that's probably less important than something that takes 20% of total time. And so profilers help us to not only just see all the code that's being run but also to know where that time goes and what time corresponds to what parts of the code. STEPH: Yeah, that's often the code that makes me the most nervous is where it's code that I suspect is being run or maybe being run, but I don't understand why it's there and then figuring out if it can be removed and then figuring out ways to perhaps even log when a call is being made to that code to determine if it's truly in use or not or at least supported by a code path that a user is hitting. You have a blog post that I read recently that I really appreciated that talks about essentially gaming benchmarking where you talk about the importance of having context around benchmarks. So if someone says, “I've improved something where it is now 10% faster.” It's like, well, what is that 10% relative to? And if it's a tool that other people are using, what does that mean for them? Or did you improve something that was already very fast, and you made it 10% faster? Was that a really valuable use of your time? NATE: Yeah. You know, something that I read recently that made me think of that again was this Hacker News post that went viral. That was like, how I optimize an AWS EC2 instance to take 1.5 million requests per second on my JSON API. And out of the box, it was like 500 requests per second, and then he got it to 1.5 million. And the whole article was presented with relative numbers. So it was like, “I made this change, and things got 33% faster. And if you do the whole thing right, 500 to 1.5 million requests per second, it's like my app is three times faster now,” or whatever. And that's true, but it would probably be more accurate to say, “I've taken three-millionth of a second out of every request in my app.” That's two ways of saying the same thing because latency and throughput are just related that way. But it's probably more accurate and more useful to say the absolute number, but it doesn't make for great blog posts, so that doesn't tend to get said. The kinds of improvements that were discussed in this article were really, really low-level stuff. That was like if you turn off...I think it was like turn off iptables or something like that. And it's like, that shaves a microsecond off of every time we make a syscall or something. And that is useful if your performance goal is to serve 1.5 million requests per second Hello World responses off of my EC2 instance, which is what this person admittedly was doing. But there's a tendency to walk that back to if I do all things in this article, my application will be three times faster. And that's just not what the evidence says. It's not what you were told. So there's just a tendency to use relative numbers when absolute numbers would be more useful to giving you the context of like, oh, well, this will improve my app or it won't. We get this a lot in Puma. We get benchmarks that are like, hey, this thing is going to help us to do 50,000 requests per second in Puma instead of 10,000. And another way of saying that is you took a couple of nanoseconds off of the overhead of every single request to Puma. And most Puma applications have a hundred millisecond response time. So it's like, yeah, I guess it's cool that you took a nanosecond off, and I'm sure it's going to help us have cool benchmarks, but none of our users are going to care. No one that's used Puma is going to care that their requests are one nanosecond faster now. So what did we really gain here? STEPH: Yeah, it makes sense that people would want to share those more...I want to call them sparkly stats and something that catches your attention, but they're not necessarily something that's going to translate to us in the way that we hoped that they will in terms of it's not going to speed up our app 30% or have those same rewards or benefits. Speaking of Puma, how is it being a co-maintainer of Puma? And how do you balance that role with all of your other work? NATE: Actually, it doesn't take all that much of my time. I try to spend about 15 minutes a day on it. And that's really possible because of the philosophy I have around open-source maintenance. I think that open source projects are fundamentally about collaboration and about sharing our hard-fought extractions and fixes and knowledge together. And it's not about a single super contributor or super maintainer who is just out of the goodness of their heart releasing all of their incredible work and time into the public domain or into a free software license. Puma is a pretty popular piece of Ruby software, so a lot of people use it. And I have things on my back burner of if I ever got 20 hours to work on Puma, here's stuff I would do. But there are a lot of other people that have more time than me to work on Puma. And they're just as smart, and they have other tools they've got in their locker that I don't have. And I realized that it was more important that I actually find ways to recruit and then unblock those people than it was for me to devote as much time as I could to Puma. And so my work on Puma now is really just more like management than anything else. It's more trying to recruit new contributors and trying to give them what they need to help Puma. And contributing to open source is a really fraught experience for a lot of people, especially their first time. And I think we should also be really conscious of that. Like, 95% of software developers have really never contributed to open source in a meaningful way. And that's a huge talent pool of people that could be helping us that aren't. So I'm less concerned about the problems of the 5% that are currently contributing than I am about why there are 95% of us that don't do anything. So that's what gets me excited to work on Puma now, is trying to change that ratio. STEPH: I really like that mindset of where you are there to provide guidance but then essentially help unblock others as they're making contributions to the project but then still be there to have the history and full context and also provide a path forward of a good direction for Puma to head. In regards to encouraging more people to contribute to open source projects, I've often heard people say how challenging that is, where they have an open-source project that they would really love people to contribute to but finding people is really hard or just letting people know that they're interested in contributions. Have you had any strategies that have been successful for you in encouraging people to contribute? NATE: Yeah. So first thing, the easiest thing is we have a contributing.md file. So that's something I think more projects should adopt is have an actual file in your project that says everything about how to contribute. Like, what kinds of contributions do you want? Different projects have different things that they want. Like, Rails doesn't want to refactor PRs. Don't send a refactor PR to Rails because they'll reject it. Puma, I'm happy to accept those. So letting people know like, “Hey, here's how we work here. Here's the community we're creating, and here's how it works. Here's how to get involved.” And I think of it as hanging out the shingle and saying, “Yes, I want your contributions. Here's how to do it.” That alone puts you a step above other projects. The second thing I would say is you need to have contributor-only communication channels. So we have Matrix chat. So Matrix is like this successor to IRC. So we have a chat channel basically, but it's like contributors only. I don't enforce that, but I just don't want support requests in there. I don't want people coming in there and being like, “My Puma config doesn't work.” And instead, it's just for people that want to contribute to Puma, and that want to help out. If you have a question and come in there, anyone can answer it. And then finally, another thing that I've had success with is doing one-on-one stuff. So I will actually...I have a Calendly invite that I think is in contributing.md now that you can just book 30 minutes with me anytime about contributing to Puma. And I will get on a Zoom call with you and talk to you about what are your concerns? Where do I think you can help? And I give my time away that way. The way I see it is like if I do that 20 times and I create one super contributor to Puma, that is worth more than me spending 10 hours on Puma because that person can contribute 100, 200, 1,000 hours over their lifetime of contributing to Puma. So that's actually a much more higher leverage contribution, really from my perspective. It's actually helping other people contribute more. STEPH: Yeah, that's huge to offer people to say, “Hey, you can book time with me, and I will walk you through and let you know where you can start making an impactful contribution right away,” or “Here are some areas that I think you'd be interested, to begin with.” That seems like such a nice onboarding for someone who says, “I'm interested, but I'm nervous,” or “I'm just not sure about where to get started.” Also, I love your complaint department voice for the person who their Puma config doesn't work. That was delightful. [chuckles] NATE: I think it's a little bit part of my open-source philosophy that, especially at a certain scale like Puma is at that we really kind of over-prioritize users. And I'm not really here to do support; I'm here to make the project better. And users don't actually contribute to open source projects. Users use the thing, and that's great. That's the whole reason we're open-sourcing is so more people use it. But it's important not to prioritize that over people who want to make the project better. And I think a lot of times; people get caught up in this almost clout chasing of getting the most GitHub stars that they think they need and users they think they need. And you don't get paid for having users, and the product doesn't get any better either. So I don't prioritize users. I prioritize the quality of the project and getting contributors. And that will create a better project, which will then create more users. So I think it's easy to get sidetracked by people that ask for your time when they're not giving anything back to the project in return. And especially at Puma's scale, we have enough people that want my time or the time of other maintainers at Puma so that they can contribute to the project. And putting user support requests ahead of that is not good for the project. It's not the biggest, long-term value increase we could be making, so I don't prioritize them anymore. STEPH: Yep. That sounds like more the pursuit of sparkly stats and looking for all those GitHub stars or all of those likes. Well, Nate, if you're game, I have two listener questions that I'd like to run by you because I shared with some folks that you are going to be on The Bike Shed today. And they're very excited and have two questions that they'd like me to run by you. How does that sound? NATE: Yeah, all right. STEPH: So the first question is, are there any paradigms or trends in Rails that inherently hurt performance? NATE: Yeah. I get this question a lot, and I will preface it with saying that I'm the performance guy, and I'm not the software design guy. And I get a lot of questions about does such and such software design...how does that impact performance? And usually, there's like a way to do anything in a performant way. And I'm just here to help people to find the performant way and not to prescribe “You must always do X, Y, or Z,” or “ActiveRecord is bad. Never use it.” That's not my job here. And in my experience, there's a fast way to do almost anything. Now, one thing that I think is dying, I guess, or one approach or one common...I don't know what to call it. One common mistake that is clearly wrong is to not do any form of server-side rendering in a web application. So I am anti-client-side app. But there are ways to do that and to do it quickly. But rendering a basically blank document, which is what most of these applications will do when they're using Rails as a back-end…you'll serve this basically blank document or a document with maybe some Chrome in it. And then, the client-side app has to execute, compile JavaScript, make XHR requests, and then render the page. That is just by definition slower than serving somebody a server-side rendered page. Now, I am 100% agnostic on how you want to generate that server-side rendering. There are some people that are working on better ways to do that with Rails and client-side apps. Or you could just go the Hotwire Turbolinks way. And it's more progressive enhancement where the back-end is always just serving the server-side rendered response. And then you do some JavaScript on top of that. So I think five years from now, nobody will be doing this approach of serving blank documents and then booting client-side apps into that. Or at least it will be seen as outdated enough that you should never design a project that way anymore. It's one of those few things where it's like, yeah, just by definition, you're adding more steps into a rendering flow. That means, by necessity, it has to be slower. So I think everybody should be thinking about server-side rendering in their project. Again, I'm totally agnostic on how you want to implement that. With React, whatever front-end flavor of the month you want to go with, there's plenty of ways to do that, but I just think you have to be prioritizing that now. STEPH: All right. Well, I like that five-year projection of where we're headed. I have found that it's often the admin-side where people will still bring in a lot of JavaScript rendering, just to touch on a bit of what you're saying, in terms of let's favor the server-rendered HTML versus over-optimizing a space that one, probably isn't a profitable space in terms that we do want our admins to have a great experience for our product. But if they are not necessarily our users, then it also doesn't need to be anything that is over the top or fancy or probably uses a lot of JavaScript. And instead, we can start simple. And there's a number of times that I've been on projects where we have often walked the admin back to be more server-rendered because we got to a point where someone was very excited to make the admin very splashy and quick but then couldn't keep up with the requests because then they were having to prioritize the user experience first. So it was almost like optimizing the admin, but then it got left out in the cold. So then it's just sort of this poor experience. NATE: Yes. Shopify famously walked back their admin from I think it was Backbone to Turbolinks. And I think that that has now moved back to React is my understanding. But Shopify is a huge company, so they have plenty of time and resources to be able to do that. But I just remember that happening at the time where I was like, oh wow, they just rolled the whole thing back to Turbolinks again. And now, with the consolidation that's gone on in the React world, it's a little bit easier to pipe a server-side rendering into a React app. Whereas with Backbone, it was like no one knew what you were doing. So there was less knowledge about how to server-side render this stuff. Now it doesn't seem to be so much of a problem. But yeah, I mean, Rails is really good at CRUD apps, and admin is like 99% CRUD. And adhering as closely as possible to the Rails Golden Path there in an admin seems to be the most productive way to work on that kind of feature. STEPH: All right. Ready for your second question? NATE: Yes. STEPH: Okay. This one's a bit more in-depth. They also mentioned a particular project name. So I am going to swap it out with a different name. So on project cinnamon roll, we found a really gnarly time-consuming API endpoint that's getting hammered. And on a first pass, we addressed a couple of N+1 issues and tuned the performance, and felt pretty confident that they had addressed the issue. But it was still fairly slow. So then they took some additional incremental steps. So they swapped out to use OJ for serialization that shaved off an additional 10% but was still slow. They also went the route of going straight to Rails cache with a one-minute expiration. So that way, they could avoid mucking with cache busting because they confirmed with the client that data could be slightly stale. And this was great. It worked out well. So it dropped their average response time down to less than 70 milliseconds. With all that said, that journey took a few hours over a few days, and multiple production deploys. And had they gone straight to the cache, then they would have had a 15-minute fix with a single deploy. So this person's wondering, are there any other examples like that where, rather than taking these incremental seemingly obvious performance whims, there are situations where you want to be much more direct with your path? NATE: I guess I'd say that profiling can help you to understand and form better hypotheses about what will make things faster and what won't. Because a profiler can't really lie to you about where time goes, either you spent 20% of your time in this method, or you didn't. So I don't spend any time in any of my material talking about what JSON serializer you use. Because really that's actually never...that's really never anybody's bottleneck. It's never a huge proportion of people's total percentage of time. And I know that because I've looked at enough profiles that the issues are usually in other places. So I would say that if your hypotheses that you're generating are not working, it's because you're not generating good enough hypotheses. And profiling is the place to do that. So having profilers running in production is probably the biggest level-upscale-like that most teams could take. So having profilers that you can access as on production servers as a user is probably the biggest level up that you could make to generating the hypotheses because that'll have real production data, real production servers, real production environment. And it's pretty common now that pretty much every team that I work with either has that already, or we work on implementing it. It's something that I've seen in production at GitHub and Shopify. You can do it yourself with rack-mini-profiler. It's all about setting up the authorization, just making sure that only authorized users get to see every single SQL query generated in the flame graph and all that. But other than that, there's no reason you shouldn't do it. So I would say that if you're not generating the right hypotheses or you don't...if the last hypothesis out of 10 is the one that works, you need better hypotheses, and the best way to do that is better profiling. STEPH: Okay, better profiling. And yeah, it sounds like there's also a bit of experience in there in terms of things that you're used to seeing, that you've noticed that could be outliers in terms of that they're not necessarily the thing that you want to improve. Like you mentioned spending time on how you're serializing your JSON is not somewhere that you would look. But then there are other areas that you've gained experience that you know would be likely more beneficial to then focus on to form that hypothesis. NATE: Yeah, that's a long way of saying experience pays off. I've had six years of doing this every single day. So I'm going to be pretty good at...that's what I get paid for. [laughs] So if I wasn't very good at that, I probably wouldn't be making any money at it. STEPH: [laughs] All right. Well, thanks, Nate, so much for coming on the show today and talking so much about performance. On that note, I think it's a good place for us to wrap up. If people are interested in following along with what you're working on and they want to keep up with your latest and greatest workshops that are coming out, where can they find you on the internet? NATE: speedshop.co is my site. @nateberkopec on Twitter. And speedshop.co has a link to my newsletter, which is where I'm actively thinking every week and publishing stuff too. So if you want to get the drip of news and thoughts, that's probably the best place to go. STEPH: Perfect. All right. Well, thank you so much. NATE: No problem. STEPH: The show notes for this episode can be found at bikeshed.fm. CHRIS: This show is produced and edited by Mandy Moore. STEPH: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or a review in iTunes as it helps other people find the show. CHRIS: If you have any feedback for this or any of our other episodes, you can reach us @bikeshed on Twitter. And I'm @christoomey. STEPH: And I'm @SViccari. CHRIS: Or you can email us at hosts@bikeshed.fm. STEPH: Thanks so much for listening to The Bike Shed, and we'll see you next week. Together: Bye. Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

google pr speaking zoom practice writing performance turning matrix memory discord react user shopify users api gdpr chrome contrast github oj puma apis javascript eureka html rails wiki gusto versus backbone complete guide css speedy hello world sql calendly prs drm kanban mandy moore irc sentry json ruby on rails jira cli datadog hacker news new relic crud hotwire core web vitals ec2 devtools cgrp so google speedshop railsconf activerecord json api sidekiq bugsnag google's core web vitals xhr chris toomey in rails

23 выпуск 09 сезона. Vue.js 3.1.0, Vagrant 3.0, Mocha.js 9.0.0, Sidekiq good practices, Rux, Astro, React 18, FlexSearch и прочее

jobs beginners programming coding rails refactoring screencasts sidekiq

Play Episode Listen Later Jun 13, 2021 47:20

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Rails 7 ensures has_one autosave association callbacks get called once Moving ActionCable over to Webpacker Conditional HTTP GET: The fastest requests need no response body Sidekiq good practices Review: Ruby Installers and Ruby Switchers 10 Years of Open Source Rux - a JSX-inspired way to write HTML tags in your Ruby code Toward Vagrant 3.0 Web The Plan for React 18 React 17 runs useEffect cleanup functions asynchronously Vue.js 3.1.0 (Pluto) Released Managing Shared State In Vue 3 Introducing the Memory Inspector Introducing Astro: Ship Less JavaScript Mocha.js 9.0.0 Released FlexSearch - next-generation full text search library for Browser and Node.js Lowdefy - an open-source low-code framework that lets you build web apps with YAML or JSON configuration files RWpod Cafe 23 (03.07.2021) Сбор и голосование за темы новостей

practices react astro html browsers node vue mocha json vagrant yaml vuejs jsx sidekiq ruby rails

Rails for Beginners Part 36: Background Jobs with Sidekiq

GoRails Screencasts

Play Episode Listen Later Jan 27, 2021 6:20

Positioning Open Source Projects with Sam Selikoff

The Business of Open Source

Play Episode Listen Later Nov 25, 2020 38:31

This conversation covers: Mirage's role as an API mocking library, the value that it offers for developers, and who can benefit from using it. How Mirage empowers front end developers to create production-ready UIs as quickly as possible. How Mirage evolved into an API mocking library How Mirage differs from JSON Server Sam's relationship to Mirage, and how it fits in with his business. Sam also talks about open source business models, and whether Mirage could work as a SaaS offering. One interesting use case for Mirage, which involves demoing software and driving sales. Links Mirage Sam's teaching site Follow Sam on Twitter Subscribe to Sam's YouTube Channel TranscriptEmily: Hi everyone. I'm Emily Omier, your host, and my day job is helping companies position themselves in the cloud-native ecosystem so that their product's value is obvious to end-users. I started this podcast because organizations embark on the cloud naive journey for business reasons, but in general, the industry doesn't talk about them. Instead, we talk a lot about technical reasons. I'm hoping that with this podcast, we focus more on the business goals and business motivations that lead organizations to adopt cloud-native and Kubernetes. I hope you'll join me.Emily: Welcome to the Business of Cloud Native. My name is Emily, I'm your host, and today I'm chatting with Sam Selikoff. Thank you so much for joining us, Sam.Sam: Thanks for having me.Emily: Yeah. So, today, we're going to do something a little bit different, and we're going to talk about positioning for open source projects. A lot of people talk about positioning for companies, which is also really important. And they don't always think about how positioning is important for open source. Open source maintainers often don't like to talk about marketing because you're not selling anything. But you are asking people to give you their time which, at least for some people, is actually more valuable than their money. And that means you have to make a compelling case for why it's worth it to contribute to your project, and also why they should use it, why they should care about it? So, anyway, we're going to talk with Sam, about Mirage. But first, I should let you introduce yourself. Sam, thank you so much for joining me, and can you introduce yourself a little bit?Sam: Sure. My name is Sam Selikoff. These days, I spend most of my time teaching people how to code in the form of videos on my YouTube channel, and my website, embermap.com. Most of it is front end web development focused. So, we focus on JavaScript. I have a business partner who also works with me. And then we also do custom app development, you know, some consulting throughout the year.Emily: Cool. And then tell me a little bit about Mirage.Sam: Yeah, so Mirage is the biggest open source project I've been a part of since falling into web development, I'd say about eight years ago, I got into open source pretty early on in programming, kind of what made me fall in love with web development and JavaScript. So, I was starting to help out and just get involved with existing projects and things that I was using. Eventually, I made my way to TED Talks, the conference company where I was a front end developer, and that's actually where I met my business partner, Ryan. And we were using Ember.js, which is a JavaScript framework, and we had lots of different apps at TED that were helping with various parts of publishing talks, and running conferences, and all that stuff. And we were seeing some common setup code that we were using across all these apps to help us test them, and that's where Mirage came from. There was another project called Pretender, which helped you mock out servers so that you could test your front end against different server states. And we first wrapped that with something called Pretenderify, and then it grew in complexity. So, I was working on it on my learning Wednesdays, renamed it to Mirage, and then I've been working on it basically ever since. And then, the other big step, I guess, in the history is that originally was an Ember only project, and then last year, we worked on generalizing it so that it can be used by React developers, React Native developers, Vue developers, so now it's just a general-purpose JavaScript API mocking library.Emily: So, we would say that the position is an API mocking library. And—does that sound right?Sam: Yeah. If I had to say what it is, I would say it's a mocking library that helps front end developers mock out backend API's so that they can develop and test the user interfaces without having to rely on back end services.Emily: Why does that matter?Sam: It matters because back end services can be very complicated, there can be multiple back end services that need to run in order to support a UI, and if you're a front end developer, and you just want to make a change and see what the shopping cart looks like when it's empty. What does the shopping cart look like when there's one item? What does it look like when there's 100 items, and we have to have multiple pages? All three of those states correspond to different data in some back end service, usually in a database. And so, for a front end developer, or anyone working on the user interface, really, it can be time-consuming and complex to put that actual server in that state that they need to help them develop the UI. That can involve anything from running, like, a Rails server on their computer to getting other API's that other teams manage into the state they need to develop the UI. So, Mirage lets them mock that out and basically have a fake server that they control and they can put into any state they need. So, it's like a simplified version of back end services that the front end developer can control to help them develop and test the UI.Emily: And when you first started Mirage, did you think of it as an API mocking library?Sam: Not exactly. We used it mostly because of testing. So, in a test, it's usually a best practice to not have your test rely on an actual network. You want to be able to run your test suite of your user interface anywhere, let's say on an airplane or something like that. So, if your user interface relies on live back end services, that's usually where you would bring in a mocking library. And then you would say, okay, when the user visits amazon.com/cart, normally, it would go try to fetch the items in your cart from a real server, but in the test, we're going to say, “Oh, when my app does that, let's just respond with zero items. And then in this next test, when my app does that, let's respond with three items.” So, that's the motivation originally, is in a testing environment, giving the UI developer control over that. And then what happened was that it was so useful, we started using it in development as well, just to help during normal times, just because it was faster than working with the real back end services.Emily: Do you think there are any other projects that do something similar?Sam: Yeah, for sure. I think the most popular one is called JSON Server, which is a popular open source library that lets a front end developer put some data in a file, and then you point JSON Server to it, and then it just gives you an instant mock server you can use to help develop your app.Emily: So, what's the difference between Mirage and JSON Server?Sam: The difference is that JSON Server is made—it's really optimized for giving you a development—kind of a fake server as fast as possible, but it comes with a certain format that it gives you the data in. So, what ends up happening is that it can help you get feedback and build your UI faster, but eventually, you're going to need to point your app at a real API server, whatever you planning on using in production. And so the way JSON Server works might not correspond—in fact, often doesn't correspond with your actual API. So, Mirage fills that gap because Mirage is designed to be able to faithfully reproduce any production API; there's ways to customize how the data comes back so that it matches so that as you're developing your actual user interface against Mirage, you can have confidence that it'll work once you switch over to production.Emily: Is Mirage slower than other options?Sam: Not performance-wise because they're all JavaScript code that runs in the browser, but JSON Server is really optimized for just getting started as fast as possible because it comes with all of those pre-baked conventions about how the data is going to be moving back and forth. So, with Mirage—it can be faster, it depends—but with Mirage, you need to learn a little bit more in order to understand how to faithfully reproduce your production API. But I think it's faster because in the long run, if you're writing code against a mock server that doesn't match the interface of your production API, then you're just going to be having to change that application code that you're writing.Emily: How much do you talk to other people in the Mirage community, and talk about how they're actually using it?Sam: I felt more in touch with the users when it was an Ember project only because Ember is a more niche-type community. Whereas now, there's folks using it in React and Vue, like I was saying, and Angular. And so, we get issues almost every day on the project. It's not like a mega-popular project, but it does have enough people using it that people will ask questions, or open an issue almost every single day. And so I try to stay in touch with the users through that, basically. And then when I went to conferences—you know, before 2020—I would love to talk with people about it, or people would just bring it up. That's kind of how a lot of people know me on the internet. So, I would say that I do it in kind of a passive way. I haven't actively gone out to talk to them, partly because it's an open source project so it doesn't contribute to our revenue. We have some ideas for how that might happen one day, but as of right now, we can't justify doing proper product development on it in the sense of spending time doing customer interviews and stuff because it's a free project right now.Emily: That is my next question, which is, how does it fit in with your business in the larger sense? You know, how you put food on the table? And what are your goals for the project?Sam: A good question. I mean, it's really aligned with our overall mission of, of everything that we do because me and my business partner, Ryan, we really want to just help people get better at UI development, we want it to be easier, we want to help empower more people to do it because we think it's powerful tool in the world, and it's just too hard right now; there's so many things that are hard about it. So, that goes back to our consulting and our teaching; mainly our teaching. That's our main mission. And then Mirage is really—the purpose of Mirage is to enable front end developers to do more kind of with less, so they don't have to run a Docker container or get an SQL database up just to change some CSS for a given server state. So, it fits into that mission. I've been doing open source long enough to know the pattern, and it happens over and over again, where people work on something, it gets popular enough that they start opening issues, and it becomes a maintenance burden for the maintainers, and then they try to stay up late closing issues, they get burned out, and then the project kind of rots. So, that's something that happens a lot in open source. And so, over the last five years or so of working on Mirage, I've been more involved, or sometimes step back if I need to spend more time on other things, but I'm really interested in making it sustainable, and I know some people in the open source community who have made their projects sustainable financially, other with a pro plan, or support plan, or different related services. So, if we could snap our fingers, that would be what would happen, and that's what we're working on now.Emily: Which one of those open source business models do you think is most appropriate?Sam: At this point, we basically are trying to not guess that answer because we think the way to find out which one it is, is to get a critical mass of users and listen to what they're saying. So, on our podcast, we interviewed Mike Perham from Sidekick, who runs Sidekiq and Sidekiq Pro in the Rails community for about 10 years, and he makes really good money. Sidekiq Pro came about because the enterprise customers of Sidekiq were asking for more robust job servers and all this kind of stuff, so there was a natural path for him to making a pro version that he could sell to enterprise clients. And so that's worked out really well for him, and the rest of the community gets to use the base version for free. And I love that because I do love this zero-cost to entry to open source. And then my other friend, Adam Wathan who works on Tailwind, he's made money to help Tailwind be sustainable through education and courses, and then more recently, a project called Tailwind UI, which is pre-built UI components with Tailwind. And that, again, came about from people asking for that after he was working on Tailwind. So, I think the best way to do it is to get that critical mass of users to the point where you know what they're asking for, you hear over and over again, and then it makes sense to go forward with that.Emily: There's also a third option, which is a cloud service, and I'm just curious—like a complete SaaS offering—have you ever considered that?Sam: Absolutely. There's a really cool opportunity there for Mirage because your Mirage server usually lives alongside your front end code so that when every individual front end developer pulls the project down, starts working on it locally, they're running their own Mirage server. But some of the things that people have done organically with Mirage is, create a certain configuration of a server state—let's say for a demo—so let's say they create a shopping cart, or let's say they're working on a financial piece of software, and they need to show what it looks like with three clients and four contracts, and here are the products that we sold and how much money they are. People will make up a Mirage scenario with that specific set of data that their salespeople take and use on sales calls. And that way, again, the user interface looks fully realistic, it's a fully working UI that is talking to Mirage, but now you don't have to worry about the actual back end servers going down or anything like that. And so, we've had this thought of a hosted Mirage, basically like a Mirage cloud, where even non-technical people could tweak the data there, and then again, they don't have to get involved with ops people or anything like that. And that could be really powerful. So, there's a ton of ideas there. I think our hesitation with our company, Embermap, it's worked to some extent, but it's not grown to the point where it can sustain us, and so that's partly because of the market issue, Ember being a little smaller. So, we're nervous to jump into any particular solution before we feel there's a proven market need for it. And so that's why, even though we have a lot of these fun ideas that I think could really work out, we first want to wait until we get that critical mass, the audience size that we'd feel comfortable could sustain a business.Emily: How many people is that? What is the audience size?Sam: That's a tough question. Let's say Mirage cloud was the goal. I think instead of waiting to a certain point, and then trying to build Mirage cloud, we would do a few things in the interim that would be little experiments and lower risk. So, I think the first thing would be, let's say, a Mirage course. And if we can sell a course on Mirage—or even we make a free Mirage course—that gets enough attention, that would tell us that there is enough of a need there, that the positioning resonates, that people are seeing it's a valuable thing, and that would tell us, okay, let's try the next thing. So, we've been trying to do that. I make some YouTube videos over this last year, and I've been tweeting more about it and the work we've been doing, and so all of those are little experiments that I'm trying to pay attention to what resonates. Again, we have users who really love Mirage, and it's changed their workflow, but it's not at the point where I feel like it's a slam dunk and it makes sense to go on the next phase. So, I think there's been some frictions with Mirage, as we've brought it out to the wider JavaScript community. So, we have some things we want to tweak, and then ship a 1.0, and then I think maybe a 10 video course, would be a good next step, and just see the response to that, and basically take it from there.Emily: Yeah. It's always interesting to think like, how will that you have reached the critical mass?Sam: Yeah, I mean—Emily: Hard. Sam: It's really hard. And there's other strategies. I mean, a lot of businesses just have an idea and go try it out. There's this book I read, called Nail It and Scale It, and they talked about finding a problem. And we've have had some users of Mirage who use it at big, bigger companies, and it's really a big part of their workflow. And we've thought about, “Hey, what if we were to build something for them that we could generalize?” They actually have a need for something like a Mirage cloud because every time they develop a feature, they have to sit down with their product people, and their front end developer will basically run through all these different Mirage scenarios to show them how the feature works in every case, and they would love to be able to just send a link to a hosted version where they could do that. So, we've thought about building that kind of thing. But again, it's just a big risk. And then the market risk is there. So, what I've seen, I feel like, working in the past few years with a small circle of small business people and open source that I hang out with is this audience-first approach. So, it's like, if you're delivering a lot of value in the form of open source, and education, and talks, and you build an audience that believes what you have to say, and likes your opinion on things, likes your point of view, and again, resonates with the value of your work, then it becomes more and more obvious. You have a lot of people giving you feedback, you start seeing the same thing over and over. And that's just what we do with developing Mirage itself. We know a lot of what we work on next is driven by the same issues that come up, the same problems that come up in the issues, over and over again. So, I think it is hard to know, but it's one of those lukewarm things. And basically, right now, it feels too early. You know?Emily: And do you feel like, when you say to somebody, let's say, somebody who isn't involved with Mirage at the moment, maybe you're at a developer meetup or something, and you say, “I created Mirage. It's an API mocking library.” Do you have an aha moment? Are they like, “Oh, yeah. I know what that is. I know why you would use that.”Sam: Mm-hm. Sometimes yes, and sometimes no. There is a form of position I feel like, sometimes, really resonates well with people, which is like, “Don't get blocked by your back end developers.” So, if you're a front end developer, and the API is not ready, you can still build your app, including all the dynamic parts of it which would normally require a real server to be running. So, you're the front end developer, and you're trying to wire up the interface, and what happens when you click save on a cart, and it saves it in the back end, and then you show a success message. But your API is not ready, and you're working with another team that's working on the API, so you're kind of frustrated because you're stuck and you feel like you can't do anything, well, that's where Mirage can come in because you don't have to wait on them at all. You can just build it out yourself with Mirage. It's much simpler because it abstracts away all the complexity of a real server enough that you can actually build your UI against this kind of faithful reproduction of the back end. So, people really like that. And then people really like the testing use cases as well. They get confused about the best way to test user interfaces in this new distributed world we're living in where you're maybe working on a React app, and the back end is a separate API that's also serving up iPhone clients and things like that. So, there's really multiple apps involved with running a website like amazon.com. So, the question is, how do you test the front end? And Mirage is a great answer for that as well. And that's where it originally came from, so.Emily: Yeah. I can definitely see the positioning is sort of like a way for front end developers to decrease their dependence on their back end colleagues.Sam: Yeah, exactly. It's hard because there's a lot that it helps you with. And the people who use Mirage the most and have used it the most, it's really transformed their workflow because it lets the front end developer just move so much faster. So, it's really just, like, the fastest way to build a front end. That's really what the whole point of—every change we make to the library, every decision that went into it, it's how to empower a front end developer to build a production-ready user interface as fast as possible. And that includes accounting for all these different server states that are usually a pain in the butt to get.Emily: I'm just taking notes. I mean, that that actually sounded really powerful. Like, “Mirage lets front end developers work as fast as possible,” basically. It's fairly high level, but ultimately, that is a pretty compelling value statement.Sam: Yeah. And I believe it to be true, too. And I mean, that's how—I mean, I've been working on apps recently, and I still use Mirage because it's just faster than using a real server because it's right there in your code. It's right alongside your front end code, so you don't have to switch over to another process running, or open up a browser and see it; it's all right there. If you want to switch from being an admin to being a user, it's like you just uncomment a line alongside the code that you're already writing. So, I truly believe it is the best way to build a user interface. I think the positioning question is interesting because that is high-level. Sometimes developers in open source, they just want to know what it is, so there are some people who would see, “Just tell me what it is.” “It's an API mocking library.” “Okay, got it.” And that's what they want to know. “And what makes it different from other API mocking libraries?” Okay, we can talk about that. But then there's other people I think, who could stand to benefit from it, and if they saw, “Mirage is an API mocking library,” they're going to be like, “I don't really need that.” But then they go back to their job, and they have to work on their app, and they find themselves spinning up a Docker container, going to auth0 to sign in just so they can run their app in an authenticated state, and it's like, Mirage would actually be perfect for this. You could just mock all that stuff out in Mirage once, all your front end developers could use it. And now anytime you want to just work on your app in an auth state, it's just right there in Mirage, you don't have to deal with any of the complexities of the production services.Emily: Yeah. I mean, I also like the idea of sort of the fastest way to build a front end app.Sam: Yeah.Emily: Or to build a UI. The fastest way to build a UI.Sam: Yeah. That is, that's nice. I mean, it's interesting. It's a pretty interesting thing to say, and it's pretty compelling, and it's like, if you can back it up, that's a pretty strong claim.Emily: Yeah. And I mean, one of the things that I also talk to clients about—so I work with all technical founders, usually, and we're often really focused on features, all the cool features, but—you don't have to ignore the features, but you sort of use them to prove that the thing that you're talking about, that the value you say you provide is not BS. But you still want to connect the dots. Like you were saying, sometimes even the most technical person is like, “Oh, a mocking library. But I don't need that.” They're not going to necessarily connect the dots that, like, “Oh, that's going to make my workflow way simpler.” Sam: Right.Emily: Does everybody know what a mocking library is? Like, everybody who needs one?Sam: Mm, depends. If you're a front end developer. I mean, these days, people are becoming more and more specialized, so there's some front end developers who don't even deal with the data fetching side of an app at all. They're working on a part of the app that already has the data, and they're just working on maybe styling, layout, things like that, but they're never writing code or refactoring code that actually interacts with the server. And then even if you are doing that, you might not be writing tests. So, a lot of people just do the lowest friction thing, which is, just point their local UI at whatever server they can. Maybe their company has a staging server or maybe they run one locally in development and they just build it like that. But again, that's the lowest friction, but it's slow because now you're running a Rails server. If you want to change the data, you have to know how to do it in Rails, and you have to run different commands. And then it comes to testing, it's really hard, too. So, I think most people would know, if you were to say, “Yeah, it mocks out the API.” Most people would know that. But people might have different conceptions of what that means. So, there are some approaches to mocking—or stubbing, or faking, there's some technical difference between those terms, but for the purpose of this conversation, it's just faking out that functionality—there's some people who would hear that and say, “Oh, okay, I get it. A function that my code calls when it normally goes to the server, I'm going to replace it with a different function that returns this fake data.” But Mirage works differently because it operates at the boundary of the app. So, when you mock your API with Mirage, you don't have to change anything about your application because it intercepts the network request before it goes to the server. So, that's another big benefit of Mirage, that Mirage has over other solutions for mocking out network functionality is that your application code stays exactly the same. And that's an important point because the whole goal with Mirage is to be the fastest way to build a production-ready user interface. And by production-ready, I mean an interface that can be plugged into your production API and you'll have confidence that it works. So, that boundary thing is another important point of Mirage. So, that would maybe be the one thing that people could mean different things when they say ‘mocking.' But by and large, I would say most front end developers would understand if you said ‘API mocking,' what they mean.Emily: And do you think they generally are also able to figure out what that's used for, why they should care?Sam: Yeah, that's the thing is that that's where I think the real opportunity is because I think, if you were to say ‘API mocking,' people are going to immediately go to testing because that's the kind of environment where you would want to mock the API so you have control over it. But people who haven't used Mirage or something like Mirage, don't realize how powerful it can be, to mock it just during your normal development flow. So, again, once people use it and get it, they use Mirage for everything; it's just part of their workflow. I just start up my UI, I'm running Mirage in a specific scenario, and if I need to see the UI in a different state, I just changed my Mirage scenario, or put some new data in there, or empty out the database there. And so it totally affects your whole workflow because now you're just disconnected from the back end services. So, I think a lot of people aren't doing that. They are just stuck—you know, they're just used to the hassle of getting these three services up and running just so that they can run their UI, and it's a pain in the butt. I mean, I was just talking to someone on another podcast, and he was saying it's the same thing. And it's really hard when they onboard new people because they have to get them set up with all these auth keys just so they can run the front end, even though they don't really care about the auth setup, they just want to start working on this page. So, again, the companies that have used Mirage and adopted it don't have any of those problems because the Mirage server is right there with the user interface, and they don't have to worry about it. So, I think that is a big gap that we could probably do a lot better job of closing with information on the homepage, or examples, or something like that.Emily: So, Mirage also helps new hires get up to speed a lot faster, or get productive, I should say?Sam: So, yeah. If you were hired by Facebook, and you're a front end developer, and your first task is to update the way the colors look on the home feed, to get that running on your computer so you can make changes and see how it looks in the code, at many companies—probably most companies—it's going to involve a lot of moving pieces because you have this React app on the front end, but it has to fetch data from somewhere so you can see how it works. Maybe they have a server that is just used for development that the person can point to. But again, if you have this shared hosted server, you only get what's on there; maybe you don't have an easy way to change what data it gives you. So, usually, front end developers need to see a dynamic user interface in all these different states. But if you're using a shared server, you don't get all those different states. But it's easier to set up because it's already set up. So, the alternative is to say, “All right, Sam, you're new here. Let's get you set up so that you can run a copy of Facebook locally.” So, that usually involves a ton of steps. I mean, I've seen that take, like, a week just to get all the pieces that are required to run the back end up so that you can actually power your front end. So, yeah, companies definitely, definitely have a problem with this. They have a problem with shared staging servers, I've talked to tons of people over the years who hate their staging servers, the back end teams, the ops teams hate them because everyone's always trying to change it for the salespeople to go take demo calls or for people to test. And so basically a shared server like that is just a nightmare because it's basically another production server to maintain that your internal team is trying to use and people want it to do different things. So, that's why Mirage is nice because it's just local to the code, and every front end developer can just tweak it in exactly the way they need for what they're doing at that time. So, I think that's a big opportunity as well.Emily: One of the most interesting things, I think, about this conversation is that you've touched on this idea of salespeople doing demos, and I really like that because it's so different. It's such a different use case.Sam: Yeah. We had a thought about that, actually because we were talking to a handful of companies—did interviews with them, actually, a couple years ago—and we're like, this could be a cool product. And it's like, just the easiest way to show off your software. And again, we've even done consulting for companies that have had basically another server, just for the purposes of a demo. And again, it's just a pain in the butt because it's another production server to manage, you have to reset the database after each call you do, and with Mirage is not like that; it just restarts with the app. It's just heavier, it's a real server to manage, where again if you just need to show off the UI, you can do it with Mirage. So, we thought about doing that. It's an interesting use case, for sure.Emily: I think it's a really good illustration of how you can take the same technology, and reposition it, basically, to do something totally different, have a totally different set of value proposition. The people who would be actually benefiting from its use would be totally different. I mean, you have salespeople versus front end developers.Sam: Yep. We've thought about that. What would that look like to market to those people? And maybe one way you could do it would be, you know, the salespeople get frustrated because they want to just change the numbers in this part of the app on the spreadsheet summary because they know it's going to help them communicate the value to the people that they're talking to, but now they have to go and ask some back end developer or someone on the ops team, “Hey, can you change this database column so that my demo works better?” What if they had a Mirage-powered demo and they could tweak the Mirage data, maybe in some interface themselves, without having to involve anybody. And that's pretty compelling because Mirage, again, is designed to work with any API. So, no matter what your tech stack is on the back end, Mirage works for the front end team. And so now the back end team can do all their stuff, they can be switching from Rails to Elixir, they can be switching to Go or microservice architecture, and they have all this stuff going on, so they're the only ones who really know how to actually get this number from 100 to 200 in the system. Like, it's complicated. And so that's, again, a benefit of just having Mirage because, on the sales calls, the people are just wanting to show what it looks like when the data is a certain way. So, yeah, that was definitely a use case that we didn't anticipate at all.Emily: Yeah. And even you could have five salespeople trying to do calls at the same time and—Sam: Exactly.Emily: Wanting to have different data reflected, and I'm sure that's just like—Sam: A nightmare. I mean, people have told us just, it's the worst. Basically, the shared staging server is a—yeah, it's a huge problem for a lot of teams.Emily: It's so interesting. Well, anyway, taking us back from the rabbit hole, although it—I mean, it really is interesting, and just fascinating how you could change this to be something that's targeted at that totally different market.Sam: Right.Emily: To wrap up. I mean, I was thinking about how you're saying that the fastest way to build a production-ready UI, that possibly is the most powerful thing I think you've said.Sam: Yeah. We actually—I think that's how we had it when we first did the redesign of the site. And it was like, I don't know if that stuck in the sense of, what does production-ready mean? “Oh, I'm already writing production-ready code.” But then it's like, you have to—like you said, connect the dots and explain how you're not writing production-ready code unless the way you're mocking your API is matching your production API, so we kind of switched it to what it says now, which is, “Build complete front end features, even if your API doesn't exist.” Which basically came from the mouth of someone who was using it, and when they had their aha moment that was what they were saying. I've been thinking if we get to the 1.0 launch, I think I would want to go back to something like that because I do think it's compelling. So, “It's the fastest way to develop a UI, test it, and then share a working demo of it,” because that's is really the motivation for the library. So, yeah, it might be interesting to think about doubling down on that as the unique value proposition.Emily: Yeah. I'm curious what your immediate plans are.Sam: So, we've been working on this REPL playground area of the site where people can learn Mirage and see examples, and we can share them and stuff. So, that will hopefully—you'll see a lot of that kind of thing in the JavaScript community. If you go to Svelte, which is another front end framework, you can create little sandboxes and play around with Svelte and learn it. So, it's a really good way to learn things and just see how it works quickly right in the browser without having to install anything. So, we're about wrapping that up. And then I think it'll just be a matter of carving out the time to go through some of the issues and bugs that people have found, and getting it to a point where we feel good about slapping a 1.0 on it. At which point, we can take a look at all the feature requests and all the issues that people have run into and figure out okay, where do we want to focus our time? Again, considering, like, is there a story here where we can make it sustainable, and hopefully, dedicate more of our time to it? Because that's really, again, I think it's the biggest impact work I've done in my career so far, but it's just, sustainability in open source is tough. So, it's a matter of not jumping too early on any particular idea for how to sustain it and monetize it: if it's a pro version, if it's a support plan, people have asked for all these things, but not maybe in the strongest numbers that would make me feel comfortable diving in on that. But I do believe that it can work because I believe it's a really good idea and I know a lot of companies have gotten a lot of value from it. So, that's what our short term plan is.Emily: Well, fabulous. Thanks so much for talking about Mirage. This has been really interesting.Sam: Yeah, thanks a lot for having me, and I appreciate your input. It was fun to revisit the positioning stuff. It's been a while, so it's always good to be thinking about that.Emily: All right. I have one last question for you, which is what is an engineering tool you can't live without?Sam: These days, it's got to be Tailwind. It's a styling framework. And it's just really excellent. So, I would not want to work on a site without it because it would just be painful. So, [laugh] I love Tailwind. Yeah, it's a really good library.Emily: All right, cool. Well, thank you so much. And—oh, what—the very last thing it should be, how can listeners find you, follow you, keep up with you?Sam: Yeah, best place is Twitter, at @samselikoff. And I'm also making YouTube videos more and more these days: youtube.com/samselikoff. So, those would be the places to go.Emily: Right. Excellent.Emily: Thanks for listening. I hope you've learned just a little bit more about The Business of Cloud Native. If you'd like to connect with me or learn more about my positioning services, look me up on LinkedIn: I'm Emily Omier—that's O-M-I-E-R—or visit my website which is emilyomier.com. Thank you, and until next time.Announcer: This has been a HumblePod production. Stay humble.

business open iphone fall in love bs ted talks saas react real world api positioning ui mirage javascript rails css elixir sql docker sidekick kubernetes pretender vue tailwinds angular cloud native react native uis open source projects samit svelte repl adam wathan sidekiq nail it javascript apis mike perham emily it tailwind ui scale it sam so sam selikoff emily well humblepod

36 выпуск 07 сезона. Sidekiq 6.0, TypeScript 3.6, String corruption in 2.6.4, Ferrum, Electron Fiddle, Trumbowyg и прочее

Play Episode Listen Later Sep 8, 2019 27:59

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby String corruption in 2.6.4, Welcome to Sidekiq 6.0, Ruby 2.7 adds warning for Proc.new and proc without block and error for lambda without block и Rails 6 adds filter_attributes on ActiveRecord::Base Ruby Pattern Matching и Ferrum - fearless Ruby Chrome driver Web Announcing TypeScript 3.6 и Introducing Feathers 4: A framework for real-time apps and REST APIs Electron Fiddle, Apexcharts.js - Modern & Interactive Open-source Charts, Trumbowyg - a lightweight WYSIWYG editor и Iosevka - a slender monospace sans-serif and slab-serif typeface

modern corruption charts string rails proc fiddle electron typescript wysiwyg ferrum sidekiq

31 выпуск 07 сезона. Rails 6.0.0 rc2, Electron 6.0.0, Sidekiq Batches, Lefthook, Minecraft clone, Cube.js, Esprint и прочее

Play Episode Listen Later Aug 4, 2019 44:59

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Rails 6.0.0 rc2 released, Rails 6 deprecates where.not working as NOR and will change to NAND in Rails 6.1 и Rails 6 adds support for Optimizer Hints Sidekiq Batches: Tips & Tricks, Don't change the signature of Sidekiq jobs running in production и 7 Ways to Selectively Run RSpec Tests Runbook: A Ruby DSL for Gradual System Automation, Lefthook: Knock your team's code back into shape и Artichoke is a Ruby implementation written in Rust Web Electron 6.0.0, Here's How Not to Suck at JavaScript и Do React Hooks Replace Redux? Detecting incognito mode in Chrome 76 with a timing attack, String.prototype.replace supports replacement patterns и Making a Realistic Glass Effect with SVG Open source Minecraft clone, Cube.js - Open Source Analytics Framework и Esprint - a fast eslint runner

tricks suck minecraft chrome cube string clone rails detecting electron artichoke nand batches sidekiq ruby rails

49 выпуск 06 сезона. Ruby 2.6.0-rc1, Truffleruby 1.0 RC10, Microsoft is building its own Chrome browser to replace Edge, Sharp, ForgJs и прочее

future microsoft passing released bridging sharp javascript browsers node ruby on rails chrome browser activerecord sidekiq ruby ruby postcss

Play Episode Listen Later Dec 9, 2018 36:48

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Ruby 2.6.0-rc1 Released, Truffleruby 1.0 RC10 и Big on Heroku: Scaling Fountain without losing a drop Passing current_user by default in Sidekiq, ActiveRecord on MySQL— Iterating over large tables with conditions и How to store emoji characters with Ruby on Rails & MySQL JavaScript Microsoft Edge: Making the web better through more open source collaboration и Bridging the Gap Between CSS and JavaScript: CSS Modules, PostCSS and the Future of CSS The fastest way to pinpoint frustrating user experiences, 6 JavaScript User Authentication Libraries for 2019, Sharp - high performance Node.js image processing и ForgJs - a JavaScript lightweight object validator

10 выпуск 06 сезона. Using Genetic Algorithms in Ruby, Standardizing lessons learned from AMP, AppBandit, Risk, Coördinator, Mutag и прочее

Play Episode Listen Later Mar 11, 2018 39:28

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Refactoring views with Ruby on Rails' ActiveSupport helpers и Ruby on Rails – your own slow query log, no sql configuration required Using Genetic Algorithms in Ruby, Grpc Tutorial With Ruby и Write your own scss-compiler RabbitMQ is more than a Sidekiq replacement и Encrypted Credentials in Rails 5.2 JavaScript Standardizing lessons learned from AMP, AMP is not the issue, it's Google, The Problem with Webpack and Why It Is (Kind of) Our Fault и Choosing Between Progressive Web Apps, React Native & NativeScript in 2018 How I organize CSS in large projects using UFOCSS - Part 1 и Get Started with ngUpgrade: Going from AngularJS to Angular AppBandit - Web Security Proxy in NodeJS and Electron, Driver - a light-weight, no-dependency, vanilla JavaScript engine to drive the user's focus across the page, Risk - an implementation of the popular board game Risk, Coördinator - turn an SVG into XY coördinates, Mutag - a simple MP3 file tag parser и Awaity.js - a functional, lightweight alternative to bluebird.js, built with async / await in mind

06 выпуск 06 сезона. Ruby 2.5 added Hash#slice method, Stimulus JS, Chiccocoin, Eloquent JavaScript и прочее

Play Episode Listen Later Feb 11, 2018 26:50

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Ruby 2.5 added Hash#slice method, How Fast is Ruby 2.5.0? и Playing with ruby's new JIT: MJIT Happy 6th Birthday, Sidekiq, Serving your own Commercial Rubygems и Stimulus JS Creating a simple GTK+ ToDo application with Ruby и Spying on a Ruby process's memory allocations with eBPF JavaScript The Top JavaScript Trends to Watch in 2018, Rollup now has code-splitting! And we need your help и React's ⚛️ new Context API JavaScript Date has too many traps и Five Common Problems in GraphQL Apps (And How to Fix Them) Chiccocoin: Learn what is a Blockchain by creating one in NodeJS, Web Assembly Studio и Eloquent JavaScript, 3rd edition

playing serving method blockchain react added stimulus slice javascript hash roll up nodejs eloquent sidekiq ruby ruby

29 выпуск 05 сезона. Monitor and Debug Bottlenecks in Sidekiq, Enclose.IO, RenderReact, Popmotion, GPU.js и прочее