Podcasts about crdts

Ben Holmes, product engineer at Warp, joins PodRocket to talk about local-first web apps and what it takes to run a database directly in the browser. He breaks down how moving data closer to the user can reduce latency, improve performance, and simplify frontend development. Learn about SQLite in the browser, syncing challenges, handling conflicts, and tools like WebAssembly, IndexedDB, and CRDTs. Plus, Ben shares insights from building his own SimpleSyncEngine and where local-first development is headed! Links https://bholmes.dev https://www.linkedin.com/in/bholmesdev https://www.youtube.com/@bholmesdev https://x.com/bholmesdev https://bsky.app/profile/bholmes.dev https://github.com/bholmesdev We want to hear from you! How did you find us? Did you see us on Twitter? In a newsletter? Or maybe we were recommended by a friend? Let us know by sending an email to our producer, Emily, at emily.kochanekketner@logrocket.com (mailto:emily.kochanekketner@logrocket.com), or tweet at us at PodRocketPod (https://twitter.com/PodRocketpod). Follow us. Get free stickers. Follow us on Apple Podcasts, fill out this form (https://podrocket.logrocket.com/get-podrocket-stickers), and we'll send you free PodRocket stickers! What does LogRocket do? LogRocket provides AI-first session replay and analytics that surfaces the UX and technical issues impacting user experiences. Start understand where your users are struggling by trying it for free at [LogRocket.com]. Try LogRocket for free today.(https://logrocket.com/signup/?pdr) Special Guest: Ben Holmes.

In today's episode, Charles and AJ are joined by Anselm Eickhoff, a trailblazing full-stack software engineer. Anselm delves into the world of "Local First" software development and the innovative tool he's developed, "Jazz." They explore how Jazz leverages schema information for type inference and runtime validation, enabling real-time collaboration and seamless offline functionality. They dive into the simplifications brought by Jazz's API changes, the flexibility offered through open protocols, and the local-first approach using CRDTs.They also tackle Jazz's impressive backend infrastructure and its shift to direct disk storage, along with insights into the challenges and emerging support structures faced by early adopters. Anselm shares real-world applications, advanced analytics capabilities, and the future outlook for Jazz, emphasizing performance, resilience, and data privacy.Plus, stay tuned for a lighthearted detour into board games and TV recommendations, as well as updates on Charles' AI Dev Boot Camp and exciting developments on the Top End Devs platform. This episode is packed with cutting-edge insights and practical advice for developers looking to integrate advanced real-time features and streamline their app development processes. SocialsLinkedIn: Anselm EickhoffPicksAJ - Dune 2-Film Collection (Blu-Ray + Digital)Charles - Imperial Miners | Board GameBecome a supporter of this podcast: https://www.spreaker.com/podcast/javascript-jabber--6102064/support.

Justin Tormey, co-founder of Castmagic, joins the Elixir Wizards to discuss building an AI-powered content creation app. Castmagic repurposes audio and video into social media posts, blog articles, newsletters, and more. The tech stack leverages OpenAI and Anthropic LLMs with Elixir as the coordination layer and Phoenix LiveView powering the front end. Justin dives into the technical details of Castmagic, including the integration of JavaScript libraries like ProseMirror and TipTap through LiveSvelte, as well as enabling real-time collaboration with CRDTs and YDoc. He shares the benefits and challenges of using Elixir for rapid development, as well as the trade-offs between custom code and off-the-shelf solutions. The conversation also covers Justin's entrepreneurial journey, highlighting the advantages of bootstrapping over venture capital, the importance of acquiring early customers, and creative marketing strategies like affiliate programs and software marketplaces. Whether you're an Elixirist exploring machine learning or an aspiring tech founder, tune in to learn more about AI with Elixir, navigating startup challenges, and turning ideas into impactful software. Topics discussed in this episode: Building Castmagic to generate content from audio and video for creators Using AI services like OpenAI and Anthropic for transcription and workflows Elixir as the coordination layer for complex processes Rapid UI development with Phoenix LiveView Integrating rich text editing libraries through LiveSvelte Enabling collaborative editing with CRDTs and YDoc Balancing offline functionality with cloud-based AI tools Challenges of working with external AI services Exploring the future of multimodal AI in product development Bootstrapping vs. venture capital: benefits and challenges Strategies for finding distribution channels and early adopters Creative approaches to marketing, including affiliates and marketplaces Balancing engineering efforts with customer and business needs Practical advice for navigating the early stages of a startup Links mentioned: https://www.castmagic.io/ https://pragprog.com/titles/jaerlang2/programming-erlang-2nd-edition/ https://www.blockchain.com/ https://fly.io/ https://hexdocs.pm/livesvelte/readme.html https://github.com/woutdp/livesvelte https://prosemirror.net/ https://tiptap.dev/ https://docs.yjs.dev/api/y.doc https://hexdocs.pm/polymorphicembed/readme.html https://github.com/elixir-nx/nx https://github.com/elixir-nx/tokenizers https://github.com/thmsmlr/instructorex https://openai.com/ https://www.anthropic.com/ https://getoban.pro/ https://github.com/wojtekmach/req https://ollama.com/ https://x.com/j_tormey Special Guest: Justin Tormey.

Nathan Sobo is co-founder of Zed, a super-fast, collaborative, AI-powered, code editor. We chat about his journey to build the ultimate code editor: lessons learned from building Atom, Electron and its challenges, CRDTs, Rust native GPU GUIs, AI Code Assistants, and more CRDTs. See also: ⁠Nathan on the Software Unscripted podcast⁠ Discuss this episode: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠discord.gg/XVKD2uPKyF⁠⁠

News includes Chris McCord's speedrun video on adding a self-hosted llama2-7b to an existing application, Tyler Young's release of parameterized_test v0.2.0, major updates in Oban Pro's new launch week, potential for CRDTs being added to Mnesia DB, Zach Daniel's blog post on Igniter for code generation, and a preview of ElixirConf 2024 with exciting speakers and topics, and more! Show Notes online - http://podcast.thinkingelixir.com/213 (http://podcast.thinkingelixir.com/213) Elixir Community News - https://x.com/chris_mccord/status/1815409966611648705 (https://x.com/chris_mccord/status/1815409966611648705?utm_source=thinkingelixir&utm_medium=shownotes) – Chris McCord does a YouTube video speedrun of adding a self-hosted llama2-7b to an existing application. He's running it against Ollama and making REST API calls to it, showing how to run the Ollama server on a private Fly.io IPv6 network using auto-stop and auto-start features. - https://x.com/TylerAYoung/status/1815391743484870980 (https://x.com/TylerAYoung/status/1815391743484870980?utm_source=thinkingelixir&utm_medium=shownotes) – Tyler Young shared a new release of his library parameterizedtest, version v0.2.0, which includes support for longer test names, comments in tables, and Obsidian markdown table format. - https://github.com/s3cur3/parameterized_test (https://github.com/s3cur3/parameterized_test?utm_source=thinkingelixir&utm_medium=shownotes) – GitHub repository for Tyler Young's parameterizedtest library that makes it easier to create tests using multiple combinations in markdown tables. - https://x.com/Exadra37/status/1815694986345611683 (https://x.com/Exadra37/status/1815694986345611683?utm_source=thinkingelixir&utm_medium=shownotes) – The Mnesia database may receive native support for automated conflict resolution via CRDTs, sponsored by ErlangSolutions and developed by Vincent Lau. - https://www.youtube.com/watch?v=tHdPRyMjmW8 (https://www.youtube.com/watch?v=tHdPRyMjmW8?utm_source=thinkingelixir&utm_medium=shownotes) – Vincent Lau spoke at Code BEAM Europe 2023 about his work on adding CRDTs to Mnesia for automated conflict resolution. - https://www.erlang.org/doc/apps/mnesia/mnesia.html (https://www.erlang.org/doc/apps/mnesia/mnesia.html?utm_source=thinkingelixir&utm_medium=shownotes) – Documentation on Mnesia, a distributed key-value DBMS built into Erlang. - https://x.com/sorentwo/status/1791166342034255938 (https://x.com/sorentwo/status/1791166342034255938?utm_source=thinkingelixir&utm_medium=shownotes) – Oban Pro's launch week introduces unified migrations, worker aliases, better support for distributed databases, faster unique job checks, and the @job decorator for small jobs. - https://x.com/sorentwo/status/1807155900609904973 (https://x.com/sorentwo/status/1807155900609904973?utm_source=thinkingelixir&utm_medium=shownotes) – Improvements in Oban Pro include better batch workflows with mermaid visualizations. - https://getoban.pro/articles/pro-1-5-launch-week-day-1 (https://getoban.pro/articles/pro-1-5-launch-week-day-1?utm_source=thinkingelixir&utm_medium=shownotes) – Article on Oban Pro's launch week, detailing new features and improvements. - https://getoban.pro/articles/pro-1-5-launch-week-day-2 (https://getoban.pro/articles/pro-1-5-launch-week-day-2?utm_source=thinkingelixir&utm_medium=shownotes) – Second day of Oban Pro's launch week article series. - https://getoban.pro/articles/pro-1-5-launch-week-day-3 (https://getoban.pro/articles/pro-1-5-launch-week-day-3?utm_source=thinkingelixir&utm_medium=shownotes) – Third day of Oban Pro's launch week article series. - https://alembic.com.au/blog/igniter-rethinking-code-generation-with-project-patching (https://alembic.com.au/blog/igniter-rethinking-code-generation-with-project-patching?utm_source=thinkingelixir&utm_medium=shownotes) – Blog post by Zach Daniel about Igniter, a tool for rethinking code generation with project patching, useful for installing libraries into existing Phoenix applications. - https://2024.elixirconf.com/ (https://2024.elixirconf.com/?utm_source=thinkingelixir&utm_medium=shownotes) – ElixirConf 2024 Preview with details on scheduled speakers and topics. Do you have some Elixir news to share? Tell us at @ThinkingElixir (https://twitter.com/ThinkingElixir) or email at show@thinkingelixir.com (mailto:show@thinkingelixir.com) Find us online - Message the show - @ThinkingElixir (https://twitter.com/ThinkingElixir) - Message the show on Fediverse - @ThinkingElixir@genserver.social (https://genserver.social/ThinkingElixir) - Email the show - show@thinkingelixir.com (mailto:show@thinkingelixir.com) - Mark Ericksen - @brainlid (https://twitter.com/brainlid) - Mark Ericksen on Fediverse - @brainlid@genserver.social (https://genserver.social/brainlid) - David Bernheisel - @bernheisel (https://twitter.com/bernheisel) - David Bernheisel on Fediverse - @dbern@genserver.social (https://genserver.social/dbern)

This is a recap of the top 10 posts on Hacker News on July 29th, 2024.This podcast was generated by wondercraft.ai(00:40): One-man SaaS, 9 Years InOriginal post: https://news.ycombinator.com/item?id=41104293&utm_source=wondercraft_ai(01:50): CrowdStrike's impact on aviationOriginal post: https://news.ycombinator.com/item?id=41103101&utm_source=wondercraft_ai(03:01): SAM 2: Segment Anything in Images and VideosOriginal post: https://news.ycombinator.com/item?id=41104523&utm_source=wondercraft_ai(04:09): Movable tree CRDTs and Loro's implementationOriginal post: https://news.ycombinator.com/item?id=41099901&utm_source=wondercraft_ai(05:30): MeTube: Self-hosted YouTube downloaderOriginal post: https://news.ycombinator.com/item?id=41098974&utm_source=wondercraft_ai(06:38): Understanding the design of the the Super Nintendo video systemOriginal post: https://news.ycombinator.com/item?id=41098141&utm_source=wondercraft_ai(07:51): Attribution is dying, clicks are dyingOriginal post: https://news.ycombinator.com/item?id=41101948&utm_source=wondercraft_ai(09:08): Is Cloudflare overcharging us for their images service?Original post: https://news.ycombinator.com/item?id=41100958&utm_source=wondercraft_ai(10:19): FastHTML – Modern web applications in pure PythonOriginal post: https://news.ycombinator.com/item?id=41104305&utm_source=wondercraft_ai(11:26): Children should be allowed to get bored (2013)Original post: https://news.ycombinator.com/item?id=41098488&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

Johannes Schickling (@schickling | schickling.dev⁠) gets us up-to-speed on Effect, the ZIO-inspired Effect System for TypeScript, and the Local-First movement. Resources: Local-First Podcast: www.localfirst.fm Ink & Switch's Local-First Essay: www.inkandswitch.com/local-first Effect (TypeScript Library): effect.website Riffle research project: riffle.systems LiveStore: github.com/livestorejs Overtone: overtone.pro Related Episodes: #81 TypeScript & Effects with Michael Arnaldi #62 Musing About Local-First Sync & CRDTs with Mark McGranaghan Discuss this episode: ⁠⁠⁠⁠⁠⁠⁠discord.gg/XVKD2uPKyF

This is my conversation with Varun Srinivasan - cofounder of Merkle Manufactory, the company building the Farcaster protocol and the Warpcast client.Timestamps:- 00:00:00 intro- 00:01:34 sponsor: Optimism- 00:02:44 Farcaster origins- 00:05:59 sufficient decentralization, namespaces, hubs and CRDTs- 00:16:02 type 1 vs type 2 decisions- 00:21:23 the protocol, channels, clients, spam- 00:30:13 direct messaging and end-to-end encryption- 00:36:38 a turing complete social protocol- 00:41:58 sponsor: Privy- 00:43:19 why frames- 00:52:14 Facebook, Twitter, Farcaster- 01:03:25 backstory, growing up in India, Microsoft, YC- 01:08:11 learnings from Coinbase- 01:15:13 building a company- 01:18:53 doing the one thing that matters- 01:28:07 outroThank you to our sponsors for making this podcast possible:Optimism - https://optimism.ioPrivy - https://privy.ioInto the Bytecode:Twitter -  https://twitter.com/sinahabFarcaster - https://warpcast.com/sinahabOther episodes - https://intothebytecode.comDisclaimer: this podcast is for informational purposes only. It is not financial advice or a recommendation to buy or sell securities. The host and guests may hold positions in the projects discussed.

This interview was recorded at GOTO Amsterdam for GOTO Unscripted.http://gotopia.techRead the full transcription of this interview hereMartin Kleppmann - Researcher at the Technical University of Munich & Author of "Designing Data-Intensive Applications"Jesse Anderson - Managing Director of Big Data Institute, Host of The Data Dream Team PodcastRESOURCESJesse Anderson: https://youtu.be/cWSCI1LpoGYMartin Kleppmann: https://youtu.be/esMjP-7jlREPrag. Dave Thomas: https://youtu.be/ug8XX2MpzEwhttps://automerge.orgMartinhttps://martin.kleppmann.comhttps://twitter.com/martinklhttps://nondeterministic.computer/@martinhttps://linkedin.com/in/martinkleppmannJessehttps://twitter.com/jessetandersonhttps://www.jesse-anderson.comhttps://sodapodcast.libsyn.com/sitehttps://linkedin.com/in/jessetandersonhttps://www.jesse-anderson.com/category/blogDESCRIPTIONJesse Anderson, director at Big Data Institute, and Martin Kleppmann, author of "Designing Data-Intensive Applications" explore together the evolving data landscape. They start with the origins of Martin's book, emphasizing the crucial art of asking the right questions. Martin unveils industry shifts since 2017, spotlighting the transformative rise of cloud services.The conversation then takes a twist as Martin delves into academia, sharing insights on local-first collaboration software and the fascinating world of Automerge. Aspiring software engineers are treated with some advice on how to navigate the delicate balance between simplicity and adaptability.The interview concludes with a glimpse into diverse career paths in the dynamic realm of data engineering, making it a must-watch for professionals at every stage of their journey.RECOMMENDED BOOKSMartin Kleppmann • Designing Data-Intensive ApplicationsMartin Kleppmann • Secret Colors: A Gentle Introduction to CryptographyJesse Anderson • Data TeamsJesse Anderson • Data Engineering TeamsJesse Anderson • The Ultimate Guide to Switching Careers to Big DataViktor Gamov, Dylan Scott & Dave Klein • Kafka in ActionFabian Hueske & Vasiliki Kalavri • Stream Processing with Apache FlinkTwitterInstagramLinkedInFacebookLooking for a unique learning experience?Attend the next GOTO conference near you! Get your ticket: gotopia.techSUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted daily!

Happy 2024! We appreciated all the feedback on the listener survey (still open, link here)! Surprising to see that some people's favorite episodes were others' least, but we'll always work on improving our audio quality and booking great guests. Help us out by leaving reviews on Twitter, YouTube, and Apple Podcasts!

In this episode I give a brief description of how to become a dental hygienist. This episode is based off of my personal experience and requirements through community college. Find a CODA accredited program Understand all requirements and prerequisites Complete prerequisites (approx 2 years) Apply to dental hygiene program Complete 2 year dental hygiene program Pass NBDHE Pass WREB or CRDTS or whichever exam is required in your state Pass law and ethics exam Apply for licensure Every state and program is different. It is highly recommended to complete your own research on this topic for a clear understanding of requirements to become a RDH in your area.

Kubernetes 13. Oktober 2023, Jochen Überraschung! Schon wieder Zeit für eine neue Podcast-Episode

In this conversation I caught up with the second part to my conversation with Scott Jenson to explore how Designer can actively get more involved with the free and open source software movement. We speak more about why this is of such an interest to Scott, and he gives his awesome advice to others on how they can get involved. Listen to Episode 1:  Listen to Scott Jenson 'From Apple, Symbian to Google - Exploring the World of Free and Open Source Software (FOSS)'Links from this episode; Open Source Job Postings Job Postings Youtube FOSSBack links #FOSSBack: Scott Jenson - Why is UX so hard to contribute to FOSS 2021 (The problem) #FOSSBack: Scott Jenson – Calling all UX Designers! 2022 (How maintainers can attract UX designers) #FOSSBack: Scott Jenson – Plan Like Einstein 2023 (How to move the culture to include UX) Open source research about using CRDTs (a new experimental file format) to have an alternative to Google Docs Upwelling: Combining real-time collaboration with version control for writers. A 1-pager on how to discuss design in a more thoughtful and cooperative way Guidelines for Discussing UX NEW COURSE: Service Design : https://www.thisishcd.com/courses/video-on-demand-introduction-to-human-centered-service-design-using-journey-mapping Become a Patron of This is HCD / Become a Patron Sign up to This is HCD Newsletter / Stay up to date with This is HCD Learn more with This is HCD / Courses on Service Design, Human Centered Design, UX Design, Research Coaching for Change-Makers / Coaching & Mentoring for Innovators & Change-Makers FREE GUIDE - 7 Days of Awesome Journey Map Tips: FREE 7-DAY GUIDE - Killer Journey Mapping Tips Follow Gerry Scullion on Twitter / gerrycircus Follow This is HCD on Twitter / thisishcd Our partner links We recommend Webflow for all web design and development needs.‍ Webflow: Create a custom website | No-code website builder We used Descript to power our podcast ‍Descript | All-in-one video & podcast editing, easy as a doc Our podcast is edited by the A-Team at CastUp / Enquire with them about editing. ‍Learn more and connect with Castup Learn more about your ad choices. Visit megaphone.fm/adchoices

In this conversation I caught up with the second part to my conversation with Scott Jenson to explore how Designer can actively get more involved with the free and open source software movement. We speak more about why this is of such an interest to Scott, and he gives his awesome advice to others on how they can get involved. Listen to Episode 1:  Listen to Scott Jenson 'From Apple, Symbian to Google - Exploring the World of Free and Open Source Software (FOSS)'Links from this episode; Open Source Job Postings Job Postings Youtube FOSSBack links #FOSSBack: Scott Jenson - Why is UX so hard to contribute to FOSS 2021 (The problem) #FOSSBack: Scott Jenson – Calling all UX Designers! 2022 (How maintainers can attract UX designers) #FOSSBack: Scott Jenson – Plan Like Einstein 2023 (How to move the culture to include UX) Open source research about using CRDTs (a new experimental file format) to have an alternative to Google Docs Upwelling: Combining real-time collaboration with version control for writers. A 1-pager on how to discuss design in a more thoughtful and cooperative way Guidelines for Discussing UX NEW COURSE: Service Design : https://www.thisishcd.com/courses/video-on-demand-introduction-to-human-centered-service-design-using-journey-mapping Become a Patron of This is HCD / Become a Patron Sign up to This is HCD Newsletter / Stay up to date with This is HCD Learn more with This is HCD / Courses on Service Design, Human Centered Design, UX Design, Research Coaching for Change-Makers / Coaching & Mentoring for Innovators & Change-Makers FREE GUIDE - 7 Days of Awesome Journey Map Tips: FREE 7-DAY GUIDE - Killer Journey Mapping Tips Follow Gerry Scullion on Twitter / gerrycircus Follow This is HCD on Twitter / thisishcd Our partner links We recommend Webflow for all web design and development needs.‍ Webflow: Create a custom website | No-code website builder We used Descript to power our podcast ‍Descript | All-in-one video & podcast editing, easy as a doc Our podcast is edited by the A-Team at CastUp / Enquire with them about editing. ‍Learn more and connect with Castup Learn more about your ad choices. Visit megaphone.fm/adchoices

This week we are joined by Richael Cobler, the Executive Director at CRDTS or Central Regional Dental Testing Services, Inc as she explains options for remediation. Remediation can be for a clinician who has requirements delivered by the state but can also be for the every day practitioner who maybe has taken some time off and wants to get back in the game.  Resources: www.crdts.org Instagram @crdts_dental Email: Richael@crdts.org or call 785-273-0380  

This week we are joined by Richael Cobler, the Executive Director at CRDTS or Central Regional Dental Testing Services, Inc as she explains options for remediation. Remediation can be for a clinician who has requirements delivered by the state but can also be for the every day practitioner who maybe has taken some time off and wants to get back in the game.  Resources: www.crdts.org Instagram @crdts_dental Email: Richael@crdts.org or call 785-273-0380  

This week's TIPisode is for the students and the educators out there! We are joined by Kim Laudenslager, RDH, MPA, CDIPC as she explains some essential tips to passing the CRDTS clinical board exams.  To learn more head over to http://www.crdts.org or send email to info@crdts.org

This week's TIPisode is for the students and the educators out there! We are joined by Kim Laudenslager, RDH, MPA, CDIPC as she explains some essential tips to passing the CRDTS clinical board exams.  To learn more head over to http://www.crdts.org or send email to info@crdts.org

James Arthur shares his project Vaxine.io, an Elixir layer built on top of a CRDT based distributed Erlang database called Antidote DB. We cover what CRDTs are and introduce how they work. We learn more about Antidote DB, the CURE protocol and especially the Vaxine.io project that adds Ecto types and makes it more approachable to Elixir applications. As applications become more global, the need for strongly consistent distributed writes becomes much more important. Show Notes online - http://podcast.thinkingelixir.com/103 (http://podcast.thinkingelixir.com/103) Elixir Community News - https://www.elixirconf.eu/talks/typecheck-effortless-runtime-type-checking/ (https://www.elixirconf.eu/talks/typecheck-effortless-runtime-type-checking/) – Marten shared an update on the TypeCheck project from ElixirConf.EU (June 9-10) - https://podcast.thinkingelixir.com/72 (https://podcast.thinkingelixir.com/72) – Episode with Martin about TypeCheck - https://twitter.com/elixirphoenix/status/1532707770415325185 (https://twitter.com/elixirphoenix/status/1532707770415325185) - https://twitter.com/wojtekmach/status/1532662628077785088 (https://twitter.com/wojtekmach/status/1532662628077785088) – Screenshot showing the single-file LiveView page - https://github.com/wojtekmach/mixinstallexamples/blob/main/phoenixliveview.exs (https://github.com/wojtekmach/mix_install_examples/blob/main/phoenix_live_view.exs) - Mix Install Examples - Phoenix LiveView app in ~70 LOC (Mix Install Examples - Phoenix LiveView app in ~70 LOC) - https://twitter.com/polvalente/status/1532439823964946432 (https://twitter.com/polvalente/status/1532439823964946432) – New Nx library called nx-signal was shared by the author, Paulo Valente - https://github.com/polvalente/nx-signal (https://github.com/polvalente/nx-signal) - https://twitter.com/josevalim/status/1533136904736198656 (https://twitter.com/josevalim/status/1533136904736198656) – José's cryptic tweet about Torchvision, ONNX, and a LiveView app - https://pytorch.org/vision/stable/index.html (https://pytorch.org/vision/stable/index.html) – Torchvision docs - https://onnx.ai/ (https://onnx.ai/) – ONNX a format for transporting trained machine learning models - https://github.com/thehaigo/live_onnx (https://github.com/thehaigo/live_onnx) – LiveOnnx project that combines the previous things with Axon and LiveView - https://github.com/oestrich/aino (https://github.com/oestrich/aino) – Aino released 0.5 - https://twitter.com/ericoestrich/status/1533995968793919488 (https://twitter.com/ericoestrich/status/1533995968793919488) – Eric explained v0.5 Aino changes - https://twitter.com/josevalim/status/1533907809942880261 (https://twitter.com/josevalim/status/1533907809942880261) – José Valim tweeted a new graphic, teasing something new in Nx land. - https://twitter.com/josevalim/status/1534120503182602240 (https://twitter.com/josevalim/status/1534120503182602240) – José mentioned that there are 3 major announcements this month starting at ElixirConfEU. Stay tuned! Do you have some Elixir news to share? Tell us at @ThinkingElixir (https://twitter.com/ThinkingElixir) or email at show@thinkingelixir.com (mailto:show@thinkingelixir.com) Discussion Resources - https://vaxine.io (https://vaxine.io) – Vaxine.io website - https://antidotedb.eu (https://antidotedb.eu) – Antidote DB website - https://crdt.tech (https://crdt.tech) – CRDT information website - https://vaxine.io/tech/how-it-works (https://vaxine.io/tech/how-it-works) - https://github.com/vaxine-io (https://github.com/vaxine-io) - https://github.com/AntidoteDB/antidote (https://github.com/AntidoteDB/antidote) – Erlang project by a different group "A planet scale, highly available, transactional database built on CRDT technology" - https://www.antidotedb.eu/ (https://www.antidotedb.eu/) - https://github.com/vaxine-io/vaxine (https://github.com/vaxine-io/vaxine) - https://github.com/vaxine-io/vax (https://github.com/vaxine-io/vax) – Data access library and Ecto integration - https://github.com/vaxine-io/examples (https://github.com/vaxine-io/examples) – Example and demo apps - https://www.foundationdb.org/ (https://www.foundationdb.org/) - https://riak.com/index.html (https://riak.com/index.html) - https://www.cockroachlabs.com/ (https://www.cockroachlabs.com/) - https://en.wikipedia.org/wiki/CockroachDB (https://en.wikipedia.org/wiki/CockroachDB) - https://supabase.com/ (https://supabase.com/) - https://lunar.vc/ (https://lunar.vc/) Guest Information - https://twitter.com/VaxineIO (https://twitter.com/VaxineIO) – Vaxine.io on Twitter - https://github.com/vaxine-io/ (https://github.com/vaxine-io/) – Vaxine Github Organization - https://vaxine.io (https://vaxine.io) – Vaxine.io website - https://vaxine.io/blog (https://vaxine.io/blog) – Blog Find us online - Message the show - @ThinkingElixir (https://twitter.com/ThinkingElixir) - Email the show - show@thinkingelixir.com (mailto:show@thinkingelixir.com) - Mark Ericksen - @brainlid (https://twitter.com/brainlid) - David Bernheisel - @bernheisel (https://twitter.com/bernheisel) - Cade Ward - @cadebward (https://twitter.com/cadebward)

Mark McGranaghan joins us to talk about how the Muse app uses Conflict-free Replicated Data Types (CRDTs) for local-first data synchronization. More details on Local-first: https://www.inkandswitch.com/local-first/ Referenced article about hybrid logical clocks: https://archive.jlongster.com/using-crdts-in-the-wild Discuss this episode: https://discord.gg/nPa76qF

Steph has a baby update and thoughts on movies, plus a question for Chris related to migrating Test Unit tests to RSpec. Chris watched a video from Google I/O where Chrome devs talked about a new feature called Page Transitions. He's also been working with a tool called Customer.io, an omnichannel communication whiz-bang adventure! Page transitions Overview (https://youtu.be/JCJUPJ_zDQ4) Using yield_self for composable ActiveRecord relations (https://thoughtbot.com/blog/using-yieldself-for-composable-activerecord-relations) A Case for Query Objects in Rails (https://thoughtbot.com/blog/a-case-for-query-objects-in-rails) Customer.io (https://customer.io/) Turning the database inside-out with Apache Samza | Confluent (https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/) Datomic (https://www.datomic.com/) About CRDTs • Conflict-free Replicated Data Types (https://crdt.tech/) Apache Kafka (https://kafka.apache.org/) Resilient Management | A book for new managers in tech (https://resilient-management.com/) Mixpanel: Product Analytics for Mobile, Web, & More (https://mixpanel.com/) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: CHRIS: Golden roads are golden. Okay, everybody's got golden roads. You have golden roads, yes? That is what we're -- STEPH: Oh, I have golden roads, yes. [laughter] CHRIS: You might should inform that you've got golden roads, yeah. STEPH: Yeah, I don't know if I say might should as much but might could. I have been called out for that one a lot; I might could do that. CHRIS: [laughs] STEPH: That one just feels so natural to me than normal. Anytime someone calls it out, I'm like, yeah, what about it? [laughter] CHRIS: Do you want to fight? STEPH: Yeah, are we going to fight? CHRIS: I might could fight you. STEPH: I might could. I might should. [laughter] CHRIS: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, what's new in your world? STEPH: Hey, Chris. I have a couple of fun updates. I have a baby Viccari update, so little baby weighs about two pounds now, two pounds. I'm 25 weeks along. So not that I actually know the exact weight, I'm using all those apps that estimate based on how far along you are, so around two pounds, which is novel. Oh, and then the other thing I'm excited to tell you about...I'm not sure how I should feel that I just got more excited about this other thing. I'm very excited about baby Viccari. But the other thing is there's a new Jurassic Park movie coming out, and I'm very excited. I think it's June 10th is when it comes out. And given how much we have sung that theme song to each other and make references to what a clever girl, I needed to share that with you. Maybe you already know, maybe you're already in the loop, but if you don't, it's coming. CHRIS: Yeah, the internet likes to yell things like that. Have you watched all of the most recent ones? There are like two, and I think this will be the third in the revisiting or whatever, the Jurassic World version or something like that. But have you watched the others? STEPH: I haven't seen all of them. So I've, of course, seen the first one. I saw the one that Chris Pratt was in, and now he's in the latest one. But I think I've missed...maybe there's like two in the middle there. I have not watched those. CHRIS: There are three in the original trilogy, and then there are three now in the new trilogy, which now it's ending, and they got everybody. STEPH: Oh, I'm behind. CHRIS: They got people from the first one, and they got the people from the second trilogy. They just got everybody, and that's exciting. You know, it's that thing where they tap into nostalgia, and they take advantage of us via it. But I'm fine. I'm here for it. STEPH: I'm here for it, especially for Jurassic Park. But then there's also a new Top Gun movie coming out, which, I'll be honest, I'm totally going to watch. But I really didn't remember the first one. I don't know that I've really ever watched the first Top Gun. So Tim, my partner, and I watched that recently, and it's such a bad movie. I'm going to say it; [laughs] it's a bad movie. CHRIS: I mean, I don't want to disagree, but the volleyball scene, come on, come on, the volleyball scene. [laughter] STEPH: I mean, I totally had a good time watching the movie. But the one part that I finally kept complaining about is because every time they showed the lead female character, I can't think of her character name or the actress's name, but they kept playing that song, Take My Breath Away. And I was like, can we just get past the song? [laughs] Because if you go back and watch that movie, I swear they play it like six different times. It was a lot. It was too much. So I moved it into bad movie category but bad movie totally worth watching, whatever category that is. CHRIS: Now I kind of want to revisit it. I feel like the drinking game writes itself. But at a minimum, anytime Take My Breath Away plays, yeah. Well, all right, good to know. [laughs] STEPH: Well, if you do that, let me know how many shots or beers you drink because I think it will be a fair amount. I think it will be more than five. CHRIS: Yeah, it involves a delicate calibration to get that right. I don't think it's the sort of thing you just freehand. It writes itself but also, you want someone who's tried it before you so that you're not like, oh no, it's every time they show a jet. That was too many. You can't drink that much while watching this movie. STEPH: Yeah, that would be death by Top Gun. CHRIS: But not the normal way, the different, indirect death by Top Gun. STEPH: I don't know what the normal way is. [laughs] CHRIS: Like getting shot down by a Top Gun pilot. [laughter] STEPH: Yeah, that makes sense. [laughs] CHRIS: You know, the dogfighting in the plane. STEPH: The actual, yeah, going to war away. Just sitting on your couch and you drink too much poison away, yeah, that one. All right, that got weird. Moving on, [laughs] there's a new Jurassic Park movie. We're going to land on that note. And in the more technical world, I've got a couple of things on my mind. One of them is I have a question for you. I'm very excited to run this by you because I could use a friend in helping me decide what to do. So I am still on that journey where I am migrating Test::Unit test over to RSpec. And as I'm going through, it's going pretty well, but it's a little complicated because some of the Test::Unit tests have different setup than, say, the RSpec do. They might run different scripts beforehand where they're loading data. That's perhaps a different topic, but that's happening. And so that has changed a few things. But then overall, I've just been really just porting everything over, like, hey, if it exists in the Test::Unit, let's just bring it to RSpec, and then I'm going to change these asserts to expects and really not make any changes from there. But as I'm doing that, I'm seeing areas that I want to improve and things that I want to clear up, even if it's just extracting a variable name. Or, as I'm moving some of these over in Test::Unit, it's not clear to me exactly what the test is about. Like, it looks more like a method name in the way that the test is being described, but the actual behavior isn't clear to me as if I were writing this in RSpec, I think it would have more of a clear description. Maybe that's not specific to the actual testing framework. That might just be how these tests are set up. But I'm at that point where I'm questioning should I keep going in terms of where I am just copying everything over from Test::Unit and then moving it over to RSpec? Because ultimately, that is the goal, to migrate over. Or should I also include some time to then go back and clean up and try to add some clarity, maybe extract some variable names, see if I can reduce some lets, maybe even reduce some of the test helpers that I'm bringing over? How much cleanup should be involved, zero, lots? I don't know. I don't know what that...[laughs] I'm sure there's a middle ground in there somewhere. But I'm having trouble discerning for myself what's the right amount because this feels like one of those areas where if I don't do any cleanup, I'm not coming back to it, like, that's just the truth. So it's either now, or I have no idea when and maybe never. CHRIS: I'll be honest, the first thing that came to mind in this most recent time that you mentioned this is, did we consider just deleting these tests entirely? Is that on...like, there are very few of them, right? Like, are they even providing enough value? So that was question one, which let me pause to see what your thoughts there were. [chuckles] STEPH: I don't know if we specifically talked about that on the mic, but yes, that has been considered. And the team that owns those tests has said, "No, please don't delete them. We do get value from them." So we can port them over to RSpec, but we don't have time to port them over to RSpec. So we just need to keep letting them go on. But yet, not porting them conflicts with my goal of then trying to speed up CI. And so it'd be nice to collapse these Test::Unit tests over to RSpec because then that would bring our CI build down by several meaningful minutes. And also, it would reduce some of the complexity in the CI setup. CHRIS: Gotcha. Okay, so now, having set that aside, I always ask the first question of like, can you just put Derek Prior's phone number on the webpage and call it an app? Is that the MVP of this app? No? Okay, all right, we have to build more. But yeah, I think to answer it and in a general way of trying to answer a broader set of questions here... I think this falls into a category of like if you find yourself having to move around some code, if that code is just comfortably running and the main thing you need to do is just to get it ported over to RSpec, I would probably do as little other work as possible. With the one consideration that if you find yourself needing to deeply load up the context of these tests like actually understand them in order to do the porting, then I would probably take advantage of that context because it's hard to get your head into a given piece of code, test or otherwise. And so if you're in there and you're like, well, now that I'm here, I can definitely see that we could rearrange some stuff and just definitively make it better, if you get to that place, I would consider it. But if this ends up being mostly a pretty rote transformation like you said, asserts become expects, and lets get switched around, you know, that sort of stuff, if it's a very mechanical process of getting done, I would probably say very minimal. But again, if there is that, like, you know what? I had to understand the test in order to port them anyway, so while I'm here, let me take advantage of that, that's probably the thing that I would consider. But if not that, then I would say even though it's messy and whatnot and your inclination would be to clean it, I would say leave it roughly as is. That's my guess or how I would approach it. STEPH: Yeah, I love that. I love how you pointed out, like, did you build up the context? Because you're right, in a lot of these test cases, I'm not, or I'm trying really hard to not build up context. I'm trying very hard to just move them over and, if I have to, mainly to find test descriptions. That's the main area I'm struggling to...how can I more explicitly state what this test does so the next person reading this will have more comprehension than I do? But otherwise, I'm trying hard to not have any real context around it. And that's such a good point because that's often...when someone else is in the middle of something, and they're deciding whether to include that cleanup or refactor or improvement, one of my suggestions is like, hey, we've got the context now. Let's go with it. But if you've built up very little context, then that's not a really good catalyst or reason to then dig in deeper and apply that cleanup. That's super helpful. Thank you. That will help reinforce what I'm going to do, which is exactly let's migrate RSpec and not worry about cleanup, which feels terrible; I'm just going to say that into the world. But it also feels like the right thing to do. CHRIS: Well, I'm happy to have helped. And I share the like, and it feels terrible. I want to do the right thing, but sometimes you got to pick a battle sort of thing. STEPH: Cool. Well, that's a huge help to me. What's going on in your world? CHRIS: What's going on in my world? I watched a great video the other day from the Google I/O. I think it's an event; I'm not actually sure, conference or something like that. But it was some Google Chrome developers talking about a new feature that's coming to the platform called Page Transitions. And I've kept an eye on this for a while, but it seems like it's more real. Like, I think they put out an RFC or an initial sort of set of ideas a while back. And the web community was like, "Oh, that's not going to work out so well." So they went back to the drawing board, revisited. I've actually implemented in Chrome Canary a version of the API. And then, in the video that I watched, which we'll include a show notes link to, they demoed the functionality of the Page Transitions API and showed what you can do. And it's super cool. It allows for the sort of animations that you see in a lot of native mobile apps where you're looking at a ListView, you click on one of the items, and it grows to fill the whole screen. And now you're on the detail screen for that item that you were looking at. But there was this very continuous animated transition that allows you to keep context in your head and all of those sorts of nice things. And this just really helps to bridge that gap between, like, the web often lags behind the native mobile platforms in terms of the experiences that we can build. So it was really interesting to see what they've been able to pull off. The demo is a pretty short video, but it shows a couple of different variations of what you can build with it. And I was like, yeah, these look like cool native app transitions, really nifty. One thing that's very interesting is the actual implementation of this. So it's like you have one version of the page, and then you transition to a new version of the page, and in doing so, you want to animate between them. And the way that they do it is they have the first version of the page. They take a screenshot of it like the browser engine takes a screenshot of it. And then they put that picture on top of the actual browser page. Then they do the same thing with the next version of the page that they're going to transition to. And then they crossfade, like, change the opacity and size and whatnot between the two different images, and then you're there. And in the back of my mind, I'm like, I'm sorry, what now? You did which? I'm like, is this the genius solution that actually makes this work and is performant? And I wonder if there are trade-offs. Like, do you lose interactivity between those because you've got some images that are just on the screen? And what is that like? But as they were going through it, I was just like, wait, I'm sorry, you did what? This is either the best idea I've ever heard, or I'm not so sure about this. STEPH: That's fascinating. You had me with the first part in terms of they take a screenshot of the page that you're leaving. I'm like, yeah, that's a great idea. And then talking about taking a picture of the other page because then you have to load it but not show it to the user that it's loaded. And then take a picture of it, and then show them the picture of the loaded page. But then actually, like you said, then crossfade and then bring in the real functionality. I am...what am I? [laughter] CHRIS: What am I actually? STEPH: [laughs] What am I? I'm shocked. I'm surprised that that is so performant. Because yeah, I also wouldn't have thought of that, or I would have immediately have thought like, there's no way that's going to be performant enough. But that's fascinating. CHRIS: For me, performance seems more manageable, but it's the like, what are you trading off for that? Because that sounds like a hack. That sounds like the sort of thing I would recommend if we need to get an MVP out next week. And I'm like, what if we just tried this? Listen, it's got some trade-offs. So I'm really interested to see are those trade-offs present? Because it's the browser engine. It's, you know, the low-level platform that's actually managing this. And there are some nice hooks that allow you to control it. And at a CSS level, you can manage it and use keyframe animations to control the transition more directly. There's a JavaScript API to instrument the sequencing of things. And so it's giving you the right primitives and the right hooks. And the fact that the implementation happens to use pictures or screenshots, to use a slightly different word, it's like, okey dokey, that's what we're doing. Sounds fun. So I'm super interested because the functionality is deeply, deeply interesting to me. Svelte actually has a version of this, the crossfade utility, but you have to still really think about how do you sequence between the two pages and how do you do the connective tissue there? And then Svelte will manage it for you if you do all the right stuff. But the wiring up is somewhat complicated. So having this in the browser engine is really interesting to me. But yeah, pictures. STEPH: This is one of those ideas where I can't decide if this was someone who is very new to the team and new to the idea and was like, "Have we considered screenshots? Have we considered pictures?" Or if this is like the uber senior person on the team that was like, "Yeah, this will totally work with screenshots." I can't decide where in that range this idea falls, which I think makes me love it even more. Because it's very straightforward of like, hey, what if we just tried this? And it's working, so cool, cool, cool. CHRIS: There's a fantastic meme that's been making the rounds where it's a bell curve, and it's like, early in your career, middle of your career, late in your career. And so early in your career, you're like, everything in one file, all lines of code that's just where they go. And then in the middle of your career, you're like, no, no, no, we need different concerns, and files, and organizational structures. And then end of your career...and this was coming up in reference to the TypeScript team seems to have just thrown everything into one file. And it's the thing that they've migrated to over time. And so they have this many, many line file that is basically the TypeScript engine all in one file. And so it was a joke of like, they definitely know what they're doing with programming. They're not just starting last week sort of thing. And so it's this funny arc that certain things can go through. So I think that's an excellent summary there [laughs] of like, I think it was folks who have thought about this really hard. But I kind of hope it was someone who was just like, "I'm new here. But have we thought about pictures? What about pictures?" I also am a little worried that I just deeply misunderstood [laughs] the representation but glossed over it in the video, and I'm like, that sounds interesting. So hopefully, I'm not just wildly off base here. [laughs] But nonetheless, the functionality looks very interesting. STEPH: That would be a hilarious tweet. You know, I've been waiting for that moment where I've said something that I understood into the mic and someone on Twitter just being like, well, good try, but... [laughs] CHRIS: We had a couple of minutes where we tried to figure out what the opposite of ranting was, and we came up with pranting and made up a word instead of going with praising or raving. No, that's what it is, raving. [laughs] STEPH: No, raving. I will never forget now, raving. [laughs] CHRIS: So, I mean, we've done this before. STEPH: That's true. Although they were nice, I don't think they tweeted. I think they sent in an email. They were like, "Hey, friends." [laughter] CHRIS: Actually, we got a handful of emails on that. [laughter] STEPH: Did you know the English language? CHRIS: Thank you, kind Bikeshed audience, for not shaming us in public. I mean, feel free if you feel like it. [laughs] But one other thing that came up in this video, though, is the speaker was describing single-page apps are very common, and you want to have animated transitions and this and that. And I was like, single-page app, okay, fine. I don't like the terminology but whatever. I would like us to call it the client-side app or client-side routing or something else. But the fact that it's a single page is just a technical consideration that no user would call it that. Users are like; I go to the web app. I like that it has URLs. Those seem different to me. Anyway, this is my hill. I'm going to die on it. But then the speaker in the video, in contrast to single-page app referenced multi-page app, and I was like, oh, come on, come on. I get it. Like, yes, there are just balls of JavaScript that you can download on the internet and have a dynamic graphics editor. But I think almost all good things on the web should have URLs, and that's what I would call the multiple pages. But again, that's just me griping about some stuff. And to name it, I don't think I'm just griping for griping sake. Like, again, I like to think about things from the user perspective, and the URL being so important. And having worked with plenty of apps that are implemented in JavaScript and don't take the URL or the idea that we can have different routable resources seriously and everything is just one URL, that's a failure mode in my mind. We missed an opportunity here. So I think I'm saying a useful thing here and not just complaining on the internet. But with that, I will stop complaining on the internet and send it back over to you. What else is new in your world, Steph? STEPH: I do remember the first time that you griped about it, and you were griping about URLs. And there was a part of me that was like, what is he talking about? [laughter] And then over time, I was like, oh, I get it now as I started actually working more in that world. But it took me a little bit to really appreciate that gripe and where you're coming from. And I agree; I think you're coming from a reasonable place, not that I'm biased at all as your co-host, but you know. CHRIS: I really like the honest summary that you're giving of, like, honestly, the first time you said this, I let you go for a while, but I did not know what you were talking about. [laughs] And I was like, okay, good data point. I'm going to store that one away and think about it a bunch. But that's fine. I'm glad you're now hanging out with me still. [laughter] STEPH: Don't do that. Don't think about it a bunch. [laughs] Let's see, oh, something else that's going on in my world. I had a really fun pairing session with another thoughtboter where we were digging into query objects and essentially extracting some logic out of an ActiveRecord model and then giving that behavior its own space and elevated namespace in a query object. And one of the questions or one of the things that came up that we needed to incorporate was optional filters. So say if you are searching for a pizza place that's nearby and you provide a city, but you don't provide what's the optional zip code, then we want to make sure that we don't apply the zip code in the where clause because then you would return all the pizza places that have a nil zip code, and that's just not what you want. So we need to respect the fact that not all the filters need to be applied. And there are a couple of ways to go about it. And it was a fun journey to see the different ways that we could structure it. So one of the really good starting points is captured in a blog post by Derek Prior, which we'll include a link to in the show notes, and it's using yield_self for composable ActiveRecord relations. But essentially, it starts out with an example where it shows that you're assigning a value to then the result of an if statement. So it's like, hey, if the zip code is present, then let's filter by zip code; if not, then just give us back the original relation. And then you can just keep building on it from there. And then there's a really nice implementation that Derek built on that then uses yield_self to pass the relation through, which then provides a really nice readability for as you are then stepping through each filter and which one should and shouldn't be applied. And now there's another blog post, and this one's written by Thiago Silva, A Case for Query Objects in Rails. And this one highlighted an approach that I haven't used before. And I initially had some mixed feelings about it. But this approach uses the extending method, which is a method that's on ActiveRecord query methods. And it's used to extend the scope with additional methods. You can either do this by providing the name of a module or by providing a block. It's only going to apply to that instance or to that specific scope when you're using it. So it's not going to be like you're running an include or something like that where all instances are going to now have access to these methods. So by using that method, extending, then you can create a module that says, "Hey, I want to create this by zip code filter that will then check if we have a zip code, let's apply it, if not, return the relation. And it also creates a really pretty chaining experience of like, here's my original class name. Let's extend with these specific scopes, and then we can say by zip code, by pizza topping, whatever else it is that we're looking to filter by. And I was initially...I saw the extending, and it made me nervous because I was like, oh, what all does this apply to? And is it going to impact anything outside of this class? But the more I've looked at it, the more I really like it. So I think you've seen this blog post too. And I'm curious, what are your thoughts about this? CHRIS: I did see this blog post come through. I follow that thoughtbot blog real close because it turns out some of the best writing on the internet is on there. But I saw this...also, as an aside, I like that we've got two Derek Prior references in one episode. Let's see if we can go for three before the end. But one thing that did stand out to me in it is I have historically avoided scopes using scope like ActiveRecord macro thing. It's a class method, but like, it's magic. It does magic. And a while ago, class methods and scopes became roughly equivalent, not exactly equivalent, but close enough. And for me, I want to use methods because I know stuff about methods. I know about default arguments. And I know about all of these different subtleties because they're just methods at the end of the day, whereas scopes are special; they have certain behavior. And I've never really known of the behavior beyond the fact that they get implemented in a different way. And so I was never really sold on them. And they're different enough from methods, and I know methods well. So I'm like, let's use the normal stuff where we can. The one thing that's really interesting, though, is the returning nil that was mentioned in this blog post. If you return nil in a scope, it will handle that for you. Whereas all of my query objects have a like, well, if this thing applies, then scope dot or relation dot where blah, blah, blah, else return relation unchanged. And the fact that that natively exists within scope is interesting enough to make me reconsider my stance on scopes versus class methods. I think I'm still doing class method. But it is an interesting consideration that I was unaware of before. STEPH: Yeah, it's an interesting point. I hadn't really considered as much whether I'm defining a class-level method versus a scope in this particular case. And I also didn't realize that scopes handle that nil case for you. That was one of the other things that I learned by reading through this blog post. I was like, oh, that is a nicety. Like, that is something that I get for free. So I agree. I think this is one of those things that I like enough that I'd really like to try it out more and then see how it goes and start to incorporate it into my process. Because this feels like one of those common areas of where I get to it, and I'm like, how do I do this again? And yield_self was just complicated enough in terms of then using the fancy method method to then be able to call the method that I want that I was like, I don't remember how to do this. I had to look it up each time. But including this module with extending and then being able to use scopes that way feels like something that would be intuitive for me that then I could just pick up and run with each time. CHRIS: If it helps, you can use then instead of yield_self because we did upgrade our Ruby a while back to have that change. But I don't think that actually solves the thing that you're describing. I'd have liked the ampersand method and then simple method name magic incantation that is part of the thing that Derek wrote up. I do use it when I write query objects, but I have to think about it or look it up each time and be like, how do I do that? All right, that's how I do that. STEPH: Yeah, that's one of the things that I really appreciate is how often folks will go back and update blog posts, or they will add an addition to them to say, "Hey, there's something new that came out that then is still relevant to this topic." So then you can read through it and see the latest and the greatest. It's a really nice touch to a number of our blog posts. But yeah, that's what was on my mind regarding query objects. What else is going on in your world? CHRIS: I have this growing feeling that I don't quite know what to do with. I think I've talked about it across some of our conversations in the world of observability. But broadly, I'm starting to like...I feel like my brain has shifted, and I now see the world slightly differently, and I can't go back. But I also don't know how to stick the landing and complete this transition in my brain. So it's basically everything's an event stream; this feels true. That's life. The arrow of time goes in one direction as far as I understand it. And I'm now starting to see it manifest in the code that we're writing. Like, we have code to log things, and we have places where we want to log more intentionally. Then occasionally, we send stuff off to Sentry. And Sentry tells us when there are errors, that's great. But in a lot of places, we have both. Like, we will warn about something happening, and we'll send that to the logs because we want to have that in the logs, which is basically the whole history of what's happened. But we also have it in Sentry, but Sentry's version is just this expanded version that has a bunch more data about the user, and things, and the browser that they were in. But they're two variations on the same event. And then similarly, analytics is this, like, third leg of well, this thing happened, and we want to know about it in the context. And what's been really interesting is we're working with a tool called Customer.io, which is an omnichannel communication whiz-bang adventure. For us, it does email, SMS, and push notifications. And it's integrated into our segment pipeline, so events flow in, events and users essentially. So we have those two different primitives within it. And then within it, we can say like, when a user does X, then send them an email with this copy. As an aside, Customer.io is a fantastic platform. I'm super-duper impressed. We went through a search for a tool like it, and we ended up on a lot of sales demos with folks that were like, hey, so yeah, starting point is $25,000 per year. And, you know, we can talk about it, but it's only going to go up from there when we talk about it, just to be clear. And it's a year minimum contract, and you're going to love it. And we're like, you do have impressive platforms, but okay, that's a bunch. And then, we found Customer.io, and it's month-by-month pricing. And it had a surprisingly complete feature set. So overall, I've been super impressed with Customer.io and everything that they've afforded. But now that I'm seeing it, I kind of want to move everything into that world where like, Customer.io allows non-engineer team members to interact with that event stream and then make things happen. And that's what we're doing all the time. But I'm at that point where I think I see the thing that I want, but I have no idea how to get there. And it might not even be tractable either. There's the wonderful Turning the Database Inside Out talk, which describes how everything is an event stream. And what if we actually were to structure things that way and do materialized views on top of it? And the actual UI that you're looking at is a materialized view on top of the database, which is a materialized view on top of that event stream. So I'm mostly in this, like, I want to figure this out. I want to start to unify all this stuff. And analytics pipes to one tool that gets a version of this event stream, and our logs are just another, and our error system is another variation on it. But they're all sort of sampling from that one event stream. But I have no idea how to do that. And then when you have a database, then you're like, well, that's also just a static representation of a point in time, which is the opposite of an event stream. So what do you do there? So there are folks out there that are doing good thinking on this. So I'm going to keep my ear to the ground and try and see what's everybody thinking on this front? But I can't shake the feeling that there's something here that I'm missing that I want to stitch together. STEPH: I'm intrigued on how to take this further because everything you're saying resonates in terms of having these event streams that you're working with. But yet, I can't mentally replace that with the existing model that I have in my mind of where there are still certain ideas of records or things that exist in the world. And I want to encapsulate the data and store that in the database. And maybe I look it up based on when it happens; maybe I don't. Maybe I'm looking it up by something completely different. So yeah, I'm also intrigued by your thoughts, but I'm also not sure where to take it. Who are some of the folks that are doing some of the thinking in this area that you're interested in, or where might you look next? CHRIS: There's the Kafka world of we have an event log, and then we're processing on top of that, and we're building stream processing engines as the core. They seem to be closest to the Turning the Database Inside Out talk that I was thinking or that I mentioned earlier. There's also the idea of CRDTs, which are Conflict-free Replicated Data Types, which are really interesting. I see them used particularly in real-time application. So it's this other tool, but they are basically event logs. And then you can communicate them well and have two different people working on something collaboratively. And these event logs then have a natural way to come together and produce a common version of the document on either end. That's at least my loose understanding of it, but it seems like a variation on this theme. So I've been looking at that a little bit. But again, I can't see how to map that to like, but I know how to make a Rails app with a Postgres database. And I think I'm reasonably capable at it, or at least I've been able to produce things that are useful to humans using it. And so it feels like there is this pretty large gap. Because what makes sense in my head is if you follow this all the way, it fundamentally re-architects everything. And so that's A, scary, and B; I have no idea how to get there, but I am intrigued. Like, I feel like there's something there. There's also Datomic is the other thing that comes to mind, which is a database engine in the Clojure world that stores the versions of things over time; that idea of the user is active. It's like, well, yeah, but when were they not? That's an event. That transition is an event that Postgres does not maintain at this point. And so, all I know is that the user is active. Maybe I store a timestamp because I'm thinking proactively about this. But Datomic is like no, no, fundamentally, as a primitive in this database; that's how we organize and think about stuff. And I know I've talked about Datomic on here before. So I've circled around these ideas before. And I'm pretty sure I'm just going to spend a couple of minutes circling and then stop because I have no idea how to connect the dots. [laughs] But I want to figure this out. STEPH: I have not worked with Kafka. But one of the main benefits I understand with Kafka is that by storing everything as a stream, you're never going to lose like a message. So if you are sending a message to another system and then that message gets lost in transit, you don't actually know if it got acknowledged or what happened with it, and replaying is really hard. Where do you pick up again? While using something like Kafka, you know exactly what you sent last, and then you're not going to have that uncertainty as to what messages went through and which ones didn't. And then the ability to replay is so important. I'm curious, as you continue to explore this, do you have a particular pain point in mind that you'd like to see improve? Or is it more just like, this seems like a really cool, novel idea; how can I incorporate more of this into my world? CHRIS: I think it's the latter. But I think the thing that I keep feeling is we keep going back and re-instrumenting versions of this. We're adding more logging or more analytics events over the wire or other things. But then, as I send these analytics events over the wire, we have Mixpanel downstream as an analytics visualization and workflow tool or Customer.io. At this point, those applications, I think, have a richer understanding of our users than our core Rails app. And something about that feels wrong to me. We're also streaming everything that goes through segment to S3 because I had a realization of this a while back. I'm like, that event stream is very interesting. I don't want to lose it. I'm going to put it somewhere that I get to keep it. So even if we move off of either Mixpanel or Customer.io or any of those other platforms, we still have our data. That's our data, and we're going to hold on to it. But interestingly, Customer.io, when it sends a message, will push an event back into segments. So it's like doubly connected to segment, which is managing this sort of event bus of data. And so Mixpanel then gets an even richer set there, and the Rails app is like, I'm cool. I'm still hanging out, and I'm doing stuff; it's fine. But the fact that the Rails app is fundamentally less aware of the things that have happened is really interesting to me. And I am not running into issues with it, but I do feel odd about it. STEPH: That touched on a theme that is interesting to me, the idea that I hadn't really considered it in those terms. But yeah, our application provides the tool in which people can interact with. But then we outsource the behavior analysis of our users and understanding what that flow is and what they're going through. I hadn't really thought about those concrete terms and where someone else owns the behavior of our users, but yet we own all the interaction points. And then we really need both to then make decisions about features and things that we're building next. But that also feels like building a whole new product, that behavior analysis portion of it, so it's interesting. My consulting brain is going wild at the moment between like, yeah, it would be great to own that. But that the other time if there's this other service that has already built that product and they're doing it super well, then let's keep letting them manage that portion of our business until we really need to bring it in-house. Because then we need to incorporate it more into our application itself so then we can surface things to the user. That's the part where then I get really interested, or that's the pain point that I could see is if we wanted more of that behavior analysis, that then we want to surface that in the app, then always having to go to a third-party would start to feel tedious or could feel more brittle. CHRIS: Yeah, I'm definitely 100% on not rebuilding Mixpanel in our app and being okay with the fact that we're sending that. Again, the thing that I did to make myself feel better about this is stream the data to S3 so that I have a version of it. And if we want to rebuild the data warehouse down the road to build some sort of machine learning data pipeline thing, we've got some raw data to work with. But I'm noticing lots of places where we're transforming a side effect, a behavior that we have in the system to dispatching an event. And so right now, we have a bunch of stuff that we pipe over to Slack to inform our admin team, hey, this thing happened. You should probably intervene. But I'm looking at that, and we're doing it directly because we can control the message in Slack a little bit better. But I had this thought in the back of my mind; it's like, could we just send that as an event, and then some downstream tool can configure messages and alerts into Slack? Because then the admin team could actually instrument this themselves. And they could be like; we are no longer interested in this event. Users seem fine on that front. But we do care about this new event. And all we need to do as the engineering team is properly instrument all of that event stream tapping. Every event just needs to get piped over. And then lots of powerful tools downstream from that that can allow different consumers of that data to do things, and broadly, that dispatch events, consume them on the other side, do fun stuff. That's the story. That's the dream. But I don't know; I can't connect all the dots. It's probably going to take me a couple of weeks to connect all these dots, or maybe years, or maybe my entire career, something like that. But, I don't know, I'm going to keep trying. STEPH: This feels like a fun startup narrative, though, where you start by building the thing that people can interact with. As more people start to interact with it, how do we start giving more of our team the ability to then manage the product that then all of these users are interacting with? And then that's the part that you start optimizing for. So there are always different interesting bits when you talk about the different stages of Sagewell, and like, what's the thing you're optimizing for? And I'm sure it's still heavily users. But now there's also this addition of we are also optimizing for our team to now manage the product. CHRIS: Yes, you're 100%. You're spot on there. We have definitely joked internally about spinning out a small company to build this analytics alerting tool [laughs] but obviously not going to do that because that's a distraction. And it is interesting, like, we want to build for the users the best thing that we can and where the admin team fits within that. To me, they're very much customers of engineering. Our job is to build the thing for the users but also, to be honest, we have to build a thing for the admins to support the users and exactly where that falls. Like, you and I have talked a handful of times about maybe the admin isn't as polished in design as other things. But it's definitely tested because that's a critical part of how this application works. Maybe not directly for a user but one step removed for a user, so it matters. Absolutely we're writing tests to cover that behavior. And so yeah, those trade-offs are always interesting to me and exploring that space. But 100%: our admin team are core customers of the work that we're doing in engineering. And we try and stay very close and very friendly with them. STEPH: Yeah, I really appreciate how you're framing that. And I very much agree and believe with you that our admin users are incredibly important. CHRIS: Well, thank you. Yeah, we're trying over here. But yeah, I think I can wrap up that segment of me rambling about ideas that are half-formed in my mind but hopefully are directionally important. Anyway, what else is up with you? STEPH: So, not that long ago, I asked you a question around how the heck to manage themes that I have going on. So we've talked about lots of fun productivity things around managing to-dos, and emails, and all that stuff. And my latest one is thinking about, like, I have a theme that I want to focus on, maybe it's this week, maybe it's for a couple of months. And how do I capture that and surface it to myself and see that I'm making progress on that? And I don't have an answer to that. But I do have a theme that I wanted to share. And the one that I'm currently focused on is building up management skills and team lead skills. That is something that I'm focused on at the moment and partially because I was inspired to read the book Resilient Management written by Lara Hogan. And so I think that is what has really set the idea. But as I picked up the book, I was like, this is a really great book, and I'd really like to share some of this. And then so that grew into like, well, let's just go ahead and make this a theme where I'm learning this, and I'm sharing this with everyone else. So along that note, I figured I would share that here. So we use Basecamp at thoughtbot. And so, I've been sharing some Basecamp posts around what I'm learning in each chapter. But to bring some of that knowledge here as well, some of the cool stuff that I have learned so far...and this is just still very early on in the book. There are a couple of different topics that Laura covers in the first chapter, and one of them is humans' core needs at work. And then there's also the concept of meeting your team, some really good questions that you can ask during your first one-on-one to get to know the person that then you're going to be managing. The part that really resonated with me and something that I would like to then coach myself to try is helping the team get to know you. So as a manager, not only are you going out of your way to really get to know that person, but how are you then helping them get to know you as well? Because then that's really going to help set that relationship in regards of they know what kind of things that you're optimizing for. Maybe you're optimizing for a deadline, or for business goals, or maybe it's for transparency, or maybe it would be helpful to communicate to someone that you're managing to say, "Hey, I'm trying some new management techniques. Let me know how this goes." [chuckles] So there's a healthier relationship of not only are you learning them, but they're also learning you. So some of the questions that Laura includes as examples as something that you can share with your team is what do you optimize for in your role? So is it that you're optimizing for specific financial goals or building up teammates? Or maybe it's collaboration, so you're really looking for opportunities for people to pair together. What do you want your teammates to lean on you for? I really liked that question. Like, what are some of the areas that bring you joy or something that you feel really skilled in that then you want people to come to you for? Because that's something that before I was a manager...but it's just as you are growing as a developer, that's such a great question of like, what do you want to be known for? What do you want to be that thing that when people think of, they're like, oh, you should go see Chris about this, or you should go see Steph about this? And two other good questions include what are your work styles and preferences? And what management skills are you currently working on learning or improving? So I really like this concept of how can I share more of myself? And the great thing about this book that I'm learning too is while it is geared towards people that are managers, I think it's so wonderful for people who are non-managers or aspiring managers to read this as well because then it can help you manage whoever's managing you. So then that way, you can have some upward management. So we had recent conversations around when you are new to a team and getting used to a manager, or maybe if you're a junior, you have to take a lot of self-advocacy into your role to make sure things are going well. And I think this book does a really good job for people that are looking to not only manage others but also manage themselves and manage upward. So that's some of the journeys from the first chapter. I'll keep you posted on the other chapters as I'm learning more. And yeah, if anybody hasn't read this book or if you're interested, I highly recommend it. I'll make sure to include a link in the show notes. CHRIS: That was just the first chapter? STEPH: Yeah, that was just the first chapter. CHRIS: My goodness. STEPH: And I shortened it drastically. [laughs] CHRIS: Okay. All right, off to the races. But I think the summary that you gave there, particularly these are true when you're managing folks but also to manage yourself and to manage up, like, this is relevant to everyone in some capacity in some shape or form. And so that feels very true. STEPH: I will include one more fun aspect from the book, and that's circling back to the humans' core needs at work. And she references Paloma Medina, a coach, and trainer who came up with this acronym. The acronym is BICEPS, and it stands for belonging, improvement, choice, equality, predictability, and significance. And then details how each of those are important to us in our work and how when one of those feels threatened, then that can lead to some problems at work or just even in our personal life. But the fun example that she gave was not when there's a huge restructuring of the organization and things like that are going on as being the most concerning in terms of how many of these needs are going to be threatened or become vulnerable. But changing where someone sits at work can actually hit all of these, and it can threaten each of these needs. And it made me think, oh, cool, plus-one for being remote because we can sit wherever we want. [laughs] But that was a really fun example of how someone's needs at work, I mean, just moving their desk, which resonates, too, because I've heard that from other people. Some of the friends that I have that work in more of a People Ops role talk about when they had to shift people around how that caused so much grief. And they were just shocked that it caused so much grief. And this explains why that can be such a big deal. So that was a fun example to read through. CHRIS: I'm now having flashbacks to times where I was like, oh, I love my spot in the office. I love the people I'm sitting with. And then there was that day, and I had to move. Yeah, no, those were days. This is true. STEPH: It triggered all the core BICEPS, all the things that you need to work. It threatened all of them. Or it could have improved them; who knows? CHRIS: There were definitely those as well, yeah. Although I think it's harder to know that it's going to be great on the way in, so it's mostly negative. I think it has that weird bias because you're like, this was a thing, I knew it. I at least understood it. And then you're in a new space, and you're like, I don't know, is this going to be terrible or great? I mean, hopefully, it's only great because you work with great people, and it's a great office. [laughs] But, like, the unknown, you're moving into the unknown, and so I think it has an inherent at least questioning bias to it. STEPH: Agreed. On that note, shall we wrap up? CHRIS: Let's wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

The foundational technology for Muse 2 is local-first sync, which draws from over a decade of computer science research on CRDTs. Mark, Adam Wiggins, and Adam Wulf get technical to describe the Muse sync technology architecture in detail. Topics include the difference between transactional, blob, and ephemeral data; the “atoms” concept inspired by Datomic; Protocol Buffers; and the user's data as a bag of edits. Plus: why sync is a powerful substrate for end-user programming. @MuseAppHQ hello@museapp.com Show notes Adam Wulf @adamwulf Fantastical Loose Leaf Wulf's iOS ink libraries OpenGL Bézier curves Houston Muse 2.0 launches May 24 Metamuse episode on local-first software Core Data Pocket Clue, Wunderlist CouchDB, Firebase Adam's writeup on sync technologies from 2014 Evernote Pixelpusher Slow Software CRDTs, operational transform Automerge Actual Budget last write wins Actual open source hybrid logical clock, vector clock CloudKit lazy loading API versioning Protocol Buffers Wulf's article on atoms Datomic “put a UUID and a version number on everything” Swift property wrappers functional reactive programming Sourcery Sentry HDD indicator light Muse job post for a local-first engineer Local-first day at ECOOP 2022

00:00:15 Introduction00:01:24 Gaming setups00:12:25 News 00:12:27 I was wrong, CRDTs are the future 00:17:18 How we lost 54k Github stars 00:21:10 DALL-E  00:25:45 Inside the Longest Atlassian Outage of All Time 00:35:11: Sponsor00:36:22 Book of the Show 00:36:38 Indie Boardgame Designers Podcast 00:37:24 The Laundry Files 00:40:35 Tool of the Show 00:40:39 Zapier 00:42:21 Earthly 00:46:46 Funding open-source projects01:19:44 How to get funding for open-source projects01:22:47 Farewells  Resources mentioned in this episode:Media: The King of Kong: A Fistful of Quarters (2017) Class Action Park (2020) Indie Boardgame Designers Podcast: https://indieboardgamedesigners.com/ GitHub Stars Won't Pay Your Rent: https://medium.com/@kitze/github-stars-wont-pay-your-rent-8b348e12baed News: I Was Wrong, CRDTs Are The Future: https://josephg.com/blog/crdts-are-the-future/ How We Lost 54k GitHub Stars: https://httpie.io/blog/stardust DALL-E: https://openai.com/blog/dall-e/ Inside the Longest Atlassian Outage of All Time: https://newsletter.pragmaticengineer.com/p/scoop-atlassian?s=r Books: Indie Board Game Designers Podcast The Laundry Files: https://amzn.to/3kdWWQg Tools: Zapier: https://zapier.com/ N8n: https://n8n.io/ Earthly: https://earthly.dev/ Adam Gordon Bell: Twitter: https://twitter.com/adamgordonbell Website: https://adamgordonbell.com/ LinkedIn: https://www.linkedin.com/in/adamgordonbell/ CoRecursive: https://corecursive.com/   If you've enjoyed this episode, you can listen to more on Programming Throwdown's website: https://www.programmingthrowdown.com/ Reach out to us via email: programmingthrowdown@gmail.com You can also follow Programming Throwdown on Facebook | Apple Podcasts | Spotify | Player.FM  Join the discussion on our DiscordHelp support Programming Throwdown through our Patreon★ Support this podcast on Patreon ★

Hello everyone! My name is Martin Kleppmann. I'm a researcher at the University of Cambridge. And I would like to tell you briefly about the technology called CRDTs. So, CRDT stands for Conflict-free Replicated Data Type. It's a type of data structure that you can use to build collaboration software. So think software like Google Docs for example. Or Figma. Or Trello. Or a TODO list that syncs between your computer and your phone. You can build this type of software using CRDTs. Read more: https://nurkiewicz.com/70 Get the new episode straight to your mailbox: https://nurkiewicz.com/newsletter

Another episode Beyond Chat. Again? Yes! A lot of cool things are going on! And today my guest is going to tell you how they're building awesome collaborative apps with Matrix. Le menu du jour is in la description ⏬ 00:00 Bonjour 00:29 CRDTs are easier to understand if you have seen them in practice. Let's start with a demo! 08:37 So, what are CRDTs? 10:05 What is Yjs and what is a provider for it? 12:20 Why do you see Matrix as a good fit for document/data storage and syncing? 16:32 Are we Maximised Widgets yet? 17:57 What would make widgets better? 18:47 How is matrix-CRDT funded (featuring the noisy microphone Jitsi notification) 20:27 Is matrix-CRDT used in a commercial project? If yes, what can you share about it? 27:02 What is SyncedStore? How different is it from Yjs? 29:07 Can you expand on local-first applications? 33:25 Final Thoughts? The paper on local-first applications: https://www.inkandswitch.com/local-first/ A talk Yousef recommends to go further on CRDTs: https://www.youtube.com/watch?v=yCcWpzY8dIA Matrix-CRDT

In this episode of the InfoQ podcast Charles Humble talks to Michael Perry about his book “The Art of Immutable Architecture”. They discuss topics including the eight fallacies of distributed computing: a set of assertions made by L Peter Deutsch and others at Sun Microsystems describing false assumptions that programmers new to distributed applications invariably make. Other topics include Pat Helland's paper “Immutability Changes Everything”, Eric Brewer's CAP Theorem, eventual consistency, location independent identity, and CRDTs. They also discuss how the approach to building distributed systems advocated by Perry could be introduced to a real-world enterprise app that needs to integrate with mutable downstream systems. Read a transcript of this interview: https://bit.ly/3wWMNvp Subscribe to our newsletters: - The InfoQ weekly newsletter: bit.ly/24x3IVq - The Software Architects' Newsletter [monthly]: www.infoq.com/software-architects-newsletter/ Upcoming Virtual Events - events.infoq.com/ InfoQ Live: https://live.infoq.com/ - August 17, 2021 - September 21, 2021 - October 19, 2021 QCon Plus: https://plus.qconferences.com/ - November 1-5, 2021 Follow InfoQ: - Twitter: twitter.com/InfoQ - LinkedIn: www.linkedin.com/company/infoq - Facebook: bit.ly/2jmlyG8 - Instagram: @infoqdotcom - Youtube: www.youtube.com/infoq

Show notes:Links:Mike MondragonCRDTShip of TheseusExceptional CreaturesShiba Inu Full Transcript:Ben:I'm just gonna dive on in there. I'm so eager. I'm so excited. It's actually weird because Starr is the one that typically starts us off. Josh:Yeah. I thought we were just going to start with our just general banter, and then not introduce the guest until 30 minutes later.Ben:By the way.Josh:It is also our tradition.Ben:Yeah. Well we're getting better at this thing.Josh:Where we say, "Oh, by the way, if Starr doesn't sound like Starr..."Ben:Right, yes. Today Starr doesn't sound like Starr because today's star is Mike Mondragon instead. Welcome Mike.Josh:Hey Mike.Mike:Hey.Ben:Mike is a long time friend of the show, and friend of the founders. Actually, Mike, how long have we known each other? It's been at least 10, maybe 15 years?Mike:Probably 2007 Seattle RB.Ben:Okay.Josh:Yeah. I was going to say you two have known each other much longer than I've even known Ben.Ben:Yeah.Josh:So you go back.Ben:Way back.Mike:Yep.Josh:Yeah.Ben:Yeah.Josh:Because I think Ben and I met in 2009.Ben:Mm-hmm (affirmative).Josh:Or something.Mike:Okay.Ben:Yeah, Mike and I have been hanging out for a long time.Mike:Yeah.Ben:We've known each other through many, many different jobs, and contracts, and so on. It's been awesome.Josh:Yeah, Mike, I feel like I've heard your name since... Yeah, for the last, at least, 10 years just working with Ben. You've always been in the background. And we've realized this is the first time we've actually met face to face, which is crazy. But it's great to... Yeah.Mike:Yeah.Josh:... have a face to put with the little... What is it, a cat avatar? Is a cat in your avatar? You've had that avatar for a really long time I feel like.Mike:Yeah, that's Wallace.Josh:Okay.Mike:So I'm Mond on GitHub and Twitter, and that cat avatar is our tuxedo cat, Wallace. And he is geriatric now. Hopefully he'll live another year. And if you remember in that era of Ruby, all of the Japanese Rubyists had cat icons. And so that was... I don't know. That's why Wallace is my icon.Josh:Yeah. Nice.Ben:So, so do Wallace and Goripav know each other?Mike:No, no, they don't. They're like best friends, right? They had to have met at Seattle RB.Ben:Yeah. Internet friends.Mike:Internet friends, yeah.Ben:Yeah. So, Mike is old school Ruby, way back, way back, yeah. But the other funny thing about the old Rubyists, all those Japanese Rubyists, I remember from RubyConf Denver... Was that 2007? Somewhere around there. I remember going to that and there were mats and a bunch of friends were sitting up at the front, and they all had these miniature laptops. I've never seen laptops so small. I don't know what they were, nine inch screens or something crazy.Mike:Mm-hmm (affirmative).Ben:I was like, "How do you even type on that thing?" But it's a thing. So I guess... I don't know. I haven't been to Japan.Mike:There are laptops that you could only get in Japan and they flash them with some sort of Linux probably.Ben:Yeah. Yeah.Mike:Mm-hmm (affirmative).Josh:Okay. I wonder how long it took them to compile C on there.Mike:Yeah. So, about the orbit with the founders. So, I think I'd put it in my notes that I... And I consider myself a sliver of a Honeybadger in that I did have a conversation with Ben about joining the company. And then in 2017, I did do a little contracting with you guys, which is ironic in that... So we're probably going to talk about cryptocurrencies and Bitcoin. So the Bitcoin protocol is, essentially, on a four-year timer. And in 2017 was the last time that we were building up to, I guess, an explosive end to that cycle. And I had just been working at Salesforce at Desk.com, And I left because of Bitcoin. And then this year, four years later, I, again, just left Salesforce, but I just left from Heroku. And I didn't leave so much because of Bitcoin, I just got a better opportunity, and I'm a principal engineer at Okta, and I'm in the developer experience working on SDKs, primarily, the Golang SDK.Mike:So I think one of the things that they were happy about was that I had experience carrying the pager, and knowing what that's like, and they wanted to have an experienced engineer that would have empathy for the engineers to main the SDK. So I'm really excited to be here, because I'm not going to be carrying the pager, and it is the fun programming. What I imagine, listening to the founders, about the kind of fun programming that you guys get to do, working with different languages and whatnot. So, obviously right now, I'm starting out with Golang. We don't have a Ruby SDK, because OmniAuth provider is the thing that most people use. But, there's also PHP, and some Java, so I'm just looking forward to being able to do a bunch of different languages.Josh:Yeah. That's awesome. Yeah. We don't know anything about SDK teams, Honeybadger. But yeah, it sounds like we have very similar jobs at the moment. So that's cool. We'll have to trade tips at some point. Yeah.Ben:Yeah, I'm excited that you're there, because I'm definitely going to hit you up on the SAML stuff, because SAML's a pain in the tuchus yeah, I'm sure you'll have some insights from your time there.Mike:Well, that was how I was even open-minded to talking to Okta, was the recruiter had contacted me and I think actually it was the recruiter... I don't know the structure of how this works, but a lot of companies have a prospecting recruiter. And I think that a veteran oriented prospecting recruiter contacted me. And so being a veteran, I'll usually entertain those cold calls. And so then when I was at Desk, I wrote... So Desk was a big Rails monolith. I wrote a microservice to break some of the SSO off of the monolith itself. And in writing the API documentation that was on desk.com, I actually used Okta as one of the examples as a SSO identity provider using SAML. So yeah, I have had a little bit of experience from the outside of Okta with SAML. And so maybe I'll have more experience here to answer your questions.Ben:Yeah. We'll have to have you back and we can just do a whole hour on that. It's a fun world.Josh:After we do an hour on SDKs.Ben:Yeah, and your code that you wrote for us still lives on in Honeybadger.Josh:Yeah. Was it the webpack? That was some of the work, right?Ben:Some of it, yeah.Mike:Yep.Josh:Yeah.Ben:And some GitHub integration work.Josh:And the integrations, yeah.Mike:Yeah, well if I remember correctly with the GitHub integration, I did do some GitHub integration, and it tickled your enthusiasm, Ben, and then I think you went in and like refactored that a little bit.Ben:Well, if you have a monolith like Redo that's been around for as long as ours has, things don't... It's like, what was that Theseus' ship, it's goes around the world but you replace things as it goes, and it's never the same app, right?Mike:Yeah, that's the thing, we had discussed this in the prelude around just software engineering in general and how hard it is to maintain a monolith, especially as a company grows and as developers come rolling into a project, you get all of these... Over time you get engineers with different goals, different techniques, different styles of touching your code base, to the point that it becomes very hard to maintain a project. And I think, I don't know if we're going to talk about Heroku at all, but I think that Heroku suffers from a little bit of that, where there's very few original Heroku that are involved in the runtime at least. And I just came from being on the runtime in the control plane. And, definitely, the code base there is... There's maybe one or two people that are still around that have touched that code base from the beginning.Ben:Yeah, let's dive into that, because that's fascinating to me. I know that there's been chatter on Twitter recently that people feel that Heroku is stagnated. That they haven't really brought a lot of innovative stuff to market recently. I remember, actually a funny story, I'm going to tell it myself. I can't remember what year this was, it were way... I don't know, I don't know, early 2000s. I was sitting as part of a focus group, and I can't reveal a lot of information because secrecy and stuff. But anyway, I was part of this focus group and I was asked as part of this group, what as a developer working on Ruby applications and Rails applications, what I thought about this new thing called Heroku. And had it explained to me, "Oh, you just get push", and "Blah, blah, blah", and I poo-pooed the idea. I was like, "Nah, I'm not interested", because I already know how to deploy stuff. I've got Mongrel, I got a DVS.Josh:Say Mongrel.Ben:I know how to use SEP, why do I need this? Like Math, never going to catch on. And so don't follow me for investing advice.Mike:Yeah, totally.Josh:I got my Linodes.Mike:Yeah. Or even back then, I wrote all of my own chef, so I got my own recipes I can-Ben:Right, exactly.Mike:... bare metal at will.Ben:Exactly. So, what do you think, you've been at Heroku, you've seen this process of people having to maintain this code base over a long period of time. What are some tips for people who might be a little earlier on the process? Looking down the road, what do you suggest people think about for having a more maintainable application?Mike:That's interesting. I really think that there is not one size fits all, and actually some of the things that are specific to Heroku, and actually to desk.com when I was there previously, that some of the issues actually stem from Salesforce culture and the way that Salesforce manages its businesses. And so, I guess the thing that I've always liked about Rails, specifically, is that the conventions that are used in Rails, you can drop an experienced Rails developer pretty much into any Rails app and they're going to know the basic conventions. And that saves you so much time to ramping up and bringing your experience into a project. Whereas when you get into bespoke software, then you run into well what were the architectural design patterns 10 years ago compared to now? How much drift has there been in libraries and the language, depending.Mike:And so that is... I don't... That's a very hard question to nail down in a specific way. I would just say in spit balling this, conventions are very important, I would say. So as long as you have a conventions using a framework, then I think that you'll get to go a long ways. However, if you start to use a framework, then you get the everything is a nail and I'm going to use my hammer framework on that. Which is its own thing that I've seen in Ruby, where if you start a project with Rails, I don't think everybody realizes this, but you are essentially going to be doing a type of software development that is in the mindset of Basecamp, right? And if you have an app that is not quite like Basecamp, and then you start to try to extending Rails to do something different, then you're going to start running into issues. And I think that... It makes me sad when I hear people talk poorly about Rails, because oftentimes people are just pushing it into a direction that it's not built to do. Whether they're, like in the old days, like monkey-patching libraries, or whatnot.Ben:Yeah, I think we saw that with the rise of Elixir and Phoenix, right? José just got frustrated with wanting to do some real time stuff. And that really wasn't the wheelhouse for Rails, right? And so he went and built Elixir and Phoenix, and built on top of that. And that became a better hammer for that particular nail than Rails, right? So now if you come into a new project and you're like, "Well, I'm going to do a lot of highly concurrent stuff", well, okay, maybe Rails isn't the best solution. Maybe you should go look at Elixir and Phoenix instead.Mike:Yeah. Yeah. So, with Heroku, I just want to say that it was so awesome to work at Heroku, and the day that I got a job offer to work there, it was like... I still, if I'm having a bad day, I still think about that, and the... I've never used hard drugs, but I would think that somebody that was cocaine high, that's probably what I was feeling when I got the offer from Heroku. I started using Heroku in 2009, and it has a story within our community, it's highly respected. And so I just want to say that I still think very highly of Heroku, and if I was to be doing just a throwaway project, and I just want to write some code and do git push main, or git push Heroku main, then I would definitely do that.Mike:And we were... And I'm not very experienced with the other kinds of competitors right now. I think, like you pointed him out, is it Vercel and Render?Ben:Render. Mm-hmm (affirmative).Mike:Yeah. So I can't really speak to them. I can really just speak to Heroku and some of the very specific things that go on there. I think one of the issues that Heroku suffers from is not the technology itself, but just the Salesforce environment. Because at Salesforce, everything eventually has to be blue, right? And so, Heroku, I don't think they ever could really figure out the right thing to do with Heroku. As well as, the other thing about enterprise software is that if I'm selling Salesforce service cloud or whatever, I'm selling, essentially, I'm selling seats of software licenses. And there's no big margin in selling Compute, because if I'm buying Compute, I expect to be using that.Mike:And so, as a salesperson, I'm not incented to sell Heroku that much because there's just not margins for me in the incentive structure that they have at sales within Salesforce. So I think that's the biggest thing that Heroku has going against it, is that it's living in a Salesforce environment. And as, I guess, a owner of Salesforce being that I have Salesforce stock, I would hope that they would maximize their profits and actually sell Heroku. Who knows, maybe a bunch of developers get together and actually buy the brand and spin that off. That would be the best thing, because I think that Salesforce would probably realize a lot more value out of Heroku just by doing that, even if there's some sort of profit sharing, and then not have to deal with all the other things.Ben:Yeah, that's really interesting. Yeah. The thing about billing, and then selling per user, versus the compute- That's definitely a different world. It's a totally different mindset. And I think Josh that we have now been given a directive step. We should acquire Heroku as part of Honeybadger.Josh:I was going to say, maybe we can acquire it with all of our Doge profits in five or 10 years from now.Mike:Well, yeah. Somebody spin a Heroku coin, a ERC20 token on Ethereum and get everybody to dump their Ethereum into this token.Josh:Mm-hmm (affirmative).Mike:Get that pot of money together. And then that is the Heroku Foundation. Yeah, exactly.Josh:Okay, yeah.Mike:The Heroku Foundation that buys the Heroku brand. I know that we're laughing about it, but actually this is what is possible today. And, I was telling Ben... Well, let me just say a couple of things about the FounderQuest and how it relates to me, is I've been listening to FounderQuest from the first episode, and I'm an only child, and I like to listen to podcasts. So I'll be on my afternoon walk, and I'll be hearing you guys talk, and I'm having this conversation along with you guys listening to the podcasts.Mike:And so, I think, in January, you guys were talking about, or maybe Ben was talking about, $30,000 Bitcoin, and you guys just had your yucks and laughs about it. And it actually made me think critically about this, because I've been involved with Bitcoin since about 2012, and it's like, "Do I have a tinfoil hat on?" Or what do I think? And so, I'm not joking about this, listening to you guys actually has helped me concretely come up with how I feel about this. And first off, I think, I'm bullish on technology. And this is the first epiphany that I had, is all of us have had a career close to Linux, close to Ruby, building backend services, close to virtualization and orchestration. Fortunately, that's been my interest, and fortunately that's been where our industry has gone. And so, when Bitcoin came out, as technologists, all you ever hear, if you don't know anything about Bitcoin, you just hear currency. And you're thinking internet money, you're not thinking about this as a technologist.Mike:And so that was the thing. I wish that Bitcoin had been talked about as a platform, or a framework.Josh:Mm-hmm (affirmative).Mike:And not even called it coin. Because that confuses the issue-Josh:The whole coin thing, just... Yeah.Mike:Yeah, totally. And mining the metaphors-Josh:That alone.Mike:... just totally throws everything off. Because we are talking, we're laughing about it, but this is really possible today. We could come up with a Foundation to buy Heroku with a cryptocurrency, and it would... Yeah. So that's one thing that Ben helped me realize in my thinking around Bitcoin and cryptocurrencies. And I think I'm just bullish on technology. And so to me, again, across our career, there's been so much change. And why would we look at Bitcoin and cryptocurrencies any differently than any other kind of technology? Even a hundred dollar bill with all the holograms on it, that is a kind of financial technology. And so we're just talking about a digital technology, we're not talking about coins I guess.Josh:That's the appeal, a lot of the Altcoins, right? They give everyone a way to invest in those companies, whereas before you would have to... Whatever, be an accredited investor or something to be able to get involved. Is that part of the appeal? I'm probably showing what I know about crypto, which is very little, but I'm excited to... Yeah, maybe you can...Mike:Yeah. Yeah, so I feel like these projects are... I'm not a VC, and I'm not an insider, but from what I can see from afar, in Silicon Valley there's a close group of people that have access to all of these ideas. And there's Angel clubs, and VC clubs, and whatnot, that are funding these startups. And to me, I feel like these crypto projects are the same kind of thing, except for they're just available to the public. And so, I think if I was speaking to another technologist that was interested in cryptocurrencies, is you probably need to get your hands on some of the technology in order to get experience with it.Mike:And so if that means you figure out how to maybe mine some coin on your laptop, or whatever, or you actually pay for it, you should at least have some in your possession, and at least learn about the custodial part of it. Also, there's different software libraries now to actually do programming against it, and platforms, I believe. So that'd be another way to at least tickle your curiosity, is by actually touching the technology and not thinking about the value. So yeah.Ben:Yeah. That, to me, that's one of the most interesting things about the whole coin thing. My younger son is really interested in the crypto space, in the coin and in the other parts of a distributed ledger, and what does that mean, and how does that work? And before I heard about NFTs, he was talking about NFTs. And so it's really interesting to me to see this coming from him. Just yesterday, we had a conversation about CRDTs, right? Because we're talking about how do you merge transactions that are happening in distributed fashion? Right? I was like, "Oh yeah", and it's so weird to have my teenage sons' world colliding with my world in this way.Josh:Yeah.Ben:But it's a lot of fun. And I've got to say, Mike, I got to give you back some credit, talking about the whole coin thing. As you've heard, we're pretty coin skeptical here at Honeybadger, the Founders, but you made a comment in our pre-show conversation. And maybe you didn't make this explicitly, but maybe it's just a way that I heard it. But I think... Well what I heard was, and maybe you actually said this, was basically think about this like an index fund, right? You put dollar cost to averaging, right? You put some money into coin, you put a little bit, it's not going to be your whole portfolio, right? But you don't treat it like a gamble, and you just treat it like an investment, like you would other things that may appreciate in value. And of course you may not.Ben:And so, as a result, I decided, "Okay, I can do that. I can put a little bit of my portfolio into coins". So just this week, and this is the funny part, just this week-Josh:I'm just finding this out now, by the way.Ben:Yeah, yeah. Josh is like... I told my wife about this last night and she was like, "What's Josh going to say?" "Like, I don't know". So anyway, just this week I put a little bit of money into Bitcoin and Ethereum. And that was... When did Elon do his thing about Bitcoin? Was that Thursday morning?Josh:Oh yeah.Ben:I bought, two hours before Elon did his thing, and Bitcoin lost 15% of its value.Mike:That's awesome.Ben:I'm like, "It's okay. It's okay, I'm just putting-Josh:Yeah, you don't sell, it doesn't matter.Mike:What was your emotion? What was your emotion?Ben:Yeah, totally. Yeah. In fact, my first buy, I used Coinbase. And Coinbase was like, "Oh, do you want to do this periodically?" I'm like, "Yes, I do. Every month". Boom.Mike:Oh.Ben:I went ahead and set that up like so, yeah.Mike:Oh, I did not know you could do that.Ben:I'm in it to win it, man.Mike:You should get a hardware wallet. That's the next thing, is you need to learn how to handle your own custody, so-Josh:Right, yeah. You got to... Yeah.Mike:Not leave it on the exchange. Interesting.Josh:Get those hard drives.Mike:Yeah.Josh:Yeah. Ben's a veteran indexer though. So you can handle some dips. Some volatility.Ben:Yeah. Yeah.Josh:I actually, I did make some money off of Bitcoin back in the day, and probably if I would've just held onto it, I would've made a lot more, of course.Mike:Same.Josh:So I accidentally... Back, I don't know when this was, it was maybe five years ago or something, when Bitcoin was going through one of its first early hype cycles, and I was like, "I'll check it". I was learning about it, of course. And so I went and bought some and I think I ran a blockchain Elixir app that someone made, to see how the transactions work and stuff. Read some books on Bitcoin. But I bought some Bitcoin, I can't remember how much, but just left it. I think this was after Coinbase had launched, I'm pretty sure I bought it through Coinbase. But yeah, I just left it, and then that was when it was in the first huge push of Bitcoin where it went up to 20,000 or something. And I remembered that I had it, and I went and looked and oh yeah, I made five grand or something. I put hardly anything into it initially. So I forget what I actually bought with that money. I just sold it and it's like cool, free money.Mike:So you just sold it this year? Or you sold it...Josh:No, I sold it back-Mike:In 17?Josh:I think I sold it at 20... Yeah, this would have been at 17 that I actually sold it, probably.Mike:Did you report it on your taxes, your capital gains?Josh:I did, yes. Yeah, I did.Ben:That's the benefit of having an accountant, because your accountant reminds you, "You know what? You did have some Bitcoin transactions, you should probably look at those".Josh:Can I say on here that I actually put some of it through a Bitcoin tumbler though, just to see how those work?Mike:Yeah, I mean...Josh:And that was a very small amount of money, but I didn't actually report that on my taxes. Because I think I actually forgot where it was or something.Ben:You'll have to explain what a Bitcoin tumbler is.Josh:So a Bitcoin tumbler... Well, I'll try, and then maybe Mike might explain it better, but a Bitcoin tumbler is basically how you anonymize your Bitcoin transaction. If you have some Bitcoin and you want to buy some drugs on the dark web or something, you go and you send your Bitcoin to this tumbler, and then it distributes it to a bunch of random Bitcoin addresses that it gives you. And then you have those addresses, and they're anonymized, because they've been sent through a bunch of peoples' wallets, or something like that.Mike:Yep. That's basically it.Ben:So it's basically money laundering.Josh:Yeah, it's laundering.Mike:Yeah. But if your privacy... I mean, okay-Josh:Yeah, no, I get it. Yeah. I mean, yeah. Because part of the appeal of Bitcoin is some people are just like, "Oh yeah, good money, credit card transactions are so... The governments are recording them and stuff, the NSA probably has a database of them". So Bitcoin is anonymous, but it's not. It's not anonymous. And yeah. So that's why people do this, right?Mike:Yeah. Well that, to me, that's if you want to... So the value of Bitcoin, if you want to get bullish on the value of Bitcoin, the traditional outlook is yeah, the silk road was going on and there's all this illegal stuff going on. Therefore it must be bad. But actually, to me, that's the thing, you know it's good if there's illicit stuff going on, because what's the number one currency that's used right now for illicit transactions? It's dirty US dollar bills. And if you're a drug dealer in central South America, you are collecting, dollar bills United States. You're paying some sort of transport probably at 10, 15% cost to get those dollars back to wherever you're going to hold them. And so, if you're using Bitcoin, you're probably not going to pay that fee. So, to me, it's like okay, that actually proves, at least in my mind, that there is value. That it's being used, right?Josh:Yeah. And you also, you don't want to see... Some people are fanatics about cash going away, even just because as more people move to digital transactions, whether it's just through, whatever, traditional networks, or through crypto. People are using less and less cash. And I feel like, whatever... Like Richard Stallman, he pays for everything in cash though, because he thinks that cash is going to go away someday. And that's a problem for privacy, because you do want a way to pay for things in private in some cases.Mike:Yep. I agree.Josh:Yeah.Ben:My only real beef with Bitcoin, well, aside from the whole requiring power plants just to do a transaction, is that there is Badger coin. This company that is named Honeybadger, it's all about Bitcoin. And they have these ATM's in Canada, and we constantly get support requests from people.Mike:Oh really?Josh:Is this the reason that we've been so down on cryptocurrencies in the past?Ben:I think so.Josh:Because ever since the beginning, since people started making coins, Badger coin came out and then it's been our primary exposure to be honest.Ben:It has been, yeah.Josh:Throughout the past... I don't know how many years it's been. Has it been six-Ben:Yeah, six-Josh:... to eight years?Ben:Yeah, something like that. It's been nuts.Josh:I'd say.Mike:You should send them an invoice, and they actually-Ben:Yeah, so what happens is they had these kiosks where you can buy Bitcoin, right? You put your real money in, and you get your fake money out, right? And the name on the top of the kiosk is Honeybadger. So, someone puts in some money, real money, and they don't get their fake money, then all of a sudden they're upset, right?Mike:Yeah.Ben:And so they... For whatever reason, it doesn't go through, right, I don't know how this works, I've never bought Bitcoin at a kiosk. But so, they're like, "Okay, Honeybadger". And so they Google Honeybadger, and the first result for Honeybadger is us. And so they're like, "Oh, here's a phone number I can call". And they call us. And they're like, "Where's my Bitcoin?" That's like, "Uh, I really can't help you with that".Josh:They do.Ben:"You stole my Bitcoin". It's like, "No, that's not us".Josh:Something just occurred to me. I wonder how many of them are just confused over the fact that Bitcoin transactions can take a while to arrive now, right? It's not always instantaneous, where it used to be a lot faster, but now I know that it can take a while to clear. So I wonder how many of those people are emailing us in the span... Maybe that's why they eventually always go away and we don't hear from them again. Maybe it's not that they're getting help, but it's just that their Bitcoins are arriving. Yeah. I have a feeling that there's some sort of... I'm guessing these are mostly regular normies using, and interacting with this very highly technical product and experience, and even if you're walking up to a kiosk, but there's still a highly technical aspect of it that, like you said Mike, people are thinking coin, they're thinking... The way this maps to their brain is it's like dollar bills. So they're looking at it like an ATM. Yep.Mike:Yeah. When it comes to cryptocurrency and the technology, I don't want to have to think about custody, or any of that other kinds of stuff. It'll be successful when it just is happening, I'm not thinking about it. They're already... In some... I don't know all of the different mobile devices, but I do carry out an iPhone. And so, the wallet on iPhone is pretty seamless now, right? And so I'm not thinking about how that technology is working. I had to associate an Amex with it originally, right? But once I've done that, then all I do is click my button to pay. And there you go. And so I do think that the cryptocurrency technology has a long way to go towards that, because if normal people, the non nerds, have to think about it, then it's not going to be useful. Because in the end-Josh:Yeah.Mike:... humans use tools, right? And so, whatever the tool is, they're going to use it especially if it's easy and it makes their life easier.Ben:So what I really want to know, Mike, is what are your feelings about Dogecoin? Are you bullish on Doge?Mike:Well, I'll answer that, but I wanted to come back to the bit about the NFT, and just talking about the possibilities with technology. And I think that you guys could profit from this.Ben:I like where it's going.Mike:You'll have to do some more research. But I think what you could do... See, I love the origin story of Honeybadger. And maybe not everybody knows about the Honeybadger meme from what is... When was this, two thousand...Ben:2012? 2011?Mike:Yeah, okay. So not everybody... Yeah, bot everybody knows about the meme. I guess, just go Google-Ben:I can link it in the show notes.Josh:It's long dead. This meme is long dead.Mike:Is it? Well it's still awesome. I still love it.Josh:It is.Mike:So, there's so many facets of this that I love. The first one is that... Can I name names on competitors-Ben:Of course.Mike:... in the origins? Okay. So the first one was is that Airbrake, an exception reporting service, was doing a poor job with their customer service. And you guys were like, "We're working on this project, we need exception reporting. It's not working". It's like, "Well, can we just take their library, and build our own backend?" Right? And to me, that is beautiful. And in thinking about this episode, in Heroku, the same opportunity lies for an aspiring developer out there where you could just take the Heroku CLI and point it at your own false backend until you figure out all of the API calls that happen. And I don't know, you have that backed by Kubernetes, or whatever orchestration framework is...Mike:There is the possibility that you could do the same Honeybadger story with Airbrake SDK, as there is with the Heroku CLI. So that's the first thing I love about the Honeybadger story, and the fact the name goes along with the fact that Airbrake had poor customer support, and you guys just were like, "F it, we're going to build our own exception reporting service". Now, in the modern context with NFTs is... I have old man experience with the NFTs in that GIFs, or GIFs, and JPEGs, this is BS that people are gouging for profit. However, the technology of the NFT... This is the thing that I think is beautiful, is that... And I'm not sure which of the NFTs does this, but there is the possibility that you could be the originator of a digital object, and then you sell that digital object. And then as that digital object is traded, then you, as the, I guess, the original creator, you can get a percentage of the sales for the lifetime of that digital asset.Ben:Yeah.Mike:And, I'm not sure which of the NFTs allows that, but that is one of the things, that's one of the value propositions in NFT. So what I was thinking is if you guys did an NFT on the shaw of the original Honeybadger Ruby SDK check-in, that this could be the thing that you guys have an experiment with, is you have real skin in the game, you're playing with the technology and see if that works. And, let me know if you do that, because I might try to buy it. So, we'll see.Josh:Well, we've already got a buyer, why wouldn't we?Mike:Yeah, so..Ben:Indeed, yeah.Josh:See I was thinking maybe you could own various errors or something in Honeybadger.Mike:Yeah, I mean... Whatever digital signature you want to... Whatever you want to sign, and then assign value to.Josh:Yeah, we could NFT our Exceptional Creatures.Mike:Yeah.Josh:Have you seen that, Mike? Have you seen that project?Mike:Yep, yep.Josh:Okay.Mike:I'm well aware of that. Yep.Ben:Yeah. I'm thinking what about open source maintainers, right? Let's say you have this project and someone really wants a particular feature, right? Or they're really happy about a particular feature that you've already done, right? You can sell them that shaw, that commit, that put it into name, right?Mike:Yeah, totally.Ben:You are the proud owner of this feature. Thank you.Mike:Yeah, totally. Yeah, I was hoping that I would come with some ideas. I hope someday in the future that I run into somebody and it's like, "Oh, we heard that podcasts were where ideas were free ideas that were worth a lot of money were thrown about. And I did this project, and now I'm retired. Thank you, Mike". Honeybadgers.Josh:Wait, so Ben are you saying that, so as a committer, so say I commit something to Rails, submit a PR, so then I own that PR once it's merged and it would be like I could sell that then to someone? Is that along the lines of what you're saying?Ben:No, I'm thinking the owner of the project. So, if you commit something to Rails, and you're really excited about it, and you for some reason want to have a trophy of that commit-Josh:Right.Ben:... on a plaque on the wall, right? Then the Rails core group could sell you that token.Josh:Okay. Gotcha.Ben:That trophy, that certificate, like, "Yep. This is your thing. Commissioned by..." It's like naming a star, right?Josh:Yeah.Ben:You buy the rights to a star, and it's fake stuff, right? We're naming stars. But that's the same idea.Josh:Yeah. So you could use that same idea to incentivize open-source contribution. So if you make the PR to Rails and it gets merged, you get this NFT for the PR merge, which you could then actually profit for if it was... Say it was, I don't know, turbo links or something, whatever. Years later, when it's a huge thing and everyone in Rails is using it, maybe Mike's going to come along and be like, "Hey, I'll buy... I want to own the PR for turbo links".Ben:Right.Josh:Yeah. And of course then, you, as the owner, would also profit from any sale between parties later on too. You'd get that little percentage.Mike:Yeah. Well, so when somebody comes up with committer coin, just remember me, I want to airdrop of some committer coin.Josh:We have a name. We've got a name for it. Commit coin.Ben:I've got a new weekend project ahead of me.Mike:Yeah.Josh:Cool. Well, that helps me understand NFTs.Ben:Yeah, I really like the idea of being able to sell ownership rights to a digital asset. That I think a good idea. I don't know that the current implementation that we see on the news is a great implementation of that idea. Buying the rights for a copy of a JPEG, it feels kind of sketchy to me. But maybe there's some sort of, I don't know, PDF document that has some sort of value for some reason. And you can give that, sell that to someone. And to me, it's not so much about the profit, or the transaction, it's the ownership. You can say I am the owner of this thing. Yeah, there can be copies all over the place, but I'm the person that has the ownership, quote unquote, of this thing.Josh:Yeah, yeah. But then you've got to define value Ben. What is value? Okay, so, what makes a PDF more valuable than a JPEG?Mike:Yeah. Yeah. Bring this back to Dogecoin, and value propositions, and whatnot. What is valuable? When you're talking about the value of a JPEG, this reminded me of a conversation I was having with my son. He's 10 years old and he wanted some money to buy, I don't know what it was, and old man voice came out of me and it's like, "That's BS. I don't think that's valuable". And he looked at me and he was like, "It's valuable to me". And it's like, "Oh, you just put a dagger in my heart. I'm killing your dream". And one person's value may not be another person's value. So, on the Dogecoin, that's interesting. Dogecoin is very interesting to me, because I feel like I'm in a quantum state with a Dogecoin where it is a joke, but at the same time it apparently it has value.Mike:And I don't know where I stand on that threshold. I know how to trade Dogecoin. And I know the behavior of Dogecoin, and the behaviors, from a trading standpoint, has changed substantially in the last six months. Before it was a pump and dump kind of thing. Well, actually, you know what? When Dogecoin was first created, its purpose was highlighted by the community. People in podcast land don't realize this, but I'm wearing a 2017 Dogecoin shirt from when the Dogecoin community sponsored the number 98 NASCAR. And the thing of the community was like, "Oh, we have all this money, and we're just being altruistic and we're giving it away". And so they were exercising their belief with this currency, right?Mike:And from then, till now, there was a bit of a cycle to Dogecoin where you could, if you acquired Dogecoin for say under a hundred Satoshis, this is the Dogecoin BTC pair, that was actually a good buy. Just wait for the next pump when somebody does something, and Dogecoin goes over 200, or 300 Satoshis, and then you dump it. And that's basically what I did on this in the last six months. I had a small bag of Dogecoin waiting for the next pump and dump. And I actually did that, but it kept on getting pumped, and then it would stabilize. And then now we're at the point where apparently Elon Musk and Mark Cuban are saying that there's value to it.Mike:And to me, I actually put a lot of credence to that, because these are two public persons that they cannot... If they're pumping things in the public domain, then they have risk, right? And so you can't be those two people, and be pumping, and not run the risk of the FTC of the United States government coming in and saying, "Hey, why were you doing this?" So there's the, I guess for me, a small bit of a guarantee that maybe there is something to Dogecoin.Josh:Yeah. See, the way I think, when you first started you were saying it is a joke, but you're in this dual state, and my initial or immediate thought was it is a joke, but this is the internet, and the internet loves to make silly things real.Mike:Yeah, yeah.Josh:Especially these days.Ben:Yeah. It's pretty funny for all those people that made a bunch of money on GameStop, right? Yeah.Mike:Yeah. Well that's the thing, is in Dogecoin, Doge is, of itself, from a meme from the same time period as Honeybadger, right? The Iba Shinu doggie, right? So, the other thing I don't understand, or the thing that I understand but I don't know how to quantify it for myself, is that, to me... So there's no pre-mine on Dogecoin. There's no one person that owns a lot of Dogecoin from the beginning. Whereas if we're talking about Ethereum, Vitalik Buterin, the founder, or one of the founders of Ethereum, they pre-mined Ethereum, and there's a ton of Ethereum that's owned by the founders. Whereas you compare that to, say, Litecoin, Charlie Lee cloned Bitcoin and created Litecoin. He sold all of his Litecoin. I believed in him when he said he's sold it all. He's a software engineer, just like us. He was Director of Engineering at Coinbase.Mike:He doesn't seem like he's wearing tinfoil hat out there, doing conspiracies. So when he says that he sold his coin in 2017, all of his Litecoin, I totally believe that. Yet today, he is the chairperson of the Litecoin foundation. And so, to me... I actually do have, I placed some value in the benevolence of Litecoin and Dogecoin, because there's not any one person that actually controls it. I guess Charlie Lee, he probably has a stronger voice than most. But he doesn't control the levers.Josh:Not financially.Mike:Yeah.Josh:Yeah.Mike:Yeah. And so then with Dogecoin... So Dogecoin, it'll be awesome if it gets above a dollar, but the structure of Dogecoin will be such as they cannot maintain that.Josh:Right.Mike:Because it's an inflation-Josh:There's no cap, right?Mike:Right.Josh:Yeah.Mike:It's inflation. And so, I don't know the number, I think it's a million Dogecoin are minted every day. So, 10 years from now, if Dogecoin is worth a dollar still, then that means Bitcoin will be worth a lot more than that. So I guess that'd be awesome if Dogecoin stays a dollar. However, the point I'm trying to make is actually there is value in having an inflationary currency, especially if we're talking about living in the structure of our current financial... The way that our current financial markets work, where there is an inflation.Mike:And so if I want to be transacting with a digital currency, I don't want to have to be, say, like having an Argentina kind of moment where my one Dogecoin is worth $5 American today, and then maybe only $3 American a week from now. So to me, I think there is value in Dogecoin in that it's inflationary, and that it will not be as susceptible to speculation bubbles as other currencies. And so, I don't know if that answers your questions on the value of Dogecoin, but those are a couple of reasons why I think that Dogecoin is valuable. Now, am I going to be holding a big bag of Dogecoin in 2022? Probably not. Just to be honest.Ben:We're all about honesty at Honeybadger. I love the episodes where we have to have a disclaimer, this is not financial advice. Please consult competent professionals before investing, et cetera, et cetera. Mike, it has been a delight to have you with us. We appreciate your counterbalance to our coin pessimism that we have amongst the Honeybadger fan base.Josh:Yeah, I think we needed this.Ben:Yeah.Josh:We really needed this.Ben:We really did.Josh:So thank you.Ben:It's been good.Mike:Yeah. Oh, I got one more idea out there. Hopefully, somebody can run with this, is I've been trying to get motivated to do some experimentation with the Bitcoin lightning network. We didn't really talk about these a layer two solutions for scaling, but I think that there is a lot of potential in coming up with an interesting project that lays within the Litecoin* network, it has its value in and of itself, but there's a secondary value of being a note on the Litecoin* network where if there's transactions going through your node, let's say, I don't know how you'd instrument this, but let's say that Honeybadger actually was... That you guys were taking your payments across your own lightning node, then all of the transactions that are going across the lightning network, you're getting a small fee, right? So I think that there's the possibility of a micropayments kind of play there, like for instance, paying by the exception. I mean, literally-*Editor's note from Mike - "in my excitement talking about the Lighting Network I slipped and said Litecoin a couple of times between Lightning Network. Lightning Network is a layer 2 protocol that is primarily intended for scaling Bitcoin and that was what I meant. However, Lightning can be implemented to run on top of Litecoin and Ethereum."Josh:That has come up that has come up in the past, I think at one point.Mike:You can't do micro payments on a credit card.Josh:Yeah.Mike:Right? But you can do micropayments on lightening network. And I'm not selling you guys on this, but I'm saying that there's going to be some nerd out there that it's like, "Oh my God micropayments are here, I can do micropayments on lighting network". And then they're going to do well on that product, but then they're also going to do well on the commission that they're earning on payments going through their node.Josh:This could be used for usage base software as a service billing model.Ben:Totally. And then you get the skim off the top, just like a good affiliate does.Mike:Yes.Ben:I love it.Mike:Yes.Ben:I love it. All right. All right, Mike, we're going to have to do some scheming together. Well, any final words, any parting words besides go by all the Dogecoin that you can?Mike:Yeah. Don't put all your money into the cryptocurrencies. Yeah.Josh:Seems like good advice.Ben:Be smart

Annette Bieniusa, Co-founder of Concordant, Joins Adi Polak to discuss how Concordant leverage CRDTs for enabling multiple Edges/IoT devices to share consistent data more efficiently without relying on network connections. How to use it and what is the exciting open-source solution, AntidotDB that they developed, helped the community, Follow @CH9 http://www.twitter.com/ch9 Follow @TechExceptions https://twitter.com/TechExceptions Follow @Anne_biene https://twitter.com/anne_biene Follow @AdiPolak https://twitter.com/AdiPolak

В этом выпуске выпуске: Event Sourcing со всех сторон, проходим аудит разными способами, современный CRDT, Rust как единственный современный язык для embedded и, конечно, темы слушателей Шоу нотес: [00:05:18] «Change Data Capture» vs «Event Sourcing» [01:06:54] Коллаборативное редактирование на основе CRDT глазами соавтора Wave I was wrong. CRDTs are the future CRDTs: The Hard Parts —… Читать далее →

I saw Martin Kleppmann's talk a few weeks ago about CRDTs, and I felt a deep sense of despair. Maybe all the work I've been doing for the past decade ... https://josephg.com/blog/crdts-are-the-future/ Martin Kleppmann’s talkJupiter paperShareJSglorified for() loopsome helper functionsoptimized CFirepadShareDBa schemeI gave a talkBenchmarks - look for B4handles this fineMartin’s talk54 minutes into his talkimplementation of automerge in only about 100 lines of codesimple proof of concept CRDT implementation in Rust6 million edits per seconditerationY.jsin the Y.js benchmarksfinanced by advertisers bidding for my eyeballslocal first softwareoperationaltransformcodeAnd that matters.

Today on The InfoQ Podcast, Wes Reisz talks with Peter Bourgon. Peter is a distributed system engineer working on Fastly. His area of interest is around coordination free replicated systems. The two engineers talk about the space of Conflict-Free Replicated Data Types (CRDTs) specifically in the context of edge compute. Topics covered on the podcast include Edge compute, CRDTs, CAP Theorem, and challenges around building distributed systems. Why listen to this podcast: - CRDTs (conflict-free replicated data types) are a class of coordination free replication systems (or systems that don’t require strategies such as leader election). - An easy way to think of a CRDT is as an associative, commutative, and idempotent data structure plus the operations needed to use it. The edge is an overloaded term that people tend to define based on where they sit on a spectrum between the customer and the data center. Fastly’s edge is away from the data center and but not to the telephone pole or handset. - RAFT and Gossip are two alternative approaches to using a coordination free replication system like CRDTs. To get the properties of a CRDT and have useful data types, you have to pay a cost in size and often bytes on the wire. These are challenges that continue to need solutions. - Modern Distributed systems and data structures like CRDTs require you to start thinking about state in the system itself. It’s not unusual for a system today to give you multiple results back that the system will have to handle or merge. More on this: Quick scan our curated show notes on InfoQ https://bit.ly/3dTZOO4 You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: https://bit.ly/3dTZOO4

In this episode of the CaSE podcast, Lucas talks to Lars Hupel of Typelevel-fame about algebraic design. Lars starts to explain how the expression problem views the difference between the functional and Object Oriented style of programming. He then explains the concept of an algebra – from coupling trains to CRDTs. This leads to a discussion on comprehensive and iterative thinking styles and how different styles of testing and proving work. Finally, they bring it all back to type systems.

# Episode 157 – LiveView & The Future of Phoenix ft. Chris McCord Today on ElixirTalk we’re joined by Chris McCord who you may know as the creator of the Phoenix framework, and a second time guest on ElixirTalk. In this episode we talk quite a bit about LiveView, the journey of taking it from a prototype to release, the optimizations that have come along the way, and some of the great examples of LiveView running today. We get into some of our previous discussions about the balance of LiveView vs Javascript with Chris and how we frame the benefits of using the technology today. Additionally we hear about what’s on the roadmap for Phoenix, and the roadmap outside of the Phoenix core library itself. Thanks as always to Chris and the Phoenix Core team for their work! ## Links * ElixirConf 2018 – Announcing LiveView: https://www.youtube.com/watch?v=Z2DU0qLfPIY * ElixirConf 2019 – Chris’ Keynote: https://www.youtube.com/watch?v=txk4WAlabvI * What’s new and next for LiveView: https://dockyard.com/blog/2019/09/06/whats-new-and-next-for-phoenix-liveview * LiveView JavaScript Hooks: https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#module-javascript-client-specific * LiveView in Marlus’ ElixirConf talk – https://www.youtube.com/watch?v=tPu-P97-cbE * Phoenix Presence and CRDTs: https://dockyard.com/blog/2016/03/25/what-makes-phoenix-presence-special-sneak-peek * Dark Lang: https://darklang.com * Oban Job Library: https://github.com/sorentwo/oban * Our discussion on LiveView: https://soundcloud.com/elixirtalk/episode-154-a-great-intro-elixirconf-recap-javascript-v-liveview * Phoenix Phrenzy Competition: https://phoenixphrenzy.com/ * Dockyard: http://dockyard.com

This episode explores the intersections between various flavors of math and programming, and the ways in which they can be mixed, matched, and combined. Michael Arntzenius, "rntz" for short, is a PhD student at the University of Birmingham building a programming language that combines some of the best features of logic, relational, and functional programming. The goal of the project is "to find a sweet spot of something that is more powerful than Datalog, but still constrained enough that we can apply existing optimizations to it and imitate what has been done in the database community and the Datalog community." The challenge is combining the key part of Datalog (simple relational computations without worrying too much underlying representations) and of functional programming (being able to abstract out repeated patterns) in a way that is reasonably performant. This is a wide-ranging conversation including: Lisp macros, FRP, Eve, miniKanren, decidability, computability, higher-order logics and their correspondence to higher-order types, lattices, partial orders, avoiding logical paradoxes by disallowing negation (or requiring monotonicity) in self reference (or recursion), modal logic, CRDTS (which are semi-lattices), and the place for formalism is programming. This was a great opportunity for me to brush up on (or learn for the first time) some useful mathematical and type theory key words. Hope you get a lot out of it as well -- enjoy! The transcript for this episode was sponsored by Repl.it and can be found at https://futureofcoding.org/episodes/040#full-transcript

Panel: Charles Max Wood Lucas Reis Justin Bennett Special Guests: Peter Mbanugo In this episode, the panel talks with guest speaker, Peter Mbanugo. Peter is a computer software specialist who works with Field Intelligence and writes technical articles for Progress Software and a few others. He studied at SMC University and currently resides in Nigeria. They talk about his creation, Hamoni Sync, and article, Real-time editable data grid in React. Also, other topics such as Offline-First, Speed Curve, Kendo UI are talked about, too. Check out today’s episode Show Topics: 1:30 – Chuck: Let’s talk about what you built and how it works. Topic: Real-time editable data grid in React. 1:40 – Peter: Real time editing. It allows you to edit and have the data go across the different devices. Synchronizing your applications. For the  2:47 – I saw that you built also the... 2:58 – Peter: Yes, I built that with Real-time. Most of the time I have to figure out how to build something to go across the channel, such as the message. Then I built the chats. Next month 4:33 – Justin: It says that it can go offline. That is challenging. How are you going about that? 4:51 – Peter answers the question. Peter: When you loose connections and when the network comes back on then it will try to publish anything to the server while offline. If you are trying to initialize the... 5:42 – Awesome. 5:45 – Peter continues his thoughts. 5:56 – Lucas: This is really interesting. Form something really simple to tackle this problem. I have gotten into so many problems. Congratulations on at least having the courage to try such a system. 6:35 – Justin: When you have someone interacting with one of these applications, lose connectivity, is the service handling this behind the scenes? 6:56 – Peter: Yes.  Peter goes into detail. 7:19 – Justin: Neat. That would be interesting to dig more into that. 7:35 – Lucas: I had a friend who sent me links and I was like WHOAH. It’s not an easy task. 7:57 – Peter: Yes, offline – I am learning each and everyday. There are different ways to go about it. Then I go write something about conflict free of different types. I thought that was the way to go. I didn’t want it to be something of the declines. 8:50 – Lucas: How did React work for you? 9:24 – Peter answers the question. 9:58 – Panelist: I was trying to synchronize the system. There are 2 types: Operational Transformations and CRDTs. It’s a really hard problem. 10:35 – Lucas: Now we have multiple devices and they can be far away from each other. Updates to send to the same server. I think that this is really complicated world. Even consider new techniques that we use in RI. You have a long in process. You need to react to them. Maybe dates that you cannot get. Hard problem we are solving now. 11:56 –Justin: Even interacting with applications that ... it has made our products that aren’t approachable if someone doesn’t have a good Internet connection. Synchronizing connections while offline. So you can have offline support. These are problems that we can resolve hopefully. 13:01 – Lucas: It affects everyone. Back in Brazil we had problems with connections, because it’s connections. Now I live in NY but the subway my connection is hurt. 13:40 – Peter: Yes, I agree. Peter talks about his connections being an issue while living in Africa. 14:52 – Justin: How does that affect your development workflow? 15:08 – Peter answers the question. 17:23 – Justin: Shout-out to the Chrome team. Tool called LIGHTHOUSE. It can test for accessibility, SEOs and etc. Good same defaults and trying to test Mobile First. When I was learning about performance I wasn’t thinking about the types of devices that people would use. The edits tab really helps think about those things. 18:41 – Digital Ocean’s Advertisement 19:18 – Justin: Any tools to help test your download speeds or anything authentication tools? 19:36 – Peter answers this question. 20:15 – Panelist asks the same question to Lucas. 20:22 – Lucas: interesting question. Even though the website was doing pretty well we were in the dark. We did a huge migration and it wasn’t clear about the performance. So my first mission here was start using a tool called SPEED CURVE. It only gets better. For a company who needs to acquire a tool SPEED CURVE is great. They have the LIGHTHOUSE measurements in their dashboards. So it can follow through time your scores and all of your analysis. These are the tools we use today. They have both synthetic and real user monitoring. So when we are measuring things on our Chrome it is a picture of your machine (biased picture) they make it both synthetic and film your page and compare through time. Analyze your assets. Some code on your application and collects statistics for each user. Relic I have used before, too. I do believe those tools are of great help. I am sure there are opensource initiatives, but I haven’t played 22:56 – Peter: Have you tried...? 23:07 – Lucas continues. LIGHTHOUSE. 23:56 – Justin: It gives great visualizations for people to see. SPEED CURVE. Where we are at – so they can see that – it’s powerful. 24:40 – Lucas: Interesting story we used SPEED CURVE. Real users and synthetic measurements; our website was getting slower and slower. We couldn’t figure it out. What is happening to our application? It turned out that the app more people were using it on the mobile. The real user speed was going up because they were using mobile. The share of mobile users and performance was getting better. You look at the overall average it was getting slower. Interesting lesson on how to look at data, interpret data and insights. It was really interesting. 26:21 – Peter. 26:25 – Lucas continues the previous conversation from 24:40. 27:00 – Justin: Taking the conversation back. It’s always a challenging problem because the implications are hard to use. What was your experience with React Table? What are the pros and cons? 27:40 – Peter: React Table is quite light. It is pretty good on data. I haven’t had much of a problem. It is okay to use. The other ones I haven’t tried them, yet. 28:08 – Justin: Same question to Charles and to Lucas. 28:21 – Lucas: I have never worked with big tables to render the massive data or tables that need to be edits and stuff like that. I don’t have experience with those components. Play here and there. It is interesting, because it is one of those components that are fighting the platform and it’s a good source of interesting solutions. 29:05 – Chuck: Kendo UI has one. I need something that his more barebones. AG Grid. 30:03 – Justin: React Windows. It optimizes long lists. It just renders what is in the current window. 30:22 – Ryan Vaughn. 30:28 – Justin: Cool library. 30:36 – Lucas: Use it as a learning tool. How do you all decide when to actually start using a library? As early as you can? Libraries to solve our problems? 31:19 – Peter: It depends on what I am doing. 31:53 – Fascinating question. Not one size fits all. It’s a balance between product deliverable needs and... There can be risks involved. Fine balance. I find myself doing a lot is I will default using a library first. Library that isn’t too large but what I need for that project. If there is a hairy feature I will use the library until my needs are met.  33:49 – Lucas adds his comments. Lucas: You want to differentiate yourself. I love GitHub. 35:36 – Question to Charles: I know you have tons of stuff going on. What’s your thought process? 35:53 – Chuck: If I can find stuff on the shelf I will pay for it. My time adds up much more quickly then what the dollars do. I will pay for something off the shelf. I only mess around for a while but if I can’t find something to help me then I will go and build something of my own. I got close with Zapier, but I got to the point that I wanted to put something together that I built my own thing through Ruby on Rails. Generally I will pay for it. 37:07 – Panelist: Yes, I don’t think we all don’t value our time and how expensive time is. 37:25 – Chuck: I own the business. My time is of value – it’s more important to me. It’s a trap that people fall into not to value their time. 38:11 – Lucas: We are not all working on what we SHOULD be working on. This isn’t going to bring business Productive time that we are using with stuff that is not our business or our main focus. Focus on the core product. Try to get the customers to have a better life. The mission of the company. The web community that started that most is the Ruby community. Having solutions and focusing on the problem. I think that JavaScript is now doing a better job of this. As we know it’s easy to fall into this trap and play with building blocks. 39:52 – Chuck: I have had a few people remind me that I am a DEVELOPER! 40:19 – Justin: The thing I have estimating is the difficulty of something. I can build it because I am a developer. Is it valuable for me? 41:10 – Lucas: The sunken costs sink in – I have done all this work and now look where I am at? 41:33 – Chuck: Anything else? 41:43 – Peter: Check out me through Twitter and the Dev blog. Message me anytime. 42:13 – Chuck: Picks! 42:18 – Advertisement. Links: Kendo UI Ruby on Rails Angular Get A Coder Job Redux Agile Real-time editable data grid in React Peter Mbanugo’s Twitter Peter Mbanguo’s LinkedIn Peter Mbanguo’s Dev.To Peter Mbanguo’s GitHub Peter Mbanguo’s WordPress Lucas Reis’ Email: lucasmreis@gmail.com Charles Max Wood’s Twitter Sponsors: Kendo UI Digital Ocean Get A Coder Job Picks: Charles Book: The ONE Thing Get A Coder Job – It will be out next week! T-Shirts & Mugs – Podcast Artwork - SWAG Kickstarter – Code Badge.Org Justin RC BLOG Podcast: Indie Hackers Indie Hackers Lucas Blog Post: Thirteen Ways of Looking at a Turtle Peter Library – Opensource Masters of Skill – Podcast Book: Ego is the Enemy Book

Panel: Charles Max Wood Lucas Reis Justin Bennett Special Guests: Peter Mbanugo In this episode, the panel talks with guest speaker, Peter Mbanugo. Peter is a computer software specialist who works with Field Intelligence and writes technical articles for Progress Software and a few others. He studied at SMC University and currently resides in Nigeria. They talk about his creation, Hamoni Sync, and article, Real-time editable data grid in React. Also, other topics such as Offline-First, Speed Curve, Kendo UI are talked about, too. Check out today’s episode Show Topics: 1:30 – Chuck: Let’s talk about what you built and how it works. Topic: Real-time editable data grid in React. 1:40 – Peter: Real time editing. It allows you to edit and have the data go across the different devices. Synchronizing your applications. For the  2:47 – I saw that you built also the... 2:58 – Peter: Yes, I built that with Real-time. Most of the time I have to figure out how to build something to go across the channel, such as the message. Then I built the chats. Next month 4:33 – Justin: It says that it can go offline. That is challenging. How are you going about that? 4:51 – Peter answers the question. Peter: When you loose connections and when the network comes back on then it will try to publish anything to the server while offline. If you are trying to initialize the... 5:42 – Awesome. 5:45 – Peter continues his thoughts. 5:56 – Lucas: This is really interesting. Form something really simple to tackle this problem. I have gotten into so many problems. Congratulations on at least having the courage to try such a system. 6:35 – Justin: When you have someone interacting with one of these applications, lose connectivity, is the service handling this behind the scenes? 6:56 – Peter: Yes.  Peter goes into detail. 7:19 – Justin: Neat. That would be interesting to dig more into that. 7:35 – Lucas: I had a friend who sent me links and I was like WHOAH. It’s not an easy task. 7:57 – Peter: Yes, offline – I am learning each and everyday. There are different ways to go about it. Then I go write something about conflict free of different types. I thought that was the way to go. I didn’t want it to be something of the declines. 8:50 – Lucas: How did React work for you? 9:24 – Peter answers the question. 9:58 – Panelist: I was trying to synchronize the system. There are 2 types: Operational Transformations and CRDTs. It’s a really hard problem. 10:35 – Lucas: Now we have multiple devices and they can be far away from each other. Updates to send to the same server. I think that this is really complicated world. Even consider new techniques that we use in RI. You have a long in process. You need to react to them. Maybe dates that you cannot get. Hard problem we are solving now. 11:56 –Justin: Even interacting with applications that ... it has made our products that aren’t approachable if someone doesn’t have a good Internet connection. Synchronizing connections while offline. So you can have offline support. These are problems that we can resolve hopefully. 13:01 – Lucas: It affects everyone. Back in Brazil we had problems with connections, because it’s connections. Now I live in NY but the subway my connection is hurt. 13:40 – Peter: Yes, I agree. Peter talks about his connections being an issue while living in Africa. 14:52 – Justin: How does that affect your development workflow? 15:08 – Peter answers the question. 17:23 – Justin: Shout-out to the Chrome team. Tool called LIGHTHOUSE. It can test for accessibility, SEOs and etc. Good same defaults and trying to test Mobile First. When I was learning about performance I wasn’t thinking about the types of devices that people would use. The edits tab really helps think about those things. 18:41 – Digital Ocean’s Advertisement 19:18 – Justin: Any tools to help test your download speeds or anything authentication tools? 19:36 – Peter answers this question. 20:15 – Panelist asks the same question to Lucas. 20:22 – Lucas: interesting question. Even though the website was doing pretty well we were in the dark. We did a huge migration and it wasn’t clear about the performance. So my first mission here was start using a tool called SPEED CURVE. It only gets better. For a company who needs to acquire a tool SPEED CURVE is great. They have the LIGHTHOUSE measurements in their dashboards. So it can follow through time your scores and all of your analysis. These are the tools we use today. They have both synthetic and real user monitoring. So when we are measuring things on our Chrome it is a picture of your machine (biased picture) they make it both synthetic and film your page and compare through time. Analyze your assets. Some code on your application and collects statistics for each user. Relic I have used before, too. I do believe those tools are of great help. I am sure there are opensource initiatives, but I haven’t played 22:56 – Peter: Have you tried...? 23:07 – Lucas continues. LIGHTHOUSE. 23:56 – Justin: It gives great visualizations for people to see. SPEED CURVE. Where we are at – so they can see that – it’s powerful. 24:40 – Lucas: Interesting story we used SPEED CURVE. Real users and synthetic measurements; our website was getting slower and slower. We couldn’t figure it out. What is happening to our application? It turned out that the app more people were using it on the mobile. The real user speed was going up because they were using mobile. The share of mobile users and performance was getting better. You look at the overall average it was getting slower. Interesting lesson on how to look at data, interpret data and insights. It was really interesting. 26:21 – Peter. 26:25 – Lucas continues the previous conversation from 24:40. 27:00 – Justin: Taking the conversation back. It’s always a challenging problem because the implications are hard to use. What was your experience with React Table? What are the pros and cons? 27:40 – Peter: React Table is quite light. It is pretty good on data. I haven’t had much of a problem. It is okay to use. The other ones I haven’t tried them, yet. 28:08 – Justin: Same question to Charles and to Lucas. 28:21 – Lucas: I have never worked with big tables to render the massive data or tables that need to be edits and stuff like that. I don’t have experience with those components. Play here and there. It is interesting, because it is one of those components that are fighting the platform and it’s a good source of interesting solutions. 29:05 – Chuck: Kendo UI has one. I need something that his more barebones. AG Grid. 30:03 – Justin: React Windows. It optimizes long lists. It just renders what is in the current window. 30:22 – Ryan Vaughn. 30:28 – Justin: Cool library. 30:36 – Lucas: Use it as a learning tool. How do you all decide when to actually start using a library? As early as you can? Libraries to solve our problems? 31:19 – Peter: It depends on what I am doing. 31:53 – Fascinating question. Not one size fits all. It’s a balance between product deliverable needs and... There can be risks involved. Fine balance. I find myself doing a lot is I will default using a library first. Library that isn’t too large but what I need for that project. If there is a hairy feature I will use the library until my needs are met.  33:49 – Lucas adds his comments. Lucas: You want to differentiate yourself. I love GitHub. 35:36 – Question to Charles: I know you have tons of stuff going on. What’s your thought process? 35:53 – Chuck: If I can find stuff on the shelf I will pay for it. My time adds up much more quickly then what the dollars do. I will pay for something off the shelf. I only mess around for a while but if I can’t find something to help me then I will go and build something of my own. I got close with Zapier, but I got to the point that I wanted to put something together that I built my own thing through Ruby on Rails. Generally I will pay for it. 37:07 – Panelist: Yes, I don’t think we all don’t value our time and how expensive time is. 37:25 – Chuck: I own the business. My time is of value – it’s more important to me. It’s a trap that people fall into not to value their time. 38:11 – Lucas: We are not all working on what we SHOULD be working on. This isn’t going to bring business Productive time that we are using with stuff that is not our business or our main focus. Focus on the core product. Try to get the customers to have a better life. The mission of the company. The web community that started that most is the Ruby community. Having solutions and focusing on the problem. I think that JavaScript is now doing a better job of this. As we know it’s easy to fall into this trap and play with building blocks. 39:52 – Chuck: I have had a few people remind me that I am a DEVELOPER! 40:19 – Justin: The thing I have estimating is the difficulty of something. I can build it because I am a developer. Is it valuable for me? 41:10 – Lucas: The sunken costs sink in – I have done all this work and now look where I am at? 41:33 – Chuck: Anything else? 41:43 – Peter: Check out me through Twitter and the Dev blog. Message me anytime. 42:13 – Chuck: Picks! 42:18 – Advertisement. Links: Kendo UI Ruby on Rails Angular Get A Coder Job Redux Agile Real-time editable data grid in React Peter Mbanugo’s Twitter Peter Mbanguo’s LinkedIn Peter Mbanguo’s Dev.To Peter Mbanguo’s GitHub Peter Mbanguo’s WordPress Lucas Reis’ Email: lucasmreis@gmail.com Charles Max Wood’s Twitter Sponsors: Kendo UI Digital Ocean Get A Coder Job Picks: Charles Book: The ONE Thing Get A Coder Job – It will be out next week! T-Shirts & Mugs – Podcast Artwork - SWAG Kickstarter – Code Badge.Org Justin RC BLOG Podcast: Indie Hackers Indie Hackers Lucas Blog Post: Thirteen Ways of Looking at a Turtle Peter Library – Opensource Masters of Skill – Podcast Book: Ego is the Enemy Book

As we scale our systems to handle larger volumes of data, geographically distributed users, and varied data sources the requirement to distribute the computational resources for managing that information becomes more pronounced. In order to ensure that all of the distributed nodes in our systems agree with each other we need to build mechanisms to properly handle replication of data and conflict resolution. In this episode Christopher Meiklejohn discusses the research he is doing with Conflict-Free Replicated Data Types (CRDTs) and how they fit in with existing methods for sharing and sharding data. He also shares resources for systems that leverage CRDTs, how you can incorporate them into your systems, and when they might not be the right solution. It is a fascinating and informative treatment of a topic that is becoming increasingly relevant in a data driven world.

Nachdem wir in unserer ersten Folge verschiedene Arten von verteilten Datenbanken vorgestellt haben, geht es nun um Probleme mit verteilten Datenbanken. Welchen Einfluss hat das Netzwerk auf eine verteilte Datenbank? Was genau ist eigentlich das CAP Theorem? Wieso können wir uns auf Uhren nicht verlassen und welche anderen Möglichkeiten haben wir? Und was sind CRDTs?

Peter Bourgon discusses his work at Weaveworks, discovering and imlemeting CRDTs for time-stamped events at Soundcloud, Microservices in Go with Go Kit and the state of package management in Go. Why listen to this podcast: - We’ve hit the limits of Moore’s law so when we want to scale we have to think about how we do communication across unreliable links between unreliable machines. - In an AP algorithm like Gossip you still make forward progress in case of a failure. In Paxos you stop and return failures. - CRDTs give us a rigours set of rules to accommodate failures for maps, sets etc. in communication that result in am eventually consistent system. - Go is optimised to readers/maintainers vs. making the programmers’ life easier. Go is closer to C than Java in that it allows you to layout memory very precisely, allowing you to, for example, optimise cache lines in your CPU. - Bourgon started a project called Go Kit, which is designed for building microservices in Go. It takes inspiration from Tiwtter’s Scala-based Finagle which solved a lot of Micoservice concerns. - Go has a number of community-maintained package managers but no good solution; work in ongoing to try and resolve this. Notes and links can be found on: http://bit.ly/2kaHC9k Work at Weaveworks Gossip vs. Paxos CRDTs at SoundCloud Go Go in large teams Go and Java package management Microservices in Go with Go Kit Logging and tracing in a distributed environment More on this: Quick scan our curated show notes on InfoQ. http://bit.ly/2kaHC9k You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

01:57 - Chris McCord Introduction Twitter GitHub Blog DockYard Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and Jose Valim Metaprogramming Elixir: Write Less Code, Get More Done (and Have Fun!) by Chris McCord 02:09 - The Phoenix Framework The Elixir Programming Language The Erlang Programming Language 02:46 - Ruby vs Elixir José Valim “Programming should be enjoyable.” sync WhatsApp Metaprogramming 12:12 - Phoenix as a Framework; Similarity to Rails 15:38 - ecto 16:45 - Performance 18:06 - Generating a Phoenix Project Mix 20:44 - Getting Started with Phoenix; Creating and Maintaining 25:29 - Guidance and Decision Making Development and Documentation 30:22 - Phoenix and JavaScript; Build Tools Brunch 37:22 - Phoenix and Elm Chris McCord and Evan Czaplicki: Phoenix and Elm: Making the Web Functional @ Erlang Factory SF 2016 38:45 - Maintainability OTP Framework 41:52 - Hosting Phoenix Heroku 42:48 - Object-Oriented vs Functional The Actor Model Smalltalk 50:29 - Debugging; Tooling UberAuth 01:01:08 - Phoenix Presence Picks Radical Candor — The Surprising Secret to Being a Good Boss | First Round Review (Jessica) Flex ('Mancer) by Ferrett Steinmetz (Jessica) The Teaching Company: How to Listen to and Understand Great Music (Jessica) Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and José Valim (Chris) Jose Valim: Phoenix a web framework for the new web @ Lambda Days 2016 (Chris) The 4-Hour Workweek: Escape 9-5, Live Anywhere, and Join the New Rich by Timothy Ferriss (Avdi) Efficient State-based CRDTs by Delta-Mutation (Chris)

01:57 - Chris McCord Introduction Twitter GitHub Blog DockYard Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and Jose Valim Metaprogramming Elixir: Write Less Code, Get More Done (and Have Fun!) by Chris McCord 02:09 - The Phoenix Framework The Elixir Programming Language The Erlang Programming Language 02:46 - Ruby vs Elixir José Valim “Programming should be enjoyable.” sync WhatsApp Metaprogramming 12:12 - Phoenix as a Framework; Similarity to Rails 15:38 - ecto 16:45 - Performance 18:06 - Generating a Phoenix Project Mix 20:44 - Getting Started with Phoenix; Creating and Maintaining 25:29 - Guidance and Decision Making Development and Documentation 30:22 - Phoenix and JavaScript; Build Tools Brunch 37:22 - Phoenix and Elm Chris McCord and Evan Czaplicki: Phoenix and Elm: Making the Web Functional @ Erlang Factory SF 2016 38:45 - Maintainability OTP Framework 41:52 - Hosting Phoenix Heroku 42:48 - Object-Oriented vs Functional The Actor Model Smalltalk 50:29 - Debugging; Tooling UberAuth 01:01:08 - Phoenix Presence Picks Radical Candor — The Surprising Secret to Being a Good Boss | First Round Review (Jessica) Flex ('Mancer) by Ferrett Steinmetz (Jessica) The Teaching Company: How to Listen to and Understand Great Music (Jessica) Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and José Valim (Chris) Jose Valim: Phoenix a web framework for the new web @ Lambda Days 2016 (Chris) The 4-Hour Workweek: Escape 9-5, Live Anywhere, and Join the New Rich by Timothy Ferriss (Avdi) Efficient State-based CRDTs by Delta-Mutation (Chris)

01:57 - Chris McCord Introduction Twitter GitHub Blog DockYard Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and Jose Valim Metaprogramming Elixir: Write Less Code, Get More Done (and Have Fun!) by Chris McCord 02:09 - The Phoenix Framework The Elixir Programming Language The Erlang Programming Language 02:46 - Ruby vs Elixir José Valim “Programming should be enjoyable.” sync WhatsApp Metaprogramming 12:12 - Phoenix as a Framework; Similarity to Rails 15:38 - ecto 16:45 - Performance 18:06 - Generating a Phoenix Project Mix 20:44 - Getting Started with Phoenix; Creating and Maintaining 25:29 - Guidance and Decision Making Development and Documentation 30:22 - Phoenix and JavaScript; Build Tools Brunch 37:22 - Phoenix and Elm Chris McCord and Evan Czaplicki: Phoenix and Elm: Making the Web Functional @ Erlang Factory SF 2016 38:45 - Maintainability OTP Framework 41:52 - Hosting Phoenix Heroku 42:48 - Object-Oriented vs Functional The Actor Model Smalltalk 50:29 - Debugging; Tooling UberAuth 01:01:08 - Phoenix Presence Picks Radical Candor — The Surprising Secret to Being a Good Boss | First Round Review (Jessica) Flex ('Mancer) by Ferrett Steinmetz (Jessica) The Teaching Company: How to Listen to and Understand Great Music (Jessica) Programming Phoenix: Productive |> Reliable |> Fast by Chris McCord, Bruce Tate, and José Valim (Chris) Jose Valim: Phoenix a web framework for the new web @ Lambda Days 2016 (Chris) The 4-Hour Workweek: Escape 9-5, Live Anywhere, and Join the New Rich by Timothy Ferriss (Avdi) Efficient State-based CRDTs by Delta-Mutation (Chris)

Robert Blumen talks to Christopher Meiklejohn about conflict-free replicated data types (CRDTs). The discussion covers forms of consistency in distributed systems; conflicts and conflict resolution algorithms; consistency and concurrency; the discovery of CRDTs; state-based and operations-based CRDTs; examples of some well-known CRDTs, including counters and sets; time and space complexity; uses of CRDTs in chat […]

Robert Blumen talks to Christopher Meiklejohn about conflict-free replicated data types. The discussion covers consistency in distributed systems, CRDTs, and their use in NoSQL databases.

Distributed systems programming will always be a world of tradeoffs -- there is no silver bullet in the future. But life can be made easier with tactics such as the actor pattern and the use of conflict-free replicated data types (CRDTs). Caitie McCaffrey is a distributed systems engineer who currently works at Twitter. She previously worked on Halo 4 at Microsoft and 343 Industries. At QCon San Francisco, she will be hosting the track Taming Distributed Architecture. Continue reading… The post Taming Distributed Architecture with Caitie McCaffrey appeared first on Software Engineering Daily.

A discussion of causality, vector clocks, version vectors, and CRDTs.

Paul deGrandis (@ohpauleez) and Kevin Lynagh (@lynaghk) are two anchors of the Clojure community, perhaps especially of the ClojureScript wing.  Both Portlanders, they’ve been elbow-deep in core.logic and a ton of ClojureScript tools and libraries like shoreleave, cljx, c2, and more.  They’ve stormed the Clojure world in the past year or two, going from zero […]

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: I think that the lack of interoperability or standardization between digital tools today really it means that all work created within a tool is confined to that tool, and to me that seems very clearly antithetical to creativity and specifically the collaborative aspect of creativity. 00:00:28 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad. This podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. I’m Adam Wiggins here with Mark McGranaghan. Hey, Adam, and our guest today, Molly Milky. 00:00:43 - Speaker 1: Hey there. 00:00:45 - Speaker 2: And how was your spring break, Molly? 00:00:47 - Speaker 1: It’s pretty good, not long enough, but it was a lovely little escape in Berkeley, and I worked on a final project for my producing class, which was a pitch on a feature film on the Whole Earth Catalog, which didn’t go over as well as I had hoped, but I’m still fingers crossed that it’ll become something. 00:01:10 - Speaker 2: And the whole Earth Catalog here being the Stewart brand work from what was the 70s or 80s. 00:01:15 - Speaker 1: Confirmed, yes, it was basically a biopic on him and the era of the whole Earth Catalog, and it was very dramatic. 00:01:23 - Speaker 2: Oh, I love that. First of all, I just love biopics. I’m a big fan of like abstract. Act on Netflix or that sort of like kind of maker documentary, but when you throw in like the weird history, I feel like the whole Earth that catalog was sort of, I don’t know, psychedelic culture meets rebel computing or something like that. 00:01:41 - Speaker 1: 100% agree, yes. In a very interesting way that I think would translate really well to film, but we’ll see. 00:01:48 - Speaker 2: Yeah, absolutely. So yeah, let me know where I can sign up to screen that I guess. 00:01:53 - Speaker 1: Amazing, yes, you’ll be the first to know. 00:01:55 - Speaker 3: Wasn’t there actually another film about Stewart Brand in general that came out recently? 00:02:00 - Speaker 1: Yep, Stripe is on it. They made a documentary that’s coming out very soon, actually, I think, and it’s as part of the SF Film Festival currently, and it was more of like looking at his whole life and his impact legacy and also the more recent like environmental stuff he’s been doing, which is much more comprehensive and honestly a much better idea. But I started this project my freshman year, so I’m pretty committed at this point. 00:02:26 - Speaker 3: Yeah, yeah, for sure. I think it just shows that there’s a lot of interest in his work. 00:02:30 - Speaker 1: It’s really interesting, like, the deeper you dig, the more you find, and the more like of a web you discover, especially on Wikipedia, in the best way, so. 00:02:39 - Speaker 2: You seem to enjoy some unearthing the history of weird characters here, your collection of computing history, folks. I’ll link that in the show notes here as well. But before we get on to that, I think the folks would love to hear your background. You’re quite early in your career and yet already have a very impressive CV here. You’ve worked at Figma, you’re now at Notion, and you just finished a thesis at UCLA, so I think we all just want to know. What’s your productivity hack? How can we all be as uh as productive as you so early on? 00:03:11 - Speaker 1: Oh God, that’s not. First of all, my little background blurb. My name is Molly. I’m currently a student at UCLA. I studied digital media, and I’m in my last year. I only have a couple more weeks left, which I’m very excited about. 00:03:26 - Speaker 2: Wow, congratulations. 00:03:27 - Speaker 1: I know, so close, yet so far. 00:03:31 - Speaker 2: The senioritis kicked in already? 00:03:32 - Speaker 1: Oh man, yes, it has been very, very present in my life ever since like September of last year. Every single week is like counting down the days, but we’re getting there. And I’m currently designing a notion, and I will be returning to Sigma at the end of the year. And I come from more of a background in visual design and storytelling, specifically filmmaking, and I got my start leading design at a startup in the Bay Area while I was transferring schools, and through that I found product design specifically, and I found that it was like this very unique fusion of the creative and the analytical at the same time, that just really clicked for me. And ever since then I basically was just exploring kind of different industries and company sizes and problem spaces more broadly, and through that and working at startups and Sigma and most recently notion, I found that creative tools were what I was the most like just completely pulled towards and really wanted to just dig deeper and explore what impact they could have. I think that there’s something about making something that enables other people to make other things that is just like incredibly gratifying for me in a way that no other product design projects really touch. And I think more broadly, I’m really interested in the combined power of like design and tech to foster creativity and community across the board, and that was definitely like the inspiration behind this thesis and also like a through line to just things that interest me across the board and in terms of like doing school and work at the same time, I think it’s really just about The space that the pandemic has provided for free time, sadly, I definitely have profited. 00:05:30 - Speaker 2: Uh, so that’s your productivity hack is be doing this all during a massive lockdown that prevents other kinds of fun things that. 00:05:38 - Speaker 1: Exactly, it’s the best one. I highly recommend. No, it’s kind of the worst, and I feel honestly a little bit guilty to have like done so well during such a terrible time, but then at the same time, I’m very grateful. So there we have it. 00:05:55 - Speaker 2: Yeah, a lot of that hits on things that speak very much to me, and I think others that I feel like are in our field, however you want to define that, they’re making tools to help others create, which I think is in many ways a harder or more interesting product design problem. It’s one that maybe historically has not been seen as very sexy when you think of, I don’t know, productivity tools, whether it’s a word processor or a video editing tool or something like that. They don’t have the same kind of sleek attention to detail that often more consumer products do. Maybe that’s starting to change now and at least I hope a little bit this concept of a tools for thought field which we talked about with all the way back in our podcast episode with Andy Matuschek about kind of transforming. From the stodgy idea of, I don’t know, word processors have been the same for 25 years and very utilitarian and just the word design doesn’t get associated with them. Maybe that’s starting to change now, which I’m very excited about. 00:06:53 - Speaker 1: Yeah, I completely agree. And also it’s interesting because I think I’m young enough to have grown up with those tools and like been in Photoshop from a very young age, and there’s something. Definitely about them that is just so intimidating and so difficult to comprehend from somebody who is not like acclimated to the environment and doesn’t understand the principles that they operate on, and I think that that’s slowly changing, but it’s definitely like, it’s still happening, we’re still figuring out the best way to do it cause it is complicated, and they’re offering a lot of different things in the same place. 00:07:27 - Speaker 2: Yeah, one thing about computing in general and creative tools in particular is they’re just so new on a relative time scale. We’re still figuring it all out. There’s some established practices, but when you compare it to a lot of other fields where I don’t know if you’re a woodworker, the best tools for doing woodworking have been slowly refined over the course of hundreds of years, and here in computing we’re still kind of just banging two rocks together to figure out how to make things, so. 00:07:52 - Speaker 1: 100%, yeah, we’re definitely still figuring things out. 00:07:56 - Speaker 2: Have you found there’s any particular, I don’t know, skills or approaches that came from this kind of film visual design background that you talked about that translate well and give you unique insights that maybe some of your colleagues don’t have doing digital product design? 00:08:10 - Speaker 1: Hm, that’s an interesting question. I mean, inherently audio and video software is. Incredibly hard to understand, and I think that it takes a preexisting like knowledge and investment and really being able to go into these tools that are just like an incredibly blank slate, and they offer so much possibility, but where it is is like up to you to really figure out and even understand what you’re looking for. And so I think Having that background in feeling comfortable just tackling these like interfaces that are very unfriendly, honestly, it does help, and I think it also helps me to understand a lot of the principles that some of the other creative tools are just beginning to adopt, and there’s a lot of like efficiency and abstraction work that has been developed and cultivated in Video and audio tools that is just beginning to kind of pop up its head in just more simple, more like consumer everyday creative tools. And I also think that fundamentally having a background in like video is also just like a Background in storytelling, which is applicable everywhere, and I think it’s becoming even more applicable in tools like design tools and writing tools and being able to help foster those stories and also to kind of weave in the story of the tool is kind of an underrated thing. And it’s not the primary concern, but it definitely is a piece of the broader puzzle of getting people to feel comfortable enough to create in the tools. So there’s something interesting there, but it’s definitely still in its nascent form. 00:09:52 - Speaker 3: Molly, it’s interesting that you mentioned growing up with complex tools like Photoshop and that being a help in using other tools in the future. I didn’t grow up on Photoshop, I grew up on Kipics. I remember when I first tried to learn programming, the tools were so foreign and unapproachable that I almost completely bounced off the field. It was like VI, which is an incredible maze and like all the Java server side stuff. It was just completely wild. And it was only because of Ruby on Rails that I found something that I could basically get working and running end to end. And once you go through it a few times, you kind of calibrate on like how terrible things should be when you’re first learning something. But I do think a lot of people just bounce off these complex Pro Tools for a reason like that. 00:10:32 - Speaker 1: 100%. I feel very lucky to have become comfortable in them at a very young age, and that was through like pirating Photoshop and getting gifted a Wacom tablet and just really starting by making really, really rudimentary like digital art and things like that. But it definitely was like the type of thing that I would try to teach my friends and things like that and kind of bring it into other areas, and it was just not adopted. It was like my understanding and knowledge of the tool was something that I definitely took for granted for a very long time. And it definitely has made me think differently too about creative tools across the board of like, wow, if you really invest in like getting people in these tools at a young age and really acclimated and understanding how they work, like there’s a lot of potential there, but it’s not scalable. So like there has to be other approaches other than that, so interesting problem that we’re only beginning to run up into. 00:11:31 - Speaker 2: So our topic today is computers and creativity, which is not at all coincidentally, the name of your thesis which you published recently, and of course I’ll link that in the show notes here, and I recommend everyone go read it. Not only is it great content, but a beautiful presentation that really takes good advantage of sort of the web as an article format. So naturally folks can go read it, but maybe for those that haven’t yet, just to prime the discussion here, maybe you want to give us a brief summary of its contents. 00:12:01 - Speaker 1: Most definitely, yeah, so my thesis is really about how can digital creative tools best augment human creativity and collaboration. And it’s really looking at the potential of creative tools as co-creators with human beings and examining kind of returning to the original vision of creative tools and how we can extract some of the things that were realized and some of them that weren’t and kind of analyze that for the present of creative tools and to kind of contextualize that with an observation, from my vantage point, I really think that the power of tools lies in their ability to Amplify human action or thought versus the power of human beings is really about our ability to think creatively. And so if that’s true, then why do computers often ask us to act as almost execution machines ourselves to create something when that’s like very uniquely the computer’s strong suit. So the paper delves into a lot of different areas and kind of the history and analyzing the present, but The main point here and like the TLDR that I kind of reach is that to foster optimal human innovation, digital creative tools really need to be interoperable or basically talk to each other. They need to be moldable or customizable to different phases of the creative process. They need to be efficient abstracted, which is similar to moldable. They basically just need to Accommodate more or less complexity at different stages, and lastly, they just need to be community driven so that people can be inspired and get help when they’re creating. So that is the very abbreviated version of my very long blog post, but I’d love to dig deeper into all of that. 00:13:52 - Speaker 2: Yeah, all of that resonates very much with stuff we’d love to talk about here and things Mark and I spent a lot of time talking about. Yeah, I guess maybe to dig in a little bit on, for example, that first section where you look back at what you called the original vision or or sort of the history. And folks who’ve been banging around in the tool space for some time will certainly recognize a lot of this, Engelbart and K and Hypercard and Flash, and Dynabook and so on, but I think it’s one of the nicer collections of summarizing all that, that isn’t, I don’t know, a super long book, so it’s a nice way to get up to speed on that. Now, it is interesting with Sort of look at this history, which I think is often presented as kind of yeah, there was these amazing visionaries who saw the potential for computers and creativity, sort of laid out a vision way back in what seems like just the Stone Ages to us, the 1960s, the 1970s, and then in some ways we lost our way and we ended up with, I don’t know, social media and Kind of lock down appliance like smartphones and in fact there’s this glorious world of I don’t know, small talk and dying a book and mother of all demos style stuff that we still need to build or we haven’t built or something like that. Do you see it as like that’s an unfulfilled vision or the flip side could be, OK, well, they had some cool ideas, some of those worked out, practiced, some of them didn’t. The reason. We don’t have everything there is that maybe some of it wasn’t practical, and I’m never fully sure how to think about the kind of lionization that we do some of these past figures. 00:15:31 - Speaker 1: Yeah, I feel the same way. I think that there’s a lot of tension and just basically more analysis that needs to be done there. I think that it’s very easy to put these people on a pedestal and just say, wow, look at this incredible vision that they outlined, and we do that, and I think that they do present some really compelling ideas and their way of framing computers as being a tool to almost augment human intelligence is something that I particularly am pretty compelled by, but obviously a lot of their ideas did fail and there’s reasons for that. Um, and I also think that they were operating in an environment that was largely kind of independent from the actual business environment and like the technology sector as we see it today. So, like, will those ideas actually thrive in reality and especially in the consumer preferences and like relationship we have with tech today. Maybe not, but I think that they Still present some really interesting kind of principles and ways of looking at computers that we can definitely take some inspiration from. And I also think that like we rely on a lot of the principles that they established. And I think it’s just really important to like recognize that and kind of piece apart what we took and what we didn’t, and maybe what we can take more of or what we should reconsider. I just think that fundamentally This is great of history, especially in a field like tech, which is kind of in some ways pretty disconnected from its own history. And there’s almost kind of like a pride in that of moving so quickly that we don’t even look to the past. 00:17:10 - Speaker 2: Yeah, or I’d almost listed as a sort of willful disregard of history because I think there’s the classic, I don’t know, why combinator or startup founder. Out of school, it’s actually their naivety that allows them to reinvent, you know, they’re not dragged down by the legacy baggage of how we do things today. They can just think about it in kind of a green field way and dream up a new idea and maybe technology has changed enough that there’s new parameters and they can really do something new, but that comes at the expense of, well, actual naivete and reinventing everything. And not using scholarship of the past to learn what’s worked and what hasn’t in order to kind of stand on the shoulders of giants or build the way that any other field would. Of course, you learn from the past and then you use that to inform what you should do going forward into the future. And yeah, the young naive startup founder or other types that we hold up as our role models sometimes in technology are not into scholarship of the past, let’s say. 00:18:11 - Speaker 1: Very well put. I couldn’t agree more. Yeah. I’m very curious though to hear what both of you think of, as you put it like the lionization of Engelbart and Kay and all of those people, cause it seems to be a pretty disputed topic. 00:18:25 - Speaker 3: Yeah, this opens a quite an interesting door for me. My sense is that a lot of people look at what these early pioneers did, and intuitively they feel like that is good, it should have succeeded. Why don’t we have this? They had it 50 years ago, what’s going on? And at the same time, the current reality, like you were saying, is not exactly that. And I think it’s really important to understand why that is, and I think you were alluding to what’s happening where a lot of this study and analysis has been at the level of the tools. So it’s like what’s on the screen, how do you program it? What’s the user interface even, but there’s an entire complex system around how software is developed and used. And like you were saying, I think the reason that the vision for the software and the reality of the software don’t line up is because we haven’t understood or Accounted for how that ecosystem works. So sometimes I call this the political economy of software development. There’s weird path dependence, there’s economic incentives. You got to understand all of that if you are going to understand how we came to where we are now. And on the flip side, if you want to predict and guide the future in that direction, you need to become a sort of political economist of software and get in not only the interfaces in the code, but also the funding and the incentives and legal stuff and all that. 00:19:39 - Speaker 1: Hm, yeah, that’s very well put too. I definitely agree. I think there’s so much complexity and also just like context that’s missing from so many of the analysis of these past tools, and they’re very like independent floating ideas versus actually tangible grounded concepts that could be turned into something real. 00:19:58 - Speaker 2: I think a lot of what you both said to me kind of just describes that these folks were visionaries in the sense of also being sort of ivory tower academics or whether or not they were an academic, they were purposefully somewhat disconnected from, for example, commercial realities and that is part of what allowed them to have big dreams. And those dreams are still inspiring to this day, but then if those dreams are to become reality, at some point they do have to be connected to the real world, and this is a huge problem in research generally, which is there’s a technology transfer, how does something go from the lab or From that more idea space that science excels at into something practical that you can use and there isn’t a good path. This is something that the I can switch research lab where Mark and I are both participants is trying to improve upon, but yeah, it’s a really hard problem because a lot of times the same people, it’s a very different kind of person that can have the big dreams versus that can kind of make it into reality. And when you think of one of the most famous examples, Xerox Park, and some of the ideas they had there, and Steve Jobs basically got a glimpse of it. He was a guy that was good at actualizing things. He got a glimpse of it and then basically stole it and then went and made a practical thing. And of course, often the visionaries feel, no, you left out important parts, but leaving out parts is actually part of how you make something come to reality. So I don’t want to dismiss these historic folks as The academics that don’t know how to bring their ideas to reality. In many cases they did make great working software or even hardware that in some cases went on to turn into underpinnings of tools we have today, right? Small talk turned into Objective C and that, you know, fed into Ruby and. SWF and other languages that, for example, we use heavily on the Muse team, you know, these are very much things that are in the real world. But maybe there is an acknowledgement that the big dreams aren’t enough, you need to do something to connect it to reality. Yeah. 00:21:58 - Speaker 3: Yeah, and I also think there are two separate axes here. So there’s the axis of What are you looking at? So it could be pure software, or it could be called the software ecosystem, and then there’s the axis of visionary and idealist versus in the weeds pragmatist. And I think in our discussion that we might have been kind of conflating those two things, but in fact, I think you can have, and I think we need more visionary idealists on the political economy side of software. Probably the closest thing we’ve seen to that is the original free software movement and that obviously got some traction and made some progress, but I think we need to re-date that for the world of cloud and mobile, where the original free software vision basically broke down, I would say. Just as an aside, this is one of my favorite creativity techniques where you identify the axes, you know, the rows and the columns of the spreadsheet, and you label each row in each column, and you see often you know what the entries in certain of those boxes are, but you can perhaps intuit that one of the boxes hasn’t been filled. yet or given a name or explored and just by sort of drawing the map like that, you can identify new quadrants. There’s a cool research paper that I read on this about data structures where they kind of identified all the different ways you can build data structures and then found the blank spots in the maps and went and synthesized those new data structures just on the basis of this cell in the spreadsheet should exist. 00:23:24 - Speaker 1: So fascinating. That is awesome. I can like visually imagine that in my brain. It’s great. 00:23:30 - Speaker 2: Yeah, if it sounds like I have a critique for some of these historic visionaries, let me bring the positive side, which is I do totally agree that they did lay out a vision for computing that is grounding in a world where we do seem stuck in, yeah, social media, consumer, I’m not. Exactly sure what everything is oriented around commerce and again, things that are all good commerce, entertainment, these things are fine. I consume these, but the reason I got interested in computers at a very young age is seeing their potential for creativity and unlocking the noblest parts of the human spirit. And it’s a good reminder to go back and look at some of this history, maybe especially because these folks didn’t have any of that prior stuff. Computers were still so new, particularly personal computing was essentially, you know, they were in the process of inventing it, thinking what could people do if they had access to computers with graphics and networking and all the things that nowadays we take for granted, but they dreamed of something very different from the world we have today, and that can be very grounding to look back at that and take a bit of a blank slate from where we are today. So yeah, I personally take a lot of inspiration from all their work as well. 00:24:44 - Speaker 1: Totally agree. I think there’s something too very compelling about. At least for me when I was reading these texts, how they kind of frame computers as partners with human beings, and I kind of integrate that as like a co-creation relationship, which is definitely a very squishy one that I think we’re still defining, but there’s something that feels very like a breath of fresh air to think about the computer as like a counterpart instead of something that is Potentially replacing us or stealing our attention or something of that sort, even just asking so much of us. It’s more like, oh, the computer is here to help. And I think that that in particular is something that I hope we optimize more for in creative tools specifically, and there’s a lot of potential there. 00:25:26 - Speaker 2: Very well said. I do feel like more often than not in the modern world you’re stealing your attention as one example. You’re sort of fighting against the computer and perhaps it’s not the computer itself, it’s the whole world of computing, the internet, or email inboxes, notifications, the way that the web works, and so on that you’re often either fighting. Again this thing trying to make you do things you don’t want to do or take away your attention or distract you, or it wants you to do its chores, you know, click this, update this, do this, fill out this box, and it should be a tool quietly waiting for what you’re asking of it and to, as you said, co-create and help you in what you’re trying to do. I like this quote from the original Tron movie which is at one point the bad guy basically says, look, you know, the systems are overloaded because we don’t have time to handle every little user request, and the guy he’s speaking to is kind of the wise and old computer sciences, actually user requests is what computers are for, and I feel like it’s so often forgotten. They are here to serve us and sometimes it feels more often the human has to serve the computer or perhaps the business. Interests and I’m a capitalist, so don’t get me wrong, but the business needs, the KPI of whoever designed the product, it’s asking me to do things to serve that rather than my needs. 00:26:49 - Speaker 1: I love that quote. That is fantastic. I want that on a bumper sticker. 00:26:54 - Speaker 3: It’s great. Related to this, Molly, one thing I really appreciate about your thesis was you surface this idea of, I forget what you call it, but I would call it like vibe, basically, it’s like emotion, motivation, valence. I think that’s so important because if you have software that’s giving you a hard time, it’s not just a tactical or mechanical issue. It’s now you’re in a whole different mindset of, uh, you know, I’m dealing with the check boxes or whatever, and you’re much less likely to be creative and to keep doing it going forward and so I thought maybe you could talk a little bit in your own words about that aspect of creative software. 00:27:23 - Speaker 1: Yeah, I think that this is one that is like just beginning to form and it’s mainly because we’ve advanced to a point where there’s enough competition that we can actually focus on vibes or whatever you want to call it. And I think when I was writing about this, and it’s something I think about a lot. I definitely think about software like Figma, which I think that there’s something to be said for just bringing a more playful approach and just treating the user with more respect and really trying to validate them, not get in their way. It really comes back to establishing the baseline of being like a very good piece of software that does the job well. But beyond that, how you can actually differentiate the piece of software, especially in creative tools, it’s really just about like the personality and the kind of attitude that the software brings to the user, and I think you see that reflected in the way that it talks to the user and the colors and just little visual things and even just like the ambient environment of their landing page. It’s just very small things, but they do add up, and it in the increasing A larger landscape of creative tools, people are going to pick the one that they identify more with. And I think that that is incredibly interesting to me personally, from like a storytelling perspective of like how we can try to create things that are more inclusive to more people and just try to get more people in the tool that might not have a background and experience scaling these tools and really navigating these usually dark gray interfaces. But yeah, I think vibe is, it’s a nascent field for software. We’re still figuring it out. 00:29:04 - Speaker 2: So there’s a section in here titled Standardization, which I think is about file formats and ultimately is how tools work together and actually something we’ve talked about on this podcast before, including with Balant from Kraft talking about the different ways he wanted to try to have essentially toolmaker humility, which is realizing that the tool you were creating for your users. One of many that they are using and you should try to as much as possible, be a good citizen and work together, although in many ways it seems with the highly sandboxed world that we get in kind of mobile apps as well as to some degree, maybe the web and cloud, you have these silos and they just aren’t really designed to work together. So what do you see as kind of the future going forward from here for, I don’t know, tools working together? 00:29:51 - Speaker 1: I think honestly, if I had to pick one concept for this project that I really like strongly stand behind and is like the hill that I’m willing to die on, it would probably be this one. I think that the lack of interoperability or standardization between digital tools today really it means that all work created within a tool is confined to that tool, and to me that seems very clearly antithetical to creativity. And specifically the collaborative aspect of creativity. I think that there’s so much to be said for tools amplifying the power of our brains and really taking over the mechanical aspects of human thought and limiting creation to a single piece of software’s capabilities is just kind of crazy if you step back and think about it. And I just think that standardization and having tools talk to each other would just fundamentally change the tide of how we use them and introduce in more collaborators and really just expand the project’s constraints beyond any One tool. And this is a really hard one. Like, solving this problem is something that I feel like is a huge problem that I just don’t even know how to approach because it is pretty much in direct contradiction to the current business models of most creative tool companies. But I’d love to hear both of your thoughts on this because it’s a huge topic and it’s definitely one ripe with controversy. 00:31:28 - Speaker 3: Yeah, and just to expand on the motivation here, I think collaborating across tools can mean several different things. It can mean, like you were saying, you have a given project and at different stages of the project, you want to be able to use different tools. That’s one case where an open format would help. You might want to collaborate with other people. And they might want to use their own tools, which is different from yours. It’s another case. And also there’s this element of time where over time software tools tend to atrophy. Companies come and go, you know, platforms change, but you at least want your data and to be able to carry that with you in some sort of archive at least. So there are many cases where having such interoperability would be helpful. Yes, it’s extremely hard and by the way, I think this is a prime example of the political economy issue. It’s very easy to say we should have X, and even if X is relatively easy to do, which is not in this case, there’s still this huge issue of the. We should. That’s quite the weasel phrase, right? Really, it’s, if we were to accomplish this, we would need a bunch of companies or individual developers to temporarily make more work for themselves, lower their profitability, make their products worse for the customers in the short term to get to some other global maximum. It’s a case where the coordination problem is really important. 00:32:36 - Speaker 1: Completely, yeah, and it’s definitely like invisible work that does not really result in much actual profit for the company, it’s much more of like a long term investment that would require all the companies getting on the same page and really agreeing to terms and it’s really a long term relationship with each other too, which is kind of crazy to even fathom how that could happen. 00:33:00 - Speaker 2: I guess to highlight what I consider a bright spot or a positive version of this, I do think files on some of these flat file formats, which includes plaintext.txt, markdown. Image formats, PNG, JPEG, probably yeah, MP4 movies increasingly audio clips, PDFs. Now PDFs come with a lot of baggage. They are very complex to render, but ultimately there are pretty standardized ways to do that. And importantly, yeah, PDF does not demand. You have, for example, Adobe Acrobat, maybe it did at one time, but now it’s a tool you can open with. very standard viewers on any platform you can edit it and so on. It’s something we strive for in Muse because we kind of have this value but again where we are subject to the same constraints as others working with, especially making an app on a platform like iOS. But for example, we do store most of the raw, you know, when you drag an image in, we store that as a raw image in one of these standard file formats and in fact, if you do a bundle. Export you just get a zip archive that it contains as much as possible formats, you know, the ink is in SVG and that sort of thing. So we try to do that as much as we can. Now in practice, a muse bundle zip archive that has a bunch of loose media in it and is not sort of you know arranged on this board maybe is of mixed value. So I guess that does lead into maybe one of the more standard objections. The standardization, which is essentially that it is maybe counter to innovation. It creates a lowest common denominator. If every markdown editor, for example, has to support that format, if you want to do something interesting like make it really easy to embed video with captions of particular time clips, and that’s just not part of the format, so you just can’t do it or you break away and do something, you basically break the format in order to add that innovation to your tool. 00:34:51 - Speaker 3: I do remain optimistic that it can, and in fact will be solved. I think we will get a general purpose data medium that’s kind of like JSON is for the synchronous single user case. It natively allows collaboration. Obviously we’ve been working a little bit towards this with automerge and so forth in the lab, but I think it’s eventually going to happen, but it’s gonna take a lot of work and I suspect it’s probably not gonna happen by a bunch of people getting in a, you know, enormous room and everyone saying, OK, let’s form the consortium for X and do a two year study, and blah blah blah. I think it will be an organic, messy process led by some champions somewhere, whether they’re individuals or companies, but I do think it’s possible. And when we get there, it’ll be great. And like you were saying, we are, I don’t know if you were saying this on the podcast or if I read this in your thesis, but we’re relatively early in this world of collaborative software. It seems so obvious to us that you have Google Docs and Figma, but that’s I don’t know what, 10 years old or something, so also just gotta give it a little bit of time. 00:35:49 - Speaker 1: Completely, yeah, I think we’re still figuring it out and really trying to understand like what to prioritize and what is the most important in the long term and just beginning to think long term, that this is going to be around and I think we’re still like even developing the social norms and values that we as like the users and the makers like care about. There’s a lot of development still happening. There that is like incredibly interesting and I think it’ll all shake out OK, but we just have to like really nail down what’s important and how we’re gonna like think about this in the long term, because even though things like standardization are not particularly enticing, like if we want it to be around for a while and if we want our work to be compounding, then it’s like you said, increasingly important. 00:36:36 - Speaker 2: And we’ll offer as a counter example to the, you know, standardization and innovation dilemma, the web, where essentially there has been a lot of innovation on the web, but no one company owns that format, and perhaps there’s some complaints you can have about a particular browser monoculture at any given time, Google Chrome at the moment. It is truly an open format, you can parse it with a lot of different tools, and it will have, I think, the longevity that will go beyond any particular browser. 00:37:04 - Speaker 3: Yeah, and I think there’s some, we do a whole podcast on protocols and stuff, but I do think there are some important lessons in the web stack, one of which is they’re relatively thin layers, or at least the layers that work the best are pretty thin. So whenever you make a layer that’s an abstraction or protocol, you get the benefit of aligning some decision space, and if it’s a relatively thick layer, you get the benefit of you’re aligning a lot of decisions together, so there’s a lot of interoperability, but then you run a sort of exponential risk of one of those things being wrong and then the whole game breaks apart. So the layers for the web are thin enough that, at least in the lower layers, you could plausibly say there aren’t huge mistakes, such that people would want to go off and do something totally different, at least for the original web use case. So here in this case, I think we’re more likely, for example, to have success with the interoperability standard that’s more like JSON and less like address book standard format, right? Something that’s less like the business objects, or if you have those, they emerge kind of organically out of more general purpose data medium, so I don’t know, we’ll see. 00:38:03 - Speaker 1: Can’t wait to see. 00:38:04 - Speaker 2: We’ve hinted a few times, I think you’ve mentioned a few times kind of the the relationship between collaboration and creativity and the co-creation element, and from my perspective, this is a relatively new element of computing creativity. You mentioned using Photoshop, growing up on Photoshop. That was a private activity. Maybe you could send a file to someone else at very great effort by putting it on a floppy disk and carrying it over to them. But you didn’t really do that very often. It was typically a private activity and furthermore, I think for many creativity is often something that is a little bit done in private. It’s sort of this vulnerable act, but then perhaps that’s changing partially because of collaborative software like Google Docs and FigMA and Notion and others. And in fact, we had a whole episode with Nicholas Cline from Sigma, who I think you might know, basically talking about, he’s also a younger guy, and I think, you know, for him, there is less of this creativity is this thing done in private, of course you make stuff together with friends, with colleagues. That’s just how it’s been. So maybe that’s culture is changing partially because the tools are changing. But for the purposes of computers and creativity and how you see it Molly, what do you see as the relationship between creating together versus a more private activity? 00:39:25 - Speaker 1: I think this is a really interesting one, and I think we’re still figuring it out from my perspective. I think creative tools, ideally should accommodate for both, um, from my perspective, I think right now they kind of still fall into two buckets of either solo or collaborative and collaborative in like the Google Docs or FIMA sense. And I think there’s immense value in having tools that do both. They optimize for incredible solo creation and incredible multiplayer building upon each other’s ideas, and I admittedly, I think I lean more in the direction of like how Nico thinks about these things of allowing in more collaborators earlier on and feeling comfortable doing so because I grew up with these tools in a fully collaborative Google Docs form. But I do think that what’s interesting here is that these tools are so new, and we’re still just like as human beings figuring out what is expected and like what does ownership mean in these environments and just trying to establish like social norms there, and that is like a very squishy one that I think will just take time. But for me, this really just like reinforces the value of moldability and ideally the tool would just accommodate, like I said, both solo and collaborative work and provide you like the resources and tools that you need to create those environments for yourself because I think Tools being less opinionated about an assumptive about what you need in those modes is going to be a great thing. I would love to see for tools to give you the features that you need to really create your unique creative space, whatever that looks like. And I think this also comes back to what Niko was talking about when he was talking about like the flywheel effect of collaboration. And really creating in the same spaces and building upon each other’s ideas. I think that that is a very different mode than like the solo creation kind of brainstorming, but ideally the tool could scale to both. So that’s like my current thinking. But I think that’s really hard, and I think that that’s two completely different things and optimizing for very, very different, almost in some ways audiences like those are sometimes the same person, but oftentimes they’re not or they’re a different subset of people and I don’t know, I think news is an interesting example here too, and I’m curious to hear what both of your thinking is because obviously that is optimizing more for the generative like solo environment in a really wonderful way. 00:42:03 - Speaker 2: Yeah, the challenge of a true thinking tool and really, you know, we’re trying to cover the very earliest part of the ideation funnel, let’s call it, or the creation funnel, which is that early ideation where you normally use a sketchbook or a whiteboard, something that is not at all intended to be a final artifact, but is about figuring out what you want to make in the first place or making a decision or just forming up your vision rather than any deliverable artifact. And that is something that does tend to be maybe creativity at its most private, like something about a sketchbook is just something that you really feel is truly private. And in fact, you know, we’ve been looking into things to try to add some collaborative capabilities, hopefully building on our values around privacy and sort of a calm sanctuary and all that sort of thing, but it is a real challenge. We could easily lose what’s good and we have even heard from Users and customers, they say no, or they’re worried, right? They say, I don’t necessarily want you to add that because then it’ll turn into this more chaotic environment that I associate with these team spaces, for example. So, I think there is a way to cut that Gordian knot, but it’s a huge design challenge, obviously. 00:43:16 - Speaker 3: Yeah, the goal is definitely to eventually accommodate all the different types or topologies of social or non-social collaboration, and it is my hope that we’re able to eventually do it in one tool, because as you have a project, you tend not to want to be jumping around through different tools, or at least to do so only with very good interoperability, and every time you do do a jump there’s a bit of an activation energy costs. And yeah, as we’ve studied the creative process by talking with creative professionals and in other ways, we have found that there’s maybe a half dozen different typical topologies. There’s you’re basically ideating alone, there’s call and response feedback, there’s real time, kind of everyone at the whiteboard collaboration, there’s a sync building up a corpus together like a tracker, uh and there’s like presentation and sharing in real time. And I think it’s possible to get all of those in one tool, but it will take some time. The reason we started with the initial ideation phase was a felt like that was the most underserved, and the one we had the most unique angle on, and also there’s something to doing the first step first, if you will, just in terms of building up the full user journey over time. 00:44:25 - Speaker 2: One thing I do imagine with any tool that has both collaborative capabilities as well as solo capabilities, and by the way, exactly as Mark said, collaborative actually covers a whole host of different modalities, even just talking about synchronous versus asynchronous, for example, I think one of the big things we’ve learned. From Google Docs, it’s not really about the real-time collaboration. It’s about having a document you know is up to date and in practice it probably is asynchronous. You sent it out, you shared it out, and someone added comments or added something to it while you were asleep, and then you’re looking at it again later, so it’s asynchronous, but you know it’s up to date. But I think if you’re clever or if you’re able to find the right combination, it shouldn’t be hopefully you’re serving those two audiences or the whatever all the modalities are, but that each one needs their own features and then pretty soon you’ve got this overstuffed product that does too many things that in fact you can find things that serve many or all of those cases. One great example to me, which is very much about creative process and how you work as version control as a developer. The first really good quality version control system I used was something called CBS many years ago. It’s kind of a precursor to this version, and then that was kind of replaced by Git in the world of decentralized revision control. But in any case, when I discovered revision control was sort of pitched as well, this is so you can work with someone else. And so in theory you don’t need it if you’re on a solo project, but I really quickly found, oh actually this is really nice. It brings a sense of OK, I’m going to work on something for a while and then package that up into what I would now call a commit, give that committed name. I can look back at my own history. I get kind of a log, you know, an undo, sort of like a large scale undo history, but it also creates a lot more structure for my own thinking about it. Obviously that’s made its way into now this collaborative space as well, which is when you’re writing the commit message, it’s for yourself, understanding what you’ve done, but also for your colleagues, so they’ll be able to see what you’re doing. And so it feels like a lot of the tools of revision control or a lot of the features of it, including how the discs work and how commits work, and all that sort of thing, both serve an individual working on a solo basis, maybe collaborating with themselves through time, you might say, and a small team or a big team working together on something. 00:46:44 - Speaker 1: Yeah, 100%. I think it’s also just like retraining ourselves a little bit to once we acclimate to the standards of like a collaborative tool or something that’s optimized for that, usually that actually directly translates to the more solo experience, not always, but I mean, having different practices in different areas, that doesn’t seem particularly intuitive either, um, and a lot of these. Processes for organization are applicable everywhere. There’s a lot of crossover between the features. I think it’s just about like establishing where we are and really like, I think making people more aware of where they are in their creative process is something that’s going to become increasingly relevant to, and that’s something that we’re still kind of figuring out in creative tools is like, which tool is used for what and like how do they, again, how do they talk to each other? Can they talk to each other? And how are we going to like use them together, which is like the bigger question and very difficult today. 00:47:39 - Speaker 2: I’m definitely a fan of the pipeline approach, at least in my own work, which is, it’s less about that I want to use 3 different tools simultaneously. At one stage, but more at a particular stage, I’m using a particular tool, so that’s the case for something like writing, where when I’m trying to figure out what I want to say, I’m using news or sketchbook or some other ideation tool for thought thing. But when I’m writing, that’s actually not the right thing. Now I want a writing tool, a scrivenner, a craft, a Google Docs. But that’s not my publishing platform. From there I’m going to go to something that’s usually on the web, but it might also be in PDF or it’s Lawtech if it’s an academic format, and sort of at each stage, in a way, the transition to the new tool, which does involve some labor to translate it across, even when they’re fairly interoperable. For me, it’s almost good for my creative process because there’s this little ritual of now I’m ready to jump over into this next stage, it’s graduated. 00:48:37 - Speaker 1: Totally, yeah, and I think acknowledging that process and paving the way and making it as seamless, but also I don’t know, building in the opportunity for you to use that as a point of reflection and almost editing cause I totally relate to that as well as like moving from ideation to first draft or something like that. That’s really like also uh editing and refinement moment as well, and you don’t want to cut that out completely. So it’s again kind of letting people choose how they want the tool to behave. I think it’s gonna become increasingly relevant for creative tools. 00:49:12 - Speaker 3: Yeah, I think this idea of acknowledging is really important. So there is the underlying platonic ideals of multi-step creative processes of social creative processes, and in fact it’s like always has been, and we can link to the always has been me here. But people have always been, you know, taking pictures of their whiteboard under their phone or like shuttling around USB sticks in the case of social collaboration. So I think if you do the careful ethnographic research and take off your blinders about what software we currently do or don’t have, you’ll see these underlying patterns and a lot of what we’re doing with Muse and a lot of what we talked on this podcast is how do you align the software with those platonic ideals of creative work. 00:49:51 - Speaker 2: All right, Mark, I think you’ve signed yourself up to create the always has been meme with that content that we can include in the Twitter thread for this episode. 00:50:00 - Speaker 3: All right, I’ll bust out the meme editor. 00:50:03 - Speaker 1: Cannot wait to see it. 00:50:06 - Speaker 2: Now when I’ve worked on really long pieces, sometimes 5 or 10,000 word pieces we did for ink and Switch or the 12factor app or other larger pieces, for me it’s the case that you ship not by finishing writing everything you want to write, but by choosing to cut out a lot of it. And so I’m curious what things might have ended up on the cutting room floor that you think are worth telling us about here. 00:50:29 - Speaker 1: Oh man, so many. I completely relate to, I think it’s so hard to know when to ship something like this, and my current rule of thumb is like, if I have way more questions, but I know just how long it’ll take to investigate them. And it’ll kind of distract from the focus of the piece. That’s when I’m like, OK, maybe I’m getting closer than I thought I was. But in terms of ideas that I’ve cut, this project actually started off focused on flow state, and I was very interested in how software could facilitate more flow state in human beings. And that’s a very broad question. I realized that’s exactly why I cut it, is because it is actually, the deeper you dig into. flow state, the more you discover that it’s very subjective and the definition of it is still kind of up in the air, depending on the discipline that you look at it through. So while it’s super interesting, that is definitely something I cut, but not before doing a lot of research on kind of the psychological conditions and what goes into flow state and how people report to experience it, which I think is really interesting still. And I would love to write a whole another thesis on that. But it’s still a tough topic to nail down. 00:51:46 - Speaker 2: And just to briefly define that one this is probably one of the most quoted or cited concepts from modern psychology, which is there’s a state that I think originally they were looking at athletes maybe when they’re sort of at their peak performance, but maybe in our own lives we’ve experienced this on running. Or doing some kind of sports or something where you’re just in this well state of flow where everything seems to come effortlessly and it seems like you’re higher, somehow you’re questioning brain narrator shuts itself down and you’re just in the moment in a way that’s very satisfying. And we talk about this a lot in the tech industry because of, I don’t know, even just a simple thing like making sure you have big blocks of time to really focus on stuff. We’ve talked about deep work, for example, that concept here before, but the idea that you want to optimize for flow state and yet technology and especially The internet now is so kind of anti-optimized for that that it wants to offer you information about things that are happening in the world and messages and notifications about everything that in the right moment can be connecting, but when you’re in flow state can be distracting. 00:52:53 - Speaker 1: Yeah, it’s a really rich topic and it is interesting too something I realized in some feedback that I actually got when I was focused on flow state specifically was more academics, but they were like, what is the relevance of this? This seems like something that you’re just throwing in as like a buzzword to get people to immediately understand that you’re talking about deep creativity, but do I actually know what flow state is? Not really. And I hear that. I think that that’s true. I think we still need to kind of define what it looks like in different contexts. And that was kind of the reason that I decided to broaden up the inquiry to just look at creativity more broadly, because I think it Functions in a lot of forms, then you don’t have to be completely 100% into your work and thinking of nothing else and just ideas are flowing. Like there’s other forms where it’s more generative, or maybe you’re building upon other people’s ideas, and that’s not encapsulated into flow state, which is interesting, and I almost think that that calls for a more definition of like what creativity looks like in human beings, but that’s another topic entirely. Another topic that ended up on the cutting room floor was actually just more closely examining the emergence of more collaborative software and like what that looks like and actually basically examining the social conditions and the psychological needs that we have when we’re in collaborative environments because from my vantage point and I feel like the collective experience of most people, it’s kind of just been a free for all, and we’re still figuring it out, and there’s a lot of potential, obviously, and we’re already benefiting from it, but it’s interesting to think about kind of returning to Some of the work that’s already been done in like academia that looks at what people need to feel comfortable collaborating and almost like in the context of arts education and creativity research, there’s a lot that can be pulled from that that is obviously much more squishy. But it also has a lot of applicability to thinking about plopping people into creative environments and expecting them just to immediately generate ideas. I think that that is a common theme in a lot of tools today, especially collaborative tools, and I’m very curious how we can try to kind of break down what we know about human beings to address like their common concerns and possible drawbacks from the current experience. 00:55:24 - Speaker 2: Yeah, obviously when people talk about collaborative software, it’s very easy for that to quickly get into design or even very technical things of, you know, is it using operational transform or CRDTs, and there’s very hard technical challenges that we’re still working on, but the social side of it, the social norms as you’ve mentioned, and also people adjusting their own attitudes about what’s expected of them or what they can feel comfortable doing. You need to be comfortable to be creative, and we’re still figuring out how to do that well. I’m reminded of Tuckman’s stages of group development, which is sort of a psychologist looking at just how teams work together, but really just any group of people, and that there’s kind of this process, these five stages that that they define, which is this forming, storming, norming, performing, and mourning, just sort of how the team comes together, but what I thought was really interesting is once I’ve read this, I can spot this. Not with just any kind of team loosely defined. As a company or a subset of a company, but really any combination of people doing anything, even friends planning an event together or something like that, and there’s typically these early stages where everyone’s super polite and they don’t wanna step on any toes, but actually that stops you from really getting into it and really the true creativity happening, and then there has to be some level of conflict and discovering of roles in the group through, yeah, friction and problems and Even fights or whatever, and then social norms emerge from that, and then that’s when you really go into sort of the magic time, they’re performing stage because it’s sort of all figured out how to do things, and then you can be truly creative. 00:57:05 - Speaker 3: This also reminds me of the satir change model, which is a similar idea, maybe just generalize a little bit, where when things change, they don’t get uniformly better. It’s not all up to the right. You have some foreign element that comes in and instigates the change, and then you go through a period of chaos where your performance is worse, people are scared and they’re reluctant, and then eventually you got to find some transforming idea to bring you into the period of better performance. The way this connects back to this collaborative software discussion is. I think when we first introduced from a technological perspective, the ability to have real-time collaboration, that was a sort of foreign element where you have some of the things that you would expect with collaboration, like you can see what other typing, but you don’t have, for example, body language on facial expressions, you don’t have vocal intonation. And and so it feels like weird, like, basically you’re in the chaos of this Google Doc feels weird or something. But then we have things like, you know, emojis and so and so is typing and things like that and avatars that float around to show you where people are in the document. And so we’re building up the set of practices that will eventually allow people to have higher performances teams. 00:58:10 - Speaker 2: Even an initial negative reaction to why would you even want that. I remember when Google Docs came along, I actually used it when it was right before they were acquired, and that collaborative element, I said, wow, this is great that I can send a document to someone they don’t need to have Microsoft Word installed. We always know there’s the wrong latest version, and I tried to pitch people that I was working with on using it or saying, look, let’s use this tool because it seems so obvious to me this is A good way to do things and very often reaction was like, oh, like I don’t want people to be able to like see me typing or I don’t want other people to be able to edit my stuff, you know, I think maybe Figma relative to sketch actually had some of the same pushback as well. I don’t want people messing with. My designs that kind of a thing, and I think that’s quite natural, which is when you have existed in one paradigm in one set of capabilities, you take for granted that those capabilities or restrictions that that box is exactly the shape box that you want something new coming along offering new capabilities, you might even see those as anti-features. 00:59:16 - Speaker 1: 100%, yeah, I think that there’s there’s so much just push back and discovery we still need to do about people’s expectations in collaborative environments, and I think that there’s a delicate balance to be had to where

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: And I feel like this idea of really changes the abstractions that operating systems should provide because maybe OSs should not just be providing this model of files as a sequence of bytes, but this higher level CRDT like model and how does that impact the entire way how software is developed. 00:00:23 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad. This podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. I’m Adam Wiggins here with my colleague Mark McGranaghan. Hey, Adam, and joined today by Martin Klutman from the University of Cambridge. Hello. And we talked before about Mark’s dabbling in playing the piano. I understand this is a hobby you’re starting to look into as well, Martin. 00:00:49 - Speaker 1: Oh yes, I’ve been playing the piano, like trying to do it a bit more consistently for the last year and a half or so. My lockdown projects. 00:00:57 - Speaker 2: And do you have a technique for not annoying your neighbors, or is this an electronic piano, or how do you do that? 00:01:03 - Speaker 1: It’s an electric piano, although I don’t think it’s too bad for the neighbors. Lately I’ve been trying to learn a WBC 400 piece that I can play together with my wife, so she’ll play two hands and I’ll play the other two. 00:01:15 - Speaker 2: Nice. I suspect a lot of our listeners know you already, Martin. I think you’re within your small, narrow niche, you’re a pretty high profile guy, but for those that don’t, it’d be great to hear a little bit about your background. What brought you on the journey to the topic we’re gonna talk about today? 00:01:31 - Speaker 1: Yeah, well, I’m a computer scientist, I guess. I started out as an entrepreneur and started two startups some years ago. I ended up at LinkedIn through the acquisition of the 2nd startup. And they worked on large scale stream processing with Apache Kafka and was part of that sort of stream processing world for a while. And then I wanted to share what I had learned about building large scale distributed data systems. And so I then took some time out to write a book which is called Designing Data Intensive Applications, which has turned out to be surprisingly popular. 00:02:10 - Speaker 2: Yeah, you wrote a nice kind of tell-all, you showed the numbers on it, which it’s been financially successful for you, but also one of the more popular O’Reilly books just by kind of copy sold in recent times. I like that post, like the candor there, but yeah, it makes you a pretty successful author, right? 00:02:28 - Speaker 1: Yeah, it’s sold over 100,000 copies, which is, wow, way more than what I was expecting for something that it’s a pretty technical, pretty niche book, really. But the goal of the book really is to help people figure out what sort of storage technologies and data processing technologies are appropriate for their particular use case. So it’s a lot about the trade-offs and the pros and cons of different types of systems. And there’s not a whole lot on that sort of thing out there, you know, there’s a lot of sort of vendor talk hyping the capabilities of their particular database or whatever it might be, but not so much on this comparison between different approaches. So that’s what my book tries to provide. Yeah, and then after writing that book, I sort of slipped into academia, sort of half by accident, half by design. So I then found a job at the University of Cambridge where I could do research full time. And since then I’ve been working on what we have come to call the first software, which we’re going to talk about today. The nice thing there is that now then academia compared to the startup world, I have the freedom to work on really long term ideas, big ideas which might take 5 or 10 years until they turn into like viable technologies that might be used in everyday software development. But if they do work, they’ll be really impactful and really important and so I’m enjoying that freedom to work on really long term things now as an academic. 00:03:53 - Speaker 2: And certainly it struck me when we got the chance to work together through these Ink & Switch projects that because you have both the commercial world, including startup founder, but obviously you’re very immersed in the academic kind of machinery now and again just that long-term mindset and thinking about creating public goods and all that sort of thing. And I found that I actually really like now working with people that have both of those. Another great example there would be another former podcast guest Jeffrey Litt. He was also in the startup world, now he’s doing academic work at MIT. 00:04:26 - Speaker 1: Yes, and I’m actually doing a project with him right now, right, I forgot about that. 00:04:29 - Speaker 2: There’s a current Ink & Switch project there. So I find that maybe if you live your whole life in one of those two kind of commercial slash industry or academia, you get like a fish doesn’t know what water is kind of thing, but if you have experienced both models, then it’s easier to know the pros and cons and understand the shape of the venue you’re doing your work in in the end. The point is to have some meaningful impact on humanity through your work, whatever small piece of the world you hope you’re making better. In our case, it’s computer things, but that the venue you’re in is not the point, that’s just a vehicle for getting to where you want to go, and each of these styles of venue have different trade-offs, and being aware of those maybe makes it easier to have your work have an impact. 00:05:19 - Speaker 1: Yes, I think it is really helpful to have seen both sides and I find it allows me to be a little bit more detached from the common mindset that you get like in every domain you get, you know, there are certain things that everyone believes, but you know, they’re kind of unspoken, maybe not really written down either. And so like in academia, that’s like the publishing culture and the competitiveness of publication venues and that sort of stuff, which seems ridiculous to outsiders. But if you’re in it, you kind of get accustomed to it. And likewise in startups, it’s like the hype to be constantly selling and marketing and promoting what you’re doing to the max crushing it, always crushing it, exactly, and to an outsider that seems really. it’s kind of a ridiculous show that people put on frankly. But to an insider, you know, you just get used to it and that’s just your everyday life. I find that having seen both makes me a bit more detached from both of them and I don’t know, maybe I see a little bit more through the bullshit. 00:06:21 - Speaker 2: So as you hinted, our topic today is local first software. So this is an essay that I’ll link to in the show notes. It’s about 2 years old, and notably there’s 4 authors on this paper, 3 of them are here, kind of almost a little reunion, and actually the 4th author, Peter van Hardenberg, we hope to have on as a future guest. But I thought it would be really fun to not only kind of summarize what that philosophy is, particularly because we’re actively pursuing that for the Muse sinking persistence model, but also to look at sort of what we’ve learned since we published that essay and revisiting a little bit. What do we wish we’d put in, how’s the movement, if that’s the right word for it, how’s that evolved, what have we learned in that time? But I guess before getting into all that, maybe Martin, you can give us the elevator pitch, if I’m to reference the startup terminology, the brief summary of what is local first software. 00:07:18 - Speaker 1: Yeah, local first software is a reaction to cloud software and so with cloud software, I mean things like Google Docs, where you have a browser window and you type into it and you can share it really easily. You can have several people contributing to a document really easily, you can send it for comments. Very easily and so on. So, it has made collaboration a ton easier, but it’s come at a great cost to our control and our ownership of the data, because whenever you’re using some cloud software, the data is stored on the cloud provider servers, like Google servers, for example. And you know, as users, we are given access to that data temporarily. Until that day where Google suddenly decides to lock your account and you are locked out of all of the documents that you ever created with Google Docs, or until the startup software as a service product you’re using, suddenly goes bust and decides to shut down their product with 2 weeks' notice and maybe allows you to download a zip file full of JSON files as your data export. And I find that tragic because as creative people, we put a ton of effort, time and our souls and really our personalities into the things that we create. And so much now the things that we create are computer-based things, you know, whether you’re writing the script for a play or whether you’re negotiating a contract or whether you’re doing any sort of endeavor, it’s probably a file on a computer somewhere. And if that file is in some cloud software, then there’s always this risk that it might disappear and that you might lose access to it. And so what we try to do with local first software is to articulate a vision for the future where that does not happen, where We have the same convenience that we have with cloud software that is we have the same ability to do real-time collaboration. It’s not back to the old world of sending files back and forth by email. We still want the same real-time collaboration that we get with Google Docs, but at the same time we also want the files stored. On our own computers. Because if there are files on our own computers, then nobody can take them away. They are there, we can back them up ourselves. We can optionally back them up to a cloud service if we want to. There’s nothing wrong with using a cloud service as long as the software still continues working without the cloud service. Moreover, we want the software to continue working offline so that if you’re working on a plane or working on a train that’s going through a tunnel or whatever, the software should just continue to work. And we want better security and privacy because we don’t want cloud services scanning through the content of all of our files. I think for creativity, it’s important to have that sense of privacy and ownership over your workspace. And so those are some of the ideas that we try to encapsulate in this idea of local first software. So how can we try to have the best of both worlds of the convenience of cloud software. But the data ownership of having the files locally on your own device. 00:10:15 - Speaker 2: Yeah, for me, the core of it is really agency and much of the value of cloud, and I think there’s a version of this also for mobile apps, let’s say and app stores and that sort of thing, which is not what we’re addressing in the paper, but maybe there’s a theme in computing that we’ve made computers vastly more accessible by In many cases, taking agency from people, and that’s actually a good thing in many cases, right? You don’t need to defrag your hard drive anymore. You lose your device, your email, and your photos and all those things are still in this cloud that’s managed by experiencedmins and product managers and so forth at companies like Google and So forth, and they can often do a better job of it in a lot of cases than an individual can. I mean, I think of managing my own email servers, SMTP servers, years back and needing to deal with data backup and spam filtering and all that kind of thing, and Gmail came along and I was just super happy to outsource the problem to them. Absolutely. They did a better job managing it. So I think that’s basically in many ways a good trend or as a net good in the world, and I don’t think we feel like we necessarily want to go back to everyone needs to do more of those data management tasks, but I think for the area of creative tools or more, I guess you call them power users, but it’s like you said, if you’re writing a play, that’s just a very different kind of interaction with a computer than the average person doing some calendar and email and messaging. Yeah, maybe they want different trade-offs. It’s worth doing a little bit more management and taking a little more ownership to get that greater agency over something like, yeah, my work product, the script of my play or my master thesis or whatever it is that I’m working on is something that really belongs to me and I want to put a little extra effort to have that ownership. 00:12:08 - Speaker 1: Right, exactly. And I feel like it’s not reasonable to expect everyone to be a sys admin and to set up their own services, you know, you get this self-hosted cloud software, but most of it is far too technical for the vast majority of users, and that’s not where we want to go with this. I think you still want exactly the same kind of convenience of clouds software that, you know, it just works out of the box and you don’t have to worry about the technicalities of how it’s set up. But one part of local first software is that because all of the interesting app specific work happens client side on your own device, it now means that the cloud services that you do use for syncing your data, for backing up your data, and so the cloud services become generic. And so you could imagine Dropbox or Google Drive or AWS or some other big cloud provider just giving you a syncing service for local first apps. And the way we’re thinking about this, you could have one generic service that could be used as the syncing infrastructure for many different pieces of software. So regardless of whether the software is a text editor or a spreadsheet or a CAD application for designing industrial products or music software or whatever it might be, all of those different apps. Could potentially use the same backup and syncing infrastructure in the cloud, and you can have multiple cloud providers that are compatible with each other and you could just switch from one to the other. So at that point, then it just becomes like, OK, who do you pay 6 cents a month to in order for them to store your data, it becomes just a very generic and fungible service. And so that’s what I see makes actually the cloud almost more powerful. Because it removes the lock-in that you have from, you have to use a single, the, the cloud service provided by the software offer. Instead, you could switch from one cloud provider to another very easily and you still retain the property that you’re using all of the cloud providers' expertise in providing a highly available service and you don’t have to do any admin yourself. It’s not like running your own SMTP server. So I feel like this is a really promising direction that local first software enables. 00:14:24 - Speaker 3: Yeah, for sure, indeed, and you could even describe local first software, I think, as sort of generalizing and distributing the capabilities of the different nodes. So in the classic cloud model, you have these thin clients, they can dial into the server and render whatever the server tells them. And then you have the servers and they can store data and process it and return to clients and when you have both of those at the same time, you know, it works great, but then if you’re a client like you said, who’s in a tunnel, well too bad you can’t do anything, and the local first model is more that any node in that system can do anything, it can. Process the data, I can validate it, it can store it, it can communicate it, it can sync it, and then you can choose what kind of typologies you want. So it might be that you just want to work alone in your tunnel, or it might be that you want to subscribe to a cloud backup service that does the synchronization storage part for you while you still maintain the ability to process and render data locally. This actually gets to how I first got into what we’re now calling local first software. I was in a coffee shop with Peter Ben Hartenberg, who’s one of the other authors that Adam mentioned. And we’re talking about working together at the lab when he was a principal there, he’s now the director, and he showed me the pixel pusher prototype. So Pixel Pusher was this Pixel art app where you color individual pictures to make a kind of retrographic thing, and it was real time collaborative, but the huge thing was that there was no server, so that you had this one code base and this one app, and you got real time collaboration. Across devices, and that was the moment that I realized, you know, I was a fish in the cloud infrastructure water and I didn’t realize it. Just assumed, oh, you need servers and AWS need a whole ops team, you’re gonna be running that for the rest of your life, it’s the whole thing. Well, actually, no, you could just write the app and point at the other laptop and there you go. And we eventually kind of realized all these other benefits that we would eventually articulate as the desiderao property to the local for software article, but that was the thing that really actually kicked it off for me. 00:16:19 - Speaker 1: Yeah, and that’s aspect that the apps become really self-contained and that you just don’t have a server anymore, or if you have a server, it’s like a really simple and generic thing. You don’t write a specific server just for your app anymore. That’s something that I’m not sure we really explored very well in the local first asset as it was published, but I’ve been continuing to think about that since, you know, this has really profound implications for the economics of software development. Because right now, as you said, like if you’re a startup and you want to provide some SAS product, you need your own ops team that is available 24/7 with one pager duty so that when the database starts running slow or a node falls over and you need to reboot something or whatever, you know, there’s just all this crap that you have to deal with, which makes it really. to provide cloud software because you need all of these people on call and you need all of these people to write these scalable cloud services and it’s really complicated, as evidenced by my book, a lot of which is basically like, oh crap, how do I build a scalable cloud service. And with local first software, potentially that problem simply goes away because you’ve just got each local client which just rights to storage on its own local hard disk. You know, there are no distributed systems problems to deal with, no network time outs and so on. You just write some data locally and then you have this syncing code which you just use an open source library like automerge, which will do the data syncing between your device and maybe a cloud service and maybe the other services. And the server side is just non-existent. And you’ve just like removed the entire backend team from the cost of developing a product and you don’t have the ops team problem anymore because you’re using some generic service provided by some other cloud provider. And you know, that has the potential to make the development of collaborative software so much cheaper. Which then in turn will mean that we get more software developed by smaller teams, faster, it’ll improve the competitiveness of software development in general, like it seems to have so many positive effects once you start thinking it through. 00:18:22 - Speaker 2: Yeah, absolutely. For me, yeah, maybe similar to both of you, my motivations were, well, both as a user and as a, let’s say a software creator or provider on the user side, we have these 7 different points we articulate and I think you can, in fact, we even set it. Up this way is you can give yourself a little scorecard and see which of the boxes you tick. It’ll be fun to do that for the muse syncing service when that’s up and running, but the offline capability is a huge one to me, and it’s not just the convenience. I mean, yeah, it’s. Every time I’m working on the train and my train goes through a tunnel, and suddenly I can’t type into my documents anymore, for example, or I don’t know, I like to go more remote places to work and have solitude, but then I can’t load up Figma or whatever else, and Yeah, that for me as a user is just this feeling of it comes back to the loss of agency, but also just practically it’s just annoying and you know we assume always on internet connections, but I wonder how much that is because the software engineers are people sitting in office. Or maybe now at home in San Francisco on fast internet connections with always connected devices versus kind of the more realities of life walking around in this well connected but not perfectly so world we all live in. That’s on the user side. 00:19:42 - Speaker 1: Yeah, I feel like there’s a huge bias there towards like, oh, it’s fine, we can assume everyone always has an internet connection because yes, we happen to be that small segment of the population that does have a reliable internet connection most of the time. There’s so many situations in which you simply can’t assume that and that might be anything from a farmer working on their fields using an app to manage what they’re doing to their crops and something like that and you know, they won’t necessarily have reliable cellular data coverage even in industrialized countries, let alone in other parts of the world where you just can’t assume that sort of level of network infrastructure at all. 00:20:17 - Speaker 3: Yeah. Yeah, it’s funny you mention this because we often run into this on the summits that we have for Muse. So we were recently in rural France and we had pretty slow internet, especially on upload. I think it was a satellite connection, and we always had this experience where there are 4 of us sitting around a table and you’re looking at each other, but you can’t, you know, send files around cause it needs to go to, you know, whatever, Virginia and come all the way back. 00:20:43 - Speaker 1: It’s crazy if you think about it, it’s ridiculous. 00:20:46 - Speaker 2: Yeah, and I don’t think you even need to reach as far as a farmer or a team summit at a remote location. I had a kitchen table in the house I lived in right before this one that was like a perfect place to sit and work with my laptop, but the location of the refrigerator, which it really couldn’t be any other place, just exactly blocked path to my router, and the router couldn’t really be any other place. I guess I could run a wire or something, but I really wanted to sit right there and work. But again, it’s this ridiculous thing where you can’t even put a character into a document and I could pick up the laptop and walk a meter to the left, and now suddenly I can type again and you can be that to something like. G, which does have more of a local is probably one of the closest thing the true local first software where you can work and yes, you need an internet connection to share that work with others, but you’re not stopped from that moment to moment typing things into your computer. 00:21:38 - Speaker 3: Yeah, and furthermore, from the implementation perspective, even when you have a very fast internet connection, you’re still dealing with this problem. So if I’m using an app and I type in a letter on my keyboard, between the time when I do that and when the round trip happens with the AWS server, which might be 50 or 100 milliseconds, the app needs to do something useful. I can’t wait for that full round trip. It needs to immediately show. Me what’s happening. So you inevitably have this little distributed system where you have your local process and app trying to display something immediately, and you have the remote server. And the great elegance, I think, of the local first approach is that that’s just like another instance of the general problem of synchronizing across nodes, whereas often in other apps, that’s sort of like an ad hoc kind of second special case thing, like, oh, It’s only going to be like this for 100 milliseconds. So just kind of do a hacky solution and make it so that most of the time the right letter shows up. And that’s why you have this behavior where apps will have like an offline mode, but it like never works, because I think we mentioned this on the podcast before, there’s systems that you use all the time and systems that don’t work. This is a maximum we can link to. But again, with local first, you’re kind of exercising that core synchronization approach all the time, including when it’s just you and another server on a good connection. 00:22:50 - Speaker 1: Yeah, and from a sort of fundamentals of distributed systems point of view, I find that very satisfying because I just see this as different amounts of network latency. Like if you’re online you have network latency of 50 or 100 milliseconds. If you’re offline, you have network latency of 3 hours or however long it’s going to be until you next come back online again. To me those are exactly the same, you know, I don’t care if it’s a few orders of magnitude apart. Both the network latency both need to be dealt with and if we can use the same techniques for dealing with both standard online latency and being offline, that just simplifies the software dramatically. 00:23:25 - Speaker 2: Going back to sort of the infrastructure, fewer moving parts thing and speaking to our personal motivations, for me, the experience of running Hiroku was a big part of my motivation or fed into my interest in this because Hiokku was an infrastructure business. I didn’t quite grasp what that meant when we went into it. I just wanted a better way to deploy apps, and in the end, I enjoy writing software, I enjoy creating products that solve problems for people, but infrastructure is a whole other game. And you know, it became the point where once you’re, I don’t know if mission critical is the right word, but just something people really care about working well and you’re in the critical path. So for example, our routing infrastructure, if it was down for 3 seconds, people would complain. So the slightest hiccup, and as they should, that was part of the service that that company is providing, and so that’s fair enough. But then when I go, OK, well, I’m building software, when I think of, for example, Muse, where I’m providing this productivity tool to help people think and that sort of thing. I don’t want to be paged because someone went to move a card 5 centimeters to the right and our server was down or overloaded or something, so then they can’t move the card and so then they’re writing into support angrily. I’m pretty comfortable with there’s some kind of cloud syncing problem and OK, I can’t easily like push my changes to someone else, and that is still a problem, but it feels like it’s on this slightly different timeline. You’re not just blocking the very most basic fundamental operation of the software. And so the idea that exactly as you said, it changes the economics, for me personally, I want to spend more of my time writing software and building products and less of my time setting up, maintaining and running infrastructure. So I guess looking back on the two years that have elapsed, I would say that this is probably, it’s hard to know for sure, but the Inkot switch essays, there’s a number of them that I think had a really good impact, but I think this one probably just my anecdotal feeling of seeing people cite it by. it in Twitter comments and things like that. It feels like one of the bigger impact pieces that we published, and I do really see quite a lot of people referencing that term, you know, we’ve sort of injected that term into discussion again, at least among a certain very niche narrow world of things. So, yeah, I’d be curious to hear from both of you, first, whether there’s things that looking back you wish we’d put in or you would add now, and then how that interacts with what you make of local first movement or other work that people are doing on that now. 00:25:59 - Speaker 1: I’m very happy that we gave the thing a name. There’s something we didn’t have initially when we started writing this and we’re just writing this like manifesto for software that works better basically. And then at some point we thought like it would be really good to have some way of referring to it and you know, people talk about offline first or mobile first, and these were all kind of established things and terms that people would throw around. And we also wanted some term X where we could say like I’m building an X type app. And so I’m very glad that we came up with this term they first because I’ve also seen people even outside of our direct community starting to use it. And just, you know, put it in an article casually without even necessarily explaining what it means and just assuming that people know what it is. And I think that’s a great form of impact if we can give people a term to articulate what it is they’re thinking about. 00:26:50 - Speaker 2: Yeah, language, a shared vocabulary to describe something as a very powerful way to, one, just sort of advance our ability to communicate clearly with each other, but also, yeah, there’s so many ideas. I mean, it’s a 20 something page paper and there’s so many ideas, but you wrap this up in this one term and for someone who has downloaded some or most of these ideas, that one term can carry all the weight and then you can build on that. You can take all that as a given and then build forward from there. 00:27:19 - Speaker 1: Yeah. One thing I wish we had done more on is I think trying to get a bit more into the economic implications of it. I guess that would have made the essay another 5 pages longer and so at some point we just have to stop. But I feel like it’s quite an important aspect like what we talked about earlier of not having to worry about back ends or even just like not having to worry generally about the distributed systems problem of like you make a request to a server, the request times out. You have no idea whether the server got the request or not. Like, do you retry it? If so, how do you make the retry? Is that potent so that it’s safe to retry and so on. Like all of those problems just go away if you’re using a general purpose syncing infrastructure that somebody else has written for you. And there are other implications as well that are less clear of what about the business model of software as a service. Because there a lot of companies' business model right now is basically pay us, otherwise you’re going to get locked out of your data. So it’s using this idea of holding data hostage almost as the reason why you should pay for a product. And you know, it’s like that with Slack. Like you put all of your messages in Slack those messages were written by you and your colleagues. There’s nothing really Slack did to own those, they just facilitated the exchange of those messages between you and your colleagues. But then once you go over the, whatever it is, 10,000 messages limit, then suddenly you have to pay slack to see the messages that you wrote yourself. And generally that’s the business model with a lot of software as a service. And with local first, it’s not clear that that business model will still work so clearly. But of course, software developers still have to be paid for their time somehow. So how do we find a way of building sustainable software businesses for collaboration software but without holding data hostage? I think that’s a really deep and interesting question. 00:29:07 - Speaker 3: Yeah, I think as an aside that uh you might call it the political economy of software is understudied and underconsidered, and I would put in here like the economics of software business, but also the interaction with things like regulation and governments and the huge amount of path dependence that’s involved. I think that’s just a huge deal. I think we’re starting to realize it, but yeah, there’s a ton of stuff we could do and think about just for local first. Like just one kind of branch that I hope we get to explore is we mentioned how local first enables you to do totally different topologies. So with cloud software almost by definition, you have this hub and spoke model where everything goes through. The central server and the central corporation. Well with local first, you can very easily, for example, have a federated system where you dial into one of many synchronization nodes and you could even have more like a mesh system where you request and incentivize packets to be forwarded through a mesh to their destination, sort of like TCPIP networking, but for like the application layer, and it may be, you know, it’s still kind of TBD but it may be that a mesh or a distributed approach has totally different political implications from a centralized node and that might become important, so. I just think there’s a lot to think about and do here. 00:30:15 - Speaker 1: Yeah, I think so too. And like, I would take email as an analogy maybe, which is a federated system just like what you described, like you send your email to your local SMTP server and it forwards it to the recipient’s SMTP server and the system works really well. Certainly as criticisms like spam filtering is difficult in a really decentralized way. Maybe spam is not a problem that local first software will have as much because it’s intended more like for collaboration between people who know each other rather than as a way of contacting people you don’t know yet. But certainly like I think taking some inspiration from that federation and seeing how that can be applied to other domains, I think would be very interesting. 00:30:55 - Speaker 3: Yeah, and this brings us to the topic of like industrialization and commercialization, and I feel like there’s more promise than ever around local First and more people are excited about it, but I still feel like we’re just in the beginning phases of getting the ball rolling on industrial and commercial applications. And if I’m being really honest, I feel like it might have been slower than I had initially hoped over the past few years, so I’m just curious if Adam and Martin, you would reflect on that. 00:31:21 - Speaker 2: It’s always hard to say, right? The thing with any technology, but certainly my career in computing, this has always proven to be the case, is that something seems right around the corner and it stays so for 20 years. I don’t know, maybe VRs in that category, but then there’ll be a moment, suddenly it’ll just be everywhere, broadband internet or something like that. So as a people who are both trying to advance the state of the art but also making Business decisions, you know, should I start a company? Should I invest in a company? Should I keep working on the company I’m working on based on what technologies exist or where you see things going? Yeah, you’re always trying to make accurate predictions. So yeah, I agree on one hand it felt very close to me on the basis of the prototypes we’d built, the automerged library, the reference, Martin I’ll link that in the notes here, but basically that’s a JavaScript implementation of something called CRDTs which Just I guess as a sidebar, it could be easy to think that CRDTs and local first software are kind of one and the same because they are often mentioned together and in fact our paper talks about them together, but CRDTs are a technology we find incredibly promising for helping to deliver local first software, but local first is a set of principles. It doesn’t require any particular technological solution. Yeah, based on the strength of those prototypes, many of which worked really well, there’s the networking side of it and whether you can have that be fully kind of decentralized versus needing more of a central coordination server. But once you get past that hump, it does work really, really well. But I think that comes back to the point you both made there about the economic model side of things, which is we have a whole software industry that’s built around people will pay for software when there’s a service connected to it. Right, so Sass, in particular B2BASS is just a fantastic business to be in, and as a result, we’ve seen a huge explosion of software around that, but connected to that is, for example, the Fremium model, exactly like what you mentioned with Slack, the docs is one of those. Notion is one of those. They do this kind of free for individuals, but then you pay when you’re a business and then you need to come up with the feature stuff, the kinds of features that seem to be selecting for you being a business with more serious needs and something like retaining your message history is there. I wrote a whole other shorter essay about paying for software. I’ll link that in the notes, but I think we got into a weird corner. The industry got itself into a weird painted itself into a corner because Things like Google giving you so much incredibly high quality software, Gmail, Google Docs, Google Maps, etc. for quote unquote free, but then how you’re really paying for it is your attention and your data, right? And that being kind of montizable through being able to essentially serve you ads. And I think that’s fine and I’m very glad for Google’s existence and they found that model, but it almost feels like then it taught people that good software should be free and that you shouldn’t pay for, maybe that’s a little bit connected to the concept that software R&D basically costs nothing to make additional copies of it. So therefore, if you make this big upfront investment and then the software exists and you can duplicate it endlessly, but I think there’s a lot of things flawed about all of that. But the end place that gets you to is, OK, if someone has my data and I’m paying them to maintain it and run the servers that it’s on, I can stomach that. OK, now I’ll pay $5 a month, $10 a month for my Dropbox account or something like that. But other than that, we’ve become accustomed to, oh, if it’s an app on the App Store, the App Store is a good example of these kind of consumer economics, we just expect it to be vastly lower cost or free and good software costs money to make, and as we kind of talked about earlier, I would rather be building the software, not maintaining the infrastructure, but when you set it up so that The only way you can make money is to build software that has infrastructure, you’re actually incentivized, build that back end as soon as you can and get the user’s data in there and not necessarily hold it hostage, but just take ownership of it, because that’s what people will pay for. They won’t pay for software where they own the data themselves. 00:35:34 - Speaker 1: Yes, one thing that a friend has suggested is that when talking about the business model of local first software, we should just call it SA, like label it as SAS, market it in exactly the same way as SAS. Don’t even tell people that it’s local first software and just use the fact that it’s a lot cheaper and easier to implement local first software and use that for your own benefit in order to build the software more cheaply. But don’t actually market the local first aspect. And I thought that’s quite an interesting idea because, you know, it is an idea that people are accustomed to and to be honest, I think the amount of piracy that you would get from people like ripping out the sinking infrastructure and putting it for something else and then continuing to use the app without paying for it, it’s probably pretty limited. So you probably only need to put in a very modest hurdle there of say, OK, like this is the point at which you pay. Regardless of whether that point for payment is, you know, necessarily enforced in the infrastructure, it might just be an if statement in your client side up and maybe that’s fine. 00:36:38 - Speaker 2: Muse is basically an example of that. We have this membership model that is, you know, subscription is your only option and there are a lot of folks that complain about that or take issue with it, and I think there are many valid complaints you can make, but I think in many cases it is just a matter of what. Folks are accustomed to and we want to be building and delivering great software that improves and changes over time and maps to the changing world around it, and that’s something where as long as you’re getting value, you pay for it and you’re not getting value anymore, you don’t have to pay anymore. And then a model like that basically works best for everyone, we think, again, not everyone agrees. But then again, you do get this pushback of we are running a small service, but it’s not super critical to the application, but maybe that would be a good moment to speak briefly about the explorations we’re doing on the local first sync side, Mark. 00:37:32 - Speaker 3: Yeah, so right now Muse is basically a local only app, like it’s a traditional desktop app where files are just saved to the local device and that’s about it, and you can manually move bundles across devices, but otherwise it just runs locally and the idea is to extend be used with first syncing across your devices and then eventually collaboration across users using a local first approach. Now we don’t plan to do, at least initially the kind of fully distributed mesh networking peer to peer thing. It will be a sync service provided by Muse and kind of baked in to the app, but it will have all those nice local first properties of it works offline, it’s very fast, all the different nodes are first class and so forth while eventually supporting syncing and and collaboration. So, yeah, we’re going through this journey of, we had a lot of experience with um basic prototypes in a lab, but there’s a big jump to have a commercialized and industrialized product, not just in terms of charging for the business model and stuff, but in terms of the performance and it’s like all the weird things that you deal with in the real world, like versioning and schemas and the idiosyncrasies of networking and All the things that go around the core functionality, like one thing we’re thinking a lot about is visibility into the sync status and how it’s different in a local first world. Yeah, so I’m excited that we are now investing a lot in bringing local first into the real world with MS. 00:38:57 - Speaker 1: Yeah, and I feel like more generally, if we want the local first ideas to be adopted, we need to make it easy in a way that people can just take an open source library off the shelf, not have to think too much about it, plug it into their app, have a server that’s ready to go, that’s either it’s already hosted for them or they can spin up their own server and make that path. Super easy and straightforward and that’s kind of where my research is focusing of trying to get the technologies to that point. So right now, we have some basic implementations of this stuff. So automerge is a library that does this kind of data synchronization. It includes a JSON like data model that you can use to store the state of your application. It has a sort of basic network protocol that can be used to sync up two nodes. But there’s so much more work to be done on making the performance really good, like at the moment it’s definitely not very good. We’re making progress with that, but it’s still a long way to go. Making the data sync protocol efficient over all sorts of different types of network link in different scenarios, making it work well if you have large numbers of files, for example, not just single file and so on. And so there’s a ton of work still to be done there on the technical side. I think before this is really in a state where people can just pick up the open source library and run with it. Part of it is also like just uh getting the APIs right, making sure it has support across all the platforms, just having a JavaScript implementation is fine for initial prototypes, but Obviously, iOS apps are written in SWIFT and Android apps will be written in coin or whatever people use and so you need to have support across all of the commonly used platforms and we’re gradually getting there, but it’s a ton of work. 00:40:43 - Speaker 2: And conceptually seeing how autoerge is evolving. And how people are trying to use it sometimes very successfully, sometimes less so, but I see this as a case of technology transfer, which is an area I’m incredibly interested in because I think it’s kind of a big unsolved problem in HCI research, computer science, honestly, maybe all research, but I’ll stick to my lane in terms of what I know, which is there is often this very excellent cutting edge research that does sit in the labs. So to speak, and never graduates or it’s very hard or there isn’t a good path often for it to jump over that hump into what’s needed in the production world and of course in the research world you’re trying to do something new and different and push the boundaries of what was possible before and in the production commercial side you want to choose boring technologies and do things that are really reliable and known and stable, and those two, there’s often a bridge that’s hard to divide there. Sitting in your seat as again someone who’s enmeshed in the academic world right now and you’re creating this library, you know, started as a called a proof of concept for lack of a better term, and then you have customers if that’s the right way to put it, but as an academic you would Shouldn’t have customers, but you sort of do because people want to use this library and in fact are for their startups and things like that. How do you see that transition happening or is there a good role model you’ve seen elsewhere or just kind of figure it out as you go? 00:42:16 - Speaker 1: Well, I think where we’re trying to go with this is it’s great for automerge to have users. I don’t think of them as customers. I don’t care about getting any money from them. But I do care about getting bug reports from them and experienced reports of how they’re getting on with the APIs and reports of performance problems and so on. And those things are all tremendously valuable because they actually feed back into the research process and so I’m essentially using the open source users and the contributors as a source of research problems. So with my research hat on, this is great because I have essentially here right in front of me a goldmine of interesting research problems to work on. I just take like the top issue that people are complaining about on GitHub, have a think about how we might solve that. And often there’s enough of a nugget of research problem in there that when we solve the problem, we can write a paper about it. It can be an academic contribution as well as moving the open source ecosystem gradually towards a point where we’ve ironed out all of those major technical problems and hopefully made something that is more usable in production. So, I actually feel those worlds are pretty compatible at the moment. There are some things which are a bit harder to make compatible like sort of the basic work of porting stuff to new languages on new platforms, that’s necessary for real life software engineering, but there’s no interesting research to be done there, to be honest. But so far I’ve found that quite a lot of the problems that we have run into actually do have interesting research that needs to be done in order to solve them. And as, as such, I think they’re quite well compatible at the moment. 00:43:56 - Speaker 2: And like imagining or the mental picture of someone submits a bug report and one year later you come back and say, here’s the fix and also the paper we published about it. 00:44:08 - Speaker 1: I’ve literally had cases where somebody turns up on Slack and says, I found this problem here. What about it? And I said, oh yeah, I wrote a paper about it. And the paper has a potential algorithm for fixing it, but I haven’t implemented it yet, sorry. And they go like, WTF what you put all of this thought into it. You’ve written the paper. And you haven’t implemented. And I go, well, actually sorry for me, that’s easier because if I want to implement it, I have to put in all of the thoughts and convince myself that it’s correct. And then I also have to implement it and then I also have to write all the tests for it and then I have to make sure that it doesn’t break other features and it doesn’t break the APIs and need to come up with good APIs for it and so on. So for me actually like implementing it is a lot more work than just doing the research in a sense, but actually, Doing the research and implementing it can be a really useful part of making sure that we’ve understood it properly from a research point of view. So that at the end what we write in the paper is ends up being correct. In this particular case, actually, it turned out that the algorithm I had written down in the paper was wrong because I just haven’t thought about it deeply enough and a student in India emailed me to say, hey, there’s a bug in your algorithm, and I said, yeah, you’re right, there’s a bug in our algorithm. We better fix it. And so probably through implementing that maybe I would have found the bug, maybe not, but I think this just shows that it is hard getting this stuff right, but the engagement with the open source community, I found a very valuable way of Both working towards a good product but also doing interesting research. 00:45:39 - Speaker 3: I think it’s also useful to think of this in terms of the research and development frame. So research is coming up with the core insights, the basic ideas, those universal truths to unlock new potential in the world, and it’s my opinion that with local first, there’s a huge amount of development that is needed, and that’s a lot of what we’re doing with Muse. So analogy I might use is Like a car and an internal combustion engine. If you came up with the idea of an internal combustion engine, that’s amazing. It’s pretty obvious that that should be world changing. You can spin this shaft at 5000 RPM with 300 horsepower, you know, it’s amazing, but you’re really not there yet. Like you need to invent suspension and transmission and cooling and It’s kind of not obvious how much work that’s gonna be until you go to actually build the car and run at 100 miles an hour. So I think there’s a lot of work that is still yet to be done on that front. And eventually, that kind of does boil down or emit research ideas and bug reports and things like that, but there’s also kind of its own whole thing, and there’s a lot to do there. There’s also a continuous analogy. I think once the research and the initial most obvious development examples get far enough along, you should have some. Unanticipated applications of the original technology. So this should be someone saying like, what if we made an airplane with an internal combustion engine, right? I don’t think we’ve quite seen that with local first, but I think we will once it’s more accessible, cause right now to use local first, you gotta be basically a world expert on local first stuff to even have a shot. But once it’s packaged enough and people see enough examples in real life, they should be able to more easily come up with their own new wild stuff. 00:47:11 - Speaker 1: Yeah, we have seen some interesting examples of people using our software in unexpected ways. One that I quite like is the Washington Post, as in the newspaper, everyone knows. They have an internal system for allowing several editors to update the layout of the home page. So the placement of which article goes where, with which headline, with which image, in which font size, in which column. All of that is set manually, of course, by editors. And they adopted automerge as a way of building the collaboration tool that helps them manage this homepage. Now, this is not really a case that needs local first, particularly because it’s not like one editor is going to spend a huge amount of time editing offline and then sharing their edits to the homepage. But what I did want is a process whereby multiple editors can each be responsible for a section of the homepage and they can propose changes to their section. And then hand those changes over to somebody else who’s going to review them and maybe approve them or maybe decline them. And so what they need essentially is this process of version control, Git style version control almost, but for the structure of representing the homepage. And they want the ability for several people to update that independently. And that’s not because people are working offline, but because people are using essentially branches using the Git metaphor. So different editors will be working on their own local branch until they’ve got it right and then they’ll hit a button where they say, OK, send this to another editor for approval. And that I found really interesting. It’s sort of using the same basic technologies that we’ve developed with CRTTs, tracking the changes to these data structures, being able to automatically merge changes made by different users, but applying it in sort of this interesting unexpected context. And I hope like as these tools mature, we will expand the set of applications for which they can be sensibly used and In that expansion, we will then also see more interesting unexpected applications where people start doing things that we haven’t anticipated. 00:49:19 - Speaker 2: Maybe this reflects my commercial world bias or maybe I’m just a simple man, but I like to see something working more than I like to read a proof that it works. And both are extremely important, right? So the engineering approach to seeing if something works is you write a script, you know, fuzz testing, right, you try a million different permutations and if it all seemed to work, kind of the Monte Carlo simulation test of something, and it seems to work in all the cases you can find, so it seems like it’s working. And then there’s, I think, the more proof style in the sense of mathematical proof of here is an airtight logical deductive reasoning case or mathematical case that shows that it works in all scenarios that it’s not a Monte Carlo calculation of the area under the curve, it’s calculus to determine precisely to infinite resolution area to the curve. And I think they both have their place kind of to Mark’s point, you need to both kind of conceptually come up with the combustion engine and then build one, and then all the things that are gonna go with that. And I think we all have our contributions to make. I think I probably, much as I like the research world at some point when there’s an idea that truly excites me enough, and local first broadly and CRDT. in this category, and I wanna see it, I want to try it, I want to see how it feels. In fact, that was our first project together. Martin was, we did this sort of trello clone essentially that was local first software and could basically merge together of two people worked offline, and they had a little bit of a version history. I did a little demo video, I’ll link that in the show notes. But for me it was really exciting to see that working, and I think maybe your reaction was a bit of a like, well, of course, you know, we have 5 years of research. Look at all these papers that prove that it would work, but I want to see it working and moreover feel what it will be like because I had this hunch that it would feel really great for the end user to have that agency. But seeing it slash experiencing it, for me that drives it home and creates internal motivation far more than the thought experiment is the right word, the conceptual realm work, even though I know that there’s no way we could have built that prototype without all that deep thinking and hard work that went into the science that led up to it. 00:51:42 - Speaker 1: Yeah, and it’s totally amazing to see something like working for the first time and it’s very hard to anticipate how something is going to feel, as you said, like you can sort of rationalize about its pros and cons and things like that, but that’s still not quite the same thing as the actual firsthand experience of really using the software. 00:52:01 - Speaker 2: All right, so local first, the paper and the concept, I think we are pretty happy with the impact that it made, how it’s changed a lot of industry discussion, and furthermore, that while the technology maybe is not as far along as we’d like, it has come a long way, and we’re starting to see it make its way into more real world applications, including news in the very near future. But I guess looking forward to the future for either that kind of general movement or the technology. What do you both hope to see in the coming, say, next 2 years, or even further out? 00:52:38 - Speaker 3: Well, the basic thing that I’d like to see is this development of the core idea and see it successfully applied in commercial and industrial settings. Like I said, I think there’s a lot of work to do there and some people have started, but I’d like to see that really land. And then assuming we’re able to get the basic development landed, a particular direction I’m really excited about is non-centralized topologies. I just think that’s going to become very important and it’s a unique potential of local first software. So things like federated syncing services, mesh topologies, end to end encryption, generalized sync services like we talked about, really excited to see those get developed and explored. 00:53:18 - Speaker 1: Yeah, those are all exciting topics. For me, one thing that I don’t really have a good answer to but which seems very interesting is what does the relationship between apps and the operating system look like in the future? Because like right now, we’re still essentially using the same 1970 Unix abstraction of we have a hierarchical file system. A file is a sequence of bytes. That’s it. A file has a name and the content has no further structure other than being a sequence of bytes. But if you want to allow several users to edit a file at the same time and then merge those things together again, you need more knowledge about the structure of what’s inside the file. You can’t just do that with an opaque sequence of bytes. And ICRDTs is essentially providing a sort of general purpose higher level file format that apps can use to express and represent the data that they want to have just like Jason and XML are general purpose data representations and CRDTs further refined this by not just capturing the current state, but also capturing all the changes that were made to the state and thereby they much better encapsulate the what was the intent of the user when they made a certain change and then capturing those intents of the user through the operations they perform that then allows different users changes to be merged in a sensible way. And I feel like this idea of really changes the abstractions that operating systems should provide because maybe OSs should not just be providing this model of files as a sequence of bytes, but this higher level CRDT like model and how does that impact the entire way how software is developed. I think there’s a potential for just rethinking a lot of the stack that has built up a huge amount of craft over the past decades. And potential to like really simplify and make things more powerful at the same time. 00:55:17 - Speaker 2: Yeah, local first file system to me is kind of the end state, and maybe that’s not quite a file system in the sense of how we think about it today, but a persistence layer that has certainly these concepts baked into it, but I think also just reflects the changing user expectations. People want Google Docs and notion and Figma. And they expect that their email and calendar will seamlessly sync across all their devices and then you have other collaborators in the mix, so your files go from being these pretty static things on the disk, you know, you press command S or ControlS and every once in a while it does a binary dump of your work that you can load later and instead it becomes a continuous stream of changes coming from a lot of different sources. They come from, I’ve got my phone and my computer and Mark’s got his tablet and his phone, and Martin, you’ve got your computer and we’re all contributing to a document and those changes are all streaming together and need to be coalesced and made sense of. And I think that’s the place where, for example, Dropbox, much as I love it, or iCloud, which I think in a lot of ways is a really good direction, but both of those are essentially dead ends, because they just take the classic static binary file and put it on the network, which is good, but it only takes you so far, cause again, people want Google Docs, that’s just the end of it. And that puts every single company that’s going to build an application of this sort. They have to build the kind of infrastructure necessary to do that. And we’ve seen where I think FIMA is the most dramatic example, they just took sketch and ported it to a kind of a real-time collaborative web first environment, and the word just there is carrying a lot of weight. Because in fact, it’s this huge engineering project and they need to raise a bunch of venture capital, but then once you had it, it was so incredibly valuable to have that collaboration, and then, of course, they built far beyond the initial, let’s just do sketch on the web. But any company that wants to do something like that, and increasingly that’s not table stakes from user expectation standpoint, they want to be able to do that, but you have to do that same thing. You’ve gotta drop, you know, tens of millions of dollars on big teams to do that. It seems strange when I think many and most at least productivity applications want something similar to that. So, if that were built into the operating system the same way that a file system is, or, you know, we proposed this idea of like a firebase style thing for local first and CRDTs which could be maybe more developer infrastructure, maybe that’s also what you guys were speaking about. Earlier with kind of like AWBS could run a generic sync service. I don’t know exactly what the interface looks like, it’s more of a developer thing or an end user thing, but basically every single application needs this, and the fact that it is a huge endeavor that costs so much money and requires really top talent to do at all, let alone continue running over time and scale up, just seems like a mismatch with what the world needs and wants right now. 00:58:25 - Speaker 3: Yeah, and now I want to riff on this, Adam, because the strongest version of that vision is

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: And I don’t hold this against Apple or anybody else. There are billions of people on Earth who need great computers, and most of them are not scientists or authors or policymakers, but we believe that this has left a gap where there just simply isn’t a major computing group that’s actually focused on how to use computers in this intelligence amplifying way. 00:00:25 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad and Mac. This podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. I’m Adam Wiggins here today with my colleague Mark McCrannigan. Hey, Adam, and my frequent collaborator, Peter Van Hardenberg from I and Switch. Hi, it’s good to be here. And Peter, one of the things that I find fascinating to talk to you about is your varied hobbies outside of the world of computing. What’s your latest interest? 00:00:56 - Speaker 1: Oh yeah, I mean, when I’m not, you know, brewing beer or getting into some strange creative hobby of one form or another, well, I guess I’m always picking up another one. So lately I’ve been doing a lot of hydroponic gardening, and I’m not growing anything of legal dubiousness, but with the pandemic, first I was focused on trying to basically have some green space in my home, because I was living in San Francisco and we didn’t have a yard. And then more recently, I have moved back to my ancestral homeland of Canada, and it gets real dark and cold here in the winter, and I wanted to make sure I could secure a steady supply of Mexican ingredients, particularly herbs and chili peppers and things that you just can’t find. Sort of north of the wall in the offseason. So that’s been a lot of fun. I’ve been learning a ton about electrical conductivity and how to take pH measurements and having a lot of fun with automating all the components of this system over time. So these are going to be the most expensive cherry tomatoes in the world by the time I’m done, but it’s just a blast learning about all this stuff and, you know, what kind of absorption wavelengths are better for different kinds of plants and so on and so forth. 00:02:09 - Speaker 2: And correct me if I’m wrong, but I think the term hydroponics typically refers to growing plants without soil or with minimal soil. That’s right, I assume here you also have the artificial light element as well with the northernly climate. 00:02:22 - Speaker 1: Yeah, and basically what I have is sort of, if you imagine like a 2 m by 2 m bed or maybe a 1.5 garden bed, that’s sort of cut into 3 pieces and then stacked into a bookshelf. And then nutrient and oxygen-rich water circulates sort of around these U-shaped trays and then cascades back down to a reservoir in the bottom, where it gets pumped back up to the top in a loop. And so I routinely come in and I have to top up the water, there’s a lot of water loss due to transpiration, which is where basically the water goes out to the leaves and then evaporates, and then you have to sort of monitor how much water there is, but also regularly top up the nutrients to make sure that all of the vital macro and micro nutrients that the plants need to live are present in the solution. It’s a fun little chemistry project. 00:03:10 - Speaker 2: And I would be disappointed if there wasn’t, I don’t know, a raspberry pie or something connected in this mix somewhere. 00:03:17 - Speaker 1: Oh yeah, there are 2 computers now involved. There’s one that doses out the nutrients using peristaltic pumps. And there’s a separate one that came with the unit, which handles notifying me when the water levels are too low, there’s a little ultrasonic sensor that can tell when the reservoir tank gets low on water, and also handles scheduling the lights off and on during the day. I’m still hopeful about automating the nutrient sensors, but it turns out that monitoring pH over time is actually a surprisingly difficult problem, and that a good hardware solution actually involves a fair amount of like upkeep and maintenance and expense just in sensor probes alone. So so far I haven’t quite taken the plunge there, but I think it’s just a matter of time. We’ll get there, don’t worry. The end state, of course, is to build a robot arm that will like use computer vision to spot when a cherry tomato is ripe, and then pluck it for me and place it on a conveyor belt. But uh, you know, I think we’re still a few years out from that, both in terms of like technological feasibility, but also just in terms of like, I got a lot of other projects on the go these days. 00:04:23 - Speaker 2: And surely plucking the ripe literal fruits of your labor is the funnest part of the whole thing, so save that for last to automate. 00:04:33 - Speaker 1: Yeah, my kids really enjoy going into the garden. They get the little step ladders out and climb up and peer into the different tiers and pluck leaves to sample. Honestly, this thing is the only way I’ve ever convinced them to eat green leaves, because they’ve learned now that there’s lots of interesting green leaves they’re allowed to eat in the garden. But if you put a salad on their plate, they won’t eat it, but they’ll go and like forage some mint leaves or basil or things like that. They’re into that. 00:04:57 - Speaker 2: And I mentioned we’re frequent collaborators, so the three of us worked together all the way back in the Hiroki days. Nowadays you are leading, administrating, directing the I and Switch Research Lab, which was a position I had previously held before entering the new spin out, which we can talk about the relationship there, but I’m sure the audience would love to hear about your background. How did you end up connected to this unusual band of misfits and unusual space in computing? 00:05:26 - Speaker 1: I always tell people that I like to move like a knight across the chessboard of the industry. I don’t like to go in a straight line and I don’t like to follow an obvious path, so I move a little bit to one side and a little bit across. So before Ink and Switch, I had been working at Hiroku with the two of you. But my background is super varied. I have been an Arctic oceanographer and spent time collecting data at sea. I’ve had waves roll over the side of a flat bottomed research vessel and swamp my boots while I was working on the aft deck. I’ve worked in game development and written physics engines for the Game Boy DS. You know, it’s really hard to ride a physics engine using only integer math. You really sort of take the ability to like divide numbers for granted in day to day computing, and so that was a really cool challenge. Yeah, I’ve done computational Shakespeare, which was cool. There’s the Internet Shakespeare editions are like a scholarly Shakespeare, so if you want to be able to ask questions about, like, can we tell who typeset this page of an original edition by analyzing their use of combined letter ligatures. You know, because they think they can tell different typesetters had different habits for where they would draw type out of the frames. But all those kinds of questions about provenance and genuineness of Shakespeare, we were building technology to enable that kind of computationally. So, yeah, I’ve done a lot of weird and wonderful things before I wound up at I and Switch, and if I’m lucky, I hope to do many more in my career, so, so far so good. 00:06:57 - Speaker 2: And then another important piece here is, what actually is Ik and Switch? What does it do? 00:07:01 - Speaker 1: Right, yeah, I and Switch is an independent industrial research group. We’ll break that down a little bit more, but basically, the way I describe what we do to people who are sort of outside the field is that we think that computers have much more potential to Make us smarter and more capable than we’re really seeing happen with the kind of technology platforms we have today, and we know that that’s true, but we don’t really know how best to pursue it, and we sort of feel like a lot of the fundamental requirements to be able to do that work aren’t in place yet. And so, I can switch research. It is about trying to light the way to make possible these kinds of breakthroughs in the way people build software and the kinds of software people can build by unblocking and discovering paths forward for people. 00:07:57 - Speaker 2: And if one were to flip through the publications on the I can switch website, I’ll link in the show notes, you’d quickly get a feel for, I think a lot of research areas that are things that Mark and I mention and with our guests here all the time on this podcast. So end user programming, for example. Yeah, what a coincidence, increasing the accessibility of programming, obviously kind of tablet touch interfaces or next generation interfaces generally. Local first software is a huge one. We had Mark Klepman on just. Recently to talk about that, and there’s many kind of sub branches of that sort of you think of it as like oh what comes after the cloud or how do we give data agency back to creative people. So those topics, many, many of the things that we talk about here really do come from or have their roots in the research there, and the research is ongoing, so those ideas are continuing to be developed and pushed forward by you and your team. 00:08:48 - Speaker 1: Yeah, there’s an endless amount of work to do, and I think the hard part is really just sort of continuing to find the efficient frontier, right, which is where are the most important unanswered questions, who are the people that can do that work, and how can we convince them to come and spend a bit of time with our ragtag band of adventurers? 00:09:09 - Speaker 2: Could you give us an example of something you’ve published recently on, just to give a taste for the kind of research work that’s undergoing there? 00:09:18 - Speaker 1: I’ll give a little preview of two unreleased papers that are coming up. Maybe that’ll be even more exciting for folk if anyone out there follows us. We have two pieces that are currently in editing and sort of preparation for publication. One of them is called Inkbase and the other one is Paratext. Now, Ibase was led by James Lindenbaum with Shimon Kliski, and it was an exploration of how you could program ink. Now, One way of looking at this is to say, well, what should the programming language for Ik be? But in fact, we sort of deliberately cut that out of scope. What we wanted to know was what would an ink programming environment feel like? What would it be capable of? You know, how could you use it if you had it, even though we didn’t have it, right? As a research project, what we wanted to know was kind of like, well, what would happen if you did? This was also in partnership with Josh Horowitz, who came to us from Dynamic land on his way to grad school. 00:10:22 - Speaker 2: So we built an environment that allowed you to program ink drawings to make them interactive, and ink here you mean the digital ink, the same exact kind that is in use, except there you scribble on your board or whatever, and that ink can be moved, it can be deleted, it can respond to To your interactions and create interactions of its own. 00:10:31 - Speaker 1: And so we saw really cool ideas explored there like Shimon built a music sequencer that produced MIDI out of the system and let him drive sound synthesis engines using drawn lines. James demonstrated how to build like a fitness tracker, you know, sort of like the Seinfeld streak that people often use. What if you could make that a programmable thing or like a to do list? You know, like a Zettel casting or a personal GTD kind of system, but one that you could mold and actually make smarter over time by adding interactions as you develop them the same way that sort of like an Excel spreadsheet grows out of a simple table of numbers into some fully featured and powerful computational environment. We want to know what happens if a drawing evolves over time to become a computational environment that responds to your needs. And your discovered requirements. So that’s the ink-based paper. It was presented at live very briefly this year, but in classic Ink & Switch style, we have a massive paper going into all kinds of detail and with links to some code coming soon. And that kind of represents one angle of our work. Another piece that we’ve been working on, we’ve talked about local first, you mentioned earlier, we’ve been exploring something called CRDTs, that’s a conflict-free replicated data types. A really gratuitously confusing name for what is basically maybe more usefully to think about like Git for data structures, mergeable data maybe mergeable data, yeah, if you have like a JSON file in Git and then you check in changes and then somebody else checks in changes and then you go to merge and you just have like some kind of nightmare of text editing and comma placement and figuring out what really happened. The idea behind automerge is like, well, what if the data structures were smart enough to synchronize themselves and what kind of new capabilities would you get that? And that’s really been sort of foundational research that’s enabled a lot of local first work over the last few years and we’re by no means the only game in town, but one of the problems we keep running into with that is we write a lot of these big essays and we want to have local first tools for it. But today the kind of state of the art is like, OK, well, you can do plain text editing. Or you’ve got nothing, and our essays are these like complex multimedia documents with videos and asides, and we won’t be able to make suggestions and comments all throughout them as we edit. And so, what we needed was a rich text CRDT and that is to say a CRDT that was aware of formatting and spans. And we talked a lot in the Paraex paper, which is coming out soon, I hope, is in concert with Jeffrey Litt, who led that project with Slim. 00:13:12 - Speaker 2: Notion, former guest of the podcast, in fact. Yeah. 00:13:15 - Speaker 1: So we’re looking forward to publishing that because it turns out to be a surprisingly subtle problem. If you’ve ever dealt with non-hierarchical data, like extent data overlapping extents, it just leads to a lot of like really interesting and subtle, quite literal edge cases. So, we’re eager to present that to everybody as soon as we stop finding bugs in the implementation. I think we’re close, we’re close. 00:13:43 - Speaker 2: I think we said before we talked with Lis Lee in a previous podcast about whether software is ever done and certainly research is never done. You have to just kind of draw a line in the sand somewhere and say we’ve advanced the state of the art enough that we think others can build on this. Let’s publish. 00:13:59 - Speaker 1: Oh, I have a great Dune quote for this, it’s right in the zeitgeist now. I’ve been using this line for years, but the way I think about this is in the words of Frank Herbert, Araki teaches the way of the knife, it is finished because we stopped here. Yeah. So by that, I mean, you know, you kind of go into a space and you work for a while, and you can spend your whole career bashing away at one problem, but one of the things I think we do really well and actually that we’ve inherited from your time as lab director is like this ruthless discipline about sort of pencils down on a given date and moving on. If you don’t do that, it’s really easy to get caught up in what you’re doing and to continue to pull that thread, you could be at it for the rest of your career. And maybe that’s good, but by stepping back and sort of re-evaluating, it forces you to kind of Both take the time to publish, but also to sort of take the time to think about what’s next and what’s really important now. 00:14:49 - Speaker 2: I think that maybe this is different for other researchers or labs, but Frankenswitch, I think we want to take kind of a breath first search of the adjacent possible for how we can improve creative computing rather than sort of the depth first, follow one fork in the trail and go as deep as we can. 00:15:07 - Speaker 1: Yeah, the way I talk about this is I often describe us as a lighthouse, you know, we can’t necessarily walk the path for everybody, but we can light the way and hopefully show people how to get to where they need to be and show them opportunities that may not have existed before. And related to that, one of our core values is to produce actual demonstrations of our ideas, right? Real tangible objects, computational objects that a human can use. You know, one is this just kind of comes from our background as like hackers and entrepreneurs, you know, you have to go and build it, but it also kind of comes from a DARPA sort of motto. DARPA, of course, is the American Department of Defense Research Group. And they talk about how you really have to show somebody a robot climbing a wall before they’ll believe you. It’s one thing to say, oh yes, robots can climb walls now, and to write papers showing sort of thrust vectors and forces and friction and so on. But if you walk into a room and a robot climbs a wall, then you believe. So we like to show people the robot climbing the wall, that’s what we do. That’s a long way from having robots in every home, but, you know, once you’ve seen it happen once, then you believe. 00:16:18 - Speaker 2: Well, I think we sort of naturally slid into the topic here, which is independent research and actually you previously described I can switch as being an independent industrial research lab, so it seems like it’d be worth breaking those down a little bit. I guess I’d like to start actually really fundamentally what is research and how does it differ from other kinds of work you could do in the computing field, and then what do those modifiers mean for you? 00:16:43 - Speaker 1: Yeah, so, OK, what is research? In the words of Richard Hamming, who I think is quite an influential figure in this space, his art of doing science and engineering, essay, lectures and book. There’s a really great book from Richard Hamming, published by Stripe Press. If you haven’t got a copy of that, you should definitely pick it up if this stuff interests you. But he said, if you know what you’re doing, and it’s science, you should not be doing it. And if it’s engineering and you don’t know what you’re doing, you should not be doing it. So science is oriented fundamentally towards the unknown. That’s what research is. If you know how to do it, you shouldn’t be doing it. You should be oriented towards the unknown. So, as a research group, we are oriented towards how to solve questions, how to answer questions that nobody knows the answer to, or at least that we don’t know the answer to and can’t find, you know. There’s an old joke that 2 years in the lab could save you 2 hours in the library. So we do try and find prior art and do some survey of the space before we do research, but, you know, sometimes you miss things and then you find out when you go to publish that somebody else had already written about that. But that’s OK. We’re learning as much for ourselves as for anything else. So that’s research, but of course, research comes in a lot of flavors, right? There’s sort of pure research, you know, if you think about the National Science Foundation will fund you. To do things potentially if you have the right connections and proposals that may not bear fruit or may not have any direct industrial application ever, or certainly not for hundreds of years. And a great example of this would be prime number theory. Prime number theory is at the very heart, the very heart of cryptography. Now it is extremely important mathematical research. But most of the core of that research happened like 100 years before anybody found any use for it. So it’s undeniably good for society that this research was done. I mean, depends how you feel about cryptography, but let’s say that it’s good for society that this research was done, but the people who actually did the research, they were dead for centuries or decades at least before any real fruit of this sort of transformational work was known. And that’s not to say that there isn’t value in it, it’s just industrial research. What we’re trying to do is be much closer to what’s happening in the world today, and to sort of connect problems that we see in the real world of humans and software authors and businesses back through this sort of lens of what we care about to doing research. So we want to identify problems or we don’t see solutions. And then we want to develop candidate solutions that could be applied in at least the span of our lives or hopefully over the next few years. 00:19:27 - Speaker 3: Yeah, and I think importantly, it’s a two-way relationship with the industrial ecosystem. You’re solving problems that have some potential industrial application, but you’re also drawing from the experience of industry, you’re talking to the customers, you’re understanding the current technology and what’s possible, you’re aware of the trends and things that are happening and it’s that sort of alchemy of not only solutions to problems, but potential that you’re identifying from the industry. 00:19:51 - Speaker 1: That’s right. And so I think you often find industrial research groups inside big companies like Microsoft Research or Xerox PARC, or Bell Labs. So there have been a number of these very impactful industrial research groups in recent decades, but of course, all of those examples I just listed. are anything but independent. And this brings us to sort of what is perhaps most strange and most unique about ink and Switch, which is that we are not part of an academic body and we are not part of another corporation. We are autocephalus, we are our own head, there’s nothing above us, much like the strange Greek monasteries of the Orthodox church that date back their independence thousands of years. Well, OK, we don’t have thousands of years, but, you know, the independence is a blessing, right? We’re not sort of tied to the Quarterly returns cycle of some business, but it also creates all of these kinds of pressure and constraints. We don’t have this like cozy relationship with the money fountain that will keep trickling us budget year to year. We have to kind of find ways to carve out our existence as we go. 00:21:03 - Speaker 2: And if you’re interested in kind of all the ways that research and labs work in the world, I point the audience to Ben Reinhardt’s work. He’s got a pretty Extensive set of writings on this. He writes quite a bit about DARPA, which you previously mentioned there, Peter, just because they’re one of the more interesting long running government sort of research funding institutions. And he speaks about ink and which specifically as sort of inverting this normal relationship between an innovation organization and it’s money machine, right, which is that the corporate research labs, yeah, Bell Labs certainly is has been a creator of so many incredible things from lasers to GPS. To the transistor, for example, and Xerox PARC, of course, for its short run was legendary. There’s many others like that, say the skunkworks, but typically you have a corporate parent whose job is they’re an industrial company, their job is to commercialize to sell things to make money, and then the money they make from that, they want to put into this research arm. But it does mean that always that that organization, of course, is subservient to that. So with it could switch the idea here a little bit is to kind of switch that around a bit, or at a minimum, make it kind of standalone. 00:22:15 - Speaker 1: Right, and so for us, you know, there’s sort of two puzzles, one which is, you know, how to get our work out there, how to have an impact, because You know, as an industrial research group doing work but not actually impacting the industry, you know, what was the point, you know, you could have just stayed home and watched Netflix all day, would have had the same end result. So it’s important to us to actually sort of change how things happen, but we also need to find ways to do that sustainably over time. So we have a number of hypotheses, you know, as with all kinds of like, long term projects, you tend to have long cycles before you see results. But we believe that there’ll be a combination of strategies that ultimately make this thing self-sustaining, and one of those is to commercialize our technology through producing spin-out companies and then maintaining sort of a share in those companies. And of course, the flagship example of this is Muse itself, right? 00:23:13 - Speaker 2: I mean, that’s kind of the videos. 00:23:14 - Speaker 1: Yeah, spoiler alert. Here’s the big reveal, you know, Muse came out of work we were doing at I can Switch. And the original version of Muse was a prototype built at the lab, and of course, the 3 founding partners, 3 of the 4 founding partners, were there 4 of you in the beginning? 00:23:32 - Speaker 2: We were 3 actually, so they were all 3 lab participants. 00:23:35 - Speaker 1: Yeah, so, all 3 of the founding partners of Muse were ink and Switch staff, and of course, that was the 2 of you, and then Julia as well. So the lab helped develop the concepts that led to Muse. It brought the team together and it built the initial prototypes, and then as a spin out from the lab, the lab retains some stake in the company. So we’re not only delighted to see our ideas put into practice, we’re incredibly excited to see the work that you’re doing and the testing of our values in the world, but we’re also sort of directly incentivized to see you succeed. And so I really love this kind of like Symbiotic relationship where we have both proof in the market of our ideas being feasible, but we also have this incentive to follow closely and make sure that, you know, we’re doing research that can help and that we’re communicating with you and vice versa. So I think it’s a really great relationship and I’m looking forward to many more of those as the years go on. 00:24:36 - Speaker 2: Yeah, I hope at least the flywheel that we’re trying to go for here, but as you said, it’s a very long cycle to prove that out is do research that is driven by real world problems. And of course, basic science and just the pursuit of truth for its own sake is absolutely incredibly valuable, the prime number theory you mentioned or maybe Gregor Mendel sitting there breeding his pea plants just to learn about how the world works or to learn about mathematics or computing’s most basic principles is worthwhile, but in it which really goes after things that are related to real world problems, all the ones that are maybe too far out, for example. Startup or a commercial entity to tackle. 00:25:16 - Speaker 1: What a great segue. Let’s talk about why Ink and Switch exists a little more. What we’ve talked about is the notion of like independent industrial research in the abstract. You could do this kind of independent industrial research in any field, material science, you could build spaceships if you had enough capital, which apparently some people do, you know, you could study journalism, as long as you have this sort of like connective loop. But in the words of Thoreau, it’s not enough to be busy, so are the ants. The question is, what are we busy about? And what we’re busy about is sort of this despair that computers have increasingly become mechanisms entirely for consumption, and that so many of the groups and bodies that were building the bicycles for the mine that were pursuing this sort of vision of computers as these intelligence amplifying devices have kind of retreated from that vision towards a more Natural consumer demand. And I don’t hold this against Apple or anybody else, you know, there are billions of people on Earth who need great computers, and most of them are not scientists or authors or policymakers, and so it’s natural that they would sort of follow the pull of their users and go to where the market is. But we believe that this has left a gap where there just simply isn’t a major computing group that’s actually focused on solving problems. Around how to use computers in this sort of intelligence amplifying way. And specifically, the reason why a research group exists to solve this problem is that people are, oh, well, startups are these great innovators. That’s only true in a very narrow way. All three of us on this call here have plenty of background in startups, both working in them, and founding them, advising them, etc. And startups can take certain kinds of risks, but they can’t take every kind of risk, and it’s sort of like the old advice about only break one law at a time, you know, it’s the same thing with startups, which is you should only take one risk at a time, and generally for the startup, there’s like a core hypothesis to the startup that they are trying to test. It’s Some new kind of product or in some new kind of market. And so the advice people give startups is, well, you should be conservative in all your other choices because you’re already taking a massive risk on this one axis. And so the way we see that play out is lots of people are building startups that would be much more effective from, we think a software and user perspective as local first software, but they build them as cloud sass because that’s what the market expects and that’s what’s easy, that’s what’s known how to do. It’s also what venture capitalists expect. Right, venture capitalism is, let’s be honest, mostly about pattern matching and comparing with other past successes and trying to into it based on this sort of both guiding and being guided by the structure of what the market is doing right now. And that’s fine too. It’s just, this doesn’t leave much space for the kinds of major transformative direction. That a research group can pursue. We can think a decade out. A startup can only think 2 or 3 years out at most. And so that means that we have the opportunity to do work that simply isn’t possible in the context of a startup. And in so doing, we can bring those time horizons in closer and bring things into sort of striking range for a startup, and then those startups can go and pursue these ideas. And in fact, we’re seeing that all the time these days, not just with Muse, but You know, we’re seeing lots of startups these days who reach out and contact us and say, hey, thanks so much for your essays, they were really inspiring, they’ve been really influential on how we’re thinking about this, and we’re hearing from venture capitalists when there are local first startups in some new domain, we’re seeing tangible results of this work on the industry today. 00:29:10 - Speaker 2: Your point about markets and the fact that sort of even startups or commercial ventures, I think just generally speaking, need to follow market trends. It would be dumb not to. In fact, that’s almost the definition of a good opportunity is capitalizing on an immediate trend and In fact, what we see that for example, ARPA slash DARPA, what they do is try to change the whole industry. So one famous example of theirs is they decided in the early 2000s, we think self-driving cars should be a thing, and they put their money to work with prizes and other kinds of funding grants to Get a bunch of both researchers but also companies and investors interested in that, and they basically kicked off that whole revolution that has yet to perhaps fully yield fruit, but at least you see they got a bunch of people caring about it, they got a bunch of money into it, they got a bunch of smart people who want to make that future come true. 00:30:05 - Speaker 1: Yeah, and DARPA refers to this as the challenge-based model. So they use this trick everywhere. DARPA doesn’t actually do any research in-house. They are fundamentally a funding agency. And so, with self-driving cars, they said, hey, we’re gonna give you this challenge, which is you have to drive this car across the desert. And the first year, I think none of the contestants made it to the finish line at all, but each year it got closer and closer until they were eventually, there were entrants that were able to traverse this whole landscape and they won some prize money, but they talk about at DARPA how The way they evaluate projects is that they want things that are really ambitious in their outcome, but they also want things that are tangible in their direction. And so they need to be both very hard, they call it DARPA hard as sort of like a discriminator of ambition. They don’t want something that’s just difficult, they want something that’s really difficult, but they also want something that can be articulated in like a real world results that can be measured or evaluated. 00:31:06 - Speaker 2: Yeah, so obviously I can switch doesn’t have that same structure. We do the research in-house. We don’t have government money to spend, etc. but I think just broadly this idea of moving the market rather than going with the trends that exist, we’re saying we actually don’t like some of the trends where the massive consumerization of computers, which has many nice benefits like most people use the computers in their pockets as Messaging devices, right, and stay in touch with friends and family. Great, that’s good, but then who is working on the creative tools? Creative tools are just not sexy, not interesting, not where it’s seen to be the place that smart people or smart investors put their effort. 00:31:45 - Speaker 1: Well, the smart investors is, I think the secret here. This is the core, right, which is that if you’re an investor, you need a Big market and it turns out that a bunch of novelists and like researchers, there aren’t a billion novelists and researchers to buy your product and if there were, they probably don’t have the budget themselves to go after it. So I think it’s perceived as a relatively small market, though we think there’s a lot of opportunity there if you build tools focused for them. 00:32:10 - Speaker 2: I don’t know, Microsoft Office did OK back in the day. Adobe seems to be doing all right. 00:32:14 - Speaker 3: That’s true. Yeah, so here’s my variant on this. I think there are actually 3 big markets that are already very well funded and being effectively pursued. One is what I would call consumer software, this is ad funded, engagement based software, Facebook, and so forth. There’s enterprise software, which is perhaps actually the most profitable domain right now. This is now Microsoft. AWS stripe, things like that. A third one actually that we often don’t mention is like surveillance and weapons related stuff, like kind of government stuff. That’s all very well funded. So this is an area with individuals that’s perceived to be less economically lucrative. So the entirely predictable and predicted outcome is that there’s just much less investment and therefore, much less progress in it. And we’re trying to jumpstart that flywheel a little bit and get some more progress in this area. 00:33:00 - Speaker 2: But I do think it’s not just on the investor side, we talked to Jason Yuan about this, which is, for example, doing design work for designers, do they want to go to work on the next Excel or do they want to go to work on the next, I don’t know, Snapchat and. At least in the recent past, I feel like these consumer facing companies because they are so big because they have such big outsized brands, but you’ve heard of them, but also they have so much money to put into it, they can do a lot to build those brands and be perceived as a place that the best talent comes to. So I might think of one of the goals of use and I can switch and all the other folks who are in our sphere of creative tools and tools for thought and so forth is just making that a cool area that people are drawn to work in. 00:33:47 - Speaker 1: Obviously you got to pay your rent and Nothing says cool like 6000 word essays on esoteric technical subjects, and then we all got our own definition of cool. 00:33:55 - Speaker 1: Oh yeah. 00:33:56 - Speaker 3: I think that’s true. I think that gets to this idea of institutional alchemy, which is it’s not a matter of just going to a space and working on it or even of just providing funding for it. You gotta bring together all these different factors. There’s funding, there’s vision, there’s people. There’s memes, you know, there’s lines of research, you need it all to be there. Communities, yeah, exactly. And one of the things I’m really proud of with ink and Switches I think we’ve managed to do that somewhat for this space. There’s this nexus, this scene that is formed around the work. 00:34:29 - Speaker 1: Yeah, I think that’s true. It’s very exciting to see a growing space, and of course, we’re talking about Ink & Switch and so I’m sort of focused on that, but I want to note that like we’re by no means operating in a vacuum, and there are plenty of other people like Gordon Brander out there who are writing about this, and there’s lots of startups like Rome Research, who are out there building products in addition to Muse, and I like to think that we are. Certainly a significant player in that ecosystem, but I don’t mean to sort of imply ownership over it or take credit for what is by every indication. A rapidly growing and very dynamic space. I think that’s true. One thing I would like to talk about is why this is important, and why we’re doing this. Now, I think all of us, sort of longtime members of the lab. Believe that the world is kind of in a rough spot right now, like in a big picture sense, right, we have major climate change problems ahead of us, we’ve got this sort of breakdown of the old social order, which is largely driven by new dynamics and communication tools through social media, without getting too much into that, there’s these attentional economy problems, right? You know, you go to unlock your phone and there’s 75 push notifications waiting for you from a mixture of actually useful things and marketers that have somehow managed to slide into your mentions. So you’ve got like all of these kinds of, like, real major issues, and here we are monkeying around with CRDTs and programmable ink. You know, I’ve had friends who are like, if you think these problems that are facing the world are so important, why are you playing with computers? Why aren’t you out there working on the front lines of climate change? I think it’s a good critique and it’s a fair question, but I think, specifically our work. And our motivation behind our work is that we want to empower the people who have those first order skills to be able to attack these problems. We want to support physicists, we want to support journalists, we wanna support writers and authors, and there’s this whole pipeline of knowledge. Pipeline is Gordon Brander, if you’re listening. There’s a systems theory explanation here. It’s not a pipeline, there’s lots of feedback loops at every step along the way, but there’s this system of cultural change that occurs where real knowledge about the universe produces change in the way that we live. And at one end of the pipeline are scientists and social theorists and other people, activists, and then, you know, it goes through a long process that involves a lot of different parties and systems like journalists and writers who bring these ideas to the public and talk about them and communicate them and eventually winding up with policymakers and entrepreneurs and government officials who either change public policies, introduce new laws, or start new businesses and pursue opportunities revealed by this. And so all of that kind of structure. You know, we believe that there’s a lot of unnecessary friction and loss of productivity caused by these bad tools. Every time someone who could be working on climate change loses their data because whatever, the computer reboots to run an update or, you know, they have some great idea and they go to take a note, but they get distracted before they take that note, because their phone is telling them to play whatever Flappy Bird or something. Like anytime those kinds of things happen. That is slowing down and taxing our ability to respond to all of these problems. And so we as a group, the people in the lab, don’t have physics degrees. We’re not social theorists, but we hope that by working with journalists and talking to them, and working with scientists and talking to them, and working with writers and talking to them, that we can learn what the challenges these people have are. And develop a set of values and theories and practices that we can then build infrastructure around that will enable these people to be more effective. But why that, right? The answer is just, this is what we’re good at, this is what we know how to do, right? We’re taking our skills, our unique advantage, and we’re trying to apply them indirectly, perhaps, but sincerely and over a prolonged time scale and intentionally towards enabling those kinds of changes. 00:38:58 - Speaker 2: 80,000 hours has a kind of whole set of essays and theory about how to apply your career in the most effective way as possible to do good in the world, and they reference exactly that point you just mentioned, which is you have to find the overlap between a meaningful problem in the world. What you specifically are great at, drawn to or passionate about, or you have skills or knowledge or experience that very few others have. And the third one actually is an area that not a lot of people are working on. And so I think that’s part of what drew us to this space for for and switch. 00:39:31 - Speaker 1: Not a lot of people were working on this when we started here. Certainly there have been times when people have kind of looked at me and been like, why aren’t you doing AI or ML or blockchain or cloud software or like any of the things that anybody ever raises any money to work on. It’s like, well, doing what everybody else does takes you where everybody else is going. If you want to solve interesting problems, you’ve got to work on things that other people aren’t already doing. That’s part of it. 00:39:57 - Speaker 3: Yeah. Another angle on the why that I would give in addition to this like kind of productivity and effectiveness motivation is one of just aesthetics and beauty and sentiment. I talk about how we are using these tools 468 hours a day, and it’s incredible. demoralizing if they’re not beautiful, if they don’t feel good. And I think that is the state of a lot of our tools. And you would feel terrible and in fact, refuse to go to work on a really creative project, 8 hours every day in like some terrible concrete block building, right? It would be demoralizing. That’s a situation that I feel like we are in with a lot of our tools. And just making people feel better about what they’re working with well I think also enable more good work. 00:40:37 - Speaker 1: Yes, we’re asking all of the artists and scientists of the world to do their best work inside a Walmart full of billboards, while the internet is screaming at them. Yeah. It’s not ideal. 00:40:49 - Speaker 2: Yeah. Yeah, to sort of summarize what both of you said or perhaps my take on it would be that computers and software and the internet have become a really fundamental infrastructure in the world. They’re like roads and bridges, they’re like our agricultural systems and our economy, and those things need to work well and for us to do anything else, solve any other worthwhile problems or do. Things like make great art, create beauty in the world, and computers have gone from being something that is maybe more of a toy or a novelty to being something that is fundamental and everywhere. And so them supporting the humanity’s best aims and noblest pursuits versus being swept up in that Walmart of billboards demanding your attention is something we think will produce returns for all of society. 00:41:41 - Speaker 1: Talking about all of society, I think one of the really valid criticisms of both our work and a lot of the other work in this space is that it is rooted in this kind of technocratic elitist view that there are experts and brilliant people out there in the world, and what we really need to do is enable them to do better work. You know, I have young kids and I was at sort of some 3 year old’s birthday party in a playground. And I got to talking to a golf course equipment manager who lives here in this town, super nice guy, sort of opened with, oh, I don’t know too much about those computer things. But he’s interested in what we did, so I was trying to sort of translate my work for him because, you know, it’s nice to hear about what other people do and share what you do. 00:42:23 - Speaker 2: Just as a note, the cocktail party conversation, which is exactly what I described this, trying to bridge, you know, describe your nichey work to someone that’s completely outside the field, was one of the biggest challenges for me working on Incode switches very hard to summarize that. It’s gotten a little easier since I’ve been onus cause I can just say, yeah, we’re making an iPad app. 00:42:43 - Speaker 1: Well, so, funny enough, this gentleman who claimed not to know much about computers was actually more plugged into the problems of our industry than anybody I know in Silicon Valley that you just pick off the street. He starts telling me about how he doesn’t trust the tractor companies. Because they’re taking all their data and putting it in cloud services and it’s not necessarily to the benefit of him or the equipment managers at the golf course, and how he’s been building all his own software and FileMaker Pro to do all of this stuff in-house, because he doesn’t like that all this data is going to these. Enterprises who are deciding what they pay and how much maintenance is going to cost and all these things, and he’s trying to maintain his independence and he doesn’t trust them not to change the data. I’m like, oh man, you get this work better than almost anybody I talked to. You understand the problems cause you’re out there on the edge. You’re the guy who is suffering because of all these venture capital backed startups and this big push to cloud sass, like, yeah, there’s a lot of benefit, but there’s a lot of cost. And this guy sees it in a way that most Silicon Valley software people don’t, because he’s feeling the pain. And so one of the things that I think we need all to do as a group is do a better job of like. Realizing that climate change is not something for scientists to solve, it’s too late for that, right? We need resilience at every level of our communities. Everybody is going to be finding new ways to live and working on new solutions to these problems, right? Social media driven sort of like cultural collapse. It is not something that only media theorists are going to solve. It’s all of us on our phones, on our computers, in our lives are going to find new ways to communicate. We’re going to need new tools to find that quiet space for ourselves, whether you’re, you know, designing knitting patterns or whether you’re Organizing neighborhood barbecues and potlucks, right? Like this is not solely a problem that affects sort of elite thinkers in society, and we should be building tools for everybody and thinking really about deep resilience through all of society. And so I think that’s something that I’ve been challenged on lately and I’ve been thinking a lot about and so I want to encourage everybody listening to remember that like this is an all of society problem and so we should be building all of society responses. 00:44:57 - Speaker 2: Well, after that grand and sweeping discussion of the big picture stuff, I’d actually be curious to just get a little nuts and bolts. What does an Ink & Switch project look like? Mark and I have talked before here about the Hollywood model a little bit, and you talked in the very beginning about the pens down, the strict pens down at the end of the project, but for example, two of these two projects you mentioned earlier, what does it actually look like start to finish? 00:45:22 - Speaker 1: So, we have a number of sort of active areas of research, and there are other areas we need to expand into and haven’t, but that’s a separate problem. So, to do a project, the first step is to find a problem, and so we have a process of sort of developing project briefs we call pre-infusion. And so, you know, around the lab and in our community of past and future collaborators are a lot of people who are sort of tracking these kinds of interesting problems, but a good Ink & Switch problem is one that we think is really important. That we don’t know how to solve and that we don’t think somebody else will solve for us if we just wait a little 00:46:03 - Speaker 1: bit. 00:46:04 - Speaker 2: So maybe an example here might be the rich text CRDT you mentioned earlier. We’ve done lots of projects with CRDTs because that’s a great technology for collaboration and local first, but then anytime we go to add text, you go, can we make this rich text? Nope, because there’s no such thing as a rich text CRDT. OK, well, put that in the future research bin somewhere. 00:46:26 - Speaker 1: Right, and I do want to point out that Kevin Jan’s uh YJS project has a rich tech CRDT, but it had some properties that we felt didn’t make it suitable for the particular task we had at hand, and also, while he’s done a ton of great work there, There’s no sort of documentation about how to do this that other people could pick up and draw from because it’s, you know, an implementation he wrote to solve his problems. So we wanted to write something about this, both so that we could implement it as well, but also so that other people could learn from this and build on this work in a more sort of open way. So that’s a great example of a problem. So basically do some research upfront. We usually have one person doing this sort of pre-infusion thing, they’ll look at the state of the art, they’ll try and scope a problem, they’ll define what we need to do, and they’ll kind of write up like a 1, maybe 2. Page description of what’s the problem we’re trying to solve and how are we going to solve it and who do we need on the team to be able to solve that. And that’s important because as you’ve talked about the Hollywood model before, but for folks who haven’t heard that episode, we sort of treat each research project kind of like a film production. We put together a team of experts or domain knowledge or generalists with the right kind of philosophy, depending on the project. To execute on some vision, and then when it’s over, we ship the paper and most people move on and go on to whatever they’re doing next, just like you would if you were building a film, rather than these sort of like permanent staff researchers. So we do have a couple of folks like that around as well. And so that’s kind of the pre-infusion. 00:47:57 - Speaker 2: One thing I’ll just note briefly on the staff side is that being able to hire people kind of as short term for these projects, which are usually a couple of months, means that you can get the exact combination of skills that you need. When we’ve done more kind of innovation oriented R&D within an existing company that has a standing staff, you know, when it comes time to say, well, we need to implement this back end system, what are we going to use? Well, we use Erlang because we have an Erlang programmer on the team rather than coming at it from what is actually the best technology or the specific skills that we need and go. Find those people. I think one of the more dramatic examples to that for me, Peter, was when you came into the lab and started advocating for CRDTs as a solution to the kind of sinking problems we’ve been thinking about a little bit, and then we decided to do a project around that, you know, this was 2016 or something, because this was such an early time for that. There was probably only 10 or 12 people in the world who you could consider experts or even reasonably knowledgeable about CRDTs. We literally made a list, you made a spreadsheet and contacted every one of them and saw who might like to work with us or who did we vibe with, and that’s how we met Martin Kleppman. 00:49:08 - Speaker 1: Yeah, and we’ve been working with him ever since. 00:49:11 - Speaker 2: Yeah, it turned out to be a great collaboration, but it’s a good example of where You don’t start from, you know, you say, well, we’ve got some post graphs experts on the team, let’s do post graphs. No, you start from what is actually needed if you look at the whole world of skills and technology and design needs and what have you, and then go find the specific person or people who have that exact skill. 00:49:33 - Speaker 1: Yeah, and one of the other sort of killer advantages of the lab is that there’s Code for America, and Code for America is this great organization that basically gets engineers to spend some time doing basically civic service, building software for some kind of governmental body or agency and. You know, we could debate sort of what percentage of that work ends up being sort of abandonwa or how effective that is, but I think the model is great, which is it sort of recognizes that in your career, you may have these gaps between things, you’re finished one thing, you’re not quite ready to start something new. And so Code for America, it’s sort of like an escape hatch that people use when they’re tired of their job or just need a break or want to do something different. Similarly, because we run these sort of short term projects, We’ve had this amazing ability to bring in incredibly talented people who will be in eventually are snapped up by the top companies in our industry. Moments later, but we get to work with them for a few months on some really interesting project and not only do they influence our thinking, but we influence their thinking, and so we’ve really been able to scale our influence on the world. And to bring in a ton more insight and knowledge and sort of inspiration by working short term with people than we would have by working long term. And so I think that’s one of our like secret weapons for sure at the lab. So, once we scope a project and kind of say, OK, well, we want to do this thing, we’re going to need these kinds of people, we go into recruiting mode, as we said, and we bring in a team. And then at that moment, our goal is like on kickoff day for a project, not always, but this is sort of the archetype. We want to have like a really clear sense of, you know, what’s the tech stack, what are we going to build, what are some vague milestones along the way. At this point, we’ve done prep, and you know, no plan survives contact with the enemy. Once the project gets started, things tend to change, but we want to come in with like a really clear direction and hit execution mode, and the team just focuses, they go head down. And they just engineer the crap out of it for like 2 or 3 months. We always pick sort of a fairly strict end date at the start of the project. You know, the fear of like impending deadlines helps everybody stay focused. And the other thing we do throughout the project is we do sort of a show and tell every other week where teams come and say like, hey, this is what we’ve been up to, you know, share with the rest of the lab, get feedback, but that sort of regular cadence of showing your work both helps people to stay focused and on task, but it also gives you kind of like regular mile. Stone so you don’t end up rabbit holing too bad. You know, it’s not like there’s like a law, but you have to show up on this day and have something new to demo. But if you’re going to turn up on Friday, on Monday, you know, this week we’re going to be showing our work, so you sort of make that extra little bit of effort to like make sure things work a little better or to polish something up a little bit or just to make sure you don’t break the build on Thursday kind of thing. So I really like that cadence. And then at the end of that time, it’s sort of like pencils down, project is done. And in the beginning, we really just sort of like at the end of the project, someone on the project, the project lead, would usually write a little internal Google doc talking about what we did, and in the really early days, everybody at the lab was on the same project, so everybody knew what the project did because you did it together. But at the end of the project, these days we’re spending more and more time writing, editing and publishing. And today that process generally takes 2 months, 2 to 3 months of work, and it’s certainly not full time work and often overlaps with pre-infusion for the next project, but it’s very much this kind of like process of not just publishing to share the knowledge with everybody else, but also mining your memories of that experience for insight and figuring out what happened. So that you can tell that story, and the process of writing, I think, really turns the work into knowledge in a really powerful way. And so as valuable as it is to publish this work to everybody outside of the lab, I think it’s at least as valuable to those of us inside the lab because it adds this extra level of rigor and understanding that we all benefit from. And then at the end of the publication, we put up these massive essays with lots of cool interactive elements and links on the sides. And then you dust your hands and take a quick breather and head back into pre-infusion and do another. 00:54:01 - Speaker 2: I think the you can switch essay style is something that developed over time. I think you wrote one of the first pieces just kind of as a medium post, you said, hey, we should publish about this work, and then we’ve gotten more comprehensive over time and really putting a lot of effort, like you said, basically now, at least in terms of wall clock time spending essentially as much time writing. And distilling the results to understand what’s important, what are the real findings, and then put it into this very nicely comprehensible thing that others can read and learn from. And certainly I’ve experienced that thing you talk about, which is taking the time to do that. I feel like I extract more insights from the project through the writing process. And then in terms of remembering it both in terms of institutional memory, but also in terms of just as an individual, I remember the things better or I can go reference my own work. It happens with some frequency of muse where we say a design question comes up or a technical question. I say, you know, actually we researched that exact thing at I can switch, let me go pull up the reference and. Sometimes we have some internal docs, but those tend to be pretty rough and they haven’t been through the same clarifying process of writing for an external audience. But when we’ve published about it publicly, then it’s much easier to both find what you’re looking for, but then also be able to draw from that and our team can use that to make decisions about something we’re working on the product. 00:55:24 - Speaker 1: Oh, it’s great to hear that that stuff is benefiting you and that you’re able to take advantage of that. I mean, obviously you’re very close to the lab, but it’s nice to know that our target audience is able to get that kind of yield. I’m always asking how we can reinvent the lab, you know, I’ve just talked about what the lab is, but I’m always thinking to myself about what’s next. I’m always thinking about how can we change. We’ve gotten much better at writing, we’ve developed this, we call it the academic voice, it’s not academic, academia has too much jargon and It has a really powerful culture around like these latexch publications. We try to make our writing more accessible than academic papers generally are in terms of tone and language, but we try to take a lot of inspiration from academic writing in terms of maintaining a sedate tone, avoiding superlatives, supporting any statements that we make. We do actually publish a lot of our work in various academic venues these days. But first and foremost, we published for the web and for this sort of industry audience and for ourselves, and the academic version sort of follows on from that. 00:56:31 - Speaker 2: Yeah, the academic format, you might think of as being an illustration of how the lab sits on the innovation spectrum between startups and classic academia, where, yeah, academia, you know, you typically have, yeah, these PDFs, they’re published in particular venues, you’ve got peer review, you’ve got a lot of conventions about citations and things designed to enforce rigor. And then in the startup world, people do publish, but it’s usually some engineer tried a framework one time and they wrote a blog post about their insights on that. 00:57:02 - Speaker 1: Right, or the point of it is like this marketing exercise for a product. 00:57:05 - Speaker 2: Yeah, exactly, marketing for startups, yeah. 00:57:08 - Speaker 1: We try to be very sober in our presentation and to be really explicit about trade-offs and shortcomings and open questions, and I think that serves us well and it serves our readers well because we try to make sure that we portray the work we do with the inspiration and the context that we hope will make people excited about it. We also try to be really transparent about what we have and haven’t done and what remains to be done as well. 00:57:35 - Speaker 2: Well, as a place to end, you mentioned what’s next for the lab or how might it expand or reinvent itself, what exactly is on your mind there? 00:57:45 - Speaker 1: Well, to me, the biggest areas where we can improve are both growing the scope of the kind of work that we do in terms of both getting broader and deeper, I want to see us expand into other adjacent areas, hardware, and I’d like to see us expanding into ever more fields of research, but some of that comes down to just sort of like funding bandwidth and people around. But I’m also interested in Expanding the ways that we work and opening up the lab to more people. We have this vision of a new kind of computer, and you know, it’s that vision of a different kind of way of using computing, not just programming, but computing that is more empowering, that makes us better versions of ourselves, that lets us do more things to do better work, to work better together, to think deeper. And I just think that our current working model is doing fine, but there has to be so much more opportunity for us to take what we’re doing to a wider audience and to get more people involved. And I think some of that is writing more about the why behind the

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: There’s been very little innovation and research more generally into what is a good interface for inputting equations. So I think most people are probably familiar with Microsoft Word or Excel have these equation editors where you basically open this palette and there is a preview and there is a button for every possible mathematical symbol or operator you can imagine. 00:00:28 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad and Mac. This podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. I’m Adam Wiggins here today with my colleague Mark McGranaghan. Hey, Adam. And joined by our guest Sarah Lim, who goes by Slim. Hello, hello, and Slim, you’ve got various interesting affiliations including UC Berkeley, Notion, Inc and Switch, but what I’m interested in right now is the lessons you’ve learned from playing classic video games. Tell me about that. 00:01:01 - Speaker 1: So this arose when I was deciding whether to get the 14 inch or 16 inch M1 MacBook Pro and a critical question of our age, let’s be 00:01:10 - Speaker 1: honest. Exactly, exactly. I couldn’t decide. I posted a request for comments on Twitter, and then I had this realization that when I was 6 years old playing Organ Trail 5, which is a remake of Organ Trail 2, which is itself a remake of the original. I was in the initial outfitting stage, and you have 3 choices for your farm wagon. You can get the small farm wagon, the large farm wagon, and the Conestoga wagon. I actually don’t know if I’m pronouncing that correctly, but let’s assume I am. So I just naively chose the Conestoga wagon because as a 6 year old, I figured that bigger must be better and being able to store more supplies for your expedition would make it more successful. I eventually learned that the fact that the wagon is much larger and can store a lot more weight means that it’s a lot easier to overload it. Among other things, this requires constantly abandoning supplies to cut weight. It makes the roover forwarding minigame much more perilous. It’s a lot harder to control the wagon. And yeah, I never chose that wagon again on subsequent playthroughs, and I decided to get the 14-inch laptop. 00:02:12 - Speaker 2: Makes perfect sense to me and and what a great lesson for a six year old trade-offs, I feel like it’s one of the most important kind of fundamental concepts to understand as a human in this world, and I think many folks struggle with that well into adulthood. At least I feel like I’ve often been in certainly business conversations where trying to explain trade-offs is met with confusion. 00:02:35 - Speaker 1: They should just play Organ Trail. 00:02:37 - Speaker 2: Clearly that’s the solution. And tell us a little bit about your background. 00:02:42 - Speaker 1: Yeah, so I’ve been interested in basically all permutations really of user interfaces and programming languages for a really long time, so this includes the very different programming languages as user interfaces and programming languages for user interfaces, and then, you know, the combination of the two. So right now I’m doing a PhD in programming languages, interested in more of like the theoretical perspective, but in the past, I’ve worked on I guess, end user computing, which is really the broader vision of notion, I was at Khan Academy for a while on the long term research team. 00:03:18 - Speaker 2: Yeah, and there I think you worked with Andy Matuschek, who’s a good friend of ours and uh previous guest on the podcast. 00:03:24 - Speaker 1: Yes, definitely. That was the first time I worked with Andy in real depth, and I still really enjoy talking to him and occasionally collaborating with him today. So, I guess, prior to that, I was doing a lot of research at the intersection of HCI or human computer interaction and programming tools, programming systems, I guess. So, one of the big projects that I worked on as an undergrad was focused on inspecting. CSS on a webpage or more generally trying to understand what are the properties of like the code that influence how the page looks or a visual outcome of interest, and there I was really motivated by the fact that you have these software tools have their own Mental model, I guess, or just model of how code works and how different parts of the program interact to produce some output and then you have the user who has often this entirely different intuitive model of what matters, what’s important. So they don’t care if this line of code is or isn’t evaluated, they care whether it actually has a visible effect on the output. So trying to reconcile those two paradigms, I think is a recurring theme in a lot of my work. 00:04:30 - Speaker 2: And I remember seeing a little demo maybe of some of the, I don’t know if it was a prototype or a full open source tool, but essentially a visualizer that helps you better understand which CSS rules are being applied. Am I remembering that right? 00:04:43 - Speaker 1: Yeah, so that was both part of the prototype and the eventual implementation in Firefox, but the idea there is The syntax of CSS really elides the complexity, I think, because syntactically it looks like you have all of these independent properties like color, red, you know, font size, 16 pixels, and they seem to be all equally important and at the same level of nesting, I guess, and what that really hides is the fact that there are a lot of dependencies between properties, so a certain property like Z index, you know, the perennial favorite Z index 9999999. Doesn’t take effect unless the element has like position relative, for example, and it’s not at all apparent if you’re writing those two properties that there is a dependency between them. So I was working on visualizing kind of what those dependencies were. This actually arose because I wrote to Burt, who is one of the co-creators of CSS and was like, Hi, I’m interested in building a tool that visualizes these dependencies. Where can I find the computer readable list of all such dependencies? And he was like, oh, we don’t have one, you know, we have this SVG that tries to map out the dependencies between CSS 2.1 modules, and even there you can see all these circular dependencies, but we don’t have anything like what you’re looking for. That to me was totally bananas because it was the basic blocker to most people being able to go from writing really trivial CSS to more complicated layouts. So I was like, well, I guess this thing doesn’t exist, so I’d better go invent it. 00:06:12 - Speaker 2: Perfect way to find good research problems. Now, more recently, two projects I wanted to make sure we reference because they connect to what we’ll talk about today, which is recently worked on the equation editor at Ocean, and then you worked on a rich text CRDT called Paratext at In and Switch. Uh, would love to hear a little bit about those projects. 00:06:34 - Speaker 1: Yeah, definitely. So I guess the Paroex project, which was the most recent one was collaboration with Jeffrey Litt, Martin Klutman and Peter Van Harperberg, and that one was really exciting because we were trying to build a CRDT that could handle rich text formatting and traditionally, you have all of these CRDTs that are designed for fairly bespoke applications. They’re things like a counter data type or a set data type that has certain behavior when you combine two sets, and we’re still at the stage of CRDT development where aside from things like JSON CRDTs like automerge, we don’t really have a one size fits all CRDT framework or solution. You still mostly have to hand design and implement the CRDT for a given application. And it turns out that in the case of something like rich text, it’s a lot harder than just saying, oh, you know, we’ll store annotations in an array and call it a day, because the semantics for how you want different types of formatting to combine when people split and rejoin sessions and things like that are all very complex and it turns out that we have a lot of learned behaviors that arise, even from just like, Design decisions in Microsoft Word, where you expect certain annotations to be able to extend, certain annotations to not extend, things like that. Capturing all of the nuance in that behavior turns out to be really difficult and requires a lot of domain specific thinking. But we think we have an approach that works and I would really encourage everyone to read the essay that we published and try to poke holes in it too. This was like the 5th version of the. algorithm, right? So like months ago, we were like, all right, let’s start writing and then Martin, who has just an incredible talent for these things is like, hey, everyone, you know, I found some issues with the approach and, you know, oh no, 00, and sort of we fix those, we’re like, all right, you know, this one’s good and just repeat this like week after week. So I really have to give him a ton of credit for both coming up with a lot of these problems and also figuring out ways to work around it. 00:08:33 - Speaker 2: We talked with Peter a little bit recently, Peter van Hardenberg, about the pencils down element of the lab, but also just research generally, which is there’s always more to solve, you know, it’s the classic XKCD, more research needed is always the end of every paper ever written, which is indeed the pursuit of the unknown. That’s part of what makes science and Seeking new knowledge, exciting and interesting, but at some point you do have to say we have a new quantum of knowledge and it’s worth publishing that. But then I think if it’s just straight up wrong or you see major problems that you feel embarrassed by, then if you want to invest more. 00:09:09 - Speaker 1: Right, exactly. I think in this case. There was a distinction between, there’s always more we can tack on versus we wanted to get it right, you know, and in particular, the history of both operational transforms or OT and CRDT for rich text, just text in general is such that it’s this minefield of I guess to use kind of a gruesome visual metaphor, just dead bodies everywhere. You’re like, oh, you know, such and such algorithm was published and it’s such and such time and it was new hotness for a while and then we realized, oh, it was actually wrong and this new paper came out which proved like 4 of the algorithms were wrong and so on. And so with correctness being such an important part of any algorithm, of course, but also kind of this white whale in the rich text field, we thought it was important to at least make a credible effort to having a correct algorithm. 00:09:57 - Speaker 2: Yeah, makes sense. Yeah, I can highly recommend the Paroex essay. One of the things I found interesting about it, maybe just for anyone who’s listening, whose head is spinning from all the specialized jargon here, CRDTs are a data structure for doing collaborative software, collaborative documents, and then, yeah, rich text, the Microsoft Word is the canonical example there. You can bold things, you can italic things, you can make things bigger and smaller. Well, part of what I enjoyed about this paper was actually that I felt, even if you have no interest in CRDTs, it has these lovely visualizations that show kind of the data representation of a sentence like the quick brown fox, and then if you bold quick, and then later someone else bolds fox, you know, how do those things merge together. But even aside from the merging and the collaborative aspect, which obviously is the research, the novel research here. I felt it gave me a greater understanding of just how rich text editing works under the hood, which I guess I had a vague idea of, but hadn’t thought about it so deeply. So, highly recommend that paper. Just give them the figures, even if you don’t want to read the thousands of words. 00:11:05 - Speaker 1: I’m glad you like the figures. They were a real labor of sigma. 00:11:08 - Speaker 2: Perfect, yeah, so. 00:11:10 - Speaker 1: The one thing I would add is that CRDTs are a technology for collaboration, but the way they differ from operational transforms or OTs is that a CRDT is basically designed to operate in a decentralized setting, so you don’t need a persistent network connection to all the parts. you don’t need a centralized server. The idea is you can fluidly recover from network partitions by merging all of the data and operations that happened while you were offline, and this turns out to be really important to our vision of how collaborative editing should work because we think it’s really important for people to be able to do things like not always be editing in the same document at the same time as everyone. Maybe I want to take some space for myself to write in private and then have my changes sync up with everyone else thereafter. Maybe I’m, you know, self-conscious about other people editing. are seeing my work in progress, but I think that it would be interesting and helpful to look at what the main document looks like and how that’s evolving while I’m working in private, and you can have that kind of one way visibility with something like a CRDT versus with something like Google Docs, where it’s just sort of always online or always not editing in your own personal editor. Conversely, maybe I’m OK with everyone else seeing the work that I’m doing in progress, but I just find it really visually jarring to have all these cursors and different colors jumping around and People inserting text, bumping my paragraphs down the page. I’ve definitely been there. I’m not particularly precious about people seeing my work in progress, but I just cannot focus on writing when the page is just changing all around me. So in that situation, maybe I would want to allow other people to see my work in progress, so that we don’t duplicate effort or something like that, but I just have like a focus mode where incoming changes don’t disrupt my writing environment and these kinds of fork join one way window. Microgit style branching paradigms are really only enabled by a technology like CRDTs where you have the flexibility to separate and then come back together. 00:13:12 - Speaker 2: And I’m incredibly excited by the design research that needs to go into that. Now at this point, I think we’re still on the technology level, you know, one way to think of it is Google Docs came along, I don’t know, 15, it’s almost 20 years ago now, I can’t even remember, let’s say 15 years ago, and this novel idea that We could both have a shared document or several people could have a shared document, all see the up to-date version and type into it and get, you know, a reasonable response or have that be coherent was an amazing breakthrough at the time and has since been kind of widely copied notion, Figma, many others. But now maybe we can go beyond that, much more granularity, like you said, maybe borrowing from the developer version control workflows a little bit in a lightweight way, giving a lot more control and flexibility, and giving us a lot more choices about how we want to work most effectively. But before we can even get onto those design decisions and how do we present all these different things to the user, what are the different options? We need this like fundamental underlying merge technology, hence the endless fascination that we have the lab and increasingly the technology industry generally has with CRDTs because it has the potential to enable all that. 00:14:23 - Speaker 1: Yeah, when we were working on the Paratax project, Peter was pushing really hard for, don’t make this just a technology project. It’s a socio-technical endeavor and we need to invest a lot of time in the design component, also just doing user interviews, identifying how people interact with and. How people collaborate in the status quo on text and Jeffrey and I actually did do a bunch of user interviews with people from all kinds of backgrounds. We’ve talked to people who write plays, people who produce a dramatic podcast kind of in this style of Night Vale. I love Night Vale. Yeah, people who are in the writer’s room kind of working together with their collaborators on that, people who write lessons, video lessons for educational platforms. And there was a ton of really interesting Insights into user behavior around collaborative text. We ended up just torn because we had this 12 week project and we were like, how should we best spend our time? Clearly, this is not just a technical area and we need to invest a lot in getting the design right, understanding what the design space even looks like since it hasn’t really been explored. I really want to avoid, and this is a recurring theme in my work, I really want to avoid publishing or shipping something. And having it be this like, very broad, very shallow exploration into all the things that are possible. I think that this kind of work plays an important role, and there are a lot of people who do this well, just fermenting the space of possibilities and getting these ideas in a lot of people’s heads, who can then go on and do really cool things with them. My personal style, I never want to feel like something is half baked, I guess, I would much rather ship this cohesive contribution like, here is an algorithm for building rich text. We think that this is a technical prerequisite to all of these interesting design choices, but the alternative with a 12 week period, and in fact, you know, this, the correctness and revision phase extended way over that. So thanks a lot to Martin and Jeffrey for leading during that part. But it’s just already so hard to get it correct that trying to tack on a really substantive design exploration that does the area justice on top of that, I was just really worried it would stretched too thin. So absolutely lots of room for future work in this particular. project. It’s very much a challenge in any area where you have simultaneously this rich design space that’s just asking to be explored with tons of prototypes and things like that, and then also to even realize the most simple of those prototypes, you require fundamentally new technology. 00:16:53 - Speaker 2: Yeah, I’ve been down that same path on many research projects as well, and often it’s that I’m excited for what the technology will enable, but also that in many cases it’s a combination, you know, some kind of peer to peer networking thing, but with that will enable us to provide a certain benefit to the user and I want to explore both of those things, but then that’s too much and then the whole thing is half baked exactly as you said. I’ve never found a perfect or even a good. Way to really manage that tradeoff. You just kind of pick your battles and hope for the best. Yeah, definitely. Well, I do want to hear about the equation editor project, but first I feel I should introduce our topic here, which I think folks could probably have gleaned is going to be rich text and rich text editing, and maybe we could just step back a moment and define that a little bit. I think we know that texts, you know, symbolic representation of language is a pretty key thing, writing and the printing press and all that sort of thing. We wrote about that a little bit in our text blocks memo, which I’ll like in the show notes. But typically, I think computers for a lot of their early time and even now with something like computer code is typically plain text, that’s the dot TXT file is kind of almost the native style of text that you have and then rich text typically layers something on top of that. I don’t know, so maybe you could better define rich text for us to have a more concrete discussion about it. 00:18:21 - Speaker 1: Yeah, I think rich text for most people basically evokes things like bold, italic, underline, the ability to augment plain text with annotations that are useful in formatting, actually, I think. Notepad to word pad is the archetypal jump in software, if you’re thinking about it from the old Windows perspective. In the past few years, I think we’ve started to see a real expansion of what rich text can look like. So, of course, we started out with something like Markdown, which is, of course, a plain text representation. But it’s designed to be able to capture more nuance in plain text and be rendered to something like HTML which very much supports rich text. So in Markdown, you have not only these kinds of inline formatting elements like bold and italic and hyperlinks as well. You also have support for images, which you could think of as more block level rich text elements, I guess, and I don’t think there’s a real clear consensus across editors on how block level rich text elements should be displayed. Of course, in between you have things like bulleted lists and those tend to be handled in a fairly standard manner with nested lists and so on, but it quickly becomes like a question of taste. Which kinds of annotations you support. So in editors like Coto or Notion, you have all these different block types where the block is really the atom of collaboration and editing, and then you can have things like, you know, file embeds or even database views, things like that. So I think we’re at a point now where both block-based editors, I’m using block based editors in like the text or writing sense, not the structured editors for programming sense, although I have other things to say about that, but we’re at a point where you’re starting to see these block-based editors appear and I think that there are a lot of really interesting patterns that this permits that the paragraphs via linear sequence of characters, including new lines and whitespace does not permit, or at least doesn’t allow you to build as structured tooling around. 00:20:30 - Speaker 2: I’m trying to think what is actually the core of the difference between a block-based editor, that’s a notion, a RO uses working on its own block text implementation and a flow of characters, so that’s Microsoft Word, Google Docs, maybe even text editors. I guess it’s sort of like paragraphs are separated by like these sort of nested. Elements or have a parent to the document versus like two new lines embedded in the stream of characters, but I don’t know, that seems too unsophisticated, maybe have a better definition for us. 00:21:03 - Speaker 1: So, I actually think about this very similarly to in the like programming languages and editor tools space. There is a distinction between structured editors and regular plain text editors for programs. The idea is that you might have a text-based programming language and you can write that perfectly fine in any buffer that allows you to put sequential characters, often AI is sufficient for some languages, and then on the other hand, These programs might have a lot of inherent structure. A simple example is with lisps which are built out of these parenthesis S expressions, everything is, you know, an S expression. You can think about like the structure of the tree formed by, I guess a forest, formed by having like these S expressions with subelements and stuff. that, and then you can do manipulations directly on the structure in a way that allows you to always have a syntactically correct program or at least a partial syntactically correct program by doing things like I’m just going to take this subtree, which is a sub-expression and move it somewhere else where there’s room for another subexpression. So, I think of block-based editors as capturing a very similar zeitgeist to structured editors for code, because instead of just having this linear buffer of characters that can have, you know, formatting or things like that, you can have new lines, you actually have more of a forest structure where you have lots of like individual blocks, and then you can have blocks that are children of other blocks and so on, and that allows you to Do things like move an entire subtree representing an outline to another position in the document without selecting all of the characters, you know, cut them and then paste them somewhere else. So things like reparenting becomes a lot easier, things like setting the background of an entire subtree becomes a lot easier. Just in general, you have more structure and there’s more things you can do with that structure, I guess is how I would phrase it. One of my favorite things that you can do with this model in notion is you can change the type of a block very easily. So let’s say I have a bullet list item, and then I hit enter and enter these like subnote or something like that as children of the initial bullet list item. I can turn the bullet list item into a page, and then all of a sudden it’s just a subpage in the document, and the sub bullets that were there before are just like top level bullets in that page. And this is particularly important for my workflow because I care a lot about starting out with something like really rough and sketchy and then progressively improving it or moving up and down the ladder of like fidelity into something more polished. So you might, for instance, start off with just an outline list or even a one dimensional list of to do blocks when you’re trying to do project planning or something. And then later on, let’s say I want to put these into like a tasks database with support for like a conbond view or something like that. I don’t actually want to sit there and like recreate all of these tasks in Jira. I’ve been there, you know, I’ve been the person making all the tasks in Jira after the meeting and then assigning them to people. What the workflow that I think notion is poised to enable and can certainly do a better job in this regard, but already offers some benefits on is like, can I just highlight all of these blocks because everything is a block, move them into some existing database and have them match the schema. That kind of like allowing people to do fast and loose prototyping with very unstructured primitives and then promote them into something more structured like in a relational database setting or similar, I think is the sweet spot, structured editing provides the sweet spot between like just completely unstructured text and these very high fidelity, high effort interfaces that allows you to kind of move between them. 00:24:47 - Speaker 3: Yeah, I really like that direction and framing, and if I can extend it a little bit, I think we can also look at a continuum of richness in terms of the content itself. So you have plain text, what you might classically call rich text with links and bold and underlying. And then you maybe start to throw a few images in, and then what if you can put it in videos and what if you have a whole table, and that table is actually a database query, and you can nest the figment document, and this way you can see that there’s sort of continuum on the richness of the document. One reason I think Notion has been so successful, they’ve been pushing along that continuum while maintaining a sort of foundation of rich textness, which is very familiar and the important basic use case for a lot of people. A related idea is that I think we’re seeing a lot of the classic document types converge. So if you look at a rich text like a Microsoft Word and a PowerPoint and increasingly spreadsheets, those all used to be 3 distinct Microsoft Office applications, and we’re seeing the value of them being in or being the same document. This is actually one of the motivating ideas behind Muse and a lot of the research we’ve done in the lab, and the kind of something Slim was saying, you want to take your idea continuously through different media and different modalities and different degrees of fidelity, and you don’t want to jump between different applications do that. You want to be able to do it on the same canvas. That’s by the way, one of the reasons I like Canvas. It’s not only because it’s a free multimedia surface, but also it evokes this idea of like flexibility and potentiality, and I think that’s one of the things that’s really excited about these mixed media documents. 00:26:16 - Speaker 2: And I know if Jeffrey were here, he might jump in and say that one downside to our current application silo world is that the only way to have this deeply rich text where it’s images, video, a table, a database query, something like that, is to have the Uber application, to have the everything app, and certainly notion has probably gotten pretty far on that, but others kind of in In some ways are forced to do that, like we have to do some of that in Muse as well. People come in and ask for all these different types here as well, and there’s more of like an open doc inspired or Unix inspired future that maybe Jeffrey and others, including me, would hope for, which would be more that applications could be these individual data types and you could put them all together through some kind of more operating system connection. But that is so completely reversed from kind of how all our computing devices work today. It’s hard to see how we might get to that. 00:27:14 - Speaker 3: Yeah, I’m certainly sympathetic to that concern, although I suspect the way out is through, and you get platforms from working killer apps. And so the way we got the whole unit ecosystem was they wanted to build a computer for, you know, writing and running programs and then eventually got all this generalized text processing stuff, but it’s not like they started in like, oh, I’m gonna make a generalized text processing machine. I don’t think that was really the way that they approached it and developed a success. So, I’m still hopeful we could do this, but I think you got to extract it from something that’s already working as an app, but it always helps to have an eye towards that, and I think we’ve done some of that with Muse. 00:27:46 - Speaker 1: I was just going to say that it’s not me talking about texts, unless I bring up my favorite piece of software of all time, which is Pandok. And I think that Pandok actually is very relevant to this discussion. So for those who aren’t as familiar with it, Panok brands itself as this Swiss Army knife for document formats, and it’s sort of headline contribution is that it allows you to convert between all kinds of documents. For instance, I can take a Word document and convert it to a PDF Word documents to something like, I don’t know, IPython notebook, Jupiter notebook, back and forth across this incredible bipartite graph of formats, but I think that the subtler contribution that Pandokc makes, which is extremely significant, is that Pandok has this form of markdown called Pandok markdown that essentially aligns and supersedes all of the different fragments of markdown that we’ve seen before. So the problem with markdown basically is that the original specification is sort of ill-defined. There are several cases in which the behavior is not super clear and then on top of that, it’s not very expressive. There aren’t very many constructs. So things like fenced code blocks, which many people associate very closely with Markdown today, that was only added by GitHubb flavored markdown, which is certainly widely used among the programming community, but not everyone is on GitHub, of course. And then you have things like table formatting or even like strike through really strike Through wasn’t defined in the original markdown specification either. And so you have markdown and then you have like GitHub flavored markdown, common mark is sort of this unifying effort remark down all these different is the markdown cinematic universe. I tried to make a joke about this. I had this joke ready for the markdown Cinematic universe when the last Marvel. Movie came out. But then like, it didn’t get nearly the traction in my timeline as the Dune did, perhaps understandably. So really, I’m just going to have to wait till the next movie comes out. It’s a real, real tragedy. No, but like, I guess you have this real pluralism of forms and it becomes very difficult to use markdown truly as a portable format because the way it renders in one editor or even parses can very much differ from editor to editor. So, Pandoc provides this format that essentially serves as an IR or intermediate representation between all these kinds of documents using a markdown supersets that somehow magically encapsulates everything. 00:30:18 - Speaker 2: And that includes not just markdown, but also like PDFs or Microsoft Word, that seems. 00:30:24 - Speaker 1: Well, so the way it works is it’s this compilation pipeline, I guess, that allows you to go from a markdown document. It compiles it to PDF using PDF Lawtech or something. It outputs Lawtech, it outputs HTML various things, and you can think of it as being this intermediate representation because you start with this like Word document, you can turn that into markdown and you can go from that markdown format into any of these output formats, which turns out to be like really powerful because the main issue with these kinds of conversions is that it’s often lossy, there are features that are supported by Law tech, for instance, that aren’t supported by the web natively, there are features that are part of like Word documents that aren’t necessarily supported by HTML and so on and so forth. So Pandok serves this role of like basically saying, OK, what is an intermediate language that can encapsulate all the different implementations of the same concept across different input and output formats. And what I think is so remarkable about it is that oftentimes when you are using an AP. of software and you’re like, oh darn, you know, now I need to support this other thing too. You quickly end up in a situation where you have the snowball and things start to feel tacked on. So you’re like, Oh man, it’s very clear that they just glommed on this additional syntax for this feature. And with Pandok, everything feels like very principled in its inclusion. And at the same time, whenever I’m using Pandok and I’m like, darn, I really wish there was a construct that I could use to express this. particular thing, I look up in the documentation and it’s always supported. So, as one of my favorite examples, one of the output formats that Handok supports is various slideshow frameworks. So Beamer for people who use Lawtech and Reveal JS for people who use HTML and CSS and these slideshow frameworks basically allow you to replace something like PowerPoint, Keynote, Google slides with essentially like a text-based format. I really like doing slideshows in Pandock markdown. There are a few reasons for that. The first reason is that it’s really useful to be able to reuse some of the same content from like my blog post or essay even in the slideshow. There are some really minor and almost petty, but really significant reasons. Like, I like to have equations or code blocks with syntax highlighting in my slideshows, and there’s not really a good solution to putting like a syntax highlighted code block in Keynote right now. 00:32:39 - Speaker 2: Last I remembered, the gold standard at the Ruby conferences I used to frequent was to take screenshot of Textmate and paste that in. 00:32:47 - Speaker 1: Yeah, it’s awful. I don’t want to see your like monochai editor with like the weird background that contrasts weirdly with the slide background. I just, ah, and it doesn’t scale on a huge conference display anyway, I digress, but The other reason why I really like doing my slideshows in text is actually that there is often a hierarchical structure to my presentations, right? I’ll have like these main top level sections and then I’ll have subsections, and then I’ll have like sub subsections and all of these manifest and slides. But in the gooey thumbnail view of most of these existing Slideshow editors like PowerPoint or Google slides, it reduces it all to like this linear list. It’s like, here are all of your thumbnails in order. And it makes it very hard, as soon as I have like an hour-long conference talk, how do I like jump to this subsection that I know exists, aside from like scrolling past like 117 thumbnails and trying to find the right one, right? And moreover, let’s say I want to Reorder a certain part of the talk because I think it better fits the narrative structure. Now I have to like figure out which thumbnails I need to drag to which other place or worse, go into the individual slide, select the text from that, move that somewhere else, and it’s just way, way clunkier actually than reordering some text in like a bullet list outline in my editor. And then the other part is that I was talking about how Pandok has really great support, expressive support for idioms of different formats, and one thing you often have in slideshows is that I have some element on the screen and then I press, you know, the next button again and then another element will appear. So in Pandoc you can denote this with just like an ellipsis basically so like dot dot dot and then if I have a slide where I have a paragraph and then the dot dot dot and then another paragraph, it will render with just the first paragraph visible and then I press next and then like the subsequent paragraph comes in. And that’s like just a very lightweight way to handle these stepped. Animations compared to going to the animation pane and then clicking the element that I want to animate in and so on and so forth. So it started off with me being like, I’ll just prototype in this format, but then it ended up supporting columns, it supports all these things that you actually want. And I was like, this is in many ways a more ergonomic way to handle long technical slideshows. Anyway, I have to chill for Pandok anytime I talk about rich text, I’m contractually obligated to do so. 00:35:08 - Speaker 2: Yeah, it’s a great piece of software, use it here and there. I think I was doing some Asky doc kind of manuals many years ago and yeah, just in general, it’s also worth looking at the homepage that you mentioned the plot they have where it shows all the different formats that can convert between is quite fun. You click on that, you can zoom in. 00:35:26 - Speaker 1: Yeah, I had this really elaborate plan when I decided to go to Berkeley, that I was going to print out a door-sized poster of like that graph that shows all the formats they convert between and then show up at John McFarlane’s door and ask him to sign it. But then the pandemic interfered with some of those plans. Nonetheless, it remains on my list. 00:35:48 - Speaker 2: Good bucket list item, pretty unique one at that. 00:35:51 - Speaker 1: Also, I found my tweet, or I found the draft of my tweet, which is about eternals, and I said, directed by Chloe Zhao, the latest entry in the Markdown Cinematic Universe features an ensemble cast of multi markdown, GitHubb flavored markdown, PHP Markdown Extra, R Markdown, and Common Mark as they joined forces in battle against mankind’s ancient enemy, Doc X. Nice. 00:36:12 - Speaker 2: Wow. You would have gotten the like from me. 00:36:16 - Speaker 1: Yeah, we’ll see if it ever sees the light of Twitter.com. 00:36:20 - Speaker 2: You briefly mentioned there equations and La tech, and maybe that’s a good chance to talk about the equation project you did for notion. And part of what I thought was so interesting or what I think in general is interesting about equations is that they are obviously an extremely important symbolic format, but in many ways extremely different from the pros we’ve been talking about. So English or other languages, even languages that are right to left or something like that, they all have the same kind of basic flow and the way that we represent sound. So with these little squiggly symbols, even though the symbols themselves and sounds vary and how we put them together into words across languages, that’s a common thing. If you go to the mathematical realm, you have symbolic representation, but equations are the whole own beast, and I think one that has gotten a lot less attention from kind of the software and editing world. So tell us about that rabbit hole. 00:37:16 - Speaker 1: Yeah, so just as context for people, notion and many other applications actually have long supported block equations, an equation that basically takes up, you know, most of the page horizontally. What is much more uncommon in editors is support for inline equations and so this can be something as simple as saying, You want to type let X be a variable, and X should be formatted or stylized mathematically. Being able to refer to elements of a block level equation in inline text is a prerequisite for being able to do any kind of serious mathematical writing, yet because this is kind of this niche area that has historically been the purview of Overleaf and other law tech editors, it’s really not implemented. In most editors. So I pushed really hard to add inline equations and inline math to notion, because I was like, there’s a huge opportunity for people to write scientific or mathematical documents that take advantage of all of notion’s other features like being able to embed FIMA or embed illustrations and things like that, right? So, it turns out that it’s kind of difficult, exactly as you’re describing to do this equation format. There’s been very little innovation and research more generally into what is like a good interface for inputting equations. So I think most people Probably familiar with Microsoft Word or Excel have these equation editors, or even like operating system level sometimes where you basically like open this palette, and there is a preview and there is a button for every possible mathematical symbol or operator you can imagine. And then for composite symbols like the fraction bar or integral or something like that, you find the button for that, you click it, and then you click into like the little subboxes and then you find whatever symbol you want and you put those there too. So it’s kind of a structured editor, but like in an unimaginably cumbersome interface. This is what I used to do my lab reports in high school, for example. And then at the other end of the spectrum, you have things like law tech. Law tech is basically how everyone in at least in computer science and mathematics chooses to typeset their work, typesets complex mathematics. One of the real selling points of law tech, I think is that It turns out that operator spacing is really important, and there’s a big difference between, say, a dash that’s used like a hyphen or a dash character that’s used in text, and a hyphen or a dash character that’s used as a minus sign in an equation, the spacing is subtly different. And one of the big things that Lawtech does is it basically allows you to declare certain operations in certain contexts as like a math operator versus just a symbol versus just like a tagged group of characters, and it correctly handles the spacing depending on what kinds of characters are around the operator in question. And so Lawtech basically produces really nice looking mathematics at the cost of this markdown which looks like I kind of smashed my keyboard that only had like 3 characters. It’s the exact opposite of the equation editors instead of having a button for every imaginable character, you only have 3 buttons. The buttons are backslash, open curly brace, and closed curly brace, and somehow like permuting those characters is supposed to get you like any possible mathematical outfit. There’s just two ends of the spectrum. 00:40:41 - Speaker 3: Yeah, I used to do my analysis homework in college in law tech, and I remember when I first looked up how you would input in law tech these formulas, like, that can’t be right. This is not the best way in the world to do this. In fact, that’s it, that’s the one and only way. 00:40:53 - Speaker 1: It really is, it’s terrifying. It’s the one and only way and the wild part is there are people who are like super, super good at law tech. They can like live tech their lecture notes. I was never nearly like that fast, but some people can do it usually with extensive use of macros, which macros are another selling point of law tech as you can define these kind of custom shorthand for operators you use a lot. But anyway, yeah, so you have a lot of tech sort of at the other end of the spectrum, like really quite unreadable, oftentimes, like, it’s like a right only format, many times. 00:41:23 - Speaker 2: And of a regular expressions come to mind on that as well, yeah. 00:41:26 - Speaker 1: It’s exactly the same zeitgeist, I think. It turns out that figuring out how to have like a combination, gooey, plain text interface that allows you to be like in a rich text editor like notion, then. into an inline equation field to have like an inline symbol and then go back into the GUI editor was like just very unexplored territory. And it kind of makes sense that lots of people don’t prioritize this because many people that notion rightfully had the question like, oh, is this something we should be working on? But first of all, it turned out that if you actually tallied up like our user requests, inline math was like near the top. Of editor feature based requests. And then more generally, it turns out that because this is like a prerequisite for many researchers and for students, you can get a lot of people on your platform who rely on it, you know, as a student to take notes and something like that, because there’s literally no alternative. And then they are able to stick around and use the platform for all kinds of other things. So this is just kind of a plug that more editors should implement this. But Yeah, I thought that this project was really interesting because in the interaction paradigm, you want to capture a lot of the things that are very fluid about editing regular text. So for instance, we knew it was important that you should be able to use the arrow keys to move left and right, kind of straight through a token without editing it if you wanted, or if you wanted to be able to go. Into a token and edit it using the arrow keys, you shouldn’t have to like use the mouse to click, although, of course, you should also be able to use the mouse to click. And when you have this formatted equation, we made the decision that the rendered equation would be represented as this atomic token. So if you were highlighting text to copy and paste and move around, it would be like highlighting a single character that would just be like the whole equation. But of course, you could go in and edit the equation. Any way you want it in kind of this pop up text editing interface. I think another thing that’s the subtle interface challenge here is that like Mark was saying, there is often a Uh, disproportionately large number of characters used to represent the equivalent of like one character with a formatted output. And so that’s something you don’t really take into account. The output is like X with a hat in San Sara font, and then there’s like 25 characters of markup that goes into that, and you just need to like scale the interface appropriately to take that into account. But I think that it’s really interesting because It shows the power of combining different input and output formats in like the same atom, right? So you have like a single line of text, and you want to have rich text that’s formatted and stylized and so on, hyperlinks, and then also equations or whatever inline rendered output of another input format that you have. I think that that’s really where GUI editors and whizzy wig editors can shine is being able to combine these like, Input formats and output formats like in the same line in Chu, yeah, I guess you can’t really do that at all with the terminal or something like that, and I say this as someone who uses like CLIIM for everything. 00:44:34 - Speaker 3: This is bringing back so many memories. I wish I had notion with equation support back when I was a math undergrad. It’s so nice. 00:44:41 - Speaker 1: I’m like the notion math stand guardian, I don’t know, something like that. And I’m always keeping track of like all the cool things people are doing using equations and notion. A lot of people are doing like math blogs in notion, which is really awesome for me to see. Also, I just feel like they’re having tried lots of other things. They’re just like really isn’t. A good alternative short of like actually writing lots like for your blog, which no one really likes. And yeah, I mean, certainly it’s the kind of thing that I implemented originally, kind of, I was like, I’m gonna do this for myself, and then realized that lots of people would be able to benefit from it. It’s been really cool to see a bit of reception it gets, like the inline math tweets on the notion, uh Twitter account overwhelmingly get the most engagement and interaction. And initially, like the marketing team was shocked. They thought this would be the super niche feature, but no, it turns out that people love math and like, they may not be the most vocal proponents or they’re used to no one caring about math type setting, things like that. For a while, I think it was the case that when I did find an editor that had support for equations of some kind, to me, it was overwhelmingly obvious that the people who implemented it did not regularly use equations for writing. I think you can often tell that with different features. So I think that having that kind of Representation is not quite the right word, but being able to see a feature that was designed by someone who really cares about using it themselves is really cool for people who are interested in typesetting, students, researchers, people who are interested in typesetting more mathematical text. 00:46:11 - Speaker 3: Yeah, and I think it’s really important, like you were saying that it’s mixed media because you’re combining the equations, the inline equation and the block equation, by the way, in the world class form, which is a lot tech based with a world class rich text editor with text and images and stuff. It’s really nice. I do think there’s still one frontier here, especially for math, which is the fully gradual process from you’re taking handwritten notes and you’re working out a problem and you’re drawing squiggly diagrams all the way up through your finished homework. I remember when I was at math undergrad. I would basically have to do the homework twice. You do it once on paper. Nobody could read that, including myself, so that, you know, do it in lot again. And I always wish there was a way to do it incrementally. You sort of changed equation by equation and diagram by diagram into the final product. And I know there has been some research on uh turning equations into lot tech formulas with machine learning. I don’t know if I can do handwriting, but perhaps someday we’ll get the new support for equations and you can go all the way to the end. 00:47:02 - Speaker 1: Yeah, like you, I share exactly the same frustration that you have to essentially do lots of things twice, and the relative position of everything is ambiguous, and Lawtech is what allows you to do things like have subscripts of subscripts, which would be really inscrutable in most people’s handwriting, including my own, and, you know, subscripts of subscripts along with super scripts and things like that. There are just so many ambiguous details and it turns out in my experience with like, anything that tries to automate the transition is that I always end up Going through and like really rewriting all of the details to be structured in a readable way. You have this other problem which back in the days of like Wizzy Wig web editors like Dreamweaver and Microsoft Front Page and things like that, you would often end up with this problem where you try to do like any edit in the Wizzy Wig side and then you look at the generated HTML and it’s ridiculous. There’s just like 16 nested empty span tags, and no one would ever be able to maintain that. And my worry is basically that when you automatically create Markup for something that has a very complex graphical representation, it’s really like one way, you know, maybe it will help you produce a compiled output, but it doesn’t actually help you go back in and like edit and tweak the representation later or it’s just so inscrutable if you do that it’s kind of also a reg x type situation. I think we really need to get to some kind of like good intermediate representation that allows you to flexibly go both ways. And that goes back to something that I think Adam and I were chatting about earlier, which is that a lot of people gripe and complain that like law tech is the best we have and, you know, I’m one of them, but It really is the case that, you know, lottech was just this like monumental effort by really a few people and amount of effort that would be like considered really impressive if I were to try to do the same thing but better today and not a lot of people just have like spare time to do this all in one text formatting, packaging, document representation project, even though it would have huge impact on the way people write and publish these kinds of documents. And so in many ways we’re sort of just bottlenecked on the fact that It’s hard to do incremental improvements to this particular area. We really depend on these like software monoliths to keep us afloat. 00:49:19 - Speaker 2: I’m not nearly as mathy as either of you, but I can’t help but make the comparison on these equation editing to what you mentioned earlier with kind of structured editors and programming, where whether there’s lightweight help from your text editor, things like code folding, syntax highlighting and autocomplete, or full structured editing, some of the visual programming stuff we talked about with Maggie Appleton, like Scratch, for example, or these flow based systems that are fully graph. and you sort of can’t have it in a bad state. And I can’t help but to think there might be some direction like that that is not necessarily the right only inscrutable tech, but is not the Microsoft Word one button literally for every symbol you might ever want. It does seem like there might be some other path, and yeah, I agree it’s a monumental effort, but I mean, mathematics is so important and foundational and so much of human endeavor that certainly seems like one worth investing in, although perhaps hard to reap a profit from, and that makes it harder to put concentrated capital behind it. 00:50:20 - Speaker 1: Yeah, I think that there’s definitely very clear demand for I think something exactly like what you’re describing, which is somewhere in between the two extremes, and it is really relevant because ACM, which is the Association for Computing Machinery, the academic and professional body really for computer science, they are currently undergoing this. Fiasco, maybe, I probably shouldn’t go on the record as calling it a fiasco. The ACM is currently undergoing this initiative called TAPS, which is the ACM Publishing System, where they are attempting to revise the template by which all computer science research is published and disseminated, and the idea behind this is that right now, computer science research is published to these PDFs. Initially they were all two column PDFs, now I think there’s some one column PDFs. They want to output HTML as the archival format for various reasons, including that it offers much better reading experience on different screen widths, so like phones or tablets, which are increasingly how people are reading papers, not just printed out. And they are much more accessible than PDFs. PDFs are just like really quite inaccessible, especially to screen readers and other assistive technologies that are trying to parse out all the different math or whatever arbitrary formatting you’ve decided to use. The upshot of this, I guess, is that there are currently a group of very smart people who are trying to figure out how in the world we’re going to get people to start writing all of their papers and outputting them in a different format, in a world where everyone is already used to preparing. Their publications and preprints in law tech. And turns out that even if you solve the problem of like what the input syntax should be, rendering math in the browser is like an extremely unsolved problem. 00:52:05 - Speaker 3: Yeah, isn’t the state of the art that it like generates PNG and sticks it in the web page? 00:52:09 - Speaker 1: Not exactly, but like almost. OK. So MathML, which is like an XML dialect or like mathematical markup language, was this effort to build. HTML XML style syntax for typesetting mathematics. Naturally, it is only implemented in Firefox, so that’s really unfortunate. So in terms of the state of the art, there are basically two libraries that you can use to typeset mathematics. There’s math Jack and Caltech. Mathjax supports basically all valid law tech, including, you know, different. Environments and equations and things like that. The problem is that Mathjacks is very slow. So if you ever go on math overflow or another like related stock exchange and you see like all of these answers with like weird gaps, and then as you watch before you, the page starts to like load all of the rendered equations like bumping everything down one level at a time. That’s math Jackson action. And oftentimes it is doing what you’re describing where it is outputting like an SBG or a PNG or something like that, and it’s just like reflowing the page with every equation. So then you have Caltech, which was a library developed at Konn Academy where they realized that math Jack’s performance was basically just like not satisfactory for their exercises and things like that. Sootte supports a much more limited subset of all of Law tech syntax, but it does it all using CSS basically, and it doesn’t reflow the page for every equation. It’s basically instant surrender. So tech is what we use at Notion, it’s also what’s used in like Facebook Messenger, which supports equations if you ever tried that, and many other websites, and basically it means that your options, if you want to render math are only target Firefox. Use a limited subset of math that’s supported by Kottech and Consign yourself to like extremely slow, dozens of reflow, full expressive power rendering to inline PNG’s. And so that’s just not like a great situation to be in, and we haven’t even gotten to the question of like how people write math. So I would say that people underestimate like how open this problem spaces. 00:54:17 - Speaker 3: Yeah, man. 00:54:19 - Speaker 1: Just take a moment of silence to like recognize the gravity of the situation. 00:54:23 - Speaker 3: This is an aside, I don’t know if you want to put this in the episode, but now I’m curious. It sounds like both of those are interpreted in the sense that the equations are rendered at load time instead of being compiled down to some like HTML and CSS that you can render without JavaScript. Like, basically, do you need JavaScript to render these pages? 00:54:39 - Speaker 1: Yeah, basically, I should say you also need JavaScript, unless you’re doing the pre-compied to MathML and then hope that people are using Firefox. 00:54:47 - Speaker 3: Man, I feel like there’s no way that that stuff loads in 10 years, but we’ll see. 00:54:52 - Speaker 1: I actually had this exact argument, again, I don’t know if you want to put this in the episode. I had this exact argument with Jonathan Aldrich, who’s on the taps committee when we were talking about this, and I think the point was not so much that you can guarantee that the artifact loads. Exactly the same way in 10 years, but that the representation is rich enough that one could feasibly build software that renders it the same way in 10 years. So it’s more about the fidelity of the like underlying representation where like a team of, I guess, digital, you know, archaeologists could recover the work that we were doing and not so much like we trust in the vendors to like keep everything stable, which is obviously never going to happen. You know, the only reason like PDFs are stable is because how many trillions of dollars of IP depend on being able to load the PDF the same way as it was written, you know, 30 years ago. 00:55:45 - Speaker 3: Yeah, interesting. 00:55:46 - Speaker 1: Nice. Going back to this idea earlier that Mark mentioned of the spectrum of like plain text, rich text, Wizzy wig editors. One recurring theme for me is thinking about decoupling this spectrum into like what is the format and then what are like the editors and tools that we can use to interact with this format, so they structured, unstructured, etc. I want to call outAR, which is a native application for Mac OS and iOS that does a really great job with this, which is that Bear is basically Something in between a whizzy wig and a plain text editor in that you’re always editing markdown documents and indeed, when you have something that’s bold, you can see the like asterisks around it that delimits that character. But all of these standard, you know, Control B, U, editor shortcuts work as you would expect. And more importantly, you can see like the formatting applied in real time. So That when you do star star, hello star star, he suddenly becomes bold face in this gooey. And so in many ways it combines like the fluidity and the real-time preview of a rich text editor or previewer with the flexibility of like ultimately just writing plain text characters. And I think this is like really unexplored area. I don’t just mean something like Open VS code or VIM and type characters and then see like different formatting labels attached to the results. I mean like a native application that’s really designed like for end use or end users, that doesn’t fully obscure the input syntax but does real time rendering in place. It’s not even like in monospace font, right? It makes it feel much more like this is actually the output that you’re targeting. And not just like an input step that needs to be pre-processed. I think that there is a lot of room for applications that are kind of in between and in that same spaces where it doesn’t entirely obscure what you are writing, but it does give you a lot of the benefits of previewing things and having like a GUI application outside of the terminal in terms of like capturing the richness of the possible results. 00:57:52 - Speaker 3: Yeah, I like the bear approach a lot. Now, are there particular domains or types of documents that you think would be susceptible to this approach, or it just for rich tech specifically? 00:58:01 - Speaker 1: So I was making a list of like all of the different traditionally graphical outputs that have corresponding plain text representations and a lot of them I was thinking about, for example, in engraving sheet music, right, traditionally you would use a desktop program like Finae or Sibelius nowadays you have options like new score and flat, which are more web-based editors, but you see the staff and you click notes. In the staff like corresponding to where you want the note, and you know you use the quarter note or the 8th note cursor to pick the duration and so on. And then at the other end of the spectrum you have Lily Pond, which is kind of like law tech I guess for engraving sheet music where you type a very like law tech-esque syntax and out comes, you know, beautifully typeset sheet music. For me this

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: But this totally changes how the data is persisted, and I think that’s important because the only way you get good results on sync systems, especially when you’re talking about offline versus online and partially online, it has to be the one system that you use all the time. You can’t have some second path that’s like the offline cache or offline mode that never works. It needs to be the one true data synchronization persistence layer. 00:00:29 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad and Mac, but this podcast isn’t about Muse the product, it’s about me as the company and the small team behind it. I’m here today with two of my colleagues, Mark McGranaghan. 00:00:43 - Speaker 3: Hey, Adam. 00:00:44 - Speaker 2: And Adam Wulf. 00:00:46 - Speaker 3: Yeah, happy to be here. 00:00:48 - Speaker 2: Now Wulf, you are not at all new to the Muse team, I think you’ve been with us for coming up on 2 years now, but it is your first appearance here on this podcast, a long overdue one I would say. So we’d love to hear a little bit about your background and how you came to the team. 00:01:03 - Speaker 3: Yeah, thanks, it’s exciting. Before Muse, I worked for a number of years with Flexits on their calendar app, Fantastical, both on the Mac and the iPhone and iPad. Really enjoyed that. At the same time, I was also working on an iPad app called Luose Leaf, which was an open source just paper inking app, kind of note taking app of sorts, really enjoyed that as well. 00:01:28 - Speaker 2: And I’ll know when we came across your profile, let’s say, and I was astonished to see loose leaf. It felt to me like a sort of the same core vision or a lot of the same ideas as Muse, this kind of like open-ended scratch pad, multimedia inking fluid environment, but I think you started in what, 2013 or something like that, the Apple pencil didn’t even exist, and you were doing it all yourself and, you know, in a way maybe too early and too much for one person to do, but astonishing to me when I saw the similarity, the vision there. 00:02:03 - Speaker 3: Yeah, thanks. I think the vision really is extremely similar. I really wanted something that felt physical, where you could just quickly and easily get to a new page of paper and just ink, and the, the app itself got out of your way, and it could just be you and your content, very similar to you sitting at your desk with some pad of paper in front of you. But yeah, it was, I think I started when the iPad 2 was almost released. And so the hardware capabilities at the time were dramatically less, and the engineering problems were exponentially harder as a result of that, and it was definitely too early, but it was a lot of fun at the time. 00:02:42 - Speaker 2: And I think one of the things that came out of that, if I remember correctly, is this open source work you did on ink engines, which is how we came across you. Tell us what you did there. 00:02:52 - Speaker 3: Yeah, there’s a few different libraries I ended up open sourcing from that work. One was the ink canvas itself, which that was the most difficult piece for me. The only way to get high performance ink on the iPad at the time was through OpenGL, which is a very low level. Usually 3D rendering pipeline. I had no background in that, and so it was extremely difficult to get something up and running with that low level of an architecture. And so, once I had it, I was excited to open source it and hopefully let other people use it without having to go through the same pain and horror that I did to make it work. But then one of the other things that was very useful that came out of loose leaf was a clipping algorithm for Bezier curves, which are just fancy ways to define ink strokes, basically, or fancy ways to describe long curvy, self-intersecting lines. And that work has also been extremely important for Muse as well. We use that same library and that same algorithm to implement our eraser and our selection algorithms. 00:04:05 - Speaker 2: And when you’re not deep in the bowels of inking engines, or as we’ll talk about soon sinking engines, what do you do with your time? 00:04:13 - Speaker 3: Oh, I live up in northwest Houston in Texas with my wife Christie and my daughter Kaylin. And she is in high school now, which is a little terrifying, and learning to drive and we’re starting that whole adventure, so that’s been fun for us. I try and get outside as much as I can. I’ll go backpacking or hiking a little bit. That can be fun, and the Houston summer, it’s rather painful, but the springs and the falls, we have nice weather for outdoors and so. 00:04:42 - Speaker 2: What’s the terrain like in the day trip kind of range for you? Is it deserty? Are there mountainous or at least hilly areas, or is it pretty flat? 00:04:52 - Speaker 3: It is extremely flat and lots and lots of pine trees, and that’s pretty much it. Just pine trees and flat land. Sometimes I’ll drive a few hours north. We have some state parks that are nice and have a bit of variety compared to what’s immediately around Houston, so that’s a good backup plan when I have the time. 00:05:14 - Speaker 2: Flat with a lot of trees sounds surprisingly similar to the immediate vicinity of Berlin. I would not have expected Texas and northern Germany to have the commonality there. It gave me a lot of appreciation for the San Francisco Bay Area, while that city didn’t quite suit. Me, as we’ve discussed in the past, one thing that was quite amazing was the nature nearby and a lot of that ends up being less the foliage or whatever, but more just elevation change. Elevation change makes hikes interesting and views interesting and I think itself leads to, yeah, just landscape elements that engage you in a way that flatness does not. 00:05:55 - Speaker 3: Yeah, absolutely. I lived in the Pacific Northwest for a while, and the trees there are enormous, and the amount of green and elevation change there is also enormous. And so when we moved back to Houston, it was a bit of a shock almost to see what I used to think were tall trees in Houston are really not very tall compared to what I lived around up in Portland, Oregon. 00:06:21 - Speaker 2: So our topic today is sync. Now Muse 2.0 is coming out very soon. We’ve got a launch date May 24th. Feels like tomorrow for our team scrambling to get all the pieces together here, but the biggest investment by far, even though we have the Mac app and we have text blocks are a part of it, the biggest kind of time, resource, energy, life force investment by far has been the local first sinking engine. And we’ve spoken before about local first sync as a philosophy generally in our episode with Martin Klapman, but I thought it would be good to get really into the details here now that we have not only built out this whole system, both the client side piece and the server piece. But also that we’ve been running it in, won’t quite call it production, but we’ve been running it for our beta for a few months now, and we have quite a number of people using that, some for pretty serious data sizes, and so we’ve gotten a little glimpse of what it’s like to run a system like this in production. So first, maybe Mark, can you describe a little bit how the responsibilities breakdown works in terms of between the two of you on the implementation? 00:07:32 - Speaker 1: Yeah, so I’ve been developing the back end or the server component of our sync system, and Wulf has been developing our iOS client that is the core of the actual app. 00:07:45 - Speaker 2: Yeah, on that side, I kind of think of the client persistence or storage layer as being the back end of the front end. So that is to say it’s in the client, which obviously is a user interface heavy and oriented thing, but then it persists the user data to this persistence layer which in the past was core data, is that right? Well the kind of standard iOS storage library thing. 00:08:08 - Speaker 3: Yeah, that’s exactly right. Yeah, we used core data, which is Apple’s fancy wrapper on top of a SQL light database. And that just stores everything locally on the iPad, like you were saying, so that way the actual interface that people see, that’s what it talks to. 00:08:25 - Speaker 2: And then that persistence layer within the client can talk to this back in the mark has created. And much more to say about that, I think, but I thought it would be nice to start with a little bit of history here, a little bit of motivation. I’ll be curious to hear both of your stories, but mine actually goes back to using my smartphone on the U-Bah, so that’s the subway system here in Berlin, when I was first working with some startups in the city back in, I guess it would have been 2014, so, 8 years ago I had this experience of using different apps and seeing how they handled both the offline state but actually the kind of unstable state because you have this thing where the train car goes in and out of stations and when you’re in the station, you usually have reception, weak reception, and you leave the station that fades off to you essentially fully offline, and so you’re in this kind of unreliable network state all the time. And two that I remember really well because they were really dramatic, was one was pocket, which is the relator tool I was using at the time, and it handled that state really well. If it couldn’t load an article, it would just say you’re offline, you need to come back later, but the things it had saved, you could just read. The other one I was using was the Facebook mobile app, and there I was amazed how many errors and weird spinners, and you go to load a thing and it would get half of it, but not the rest of it, and the app just seemed to lose its mind because the network was unreliable, and I found myself thinking, what would make it possible to make more apps to work the way the pocket does and less the way that Facebook works. And I also had the opportunity to work with some startups here, including Clue and Wunderlust and some others that had their own. Essentially everyone needs this. Everyone needs syncing because they want either one, the user to be able to access their stuff from different devices, or 2, they want some kind of sharing, and I think Vonunderlust was an interesting case because they built out this crack engineering team. To develop really good real-time syncing for a very simple case. It’s just a to do list, and the common case that people use it for, I think was, you know, a couple that’s grocery shopping and they want to like, make sure they don’t overlap and pick the same things in the cart. But it worked really well, but they built this huge, I think it was like a 15 person engineering team that spent years of effort to make really good real-time sin, and it seemed strange to me that you need this big engineering team to do what seems like a simple thing that every app needs. We went down this road of trying CouchDB and Firebase and a bunch of others, and all were pretty unsatisfying. And then that further led in, you know, that kind of idea, the sync problem lodged in my mind and then when we got started at ink and Switch, some of our early user studies there were on sync and how people thought about it. And one thing that stuck with me from those was we looked into just kind of syncing on. And note taking apps and talked to a whole bunch of people about this, and we didn’t have a product at the time, so it was just kind of a user research study, but we went and talked to a bunch of folks, most of whom were using Evernote was kind of the gold standard at the time. And almost everyone we talked to, when I asked what’s your number one most important feature from your notes app, they said sync and said, OK, so that’s why you chose Evernote, and they said, yeah, and they said, how well does it work? And they said terribly, it fails all the time. You know, I write a note on my computer, I close the lid, I go to lunch. Half an hour later, I go to pull it up on my phone. It’s not there. I have no idea why. And so some combination of those experiences sort of lodged this thing in my mind of the technology industry can just do so much better, and this is important and everyone needs it. What’s the missing piece. And I wasn’t really sure, but that led into once I met up with folks in the research world who indeed had been working on this problem for a while, and I got excited about the technologies they had to offer. 00:12:15 - Speaker 1: Yeah, and then I guess I was downstream of that because I got introduced to space by Peter Van Hartenburg with time was a principal at the Inn Switch Research Lab, and it’s now the director of the lab. And he showed me a demo of the Pixel pusher project, and we can link to the article on this, but essentially this is a Pixel art editing tool that was peer to peer collaborative, and the app itself is very standard, but was amazing to me was he had implemented this app and he had 2 devices or 2 windows on the same device, and they were doing real-time collaboration, but there was no server. And I had come from this world of wherever you add a feature to an app, you gotta write the front end and then you gotta write the back end, you gotta make sure they line up whenever anything changes, it’s a whole mess, and it was just magical to me that you could just type up this JavaScript app and have it collaborating with another client in real time. So I went down that rabbit hole, and there was the obvious attractions of the austere locations and, you know, minimal network connectivity and things like that. And also at the time the research was very oriented around P2P, so there was this notion of the user having more control of their data and perhaps not even requiring a central server, but a couple of things became even more appealing to me as I researched it more. One was that Potential of higher performance. And I ended up writing a whole article about software performance that we can link to. But one of the key insights was that it’s not physically possible to have acceptably fast software if you have to go anywhere beyond the local SSD. Now, certainly if you’re going to a data center in Virginia or whatever, you’re totally hosed. So it was very important to incorporate this performance capability into Muse. 00:13:49 - Speaker 2: Yeah, that article was eye opening for me and that you connected the research around human factors, things that looked at what level of latency you needed for something to feel snappy and responsive, and then separately the speed of light, which is how sort of the maximum possible speed that information can travel, and if you add those together or do very simple arithmetic on that, you can instantly see it’s not about having a faster network connection. You literally cannot make something that will feel fast in the way that we’re talking about if you have to make a network round trip. 00:14:21 - Speaker 1: Yeah, and the one other thing that was really interesting to me about this space was the developer experience. I alluded to this earlier with the Pixel Pusher demo, but in the before times there were two ways to develop apps. You had the local model where you were typically programming against the SQL database, and everything was right there and it sort of made perfect sense. You would query for what you need and you write when you have new information and so on. And then there was the remote model of you would make rest calls, for example, out to some service like admit this edit or add a new post or whatever. But then these two worlds were colliding where we always wanted to be adding sync and collaborative capabilities to our apps, we would try to kind of jam one into the other, like you would try to patch some rest onto the database or you try to patch some database on yours and it just wasn’t working, and I realized we need to do a pretty fundamental rethink of this whole architecture, which is what we end up doing in the research lab and then now with Muse. The last thing I’ll mention about my journey here was my background was in back in engineering and distributed systems engineering, and so I had encountered variants of the sync problem several times, for example, at Hiroku, Adam. We had this challenge of we had these routers that were directing HTTP requests to a back end that was constantly changing based on these dinos coming up and down, and the routers needed to maintain in memory router tables based on the control plan that was being adjusted by the API. And so we had a similar problem if you need to propagate consistently in real time state to the in-memory databases of all these router nodes, and sure enough that work kind of came full circle and we were applying some of the same lessons here with Muse. So it’s a problem I’ve had the opportunity, for better or worse, to make a few passes at in my career. 00:15:57 - Speaker 3: Yeah, I think it’s an extremely hard problem that comes up so often across so many projects is eventually you need data over here in Box A to look the exact same as data over here in Box B. and it’s one of those problems that’s just surprisingly hard to get right, and there just aren’t that many libraries and existing solutions for it to drop in and implement. A lot of other libraries you can just go out and find it, and there’s code out there, or you can license it or open source, whatever, but for whatever reason, sync is one of those things that’s for every project, it needs to be custom baked to that project, just about every time. 00:16:38 - Speaker 2: And that’s part of what blew my mind back 8 years ago when I was looking for a sinking layer for clue and realizing that, yeah, I just had this feeling like surely everyone has this problem, everyone needs it, everyone needs the same thing. It’s really hard, you know, an individual company shouldn’t be distracting from their core competency of building their app to create the sinking layer, and yet to my surprise, there really wasn’t much, and that continues to basically be true today. 00:17:06 - Speaker 1: Yeah, and this gets into our collaboration with Martin Klutman on CRDTs. So briefly you can think of there being two pieces to this problem. One is conveying the physical data around, and the other is, OK, you have all this data that synchronize, what do you do with it, because it’s all a bunch of conflicting edits and so on. And that’s where the CRDT technology came in. I think one of the reasons why we haven’t seen widespread standard libraries for this stuff is the thinking problem is hard. We’ll talk more about that. But another is that we haven’t had the computer science technology to make sense of all of these edits. Well, we sort of did. There was like operational transforms, but you literally need to hire a. Team of PhD computer scientists have any shot at doing stuff like that. And so Google Docs basically had it and maybe a few others, but normal humans couldn’t do anything with it. But the CRDT technology and automerge, which we’ll talk more about, made it much more accessible and possible to make sense of all these conflicting edits and merge them into some useful application state. So that’s the kind of why now of why now is a good time I think to be pursuing this. 00:18:06 - Speaker 3: Yeah, and I think almost surprisingly to me, the solution we came up with at Muse, I think is actually really generic, and I think we solve it in a really elegant way that’s even more foundational to the technology than solving just for use. I think the solution we have. Can certainly solve from use in the future and is futureproof in that regard, but is broad enough to be applicable to a whole number of different uses and applications, which I think is really exciting too. 00:18:37 - Speaker 2: Maybe it’s worth taking a moment to also mention why we think local first in the style of sync is important for you specifically. I think certainly Mark and I have had a long time interest in it. Well, if you have an interest in it, so it’s just something that’s more like we’d like to see more software working in this way where the user has a lot more sort of control and literal ownership over the data because it’s on their device. In addition to being mirrored in the cloud, certainly the performance element is huge for me personally, and I think for all of us on the team. But I think Muse, as we go to this multi-device world, on one hand, we think that every device has its own kind of unique mood. The iPad is this relaxed space for reading and annotating, whereas the Mac or a desktop computer is for focus, productivity, you know, the phone is for quick capture, the web is good for sharing. OK, so really you need your work to be seamlessly across all of them. But at the same time, you know, we want that sense of intimacy and certainly the performance and the feeling that it’s in your control and you own it, and it belongs to you. I think that maybe matters less for some consumer products, or maybe it matters less for more kind of B2B, you know, enterprisey products, but for this tool, which is for thinking. Which is very personal, which is very kind of needs to be at your fingertips and friction free. I think the local first approach would be a good fit for a lot of software, but I think Muse needs it even more than most. So that’s why I’m really excited to see how this works out in practice as people try it out, and we really don’t know yet, right? It may be that we’ve made this huge engineering investment and in the end customers just say, I’d be happy with the cloud, yeah, it’s fine. I have some spinners, I can’t access my work offline. I hope not. But that could happen. We could be like falsifying the business hypothesis, but I really believe that for our specific type of customer, you’ll go to use this product with the sinking layer, you know, once we shake out all the bugs and so on and say, you know, this feels really fundamentally different from the more cloud-based software that I’m used to like an ocean and also fundamentally different from the non syncing pure local apps that I might use. 00:20:51 - Speaker 3: Yeah, I really think that with as connected as this world is and is becoming, there’s always going to be places of low connectivity, there’s always going to be places of just dodgy internet, and having an application that you know just always works, no matter what’s going on, and figures itself out later once it has good internet, is just so freeing compared to Those times when, you know, your device is switching Wi Fi networks or the LTE is just not quite what it needs to be to make things happen. I think it really does make a really huge difference, especially when you’re deep in thought, working on your content in use, the last thing you want is to be interrupted for even half of a second with a small spinner that says please connect to the internet. And so just being able to free the application and free the user from even worrying about the internet at all, even if it works 99% of the time, it’s that 1% of the time that breaks your train of thought that is just really frustrating. And I think that’s what’s exciting about being able to be purely offline is it fixes that really huge problem of that really small percentage of time that it happens. 00:22:10 - Speaker 2: Very well said. Now with that, I’ll do a little content warning. I think we’re about to get a lot more technical than we ever have on this podcast before, but I think this is a topic that deserves it. So I’d love to, and me especially as someone who’s not deep in the technology and just observing from the sidelines, I’d love to hear about what’s the high level architecture, what are all the pieces that fit together here that make this syncs when you’re on the internet and lets you keep working even when you’re not? What is it that makes all that work? 00:22:41 - Speaker 1: Yeah, I’ll give a really quick overview and then we can dive into some of the specific pieces. So to start with the logical architecture, the basic model is a user has a bag of edits, so you might have 1000 edits or a million edits where each edit is something like I put this card here or I edit this picture, and over time the user is accumulating all these edits and the job of the sync system is to ensure that eventually all of the users' devices have the same bag of edits. And it passes those edits around as opaque blobs and different flavors of blobs we’ll talk about. Basically there’s a bunch of bits of binary data that all devices need to have the same, and then it’s the device’s responsibility to make sense of those edits in a consistent way. So given the same bag, each device needs to come up with the same view of the muse corpus of that individual user, what boards are alive and what cards are on them and so forth. And then briefly in terms of the physical architecture, there’s a small server that’s running on Hiokku, data is stored in post grass and S3 and it’s implemented in Go, and again the server is just shuffling binary blocks around basically. And then there’s different front ends, different clients that implement this synchronization protocol and present a use corpus model up to the application developers. So the most important of these is the SWF client. We also have a JavaScript client and both of these back to SOI databases locally. 00:24:09 - Speaker 3: Yeah, and I think what’s really interesting about this architecture is that we actually maintain the entire bag of edits. Edits only get added into the bag, but they never really get removed. And so the current state of the application is whatever the most recent edit is. So if I make a card bigger on my Mac, and then I go to my iPad and I make that same card smaller. And then we synchronize those two things. Well, at the end of the day, either the card is going to be smaller on both devices, or the card is gonna be bigger on both devices, and we just pick the most recent one. And that strategy of just picking the most recent edit actually makes conflicts essentially invisible or so small and so easy to fix that the user can just, oh, I want that big, let me make it big again. It’s really easy to solve. For the user side without showing up one of those really annoying, hello, there’s been an edit. There’s a conflict here. Would you like to choose copy A or copy B? Just being able to automatically resolve those is more than half of the magic, I think, of this architecture. 00:25:13 - Speaker 2: I also note this is a place where I think the muse domain, if you want to call it that, of the cards on a canvas model works pretty well with this sort of automated resolution, which is if you moved a card in one direction on one device and you moved it somewhere else on the other device, it’s not really a huge deal which one it picks as long as it’s all kind of like flows pretty logically. By comparison, text editing, so what you have in a Google Docs or certainly I know auto merge team and the incode switch team has done a huge amount of work on this, is a much harder space where you can get into very illogical states if you can merge your edits together, strangely, but I think a card move, a card resize, add remove, even some amount of reparenting within the boards, those things just are pretty natural to merge together, I think. 00:26:02 - Speaker 3: Yeah, I think so, and I think even with the new text block feature in Muse, we end up slicing what would be a really long form text document into much smaller sentences or paragraphs. And so then text edits, even though we’re only picking the kind of the most recent one to win, we’re picking that most recent at the granularity of the sentence or of the the paragraph, and so. Conflicts between documents for us are larger than they would be for automerge or for Google Docs, but are small enough that it’s still ignorable for the user and easily solvable by the user. 00:26:42 - Speaker 2: Which incidentally I think is a trick we sort of borrowed from FIMA, at least on the tech side, which is in FIGA and also in Muse. If one person edits, you know, the red car and someone else edits the blue car, you don’t get the red blue car, you just get one or the other, and it turns out for this specific domain, that’s just fine. 00:27:03 - Speaker 3: Yeah, I think we kind of lucked out having such a visual model, and we don’t need to worry about intricacies of multi-user live document editing. 00:27:13 - Speaker 1: Yeah, I would point to both Sigma and actual budget as two very important inspirations for our work. I would say those are two of the products that were most at the forefront of this space, and thought about it most similarly to how we did. And notably they, as well as us sort of independently arrived at this notion of basically having a bunch of last white wins registers. As the quote unquote CRDTs. So these are very, very small, simple, almost degenerate CRDTs where the CRDT itself is just representing one attribute, for example, the X coordinate of a given card. But this is an important insight of the industrial application of this technology, if you will. That’s a good trade-off to make it. It basically covers all the practical cases, but it’s still very simple to implement, relatively speaking. 00:28:03 - Speaker 2: I also mentioned briefly actual budget, great in the basically made by one person app and recently open source, so you can actually go and read the custom CRDT work there and maybe learn a thing or two that you might want to borrow from. 00:28:17 - Speaker 3: I think one of the really interesting problems for me about the CRDT was Deciding which edit is the most recent because it just makes logical sense to say, oh well, it’s 3 o’clock, and when I make this edit at 3 o’clock and I make a different edit at 3:02, obviously the one at 3:02 wins. But since computer clocks aren’t necessarily trustworthy, sometimes I have games on my iPad that reset every day and so I’ll set my clock forward or set my clock backward. Or if I’m on an airplane and there’s time zones, and there’s all kinds of reasons the clock might jump forward or jump backward or set to different problems, and so using A fancier clock that incorporates a wall clock, but also includes a counter and some other kind of bits of information, lets us still order edits one after the other, even if one of those clocks on the wall is a year ahead of schedule compared to the other clocks that are being synchronized. I don’t know how in depth we want to get on that, but it’s it’s called a hybrid logical clock. 00:29:23 - Speaker 1: Yeah, I think this is another great example along with choosing very simple CRDT structures of industrial style architecture where you could go for a full blown vector clock, and that gives you perfect logical ordering and a bunch of other nice properties, but it’s quite large and it’s expensive to compute and so on. Whereas if you choose a simpler fixed size clock, that can give you all the benefits that you need in practice, it can be easier to implement, it could be faster to run, and so on. 00:29:52 - Speaker 3: Like everything in life, it’s all about trade-offs, and you can get accuracy, but it costs more, or you can get a little bit less accuracy, and it costs a lot less, and for us that was the better trade-off to have a fixed size clock that gives us Enough of the ordering to make sense, but might not be exactly perfect ordering. 00:30:13 - Speaker 1: And we’ve been alluding to trade-offs and different options, so maybe it’s time to address it head on in terms of the other options that we considered and why they weren’t necessarily as good of a fit for us. So I would include in this list both iCloud and what you call like file storage. 00:30:27 - Speaker 2: It might be like cloud kit or something, but yeah, they have one that’s more of a blob, kind of, you know, save files, what people will think of with their sort of iCloud drive, almost kind of a Dropbox thing, and then they also have a cloud kit. I feel like it’s a key value store, but in theory, those two things together would give you the things you need for an application like ours. 00:30:47 - Speaker 1: Yeah, so there’s iCloud as an option, Firebase, automerge. CouchDB maybe, then there’s the role you’re on which we ended up doing. 00:30:57 - Speaker 2: Yeah, the general wisdom is, you know, you don’t write your own, if there’s a good off the shelf solution, you name some there that are commercial, some are built into the operating system we’re using, some are indeed research projects that we’ve been a part of, what ultimately caused us to follow our own path on that. 00:31:15 - Speaker 1: Yeah, so there was a set of issues that tended to come up with all of these, and it was more or less in different cases, but I think it’d be useful to go through the challenge that we ran into and talk about how they emerged in different ones of these other solutions. So one simple one, it would seem it’s just like correctness slash it works. And the simple truth is, a lot of the singing systems out there just do not work reliably. Hate to pick on Apple and iCloud, but honestly, they were the toughest in this respect where sometimes you would, you know, admit data to be synchronized and just wouldn’t show up, and especially with opaque closed source solutions and third party solutions, stuff would not show up and you couldn’t do anything about it, like you couldn’t see what went wrong or when it might show up or if there was some error. And then bizarrely, sometimes the stuff would pop up like 5 or 10 minutes later. It’s like, oh, it’s actually sort of worked, but it’s off by You know, several zeros in terms of performance. So that was a really basic one, like the syncing system has to be absolutely rock solid and it kind of goes back to the discussion Wulf had around being offline sometimes. If there’s any chance that the sync system is not reliable, then that becomes a loop in the user’s mind. Am I gonna lose this data? Is something not showing up because the sync system is broken. Our experience has been that if there’s any lack of reliability or lack of visibility into the synchronization layer. It really bubbles up into the user’s mind in a destructive way, so we want it to be absolutely rock solid. Another important thing for us was supporting the right programming model. So we’ve been working on news for several years now. We have a pretty good idea of what capabilities the system needed to have, and I think there were 4 key pillars. One is the obvious transactional data. It’s things like what are the cards and where are they on the board. This is data that you would traditionally put in a SQL database. Another thing that’s important to have is blob support, to a lot of these binary assets in use, and we wanted those to be in the same system and not have to have another separate thing that’s out of band, and they need to be able to relate to each other correctly. 00:33:09 - Speaker 2: This is something where a 10 megabyte PDF or a 50 megabyte video just has very different data storage needs than the tiny little record that says this card is at this X and Y position and belongs to this board. 00:33:23 - Speaker 1: Right, very different, and in fact you’re gonna want to manage the networking differently. Basically you want to prioritize the transactional data and then load later, or even lazily, the binary data, which is much larger. Yeah, so there was transactional data, blob data, and then real-time data slash ephemeral data. So this is things like you’re in the middle of an ink stroke or you’re in the middle of moving a card around and this is very important to convey if you’re gonna have real time and especially multi-user collaboration, but again, you can’t treat this the same as certainly blob data, but even transactional data, because if you store every position a card ever was under your finger for all time, you’re gonna blow up the database. So you need those 3 different types of data, and they all need to be integrated very closely. So for example, when you’re moving a card around, that’s real time, but basically the last frame becomes a bit of transactional data, and those two systems need to be so lined up to each other that it’s as simple as changing a flag. If you’re going on a 2nd or a 3rd band for real-time data and need to totally change course for saving the transactional data, it’s not gonna be good. It was quite rare. I don’t know if we found any systems that support all three of these coherently. 00:34:33 - Speaker 2: The ephemeral data element I found especially interesting because you do really want that real timey feeling of someone wiggles a card with their finger and you can see the wiggling on the other side. That just makes the thing feel live and Just responsive in a way that it doesn’t otherwise. But yeah, at the same time, you also don’t want hundreds of thousands of records of the card moved 3 pixels right, and then 3 pixels left. And one thing I thought was fascinating, correct me if I misunderstood this, but is that because the client even knows how many other devices are actively connected to the session, it can choose to not even send that ephemeral data at all. It doesn’t even need to tap the network. If no one else is listening, why bother sending ephemeral data? All you need is the transactions over time. 00:35:21 - Speaker 1: Right, this is actually a good example of how there’s a lot of cases where different parts of the system need to know or at least benefit from knowing about other parts. So it becomes costly or or maybe just an outright bad idea to separate them, especially as we’re still figuring out as industry how they should work. I think there’s actually quite a bit of benefits to them being integrated. Another. that we could talk about eventually is prioritizing which data you download and upload, you might choose to first download blobs that are closer to you in board space, like it’s in your current room or it’s in adjacent rooms, and then later you can download other blobs. So that’s something you could do if the application had no notion of the networking layer. It actually brings us to Another big challenge we saw with existing systems, which is multiplexing. So I’ll use an example of automerge here, and this is something we’ve seen with a lot of research oriented CRDT work. It’s very focused on a single document, so you have a document that represents, you know, say a board or whatever, and a lot of the work is around how do you synchronize that document, how do you maintain correctness, even how do you maintain performance when you’re synchronizing that document across devices. Well, the challenge with Muse with our model. You might have, you know, easily 1000, but, you know, potentially tens of thousands up to millions of documents in the system corresponding to all your individual cards and so on. And so if you do anything that’s order and in the number of documents, it’s already game over. It needs to be the case that, here’s a specific challenge that I had in mind for the system. You have a corpus, let’s say it’s a million edits across 10,000 documents or something like that, and it’s 100 megabytes. I wanted the time to synchronize a new device that is to download and persist that entire corpus, to be roughly proportional to the time it would take to just physically download that data. So if you’re on a 10 megabyte connection, 100 megabyte connection, maybe that’s 10 seconds. But the only way to do that is to do a massive amount of like multiplexing, coalescing, batching, compression, so that you’re taking all these edits and you’re squeezing them into a small number of network messages and compressing them and so on. So you’re sort of pivoting the data, so it’s better suited to the network transfer and the persistence layer. And again, you need to be considering all these things at once, like how does the application model relate to the logical model, relate to the networking protocol, relate to the compression strategy, and we weren’t able to find systems that correctly handle that, especially for when you’re talking about thousands or millions of documents being synchronized in parallel. And the last thing I’ll mention is what I call industrial design trade-offs. We’ve been alluding to it in the podcast so far, but things like simplicity, understandability, control, these are incredibly important when you’re developing an industrial application, and you tend not to get these with early stage open source projects and third party solutions and third party services. You just don’t have a lot of control and it was too likely to my mind that we would just be stuck in the cold at some point where system didn’t work or it didn’t have some capability that we wanted, and then you’re up a dead end road, and so what do you do? Whereas this is a very small, simple system. You could print out the entirety of the whole system it would be probably a few pages, well it’s a few 1000 lines of code, it’s not a lot of code, and it’s across it’s a couple code bases, and so we can load the whole thing into our head and therefore understand it and make changes as needed to advance the business. 00:38:38 - Speaker 3: Yeah, I think that last point might honestly be the most important, at least for me. I think having a very simple mental model of what is happening in sync makes things so much easier to reason about. It makes fixing bugs so much easier. It makes preventing bugs so much easier. We’ve been talking about how sync is hard and how almost nobody gets it right, and that’s because it’s complicated. There’s a bajillion little bitty edge cases of if this happens, but then this happens after this happens, and then this happens. What do we do? And so making something really really simple conceptually, I think was really important for the muse sync stability and performance at the end of the day. 00:39:21 - Speaker 2: I’m an old school web developer, so when I think of clients and servers, I think of rest APIs, and you maybe make kind of a version API spec, and then the back end developer writes the endpoint to be called to and the front end developer figures out how to call that with the right parameters and what to do with the response. What’s the diff between a world that looks like that and how the new sync service is implemented? 00:39:50 - Speaker 1: Yeah, well, a couple things. At the network layer, it’s not wildly different. We do use protocol buffers and binary encoding, which by the way, I think would actually be the better thing for a lot of services to do, and I think services are increasingly moving in that direction, but that core model of you have, we call them endpoints. You construct messages that you send to the endpoint and the server responds with a response message. That basic model is pretty similar, even if it’s implemented in a way that’s designed to be more efficient, maintainable, and so on than a traditional rest server. But a big difference between A traditional rest application and the muse sync layer is that there are two completely separate layers, what we call the network layer and the app layer. So the network layer is responsible for shuffling these binary blobs around the transactional data, the ephemeral data, and the big binary assets. And the server knows absolutely nothing about what’s inside of them by design, both because we don’t want to have to reimplement all of the muse logic about boards and cards or whatever in the server, and also because we anticipate eventually end to end encrypting this, and at that point, of course, the server can’t know anything about it, it’s not gonna be possible. So that’s the networking layer and then if you sort of unwrap that you get the application layer, and that’s the layer that knows about boards and cards and edits and so on. And so it is different, and I would say it’s a challenge to think about these two different layers. There’s actually some additional pivots that go on in between them, versus the traditional model of you would like post V1 slash boards directly and you’d put the parameters of the boards and then the surfer would write that to the boards table and the database. There’s a few different layers that happen with this system. 00:41:30 - Speaker 2: So if we want to add a new card type, for example, or add a new piece of data to an existing card, that’s purely in the application layer on the back end, or it doesn’t know anything about that or no changes are needed on the back end. 00:41:44 - Speaker 1: Yeah, no changes are needed. In fact, one of the things I’m most proud about with this project is we basically haven’t changed the server since last year, December, and we’ve been, you know, rigorously iterating on the app, you know, adding features, changing features, improving a bunch of stuff, and the servers, it’s basically the same thing that was up 4 months ago, just chunking along, and that’s a benefit. It’s a huge benefit, I think, of this model of separating out the application model and the network model, because the network model is eventually gonna move very slowly. You basically figure that out once and I can run forever. And the application model has more churn, but then when you need to make those changes, you only need to make them in the client or the clients that maybe you update the application schema so that current and future clients can understand that, and then you just start including those data in the bag of edits. 00:42:26 - Speaker 3: Yeah, I think one thing that’s really nice is that those protocol buffers that you were talking about are type safe and kind of statically defined, so that way it’s when we’re sending that message over the wire, we know 100% we’re sending exactly the correct messages no matter what, and that guarantee is made at compile time, which I think is really nice because it means that a lot of bugs that could otherwise easily sneak in if we’re using kind of a generic JSON framework, we’re gonna find out about when we hit the please build muse button. Instead of the I’m running views and I randomly hit a bug button. And that kind of confidence early on in the build process has been really important for us as well to find and fix issues before they even arise. 00:43:11 - Speaker 1: Yeah, to my mind this is the correct way to build network clients. You have a schema and it generates typesa code in whatever language you want to use. There’s just enormous benefits to that approach. I think we’re seeing it with this on use and again, I think more systems, even more traditional B2B type systems are moving in this direction. By the way, everyone always made fun of Amazon’s API back in the day. I had this crazy XML thing where There’s a zillion endpoints. I actually think they were closer to the truth and the traditional, you know, nice rest crud stuff because their clients are all auto generate and sure enough they have like literally a zillion endpoints, but everything gets generated for free to a bunch of different languages. Anyways, one challenge that we do have with this approach is, you know, one does not simply write a schema when you have these multiple layers. So again, if you look at a traditional application, you have a protocol buffer definition of, say, a board B board and probuffs. And that would have fields like title and width and height or whatever. And when you want to update the board, you would populate a memory object and you would encode this to a protocol buffer and you would send this off to the server. Well, it’s not quite that simple for us because we have this model of the small registers that we call atoms. So an atom is the entity, say a board, the attributes say the title, the value say use podcast, and the time stamp. And your bag of edits is comprised of all these different atoms, but the problem is, how do you encode both how you’re gonna send an atom, which is as those twopos, as well as what a logical board is, you know, what the collection of atoms is meant to look like, you know, it’s gonna have a title and the width and height and so on. So that’s been honestly a pretty big challenge for us where it doesn’t fit into any of the standard schema definition approaches, certainly not the regular protocol buffer schema, which again we use for the network and for encoding the messages that are wrapped up in the network, but you need a separate layer that encodes the application model, as we call it, you know, what is a board, what is a card, what attributes that they have and so on. 00:45:06 - Speaker 2: And Wulf, if I recall you have a blog post about these atomic attributes. I’ll link that in the show notes for folks. 00:45:14 - Speaker 3: Yeah, so unfortunately no relation between my name and Adam. It’s a TOM. 00:45:18 - Speaker 2: Yes, we have two Adams on this podcast. The ADAM is different from the ATOM. 00:45:25 - Speaker 1: Yeah. A big inspiration on this, by the way, is Tomic, I don’t know if we’ve mentioned that yet on this podcast, but Atomic is a database system developed by Rich Hickey and team who is also the creator of Closure. And it uses this model in contrast to the traditional relational model you have tables and columns and rows. The atomic model is more like a bag of time stamped attributes where you have an entity, an attribute, a value and a time stamp. And from that, it could be more challenging to work with that model, but it’s infinitely flexible. You can sort of put whatever model you want on top of that, and it works well for creating a generic database system. You know, you couldn’t have a generic post graphs, for example, that could run any application. You need to first create tables that correspond to the actual application you’re trying to build, whereas with an atom oriented database, you basically have one huge table which is atoms. So it’s useful again for having this slower moving more stable synchronization layer that handles moving data around that you build the application on top of that moving quickly. 00:46:27 - Speaker 3: Yeah, and like we talked about earlier, it’s so much simpler to reason about. All of the problems of my iPad is on version 1, my Mac is on version 2, and my iPad Mini is on version 3. They’re sending data back and forth. At the end of the day, every single database on all three of those clients is gonna look the same, even though they have completely different logic, maybe different features. But all the simplicity of that data store makes it much, much easier to reason about as the application gets upgraded or as two different versions of the client are synchronizing back and forth. 00:47:03 - Speaker 2: How does that work in practice? So I can certainly imagine something where all of the data is sent to every client, but a V1 client just doesn’t know what to do with this new field, so just quietly stores it and doesn’t worry about it. But in practice, what happens if I do have pretty divergent versions between several different clients? 00:47:23 - Speaker 1: Recall some podcasts ago, we suggested that everything you emit should have a UU ID and a version. Well sure enough that’s advice that we take to heart with this design, where all the entities, all the messages, everything has a UU ID and also everything’s version, so there’s several layers of versioning. There’s the network protocol is versioned and the application schema is versioned. So by being sure to thread those versions around everywhere, the application can then make decisions about what it’s gonna do and Wulf can speak to what the application actually chooses to do here. 00:47:54 - Speaker 3: Yeah, exactly. If we’re sending maybe a new version of a piece of data on the network layer that device A just doesn’t physically know how to decode from that work, then it’ll just save it off to the side until it eventually upgrades and then it’ll actually read it once it knows what that version is. 00:48:11 - Speaker 2: So is there someone like, can I make a crude metaphor here, someone emails me a version of a Word doc from a later version that I don’t have yet, I can save that on my hard drive, and later on when I get the new version, I’ll be able to open the file. 00:48:25 - Speaker 3: Yeah, exactly right. It’s very similar to that. And then I think there’s a different kind of upgrade where we’re actually talking the same language, but I don’t know what one of the words is that you said. So let’s say we add a completely new content type to muse called the coffee cup, and everyone can put coffee cups on their boards, right? That coffee cup is gonna have a new type ID attribute that kind of labels it as such. New clients are gonna know what type 75 means coffee cup, and old clients are gonna look at type 75 and say, oh, I don’t know about Type 75, so I’ll just ignore it. And so the data itself is transferred over the network schema and kind of the app schema and understands those versions, but it might not understand the physical data that arrives in the value of that atom. And in that case, it can happily ignore it and will eventually understand what it means once the client upgrades. And so there’s a number of different kind of safety layers where we version something. If we’re unable to even understand kind of the language that’s being spoken, it’ll be saved off to the side. If we do understand the language that’s spoken, but we don’t understand the word, we can just kind of safely ignore it, and then once we are upgraded, we can safely understand both the language and the word. 00:49:47 - Speaker 1: Yeah, so maybe to recap our discussion of the low level synchronization protocol before we go on to the developer experience and user experience, might be useful to walk through a sort of example. So suppose you are doing a thinking session in your nice comfy chair on your iPad, you’re offline, you’re making a few dozen edits to a series of different boards and cards in your corpus. Those are going to write. New atoms in your database, and those are essentially gonna be flagged as not yet synchronized, and then when you go online, those atoms, assuming it’s some plausible number, you know, it’s maybe less than 1000 or so. Those are all gonna be combined into a single network message. So this is that multiplexing efficiency where you certainly don’t need to check every document in your corpus, and you don’t even need to do one network right per every edit or even one network right per document. You can just bundle up all of your recent changes into a single protocol buffer message and could potentially compress it all with GSIP, and then you send that out to the server. The server doesn’t know anything about these edits, you know it’s just one big binary packet. The server persists that, and then it sends a response message back to the client and says, OK, I’ve successfully received this. You can now treat this as synchronized and the server will take responsibility for broadcasting it out to all the clients. And then clients as they come online, if they’re not already online, they will immediately receive this new packet of data called a pack. And then they can decompress and unpack that into its constituent atoms, and once they’ve processed those, tell the server, I have successfully saved this on my device and in the background of the server is essentially maintaining a high watermark of for each device that’s registered for this user, what’s the latest pack or block they successfully persisted, and that way as devices come on and offline, the server knows. What new data needs to send to each individual device, and that works both for essentially conveying these updates in near real time as they happen, as well as for doing big bulk downloads if a device has been offline for a long time. And I know we’ve mentioned a few times, but to my mind this multiplexing and batching and compression is so important, so it’s the only thing that makes this even remotely feasible with the Muse data model of having a huge number of objects. And then I think this leads pretty naturally to a discussion of the developer experience. So we’ve talked about this sort of sync framework, and that essentially is gonna present a developer interface up to the application developer. So Wulf, maybe you can speak a little bit to that. 00:52:19 - Speaker 3: Yeah, we’ve talked some about the simplicity that we’re aiming for, just conceptually and how synchronization works. I think it’s equally important for this to be extremely simple for the end developer to use as we’re building new features in use or as we’re, you know, changing the user interface around. That developer, whether it’s me or Julia or anybody else working on Muse, doesn’t need to be able to think in terms of sync at all. We just need to be able to write the application is the ideal world, needs to be very, very simple. And so keeping that developer experience simple was a really big piece of designing what sync looks like inside of Swift for iOS. Since we had been built on core data beforehand, a lot of that developer interaction ends up looking extremely similar to core data. And so we build our models in Swift, it’s a Swift class. We have all of the different attributes where there’s position, and size, and related document, and things like that, and we just stick what’s called in Swift a property wrapper, it’s just a small little attribute. In front of that property that says, oh by the way, this thing, this thing belongs in the sync database. This property is gonna be merged, and that one little piece of code, that one little kind of word in the code program is what fires up the sync database and the sync engine behind it to make all of this stuff work. And that has been really important both for conceptually building new features, but also for migrating from core data to sync. Because the code that core data looks like beforehand, and the code that sync looks like now, is actually extremely similar. Early on in the development process, our very first kind of internal beta, internal alpha, pre-alpha, whatever version you want to call it. Very early on in the process, we actually ran both core data and the sync engine side by side. So some of the data in Muse would load from core data and other bits and pieces would load from sync, but both of those, because they looked very, very similar from the developer’s perspective, from kind of how we use both of those frameworks. It allowed us to actually slowly migrate use over from one to the other, by replacing bits of core data with bits of sync, and then this little bit of core data with this little bit of sync. I mean there’s, you know, thousands and 10s of thousands of lines of custom logic to make muse muse. And so it was really important to keep all of that logic running, and to keep all that logic separate from the physical data that logic was manipulating. And so making those appear similar to the developer, let us do that. It let us keep all of that logic essentially unchanged in use while we kind of swap out that foundation from underneath it. 00:55:15 - Speaker 2: And I remember when you implemented the first pass at this persistence library, and I forget if Yuli was maybe away on holiday or maybe she was just working on another project, but then she came in to use your kind of first working version and start working on the sort of porting it across and she had a very positive reaction on the developer experience, you know, you are sort of developing the persistence layers so you naturally like it because. Here, maybe the way you like it and you’re thinking about the internals, she’s coming at it more from the perspective as a consumer of it or a client or a user, and the developer experience, and I found that to be promising because I mean, she, like most, I think iOS developers has spent many, many years in her career using core data. Which is a long developed and well thought through and very production ready persistence layer that has a lot of edge cases covered and is well documented and all that sort of thing. So in a way it’s a pretty high bar to come in and replace something like that and have someone just have a positive reaction to using it as a developer. 00:56:21 - Speaker 3: Yeah, I was so happy when she said that she kind of enjoyed using it and kind of understood how it worked, because of course every developer likes their own code, but when a developer can use and is comfortable with another developer’s code, that’s really, really important. And that was absolutely one of my goals is to make sure that it was simple for Julia to use and simple for any other developer that comes onto the Muse team that doesn’t have background in Muse and in our code base. For them to be able to jump in quickly and easily and understand what’s going on, was a really important piece of how this framework was built. 00:56:57 - Speaker 1: Yeah, I think this is a really important accomplishment, and Wulf is maybe even underselling himself a little bit, so I want to comment on a few things. One is, while there’s just this simple annotation of I think it’s at merged is that it Wulf. Yeah, that’s right. Sometimes when you see that, that annotation instructs the framework to do something additionally on the side, on top of the existing standard relational database, you know, like basically do your best, try to synchronize this data out of band with some third party service or whatever. But this in fact totally changes how the data is persisted and managed in the system, so it’s sort of like a whole new persistent stack for the application. And I think that’s important because we constantly see that the only way you get good results on sync system, especially when you’re talking about offline versus online and partially online, it has to be the one system that you use all the time. You can’t have some second path, that’s like the offline cache or offline mode that never works. It needs to be the one, you know, true data synchronization and persistence layer. So I think that’s as important though. There’s another subtle piece here, which is the constraints that you have with the industrial setup. So a lot of the research on CRDTs and synchronization basically assumes that you have one or a small number of documents in memory, and that if you’re going to be querying these documents or managing synchronization of these documents that you have access to the full data set in memory. But it’s not practical for our use case, both because of the total size of memory and the number of documents we’d be talking about. So a lot of critical operations can take place directly against the database or the data layer can smartly manage bringing stuff in out of memory. It’s not like we have the whole new corpus up in memory at any one time, the system has to smartly manage what gets pulled up from disk in the memory and what gets flushed back, and then that. Introduce a w

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: To borrow the Hobbit, one does not simply build a new sync layer. You start with a product and you want to build a sync layer, and now you have two products. You had a huge undertaking to build this kind of a system on the server and on the client and should not be a default answer, I think, for anyone, and it was not our default answer, but I think it worked out and was the correct decision for us in the end. 00:00:27 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac, but this podcast isn’t about Muse product, it’s about the small team and the big ideas behind it. I’m Adam Wiggins here today with my colleagues, Mark McGrenigan. Hey, Adam. And Adam Wulf. 00:00:44 - Speaker 1: Hey guys, it’s great to be back. 00:00:46 - Speaker 2: Now Wulf, you took a little staycation recently, and I understand you’re working on a little side hack project there called Developer Duck. Tell us about that. 00:00:56 - Speaker 1: Yeah, it was kind of nice because the vacation just happened to align with all of the chat GPT AI magic that’s been released lately. So I spent a fair bit of the vacation just working on a prototype for a developer tool that uses AI to help developers kind of build faster to take over some of the tedious tasks so the developer can work on the more meaningful tasks. So it’s been a really fun adventure. It’s got, you know, X code integration and it’ll edit your source files for you or add comments or explain code you don’t understand or fill out code and replace comments with code and it’s got a chat feature as well, just like chat GPT but it’s focused and prompted more specifically for developers, so it’s a bit less for both and a code highlighting and all sorts of stuff. So it was just a really fun. Yeah, adventure to kind of see what can these AIs do now and is the hype real or is it really just someone behind a curtain? 00:01:59 - Speaker 2: And I think the name there is a reference to the uh rubber duck from the pragmatic programmer, am I wrong? 00:02:07 - Speaker 1: No, that’s exactly right. It’s a really common metaphor for programmers to talk to a rubber duck and just verbalizing the problem out loud and kind of pushing it through the language center of our brain helps us clarify what it is that we’re actually looking for. And so this is that sort of a thing. You can chat with the robot, express your problems, it’s able to prompt and kind of guide you towards solutions sometimes. But it’s really a great first step before you go interrupting a coworker and pulling them out of their flow state. You can stay in your flow state and talk to the rubber duck and hopefully get a solution without much delay in either your day or your coworker’s day. 00:02:52 - Speaker 3: Yeah, I found rubber ducking to be unreasonably effective, and so I can only imagine with the supercharging of Jet GPT it’s even better. 00:03:00 - Speaker 1: Yeah, it’s actually been really fun because a lot of times I’ll know what I need to do, but I don’t know the right jargon word to search Stack Overflow or to search Google. And so sometimes even just that, you know, the 1st 5 or 10 minutes of searching around is just understanding what is this problem actually called? Like, what is it that I actually need to do? I know the problem I need to solve, but I don’t know what the name of the solution is. And so just typing the problem into something like developer duck, it actually prompts back with, oh, these are actually great bunch of keywords and ideas and even possible solutions that I should go try, and then it’s something I can go and dig deeper on on Stack Overflow or Google or whatnot to find the final answer. 00:03:44 - Speaker 2: So our topic today is local first, one year later. So this is a reference to or let’s call it a sequel to an episode, the 3 of us recorded about a year ago where we dive deep on the technical architecture from Muse’s sync system, and I’ll link that episode and show notes. Now, as a reminder, local first is this idea that we want to get the benefits of cloud software. Think of Google Docs, where you can share and collaborate really easily with other people. But also gives you the benefits of call the more traditional style of just saving a file to your hard drive, right? It’s always fast, it’s available offline, and then you just have more data ownership generally. And part of how we do this is using a technology that comes pretty recently out of the computer science world called CRDTs. So at the time we we recorded them, we were still in beta with this device syncing, we’ve launched Muse 2.0, which included that and lots of people are using that in production. And now we have our next iteration, which is Muse for Teams, which uses the same local first sync technology, but now for multiplayer that you can have many people on a team, each with multiple devices that are all in a single board or single workspace. So I guess the prompt for this episode then is what have we learned in the past year of running this system in production at scale? And maybe as a starting question, I’ll ask each of you, how does it compare to your experience working on call them traditional client server apps like a web app or an iOS app that calls out to a retrographQL API. 00:05:11 - Speaker 1: I think the biggest difference. Just using it day to day on the client. I how nice it is to just be able to work with data as if everything is 100% local on the device. The network has been almost entirely abstracted away, and so there’s no waiting for the rest API call. There’s no error handling if the API is down. There’s no HTTP error 404 verse 401 verse 500, like none of that exists, which is So pleasant to work with because you just, oh, I’m gonna load some data, and then I edit some data, and then I save it back and I’m done, and all of the network stuff is handled. At a lower level, completely abstracted away, which meant that building new features. Has been dramatically faster than when I build stuff with a traditional rest API or traditional server-based API. It’s just let us iterate. Much, much quicker than we could otherwise, I think. 00:06:17 - Speaker 2: That was a surprise to me, which was Julia reported, you know, she does more of the interface engineering, she’s essentially a client of the persistence layer you’ve created. So when we use that persistence layer to replace core data, which of course is so well established and has decades of development, it’s well hooked into all of the iOS interface APIs. She’s worked with it for a long time and I thought, OK, well our homegrown system is going to be Almost certainly less nice to work with, but she was really pleased with how easy it made everything and how few error states you need to deal with, and it just simplifies things because it exactly as you said, it feels like writing an old school program that just loads and saves stuff to the disk and hold things in memory, and just all of the complexities of distributed systems that you’re essentially forced to deal with one way or another through network errors and things like you mentioned. Aren’t a consideration in building your client side software. 00:07:15 - Speaker 1: Right, exactly. The core data has a number of just kind of programming niceties for how to annotate which data gets saved in the database and which data is just kind of ephemeral in memory. And The way that we architected this sink layer is Really modeled on a lot of how the developer interaction with core data works. It ends up being almost the exact same. Interaction at the code level with our sync layer as it is with core data, which is really nice because then it meant that we could just Switch some model files from core data annotated model files to now sync annotated data files, and it was almost a 1 to 1 mapping, so it was really easy to get in on board. It’s really easy. To take a lot of the experience that developers already have after years of working with core data, a lot of that mapped almost 1 to 1 with a new system. And so that minimizing developer learning. I think it’s been probably just as important, if not more important than Some of the technical details of how it physically works behind the scenes, making sure that that interface for the developer is easy and convenient. It’s let us work much faster than we could otherwise. 00:08:37 - Speaker 2: Mark, what have you learned in a year of running these systems? Particularly interested in things that maybe were surprising to you. 00:08:45 - Speaker 3: Well, the good news is it hasn’t been that surprising. And by that I mean, I think we’ve realized all the benefits that we expected and hoped for. It’s, you only need to update code in one place. It obviously doesn’t require the device to be online. All the data is there on the device when you need it, and so on. And also, we’ve been working through all the challenges that we expected to deal with versioning differently, performance becomes more sensitive, you’re downloading a lot of data, you know, all these things that we basically expected from our research in the lab. So, I think we’ve got a lot more texture around all those benefits and challenges, especially the challenges. But no, to me, real big surprises. One thing that has played out a little bit differently than I had hoped and expected was how content aware the server is. So the very earliest research prototypes we had, the server really had no idea what was going on. You created these very abstract like channels and mailboxes, and the server had no idea how that mapped to people who were collaborating or documents or anything like that. You basically just shuffled bits around totally the direction of clients. And the motivation for that was to keep the server as simple as possible and to retain the option of doing straightforward and and encryption. And we tried going down that road for a while, but it’s proven really tough. I think the two reasons are, one, it does put a very big burden on the clients to have to basically route all their own mail everywhere and not being able to rely on any server, for example, saying, you know, these people are working together on this document that’s where these packets go to these people. And also, of course, it does become challenging if the server can never know anything about the content. You know, if you want to send an email notification about an update, server needs to know about that unless you do something really wild. So that’s been a little bit different. It’s not a huge difference, but Something I wanted to mention. 00:10:34 - Speaker 2: Yeah, I think we’d hoped both from the perspective of call it general kind of privacy principles, but also from the perspective of the simplicity of the server that it could be a completely dumb pipe for the data and we talked about this in our episode with Martin Kleppman about even the idea that someday you might bring your own sync service, that you know, AWS and other providers offer you kind of a pipe or a sync service that you could plug into any application the same way that a file system could plug into any application. And I still like that idea, but it does restrict things you can do server side. You mentioned the emails notifications is another one like, I want to get notified when someone replies to my comment thread when none of my apps are logged in, but like there needs to be some system somewhere on the back end that can make some kind of basic interpretation of the data in order to offer features like that. 00:11:26 - Speaker 1: I think one interesting thing that I’ve seen from kind of the outside of the server. Is that a lot of those abstractions are still true. It is in still some ways a very simple pipe that just routes data back and forth. Just now it’s kind of a pipe with a window, so to speak, so you can kind of look inside the pipe on the server and look at things that you really need to look for. But the other thing that, as I understand it, has been challenging is a traditional server architecture will have The state of the world inside of its database, done. So you just kind of query the state of the world and you get the answer back and you know exactly who’s talking about what. But because we’re working primarily on local data, It means that the server doesn’t necessarily have one single view of the world, one single, what’s the latest state of the world for any particular client or team, but the server needs to actually kind of Go back and read all of the logs from all of the different devices recently to kind of recreate the state of the world and determine what’s changed. Mark, maybe you can talk a little bit about that and just kind of what’s been different. And how it’s architected versus kind of a traditional rest API architected to do things even as simple as notifications. 00:12:46 - Speaker 3: Yeah, OK, so there’s two things going on here. There’s, do you have an event-based or a state-based view of the world, and is it a push versus pull? So in a traditional database you have a state-based view of the world and it’s pull. So for example, you have an object that represents a cart in a database and what’s in the database is basically just the current value of the cart. 00:13:13 - Speaker 2: And then when a client needs a car, it will ask the database for this card snapshot, and the surfer will reply, select star from shopping carts where card ID equals whatever and you get back a bunch of products at the end, right. 00:13:19 - Speaker 3: So the event-based variant of that would be instead of a cart, you have a log of cart changes. Add this item, remove this item, change this quantity, and if you roll up all those changes over time, you can build a view of the current cart. You don’t necessarily persist that. And so we use this event-based approach. I found that to be fine. It does take a little bit of work and as we’ll talk about in the client, it does become more challenging if you want to do queries like select all cart where it contains pencils or whatever, that becomes more challenging. That’s not too bad in my mind. Then there’s also this push versus pull thing. So it’s I think relatively easy on standard servers when you get a pull request, it says, give me the cart with this ID while you go to your index on disk, you crawl a bee tree, and there you are, you’re at a cart with an ID, whereas in our model, what happens is you receive a cart change event. And then you need to figure out all the devices in the world that are potentially interested in cart change events for this cart and send each of them that update. And by the way, they might not be logged in right now, so you gotta do it, you know, now versus later and keep track of who’s received it and when and stuff like that. So that that part has been challenging, you know, it’s basically it’s an engineering problem, it’s not insurmountable, I don’t think, but I do think it’s harder than traditional poll style database. 00:14:41 - Speaker 2: Yeah, perhaps partially harder just because it’s less well precedented. We’re operating on a model that is, I think in the long run is and could be superior, we’ve already seen benefits from that, but we don’t have the same tooling or knowledge about it that we do on these more classic state-based systems. I’ll give a little anecdote where sort of there’s a let’s call it a user benefit or as a person on the team who’s not part of the engineering and but I am testing internal builds and things like this. I’ve been very impressed by how good the data integrity is, and I think we talked about this in our first episode because essentially you’ve got the servers just shipping around these bundles of data. That it doesn’t have any insight into what’s in them, but even on the client, there seems to be a network layer. Wulf, you can probably correct me if I’m interpreting this wrong, but it seems like it receives those bundles and then once it saves them, then it figures out what to do with them. And in some cases if you have a new app schema, so that is to say we’ve added a new data type or something or changed the data type, the newer clients can interpret that, but actually if you’re on an older client. It doesn’t know how to interpret it, but it still leaves the data laying there and so you may be able to interpret it in a future client. And so we’ve had situations, obviously we’ve had internal builds for testing, whatever scheme has changed, etc. etc. and so you might have something where you’re on the wrong version of the client and you can’t see some of the data because it can’t interpret it, but it’s all there. It’s receiving it through the network even if people are in real time doing edits and your client is receiving them, it doesn’t know what to do with them, but it just sort of leaves them sitting in the local data store and then when Get the new version of the client, now you can see all the new stuff and that’s basically resulted in any time there’s been data problems, it’s always transient, it isn’t like people being on different client versions results in some kind of data loss. It’s just sort of confusion and how it’s viewed. And that actually it is really nice from the perspective of me having more trust in, OK, the data is always there, it’s just how it’s interpreted, how it’s materialized, is going to depend on sort of the client version and the current code that’s showing it to me. 00:16:48 - Speaker 1: Yeah, that’s exactly right. I think one of the really important things we did really from day one was clarify how the client talks to the server and make sure that that what we call the network schema was well versioned so we had a very clear definition of kind of how we ship boxes to and from the client. And then we also have the app schema version, which is how do we version the data that’s inside of the boxes. And so what that lets us do is change and upgrade how the app understands the data model, regardless of how that data is sent to and from the server. And similarly, we’re able to upgrade and make more efficient how the data is sent to and from the server, regardless of the type of data that the app is sending. And so having a very clear definition about how the data is shaped in the application versus how the data is transmitted has been very helpful to make sure that everything that arrives on the client is in a very well known state, and if the app doesn’t currently know how to interpret that state, that’s OK. It just saves on disk. Next time the app updates it checks again, oh, I’m a new app. Look, I have some data here that I didn’t know how to look at last time. Let me try again. Oh yeah, now I understand it. OK, great, and it can continue on just fine, and that lets old clients on maybe a V9 or V10 of the app schema, continue to operate just fine cause they’re talking in The version 10 of their language, and whenever a client logs in with a version 11 of the A schema language, It can still understand all of the V10 things, and it starts talking in V11 language, and all of those V10 devices that are still on the network, hear that and go, I don’t understand this yet. Let me just save it. And then once they upgrade, then everyone’s talking the same language again and everything works fine. So it lets each device speak in the way that is comfortable to the other devices, interpret in the way that is comfortable from the other devices, and when it gets confused, that’s OK, it can just wait, and having those three states. Has meant that it’s been important as we’ve planned for new features to make sure that we’re planning with regards to those three states, that we’re planning with regards for backwards compatibility, and potential future compatibility, and What happens when two devices are talking with two slightly different languages and making sure that the correct updates are still visible while then fancy new updates, it’s OK for them to be invisible, and to make sure that the application still behaves in a correct way. 00:19:30 - Speaker 3: Yeah, and looking back, we’ve had a lot of dynamism on the so-called app schema. That’s what’s in the boxes, the boards and the ink and the text blocks and all that stuff. The network model of binary data plus transactional data plus ephemeral data, which I argued you could like intuit from first principles cause all applications look like that. That’s proven correct, like that basically hasn’t changed. Some small stuff around the edges to make it a little bit more efficient and support collaboration. But we’re still shipping stuff around in the same boxes we used to, more or less, even while the the application has changed pretty dramatically inside the boxes. 00:20:07 - Speaker 2: Maybe the comparison there is wire protocol to, yeah, database schema, but I think it is the case in the kind of cloud style client serverE applications that I’m familiar with and I spent a good bit of my career working with in the last couple of decades where, OK, we’re gonna add a way to turn things red, so therefore we need a new field called color that goes in our back end database which is SQL. The back end code needs to handle that. We need to add it to the API, which might be a crowd API, might be graphQL or something like that, and the client needs to interpret that. It’s going to display that in the interface, and the reason that we get that dynamism, as you said, Mark, is the client can essentially update its app schema, add that thing in there, but know that it will be transmitted through without needing the back end to change, know anything about it. And so you essentially only need to really change it in one place, which is the client side code. Now I think it does create a locus of complexity on the client for us, whereas maybe those client server applications, standard cloud applications we’re familiar with, to end up with more balanced complexity between the back end and the front end, but the not needing to coordinate. You know, OK, the back end engineer is going to design the API and the front end engineer is gonna build the interface and needs to consume that and all that back and forth for every little thing we wanna add. I think that’s a real boon to development speed. 00:21:33 - Speaker 1: Yeah, I do too. I think that’s been so important for us to be able to completely abstract away how data is transmitted to and from the server versus what that data means. And the complexity is almost all entirely on the client, instead of being shared between the client and the server, but I don’t think we’ve ended up with double complexity on the client. I think we have essentially halved the complexity of any app schema change, and so the client complexity ends up being kind of the same amount as it would have been with a rest server, but now we’ve could just completely removed a lot of the server complexity, and so the The actual amount of work, the actual amount of difference. Ends up being about half, in my view, compared to a more traditionalru-based classic server architecture. 00:22:28 - Speaker 3: Yeah, I think that’s right. And I think there’s a little bit of additional complexity on our client versus the typical one, but that’s due to the client being stateful versus stateless. And if you had a stateful arrest client, it would have the same issues with migrations and so on, which we’ll talk about. 00:22:43 - Speaker 2: Let’s talk about the operational side of it, so. I’ve spent a good bit of my career, say, carrying a pager, although mostly that’s metaphorically, it’s notifications to my phone, but you know, you write code, especially in the early days of a startup, you don’t have a dedicated ops team and you set yourself up to receive notifications for your monitoring systems or your pingdom or whatever it is, that’s kind of just tracking whether the systems you’ve built are online. You also have things like migrations where you need to, OK, we’re going to do it. I don’t know, let’s do it at 10 p.m. at night because we know the system needs to be offline for 10 minutes while we do this thing and it’s kind of stressful, and we don’t want to stay up too late cause we’ll be tired, but we also want to do it at night when volume is low. And so one thing I’ve observed just kind of again from the On the inside, but outside the engineering team perspective that I have is, first of all, just the operational cost in general has been lower, that we spend less time on that. We have pager systems and what have you, but they don’t just seem to occupy as much of our teams. Mental and energy bandwidth. But the other thing is, I think we’ve had to do one or two downtime just for migrations. There was one recently and yeah, I remember it was longish, I think it was like 20 minutes, but when the only impact is your little filled-in circle turns to empty and Your devices don’t sync for a little bit. Maybe that’s annoying if you’re in the middle of something that you wanted to use between two devices or a collaborative session, but you’re not stopped from working, you can access everything. It isn’t the whole O is slack down as notion down as GitHub down, my work is completely interrupted. You can really continue as is, it’s much more of a low-key event and both of those sides of it, like the team needing to spend less time and energy and the impact of customers for downtime. Being a little bit less. Both of those seem like dramatic wins to me, but I’m curious how you both see it. 00:24:36 - Speaker 1: Yeah, from my side, I think that’s been one of the really nice things is that the downtime that we do have has almost no client impact at all. Certainly from the code perspective, because I can just save and read from the disk and the network is entirely abstracted away. There are not any new error handlers that I need to cover. There’s no alerts or Error states with the data that need to be managed. I know that everything will eventually sync and everything will eventually come back down, and the state will be shared between all devices. And so a lot of that complexity is just kind of gone from the client, which is so nice. And like you said, the users can still work, they still get, you know, certainly more than 80% of the the benefits of used by having all of their data locally. So downtime is While still rare from use, very rare, it’s certainly not an impact for users when it does happen. 00:25:36 - Speaker 3: Yeah, the server downtime is less common than it would be otherwise. It’s less impactful on users and it’s much less stressful for the operators. Which I think is actually quite valuable when you’re a very small team, you can’t actually staff a full standard pager rotation with a team of the size, so you either need to let that slip or have more flexibility like we do with the local first set up. And I would emphasize something Wulf mentioned, which is In our client architectures, the offline case is the standard case. The way the client works is it basically saves data as if it was offline, and if it happens to be online, it goes and like does another step, which is send it to the server. And I’ve long been an advocate of this idea that if you don’t treat the error cases as a standard case that happens all the time, they’re just not gonna work. And we have the flip of that here where it’s the thing we’re doing all the time, so it works fine when the server goes out. 00:26:28 - Speaker 2: I like how again another reference to our episode with Martin Kleppman, he phrased it as, you always have latency, and that latency might be 50 milliseconds. If you’re lucky, it might be. second or a second if you’re further away, your network’s slow or something like that, but it might be 3 hours because that one of the systems in the way is offline. And in a way you can treat those as all classes of the same problem, which is you just need to queue up what you want to send and send it and merge it in when things are back online again. That sounds so simple in concept, but in practice, when you have systems that are built around the expectation of things being online and needing a central server to resolve conflicts. Then you end up special casing all these different things, but if you treat it as classes of the same problems, which we do by assuming you’re offline and then saving it to a place that it can later be streamed, and of course in a format that it can be merged together, that’s a really Key element that’s obviously the CRDT technology, but it’s also to some extent this separation of the app schema from the data packets that are either on disk or in transit as something that can be completely disconnected from the application function transmitted at any time. Well, maybe it would be a good time now to get into what we’re working on now and have been working on for the last 6 months or so, which is multi-user. So when we started on this path, we knew that we wanted the single player app, then it syncs to your devices and then it’s multi-user, and that indeed the same technology that syncs between your devices in this local first manner kind of offline as the default case could also be used for multi-user to get that either in real time that Google Docs figMA type experience, but also something more asynchronous and Indeed, that is what we have working now. So what was some of the big changes or what was some of the big architecture decisions to make going from, OK, we can sync reliably between a person’s Mac and iPad. Now we want to have several people connected to the same board or workspace with their Macs or iPads and have all that fit together. What did that transition look like? 00:28:52 - Speaker 3: Yeah, so we retained our same basic model, but had to add one layer of indirection. So recall in the single user case, the model was a user’s muse consists of basically all the edits they’ve ever done in one big log. And if you have a copy of that log, say on a different device, you can replay it and get the state of that news corpus on that other device and therefore, when one device writes an update, the update needs to be conveyed to the other device and so on. And when a new device comes online, it gets a full copy of just that log. With the multi-user case, basically have two logs now, every user still has a log, which is like all the edits that they’re supposed to see, you know, all that’s dated, of course, but also at the edits that people who are on documents that they’re collaborated on made and each, let’s call it document for now, each document has a log that consists of all the edits ever made in, say, a board. And now the job is you need to maintain the logs for each of these users by like stitching together all of the logs of the constituent documents they’re collaborate on. So that’s the layer of the direction. So you still have the same core model of you have immutable edits, you have logs, the users have a log, the devices catch up by maintaining a high water mark in the user’s log. That’s all the same, but then the addition is you now have this additional layer of abstraction with the documents and then threading those to all the constituent users. 00:30:23 - Speaker 1: The way that I’ve thought about it, having not worked on the server code, I’m curious, Mark, how much this is accurate or is a fairy tale. But if there’s 1000 users and they each have 1000 records, then when a user talks to the server kind of in the old world, it was relatively easy for the server to say, oh yeah, you’re user A, great, you’re definitely within these 1000 records, no big deal. Let’s figure out which of these 1000 records you need to talk to. But now because any particular user can be on a team with any other particular user, if you have 1000 users and 1000 records and a user logs in, instead of looking at just 1000 records, now you have to look at all 1 million records because you don’t know which other user might have written something to that person. So it’s really a huge kind of order of magnitude harder problem to solve now in many ways just because of that one extra layer of indirection that we ended up adding. 00:31:18 - Speaker 3: Yeah, the engineering is quite a bit harder on the server now because as you said, it used to be that you had N or N is equal to the number of users, totally isolated islands of data, you know, all the users' data, their devices, the high water marks, everything could be siloed off. Whereas now, if you think about these entries in a database are basically overlapping islands because you have, you know, a document is shared by many users, but then those users, some of them are also on other documents together, but those aren’t exactly the same. And furthermore, we’ve wanted to keep it general so that we can support future product changes, so we could have made it a little bit easier on ourselves by saying that there are K islands where K is the number of teams and used for teams, but then we’d have to go solve this problem again when we went to the fully general case so you can collaborate across teams. So yeah, it is pretty challenging. One other example I’ll give here to add a bit of texture. So it used to be that the only types of updates in the system were edits, like when you add a text or whatever, but now we have updates around the configuration of teams and permissions, so you can add a member to a team and you can add a team to a document and so on. So it used to be the kind of hardest thing for the server to do would be to get a new device online cause then that device needs to be caught up for all the Entries ever written for that user, but at least it’s already in one log, it’s not too bad. Now the hardest case is you have a new user added to a team. So for that, the server needs to say, OK, who are all the users in this team, and what are all the documents that have ever been added to the team and what are all the updates in any of those documents and then zip together a huge log and send it over to that new user to be added into their existing huge log. So it’s just an example of how it can become quite challenging even though there’s just one additional layer of indirection. 00:33:04 - Speaker 2: The one thing I’m really interested to learn more about is what we usually call the unit of sharing. Some of the things we’ve explored on the product side include things like sharing an individual board the same way that you can with a lot of products like a notion or craft or something where there’s a big share button and you can share kind of just that one document, something like a complete team workspace, which is the main thing we’re supporting right now. But ultimately, as you said, there’s those islands, those overlapping islands that we need to think about it. I’ve heard terms fly around like scope and scope set, and I haven’t fully followed all that. So maybe you guys can be up to date, not just on how we think about the unit of sharing, but maybe what were some of the things we tried that didn’t work or how do we land on the architecture that we have now, I guess is what I’m curious to know. 00:33:48 - Speaker 3: Yeah, so the thinking with designing the sharing protocol was we wanted to keep all appropriate options open for the product, because this was before we even had sharing at all, you know, much less the variations of sharing that we’ve thought about, including sharing individual boards, whole team spaces, subspaces, boards, but their children as well, right? There’s all kinds of variants possible. So the idea was to keep that option open and we did that by creating this unit called a scope, and a scope is the smallest possible unit of sharing. It’s sort of like a physical analogy might be a book. And let’s do something crazy like go and cut the book out, you kind of got to give someone the book or not, but you can also choose to form those in the libraries and give people access to the whole library, right? So it’s not like the only thing you can give someone access to is one book at a time, there’s there’s ways to aggregate those things, but that’s sort of like the smallest unit that you would plausibly lend out. And we specifically designed them and named them such that they weren’t any of the things in our system. They’re not boards, they’re not teams, they’re not nested boards, they’re not spaces. These are all things that might be defined as a set of scopes. But that’s not baked in. And then to emphasize when you’re sharing a scope actually in the system it’s what we call scope set. So you basically bundle up a set of scopes, which of course in the degenerate case can be one scope, but in full generality it could be any number of scopes up to and including the whole team corpus. And furthermore, the system does allow those currently to overlap, so you could share a team space with some set of people and then share a subset of that with a larger group of people, for example. So basically the idea was to allow this full generality. And so the key decision that we had to make at the beginning, which would be hard to reverse, as we’ll talk about, was what is the smallest unit that we might plausibly want to share in the future. That is, this is the unit that’s kind of all or nothing when I share it with another person, and what we decided was the board. So you’re not gonna be able to share, or we didn’t foresee the need to share like a subset of a board. That seems plausible to be all or nothing, but we didn’t want it to be bigger than that, so we wanted to retain the option of being able to share just a board, even if we didn’t exercise that option in the short term or even ever. And I think that decision has proved pretty much correct. We have one little nugget in there with text blocks which Wulf can talk about. But I felt pretty good about that decision. I should say that there’s a bit of a downside if you allow for all this generality and then never exercise it. So say you had this scope system and you can share it down to a board, then the only thing you ever did was allow teams to share their whole workspace. Well, then you’re basically doing a lot of this accounting with scope sets and scopes and, you know, it’s a lot of records to deal with for never taking advantage of it. So you would hope that you would eventually get to using most of that granularity, but you don’t have to. 00:36:33 - Speaker 1: As you alluded to, I think the text blocks are an interesting case, and they are an interesting case only because of the history of the Muse database and the fact that we came from core data. 00:36:45 - Speaker 2: Maybe I’ll just interject here quickly and mention the text blocks are basically double click anywhere on the canvas and you can just start typing, including a copy paste, like a long form article. I’ve written a lot of longish articles this way with the concept of blocks is something that appears in Notion, Rome, others where you can kind of freely reorder them, drag them around. So in a sense it is sort of like an exploded long form text note, but it’s a really important data type for the use canvas. 00:37:13 - Speaker 3: Right, and furthermore, because they tend to be like lines or small paragraphs, you can have a lot on a board, you know, dozens, hundreds on a board. 00:37:20 - Speaker 1: Yeah, exactly. I think having scopes as the unit of sharing and translating that to boards has made a lot of sense for us because it’s given us enough flexibility while not being kind of too strict in what we need to do. The way that we initialize those scopes is for boards, it’s very obvious it’s a board, but we also have lots of other data types. We have images, we have files, we have PDFs, we have note cards, we have text blocks which are kind of paragraphs of longer form text. So there’s lots of different content types, and each one of those content types is a scope. And for historical reasons, back in the old core data database world presync, each of those types inherited from kind of its foundational document type. And so when we migrate it to sync. The most natural connection between core data and this new scoped sync world was to say, OK, every single document in core data now gets its own scope in the sync world, which works great because the scope and a document are essentially the same thing. A document was aboard, now a scope is a board. A document was a PDF. Now a scope is a PDF. But where that fell down for us was that text blocks, each paragraph, each kind of sentence on a board. was technically its own document in core data. And so then that meant we had this explosion of scopes in the sync world where a single board with a fair bit of text on it might suddenly be hundreds of scopes that we would need to synchronize. 00:39:00 - Speaker 2: And I assume that each scope comes with a certain amount of bookkeeping because of its ability to be a shareable unit. 00:39:08 - Speaker 3: Right, you gotta say this little bullet point needs to go into a set that can be shared with the users, this little bullet point, and so on. 00:39:15 - Speaker 2: Right, and really there’s no world where you’d want to be able to share these 3 text blocks, but not the 3 that are under it, you know, or share them with different people there from a user perspective, we know that those are going to be all or nothing. 00:39:28 - Speaker 3: Yeah, and furthermore, unlike big binary blobs like movies, there’s also no incentive to try to avoid copying, so you might want to put a blob in a separate scope, so if it moves to a different board, it can be changed by pointer only, whereas with the text block if it’s 10 characters, you might as well just, you know, rewrite it under the new scope. 00:39:49 - Speaker 2: OK, if I pick up a heavy video and move it to another board, including one that’s even shared with other people, that doesn’t require reuploading the video or something like that because it’s referencing that same blob, whereas if I copy a paragraph of text that just under this new system where a text block is not a scope, we might as well just bring all the text across. It’s fine that it copies it because it’s just a trivial amount of data. 00:40:15 - Speaker 1: Yeah, exactly, and importantly, all of that bookkeeping for text box being scopes needed to happen on both the client and the server, and so that explosion of scopes kind of Add a lot of extra headache for the entire development team on kind of every aspect of the Sinclair. And so that’s what I would consider a major migration of the app schema is unscoping all of these text blocks, so that way they become just very simple kind of attributes or many objects inside of the board, instead of an entire document and all of the kind of the heavy weight that goes with it, that it was before. 00:40:56 - Speaker 2: Well, speaking of unit of sharing, I’d also like to hear about, let’s call it edge cases. So in this local first world and this almost offline by default point of view, and Mark you described this event-based model and alluded to maybe that sharing, adding or removing someone to a unit of sharing a scope, I assume is itself an event. What actually happens or how do we handle, I’m offline and I remove someone from the team, but in the meantime, they’re online and they’ve been making changes. How do those things that are not really data edits, but are really more permission changes get handled in this world. 00:41:36 - Speaker 3: Yeah, there are quite a few interesting edge cases, and generally the way that we handle it is that the server decides the sequence of events. So unlike content edits, where we use a vector clock to allow clients to resolve them without the server needing to make any decisions, because semantically it’s kind of fine for people to be concurrently editing things. With respect to permissions, there really needs to be one order that everyone agrees on, and the only place that that can happen is a server. So when events come into the server, not when they’re written on the client, when they come into the server, they’re sequenced monotonically and the server can use that to decide who can do what. If you think really carefully about it, you can get some weird behaviors out of that, I guess, but like with our experience on distributed handling of content edits, it’s just not a practical problem for us right now. As far as I know, we haven’t had any of these weird edge cases in practice. The real problems that we have are much more mundane. But it is there. 00:42:36 - Speaker 2: There are other things we’ve learned from, obviously we’re not running the multi-user stuff at anywhere near the scale of the multi-device stuff that’s, you know, live in a production product in the App Store, but still, we’ve had a good number of users and edits come through the system over the last few months that it’s been live, including our own team’s use, which is pretty heavy. What have we learned from that? Have there been surprises? Have there been things we’ve wanted to modify as part of, you know, working on this alpha slash beta software? 00:43:06 - Speaker 3: I’ll give you one little use case, it’s been a challenge in multi-user. This is what I call the link click experience. So it’s very common with SAS that you’re working with a team on like a weekly planning doc, and the team leader in the morning goes and prepares the document, and then they paste a link, a URL to it in Slack, and then everyone goes and clicks on the link. So I really didn’t like the behavior of this in traditional SAS apps where what happens is when you’re a team member clicking on the link, what you see is a spinner for 3 seconds. I was incredibly annoyed by that. We’re gonna fix that by doing local first. Well, sort of. So now there’s a potential problem where you click on the link, it opens up your muse app and it downloads the whole weekend of updates from the entire team, right? Or the whole day of work from people in Europe, if you’re in the states. And you click the link, you expect like a normal SAA for it to go to the weekly planning board, but at first has to basically download and process all the updates for the day. And that’s an extremely critical flow for a SAA, and it’s one that, to be totally frank, don’t handle great yet. So that has been, that’s an example of a challenge that’s unique to local first. 00:44:10 - Speaker 1: I think that’s a really interesting one because I think the superpower of local first is that you always have your data and you can always work offline, everything always works as you expect. But that same superpower becomes your kryptonite when you’ve been offline for a weekend or offline for a vacation, and you need to come back to the office and suddenly you have to wait for, you know, even if it’s just 15 or 30 seconds. When all of the rest of your work has been almost instant, suddenly that 30 seconds feels like a lifetime. And cleaning up that user experience, I think is certainly on our list, but it’s definitely one of the weird kind of edge KC things of this type of architecture. 00:44:51 - Speaker 3: Yeah, and to be clear, it’s absolutely fixable. It’s not really a flaw of the approach, it’s an engineering deficiency, and we will fix it eventually. But you are quite susceptible when you’re trying to bring all the content in locally. If you are slow, for example, that becomes very apparent when you’re trying to bring in the whole corpus versus when you’re just trying to bring in a page in a traditional set up. 00:45:11 - Speaker 2: Yeah, I think we anticipated this in the research phases many years ago, and we always speculated that in our dream world, and we’ve talked about this with not only among ourselves, but many of our guests here, including folks like Martin Kleppman and Jeffrey Litt from the local first world that something built into the operating system where you essentially had a syncD, you know, Damon that’s running in the background or it’s part of your file system that it is always syncing those changes whether or not you have the app open. And so when the app opens, it can just interpret kind of what’s already there on disk, that sort of thing where subscribing to events is something that’s part of the this kind of next generation file system. I think that’s what we always hoped for, and you could basically get a similar effect with Muse if you just left it open all the time and minimized so that you’re always streaming down the changes, as long as your computer’s kind of awake and online, and then when you click on the link, it just sort of brings it to the foreground versus needing to go get a bunch of updates. 00:46:07 - Speaker 3: Yeah, although, well, I feel like we need to look into this, cause whenever my muse is running in the background, the Mac has like no problems looking for system updates and all kinds of ridiculous stuff that I don’t want to happen, but I’m not sure if it’s always getting the updates from the sync server when it’s sleeping, basically, so we should look into it. 00:46:23 - Speaker 1: Yeah, absolutely, it’s one of those, uh, many things on the list. But certainly, as you said, a solvable problem, which is always nice. 00:46:32 - Speaker 2: This is a downside to pioneering a new architecture is that, you know, you gotta kind of build a lot of the foundations yourself and you might run into things that are sort of handled or solved or just standardized, let’s say clicking on that link and getting that spinner is pretty well established for the classic web sass, but we’re sort of reinventing some of the primitives of the universe that we’re working within, so that creates an ongoing stream of work for us. I’ll also mention kind of on the user experience side, I think another closely related thing is not just the weekend, but if you take a vacation and you have a team that’s very active generating a lot of content, particularly again, if you use video or heavy PDFs or lots of ink, that sort of things, you can essentially fire up your iPad, fire up your Mac, and find you have a gigabyte of updates to download. And again, it’s not a huge deal, but it may be surprising for, you know, if you’re compared to cloud software where you can open any page and notion without needing to download all the rest of what has changed since the last time you opened it. So that’s a benefit to that like really shallow cache of the web kind of standard model. Now of course the flip side of that is I go to open a page that I had opened 30 seconds ago and my browser’s already forgotten about it, but notion’s offline or I’m offline or something and I can’t load it and I just get a spinner. So you know there’s trade-offs there, but the way I see it is that part of what we’re doing here is seeing how this local first architecture can work in practice, what it means for users, both benefits, but also these edge cases or downsides, and what can we do to mitigate those, or are there places where there’s just truly trade-offs where this is going to be sort of a worse experience than other approaches like client server or even classic desktop software. So we’re in the process of exploring that and it’s part of what’s fun and exciting about doing this work. 00:48:24 - Speaker 3: One other fun one that I’ll mention that’s become apparent with multi-user is the dancing cards. So when you come back from vacation and you’re receiving this big sink down at the data layer, it’s pretty clear that we should be telling the user, like there’s a sync in progress, the little indicator on the bottom right can be flashing or whatever, and then when it’s done, it’s done. But then what you display on the actual boards is a pretty interesting question. You could like make it impossible to interact until it’s fully caught up, but that doesn’t feel great. What we currently do is that as we’re getting updates, we’re like replaying them and the cards are literally dancing around, reflecting the moves that they’ve made in the last week and the additions and subtractions and so on. So it’s kind of funny to watch. We still need to figure out what the exact right user experience is there, not to mention the technical problem of how often do you to bounce the re-renders and so on. 00:49:09 - Speaker 1: Yeah, I think there’s a lot of interesting things that we could end up doing that we’ve certainly talked about and planned for, compaction being one of them, where if I move a card and move a card the second time, then that first move, we really don’t need to store anymore because it’s just never gonna happen. And so being able to remove old content that we know is going to be overwritten from the database, that’s gonna dramatically reduce the number of things that a user would ever need to download. I think we can probably reorder the types of updates and so that way, instead of going from past to future, we can Possibly send back updates from future backwards to past, and so that way the most recent arrives on the device first. So I think there are a lot of technical things that we can do to minimize kind of the strangeness of some of these updates. But yeah, it’s definitely a new frontier and so we’ll find strange new experience. OK, now we have the technical solution, let’s implement that and then that just moves us to a different strange experience that’s slightly less common, that’s a little bit further down the road, and it’s just a continuing adventure. 00:50:15 - Speaker 3: Yeah, well then I’m looking forward to working on there as prioritization. And we kind of alluded to it. Right now we do some prioritization because we prefer to download the packs relative to the blobs, which is good because packs are much smaller and they’re more important. They tell you what boards are where and what cards are where and so on. But I would also like to see us prioritize blobs. You can imagine constructing and continuously updating a priority tree based on how close a blob is to your current position in the hierarchy, or other factors like how often do you open up a known board, and based on that be constantly prioritizing your blob download so that You, for example, prioritize the ones that are on your board, and also on the board that’s like 1 hop away and then 2 hops away, and that’s always being reshuffled according to where you are on the board. I think that’d be cool. 00:50:58 - Speaker 2: So as a closing topic or a potentially very substantial closing topic, a question I get fairly frequently from other teams is basically what should we do? We agree broadly with the principles of local first, we need to sync our application data as basically everyone does in this modern world. How should we do that? What should we use? Should we build it yourself the way you’ve done it? There’s now an increasing number of commercial products that offer off the shelf solutions like replicache and live blocks and party kit. There’s libraries like Amerge that we worked on at Inco Switch or YJS. But then of course, we chose to build our own and we talked a little bit about the motivation for that back in the first episode. So yeah, I guess if a team came to you today and said that they’re working on a piece of application software that is in the productivity space and wanted to implement syncing between devices and multi-user collaborative support, both real time and asynchronous, what would we recommend them or what would you both recommend them in terms of the best way to do that today? 00:52:08 - Speaker 3: I’m shaking my head on the podcast cause it’s really tough. There’s not an easy answer here. There isn’t, as far as I know, any satisfactory, fully integrated solution. Now, we haven’t developed that ourselves, even for our internal use, and I can explain what the gaps might be. And there are also some existing commercial offerings and maybe some open source offerings, but as far as I know they’re still quite a ways from the full solution. So it’s tough. What might I do if I was starting again? Well, first of all, it’s very hard to recommend that someone roll their own unless perhaps they’re extremely interested in it and they’re an experienced systems engineer. I think if both of those aren’t true, you’re really asking for trouble, and then what do you do? I think it’s worth trying out the existing open source and commercial implementations and doing like a smoke test or a bake off where you try to build a basic app, like a to do list or something like that, and each of these things, I think you would find that none of them are there yet, especially if you do some reasonable anticipation of the things that you’re gonna need in a sophisticated production app. So it’s tough. I want the industry to keep trying, but there really isn’t an off the shelf solution that’s satisfactory right now. And I think for that reason, you’re just wearing your business hat. I don’t think it’s fair to make the immediate turn on the decision tree towards local first, as much as I like it, right? I think you got to be open to the possibility that it’s not appropriate to spend your innovation tokens there. We can link to the talk for this, to spend your innovation tokens on the persistence technology. You might want to spend it on a new business model or a new market or something, right? Yeah, I mean, it’s a bummer to say that, but it’s still early. 00:53:41 - Speaker 1: I agree. I think there’s a lot more options today than there were when we started, what, almost 1.5, 2 years ago now, on our own sync engine. 00:53:50 - Speaker 3: I mean, I’ve been working on this for like 5 years. 00:53:53 - Speaker 1: Yeah, yeah. And it’s such a tough thing, it does keep getting better and, you know, like you said, every product has its own needs and will be able to accept slightly different trade-offs or dramatically different trade-offs than any other app, and so it’s In building anything, the should I roll my own question is almost always answered with, of course not, that’s generally a bad idea. You know, leverage what other people have done. I think because it is such a young world for sync and for CRDTs and for local sync in particular, it is a reasonable option. I think if I went back and talked to ourselves a number of years ago, I think I would still recommend that we do this, cause we’re able to decide. Which trade-offs we care about and which ones we don’t, and make something that works really well for use and for our use case, but You know, to borrow the Hobbit, one does not simply build a new sync layer. You start with a product and you want to build a sink layer, and now you have two products. You had a huge undertaking to build. This kind of a system on the server and on the client and should not be a default answer, I think for anyone, and it was not our default answer, but I think it worked out and was the correct decision for us in the end. 00:55:14 - Speaker 3: Yeah, I’m, I’m happy we did it, and I think it was a good call. I just think it needs to be, you know, appropriately tempered for people ask us that question of what should we do. Wulf your comment reminds me of something that the Hoku postgrads team told me back in the day, which is, I, I hope I’m quoting this right, but they said something like, plan for 10 years to build a database, and I’m not sure if that was their original coinage or if that’s due to the post grass community, but it’s been my experience that’s basically correct. I, I’ve seen now quite a few database projects attempted and You know, often in year 5 they’re kind of just getting their sea legs and starting to figure stuff out. So it’s not too surprising in that respect. This is basically building a database of sorts. I’ve been thinking in anticipation of this episode, like what would a satisfactory commercial solution look like, like what problems would need to solve, and I think it’d be fun just to enumerate my list here, so just to quickly run through it, I have. Obviously you need basic persistence and networking to be able to get the data back and forth, but importantly, you also need built-in batch line pipelining and compression. It’s something that I think a lot of systems miss. You need compaction and excision. Eventually you need anti-entropy. We don’t have that yet. You need the ability to handle this on in the background. You need some sort of two-way interface like an ORM comparable for both crude and rendering. You need versioning, migrations, handling all those cases that we’re talking about with the disjoint worlds. You need something around like queries, indexing, declarative views, something like that. You eventually need prioritization and for similar reasons you need the ability to declaratively load and unload, so everything isn’t either all or nothing on disk or memory. That’s a lot of stuff. It takes a long time to build. It’s actually quite hard to get all of that right, I think, unless you’ve seen the very specific problems in production. So it’s tough. 00:56:51 - Speaker 2: And I’ll add to that list developer experience, right? You’re designing an API, you’re often asking developers, you know, in the iOS world, they’re used to working with say core data or maybe something a firebase in the web world, they might be used to working with something like local browser storage or, you know, in the rails world, you do yeah ORMs on the back end, that sort of thing, and those things are really well established. And developers know how to get their data out of whatever persistence layer they happen to be using and turn it into their rendering layer, and now you’re offering them this new as exactly as you said, a new database with new contours and new capabilities and behaves in different ways, hopefully better ways in the long run in terms of user experience, but that is a whole new surface area, a whole new API, a whole new experience for the developer, and certainly I know that that’s something that a lot of these projects like Amerge and YJS and the commercial ones I mentioned. Spend a lot of their time on it’s just like what’s the right API that feels familiar and can be used in a way that’s maybe similar to other ORMs or persistence layers, but also offers the things that make this sync-based approach and local databased approach unique. 00:58:01 - Speaker 3: Yeah, and this developer experience thing reminds me, we’ve done this whole podcast basically talking about this world of Swift, which we use both for iOS and Mac, and we’ve also implemented a little bit of this client side protocol and node for testing

Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: No one would ever write a blog post or a book by hand or on a typewriter in 2023. Yet reading still pretty much takes place in the physical world, at least nonfiction, 90% of nonfiction reading is still paper books, and so our lofty ambition would be, we’ve created such a better reading experience using software that people are motivated to switch and read digitally. 00:00:28 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac, but this podcast isn’t about Muse the product, it’s about the small team and the big ideas behind it. I’m Adam Wiggins here today with guests Dan Doyen and Tristan Holmesy of ReadW Wise. 00:00:44 - Speaker 3: Hi, I’m Tristan, co-founder and technically CEO of ReadW Wise. 00:00:49 - Speaker 1: Excited to be here, Honored to be a guest, huge fan of the podcast. 00:00:54 - Speaker 2: And I understand you two had some fun with Falcon recently. 00:00:58 - Speaker 3: Yeah, that’s right. We were just off on a team offsite in Ireland. We’re a remote company, so we try and get the team together about twice a year, you know, with scheduling and all that. We’re about 14 people now, so getting 14 people out of one place is a little non-trivial these days, but we chose Ireland because we have an engineer. who already lives there, so you can kind of give us a good local experience. And as part of that, we had a fun little falconry event where, you know, you have the little glove. I think you’ve done it before, Adam, and the falconry instructor puts a little meat on your glove and then the birds fly onto your glove and eat the meat. 00:01:38 - Speaker 1: Dan, is that a good overview of, yeah, we were able to play with a falcon, a hawk, and 4 breeds of owls. 00:01:43 - Speaker 3: Yeah, so the fun part was the first owl was like this cute little, I think it was like a snowy owl or something, Barn owl, oh, barn owl, OK. So we take turns each like having the owl fly onto our glove. It flew on to dance, I think pretty successfully, and then we moved on to the next person, I think like one of our engineers or something, and instead of the bird flying onto the glove of the engineer, which had me, it flew back. On to Dan’s glove, which he wasn’t even holding properly or anything, and kind of gained this funny attachment to Dan. Then the instructor was like, Oh, that’s weird. Grabbed the bird and was like, OK, go over here. Go over here. He eventually got it onto the engineer’s glove, and then we did the next person, and the owl again flew onto Dan’s glove. And then that basically repeated for, you know, the next like 10 minutes until Dan had to like hide his glove and basically hide from the bird before it would stop flying to him, so. 00:02:33 - Speaker 1: Yeah, I grew up reading My Side of the Mountain, so I fancy myself a bit of a bird of prey expert. 00:02:40 - Speaker 2: Missed your calling as a bird whisperer. Yeah, I briefly had the chance to do the glove thing with the yeah, I think it was a falcon, and they’re just obviously really beautiful creatures, also very alien in a way, the way they move and the way they Obviously feathers and all this sort of thing, and obviously we see birds in our daily life, but birds of prey really feel like a different thing, I guess, and being that close and being aware of the weapons, you know, their claws or talons, I guess they’re called, and their beak and that sort of thing. It’s just, yeah, it can be a powerful experience. 00:03:15 - Speaker 1: Yeah, the descendants of dinosaurs. 00:03:18 - Speaker 2: Exactly right. And tell me a little about Reidwise. 00:03:22 - Speaker 3: Sure, so Readwise at a high level, we basically make software that helps people read better. There’s a lot of ways we do that. Most recently, we’ve been working on this app, which we call Reader or the Readise Reader, which is this piece of software which lets you read basically any type of content from PDFs to books, to articles, to YouTube videos, including their transcripts. To newsletters, to Twitter threads, to a lot more. We let you read that stuff, but more importantly, kind of save it, highlight it, and just kind of get the most out of your reading. And before we built Reader, we also built a slightly more niche app, which we now call ReadW 1.0, which basically helps you retain more from your reading and kind of manage all of your digital highlights. 00:04:06 - Speaker 2: And could you tell me a little bit about the journey you each took in your careers that led you to this venture? 00:04:11 - Speaker 3: Yeah, I think the Re Wi story really starts with Dan, so he can probably pick that off. 00:04:15 - Speaker 1: Yeah, I’ll start. My journey was a little bit more non-traditional than Tristan’s or probably the typical guest on this podcast. In high school, I was very interested in computers and entrepreneurship. I actually had a web development agency. This is in the late 90s, early 2000s, when every Main Street business was realizing they needed to have a presence on the World Wide Web. As an expat in Germany, you might find it amusing that we called the company Glowing Pear, because we were studying German and learned that light bulb in German is Glue Bena. So that was a funny name. But I went to school to study business, not realizing that studying business is really finance, so I took a 10 year detour from software and entrepreneurship to go to Wall Street, but then in my early 30s, I had an early midlife crisis, quit my job, sold all my things, and Traveled the world for a little over a year. It was during that time that I became a power user of Kindle, because we were backpacking and couldn’t really lug around all these books. And it was also during this time that I was reading about one challenging book per week, and I got really frustrated with the fact that I could barely tell you the name of a book I’d read a month ago, nevertheless, the key takeaways. So, when that frustration hit me, I was just ascending the learning curve of this challenging flashcard app called Onki, and I had this idea, oh, I’m taking all these highlights while I read in Kindle, not doing anything with them. What would happen if I put them into Anki and review them for 5 minutes every morning? 00:05:46 - Speaker 2: Nke just briefly is kind of the original tool for thought in some ways, a space repetition system that basically uses this technique to help you remember things better. Beloved by language learners for sure, but also could basically help you remember any knowledge. 00:06:02 - Speaker 1: Yeah, Onki is a very powerful piece of software, but it’s a little bit hard to use. I had to read like a 40 page PDF to kind of get up the learning curve. So this sabbatical afforded me that opportunity to learn it. So, yeah, I did a prototype where I wrote a script that would import my Kindle highlights and then I would review them for a few minutes every morning. There was no commercial interest here, this is purely a hobby, just a personal life hack, but I really became interested when Not only did it help me remember the books, but there were all these higher order second level benefits that I hadn’t anticipated, and that really opened my eyes to what just I now call reading tech or reading technology, which is this idea that the software is eating the world, but it’s barely scratched the surface on the practice of reading. Um, so this was in 2016, and I was in a place where I was looking for what to do next, and that kind of led me to Tristan. 00:06:57 - Speaker 3: Yeah, I have a little more traditional background than Dan, I guess, for software founders, I suppose. I was studying computer science at the University of Waterloo. I’d worked at a couple companies, maybe like some startups, a notable one might be Stripe, and another Superhuman. I was actually doing an internship at Superhuman. Even before that, I had a little bit of a background and kind of reading technology. I guess it started when I was working at Stripe, you know, we had these brilliant founders, John and Patrick Carlson, and they’d always talk about how much they’re reading, how much they love reading. How beneficial it is for you, how it helped them form the company, how it helps them be successful, and I was kind of struck by this weird dichotomy that like here were these like super high tech founders who rely on and love books so much and find them so beneficial, and obviously I agreed, I read a lot too, and yet, you know, they were reading basically paper books, there was basically no innovation whatso all. From the practice of reading from what, like, their grandparents or people 1000 years ago would have done. So, I found that pretty interesting, and that kind of got me onto the vein of reading technology. So I had my own side project, similar to Dan, which I called Reading List.io, which was a slightly different tact, but basically, I was trying to build a tool which would help you prioritize. what you want to read. So not discovery and not really trying to be social like Goodreads or anything, but myself and a lot of people I knew just had lists of books they wanted to read in a text file or Amazon card or something, like hundreds of books, but they had no good way of prioritizing them, analyzing that list, and deciding what to read next. So I actually posted a comment to Hacker News, I just on some random posts to do with reading. And I left my email, I mentioned I was working on this product, and Dan shot me an email, which was how we first met. And then I happened to be living in San Francisco a couple months later, working at Superhuman, and Dan was also living in San Francisco, so we actually met up in person, and we became friends and decided ultimately to start on the first version of Reid Wise way back then in 2017, and yeah, that’s how we got started. 00:08:54 - Speaker 2: And how is this vision of reading tech evolved since you joined forces? 00:09:01 - Speaker 1: In some ways it stayed the same. Our initial mission, I remember vividly being in San Francisco with Tristan in 2017, we formed the mission to improve the practice of reading by an order of magnitude using software. That’s still our mission, still our vision. The way we conceptualized it is we broke the process of reading down into 3 stages before you read, while you read, and after you read, and we hypothesized that if we could double the benefit in each of those, we’d get 23 power or an 8 times improvement which would round up to an order of magnitude. So, we decided to start with actually after you read, based on my experimentation with putting Kindle highlights in Anki and the profound benefits I was getting from retaining more of what we read. We found it was a pretty salient problem, like, we could talk to most people and be like, hey, have you invested 10 hours in reading a book and, you know, ever get frustrated that you are nodding your head the whole time you’re reading it, and like, wow, this is amazing, this is gonna change my life. But then in a month later, nothing happens. A lot of people resonate with that, and There really was nothing else out there trying to provide a solution to that problem. So, that’s where we started with Read Wise, what we call now Readwise 1.0, and it’s a very niche product, don’t get me wrong, but people who like it tend to love it, and it’s very sticky, and they use it for Years on a daily basis, and that was kind of the platform for us to move backwards through the process of reading with our newer app, which we call Reader, we were now able to get into while you read and start innovating there, and also to a lesser extent before you read. 00:10:46 - Speaker 3: Yeah, and there’s a long journey between us kicking off on that first Readwise 1.0, help you remember more product and to even when we started Reader, but yeah, a long slow journey, but, you know, we’ve been working on it for basically exactly 6 years now. 00:11:02 - Speaker 2: Our topic today is read later apps. Now, in this framework you’ve already described, this refers to, I guess, kind of before and during reading. This is an area I’ve been interested in for a long time because I have used a lot of Rela apps over the years, starting with Pocket was the first one I fell in love with. And indeed that brought me to your product, Reader, which is looking for that kind of perfect relator app. And so I thought it would be interesting to talk about the history of those and other approaches to this, but particularly within this larger vision that you have about reading technology and improving all stages of this. So maybe we can start with just how you both think about the category of rel apps and how you think about why you wanted to enter that category. 00:11:50 - Speaker 1: Yeah, for sure. So, starting with the history of read it later apps, there were really two prerequisites that were required before that category emerged. The first is kind of obvious, but the internet and This profusion of permissionless content, notably in the form of blogging, you really needed that content out there before there would be anything to save to read a later app. And then the second innovation was obviously the smartphone. This is the first time that computing devices were less about productivity, right? Like a personal computer is more about doing work. Of course people would play games or Watch movies on a computer occasionally, but it was really productivity first, and the mobile phone, smartphone was a consumption first device. So with those two prerequisites in the late 2000s, we saw the first read it later app. There’s some controversy as to who came first, but according to our conversations and reading of the history, it seems like Marco Ormant created Insta paper. And was the first person to kind of check all the table stakes boxes of what makes a real later app, need the ability to save web pages, to read later from both your computer and your mobile phone. You need to parse those web pages into distraction free clean HTML and then you need the ability to read those across devices. So he really kicked off the category with Pocket as a fast follower, and they really kind of dominated the category. There was a lot of enthusiasm behind these. I think Marco became a little frustrated with the whole competition with Pocket, so he sold Insta paper to a venture capital studio in New York called Betaworks, and they took the baton and continued to grow the platform while Pocket raised a bunch of money. And continue to grow the platform. But then around like 2015, I think they reached an asymptote in terms of their growth and kind of put these apps into maintenance mode. Betaworks spun Ins the paper out to Pinterest, where it kind of lived dormant for three years. Pinterest then spun it back out to the guy from Betaworks who ran it in 2018, Brian Donahue, good friend of ours, a lot of respect for him. And then Pocket was acquired by Mozilla, which is why you see Pocket built into the Firefox web browser. I share that history because that’s part of the reason that we entered this category is because we love these apps. I’ve been a user of Insta Paper since, I think, 2011, I tried to go back and figure out when I first saved a document there, but from about 2015, 2016 on, they were just maintained as opposed to adding features, so we felt like there was a little bit of a void in terms of innovation and continuing to build on to the great foundation that they started. So we saw an opportunity there, at least for our user, which is very much a power user to enter that space and kind of carry the baton forward. 00:14:55 - Speaker 3: Yeah, and you can kind of take the path to read it later apps forward from us, we kind of start a new one. I’d say the existing ones this paper and pocket basically been left alone, just kind of maintained, but you can also go a little further back, it’s kind of interesting, and this is something that Dan actually has pointed out to me and we’ve kind of discovered, or at least discovered the significance of only like rather late into our journey to Reader, like, there was a fundamental activity that came before Read it later apps, which was, as you probably know, Adam bookmarking. There have been bookmarking apps, basically, I think, as long as the web browser has existed. I’m not really sure what the first one was probably built into Netscape or something, but you know, there’s this fundamental habit of just bookmarking a web page or URL that you want to come back to in the future. All major web browsers have this built in, there’s a lot of dedicated bookmarking apps. Such as pinboard and many others. What’s kind of interesting is that fundamental activity, saving a link kind of forked when the smartphone came out. So, One Direction is the one that Dan mentioned, which is that people took bookmarking, added in, you know, distraction free reading, added in the cross platform syncing, and you kind of got real later apps. There was another fork though, which was pretty interesting, which was basically what Pinterest did. Pinterest at its core is bookmarking. It’s much more visual bookmarking. It’s a very different market from the read it later out market. It’s a lot less of a focus on maybe like the traditional type of like blog post reading that would happen in its paper rather, it’s a lot more focus on inspiration, stuff like saving recipes, but at its core, that’s another fork of bookmarking, so it’s kind of interesting to see. And what’s also striking about this category is, we’re not venture successes at all. Its paper wasn’t trying to be one, pocket was. And I’m sure the exit was still successful, but it hasn’t become this like household name, whereas Pinterest really took that fork and it was massively successful with it, which is kind of an interesting tidbit. And of course you could maybe view a third fork as social networking apps, especially if you focus on news. And just a more social angle, those are also being massively successful and at their core, a lot of what people are doing is sharing bookmarks. So there’s some kind of interesting history there, we could probably nerd out about the history forever, but that’s probably as deep as we should go. 00:17:07 - Speaker 2: Yeah, I agree. It feels like the fundamental building blocks or primitives that eventually became what we now think of as relator is bookmarking, obviously the web and as you said earlier, Dan, the sort of permission free, I can just click a link and start reading it style that exists on the web, putting paywalls aside, of course. And then I also feel like RSS readers and the sort of blogosphere circa, you know, mid 2000s, maybe as part of this equation as well. But looking at the kind of relative enthusiasm for those early movers and paper and pocket and then how they both seem to peter out and eventually go into maintenance mode, it does seem like this is a beloved category for a certain class of person, 3 of us included, but that class of person there just isn’t enough of to make a venture scale business out of. And so, yeah, when things cap out and then they can’t give the proper returns to their investors, and I seem to recall pocket kind of thrashing around a bit and adding social features and adding this, that and the other thing that I think power users who are focused on their reading experience didn’t want in hopes of, yeah, trying to achieve that scale. What have you learned from this history, and maybe it’s partially just how to think about the market, but also, what are you gonna do to avoid the, you’re walking down a road and you see these bodies alongside of it. What does that tell you about the dragons that lurk in the bushes? 00:18:34 - Speaker 1: Yeah, well, the first thing is raising venture capital into a reading app is difficult, which is one of the core reasons back in 2018 we actually published a blog post that we referenced all the time. We decided to bootstrap readwise, meaning, you know, fund it from Revenue from our customers as opposed to venture capital, that was one key lesson we learned and we think enables us to build for the long term. Tristan, you wanna take it from there? 00:19:00 - Speaker 3: Yeah, well, the other is kind of the fact that we look at the history at all, and we looked at similar startups when we first started Read Wi, similar companies that had tried to build on top of the Kindle platform, or what have you, you know, I think there’s that quote from Alan Kay, where he’s like, computer science is like a pop culture, where no one ever looks back in time at all. And we’ve actually just gotten a surprising amount of value, just by looking back, you know, looking at the history of Insta paper and pocket, how it turned out, like what People liked about those, what they didn’t, what growth strategies worked for these companies. It just says a meta point that’s super underrated in startups. And you know, we’re not near one of the internet here. This isn’t like 1995 or whatever. There’s a lot of data to go back on, and that’s not to say that means these ideas are dead ends. Obviously a lot of ideas that didn’t work in the past can now work because of better distribution or better internet speeds or hardware or whatever, but there’s just a lot of value to going back and looking at those. 00:19:52 - Speaker 1: Yeah, I think another lesson is from a business model perspective, they were both trying to reach a social scale that could be funded through ads and just having so much attention and eyeballs. I think that’s a pitfall that we avoided by making premium software. Another thing we noticed is It’s hard to get accurate estimates of how many signups or users they’ve ever attained, but it seems like around like 60 to 80 million people have tried insta paper pocket, yet I would venture, there’s probably a million active users, so there’s some sort of huge drop off where people fail to adopt the core behavior. That’s definitely not something we have solved. Let me be very clear, but that is one of the things that we aspire to figure out and solve at some point. 00:20:41 - Speaker 2: I do have to wonder whether there’s sort of a gym membership effect here, which is people aspire to read more, to engage more thoughtfully with, you know, whatever content they feel is important to them. But in the end, the low friction path is let the algorithm pick out a video for me to watch on YouTube or Netflix, for example. I mean, the entertainment and general kind of media consumption options we have at our disposal in this day and age are just unbelievable, and so you have to be fairly dedicated to stick with that, I think. 00:21:10 - Speaker 1: Yeah, we love that fitness gym membership metaphor. We use it all the time internally to describe what we do. 00:21:18 - Speaker 3: Yeah, and it’s pretty motivating, right? Because we are often eating time from, you know, somebody scrolling reels on Instagram, or scrolling TikToks or scrolling through their Twitter feed. We aspire to do better than that, and it’s very motivating when you know that you’re helping people read stuff that they’ve consciously picked out. It’s a lot harder, absolutely, it’s a huge challenge and to be clear, we lose to these. Addictive social media apps all the time, but we can feel good about building a future which really grows engagement, which really makes the app sticky, which really, you know, sends the user a push notification every morning, and we don’t have to feel bad about it, and the user will generally leave a session of the reader app or the Read Wi app feeling very happy about the time they spent. And I think While it is a little bit of an uphill battle, just like a gym membership. Once you can do it, I think people become very loyal customers. They have very high retention, and they really like coming back to you. It’s just a a lot harder, but, you know, in the same way, we can feel good, just like a gym membership. Maybe most gym members won’t end up using that, you know, you can go and invent like a Barry’s boot camp type of class. Or something, or you can go invent a new workout machine or workout process or workout plan, which really does help people actually get better outcomes and read more and spend more time reading and feeling better. It’s super motivating to get up every morning and work on that problem as opposed to how can we make reels, you know, 1% more addictive and extract, you know, 5% more ad revenue. We’re not like these huge social media haters. We go on these tirades, like they’re tearing apart society, especially Twitter, definitely has its place, and there’s a lot of value to very like low information latency communication and stuff like that, but there’s also a place for long form reading that there’s probably a suboptimal amount going on there if you would ask most people how they’d like to spend their time. 00:23:01 - Speaker 2: Well, actually, since we’ve mentioned social media a few times and I think this is probably also related to that proto aspect of linked discovery, which was RSS feeds, for me, there’s actually a very natural pairing between the social media that I do use, which includes Reddit and Twitter and Re later because I find that when I’m in that linked discovery mode of scrolling or grazing at the content. It’s a little bit too disorienting to then dive really deep into a longer article. And actually I noticed this in my career as a writer on the web, and particularly when I’m writing content for Yeah, like a blog somewhere where we are thinking about, OK, people are going to follow this link from Twitter hacker news, whatever, and we want to keep them engaged and there’s somewhere around 2000 words is kind of like your maximum. No matter how interesting it is, people are going to drop off, not because they’re not interested, but because they’re not in that mindset of going deep in the long form reading. And one effect, you know, if you don’t have a tool like this or some kind of discipline built into your consumption habits, is that just gets lost, you know, maybe you bookmark it or you leave it in an open tab, you know, I’m going to read this later, and I think that system doesn’t often work that well for people, but for me, it works really well. To have a place to put something that I want to read, I do want to engage deeply with, but I want to be in a place and a mindset where that’s what I want. I’ve just settled down into my seat on a long airplane ride, and yeah, of course I can get a book, but I also might want to get out a long form web article, for example. How do you see reader fitting together with, yeah, social media, the linked discovery problem generally, and how does that fit in also to that framework of the, you know, before, during, and after reading. 00:24:49 - Speaker 1: Yeah, there are a lot of interesting topics there. I’ll start with first, the type of reading that we’re focused on. So in books, which is where we started, believe it or not, before read it later, they typically bifurcate reading into nonfiction and fiction. We think that’s a little bit of a crude dichotomy. It doesn’t really apply to this reading. We break it into reading for betterment and reading for entertainment, and our mission is to improve the practice of reading for betterment by an order of magnitude. Reading for betterment might be learning knowledge, learning know-how, which would be like a skill, trying to better your career, your relationships, listening to a podcast in your industry like Meta Muse, reading a newsletter about your career. Those are examples of reading for betterment. Reading for entertainment would obviously be the kind of reading you would do instead of watching Netflix, like maybe reading Harry Potter, listening to a fun audiobook about science fiction. And news is one of these that kind of falls somewhere in between. In many cases it is just a form of entertainment. Occasionally it is a form of reading for entertainment, so we often see it like things blur for us when we’re talking about news and RSS. When we started Reader, we spent a lot of time trying to come up with these really robust abstractions that would enable us to build for a long time, and we tried to anticipate all the pitfalls and the dead ends and the idea maze. I think one area where we were a little naive was the pairing of read it later with RSS. That was probably one of the hardest challenges. For me, it’s one of the hardest challenges we faced, because you have so many different RSS use cases. So, you have this old school generation of RSS readers, people who are around back in the golden era of Google Reader, who are just absolute infovores. They’re getting like 1000 new links a day, and they just want to churn through them. We haven’t yet built our RSS feeder to accommodate them. We’ve really Built it towards the more modern RSS reader, which is born out of substack and email newsletters, where you’re getting maybe 2 or 3 long form pieces per week, and that’s more the volume you can deal with. But obviously RSS has been a category that had a lot of enthusiasm in the late 2000s, early 2010s, but from our perspective, I think most people would agree, social media kind of replaced or supplanted RSS. So yeah, we’ve not been trying to go back and recreate what RSS used to be and kind of fight against that current. We’ve really been building for the more modern RSS use case, which is less about like getting your TechCrunch and yournggadget, 100 links a day, and more about your substack writer who posts 3 times a week, more to that like blogosphere era. 00:27:40 - Speaker 2: Substack is an interesting one to talk about. I think Reader and Substack at the moment are sort of my two apps on my phone that are the ones when I think, OK, I want to go for an in-depth piece, I have some time to sit down. I’m eating lunch, I’m, yeah, on the plane, I’m whatever, those are the two apps I’m likely to go to, but they’re completely different. Reader is part of, yeah, open web, open protocols, you can save any link. Indeed, you support media types outside of the web, including PDF and Video and others like you’ve mentioned, whereas SubStack is a closed network that includes the publishing, the content, the linked discovery, the whole, you know, subscribe and follow the payment as well as now because they have a really nice mobile app, the reading experience. How do you think about, I don’t know, substack fitting into your world, part competitor, part adjacent complimentary product, I would imagine. Yeah, how do you think about that? 00:28:38 - Speaker 1: Yeah, I mean, we definitely have a lot of respect for those guys. The quality of content coming from them is off the charts. One interesting thing that Tristan and I always talk about, and maybe he can expand on this is one of the opportunities we saw when we started working on Reader is, but for ins the paper and pocket, there’s really never been a reading app that’s been built for the reader. Most reading apps come from the supply side, and they’re really Trying to benefit the creator of the content as opposed to the consumer of the content. So we saw that as an opportunity for us in a point of differentiation, where we’re putting the interests of the reader first, as opposed to the interests of the creator. Now, oftentimes there’s an alignment of interest there, but not always. 00:29:22 - Speaker 2: Right, you can think of Substack like medium before it as something where it’s a two-sided network, and they want to get the readers and they want to get the writers or the content creators and put them together, but even, yeah, for example, part of the substack, call it reading experience is that when you’re scrolling through an article, you know, it’s prompting you to subscribe to their newsletter, either to get the emails or to subscribe as and become a paying subscriber or even once you are a paying subscriber. then it still says, do you want to give this as a gift to someone else, that sort of thing. And obviously all of that serves the interests more of the platform and of the content creator, and it’s a minor inconvenience, so I don’t worry about it, but it’s obviously not putting the reader experience first the way that you can. 00:30:05 - Speaker 3: Yeah, and to be clear, there’s great value in serving the writers. I think they are an important part of the ecosystem, you know, with writers, there’s no reading, but just for us, we’re like laser focused just on the readers, you know, we don’t want to offend the writers, we’re not gonna do anything that takes money out of the writers' pockets or try to hurt them at all, but at the end of the day, every day we wake up and we build software for the readers. We think about how does this reader want to spend their day, how do they want to spend their time reading, how will they feel good about their reading time after, and how can we help them do that? And ultimately there are customers too, we’re just very, very laser focused, and I think that allows us to go a little deeper in the IDAs of reading. We just solely think about how we can make reading better, and I think that helps us a lot. I think we’re pretty lucky to be able to do that. Of course, you know, it’s arguably a worse business model. Dan and I can go on and we have for days and days about the challenges with consumer sass and charging consumers money and how much higher the bar is there compared to enterprise and all the unique challenges like having churn, non-negative churn for your business. So there’s a lot of downsides to it, but at the end of the day, at least we get to have that benefit. So we try and enjoy it as much as we can, I guess. 00:31:13 - Speaker 2: Yeah, I’m generally a big fan of prosumer software and indeed that’s what Muses, but yeah, it comes with a lot of challenges. You get the fickle behavior of consumers and the desire not to pay any money. Often in a surprising contradiction where what they’re willing to pay for physical goods or experiences like eating out at a restaurant, don’t seem to match up with what they’re willing to pay for digital goods. Happily, I think that has changed a lot and actually maybe this is to your point about your ability to succeed where some of the earlier folks didn’t, you know, I remember when Pocket experimented with, I became a paying subscriber, they had a, I can’t remember what it was, $40 a year or something. Subscription, I was happy to support them, but I think at that time, this would have been, yeah, 8 or so years ago, the idea of prosumer software paying subscriptions at that price point, which is kind of what you need for the number of people in your addressable markets. I was just like offensive, somehow and people were just like, you know, up in arms because they were used to getting all this free stuff from, you know, Facebook and Google and everything like that. Happily, I think that has started to change both as software developers, especially in the software developers realize subscription. are a way to make a sustainable business and do well by your customers and increasingly I think customers are starting to recognize that. But yeah, when you compare to the ease of getting a business, a B2B customer to pay, where if you can do something to improve their efficiency and help them do their business better, they’ll happily pay for a tool, an individual will hesitate on that a lot more. 00:32:47 - Speaker 1: Yeah, it’s always funny how I’ll slap down the corporate card to buy a $30 a month form building software without even thinking, but many times we’ll get users right in that this is kind of like a family decision. They, they have to talk to their significant other about whether or not they can afford this expense at dinner. Completely different psychology around these expenditures. 00:33:09 - Speaker 3: Yeah, and while you’re right that there is a lot more willingness to pay and subscriptions have become a little normalized, I think, whereas in the past, you might have got more of a comparison to Google or to Facebook and be like, oh, online software should simply be free. Now we get a lot more of the comparisons to Netflix or Spotify, and they’ll be like, well, Netflix gives me millions of videos, Spotify gives me unlimited music, and, you know, they cost the same per month as your product, so like. How can you justify that? And of course the optimal amount of like pushback you’re getting from these customers is not zero. If you’re getting no pushback, it’s probably because you’re charging like something so ridiculously low that your company is not sustainable. So you kind of have to take a lot of that feedback and stride and be like, oh, well, if you value Netflix more than reading better, then, you know, you’re probably just not our target user, and that’s OK. Maybe you can go use the free pocket version and that’s actually better for you. 00:33:57 - Speaker 1: Yeah, the place where we succeed is when we’re dealing with someone who views reading not as a form of consumption or entertainment, but as an investment in themselves, almost like a form of education, and when it’s framed that way for them, the expense is a no brainer because they can get such a higher return on time invested in reading than they can otherwise. So that’s where we really win. 00:34:21 - Speaker 3: Yeah, and that is the sweet spot, right? I think there are a lot of apps out there, like, you know, Pocket is better for the price obviously, than Reader. You could argue maybe like Pocket Earns paper has a very simplified, streamlined, very simple approach, and maybe that’s better for some people who just want something dumb simple, and that’s fine. There’s a lot of aspects in my life. I might prefer an app like that, but if you are someone like Dan mentioned who views reading as an investment, and you’re kind of coming into the same frame with it as you would a online. Course, you know, they have these online courses that teach you to write better, to be more productive in your job. They’ll usually be like $3000 4000 dollars for a course. And, you know, a lot of people consider that a worthwhile investment in themselves, and it probably is, it probably is worthwhile. So for people who actually value that investment, I think we can say Reader is the best product. For those types of serious readers and people who are reading for betterment, and that’s kind of the sweet spot we aim to be at. Again, because we’re bootstrapped, we’re not trying to get a billion users, probably like Pocket was trying to do, I would guess. We don’t want a billion users, we want the million, the hopefully someday 10 million people who really, really want the absolute best reading experience for productivity and betterment. 00:35:33 - Speaker 2: Now speaking of comparing to free alternatives, how do you think about what’s now built into the sort of iOS and Apple operating systems with there’s a reading list and the readability mode for websites, that’s kind of one tap away and yeah, it gets all these first party advantages in terms of it’s a single tap to say something to your reading list versus needing to go through a share sheet and set things up to reach your app. How do you find customers compare you? How do you think about positioning on that? 00:36:01 - Speaker 3: Yeah, that’s a great question. That actually speaks to like the main benefit of reader, you know, we’ve talked about we want it to be better, but how is it actually better than other ways of reading right now. The main value prop that we pitch, we basically call all in one, and this goes back to comparisons to substack 2. The main goal we had for reader as a product for the 1st 2 years of building it was that a user can save absolutely any type of content they want to read. As long as it’s reading for betterment, they can save that to readers. So, again, that’s from books to articles, to email newsletters, to RSS feeds, Twitter threads, YouTube videos. Basically, you can get all of that stuff into one place, so when you’re, you know, sitting down on a flight, like you mentioned, Adam, you have everything that you’re consuming in one place rather than say in Safari, you’ll just have your web articles, or on Twitter you’ll just have your Twitter bookmarks or something. The reader can support all of those things and more. That seems to be the thing that users tell us they like the best of our reader. 00:36:58 - Speaker 1: Yeah, this is kind of like a classic substitute. What’s a substitute for what you’re building. And I will say the biggest substitute is just having a bunch of tabs opening your browser. And I’d be lying if we said we had figured out how, not with a power user, but with more of what we call like a normie user, we’d figured out how to persuade someone that it’s better to be saving those things to reader and aggregating them in one place and kind of taking control of that fire hose of content. In reader, but that’s probably our biggest aspiration for the remainder of 2023, trying to figure out how to make that value prop more tangible and the onboarding experience to experience it much faster and smoother. 00:37:42 - Speaker 2: I’d be curious to ask about a few technical aspects of how you’ve built this product, and the first one that comes to mind is, yeah, let’s call it that readability mode or the parsing, the stripped down view. You’re not just a bookmark where when I call up the article that then it’s displaying it exactly as I would see it in my browser. I get this simplified view, everything’s kind of in the same typeface, etc. and my experience with read letter apps has always been that. Even a very well made 1, 80%, 90% work perfectly and then there’s also a pretty big list that worked pretty well, but for some reason, the first paragraphs missing or the image should be there that isn’t or there’s an image that’s actually part of an ad that is, and then on the far end of the tail there somewhere is articles that just don’t work at all or whatever and you need to open in your browser. How do you implement that parsing, whatever it is you call that part of the product. 00:38:37 - Speaker 3: Yeah, that’s a great question, and it is a really hard problem. It’s definitely one of the harder problems we had to tackle. You know, there’s a lot of secret sauce there. There’s a lot of just manual work over a long period of having the right libraries to do the right type of extraction, the right type of labeling, third party providers, machine learning. There’s a lot that goes into it, but I’d say there were Maybe a couple like interesting decisions we made that I think have made us pretty successful at this. Obviously, we still have parsing issues, and I’ll explain in a bit how I know this. Like, I think we have basically the best parsing of any read itator app that we’ve tried. So the first major decision I think you have to make is, do you do the parsing on device locally or do you do it in the cloud? And we decided to do it in the cloud. There are a lot of libraries. There’s a pretty popular readability.js one that I think Firefox maintains that they use for their reading mode, and I don’t actually think they use it for a lot of pocket, but it is used for the Firefox reader mode, and who knows what else. Google Chrome has their own version too, that’s even a lot simpler than Readability JS, so we could have done it on the client, and there still might be situations in the future where we want to do it on the client, but We just found them really lacking. So ultimately we decided to join the cloud, which gives you a lot more ability to debug what went wrong, to improve your process, to fix the parsing from our side when it’s broken, and just really collect the data we need to build this ongoing process. So that was one decision we made earlier on, which we really have no regrets on. We do a lot of other stuff on the client, you know, our reader as an app runs what we call offline first, basically, most stuff can happen offline and the user’s client, but this was one thing where we’re really glad. We did it on the server. Another thing we did early on, which I guess we just took for granted as like a software engineering thing to do, but has really helped us is basically, before we even started building parsing, we went into Readwise, we found the top 100 articles that people had highlighted the most. And read the most. And basically this is just like a proxy of the types of articles that our users would want to read. We went into Insta paper, we went into pocket, and we basically manually labeled every single one, like, did instant paper succeed at parsing this article perfectly. If it wasn’t perfect, what did it have missed? Did it miss some images? Did it miss some text? Did it include stuff it shouldn’t have included, and you can have different. Tables for these kinds of things, you know, including stuff you shouldn’t have included isn’t nearly as bad as missing content. So we label it that way, and it was basically just this giant Google sheet, very manual things that don’t scale process. But then when we were trying different prising solutions, we would just evaluate them against the spreadsheet, which really, really helped us. And now, you know, we can confidently say, at least with the types of articles that our users read. By this benchmark, we actually definitely outperform ins paper and pocket in almost all cases. Obviously there’s still a few that I will miss, but, and then again, we’re able to go onto that spreadsheet and prove the parsing and fix even more. So that was a really awesome decision we made early on, is just building that manual benchmark and manually going through early on and building that. 00:41:28 - Speaker 2: Yeah, it’s great to have like a really clear benchmark because there’s no such thing as perfection, but I’m reminded maybe of like the web standards acid tests or something like that where engineers and builders in general are really good at, you know, working towards a clear targets, but when it’s just like, I don’t know, try to parse most articles on the web correctly, hand wave, hand wave, that’s not an achievable thing or a measurable thing. So yeah, I can see why that would be a really nice way to concretely compare the approaches you’re taking. And also seems like a natural just integration test, basically. 00:42:02 - Speaker 3: Yes, so that’s the benchmark, and then the last thing I think we did that was maybe unique to us, I don’t know how pocket Instapa did it back in their day. Instant paper was basically just Marco, so I know he didn’t do this, but basically we have a dedicated parsing engineer. Started off just a role that one of our engineers took on part time, but now we have a contractor who basically works on fixing parsing issues full time. We have a whole test suite. We know that like when we update parsing, it’s not breaking anything that was previously working, and we just have this engineer working full time on going through. We have a whole system where we have users report parsing issues, so that goes into this like database we And then we sort them by the things that are most read that are reported broken, and then things that are most reported broken by our users, and we’re able to just kind of privatize that way. And so we still have an engineer going in every day and fixing the top websites, fixing the top articles that the user report are broken, and that has been pretty good. There’s still a lot left to do, but at least we can have this confidence that it is getting better over time rather than slowly decaying, which I think is the norm with this open web kind of stuff. 00:43:02 - Speaker 1: Yeah, to that point, it truly is a game of cat and mouse. There are some platforms out there, particularly the old school media outlets that really don’t want people to take content off the platform and read it elsewhere. So, you may have good parsing for a month, but then something changes and you have to go back and fix it. So, it’s a constant battle, you’re never done. One final thing we did from a product perspective as opposed to, well, it’s heavy engineering, but from the user experience level. We also have a powerful browser extension that enables you to highlight the native web page. So we consider that our failsafe in the event that parsing fails, you can always open the original, and then if you have the reader browser extension installed, you can highlight on the native page. So that’s good for these parsing exceptions, it’s also good for instances where the original creator put a lot of effort into formatting or creating a reading experience that It is actually better than the distraction free clean experience. I’d say the distraction free is better 98 times out of 100, but there are some times where the website is like, truly unique and special, and you want to read it there. 00:44:14 - Speaker 2: Yeah, if you ever read like a bread Victor post or article, yeah, explorable explanations was the first thing that came to my mind. But yeah, I also know that exactly some people put a lot of effort into the typography into the diagrams, so the things I actually want there are bespoke. Experience, particularly if they made it work reasonably well on mobile. It’s just as a default and it’s both the sort of actively user hostile aspects of ads and irrelevant junk that I don’t want to see and clutter up, especially a mobile screen, but also it’s just like, I don’t know, there’s many pieces of good content that are also on a website that just doesn’t have legible typography. And I just want to be able to read it. That’s all, and this, you know, basically pulls out the content and puts it into a kind of a standard size box that’s often better than whatever wild west is out there on the internet. 00:45:02 - Speaker 3: Yeah, and luckily for us, the web page you do actually want to read the author’s version of is quite rare, you know, like Dan was saying, I would say it’s, you know, definitely single digit percentage, often less, and then that actually extends to other format types too. For example, we support Twitter threads. A lot of people prefer saving their Twitter threads to reader and just being able to read highlight across tweets, for example, in the thread, be able to read it, you know, without having to scroll through and There is some cool benefits to Twitter threads where you can click into a specific tweet and see the comments, but for the most part, that’s better. And then even in PDFs, PDFs are kind of like notoriously annoying to read on mobile. You have to kind of like pinch and zoom to like read a sentence and then scroll it across. 00:45:42 - Speaker 2: The two column academic format was never made with the mobile form factor in mind. 00:45:47 - Speaker 3: Yeah, I mean, just PDFs weren’t in general, right? And so we do. Have a feature now where we’ll parse your PDFs basically into clean HTML. So when you’re on your phone, you can just switch to that. It doesn’t work great for graphic heavy PDFs, obviously, but, you know, if you’re basically just reading some text exported to PDF, it works a lot better. And so, you know, there’s really no shortage of benefits for the clean view. Luckily for us. Maybe in the future, all websites will be formatted so nicely and have such great interactive experiments. We’ll need to change that up. But for now, it works quite well. 00:46:18 - Speaker 2: Another technical area that listeners of the podcast will know I’m very interested in is everything having to do with sync, and I know you’ve mentioned there being offline first and maybe I’ve even heard you use the term local first or seen that in your marketing somewhere, but certainly coming back to that personal example of the reader app and the substack app. Both do pretty well at allowing you to read stuff offline, but even there I can see where you put some effort into that that goes beyond what substack folks do as one small example that I run into pretty frequently, I read something on Substack while I’m, yeah, again, on a plane or in the elevator in my apartment or some other place where I don’t have network, and the archive button just basically freezes the app if you tap it, so you can’t archive an article while you’re offline. You could read it, but you can’t archive it, whereas that does work for readers. So I’m curious to hear, yeah, how do you do the syncing? Obviously the cross-platform element is actually an important part of the relator experience because you do often read on mobile, capture on mobile, but the desktop is also really important. Tell me about all that. 00:47:24 - Speaker 3: Yeah, of course. So there is a lot there. I will say our app is in technically local first, and we’ve taken a lot of product engineer, especially engineering inspiration from what you guys have put out the Ink & Switch. So hugely thankful for that. It was actually really, really useful reading through all of that, and Reader isn’t. Technically, local first, I think by the proper definitions, your client will need to sync with the server to start. You can’t just start using it offline. For the first time you use a new device, it needs to basically sync some state from the server, but after that you can basically use it offline indefinitely. And there are some things, you know, obviously syncing documents saved on other platforms, etc. parsing, all that stuff only works online. There are parts that require working online, but for the most part, yeah, you can do everything offline for as long as you want, and then just when you come back online, all your changes will be synced. 00:48:13 - Speaker 2: Well, I mean, it’s pretty fair that parsing only works online because if I save a web link, somehow. Now I don’t know, I had it in my copy paste buffer into my relator app and I’m offline. I can’t load the webpage regardless. So it doesn’t really matter if I can parse it or not. 00:48:27 - Speaker 3: Yeah, that was our thinking there too is there’s a lot of stuff which is just naturally paired with online activity anyways. Another thing that doesn’t work is adding an RSS feed that won’t work offline, where obviously you need to connect to the external RSS feed. So yeah, it mostly works. So that’s why we kind of tried to say offline first rather than local first. It isn’t quite the vision of maybe like some of the demo apps you guys have shown us, like an app where you can take notes, and that could just work fully offline and never need to connect to the server. It’s not quite that, but basically once it’s connected up front, you’re good to go. So syncing is actually interesting. Believe it or not, we actually tried to use automerge was one of the things we first tried when I was first building the first version of. Reader. Unfortunately, Reader supports a lot of data. Our users will commonly save thousands, tens of thousands of documents to reader. And at the time, automerge performance was just too slow to load that stuff into memory. I do know there’s been like a whole new version of Amerge recently, I think, written in like RT or something, which actually might work great. I don’t know if we could switch to it at this point, but yeah, so we evaluated automerge, didn’t really work for us at the time. We evaluated CRDTs, we’re like, oh well, you know, we could use some other type of CRDTs and actually because our app isn’t multiplayer, it’s really just single player. We actually just settle on a very simple syncing solution, basically, a user will make a change locally. The whole local state is basically just JavaScript objects, JavaScript arrays, etc. So what we do is we take the state before, we take the state after, we generate a JSON patch between the two, and we generate the reverse JSON patch, which is very important as well, and we bundle those up into an update object which we then cache locally on the client and send off to the server when it’s ready, and just the JSON patches which are There’s still going to be some conflicts sometimes if you do some stuff offline, then change the same data on a different client, but conflict resolution is very easy for us because it’s just the same person twice, so we can kind of just do last right, almost always wins. And so, yeah, Jason Patch’s simplicity for the win, I guess, has actually worked out great for us. Now, if we ever do build multiplayer features, which we’re not really planning on, but if we did, maybe that would bite us in the butt, but. 00:50:36 - Speaker 2: Now that makes sense to me, and I think the domain you’re in as well, almost all the operations that just come to mind, whether it’s saving a link, whether it’s archiving something, whether it’s highlighting something, these are mostly items operations, right? If I archive on my desktop, but I had previously done that offline on my phone, and then I come back online with the phone and that syncs up, just the two archive operations don’t matter that much. So it seems to me like a really simple solution there is a good one, but ultimately is a syncing. Solution rather than a classic cloud solution, which is I don’t want to tap on my article to read just as I settle down in my comfy chair and get a spinner. I want the whole idea. I mean, I think the product brand pocket was an excellent one and instant paper also encompasses this. Both of them seem to capture the sense of like, it’s a thing you have or you can hold. Rather than the web is a place you, you go out to, you visit a web page, but I don’t want to visit a web page. I want to take this link and save it into my pocket or my instant piece of paper and have it. So then when I sit down to read, I know it’s there and I’m not subject to the whims of the network for that. 00:51:44 - Speaker 1: Yeah, and that whole notion of capturing or archiving or saving is another value prop of read it later that we haven’t talked about, but it’s very important to people to kind of snag this content and a place and time and know that it’ll be there when they come back, which you don’t always get with just the URL. 00:52:02 - Speaker 2: Yeah, I wonder also if there’s an element of personal curation here somewhat, which is one effect of the call the mainstream sort of digital media consumption places is that you typically have this stream of stuff that doesn’t belong to you in any sense when I’m browsing through TikTok. None of this is like mine unless I actually create it. It’s just a stream of stuff that’s started going by, and even something like my Twitter feed or whatever, doesn’t have that sense. I guess you can like your bookmark things there and hopefully those are retrievable later, but I think there’s something to, I want to pluck this thing out of Again, my late discovery mechanism, whether that’s hacker news, Reddit, Twitter, whatever else it may be, and say this is mine now, it’s something I want to read. And then furthermore, maybe this goes to your kind of after reading part of the experience. I want to say I’ve actually highlighted some things because I’ve got some value from some of the concepts in this article, and these highlights are mine. The article was written by someone else, but these highlights were for me, the things that mattered or were relevant or sparked some insight in me. And then furthermore, then I can tag it as a favorite or something like that and be able to go and search that. Later, what was that article I read, I think it was like a month ago that had this concept, I want to share that with someone else, but I can’t remember the name of the article, and searching the web for it is a much bigger haystack to search for that needle than just searching articles that I’ve read or favored. So I feel like this curation aspect, which I think again is a downside, most people don’t want to do that curation work. They don’t want to curate even a playlist on Spotify, they just want to turn on the radio thing and just say play stuff I like for me, whereas maybe again coming back more to that power user, you’re investing in your reading experience. In the kinds of content that you want to read, and it’s worth doing a little bit of effort, especially if it’s a tool you enjoy in order to get that curated set of stuff. This is my stuff. There’s a whole internet versus full of stuff, but here’s stuff that I’m gonna say is mine. 00:54:05 - Speaker 1: Yeah, we’ve definitely benefited from this whole knowledge management or second brain movement. Uh, those weren’t really in effect back when instant paper and pocket were around, so another framing of reader is kind of a read it later app for that community, which is exactly what you’re talking about, this desire to kind of create your own library and anti-library that’s bespoke to you in the digital realm, not just the physical realm. 00:54:32 - Speaker 3: Yeah, I will also say from the technical side, we did make some decisions that kind of reflect that. So, just if you’re, you know, building a table, say, you know, you have an external web page, and then you have a user, and then you have the fact that this user saved this web page, and so maybe you represent that as like a user save model or something, would maybe be the traditional way you do that, and that’s super normalized, that’s super efficient. We didn’t actually really do that, and maybe Dan can get into more why we did this. It’s more of like a product philosophical thing, but we have this model that we just call the document. And everything you save to reader becomes your own document, so it has its own title, its own author, its own URL, its own metadata, it’s own reading progress, all this stuff. So when you save an external web page, we do have like a cloud representation of, say, uh, Ink & Switch blog post. We have the cloud version of that and we say. You know, 10 users have saved that version, but then each of those users gets their own document of that external URL, and of course the document points at the cloud version in our database, but each user gets their own version of that document, which they can then change, they can change the author, they can change the title, they can add highlights to, they can make progress on. It’s largely a product. Decision is largely led by product decisions, but it’s also just a little bit of a denormalized database solution and it’s actually served us quite well, because now we basically allow users to modify those documents however they want. They can go and edit the author title, they can change the reading position. The client is actually agnostic or the server is agnostic. The client can say they want to change any field on this document, it’s just possible. The user is just updating their own version of that document. Now, of course, there’s some complexity there because You have 2 copies. What if the cloud version actually changes? And what if you make an update to your ink