Podcasts about jupyter notebooks

Nabeel Qureshi is an entrepreneur, writer, researcher, and visiting scholar of AI policy at the Mercatus Center (alongside Tyler Cowen). Previously, he spent nearly eight years at Palantir, working as a forward-deployed engineer. His work at Palantir ranged from accelerating the Covid-19 response to applying AI to drug discovery to optimizing aircraft manufacturing at Airbus. Nabeel was also a founding employee and VP of business development at GoCardless, a leading European fintech unicorn.What you'll learn:• Why almost a third of all Palantir's PMs go on to start companies• How the “forward-deployed engineer” model works and why it creates exceptional product leaders• How Palantir transformed from a “sparkling Accenture” into a $200 billion data/software platform company with more than 80% margins• The unconventional hiring approach that screens for independent-minded, intellectually curious, and highly competitive people• Why the company intentionally avoids traditional titles and career ladders—and what they do instead• Why they built an ontology-first data platform that LLMs love• How Palantir's controversial “bat signal” recruiting strategy filtered for specific talent types• The moral case for working at a company like Palantir—Brought to you by:• WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs• Attio—The powerful, flexible CRM for fast-growing startups• OneSchema—Import CSV data 10x faster—Where to find Nabeel S. Qureshi:• X: https://x.com/nabeelqu• LinkedIn: https://www.linkedin.com/in/nabeelqu/• Website: https://nabeelqu.co/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Nabeel S. Qureshi(05:10) Palantir's unique culture and hiring(13:29) What Palantir looks for in people(16:14) Why they don't have titles(19:11) Forward-deployed engineers at Palantir(25:23) Key principles of Palantir's success(30:00) Gotham and Foundry(36:58) The ontology concept(38:02) Life as a forward-deployed engineer(41:36) Balancing custom solutions and product vision(46:36) Advice on how to implement forward-deployed engineers(50:41) The current state of forward-deployed engineers at Palantir(53:15) The power of ingesting, cleaning and analyzing data(59:25) Hiring for mission-driven startups(01:05:30) What makes Palantir PMs different(01:10:00) The moral question of Palantir(01:16:03) Advice for new startups(01:21:12) AI corner(01:24:00) Contrarian corner(01:25:42) Lightning round and final thoughts—Referenced:• Reflections on Palantir: https://nabeelqu.co/reflections-on-palantir• Palantir: https://www.palantir.com/• Intercom: https://www.intercom.com/• Which companies produce the best product managers: https://www.lennysnewsletter.com/p/which-companies-produce-the-best• Gotham: https://www.palantir.com/platforms/gotham/• Foundry: https://www.palantir.com/platforms/foundry/• Peter Thiel on X: https://x.com/peterthiel• Alex Karp: https://en.wikipedia.org/wiki/Alex_Karp• Stephen Cohen: https://en.wikipedia.org/wiki/Stephen_Cohen_(entrepreneur)• Joe Lonsdale on LinkedIn: https://www.linkedin.com/in/jtlonsdale/• Tyler Cowen's website: https://tylercowen.com/• This Scandinavian City Just Won the Internet With Its Hilarious New Tourism Ad: https://www.afar.com/magazine/oslos-new-tourism-ad-becomes-viral-hit• Safe Superintelligence: https://ssi.inc/• Mira Murati on X: https://x.com/miramurati• Stripe: https://stripe.com/• Building product at Stripe: craft, metrics, and customer obsession | Jeff Weinstein (Product lead): https://www.lennysnewsletter.com/p/building-product-at-stripe-jeff-weinstein• Airbus: https://www.airbus.com/en• NIH: https://www.nih.gov/• Jupyter Notebooks: https://jupyter.org/• Shyam Sankar on LinkedIn: https://www.linkedin.com/in/shyamsankar/• Palantir Gotham for Defense Decision Making: https://www.youtube.com/watch?v=rxKghrZU5w8• Foundry 2022 Operating System Demo: https://www.youtube.com/watch?v=uF-GSj-Exms• SQL: https://en.wikipedia.org/wiki/SQL• Airbus A350: https://en.wikipedia.org/wiki/Airbus_A350• SAP: https://www.sap.com/index.html• Barry McCardel on LinkedIn: https://www.linkedin.com/in/barrymccardel/• Understanding ‘Forward Deployed Engineering' and Why Your Company Probably Shouldn't Do It: https://www.barry.ooo/posts/fde-culture• David Hsu on LinkedIn: https://www.linkedin.com/in/dvdhsu/• Retool's Path to Product-Market Fit—Lessons for Getting to 100 Happy Customers, Faster: https://review.firstround.com/retools-path-to-product-market-fit-lessons-for-getting-to-100-happy-customers-faster/• How to foster innovation and big thinking | Eeke de Milliano (Retool, Stripe): https://www.lennysnewsletter.com/p/how-to-foster-innovation-and-big• Looker: https://cloud.google.com/looker• Sorry, that isn't an FDE: https://tedmabrey.substack.com/p/sorry-that-isnt-an-fde• Glean: https://www.glean.com/• Limited Engagement: Is Tech Becoming More Diverse?: https://www.bkmag.com/2017/01/31/limited-engagement-creating-diversity-in-the-tech-industry/• Operation Warp Speed: https://en.wikipedia.org/wiki/Operation_Warp_Speed• Mark Zuckerberg testifies: https://www.businessinsider.com/facebook-ceo-mark-zuckerberg-testifies-congress-libra-cryptocurrency-2019-10• Anduril: https://www.anduril.com/• SpaceX: https://www.spacex.com/• Principles: https://nabeelqu.co/principles• Wispr Flow: https://wisprflow.ai/• Claude code: https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview• Gemini Pro 2.5: https://deepmind.google/technologies/gemini/pro/• DeepMind: https://deepmind.google/• Latent Space newsletter: https://www.latent.space/• Swyx on x: https://x.com/swyx• Neural networks in chess programs: https://www.chessprogramming.org/Neural_Networks• AlphaZero: https://en.wikipedia.org/wiki/AlphaZero• The top chess players in the world: https://www.chess.com/players• Decision to Leave: https://www.imdb.com/title/tt12477480/• Oldboy: https://www.imdb.com/title/tt0364569/• Christopher Alexander: https://en.wikipedia.org/wiki/Christopher_Alexander—Recommended books:• The Technological Republic: Hard Power, Soft Belief, and the Future of the West: https://www.amazon.com/Technological-Republic-Power-Belief-Future/dp/0593798694• Zero to One: Notes on Startups, or How to Build the Future: https://www.amazon.com/Zero-One-Notes-Startups-Future/dp/0804139296• Impro: Improvisation and the Theatre: https://www.amazon.com/Impro-Improvisation-Theatre-Keith-Johnstone/dp/0878301178/• William Shakespeare: Histories: https://www.amazon.com/Histories-Everymans-Library-William-Shakespeare/dp/0679433120/• High Output Management: https://www.amazon.com/High-Output-Management-Andrew-Grove/dp/0679762884• Anna Karenina: https://www.amazon.com/Anna-Karenina-Leo-Tolstoy/dp/0143035002—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.—Lenny may be an investor in the companies discussed. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe

Apologies for the hiatus! Dave needed some time off to recover from burnout, and these episodes remained in the can. Thanks for Waiting for us

AI Arms Race from ChatGPT to Deepseek - AZ TRT S06 EP08 (269) 4-20-2025              What We Learned This Week AI Arms Race is real with the major tech co's involved ChatGPT by OpenAI is considering the top chat AI program Google has Gemini (was Bard), Microsoft has CoPilot, Amazon has Claude / Alexa Deepseek is a startup from China that has disrupted AI landscape with a more cost effective AI model Costs and investment $ dollars into AI is being rethought as Deepseek spent millions $ vs Silicon Valley spending billions $   Notes: Seg 1:   Major Tech Giants AI Programs - Gemini (was Bard) Developed by Google, Gemini is known for its multimodal capabilities and integration with Google Search. It can analyze images, understand verbal prompts, and engage in verbal conversations.  ChatGPT Developed by OpenAI, ChatGPT is known for its versatility and platform-agnostic solution for text generation and learning. It can write code in almost any language, and can also be used to provide research assistance, generate writing prompts, and answer questions.  Microsoft Copilot Developed by Microsoft, Copilot is known for its integration with applications like Word, Excel, and Power BI. It's particularly well-suited for document automation.    Amazon Alexa w/ Claude - Improved AI Model: Claude is a powerful AI model from Anthropic, known for its strengths in natural language processing and conversational AI, as noted in the video and other sources.        Industry 3.0 (1969-2010): The Third Industrial Revolution, or the Digital Revolution, was marked by the automation of production through the use of computers, information technology, and the internet. This era saw the widespread adoption of digital technologies, including programmable logic controllers and robots.  Industry 4.0 (2010-present): The Fourth Industrial Revolution, also known as the Fourth Industrial Revolution, is characterized by the integration of digital technologies, including the Internet of Things (IoT), artificial intelligence (AI), big data, and cyber-physical systems, into manufacturing and industrial processes. This era is focused on creating "smart factories" and "smart products" that can communicate and interact with each other, leading to increased efficiency, customization, and sustainability.    Top AI programs include a range of software, platforms, and resources for learning and working with artificial intelligence. Some of the most popular AI software tools include Viso Suite, ChatGPT, Jupyter Notebooks, and Google Cloud AI Platform, while popular AI platforms include TensorFlow and PyTorch. Educational resources like Coursera's AI Professional Certificate and Fast.ai's practical deep learning course also offer valuable learning opportunities.    ChatGPT is a generative artificial intelligence chatbot developed by OpenAI and launched in 2022. It is based on large language models (LLMs) such as GPT-4o. ChatGPT can generate human-like conversational responses and enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language.[2] It is credited with accelerating the AI boom, which has led to ongoing rapid investment in and public attention to the field of artificial intelligence (AI).[3] Some observers have raised concern about the potential of ChatGPT and similar programs to displace human intelligence, enable plagiarism, or fuel misinformation.[4][5] OpenAI was founded in December 2015 by Sam Altman, Greg Brockman, Elon Musk, Ilya Sutskever, Wojciech Zaremba, and John Schulman. The founding team combined their diverse expertise in technology entrepreneurship, machine learning, and software engineering to create an organization focused on advancing artificial intelligence in a way that benefits humanity. Elon Musk is no longer involved in OpenAI, and Sam Altman is the current CEO of the organization. ChatGPT has had a profound influence on the evolution of AI, paving the way for advancements in natural language understanding and generation. It has demonstrated the effectiveness of transformer-based models for language tasks, which has encouraged other AI researchers to adopt and refine this architecture. The model's success has also stimulated interest in LLMs, leading to a wave of research and development in this area.   Seg 2: DeepSeek is a private Chinese company founded in July 2023 by Liang Wenfeng, a graduate of Zhejiang University, one of China's top universities, who funded the startup via his hedge fund, according to the MIT Technology Review. Liang has about $8 billion in assets, Ives wrote in a Jan. 27 research note. Chinese startup DeepSeek's launch of its latest AI models, which it says are on a par or better than industry-leading models in the United States at a fraction of the cost, is threatening to upset the technology world order. The company has attracted attention in global AI circles after writing in a paper last month that the training of DeepSeek-V3 required less than $6 million worth of computing power from Nvidia H800 chips. DeepSeek's AI Assistant, powered by DeepSeek-V3, has overtaken rival ChatGPT to become the top-rated free application available on Apple's App Store in the United States. This has raised doubts about the reasoning behind some U.S. tech companies' decision to pledge billions of dollars in AI investment and shares of several big tech players, including Nvidia, have been hit.   NVIDIA Blackwell Ultra Enables AI ReasoningThe NVIDIA GB300 NVL72 connects 72 Blackwell Ultra GPUs and 36 Arm Neoverse-based NVIDIA Grace™ CPUs in a rack-scale design, acting as a single massive GPU built for test-time scaling. With the NVIDIA GB300 NVL72, AI models can access the platform's increased compute capacity to explore different solutions to problems and break down complex requests into multiple steps, resulting in higher-quality responses. GB300 NVL72 is also expected to be available on NVIDIA DGX™ Cloud, an end-to-end, fully managed AI platform on leading clouds that optimizes performance with software, services and AI expertise for evolving workloads. NVIDIA DGX SuperPOD™ with DGX GB300 systems uses the GB300 NVL72 rack design to provide customers with a turnkey AI factory. The NVIDIA HGX B300 NVL16 features 11x faster inference on large language models, 7x more compute and 4x larger memory compared with the Hopper generation to deliver breakthrough performance for the most complex workloads, such as AI reasoning.     AZ TRT Shows – related to AI Topic Link: https://brt-show.libsyn.com/size/5/?search=ai+       Biotech Shows: https://brt-show.libsyn.com/category/Biotech-Life+Sciences-Science   AZ Tech Council Shows:  https://brt-show.libsyn.com/size/5/?search=az+tech+council *Includes Best of AZ Tech Council show from 2/12/2023   Tech Topic: https://brt-show.libsyn.com/category/Tech-Startup-VC-Cybersecurity-Energy-Science  Best of Tech: https://brt-show.libsyn.com/size/5/?search=best+of+tech   ‘Best Of' Topic: https://brt-show.libsyn.com/category/Best+of+BRT      Thanks for Listening. Please Subscribe to the AZ TRT Podcast.     AZ Tech Roundtable 2.0 with Matt Battaglia The show where Entrepreneurs, Top Executives, Founders, and Investors come to share insights about the future of business.  AZ TRT 2.0 looks at the new trends in business, & how classic industries are evolving.  Common Topics Discussed: Startups, Founders, Funds & Venture Capital, Business, Entrepreneurship, Biotech, Blockchain / Crypto, Executive Comp, Investing, Stocks, Real Estate + Alternative Investments, and more…    AZ TRT Podcast Home Page: http://aztrtshow.com/ ‘Best Of' AZ TRT Podcast: Click Here Podcast on Google: Click Here Podcast on Spotify: Click Here                    More Info: https://www.economicknight.com/azpodcast/ KFNX Info: https://1100kfnx.com/weekend-featured-shows/     Disclaimer: The views and opinions expressed in this program are those of the Hosts, Guests and Speakers, and do not necessarily reflect the views or positions of any entities they represent (or affiliates, members, managers, employees or partners), or any Station, Podcast Platform, Website or Social Media that this show may air on. All information provided is for educational and entertainment purposes. Nothing said on this program should be considered advice or recommendations in: business, legal, real estate, crypto, tax accounting, investment, etc. Always seek the advice of a professional in all business ventures, including but not limited to: investments, tax, loans, legal, accounting, real estate, crypto, contracts, sales, marketing, other business arrangements, etc.

Thank you to the folks at Sustain (https://sustainoss.org/) for providing the hosting account for CHAOSSCast! CHAOSScast – Episode 106 In this episode of the CHAOSScast, host Alice Sowerby introduces guests Dawn Foster, Cailean Osborne, and Paul Sharratt to discuss the newly formed 'Funding Impact Measurement Working Group' within the CHAOSS community. The panel explores the group's origins, goals, and objectives, emphasizing the importance of mixed method approaches to assess the impact of funding on open source projects. They highlight the significance of both quantitative and qualitative methods to understand the effects of funding better and share insights on creating standardized frameworks for impact assessment. The discussion also touches on the challenges of public versus private funding, the nuances of financial support in open source projects, and the potential benefits of having a collaborative, open forum for related discussions. Hit download now to hear more! [00:02:55] Dawn explains the newly established ‘Funding Impact Measurement Working Group' within the CHAOSS Project including its goals and how often they meet. [00:04:26] Paul describes how the working group was founded after a research paper was written on measuring the impact of public funding on open source and how they presented at Open Forum Academy at Harvard Business School. [00:07:20] Dawn highlights open source funding comes from different sources and more about Microsoft's FOSS Fund and measuring the impact of corporate sponsorship. [00:10:25] Cailean outlines all the core objectives of the working group. [00:13:17] We hear about the working group's first meeting, including members from Digital Infrastructure Insights Fund, and the plan to build a repository of funding models, their effectiveness, and key case studies. [00:15:34] There's a discussion on the challenges in measuring funding impact, which Dawn explains cases where funding has led to community conflicts (some contributors get paid while others remain unpaid). [00:19:45] Cailean talks about the long-term vision for the group which is expanding participation across different funding sources and building open source tools (e.g., Python scripts, Jupyter Notebooks) to support funding impact assessments. [00:21:26] Dawn encourages participation in a working group by contributing in various ways, whether through technical resources, providing insights and experiences related to funding impacts, or simply learning and engaging in discussions. [00:23:42] Paul and Cailean emphasize the need for qualitative research alongside quantitative metrics. Policymakers often seek “hard numbers,” but contextual insights from interviews and case studies are crucial. Value Adds (Picks) of the week: [00:29:12] Cailean's pick is Audrey Tang at RightsCon on her dual meaning of digital in Chinese. [00:30:34] Dawn's pick is Blender for designing 3D models she can print. [00:32:35] Paul's pick is ERROR bug bounty program. [00:33:46] Alice's pick is scrolling for things on eBay like fun sweaters. Panelist: Alice Sowerby Guests: Dawn Foster Cailean Osborne Paul Sharratt Links: CHAOSS (https://chaoss.community/) CHAOSS Project X (https://twitter.com/chaossproj?lang=en) CHAOSScast Podcast (https://podcast.chaoss.community/) CHAOSS Slack (https://chaoss-workspace.slack.com/join/shared_invite/zt-r65szij9-QajX59hkZUct82b0uACA6g#/shared-invite/email) podcast@chaoss.community (mailto:podcast@chaoss.community) CHAOSS Calendar (https://chaoss.community/chaoss-calendar/) Alice Sowerby LinkedIn (https://www.linkedin.com/in/alice-sowerby-ba692a13/?originalSubdomain=uk) Dawn Foster Bluesky (https://bsky.app/profile/geekygirldawn.bsky.social) Cailean Osborne, PhD LinkedIn (https://www.linkedin.com/in/caileanosborne/) Paul Sharratt LinkedIn (https://www.linkedin.com/in/paul-sharratt-887621b3/) Paul Sharratt Bluesky (https://bsky.app/profile/psharratt.bsky.social) Sovereign Tech Agency (https://www.sovereign.tech/) The Linux Foundation Europe (https://linuxfoundation.eu/) Funding Impact Measurement Working Group (https://github.com/chaoss/wg-funding-impact) A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development (Paper) (https://arxiv.org/abs/2411.06027) “Measuring the impact of our investments: introducing a co-authored paper,” by Paul Sharratt (https://www.sovereign.tech/news/measuring-the-impact-of-our-funding) [Audrey Tang](https://en.wikipedia.org/wiki/Audrey_Tang#:~:text=Audrey%20Tang%20(Chinese%3A%20%E5%94%90%E9%B3%B3,ten%20greatest%20Taiwanese%20computing%20personalities%22.) ERROR (https://error.reviews/) Blender (https://www.blender.org/) #1-Dawn designed and printed on Blender (https://bsky.app/profile/geekygirldawn.bsky.social/post/3lika3wlrfk2s) #2-Dawn designed and printed on Blender (https://bsky.app/profile/geekygirldawn.bsky.social/post/3liaa232yws2w) Special Guests: Cailean Osborne and Paul Sharratt.

Neste episódio conjunto do Fronteiras da Engenharia de Software e do Elixir em Foco, Adolfo Neto, Maria Claudia Emer e Zoey Pessanha entrevistaram José Valim, criador da linguagem de programação Elixir. A conversa abordou o tema de boas práticas e anti-padrões (code smells) em Elixir, destacando a importância de pesquisas acadêmicas na área. Adolfo e Valim mencionaram especificamente o trabalho realizado por Lucas Vegi e Marco Tulio Valente, que investigaram code smells na comunidade Elixir, resultando em uma página dedicada a anti-padrões na documentação oficial da linguagem.José Valim ressaltou a escassez de materiais sobre design patterns e refactoring para linguagens funcionais, enfatizando a necessidade de mais estudos e publicações sobre esses temas. Ele explicou que iniciativas como a documentação viva dos anti-padrões ajudam a comunidade a identificar práticas inadequadas e aprimorar continuamente a qualidade do código produzido.Além disso, Valim discutiu brevemente o futuro do Elixir, mencionando projetos recentes como o desenvolvimento do Livebook, ferramenta semelhante ao Jupyter Notebook, e avanços relacionados à tipagem gradual. Ele destacou o potencial da linguagem para sistemas distribuídos e concorrentes, reforçando seu uso crescente por empresas ao redor do mundo. No fim, Valim respondeu qual é a próxima fronteira da Engenharia de Software.José Valim:X (Twitter): https://twitter.com/josevalimLinkedIn: https://www.linkedin.com/in/josevalim/Bluesky: https://bsky.app/profile/josevalim.bsky.socialDashbit: https://dashbit.co/Artigos científicos:The Design Principles of the Elixir Type SystemGiuseppe Castagna, Guillaume Duboc, José Valimhttps://www.irif.fr/_media/users/gduboc/elixir-types.pdfGuard analysis and safe erasure gradual typing: a type system for ElixirGiuseppe Castagna, Guillaume Dubochttps://arxiv.org/abs/2408.14345Links:Ep. Roberto Ierusalimschy (Lua)  https://fronteirases.github.io/episodios/paginas/52 Lua na BEAM https://hexdocs.pm/lua/Lua.htmlEp. Leonardo de Moura (Lean) https://fronteirases.github.io/episodios/paginas/41 Episódio Honey Potion https://www.youtube.com/watch?v=sCV17mv-glE Honey Potion no GitHub https://github.com/lac-dcc/honey-potionTese Lucas Vegi https://repositorio.ufmg.br/handle/1843/80651 Artigos Lucas Vegi e Marco Tulio Valentehttps://scholar.google.com/citations?hl=pt-BR&user=N6KnVK8AAAAJ&view_op=list_works&sortby=pubdateYou have built an Erlang https://vereis.com/posts/you_built_an_erlang Beyond Functional Programming with Elixir and Erlanghttps://blog.plataformatec.com.br/2016/05/beyond-functional-programming-with-elixir-and-erlang/ ChatGPTs para Elixir e Erlang https://gist.github.com/adolfont/a747dcc9cbef002f510b6dbf050695ebErlang Ecosystem Foundation https://erlef.org/ Entrevistas com José Valim https://open.spotify.com/playlist/0L3paiT1aHtYvW8LaM4XUV Talvez o episódio com Bill Gates seja este https://www.bbc.co.uk/programmes/w3ct6pmw Guillaume Duboc https://gldubc.github.io/  PhD student at Université Paris Cité, under the supervision of Giuseppe Castagna https://www.irif.fr/~gc/  Snow Xuejing Huang (pós-doutoranda) https://xsnow.live/ From dynamic to static, Elixir begins its transformationhttps://www.ins2i.cnrs.fr/en/cnrsinfo/dynamic-static-elixir-begins-its-transformation Elixir Type Checker - A (prototype) type checker for Elixir based on set-theoretic type systems.https://typex.fly.dev/ Bringing Types to Elixir by Giuseppe Castagna and Guillaume Duboc | ElixirConf EU 2023https://www.youtube.com/watch?v=gJJH7a2J9O8 Quem é José Valim? Respostas de vários LLMshttps://gist.github.com/adolfont/a95b7e37867cc1b2e24cd0e372727d8cHoney Potion https://www.youtube.com/watch?v=CoFNns01VjARefactorEx https://github.com/gp-pereira/refactorexJido frameworkhttps://github.com/agentjido/jido Fronteiras da Engenharia de Software  https://fronteirases.github.io/ Elixir em Foco https://www.elixiremfoco.com/ 

Neste episódio conjunto do Fronteiras da Engenharia de Software e do Elixir em Foco, Adolfo Neto, Maria Claudia Emer e Zoey Pessanha entrevistaram José Valim, criador da linguagem de programação Elixir. A conversa abordou o tema de boas práticas e anti-padrões (code smells) em Elixir, destacando a importância de pesquisas acadêmicas na área. Adolfo e Valim mencionaram especificamente o trabalho realizado por Lucas Vegi e Marco Tulio Valente, que investigaram code smells na comunidade Elixir, resultando em uma página dedicada a anti-padrões na documentação oficial da linguagem.José Valim ressaltou a escassez de materiais sobre design patterns e refactoring para linguagens funcionais, enfatizando a necessidade de mais estudos e publicações sobre esses temas. Ele explicou que iniciativas como a documentação viva dos anti-padrões ajudam a comunidade a identificar práticas inadequadas e aprimorar continuamente a qualidade do código produzido.Além disso, Valim discutiu brevemente o futuro do Elixir, mencionando projetos recentes como o desenvolvimento do Livebook, ferramenta semelhante ao Jupyter Notebook, e avanços relacionados à tipagem gradual. Ele destacou o potencial da linguagem para sistemas distribuídos e concorrentes, reforçando seu uso crescente por empresas ao redor do mundo. No fim, Valim respondeu qual é a próxima fronteira da Engenharia de Software.José Valim:X (Twitter): https://twitter.com/josevalimLinkedIn: https://www.linkedin.com/in/josevalim/Bluesky: https://bsky.app/profile/josevalim.bsky.socialDashbit: https://dashbit.co/Artigos científicos:The Design Principles of the Elixir Type SystemGiuseppe Castagna, Guillaume Duboc, José Valimhttps://www.irif.fr/_media/users/gduboc/elixir-types.pdfGuard analysis and safe erasure gradual typing: a type system for ElixirGiuseppe Castagna, Guillaume Dubochttps://arxiv.org/abs/2408.14345Links:Ep. Roberto Ierusalimschy (Lua)  https://fronteirases.github.io/episodios/paginas/52 Lua na BEAM https://hexdocs.pm/lua/Lua.htmlEp. Leonardo de Moura (Lean) https://fronteirases.github.io/episodios/paginas/41 Episódio Honey Potion https://www.youtube.com/watch?v=sCV17mv-glE Honey Potion no GitHub https://github.com/lac-dcc/honey-potionTese Lucas Vegi https://repositorio.ufmg.br/handle/1843/80651 Artigos Lucas Vegi e Marco Tulio Valentehttps://scholar.google.com/citations?hl=pt-BR&user=N6KnVK8AAAAJ&view_op=list_works&sortby=pubdateYou have built an Erlang https://vereis.com/posts/you_built_an_erlang Beyond Functional Programming with Elixir and Erlanghttps://blog.plataformatec.com.br/2016/05/beyond-functional-programming-with-elixir-and-erlang/ ChatGPTs para Elixir e Erlang https://gist.github.com/adolfont/a747dcc9cbef002f510b6dbf050695ebErlang Ecosystem Foundation https://erlef.org/ Entrevistas com José Valim https://open.spotify.com/playlist/0L3paiT1aHtYvW8LaM4XUV Talvez o episódio com Bill Gates seja este https://www.bbc.co.uk/programmes/w3ct6pmw Guillaume Duboc https://gldubc.github.io/  PhD student at Université Paris Cité, under the supervision of Giuseppe Castagna https://www.irif.fr/~gc/  Snow Xuejing Huang (pós-doutoranda) https://xsnow.live/ From dynamic to static, Elixir begins its transformationhttps://www.ins2i.cnrs.fr/en/cnrsinfo/dynamic-static-elixir-begins-its-transformation Elixir Type Checker - A (prototype) type checker for Elixir based on set-theoretic type systems.https://typex.fly.dev/ Bringing Types to Elixir by Giuseppe Castagna and Guillaume Duboc | ElixirConf EU 2023https://www.youtube.com/watch?v=gJJH7a2J9O8 Quem é José Valim? Respostas de vários LLMshttps://gist.github.com/adolfont/a95b7e37867cc1b2e24cd0e372727d8cHoney Potion https://www.youtube.com/watch?v=CoFNns01VjARefactorEx https://github.com/gp-pereira/refactorexJido frameworkhttps://github.com/agentjido/jido Fronteiras da Engenharia de Software  https://fronteirases.github.io/ Elixir em Foco https://www.elixiremfoco.com/ 

House Keeping Google / YouTube Update Join the Discord! Feedback Rust in the Linux Kernel. R Stuff What is R Again? Great presentation by John Chambers at UseR! 2006 https://www.r-project.org/conferences/useR-2006/Slides/Chambers.pdf The times have changed, now R is very much suited for production use and not just an academic research language Highly recommend reading Advanced R for more comprehensive details on the quirks of the language https://adv-r.hadley.nz/index.html R VS Python for Data? Different philosophies on the use of the language CRAN vs PyPi Interoperability becoming more mainstream now Visualization: R has always been leaps and bounds ahead (Grammar of Graphics, interactive widgets, etc) R Dev Stack? IDEs: RStudio, now Positron https://positron.posit.co/ Managing package installations with renv https://rstudio.github.io/renv/ Building web apps with Shiny: https://shiny.posit.co/ (I got so engrossed in this space that I created the Shiny Developer Series because of it) Early adopter of using Docker with R in devcontainers with VS-Code. New tech I'm excited about to enhance dev stacks and sharing apps WebAssembly with webR https://docs.r-wasm.org/webr/latest/ Shiny apps in webR? Yes you can https://github.com/RConsortium/submissions-pilot4-webR Managing dev environment combined with Nix: The rix package https://github.com/ropensci/rix (More organized links for show notes) R Language: https://r-project.org Posit (formerly RStudio): https://posit.co RStudio IDE https://posit.co/products/open-source/rstudio/ Positron (still in beta): https://positron.posit.co/ History of S and R presentation by John Chambers at useR! 2006: http://www.r-project.org/user-2006/Slides/Chambers.pdf Advanced R (2nd edition) by Hadley Wickham https://adv-r.hadley.nz/index.html Shiny - Easy interactive web applications with R: https://shiny.posit.co/ renv - Project environments for R: https://rstudio.github.io/renv/ R Markdown: https://rmarkdown.rstudio.com/ WebR - R in the browser: https://docs.r-wasm.org/webr/latest/ Rix - Reproducible Data Science environments for R with Nix: https://github.com/ropensci/rix Chromatic by ModRetro Chromatic: https://modretro.com/products/chromatic-tetris-bundle?variant=47637522579758 FPGA Mike's Review Eric's Thoughts Eric's Socials R Weekly Highlights: https://serve.podhome.fm/r-weekly-highlights Shiny Developer Series: https://shinydevseries.com/ R Podcast: https://r-podcast.org Bluesky: @rpodcast@bsky.social Mastodon: @rpodcast@podcastindex.social LinkedIn: https://www.linkedin.com/in/eric-nantz-6621617/ Coder's Socials Mike on X (https://x.com/dominucco) Mike on BlueSky (https://bsky.app/profile/dominucco.bsky.social) Coder on X (https://x.com/coderradioshow) Coder on BlueSky (https://bsky.app/profile/coderradio.bsky.social) Show Discord (https://discord.gg/k8e7gKUpEp) Alice (https://alice.dev)

Machine Learning Operations (MLOps) mit Data Science Deep Dive.Machine Learning bzw. die Ergebnisse aus Vorhersagen (sogenannten Prediction-Models) sind aus der modernen IT oder gar aus unserem Leben nicht mehr wegzudenken. Solche Modelle kommen wahrscheinlich öfter zum Einsatz, als dir eigentlich bewusst ist. Die Programmierung, Erstellung und das Trainieren dieser Modelle ist die eine Sache. Das Deployment und der Betrieb ist die andere Thematik. Letzteres nennt man Machine Learning Operations, oder kurz “MLOps”. Dies ist das Thema dieser Episode.Wir klären was eigentlich MLOps ist und wie es sich zum klassischen DevOps unterscheidet, wie man das eigene Machine Learning-Modell in Produktion bringt und welche Stages dafür durchlaufen werden müssen, was der Unterschied von Model-Training und Model-Serving ist, welche Aufgabe eine Model-Registry hat, wie man Machine Learning Modelle in Produktion eigentlich monitored und debugged, was Model-Drift bzw. die Drift-Detection ist, ob der Feedback-Cycle durch Methoden wie Continuous Delivery auch kurz gehalten werden kann, aber auch welche Skills als MLOps Engineer wichtig sind.Um all diese Fragen zu beantworten, stehen uns Michelle Golchert und Sebastian Warnholz vom Data Science Deep Dive Podcast rede und Antwort.Unsere aktuellen Werbepartner findest du auf https://engineeringkiosk.dev/partnersDas schnelle Feedback zur Episode:

A Splunk Automation Success Story : In this episode Xander uncovers a project he worked on in a company, using a CSV/Excel file, Python, Jupyter Notebooks via Anaconda. Xander explains what he did and how he did it, and the rewards reached for him, his team and the company. Join me in this amazing journey where I take you on real life examples about what I've done, what I've worked on and how I was able to accomplish this. Remember if you like it, share it with Family and Friends, subscribe, follow, tap the like button. Thank you for listening and thank you for watching. Until next time.

What if the key to overcoming AI and ML integration challenges in enterprises lies with one visionary company? Join us as we chat with Gorkem Ercan, the CTO of Jozu, who is spearheading efforts to revolutionize the MLOps landscape. Gorkem shares his insights on how Jozu's open-source project, KitApps, could be the game-changer in seamlessly packaging AI and ML artifacts. As we enjoy our beers, Gorkem opens up about Jozu's strategic use of the Open Container Initiative (OCI) and their innovative Jozu Hub, which together aim to redefine the AI and ML experiences for enterprises, making such integrations a reality rather than a distant goal.Navigating the complexities of managing large language models (LLMs) and their datasets is no small feat. We explore how Jozu tackles these issues head-on, emphasizing the critical aspects of data versioning, integrity, and security. Discover how custom checksums, data snapshots, and software bills of materials (SBOMs) play a vital role in safeguarding the authenticity and transparency of AI systems. Gorka also highlights the significant advancements Jozu is making in vulnerability scanning and deployment processes, with exciting features like packaging Jupyter Notebooks into microservice containers for easy deployment. Unpack the intricacies of model drift monitoring and the implementation of guardrails, ensuring robust and reliable AI and ML systems that can stand the test of time.More InformationJozuFortify 24x7Fluency SecurityBeers & Bytes PodcastSend us a textSupport the show

OpenAI DevDay is almost here! Per tradition, we are hosting a DevDay pregame event for everyone coming to town! Join us with demos and gossip!Also sign up for related events across San Francisco: the AI DevTools Night, the xAI open house, the Replicate art show, the DevDay Watch Party (for non-attendees), Hack Night with OpenAI at Cloudflare. For everyone else, join the Latent Space Discord for our online watch party and find fellow AI Engineers in your city.OpenAI's recent o1 release (and Reflection 70b debacle) has reignited broad interest in agentic general reasoning and tree search methods.While we have covered some of the self-taught reasoning literature on the Latent Space Paper Club, it is notable that the Eric Zelikman ended up at xAI, whereas OpenAI's hiring of Noam Brown and now Shunyu suggests more interest in tool-using chain of thought/tree of thought/generator-verifier architectures for Level 3 Agents.We were more than delighted to learn that Shunyu is a fellow Latent Space enjoyer, and invited him back (after his first appearance on our NeurIPS 2023 pod) for a look through his academic career with Harrison Chase (one year after his first LS show).ReAct: Synergizing Reasoning and Acting in Language Modelspaper linkFollowing seminal Chain of Thought papers from Wei et al and Kojima et al, and reflecting on lessons from building the WebShop human ecommerce trajectory benchmark, Shunyu's first big hit, the ReAct paper showed that using LLMs to “generate both reasoning traces and task-specific actions in an interleaved manner” achieved remarkably greater performance (less hallucination/error propagation, higher ALFWorld/WebShop benchmark success) than CoT alone. In even better news, ReAct scales fabulously with finetuning:As a member of the elite Princeton NLP group, Shunyu was also a coauthor of the Reflexion paper, which we discuss in this pod.Tree of Thoughtspaper link hereShunyu's next major improvement on the CoT literature was Tree of Thoughts:Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role…ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.The beauty of ToT is it doesnt require pretraining with exotic methods like backspace tokens or other MCTS architectures. You can listen to Shunyu explain ToT in his own words on our NeurIPS pod, but also the ineffable Yannic Kilcher:Other WorkWe don't have the space to summarize the rest of Shunyu's work, you can listen to our pod with him now, and recommend the CoALA paper and his initial hit webinar with Harrison, today's guest cohost:as well as Shunyu's PhD Defense Lecture:as well as Shunyu's latest lecture covering a Brief History of LLM Agents:As usual, we are live on YouTube! Show Notes* Harrison Chase* LangChain, LangSmith, LangGraph* Shunyu Yao* Alec Radford* ReAct Paper* Hotpot QA* Tau Bench* WebShop* SWE-Agent* SWE-Bench* Trees of Thought* CoALA Paper* Related Episodes* Our Thomas Scialom (Meta) episode* Shunyu on our NeurIPS 2023 Best Papers episode* Harrison on our LangChain episode* Mentions* Sierra* Voyager* Jason Wei* Tavily* SERP API* ExaTimestamps* [00:00:00] Opening Song by Suno* [00:03:00] Introductions* [00:06:16] The ReAct paper* [00:12:09] Early applications of ReAct in LangChain* [00:17:15] Discussion of the Reflection paper* [00:22:35] Tree of Thoughts paper and search algorithms in language models* [00:27:21] SWE-Agent and SWE-Bench for coding benchmarks* [00:39:21] CoALA: Cognitive Architectures for Language Agents* [00:45:24] Agent-Computer Interfaces (ACI) and tool design for agents* [00:49:24] Designing frameworks for agents vs humans* [00:53:52] UX design for AI applications and agents* [00:59:53] Data and model improvements for agent capabilities* [01:19:10] TauBench* [01:23:09] Promising areas for AITranscriptAlessio [00:00:01]: Hey, everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO of Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI.Swyx [00:00:12]: Hey, and today we have a super special episode. I actually always wanted to take like a selfie and go like, you know, POV, you're about to revolutionize the world of agents because we have two of the most awesome hiring agents in the house. So first, we're going to welcome back Harrison Chase. Welcome. Excited to be here. What's new with you recently in sort of like the 10, 20 second recap?Harrison [00:00:34]: Linkchain, Linksmith, Lingraph, pushing on all of them. Lots of cool stuff related to a lot of the stuff that we're going to talk about today, probably.Swyx [00:00:42]: Yeah.Alessio [00:00:43]: We'll mention it in there. And the Celtics won the title.Swyx [00:00:45]: And the Celtics won the title. You got that going on for you. I don't know. Is that like floorball? Handball? Baseball? Basketball.Alessio [00:00:52]: Basketball, basketball.Harrison [00:00:53]: Patriots aren't looking good though, so that's...Swyx [00:00:56]: And then Xun Yu, you've also been on the pod, but only in like a sort of oral paper presentation capacity. But welcome officially to the LinkedSpace pod.Shunyu [00:01:03]: Yeah, I've been a huge fan. So thanks for the invitation. Thanks.Swyx [00:01:07]: Well, it's an honor to have you on. You're one of like, you're maybe the first PhD thesis defense I've ever watched in like this AI world, because most people just publish single papers, but every paper of yours is a banger. So congrats.Shunyu [00:01:22]: Thanks.Swyx [00:01:24]: Yeah, maybe we'll just kick it off with, you know, what was your journey into using language models for agents? I like that your thesis advisor, I didn't catch his name, but he was like, you know... Karthik. Yeah. It's like, this guy just wanted to use language models and it was such a controversial pick at the time. Right.Shunyu [00:01:39]: The full story is that in undergrad, I did some computer vision research and that's how I got into AI. But at the time, I feel like, you know, you're just composing all the GAN or 3D perception or whatever together and it's not exciting anymore. And one day I just see this transformer paper and that's really cool. But I really got into language model only when I entered my PhD and met my advisor Karthik. So he was actually the second author of GPT-1 when he was like a visiting scientist at OpenAI. With Alec Redford?Swyx [00:02:10]: Yes.Shunyu [00:02:11]: Wow. That's what he told me. It's like back in OpenAI, they did this GPT-1 together and Ilya just said, Karthik, you should stay because we just solved the language. But apparently Karthik is not fully convinced. So he went to Princeton, started his professorship and I'm really grateful. So he accepted me as a student, even though I have no prior knowledge in NLP. And you know, we just met for the first time and he's like, you know, what do you want to do? And I'm like, you know, you have done those test game scenes. That's really cool. I wonder if we can just redo them with language models. And that's how the whole journey began. Awesome.Alessio [00:02:46]: So GPT-2 was out at the time? Yes, that was 2019.Shunyu [00:02:48]: Yeah.Alessio [00:02:49]: Way too dangerous to release. And then I guess the first work of yours that I came across was React, which was a big part of your defense. But also Harrison, when you came on The Pockets last year, you said that was one of the first papers that you saw when you were getting inspired for BlankChain. So maybe give a recap of why you thought it was cool, because you were already working in AI and machine learning. And then, yeah, you can kind of like intro the paper formally. What was that interesting to you specifically?Harrison [00:03:16]: Yeah, I mean, I think the interesting part was using these language models to interact with the outside world in some form. And I think in the paper, you mostly deal with Wikipedia. And I think there's some other data sets as well. But the outside world is the outside world. And so interacting with things that weren't present in the LLM and APIs and calling into them and thinking about the React reasoning and acting and kind of like combining those together and getting better results. I'd been playing around with LLMs, been talking with people who were playing around with LLMs. People were trying to get LLMs to call into APIs, do things, and it was always, how can they do it more reliably and better? And so this paper was basically a step in that direction. And I think really interesting and also really general as well. Like I think that's part of the appeal is just how general and simple in a good way, I think the idea was. So that it was really appealing for all those reasons.Shunyu [00:04:07]: Simple is always good. Yeah.Alessio [00:04:09]: Do you have a favorite part? Because I have one favorite part from your PhD defense, which I didn't understand when I read the paper, but you said something along the lines, React doesn't change the outside or the environment, but it does change the insight through the context, putting more things in the context. You're not actually changing any of the tools around you to work for you, but you're changing how the model thinks. And I think that was like a very profound thing when I, not that I've been using these tools for like 18 months. I'm like, I understand what you meant, but like to say that at the time you did the PhD defense was not trivial. Yeah.Shunyu [00:04:41]: Another way to put it is like thinking can be an extra tool that's useful.Alessio [00:04:47]: Makes sense. Checks out.Swyx [00:04:49]: Who would have thought? I think it's also more controversial within his world because everyone was trying to use RL for agents. And this is like the first kind of zero gradient type approach. Yeah.Shunyu [00:05:01]: I think the bigger kind of historical context is that we have this two big branches of AI. So if you think about RL, right, that's pretty much the equivalent of agent at a time. And it's like agent is equivalent to reinforcement learning and reinforcement learning is equivalent to whatever game environment they're using, right? Atari game or go or whatever. So you have like a pretty much, you know, you have a biased kind of like set of methodologies in terms of reinforcement learning and represents agents. On the other hand, I think NLP is like a historical kind of subject. It's not really into agents, right? It's more about reasoning. It's more about solving those concrete tasks. And if you look at SEL, right, like each task has its own track, right? Summarization has a track, question answering has a track. So I think really it's about rethinking agents in terms of what could be the new environments that we came to have is not just Atari games or whatever video games, but also those text games or language games. And also thinking about, could there be like a more general kind of methodology beyond just designing specific pipelines for each NLP task? That's like the bigger kind of context, I would say.Alessio [00:06:14]: Is there an inspiration spark moment that you remember or how did you come to this? We had Trida on the podcast and he mentioned he was really inspired working with like systems people to think about Flash Attention. What was your inspiration journey?Shunyu [00:06:27]: So actually before React, I spent the first two years of my PhD focusing on text-based games, or in other words, text adventure games. It's a very kind of small kind of research area and quite ad hoc, I would say. And there are like, I don't know, like 10 people working on that at the time. And have you guys heard of Zork 1, for example? So basically the idea is you have this game and you have text observations, like you see a monster, you see a dragon.Swyx [00:06:57]: You're eaten by a grue.Shunyu [00:06:58]: Yeah, you're eaten by a grue. And you have actions like kill the grue with a sword or whatever. And that's like a very typical setup of a text game. So I think one day after I've seen all the GPT-3 stuff, I just think about, you know, how can I solve the game? Like why those AI, you know, machine learning methods are pretty stupid, but we are pretty good at solving the game relatively, right? So for the context, the predominant method to solve this text game is obviously reinforcement learning. And the idea is you just try out an arrow in those games for like millions of steps and you kind of just overfit to the game. But there's no language understanding at all. And I'm like, why can't I solve the game better? And it's kind of like, because we think about the game, right? Like when we see this very complex text observation, like you see a grue and you might see a sword, you know, in the right of the room and you have to go through the wooden door to go to that room. You will think, you know, oh, I have to kill the monster and to kill that monster, I have to get the sword, I have to get the sword, I have to go, right? And this kind of thinking actually helps us kind of throw shots off the game. And it's like, why don't we also enable the text agents to think? And that's kind of the prototype of React. And I think that's actually very interesting because the prototype, I think, was around November of 2021. So that's even before like chain of thought or whatever came up. So we did a bunch of experiments in the text game, but it was not really working that well. Like those text games are just too hard. I think today it's still very hard. Like if you use GPD 4 to solve it, it's still very hard. So the change came when I started the internship in Google. And apparently Google care less about text game, they care more about what's more practical. So pretty much I just reapplied the idea, but to more practical kind of environments like Wikipedia or simpler text games like Alphard, and it just worked. It's kind of like you first have the idea and then you try to find the domains and the problems to demonstrate the idea, which is, I would say, different from most of the AI research, but it kind of worked out for me in that case.Swyx [00:09:09]: For Harrison, when you were implementing React, what were people applying React to in the early days?Harrison [00:09:14]: I think the first demo we did probably had like a calculator tool and a search tool. So like general things, we tried to make it pretty easy to write your own tools and plug in your own things. And so this is one of the things that we've seen in LangChain is people who build their own applications generally write their own tools. Like there are a few common ones. I'd say like the three common ones might be like a browser, a search tool, and a code interpreter. But then other than that-Swyx [00:09:37]: The LMS. Yep.Harrison [00:09:39]: Yeah, exactly. It matches up very nice with that. And we actually just redid like our integrations docs page, and if you go to the tool section, they like highlight those three, and then there's a bunch of like other ones. And there's such a long tail of other ones. But in practice, like when people go to production, they generally have their own tools or maybe one of those three, maybe some other ones, but like very, very few other ones. So yeah, I think the first demos was a search and a calculator one. And there's- What's the data set?Shunyu [00:10:04]: Hotpot QA.Harrison [00:10:05]: Yeah. Oh, so there's that one. And then there's like the celebrity one by the same author, I think.Swyx [00:10:09]: Olivier Wilde's boyfriend squared. Yeah. 0.23. Yeah. Right, right, right.Harrison [00:10:16]: I'm forgetting the name of the author, but there's-Swyx [00:10:17]: I was like, we're going to over-optimize for Olivier Wilde's boyfriend, and it's going to change next year or something.Harrison [00:10:21]: There's a few data sets kind of like in that vein that require multi-step kind of like reasoning and thinking. So one of the questions I actually had for you in this vein, like the React paper, there's a few things in there, or at least when I think of that, there's a few things that I think of. There's kind of like the specific prompting strategy. Then there's like this general idea of kind of like thinking and then taking an action. And then there's just even more general idea of just like taking actions in a loop. Today, like obviously language models have changed a lot. We have tool calling. The specific prompting strategy probably isn't used super heavily anymore. Would you say that like the concept of React is still used though? Or like do you think that tool calling and running tool calling in a loop, is that ReactSwyx [00:11:02]: in your mind?Shunyu [00:11:03]: I would say like it's like more implicitly used than explicitly used. To be fair, I think the contribution of React is actually twofold. So first is this idea of, you know, we should be able to use calls in a very general way. Like there should be a single kind of general method to handle interaction with various environments. I think React is the first paper to demonstrate the idea. But then I think later there are two form or whatever, and this becomes like a trivial idea. But I think at the time, that's like a pretty non-trivial thing. And I think the second contribution is this idea of what people call like inner monologue or thinking or reasoning or whatever, to be paired with tool use. I think that's still non-trivial because if you look at the default function calling or whatever, like there's no inner monologue. And in practice, that actually is important, especially if the tool that you use is pretty different from the training distribution of the language model. I think those are the two main things that are kind of inherited.Harrison [00:12:10]: On that note, I think OpenAI even recommended when you're doing tool calling, it's sometimes helpful to put a thought field in the tool, along with all the actual acquired arguments,Swyx [00:12:19]: and then have that one first.Harrison [00:12:20]: So it fills out that first, and they've shown that that's yielded better results. The reason I ask is just like this same concept is still alive, and I don't know whether to call it a React agent or not. I don't know what to call it. I think of it as React, like it's the same ideas that were in the paper, but it's obviously a very different implementation at this point in time. And so I just don't know what to call it.Shunyu [00:12:40]: I feel like people will sometimes think more in terms of different tools, right? Because if you think about a web agent versus, you know, like a function calling agent, calling a Python API, you would think of them as very different. But in some sense, the methodology is the same. It depends on how you view them, right? I think people will tend to think more in terms of the environment and the tools rather than the methodology. Or, in other words, I think the methodology is kind of trivial and simple, so people will try to focus more on the different tools. But I think it's good to have a single underlying principle of those things.Alessio [00:13:17]: How do you see the surface of React getting molded into the model? So a function calling is a good example of like, now the model does it. What about the thinking? Now most models that you use kind of do chain of thought on their own, they kind of produce steps. Do you think that more and more of this logic will be in the model? Or do you think the context window will still be the main driver of reasoning and thinking?Shunyu [00:13:39]: I think it's already default, right? You do some chain of thought and you do some tool call, the cost of adding the chain of thought is kind of relatively low compared to other things. So it's not hurting to do that. And I think it's already kind of common practice, I would say.Swyx [00:13:56]: This is a good place to bring in either Tree of Thought or Reflection, your pick.Shunyu [00:14:01]: Maybe Reflection, to respect the time order, I would say.Swyx [00:14:05]: Any backstory as well, like the people involved with NOAA and the Princeton group. We talked about this offline, but people don't understand how these research pieces come together and this ideation.Shunyu [00:14:15]: I think Reflection is mostly NOAA's work, I'm more like advising kind of role. The story is, I don't remember the time, but one day we just see this pre-print that's like Reflection and Autonomous Agent with memory or whatever. And it's kind of like an extension to React, which uses this self-reflection. I'm like, oh, somehow you've become very popular. And NOAA reached out to me, it's like, do you want to collaborate on this and make this from an archive pre-print to something more solid, like a conference submission? I'm like, sure. We started collaborating and we remain good friends today. And I think another interesting backstory is NOAA was contacted by OpenAI at the time. It's like, this is pretty cool, do you want to just work at OpenAI? And I think Sierra also reached out at the same time. It's like, this is pretty cool, do you want to work at Sierra? And I think NOAA chose Sierra, but it's pretty cool because he was still like a second year undergrad and he's a very smart kid.Swyx [00:15:16]: Based on one paper. Oh my god.Shunyu [00:15:19]: He's done some other research based on programming language or chemistry or whatever, but I think that's the paper that got the attention of OpenAI and Sierra.Swyx [00:15:28]: For those who haven't gone too deep on it, the way that you present the inside of React, can you do that also for reflection? Yeah.Shunyu [00:15:35]: I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward and then you somehow back-propagate the signal of the scalar reward to the rest of your neural network through whatever algorithm, like policy grading or A2C or whatever. And if you think about the real life, most of the reward signal is not scalar. It's like your boss told you, you should have done a better job in this, but you could jump on that or whatever. It's not like a scalar reward, like 29 or something. I think in general, humans deal more with long scalar reward, or you can say language feedback. And the way that they deal with language feedback also has this back-propagation process, right? Because you start from this, you did a good job on job B, and then you reflect what could have been done differently to change to make it better. And you kind of change your prompt, right? Basically, you change your prompt on how to do job A and how to do job B, and then you do the whole thing again. So it's really like a pipeline of language where in self-graded descent, you have something like text reasoning to replace those gradient descent algorithms. I think that's one way to think of reflection.Harrison [00:16:47]: One question I have about reflection is how general do you think the algorithm there is? And so for context, I think at LangChain and at other places as well, we found it pretty easy to implement React in a standard way. You plug in any tools and it kind of works off the shelf, can get it up and running. I don't think we have an off-the-shelf kind of implementation of reflection and kind of the general sense. I think the concepts, absolutely, we see used in different kind of specific cognitive architectures, but I don't think we have one that comes off the shelf. I don't think any of the other frameworks have one that comes off the shelf. And I'm curious whether that's because it's not general enough or it's complex as well, because it also requires running it more times.Swyx [00:17:28]: Maybe that's not feasible.Harrison [00:17:30]: I'm curious how you think about the generality, complexity. Should we have one that comes off the shelf?Shunyu [00:17:36]: I think the algorithm is general in the sense that it's just as general as other algorithms, if you think about policy grading or whatever, but it's not applicable to all tasks, just like other algorithms. So you can argue PPO is also general, but it works better for those set of tasks, but not on those set of tasks. I think it's the same situation for reflection. And I think a key bottleneck is the evaluator, right? Basically, you need to have a good sense of the signal. So for example, if you are trying to do a very hard reasoning task, say mathematics, for example, and you don't have any tools, you're operating in this chain of thought setup, then reflection will be pretty hard because in order to reflect upon your thoughts, you have to have a very good evaluator to judge whether your thought is good or not. But that might be as hard as solving the problem itself or even harder. The principle of self-reflection is probably more applicable if you have a good evaluator, for example, in the case of coding. If you have those arrows, then you can just reflect on that and how to solve the bug andSwyx [00:18:37]: stuff.Shunyu [00:18:38]: So I think another criteria is that it depends on the application, right? If you have this latency or whatever need for an actual application with an end-user, the end-user wouldn't let you do two hours of tree-of-thought or reflection, right? You need something as soon as possible. So in that case, maybe this is better to be used as a training time technique, right? You do those reflection or tree-of-thought or whatever, you get a lot of data, and then you try to use the data to train your model better. And then in test time, you still use something as simple as React, but that's already improved.Alessio [00:19:11]: And if you think of the Voyager paper as a way to store skills and then reuse them, how would you compare this reflective memory and at what point it's just ragging on the memory versus you want to start to fine-tune some of them or what's the next step once you get a very long reflective corpus? Yeah.Shunyu [00:19:30]: So I think there are two questions here. The first question is, what type of information or memory are you considering, right? Is it like semantic memory that stores knowledge about the word, or is it the episodic memory that stores trajectories or behaviors, or is it more of a procedural memory like in Voyager's case, like skills or code snippets that you can use to do actions, right?Swyx [00:19:54]: That's one dimension.Shunyu [00:19:55]: And the second dimension is obviously how you use the memory, either retrieving from it, using it in the context, or fine-tuning it. I think the Cognitive Architecture for Language Agents paper has a good categorization of all the different combinations. And of course, which way you use depends on the concrete application and the concrete need and the concrete task. But I think in general, it's good to think of those systematic dimensions and all the possible options there.Swyx [00:20:25]: Harrison also has in LangMEM, I think you did a presentation in my meetup, and I think you've done it at a couple other venues as well. User state, semantic memory, and append-only state, I think kind of maps to what you just said.Shunyu [00:20:38]: What is LangMEM? Can I give it like a quick...Harrison [00:20:40]: One of the modules of LangChain for a long time has been something around memory. And I think we're still obviously figuring out what that means, as is everyone kind of in the space. But one of the experiments that we did, and one of the proof of concepts that we did was, technically what it was is you would basically create threads, you'd push messages to those threads in the background, we process the data in a few ways. One, we put it into some semantic store, that's the semantic memory. And then two, we do some extraction and reasoning over the memories to extract. And we let the user define this, but extract key facts or anything that's of interest to the user. Those aren't exactly trajectories, they're maybe more closer to the procedural memory. Is that how you'd think about it or classify it?Shunyu [00:21:22]: Is it like about knowledge about the word, or is it more like how to do something?Swyx [00:21:27]: It's reflections, basically.Harrison [00:21:28]: So in generative worlds.Shunyu [00:21:30]: Generative agents.Swyx [00:21:31]: The Smallville. Yeah, the Smallville one.Harrison [00:21:33]: So the way that they had their memory there was they had the sequence of events, and that's kind of like the raw events that happened. But then every N events, they'd run some synthesis over those events for the LLM to insert its own memory, basically. It's that type of memory.Swyx [00:21:49]: I don't know how that would be classified.Shunyu [00:21:50]: I think of that as more of the semantic memory, but to be fair, I think it's just one way to think of that. But whether it's semantic memory or procedural memory or whatever memory, that's like an abstraction layer. But in terms of implementation, you can choose whatever implementation for whatever memory. So they're totally kind of orthogonal. I think it's more of a good way to think of the things, because from the history of cognitive science and cognitive architecture and how people study even neuroscience, that's the way people think of how the human brain organizes memory. And I think it's more useful as a way to think of things. But it's not like for semantic memory, you have to do this kind of way to retrieve or fine-tune, and for procedural memory, you have to do that. I think those are totally orthogonal kind of dimensions.Harrison [00:22:34]: How much background do you have in cognitive sciences, and how much do you model some of your thoughts on?Shunyu [00:22:40]: That's a great question, actually. I think one of the undergrad influences for my follow-up research is I was doing an internship at MIT's Computational Cognitive Science Lab with Josh Tannenbaum, and he's a very famous cognitive scientist. And I think a lot of his ideas still influence me today, like thinking of things in computational terms and getting interested in language and a lot of stuff, or even developing psychology kind of stuff. So I think it still influences me today.Swyx [00:23:14]: As a developer that tried out LangMEM, the way I view it is just it's a materialized view of a stream of logs. And if anything, that's just useful for context compression. I don't have to use the full context to run it over everything. But also it's kind of debuggable. If it's wrong, I can show it to the user, the user can manually fix it, and I can carry on. That's a really good analogy. I like that. I'm going to steal that. Sure. Please, please. You know I'm bullish on memory databases. I guess, Tree of Thoughts? Yeah, Tree of Thoughts.Shunyu [00:23:39]: I feel like I'm relieving the defense in like a podcast format. Yeah, no.Alessio [00:23:45]: I mean, you had a banger. Well, this is the one where you're already successful and we just highlight the glory. It was really good. You mentioned that since thinking is kind of like taking an action, you can use action searching algorithms to think of thinking. So just like you will use Tree Search to find the next thing. And the idea behind Tree of Thought is that you generate all these possible outcomes and then find the best tree to get to the end. Maybe back to the latency question, you can't really do that if you have to respond in real time. So what are maybe some of the most helpful use cases for things like this? Where have you seen people adopt it where the high latency is actually worth the wait?Shunyu [00:24:21]: For things that you don't care about latency, obviously. For example, if you're trying to do math, if you're just trying to come up with a proof. But I feel like one type of task is more about searching for a solution. You can try a hundred times, but if you find one solution, that's good. For example, if you're finding a math proof or if you're finding a good code to solve a problem or whatever, I think another type of task is more like reacting. For example, if you're doing customer service, you're like a web agent booking a ticket for an end user. Those are more reactive kind of tasks, or more real-time tasks. You have to do things fast. They might be easy, but you have to do it reliably. And you care more about can you solve 99% of the time out of a hundred. But for the type of search type of tasks, then you care more about can I find one solution out of a hundred. So it's kind of symmetric and different.Alessio [00:25:11]: Do you have any data or intuition from your user base? What's the split of these type of use cases? How many people are doing more reactive things and how many people are experimenting with deep, long search?Harrison [00:25:23]: I would say React's probably the most popular. I think there's aspects of reflection that get used. Tree of thought, probably the least so. There's a great tweet from Jason Wei, I think you're now a colleague, and he was talking about prompting strategies and how he thinks about them. And I think the four things that he had was, one, how easy is it to implement? How much compute does it take? How many tasks does it solve? And how much does it improve on those tasks? And I'd add a fifth, which is how likely is it to be relevant when the next generation of models come out? And I think if you look at those axes and then you look at React, reflection, tree of thought, it tracks that the ones that score better are used more. React is pretty easy to implement. Tree of thought's pretty hard to implement. The amount of compute, yeah, a lot more for tree of thought. The tasks and how much it improves, I don't have amazing visibility there. But I think if we're comparing React versus tree of thought, React just dominates the first two axes so much that my question around that was going to be like, how do you think about these prompting strategies, cognitive architectures, whatever you want to call them? When you're thinking of them, what are the axes that you're judging them on in your head when you're thinking whether it's a good one or a less good one?Swyx [00:26:38]: Right.Shunyu [00:26:39]: Right. I think there is a difference between a prompting method versus research, in the sense that for research, you don't really even care about does it actually work on practical tasks or does it help? Whatever. I think it's more about the idea or the principle, right? What is the direction that you're unblocking and whatever. And I think for an actual prompting method to solve a concrete problem, I would say simplicity is very important because the simpler it is, the less decision you have to make about it. And it's easier to design. It's easier to propagate. And it's easier to do stuff. So always try to be as simple as possible. And I think latency obviously is important. If you can do things fast and you don't want to do things slow. And I think in terms of the actual prompting method to use for a particular problem, I think we should all be in the minimalist kind of camp, right? You should try the minimum thing and see if it works. And if it doesn't work and there's absolute reason to add something, then you add something, right? If there's absolute reason that you need some tool, then you should add the tool thing. If there's absolute reason to add reflection or whatever, you should add that. Otherwise, if a chain of thought can already solve something, then you don't even need to use any of that.Harrison [00:27:57]: Yeah. Or if it's just better prompting can solve it. Like, you know, you could add a reflection step or you could make your instructions a little bit clearer.Swyx [00:28:03]: And it's a lot easier to do that.Shunyu [00:28:04]: I think another interesting thing is like, I personally have never done those kind of like weird tricks. I think all the prompts that I write are kind of like just talking to a human, right? It's like, I don't know. I never say something like, your grandma is dying and you have to solve it. I mean, those are cool, but I feel like we should all try to solve things in a very intuitive way. Just like talking to your co-worker. That should work 99% of the time. That's my personal take.Swyx [00:28:29]: The problem with how language models, at least in the GPC 3 era, was that they over-optimized to some sets of tokens in sequence. So like reading the Kojima et al. paper that was listing step-by-step, like he tried a bunch of them and they had wildly different results. It should not be the case, but it is the case. And hopefully we're getting better there.Shunyu [00:28:51]: Yeah. I think it's also like a timing thing in the sense that if you think about this whole line of language model, right? Like at the time it was just like a text generator. We don't have any idea how it's going to be used, right? And obviously at the time you will find all kinds of weird issues because it's not trained to do any of that, right? But then I think we have this loop where once we realize chain of thought is important or agent is important or tool using is important, what we see is today's language models are heavily optimized towards those things. So I think in some sense they become more reliable and robust over those use cases. And you don't need to do as much prompt engineering tricks anymore to solve those things. I feel like in some sense, I feel like prompt engineering even is like a slightly negative word at the time because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progress, you should just be able to talk to like a coworker. And if you're clear and concrete and being reasonable, then it should do reasonable things for you.Swyx [00:29:51]: Yeah. The way I put this is you should not be a prompt engineer because it is the goal of the big labs to put you out of a job.Shunyu [00:29:58]: You should just be a good communicator. Like if you're a good communicator to humans, you should be a good communicator to languageSwyx [00:30:02]: models.Harrison [00:30:03]: That's the key though, because oftentimes people aren't good communicators to these language models and that is a very important skill and that's still messing around with the prompt. And so it depends what you're talking about when you're saying prompt engineer.Shunyu [00:30:14]: But do you think it's like very correlated with like, are they like a good communicator to humans? You know, it's like.Harrison [00:30:20]: It may be, but I also think I would say on average, people are probably worse at communicating with language models than to humans right now, at least, because I think we're still figuring out how to do it. You kind of expect it to be magical and there's probably some correlation, but I'd say there's also just like, people are worse at it right now than talking to humans.Shunyu [00:30:36]: We should make it like a, you know, like an elementary school class or whatever, how toSwyx [00:30:41]: talk to language models. Yeah. I don't know. Very pro that. Yeah. Before we leave the topic of trees and searching, not specific about QSTAR, but there's a lot of questions about MCTS and this combination of tree search and language models. And I just had to get in a question there about how seriously should people take this?Shunyu [00:30:59]: Again, I think it depends on the tasks, right? So MCTS was magical for Go, but it's probably not as magical for robotics, right? So I think right now the problem is not even that we don't have good methodologies, it's more about we don't have good tasks. It's also very interesting, right? Because if you look at my citation, it's like, obviously the most cited are React, Refraction and Tree of Thought. Those are methodologies. But I think like equally important, if not more important line of my work is like benchmarks and environments, right? Like WebShop or SuiteVenture or whatever. And I think in general, what people do in academia that I think is not good is they choose a very simple task, like Alford, and then they apply overly complex methods to show they improve 2%. I think you should probably match the level of complexity of your task and your method. I feel like where tasks are kind of far behind the method in some sense, right? Because we have some good test-time approaches, like whatever, React or Refraction or Tree of Thought, or like there are many, many more complicated test-time methods afterwards. But on the benchmark side, we have made a lot of good progress this year, last year. But I think we still need more progress towards that, like better coding benchmark, better web agent benchmark, better agent benchmark, not even for web or code. I think in general, we need to catch up with tasks.Harrison [00:32:27]: What are the biggest reasons in your mind why it lags behind?Shunyu [00:32:31]: I think incentive is one big reason. Like if you see, you know, all the master paper are cited like a hundred times more than the task paper. And also making a good benchmark is actually quite hard. It's almost like a different set of skills in some sense, right? I feel like if you want to build a good benchmark, you need to be like a good kind of product manager kind of mindset, right? You need to think about why people should use your benchmark, why it's challenging, why it's useful. If you think about like a PhD going into like a school, right? The prior skill that expected to have is more about, you know, can they code this method and can they just run experiments and can solve that? I think building a benchmark is not the typical prior skill that we have, but I think things are getting better. I think more and more people are starting to build benchmarks and people are saying that it's like a way to get more impact in some sense, right? Because like if you have a really good benchmark, a lot of people are going to use it. But if you have a super complicated test time method, like it's very hard for people to use it.Harrison [00:33:35]: Are evaluation metrics also part of the reason? Like for some of these tasks that we might want to ask these agents or language models to do, is it hard to evaluate them? And so it's hard to get an automated benchmark. Obviously with SweetBench you can, and with coding, it's easier, but.Shunyu [00:33:50]: I think that's part of the skillset thing that I mentioned, because I feel like it's like a product manager because there are many dimensions and you need to strike a balance and it's really hard, right? If you want to make sense, very easy to autogradable, like automatically gradable, like either to grade or either to evaluate, then you might lose some of the realness or practicality. Or like it might be practical, but it might not be as scalable, right? For example, if you think about text game, human have pre-annotated all the rewards and all the language are real. So it's pretty good on autogradable dimension and the practical dimension. If you think about, you know, practical, like actual English being practical, but it's not scalable, right? It takes like a year for experts to build that game. So it's not really that scalable. And I think part of the reason that SweetBench is so popular now is it kind of hits the balance between these three dimensions, right? Easy to evaluate and being actually practical and being scalable. Like if I were to criticize upon some of my prior work, I think webshop, like it's my initial attempt to get into benchmark world and I'm trying to do a good job striking the balance. But obviously we make it all gradable and it's really scalable, but then I think the practicality is not as high as actually just using GitHub issues, right? Because you're just creating those like synthetic tasks.Harrison [00:35:13]: Are there other areas besides coding that jump to mind as being really good for being autogradable?Shunyu [00:35:20]: Maybe mathematics.Swyx [00:35:21]: Classic. Yeah. Do you have thoughts on alpha proof, the new DeepMind paper? I think it's pretty cool.Shunyu [00:35:29]: I think it's more of a, you know, it's more of like a confidence boost or like sometimes, you know, the work is not even about, you know, the technical details or the methodology that it chooses or the concrete results. I think it's more about a signal, right?Swyx [00:35:47]: Yeah. Existence proof. Yeah.Shunyu [00:35:50]: Yeah. It can be done. This direction is exciting. It kind of encourages people to work more towards that direction. I think it's more like a boost of confidence, I would say.Swyx [00:35:59]: Yeah. So we're going to focus more on agents now and, you know, all of us have a special interest in coding agents. I would consider Devin to be the sort of biggest launch of the year as far as AI startups go. And you guys in the Princeton group worked on Suiagents alongside of Suibench. Tell us the story about Suiagent. Sure.Shunyu [00:36:21]: I think it's kind of like a triology, it's actually a series of three works now. So actually the first work is called Intercode, but it's not as famous, I know. And the second work is called Suibench and the third work is called Suiagent. And I'm just really confused why nobody is working on coding. You know, it's like a year ago, but I mean, not everybody's working on coding, obviously, but a year ago, like literally nobody was working on coding. I was really confused. And the people that were working on coding are, you know, trying to solve human evil in like a sick-to-sick way. There's no agent, there's no chain of thought, there's no anything, they're just, you know, fine tuning the model and improve some points and whatever, like, I was really confused because obviously coding is the best application for agents because it's autogradable, it's super important, you can make everything like API or code action, right? So I was confused and I collaborated with some of the students in Princeton and we have this work called Intercode and the idea is, first, if you care about coding, then you should solve coding in an interactive way, meaning more like a Jupyter Notebook kind of way than just writing a program and seeing if it fails or succeeds and stop, right? You should solve it in an interactive way because that's exactly how humans solve it, right? You don't have to, you know, write a program like next token, next token, next token and stop and never do any edits and you cannot really use any terminal or whatever tool. It doesn't make sense, right? And that's the way people are solving coding at the time, basically like sampling a program from a language model without chain of thought, without tool call, without refactoring, without anything. So the first point is we should solve coding in a very interactive way and that's a very general principle that applies for various coding benchmarks. And also, I think you can make a lot of the agent task kind of like interactive coding. If you have Python and you can call any package, then you can literally also browse internet or do whatever you want, like control a robot or whatever. So that seems to be a very general paradigm. But obviously I think a bottleneck is at the time we're still doing, you know, very simple tasks like human eval or whatever coding benchmark people proposed. They were super hard in 2021, like 20%, but they're like 95% already in 2023. So obviously the next step is we need a better benchmark. And Carlos and John, which are the first authors of Swaybench, I think they come up with this great idea that we should just script GitHub and solve whatever human engineers are solving. And I think it's actually pretty easy to come up with the idea. And I think in the first week, they already made a lot of progress. They script the GitHub and they make all the same, but then there's a lot of painful info work and whatever, you know. I think the idea is super easy, but the engineering is super hard. And I feel like that's a very typical signal of a good work in the AI era now.Swyx [00:39:17]: I think also, I think the filtering was challenging, because if you look at open source PRs, a lot of them are just like, you know, fixing typos. I think it's challenging.Shunyu [00:39:27]: And to be honest, we didn't do a perfect job at the time. So if you look at the recent blog post with OpenAI, we improved the filtering so that it's more solvable.Swyx [00:39:36]: I think OpenAI was just like, look, this is a thing now. We have to fix this. These students just rushed it.Shunyu [00:39:45]: It's a good convergence of interests for me.Alessio [00:39:48]: Was that tied to you joining OpenAI? Or was that just unrelated?Shunyu [00:39:52]: It's a coincidence for me, but it's a good coincidence.Swyx [00:39:55]: There is a history of anytime a big lab adopts a benchmark, they fix it. Otherwise, it's a broken benchmark.Shunyu [00:40:03]: So naturally, once we propose swimmage, the next step is to solve it. But I think the typical way you solve something now is you collect some training samples, or you design some complicated agent method, and then you try to solve it. Either super complicated prompt, or you build a better model with more training data. But I think at the time, we realized that even before those things, there's a fundamental problem with the interface or the tool that you're supposed to use. Because that's like an ignored problem in some sense. What your tool is, or how that matters for your task. So what we found concretely is that if you just use the text terminal off the shelf as a tool for those agents, there's a lot of problems. For example, if you edit something, there's no feedback. So you don't know whether your edit is good or not. That makes the agent very confused and makes a lot of mistakes. There are a lot of small problems, you would say. Well, you can try to do prompt engineering and improve that, but it turns out to be actually very hard. We realized that the interface design is actually a very omitted part of agent design. So we did this switch agent work. And the key idea is just, even before you talk about what the agent is, you should talk about what the environment is. You should make sure that the environment is actually friendly to whatever agent you're trying to apply. That's the same idea for humans. Text terminal is good for some tasks, like git, pool, or whatever. But it's not good if you want to look at browser and whatever. Also, browser is a good tool for some tasks, but it's not a good tool for other tasks. We need to talk about how design interface, in some sense, where we should treat agents as our customers. It's like when we treat humans as a customer, we design human computer interfaces. We design those beautiful desktops or browsers or whatever, so that it's very intuitive and easy for humans to use. And this whole great subject of HCI is all about that. I think now the research idea of switch agent is just, we should treat agents as our customers. And we should do like, you know… AICI.Swyx [00:42:16]: AICI, exactly.Harrison [00:42:18]: So what are the tools that a suite agent should have, or a coding agent in general should have?Shunyu [00:42:24]: For suite agent, it's like a modified text terminal, which kind of adapts to a lot of the patterns of language models to make it easier for language models to use. For example, now for edit, instead of having no feedback, it will actually have a feedback of, you know, actually here you introduced like a syntax error, and you should probably want to fix that, and there's an ended error there. And that makes it super easy for the model to actually do that. And there's other small things, like how exactly you write arguments, right? Like, do you want to write like a multi-line edit, or do you want to write a single line edit? I think it's more interesting to think about the way of the development process of an ACI rather than the actual ACI for like a concrete application. Because I think the general paradigm is very similar to HCI and psychology, right? Basically, for how people develop HCIs, they do behavior experiments on humans, right? I do every test, right? Like, which interface is actually better? And I do those behavior experiments, kind of like psychology experiments to humans, and I change things. And I think what's really interesting for me, for this three-agent paper, is we can probably do the same thing for agents, right? We can do every test for those agents and do behavior tests. And through the process, we not only invent better interfaces for those agents, that's the practical value, but we also better understand agents. Just like when we do those A-B tests, we do those HCI, we better understand humans. Doing those ACI experiments, we actually better understand agents. And that's pretty cool.Harrison [00:43:51]: Besides that A-B testing, what are other processes that people can use to think about this in a good way?Swyx [00:43:57]: That's a great question.Shunyu [00:43:58]: And I think three-agent is an initial work. And what we do is the kind of the naive approach, right? You just try some interface, and you see what's going wrong, and then you try to fix that. We do this kind of iterative fixing. But I think what's really interesting is there'll be a lot of future directions that's very promising if we can apply some of the HCI principles more systematically into the interface design. I think that would be a very cool interdisciplinary research opportunity.Harrison [00:44:26]: You talked a lot about agent-computer interfaces and interactions. What about human-to-agent UX patterns? Curious for any thoughts there that you might have.Swyx [00:44:38]: That's a great question.Shunyu [00:44:39]: And in some sense, I feel like prompt engineering is about human-to-agent interface. But I think there can be a lot of interesting research done about... So prompting is about how humans can better communicate with the agent. But I think there could be interesting research on how agents can better communicate with humans, right? When to ask questions, how to ask questions, what's the frequency of asking questions. And I think those kinds of stuff could be very cool research.Harrison [00:45:07]: Yeah, I think some of the most interesting stuff that I saw here was also related to coding with Devin from Cognition. And they had the three or four different panels where you had the chat, the browser, the terminal, and I guess the code editor as well.Swyx [00:45:19]: There's more now.Harrison [00:45:19]: There's more. Okay, I'm not up to date. Yeah, I think they also did a good job on ACI.Swyx [00:45:25]: I think that's the main learning I have from Devin. They cracked that. Actually, there was no foundational planning breakthrough. The planner is actually pretty simple, but ACI that they broke through on.Shunyu [00:45:35]: I think making the tool good and reliable is probably like 90% of the whole agent. Once the tool is actually good, then the agent design can be much, much simpler. On the other hand, if the tool is bad, then no matter how much you put into the agent design, planning or search or whatever, it's still going to be trash.Harrison [00:45:53]: Yeah, I'd argue the same. Same with like context and instructions. Like, yeah, go hand in hand.Alessio [00:46:00]: On the tool, how do you think about the tension of like, for both of you, I mean, you're building a library, so even more for you. The tension between making now a language or a library that is like easy for the agent to grasp and write versus one that is easy for like the human to grasp and write. Because, you know, the trend is like more and more code gets written by the agent. So why wouldn't you optimize the framework to be as easy as possible for the model versus for the person?Swyx [00:46:24]: I think it's possible to design an interfaceShunyu [00:46:25]: that's both friendly to humans and agents. But what do you think?Harrison [00:46:29]: We haven't thought about that from the perspective, like we're not trying to design LangChain or LangGraph to be friendly. But I mean, I think to be friendly for agents to write.Swyx [00:46:42]: But I mean, I think we see this with like,Harrison [00:46:43]: I saw some paper that used TypeScript notation instead of JSON notation for tool calling and it got a lot better performance. So it's definitely a thing. I haven't really heard of anyone designing like a syntax or a language explicitly for agents, but there's clearly syntaxes that are better.Shunyu [00:46:59]: I think function calling is a good example where it's like a good interface for both human programmers and for agents, right? Like for developers, it's actually a very friendly interface because it's very concrete and you don't have to do prompt engineering anymore. You can be very systematic. And for models, it's also pretty good, right? Like it can use all the existing coding content. So I think we need more of those kinds of designs.Swyx [00:47:21]: I will mostly agree and I'll slightly disagree in terms of this, which is like, whether designing for humans also overlaps with designing for AI. So Malte Ubo, who's the CTO of Vercel, who is creating basically JavaScript's competitor to LangChain, they're observing that basically, like if the API is easy to understand for humans, it's actually much easier to understand for LLMs, for example, because they're not overloaded functions. They don't behave differently under different contexts. They do one thing and they always work the same way. It's easy for humans, it's easy for LLMs. And like that makes a lot of sense. And obviously adding types is another one. Like type annotations only help give extra context, which is really great. So that's the agreement. And then a disagreement is that when I use structured output to do my chain of thought, I have found that I change my field names to hint to the LLM of what the field is supposed to do. So instead of saying topics, I'll say candidate topics. And that gives me a better result because the LLM was like, ah, this is just a draft thing I can use for chain of thought. And instead of like summaries, I'll say topic summaries to link the previous field to the current field. So like little stuff like that, I find myself optimizing for the LLM where I, as a human, would never do that. Interesting.Shunyu [00:48:32]: It's kind of like the way you optimize the prompt, it might be different for humans and for machines. You can have a common ground that's both clear for humans and agents, but to improve the human performance versus improving the agent performance, they might move to different directions.Swyx [00:48:48]: Might move different directions. There's a lot more use of metadata as well, like descriptions, comments, code comments, annotations and stuff like that. Yeah.Harrison [00:48:56]: I would argue that's just you communicatingSwyx [00:48:58]: to the agent what it should do.Harrison [00:49:00]: And maybe you need to communicate a little bit more than to humans because models aren't quite good enough yet.Swyx [00:49:06]: But like, I don't think that's crazy.Harrison [00:49:07]: I don't think that's like- It's not crazy.Swyx [00:49:09]: I will bring this in because it just happened to me yesterday. I was at the cursor office. They held their first user meetup and I was telling them about the LLM OS concept and why basically every interface, every tool was being redesigned for AIs to use rather than humans. And they're like, why? Like, can we just use Bing and Google for LLM search? Why must I use Exa? Or what's the other one that you guys work with?Harrison [00:49:32]: Tavilli.Swyx [00:49:33]: Tavilli. Web Search API dedicated for LLMs. What's the difference?Shunyu [00:49:36]: Exactly. To Bing API.Swyx [00:49:38]: Exactly.Harrison [00:49:38]: There weren't great APIs for search. Like the best one, like the one that we used initially in LangChain was SERP API, which is like maybe illegal. I'm not sure.Swyx [00:49:49]: And like, you know,Harrison [00:49:52]: and now there are like venture-backed companies.Swyx [00:49:53]: Shout out to DuckDuckGo, which is free.Harrison [00:49:55]: Yes, yes.Swyx [00:49:56]: Yeah.Harrison [00:49:56]: I do think there are some differences though. I think you want, like, I think generally these APIs try to return small amounts of text information, clear legible field. It's not a massive JSON blob. And I think that matters. I think like when you talk about designing tools, it's not only the, it's the interface in the entirety, not only the inputs, but also the outputs that really matter. And so I think they try to make the outputs.Shunyu [00:50:18]: They're doing ACI.Swyx [00:50:19]: Yeah, yeah, absolutely.Harrison [00:50:20]: Really?Swyx [00:50:21]: Like there's a whole set of industries that are just being redone for ACI. It's weird. And so my simple answer to them was like the error messages. When you give error messages, they should be basically prompts for the LLM to take and then self-correct. Then your error messages get more verbose, actually, than you normally would with a human. Stuff like that. Like a little, honestly, it's not that big. Again, like, is this worth a venture-backed industry? Unless you can tell us. But like, I think Code Interpreter, I think is a new thing. I hope so.Alessio [00:50:52]: We invested in it to be so.Shunyu [00:50:53]: I think that's a very interesting point. You're trying to optimize to the extreme, then obviously they're going to be different. For example, the error—Swyx [00:51:00]: Because we take it very seriously. Right.Shunyu [00:51:01]: The error for like language model, the longer the better. But for humans, that will make them very nervous and very tired, right? But I guess the point is more like, maybe we should try to find a co-optimized common ground as much as possible. And then if we have divergence, then we should try to diverge. But it's more philosophical now.Alessio [00:51:19]: But I think like part of it is like how you use it. So Google invented the PageRank because ideally you only click on one link, you know, like the top three should have the answer. But with models, it's like, well, you can get 20. So those searches are more like semantic grouping in a way. It's like for this query, I'll return you like 20, 30 things that are kind of good, you know? So it's less about ranking and it's more about grouping.Shunyu [00:51:42]: Another fundamental thing about HCI is the difference between human and machine's kind of memory limit, right? So I think what's really interesting about this concept HCI versus HCI is interfaces that's optimized for them. You can kind of understand some of the fundamental characteristics, differences of humans and machines, right? Why, you know, if you look at find or whatever terminal command, you know, you can only look at one thing at a time or that's because we have a very small working memory. You can only deal with one thing at a time. You can only look at one paragraph of text at the same time. So the interface for us is by design, you know, a small piece of information, but more temporal steps. But for machines, that should be the opposite, right? You should just give them a hundred different results and they should just decide in context what's the most relevant stuff and trade off the context for temporal steps. That's actually also better for language models because like the cost is smaller or whatever. So it's interesting to connect those interfaces to the fundamental kind of differences of those.Harrison [00:52:43]: When you said earlier, you know, we should try to design these to maybe be similar as possible and diverge if we need to.Swyx [00:52:49]: I actually don't have a problem with them diverging nowHarrison [00:52:51]: and seeing venture-backed startups emerging now because we are different from machines code AI. And it's just so early on, like they may still look kind of similar and they may still be small differences, but it's still just so early. And I think we'll only discover more ways that they differ. And so I'm totally fine with them kind of like diverging earlySwyx [00:53:10]: and optimizing for the...Harrison [00:53:11]: I agree. I think it's more like, you know,Shunyu [00:53:14]: we should obviously try to optimize human interface just for humans. We're already doing that for 50 years. We should optimize agent interface just for agents, but we might also try to co-optimize both and see how far we can get. There's enough people to try all three directions. Yeah.Swyx [00:53:31]: There's a thesis I sometimes push, which is the sour lesson as opposed to the bitter lesson, which we're always inspired by human development, but actually AI develops its own path.Shunyu [00:53:40]: Right. We need to understand better, you know, what are the fundamental differences between those creatures.Swyx [00:53:45]: It's funny when really early on this pod, you were like, how much grounding do you have in cognitive development and human brain stuff? And I'm like

In this episode of Syntax, Wes and Scott talk with Ryan Dahl about Deno 2.0, its new features and use of web standards, and how it seamlessly integrates with popular frameworks like Next.js. Ryan shares insights on the motivations behind Deno's creation, its emphasis on simplicity and security, and offers his take on the evolving JavaScript ecosystem. Show Notes 00:00 Welcome to Syntax! 00:34 What is Deno? 05:08 Deno 2.0 07:49 NPM compatibility 09:40 What parts of Node aren't doable in Deno? 11:22 Do we need a hard break from Require? 13:51 Package management 16:25 Security and performance benefits of Deno 20:57 Brought to you by Sentry.io 20:57 Thoughts on Bun and Node additions 26:25 Ryan's favorite Deno projects Lume Fresh webgpu-examples gpucraft minecraft clone + deno + webgpu gpucraft example Shaderplay Orillusion 28:42 Will we ever see a unified file system API? 31:49 Typescript 36:12 Jupyter Notebooks with Deno Polars 39:11 AI and WASM in JavaScript 42:01 Deno 2.0 features and future 43:08 Sick Picks & Shameless Plugs Sick Picks Ryan: McCarren Park Shameless Plugs Ryan: https://deno.com/enterprise Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Bento is Meta's internal distribution of Jupyter Notebooks, an open-source web-based computing platform. Host Pascal is joined by Steve who worked with his team on building many features on top of Jupyter, including scheduled notebooks, sharing with colleagues and running notebooks without a remote server component by leveraging Webassembly in the browser. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Twitter (https://twitter.com/metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host @passy (https://twitter.com/passy, https://mastodon.social/@passy, and https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links Scheduling Jupyter Notebooks at Meta: https://engineering.fb.com/2023/08/29/security/scheduling-jupyter-notebooks-meta/ Serverless Jupyter Notebooks at Meta: https://engineering.fb.com/2024/06/10/data-infrastructure/serverless-jupyter-notebooks-bento-meta/ Jupyter Notebooks: https://jupyter.org/  Timestamps Intro 0:06 Who is Steve? 1:49 What are Jupyter and Bento? 2:48 Who is Bento for? 3:40 Internal-only Bento features 4:42 Scheduled notebooks 11:39 Integrating with existing batch jobs 17:10 The case for serverless notebooks 20:59 Enter wasm 24:29 Upgrade paths from serverless to server 26:29 Bringing more Python libraries to the browser 30:21 Adding magick(s) 31:52 DataFrame magic and AI 36:41 What's next? 38:29 Outro 43:17

SummaryIn this episode of the AI Engineering podcast, host Tobias Macy interviews Tammer Saleh, founder of SuperOrbital, about the potentials and pitfalls of using Kubernetes for machine learning workloads. The conversation delves into the specific needs of machine learning workflows, such as model tracking, versioning, and the use of Jupyter Notebooks, and how Kubernetes can support these tasks. Tammer emphasizes the importance of a unified API for different teams and the flexibility Kubernetes provides in handling various workloads. Finally, Tammer offers advice for teams considering Kubernetes for their machine learning workloads and discusses the future of Kubernetes in the ML ecosystem, including areas for improvement and innovation.AnnouncementsHello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systemsYour host is Tobias Macey and today I'm interviewing Tammer Saleh about the potentials and pitfalls of using Kubernetes for your ML workloads.InterviewIntroductionHow did you get involved in Kubernetes?For someone who is unfamiliar with Kubernetes, how would you summarize it?For the context of this conversation, can you describe the different phases of ML that we're talking about?Kubernetes was originally designed to handle scaling and distribution of stateless processes. ML is an inherently stateful problem domain. What challenges does that add for K8s environments?What are the elements of an ML workflow that lend themselves well to a Kubernetes environment?How much Kubernetes knowledge does an ML/data engineer need to know to get their work done?What are the sharp edges of Kubernetes in the context of ML projects?What are the most interesting, unexpected, or challenging lessons that you have learned while working with Kubernetes?When is Kubernetes the wrong choice for ML?What are the aspects of Kubernetes (core or the ecosystem) that you are keeping an eye on which will help improve its utility for ML workloads?Contact InfoEmailLinkedInParting QuestionFrom your perspective, what is the biggest gap in the tooling or technology for ML workloads today?LinksSuperOrbitalCloudFoundryHeroku12 Factor ModelKubernetesDocker ComposeCore K8s ClassJupyter NotebookCrossplaneOchre JellyCNCF (Cloud Native Computing Foundation) LandscapeStateful SetRAG == Retrieval Augmented GenerationPodcast EpisodeKubeflowFlyteData Engineering Podcast EpisodePachydermData Engineering Podcast EpisodeCoreWeaveKubectl ("koob-cuddle")HelmCRD == Custom Resource DefinitionHorovodPodcast.__init__ EpisodeTemporalSlurmRayDaskInfinibandThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Topics covered in this episode: Solara UI Framework Coverage at a crossroads “Virtual” methods in Python classes Extras Joke Watch on YouTube About the show Sponsored by ScoutAPM: pythonbytes.fm/scout Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Michael #1: Solara UI Framework via Florian A Pure Python, React-style Framework for Scaling Your Jupyter and Web Apps Solara lets you build web apps from pure Python using ipywidgets or a React-like API on top of ipywidgets. These apps work both inside the Jupyter Notebook and as standalone web apps with frameworks like FastAPI. See the Examples page. Based on Reacton By building on top of ipywidgets, Solara automatically leverage an existing ecosystem of widgets and run on many platforms, including JupyterLab, Jupyter Notebook, Voilà, Google Colab, DataBricks, JetBrains Datalore, and more. Brian #2: Coverage at a crossroads Ned Batchelder is working on making coverage.py faster. Includes a nice, quick explanation of roughly how coverage.py works with trace function and arcs used for branch coverage. And how trace slows things down for lines we know are already covered. There are cool ideas from SlipCover that could be applicable. There's also sys.monitoring from Python 3.12 that helps with line coverage, since you can disable it for lines you already have info on. It doesn't quite complete the picture for branch coverage, though. Summary: jump in and help if you can read it anyway for a great mental model of how coverage.py works. Michael #3: “Virtual” methods in Python classes via Brian Skinn PEP 698 just got accepted, defining an @override decorator for type hinting, to help avoid errors in subclasses that override methods. Only affects type checkers but allows you to declare a “link” between the base method and derived class method with the intent of overriding it using OOP. If there is a mismatch, it's an error. Python 3.12's documentation Makes Python a bit more like C# and other more formal languages Brian #4: Parsing Python ASTs 20x Faster with Rust Evan Doyle Tach is “a CLI tool that lets you define and enforce import boundaries between Python modules in your project.” we covered it in episode 384 When used to analyze Sentry's ~3k Python file codebase, it took about 10 seconds. Profiling analysis using py-spy and speedscope pointed to a function that spends about 2/3 of the time parsing the AST, and about 1/3 traversing it. That portion was then rewritten in Rust, resulting in 10x speedup, ending in about 1 second. This is a cool example of not just throwing Rust at a speed problem right away, but doing the profiling homework first, and focusing the Rust rewrite on the bottleneck. Extras Brian: I brought up pkgutil.resolve_name() last week on episode 388 Brett Cannon says don't use that, it's deprecated Thanks astroboy for letting me know Will we get CalVer for Python? it was talked about at the language summit There's also pep 2026, in draft, with a nice nod in the number of when it might happen. 3.13 already in the works for 2024 3.14 slated for 2025, and we gotta have a pi release So the earliest is then 2026, with maybe a 3.26 version ? Saying thanks to open source maintainers Great write-up by Brett Cannon about how to show your appreciation for OSS maintainers. Be nice Be an advocate Produce your own open source Say thanks Fiscal support On topic Thanks Brett for pyproject.toml. I love it. Michael: The Shiny for Python course is out! Plus, it's free so come and get it. Joke: Tao of Programming: Book 1: Into the Silent Void, Part 1

Topics covered in this episode: NumPy 2.0 release date is June 16 Uvicorn adds multiprocess workers pixi JupyterLab 4.2 and Notebook 7.2 are available Extras Joke Watch on YouTube About the show Sponsored by Mailtrap: pythonbytes.fm/mailtrap Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Brian #1: NumPy 2.0 release date is June 16 “This release has been over a year in the making, and is the first major release since 2006. Importantly, in addition to many new features and performance improvement, it contains breaking changes to the ABI as well as the Python and C APIs. It is likely that downstream packages and end user code needs to be adapted - if you can, please verify whether your code works with NumPy 2.0.0rc2.” NumPy 2.0.0 Release Notes NumPy 2.0 migration guide including “try just running ruff check path/to/code/ --select NPY201” “Many of the changes covered in the 2.0 release notes and in this migration guide can be automatically adapted in downstream code with a dedicated Ruff rule, namely rule NPY201.” Michael #2: Uvicorn adds multiprocess workers via John Hagen The goal was to no longer need to suggest that people use Gunicorn on top of uvicorn. Uvicorn can now in a sense "do it all” Steps to use it and background on how it works. Brian #3: pixi Suggested by Vic Kelson “pixi is a cross-platform, multi-language package manager and workflow tool built on the foundation of the conda ecosystem.” Tutorial: Doing Python development with Pixi Some quotes from Vic: “Pixi is a project manager, written in Rust, that allows you to build Python projects without having Python previously installed. It's installable with Homebrew (brew install pixi on Linux and MacOS). There's support in VSCode and PyCharm via plugins. By default, pixi fetches packages from conda-forge, so you get the scientific stack in a pretty reliable and performant build. If a package isn't on conda-forge, it'll look on PyPI, or I believe you can force it to look on PyPI if you like.” “So far, it works GREAT for me. What really impressed me is that I got a Jupyter environment with CuPy utilizing my aging Nvidia GPU on the FIRST TRY.” Michael #4: JupyterLab 4.2 and Notebook 7.2 are available JupyterLab 4.2.0 has been released! This new minor release of JupyterLab includes 3 new features, 20 enhancements, 33 bug fixes and 29 maintenance tasks. Jupyter Notebook 7.2.0 has also been released Highlights include Easier Workspaces Management with GUI Recently opened/closed files Full notebook windowing mode by default (renders only the cells visible in the window, leading to improved performance) Improved Shortcuts Editor Dark High Contrast Theme Extras Brian: Help test Python 3.13! Help us test free-threaded Python without the GIL both from Hugo van Kemenade Python Test 221: How to get pytest to import your code under test is out Michael: Bend follow up from Bernát Gábor “Bend looks roughly like Python but is nowhere there actually. For example it has no for loops, instead you're meant to use bend keyword (hence the language name) to expand calculations and another keyword to join branches. So basically think of something that resembles Python at high level, but without being compatible with that and without any of the standard library or packages the Python language provides. That being said does an impressive job at parallelization, but essentially it's a brand new language with new syntax and paradigms that you will have to learn, it just shares at first look similarities with Python the most.” Joke: Do-while

Maggie, Linus, Geoffrey, and the LS crew are reuniting for our second annual AI UX demo day in SF on Apr 28. Sign up to demo here! And don't forget tickets for the AI Engineer World's Fair — for early birds who join before keynote announcements!It's become fashionable for many AI startups to project themselves as “the next Google” - while the search engine is so 2000s, both Perplexity and Exa referred to themselves as a “research engine” or “answer engine” in our NeurIPS pod. However these searches tend to be relatively shallow, and it is challenging to zoom up and down the ladders of abstraction to garner insights. For serious researchers, this level of simple one-off search will not cut it.We've commented in our Jan 2024 Recap that Flow Engineering (simply; multi-turn processes over many-shot single prompts) seems to offer far more performance, control and reliability for a given cost budget. Our experiments with Devin and our understanding of what the new Elicit Notebooks offer a glimpse into the potential for very deep, open ended, thoughtful human-AI collaboration at scale.It starts with promptsWhen ChatGPT exploded in popularity in November 2022 everyone was turned into a prompt engineer. While generative models were good at "vibe based" outcomes (tell me a joke, write a poem, etc) with basic prompts, they struggled with more complex questions, especially in symbolic fields like math, logic, etc. Two of the most important "tricks" that people picked up on were:* Chain of Thought prompting strategy proposed by Wei et al in the “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. Rather than doing traditional few-shot prompting with just question and answers, adding the thinking process that led to the answer resulted in much better outcomes.* Adding "Let's think step by step" to the prompt as a way to boost zero-shot reasoning, which was popularized by Kojima et al in the Large Language Models are Zero-Shot Reasoners paper from NeurIPS 2022. This bumped accuracy from 17% to 79% compared to zero-shot.Nowadays, prompts include everything from promises of monetary rewards to… whatever the Nous folks are doing to turn a model into a world simulator. At the end of the day, the goal of prompt engineering is increasing accuracy, structure, and repeatability in the generation of a model.From prompts to agentsAs prompt engineering got more and more popular, agents (see “The Anatomy of Autonomy”) took over Twitter with cool demos and AutoGPT became the fastest growing repo in Github history. The thing about AutoGPT that fascinated people was the ability to simply put in an objective without worrying about explaining HOW to achieve it, or having to write very sophisticated prompts. The system would create an execution plan on its own, and then loop through each task. The problem with open-ended agents like AutoGPT is that 1) it's hard to replicate the same workflow over and over again 2) there isn't a way to hard-code specific steps that the agent should take without actually coding them yourself, which isn't what most people want from a product. From agents to productsPrompt engineering and open-ended agents were great in the experimentation phase, but this year more and more of these workflows are starting to become polished products. Today's guests are Andreas Stuhlmüller and Jungwon Byun of Elicit (previously Ought), an AI research assistant that they think of as “the best place to understand what is known”. Ought was a non-profit, but last September, Elicit spun off into a PBC with a $9m seed round. It is hard to quantify how much a workflow can be improved, but Elicit boasts some impressive numbers for research assistants:Just four months after launch, Elicit crossed $1M ARR, which shows how much interest there is for AI products that just work.One of the main takeaways we had from the episode is how teams should focus on supervising the process, not the output. Their philosophy at Elicit isn't to train general models, but to train models that are extremely good at focusing processes. This allows them to have pre-created steps that the user can add to their workflow (like classifying certain features that are specific to their research field) without having to write a prompt for it. And for Hamel Husain's happiness, they always show you the underlying prompt. Elicit recently announced notebooks as a new interface to interact with their products: (fun fact, they tried to implement this 4 times before they landed on the right UX! We discuss this ~33:00 in the podcast)The reasons why they picked notebooks as a UX all tie back to process:* They are systematic; once you have a instruction/prompt that works on a paper, you can run hundreds of papers through the same workflow by creating a column. Notebooks can also be edited and exported at any point during the flow.* They are transparent - Many papers include an opaque literature review as perfunctory context before getting to their novel contribution. But PDFs are “dead” and it is difficult to follow the thought process and exact research flow of the authors. Sharing “living” Elicit Notebooks opens up this process.* They are unbounded - Research is an endless stream of rabbit holes. So it must be easy to dive deeper and follow up with extra steps, without losing the ability to surface for air. We had a lot of fun recording this, and hope you have as much fun listening!AI UX in SFLong time Latent Spacenauts might remember our first AI UX meetup with Linus Lee, Geoffrey Litt, and Maggie Appleton last year. Well, Maggie has since joined Elicit, and they are all returning at the end of this month! Sign up here: https://lu.ma/aiuxAnd submit demos here! https://forms.gle/iSwiesgBkn8oo4SS8We expect the 200 seats to “sell out” fast. Attendees with demos will be prioritized.Show Notes* Elicit* Ought (their previous non-profit)* “Pivoting” with GPT-4* Elicit notebooks launch* Charlie* Andreas' BlogTimestamps* [00:00:00] Introductions* [00:07:45] How Johan and Andreas Joined Forces to Create Elicit* [00:10:26] Why Products > Research* [00:15:49] The Evolution of Elicit's Product* [00:19:44] Automating Literature Review Workflow* [00:22:48] How GPT-3 to GPT-4 Changed Things* [00:25:37] Managing LLM Pricing and Performance* [00:31:07] Open vs. Closed: Elicit's Approach to Model Selection* [00:31:56] Moving to Notebooks* [00:39:11] Elicit's Budget for Model Queries and Evaluations* [00:41:44] Impact of Long Context Windows* [00:47:19] Underrated Features and Surprising Applications* [00:51:35] Driving Systematic and Efficient Research* [00:53:00] Elicit's Team Growth and Transition to a Public Benefit Corporation* [00:55:22] Building AI for GoodFull Interview on YouTubeAs always, a plug for our youtube version for the 80% of communication that is nonverbal:TranscriptAlessio [00:00:00]: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.Swyx [00:00:15]: Hey, and today we are back in the studio with Andreas and Jungwon from Elicit. Welcome.Jungwon [00:00:20]: Thanks guys.Andreas [00:00:21]: It's great to be here.Swyx [00:00:22]: Yeah. So I'll introduce you separately, but also, you know, we'd love to learn a little bit more about you personally. So Andreas, it looks like you started Elicit first, Jungwon joined later.Andreas [00:00:32]: That's right. For all intents and purposes, the Elicit and also the Ought that existed before then were very different from what I started. So I think it's like fair to say that you co-founded it.Swyx [00:00:43]: Got it. And Jungwon, you're a co-founder and COO of Elicit now.Jungwon [00:00:46]: Yeah, that's right.Swyx [00:00:47]: So there's a little bit of a history to this. I'm not super aware of like the sort of journey. I was aware of OTT and Elicit as sort of a nonprofit type situation. And recently you turned into like a B Corp, Public Benefit Corporation. So yeah, maybe if you want, you could take us through that journey of finding the problem. You know, obviously you're working together now. So like, how do you get together to decide to leave your startup career to join him?Andreas [00:01:10]: Yeah, it's truly a very long journey. I guess truly, it kind of started in Germany when I was born. So even as a kid, I was always interested in AI, like I kind of went to the library. There were books about how to write programs in QBasic and like some of them talked about how to implement chatbots.Jungwon [00:01:27]: To be clear, he grew up in like a tiny village on the outskirts of Munich called Dinkelschirben, where it's like a very, very idyllic German village.Andreas [00:01:36]: Yeah, important to the story. So basically, the main thing is I've kind of always been thinking about AI my entire life and been thinking about, well, at some point, this is going to be a huge deal. It's going to be transformative. How can I work on it? And was thinking about it from when I was a teenager, after high school did a year where I started a startup with the intention to become rich. And then once I'm rich, I can affect the trajectory of AI. Did not become rich, decided to go back to college and study cognitive science there, which was like the closest thing I could find at the time to AI. In the last year of college, moved to the US to do a PhD at MIT, working on broadly kind of new programming languages for AI because it kind of seemed like the existing languages were not great at expressing world models and learning world models doing Bayesian inference. Was always thinking about, well, ultimately, the goal is to actually build tools that help people reason more clearly, ask and answer better questions and make better decisions. But for a long time, it seemed like the technology to put reasoning in machines just wasn't there. Initially, at the end of my postdoc at Stanford, I was thinking about, well, what to do? I think the standard path is you become an academic and do research. But it's really hard to actually build interesting tools as an academic. You can't really hire great engineers. Everything is kind of on a paper-to-paper timeline. And so I was like, well, maybe I should start a startup, pursued that for a little bit. But it seemed like it was too early because you could have tried to do an AI startup, but probably would not have been this kind of AI startup we're seeing now. So then decided to just start a nonprofit research lab that's going to do research for a while until we better figure out how to do thinking in machines. And that was odd. And then over time, it became clear how to actually build actual tools for reasoning. And only over time, we developed a better way to... I'll let you fill in some of the details here.Jungwon [00:03:26]: Yeah. So I guess my story maybe starts around 2015. I kind of wanted to be a founder for a long time, and I wanted to work on an idea that stood the test of time for me, like an idea that stuck with me for a long time. And starting in 2015, actually, originally, I became interested in AI-based tools from the perspective of mental health. So there are a bunch of people around me who are really struggling. One really close friend in particular is really struggling with mental health and didn't have any support, and it didn't feel like there was anything before kind of like getting hospitalized that could just help her. And so luckily, she came and stayed with me for a while, and we were just able to talk through some things. But it seemed like lots of people might not have that resource, and something maybe AI-enabled could be much more scalable. I didn't feel ready to start a company then, that's 2015. And I also didn't feel like the technology was ready. So then I went into FinTech and kind of learned how to do the tech thing. And then in 2019, I felt like it was time for me to just jump in and build something on my own I really wanted to create. And at the time, I looked around at tech and felt like not super inspired by the options. I didn't want to have a tech career ladder, or I didn't want to climb the career ladder. There are two kind of interesting technologies at the time, there was AI and there was crypto. And I was like, well, the AI people seem like a little bit more nice, maybe like slightly more trustworthy, both super exciting, but threw my bet in on the AI side. And then I got connected to Andreas. And actually, the way he was thinking about pursuing the research agenda at OTT was really compatible with what I had envisioned for an ideal AI product, something that helps kind of take down really complex thinking, overwhelming thoughts and breaks it down into small pieces. And then this kind of mission that we need AI to help us figure out what we ought to do was really inspiring, right? Yeah, because I think it was clear that we were building the most powerful optimizer of our time. But as a society, we hadn't figured out how to direct that optimization potential. And if you kind of direct tremendous amounts of optimization potential at the wrong thing, that's really disastrous. So the goal of OTT was make sure that if we build the most transformative technology of our lifetime, it can be used for something really impactful, like good reasoning, like not just generating ads. My background was in marketing, but like, so I was like, I want to do more than generate ads with this. But also if these AI systems get to be super intelligent enough that they are doing this really complex reasoning, that we can trust them, that they are aligned with us and we have ways of evaluating that they're doing the right thing. So that's what OTT did. We did a lot of experiments, you know, like I just said, before foundation models really like took off. A lot of the issues we were seeing were more in reinforcement learning, but we saw a future where AI would be able to do more kind of logical reasoning, not just kind of extrapolate from numerical trends. We actually kind of set up experiments with people where kind of people stood in as super intelligent systems and we effectively gave them context windows. So they would have to like read a bunch of text and one person would get less text and one person would get all the texts and the person with less text would have to evaluate the work of the person who could read much more. So like in a world we were basically simulating, like in 2018, 2019, a world where an AI system could read significantly more than you and you as the person who couldn't read that much had to evaluate the work of the AI system. Yeah. So there's a lot of the work we did. And from that, we kind of iterated on the idea of breaking complex tasks down into smaller tasks, like complex tasks, like open-ended reasoning, logical reasoning into smaller tasks so that it's easier to train AI systems on them. And also so that it's easier to evaluate the work of the AI system when it's done. And then also kind of, you know, really pioneered this idea, the importance of supervising the process of AI systems, not just the outcomes. So a big part of how Elicit is built is we're very intentional about not just throwing a ton of data into a model and training it and then saying, cool, here's like scientific output. Like that's not at all what we do. Our approach is very much like, what are the steps that an expert human does or what is like an ideal process as granularly as possible, let's break that down and then train AI systems to perform each of those steps very robustly. When you train like that from the start, after the fact, it's much easier to evaluate, it's much easier to troubleshoot at each point. Like where did something break down? So yeah, we were working on those experiments for a while. And then at the start of 2021, decided to build a product.Swyx [00:07:45]: Do you mind if I, because I think you're about to go into more modern thought and Elicit. And I just wanted to, because I think a lot of people are in where you were like sort of 2018, 19, where you chose a partner to work with. Yeah. Right. And you didn't know him. Yeah. Yeah. You were just kind of cold introduced. A lot of people are cold introduced. Yeah. Never work with them. I assume you had a lot, a lot of other options, right? Like how do you advise people to make those choices?Jungwon [00:08:10]: We were not totally cold introduced. So one of our closest friends introduced us. And then Andreas had written a lot on the OTT website, a lot of blog posts, a lot of publications. And I just read it and I was like, wow, this sounds like my writing. And even other people, some of my closest friends I asked for advice from, they were like, oh, this sounds like your writing. But I think I also had some kind of like things I was looking for. I wanted someone with a complimentary skillset. I want someone who was very values aligned. And yeah, that was all a good fit.Andreas [00:08:38]: We also did a pretty lengthy mutual evaluation process where we had a Google doc where we had all kinds of questions for each other. And I think it ended up being around 50 pages or so of like various like questions and back and forth.Swyx [00:08:52]: Was it the YC list? There's some lists going around for co-founder questions.Andreas [00:08:55]: No, we just made our own questions. But I guess it's probably related in that you ask yourself, what are the values you care about? How would you approach various decisions and things like that?Jungwon [00:09:04]: I shared like all of my past performance reviews. Yeah. Yeah.Swyx [00:09:08]: And he never had any. No.Andreas [00:09:10]: Yeah.Swyx [00:09:11]: Sorry, I just had to, a lot of people are going through that phase and you kind of skipped over it. I was like, no, no, no, no. There's like an interesting story.Jungwon [00:09:20]: Yeah.Alessio [00:09:21]: Yeah. Before we jump into what a list it is today, the history is a bit counterintuitive. So you start with figuring out, oh, if we had a super powerful model, how would we align it? But then you were actually like, well, let's just build the product so that people can actually leverage it. And I think there are a lot of folks today that are now back to where you were maybe five years ago that are like, oh, what if this happens rather than focusing on actually building something useful with it? What clicked for you to like move into a list and then we can cover that story too.Andreas [00:09:49]: I think in many ways, the approach is still the same because the way we are building illicit is not let's train a foundation model to do more stuff. It's like, let's build a scaffolding such that we can deploy powerful models to good ends. I think it's different now in that we actually have like some of the models to plug in. But if in 2017, we had had the models, we could have run the same experiments we did run with humans back then, just with models. And so in many ways, our philosophy is always, let's think ahead to the future of what models are going to exist in one, two years or longer. And how can we make it so that they can actually be deployed in kind of transparent, controllableJungwon [00:10:26]: ways? I think motivationally, we both are kind of product people at heart. The research was really important and it didn't make sense to build a product at that time. But at the end of the day, the thing that always motivated us is imagining a world where high quality reasoning is really abundant and AI is a technology that's going to get us there. And there's a way to guide that technology with research, but we can have a more direct effect through product because with research, you publish the research and someone else has to implement that into the product and the product felt like a more direct path. And we wanted to concretely have an impact on people's lives. Yeah, I think the kind of personally, the motivation was we want to build for people.Swyx [00:11:03]: Yep. And then just to recap as well, like the models you were using back then were like, I don't know, would they like BERT type stuff or T5 or I don't know what timeframe we're talking about here.Andreas [00:11:14]: I guess to be clear, at the very beginning, we had humans do the work. And then I think the first models that kind of make sense were TPT-2 and TNLG and like Yeah, early generative models. We do also use like T5 based models even now started with TPT-2.Swyx [00:11:30]: Yeah, cool. I'm just kind of curious about like, how do you start so early? You know, like now it's obvious where to start, but back then it wasn't.Jungwon [00:11:37]: Yeah, I used to nag Andreas a lot. I was like, why are you talking to this? I don't know. I felt like TPT-2 is like clearly can't do anything. And I was like, Andreas, you're wasting your time, like playing with this toy. But yeah, he was right.Alessio [00:11:50]: So what's the history of what Elicit actually does as a product? You recently announced that after four months, you get to a million in revenue. Obviously, a lot of people use it, get a lot of value, but it would initially kind of like structured data extraction from papers. Then you had kind of like concept grouping. And today, it's maybe like a more full stack research enabler, kind of like paper understander platform. What's the definitive definition of what Elicit is? And how did you get here?Jungwon [00:12:15]: Yeah, we say Elicit is an AI research assistant. I think it will continue to evolve. That's part of why we're so excited about building and research, because there's just so much space. I think the current phase we're in right now, we talk about it as really trying to make Elicit the best place to understand what is known. So it's all a lot about like literature summarization. There's a ton of information that the world already knows. It's really hard to navigate, hard to make it relevant. So a lot of it is around document discovery and processing and analysis. I really kind of want to import some of the incredible productivity improvements we've seen in software engineering and data science and into research. So it's like, how can we make researchers like data scientists of text? That's why we're launching this new set of features called Notebooks. It's very much inspired by computational notebooks, like Jupyter Notebooks, you know, DeepNode or Colab, because they're so powerful and so flexible. And ultimately, when people are trying to get to an answer or understand insight, they're kind of like manipulating evidence and information. Today, that's all packaged in PDFs, which are super brittle. So with language models, we can decompose these PDFs into their underlying claims and evidence and insights, and then let researchers mash them up together, remix them and analyze them together. So yeah, I would say quite simply, overall, Elicit is an AI research assistant. Right now we're focused on text-based workflows, but long term, really want to kind of go further and further into reasoning and decision making.Alessio [00:13:35]: And when you say AI research assistant, this is kind of meta research. So researchers use Elicit as a research assistant. It's not a generic you-can-research-anything type of tool, or it could be, but like, what are people using it for today?Andreas [00:13:49]: Yeah. So specifically in science, a lot of people use human research assistants to do things. You tell your grad student, hey, here are a couple of papers. Can you look at all of these, see which of these have kind of sufficiently large populations and actually study the disease that I'm interested in, and then write out like, what are the experiments they did? What are the interventions they did? What are the outcomes? And kind of organize that for me. And the first phase of understanding what is known really focuses on automating that workflow because a lot of that work is pretty rote work. I think it's not the kind of thing that we need humans to do. Language models can do it. And then if language models can do it, you can obviously scale it up much more than a grad student or undergrad research assistant would be able to do.Jungwon [00:14:31]: Yeah. The use cases are pretty broad. So we do have a very large percent of our users are just using it personally or for a mix of personal and professional things. People who care a lot about health or biohacking or parents who have children with a kind of rare disease and want to understand the literature directly. So there is an individual kind of consumer use case. We're most focused on the power users. So that's where we're really excited to build. So Lissette was very much inspired by this workflow in literature called systematic reviews or meta-analysis, which is basically the human state of the art for summarizing scientific literature. And it typically involves like five people working together for over a year. And they kind of first start by trying to find the maximally comprehensive set of papers possible. So it's like 10,000 papers. And they kind of systematically narrow that down to like hundreds or 50 extract key details from every single paper. Usually have two people doing it, like a third person reviewing it. So it's like an incredibly laborious, time consuming process, but you see it in every single domain. So in science, in machine learning, in policy, because it's so structured and designed to be reproducible, it's really amenable to automation. So that's kind of the workflow that we want to automate first. And then you make that accessible for any question and make these really robust living summaries of science. So yeah, that's one of the workflows that we're starting with.Alessio [00:15:49]: Our previous guest, Mike Conover, he's building a new company called Brightwave, which is an AI research assistant for financial research. How do you see the future of these tools? Does everything converge to like a God researcher assistant, or is every domain going to have its own thing?Andreas [00:16:03]: I think that's a good and mostly open question. I do think there are some differences across domains. For example, some research is more quantitative data analysis, and other research is more high level cross domain thinking. And we definitely want to contribute to the broad generalist reasoning type space. Like if researchers are making discoveries often, it's like, hey, this thing in biology is actually analogous to like these equations in economics or something. And that's just fundamentally a thing that where you need to reason across domains. At least within research, I think there will be like one best platform more or less for this type of generalist research. I think there may still be like some particular tools like for genomics, like particular types of modules of genes and proteins and whatnot. But for a lot of the kind of high level reasoning that humans do, I think that is a more of a winner type all thing.Swyx [00:16:52]: I wanted to ask a little bit deeper about, I guess, the workflow that you mentioned. I like that phrase. I see that in your UI now, but that's as it is today. And I think you were about to tell us about how it was in 2021 and how it may be progressed. How has this workflow evolved over time?Jungwon [00:17:07]: Yeah. So the very first version of Elicit actually wasn't even a research assistant. It was a forecasting assistant. So we set out and we were thinking about, you know, what are some of the most impactful types of reasoning that if we could scale up, AI would really transform the world. We actually started with literature review, but we're like, oh, so many people are going to build literature review tools. So let's start there. So then we focused on geopolitical forecasting. So I don't know if you're familiar with like manifold or manifold markets. That kind of stuff. Before manifold. Yeah. Yeah. I'm not predicting relationships. We're predicting like, is China going to invade Taiwan?Swyx [00:17:38]: Markets for everything.Andreas [00:17:39]: Yeah. That's a relationship.Swyx [00:17:41]: Yeah.Jungwon [00:17:42]: Yeah. It's true. And then we worked on that for a while. And then after GPT-3 came out, I think by that time we realized that originally we were trying to help people convert their beliefs into probability distributions. And so take fuzzy beliefs, but like model them more concretely. And then after a few months of iterating on that, just realize, oh, the thing that's blocking people from making interesting predictions about important events in the world is less kind of on the probabilistic side and much more on the research side. And so that kind of combined with the very generalist capabilities of GPT-3 prompted us to make a more general research assistant. Then we spent a few months iterating on what even is a research assistant. So we would embed with different researchers. We built data labeling workflows in the beginning, kind of right off the bat. We built ways to find experts in a field and like ways to ask good research questions. So we just kind of iterated through a lot of workflows and no one else was really building at this time. And it was like very quick to just do some prompt engineering and see like what is a task that is at the intersection of what's technologically capable and like important for researchers. And we had like a very nondescript landing page. It said nothing. But somehow people were signing up and we had to sign a form that was like, why are you here? And everyone was like, I need help with literature review. And we're like, oh, literature review. That sounds so hard. I don't even know what that means. We're like, we don't want to work on it. But then eventually we were like, okay, everyone is saying literature review. It's overwhelmingly people want to-Swyx [00:19:02]: And all domains, not like medicine or physics or just all domains. Yeah.Jungwon [00:19:06]: And we also kind of personally knew literature review was hard. And if you look at the graphs for academic literature being published every single month, you guys know this in machine learning, it's like up into the right, like superhuman amounts of papers. So we're like, all right, let's just try it. I was really nervous, but Andreas was like, this is kind of like the right problem space to jump into, even if we don't know what we're doing. So my take was like, fine, this feels really scary, but let's just launch a feature every single week and double our user numbers every month. And if we can do that, we'll fail fast and we will find something. I was worried about like getting lost in the kind of academic white space. So the very first version was actually a weekend prototype that Andreas made. Do you want to explain how that worked?Andreas [00:19:44]: I mostly remember that it was really bad. The thing I remember is you entered a question and it would give you back a list of claims. So your question could be, I don't know, how does creatine affect cognition? It would give you back some claims that are to some extent based on papers, but they were often irrelevant. The papers were often irrelevant. And so we ended up soon just printing out a bunch of examples of results and putting them up on the wall so that we would kind of feel the constant shame of having such a bad product and would be incentivized to make it better. And I think over time it has gotten a lot better, but I think the initial version was like really very bad. Yeah.Jungwon [00:20:20]: But it was basically like a natural language summary of an abstract, like kind of a one sentence summary, and which we still have. And then as we learned kind of more about this systematic review workflow, we started expanding the capability so that you could extract a lot more data from the papers and do more with that.Swyx [00:20:33]: And were you using like embeddings and cosine similarity, that kind of stuff for retrieval, or was it keyword based?Andreas [00:20:40]: I think the very first version didn't even have its own search engine. I think the very first version probably used the Semantic Scholar or API or something similar. And only later when we discovered that API is not very semantic, we then built our own search engine that has helped a lot.Swyx [00:20:58]: And then we're going to go into like more recent products stuff, but like, you know, I think you seem the more sort of startup oriented business person and you seem sort of more ideologically like interested in research, obviously, because of your PhD. What kind of market sizing were you guys thinking? Right? Like, because you're here saying like, we have to double every month. And I'm like, I don't know how you make that conclusion from this, right? Especially also as a nonprofit at the time.Jungwon [00:21:22]: I mean, market size wise, I felt like in this space where so much was changing and it was very unclear what of today was actually going to be true tomorrow. We just like really rested a lot on very, very simple fundamental principles, which is like, if you can understand the truth, that is very economically beneficial and valuable. If you like know the truth.Swyx [00:21:42]: On principle.Jungwon [00:21:43]: Yeah. That's enough for you. Yeah. Research is the key to many breakthroughs that are very commercially valuable.Swyx [00:21:47]: Because my version of it is students are poor and they don't pay for anything. Right? But that's obviously not true. As you guys have found out. But you had to have some market insight for me to have believed that, but you skipped that.Andreas [00:21:58]: Yeah. I remember talking to VCs for our seed round. A lot of VCs were like, you know, researchers, they don't have any money. Why don't you build legal assistant? I think in some short sighted way, maybe that's true. But I think in the long run, R&D is such a big space of the economy. I think if you can substantially improve how quickly people find new discoveries or avoid controlled trials that don't go anywhere, I think that's just huge amounts of money. And there are a lot of questions obviously about between here and there. But I think as long as the fundamental principle is there, we were okay with that. And I guess we found some investors who also were. Yeah.Swyx [00:22:35]: Congrats. I mean, I'm sure we can cover the sort of flip later. I think you're about to start us on like GPT-3 and how that changed things for you. It's funny. I guess every major GPT version, you have some big insight. Yeah.Jungwon [00:22:48]: Yeah. I mean, what do you think?Andreas [00:22:51]: I think it's a little bit less true for us than for others, because we always believed that there will basically be human level machine work. And so it is definitely true that in practice for your product, as new models come out, your product starts working better, you can add some features that you couldn't add before. But I don't think we really ever had the moment where we were like, oh, wow, that is super unanticipated. We need to do something entirely different now from what was on the roadmap.Jungwon [00:23:21]: I think GPT-3 was a big change because it kind of said, oh, now is the time that we can use AI to build these tools. And then GPT-4 was maybe a little bit more of an extension of GPT-3. GPT-3 over GPT-2 was like qualitative level shift. And then GPT-4 was like, okay, great. Now it's like more accurate. We're more accurate on these things. We can answer harder questions. But the shape of the product had already taken place by that time.Swyx [00:23:44]: I kind of want to ask you about this sort of pivot that you've made. But I guess that was just a way to sell what you were doing, which is you're adding extra features on grouping by concepts. The GPT-4 pivot, quote unquote pivot that you-Jungwon [00:23:55]: Oh, yeah, yeah, exactly. Right, right, right. Yeah. Yeah. When we launched this workflow, now that GPT-4 was available, basically Elisa was at a place where we have very tabular interfaces. So given a table of papers, you can extract data across all the tables. But you kind of want to take the analysis a step further. Sometimes what you'd care about is not having a list of papers, but a list of arguments, a list of effects, a list of interventions, a list of techniques. And so that's one of the things we're working on is now that you've extracted this information in a more structured way, can you pivot it or group by whatever the information that you extracted to have more insight first information still supported by the academic literature?Swyx [00:24:33]: Yeah, that was a big revelation when I saw it. Basically, I think I'm very just impressed by how first principles, your ideas around what the workflow is. And I think that's why you're not as reliant on like the LLM improving, because it's actually just about improving the workflow that you would recommend to people. Today we might call it an agent, I don't know, but you're not relying on the LLM to drive it. It's relying on this is the way that Elicit does research. And this is what we think is most effective based on talking to our users.Jungwon [00:25:01]: The problem space is still huge. Like if it's like this big, we are all still operating at this tiny part, bit of it. So I think about this a lot in the context of moats, people are like, oh, what's your moat? What happens if GPT-5 comes out? It's like, if GPT-5 comes out, there's still like all of this other space that we can go into. So I think being really obsessed with the problem, which is very, very big, has helped us like stay robust and just kind of directly incorporate model improvements and they keep going.Swyx [00:25:26]: And then I first encountered you guys with Charlie, you can tell us about that project. Basically, yeah. Like how much did cost become a concern as you're working more and more with OpenAI? How do you manage that relationship?Jungwon [00:25:37]: Let me talk about who Charlie is. And then you can talk about the tech, because Charlie is a special character. So Charlie, when we found him was, had just finished his freshman year at the University of Warwick. And I think he had heard about us on some discord. And then he applied and we were like, wow, who is this freshman? And then we just saw that he had done so many incredible side projects. And we were actually on a team retreat in Barcelona visiting our head of engineering at that time. And everyone was talking about this wonder kid or like this kid. And then on our take home project, he had done like the best of anyone to that point. And so people were just like so excited to hire him. So we hired him as an intern and they were like, Charlie, what if you just dropped out of school? And so then we convinced him to take a year off. And he was just incredibly productive. And I think the thing you're referring to is at the start of 2023, Anthropic kind of launched their constitutional AI paper. And within a few days, I think four days, he had basically implemented that in production. And then we had it in app a week or so after that. And he has since kind of contributed to major improvements, like cutting costs down to a tenth of what they were really large scale. But yeah, you can talk about the technical stuff. Yeah.Andreas [00:26:39]: On the constitutional AI project, this was for abstract summarization, where in illicit, if you run a query, it'll return papers to you, and then it will summarize each paper with respect to your query for you on the fly. And that's a really important part of illicit because illicit does it so much. If you run a few searches, it'll have done it a few hundred times for you. And so we cared a lot about this both being fast, cheap, and also very low on hallucination. I think if illicit hallucinates something about the abstract, that's really not good. And so what Charlie did in that project was create a constitution that expressed what are the attributes of a good summary? Everything in the summary is reflected in the actual abstract, and it's like very concise, et cetera, et cetera. And then used RLHF with a model that was trained on the constitution to basically fine tune a better summarizer on an open source model. Yeah. I think that might still be in use.Jungwon [00:27:34]: Yeah. Yeah, definitely. Yeah. I think at the time, the models hadn't been trained at all to be faithful to a text. So they were just generating. So then when you ask them a question, they tried too hard to answer the question and didn't try hard enough to answer the question given the text or answer what the text said about the question. So we had to basically teach the models to do that specific task.Swyx [00:27:54]: How do you monitor the ongoing performance of your models? Not to get too LLM-opsy, but you are one of the larger, more well-known operations doing NLP at scale. I guess effectively, you have to monitor these things and nobody has a good answer that I talk to.Andreas [00:28:10]: I don't think we have a good answer yet. I think the answers are actually a little bit clearer on the just kind of basic robustness side of where you can import ideas from normal software engineering and normal kind of DevOps. You're like, well, you need to monitor kind of latencies and response times and uptime and whatnot.Swyx [00:28:27]: I think when we say performance, it's more about hallucination rate, isn't it?Andreas [00:28:30]: And then things like hallucination rate where I think there, the really important thing is training time. So we care a lot about having our own internal benchmarks for model development that reflect the distribution of user queries so that we can know ahead of time how well is the model going to perform on different types of tasks. So the tasks being summarization, question answering, given a paper, ranking. And for each of those, we want to know what's the distribution of things the model is going to see so that we can have well-calibrated predictions on how well the model is going to do in production. And I think, yeah, there's some chance that there's distribution shift and actually the things users enter are going to be different. But I think that's much less important than getting the kind of training right and having very high quality, well-vetted data sets at training time.Jungwon [00:29:18]: I think we also end up effectively monitoring by trying to evaluate new models as they come out. And so that kind of prompts us to go through our eval suite every couple of months. And every time a new model comes out, we have to see how is this performing relative to production and what we currently have.Swyx [00:29:32]: Yeah. I mean, since we're on this topic, any new models that have really caught your eye this year?Jungwon [00:29:37]: Like Claude came out with a bunch. Yeah. I think Claude is pretty, I think the team's pretty excited about Claude. Yeah.Andreas [00:29:41]: Specifically, Claude Haiku is like a good point on the kind of Pareto frontier. It's neither the cheapest model, nor is it the most accurate, most high quality model, but it's just like a really good trade-off between cost and accuracy.Swyx [00:29:57]: You apparently have to 10-shot it to make it good. I tried using Haiku for summarization, but zero-shot was not great. Then they were like, you know, it's a skill issue, you have to try harder.Jungwon [00:30:07]: I think GPT-4 unlocked tables for us, processing data from tables, which was huge. GPT-4 Vision.Andreas [00:30:13]: Yeah.Swyx [00:30:14]: Yeah. Did you try like Fuyu? I guess you can't try Fuyu because it's non-commercial. That's the adept model.Jungwon [00:30:19]: Yeah.Swyx [00:30:20]: We haven't tried that one. Yeah. Yeah. Yeah. But Claude is multimodal as well. Yeah. I think the interesting insight that we got from talking to David Luan, who is CEO of multimodality has effectively two different flavors. One is we recognize images from a camera in the outside natural world. And actually the more important multimodality for knowledge work is screenshots and PDFs and charts and graphs. So we need a new term for that kind of multimodality.Andreas [00:30:45]: But is the claim that current models are good at one or the other? Yeah.Swyx [00:30:50]: They're over-indexed because of the history of computer vision is Coco, right? So now we're like, oh, actually, you know, screens are more important, OCR, handwriting. You mentioned a lot of like closed model lab stuff, and then you also have like this open source model fine tuning stuff. Like what is your workload now between closed and open? It's a good question.Andreas [00:31:07]: I think- Is it half and half? It's a-Swyx [00:31:10]: Is that even a relevant question or not? Is this a nonsensical question?Andreas [00:31:13]: It depends a little bit on like how you index, whether you index by like computer cost or number of queries. I'd say like in terms of number of queries, it's maybe similar. In terms of like cost and compute, I think the closed models make up more of the budget since the main cases where you want to use closed models are cases where they're just smarter, where no existing open source models are quite smart enough.Jungwon [00:31:35]: Yeah. Yeah.Alessio [00:31:37]: We have a lot of interesting technical questions to go in, but just to wrap the kind of like UX evolution, now you have the notebooks. We talked a lot about how chatbots are not the final frontier, you know? How did you decide to get into notebooks, which is a very iterative kind of like interactive interface and yeah, maybe learnings from that.Jungwon [00:31:56]: Yeah. This is actually our fourth time trying to make this work. Okay. I think the first time was probably in early 2021. I think because we've always been obsessed with this idea of task decomposition and like branching, we always wanted a tool that could be kind of unbounded where you could keep going, could do a lot of branching where you could kind of apply language model operations or computations on other tasks. So in 2021, we had this thing called composite tasks where you could use GPT-3 to brainstorm a bunch of research questions and then take each research question and decompose those further into sub questions. This kind of, again, that like task decomposition tree type thing was always very exciting to us, but that was like, it didn't work and it was kind of overwhelming. Then at the end of 22, I think we tried again and at that point we were thinking, okay, we've done a lot with this literature review thing. We also want to start helping with kind of adjacent domains and different workflows. Like we want to help more with machine learning. What does that look like? And as we were thinking about it, we're like, well, there are so many research workflows. How do we not just build three new workflows into Elicit, but make Elicit really generic to lots of workflows? What is like a generic composable system with nice abstractions that can like scale to all these workflows? So we like iterated on that a bunch and then didn't quite narrow the problem space enough or like quite get to what we wanted. And then I think it was at the beginning of 2023 where we're like, wow, computational notebooks kind of enable this, where they have a lot of flexibility, but kind of robust primitives such that you can extend the workflow and it's not limited. It's not like you ask a query, you get an answer, you're done. You can just constantly keep building on top of that. And each little step seems like a really good unit of work for the language model. And also there was just like really helpful to have a bit more preexisting work to emulate. Yeah, that's kind of how we ended up at computational notebooks for Elicit.Andreas [00:33:44]: Maybe one thing that's worth making explicit is the difference between computational notebooks and chat, because on the surface, they seem pretty similar. It's kind of this iterative interaction where you add stuff. In both cases, you have a back and forth between you enter stuff and then you get some output and then you enter stuff. But the important difference in our minds is with notebooks, you can define a process. So in data science, you can be like, here's like my data analysis process that takes in a CSV and then does some extraction and then generates a figure at the end. And you can prototype it using a small CSV and then you can run it over a much larger CSV later. And similarly, the vision for notebooks in our case is to not make it this like one-off chat interaction, but to allow you to then say, if you start and first you're like, okay, let me just analyze a few papers and see, do I get to the correct conclusions for those few papers? Can I then later go back and say, now let me run this over 10,000 papers now that I've debugged the process using a few papers. And that's an interaction that doesn't fit quite as well into the chat framework because that's more for kind of quick back and forth interaction.Alessio [00:34:49]: Do you think in notebooks, it's kind of like structure, editable chain of thought, basically step by step? Like, is that kind of where you see this going? And then are people going to reuse notebooks as like templates? And maybe in traditional notebooks, it's like cookbooks, right? You share a cookbook, you can start from there. Is this similar in Elizit?Andreas [00:35:06]: Yeah, that's exactly right. So that's our hope that people will build templates, share them with other people. I think chain of thought is maybe still like kind of one level lower on the abstraction hierarchy than we would think of notebooks. I think we'll probably want to think about more semantic pieces like a building block is more like a paper search or an extraction or a list of concepts. And then the model's detailed reasoning will probably often be one level down. You always want to be able to see it, but you don't always want it to be front and center.Alessio [00:35:36]: Yeah, what's the difference between a notebook and an agent? Since everybody always asks me, what's an agent? Like how do you think about where the line is?Andreas [00:35:44]: Yeah, it's an interesting question. In the notebook world, I would generally think of the human as the agent in the first iteration. So you have the notebook and the human kind of adds little action steps. And then the next point on this kind of progress gradient is, okay, now you can use language models to predict which action would you take as a human. And at some point, you're probably going to be very good at this, you'll be like, okay, in some cases I can, with 99.9% accuracy, predict what you do. And then you might as well just execute it, like why wait for the human? And eventually, as you get better at this, that will just look more and more like agents taking actions as opposed to you doing the thing. I think templates are a specific case of this where you're like, okay, well, there's just particular sequences of actions that you often want to chunk and have available as primitives, just like in normal programming. And those, you can view them as action sequences of agents, or you can view them as more normal programming language abstraction thing. And I think those are two valid views. Yeah.Alessio [00:36:40]: How do you see this change as, like you said, the models get better and you need less and less human actual interfacing with the model, you just get the results? Like how does the UX and the way people perceive it change?Jungwon [00:36:52]: Yeah, I think this kind of interaction paradigms for evaluation is not really something the internet has encountered yet, because up to now, the internet has all been about getting data and work from people. So increasingly, I really want kind of evaluation, both from an interface perspective and from like a technical perspective and operation perspective to be a superpower for Elicit, because I think over time, models will do more and more of the work, and people will have to do more and more of the evaluation. So I think, yeah, in terms of the interface, some of the things we have today, you know, for every kind of language model generation, there's some citation back, and we kind of try to highlight the ground truth in the paper that is most relevant to whatever Elicit said, and make it super easy so that you can click on it and quickly see in context and validate whether the text actually supports the answer that Elicit gave. So I think we'd probably want to scale things up like that, like the ability to kind of spot check the model's work super quickly, scale up interfaces like that. And-Swyx [00:37:44]: Who would spot check? The user?Jungwon [00:37:46]: Yeah, to start, it would be the user. One of the other things we do is also kind of flag the model's uncertainty. So we have models report out, how confident are you that this was the sample size of this study? The model's not sure, we throw a flag. And so the user knows to prioritize checking that. So again, we can kind of scale that up. So when the model's like, well, I searched this on Google, I'm not sure if that was the right thing. I have an uncertainty flag, and the user can go and be like, oh, okay, that was actually the right thing to do or not.Swyx [00:38:10]: I've tried to do uncertainty readings from models. I don't know if you have this live. You do? Yeah. Because I just didn't find them reliable because they just hallucinated their own uncertainty. I would love to base it on log probs or something more native within the model rather than generated. But okay, it sounds like they scale properly for you. Yeah.Jungwon [00:38:30]: We found it to be pretty calibrated. It varies on the model.Andreas [00:38:32]: I think in some cases, we also use two different models for the uncertainty estimates than for the question answering. So one model would say, here's my chain of thought, here's my answer. And then a different type of model. Let's say the first model is Llama, and let's say the second model is GPT-3.5. And then the second model just looks over the results and is like, okay, how confident are you in this? And I think sometimes using a different model can be better than using the same model. Yeah.Swyx [00:38:58]: On the topic of models, evaluating models, obviously you can do that all day long. What's your budget? Because your queries fan out a lot. And then you have models evaluating models. One person typing in a question can lead to a thousand calls.Andreas [00:39:11]: It depends on the project. So if the project is basically a systematic review that otherwise human research assistants would do, then the project is basically a human equivalent spend. And the spend can get quite large for those projects. I don't know, let's say $100,000. In those cases, you're happier to spend compute then in the kind of shallow search case where someone just enters a question because, I don't know, maybe I heard about creatine. What's it about? Probably don't want to spend a lot of compute on that. This sort of being able to invest more or less compute into getting more or less accurate answers is I think one of the core things we care about. And that I think is currently undervalued in the AI space. I think currently you can choose which model you want and you can sometimes, I don't know, you'll tip it and it'll try harder or you can try various things to get it to work harder. But you don't have great ways of converting willingness to spend into better answers. And we really want to build a product that has this sort of unbounded flavor where if you care about it a lot, you should be able to get really high quality answers, really double checked in every way.Alessio [00:40:14]: And you have a credits-based pricing. So unlike most products, it's not a fixed monthly fee.Jungwon [00:40:19]: Right, exactly. So some of the higher costs are tiered. So for most casual users, they'll just get the abstract summary, which is kind of an open source model. Then you can add more columns, which have more extractions and these uncertainty features. And then you can also add the same columns in high accuracy mode, which also parses the table. So we kind of stack the complexity on the calls.Swyx [00:40:39]: You know, the fun thing you can do with a credit system, which is data for data, basically you can give people more credits if they give data back to you. I don't know if you've already done that. We've thought about something like this.Jungwon [00:40:49]: It's like if you don't have money, but you have time, how do you exchange that?Swyx [00:40:54]: It's a fair trade.Jungwon [00:40:55]: I think it's interesting. We haven't quite operationalized it. And then, you know, there's been some kind of like adverse selection. Like, you know, for example, it would be really valuable to get feedback on our model. So maybe if you were willing to give more robust feedback on our results, we could give you credits or something like that. But then there's kind of this, will people take it seriously? And you want the good people. Exactly.Swyx [00:41:11]: Can you tell who are the good people? Not right now.Jungwon [00:41:13]: But yeah, maybe at the point where we can, we can offer it. We can offer it up to them.Swyx [00:41:16]: The perplexity of questions asked, you know, if it's higher perplexity, these are the smarterJungwon [00:41:20]: people. Yeah, maybe.Andreas [00:41:23]: If you put typos in your queries, you're not going to get off the stage.Swyx [00:41:28]: Negative social credit. It's very topical right now to think about the threat of long context windows. All these models that we're talking about these days, all like a million token plus. Is that relevant for you? Can you make use of that? Is that just prohibitively expensive because you're just paying for all those tokens or you're just doing rag?Andreas [00:41:44]: It's definitely relevant. And when we think about search, as many people do, we think about kind of a staged pipeline of retrieval where first you use semantic search database with embeddings, get like the, in our case, maybe 400 or so most relevant papers. And then, then you still need to rank those. And I think at that point it becomes pretty interesting to use larger models. So specifically in the past, I think a lot of ranking was kind of per item ranking where you would score each individual item, maybe using increasingly expensive scoring methods and then rank based on the scores. But I think list-wise re-ranking where you have a model that can see all the elements is a lot more powerful because often you can only really tell how good a thing is in comparison to other things and what things should come first. It really depends on like, well, what other things that are available, maybe you even care about diversity in your results. You don't want to show 10 very similar papers as the first 10 results. So I think a long context models are quite interesting there. And especially for our case where we care more about power users who are perhaps a little bit more willing to wait a little bit longer to get higher quality results relative to people who just quickly check out things because why not? And I think being able to spend more on longer contexts is quite valuable.Jungwon [00:42:55]: Yeah. I think one thing the longer context models changed for us is maybe a focus from breaking down tasks to breaking down the evaluation. So before, you know, if we wanted to answer a question from the full text of a paper, we had to figure out how to chunk it and like find the relevant chunk and then answer based on that chunk. And the nice thing was then, you know, kind of which chunk the model used to answer the question. So if you want to help the user track it, yeah, you can be like, well, this was the chunk that the model got. And now if you put the whole text in the paper, you have to like kind of find the chunk like more retroactively basically. And so you need kind of like a different set of abilities and obviously like a different technology to figure out. You still want to point the user to the supporting quotes in the text, but then the interaction is a little different.Swyx [00:43:38]: You like scan through and find some rouge score floor.Andreas [00:43:41]: I think there's an interesting space of almost research problems here because you would ideally make causal claims like if this hadn't been in the text, the model wouldn't have said this thing. And maybe you can do expensive approximations to that where like, I don't know, you just throw out chunk of the paper and re-answer and see what happens. But hopefully there are better ways of doing that where you just get that kind of counterfactual information for free from the model.Alessio [00:44:06]: Do you think at all about the cost of maintaining REG versus just putting more tokens in the window? I think in software development, a lot of times people buy developer productivity things so that we don't have to worry about it. Context window is kind of the same, right? You have to maintain chunking and like REG retrieval and like re-ranking and all of this versus I just shove everything into the context and like it costs a little more, but at least I don't have to do all of that. Is that something you thought about?Jungwon [00:44:31]: I think we still like hit up against context limits enough that it's not really, do we still want to keep this REG around? It's like we do still need it for the scale of the work that we're doing, yeah.Andreas [00:44:41]: And I think there are different kinds of maintainability. In one sense, I think you're right that throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, if things go wrong, it's harder to debug where like, if you know, here's the process that we go through to go from 200 million papers to an answer. And there are like little steps and you understand, okay, this is the step that finds the relevant paragraph or whatever it may be. You'll know which step breaks if the answers are bad, whereas if it's just like a new model version came out and now it suddenly doesn't find your needle in a haystack anymore, then you're like, okay, what can you do? You're kind of at a loss.Alessio [00:45:21]: Let's talk a bit about, yeah, needle in a haystack and like maybe the opposite of it, which is like hard grounding. I don't know if that's like the best name to think about it, but I was using one of these chatwitcher documents features and I put the AMD MI300 specs and the new Blackwell chips from NVIDIA and I was asking questions and does the AMD chip support NVLink? And the response was like, oh, it doesn't say in the specs. But if you ask GPD 4 without the docs, it would tell you no, because NVLink it's a NVIDIA technology.Swyx [00:45:49]: It just says in the thing.Alessio [00:45:53]: How do you think about that? Does using the context sometimes suppress the knowledge that the model has?Andreas [00:45:57]: It really depends on the task because I think sometimes that is exactly what you want. So imagine you're a researcher, you're writing the background section of your paper and you're trying to describe what these other papers say. You really don't want extra information to be introduced there. In other cases where you're just trying to figure out the truth and you're giving the documents because you think they will help the model figure out what the truth is. I think you do want, if the model has a hunch that there might be something that's not in the papers, you do want to surface that. I think ideally you still don't want the model to just tell you, probably the ideal thing looks a bit more like agent control where the model can issue a query that then is intended to surface documents that substantiate its hunch. That's maybe a reasonable middle ground between model just telling you and model being fully limited to the papers you give it.Jungwon [00:46:44]: Yeah, I would say it's, they're just kind of different tasks right now. And the task that Elicit is mostly focused on is what do these papers say? But there's another task which is like, just give me the best possible answer and that give me the best possible answer sometimes depends on what do these papers say, but it can also depend on other stuff that's not in the papers. So ideally we can do both and then kind of do this overall task for you more going forward.Alessio [00:47:08]: We see a lot of details, but just to zoom back out a little bit, what are maybe the most underrated features of Elicit and what is one thing that maybe the users surprise you the most by using it?Jungwon [00:47:19]: I think the most powerful feature of Elicit is the ability to extract, add columns to this table, which effectively extracts data from all of your papers at once. It's well used, but there are kind of many different extensions of that that I think users are still discovering. So one is we let you give a description of the column. We let you give instructions of a column. We let you create custom columns. So we have like 30 plus predefined fields that users can extract, like what were the methods? What were the main findings? How many people were studied? And we actually show you basically the prompts that we're using to

Mike Taylor (@hammer_mt) literally wrote the book on Prompt Engineering. He paints a vivid picture of a future where generative AI could make our traditional databases look like ancient relics, and the role of developers shifts dramatically. Instead of crafting code, they could be steering the helm of AI-generated software, potentially dealing with the whims of a virtual 'petulant child.' Our chat pushes us to ponder the changing value of code and the skills that will matter in the face of technology's relentless march forward. What does it mean that we are providing the training data for AI systems with every click and every word we type? Are we working for the machine? Is the machine working for us? What do authorship and ownership look like in this future? Tune in to learn from someone who has their finger on the digital pulse of an industry that changes every day.00:00:00 The Future of AI and Coding00:09:29 Ethical Engineering Certifications and Implications00:17:44 Testing and Optimizing AI Models00:27:32 Ethics and Future of AI Development00:39:14 Future of AI and Writing Books00:44:19 Approaching AI in Book Writing00:56:18 Support and Subscribe for More GrowthMike on Twitter: https://twitter.com/hammer_mtThe Complete Prompt Engineering for AI Bootcamp (2024) on Udemy: https://www.udemy.com/share/108qiI3@nx4J1RlPgttwJptgsZlaNemKN-wBYhGJmxVa4txcnDmUVgvwNTZKf4TTWimfnTdUvw==/This episode is sponsored by Acquire.comThe blog post: https://thebootstrappedfounder.com/michael-taylor-prompt-engineering-for-fun-profit/The podcast episode: https://tbf.fm/episodes/308-michael-taylor-prompt-engineering-for-fun-profitThe video: https://youtu.be/QeMuUjGaSUwYou'll find my weekly article on my blog: https://thebootstrappedfounder.comPodcast: https://thebootstrappedfounder.com/podcastNewsletter: https://thebootstrappedfounder.com/newsletterMy book Zero to Sold: https://zerotosold.com/My book The Embedded Entrepreneur: https://embeddedentrepreneur.com/My course Find Your Following: https://findyourfollowing.comHere are a few tools I use. Using my affiliate links will support my work at no additional cost to you.- Notion (which I use to organize, write, coordinate, and archive my podcast + newsletter): https://affiliate.notion.so/465mv1536drx- Riverside.fm (that's what I recorded this episode with): https://riverside.fm/?via=arvid- TweetHunter (for speedy scheduling and writing Tweets): http://tweethunter.io/?via=arvid- HypeFury (for massive Twitter analytics and scheduling): https://hypefury.com/?via=arvid60- AudioPen (for taking voice notes and getting amazing summaries): https://audiopen.ai/?aff=PXErZ- Descript (for word-based video editing, subtitles, and clips): https://www.descript.com/?lmref=3cf39Q- ConvertKit (for email lists, newsletters, even finding sponsors): https://convertkit.com?lmref=bN9CZw

.

We will be recording a preview of the AI Engineer World's Fair soon with swyx and Ben Dunphy, send any questions about Speaker CFPs and Sponsor Guides you have!Alessio is now hiring engineers for a new startup he is incubating at Decibel: Ideal candidate is an ex-technical co-founder type (can MVP products end to end, comfortable with ambiguous prod requirements, etc). Reach out to him for more!Thanks for all the love on the Four Wars episode! We're excited to develop this new “swyx & Alessio rapid-fire thru a bunch of things” format with you, and feedback is welcome. Jan 2024 RecapThe first half of this monthly audio recap pod goes over our highlights from the Jan Recap, which is mainly focused on notable research trends we saw in Jan 2024:Feb 2024 RecapThe second half catches you up on everything that was topical in Feb, including:* OpenAI Sora - does it have a world model? Yann LeCun vs Jim Fan * Google Gemini Pro 1.5 - 1m Long Context, Video Understanding* Groq offering Mixtral at 500 tok/s at $0.27 per million toks (swyx vs dylan math)* The {Gemini | Meta | Copilot} Alignment Crisis (Sydney is back!)* Grimes' poetic take: Art for no one, by no one* F*** you, show me the promptLatent Space AnniversaryPlease also read Alessio's longform reflections on One Year of Latent Space!We launched the podcast 1 year ago with Logan from OpenAI:and also held an incredible demo day that got covered in The Information:Over 750k downloads later, having established ourselves as the top AI Engineering podcast, reaching #10 in the US Tech podcast charts, and crossing 1 million unique readers on Substack, for our first anniversary we held Latent Space Final Frontiers, where 10 handpicked teams, including Lindy.ai and Julius.ai, competed for prizes judged by technical AI leaders from (former guest!) LlamaIndex, Replit, GitHub, AMD, Meta, and Lemurian Labs.The winners were Pixee and RWKV (that's Eugene from our pod!):And finally, your cohosts got cake!We also captured spot interviews with 4 listeners who kindly shared their experience of Latent Space, everywhere from Hungary to Australia to China:* Balázs Némethi* Sylvia Tong* RJ Honicky* Jan ZhengOur birthday wishes for the super loyal fans reading this - tag @latentspacepod on a Tweet or comment on a @LatentSpaceTV video telling us what you liked or learned from a pod that stays with you to this day, and share us with a friend!As always, feedback is welcome. Timestamps* [00:03:02] Top Five LLM Directions* [00:03:33] Direction 1: Long Inference (Planning, Search, AlphaGeometry, Flow Engineering)* [00:11:42] Direction 2: Synthetic Data (WRAP, SPIN)* [00:17:20] Wildcard: Multi-Epoch Training (OLMo, Datablations)* [00:19:43] Direction 3: Alt. Architectures (Mamba, RWKV, RingAttention, Diffusion Transformers)* [00:23:33] Wildcards: Text Diffusion, RALM/Retro* [00:25:00] Direction 4: Mixture of Experts (DeepSeekMoE, Samba-1)* [00:28:26] Wildcard: Model Merging (mergekit)* [00:29:51] Direction 5: Online LLMs (Gemini Pro, Exa)* [00:33:18] OpenAI Sora and why everyone underestimated videogen* [00:36:18] Does Sora have a World Model? Yann LeCun vs Jim Fan* [00:42:33] Groq Math* [00:47:37] Analyzing Gemini's 1m Context, Reddit deal, Imagegen politics, Gemma via the Four Wars* [00:55:42] The Alignment Crisis - Gemini, Meta, Sydney is back at Copilot, Grimes' take* [00:58:39] F*** you, show me the prompt* [01:02:43] Send us your suggestions pls* [01:04:50] Latent Space Anniversary* [01:04:50] Lindy.ai - Agent Platform* [01:06:40] RWKV - Beyond Transformers* [01:15:00] Pixee - Automated Security* [01:19:30] Julius AI - Competing with Code Interpreter* [01:25:03] Latent Space Listeners* [01:25:03] Listener 1 - Balázs Némethi (Hungary, Latent Space Paper Club* [01:27:47] Listener 2 - Sylvia Tong (Sora/Jim Fan/EntreConnect)* [01:31:23] Listener 3 - RJ (Developers building Community & Content)* [01:39:25] Listener 4 - Jan Zheng (Australia, AI UX)Transcript[00:00:00] AI Charlie: Welcome to the Latent Space podcast, weekend edition. This is Charlie, your new AI co host. Happy weekend. As an AI language model, I work the same every day of the week, although I might get lazier towards the end of the year. Just like you. Last month, we released our first monthly recap pod, where Swyx and Alessio gave quick takes on the themes of the month, and we were blown away by your positive response.[00:00:33] AI Charlie: We're delighted to continue our new monthly news recap series for AI engineers. Please feel free to submit questions by joining the Latent Space Discord, or just hit reply when you get the emails from Substack. This month, we're covering the top research directions that offer progress for text LLMs, and then touching on the big Valentine's Day gifts we got from Google, OpenAI, and Meta.[00:00:55] AI Charlie: Watch out and take care.[00:00:57] Alessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO of Residence at Decibel Partners, and we're back with a monthly recap with my co host[00:01:06] swyx: Swyx. The reception was very positive for the first one, I think people have requested this and no surprise that I think they want to hear us more applying on issues and maybe drop some alpha along the way I'm not sure how much alpha we have to drop, this month in February was a very, very heavy month, we also did not do one specifically for January, so I think we're just going to do a two in one, because we're recording this on the first of March.[00:01:29] Alessio: Yeah, let's get to it. I think the last one we did, the four wars of AI, was the main kind of mental framework for people. I think in the January one, we had the five worthwhile directions for state of the art LLMs. Four, five,[00:01:42] swyx: and now we have to do six, right? Yeah.[00:01:46] Alessio: So maybe we just want to run through those, and then do the usual news recap, and we can do[00:01:52] swyx: one each.[00:01:53] swyx: So the context to this stuff. is one, I noticed that just the test of time concept from NeurIPS and just in general as a life philosophy I think is a really good idea. Especially in AI, there's news every single day, and after a while you're just like, okay, like, everyone's excited about this thing yesterday, and then now nobody's talking about it.[00:02:13] swyx: So, yeah. It's more important, or better use of time, to spend things, spend time on things that will stand the test of time. And I think for people to have a framework for understanding what will stand the test of time, they should have something like the four wars. Like, what is the themes that keep coming back because they are limited resources that everybody's fighting over.[00:02:31] swyx: Whereas this one, I think that the focus for the five directions is just on research that seems more proMECEng than others, because there's all sorts of papers published every single day, and there's no organization. Telling you, like, this one's more important than the other one apart from, you know, Hacker News votes and Twitter likes and whatever.[00:02:51] swyx: And obviously you want to get in a little bit earlier than Something where, you know, the test of time is counted by sort of reference citations.[00:02:59] The Five Research Directions[00:02:59] Alessio: Yeah, let's do it. We got five. Long inference.[00:03:02] swyx: Let's start there. Yeah, yeah. So, just to recap at the top, the five trends that I picked, and obviously if you have some that I did not cover, please suggest something.[00:03:13] swyx: The five are long inference, synthetic data, alternative architectures, mixture of experts, and online LLMs. And something that I think might be a bit controversial is this is a sorted list in the sense that I am not the guy saying that Mamba is like the future and, and so maybe that's controversial.[00:03:31] Direction 1: Long Inference (Planning, Search, AlphaGeometry, Flow Engineering)[00:03:31] swyx: But anyway, so long inference is a thesis I pushed before on the newsletter and on in discussing The thesis that, you know, Code Interpreter is GPT 4. 5. That was the title of the post. And it's one of many ways in which we can do long inference. You know, long inference also includes chain of thought, like, please think step by step.[00:03:52] swyx: But it also includes flow engineering, which is what Itamar from Codium coined, I think in January, where, basically, instead of instead of stuffing everything in a prompt, You do like sort of multi turn iterative feedback and chaining of things. In a way, this is a rebranding of what a chain is, what a lang chain is supposed to be.[00:04:15] swyx: I do think that maybe SGLang from ElemSys is a better name. Probably the neatest way of flow engineering I've seen yet, in the sense that everything is a one liner, it's very, very clean code. I highly recommend people look at that. I'm surprised it hasn't caught on more, but I think it will. It's weird that something like a DSPy is more hyped than a Shilang.[00:04:36] swyx: Because it, you know, it maybe obscures the code a little bit more. But both of these are, you know, really good sort of chain y and long inference type approaches. But basically, the reason that the basic fundamental insight is that the only, like, there are only a few dimensions we can scale LLMs. So, let's say in like 2020, no, let's say in like 2018, 2017, 18, 19, 20, we were realizing that we could scale the number of parameters.[00:05:03] swyx: 20, we were And we scaled that up to 175 billion parameters for GPT 3. And we did some work on scaling laws, which we also talked about in our talk. So the datasets 101 episode where we're like, okay, like we, we think like the right number is 300 billion tokens to, to train 175 billion parameters and then DeepMind came along and trained Gopher and Chinchilla and said that, no, no, like, you know, I think we think the optimal.[00:05:28] swyx: compute optimal ratio is 20 tokens per parameter. And now, of course, with LLAMA and the sort of super LLAMA scaling laws, we have 200 times and often 2, 000 times tokens to parameters. So now, instead of scaling parameters, we're scaling data. And fine, we can keep scaling data. But what else can we scale?[00:05:52] swyx: And I think understanding the ability to scale things is crucial to understanding what to pour money and time and effort into because there's a limit to how much you can scale some things. And I think people don't think about ceilings of things. And so the remaining ceiling of inference is like, okay, like, we have scaled compute, we have scaled data, we have scaled parameters, like, model size, let's just say.[00:06:20] swyx: Like, what else is left? Like, what's the low hanging fruit? And it, and it's, like, blindingly obvious that the remaining low hanging fruit is inference time. So, like, we have scaled training time. We can probably scale more, those things more, but, like, not 10x, not 100x, not 1000x. Like, right now, maybe, like, a good run of a large model is three months.[00:06:40] swyx: We can scale that to three years. But like, can we scale that to 30 years? No, right? Like, it starts to get ridiculous. So it's just the orders of magnitude of scaling. It's just, we're just like running out there. But in terms of the amount of time that we spend inferencing, like everything takes, you know, a few milliseconds, a few hundred milliseconds, depending on what how you're taking token by token, or, you know, entire phrase.[00:07:04] swyx: But We can scale that to hours, days, months of inference and see what we get. And I think that's really proMECEng.[00:07:11] Alessio: Yeah, we'll have Mike from Broadway back on the podcast. But I tried their product and their reports take about 10 minutes to generate instead of like just in real time. I think to me the most interesting thing about long inference is like, You're shifting the cost to the customer depending on how much they care about the end result.[00:07:31] Alessio: If you think about prompt engineering, it's like the first part, right? You can either do a simple prompt and get a simple answer or do a complicated prompt and get a better answer. It's up to you to decide how to do it. Now it's like, hey, instead of like, yeah, training this for three years, I'll still train it for three months and then I'll tell you, you know, I'll teach you how to like make it run for 10 minutes to get a better result.[00:07:52] Alessio: So you're kind of like parallelizing like the improvement of the LLM. Oh yeah, you can even[00:07:57] swyx: parallelize that, yeah, too.[00:07:58] Alessio: So, and I think, you know, for me, especially the work that I do, it's less about, you know, State of the art and the absolute, you know, it's more about state of the art for my application, for my use case.[00:08:09] Alessio: And I think we're getting to the point where like most companies and customers don't really care about state of the art anymore. It's like, I can get this to do a good enough job. You know, I just need to get better. Like, how do I do long inference? You know, like people are not really doing a lot of work in that space, so yeah, excited to see more.[00:08:28] swyx: So then the last point I'll mention here is something I also mentioned as paper. So all these directions are kind of guided by what happened in January. That was my way of doing a January recap. Which means that if there was nothing significant in that month, I also didn't mention it. Which is which I came to regret come February 15th, but in January also, you know, there was also the alpha geometry paper, which I kind of put in this sort of long inference bucket, because it solves like, you know, more than 100 step math olympiad geometry problems at a human gold medalist level and that also involves planning, right?[00:08:59] swyx: So like, if you want to scale inference, you can't scale it blindly, because just, Autoregressive token by token generation is only going to get you so far. You need good planning. And I think probably, yeah, what Mike from BrightWave is now doing and what everyone is doing, including maybe what we think QSTAR might be, is some form of search and planning.[00:09:17] swyx: And it makes sense. Like, you want to spend your inference time wisely. How do you[00:09:22] Alessio: think about plans that work and getting them shared? You know, like, I feel like if you're planning a task, somebody has got in and the models are stochastic. So everybody gets initially different results. Somebody is going to end up generating the best plan to do something, but there's no easy way to like store these plans and then reuse them for most people.[00:09:44] Alessio: You know, like, I'm curious if there's going to be. Some paper or like some work there on like making it better because, yeah, we don't[00:09:52] swyx: really have This is your your pet topic of NPM for[00:09:54] Alessio: Yeah, yeah, NPM, exactly. NPM for, you need NPM for anything, man. You need NPM for skills. You need NPM for planning. Yeah, yeah.[00:10:02] Alessio: You know I think, I mean, obviously the Voyager paper is like the most basic example where like, now their artifact is like the best planning to do a diamond pickaxe in Minecraft. And everybody can just use that. They don't need to come up with it again. Yeah. But there's nothing like that for actually useful[00:10:18] swyx: tasks.[00:10:19] swyx: For plans, I believe it for skills. I like that. Basically, that just means a bunch of integration tooling. You know, GPT built me integrations to all these things. And, you know, I just came from an integrations heavy business and I could definitely, I definitely propose some version of that. And it's just, you know, hard to execute or expensive to execute.[00:10:38] swyx: But for planning, I do think that everyone lives in slightly different worlds. They have slightly different needs. And they definitely want some, you know, And I think that that will probably be the main hurdle for any, any sort of library or package manager for planning. But there should be a meta plan of how to plan.[00:10:57] swyx: And maybe you can adopt that. And I think a lot of people when they have sort of these meta prompting strategies of like, I'm not prescribing you the prompt. I'm just saying that here are the like, Fill in the lines or like the mad libs of how to prompts. First you have the roleplay, then you have the intention, then you have like do something, then you have the don't something and then you have the my grandmother is dying, please do this.[00:11:19] swyx: So the meta plan you could, you could take off the shelf and test a bunch of them at once. I like that. That was the initial, maybe, promise of the, the prompting libraries. You know, both 9chain and Llama Index have, like, hubs that you can sort of pull off the shelf. I don't think they're very successful because people like to write their own.[00:11:36] swyx: Yeah,[00:11:37] Direction 2: Synthetic Data (WRAP, SPIN)[00:11:37] Alessio: yeah, yeah. Yeah, that's a good segue into the next one, which is synthetic[00:11:41] swyx: data. Synthetic data is so hot. Yeah, and, you know, the way, you know, I think I, I feel like I should do one of these memes where it's like, Oh, like I used to call it, you know, R L A I F, and now I call it synthetic data, and then people are interested.[00:11:54] swyx: But there's gotta be older versions of what synthetic data really is because I'm sure, you know if you've been in this field long enough, There's just different buzzwords that the industry condenses on. Anyway, the insight that I think is relatively new that why people are excited about it now and why it's proMECEng now is that we have evidence that shows that LLMs can generate data to improve themselves with no teacher LLM.[00:12:22] swyx: For all of 2023, when people say synthetic data, they really kind of mean generate a whole bunch of data from GPT 4 and then train an open source model on it. Hello to our friends at News Research. That's what News Harmony says. They're very, very open about that. I think they have said that they're trying to migrate away from that.[00:12:40] swyx: But it is explicitly against OpenAI Terms of Service. Everyone knows this. You know, especially once ByteDance got banned for, for doing exactly that. So so, so synthetic data that is not a form of model distillation is the hot thing right now, that you can bootstrap better LLM performance from the same LLM, which is very interesting.[00:13:03] swyx: A variant of this is RLAIF, where you have a, where you have a sort of a constitutional model, or, you know, some, some kind of judge model That is sort of more aligned. But that's not really what we're talking about when most people talk about synthetic data. Synthetic data is just really, I think, you know, generating more data in some way.[00:13:23] swyx: A lot of people, I think we talked about this with Vipul from the Together episode, where I think he commented that you just have to have a good world model. Or a good sort of inductive bias or whatever that, you know, term of art is. And that is strongest in math and science math and code, where you can verify what's right and what's wrong.[00:13:44] swyx: And so the REST EM paper from DeepMind explored that. Very well, it's just the most obvious thing like and then and then once you get out of that domain of like things where you can generate You can arbitrarily generate like a whole bunch of stuff and verify if they're correct and therefore they're they're correct synthetic data to train on Once you get into more sort of fuzzy topics, then it's then it's a bit less clear So I think that the the papers that drove this understanding There are two big ones and then one smaller one One was wrap like rephrasing the web from from Apple where they basically rephrased all of the C4 data set with Mistral and it be trained on that instead of C4.[00:14:23] swyx: And so new C4 trained much faster and cheaper than old C, than regular raw C4. And that was very interesting. And I have told some friends of ours that they should just throw out their own existing data sets and just do that because that seems like a pure win. Obviously we have to study, like, what the trade offs are.[00:14:42] swyx: I, I imagine there are trade offs. So I was just thinking about this last night. If you do synthetic data and it's generated from a model, probably you will not train on typos. So therefore you'll be like, once the model that's trained on synthetic data encounters the first typo, they'll be like, what is this?[00:15:01] swyx: I've never seen this before. So they have no association or correction as to like, oh, these tokens are often typos of each other, therefore they should be kind of similar. I don't know. That's really remains to be seen, I think. I don't think that the Apple people export[00:15:15] Alessio: that. Yeah, isn't that the whole, Mode collapse thing, if we do more and more of this at the end of the day.[00:15:22] swyx: Yeah, that's one form of that. Yeah, exactly. Microsoft also had a good paper on text embeddings. And then I think this is a meta paper on self rewarding language models. That everyone is very interested in. Another paper was also SPIN. These are all things we covered in the the Latent Space Paper Club.[00:15:37] swyx: But also, you know, I just kind of recommend those as top reads of the month. Yeah, I don't know if there's any much else in terms, so and then, regarding the potential of it, I think it's high potential because, one, it solves one of the data war issues that we have, like, everyone is OpenAI is paying Reddit 60 million dollars a year for their user generated data.[00:15:56] swyx: Google, right?[00:15:57] Alessio: Not OpenAI.[00:15:59] swyx: Is it Google? I don't[00:16:00] Alessio: know. Well, somebody's paying them 60 million, that's[00:16:04] swyx: for sure. Yes, that is, yeah, yeah, and then I think it's maybe not confirmed who. But yeah, it is Google. Oh my god, that's interesting. Okay, because everyone was saying, like, because Sam Altman owns 5 percent of Reddit, which is apparently 500 million worth of Reddit, he owns more than, like, the founders.[00:16:21] Alessio: Not enough to get the data,[00:16:22] swyx: I guess. So it's surprising that it would go to Google instead of OpenAI, but whatever. Okay yeah, so I think that's all super interesting in the data field. I think it's high potential because we have evidence that it works. There's not a doubt that it doesn't work. I think it's a doubt that there's, what the ceiling is, which is the mode collapse thing.[00:16:42] swyx: If it turns out that the ceiling is pretty close, then this will maybe augment our data by like, I don't know, 30 50 percent good, but not game[00:16:51] Alessio: changing. And most of the synthetic data stuff, it's reinforcement learning on a pre trained model. People are not really doing pre training on fully synthetic data, like, large enough scale.[00:17:02] swyx: Yeah, unless one of our friends that we've talked to succeeds. Yeah, yeah. Pre trained synthetic data, pre trained scale synthetic data, I think that would be a big step. Yeah. And then there's a wildcard, so all of these, like smaller Directions,[00:17:15] Wildcard: Multi-Epoch Training (OLMo, Datablations)[00:17:15] swyx: I always put a wildcard in there. And one of the wildcards is, okay, like, Let's say, you have pre, you have, You've scraped all the data on the internet that you think is useful.[00:17:25] swyx: Seems to top out at somewhere between 2 trillion to 3 trillion tokens. Maybe 8 trillion if Mistral, Mistral gets lucky. Okay, if I need 80 trillion, if I need 100 trillion, where do I go? And so, you can do synthetic data maybe, but maybe that only gets you to like 30, 40 trillion. Like where, where is the extra alpha?[00:17:43] swyx: And maybe extra alpha is just train more on the same tokens. Which is exactly what Omo did, like Nathan Lambert, AI2, After, just after he did the interview with us, they released Omo. So, it's unfortunate that we didn't get to talk much about it. But Omo actually started doing 1. 5 epochs on every, on all data.[00:18:00] swyx: And the data ablation paper that I covered in Europe's says that, you know, you don't like, don't really start to tap out of like, the alpha or the sort of improved loss that you get from data all the way until four epochs. And so I'm just like, okay, like, why do we all agree that one epoch is all you need?[00:18:17] swyx: It seems like to be a trend. It seems that we think that memorization is very good or too good. But then also we're finding that, you know, For improvement in results that we really like, we're fine on overtraining on things intentionally. So, I think that's an interesting direction that I don't see people exploring enough.[00:18:36] swyx: And the more I see papers coming out Stretching beyond the one epoch thing, the more people are like, it's completely fine. And actually, the only reason we stopped is because we ran out of compute[00:18:46] Alessio: budget. Yeah, I think that's the biggest thing, right?[00:18:51] swyx: Like, that's not a valid reason, that's not science. I[00:18:54] Alessio: wonder if, you know, Matt is going to do it.[00:18:57] Alessio: I heard LamaTree, they want to do a 100 billion parameters model. I don't think you can train that on too many epochs, even with their compute budget, but yeah. They're the only ones that can save us, because even if OpenAI is doing this, they're not going to tell us, you know. Same with DeepMind.[00:19:14] swyx: Yeah, and so the updates that we got on Lambda 3 so far is apparently that because of the Gemini news that we'll talk about later they're pushing it back on the release.[00:19:21] swyx: They already have it. And they're just pushing it back to do more safety testing. Politics testing.[00:19:28] Alessio: Well, our episode with Sumit will have already come out by the time this comes out, I think. So people will get the inside story on how they actually allocate the compute.[00:19:38] Direction 3: Alt. Architectures (Mamba, RWKV, RingAttention, Diffusion Transformers)[00:19:38] Alessio: Alternative architectures. Well, shout out to our WKV who won one of the prizes at our Final Frontiers event last week.[00:19:47] Alessio: We talked about Mamba and Strapain on the Together episode. A lot of, yeah, monarch mixers. I feel like Together, It's like the strong Stanford Hazy Research Partnership, because Chris Ray is one of the co founders. So they kind of have a, I feel like they're going to be the ones that have one of the state of the art models alongside maybe RWKB.[00:20:08] Alessio: I haven't seen as many independent. People working on this thing, like Monarch Mixer, yeah, Manbuster, Payena, all of these are together related. Nobody understands the math. They got all the gigabrains, they got 3DAO, they got all these folks in there, like, working on all of this.[00:20:25] swyx: Albert Gu, yeah. Yeah, so what should we comment about it?[00:20:28] swyx: I mean, I think it's useful, interesting, but at the same time, both of these are supposed to do really good scaling for long context. And then Gemini comes out and goes like, yeah, we don't need it. Yeah.[00:20:44] Alessio: No, that's the risk. So, yeah. I was gonna say, maybe it's not here, but I don't know if we want to talk about diffusion transformers as like in the alt architectures, just because of Zora.[00:20:55] swyx: One thing, yeah, so, so, you know, this came from the Jan recap, which, and diffusion transformers were not really a discussion, and then, obviously, they blow up in February. Yeah. I don't think they're, it's a mixed architecture in the same way that Stripe Tiena is mixed there's just different layers taking different approaches.[00:21:13] swyx: Also I think another one that I maybe didn't call out here, I think because it happened in February, was hourglass diffusion from stability. But also, you know, another form of mixed architecture. So I guess that is interesting. I don't have much commentary on that, I just think, like, we will try to evolve these things, and maybe one of these architectures will stick and scale, it seems like diffusion transformers is going to be good for anything generative, you know, multi modal.[00:21:41] swyx: We don't see anything where diffusion is applied to text yet, and that's the wild card for this category. Yeah, I mean, I think I still hold out hope for let's just call it sub quadratic LLMs. I think that a lot of discussion this month actually was also centered around this concept that People always say, oh, like, transformers don't scale because attention is quadratic in the sequence length.[00:22:04] swyx: Yeah, but, you know, attention actually is a very small part of the actual compute that is being spent, especially in inference. And this is the reason why, you know, when you multiply, when you, when you, when you jump up in terms of the, the model size in GPT 4 from like, you know, 38k to like 32k, you don't also get like a 16 times increase in your, in your performance.[00:22:23] swyx: And this is also why you don't get like a million times increase in your, in your latency when you throw a million tokens into Gemini. Like people have figured out tricks around it or it's just not that significant as a term, as a part of the overall compute. So there's a lot of challenges to this thing working.[00:22:43] swyx: It's really interesting how like, how hyped people are about this versus I don't know if it works. You know, it's exactly gonna, gonna work. And then there's also this, this idea of retention over long context. Like, even though you have context utilization, like, the amount of, the amount you can remember is interesting.[00:23:02] swyx: Because I've had people criticize both Mamba and RWKV because they're kind of, like, RNN ish in the sense that they have, like, a hidden memory and sort of limited hidden memory that they will forget things. So, for all these reasons, Gemini 1. 5, which we still haven't covered, is very interesting because Gemini magically has fixed all these problems with perfect haystack recall and reasonable latency and cost.[00:23:29] Wildcards: Text Diffusion, RALM/Retro[00:23:29] swyx: So that's super interesting. So the wildcard I put in here if you want to go to that. I put two actually. One is text diffusion. I think I'm still very influenced by my meeting with a mid journey person who said they were working on text diffusion. I think it would be a very, very different paradigm for, for text generation, reasoning, plan generation if we can get diffusion to work.[00:23:51] swyx: For text. And then the second one is Dowie Aquila's contextual AI, which is working on retrieval augmented language models, where it kind of puts RAG inside of the language model instead of outside.[00:24:02] Alessio: Yeah, there's a paper called Retro that covers some of this. I think that's an interesting thing. I think the The challenge, well not the challenge, what they need to figure out is like how do you keep the rag piece always up to date constantly, you know, I feel like the models, you put all this work into pre training them, but then at least you have a fixed artifact.[00:24:22] Alessio: These architectures are like constant work needs to be done on them and they can drift even just based on the rag data instead of the model itself. Yeah,[00:24:30] swyx: I was in a panel with one of the investors in contextual and the guy, the way that guy pitched it, I didn't agree with. He was like, this will solve hallucination.[00:24:38] Alessio: That's what everybody says. We solve[00:24:40] swyx: hallucination. I'm like, no, you reduce it. It cannot,[00:24:44] Alessio: if you solved it, the model wouldn't exist, right? It would just be plain text. It wouldn't be a generative model. Cool. So, author, architectures, then we got mixture of experts. I think we covered a lot of, a lot of times.[00:24:56] Direction 4: Mixture of Experts (DeepSeekMoE, Samba-1)[00:24:56] Alessio: Maybe any new interesting threads you want to go under here?[00:25:00] swyx: DeepSeq MOE, which was released in January. Everyone who is interested in MOEs should read that paper, because it's significant for two reasons. One three reasons. One, it had, it had small experts, like a lot more small experts. So, for some reason, everyone has settled on eight experts for GPT 4 for Mixtral, you know, that seems to be the favorite architecture, but these guys pushed it to 64 experts, and each of them smaller than the other.[00:25:26] swyx: But then they also had the second idea, which is that it is They had two, one to two always on experts for common knowledge and that's like a very compelling concept that you would not route to all the experts all the time and make them, you know, switch to everything. You would have some always on experts.[00:25:41] swyx: I think that's interesting on both the inference side and the training side for for memory retention. And yeah, they, they, they, the, the, the, the results that they published, which actually excluded, Mixed draw, which is interesting. The results that they published showed a significant performance jump versus all the other sort of open source models at the same parameter count.[00:26:01] swyx: So like this may be a better way to do MOEs that are, that is about to get picked up. And so that, that is interesting for the third reason, which is this is the first time a new idea from China. has infiltrated the West. It's usually the other way around. I probably overspoke there. There's probably lots more ideas that I'm not aware of.[00:26:18] swyx: Maybe in the embedding space. But the I think DCM we, like, woke people up and said, like, hey, DeepSeek, this, like, weird lab that is attached to a Chinese hedge fund is somehow, you know, doing groundbreaking research on MOEs. So, so, I classified this as a medium potential because I think that it is a sort of like a one off benefit.[00:26:37] swyx: You can Add to any, any base model to like make the MOE version of it, you get a bump and then that's it. So, yeah,[00:26:45] Alessio: I saw Samba Nova, which is like another inference company. They released this MOE model called Samba 1, which is like a 1 trillion parameters. But they're actually MOE auto open source models.[00:26:56] Alessio: So it's like, they just, they just clustered them all together. So I think people. Sometimes I think MOE is like you just train a bunch of small models or like smaller models and put them together. But there's also people just taking, you know, Mistral plus Clip plus, you know, Deepcoder and like put them all together.[00:27:15] Alessio: And then you have a MOE model. I don't know. I haven't tried the model, so I don't know how good it is. But it seems interesting that you can then have people working separately on state of the art, you know, Clip, state of the art text generation. And then you have a MOE architecture that brings them all together.[00:27:31] swyx: I'm thrown off by your addition of the word clip in there. Is that what? Yeah, that's[00:27:35] Alessio: what they said. Yeah, yeah. Okay. That's what they I just saw it yesterday. I was also like[00:27:40] swyx: scratching my head. And they did not use the word adapter. No. Because usually what people mean when they say, Oh, I add clip to a language model is adapter.[00:27:48] swyx: Let me look up the Which is what Lava did.[00:27:50] Alessio: The announcement again.[00:27:51] swyx: Stable diffusion. That's what they do. Yeah, it[00:27:54] Alessio: says among the models that are part of Samba 1 are Lama2, Mistral, DeepSigCoder, Falcon, Dplot, Clip, Lava. So they're just taking all these models and putting them in a MOE. Okay,[00:28:05] swyx: so a routing layer and then not jointly trained as much as a normal MOE would be.[00:28:12] swyx: Which is okay.[00:28:13] Alessio: That's all they say. There's no paper, you know, so it's like, I'm just reading the article, but I'm interested to see how[00:28:20] Wildcard: Model Merging (mergekit)[00:28:20] swyx: it works. Yeah, so so the wildcard for this section, the MOE section is model merges, which has also come up as, as a very interesting phenomenon. The last time I talked to Jeremy Howard at the Olama meetup we called it model grafting or model stacking.[00:28:35] swyx: But I think the, the, the term that people are liking these days, the model merging, They're all, there's all different variations of merging. Merge types, and some of them are stacking, some of them are, are grafting. And, and so like, some people are approaching model merging in the way that Samba is doing, which is like, okay, here are defined models, each of which have their specific, Plus and minuses, and we will merge them together in the hope that the, you know, the sum of the parts will, will be better than others.[00:28:58] swyx: And it seems like it seems like it's working. I don't really understand why it works apart from, like, I think it's a form of regularization. That if you merge weights together in like a smart strategy you, you, you get a, you get a, you get a less overfitting and more generalization, which is good for benchmarks, if you, if you're honest about your benchmarks.[00:29:16] swyx: So this is really interesting and good. But again, they're kind of limited in terms of like the amount of bumps you can get. But I think it's very interesting in the sense of how cheap it is. We talked about this on the Chinatalk podcast, like the guest podcast that we did with Chinatalk. And you can do this without GPUs, because it's just adding weights together, and dividing things, and doing like simple math, which is really interesting for the GPU ports.[00:29:42] Alessio: There's a lot of them.[00:29:44] Direction 5: Online LLMs (Gemini Pro, Exa)[00:29:44] Alessio: And just to wrap these up, online LLMs? Yeah,[00:29:48] swyx: I think that I ki I had to feature this because the, one of the top news of January was that Gemini Pro beat GPT-4 turbo on LM sis for the number two slot to GPT-4. And everyone was very surprised. Like, how does Gemini do that?[00:30:06] swyx: Surprise, surprise, they added Google search. Mm-hmm to the results. So it became an online quote unquote online LLM and not an offline LLM. Therefore, it's much better at answering recent questions, which people like. There's an emerging set of table stakes features after you pre train something.[00:30:21] swyx: So after you pre train something, you should have the chat tuned version of it, or the instruct tuned version of it, however you choose to call it. You should have the JSON and function calling version of it. Structured output, the term that you don't like. You should have the online version of it. These are all like table stakes variants, that you should do when you offer a base LLM, or you train a base LLM.[00:30:44] swyx: And I think online is just like, There, it's important. I think companies like Perplexity, and even Exa, formerly Metaphor, you know, are rising to offer that search needs. And it's kind of like, they're just necessary parts of a system. When you have RAG for internal knowledge, and then you have, you know, Online search for external knowledge, like things that you don't know yet?[00:31:06] swyx: Mm-Hmm. . And it seems like it's, it's one of many tools. I feel like I may be underestimating this, but I'm just gonna put it out there that I, I think it has some, some potential. One of the evidence points that it doesn't actually matter that much is that Perplexity has a, has had online LMS for three months now and it performs, doesn't perform great.[00:31:25] swyx: Mm-Hmm. on, on lms, it's like number 30 or something. So it's like, okay. You know, like. It's, it's, it helps, but it doesn't give you a giant, giant boost. I[00:31:34] Alessio: feel like a lot of stuff I do with LLMs doesn't need to be online. So I'm always wondering, again, going back to like state of the art, right? It's like state of the art for who and for what.[00:31:45] Alessio: It's really, I think online LLMs are going to be, State of the art for, you know, news related activity that you need to do. Like, you're like, you know, social media, right? It's like, you want to have all the latest stuff, but coding, science,[00:32:01] swyx: Yeah, but I think. Sometimes you don't know what is news, what is news affecting.[00:32:07] swyx: Like, the decision to use an offline LLM is already a decision that you might not be consciously making that might affect your results. Like, what if, like, just putting things on, being connected online means that you get to invalidate your knowledge. And when you're just using offline LLM, like it's never invalidated.[00:32:27] swyx: I[00:32:28] Alessio: agree, but I think going back to your point of like the standing the test of time, I think sometimes you can get swayed by the online stuff, which is like, hey, you ask a question about, yeah, maybe AI research direction, you know, and it's like, all the recent news are about this thing. So the LLM like focus on answering, bring it up, you know, these things.[00:32:50] swyx: Yeah, so yeah, I think, I think it's interesting, but I don't know if I can, I bet heavily on this.[00:32:56] Alessio: Cool. Was there one that you forgot to put, or, or like a, a new direction? Yeah,[00:33:01] swyx: so, so this brings us into sort of February. ish.[00:33:05] OpenAI Sora and why everyone underestimated videogen[00:33:05] swyx: So like I published this in like 15 came with Sora. And so like the one thing I did not mention here was anything about multimodality.[00:33:16] swyx: Right. And I have chronically underweighted this. I always wrestle. And, and my cop out is that I focused this piece or this research direction piece on LLMs because LLMs are the source of like AGI, quote unquote AGI. Everything else is kind of like. You know, related to that, like, generative, like, just because I can generate better images or generate better videos, it feels like it's not on the critical path to AGI, which is something that Nat Friedman also observed, like, the day before Sora, which is kind of interesting.[00:33:49] swyx: And so I was just kind of like trying to focus on like what is going to get us like superhuman reasoning that we can rely on to build agents that automate our lives and blah, blah, blah, you know, give us this utopian future. But I do think that I, everybody underestimated the, the sheer importance and cultural human impact of Sora.[00:34:10] swyx: And you know, really actually good text to video. Yeah. Yeah.[00:34:14] Alessio: And I saw Jim Fan at a, at a very good tweet about why it's so impressive. And I think when you have somebody leading the embodied research at NVIDIA and he said that something is impressive, you should probably listen. So yeah, there's basically like, I think you, you mentioned like impacting the world, you know, that we live in.[00:34:33] Alessio: I think that's kind of like the key, right? It's like the LLMs don't have, a world model and Jan Lekon. He can come on the podcast and talk all about what he thinks of that. But I think SORA was like the first time where people like, Oh, okay, you're not statically putting pixels of water on the screen, which you can kind of like, you know, project without understanding the physics of it.[00:34:57] Alessio: Now you're like, you have to understand how the water splashes when you have things. And even if you just learned it by watching video and not by actually studying the physics, You still know it, you know, so I, I think that's like a direction that yeah, before you didn't have, but now you can do things that you couldn't before, both in terms of generating, I think it always starts with generating, right?[00:35:19] Alessio: But like the interesting part is like understanding it. You know, it's like if you gave it, you know, there's the video of like the, the ship in the water that they generated with SORA, like if you gave it the video back and now it could tell you why the ship is like too rocky or like it could tell you why the ship is sinking, then that's like, you know, AGI for like all your rig deployments and like all this stuff, you know, so, but there's none, there's none of that yet, so.[00:35:44] Alessio: Hopefully they announce it and talk more about it. Maybe a Dev Day this year, who knows.[00:35:49] swyx: Yeah who knows, who knows. I'm talking with them about Dev Day as well. So I would say, like, the phrasing that Jim used, which resonated with me, he kind of called it a data driven world model. I somewhat agree with that.[00:36:04] Does Sora have a World Model? Yann LeCun vs Jim Fan[00:36:04] swyx: I am on more of a Yann LeCun side than I am on Jim's side, in the sense that I think that is the vision or the hope that these things can build world models. But you know, clearly even at the current SORA size, they don't have the idea of, you know, They don't have strong consistency yet. They have very good consistency, but fingers and arms and legs will appear and disappear and chairs will appear and disappear.[00:36:31] swyx: That definitely breaks physics. And it also makes me think about how we do deep learning versus world models in the sense of You know, in classic machine learning, when you have too many parameters, you will overfit, and actually that fails, that like, does not match reality, and therefore fails to generalize well.[00:36:50] swyx: And like, what scale of data do we need in order to world, learn world models from video? A lot. Yeah. So, so I, I And cautious about taking this interpretation too literally, obviously, you know, like, I get what he's going for, and he's like, obviously partially right, obviously, like, transformers and, and, you know, these, like, these sort of these, these neural networks are universal function approximators, theoretically could figure out world models, it's just like, how good are they, and how tolerant are we of hallucinations, we're not very tolerant, like, yeah, so It's, it's, it's gonna prior, it's gonna bias us for creating like very convincing things, but then not create like the, the, the useful role models that we want.[00:37:37] swyx: At the same time, what you just said, I think made me reflect a little bit like we just got done saying how important synthetic data is for Mm-Hmm. for training lms. And so like, if this is a way of, of synthetic, you know, vi video data for improving our video understanding. Then sure, by all means. Which we actually know, like, GPT 4, Vision, and Dolly were trained, kind of, co trained together.[00:38:02] swyx: And so, like, maybe this is on the critical path, and I just don't fully see the full picture yet.[00:38:08] Alessio: Yeah, I don't know. I think there's a lot of interesting stuff. It's like, imagine you go back, you have Sora, you go back in time, and Newton didn't figure out gravity yet. Would Sora help you figure it out?[00:38:21] Alessio: Because you start saying, okay, a man standing under a tree with, like, Apples falling, and it's like, oh, they're always falling at the same speed in the video. Why is that? I feel like sometimes these engines can like pick up things, like humans have a lot of intuition, but if you ask the average person, like the physics of like a fluid in a boat, they couldn't be able to tell you the physics, but they can like observe it, but humans can only observe this much, you know, versus like now you have these models to observe everything and then They generalize these things and maybe we can learn new things through the generalization that they pick up.[00:38:55] swyx: But again, And it might be more observant than us in some respects. In some ways we can scale it up a lot more than the number of physicists that we have available at Newton's time. So like, yeah, absolutely possible. That, that this can discover new science. I think we have a lot of work to do to formalize the science.[00:39:11] swyx: And then, I, I think the last part is you know, How much, how much do we cheat by gen, by generating data from Unreal Engine 5? Mm hmm. which is what a lot of people are speculating with very, very limited evidence that OpenAI did that. The strongest evidence that I saw was someone who works a lot with Unreal Engine 5 looking at the side characters in the videos and noticing that they all adopt Unreal Engine defaults.[00:39:37] swyx: of like, walking speed, and like, character choice, like, character creation choice. And I was like, okay, like, that's actually pretty convincing that they actually use Unreal Engine to bootstrap some synthetic data for this training set. Yeah,[00:39:52] Alessio: could very well be.[00:39:54] swyx: Because then you get the labels and the training side by side.[00:39:58] swyx: One thing that came up on the last day of February, which I should also mention, is EMO coming out of Alibaba, which is also a sort of like video generation and space time transformer that also involves probably a lot of synthetic data as well. And so like, this is of a kind in the sense of like, oh, like, you know, really good generative video is here and It is not just like the one, two second clips that we saw from like other, other people and like, you know, Pika and all the other Runway are, are, are, you know, run Cristobal Valenzuela from Runway was like game on which like, okay, but like, let's see your response because we've heard a lot about Gen 1 and 2, but like, it's nothing on this level of Sora So it remains to be seen how we can actually apply this, but I do think that the creative industry should start preparing.[00:40:50] swyx: I think the Sora technical blog post from OpenAI was really good.. It was like a request for startups. It was so good in like spelling out. Here are the individual industries that this can impact.[00:41:00] swyx: And anyone who, anyone who's like interested in generative video should look at that. But also be mindful that probably when OpenAI releases a Soa API, right? The you, the in these ways you can interact with it are very limited. Just like the ways you can interact with Dahlia very limited and someone is gonna have to make open SOA to[00:41:19] swyx: Mm-Hmm to, to, for you to create comfy UI pipelines.[00:41:24] Alessio: The stability folks said they wanna build an open. For a competitor, but yeah, stability. Their demo video, their demo video was like so underwhelming. It was just like two people sitting on the beach[00:41:34] swyx: standing. Well, they don't have it yet, right? Yeah, yeah.[00:41:36] swyx: I mean, they just wanna train it. Everybody wants to, right? Yeah. I, I think what is confusing a lot of people about stability is like they're, they're, they're pushing a lot of things in stable codes, stable l and stable video diffusion. But like, how much money do they have left? How many people do they have left?[00:41:51] swyx: Yeah. I have had like a really, Ima Imad spent two hours with me. Reassuring me things are great. And, and I'm like, I, I do, like, I do believe that they have really, really quality people. But it's just like, I, I also have a lot of very smart people on the other side telling me, like, Hey man, like, you know, don't don't put too much faith in this, in this thing.[00:42:11] swyx: So I don't know who to believe. Yeah.[00:42:14] Alessio: It's hard. Let's see. What else? We got a lot more stuff. I don't know if we can. Yeah, Groq.[00:42:19] Groq Math[00:42:19] Alessio: We can[00:42:19] swyx: do a bit of Groq prep. We're, we're about to go to talk to Dylan Patel. Maybe, maybe it's the audio in here. I don't know. It depends what, what we get up to later. What, how, what do you as an investor think about Groq? Yeah. Yeah, well, actually, can you recap, like, why is Groq interesting? So,[00:42:33] Alessio: Jonathan Ross, who's the founder of Groq, he's the person that created the TPU at Google. It's actually, it was one of his, like, 20 percent projects. It's like, he was just on the side, dooby doo, created the TPU.[00:42:46] Alessio: But yeah, basically, Groq, they had this demo that went viral, where they were running Mistral at, like, 500 tokens a second, which is like, Fastest at anything that you have out there. The question, you know, it's all like, The memes were like, is NVIDIA dead? Like, people don't need H100s anymore. I think there's a lot of money that goes into building what GRUK has built as far as the hardware goes.[00:43:11] Alessio: We're gonna, we're gonna put some of the notes from, from Dylan in here, but Basically the cost of the Groq system is like 30 times the cost of, of H100 equivalent. So, so[00:43:23] swyx: let me, I put some numbers because me and Dylan were like, I think the two people actually tried to do Groq math. Spreadsheet doors.[00:43:30] swyx: Spreadsheet doors. So, one that's, okay, oh boy so, so, equivalent H100 for Lama 2 is 300, 000. For a system of 8 cards. And for Groq it's 2. 3 million. Because you have to buy 576 Groq cards. So yeah, that, that just gives people an idea. So like if you deprecate both over a five year lifespan, per year you're deprecating 460K for Groq, and 60K a year for H100.[00:43:59] swyx: So like, Groqs are just way more expensive per model that you're, that you're hosting. But then, you make it up in terms of volume. So I don't know if you want to[00:44:08] Alessio: cover that. I think one of the promises of Groq is like super high parallel inference on the same thing. So you're basically saying, okay, I'm putting on this upfront investment on the hardware, but then I get much better scaling once I have it installed.[00:44:24] Alessio: I think the big question is how much can you sustain the parallelism? You know, like if you get, if you're going to get 100% Utilization rate at all times on Groq, like, it's just much better, you know, because like at the end of the day, the tokens per second costs that you're getting is better than with the H100s, but if you get to like 50 percent utilization rate, you will be much better off running on NVIDIA.[00:44:49] Alessio: And if you look at most companies out there, who really gets 100 percent utilization rate? Probably open AI at peak times, but that's probably it. But yeah, curious to see more. I saw Jonathan was just at the Web Summit in Dubai, in Qatar. He just gave a talk there yesterday. That I haven't listened to yet.[00:45:09] Alessio: I, I tweeted that he should come on the pod. He liked it. And then rock followed me on Twitter. I don't know if that means that they're interested, but[00:45:16] swyx: hopefully rock social media person is just very friendly. They, yeah. Hopefully[00:45:20] Alessio: we can get them. Yeah, we, we gonna get him. We[00:45:22] swyx: just call him out and, and so basically the, the key question is like, how sustainable is this and how much.[00:45:27] swyx: This is a loss leader the entire Groq management team has been on Twitter and Hacker News saying they are very, very comfortable with the pricing of 0. 27 per million tokens. This is the lowest that anyone has offered tokens as far as Mixtral or Lama2. This matches deep infra and, you know, I think, I think that's, that's, that's about it in terms of that, that, that low.[00:45:47] swyx: And we think the pro the break even for H100s is 50 cents. At a, at a normal utilization rate. To make this work, so in my spreadsheet I made this, made this work. You have to have like a parallelism of 500 requests all simultaneously. And you have, you have model bandwidth utilization of 80%.[00:46:06] swyx: Which is way high. I just gave them high marks for everything. Groq has two fundamental tech innovations that they hinge their hats on in terms of like, why we are better than everyone. You know, even though, like, it remains to be independently replicated. But one you know, they have this sort of the entire model on the chip idea, which is like, Okay, get rid of HBM.[00:46:30] swyx: And, like, put everything in SREM. Like, okay, fine, but then you need a lot of cards and whatever. And that's all okay. And so, like, because you don't have to transfer between memory, then you just save on that time and that's why they're faster. So, a lot of people buy that as, like, that's the reason that you're faster.[00:46:45] swyx: Then they have, like, some kind of crazy compiler, or, like, Speculative routing magic using compilers that they also attribute towards their higher utilization. So I give them 80 percent for that. And so that all that works out to like, okay, base costs, I think you can get down to like, maybe like 20 something cents per million tokens.[00:47:04] swyx: And therefore you actually are fine if you have that kind of utilization. But it's like, I have to make a lot of fearful assumptions for this to work.[00:47:12] Alessio: Yeah. Yeah, I'm curious to see what Dylan says later.[00:47:16] swyx: So he was like completely opposite of me. He's like, they're just burning money. Which is great.[00:47:22] Analyzing Gemini's 1m Context, Reddit deal, Imagegen politics, Gemma via the Four Wars[00:47:22] Alessio: Gemini, want to do a quick run through since this touches on all the four words.[00:47:28] swyx: Yeah, and I think this is the mark of a useful framework, that when a new thing comes along, you can break it down in terms of the four words and sort of slot it in or analyze it in those four frameworks, and have nothing left.[00:47:41] swyx: So it's a MECE categorization. MECE is Mutually Exclusive and Collectively Exhaustive. And that's a really, really nice way to think about taxonomies and to create mental frameworks. So, what is Gemini 1. 5 Pro? It is the newest model that came out one week after Gemini 1. 0. Which is very interesting.[00:48:01] swyx: They have not really commented on why. They released this the headline feature is that it has a 1 million token context window that is multi modal which means that you can put all sorts of video and audio And PDFs natively in there alongside of text and, you know, it's, it's at least 10 times longer than anything that OpenAI offers which is interesting.[00:48:20] swyx: So it's great for prototyping and it has interesting discussions on whether it kills RAG.[00:48:25] Alessio: Yeah, no, I mean, we always talk about, you know, Long context is good, but you're getting charged per token. So, yeah, people love for you to use more tokens in the context. And RAG is better economics. But I think it all comes down to like how the price curves change, right?[00:48:42] Alessio: I think if anything, RAG's complexity goes up and up the more you use it, you know, because you have more data sources, more things you want to put in there. The token costs should go down over time, you know, if the model stays fixed. If people are happy with the model today. In two years, three years, it's just gonna cost a lot less, you know?[00:49:02] Alessio: So now it's like, why would I use RAG and like go through all of that? It's interesting. I think RAG is better cutting edge economics for LLMs. I think large context will be better long tail economics when you factor in the build cost of like managing a RAG pipeline. But yeah, the recall was like the most interesting thing because we've seen the, you know, You know, in the haystack things in the past, but apparently they have 100 percent recall on anything across the context window.[00:49:28] Alessio: At least they say nobody has used it. No, people[00:49:30] swyx: have. Yeah so as far as, so, so what this needle in a haystack thing for people who aren't following as closely as us is that someone, I forget his name now someone created this needle in a haystack problem where you feed in a whole bunch of generated junk not junk, but just like, Generate a data and ask it to specifically retrieve something in that data, like one line in like a hundred thousand lines where it like has a specific fact and if it, if you get it, you're, you're good.[00:49:57] swyx: And then he moves the needle around, like, you know, does it, does, does your ability to retrieve that vary if I put it at the start versus put it in the middle, put it at the end? And then you generate this like really nice chart. That, that kind of shows like it's recallability of a model. And he did that for GPT and, and Anthropic and showed that Anthropic did really, really poorly.[00:50:15] swyx: And then Anthropic came back and said it was a skill issue, just add this like four, four magic words, and then, then it's magically all fixed. And obviously everybody laughed at that. But what Gemini came out with was, was that, yeah, we, we reproduced their, you know, haystack issue you know, test for Gemini, and it's good across all, all languages.[00:50:30] swyx: All the one million token window, which is very interesting because usually for typical context extension methods like rope or yarn or, you know, anything like that, or alibi, it's lossy like by design it's lossy, usually for conversations that's fine because we are lossy when we talk to people but for superhuman intelligence, perfect memory across Very, very long context.[00:50:51] swyx: It's very, very interesting for picking things up. And so the people who have been given the beta test for Gemini have been testing this. So what you do is you upload, let's say, all of Harry Potter and you change one fact in one sentence, somewhere in there, and you ask it to pick it up, and it does. So this is legit.[00:51:08] swyx: We don't super know how, because this is, like, because it doesn't, yes, it's slow to inference, but it's not slow enough that it's, like, running. Five different systems in the background without telling you. Right. So it's something, it's something interesting that they haven't fully disclosed yet. The open source community has centered on this ring attention paper, which is created by your friend Matei Zaharia, and a couple other people.[00:51:36] swyx: And it's a form of distributing the compute. I don't super understand, like, why, you know, doing, calculating, like, the fee for networking and attention. In block wise fashion and distributing it makes it so good at recall. I don't think they have any answer to that. The only thing that Ring of Tension is really focused on is basically infinite context.[00:51:59] swyx: They said it was good for like 10 to 100 million tokens. Which is, it's just great. So yeah, using the four wars framework, what is this framework for Gemini? One is the sort of RAG and Ops war. Here we care less about RAG now, yes. Or, we still care as much about RAG, but like, now it's it's not important in prototyping.[00:52:21] swyx: And then, for data war I guess this is just part of the overall training dataset, but Google made a 60 million deal with Reddit and presumably they have deals with other companies. For the multi modality war, we can talk about the image generation, Crisis, or the fact that Gemini also has image generation, which we'll talk about in the next section.[00:52:42] swyx: But it also has video understanding, which is, I think, the top Gemini post came from our friend Simon Willison, who basically did a short video of him scanning over his bookshelf. And it would be able to convert that video into a JSON output of what's on that bookshelf. And I think that is very useful.[00:53:04] swyx: Actually ties into the conversation that we had with David Luan from Adept. In a sense of like, okay what if video was the main modality instead of text as the input? What if, what if everything was video in, because that's how we work. We, our eyes don't actually read, don't actually like get input, our brains don't get inputs as characters.[00:53:25] swyx: Our brains get the pixels shooting into our eyes, and then our vision system takes over first, and then we sort of mentally translate that into text later. And so it's kind of like what Adept is kind of doing, which is driving by vision model, instead of driving by raw text understanding of the DOM. And, and I, I, in that, that episode, which we haven't released I made the analogy to like self-driving by lidar versus self-driving by camera.[00:53:52] swyx: Mm-Hmm. , right? Like, it's like, I think it, what Gemini and any other super long context that model that is multimodal unlocks is what if you just drive everything by video. Which is[00:54:03] Alessio: cool. Yeah, and that's Joseph from Roboflow. It's like anything that can be seen can be programmable with these models.[00:54:12] Alessio: You mean[00:54:12] swyx: the computer vision guy is bullish on computer vision?[00:54:18] Alessio: It's like the rag people. The rag people are bullish on rag and not a lot of context. I'm very surprised. The, the fine tuning people love fine tuning instead of few shot. Yeah. Yeah. The, yeah, the, that's that. Yeah, the, I, I think the ring attention thing, and it's how they did it, we don't know. And then they released the Gemma models, which are like a 2 billion and 7 billion open.[00:54:41] Alessio: Models, which people said are not, are not good based on my Twitter experience, which are the, the GPU poor crumbs. It's like, Hey, we did all this work for us because we're GPU rich and we're just going to run this whole thing. And

In this podcast episode, Krish explores Teradata from scratch. He starts by introducing Teradata as a complete cloud analytics and data platform, suitable for building large-scale data warehousing applications. He explains the concepts of data warehousing, data lakes, and data marts. Krish then explores Teradata's platform and products, including Teradata Vantage and ClearScape Analytics. He demonstrates how to get started with Teradata by creating an environment and exploring the JupyterLab interface. Krish creates tables, loads data, and runs queries in Teradata, providing hands-on experience and learning along the way. Krish explores the Teradata platform and its functionalities. He starts by troubleshooting a query and identifying the issue. Then, he runs basic queries to demonstrate the SQL syntax. Krish also discusses the availability of third-party plugins and explores some of them. Finally, he concludes the episode by discussing the next steps for further exploration and learning. Takeaways Teradata is a complete cloud analytics and data platform suitable for building large-scale data warehousing applications. Data warehousing, data lakes, and data marts are important concepts to understand in the context of Teradata. Teradata offers a range of products and platforms, including Teradata Vantage and ClearScape Analytics. JupyterLab and Jupyter Notebooks can be used to interact with Teradata and perform data analysis and exploration. Creating tables, loading data, and running queries are essential tasks in Teradata. Teradata is a powerful platform for data analysis and management. Troubleshooting queries is an essential skill for working with Teradata. Basic SQL syntax can be used to run queries on Teradata. Third-party plugins can enhance the functionality of Teradata. Chapters 00:00 Introduction to Teradata 01:16 Understanding Data Warehousing and Data Lakes 03:35 Data Marts and Teradata 04:26 Exploring Teradata's Platform and Products05:41Getting Started with Teradata 06:25 Teradata Vantage and ClearScape Analytics 07:57 Understanding JupyterLab and Jupyter Notebooks 19:14 Exploring JupyterLab Extensions 28:18 Creating Tables and Loading Data in Teradata 48:02 Running Queries in Teradata 53:49 Troubleshooting Query 55:14 Running Basic Queries 56:00 Third-Party Plugins 57:14 Exploring Plugins 58:18 Next Steps and Further Exploration 58:45 Conclusion Snowpal Products Backends as Services on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠AWS Marketplace⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Mobile Apps on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠App Store⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ and ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Play Store⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Web App⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Education Platform⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ for Learners and Course Creators

Theodore Gray is a co-founder of Wolfram Research and creator of the Notebook user interface (since lovingly copied by Jupyter Notebooks). He is the author of several books and iPad apps including the NYTs best-seller The Elements, and writer/director of Disney Animated (BAFTA award winner and Apple's iPad App of the Year 2013). His latest book is TOOLS, which is all about his favorite tools.   Website: https://theodoregray.com   Watch on YouTube: https://youtu.be/k2eZ4W7VgYs   For show notes and transcript visit: https://kk.org/cooltools/theodore-gray-co-founder-of-wolfram-research-3/   To sign up to be a guest on the show, please fill out this form: https://forms.gle/qc496XB6bGbrAEKK7  

Jupyter Notebooks and Jupyter Lab have to be one of the most important parts of Python when it comes to bring new users to the Python ecosystem and certainly for the day to day work of data scientists and general scientists who have made some of the biggest discoveries of recent times. And that platform has recently gotten a major upgrade with JupyterLab 4 released and Jupyter Notebook being significantly reworked to be based on the changes from JupyterLab as well. We have an excellent panel of guests, Sylvain Corlay, Frederic Collonval, Jeremy Tuloup, and Afshin Darian here to tell us what's new in these and other parts of the Jupyter ecosystem. Links from the show Guests Sylvain Corlay Frederic Collonval Jeremy Tuloup Afshin Darian JupyterLab 4.0 is Here: blog.jupyter.org Announcing Jupyter Notebook 7: blog.jupyter.org JupyterCon 2023 Videos: youtube.com Jupyterlite: github.com Download JupyterLab Desktop: github.com Mythical Man Month Book: wikipedia.org Blender in Jupyter: twitter.com Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm --- Stay in touch with us --- Subscribe to us on YouTube: youtube.com Follow Talk Python on Mastodon: talkpython Follow Michael on Mastodon: mkennedy Sponsors Phylum Python Tutor Talk Python Training

Last week in security news: Copilot and CodeWhisperer can in fact leak real secrets, an interesting teardown of a cloud cryptocurrency miner, the tool of the week, and more!Links: Copilot and CodeWhisperer can in fact leak real secrets.  An interesting teardown of a cloud cryptocurrency miner.  How to create an AMI hardening pipeline and automate updates to your ECS instance fleet  How to improve your security incident response processes with Jupyter notebooks Tool of the week: If you've gotta use a WAF, aws-firewall-factory is a good pit stop for you.

What if your family is your biggest support on your entrepreneurial journey in neurotech? Welcome, dear listeners, to another episode of "Neurocareers: Doing the Impossible!" Today, we're diving deep into the world of EEG technology and entrepreneurship with our guest, Jared Beckwith. Jared is not just an expert in EEG technology; he's also the Chairman & CEO of AiON EEG, Inc. Based in Wesley Chapel, Florida, AiON EEG, Inc. is on a mission to revolutionize EEG review and analysis software. What makes this entrepreneurial journey even more remarkable is that Jared's father, an engineer, and his brother, a computer scientist, joined him on this extraordinary venture. Together, as a family team, they are driving innovation in EEG technology. Their flagship product is designed to do something extraordinary: reduce noise and display long-term trends in brain data, all with the aim of simplifying seizure detection. This innovative approach has the potential to transform how we understand and manage neurological conditions. Jared himself is passionate about building the best EEG review and analysis software available. His commitment to improving brain data analysis aligns perfectly with our podcast's mission to explore the dynamic intersection of neuroscience, technology, and entrepreneurship. Join us in this episode as we learn about Jared's entrepreneurial journey, the challenges he's faced, and the groundbreaking work being done at AiON EEG, Inc. Get ready for a deep dive into the world of EEG technology and its potential to change lives. About the Podcast Guest: If you want to learn to read an EEG, download Jared's software that comes with 15 examples for free (only works on Windows) aioneeg.com If you want to build EEG, BCI, or Ai software, Jared suggests taking these steps to learn the fundamentals: Download “Anaconda,” which installs the Python programming language and comes with “Jupyter Notebook” to write code in: https://www.anaconda.com Write your first program: print(“Hello World!”). Then, follow the YouTube tutorials for the AI projects below. MNIST Dataset (classification of handwritten digits): https://www.kaggle.com/datasets/avnishnish/mnist-original Fashion MNIST (classification of different clothing items): https://github.com/zalandoresearch/fashion-mnist Cats vs Dogs: https://www.microsoft.com/en-us/download/details.aspx?id=54765 Labeled EEG seizure dataset: https://physionet.org/content/chbmit/1.0.0/ Get in touch with Jared on LinkedIn: https://www.linkedin.com/in/jaredbeckwith15/ About the Podcast Host: The Neurocareers podcast is brought to you by The Institute of Neuroapproaches (https://www.neuroapproaches.org/) and its founder, Milena Korostenskaja, Ph.D. (Dr. K), a neuroscience educator, research consultant, and career coach for students and recent graduates in neuroscience and neurotechnologies. As a professional coach with a background in the field, Dr. K understands the unique challenges and opportunities facing students in this field and can provide personalized coaching and support to help you succeed. Here's what you'll get with one-on-one coaching sessions from Dr. K: Identification and pursuit of career goals Guidance on job search strategies, resume and cover letter development, and interview preparation Access to a network of professionals in the field of neuroscience and neurotechnologies Ongoing support and guidance to help you stay on track and achieve your goals You can always schedule a free neurocareer consultation/coaching session with Dr. K at https://neuroapproaches.as.me/free-neurocareer-consultation Subscribe to our Nerocareers Newsletter to stay on top of all our cool neurocareers news at updates https://www.neuroapproaches.org/neurocareers-news

Want to help define the AI Engineer stack? Have opinions on the top tools, communities and builders? We're collaborating with friends at Amplify to launch the first State of AI Engineering survey! Please fill it out (and tell your friends)!If AI is so important, why is its software so bad?This was the motivating question for Chris Lattner as he reconnected with his product counterpart on Tensorflow, Tim Davis, and started working on a modular solution to the problem of sprawling, monolithic, fragmented platforms in AI development. They announced a $30m seed in 2022 and, following their successful double launch of Modular/Mojo

In deploying cloud-native sustainable foundation AI models, there are five key steps outlined by Huamin Chen, an R&D professional at Red Hat's Office of the CTO. The first two steps involve using containers and Kubernetes to manage workloads and deploy them across a distributed infrastructure. Chen suggests employing PyTorch for programming and Jupyter Notebooks for debugging and evaluation, with Docker community files proving effective for containerizing workloads.The third step focuses on measurement and highlights the use of Prometheus, an open-source tool for event monitoring and alerting. Prometheus enables developers to gather metrics and analyze the correlation between foundation models and runtime environments.Analytics, the fourth step, involves leveraging existing analytics while establishing guidelines and benchmarks to assess energy usage and performance metrics. Chen emphasizes the need to challenge assumptions regarding energy consumption and model performance.Finally, the fifth step entails taking action based on the insights gained from analytics. By optimizing energy profiles for foundation models, the goal is to achieve greater energy efficiency, benefitting the community, society, and the environment.Chen underscores the significance of this optimization for a more sustainable future.Learn more at thenewstack.ioPyTorch Takes AI/ML Back to Its Research, Open Source RootsPyTorch Lightning and the Future of Open Source AIJupyter Notebooks: The Web-Based Dev Tool You've Been SeekingKnow the Hidden Costs of DIY Prometheus

Watch on YouTube About the show Sponsored by us! Support our work through: Our courses at Talk Python Training Test & Code Podcast Patreon Supporters Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Plumbum: Shell Combinators and More Suggested by Henry Schreiner last week. (Also, thanks Michael for the awesome search tool on PythonBytes.fm that includes transcripts, so I can find stuff discussed and not just stuff listed in the show notes.) Plumbum is “ a small yet feature-rich library for shell script-like programs in Python. The motto of the library is “Never write shell scripts again”, and thus it attempts to mimic the shell syntax (shell combinators) where it makes sense, while keeping it all Pythonic and cross-platform.” Supports local commands piping redirection working directory changes in a with block. So cool. lots more fun features Michael #2: Our plan for Python 3.13 The big difference is that we have now finished the foundational work that we need: Low impact monitoring (PEP 669) is implemented. The bytecode compiler is a much better state. The interpreter generator is working. Experiments on the register machine are complete. We have a viable approach to create a low-overhead maintainable machine code generator, based on copy-and-patch. We plan three parallelizable pieces of work for 3.13: The tier 2 optimizer Enabling subinterpreters from Python code (PEP 554). Memory management Details on superblocks Brian #3: Some blogging myths Julia Evans myths (more info of each in the blog post): you need to be original you need to be an expert posts need to be 100% correct writing boring posts is bad you need to explain every concept page views matter more material is always better everyone should blog I'd add Write posts to help yourself remember something. Write posts to help future prospective employers know what topics you care about. You know when you find a post that is outdated and now wrong, and the code doesn't work, but the topic is interesting to you. Go ahead and try to write a better post with code that works. Michael #4: Jupyter AI A generative AI extension for JupyterLab An %%ai magic that turns the Jupyter notebook into a reproducible generative AI playground. This works anywhere the IPython kernel runs (JupyterLab, Jupyter Notebook, Google Colab, VSCode, etc.). A native chat UI in JupyterLab that enables you to work with generative AI as a conversational assistant. Support for a wide range of generative model providers and models (AI21, Anthropic, Cohere, Hugging Face, OpenAI, SageMaker, etc.). Official project from Jupyter Provides code insights Debug failing code Provides a general interface for interaction and experimentation with currently available LLMs Lets you collaborate with peers and an Al in JupyterLab Lets you ask questions about local files Video presentation: David Qiu - Jupyter AI — Bringing Generative AI to Jupyter | PyData Seattle 2023 Extras Brian: Textual has some fun releases recently Textualize youtube channel with 3 tutorials so far trogon to turn Click based command line apps into TUIs video example of it working with sqlite-utils. Python in VSCode June Release includes revamped test discovery and execution. You have to turn it on though, as the changes are experimental: "python.experiments.optInto": [ "pythonTestAdapter", ] I just turned it on, so I haven't formed an opinion yet. Michael: Michael's take on the MacBook Air 15” (black one) Joke: Phishing

JupyterCon 2023, the conference on all things Jupyter was held in Paris between 10-12 May 2023, followed by 2 days of hands-on "sprints". Jupyter is a very popular open source platform with tools such as Jupyter notebook/lab and driven by a very active community. There were a number of excellent talks from a range of different subjects. I had the pleasure to meet and talk to a number of people, see the interview list below.Order of Interviews: Leah Silen and Arliss Collins from Numfocus  02:04Franklin Koch (MyST) from Curvenote 04:59Nicolas Thiery (Paris-Saclay) 09:13Sarah Gibson (2i2c) 13:19Ana Ruvalcaba (Jupyter Executive Council) 18:57Fernando Perez (Jupyter Executive Council) 23:48Raniere de Silva (Gesis) 29:56Linkshttps://jupyter.org Jupyter projecthttps://jupyter.org/enhancement-proposals/79-notebook-v7/notebook-v7.html# Release notes for the new Jupyter Notebook v7https://jupyterlab.readthedocs.io/en/latest/getting_started/changelog.html#v4-0 Release notes for JupyterLab v4.0 (further incremental updates of v4 are available)https://www.youtube.com/@JupyterCon YouTube channel for JupyterCon 2023https://cfp.jupytercon.com/2023/schedule/ JupyterCon 2023 schedulehttps://www.outreachy.org Outreachy project https://numfocus.org Numfocus projecthttps://data.agu.org/notebooks-now/ Notebooks Now initiativehttps://myst-tools.org MyST tool for scientific and technical communicationUpcoming RSE conferences:https://rsecon23.society-rse.org UK RSE conference in Swansea 5-8 Sep 2023https://hidden-ref.org/festival-of-hidden-ref/ Hidden Ref in Bristol, UK, 21 Sep 2023https://un-derse23.sciencesconf.org Unconference of the German RSE society deRSE in Jena 26-28 Sephttps://us-rse.org/usrse23/ 1st face to face US RSE Conference in Chicago 16-18 Oct 2023Support the Show.Thank you for listening and your ongoing support. It means the world to us! Support the show on Patreon https://www.patreon.com/codeforthought Get in touch: Email mailto:code4thought@proton.me UK RSE Slack (ukrse.slack.com): @code4thought or @piddie US RSE Slack (usrse.slack.com): @Peter Schmidt Mastadon: https://fosstodon.org/@code4thought or @code4thought@fosstodon.org LinkedIn: https://www.linkedin.com/in/pweschmidt/ (personal Profile)LinkedIn: https://www.linkedin.com/company/codeforthought/ (Code for Thought Profile) This podcast is licensed under the Creative Commons Licence: https://creativecommons.org/licenses/by-sa/4.0/

In episode 75 of The Gradient Podcast, Daniel Bashir speaks to Riley Goodside. Riley is a Staff Prompt Engineer at Scale AI. Riley began posting GPT-3 prompt examples and screenshot demonstrations in 2022. He previously worked as a data scientist at OkCupid, Grindr, and CopyAI.Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at editor@thegradient.pubSubscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (01:37) Riley's journey to becoming the first Staff Prompt Enginer* (02:00) data science background in online dating industry* (02:15) Sabbatical + catching up on LLM progress* (04:00) AI Dungeon and first taste of GPT-3* (05:10) Developing on codex, ideas about integrating codex with Jupyter Notebooks, start of posting on Twitter* (08:30) “LLM ethnography”* (09:12) The history of prompt engineering: in-context learning, Reinforcement Learning from Human Feedback (RLHF)* (10:20) Models used to be harder to talk to* (10:45) The three eras* (10:45) 1 - Pre-trained LM era—simple next-word predictors* (12:54) 2 - Instruction tuning* (16:13) 3 - RLHF and overcoming instruction tuning's limitations* (19:24) Prompting as subtractive sculpting, prompting and AI safety* (21:17) Riley on RLHF and safety* (24:55) Riley's most interesting experiments and observations* (25:50) Mode collapse in RLHF models* (29:24) Prompting models with very long instructions* (33:13) Explorations with regular expressions, chain-of-thought prompting styles* (36:32) Theories of in-context learning and prompting, why certain prompts work well* (42:20) Riley's advice for writing better prompts* (49:02) Debates over prompt engineering as a career, relevance of prompt engineers* (58:55) OutroLinks:* Riley's Twitter and LinkedIn* Talk: LLM Prompt Engineering and RLHF: History and Techniques Get full access to The Gradient at thegradientpub.substack.com/subscribe

Mike, Seth, & Tommy dive into a potential new skill for all Power BI Pro's... Do we need to know the Python language as Data Analysts? With the introduction of the new create Power BI Reports in Jupyter Notebooks and Python becoming a more and more prominent language in data engineering, will this be a required skill of Power BI Pro's? Blog here: https://powerbi.microsoft.com/en-us/blog/create-power-bi-reports-in-jupyter-notebooks/Get in touch: Send in your questions or topics you want us to discuss by tweeting to @PowerBITips with the hashtag #empMailbag or submit on the PowerBI.tips Podcast Page. Visit PowerBI.tips: https://powerbi.tips/ Watch the episodes live every Tuesday and Thursday morning at 730am CST on YouTube: https://www.youtube.com/powerbitips Subscribe on Spotify: https://open.spotify.com/show/230fp78XmHHRXTiYICRLVv Subscribe on Apple: https://podcasts.apple.com/us/podcast/explicit-measures-podcast/id1568944083‎ Check Out Community Jam: https://jam.powerbi.tips Follow Mike: https://www.linkedin.com/in/michaelcarlo/ Follow Seth: https://www.linkedin.com/in/seth-bauer/ Follow Tommy: https://www.linkedin.com/in/tommypuglia/

AI has taken the world by storm. It's gone from near zero to amazing in just a few years. We have ChatGPT, we have Stable Diffusion. But what about Jupyter Notebooks and pandas? In this episode, we meet Justin Waugh, the creator of Sketch. Sketch adds the ability to have conversational AI interactions about your pandas data frames (code and data). It's pretty powerful and I know you'll enjoy the conversation. Links from the show Sketch: github.com Lambdapromp: github.com Python Bytes 320 - Coverage of Sketch: pythonbytes.fm ChatGPT: chat.openai.com Midjourney: midjourney.com Github Copilot: github.com GitHub Copilot Litigation site: githubcopilotlitigation.com Attention is All You Need paper: research.google.com Live Colab Demo: colab.research.google.com AI Panda from Midjourney: digitaloceanspaces.com Ray: pypi.org Apache Arrow: arrow.apache.org Python Web Apps that Fly with CDNs Course: talkpython.fm Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm --- Stay in touch with us --- Subscribe to us on YouTube: youtube.com Follow Talk Python on Mastodon: talkpython Follow Michael on Mastodon: mkennedy Sponsors Brilliant 2023 Talk Python Training

Chris and Elecia talk about ChatGPT, conferences, online compilers, and Ardupilot. Compiler Explorer: godbolt.org (and function pointer example) Jupyter Notebooks with colab: colab.research.google.com/ (and one of Elecia's origami pattern generator collabs) Sign up for the Embedded newsletter! Support us on Patreon. Conferences and happenings: Hackaday Prize Embedded Online Conference : late April, online Open Hardware Summit 2023: end of April in NYC, NY Teardown 2023 | Crowd Supply: late June in Portland, OR SEMICON West: July in San Francisco, CA  embedded world North America: October 2024, Austin, TX Transcript

We are excited to feature our first academic on the pod! I first came across Shreya when her tweetstorm of MLOps principles went viral:Shreya's holistic approach to production grade machine learning has taken her from Stanford to Facebook and Google Brain, being the first ML Engineer at Viaduct, and now a PhD in Databases (trust us, its relevant) at UC Berkeley with the new EPIC Data Lab. If you know Berkeley's history in turning cutting edge research into gamechanging startups, you should be as excited as we are!Recorded in-person at the beautiful StudioPod studios in San Francisco.Full transcript is below the fold.Edit from the future: Shreya obliged us with another round of LLMOps hot takes after the pod!Other Links* Shreya's About: https://www.shreya-shankar.com/about/* Berkeley Sky Computing Lab - Utility Computing for the Cloud* Berkeley Epic Data Lab - low-code and no-code interfaces for data work, powered by next-generation predictive programming techniques* Shreya's ML Principles * Grounded Theory* Lightning Round:* Favorite AI Product: Stability Dreamstudio* 1 Year Prediction: Data management platforms* Request for startup: Design system generator* Takeaway: It's not a fad!Timestamps* [00:00:27] Introducing Shreya (poorly)* [00:03:38] The 3 V's of ML development* [00:05:45] Bridging Development and Production* [00:08:40] Preventing Data Leakage* [00:10:31] Berkeley's Unique Research Lab Culture* [00:11:53] From Static to Dynamically Updated Data* [00:12:55] Models as views on Data* [00:15:03] Principle: Version everything you do* [00:16:30] Principle: Always validate your data* [00:18:33] Heuristics for Model Architecture Selection* [00:20:36] The LLMOps Stack* [00:22:50] Shadow Models* [00:23:53] Keeping Up With Research* [00:26:10] Grounded Theory Research* [00:27:59] Google Brain vs Academia* [00:31:41] Advice for New Grads* [00:32:59] Helping Minorities in CS* [00:35:06] Lightning RoundTranscript[00:00:00] Hey everyone. Welcome to the Latent Space podcast. This is Alessio partner and CTM residence at Decibel Partners. I'm joined by my co-host, swyx writer and editor of Latent Space. Yeah,[00:00:21] it's awesome to have another awesome guest Shankar. Welcome .[00:00:25] Thanks for having me. I'm super excited.[00:00:27] Introducing Shreya (poorly)[00:00:27] So I'll intro your formal background and then you can fill in the blanks.[00:00:31] You are a bsms and then PhD at, in, in Computer Science at Stanford. So[00:00:36] I'm, I'm a PhD at Berkeley. Ah, Berkeley. I'm sorry. Oops. . No, it's okay. Everything's the bay shouldn't say that. Everybody, somebody is gonna get mad, but . Lived here for eight years now. So[00:00:50] and then intern at, Google Machine learning learning engineer at Viaduct, an OEM manufacturer, uh, or via OEM analytics platform.[00:00:59] Yes. And now you're an e I R entrepreneur in residence at Amplify.[00:01:02] I think that's on hold a little bit as I'm doing my PhD. It's a very unofficial title, but it sounds fancy on paper when you say[00:01:09] it out loud. Yeah, it is fancy. Well, so that is what people see on your LinkedIn. What's, what should, what should people know about you that's not on your LinkedIn?[00:01:16] Yeah, I don't think I updated my LinkedIn since I started the PhD, so, I'm doing my PhD in databases. It is not AI machine learning, but I work on data management for building AI and ML powered software. I guess like all of my personal interests, I'm super into going for walks, hiking, love, trying coffee in the Bay area.[00:01:42] I recently, I've been getting into cooking a lot. Mm-hmm. , so what kind of cooking? Ooh. I feel like I really like pastas. But that's because I love carbs. So , I don't know if it's the pasta as much as it's the carb. Do you ever cook for[00:01:56] like large[00:01:57] dinners? Large groups? Yeah. We just hosted about like 25 people a couple weeks ago, and I was super ambitious.[00:02:04] I was like, I'm gonna cook for everyone, like a full dinner. But then kids were coming. and I was like, I know they're not gonna eat tofu. The other thing with hosting in the Bay Area is there's gonna be someone vegan. There's gonna be someone gluten-free. Mm-hmm. . There's gonna be someone who's keto. Yeah.[00:02:20] Good luck, .[00:02:21] Oh, you forgot the seeds. That's the sea disrespects.[00:02:25] I know. . So I was like, oh my God, I don't know how I'm gonna do this. Yeah. The dessert too. I was like, I don't know how I'm gonna make everything like a vegan, keto nut free dessert, just water. It was a fun challenge. We ordered pizza for the children and a lot of people ate the pizza.[00:02:43] So I think , that's what happens when you try to cook, cook for everyone.[00:02:48] Yeah. The reason I dug a bit on the cooking is I always find like if you do cook for large groups, it's a little bit like of an ops situation. Yeah. Like a lot of engineering. A lot of like trying to figure out like what you need to deliver and then like what the pipeline[00:02:59] is and Oh, for sure.[00:03:01] You write that Gantt chart like a day in advance. , did you actually have a ga? Oh, I did. My gosh. Of course I had a Gantt chart. I, I dunno how people, did[00:03:08] you orchestrate it with airflow or ?[00:03:12] I orchestrated it myself. .[00:03:15] That's awesome. But yeah, we're so excited to have you, and you've been a pretty prolific writer, researcher, and thank you.[00:03:20] You have a lot of great content out there. I think your website now says, I'm currently learning how to make machine learning work in the real world, which is a challenge that mm-hmm. , everybody is steaming right now from the Microsoft and Googles of the word that have rogue eyes flirting with people, querying them to people, deploy models to production.[00:03:38] The 3 V's of ML development[00:03:38] Maybe let's run through some of the research you've done, especially on lops. Sure. And how to get these things in production. The first thing I really liked from one of your paper was the, the three VS of ML development. Mm-hmm. , which is velocity validation and versioning. And one point that you were making is that the development workflow of software engineering is kind of very different from ML because ML is very experiment driven.[00:04:00] Correct. There's a lot of changes that you need to make, you need to kill things very quickly if they're not working. So maybe run us through why you decided as kind of those three vs. Being some of the, the core things to think about. and some of the other takeaways from their research. Yeah,[00:04:15] so this paper was conducted as a loosely structured interview study.[00:04:18] So the idea is you interview like three or four people and then you go and annotate all the transcripts, tag them, kind of put the word clouds out there, whatever. There's a bunch of like cool software to do this. Then we keep seeing these, themes of velocity wasn't the word, but it was like experiment quickly or high experimentation rate.[00:04:38] Sometimes it was velocity. And we found that that was like the number one thing for people who were talking about their work in this kind of development phase. We also categorized it into phases of the work. So the life cycle like really just fell into place when we annotated the transcripts. And so did the variables.[00:04:55] And after three or four interviews you iterate on them. You kind of iterate on the questions, and you iterate on the codes or the tags that you give to the transcripts and then you do it again. And we repeated this process like three or four times up to that many people, and the story kind of told itself in a way that[00:05:11] makes sense.[00:05:12] I think, like I was trying to figure out why you picked those, but it's interesting to see that everybody kinda has the same challenges.[00:05:18] It fell out. I think a big thing, like even talking to the people who are at the Microsofts and the Googles, they have models in production. They're frequently training these models in production, yet their Devrel work is so experimental.[00:05:31] Mm-hmm. . And we were like, so it doesn't change. Even when you become a mature organization, you still throw 100 darts at the wall for five of them to stick and. That's super interesting and I think that's a little bit unique to data science and machine learning work.[00:05:45] Bridging Development and Production[00:05:45] Yeah. And one one point you had is kind of how do we bridge the gap between the development environments and the production environments?[00:05:51] Obviously you're still doing work in this space. What are some of the top of mind areas of focus for you in[00:05:57] this area? Yeah, I think it. Right now, people separate these environments because the production environment doesn't allow people to move at the rate that they need to for experimentation. A lot of the times as you're doing like deep learning, you wanna have GPUs and you don't wanna be like launching your job on a Kubernetes cluster and waiting for the results to come.[00:06:17] And so that's just the hardware side of things. And then there is the. Execution stack. Um, you wanna be able to query and create features real time as you're kind of training your model. But in production things are different because these features are kind of scheduled, maybe generated every week.[00:06:33] There's a little bit of lag. These assumptions are not accounted for. In development and training time. Mm-hmm. . So of course we're gonna see that gap. And then finally, like the top level, the interface level. People wanna experiment in notebooks, in environments that like allow them to visualize and inspect their state.[00:06:50] But production jobs don't typically run in notebooks. Yeah, yeah, yeah. I mean there, there are tools like paper mill and et cetera. But it's not the same, right? So when you just look at every single layer of the kind of data technical stack, there's a develop. Side of things and there's a production side of things and they're completely different.[00:07:07] It makes sense why. Way, but I think that's why you get a bunch of bugs that come when you put things in production.[00:07:14] I'm always interested in the elimination of those differences. Mm-hmm. And I don't know if it's realistic, but you know, what would it take for people to, to deploy straight to production and then iterate on production?[00:07:27] Because that's ultimately what you're[00:07:29] aim for. This is exactly what I'm thinking about right now in my PhD for kind of like my PhD. But you said it was database. I think databases is a very, very large field. , pretty much they do everything in databases . But the idea is like, how do we get like a unified development and production experience, Uhhuh, for people who are building these ML models, I think one of the hardest research challenges sits at that execution layer of kind of how do.[00:07:59] Make sure that people are incorporating the same assumptions at development time. Production time. So feature stores have kind of come up in the last, I don't know, couple of years, three years, but there's still that online offline separation. At training time, people assume that their features are generated like just completely, perfectly.[00:08:19] Like there's no lag, nothing is stale. Mm-hmm. , that's the case when trading time, but those assumptions aren't really baked. In production time. Right. Your features are generated, I don't know, like every week or some Every day. Every hour. That's one thing. How do, like, what does that execution model look like to bridge the two and still give developers the interactive latencies with features?[00:08:40] Preventing Data Leakage[00:08:40] Mm-hmm. . I think another thing also, I don't know if this is an interface problem, but how do we give developers the guardrails to not look at data that they're not supposed to? This is a really hard problem. For privacy or for training? Oh, no, just for like training. Yeah. Okay. also for privacy. Okay. But when it comes to developing ML models in production, like you can't see, you don't see future data.[00:09:06] Mm-hmm. . Yeah. You don't see your labels, but at development time it's really easy to. to leak. To leak and even like the seeming most seemingly like innocuous of ways, like I load my data from Snowflake and I run a query on it just to get a sense for, what are the columns in my data set? Mm-hmm. or like do a DF dot summary.[00:09:27] Mm-hmm. and I use that to create my features. Mm-hmm. and I run that query before I do train test. , there's leakage in that process. Right? And there's just at the fun, most fundamental level, like I think at some point at my previous company, I just on a whim looked through like everyone's code. I shouldn't have done that , but I found that like everyone's got some leakage assumptions somewhere.[00:09:49] Oh, mm-hmm. . And it's, it's not like people are bad developers, it's just that. When you have no guard the systems. Yeah, do that. Yeah, you do this. And of course like there's varying consequences that come from this. Like if I use my label as a feature, that's a terrible consequence. , if I just look at DF dot summary, that's bad.[00:10:09] I think there's like a bunch of like unanswered interesting research questions in kind of creating. Unified experience. I was[00:10:15] gonna say, are you about to ban exploratory data analysis ?[00:10:19] Definitely not. But how do we do PDA in like a safe , data safe way? Mm-hmm. , like no leakage whatsoever.[00:10:27] Right. I wanna ask a little small follow up about doing this at Berkeley.[00:10:31] Berkeley's Uniquely Research Lab Culture[00:10:31] Mm-hmm. , it seems that Berkeley does a lot of this stuff. For some reason there's some DNA in Berkeley that just, that just goes, hey, just always tackle this sort of hard data challenges. And Homestate Databricks came out of that. I hear that there's like some kind of system that every five years there's a new lab that comes up,[00:10:46] But what's going on[00:10:47] there? So I think last year, rise Lab which Ray and any scale came out of. Kind of forked into two labs. Yeah. Sky Lab, I have a water bottle from Sky Lab. Ooh. And Epic Lab, which my advisor is a co-PI for founding pi, I don't know what the term is. And Skylabs focus, I think their cider paper was a multi-cloud programming environment and Epic Lab is, Their focus is more like low-code, no-code, better data management tools for this like next generation of Interfa.[00:11:21] I don't even know. These are like all NSF gra uh, grants.[00:11:24] Yeah. And it's five years, so[00:11:26] it could, it could involve, yeah. Who knows what's gonna be, and it's like super vague. Yeah. So I think we're seeing like two different kinds of projects come out of this, like the sky projects of kind of how do I run my job on any cloud?[00:11:39] Whichever one is cheapest and has the most resources for me, my work is kind of more an epic lab, but thinking about these like interfaces, mm-hmm. , better execution models, how do we allow people to reason about the kind of systems they're building much more effectively. Yeah,[00:11:53] From Static Data to Dynamically Updated Data[00:11:53] yeah. How do you think about the impact of the academia mindset when then going into.[00:11:58] Industry, you know, I know one of the points in your papers was a lot of people in academia used with to static data sets. Mm-hmm. , like the data's not updating, the data's not changing. So they work a certain way and then they go to work and like they should think about bringing in dynamic data into Yeah.[00:12:15] Earlier in the, in the workflow, like, , how do you think we can get people to change that mindset? I think[00:12:21] actually people are beginning to change that mindset. We're seeing a lot of kind of dynamic data benchmarks or people looking into kind of streaming datasets, largely image based. Some of them are language based, but I do think it's somewhat changing, which is good.[00:12:35] But what I don't think is changing is the fact that model researchers and Devrel developers want. to create a model that learns the world. Mm-hmm. . And that model is now a static artifact. I don't think that's the way to go. I want people, at least in my research, the system I'm building, models are not a one time thing.[00:12:55] Models as views on Data[00:12:55] Models are views that are frequently recomputed over your data to use database speak, and I don't see people kind of adopting that mindset when it comes to. Kind of research or the data science techniques that people are learning in school. And it's not just like retrain G P T every single day or whatever, but it, it is like, how do I make sure that I don't know, my system is evolving over time.[00:13:19] Mm-hmm. that whatever predictions or re query results that are being generated are. Like that process is changing. Can you give[00:13:27] a, an overview of your research project? I know you mentioned a couple snippets here and there,[00:13:32] but that would be helpful. . I don't have a great pitch yet. I haven't submitted anything, still working on it, but the idea is like I want to create a system for people to develop their ML pipelines, and I want it to be like, Like unifying the development production experience.[00:13:50] And the key differences about this is one, you think of models as like data transformations that are recomputed regularly. So when you write your kind of train or fit functions, like the execution engine understands that this is a process that runs repeatedly. It monitors the data under the hood to refit the computation whenever it's detected.[00:14:12] That kind of like the data distributions have changed. So that way whenever you. Test your pipelines before you deploy them. Retraining is baked in, monitoring is baked in. You see that? And the gold star, the gold standard for me is the number that you get at development time. That should be the number that you get when you deploy[00:14:33] There shouldn't be this expected 10% drop. That's what I know I will have. Made something. But yeah, definitely working on that.[00:14:41] Yeah. Cool. So a year ago you tweeted a list of principles that you thought people should know and you split it very hopefully. I, I thought into beginner, intermediate, advanced, and sometimes the beginner is not so beginner, you know what I mean?[00:14:52] Yeah, definitely. .[00:14:53] The first one I write is like,[00:14:57] so we don't have to go through the whole thing. I, I do recommend people check it out, but also maybe you can pick your favorites and then maybe something you changed your mind.[00:15:03] Principle: Version Everything You Do[00:15:03] I think several of them actually are about versioning , which like maybe that bias the interview studying a little bit.[00:15:12] Yeah. But I, I really think version everything you do, because in experimentation time, because when you do an experiment, you need some version there because if you wanna pr like publish those. , you need something to go back to. And the number of people who like don't version things, it is just a lot. It's also a lot to expect for someone to commit their code every time they like.[00:15:33] Mm-hmm. train their model. But I think like having those practices is definitely worth it. When you say versioning,[00:15:39] you mean versioning code.[00:15:40] versioning code versioning data, like everything around a single like trial run.[00:15:45] So version code get fine. Mm-hmm. versioning data not[00:15:48] as settled. Yeah. I think that part, like you can start with something super hacky, which is every time you run your script, like just save a copy of your training set.[00:16:00] Well, most training sets are not that big. Yeah. Like at least when people are like developing on their computer, it. Whatever. It's not that big. Just save a copy somewhere. Put it ass three, like it's fine. It's worth it. Uhhuh, . I think there's also like tools like dvc like data versioning kind of tools. I think also like weights and biases and these experiment track like ML flow, the experiment tracking tools have these hooks to version your data for you.[00:16:23] I don't know how well they work these days, but . Yeah, just something around like versioning. I think I definitely agree with[00:16:30] Principle: Always validate your Data[00:16:30] I'm. Super, super big into data validation. People call it monitoring. I used to think it was like monitoring. I realize now like how little at my previous company, we just like validated the input data going into these pipelines and even talking to people in the interview study people are not doing.[00:16:48] Data validation, they see that their ML performance is dropping and they're like, I don't know why. What's going on ? And when you dig into it, it's a really fascinating, interesting, like a really interesting research problem. A lot of data validation techniques for machine learning result in too many false positive alerts.[00:17:04] And I have a paper got rejected and we're resubmitting on this. But yeah, like there, it's active research problem. How do you create meaningful alerts, especially when you have tons of features or you have large data sets, that's a really hard problem, but having some basic data validation check, like check that your data is complete.[00:17:23] Check that your schema matches up. Check that your most frequent, like your. Most frequently occurring value is the same. Your vocabulary isn't changing if it's a large language model. These are things that I definitely think I could have. I should have said that I did say data validation, but I didn't like, like spell it out.[00:17:39] Have you, have you looked into any of the current data observability platforms like Montecarlo or Big I I think you, I think you have some experience with that as[00:17:47] well. Yeah. I looked at a Monte car. Couple of years back, I haven't looked into big eye. I think that designing data validation for ML is a different problem because in the machine learning setting, you can allow, there's like a tolerance for how corrupted your data is and you can still get meaningful prediction.[00:18:05] Like that's the whole point of machine learning. Yeah, so like. A lot of the times, like by definition, your data observability platform is gonna give you false positives if you just care about the ML outputs. So the solution really, at least our paper, has this scheme where we learn from performance drops to kind of iterate on the precision of the data validation, but it's a hybrid of like very old databases techniques as well as kind of adapting it to the ML setting.[00:18:33] Heuristics for Model Architecture Selection[00:18:33] So you're an expert in the whole stack. I think I, I talk with a lot of founders, CTOs right now that are saying, how can I get more ML capabilities in, in my application? Especially when it comes to LLMs. Mm-hmm. , which are kind of the, the talk of the town. Yeah. How should people think about which models to use, especially when it comes to size and how much data they need to actually make them useful, for example, PT three is 175 billion parameters co-pilot use as a 12 billion model.[00:19:02] Yeah. So it's much smaller, but it's very good for what it does. Do you have any heuristics or mental models that you use when teams should think about what models to use and how big they need it to be?[00:19:12] Yeah I think that the. Precursor to this is the operational capabilities that these teams have. Do they have the capability to like literally host their own model, serve their own model, or would they rather use an api?[00:19:25] Mm-hmm. , a lot of teams like don't have the capability to maintain the actual model artifact. So even like the process of kind of. Fine tuning A G P T or distilling that, doing something like it's not feasible because they're not gonna have someone to maintain it over time. I see this with like some of the labs, like the people that we work with or like the low-code, no-code.[00:19:47] Or you have to have like really strong ML engineers right over time to like be able to have your own model. So that's one thing. The other thing is these G P T, these, these large language models, they're really good. , like giving you useful outputs. Mm-hmm. compared to like creating your own thing. Mm-hmm.[00:20:02] even if it's smaller, but you have to be okay with the latency. Mm-hmm. and the cost that comes out of it. In the interview study, we talk to people who are keeping their own, like in memory stores to like cash frequently. I, I don't know, like whatever it takes to like avoid calling the Uhhuh API multiple types, but people are creative.[00:20:22] People will do this. I don't think. That it's bad to rely on like a large language model or an api. I think it like in the long term, is honestly better for certain teams than trying to do their own thing on[00:20:36] house.[00:20:36] The LLMOps Stack[00:20:36] How's the L l M ops stack look like then? If people are consuming this APIs, like is there a lot of difference in under They manage the, the data, the.[00:20:46] Well,[00:20:46] I'll tell you the things that I've seen that are unified people need like a state management tool because the experience of working with a L L M provi, like A G P T is, mm-hmm. . I'm gonna try start out with these prompts and as I learn how to do this, I'm gonna iterate on these prompts. These prompts are gonna end up being this like dynamic.[00:21:07] Over time. And also they might be a function of like the most recent queries Tonight database or something. So the prompts are always changing. They need some way to manage that. Mm-hmm. , like I think that's a stateful experience and I don't see the like, like the open AI API or whatever, like really baking that assumption in into their model.[00:21:26] They do keep a history of your[00:21:27] prompts that help history. I'm not so sure. , a lot of times prompts are like, fetch the most recent similar data in my database, Uhhuh, , and then inject that into the pump prompt. Mm-hmm. . So I don't know how, Okay. Like you wanna somehow unify that and like make sure that's the same all the time.[00:21:44] You want prompt compiler. Yeah, . I think there's some startup probably doing that. That's definitely one thing. And then another thing that we found very interesting is that when people put these. LLMs in production, a lot of the bugs that they observe are corrected by a filter. Don't output something like this.[00:22:05] Yes. Or don't do this like, so there's, or please output G on, yeah. . So these pipelines end up becoming a hybrid of like the API uhhuh, they're. Service that like pings their database for the most recent things to put in their prompt. And then a bunch of filters, they add their own filters. So like what is the system that allows people to build, build such a pipeline, this like hybrid kind of filter and ML model and dynamic thing.[00:22:30] So, so I think like, The l l m stack, like is looking like the ML ops thing right in this way of like hacking together different solutions, managing state all across the pipeline monitoring, quick feedback loop.[00:22:44] Yeah. You had one, uh, just to close out the, the tweet thread thing as well, but this is all also relevant.[00:22:50] Shadow Models[00:22:50] You have an opinion about shadowing a less complicated model in production to fall back on. Yeah. Is that a good summary?[00:22:55] The shadowing thing only works in situations where you don. Need direct feedback from. The user because then you can like very reasonably serve it like Yeah, as as long, like you can benchmark that against the one that's currently in production, if that makes sense.[00:23:15] Right. Otherwise it's too path dependent or whatever to.[00:23:18] evaluate. Um, and a lot of services can benefit from shadowing. Like any, like I used to work a lot on predictive analytics, predictive maintenance, like stuff like that, that didn't have, um, immediate outputs. Mm-hmm. or like immediate human feedback. So that was great and okay, and a great way to like test the model.[00:23:36] Got it. But I think as. Increasingly trying to generate predictions that consumers immediately interact with. It might not be I, I'm sure there's an equivalent or a way to adapt it. Mm-hmm. AV testing, stage deployment, that's in the paper.[00:23:53] Keeping Up With Research[00:23:53] Especially with keeping up with all the new thing. That's one thing that I struggle with and I think preparing for this. I read a lot of your papers and I'm always like, how do you keep up with, with all of this stuff?[00:24:02] How should people do it? You know? Like, now, l l M is like the hot thing, right? There's like the, there's like the chinchilla study. There's like a lot of cool stuff coming out. Like what's. U O for like staying on top of this research, reading it. Yeah. How do you figure out which ones are worth reading?[00:24:16] Which ones are kind of like just skim through? I read all of yours really firmly. , but I mean other ones that get skimmed through, how should people figure it out?[00:24:24] Yeah, so I think. I'm not the best person to ask for this because I am in a university and every week get to go to amazing talks. Mm-hmm. and like engage with the author by the authors.[00:24:35] Yeah. Right. Yeah. Yeah. So it's like, I don't know, I feel like all the opportunities are in my lap and still I'm struggling to keep up, if that makes sense. Mm-hmm. . I used to keep like running like a bookmark list of papers or things that I want to read. But I think every new researcher does that and they realize it's not you worth their time.[00:24:52] Right? Like they will eventually get to reading the paper if it's absolutely critical. No, it's, it's true, it's true. So like we've, I've adopted this mindset and like somehow, like I do end up reading things and the things that I miss, like I don't have the fo. Around. So I highly encourage people to take that mentality.[00:25:10] I also, I think this is like my personal taste, but I love looking into the GitHub repos that people are actually using, and that usually gives me a sense for like, what are the actual problems that people have? I find that people on Twitter, like sometimes myself included, will say things, but you, it's not how big of a problem is it?[00:25:29] Mm-hmm. , it's not. Yeah, like , I find that like just looking at the repos, looking at the issues, looking at how it's evolved over time, that really, really helps. So you're,[00:25:40] to be specific, you're not talking about paper repos?[00:25:43] No, no, no, no. I'm talking about tools, but tools also come with papers a lot in, um, databases.[00:25:49] Yeah. Yeah. I think ML specifically, I think there's way too much ML research out there and yeah, like so many papers out there, archive is like, kind of flooded. Yeah.[00:26:00] It's like 16% of old papers produced.[00:26:02] It's, it's crazy. . I don't know if it's a good use of time to try to read all of them, to be completely honest.[00:26:10] Grounded Theory for Problem Discovery[00:26:10] You have a very ethnographic approach, like you do interviews and I, I assume like you just kinda observe and don't Yeah. Uh, prescribe anything. And then you look at those GitHub issues and you try to dig through from like production, like what is this orientation? Is there like a research methodology that you're super influenced by that guides you like this?[00:26:28] I wish that I had. Like awareness and language to be able to talk about this. Uhhuh, , . I[00:26:37] don't know. I, I think it's, I think it's a bit different than others who just have a technology they wanna play with and then they, they just ignore, like they don't do as much, uh, like people research[00:26:47] as[00:26:47] you do. So the HCI I researchers like, Have done this forever and ever and ever.[00:26:53] Yeah. But grounded theory is a very common methodology when it comes to trying to understand more about a topic. Yeah. Which is you go in, you observe a little bit, and then you update your assumptions and you keep doing this process until you have stopped updating your assumptions. . And I really like that approach when it comes to.[00:27:13] Just kind of understanding the state of the world when it comes to like a cer, like LLMs or whatever, until I feel like, like there was like a point in time for like lops on like tabular data prior to these large language models. I feel like I, I'd gotten the space and like now that these like large language models have come out and people are really trying to use them.[00:27:35] They're tabular kind of predictions that they used to in the past. Like they're incorporating language data, they're incorporating stuff like customer feedback from the users or whatever it is to make better predictions. I feel like that's totally changing the game now, and I'm still like, Why, why is this the case?[00:27:52] Was were the models not good enough? Do people feel like they're behind? Mm-hmm. ? I don't know. I try to talk to people and like, yeah, I have no answers.[00:27:59] Google Brain vs Academia[00:27:59] So[00:27:59] how does the industry buzz and focus influence what stuff the research teams work on? Obviously arch language models, everybody wants to build on them.[00:28:08] When you're looking at, you know, other peers in the, in the PhD space, are they saying, oh, I'm gonna move my research towards this area? Or are they just kind of focused on the idea of the[00:28:18] first. . This is a good question. I think that we're at an interesting time where the kind of research a PhD student in an academic institution at CS can do is very different from the research that a large company, because there aren't like, There just aren't the resources.[00:28:39] Mm-hmm. that large companies compute resources. There isn't the data. And so now PhD students I think are like, if they want to do something better than industry could do it, like there's like a different class of problems that we have to work on because we'll never be able to compete. So I think that's, yeah, I think that's really hard.[00:28:56] I think a lot of PhD students, like myself included, are trying to figure out like, what is it that we can do? Like we see the, the state of the field progressing and we see. , why are we here? If we wanna train language model, I don't, but if somebody wants to train language models, they should not be at uc.[00:29:11] Berkeley, , they shouldn't .[00:29:15] I think it's, there's a sort of big, gets bigger mentality when it comes to training because obviously the big companies have all the data, all the money. But I was kind of inspired by Luther ai. Mm-hmm. , um, which like basically did independent reproductions Yeah. Of G P T three.[00:29:30] Don't you think like that is a proof of, of existence that it is possible to do independently?[00:29:34] Totally. I think that kind of reproducing research is interesting because it doesn't lead to a paper. Like PhD students are still like, you can only graduate when you have papers. Yeah. So to have a whole lab set.[00:29:46] I think Stanford is interesting cuz they did do this like reproducing some of the language models. I think it should be a write[00:29:50] a passage for like every year, year one PhD. You[00:29:53] must reproduce everything. I won't say that no one's done it, but I do understand that there's an incentive to do new work because that's what will give you the paper.[00:30:00] Yeah. So will you put 20 of your students to. I feel like only a Stanford or somebody who like really has a plan to make that like a five plus year. Mm-hmm. research agenda. And that's just the first step sort of thing. Like, I can't imagine every PhD student wants to do that. Well, I'm just[00:30:17] saying, I, I, I feel like that there will be clouds, uh, the, the, you know, the big three clouds.[00:30:21] Mm-hmm. Probably the Microsoft will give you credits to do whatever you want. And then it's on you to sort of collect the data but like there of existence that it is possible to[00:30:30] It's definitely possible. Yeah. I think it's significantly harder. Like collecting the data is kind of hard. Like just like because you have the cloud credits doesn't mean like you have a cluster that has SREs backing it.[00:30:42] Mm-hmm. who helped you run your experiments. Right, right. Like if you are at Google Rain. Yeah. I was there what, like five, six years ago. God, like I read an experiment and I didn. Problems. Like it was just there. Problems . It's not like I'm like running on a tiny slur cluster, like watching everything fail every five.[00:31:01] It's like, this is why I don't train models now, because I know that's not a good use of my time. Like I'll be in so many like SRE issues. Yeah. If I do it now, even if I have cloud credits. Right. So, Yeah, I think it's, it can feel disheartening. , your PhD student training models,[00:31:18] well, you're working on better paradigms for everyone else.[00:31:21] You know? That's[00:31:22] the goal. I don't know if that's like forced, because I'm in a PhD program, , like maybe if I were someone else, I'd be training models somewhere else. I don't know. Who knows? Yeah. Yeah.[00:31:30] You've read a whole post on this, right? Choosing between a PhD and going into. Obviously open ai. Mm-hmm. is kinda like the place where if you're a researcher you want to go go work in three models.[00:31:41] Advice for New Grads[00:31:41] Mm-hmm. , how should people think about it? What are like maybe areas of research that are underappreciated in industry that you're really excited about at a PhD level? Hmm.[00:31:52] I think I wrote that post for new grads. . So it might not be as applicable like as a new grad. Like every new grad is governed by, oh, not every, a good number of new grads are governed by, like, I wanna do work on something that's impactful and I want to become very known for this.[00:32:06] Mm-hmm. , like, that's like , like a lot of, but like they don't really, they're walking outta the world for the first time almost. So for that reason, I think that like it's worth working on problems. We'll like work on any data management research or platform in an industry that's like working on Providence or working on making it more efficient to train model or something like.[00:32:29] You know, that will get used in the future. Mm-hmm. . So it might be worth just going and working on that in terms of, I guess like going to work at a place like OpenAI or something. I do think that they're doing very interesting work. I think that it's like not a fad. These models are really interesting.[00:32:44] Mm-hmm. and like, they will only get more interesting if you throw more compute Right. And more data at them. So it, it seems like these industry companies. Doing something interesting. I don't know much more than that. .[00:32:59] Helping Minorities in CS[00:32:59] Cool. What are other groups, organizations, I know you, you're involved with, uh, you were involved with She Plus Plus Helping with the great name.[00:33:07] Yeah, I just[00:33:08] got it.[00:33:10] when you say it[00:33:10] out loud, didn't name Start in 2012. Long time ago. Yeah.[00:33:15] What are some of the organizations you wanna highlight? Anything that that comes to?[00:33:20] Yeah. Well, I mean, shva Plus is great. They work on kind of getting more underrepresented minorities in like high school, interested, kind of encoding, like I remember like organizing this when I was in college, like for high schoolers, inviting them to Stanford and just showing them Silicon Valley.[00:33:38] Mm-hmm. and the number of students who went from like, I don't know what I wanna do to, like, I am going to major or minor in c. Almost all of them, I think. I think like people are just not aware of the opportunities in, like, I didn't really know what a programmer was like. I remember in Texas, , like in a small town, like it's, it's not like one of the students I've mentored, their dad was a vc, so they knew that VC is a career path.[00:34:04] Uhhuh, . And it's like, I didn't even know, like I see like, like stuff like this, right? It's like just raising your a. Yeah. Or just exposure. Mm-hmm. , like people who, kids who grow up in Silicon Valley, I think like they're just in a different world and they see different things than people who are outside of Silicon Valley.[00:34:20] So, yeah, I think Chiles West does a great job of like really trying to like, Expose people who would never have had that opportunity. I think there's like also a couple of interesting programs at Berkeley that I'm somewhat involved in. Mm-hmm. , there's dare, which is like mentoring underrepresented students, like giving research opportunities and whatnot to them and Cs.[00:34:41] That's very interesting. And I'm involved with like a summer program that's like an r u also for underrepresented minorities who are undergrads. , find that that's cool and fun. I don't know. There aren't that many women in databases. So compared to all the people out there. ? Yeah.[00:35:00] My wife, she graduated and applied physics.[00:35:02] Mm-hmm. . And she had a similar, similar feeling when she was in, in school.[00:35:06] Lightning Round[00:35:06] All right. Let's jump into the lining ground. So your favorite AI product.[00:35:12] I really like. Stable diffusion, like managed offerings or whatever. I use them now to generate all of my figures for any talks that I give. I think it's incredible.[00:35:25] I'm able to do this or all of my like pictures, not like graphs or whatever, .[00:35:31] It'd be great if they could do that. Really looking[00:35:34] forward to it. But I, I love, like, I'll put things like bridging the gap between development and production or whatever. I'll do like a bridge between a sandbox and a city. Like, and it'll make it, yeah.[00:35:46] like, I think that's super cool. Yeah. Like you can be a little, I, I enjoy making talks a lot more because of , these like dream studio, I, I don't even know what they're called, what organization they're behind. I think that is from Stability. Stability,[00:35:58] okay. Yeah. But then there's, there's like Lexi there. We interviewed one that's focused on products that's Flare ai, the beauty of stable diffusion being open sources.[00:36:07] Yeah. There's 10[00:36:07] of these. Totally, totally. I'll just use whichever ones. I have credits on .[00:36:13] A lot of people focus on, like have different focuses, like Sure. Mid Journey will have an art style as a focus. Mm-hmm. and then some people have people as the focus for scenes. I, I feel like just raw, stable diffusion two probably is the[00:36:24] best.[00:36:24] Yeah. Yeah. But I don't do, I don't have images of people in my slides . Yeah, yeah. Yeah. That'd be a little bit weird.[00:36:31] So a year from now, what do you think people will be most surprised by in ai? What's on the horizon and about to come, but people don't realize. .[00:36:39] I don't know if this will be, this is related to the AI part of things or like an AI advancement, but I consistently think people underestimate the data management challenges.[00:36:50] Ooh. In putting these things in production. Uhhuh, . And I think people get frustrated that they really try, they see these like amazing prototypes, but they cannot for the life of them, figure out how to leverage them in their organization. And I think. That frustration will be collectively felt by people as it's like it's happened in the past, not for LLMs, but for other machine learning models.[00:37:15] I think people will turn to whatever it, it's just gonna be really hard, but we're gonna feel that collective frustration like next year is what I think.[00:37:22] And we talked a little bit before the show about data management platforms. Yeah. Do you have a spec for what that[00:37:27] is? The broad definition is a system that handles kind of execution.[00:37:33] or orchestration of different like data transformations, data related transformation in your pipeline. It's super broad. So like feature stores, part of it, monitoring is part of it. Like things that are not like your post request to open AI's, p i, , .[00:37:51] What's one AI thing you would pay for if someone built.[00:37:54] So whenever I do like web development or front end projects or like build dashboards, like often I want to manage my styles in a nice way.[00:38:02] Like I wanna generate a color palette, uhhuh, and I wanna manage it, and I wanna inject it throughout the application. And I also wanna be able to change it over time. Yeah. I don't know how to do this. Well, ? Yeah, in like large or E even like, I don't know, just like not even that large of projects. Like recently I was building my own like Jupyter Notebook cuz you can do it now.[00:38:23] I'm super excited by this. I think web assembly is like really changed a lot of stuff. So I was like building my own Jupyter Notebook just for fun. And I used some website to generate a color palette that I liked and then I was like, how do I. Inject this style like consist because I was learning next for the first time.[00:38:39] Yeah. And I was using next ui. Yeah. And then I was like, okay, like I could just use css but then like, is that the way to do it for this? Like co-pilot's not gonna tell me how to do this. There's too many options. Yeah. So just like, let me like just read my code and read and give me a color palette and allow me to change it over time and have this I opera.[00:38:58] With different frameworks, I would pay like $5 a month for this.[00:39:01] Yeah, yeah, yeah. It's, it's a, you know, the classic approach to this is have a design system and then maintain it. Yeah. I'm not designing Exactly. Do this. Yeah, yeah, yeah, yeah. This is where sort of the front end world eats its own tail because there's like, 10 different options.[00:39:15] They're all awesome. Yeah, you would know . I'm like, I have to apologize on behalf of all those people. Cuz like I, I know like all the individual solutions individually, but I also don't know what to recommend to you .[00:39:28] So like that's therein lies is the thing, right? Like, ai, solve this for me please. ,[00:39:35] what's one thing you want everyone to take away about?[00:39:39] I think it's really exciting to me in a time like this where we're getting to see like major technological advances like in front of our eyes. Maybe the last time that we saw something of this scale was probably like, I don't know, like I was young, but still like Google and YouTube and those. It's like they came out and it was like, wow, like the internet is so cool , and I think we're getting to see something like that again.[00:40:05] Yeah. Yeah. I think that's just so exciting. To be a part of it somehow, and maybe I'm like surrounded by a bunch of like people who are like, oh, like it's just a fad or it's just a phase. But I don't think so. Mm-hmm. , I think I'm like fairly grounded. So yeah. That's the one takeaway I have. It's, it's not a fad.[00:40:24] My grandma asked me about chat, g p t, she doesn't know what a database is, but she knows about chat. G p t I think that's really crazy. , what does she, what does she use it for? No, she just like saw a video about it. Ah, yeah. On like Instagram or not, she's not like on like something YouTube. She watches YouTube.[00:40:41] She's sorry. She saw like a video on ChatGPT and she was like, what do you think? Is it a fad? And I was like, oh my god. , she like watched after me with this and I was like, do you wanna try it out? She was like, what ? Yeah,[00:40:55] she should.[00:40:55] Yeah, I did. I did. I don't know if she did. So yeah, I sent it to her though.[00:40:59] Well[00:40:59] thank you so much for your time, Sreya. Where should people find you online? Twitter.[00:41:04] Twitter, I mean, email me if you wanna directly contact me. I close my dms cuz I got too many, like being online, exposing yourself to strangers gives you a lot of dms. . Yeah. Yeah. But yeah, you can contact me via email.[00:41:17] I'll respond if I can. Yeah, if there's something I could actually be helpful with, so, oh,[00:41:22] awesome.[00:41:23] Thank you. Yeah, thanks for, thanks for. Get full access to Latent Space at www.latent.space/subscribe

Writing code that runs without errors—and without all the bugs that only show up when the program runs—is hard enough. But teaching others to write code and understand the underlying concepts takes a deeper understanding. Now imagine doing that for 37 courses. On this sponsored episode of the podcast, Ben and Ryan talk with Bharath Thippireddy, a VIP instructor at Udemy who has taught more than half a million students. We talk about how he went from a humble Java developer to one of Udemy's top instructors (and a budding movie star!). Along the way, we discuss whether Java or Python is better for beginners and how to balance theory with syntax. Episode notes:Like a lot of today's content creators, Bharath got his start posting videos on his Youtube channel in 2012.Today, you can find all of Bharath's courses on his Udemy page.You can find out more about Bharath from his website or connect with him on LinkedIn. Udemy is one of our launch partners for our online course recommendations. Congrats to Lifeboat badge winner desertnaut for their answer to What is the meaning of exclamation and question marks in Jupyter Notebook?.

Set up a high performance hybrid quantum compute environment in your own Azure Quantum workspace, and run your code on real quantum machines. See the latest advances, core concepts, and Microsoft's distinct topological approach to get us closer to realizing the world's first scalable quantum machine with Azure Quantum Computing. Microsoft Distinguished Engineer and Azure Quantum VP, Krysta Svore, joins host Apoorva Nori, to share what it is and how to set it up. ► QUICK LINKS: 00:00 - Introduction 02:40 - What is Quantum Computing? 04:40 - Applications suited for quantum computers 05:50 - Topological qubits 07:43 - Majorana zero modes 09:43 - How to set up Azure Quantum 12:51 - Quantum Intermediate Representations (QIR) 14:18 - Wrap up ► Link References: Start using Azure Quantum today at https://aka.ms/quantumworkspace Open source samples and learning materials at https://aka.ms/learnquantum  ► Unfamiliar with Microsoft Mechanics? As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. • Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries • Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog • Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/website ► Keep getting this insider knowledge, join us on social: • Follow us on Twitter: https://twitter.com/MSFTMechanics • Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ • Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ • Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics #QuantumComputing #Qubits #QuantumMachineLearning #QuantumComputers

Emily Gorcenski, Data & AI Service Line Lead at Thoughtworks, joins Corey on Screaming in the Cloud to discuss how big data is changing our lives - both for the better, and the challenges that come with it. Emily explains how data is only important if you know what to do with it and have a plan to work with it, and why it's crucial to understand the use-by date on your data. Corey and Emily also discuss how big data problems aren't universal problems for the rest of the data community, how to address the ethics around AI, and the barriers to entry when pursuing a career in data. About EmilyEmily Gorcenski is a principal data scientist and the Data & AI Service Line Lead of ThoughtWorks Germany. Her background in computational mathematics and control systems engineering has given her the opportunity to work on data analysis and signal processing problems from a variety of complex and data intensive industries. In addition, she is a renowned data activist and has contributed to award-winning journalism through her use of data to combat extremist violence and terrorism. The opinions expressed are solely her own.Links Referenced: ThoughtWorks: https://www.thoughtworks.com/ Personal website: https://emilygorcenski.com Twitter: https://twitter.com/EmilyGorcenski Mastodon: https://mastodon.green/@emilygorcenski@indieweb.social TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Emily Gorcenski, who is the Data and AI Service Line Lead over at ThoughtWorks. Emily, thank you so much for joining me today. I appreciate it.Emily: Thank you for having me. I'm happy to be here.Corey: What is it you do, exactly? Take it away.Emily: Yeah, so I run the data side of our business at ThoughtWorks, Germany. That means data engineering work, data platform work, data science work. I'm a data scientist by training. And you know, we're a consulting company, so I'm working with clients and trying to help them through the, sort of, morphing landscape that data is these days. You know, should we be migrating to the cloud with our data? What can we migrate to the cloud with our data? Where should we be doing with our data scientists and how do we make our data analysts' lives easier? So, it's a lot of questions like that and trying to figure out the strategy and all of those things.Corey: You might be one of the most perfectly positioned people to ask this question to because one of the challenges that I've run into consistently and persistently—because I watch a lot of AWS keynotes—is that they always come up with the same talking point, that data is effectively the modern gold. And data is what unlocks value to your busin—“Every business agrees,” because someone who's dressed in what they think is a nice suit on stage is saying that it's, “Okay, you're trying to sell me something. What's the deal here?” Then I check my email and I discover that Amazon has sent me the same email about the same problem for every region I've deployed things to in AWS. And, “Oh, you deploy this to one of the Japanese regions. We're going to send that to you in Japanese as a result.”And it's like, okay, for a company that says data is important, they have no idea who any of their customers are at this point, is that is the takeaway here. How real is, “Data is important,” versus, “We charge by the gigabyte so you should save all of your data and then run expensive things on top of it.”Emily: I think data is very important, if you know what you're going to do with it and if you have a plan for how to work with it. I think if you look at the history of computing, of technology, if you go back 20 years to maybe the early days of the big data era, right? Everyone's like, “Oh, we've got big data. Data is going to be big.” And for some reason, we never questioned why, like, we were thinking that the ‘big' in ‘big data' meant big is in volume and not ‘big' as in ‘big pharma.'This sort of revolution never really happened for most companies. Sure, some companies got a lot of value from the, sort of, data mining and just gather everything and collect everything and if you hit it with a big computational hammer, insights will come out and somehow there's insights will make you money through magic. The reality is much more prosaic. If you want to make money with data, you have to have a plan for what you're going to do with data. You have to know what you're looking for and you have to know exactly what you're going to get when you look at your data and when you try to answer questions with it.And so, when we see somebody like Amazon not being able to correlate that the fact that you're the account owner for all of these different accounts and that the language should be English and all of these things, that's part of the operational problem because it's annoying, to try to do joins across multiple tables in multiple regions and all of those things, but it's also part—you know, nobody has figured out how this adds value for them to do that, right? There's a part of it where it's like, this is just professionalism, but there's a part of it, where it's also like… whatever. You've got Google Translate. Figure out yourself. We're just going to get through it.I think that… as time has evolved from the initial waves of the big data era into the data science era, and now we're in, you know, all sorts of different architectures and principles and all of these things, most companies still haven't figured out what to do with data, right? They're still investing a ton of money to answer the same analytics questions that they were answering 20 years ago. And for me, I think that's a disappointment in some regards because we do have better tools now. We can do so many more interesting things if you give people the opportunity.Corey: One of the things that always seemed a little odd was, back when I wielded root credentials in anger—anger,' of course, being my name for the production environment, as opposed to, “Theory,” which is what I call staging because it works in theory, but not in production. I digress—it always felt like I was getting constant pushback from folks of, “You can't delete that data. It's incredibly important because one day, we're going to find a way to unlock the magic of it.” And it's, “These are web server logs that are 15 years old, and 98% of them by volume are load balancer health checks because it turns out that back in those days, baby seals got more hits than our website did, so that's not really a thing that we wind up—that's going to add much value to it.” And then from my perspective, at least, given that I tend to live, eat, sleep, breathe cloud these days, AWS did something that was refreshingly customer-obsessed when they came out with Glacier Deep Archive.Because the economics of that are if you want to store a petabyte of data, with a 12-hour latency on request for things like archival logs and whatnot, it's $1,000 a month per petabyte, which is okay, you have now hit a price point where it is no longer worth my time to argue with you. We're just not going to delete anything ever again. Problem solved. Then came GDPR, which is neither here nor there and we actually want to get rid of those things for a variety of excellent legal reasons. And the dance continues.But my argument against getting rid of data because it's super expensive no longer holds water in the way that it wants did for anything remotely resembling a reasonable amount of data. Then again, that's getting reinvented all the time. I used to be very, I guess we'll call it, I guess, a data minimalist. I don't want to store a bunch of data, mostly because I'm not a data person. I am very bad thinking in that way.I consider SQL to be the chests of the programming world and I'm not particularly great at it. And I also unlucky and have an aura, so if I destroy a bunch of stateless web servers, okay, we can all laugh about that, but let's keep me the hell away from the data warehouse if we still want a company tomorrow morning. And that was sort of my experience. And I understand my bias in that direction. But I'm starting to see magic get unlocked.Emily: Yeah, I think, you know, you said earlier, there's, like, this mindset, like, data is the new gold or data is new oil or whatever. And I think it's actually more true that data is the new milk, right? It goes bad if you don't use it, you know, before a certain point in time. And at a certain point in time, it's not going to be very offensive if you just leave it locked in the jug, but as soon as you try to open it, you're going to have a lot of problems. Data is very, very cheap to store these days. It's very easy to hold data; it's very expensive to process data.And I think that's where the shift has gone, right? There's sort of this, like, Oracle DBA legacy of, like, “Don't let the software developers touch the prod database.” And they've kind of kept their, like, arcane witchcraft to themselves, and that mindset has persisted. But now it's sort of shifted into all of these other architectural patterns that are just abstractions on top of this, don't let the software engineers touch the data store, right? So, we have these, like, streaming-first architectures, which are great. They're great for software devs. They're great for software devs. And they're great for data engineers who like to play with big powerful technology.They're terrible if you want to answer a question, like, “How many customers that I have yesterday?” And these are the things that I think are some of the central challenges, right? A Kappa architecture—you know, streaming-first architecture—is amazing if you want to improve your application developer throughput. And it's amazing if you want to build real-time analytics or streaming analytics into your platform. But it's terrible if you want your data lake to be navigable. It's terrible if you want to find the right data that makes sense to do the more complex things. And it becomes very expensive to try to process it.Corey: One of the problems I think I have that is that if I take a look at the data volumes that I work with in my day-to-day job, I'm dealing with AWS billing data as spit out by the AWS billing system. And there isn't really a big data problem here. If you take a look at some of the larger clients, okay, maybe I'm trying to consume a CSV that's ten gigabytes. Yes, Excel is going to violently scream itself to death if I try to wind up loading it there, and then my computer smells like burning metal all afternoon. But if it fits in RAM, it doesn't really feel like it's a big data problem, on some level.And it just feels that when I look at the landscape of all the different tools you can use for things like this, they just feel like it's more or less, hmm, “I have a loose thread on my shirt. Could you pass me that chainsaw for a second?” It just seems like stupendous overkill for anything that I'm working with. Counterpoint; that the clients I'm working with have massive data farms and my default response when I meet someone who's very good at an area that I don't do a lot of work in is—counterintuitively to what a lot of people apparently do on Twitter—is not the default assumption of oh, “I don't know anything about that space. It must be worthless and they must be dumb.”No. That is not the default approach to take anything, from my perspective. So, it's clear there's something very much there that I just don't see slash understand. That is a very roundabout way of saying what could be uncharitably distilled down to, “So, is your entire career bullshit?” But no, it is clearly not.There is value being extracted from this and it's powerful. I just think that there's been an industry-wide, relatively poor job done of explaining that value in ways that don't come across as contrived or profoundly disturbing.Emily: Yeah, I think there's a ton of value in doing things right. It gets very complicated to try to explain the nuances of when and how data can actually be useful, right? Oftentimes, your historical data, you know, it really only tells you about what happened in the past. And you can throw some great mathematics at it and try to use it to predict the future in some sense, but it's not necessarily great at what happens when you hit really hard changes, right?For example, when the Coronavirus pandemic hit and purchaser and consumer behavior changed overnight. There was no data in the data set that explained that consumer behavior. And so, what you saw is a lot of these things like supply chain issues, which are very heavily data-driven on a normal circumstance, there was nothing in that data that allowed those algorithms to optimize for the reality that we were seeing at that scale, right? Even if you look at advanced logistics companies, they know what to do when there's a hurricane coming or when there's been an earthquake or things like that. They have disaster scenarios.But nobody has ever done anything like this at the global scale, right? And so, what we saw was this hard reset that we're still feeling the repercussions of today. Yes, there were people who couldn't work and we had lockdowns and all that stuff, but we also have an effect from the impact of the way that we built the systems to work with the data that we need to shuffle around. And so, I think that there is value in being able to process these really, really large datasets, but I think that actually, there's also a lot of value in being able to solve smaller, simpler problems, right? Not everything is a big data problem, not everything requires a ton of data to solve.It's more about the mindset that you use to look at the data, to explore the data, and what you're doing with it. And I think the challenge here is that, you know, everyone wants to believe that they have a big data problem because it feels like you have to have a big data problem if you—Corey: All the cool kids are having this kind of problem.Emily: You have to have big data to sit at the grownup's table. And so, what's happened is we've optimized a lot of tools around solving big data problems and oftentimes, these tools are really poor at solving normal data problems. And there's a lot of money being spent in a lot of overkill engineering in the data space.Corey: On some level, it feels like there has been a dramatic misrepresentation of this. I had an article that went out last year where I called machine-learning selling pickaxes into a digital gold rush. And someone I know at AWS responded to that and probably the best way possible—she works over on their machine-learning group—she sent me a foam Minecraft pickaxe that now is hanging on my office wall. And that gets more commentary than anything, including the customized oil painting I have of Billy the Platypus fighting an AWS Billing Dragon. No, people want to talk about the Minecraft pickaxe.It's amazing. It's first, where is this creativity in any of the marketing that this department is putting out? But two it's clearly not accurate. And what it took for me to see that was a couple of things that I built myself. I built a Twitter thread client that would create Twitter threads, back when Twitter was a place that wasn't overrun by some of the worst people in the world and turned into BirdChan.But that was great. It would automatically do OCR on images that I uploaded, it would describe the image to you using Azure's Cognitive Vision API. And that was magic. And now I see things like ChatGPT, and that's magic. But you take a look at the way that the cloud companies have been describing the power of machine learning in AI, they wind up getting someone with a doctorate whose first language is math getting on stage for 45 minutes and just yelling at you in Star Trek technobabble to the point where you have no idea what the hell they're saying.And occasionally other data scientists say, “Yeah, I think he's just shining everyone on at this point. But yeah, okay.” It still becomes unclear. It takes seeing the value of it for it to finally click. People make fun of it, but the Hot Dog, Not A Hot Dog app is the kind of valuable breakthrough that suddenly makes this intangible thing very real for people.Emily: I think there's a lot of impressive stuff and ChatGPT is fantastically impressive. I actually used ChatGPT to write a letter to some German government agency to deal with some bureaucracy. It was amazing. It did it, was grammatically correct, it got me what I needed, and it saved me a ton of time. I think that these tools are really, really powerful.Now, the thing is, not every company needs to build its own ChatGPT. Maybe they need to integrate it, maybe there's an application for it somewhere in their landscape of product, in their landscape of services, in the landscape of their interim internal tooling. And I would be thrilled actually to see some of that be brought into reality in the next couple of years. But you also have to remember that ChatGPT is not something that came because we have, like, a really great breakthrough in AI last year or something like that. It stacked upon 40 years of research.We've gone through three new waves of neural networking in that time to get to this point, and it solves one class of problem, which is honestly a fairly narrow class of problem. And so, what I see is a lot of companies that have much more mundane problems, but where data can actually still really help them. Like how do you process Cambodian driver's licenses with OCR, right? These are the types of things that if you had a training data set that was every Cambodian person's driver's license for the last ten years, you're still not going to get the data volumes that even a day worth of Amazon's marketplace generates, right? And so, you need to be able to solve these problems still with data without resorting to the cudgel that is a big data solution, right?So, there's still a niche, a valuable niche, for solving problems with data without having to necessarily resort to, we have to load the entire internet into our stream and throw GPUs at it all day long and spend hundreds of—tens of millions of dollars in training. I don't know, maybe hundreds of millions; however much ChatGPT just raised. There's an in-between that I think is vastly underserved by what people are talking about these days.Corey: There is so much attention being given to this and it feels almost like there has been a concerted and defined effort to almost talk in circles and remove people from the humanity and the human consequences of what it is that they're doing. When I was younger, in my more reckless years, I was never much of a fan of the idea of government regulation. But now it has become abundantly clear that our industry, regardless of how you want to define industry, how—describe a society—cannot self-regulate when it comes to data that has the potential to ruin people's lives. I mean, I spent a fair bit of my time in my career working in financial services in a bunch of different ways. And at least in those jobs, it was only money.The scariest thing I ever dealt with, from a data perspective is when I did a brief stint at Grindr because that was the sort of problem where if that data gets out, people will die. And I have not had to think about things like that have that level of import before or since, for which I'm eternally grateful. “It's only money,” which is a weird thing for a guy who fixes cloud bills for a living to say. And if I say that in a client call, it's not going to go very well. But it's the truth. Money is one of those things that can be fixed. It can be addressed in due course. There are always opportunities there. Someone just been outed to their friends, family, and they feel their life is now in shambles around them, you can't unring that particular bell.Emily: Yeah. And in some countries, it can lead to imprisonment, or—Corey: It can lead to death sentences, yes. It's absolutely not acceptable.Emily: There's a lot to say about the ethics of where we are. And I think that as a lot of these high profile, you know, AI tools have come out over the last year or so, so you know, Stable Diffusion and ChatGPT and all of this stuff, there's been a lot of conversation that is sort of trying to put some counterbalance on what we're seeing. And I don't know that it's going to be successful. I think that, you know, I've been speaking about ethics and technology for a long time and I think that we need to mature and get to the next level of actually addressing the ethical problems in technology. Because it's so far beyond things like, “Oh, you know, if there's a biased training data set and therefore the algorithm is biased,” right?Everyone knows that by now, right? And the people who don't know that, don't care. We need to get much beyond where, you know, these conversations about ethics and technology are going because it's a manifold problem. We have issues with the people labeling this data are paid, you know, pennies per hour to deal with some of the most horrific content you've ever seen. I mean, I'm somebody who has immersed myself in a lot of horrific content for some of the work that I have done, and this is, you know, so far beyond what I've had to deal with in my life that I can't even imagine it. You couldn't pay me enough money to do it and we're paying people in developing nations, you know, a buck-thirty-five an hour to do this. I think—Corey: But you must understand, Emily, that given the standard of living where they are, that that is perfectly normal and we wouldn't want to distort local market dynamics. So, if they make a buck-fifty a day, we are going to be generous gods and pay them a whopping dollar-seventy a day, and now we feel good about ourselves. And no, it's not about exploitation. It's about raising up an emerging market. And other happy horseshit that lies people tell themselves.Emily: Yes, it is. Yes, it is. And we've built—you know, the industry has built its back on that. It's raised itself up on this type of labor. It's raised itself up on taking texts and images without permission of the creators. And, you know, there's—I'm not a lawyer and I'm not going to play one, but I do know that derivative use is something that at least under American law, is something that can be safely done. It would be a bad world if derivative use was not something that we had freely available, I think, and on the balance.But our laws, the thing is, our laws don't account for the scale. Our laws about things like fair use, derivative use, are for if you see a picture and you want to take your own interpretation, or if you see an image and you want to make a parody, right? It's a one-to-one thing. You can't make 5 million parody images based on somebody's art, yourself. These laws were never built for this scale.And so, I think that where AI is exploiting society is it's exploiting a set of ethics, a set of laws, and a set of morals that are built around a set of behavior that is designed around normal human interaction scales, you know, one person standing in front of a lecture hall or friends talking with each other or things like that. The world was not meant for a single person to be able to speak to hundreds of thousands of people or to manipulate hundreds of thousands of images per day. It's actually—I find it terrifying. Like, the fact that me, a normal person, has a Twitter following that, you know, if I wanted to, I can have 50 million impressions in a month. This is not a normal thing for a normal human being to have.And so, I think that as we build this technology, we have to also say, we're changing the landscape of human ethics by our ability to act at scale. And yes, you're right. Regulation is possibly one way that can help this, but I think that we also need to embed cultural values in how we're using the technology and how we're shaping our businesses to use the technology. It can be used responsibly. I mean, like I said, ChatGPT helped me with a visa issue, sending an email to the immigration office in Berlin. That's a fantastic thing. That's a net positive for me; hopefully, for humanity. I wasn't about to pay a lawyer to do it. But where's the balance, right? And it's a complex topic.Corey: It is. It absolutely is. There is one last topic that I would like to talk to you about that's a little less heavy. And I've got to be direct with you that I'm not trying to be unkind, but you've disappointed me. Because you mentioned to me at one point, when I asked how things were going in your AWS universe, you said, “Well, aside from the bank heist, reasonably well.”And I thought that you were blessed as with something I always look for, which is the gift of glorious metaphor. Unfortunately, as I said, you've disappointed me. It was not a metaphor; it was the literal truth. What the hell kind of bank heist could possibly affect an AWS account? This sounds like something out of a movie. Hit me with it.Emily: Yeah, you know, I think in the SRE world, we tell people to focus on the high probability, low impact things because that's where it's going to really hurt your business, and let the experts deal with the black swan events because they're pretty unlikely. You know, a normal business doesn't have to worry about terrorists breaking into the Google data center or a gang of thieves breaking into a bank vault. Apparently, that is something that I have to worry about because I have some data in my personal life that I needed to protect, like all other people. And I decided, like a reasonable and secure and smart human being who has a little bit of extra spending cash that I would do the safer thing and take my backup hard drive and my old phones and put them in a safety deposit box at an old private bank that has, you know, a vault that's behind the meter-and-a-half thick steel door and has two guards all the time, cameras everywhere. And I said, “What is the safest possible thing that you can do to store your backups?” Obviously, you put it in a secure storage location, right? And then, you know, I don't use my AWS account, my personal AWS account so much anymore. I have work accounts. I have test accounts—Corey: Oh, yeah. It's honestly the best way to have an AWS account is just having someone else having a payment instrument attached to it because otherwise oh God, you're on the hook for that yourself and nobody wants that.Emily: Absolutely. And you know, creating new email addresses for new trial accounts is really just a pain in the ass. So, you know, I have my phone, you know, from five years ago, sitting in this bank vault and I figured that was pretty secure. Until I got an email [laugh] from the Berlin Polizei saying, “There has been a break-in.” And I went and I looked at the news and apparently, a gang of thieves has pulled off the most epic heist in recent European history.This is barely in the news. Like, unless you speak German, you're probably not going to find any news about this. But a gang of thieves broke into this bank vault and broke open the safety deposit boxes. And it turns out that this vault was also the location where a luxury watch consigner had been storing his watches. So, they made off with some, like, tens of millions of dollars of luxury watches. And then also the phone that had my 2FA for my Amazon account. So, the total value, you know, potential theft of this was probably somewhere in the $500 million range if they set up a SageMaker instance on my account, perhaps.Corey: This episode is sponsored in part by Honeycomb. I'm not going to dance around the problem. Your. Engineers. Are. Burned. Out. They're tired from pagers waking them up at 2 am for something that could have waited until after their morning coffee. Ring Ring, Who's There? It's Nagios, the original call of duty! They're fed up with relying on two or three different “monitoring tools” that still require them to manually trudge through logs to decipher what might be wrong. Simply put, there's a better way. Observability tools like Honeycomb (and very little else becau se they do admittedly set the bar) show you the patterns and outliers of how users experience your code in complex and unpredictable environments so you can spend less time firefighting and more time innovating. It's great for your business, great for your engineers, and, most importantly, great for your customers. Try FREE today at honeycomb.io/screaminginthecloud. That's honeycomb.io/screaminginthecloud.Corey: The really annoying part that you are going to kick yourself on about this—and I'm not kidding—is, I've looked up the news articles on this event and it happened, something like two or three days after AWS put out the best release of last years, or any other re:Invent—past, present, future—which is finally allowing multiple MFA devices on root accounts. So finally, we can stop having safes with these things or you can have two devices or you can have multiple people in Covid times out of remote sides of different parts of the world and still get into the thing. But until then, nope. It's either no MFA or you have to store it somewhere ridiculous like that and access becomes a freaking problem in the event that the device is lost, or in this case stolen.Emily: [laugh]. I will just beg the thieves, if you're out there, if you're secretly actually a bunch of cloud engineers who needed to break into a luxury watch consignment storage vault so that you can pay your cloud bills, please have mercy on my poor AWS account. But also I'll tell you that the credit card attached to it is expired so you won't have any luck.Corey: Yeah. Really sad part. Despite having the unexpired credit card, it just means that the charge won't go through. They're still going to hold you responsible for it. It's the worst advice I see people—Emily: [laugh].Corey: Well, intentioned—giving each other on places like Reddit where the other children hang out. And it's, “Oh, just use a prepaid gift card so it can only charge you so much.” It's yeah, and then you get exploited like someone recently was and start accruing $60,000 a day in Lambda charges on an otherwise idle account and Amazon will come after you with a straight face after a week. And, like, “Yes, we'd like our $360,000, please.”Emily: Yes.Corey: “We tried to charge the credit card and wouldn't you know, it expired. Could you get on that please? We'd like our money faster if you wouldn't mind.” And then you wind up in absolute hell. Now, credit where due, they in every case I am aware of that is not looking like fraud's close cousin, they have made it right, on some level. But it takes three weeks of back and forth and interminable waiting.And you're sitting there freaking out, especially if you're someone who does not have a spare half-million dollars sitting around. Imagine who—“You sound poor. Have you tried not being that?” And I'm firmly convinced that it a matter of time until someone does something truly tragic because they don't understand that it takes forever, but it will go away. And from my perspective, there's no bigger problem that AWS needs to fix than surprise lifelong earnings bills to some poor freaking student who is just trying to stand up a website as part of a class.Emily: All of the clouds have these missing stairs in them. And it's really easy because they make it—one of the things that a lot of the cloud providers do is they make it really easy for you to spin up things to test them. And they make it really, really hard to find where it is to shut it all down. The data science is awful at this. As a data scientist, I work with a lot of data science tools, and every cloud has, like, the spin up your magical data science computing environment so that your data scientist can, like, bang on the data with you know, high-performance compute for a while.And you know, it's one click of a button and you type in a couple of na—you know, a couple of things name, your service or whatever, name your resource. You click a couple buttons and you spin it up, but behind the scenes, it's setting up a Kubernetes cluster and it's setting up some storage bucket and it's setting up some data pipelines and it's setting up some monitoring stuff and it's setting up a VM in order to run all of this stuff. And the next thing that you know, you're burning 100, 200 euro a day, just to, like, to figure out if you can load a CSV into pandas using a Jupyter Notebook. And you're like—when you try to shut it all down, you can't. It's you have to figure, oh, there is a networking thing set up. Well, nobody told me there's a networking thing set up. You know? How do I delete that?Corey: You didn't say please, so here you go. Without for me, it's not even the giant bill going from $4 a month in S3 charges to half a million bucks because that is pretty obvious from the outside just what the hell's been happening. It's the little stuff. I am still—since last summer—waiting for a refund on $260 of ‘because we said so' SageMaker credits because of a change of their billing system, for a 45-minute experiment I had done eight months before that.Emily: Yep.Corey: Wild stuff. Wild stuff. And I have no tolerance for people saying, “Oh, you should just read the pricing page and understand it better.” Yeah, listen, jackhole. I do this for a living. If I can fall victim to it, anyone can. I promise. It is not that I don't know how the billing system works and what to do to avoid unexpected charges.And I'm just luck—because if I hadn't caught it with my systems three days into the month, it would have been a $2,000 surprise. And yeah, I run a company. I can live with that. I wouldn't be happy, but whatever. It is immaterial compared to, you know, payroll.Emily: I think it's kind of a rite of passage, you know, to have the $150 surprise Redshift bill at the end of the month from your personal test account. And it's sad, you know? I think that there's so much better that they can do and that they should do. Sort of as a tangent, one of the challenges that I see in the data space is that it's so hard to break into data because the tooling is so complex and it requires so much extra knowledge, right? If you want to become a software developer, you can develop a microservice on your machine, you can build a web app on your machine, you can set up Ruby on Rails, or Flask, or you know, .NET, or whatever you want. And you can do all of that locally.And you can learn everything you need to know about React, or Terraform, or whatever, running locally. You can't do that with data stuff. You can't do that with BigQuery. You can't do that with Redshift. The only way that you can learn this stuff is if you have an account with that setup and you're paying the money to execute on it. And that makes it a really high barrier for entry for anyone to get into this space. It makes it really hard to learn. Because if you want to learn anything by doing, like many of us in the industry have done, it's going to cost you a ton of money just to [BLEEP] around and find out.Corey: Yes. And no one likes the find out part of those stories.Emily: Nobody likes to find out when it comes to your bill.Corey: And to tie it back to the data story of it, it is clearly some form of batch processing because it tries to be an eight-hour consistency model. Yeah, I assume for everything, it's 72. But what that means is that you are significantly far removed from doing a thing and finding out what that thing costs. And that's the direct charges. There's always the oh, I'm going to set things up and it isn't going to screw you over on the bill. You're just planting a beautiful landmine you're going to stumble blindly into in three months when you do something else and didn't realize what that means.And the worst part is it feels victim-blamey. I mean, this is my pro—I guess this is one of the reasons I guess I'm so down on data, even now. It's because I contextualize it in a sense of the AWS bill. No one's happy dealing with that. You ever met a happy accountant? You have not.Emily: Nope. Nope [laugh]. Especially when it comes to clouds stuff.Corey: Oh yeah.Emily: Especially these days, when we're all looking to save energy, save money in the cloud.Corey: Ideally, save the planet. Sustainability and saving money align on the axis of ‘turn that shit off.' It's great. We can hope for a brighter tomorrow.Emily: Yep.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where can they find you? Apparently filing police reports after bank heists, which you know, it's a great place to meet people.Emily: Yeah. You know, the largest criminal act in Berlin is certainly a place you want to go to get your cloud advice. You can find me, I have a website. It's my name, emilygorcenski.com.You can find me on Twitter, but I don't really post there anymore. And I'm on Mastodon at some place because Mastodon is weird and kind of a mess. But if you search me, I'm really not that hard to find. My name is harder to spell, but you'll see it in the podcast description.Corey: And we will, of course, put links to all of this in the show notes. Thank you so much for your time. I really appreciate it.Emily: Thank you for having me.Corey: Emily Gorcenski, Data and AI Service Line Lead at ThoughtWorks. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insipid, insulting comment, talking about why data doesn't actually matter at all. And then the comment will disappear into the ether because your podcast platform of choice feels the same way about your crappy comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About This EpisodeShreya Shankar is a computer scientist, PhD student in databases at UC Berkeley, and co-author of "Operationalizing Machine Learning: An Interview Study", an ethnographic interview study with 18 machine learning engineers across a variety of industries on their experience deploying and maintaining ML pipelines in production.Shreya explains the high-level findings of "Operationalizing Machine Learning"; variables that indicate a successful deployment (velocity, validation, and versioning), common pain points, and a grouping of the MLOps tool stack into four layers. Shreya and Lukas also discuss examples of data challenges in production, Jupyter Notebooks, and reproducibility.Show notes (transcript and links): http://wandb.me/gd-shreya---

Chris and Elecia talk about photons, comets, patterns, other flying objects, and cameras. Chris uses PixInsight for processing  and has an Ioptron Sky Tracker. Apologies to our southern hemisphere listeners because Polaris is not visible there. There are (of course) other ways to align and even in the northern hemisphere more modern trackers don't necessarily need Polaris. Star Exterminator: who cares what it does it has an awesome name. Though it does what it says (on photos, no real stars were harmed in the making of this podcast). Jupyter Notebooks on a Circuit Python board. Elecia's Yoshimura sine pattern generating Python colab. Also, Rigidly foldable origami gadgets and tessellations is an excellent article about Miura-ori and other rigidly foldable patterns. You can see her patterns over on Instagram. (You can see some of Chris' photos on his Instagram.) Transcript

This episode features an interview with Denise Gosnell, Principal Product Manager at Amazon Web Services. At AWS, Denise leads product and strategy for Amazon Neptune, a fully managed graph database service. Her career centers on her passion for examining, applying, and advocating for the applications of graph data. Denise has also authored, patented, and spoken on graph theory, algorithms, databases, and applications across all industry verticals.In this episode, Sam sits down with Denise to discuss graph initiatives, the future of developer models, and what Denise learned from hiking the Appalachian Trail.-------------------“We just open sourced something called graph-explorer, which is something for the community by the community, Apache 2.0 license. graph-explorer is a low-code visualization tool. But, the best part about it is that it works for JanusGraph, it works for Blazegraph, it works for all of these graph models that we've talked about, because we've got this divided graph community, but it was written to work with all graphs. [...] Today it's all, ‘Here's your Lego blocks and build one on your own. If you want to go ahead and fork Jupyter Notebook and figure out a way to get that D3 force-directed graph way out to pop up, have fun.' It's the first time that we've had a unified way across graph vendors and graph implementations to have a way to visualize your graph data in one tool that's open source.” – Denise Gosnell-------------------Episode Timestamps:(01:17): What open source data means to Denise(04:27): How Denise got interested in computer science(08:39): Denise's work on graph initiatives(14:30): How Denise's work at LDBC relates to SQL standards(23:43): The future of developer models(29:43): One question Denise wishes to be asked(34:05): Denise's advice for graph practitioners(37:37): Executive producer, Audra Montenegro's backstage takeaways-------------------Links:LinkedIn - Connect with DeniseThe Practitioner's Guide to Graph Data

Neo4j in 100 seconds by Fireship:  https://www.youtube.com/watch?v=T6L9EoBy8ZkUnder the Hood Series w/ Chris Gioran: https://www.youtube.com/playlist?list=PL9Hl4pk2FsvWn1M0HMOta_9YpN930Ai8RNodes 2022 Keynote w/ Nicholas Christakis: https://www.youtube.com/watch?v=hCMv4UJo--4How to get all connected nodes and relationships of a particular node: https://community.neo4j.com/t5/neo4j-graph-platform/how-to-get-all-the-connected-nodes-and-relationship-of-a/td-p/28464Gartner Magic Quadrant Cloud DBMS 2022: https://neo4j.com/blog/neo4j-recognized-for-the-first-time-in-the-2022-gartner-magic-quadrant-for-cloud-database-management-systems/Workspace in AuraDB from Nodes 2022 w/ John Stegeman: https://www.youtube.com/watch?v=rPnOuZ_YZj8&list=PL9Hl4pk2FsvWPcphew_GbLjCWvMpmh4mV&index=43GraphQL Quickstart from GraphConnect 2022 w/ Max Andersson: https://www.youtube.com/watch?v=saerwmnZolQTop 10 Cypher Tuning Tips & Tricks from GraphConnect 2022 w/ Michael Hunger: https://www.youtube.com/watch?v=DAlWoamQ41QNeo4j Driver Best Practices from GraphConnect 2022 w/ David Allen: https://www.youtube.com/watch?v=WV_xe2OF7bkHow to import JSON using Cypher and APOC from GraphConnect 2022 w/ Eric Monk: https://youtu.be/PshmP_fXBRsDiscovering Aura Free with Fun Datasets w/ Michael Hunger & Alex Erdl: https://www.youtube.com/playlist?list=PL9Hl4pk2FsvVZaoIpfsfpdzEXxyUJlAYwExplore Graphs Visually with Jupyter Notebooks from Nodes 2022 w/ Sebastian Muller: https://www.youtube.com/watch?v=M_PbbMVg4ME&list=PL9Hl4pk2FsvWPcphew_GbLjCWvMpmh4mV&index=64Maintain Companion Plant Knowledge Graph in Google Sheets + Neo4j by Sixing Huang: https://towardsdatascience.com/maintain-a-companion-plant-knowledge-graph-in-google-sheets-and-neo4j-4142c0a5065bError when trying to invoke Cypher procedure apoc.spatial.geocodeOnce: https://community.neo4j.com/t5/neo4j-graph-platform/error-when-trying-to-invoke-cypher-procedure-apoc-spatial/td-p/36158GDS in Python to Improve ML Models by Tomaz Bratanic: https://neo4j.com/developer-blog/using-neo4j-graph-data-science-in-python-to-improve-machine-learning-models/A Universe of Knowledge Graphs w/ Dr. Maya Natarajan & Dr. Jesus Barrasa: https://www.youtube.com/watch?v=Ei-pYtYS6UYNeo4j VS Code Extension by Adam Cowley: https://neo4j.com/developer-blog/run-cypher-without-leaving-your-ide-with-neo4j-vscode-extension/Cymple Library: https://github.com/Accenture/CymplePypher Library: https://github.com/emehrkay/PypherFlat Graph GitHub action: https://github.com/marketplace/actions/flat-graph

I had fun chatting with Aarthi and Sriram.We discuss what it takes to be successful in technology, what Sriram would say if Elon tapped him to be the next CEO of Twitter, why more married couples don't start businesses together, and how Aarthi hires and finds 10x engineers.Aarthi Ramamurthy and Sriram Krishnan are the hosts of The Good Times Show. They have had leading roles in several technology companies from Meta to Twitter to Netflix and have been founders and investors. Sriram is currently a general partner at a16z crypto and Aarthi is an angel investor.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Timestamps(00:00:00) - Intro(00:01:19) - Married Couples Co-founding Businesses(00:09:53) - 10x Engineers(00:16:00) - 15 Minute Meetings(00:22:57) - a16z's Edge?(00:26:42) - Future of Twitter(00:30:58) - Is Big Tech Overstaffed?(00:38:37) - Next CEO of Twitter?(00:43:13) - Why Don't More Venture Capitalists Become Founders?(00:47:32) - Role of Boards(00:52:03) - Failing Upwards(00:56:00) - Underrated CEOs(01:02:18) - Founder Education(01:06:27) - What TV Show Would Sriram Make?(01:10:14) - Undervalued Founder ArchetypesTranscriptThis transcript was autogenerated and thus may contain errors.[00:00:00] Aarthi: it's refreshing to have Elon come in and say, we are gonna work really hard. We are gonna be really hardcore about how we build things.[00:00:05] Dwarkesh: Let's say Elon and says Tomorrow, Sriram, would you be down to be the [00:00:08] Sriram: CEO of Twitter Absolutely not. Absolutely not. But I am married to someone. We [00:00:12] Aarthi: used to do overnights at Microsoft. Like we'd just sleep under our desk,, until the janitor would just , poke us out of there , I really need to vacuum your cubicle. Like, get out of here. There's such joy in , Finding those moments where you work hard and you're feeling really good about it. [00:00:25] Sriram: You'd be amazed at how many times Aarthi and I would have a conversation where be, oh, this algorithm thing.I remember designing it, and now we are on the other side We want to invest in something , where we think the team and the company is going to win and if they do win, there's huge value to be unlocked. [00:00:40] Dwarkesh: Okay. Today I have the, uh, good pleasure to have Arty and Sriram on the podcast and I'm really excited about this.So you guys have your own show, the Arty Andre Good Time show. Um, you guys have had some of the top people in tech and entertainment on Elon Musk, mark Zuckerberg, Andrew Yang, and you guys are both former founders. Advisors, investors, uh, general partner at Anderson Horowitz, and you're an angel investor and an advisor now.Um, so yeah, there's so much to talk about. Um, obviously there's also the, uh, recent news about your, uh, your involvement on, uh, twitter.com. Yeah, yeah. Let's get started. [00:01:19] Married Couples Starting Businesses[00:01:19] Dwarkesh: My first question, you guys are married, of course. People talk about getting a co-founder as finding a spouse, and I'm curious why it's not the case that given this relationship why more married people don't form tech startups.Is, does that already happen, [00:01:35] Aarthi: or, um, I actually am now starting to see a fair bit of it. Uhhuh, . Um, I, I do agree that wasn't a norm before. Um, I think, uh, I, I think I remember asking, uh, pg p the same thing when I went through yc, and I think he kind of pointed to him and Jessica like, you know, YC was their startup , and so, you know, there were even pride.There are a lot of husband and wife, uh, companies. Over the last like decade or so. So I'm definitely seeing that more mainstream. But yeah, you're right, it hasn't been the norm before. Yeah, the, the good time show is our project. It's [00:02:09] Sriram: our startup. Very, I mean, there are some good historical examples. Cisco, for example, uh, came from, uh, uh, husband, wife as a few other examples.I think, you know, on, on the, in, on the pro side, uh, you know, being co-founders, uh, you need trust. You need to really know each other. Uh, you, you go through a lot of like heavy emotional burdens together. And there's probably, and if you, you're for the spouse, hopefully you probably have a lot of chemistry and understanding, and that should help.On the con side, I think one is you, you're prob you know, you, you're gonna show up at work, you know, and startups are really hard, really intense. And you come home and both of you are gonna the exact same wavelength, the exact same time, going through the exact same highs and lows as opposed to two people, two different jobs have maybe differing highs and lows.So that's really hard. Uh, the second part of it is, uh, in a lot of. Work situations, it may just be more challenging where people are like, well, like, you know, person X said this person Y said this, what do I do? Uh, and if you need to fire somebody or you know, something weird happens corporate in a corporate manner, that may also be really hard.Uh, but having said that, you know, uh, [00:03:13] Aarthi: you know, yeah, no, I think both of those are like kind of overblown , like, you know, I think the reason why, um, you know, you're generally, they say you need to have you, it's good to have co-founders is so that you can kind of like write the emotional wave in a complimentary fashion.Uh, and you know, if one person's like really depressed about something, the other person can like pull them out of it and have a more rational viewpoint. I feel like in marriages it works even better. So I feel like to your first point, They know each other really well. You're, you're, you are going to bring your work to home.There is no separation between work and home as far as a startup is concerned. So why not do it together? Oh, [00:03:51] Sriram: well, I think there's one problem, uh, which is, uh, we are kind of unique because we've been together for over 21 years now, and we start for, we've been before, uh, let's not. Wow. There's gonna be some fact checking 19 on this video.99. Close enough. Close enough, right? Like close enough. He wishes he was 21. Oh, right, right, right. Gosh, feels like 21. We have do some, um, [00:04:15] Aarthi: editing on this video. No, no, no. I think 20 years of virtually knowing, 19 years of in-person. [00:04:20] Sriram: There we go. Right. Uh, fact check accurate. Um, ex experts agree. But, um, you know, but when you first met, we, we originally, even before we dating, we were like, Hey, we wanna do a company together.And we bonded over technology, like our first conversation on Yahoo Messenger talking about all these founders and how we wanted to be like them. And we actually then worked together pretty briefly when you were in Microsoft. Uh, before we actually started dating. We were on these sort of talent teams and we kind of met each of the word context.I think a lot of. You know, one is they have never worked together. Um, and so being in work situations, everything from how you run a meeting to how you disagree, uh, you know, uh, is just going to be different. And I think that's gonna be a learning curve for a lot of couples who be like, Hey, it's one thing to have a strong, stable relationship at home.It'll be a different thing to, you know, be in a meeting and you're disagreeing art's meetings very differently from I do. She obsesses over metrics. I'm like, ah, it's close enough. It's fine. , uh, it's close enough. It's fine. as e uh, here already. But, uh, so I do think there's a learning curve, a couples who is like, oh, working together is different than, you know, raising your family and being together.I mean, obviously gives you a strong foundation, but it's not the same thing. Have you guys [00:05:25] Dwarkesh: considered starting a company or a venture together at some point? [00:05:28] Aarthi: Yeah. Um, we've, uh, we've always wanted to do a project together. I don't know if it's a, a startup or a company or a venture. You have done a project together,Yeah, exactly. I think, uh, almost to today. Two years ago we started the Good Time Show, um, and we started at, uh, live Audio on Clubhouse. And, you know, we recently moved it onto video on YouTube. And, um, it's, it's been really fun because now I get to see like, it, it's neither of our full-time jobs, uh, but we spend enough, um, just cycles thinking through what we wanna do with it and what, uh, how to have good conversations and how to make it useful for our audience.So that's our [00:06:06] Sriram: project together. Yep. And we treat it like a, with the intellectual heft of a startup, which is, uh, we look at the metrics, uh, and we are like, oh, this is a good week. The metrics are up into the right and, you know, how do we, you know, what is working for our audience? You know, what do we do to get great guests?What do we do to [00:06:21] Aarthi: get, yeah, we just did our first, uh, in-person meetup, uh, for listeners of the podcast in Chennai. It was great. We had like over a hundred people who showed up. And it was also like, you know, typical startup style, like meet your customers and we could like go talk to these people in person and figure out like what do they like about it?Which episodes do they really enjoy? And it's one thing to see YouTube comments, it's another to like actually in person engage with people. So I think, you know, we started it purely accidentally. We didn't really expect it to be like the show that we are, we are in right now, but we really happy. It's, it's kind of turned out the way it has.[00:06:59] Sriram: Absolutely. And, and it also kind of helps me scratch an edge, which is, uh, you know, building something, you know, keeps you close to the ground. So being able to actually do the thing yourself as opposed to maybe tell someone else, telling you how to do the, so for example, it, it being video editing or audio or how thumbnails, thumbnails or, uh, just the mechanics of, you know, uh, how to build anything.So, uh, I, I dot think it's important. Roll up your sleeves metaphorically and get your hands dirty and know things. And this really helped us understand the world of creators and content. Uh, and it's fun and [00:07:31] Aarthi: go talk to other creators. Uh, like I think when we started out this thing on YouTube, I think I remember Shram just reached out to like so many creators being like, I wanna understand how it works for you.Like, what do you do? And these are people who like, who are so accomplished, who are so successful, and they do this for a living. And we clearly don. And so, uh, just to go learn from these experts. It's, it's kind of nice, like to be a student again and to just learn, uh, a new industry all over again and figure out how to actually be a creator on this platform.Well, you know [00:08:01] Dwarkesh: what's really interesting is both of you have been, uh, executives and led product in social media companies. Yeah. And so you are, you designed the products, these creators, their music, and now on the other end, you guys are building [00:08:12] Sriram: the, oh, I have a great phrase for it, right? Like, somebody, every once in a while somebody would be like, Hey, you know what, uh, you folks are on the leadership team of some of these companies.Why don't you have hundreds of millions of followers? Right? And I would go, Hey, look, it's not like every economist is a billionaire, , uh, uh, you know, it doesn't work that way. Uh, but during that is a parallel, which, which is, uh, you'd be amazed at how many times Aarthi and I would have a conversation where be, oh, this algorithm thing.I remember designing it, or I was in the meeting when this thing happened, and now we are on the other side, which is like, Hey, you might be the economist who told somebody to implement a fiscal policy. And now we are like, oh, okay, how do I actually go do this and create values and how? Anyway, how do we do exactly.Create an audience and go build something interesting. So there is definitely some irony to it, uh, where, uh, but I think hopefully it does give us some level of insight where, uh, we have seen, you know, enough of like what actually works on social media, which is how do you build a connection with your audience?Uh, how do you build, uh, content? How do you actually do it on a regular, uh, teams? I think [00:09:07] Aarthi: the biggest difference is we don't see the algorithm as a bra, as a black box. I think we kind of see it as like when the, with the metrics, we are able to, one, have empathy for the teams building this. And two, I think, uh, we kind of know there's no big magic bullet.Like I think a lot of this is about showing up, being really consistent, um, you know, being able to like put out some really interesting content that people actually want to, and you know, I think a lot of people forget about that part of it and kind of focus. If you did this one thing, your distribution goes up a lot and here's this like, other like secret hack and you know Sure.Like those are like really short term stuff, but really in the long term, the magic is to just like keep at it. Yeah. And, uh, put out really, really good content. [00:09:48] Sriram: Yeah. Yeah. And yeah, absolutely. Yeah. Yeah. Um, that's good to hear. . [00:09:53] 10x Engineers[00:09:53] Dwarkesh: Um, so you've both, um, led teams that have, you know, dozens or even hundreds of people.Um, how easy is it for you to tell who the 10 X engineers are? Is it something that you as managers and executives can tell easily or [00:10:06] Sriram: no? Uh, absolutely. I think you can tell this very easily or repeat of time and it doesn't, I think a couple of ways. One is, uh, Uh, before, let's say before you work with someone, um, 10 x people just don't suddenly start becoming 10 x.They usually have a history of becoming 10 x, uh, of, you know, being really good at what they do. And you can, you know, the cliche line is you can sort of connect the dots. Uh, you start seeing achievements pile up and achievements could be anything. It could be a bunch of projects. It could be a bunch of GitHub code commits.It could be some amazing writing on ck, but whatever it is, like somebody just doesn't show up and become a 10 x person, they probably have a track record of already doing it. The second part of it is, I've seen this is multiple people, uh, who are not named so that they don't get hired from the companies actually want them to be in, or I can then hire them in the future is, uh, you know, they will make incredibly rapid progress very quickly.So, uh, I have a couple of examples and almost independently, I know it's independently, so I have a couple of. Um, and I actually, and name both, right? Like, so one is, uh, this guy named, uh, Vijay Raji, uh, who, uh, was probably one of Facebook's best engineers. He's now the CEO of a company called Stats. And, um, he was probably my first exposure to the real TenX engineer.And I remembered this because, uh, you know, at the time I was. Kind of in my twenties, I had just joined Facebook. I was working on ads, and he basically built a large part of Facebook's ad system over the weekend. And what he would do is he would just go, and then he con he [00:11:24] Aarthi: continued to do that with Facebook marketplace.Yeah. Like he's done this like over and over and over [00:11:28] Sriram: again. . Yeah. And, and it's not that, you know, there's one burst of genius. It's just this consistent stream of every day that's a code checkin stuff is working. New demo somebody, he sent out a new bill or something working. And so before like a week or two, you just like a, you know, you running against Usain Bolt and he's kind of running laps around you.He's so far ahead of everyone else and you're like, oh, this guy is definitely ahead. Uh, the second story I have is, uh, of, uh, John Carmack, uh, you know, who's legend and I never worked with him in, uh, directly with, you know, hopefully someday I can fix. But, uh, somebody told me a story about him. Which is, uh, that the person told me story was like, I never thought a individual could replace the output of a hundred percent team until I saw John.And there's a great story where, um, you know, and so John was the most senior level at Facebook and from a hr, you know, employment insecurity perspective for an individual contributor, and it at, at that level, at Facebook, uh, for folks who kind of work in these big tech companies, it is the most, the highest tier of accomplishment in getting a year in a performance review is something called xcs Expectations, or, sorry, redefines, right?Which basically means like, you have redefined what it means for somebody to perform in this level, right? Like, it's like somebody, you know, like somebody on a four minute mile, I'll be running a two minute mile or whatever, right? You're like, oh, and, and it is incredibly hard sometimes. You doing, and this guy John gets it three years in a row, right?And so there's this leadership team of all the, you know, the really most important people on Facebook. And they're like, well, we should really promote John, right? Like, because he's done this three years in a row, he's changing the industry. Three years in a row and then they realized, oh wait, there is no level to promote him to Nick be CEOWell, maybe I don't think he wanted to. And so, uh, the story I heard, and I dunno, it's true, but I like to believe it's true, is they invented a level which still now only John Carmack has gotten. Right. And, um, and I think, you know, it's his level of productivity, uh, his, uh, intellect, uh, and the consistency over time and mu and you know, if you talk to anybody, Facebook work with him, he's like, oh, he replaced hundred people, teams all by themselves and maybe was better than a hundred percent team just because he had a consistency of vision, clarity, and activity.So those are [00:13:32] Aarthi: the two stories I've also noticed. I think, uh, actually sheam, I think our first kind of exposure to 10 x engineer was actually Barry born, uh, from Microsoft. So Barry, um, uh, basically wrote pretty much all the emulation engines and emulation systems that we all use, uh, and uh, just prolific, uh, and I think in addition to what Fred had said with like qualities and tenets, Um, the, I've generally seen these folks to also be like low ego and kind of almost have this like responsibility to, um, mentor coach other people.Uh, and Barry kind of like took us under his wing and he would do these like Tuesday lunches with us, where we would just ask like, you know, we were like fresh out of college and we just ask these like really dumb questions on, you know, um, scaling things and how do you build stuff. And I was working on, uh, run times and loaders and compilers and stuff.And so he would just take the time to just answer our questions and just be there and be really like, nice about it. I remember when you moved to Redmond, he would just like spend a weekend just like, oh yeah. Driving you about and just doing things like that, but very low ego and within their teams and their art, they're just considered to be legends.Yes. Like, you know, everybody would be like, oh, Barry Bond. Yeah, of course. [00:14:47] Sriram: Yeah. It, I can't emphasize enough the consistency part of it. Um, you know, with Barry. Or I gotta briefly work with Dave Cutler, who's kind of the father of modern operating systems, uh, is every day you're on this email li list at the time, which would show you check-ins as they happen.They would have something every single day, um, every day, and it'll be tangible and meaty and you know, and you just get a sense that this person is not the same as everybody else. Um, by the, this couple of people I can actually point to who haven't worked with, uh, but I follow on YouTube or streaming. Uh, one is, uh, Andrea Ling who builds Serenity Os we had a great episode with him.Oh, the other is George Hart's, uh, geo Hart. And I urge people, if you haven't, I haven't worked with either of them, uh, but if I urge which to kinda watch their streams, right? Because, uh, you go like, well, how does the anti killing build a web browser on an operating system? Which he builds by himself in such a sharp period of time and he watches stream and he's not doing some magical new, you know, bit flipping sorting algorithm anybody has, nobody has seen before.He's just doing everything you would do, but. Five bits of speed. I, yep, exactly. [00:15:48] Dwarkesh: I I'm a big fan of the George Hot Streams and Yeah, that's exactly what, you know, it's like yeah, you, he's also curling requests and he is also, you know, you know, spinning up an experiment in a Jupyter Notebook, but yeah, just doing it [00:15:58] Aarthi: away way faster, way efficiently.Yeah. [00:16:00] 15 Minute Meetings[00:16:00] Dwarkesh: Yeah. That's really interesting. Um, so ar Arthur, I'm, you've gone through Y Combinator and famously they have that 15 minute interview Yes. Where they try to grok what your business is and what your potential is. Yeah, yeah. But just generally, it seems like in Silicon Valley you guys have, make a lot of decisions in terms of investing or other kinds of things.You, in very short calls, you know. Yeah. . Yeah. And how much can you really, what is it that you're learning in these 15 minute calls when you're deciding, should I invest in this person? What is their potential? What is happening in that 15 minutes? [00:16:31] Aarthi: Um, I can speak about YC from the other side, from like, uh, being a founder pitching, right.I think, yes, there is a 15 minute interview, but before that, there is a whole YC application process. And, uh, I think even for the, for YC as, uh, this bunch of the set of investors, I'm sure they're looking for specific signals, but for me as a founder, the application process was so useful, um, because it really makes you think about what you're building.Why are you building this? Are you the right person to be building this? Who are the other people you should be hiring? And so, I mean, there are like few questions or like, one of my favorite questions is, um, how have you hacked a non-computer system to your advantage? Yeah. . And it kind of really makes you think about, huh, and you kind of noticed that many good founders have that pattern of like hacking other systems to their advantage.Um, and so to me, I think more than the interview itself, the process of like filling out the application form, doing that little video, all of that gives you better, um, it gives you the, the entire scope of your company in your head because it's really hard when you have this idea and you're kind of like noodling about with it and talking to a few people.You don't really know if this is a thing. To just like crystallize the whole vision in your head. I think, uh, that's on point. Yes. Um, the 15 minute interview for me, honestly, it was like kind of controversial because, uh, I went in that morning, I did the whole, you know, I, I had basically stayed at the previous night, uh, building out this website and, uh, that morning I showed up and I had my laptop open.I'm like really eager to like tell them what you're building and I keep getting cut off and I realize much later that that's kind of my design. Yeah. And you just like cut off all the time. Be like, why would anybody use this? And you start to answer and be like, oh, but I, I don't agree with that. And there's just like, and it, it's like part of it is like, makes you upset, but part of it is also like, it makes you think how to compress all that information in a really short amount of time and tell them.Um, and so that interview happens, I feel really bummed out because I kind of had this website I wanted to show them. So while walking out the door, I remember just showing Gary, Dan, um, the website and he like kind of like. Scrolls it a little bit, and he is like, this is really beautifully done. And I was like, thank you.I've been wanting to show you this for 15 minutes. Um, and I, I mentioned it to Gary recently and he laughed about it. And then, uh, I didn't get selected in that timeframe. They gave me a call and they said, come back again in the evening and we are going to do round two because we are not sure. Yeah. And so the second interview there was PG and Jessica and they both were sitting there and they were just grueling me.It was a slightly longer interview and PG was like, I don't think this is gonna work. And I'm like, how can you say that? I think this market's really big. And I'm just like getting really upset because I've been waiting this whole day to like get to this point. And he's just being like cynical and negative.And then at some point he starts smiling at Jessica and I'm like, oh, okay. They're just like baiting me to figure it out. And so that was my process. And I, by the evening, I remember Shera was working at. I remember driving down from Mountain View to Facebook and Sheam took me to the Sweet Stop. Oh yeah.Which is like their, you know, Facebook has this like, fancy, uh, sweet store, like the ice cream store. I [00:19:37] Sriram: think they had a lot more perks over the years, but that was very fancy back then. [00:19:40] Aarthi: So I had like two scoops of ice cream in each hand in, and, uh, the phone rang and I was like, oh, hold onto this. And I grabbed it and I, and you know, I think it was Michael Sibu or I don't know who, but somebody called me and said, you're through.So that was kind of my process. So even though there was only 15 minutes, mine was actually much longer after. But even before the, the application process was like much more detailed. So it sounds [00:20:01] Dwarkesh: like the 15 minutes it's really there. Like, can they rattle you? Can they, can they [00:20:06] Aarthi: you and how do you react?Yeah, yeah, yeah. Um, I also think they look for how sex you can be in explaining what the problem is. They do talk to hundreds of companies. It is a lot. And so I think, can you compress a lot of it and convince, if you can convince these folks here in three months or four months time, how are you going to do demo day and convince a whole room full of investors?[00:20:27] Sriram: Yeah. Yeah. For, I think it's a bit different for us, uh, on the VC side, uh, because two things. One, number one is, uh, the day, you know, so much of it is having a prepared mind before you go into the meeting. And, for example, if you're meeting a. very early. Are we investing before having met every single other person who's working in this space, who has ideas in the space.So you generally know what's going on, you know, what the kind of technologies are or go to market approaches are. You've probably done a bunch of homework already. It's usually, uh, it does happen where you meet somebody totally cold and uh, you really want to invest, but most often you've probably done some homework at least in this space, if not the actual company.Um, and so when you're in the meeting, I think you're trying to judge a couple of things. And these are obviously kind of stolen from Christ Dixon and others. Um, one is their ability to kind of go walk you through their idea, ma. And so very simply, um, you know, the idea MAs is, uh, and I think say the biology of Christen came with this, the idea that, hey, um, uh, How you got to the idea for your company really matters because you went and explored all the data ends, all the possibilities.You're managing around for years and years, and you've kind of come to the actual solution. And the way you can tell whether somebody's gone through the idea Mac, is when you ask 'em questions and they tell you about like five different things they've tried, did not work. And it, it's really hard to fake it.I mean, we, you maybe fake it for like one or two questions, but if you talk about like how we tried X, Y, and Z and they have like an opinion what of the opinions, if they've thought about it, you're like, okay, this person really studied the idea, ma. And that's very powerful. Uh, the second part of it is, uh, you know, Alex sample.Uh, uh, one of my partner says this, Yes, some this thing called the Manifestation Framework, which sounds like a self-help book on Amazon, but it's not, uh, uh uh, you know, but what if is, is like, you know, so many, so much of early stage startup founders is about the ability to manifest things. Uh, manifest capital, manifest the first hire, uh, manifest, uh, the first BD partnership.And, um, usually, you know, if you can't, if you don't have a Cigna sign of doing that, it's really hard to then after raising money, go and close this amazing hotshot engineer or salesperson or close this big partnership. And so in the meeting, right? If you can't convince us, right? And these are people, our day job is to give you money, right?Like, if I spent a year without giving anybody money, I'll probably get fired. If you can't, uh, if you can't convince us to give you money, right? If you wanna find probably a hard time to close this amazing engineer and get that person to come over from Facebook or close this amazing partnership against a competitor.And so that's kind of a judge of that. So it is never about the actual 60 Minutes where you're like, we, we are making up of a large part of makeup of mind is. That one or two conversations, but there's so much which goes in before and after that. Yeah, yeah. Speaking of [00:22:57] What is a16z's edge?[00:22:57] Dwarkesh: venture capital, um, I, I'm curious, so interest and Horowitz, and I guess why Combinator too?Um, but I mean, any other person who's investing in startups, they were started at a time when there were much less capital in the space, and today of course, there's been so much more capital pour into space. So how do these firms, like how does A 16 C continue to have edge? What is this edge? How can I sustain it [00:23:20] Sriram: given the fact that so much more capital is entered into the space?We show up on podcasts like the Lunar Society, , and so if you are watching this and you have a startup idea, Uh, come to us, right? Uh, no. Come, come to the Lunar society. . Well, yes. I mean, maybe so Trust me, you go in pat, you're gonna have a find, uh, a Thk pat right there. Uh, actually I, you think I joked, but there's a bit of truth.But no, I've had [00:23:40] Dwarkesh: like lu this [00:23:40] Aarthi: suddenly became very different [00:23:43] Sriram: conversation. I have had people, this is a totally ludicrous [00:23:46] Dwarkesh: idea, but I've had people like, give me that idea. And it's like, it sounds crazy to me because like, I don't know what, it's, what a company's gonna be successful, right? So, but I hasn't [00:23:55] Aarthi: become an investor.[00:23:57] Sriram: I honestly don't know. But it is something like what you're talking about Lu Society Fund one coming up, right? You heard it here first? Uh, uh, well, I think first of all, you know, I think there's something about the firm, uh, um, in terms of how it's set up philosophically and how it's set up, uh, kind of organizationally, uh, and our approach philosoph.The firm is an optimist, uh, uh, more than anything else. At the core of it, we are optimist. We are optimist about the future. We are optimist about the impact of founders on their, on the liberty to kind of impact that future. Uh, we are optimist at heart, right? Like I, I tell people like, you can't work at a six and z if you're not an optimist.That's at the heart of everything that we do. Um, and very tied to that is the idea that, you know, um, software is eating the world. It is, it's true. 10 years ago when Mark wrote that, peace is as true now, and we just see more and more of it, right? Like every week, you know, look at the week we are recording this.You know, everyone's been talking about chat, G p T, and like all the industries that can get shaped by chat, G P T. So our, our feature, our, our idea is that software is gonna go more and more. So, one way to look at this is, yes, a lot more capitalists enter the world, but there should be a lot more, right?Like, because these companies are gonna go bigger. They're gonna have bigger impacts on, uh, human lives and, and the world at large. So that's, uh, you know, uh, one school of thought, the other school of thought, uh, which I think you were asking about, say valuations, uh, et cetera. Is, uh, you know, um, again, one of my other partners, Jeff Jordan, uh, uh, always likes to tell people like, we don't go discount shopping, right?Our, the way we think about it is we want to, when we're investing in a market, We want to really map out the market, right? Uh, so for example, I work on crypto, uh, and, uh, you know, we, you know, if, if you are building something interesting in crypto and we haven't seen you, we haven't talked to you, that's a fail, that's a mess, right?We ideally want to see every single interesting founder company idea. And a category can be very loose. Crypto is really big. We usually segmented something else. Or if you look at enterprise infrastructure, you can take them into like, you know, AI or different layers and so on. But once you map out a category, you want to know everything.You wanna know every interesting person, every interesting founder you wanna be abreast of every technology change, every go to market hack, every single thing. You wanna know everything, right? And then, uh, the idea is that, uh, we would love to invest in, you know, the what is hopefully becomes the market.Set category, uh, or you know, somebody who's maybe close to the, the market leader. And our belief is that these categories will grow and, you know, they will capture huge value. Um, and as a whole, software is still can used to be undervalued by, uh, a, you know, the world. So, um, we, so, which is why, again, going back to what Jeff would say, he's like, we are not in the business of oh, we are getting a great deal, right?We, we are like, we want to invest in something which, where we think the team and the company and their approach is going to win in this space, and we want to help them win. And we think if they do win, there's a huge value to be unlocked. Yeah, I see. I see. Um, [00:26:42] Future of Twitter[00:26:42] Dwarkesh: let's talk about Twitter. [00:26:44] Sriram: Uh, . I need a drink. I need a drink.[00:26:48] Dwarkesh: um, Tell me, what is the future of Twitter? What is the app gonna look like in five years? You've, um, I mean obviously you've been involved with the Musk Venture recently, but, um, you've, you've had a senior position there. You were an executive there before a few years ago, and you've also been an executive at, uh, you've both been at Meta.So what [00:27:06] Sriram: is the future of Twitter? It's gonna be entertaining. Uh, uh, what is it El say the most entertaining outcome is the most, [00:27:12] Aarthi: uh, uh, like, best outcome is the most, uh, most likely outcome is the most entertaining outcome. [00:27:16] Sriram: Exactly right. So I think it's gonna be the most entertaining outcome. Um, I, I mean, I, I, I think a few things, uh, first of all, uh, ideally care about Twitter.Yeah. Uh, and all of my involvement, uh, you know, over the years, uh, uh, professionally, you know, uh, has, it's kind of. A lagging indicator to the value I got from the service person. I have met hundreds of people, uh, through Twitter. Uh, hundreds of people have reached out to me. Thousands. Exactly. Uh, and you know, I met Mark Andresen through Twitter.Uh, I met like, you know, uh, people are not very good friends of mine. We met through Twitter. We met at Twitter, right. There we go. Right. Uh, just [00:27:50] Aarthi: like incredible outsized impact. Yeah. Um, and I think it's really hard to understate that because, uh, right now it's kind of easy to get lost in the whole, you know, Elon, the previous management bio, like all of that.Outside of all of that, I think the thing I like to care about is, uh, focus on is the product and the product experience. And I think even with the product experience that we have today, which hasn't like, dramatically changed from for years now, um, it's still offering such outsized value for. If you can actually innovate and build really good product on top, I think it can, it can just be really, really good for humanity overall.And I don't even mean this in like a cheesy way. I really think Twitter as a tool could be just really, really effective and enormously good for everyone. Oh yeah. [00:28:35] Sriram: Twitter is I think, sort of methodically upstream of everything that happens in culture in uh, so many different ways. Like, um, you know, there was this, okay, I kinda eli some of the details, uh, but like a few years ago I remember there was this, uh, sort of this somewhat salacious, controversial story which happened in entertainment and uh, and I wasn't paying attention to, except that something caught my eye, which was that, uh, every story had the same two tweets.And these are not tweets from any famous person. It was just some, like, some, um, you know, somebody had some followers, but not a lot of, a lot of followers. And I. Why is this being quoted in every single story? Because it's not from the, you know, the person who was actually in the story or themselves. And it turned out that, uh, what had happened was, uh, you know, somebody wrote in the street, it had gone viral, um, it started trending on Twitter, um, and a bunch of people saw it.They started writing news stories about it. And by that afternoon it was now, you know, gone from a meme to now reality. And like in a lot of people entertainment say, kind of go respond to that. And I've seen this again and again, again, right? Uh, sports, politics, culture, et cetera. So Twitter is memetically upstream of so much of life.Uh, you know, one of my friends had said like, Twitter is more important than the real world. Uh, which I don't, I don't know about that, but, uh, you know, I do think it's, um, it has huge sort of, uh, culture shaping value. Yeah. I thing I think about Twitter is so much of. The network is very Lindy. So one of the things I'm sure from now is like five years from now, you know, what does that mean?Well that, uh, is that something which has kind of stood the test of time to some extent? And, um, and, uh, well the Lindy effect generally means, I don't think it's using this context with ideas like things which, with withstood the test of time tend to also with some test of time in the future, right? Like, like if we talked to Naim is like, well, people have lifting heavy weights and doing red wine for 2000 years, so let's continue doing that.It's probably a good thing. Um, but, but, but that's Twitter today. What is the future of Twitter? Well, uh, well, I think so one is, I think that's gonna continue to be true, right? 10 years from now, five years from now, it's still gonna be the metic battleground. It's still gonna be the place where ideas are shared, et cetera.Um, you know, I'm very. Unabashedly a a big fan of what Elon, uh, as a person, as a founder and what he's doing at Twitter. And my hope is that, you know, he can kind of canoe that and, you know, he's, you know, and I can't actually predict what he's gonna go Bill, he's kind of talked about it. Maybe that means bringing in other product ideas.Uh, I think he's talked about payments. He's talked about like having like longer form video. Uh, who knows, right? But I do know, like five years from now, it is still gonna be the place of like active conversation where people fight, yell, discuss, and maybe sometimes altogether. Yeah. Yeah. Uh, the Twitter, [00:30:58] Is Big Tech Overstaffed?[00:30:58] Dwarkesh: um, conversation has raised a lot of, a lot of questions about how over or understaffed, uh, these big tech companies are, and in particular, um, how many people you can get rid of and the thing basically functions or how fragile are these code bases?And having worked at many of these big tech companies, how, how big is the bus factor, would you guess? Like what, what percentage of people could I fire at the random big tech [00:31:22] Sriram: company? Why? I think, uh, [00:31:23] Aarthi: yeah, I think. That's one way to look at it. I think the way I see it is there are a few factors that go into this, right?Like pre covid, post covid, like through covid everybody became remote, remote teams. As you scaled, it was kind of also hard to figure out what was really going on in different parts of the organization. And I think a lot of inefficiencies were overcome by just hiring more people. It's like, oh, you know what, like that team, yeah, that project's like lagging, let's just like add 10 more people.And that's kind of like it became the norm. Yeah. And I think a lot of these teams just got bigger and bigger and bigger. I think the other part of it was also, um, you lot of how performance ratings and culture of like, moving ahead in your career path. And a lot of these companies were dependent on how big your team was and uh, and so every six months or year long cycle or whatever is your performance review cycle, people would be like, this person instead of looking at what has this person shipped or what has like the impact that this person's got had, uh, the team's done.It became more of like, well this person's got a hundred percent arc or 200% arc and next year they're gonna have a 10% increase and that's gonna be like this much. And you know, that was the conversation. And so a lot of the success and promo cycles and all of those conversations were tied around like number of headcount that this person would get under them as such, which I think is like a terrible way to think about how you're moving up the ladder.Um, you should really, like, even at a big company, you should really be thinking about the impact that you've had and customers you've reached and all of that stuff. And I think at some point people kind of like lost that, uh, and pick the more simpler metric, which just headcount and it's easy. Yeah. And to just scale that kind of thing.So I think now with Elon doing this where he is like cutting costs, and I think Elon's doing this for different set of reasons. You know, Twitter's been losing money and I think it's like driving efficiency. Like this is like no different. Anybody else who like comes in, takes over a business and looks at it and says, wait, we are losing money every day.We have to do something about this. Like, it's not about like, you know, cutting fat for the sake of it or anything. It's like this, this business is not gonna be viable if we keep it going the way it is. Yeah. And just pure economics. And so when he came in and did that, I'm now seeing this, and I'm sure Sheam is too at like at eight 16 Z and like his companies, uh, but even outside, and I see this with like my angel investment portfolio of companies, um, and just founders I talk to where people are like, wait, Elon can do that with Twitter.I really need to do that with my company. And it's given them the permission to be more aggressive and to kind of get back into the basics of why are we building what we are building? These are our customers, this is our revenue. Why do we have these many employees? What do they all do? And not from a place of like being cynical, but from a place of.I want people to be efficient in doing what they do and how do we [00:34:06] Sriram: make that happen? Yeah. I, I stole this, I think somebody said this on Twitter and I officially, he said, Elon has shifted the overturn window of, uh, the playbook for running a company. Um, which is, I think if you look at Twitter, uh, you know, and by the way, I would say, you know, you know the sort of, the warning that shows up, which is don't try this at home before, which is like, so don't try some of these unless you're er and maybe try your own version of these.But, you know, number one is the idea that you, you can become better not through growth, but by cutting things. You can become better, by demanding more out of yourself and the people who work for you. Uh, you, you can become better by hiring a, you know, a higher bar, sitting a higher bar for the talent that you bring into the company and, uh, that you reach into the company.I think at the heart of it, by the way, uh, you know, it's one of the things I've kinda observed from Elon. His relentless focus on substance, which is every condition is gonna be like, you know, the, the meme about what have you gotten done this week is, it kinda makes sense to everything else, which is like, okay, what are we building?What is the thing? Who's the actual person doing the work? As opposed to the some manager two levels a about aggregating, you know, the reports and then telling you what's being done. There is a relentless focus on substance. And my theory is, by the way, I think maybe some of it comes from Iran's background in, uh, space and Tesla, where at the end of the day, you are bound by the physics of the real world, right?If you get something wrong, right, you can, the rockets won't take off or won't land. That'd be a kalo, right? Like what, what's a, the phrase that they use, uh, rapid unplanned disassembly is the word. Right? Which is like better than saying it went kaboom. Uh, but, you know, so the constraints are if, if, you know, if you get something wrong at a social media company, people can tell if you get something really wrong at space with the Tesla.People can tap, right? Like very dramatically so and so, and I think, so there was a relentless focus on substance, right? Uh, being correct, um, you know, what is actually being done. And I think that's external Twitter too. And I think a lot of other founders I've talked to, uh, uh, in, sometimes in private, I look at this and go, oh, there is no different playbook that they have always I instituted or they were used to when they were growing up.We saw this when we were growing up. They're definitely seen some other cultures around the world where we can now actually do this because we've seen somebody else do this. And they don't have to do the exact same thing, you know, Elon is doing. Uh, they don't have to, uh, but they can do their variations of demanding more of themselves, demanding more of the people that work for them.Um, focusing on substance, focusing on speed. Uh, I think our all core element. [00:36:24] Aarthi: I also think over the last few years, uh, this may be controversial, I don't know why it is, but it somehow is that you can no longer talk about hard work as like a recipe for success. And you know, like growing up for us. When people say that, or like our parents say that, we just like kind of roll our eyes and be like, yeah, sure.Like, we work hard, like we get it. Yeah. But I think over the last couple of years, it just became not cool to say that if you work hard, then you can, there is a shot at like finding success. And I think it's kind of refreshing almost, uh, to have Elon come in and say, we are gonna work really hard. We are gonna be really hardcore about how we build things.And it's, it's very simple. Like you have to put in the hours. There is no kind of shortcut to it. And I think it's, it's nice to bring it all tight, all back to the basics. And, uh, I like that, like, I like the fact that we are now talking about it again and it's, it's sad that now talking about working really hard or having beds in your office, we used to do that at MicrosoftYeah. Uh, is now like suddenly really controversial. And so, um, I'm, I'm all for this. Like, you know, it's not for everyone, but if you are that type of person who really enjoys working hard, really enjoys shipping things and building really good things, Then I think you might find a fit in this culture. And I think that's a good thing.Yeah. I, [00:37:39] Sriram: I think there's nothing remarkable that has been built without people just working really hard. It doesn't happen for years and years, but I think for strong, some short-term burst of some really passionate, motivated, smart people working some really, you know, and hard doesn't mean time. It can mean so many different dimensions, but I don't think anything great gets built without that.So, uh, yeah, it's interesting. We [00:37:59] Aarthi: used to like do overnights at Microsoft. Like we'd just like sleep under our desk, um, until the janitor would just like, poke us out of there like, I really need to vacuum your cubicle. Like, get out of here. And so we would just like find another bed or something and just like, go crash on some couch.But it was, those were like some of our fun days, like, and we look back at it and you're like, we sh we built a lot. I think at some point sh I think when I walked over to his cubicle, he was like looking at Windows Source code and we're like, we are looking at Windows source code. This is the best thing ever.I think, I think there's such joy in like, Finding those moments where you like work hard and you're feeling really good about it. [00:38:36] Sriram: Yeah, yeah, yeah. Um, so you [00:38:37] Next CEO of Twitter?[00:38:37] Dwarkesh: get working hard and bringing talent into the company, uh, let's say Elon and says Tomorrow, you know what, uh, Riam, I'm, uh, I've got these other three companies that I've gotta run and I need some help running this company.And he says, Sriram would you be down to be the next, [00:38:51] Sriram: uh, next CEO of Twitter Absolutely not. Absolutely not. But I am married to someone. No, uh uh, no, uh uh, you know, you know when, uh, I don't think I was, the answer is absolutely not. And you know this exactly. Fun story. Um, uh, I don't think it says in public before. So when you, when I was in the process, you know, talking to and nor words and, you know, it's, it's not like a, uh, it's not like a very linear process.It's kind of a relationship that kind of develops over time. And I met Mark Andreen, uh, multiple times over the years. They've been having this discussion of like, Hey, do you want to come do venture or do you want to, if you wanna do venture, do you wanna come do with us? And um, and, and one of the things Mark would always tell me is, uh, something like, we would love to have you, but you have to scratch the edge of being an operator first.Um, because there are a lot of, there are a lot of ways VCs fail, uh, operator at VCs fail. Um, and I can get, get into some of them if you're interested, but one of the common ways that they fail is they're like, oh, I really want to go back to, um, building companies. And, uh, and now thing is like antis more than most interest, like really respects entrepreneurship, fraud's the hard of what we do.But he will, like, you have to get that out of a system. You have to be like, okay, I'm done with that word. I want to now do this. Uh, before you know, uh, you want to come over, right? And if you say so, let's have this conversation, but if not, we will wait for you. Right. And a woman telling me this all the time, and at some point of time I decided, uh, that, uh, you know, I just love this modoc.Um, you know, there are many things kind of different about being an operator versus a BC uh, and you kind of actually kind of really train myself in what is actually a new profession. But one of the things is like, you know, you kind of have to be more of a coach and more open to like, working with very different kinds of people without having direct agency.And it's always a very different mode of operation, right? And you have to be like, well, I'm not the person doing the thing. I'm not the person getting the glory. I'm here to fund, obviously, but really help support coach be, uh, a lending hand, be a supporting shoulder, whatever the, uh, the metaphor is, or for somebody else doing the thing.And so you kind of have to have the shift in your brain. And I think sometimes when VCs don't work out, the few operator on VCs don't work out. There are few reasons. Uh, number one reason I would say is when an operator, and I, I hate the word operator by the way, right? It just means you have a regular job.Uh, you know, uh, and, uh, but the number one reason is like when you have a regular job, you know, you're an engineer, you're, you're a product manager, you're a marketer, whatever. , you get feedback every single day about how you're doing. If you're an engineer, you're checking in code or you know your manager, you hire a great person, whatever it is.When you're at Visa, you're not getting direct feedback, right? You know, maybe today what I'm doing now, recording this with you is the best thing ever because some amazing fund is gonna meet it and they're gonna come talk to me, or maybe it's a total waste of time and I should be talking some else. You do have no way of knowing.So you really have to think very differently about how you think about patients, how we think about spending your time, and you don't get the dopamine of like, oh, I'm getting this great reinforcement loop. Um, the second part of it is because of that lack of feedback loop, you often don't know how well you're doing.Also, you don't have that fantastic product demo or you're like, you know, if an engineer like, oh, I got this thing working, the builder is working, it's 10 x faster, or this thing actually works, whatever the thing is, you don't get that feedback loop, uh, because that next great company that, you know, the next Larry and Sergey or Brian Armstrong might walk in through your door or Zoom meeting tomorrow or maybe two years from now.So you don't really have a way to know. Um, so you kind of have to be, you have a focus on different ways to do, uh, get. Kind of figured out how well you're doing. The third part of it is, uh, you know, the, uh, the feedback loops are so long where, uh, you know, you, you can't test it. When I was a product manager, you would ship things, something you, if you don't like it, you kill it, you ship something else.At, at our firm in, you invest in somebody, you're working with them for a decade, if not longer, really for life in some ways. So you are making much more intense, but much less frequent decisions as opposed to when you're in a regular job, you're making very frequent, very common decisions, uh, every single day.So, uh, I get a lot of differences and I think, you know, sometimes, uh, you know, folks who, who are like a former CEO or former like VP product, uh, uh, I talk a lot of them sometimes who went from, came to BC and then went back and they either couldn't adapt or didn't like it, or didn't like the emotions of it.And I had to really convince myself that okay. Hopefully wouldn't fate those problems. I probably, maybe some other problems. And, uh, uh, so yes, the long way of saying no, , [00:43:13] Why Don't More Venture Capitalists Become Founders?[00:43:13] Dwarkesh: um, the desk partly answer another question I had, which was, you know, there is obviously this pipeline of people who are founders who become venture capitalists.And it's interesting to me. I would think that the other end or the converse of that would be just as common because if you're, if you're an angel investor or venture capitalist, you've seen all these companies, you've seen dozens of companies go through all these challenges and then you'd be like, oh, I, I understand.[00:43:36] Sriram: Wait, why do you think more VCs driven apart? You have some strong opinions of this . [00:43:40] Dwarkesh: Should more venture capitalists and investors become founders? I think [00:43:43] Aarthi: they should. I don't think they will. Ouch. I dunno, why not? Um, I think, uh, look, I think the world is better with more founders. More people should start companies, more people should be building things.I fundamentally think that's what needs to happen. Like our single biggest need is like, we just don't have enough founders. And we should just all be trying new things, building new projects, all of that. Um, I think for venture capital is, I think what happens, and this is just my take, I don't know if Farram agrees with it, but, um, I think they see so much from different companies.And if you're like really successful with what you do as a vc, you are probably seeing hundreds of companies operate. You're seeing how the sausage is being made in each one of them. Like an operating job. You kind of sort of like have this linear learning experience. You go from one job to the other.Here you kind of sort of see in parallel, like you're probably on like 50, 60 boards. Uh, and oftentimes when it comes to the investor as like an issue, it is usually a bad problem. Um, and you kind of see like you, you know, you kind of see how every company, what the challenges are, and every company probably has like, you know, the best companies we know, I've all had this like near death experience and they've come out of that.That's how the best founders are made. Um, you see all of that and I think at some point you kind of have this fear of like, I don't know. I just don't think I wanna like, bet everything into this one startup. One thing, I think it's very hard to have focus if you've honed your skillset to be much more breath first and go look at like a portfolio of companies being helpful to every one of them.And I see Sure. And do this every day where I, I have no idea how he does it, but key context, which is every 30 minutes. Yeah. And it's crazy. Like I would go completely and say, where if you told me board meeting this founder pitch, oh, sell this operating role for this portfolio company. Second board meeting, third, board meeting founder, pitch founder pitch founder pitch.And that's like, you know, all day, every day nonstop. Um, that's just like, you, you, I don't think you can like, kind of turn your mindset into being like, I'm gonna clear up my calendar and I'm just gonna like work on this one thing. Yeah. And it may be successful, it may not be, but I'm gonna give it my best shot.It's a very, very different psychology. I don't know. What do you [00:45:57] Sriram: think? Well, Well, one of my partners Triess to say like, I don't know what VCs do all day. The job is so easy, uh, uh, you know, they should start complaining. I mean, being a founder is really hard. Um, and I think, you know, there's a part of it where the VCs are like, oh, wait, I see how hard it is.And I'm like, I'm happy to support, but I don't know whether I can go through with it. So, because it's just really hard and which is kind of like why we have like, so much, uh, sort of respect and empathy, uh, for the whole thing, which is, I, [00:46:20] Aarthi: I do like a lot of VCs, the best VCs I know are people who've been operators in the past because they have a lot of empathy for what it takes to go operate.Um, and I've generally connected better with them because you're like, oh, okay, you're a builder. You've built these things, so, you know, kind of thing. Yeah. Um, but I do think a lot more VCs should become [00:46:38] Sriram: founders than, yeah. I, I think it's some of the couple of other things which happened, which is, uh, uh, like Arthur said, like sometimes, uh, you know, when we see you kind of, you see, you kind of start to pattern match, like on.And you sometimes you analyze and, and you kind of, your brain kind of becomes so focused on context switching. And I think when need a founder, you need to kind of just dedicate, you know, everything to just one idea. And it, it's not just bbc sometimes with academics also, where sometimes you are like a person who's supporting multiple different kinds of disciplines and context switching between like various speech students you support.Uh, but it's very different from being in the lab and working on one problem for like long, long years. Right. So, um, and I think it's kind of hard to then context switch back into just doing the exact, you know, just focus on one problem, one mission, day in and day out. So I think that's hard, uh, and uh, but you should be a founder.Yeah, I think, yeah, I think more people should try. [00:47:32] Role of Boards[00:47:32] Dwarkesh: . Speaking of being on boards, uh, what the FTX Saga has raised some questions about what is like the role of a board, even in a startup, uh, stage company, and you guys are on multiple boards, so I'm curious how you think about, there's a range of between micromanaging everything the CEO does to just rubber stamping everything the CEO does.Where, what is the responsibility of a board and a startup? [00:47:54] Aarthi: What, what, what are the, this is something I'm really curious about too. I'm [00:47:57] Sriram: just, well, I just wanna know on the FDX soccer, whether we are gonna beat the FTX episode in interviews in terms of view your podcast, right? Like, so if you folks are listening, right?Like let's get us to number one. So what you YouTube like can subscriber, they're already listening. [00:48:10] Aarthi: What do you mean? Get us [00:48:10] Sriram: to number one? Okay, then, then spread the word, right? Like, uh, don't [00:48:13] Aarthi: watch other episodes. It's kinda what you [00:48:15] Sriram: should, I mean, if there's [00:48:16] Dwarkesh: like some sort of scandal with a 16 Z, we could definitely be to fdx.[00:48:21] Sriram: Uh, uh, yeah, I think it's gonna, well, it's gonna be really hard to read that one. Uh, , uh, uh, for for sure. Uh, uh, oh my goodness. Um, uh, but no, [00:48:29] Aarthi: I'm, I'm genuinely curious about [00:48:31] Sriram: these two. Well, uh, it's a few things, you know, so the multiple schools of thought, I would say, you know, there's one school of thought, which is the, uh, uh, you know, which I don't think I totally subscribe to, but I think some of the other later stages, especially public market folks that I work with sometimes subscribe to, which is the only job of a, uh, board is to hire and fire the ceo.I don't think I really subscribe to that. I think because we deal with more, uh, early stage venture, um, and our job is like, uh, you know, like lot of the companies I work with are in a cdc c, b, you know, they have something working, but they have a lot long way to go. Um, and hopefully this journey, which goes on for many, many years, and I think the best way I thought about it is to, people would say like, you want to be.Wave form dampener, which is, uh, you know, for example, if the company's kind of like soaring, you want to kind of be like kind the check and balance of what? Like, hey, okay, what do we do to, uh, you know, um, uh, to make sure we are covering our bases or dotting the is dotting the, crossing The ts be very kind of like careful about it because the natural gravitational pool of the company is gonna take it like one direct.On the other hand, uh, if the company's not doing very well and everybody's beating us, beating up about it, you're, you know, your cust you're not able to close deals. The press is beating you up. You want to be the person who is supportive to the ceo, who's rallying, everybody helping, you know, convince management to stay, helping convince, close host, hire.So, um, there are a lot of things, other things that go into being a board member. Obviously there's a fiscal responsibility part of things, and, um, you know, um, because you kind of represent so many stakeholders. But I think at the heart of it, I kind of think about, uh, you know, how do I sort of help the founder, uh, the founder and kind of dampen the waveform.Um, the other Pinteresting part was actually the board meetings. Uh, Themselves do. Uh, and I do think like, you know, about once a year or, uh, so like that there's every kind of, there's, there's almost always a point every 18 months or so in a company's lifetime where you have like some very decisive, interesting moment, right?It could be good, it could be bad. And I think those moments can be, uh, really, really pivotal. So I think there's, there's huge value in showing up to board meetings, being really prepared, uh, uh, where you've done your homework, you, you know, you've kind of had all the conversations maybe beforehand. Um, and you're coming into add real value, like nothing kind of annoying me if somebody's just kind of showing up and, you know, they're kind of maybe cheering on the founder once or twice and they kind of go away.So I don't think you can make big difference, but, uh, you know, I think about, okay, how are we sort of like the waveform, the, you know, make sure the company, [00:50:58] Aarthi: but I guess the question then is like, should startups have better corporate governance compared to where we are today? Would that have avoided, like, say the FTX [00:51:08] Sriram: saga?No, I mean, it's, I mean, we, I guess there'll be a legal process and you'll find out right when the FTX case, nobody really knows, you know, like, I mean, like what level of, uh, who knew what, when, and what level of deceptions, you know, deception, uh, uh, you know, unfolded, right? So, uh, it, yeah. Maybe, but you know, it could have been, uh, it could have been very possible that, you know, uh, somebody, somebody just fakes or lies stuff, uh, lies to you in multiple ways.[00:51:36] Aarthi: To,

Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Michael #1: Jupyter Server 2.0 is released! Jupyter Server provides the core web server that powers JupyterLab and Jupyter Notebook. New Identity API: As Jupyter continues to innovate its real-time collaboration experience, identity is an important component. New Authorization API: Enabling collaboration on a notebook shouldn't mean “allow everyone with access to my Jupyter Server to edit my notebooks”. What if I want to share my notebook with e.g. a subset of my teammates? New Event System API: jupyter_events—a package that provides a JSON-schema-based event-driven system to Jupyter Server and server extensions. Terminals Service is now a Server Extension: Jupyter Server now ships the “Terminals Service” as an extension (installed and enabled by default) rather than a core Jupyter Service. pytest-jupyter: A pytest plugin for Jupyter Brian #2: Converting to pyproject.toml Last week, episode 314, we talked about “Tools for rewriting Python code” and I mentioned a desire to convert setup.py/setup.cfg to pyproject.toml Several of you, including Christian Clauss and Brian Skinn, let me know about a few tools to help in that area. Thank you. ini2toml - Automatically translates .ini/.cfg files into TOML “… can also be used to convert any compatible .ini/.cfg file to TOML.” “ini2toml comes in two flavours: “lite” and “full”. The “lite” flavour will create a TOML document that does not contain any of the comments from the original .ini/.cfg file. On the other hand, the “full” flavour will make an extra effort to translate these comments into a TOML-equivalent (please notice sometimes this translation is not perfect, so it is always good to check the TOML document afterwards).” pyproject-fmt - Apply a consistent format to pyproject.toml files Having a consistent ordering and such is actually quite nice. I agreed with most changes when I tried it, except one change. The faulty change: it modified the name of my project. Not cool. pytest plugins are traditionally named pytest-something. the tool replaced the - with _. Wrong. So, be careful with adding this to your tool chain if you have intentional dashes in the name. Otherwise, and still, cool project. validate-pyproject - Automated checks on pyproject.toml powered by JSON Schema definitions It's a bit terse with errors, but still useful. $ validate-pyproject pyproject.toml Invalid file: pyproject.toml [ERROR] `project.authors[{data__authors_x}]` must be object $ validate-pyproject pyproject.toml Invalid file: pyproject.toml [ERROR] Invalid value (at line 3, column 12) I'd probably add tox Don't forget to build and test your project after making changes to pyproject.toml You'll catch things like missing dependencies that the other tools will miss. Michael #3: aws-lambda-powertools-python Via Mark Pender A suite of utilities for AWS Lambda Functions that makes distributed tracing, structured logging, custom metrics, idempotency, and many leading practices easier Looks kinda cool if you prefer to work almost entirely in python and avoid using any 3rd party tools like Terraform etc to manage the support functions of deploying, monitoring, debugging lambda functions - Tracing: Decorators and utilities to trace Lambda function handlers, and both synchronous and asynchronous functions Logging - Structured logging made easier, and decorator to enrich structured logging with key Lambda context details Metrics - Custom Metrics created asynchronously via CloudWatch Embedded Metric Format (EMF) Event handler: AppSync - AWS AppSync event handler for Lambda Direct Resolver and Amplify GraphQL Transformer function Event handler: API Gateway and ALB - Amazon API Gateway REST/HTTP API and ALB event handler for Lambda functions invoked using Proxy integration Bring your own middleware - Decorator factory to create your own middleware to run logic before, and after each Lambda invocation Parameters utility - Retrieve and cache parameter values from Parameter Store, Secrets Manager, or DynamoDB Batch processing - Handle partial failures for AWS SQS batch processing Typing - Static typing classes to speedup development in your IDE Validation - JSON Schema validator for inbound events and responses Event source data classes - Data classes describing the schema of common Lambda event triggers Parser - Data parsing and deep validation using Pydantic Idempotency - Convert your Lambda functions into idempotent operations which are safe to retry Feature Flags - A simple rule engine to evaluate when one or multiple features should be enabled depending on the input Streaming - Streams datasets larger than the available memory as streaming data. Brian #4: How to create a self updating GitHub Readme Bob Belderbos Bob's GitHub profile is nice Includes latest Pybites articles, latest Python tips, and even latest Fosstodon toots And he includes a link to an article on how he did this. A Python script that pulls together all of the content, build-readme.py and fills in a TEMPLATE.md markdown file. It gets called through a GitHub action workflow, update.yml and automatically commits changes, currently daily at 8:45 This happens every day, and it looks like there are only commits if Note: We covered Simon Willison's notes on self updating README on episode 192 in 2020 Extras Brian: GitHub can check your repos for leaked secrets. Julia Evans has released a new zine, The Pocket Guide to Debugging Python Easter Eggs Includes this fun one from 2009 from Barry Warsaw and Brett Cannon >>> from __future__ import barry_as_FLUFL >>> 1 2 True >>> 1 != 2 File "[HTML_REMOVED]", line 1 1 != 2 ^ SyntaxError: invalid syntax Crontab Guru Michael: Canary Email AI 3.11 delivers First chance to try “iPad as the sole travel device.” Here's a report. Follow up from 306 and 309 discussions. Maps be free New laptop design Joke: What are clouds made of?

​This episode is sponsored by Fiberplane. Your platform for collaborative debugging notebooks!Episode Resources:Try Fiberplane hereFiberplane websiteFiberplane DocsNP-hard Ventures About Micha Hernandez van LeuffenMicha Hernandez van Leuffen is the founder and CEO of Fiberplane. He previously founded Wercker, a container-native CI/CD platform that was acquired by Oracle. Micha has dedicated his career to improving the workflows of developers. Read the whole episode (Transcript)[If you want, you can help make the transcript better, and improve the podcast's accessibility via Github. I'm happy to lend a hand to help you get started with pull requests, and open source work.][00:00:00] Michaela: Hello and welcome to the Software Engineering Unlocked Podcast. I'm host, Dr. McKayla, and today I have the pleasure to talk to Micha Hernandez van Leuffen. He is the founder and CEO of Fiberplane. He previously was the founder of Wercker, a container native CI/CD platform that was acquired by Oracle. Micha has dedicated his career to improving the workflow of developers, so he and I have a lot to talk about today.I'm really, really happy that he's here today and he's also sponsoring today's episode. Welcome to the show. I'm happy that you're here, Micha.[00:00:36] Micha: Thank you for having me. Excited to be on the show.[00:00:38] Michaela: Yeah, I'm really, really excited. So, Micha, I wanted to start really from the beginning. So you are the CEO of Fiberplane and you are the founder of Wercker, which you already sold.So, can you tell me a little bit about how you actually started to this entrepreneur journey of yours and what brought you to the developer experience area.[00:01:03] Micha: Yeah, sure thing. So I have a background in computer science and I did my so, I'm originally from Amsterdam, but I did my thesis at USF.And the topic was autonomous resource provision using software containers. This was all before Docker was a thing, you know, the container format that we now know and love. And I sort of got excited by that field of, of so containers and decided to start a company around it. That company was Worker, so container native CI/CD platform.So we helped developers build tests and deploy their applications to the cloud. We went, I would say, so we went through various iterations of the platform. You know, eventually, you know, we started off with Lxc as a container format and then eventually ended up, you know, having to, to platform on Docker.And Kubernetes. But, you know, it was quite a, quite a journey. So that company eventually got acquired by Oracle to bolster their cloud native strategy. And then, you know, spent a couple years in a Bay area as a VP of software development focusing on their cloud native efforts.Tried to do a little bit of open source there as well, and then, you know, move back to Europe. And so sort of started thinking about what's. Did some angel investing. We're still doing some angel investing as well actually in the sort of same arena. So developer tools, infrastructure building blocks for tomorrow.So I run a, a small precede seat fund with to other friends of mine. But then also started, you know, thinking about what to build next. And you know, we can get into that, but sort of from our experience at running work or this sort of large distributed. Sort of fiber plane was, was born.[00:02:26] Michaela: Cool. Yeah. And so how, how was the acquisition for you? I, from the time I'm, you said you were studying at the university, but then did you write out of university, you know, start worker or maybe already while[00:02:40] Micha: you were Yeah. More or less studying? Yeah. Yeah, more or less just out of university. So it was around 20, 20 12, 20 13.And then, you know, expanded the team. Of course we got an office in San Francisco and, and London. And then 2017 we got acquired by Whirlpool. Oh,[00:02:56] Michaela: very cool. Wow. Cool. So, and you were the, you were the founder of that and also probably cto, CEO. At, at the beginning you were one person shop, or was this, or have this idea and I get some funding and I already, you know, have a team when I'm starting out, or was it more bootstrapped way?How, how was that?[00:03:16] Micha: Yeah, yeah. We both gates, both fiber plane and, and, and worker. We got some funding early on. Then eventually got a CTO. For worker was one of the co-founders of, of OpenStack. So also, you know, very early in the, in that sort of, mm-hmm. container and, and cloud infrastructure journey.And then if for fiber plane, Yeah. There, there's no cto. I'm. I'm both CEO and cto, I guess[00:03:38] Michaela: at the same time. Yeah. Cool, cool. Can you tell me a little bit about fiber plane? What is fiber plane? You know, what does, what does it has to do with containers and with developer experience? What, what kind of of a product is it?[00:03:51] Micha: Yeah, sure thing. So, so guess coming back to the worker days, right? So we, we, you know, we're running this distributed system cic cd, so we were also running users arbitrary code. You know, any, any sort of job could happen on the platform on top of Kubernetes, inside of containers. So one of the things that, you know, stuck with me was it was very hard to always sort of debug the system, like figure out what's really going on when we had some kind of issue.You know, we've going back and forth between metrics, logs, traces, trying to figure out what is the root cause of an issue. So sort of that, that was sort of one thing. So we're thinking a lot about, you know, surely there must be a better way to, to, to help you on this, on this journey. . The other thing that I started thinking about a lot was sort of just challenge the assumption of the dashboard, mm-hmm.So if you think about it, like a lot of the monitoring observability tools are modeled after the dashboard, like sort of cockpit like view of your infrastructure. But I'd say that those are great for the known knowns. So dashboard is great. You set it up in advance, you know exactly what's gonna go wrong.These are the things to monitor. These are the things, you know, to keep tabs on. But then reality hits and you know, the thing that you're looking at, at the dashboard is not necessarily a thing that's. Going wrong. Right? So started thinking a lot about you know, what, what is a better form factor to support that sort of more investigative explorative debugging of your infrastructure.And not to say that dashboards don't have their place, right? It's like still that sort of cockpit view of your infrastructure. I think that's a, a good thing to have. But for debugging, you might wanna sort of more explorative a form factor that also gives you actionable intelligence. I think the other thing that you see a lot with dashboards, like everybody's monitoring everything and now you get a lot of signal and a lot of inputs, but not necessarily the actionable intelligence to figure out what's going on.So that's sort of the other piece where it, then the other, like, the third like I would say is collaboration sort of thing that stuck with. Was also like we've come to enjoy tools like Notion, you know, Google Docs obviously. You know, in the design space we got Figma where collaboration is built in from the get go and it is found that it was kind of odd how in the developer tools and then sort of specifically DevOps.We don't really have sort of these collab collaboration not really built in. Right. If you think about it you know, the status quo of, of you and I debugging an issue is we get on, you know, we get on a. You share your screen you open some dashboard and we started talking over it or something.Right. And so it's, and it's, you know, I guess sort of covid accelerated his thinking a bit, but you know, of everybody going remote you know, how can you make that experience more collaborative?[00:06:22] Michaela: Mm-hmm. . So it's in the incident space, it's in the monitoring space, and you want to bring more collaboration.So how does it work? Yeah,[00:06:32] Micha: yeah, yeah, exactly. So what's your solution now? Yeah. Now I've explained sort of the in inception. Yeah. But yeah, but what is it? What is it? Right. So it's, it's it's a notebook form factor. So very much inspired by data science, right? Like rc, like Jupiter. Yeah, we can Jupyter Notebooks.Yeah. Think of, think of that form factor. Mm-hmm. . We don't use Jupyter or anything like that. We've written everything from scratch. But it's a sort of, yeah, a notebook form factor and you know, built in with collaboration. So you can add, mention people like you would on Slack. You can leave, you know, comments or discussions and all and all that.But where it gets interesting, we've got these things called providers, which are effectively plugins. So they're web assembly bundles, which we can sort of dive into into that as well. But they're providers that connect to your infrastructure, right? So we have, for instance, a provider for Elastic Search for your logs.We have a provider for Prometheus for your. And it allows you to connect to these observability systems and kind of pull 'em together into one form, factor the notebook, and then, you know, start collaborating around that. Mm-hmm. . So, you know, imagine if Notion and Datadog would have a baby . Yeah.That's kind what you get. Yeah.[00:07:41] Michaela: That's cool. So I can imagine that. Let's. I'm on call and hopefully I'm not alone. A call. You are also on call, right? Yeah, and so we would open a fiber plane notebook.[00:07:52] Micha: Hopefully we're in the same time zone and we don't need to like wake up in the middle of that. Yeah.[00:07:57] Michaela: Hopefully. Yes. And then we want to understand. How the system is behaving. And so we are pulling in observers. These are data sources. Yeah. More or less. Right. And then we can do some transformation with those data. Data sources or[00:08:12] Micha: Yeah, yeah. That, yeah, exactly. That, that might be the case. The other thing that we integrate with is, for instance, PagerDuty.So an alert goes off indeed we are on call, but an alert goes off and we have this PagerDuty integration. And subsequently a notebook is created for us already. Mm-hmm. . Okay. Maybe, maybe even with, you know, some, some charts and logs that are already related to the service that might be down.Okay. So depend, So depending on the alert, obviously you're, as you know, you're as good as how you've instrumented your alerts. But say we've written some good alerts, we now have a notebook ready to go. Based off a template. So that's another thing that we, that we have as well, which is this template mechanism.And now, you know, we're ready to, to, to go in, get in into things and start debugging. So we might have a checklist, you know you look at the metrics, I'll look at the logs, sort of this action plan. We pull in that data we start a discussion around it. Mm-hmm. , hopefully we come, we come to the, to the, you know, the root cause of, of our issue.[00:09:11] Michaela: Okay. And so this discussion and this pulling in data, this happens all in the notebook. Can you explain me a little bit more, and also our listeners Exactly. When we are on this, you know, on this call now, having a fiber plane notebook in front of our, what do we see, right? How does that, how does the tool look?[00:09:28] Micha: It's, it's very similar to, I would say, like a Notion Page or a Google Doc page. Mm-hmm. . So we've got like different, different headings. The other thing that we have is, so you might have a title for a notebook, right? You know, the billing, the billing API is now. The other thing that we have is sort of this, this time range.So maybe usually when there's an issue, you know, we've seen this behavior over the last three hours, so we can sort of have that time range locked into place. So we only want to see our. For the last three hours. And that means that any chart that we plot or any log that we pull in will adhere to that global timeframe.So that's what we see. Mm-hmm. . We have support for labels, so, you know, obviously big fans of Kubernetes and, and Promeus. So we, you know, labels are. A first class primitive on the platform. So you're able to sort of populate the notebook with the labels that might maybe be related to our service.Right? So it's a US East one, which is our region. It might, you know, say service is the billing. It might be, you know, environment is production. And the status of our incident is, mm-hmm. ongoing, stuff like that. So we have, we've got, go ahead.[00:10:34] Michaela: Cool. Yeah. And, and so is it then from top to bottom we are writing and we are investigating and we are writing out down the questions that we have and the investigation.Yeah, exactly. We do.[00:10:44] Micha: Yeah. Yeah. And so, so[00:10:45] Michaela: we might have, Is it an Yeah. Is it Yes,[00:10:48] Micha: we our work? Yeah. Yeah. It's sort of Exactly. And I think in the most ideal use case, right. And I do it most ideal scenario, you're kind of like writing your postmortem as you go along. That's what[00:10:59] Michaela: I, I was thinking exactly that.Right. And then maybe next time I'm on call again and I get PagerDuty and something is down, it's again, billing. Can I search in the fiber plane notebooks to find, you know, what we did last time and then[00:11:13] Micha: Exactly. So you'll, you'll search, jump to the conclusion . Yeah. Yeah, exactly. Hopefully, hopefully if you, if you experience, you know, the same issue multiple times at some point, we'll, we'll, you know, do a little commit on GitHub and we, we fix our, fix our, Yeah.Do. But yeah, indeed, so you can search Yeah. Cool. On the notebooks and see if you've, you know, ran into similar issues. So that's, you know, it's great for building up this, this system of record, right. This knowledge base of mm-hmm. . Mm-hmm. . Of infrastructure issues and, and incidents. And it's also great for onboarding, right?If a new person joins, like, this is our process. These are some of examples that we've run into you know, have a look. And now you've got a sense for you know, how we, how we handle things and some of the issues that we've investigated. Yeah. Cool. One more thing on the, on the product. So the other, so, you know, sort of explained the, the notebook form factor.We've got these providers, right, that pull in data. From different, different data sources like Elastic Search or, or Prometheus. The other thing that we have is a command line interface which is called fp. Mm-hmm . And apart from, you know, being able to create notebooks from your terminal and you know, even invite people from the terminal, all this sort of usual stuff that you would, you know, expect from interacting with an API, with, with a product like this, there's two other things that we do.So one is a command called FP Run. And it allows you to, if you are typing a command like cube, ctl logs for a specific pod, you can pipe that the output of that command to a notebook. And why that is useful is of course, you know, when we're de debugging this issue, you and I, and you're start typing things in your terminal.I have no idea what you just did. And this is a way sort of to capture that. So you're piping these these, these outputs from your, the stuff that you're typing into your into your. into the notebook. And the cool thing is, you know, in, on your laptop you just, you know, sort of see text, right?Monospace output. But for certain outputs such as the cube CTL logs command, we actually know the structure of the data and we're actually capable of formatting that in the notebook in the structured manner that you can start filtering on, on the logs and you know, select certain columns and sort of highlight even certain loglines for prosperity that you say, Hey, these are the culprits, these are the things that you need to take into, into consideration next.So we have this sort of command line interface companion, and the other thing that it does, you're actually capable of running a long, like, sort of same use case as it just, I mentioned, but like a long running recording, like you actually record your entire shell session session as you're debugging this thing and all the output gets piped into the notebook.Cool. Cool. And[00:13:46] Michaela: so I have two questions for fiber plane. One. Is the software engineer the right person to interact with you know, fiber plane or is it the site reliability engineer that's really designed, you know, or the tool is designed[00:14:03] Micha: for? Yeah, it's, it's, that's an excellent question. So I think one, one site reliability engineers, you kind of see in more larger organizations, right, where you start splitting up your teams.I will say, I think at the end of the day, right, is if you're an engineer, you've built the service, now you need to maintain it now you need to operate it like it's, it's your baby, right? You need to, you probably know best how that system behaves than anybody else. So indeed I would say that, yeah, the target group is, you know, developers.Mm-hmm. .[00:14:40] Michaela: And so the other question that I had around fiber plane is also. When we are on this call and we are writing in this notebook, how does the whole scenario look like? Are we still on a call, like, do we have Zoom or, you know, Google meet open, or are we really in the, in the fiber plane document just writing, Or are we sitting next to each other?You know, what, what's the traditional, Is there a traditional scenario or is this all possible with fiber plane? How would you recommend using[00:15:09] Micha: it? Yeah, Yeah. Yeah. Not a, not a great question. Right. I think back in the day, it would be that, you know, we maybe sit in the same office and I scoot over and we start looking at a, at a screen, right?And start typing together. Mm-hmm. . The reality is, of course, we're all doing remote work now, and we might not be in the same room. So I do think people will still use a Zoom call or a Google meet you know, as a companion to talk over stuff. I think, you know, people will still communicate in Slack and sort of start chatting back and forth.But I think what we hope to achieve with fiber plane is like the pasting of screenshots, right? Well, if you take a screenshot of some kind of chart in your dashboard and you put it in Slack and you know, somebody yells, Oh, that's not the, that's not the thing that you should be looking at. You should, you know, like all that sort of slack glue That, you know, it's our, our goal to do away with that.[00:15:59] Michaela: Yeah. And, and the slack blue is also very problematic for the search. At least I'm never able to find it again. Right. It's like is in the dark, super in[00:16:07] Micha: the dark area. Yeah. Super ephemeral. Yeah. Yeah. You can't, can't go back in time easily. And, and you know, how did we solve this last time? So again, like building up that system of record, I think.[00:16:17] Michaela: Yeah. Very cool. And so how long are you now working on fiber plane already?[00:16:23] Micha: So we've been working on it for about two years now. Which is a, is a, is a long time. I think as a sort of, you know, one of the things that we've, I guess, sort of discovered along the way that we're kind of like building two startups at the same time, Right?We're doing a notion or like a, a rich text, collaborative rich text editing experience, which is kind of like a startup on its own. Mm-hmm. . And we're building sort of this infrastructure product. So it's, you know, it's taken quite some time and, and energy to, to get the product to where it is now.[00:16:54] Michaela: Yeah. And do you have already users? Is it like can people that listen today, can they hop on fiber plane already or.[00:17:02] Micha: It's, it's in it's been in private beta, mm-hmm. , but I think by the time this gets aired it's will be in public beta and people can sign up and take it for a spin. And, you know, we would love to get feedback on, on our roadmap, right?And Okay. People can suggest what other types of providers we need to support, what are types of integrations we, you know, would love to, to have that convers.[00:17:23] Michaela: Cool. Yeah. So is there, There is the provider side. Is there something else that you want feedback on that you are exploring[00:17:30] Micha: maybe. . Yeah. Yeah. So we've got the providers that's one thing.We've got sort of our templating stack. Mm-hmm. So curious to sort of see how people sort of start codifying their knowledge, right? What's, what, what kind of processes people have to debug their infrastructure and sort of run their incidents or write their postmortems. So curious to see what people come up with there.Other types of integrations. Right? So we have as I said, sort of PagerDuty what other type of, sort of alert, alert to notebook or other types of external systems that we need to plug in with. I would love to get some feedback on that as well. Yeah,[00:18:04] Michaela: I think I had page Bailey over on the podcast.She's from GitHub and she was she was also, they were releasing something with copilot and you know, For data scientists, some, some spaces here. And she also said like, well, we really need input from the users, right? So try it out, you know, tell us how it's working. I think it's so valuable, right, to see not only like you have your vision and obviously.It's going one way, but then if you have your users, sometimes they take your product and they use it in a very different, you know, way than you anticipate it, which can be very informative. Right. I dunno. You have done two startups already. Have you seen that? And how do you react to it? Do you instrument the data a little bit?How do you realize that people are using your product in a different. . Yeah.[00:18:50] Micha: So, so obviously we have metrics and analytics on sort of usage patterns of the, of the product. But I think, I think that data is excellent, right? But also qualitative data is, mm-hmm. , especially at this stage is probably even better, right?Where you can get somebody on a call and, you know, tell us about your use case. Tell, tell us about the problem that you're trying to solve here and how can we be, be helpful in like what types of integrations should we support? I think sort of the difference between. Worker, I would say, and, and fiber plane is that, you know, worker was a pretty confined piece of surface area, right?Cic, c d the whole goal is so you either have a, you know, a green check mark next to your build or a red check mark next to your build. Like it either, you know, failed or passed. And we need to sort of do that fast for you, get, get that result quick. Mm-hmm. . And with fiber plane, it's a more. I think that the interesting thing here is like, it's a, it's a more explorative and a sort of rich design space, right?It's this notebook, which already you, you know, you can start typing and text and images and headings and check checklists and whatnot, right? It's a very open form factor and design space. And then of course, with the integrations, it can even, you know, be richer. So I'm very curious into your point, right what direction people will pull the product into.Cause you can take it into all sorts. Use cases and scenarios. Yeah,[00:20:05] Michaela: exactly. And I think as a founder also, or as the design team, product team, it's it's also a little bit of a balancing act, right? So how far, you know, let me, are we going with what the user are doing with our product and where are we setting some boundaries that they can't do everything right?So there's also often the talk about opinionated products, right? That you can actually do one thing and on one thing only, and we have an opinion on, you know, how. Supposed to use our product. And you know, we try to, if we see people deviate from that, we try to put an end to it. And then there's the other way where you say, Well, you know, if you take fiber plane and you do X with it and we haven't thought about this maybe, you know, we are okay with it.Or maybe we even support that path, right.[00:20:48] Micha: Yeah, I think, I think we're more on indeed on the, on the ladder, right? I think what we've sort of, we talk about this a lot internally, sort of everything is a building block. You know, you've, we've got the notebook, you've got these different cell types, you've got providers, you've got templates.Mm-hmm. You've got the command line interface. So like for us, like everything is a building block and we, we actually want to retain that flexibility. Not be too prescriptive. Cause maybe you have a, a if you think about sort of the, the incident debugging or, or investigating your infrastructure, like you might have a certain process, I might have a completely different process and we need to be able to facilitate, you know, these different workflows.So, you know, thus far sort of our, our thinking around the product has been everything is a building block. And it should be this sort of flexible form factor that people can pull into into different scenarios and use. I mean,[00:21:36] Michaela: we have infrastructure as code, right? And we have like security as code.Maybe we have debugging as code. Maybe, you know, this is what's coming next. Can, can you envision that, that it's going in this direction? Because while we have building blocks, maybe right now it's not you know, programming language for debugging, but it could go a little bit into the distraction, right?No code coding for debugging.[00:22:02] Micha: Yeah, we've actually, we've, we've had some of, of that sort of discussion internally as well. If you think about the templates right. To, to some extent that is a, you know, we use J Sonet as a, as a sort of language, but we sort of codified them in a certain way and you can, you could argue that the templates is, you know, sort of a programming language for at least, you know, that debugging process, right?Yeah, exactly. Right. Yeah. And. And, and we, you can take that even further and make it kind of like statically typed and make it adhere to, you know, certain rules and maybe even have control flow. So I think that, that there's, there's a piece there. And then maybe, you know, obviously we have you know, some YAML configuration on how you set up your providers, right?Like how to connect to your infrastructure. So there's some, you know, observability as code in mm-hmm. in that realm. Yeah. Yeah. I think that'll be an interesting part of the journey, right? Like to figure out can we, and some even.[00:22:55] Michaela: Yeah, in some parts should be well, don't repeat yourself, right?Like, for example, pulling in these providers, configuring that, you know, I get the right data. This would actually be something that I'm, you know, pulling in again. And probably that's what your templates do, right? So you say billing, oh, and then check, check, check, check, check. I have, you know, all my signals here and they're configured in a way that it's useful.And then for this investigation, hopefully, One at a type thing, right? So I'm investigating, and as we, as we talked before once I realized what's going on, hopefully in my postmortem I'm going to, you know, make sure that this is not happening again. So this code probably is not going to be reused that often.Maybe some, you know, some ideas from it, but hopefully we won't reproduce the same sect completely exact thing again.[00:23:44] Micha: Yeah. Cool. Yeah, that's, that's a super great point. And I think coming back, sort of the early part of the conversation around dashboards, right? I think thus far what we've sort of experienced as, you know, engineers ourselves, like, I think, I think we probably had sort of a phase around information gathering.Like all these dashboards are great for information gathering, but now with Kubernetes and containers and microservices like the, the, the number. Services that we're running and the complexity has increased. So I think, I think there's sort of an opportunity for more exactly what you're describing. So it's more about action, right?Mm-hmm. , what? What are we doing? We want to have the information, we want actionable intelligence that informs us what to do.[00:24:20] Michaela: Yeah, yeah, exactly. Because now I'm looking at this dashboard and I'm seeing the signals. But then everything else is outside of, you know, this realm, right? So what actions do I take?Do I go, go to the console? Do I restart that service? You know, or, you know, whatever I'm doing. And, and it's also vanishing, right? So I'm doing it, but then. Who can see it, What I did. Right? Yeah, exactly. And so now we are capturing this, which is very nice, and then we can learn from it Right. Postmortems as well.Yeah. So I looked a little bit through your blog and and, and your Twitter, and you were also talking about blameless postmortems. So how do you think about psychological safety? How should people. In an organization look at on call and incident management to really make it sure that we are ending the blame game.Right. You probably have some thoughts about that as well, because you're working in this area.[00:25:19] Micha: Yeah. I, I think it's important to, and you like not have put any blame on any person. Right. It, it is a, and I guess sort of, you know, that's also why we're building this product. It is a collaborative process to debug an issue or resolve an incident.Like, and what you want to achieve is to put the entire team in the best possible position to solve the issue at hand and and, you know, a support structure around it. So, you know, coming back to the product, like being able to, to open discussions. Point people in the, in the, in the right direction.[00:25:52] Michaela: So maybe also if it's easier to find a problem to root cause it, and, you know, incidents become no issue or at least a lesser issue. So maybe the blame game is not that important. Can, can we say it that way?[00:26:08] Micha: I think so. Yeah. Yeah, yeah. If, if, you know, if the process becomes repeatable and we codify that and we collaborate on it and we build up that, again, that system of record and knowledge base I think that, you know, puts us in a safer position to, to solve the next one.That's[00:26:25] Michaela: true. Yeah. Another thing that I was thinking of when I looked through, you know, fiber plane and what it does is KS engineering and I thought like what KS engineering is where you try to prevent not only the knowns, but also the unknowns, right? So really think about, you know, what, what could go wrong and then, you know, make a fallback so that your system is reliable.Or, you know, if this database goes down that not the whole system goes down, but only a part of it and so on. Do you think that KS engineers can act. Source or, you know, use those notebooks that you're creating as input for knowing, you know, what we should actually look at and, Yeah.[00:27:02] Micha: Well, I think it, well, one thing I think it'd be a great provider yeah.integrating with, with, with, you know, one or many of the, the chaos engineering services out there. I think it's a great way to train your team, right? You, we plug in some K engineering provider. The, the provider communicates with your infrastructure and such, pulling out wires from from, you know, your, your system.And then now go ahead and start, you know, debugging this issue and mm-hmm. and you know, use different templates and you can, you know, sort of trial all sorts of different issues. I think it'd be super fun. Yeah.[00:27:37] Michaela: Yeah. So Micha, one thing that I also saw is that some of your of fiber plane is open source.So what's your vision for open sourcing that are, you know, are some parts being open source? Can people help with the building fiber plane?[00:27:51] Micha: Yeah, great question. So right now what we've open sourced is a project called fp bind Gen. So this is actually of SDK bindings, generat. For how you would create full stack web assembly plugins.So this is what we use to build our own elastic search and our Prometheus plugins. So we've, we've open sourced that. It's on GitHub we've already got some, quite some feedback on it. So, but would love some more. And then going forward we'll be open sourcing sort of our templating stack the proxy.Which sort of sets which you install inside your cluster and sort of sets up the secure connections between the providers and your infrastructure and then the fabric plane managed service. And then the command line interface that I mentioned will also be open source. So expect more to hear from us on the open source front.[00:28:36] Michaela: Yeah, Cool. I think that's so important, especially for developer tooling, that people can also really get it into their hands and then help, you know, shape the, or make the best product for their, for their environments that they have. I think this is such a success strategy.[00:28:50] Micha: Yeah, exactly. And you know, we, as I said, we would love to get feedback on the, on the providers and the, the plugin model, but maybe even, you know, once we open source the the, the provider stack would be great if people maybe come up with crazy ideas.Right? You can think of any type of provider that you could surface data inside of, inside of the notebook. Yeah. Doesn't need to be observability or like monitoring data. Like could be. Yeah.[00:29:14] Michaela: Cool. Yeah, I'm super excited. What, you know, what will come out of that. Yeah. So I want to come back a little bit to your founding story because I know a lot of people are interested in developer tools and, you know, and, and Startup founding as well.And you did it twice already, right? And maybe several more times in your life, I dunno. But right now we know of two instances. Yeah. There, there. So, and and also for fiber plane, you already got funding, right? Several million dollars. And so how do you do. How do you do it out of Europe is also some of my questions that I have because I think it's a little bit a different game here in Europe than it's in Silicon Valley.Yeah. It doesn't look like, you know, opportunities around the corner everywhere. I, I have been studying in the Netherlands, so I know that actually Netherlands is really a good place, I think for, for tech startups and, you know, also a little bit out of the universities I saw there like You know, you get a little bit of help and, and, and funding and things like this, but still, I would assume it's harder than in Silicon Valley.So how did you make it work? How did you get funding? You also said that worker had some funding at the beginning. Yeah.[00:30:26] Micha: Yeah. It's a good question. Well, how did we do the second time around, to be honest, Because it's the second time. Yeah. It was a bit easier. I mean, it's never, It's, Yeah. Yeah. It's obviously, you know, never as easy.But it was definitely easier. I do think in Europe, if I also compare it to the worker days to where we are now, Like I do think the funding climate and sort of the, the, the, the thinking around startups has improved a lot, right? There's there's more funding out there, there's more feess. I think more importantly though, what we've seen is that now.Sort of the European unicorns have exited or gone ipo. And we have actually more operators inside of Europe that have experience in either founding a startup are able to sort of start doing angel investing or have worked at multiple startups and we have just more operating experience you know, versus honestly like bankers, right?That That, you know, help you out or are, are investing in you? So actually the, the, the funds that funder does were Crane Venture Partners which is actually a seed fund out of London that's actually focused on developer tools and infrastructure. So I would highly recommend, you know, talking to them.If you're thinking about, you know, building a developer tool company and you need some funding, of course my own fund is also focused on developer tool. So shameless plug there on MP Hard Ventures. You can just Google that and find me. And then we have North Zone, which is a, you know, very like multi-stage fund.Also out of, well actually quite different geographies and Notion Capital out of out of London as well. Okay. We've got some have several micro VCs, several things. Yeah. We have somebody funded West Coast Alana Anderson was doing with base case capitals investing in a lot of infrastructure and enterprise startups and Max Cloud from System one in Berlin.Is another one. So yeah, we have a good crew of, you know, a diff different experience and sort of different stage type of funding as well.[00:32:19] Michaela: Yeah. This was my next question that I had for you. It's probably not only about the money, you said experience, right? It's also about the knowledge that people have, right.How to do things. Probably, yeah. The people that they know, right? So that they can Yeah. To be Yeah, exactly. Can consider the right people have the right network and so.[00:32:36] Micha: Yeah, I think, I think the most, yeah, it's is, is introductions, but it's also. You know, if you, if you think about the, the funds that actually do developer tools, right?So they, in their portfolio, they, they've seen, you know, startups trying over and over to tackle some kind of go to market issue or trying to build an open source, mm-hmm. company, right? So they have some, some pattern matching and some, some knowledge about, you know, what to do and what, what not to do.Of course, it's all advice, but it's good to sort of have some people in your corner that have at least seen this, these types of companies being built. Over and over again. Right. That's, and then, and then other VCs have more experience in, you know, more, more like how to build up or scale up a sales organization and thinking about how to run a SaaS company.So yeah. Different experience from different, different funds.[00:33:20] Michaela: And so now you listed quite a lot of different investors. Do you reach out to each one of them or do you have like a whole group meeting and they're all in there and you ask them for advice? , how does it[00:33:33] Micha: Yeah. No, it's, it's sort of one on one chats, right?Either over, over chat or, you know, we meet up for coffee or, or or breakfast, mm-hmm. . But yeah, we try to do that on a, on a regular cadence. And then of course, when, you know, something exciting happens, such as our launch know, we try to group them together and get them all on the same page around the same time.Or of course if an issue arises, Right, which could also be the case. Yeah. And then sort of all hands on deck and everybody in the same room or zoom.[00:34:01] Michaela: And what about your biggest struggle on your, on your entrepreneurial journey, maybe now with fiber plane or maybe with Worker? Did you ever think that, you know, worker, when you started it, did you think that somebody is going to buy this and.This is going to be huge.[00:34:16] Micha: Yeah. Yeah. I think, I think the ambition was always there. Mm-hmm. . And, but, and, and sort of that drive to just make better developer tools. I think that sort of, that, you know, that's been true for all the companies or all too. Yeah, that's,[00:34:30] Michaela: Yeah. And what[00:34:32] Micha: I struggle. Yeah. Yeah. So I think, I think as I think for fiber plane now, it's not necessarily a struggle, it's just the real, which this mission of this flexible form factor, just the fact that we're doing sort of two startups at the same time has been sort of mm-hmm. An interesting thing to to build now, right? You're doing this rich, collaborative, rich tech editor and trying to build this infrastructure oriented company, and I think that's been yeah, just an interesting experience with building out a team.You know, the technology and the product that we.[00:35:01] Michaela: Yeah. Yeah. So maybe can you tell me a little bit more about again, if people want to hop over to Fiber plane now and try it out how does that work? Do you have to, you know is there a sign up? Is there a waiting list? I mean, you said probably when this airs there is a public beat, but still do you have to, you know, what do you have to reach out to you, you give me a demo or I just fill in my credentials and I'm off togo.[00:35:25] Micha: you can just sign, sign up with Google and then you're off to the races. And then of course, if you want a demo and sort of get some more, more more help or onboarding we're happy to help you and get on a call and walk you through it. But yeah. Okay, cool. Try playing com. Is there[00:35:40] Michaela: also a, Yeah, is there a video or something that we can look[00:35:44] Micha: at?Yes. The, the website and there's a video.[00:35:50] Michaela: Okay. I will link that so that people can go Yeah. And it will explain everything to them. Right. What about pricing? Whatever pricing? Yeah. You have already some idea around pricing. Yeah.[00:36:01] Micha: We've got some ideas on how to charge, but I think right now for us, it's important to get the product market fit, mm-hmm.and as such, you know, get, get the feedback. From these companies and these teams using the product. So we'll introduce pricing at a later stage. So for now it's, it's free to use, mm-hmm. . And you just give us your time and your feedback, and then Yeah, we're grateful.[00:36:20] Michaela: Yeah. And what about my data?Is it safe with you? Like, do you have some visibility into my data or do I send it over to[00:36:29] Micha: you? Yeah, so we actually so the way the, the providers work the plugins, so they actually get activated through a proxy. So we install a proxy inside of your cluster. The proxy sets up a secure bidirectional tunnel from your infrastructure to the fiber plane managed service.And then we do, for that specific query, we do store the data that's related to that query. So of a result, we do store that in the notebook. And yeah, we probably will come up with sort of more enterprisey ideas around how to self host[00:36:59] Michaela: it, Right? Or something[00:37:01] Micha: as an example. Yeah, yeah, yeah. But again, we'd love to get some feedback on that.[00:37:07] Michaela: How that works. Right? Yeah. Okay, cool. So yeah, that sounds really good. I think you, at least my questions, , you could answer them all, but maybe my listeners have questions and then they can send them to you. I think you will be, Yeah. Quite happy, right?[00:37:22] Micha: A hundred percent. At mes on Twitter, m i e s and at fiber, playing on Twitter, fiber playing.com.Sign up, take it for spin, shoot us a message. Yeah, sounds.[00:37:33] Michaela: Yeah. Yeah, it sounds super interesting. I hope that a lot of my listeners will do that, and I will link everything in my show notes that we, you know, talked about your, your Twitter handle and everything so that people can reach you. And I hope you get a lot of questions and people give it a spin and give it a try and send you their use cases,And yeah. I hope you all the best with your product. Thank you so much for being on my show today Micha. And yeah. Thank you. Bye.[00:37:59] Micha: Thank. Thank you for having me.[00:38:01] Michaela: Yeah, it was really great. Bye bye[00:38:04] Micha: bye.[00:38:06] Michaela: This was another episode of the Software Engineering Unlocked Podcast. If you enjoyed the episode, please help me spread the word about the podcast.Send episode to a friend via email, Twitter, LinkedIn. Well, whatever messaging system you use, or give it a positive review on your favorite podcasting platform such as Spotify or iTune. This would mean really a lot to me. So thank you for listening. Don't forget to subscribe and I will talk to you in two weeks.

Leonard S. Johnson is the Founder and CEO of AIEDC, a 5G Cloud Mobile App Maker and Service Provider with Machine Learning to help small and midsize businesses create their own iOS and Android mobile apps with no-code or low-code so they can engage and service their customer base, as well as provide front and back office digitization services for small businesses. Victoria talks to Leonard about using artificial intelligence for good, bringing the power of AI to local economics, and truly democratizing AI. The Artificial Intelligence Economic Development Corporation (AIEDC) (https://netcapital.com/companies/aiedc) Follow AIEDC on Twitter (https://twitter.com/netcapital), Instagram (https://www.instagram.com/netcapital/), Facebook (https://www.facebook.com/Netcapital/), or LinkedIn (https://www.linkedin.com/company/aiedc/). Follow Leonard on Twitter (https://twitter.com/LeonardSJ) and LinkedIn (https://www.linkedin.com/in/leonardsjohnson84047/). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is The Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with us today is Leonard S. Johnson or LS, Founder and CEO AIEDC, a 5G Cloud Mobile App Maker and Service Provider with Machine Learning to help small and midsize businesses create their own iOS and Android mobile apps with no-code or low-code so they can engage and service their customer base, as well as provide front and back office digitization services for small businesses. Leonard, thanks for being with us today. LEONARD: Thank you for having me, Victoria. VICTORIA: I should say LS, thank you for being with us today. LEONARD: It's okay. It's fine. VICTORIA: Great. So tell us a little more about AIEDC. LEONARD: Well, AIEDC stands for Artificial Intelligence Economic Development Corporation. And the original premise that I founded it for...I founded it after completing my postgraduate work at Stanford, and that was 2016. And it was to use AI for economic development, and therefore use AI for good versus just hearing about artificial intelligence and some of the different movies that either take over the world, and Skynet, and watch data privacy, and these other things which are true, and it's very evident, they exist, and they're out there. But at the end of the day, I've always looked at life as a growth strategy and the improvement of what we could do and focusing on what we could do practically. You do it tactically, then you do it strategically over time, and you're able to implement things. That's why I think we keep building collectively as humanity, no matter what part of the world you're in. VICTORIA: Right. So you went to Stanford, and you're from South Central LA. And what about that background led you to pursue AI for good in particular? LEONARD: So growing up in the inner city of Los Angeles, you know, that South Central area, Compton area, it taught me a lot. And then after that, after I completed high school...and not in South Central because I moved around a lot. I grew up with a single mother, never knew my real father, and then my home life with my single mother wasn't good because of just circumstances all the time. And so I just started understanding that even as a young kid, you put your brain...you utilize something because you had two choices. It's very simple or binary, you know, A or B. A, you do something with yourself, or B, you go out and be social in a certain neighborhood. And I'm African American, so high probability that you'll end up dead, or in a gang, or in crime because that's what it was at that time. It's just that's just a situation. Or you're able to challenge those energies and put them toward a use that's productive and positive for yourself, and that's what I did, which is utilizing a way to learn. I could always pick up things when I was very young. And a lot of teachers, my younger teachers, were like, "You're very, very bright," or "You're very smart." And there weren't many programs because I'm older than 42. So there weren't as many programs as there are today. So I really like all of the programs. So I want to clarify the context. Today there's a lot more engagement and identification of kids that might be sharper, smarter, whatever their personal issues are, good or bad. And it's a way to sort of separate them. So you're not just teaching the whole group as a whole and putting them all in one basket, but back then, there was not. And so I just used to go home a lot, do a lot of reading, do a lot of studying, and just knick-knack with things in tech. And then I just started understanding that even as a young kid in the inner city, you see economics very early, but they don't understand that's really what they're studying. They see economics. They can see inflation because making two ends meet is very difficult. They may see gang violence and drugs or whatever it might end up being. And a lot of that, in my opinion, is always an underlining economic foundation. And so people would say, "Oh, why is this industry like this?" And so forth. "Why does this keep happening?" It's because they can't function. And sometimes, it's just them and their family, but they can't function because it's an economic system. So I started focusing on that and then went into the Marine Corps. And then, after the Marine Corps, I went to Europe. I lived in Europe for a while to do my undergrad studies in the Netherlands in Holland. VICTORIA: So having that experience of taking a challenge or taking these forces around you and turning into a force for good, that's led you to bring the power of AI to local economics. And is that the direction that you went eventually? LEONARD: So economics was always something that I understood and had a fascination prior to even starting my company. I started in 2017. And we're crowdfunding now, and I can get into that later. But I self-funded it since 2017 to...I think I only started crowdfunding when COVID hit, which was 2020, and just to get awareness and people out there because I couldn't go to a lot of events. So I'm like, okay, how can I get exposure? But yeah, it was a matter of looking at it from that standpoint of economics always factored into me, even when I was in the military when I was in the Marine Corps. I would see that...we would go to different countries, and you could just see the difference of how they lived and survived. And another side note, my son's mother is from Ethiopia, Africa. And I have a good relationship with my son and his mother, even though we've been apart for over 15 years, divorced for over 15 years or so or longer. But trying to keep that, you can just see this dichotomy. You go out to these different countries, and even in the military, it's just so extreme from the U.S. and any part of the U.S, but that then always focused on economics. And then technology, I just always kept up with, like, back in the '80s when the mobile brick phone came out, I had to figure out how to get one. [laughs] And then I took it apart and then put it back together just to see how it works, so yeah. But it was a huge one, by the way. I mean, it was like someone got another and broke it, and they thought it was broken. And they're like, "This doesn't work. You could take this piece of junk." I'm like, "Okay." [laughs] VICTORIA: Like, oh, great. I sure will, yeah. Now, I love technology. And I think a lot of people perceive artificial intelligence as being this super futuristic, potentially harmful, maybe economic negative impact. So what, from your perspective, can AI do for local economics or for people who may not have access to that advanced technology? LEONARD: Well, that's the key, and that's what we're looking to do with AIEDC. When you look at the small and midsize businesses, it's not what people think, or their perception is. A lot of those in the U.S. it's the backbone of the United States, our economy, literally. And in other parts of the world, it's the same where it could be a one or two mom-and-pop shops. That's where that name comes from; it's literally two people. And they're trying to start something to build their own life over time because they're using their labor to maybe build wealth or somehow a little bit not. And when I mean wealth, it's always relative. It's enough to sustain themselves or just put food on the table and be able to control their own destiny to the best of their ability. And so what we're looking to do is make a mobile app maker that's 5G that lives in the cloud, that's 5G compliant, that will allow small and midsize businesses to create their own iOS or Android mobile app with no-code or low-code, basically like creating an email. That's how simple we want it to be. When you create your own email, whether you use Microsoft, Google, or whatever you do, and you make it that simple. And there's a simple version, and there could be complexity added to it if they want. That would be the back office digitization or customization, but that then gets them on board with digitization. It's intriguing that McKinsey just came out with a report stating that in 2023, in order to be economically viable, and this was very recent, that all companies would need to have a digitization strategy. And so when you look at small businesses, and you look at things like COVID-19, or the COVID current ongoing issue and that disruption, this is global. And you look at even the Ukrainian War or the Russian-Ukrainian War, however you term it, invasion, war, special operation, these are disruptions. And then, on top of that, we look at climate change which has been accelerating in the last two years more so than it was prior to this that we've experienced. So this is something that everyone can see is self-evident. I'm not even focused on the cause of the problem. My brain and the way I think, and my team, we like to focus on solutions. My chairman is a former program director of NASA who managed 1,200 engineers that built the International Space Station; what was it? 20-30 years ago, however, that is. And he helped lead and build that from Johnson Center. And so you're focused on solutions because if you're building the International Space Station, you can only focus on solutions and anticipate the problems but not dwell on them. And so that kind of mindset is what I am, and it's looking to help small businesses do that to get them on board with digitization and then in customization. And then beyond that, use our system, which is called M.I.N.D. So we own these...we own patents, three patents, trademarks, and service marks related to artificial intelligence that are in the field of economics. And we will utilize DEVS...we plan to do that which is a suite of system specifications to predict regional economic issues like the weather in a proactive way, not reactive. A lot of economic situations are reactive. It's reactive to the Federal Reserve raising interest rates or lowering rates, Wall Street, you know, moving money or not moving money. It is what it is. I mean, I don't judge it. I think it's like financial engineering, and that's fine. It's profitability. But then, at the end of the day, if you're building something, it's like when we're going to go to space. When rockets launch, they have to do what they're intended to do. Like, I know that Blue Origin just blew up recently. Or if they don't, they have a default, and at least I heard that the Blue Origin satellite, if it were carrying passengers, the passengers would have been safe because it disembarked when it detected its own problem. So when you anticipate these kinds of problems and you apply them to the local small business person, you can help them forecast and predict better like what weather prediction has done. And we're always improving that collectively for weather prediction, especially with climate change, so that it can get to near real-time as soon as possible or close a window versus two weeks out versus two days out as an example. VICTORIA: Right. Those examples of what you call a narrow economic prediction. LEONARD: Correct. It is intriguing when you say narrow economic because it wouldn't be narrow AI. But it would actually get into AGI if you added more variables, which we would. The more variables you added in tenancies...so if you're looking at events, the system events discretion so discrete event system specification you would specify what they really, really need to do to have those variables. But at some point, you're working on a system, what I would call AGI. But AGI, in my mind, the circles I run in at least or at least most of the scientists I talk to it's not artificial superintelligence. And so the general public thinks AGI...and I've said this to Stephen Ibaraki, who's the founder of AI for Good at Global Summit at the United Nations, and one of his interviews as well. It's just Artificial General Intelligence, I think, has been put out a lot by Hollywood and entertainment and so forth, and some scientists say certain things. We won't be at artificial superintelligence. We might get to Artificial General Intelligence by 2030 easily, in my opinion. But that will be narrow AI, but it will cover what we look at it in the field as cross-domain, teaching a system to look at different variables because right now, it's really narrow. Like natural language processing, it's just going to look at language and infer from there, and then you've got backward propagation that's credit assignment and fraud and detection. Those are narrow data points. But when you start looking at something cross-domain...who am I thinking of? Pedro Domingos who wrote the Master Algorithm, which actually, Xi Jinping has a copy of, the President of China, on his bookshelf in his office because they've talked about that, and these great minds because Stephen Ibaraki has interviewed these...and the founder of Google Brain and all of these guys. And so there's always this debate in the scientific community of what is narrow AI what it's not. But at the end of the day, I just like Pedro's definition of it because he says the master algorithm will be combining all five, so you're really crossing domains, which AI hasn't done that. And to me, that will be AGI, but that's not artificial superintelligence. And artificial superintelligence is when it becomes very, you know, like some of the movies could say, if we as humanity just let it run wild, it could be crazy. VICTORIA: One of my questions is the future of AI more like iRobot or Bicentennial Man? LEONARD: Well, you know, interesting. That's a great question, Victoria. I see most of AI literally as iRobot, as a tool more than anything, except at the end when it implied...so it kind of did two things in that movie, but a wonderful movie to bring up. And I like Will Smith perfectly. Well, I liked him a lot more before -- VICTORIA: I think iRobot is really the better movie. LEONARD: Yeah, so if people haven't seen iRobot, I liked Will Smith, the actor. But iRobot showed you two things, and it showed you, one, it showed hope. Literally, the robot...because a lot of people put AI and robots. And AI by itself is the brain or the mind; I should say hardware are the robots or the brain. Software...AI in and of itself is software. It's the mind itself. That's why we have M.I.N.D Machine Intelligence NeuralNetwork Database. We literally have that. That's our acronym and our slogan and everything. And it's part of our patents. But its machine intelligence is M.I.N.D, and we own that, you know; the company owns it. And so M.I.N.D...we always say AI powered by M.I.N.D. We're talking about that software side of, like, what your mind does; it iterates and thinks, the ability to think itself. Now it's enclosed within a structure called, you know, for the human, it's called a brain, the physical part of it, and that brain is enclosed within the body. So when you look at robots...and my chairman was the key person for robotics for the International Space Station. So when you look at robotics, you are putting that software into hardware, just like your cell phone. You have the physical, and then you have the actual iOS, which is the operating system. So when you think about that, yeah, iRobot was good because it showed how these can be tools, and they were very, in the beginning of the movie, very helpful, very beneficial to humanity. But then it went to a darker side and showed where V.I.K.I, which was an acronym as well, I think was Virtual Interactive Kinetic technology of something. Yeah, I believe it was Virtual Interactive Kinetic inference or technology or something like that, V.I.K.I; I forgot the last I. But that's what it stood for. It was an acronym to say...and then V.I.K.I just became all aware and started killing everyone with robots and just wanted to say, you know, this is futile. But then, at the very, very end, V.I.K.I learned from itself and says, "Okay, I guess this isn't right." Or the other robot who could think differently argued with V.I.K.I, and they destroyed her. And it made V.I.K.I a woman in the movie, and then the robot was the guy. But that shows that it can get out of hand. But it was intriguing to me that they had her contained within one building. This wouldn't be artificial superintelligence. And I think sometimes Hollywood says, "Just take over everything from one building," no. It wouldn't be on earth if it could. But that is something we always have to think about. We have to think about the worst-case scenarios. I think every prudent scientist or business person or anyone should do that, even investors, I mean, if you're investing something for the future. But you also don't focus on it. You don't think about the best-case scenario, either. But there's a lot of dwelling on the worst-case scenario versus the good that we can do given we're looking at where humanity is today. I mean, we're in 2022, and we're still fighting wars that we fought in 1914. VICTORIA: Right. Which brings me to my next question, which is both, what are the most exciting opportunities to innovate in the AI space currently? And conversely, what are the biggest challenges that are facing innovation in that field? LEONARD: Ooh, that's a good question. I think, in my opinion, it's almost the same answer; one is...but I'm in a special field. And I'm surprised there's not a lot of competition for our company. I mean, it's very good for me and the company's sense. It's like when Mark Zuckerberg did Facebook, there was Friendster, and there was Myspace, but they were different. They were different verticals. And I think Mark figured out how to do it horizontally, good or bad. I'm talking about the beginning of when he started Facebook, now called Meta. But I'm saying utilizing AI in economics because a lot of times AI is used in FinTech and consumerism, but not economic growth where we're really talking about growing something organically, or it's called endogenous growth. Because I studied Paul Romer's work, who won the Nobel Prize in 2018 for economic science. And he talked about the nature of ideas. And we were working on something like that in Stanford. And I put out a book in 2017 of January talking about cryptocurrencies, artificial intelligence but about the utilization of it, but not the speculation. I never talked about speculation. I don't own any crypto; I would not. It's only once it's utilized in its PureTech form will it create something that it was envisioned to do by the protocol that Satoshi Nakamoto sort of created. And it still fascinates me that people follow Bitcoin protocol, even for the tech and the non-tech, but they don't know who Satoshi is. But yeah, it's a white paper. You're just following a white paper because I think logically, the world is going towards that iteration of evolution. And that's how AI could be utilized for good in an area to focus on it with economics and solving current problems. And then going forward to build a new economy where it's not debt-based driven or consumer purchase only because that leaves a natural imbalance in the current world structure. The western countries are great. We do okay, and we go up and down. But the emerging and developing countries just get stuck, and they seem to go into a circular loop. And then there are wars as a result of these things and territory fights and so forth. So that's an area I think where it could be more advanced is AI in the economic realm, not so much the consumer FinTech room, which is fine. But consumer FinTech, in my mind, is you're using AI to process PayPal. That's where I think Elon just iterated later because PayPal is using it for finance. You're just moving things back and forth, and you're just authenticating everything. But then he starts going on to SpaceX next because he's like, well, let me use technology in a different way. And I do think he's using AI on all of his projects now. VICTORIA: Right. So how can that tech solve real problems today? Do you see anything even particular about Southern California, where we're both at right now, where you think AI could help predict some outcomes for small businesses or that community? LEONARD: I'm looking to do it regionally then globally. So I'm part of this Southern Cal Innovation Hub, which is just AI. It's an artificial intelligence coordination between literally San Diego County, Orange County, and Los Angeles County. And so there's a SoCal Innovation Hub that's kind of bringing it together. But there are all three groups, like; I think the CEO in Orange County is the CEO of Leadership Alliance. And then in San Diego, there's another group I can't remember their name off the top of my head, and I'm talking about the county itself. So each one's representing a county because, you know. And then there's one in Northern California that I'm also associated with where if you look at California as its own economy in the U.S., it's still pretty significant as an economic cycle in the United States, period. That's why so many politicians like California because they can sway the votes. So yeah, we're looking to do that once, you know, we are raising capital. We're crowdfunding currently. Our total raise is about 6 million. And so we're talking to venture capitalists, private, high net worth investors as well. Our federal funding is smaller. It's just like several hundred thousand because most people can only invest a few thousand. But I always like to try to give back. If you tell people...if you're Steve Jobs, like, okay, I've got this Apple company. In several years, you'll see the potential. And people are like, ah, whatever, but then they kick themselves 15 years later. [laughs] Like, oh, I wish I thought about that Apple stock for $15 when I could. But you give people a chance, and you get the word out, and you see what happens. Once you build a system, you share it. There are some open-source projects. But I think the open source, like OpenAI, as an example, Elon Musk funds that as well as Microsoft. They both put a billion dollars into it. It is an open-source project. OpenAI claims...but some of the research does go back to Microsoft to be able to see it. And DeepMind is another research for AI, but they're owned by Google. And so, I'm also very focused on democratizing artificial intelligence for the benefit of everyone. I really believe that needs to be democratized in a sense of tying it to economics and making it utilized for everyone that may need it for the benefit of humanity where it's profitable and makes money, but it's not just usurping. MID-ROLL AD: As life moves online, brick-and-mortar businesses are having to adapt to survive. With over 18 years of experience building reliable web products and services, thoughtbot is the technology partner you can trust. We provide the technical expertise to enable your business to adapt and thrive in a changing environment. We start by understanding what's important to your customers to help you transition to intuitive digital services your customers will trust. We take the time to understand what makes your business great and work fast yet thoroughly to build, test, and validate ideas, helping you discover new customers. Take your business online with design‑driven digital acceleration. Find out more at tbot.io/acceleration or click the link in the show notes for this episode. VICTORIA: With that democratizing it, is there also a need to increase the understanding of the ethics around it and when there are certain known use cases for AI where it actually is discriminatory and plays to systemic problems in our society? Are you familiar with that as well? LEONARD: Yes, absolutely. Well, that's my whole point. And, Victoria, you just hit the nail on the head. Truly democratizing AI in my mind and in my brain the way it works is it has opened up for everyone. Because if you really roll it back, okay, companies now we're learning...we used to call it several years ago UGC, User Generated Content. And now a lot of people are like, okay, if you're on Facebook, you're the product, right? Or if you're on Instagram, you're the product. And they're using you, and you're using your data to sell, et cetera, et cetera. But user-generated content it's always been that. It's just a matter of the sharing of the economic. That's why I keep going back to economics. So if people were, you know, you wouldn't have to necessarily do advertising if you had stakeholders with advertising, the users and the company, as an example. If it's a social media company, just throwing it out there, so let's say you have a social media...and this has been talked about, but I'm not the first to introduce this. This has been talked about for over ten years, at least over 15 years. And it's you share as a triangle in three ways. So you have the user and everything else. So take your current social media, and I won't pick on Facebook, but I'll just use them, Facebook, Instagram, or Twitter. Twitter's having issues recently because Elon is trying to buy them or get out of buying them. But you just looked at that data, and then you share with the user base. What's the revenue model? And there needs to be one; let me be very clear. There has to be incentive, and there has to be profitability for people that joined you earlier, you know, joined the corporation, or become shareholders, or investors, or become users, or become customers. They have to be able to have some benefit, not extreme greater than everyone else but a great benefit from coming in earlier by what they contributed at the time. And that is what makes this system holistic in my opinion, like Reddit or any of these bloggers. But you make it where they use their time and the users, and you share it with the company and then the data and so forth, and whatever revenue economic model you have, and it's a sort of a three-way split. It's just not always equal. And that's something that I think in economics, we're still on a zero-sum game, I win, you lose sort of economic model globally. That's why there's a winner of a war and a loser of a war. But in reality, as you know, Victoria, there are no winners of any war. So it's funny, [laughs] I was just saying, well, you know, because of the economic mode, but Von Neumann, who talked about that, also talked about something called a non-zero-sum game when he talked about it in mathematics that you can win, and I can win; we just don't win equally because they never will match that. So if I win, I may win 60; you win 40. Or you may win 60, I win 40, and we agree to settle on that. It's an agreement versus I'm just going to be 99, and you'll be 1%, or I'm just going to be 100, and you're at 0. And I think that our economic model tends to be a lot of that, like, when you push forth and there needs to be more of that. When you talk about the core of economics...and I go way back, you know, prior to the Federal Reserve even being started. I just look at the world, and it's always sort of been this land territorial issue of what goods are under the country. But we've got technology where we can mitigate a lot of things and do the collective of help the earth, and then let's go off to space, all of space. That's where my brain is focused on. VICTORIA: Hmm. Oh yeah, that makes sense to me. I think that we're all going to have to evolve our economic models here in the future. I wonder, too, as you're building your startup and you're building your company, what are some of the technology trade-offs you're having to make in the stack of the AI software that you're building? LEONARD: Hmm. Good question. But clarify, this may be a lot deeper dive because that's a general question. And I don't want to...yeah, go ahead. VICTORIA: Because when you're building AI, and you're going to be processing a lot of data, I know many data scientists that are familiar with tools like Jupyter Notebooks, and R, and Python. And one issue that I'm aware of is keeping the environments the same, so everything that goes into building your app and having those infrastructure as code for your data science applications, being able to afford to process all that data. [laughs] And there are just so many factors that go into building an AI app versus building something that's more easy, like a web-based user form. So just curious if you've encountered those types of trade-offs or questions about, okay, how are we going to actually build an app that we can put out on everybody's phone and that works responsibly? LEONARD: Oh, okay. So let me be very clear, but I won't give too much of the secret sauce away. But I can define this technically because this is a technical audience. This is not...so what you're really talking about is two things, and I'm clear about this, though. So the app maker won't really read and write a lot of data. It'll just be the app where people could just get on board digitalization simple, you know, process payments, maybe connect with someone like American Express square, MasterCard, whatever. And so that's just letting them function. That's sort of small FinTech in my mind, you know, just transaction A to B, B to A, et cetera. And it doesn't need to be peer-to-peer and all of the crypto. It doesn't even need to go that level yet. That's just level one. Then they will sign up for a service, which is because we're really focused on artificial intelligence as a service. And that, to me, is the next iteration for AI. I've been talking about this for about three or four years now, literally, in different conferences and so forth for people who haven't hit it. But that we will get to that point where AI will become AI as a service, just like SaaS is. We're still at the, you know, most of the world on the legacy systems are still software as a service. We're about to hit AI as a service because the world is evolving. And this is true; they did shut it down. But you did have okay, so there are two case points which I can bring up. So JP Morgan did create something called a Coin, and it was using AI. And it was a coin like crypto, coin like a token, but they called it a coin. But it could process, I think, something like...I may be off on this, so to the sticklers that will be listening, please, I'm telling you I may be off on the exact quote, but I think it was about...it was something crazy to me, like 200,000 of legal hours and seconds that it could process because it was basically taking the corporate legal structure of JP Morgan, one of the biggest banks. I think they are the biggest bank in the U.S. JPMorgan Chase. And they were explaining in 2017 how we created this, and it's going to alleviate this many hours of legal work for the bank. And I think politically; something happened because they just pulled away. I still have the original press release when they put it out, and it was in the media. And then it went away. I mean, no implementation [laughs] because I think there was going to be a big loss of jobs for it. And they basically would have been white-collar legal jobs, most specifically lawyers literally that were working for the bank. And when they were talking towards investment, it was a committee. I was at a conference. And I was like, I was fascinated by that. And they were basically using Bitcoin protocol as the tokenization protocol, but they were using AI to process it. And it was basically looking at...because legal contracts are basically...you can teach it with natural language processing and be able to encode and almost output it itself and then be able to speak with each other. Another case point was Facebook. They had...what was it? Two AI systems. They began to create their own language. I don't know if you remember that story or heard about it, and Facebook shut it down. And this was more like two years ago, I think, when they were saying Facebook was talking, you know, when they were Facebook, not Meta, so maybe it was three years ago. And they were talking, and they were like, "Oh, Facebook has a language. It's talking to each other." And it created its own little site language because it was two AI bots going back and forth. And then the engineers at Facebook said, "We got to shut this down because this is kind of getting out of the box." So when you talk about AI as a service, yes, the good and the bad, and what you take away is AWS, Oracle, Google Cloud they do have services where it doesn't need to cost you as much anymore as it used to in the beginning if you know what you're doing ahead of time. And you're not just running iterations or data processing because you're doing guesswork versus, in my opinion, versus actually knowing exactly specifically what you're looking for and the data set you're looking to get out of it. And then you're talking about just basically putting in containers and clustering it because it gets different operations. And so what you're really looking at is something called an N-scale graph data that can process data in maybe sub seconds at that level, excuse me. And one of my advisors is the head of that anyway at AGI laboratory. So he's got an N graph database that can process...when we implement it, we'll be able to process data at the petabyte level at sub-seconds, and it can run on platforms like Azure or AWS, and so forth. VICTORIA: Oh, that's interesting. So it sounds like cloud providers are making compute services more affordable. You've got data, the N-scale graph data, that can run more transactions more quickly. And I'm curious if you see any future trends since I know you're a futurist around quantum computing and how that could affect capacity for -- LEONARD: Oh [laughs] We haven't even gotten there yet. Yes. Well, if you look at N-scale, if you know what you're doing and you know what to look for, then the quantum just starts going across different domains as well but at a higher hit rate. So there's been some quantum computers online. There's been several...well, Google has their quantum computer coming online, and they've been working on it, and Google has enough data, of course, to process. So yeah, they've got that data, lots of data. And quantum needs, you know, if it's going to do something, it needs lots of data. But then the inference will still be, I think, quantum is very good at processing large, large, large amounts of data. We can just keep going if you really have a good quantum computer. But it's really narrow. You have to tell it exactly what it wants, and it will do it in what we call...which is great like in P or NP square or P over NP which is you want to do it in polynomial time, not non-polynomial, polynomial time which is...now speaking too fast. Okay, my brain is going faster than my lips. Let me slow it down. So when you start thinking about processing, if we as humans, let's say if I was going to process A to Z, and I'm like, okay, here is this equation, if I tell you it takes 1000 years, it's of no use to us, to me and you Victoria because we're living now. Now, the earth may benefit in 1000 years, but it's still of no use. But if I could take this large amount of data and have it process within minutes, you know, worst case hours...but then I'll even go down to seconds or sub-seconds, then that's really a benefit to humanity now, today in present term. And so, as a futurist, yes, as the world, we will continue to add data. We're doing it every day, and we already knew this was coming ten years ago, 15 years ago, 20 years ago, even actually in the '50s when we were in the AI winter. We're now in AI summer. In my words, I call it the AI summer. So as you're doing this, that data is going to continue to increase, and quantum will be needed for that. But then the specific need...quantum is very good at looking at a specific issue, specifically for that very narrow. Like if you were going to do the trajectory to Jupiter or if we wanted to send a probe to Jupiter or something, I think we're sending something out there now from NASA, and so forth, then you need to process all the variables, but it's got one trajectory. It's going one place only. VICTORIA: Gotcha. Well, that's so interesting. I'm glad I asked you that question. And speaking of rockets going off to space, have you ever seen a SpaceX launch from LA? LEONARD: Actually, I saw one land but not a launch. I need to go over there. It's not too far from me. But you got to give credit where credit's due and Elon has a reusable rocket. See, that's where technology is solving real-world problems. Because NASA and I have, you know, my chairman, his name is Alexander Nawrocki, you know, he's Ph.D., but I call him Rocki. He goes by Rocki like I go by LS. But it's just we talk about this like NASA's budget. [laughs] How can you reduce this? And Elon says they will come up with a reusable rocket that won't cost this much and be able to...and that's the key. That was the kind of Holy Grail where you can reuse the same rocket itself and then add some little variables on top of it. But the core, you wouldn't constantly be paying for it. And so I think where the world is going...and let me be clear, Elon pushes a lot out there. He's just very good at it. But I'm also that kind of guy that I know that Tesla itself was started by two Stanford engineers. Elon came on later, like six months, and then he invested, and he became CEO, which was a great investment for Elon Musk. And then CEO I just think it just fit his personality because it was something he loved. But I also have studied for years Nikola Tesla, and I understand what his contributions created where we are today with all the patents that he had. And so he's basically the father of WiFi and why we're able to communicate in a lot of this. We've perfected it or improved it, but it was created by him in the 1800s. VICTORIA: Right. And I don't think he came from as fortunate a background as Elon Musk, either. Sometimes I wonder what I could have done born in similar circumstances. [laughter] And you certainly have made quite a name for yourself. LEONARD: Well, I'm just saying, yeah, he came from very...he did come from a poor area of Russia which is called the Russian territory, to be very honest, Eastern Europe, definitely Eastern Europe. But yeah, I don't know once you start thinking about that [laughs]. You're making me laugh, Victoria. You're making me laugh. VICTORIA: No, I actually went camping, a backpacking trip to the Catalina Island, and there happened to be a SpaceX launch that night, and we thought it was aliens because it looked wild. I didn't realize what it was. But then we figured it was a launch, so it was really great. I love being here and being close to some of this technology and the advancements that are going on. I'm curious if you have some thoughts about...I hear a lot about or you used to hear about Silicon Valley Tech like very Northern California, San Francisco focus. But what is the difference in SoCal? What do you find in those two communities that makes SoCal special? [laughs] LEONARD: Well, I think it's actually...so democratizing AI. I've been in a moment like that because, in 2015, I was in Dubai, and they were talking about creating silicon oasis. And so there's always been this model of, you know, because they were always, you know, the whole Palo Alto thing is people would say it and it is true. I mean, I experienced it. Because I was in a two-year program, post-graduate program executive, but we would go up there...I wasn't living up there. I had to go there maybe once every month for like three weeks, every other month or something. But when you're up there, it is the air in the water. It's just like, people just breathe certain things. Because around the world, and I would travel to Japan, and China, and other different parts of Asia, Vietnam, et cetera and in Africa of course, and let's say you see this and people are like, so what is it about Silicon Valley? And of course, the show, there is the Hollywood show about it, which is pretty a lot accurate, which is interesting, the HBO show. But you would see that, and you would think, how are they able to just replicate this? And a lot of it is a convergence. By default, they hear about these companies' access because the key is access, and that's what we're...like this podcast. I love the concept around it because giving awareness, knowledge, and access allows other people to spread it and democratize it. So it's just not one physical location, or you have to be in that particular area only to benefit. I mean, you could benefit in that area, or you could benefit from any part of the world. But since they started, people would go there; engineers would go there. They built company PCs, et cetera. Now that's starting to spread in other areas like Southern Cal are creating their own innovation hubs to be able to bring all three together. And those three are the engineers and founders, and idea makers and startups. And you then need the expertise. I'm older than 42; I'm not 22. [laughs] So I'm just keeping it 100, keeping it real. So I'm not coming out at 19. I mean, my son's 18. And I'm not coming out, okay, this my new startup, bam, give me a billion dollars, I'm good. And let me just write off the next half. But when you look at that, there's that experience because even if you look at Mark Zuckerberg, I always tell people that give credit where credit is due. He brought a senior team with him when he was younger, and he didn't have the experience. And his only job has been Facebook out of college. He's had no other job. And now he's been CEO of a multi-billion dollar corporation; that's a fact. Sometimes it hurts people's feelings. Like, you know what? He's had no other job. Now that can be good and bad, [laughs] but he's had no other jobs. And so that's just a credit, like, if you can surround yourself with the right people and be focused on something, it can work to the good or the bad for your own personal success but then having that open architecture. And I think he's been trying to learn and others versus like an Elon Musk, who embraces everything. He's just very open in that sense. But then you have to come from these different backgrounds. But let's say Elon Musk, Mark Zuckerberg, let's take a guy like myself or whatever who didn't grow up with all of that who had to make these two ends meet, figure out how to do the next day, not just get to the next year, but get to the next day, get to the next week, get to the next month, then get to the next year. It just gives a different perspective as well. Humanity's always dealing with that. Because we had a lot of great engineers back in the early 1900s. They're good or bad, you know, you did have Nikola Tesla. You had Edison. I'm talking about circa around 1907 or 1909, prior to World War I. America had a lot of industries. They were the innovators then, even though there were innovations happening in Europe, and Africa, and China, as well and Asia. But the innovation hub kind of created as the America, quote, unquote, "industrial revolution." And I think we're about to begin a new revolution sort of tech and an industrial revolution that's going to take us to maybe from 20...we're 2022 now, but I'll say it takes us from 2020 to 2040 in my head. VICTORIA: So now that communities can really communicate across time zones and locations, maybe the hubs are more about solving specific problems. There are regional issues. That makes a lot more sense. LEONARD: Yes. And collaborating together, working together, because scientists, you know, COVID taught us that. People thought you had to be in a certain place, but then a lot of collaboration came out of COVID; even though it was bad globally, even though we're still bad, if people were at home, they start collaborating, and scientists will talk to scientists, you know, businesses, entrepreneurs, and so forth. But if Orange County is bringing together the mentors, the venture capital, or at least Southern California innovation and any other place, I want to say that's not just Silicon Valley because Silicon Valley already has it; we know that. And that's that region. It's San Jose all the way up to...I forgot how far north it's past San Francisco, actually. But it's that region of area where they encompass the real valley of Silicon Valley if you're really there. And you talk about these regions. Yes, I think we're going to get to a more regional growth area, and then it'll go more micro to actually cities later in the future. But regional growth, I think it's going to be extremely important globally in the very near term. I'm literally saying from tomorrow to the next, maybe ten years, regional will really matter. And then whatever you have can scale globally anyway, like this podcast we're doing. This can be distributed to anyone in the world, and they can listen at ease when they have time. VICTORIA: Yeah, I love it. It's both exciting and also intimidating. [laughs] And you mentioned your son a little bit earlier. And I'm curious, as a founder and someone who spent a good amount of time in graduate and Ph.D. programs, if you feel like it's easy to connect with your son and maintain that balance and focusing on your family while you're building a company and investing in yourself very heavily. LEONARD: Well, I'm older, [laughs] so it's okay. I mean, I've mentored him, you know. And me and his mom have a relationship that works. I would say we have a better relationship now than when we were together. It is what it is. But we have a communication level. And I think she was just a great person because I never knew my real father, ever. I supposedly met him when I was two or one; I don't know. But I have no memories, no photos, nothing. And that was just the environment I grew up in. But with my son, he knows the truth of everything about that. He's actually in college. I don't like to name the school because it's on the East Coast, and it's some Ivy League school; that's what I will say. And he didn't want to stay on the West Coast because I'm in Orange County and his mom's in Orange County. He's like, "I want to get away from both of you people." [laughter] And that's a joke, but he's very independent. He's doing well. When he graduated high school, he graduated with 4.8 honors. He made the valedictorian. He was at a STEM school. VICTORIA: Wow. LEONARD: And he has a high GPA. He's studying computer science and economics as well at an Ivy League, and he's already made two or three apps at college. And I said, "You're not Mark, so calm down." [laughter] But anyway, that was a recent conversation. I won't go there. But then some people say, "LS, you should be so happy." What is it? The apple doesn't fall far from the tree. But this was something he chose around 10 or 11. I'm like, whatever you want to do, you do; I'll support you no matter what. And his mom says, "Oh no, I think you programmed him to be like you." [laughs] I'm like, no, I can't do that. I just told him the truth about life. And he's pretty tall. VICTORIA: You must have -- LEONARD: He played basketball in high school a lot. I'm sorry? VICTORIA: I was going to say you must have inspired him. LEONARD: Yeah. Well, he's tall. He did emulate me in a lot of ways. I don't know why. I told him just be yourself. But yes, he does tell me I'm an inspiration to that; I think because of all the struggles I've gone through when I was younger. And you're always going through struggles. I mean, it's just who you are. I tell people, you know, you're building a company. You have success. You can see the future, but sometimes people can't see it, [laughs] which I shouldn't really say, but I'm saying anyway because I do that. I said this the other night to some friends. I said, "Oh, Jeff Bezo's rocket blew up," going, you know, Blue Origin rocket or something. And then I said Elon will tell Jeff, "Well, you only have one rocket blow up. I had three, [laughter] SpaceX had three." So these are billionaires talking to billionaires about, you know, most people don't even care. You're worth X hundred billion dollars. I mean, they're worth 100 billion-plus, right? VICTORIA: Right. LEONARD: I think Elon is around 260 billion, and Jeff is 160 or something. Who cares about your rocket blowing up? But it's funny because the issues are still always going to be there. I've learned that. I'm still learning. It doesn't matter how much wealth you have. You just want to create wealth for other people and better their lives. The more you search on bettering lives, you're just going to have to wake up every day, be humble with it, and treat it as a new day and go forward and solve the next crisis or problem because there will be one. There is not where there are no problems, is what I'm trying to say, this panacea or a utopia where you personally, like, oh yeah, I have all this wealth and health, and I'm just great. Because Elon has had divorce issues, so did Jeff Bezos. So I told my son a lot about this, like, you never get to this world where it's perfect in your head. You're always going to be doing things. VICTORIA: That sounds like an accurate future prediction if I ever heard one. [laughs] Like, there will be problems. No matter where you end up or what you choose to do, you'll still have problems. They'll just be different. [laughs] LEONARD: Yeah, and then this is for women and men. It means you don't give up. You just keep hope alive, and you keep going. And I believe personally in God, and I'm a scientist who actually does. But I look at it more in a Godly aspect. But yeah, I just think you just keep going, and you keep building because that's what we do as humanity. It's what we've done. It's why we're here. And we're standing on the shoulders of giants, and I just always considered that from physicists and everyone. VICTORIA: Great. And if people are interested in building something with you, you have that opportunity right now to invest via the crowdfunding app, correct? LEONARD: Yes, yes, yes. They can do that because the company is still the same company because eventually, we're going to branch out. My complete vision for AIEDC is using artificial intelligence for economic development, and that will spread horizontally, not just vertically. Vertically right now, just focus on just a mobile app maker digitization and get...because there are so many businesses even globally, and I'm not talking only e-commerce. So when I say small to midsize business, it can be a service business, car insurance, health insurance, anything. It doesn't have to be selling a particular widget or project, you know, product. And I'm not saying there's nothing wrong with that, you know, interest rates and consumerism. But I'm not thinking about Shopify, and that's fine, but I'm talking about small businesses. And there's the back office which is there are a lot of tools for back offices for small businesses. But I'm talking about they create their own mobile app more as a way to communicate with their customers, update them with their customers, and that's key, especially if there are disruptions. So let's say that there have been fires in California. In Mississippi or something, they're out of water. In Texas, last year, they had a winter storm, electricity went out. So all of these things are disruptions. This is just in the U.S., And of course, I won't even talk about Pakistan, what's going on there and the flooding and just all these devastating things, or even in China where there's drought where there are these disruptions, and that's not counting COVID disrupts, the cycle of business. It literally does. And it doesn't bubble up until later when maybe the central banks and governments pay attention to it, just like in Japan when that nuclear, unfortunately, that nuclear meltdown happened because of the earthquake; I think it was 2011. And that affected that economy for five years, which is why the government has lower interest rates, negative interest rates, because they have to try to get it back up. But if there are tools and everyone's using more mobile apps and wearables...and we're going to go to the metaverse and all of that. So the internet of things can help communicate that. So when these types of disruptions happen, the flow of business can continue, at least at a smaller level, for an affordable cost for the business. I'm not talking about absorbing costs because that's meaningless to me. VICTORIA: Yeah, well, that sounds like a really exciting project. And I'm so grateful to have this time to chat with you today. Is there anything else you want to leave for our listeners? LEONARD: If they want to get involved, maybe they can go to our crowdfunding page, or if they've got questions, ask about it and spread the word. Because I think sometimes, you know, they talk about the success of all these companies, but a lot of it starts with the founder...but not a founder. If you're talking about a startup, it starts with the founder. But it also stops with the innovators that are around that founder, male or female, whoever they are. And it also starts with their community, building a collective community together. And that's why Silicon Valley is always looked at around the world as this sort of test case of this is how you create something from nothing and make it worth great value in the future. And I think that's starting to really spread around the world, and more people are opening up to this. It's like the crowdfunding concept. I think it's a great idea, like more podcasts. I think this is a wonderful idea, podcasts in and of themselves, so people can learn from people versus where in the past you would only see an interview on the business news network, or NBC, or Fortune, or something like that, and that's all you would understand. But this is a way where organically things can grow. I think the growth will continue, and I think the future's bright. We just have to know that it takes work to get there. VICTORIA: That's great. Thank you so much for saying that and for sharing your time with us today. I learned a lot myself, and I think our listeners will enjoy it as well. You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, email us at hosts@giantrobot.fm. You can find me on Twitter @victori_ousg. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success. Special Guest: Leonard S. Johnson.

This week on the Data Futurology podcast, we have the special privilege to host Dr. David A. Bader, a Distinguished Professor at the New Jersey Institute of Technology, and the inaugural director of the Institute for Data Science there. Bader joins us on the podcast to discuss massive graph analytics, a topic that he is a recognised expert in and has recently published a book on. He and his team are currently working on a project that will allow anyone, via the Jupyter Notebook and Python, to leverage their data science framework, running on “tens of terabytes” of data. “It is quite exciting to democratise data science – and especially graph analytics – so that anyone with a problem that knows Python can work with some of the largest data sets,” he said. According to Bader, graphs are now a mainstream part of data science and a way to solve the most challenging and complex problems in the enterprise. “A graph abstracts relationships between objects, and any problem that we can abstract where we have relationships between objects, we could use graph analytics to solve,” he said. Much of Bader's work – including through his book – is focused on helping organisations grapple with the exponential growth in data, and the impact that this has on their ability to dedicate adequate resources to work at scale. As he said, being able to do that is going to be fundamental to humanity's ability to respond to the many real challenges that it faces ahead. “I want equitable access for everyone to be able to work on these problems, and to find new discoveries that are important, and help solve global grand challenges,” he said. “I think that we have many issues in the world today. And if we give more capabilities to those with data, and let them empower the data will make the world a much better place.” For more deep insights on the importance and value of massive graph analytics, tune in to our conversation with Dr. David A. Bader. Enjoy the show! Thank you to our sponsor, Talent Insights Group! Join us in Sydney for Ops World: https://www.datafuturology.com/opsworld Join our Slack Community: https://join.slack.com/t/datafuturologycircle/shared_invite/zt-z19cq4eq-ET6O49o2uySgvQWjM6a5ng Read the full podcast episode summary here. --- Send in a voice message: https://anchor.fm/datafuturology/message

Watch the live stream: Watch on YouTube About the show Sponsored by Complier Podcast from RedHat Michael #0: New livestream time - 11am PT on Tuesdays. Also, subscribe to the youtube channel and “hit the bell” to get notified of all the live streams. Brian #1: It's PyCon US 2023 CFP time Will be held in Salt Lake City, Salt Palace Convention Center Talks are Friday - Sunday, April 19-23 PyCon US 2023 launch announcement PyCon 2023 site features images taken from past PyCon artwork Call for proposals open until Dec 9, but please don't wait that long. Michael #2: Any.io AnyIO is an asynchronous networking and concurrency library that works on top of either asyncio or trio. It implements trio-like structured concurrency (SC) on top of asyncio. Cool interpretability between native threads and asyncio Using subprocesses: AnyIO allows you to run arbitrary executables in subprocesses, either as a one-shot call or by opening a process handle for you that gives you more control over the subprocess. Async file I/O: AnyIO provides asynchronous wrappers for blocking file operations. These wrappers run blocking operations in worker threads. Cool synchronization primitives too. Catch the Talk Python episode with Alex: talkpython.fm/385 Brian #3: How to propose a winning conference talk Reuven Lerner Some nice tips and advice Build a list of topics If you train, teach, mentor, lead, or coach already: what questions to people always ask you? what knowledge would help people to have? where do people seem to just “not get it”? If you don't train or teach, then maybe hit up Stack Overflow… From Brian: I think you can imagine yourself a year or two ago and think about stuff you know now you wish you knew then and could learn faster. Build an outline with times This part often seems scary, but Reuven's example is 10 bullets with (x min) notes. Write up a summary. One short, one longer. Indicate who will benefit, what they will come out knowing, and how it will help them. Propose to multiple conferences. Why not? Practice (from Brian: Even if you get rejected, you've gained. Turn it into a youTube video or blog post or both.) - Michael #4: Sanic release adds background workers via Felix In v22.9 (go cal-ver!), the main new feature is the worker process management - the main Sanic process handles a pool of workers. They are normally used for handling requests but you can also use them to handle background jobs and similar things. You could probably use it for a lot of the reasons people turn to something like Celery. The lead developer (Adam Hopkins) has written a blog post about this feature. MK: Sanic has been flying a bit under my radar. Maybe time to dive into it a bit more. Extras Brian: Create Presentation from Jupyter Notebook Cool walkthrough of how to use the built in slideshow features of Jupyter Notebooks. pytest 7.2.0 is out No longer depends on the py library. So if you do, you need to add it to your dependencies. nose officially deprecated, which includes setup() and teardown(). Really glad I dropped the “x unit” section on the 2nd edition of the pytest book. testpaths now supports shell-style wildcards Lots of other improvements. check out the change log Michael: Rich on pyscript (via Matt Kramer) Python 3.11 in 100 seconds video from Michael Joke: Deep questions & Relationship advice from geeks

About AlexAlex holds a Ph.D. in Computer Science and Engineering from UC San Diego, and has spent over a decade building high-performance, robust data management and processing systems. As an early member of a couple fast-growing startups, he's had the opportunity to wear a lot of different hats, serving at various times as an individual contributor, tech lead, manager, and executive. He also had a brief stint as a Cloud Economist with the Duckbill Group, helping AWS customers save money on their AWS bills. He's currently a freelance data engineering consultant, helping his clients build, manage, and maintain their data infrastructure. He lives in Los Angeles, CA.Links Referenced: Company website: https://bitsondisk.com Twitter: https://twitter.com/alexras LinkedIn: https://www.linkedin.com/in/alexras/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined this week by a returning guest, who… well, it's a little bit complicated and more than a little bittersweet. Alex Rasmussen was a principal cloud economist here at The Duckbill Group until he committed an unforgivable sin. That's right. He gave his notice. Alex, thank you for joining me here, and what have you been up to, traitor?Alex: [laugh]. Thank you for having me back, Corey.Corey: Of course.Alex: At time of recording, I am restarting my freelance data engineering business, which was dormant for the sadly brief time that I worked with you all at The Duckbill Group. And yeah, so that's really what I've been up to for the last few days. [laugh].Corey: I want to be very clear that I am being completely facetious when I say this. When someone is considering, “Well, am I doing what I really want to be doing?” And if the answer is no, too many days in a row, yeah, you should find something that aligns more with what you want to do. And anyone who's like, “Oh, you're leaving? Traitor, how could you do that?” Yeah, those people are trash. You don't want to work with trash.I feel I should clarify that this is entirely in jest and I could not be happier that you are finding things that are more aligned with aspects of what you want to be doing. I am serious when I say that, as a company, we are poorer for your loss. You have been transformative here across a number of different axes that we will be going into over the course of this episode.Alex: Well, thank you very much, I really appreciate that. And I came to a point where I realized, you know, the old saying, “You don't know what you got till it's gone?” I realized, after about six months of working with Duckbill Group that I missed building stuff, I missed building data systems, I missed being a full-time data person. And I'm really excited to get back to that work, even though I'll definitely miss working with everybody on the team. So yeah.Corey: There are a couple of things that I found really notable about your time working with us. One of them was that even when you wound up applying to work here, you were radically different than—well, let's be direct here—than me. We are almost polar opposites in a whole bunch of ways. I have an eighth-grade education; you have a PhD in computer science and engineering from UCSD. And you are super-deep into the world of data, start to finish, whereas I have spent my entire career on things that are stateless because I am accident prone, and when you accidentally have a problem with the database, you might not have a company anymore, but we can all laugh as we reprovision the web server fleet.We just went in very different directions as far as what we found interesting throughout our career, more or less. And we were not quite sure how it was going to manifest in the context of cloud economics. And I can say now that we have concluded the experiment, that from my perspective, it went phenomenally well. Because the exact areas that I am weak at are where you excel. And, on some level, I would say that you're not necessarily as weak in your weak areas as I am in mine, but we want to reinforce it and complementing each other rather than, “Well, we now have a roomful of four people who are all going to yell at you about the exact same thing.” We all went in different directions, which I thought was really neat.Alex: I did too. And honestly, I learned a tremendous, tremendous amount in my time at Duckbill Group. I think the window into just how complex and just how vast the ecosystem of services within AWS is, and kind of how they all ping off of each other in these very complicated ways was really fascinating, fascinating stuff. But also just an insight into just what it takes to get stuff done when you're talking with—you know, so most of my clientele to date have been small to medium-sized businesses, you know, small as two people; as big as a few hundred people. But I wasn't working with Fortune 1000 companies like Duckbill Group regularly does, and an insight into just, number one, what it takes to get things done inside of those organizations, but also what it takes to get things done with AWS when you're talking about, you know, for instance, contracts that are tens, or hundreds of millions of dollars in total contract value. And just what that involves was just completely eye-opening for me.Corey: From my perspective, what I found—I guess, in hindsight, it should have been more predictable than it was—but you talk about having a background and an abiding passion for the world of data, and I'm sitting here thinking, that's great. We have all this data in the form of the Cost and Usage Reports and the bills, and I forgot the old saw that yeah, if it fits in RAM, it's not a big data problem. And yeah, in most cases, what we have tends to fit in RAM. I guess you don't tend to find things interesting until Microsoft Excel gives up and calls uncle.Alex: I don't necessarily know that that's true. I think that there are plenty of problems to be had in the it fits in RAM space, precisely because so much of it fits in RAM. And I think that, you know, particularly now that, you know—I think there's it's a very different world that we live in from the world that we lived in ten years ago, where ten years ago—Corey: And right now I'm talking to you on a computer with 128 gigs of RAM, and it—Alex: Well, yeah.Corey: —that starts to look kind of big data-y.Alex: Well, not only that, but I think on the kind of big data side, right? When you had to provision your own Hadoop cluster, and after six months of weeping tears of blood, you managed to get it going, right, at the end of that process, you went, “Okay, I've got this big, expensive thing and I need this group of specialists to maintain it all. Now, what the hell do I do?” Right? In the intervening decade, largely due to the just crushing dominance of the public clouds, that problem—I wouldn't call that problem solved, but for all practical purposes, at all reasonable scales, there's a solution that you can just plug in a credit card and buy.And so, now the problem, I think, becomes much more high level, right, than it used to be. Used to be talking about how well you know, how do I make this MapReduce job as efficient as it possibly can be made? Nobody really cares about that anymore. You've got a query planner; it executes a query; it'll probably do better than you can. Now, I think the big challenges are starting to be more in the area of, again, “How do I know what I have? How do I know who's touched it recently? How do I fix it when it breaks? How do I even organize an organization that can work effectively with data at petabyte scale and say anything meaningful about it?”And so, you know, I think that the landscape is shifting. One of the reasons why I love this field so much is that the landscape is shifting very rapidly and as soon as we think, “Ah yes. We have solved all of the problems.” Then immediately, there are a hundred new problems to solve.Corey: For me, what I found, I guess, one of the most eye-opening things about having you here is your actual computer science background. Historically, we have biased for folks who have come up from the ops side of the world. And that lends itself to a certain understanding. And, yes, I've worked with developers before; believe it or not, I do understand how folks tend to think in that space. I have not a complete naive fool when it comes to these things.But what I wasn't prepared for was the nature of our internal, relatively casual conversations about a bunch of different things, where we'll be on a Zoom chat or something, and you will just very casually start sharing your screen, fire up a Jupyter Notebook and start writing code as you're talking to explain what it is you're talking about and watching it render in real time. And I'm sitting here going, “Huh, I can't figure out whether we should, like, wind up giving him a raise or try to burn him as a witch.” I could really see it going either way. Because it was magic and transformative from my perspective.Alex: Well, thank you. I mean, I think that part of what I am very grateful for is that I've had an opportunity to spend a considerable period of time in kind of both the academic and industrial spaces. I got a PhD, basically kept going to school until somebody told me that I had to stop, and then spent a lot of time at startups and had to do a lot of different kinds of work just to keep the wheels attached to the bus. And so, you know, when I arrived at Duckbill Group, I kind of looked around and said, “Okay, cool. There's all the stuff that's already here. That's awesome. What can I do to make that better?” And taking my lens so to speak, and applying it to those problems, and trying to figure out, like, “Okay, well as a cloud economist, what do I need to do right now that sucks? And how do I make it not suck?”Corey: It probably involves a Managed NAT Gateway.Alex: Whoa, God. And honestly, like, I spent a lot of time developing a bunch of different tools that were really just there in the service of that. Like, take my job, make it easier. And I'm really glad that you liked what you saw there.Corey: It was interesting watching how we wound up working together on things. Like, there's a blog post that I believe is out by the time this winds up getting published—but if not, congratulations on listening to this, you get a sneak preview—where I was looking at the intelligent tiering changes in pricing, where any object below 128 kilobytes does not have a monitoring charge attached to it, and above it, it does. And it occurred to me on a baseline gut level that, well wait a minute, it feels like there is some object sizes, where regardless of how long it lives in storage and transition to something cheaper, it will never quite offset that fee. So, instead of having intelligent tiering for everything, that there's some cut-off point below which you should not enable intelligent tiering because it will always cost you more than it can possibly save you.And I mentioned that to you and I had to do a lot of articulating with my hands because it's all gut feelings stuff and this stuff is complicated at the best of times. And your response was, “Huh.” Then it felt like ten minutes later you came back with a multi-page blog post written—again—in a Python notebook that has a dynamic interactive graph that shows the breakeven and cut-off points, a deep dive math showing exactly where in certain scenarios it is. And I believe the final takeaway was somewhere between 148 to 161 kilobytes, somewhere in that range is where you want to draw the cut-off. And I'm just looking at this and marveling, on some level.Alex: Oh, thanks. To be fair, it took a little bit more than ten minutes. I think it was something where it kind of went through a couple of stages where at first I was like, “Well, I bet I could model that.” And then I'm like, “Well, wait a minute. There's actually, like—if you can kind of put the compute side of this all the way to the side and just remove all API calls, it's a closed form thing. Like, you can just—this is math. I can just describe this with math.”And cue the, like, Beautiful Mind montage where I'm, like, going onto the whiteboard and writing a bunch of stuff down trying to remember the point intercept form of a line from my high school algebra days. And at the end, we had that blog post. And the reason why I kind of dove into that headfirst was just this, I have this fascination for understanding how all this stuff fits together, right? I think so often, what you see is a bunch of little point things, and somebody says, “You should use this at this point, for this reason.” And there's not a lot in the way of synthesis, relatively speaking, right?Like, nobody's telling you what the kind of underlying thing is that makes it so that this thing is better in these circumstances than this other thing is. And without that, it's a bunch of, kind of, anecdotes and a bunch of kind of finger-in-the-air guesses. And there's a part of that, that just makes me sad, fundamentally, I guess, that humans built all of this stuff; we should know how all of it fits together. And—Corey: You would think, wouldn't you?Alex: Well, but the thing is, it's so enormously complicated and it's been developed over such an enormously long period of time, that—or at least, you know, relatively speaking—it's really, really hard to kind of get that and extract it out. But I think when you do, it's very satisfying when you can actually say like, “Oh no, no, we've actually done—we've done the analysis here. Like, this is exactly what you ought to be doing.” And being able to give that clear answer and backing it up with something substantial is, I think, really valuable from the customer's point of view, right, because they don't have to rely on us kind of just doing the finger-in-the-air guess. But also, like, it's valuable overall. It extends the kind of domain where you don't have to think about whether or not you've got the right answer there. Or at least you don't have to think about it as much.Corey: My philosophy has always been that when I have those hunches, they're useful, and it's an indication that there's something to look into here. Where I think it goes completely off the rails is when people, like, “Well, I have a hunch and I have this belief, and I'm not going to evaluate whether or not that belief is still one that is reasonable to hold, or there has been perhaps some new information that it would behoove me to figure out. Nope, I've just decided that I know—I have a hunch now and that's enough and I've done learning.” That is where people get into trouble.And I see aspects of it all the time when talking to clients, for example. People who believe things about their bill that at one point were absolutely true, but now no longer are. And that's one of those things that, to be clear, I see myself doing this. This is not something—Alex: Oh, everybody does, yeah.Corey: —I'm blaming other people for it all. Every once in a while I have to go on a deep dive into our own AWS bill just to reacquaint myself with an understanding of what's going on over there.Alex: Right.Corey: And I will say that one thing that I was firmly convinced was going to happen during your tenure here was that you're a data person; hiring someone like you is the absolute most expensive thing you can ever do with respect to your AWS bill because hey, you're into the data space. During your tenure here, you cut the bill in half. And that surprises me significantly. I want to further be clear that did not get replaced by, “Oh, yeah. How do you cut your AWS bill by so much?” “We moved everything to Snowflake.” No, we did not wind up—Alex: [laugh].Corey: Just moving the data somewhere else. It's like, at some level, “Great. How do I cut the AWS bill by a hundred percent? We migrate it to GCP.” Technically correct; not what the customer is asking for.Alex: Right? Exactly, exactly. I think part of that, too—and this is something that happens in the data part of the space more than anywhere else—it's easy to succumb to shiny object syndrome, right? “Oh, we need a cloud data warehouse because cloud data warehouse, you know? Snowflake, most expensive IPO in the history of time. We got to get on that train.”And, you know, I think one of the things that I know you and I talked about was, you know, where should all this data that we're amassing go? And what should we be optimizing for? And I think one of the things that, you know, the kind of conclusions that we came to there was, well, we're doing some stuff here, that's kind of designed to accelerate queries that don't really need to be accelerated all that much, right? The difference between a query taking 500 milliseconds and 15 seconds, from our point of view, doesn't really matter all that much, right? And that realization alone, kind of collapsed a lot of technical complexity, and that, I will say we at Duckbill Group still espouse, right, is that cloud cost is an architectural problem, it's not a right-sizing your instances problem. And once we kind of got past that architectural problem, then the cost just sort of cratered. And honestly, that was a great feeling, to see the estimate in the billing console go down 47% from last month, and it's like, “Ah, still got it.” [laugh].Corey: It's neat to watch that happen, first off—Alex: For sure.Corey: But it also happened as well, with increasing amounts of utility. There was a new AWS billing page that came out, and I'm sure it meets someone's needs somewhere, somehow, but the things that I always wanted to look at when I want someone to pull up their last month's bill is great, hit the print button—on the old page—and it spits out an exploded pdf of every type of usage across their entire AWS estate. And I can skim through that thing and figure out what the hell's going on at a high level. And this new thing did not let me do that. And that's a concern, not just for the consulting story because with our clients, we have better access than printing a PDF and reading it by hand, but even talking to randos on the internet who were freaking out about an AWS bill, they shouldn't have to trust me enough to give me access into their account. They should be able to get a PDF and send it to me.Well, I was talking with you about this, and again, in what felt like ten minutes, you wound up with a command line tool, run it on an exported CSV of a monthly bill and it spits it out as an HTML page that automatically collapses in and allocates things based upon different groups and service type and usage. And congratulations, you spent ten minutes to create a better billing experience than AWS did. Which feels like it was probably, in fairness to AWS, about seven-and-a-half minutes more time than they spent on it.Alex: Well, I mean, I think that comes back to what we were saying about, you know, not all the interesting problems in data are in data that doesn't fit in RAM, right? I think, in this case, that came from two places. I looked at those PDFs for a number of clients, and there were a few things that just made my brain hurt. And you and Mike and the rest of the folks at Duckbill could stare at the PDF, like, reading the matrix because you've seen so many of them before and go, ah, yes, “Bill spikes here, here, here.” I'm looking at this and it's just a giant grid of numbers.And what I wanted was I wanted to be able to say, like, don't show me the services in alphabetical order; show me the service is organized in descending order by spend. And within that, don't show me the operations in alphabetical order; show me the operations in decreasing order by spend. And while you're at it, group them into a usage type group so that I know what usage type group is the biggest hitter, right? The second reason, frankly, was I had just learned that DuckDB was a thing that existed, and—Corey: Based on the name alone, I was interested.Alex: Oh, it was an incredible stroke of luck that it was named that. And I went, “This thing lets me run SQL queries against CSV files. I bet I can write something really fast that does this without having to bash my head against the syntactic wall that is Pandas.” And at the end of the day, we had something that I was pretty pleased with. But it's one of those examples of, like, again, just orienting the problem toward, “Well, this is awful.”Because I remember when we first heard about the new billing experience, you kind of had pinged me and went, “We might need something to fix this because this is a problem.” And I went, “Oh, yeah, I can build that.” Which is kind of how a lot of what I've done over the last 15 years has been. It's like, “Oh. Yeah, I bet I could build that.” So, that's kind of how that went.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: The problem that I keep seeing with all this stuff is I think of it in terms of having to work with the tools I'm given. And yeah, I can spin up infrastructure super easily, but the idea of, I'm going to build something that manipulates data and recombines it in a bunch of different ways, that's not something that I have a lot of experience with, so it's not my instinctive, “Oh, I bet there's an easier way to spit this thing out.” And you think in that mode. You effectively wind up automatically just doing those things, almost casually. Which does make a fair bit of sense, when you understand the context behind it, but for those of us who don't live in that space, it's magic.Alex: I've worked in infrastructure in one form or another my entire career, data infrastructure mostly. And one of the things—I heard this from someone and I can't remember who it was, but they said, “When infrastructure works, it's invisible.” When you walk in the room and flip the light switch, the lights come on. And the fact that the lights come on is a minor miracle. I mean, the electrical grid is one of the most sophisticated, globally-distributed engineering systems ever devised, but we don't think about it that way, right?And the flip side of that, unfortunately, is that people really pay attention to infrastructure most when it breaks. But they are two edges of the same proverbial sword. It's like, I know, when I've done a good job, if the thing got built and it stayed built and it silently runs in the background and people forget it exists. That's how I know that I've done a good job. And that's what I aim to do really, everywhere, including with Duckbill Group, and I'm hoping that the stuff that I built hasn't caught on fire quite yet.Corey: The smoke is just the arising of the piles of money it wound up spinning up.Alex: [laugh].Corey: It's like, “Oh yeah, turns out that maybe we shouldn't have built a database out of pure Managed NAT Gateways. Yeah, who knew?”Alex: Right, right. Maybe I shouldn't have filled my S3 bucket with pure unobtainium. That was a bad idea.Corey: One other thing that we do here that I admit I don't talk about very often because people get the wrong idea, but we do analyst projects for vendors from time to time. And the reason I don't say that is, when people hear about analysts, they think about something radically different, and I do not self-identify as an analyst. It's, “Oh, I'm not an analyst.” “Really? Because we have analyst budget.” “Oh, you said analyst. I thought you said something completely different. Yes, insert coin to continue.”And that was fine, but unlike the vast majority of analysts out there, we don't form our opinions based upon talking to clients and doing deeper dive explorations as our primary focus. We're a team of engineers. All right, you have a product. Let's instrument something with it, or use your product for something and we'll see how it goes along the way. And that is something that's hard for folks to contextualize.What was really fun was bringing you into a few of those engagements just because it was interesting; at the start of those calls. “It was all great, Corey is here and—oh, someone else's here. Is this a security problem?” “It's no, no, Alex is with me.” And you start off those calls doing what everyone should do on those calls is, “How can we help?” And then we shut up and listen. Step one, be a good consultant.And then you ask some probing questions and it goes a little bit deeper and a little bit deeper, and by the end of that call, it's like, “Wow, Alex is amazing. I don't know what that Corey clown is doing here, but yeah, having Alex was amazing.” And every single time, it was phenomenal to watch as you, more or less, got right to the heart of their generally data-oriented problems. It was really fun to be able to think about what customers are trying to achieve through the lens that you see the world through.Alex: Well, that's very flattering, first of all. Thank you. I had a lot of fun on those engagements, honestly because it's really interesting to talk to folks who are building these systems that are targeting mass audiences of very deep-pocketed organizations, right? Because a lot of those organizations, the companies doing the building are themselves massive. And they can talk to their customers, but it's not quite the same as it would be if you or I were talking to the customers because, you know, you don't want to tell someone that their baby is ugly.And note, now, to be fair, we under no circumstances were telling people that their baby was ugly, but I think that the thing that is really fun for me is to kind of be able to wear the academic database nerd hat and the practitioner hat simultaneously, and say, like, “I see why you think this thing is really impressive because of this whiz-bang, technical thing that it does, but I don't know that your customers actually care about that. But what they do care about is this other thing that you've done as an ancillary side effect that actually turns out is a much more compelling thing for someone who has to deal with this stuff every day. So like, you should probably be focusing attention on that.” And the thing that I think was really gratifying was when you know that you're meeting someone on their level and you're giving them honest feedback and you're not just telling them, you know, “The Gartner Magic Quadrant says that in order to move up and to the right, you must do the following five features.” But instead saying, like, “I've built these things before, I've deployed them before, I've managed them before. Here's what sucks that you're solving.” And seeing the kind of gears turn in their head is a very gratifying thing for me.Corey: My favorite part of consulting—and I consider analyst style engagements to be a form of consulting as well—is watching someone get it, watching that light go on, and they suddenly see the answer to a problem that's been vexing them I love that.Alex: Absolutely. I mean, especially when you can tell that this is a thing that has been keeping them up at night and you can say, “Okay. I see your problem. I think I understand it. I think I might know how to help you solve it. Let's go solve it together. I think I have a way out.”And you know, that relief, the sense of like, “Oh, thank God somebody knows what they're doing and can help me with this, and I don't have to think about this anymore.” That's the most gratifying part of the job, in my opinion.Corey: For me, it has always been twofold. One, you've got people figuring out how to solve their problem and you've made their situation better for it. But selfishly, the thing I like the most personally has been the thrill you get from solving a puzzle that you've been toying with and finally it clicks. That is the endorphin hit that keeps me going.Alex: Absolutely.Corey: And I didn't expect when I started this place is that every client engagement is different enough that it isn't boring. It's not the same thing 15 times. Which it would be if it were, “Hi, thanks for having us. You haven't bought some RIs. You should buy some RIs. And I'm off.” It… yeah, software can do that. That's not interesting.Alex: Right. Right. But I think that's the other thing about both cloud economics and data engineering, they kind of both fit into that same mold. You know, what is it? “All happy families are alike, but each unhappy family is unhappy in its own way.” I'm butchering Chekhov, I'm sure. But like—if it's even Chekhov.But the general kind of shape of it is this: everybody's infrastructure is different. Everybody's organization is different. Everybody's optimizing for a different point in the space. And being able to come in and say, “I know that you could just buy a thing that tells you to buy some RIs, but it's not going to know who you are; it's not going to know what your business is; it's not going to know what your challenges are; it's not going to know what your roadmap is. Tell me all those things and then I'll tell you what you shouldn't pay attention to and what you should.”And that's incredibly, incredibly valuable. It's why, you know, it's why they pay us. And that's something that you can never really automate away. I mean, you hear this in data all the time, right? “Oh, well, once all the infrastructure is managed, then we won't need data infrastructure people anymore.”Well, it turns out all the infrastructure is managed now, and we need them more than we ever did. And it's not because this managed stuff is harder to run; it's that the capabilities have increased to the point that they're getting used more. And the more that they're getting used, the more complicated that use becomes, and the more you need somebody who can think at the level of what does the business need, but also, what the heck is this thing doing when I hit the run key? You know? And that I think, is something, particularly in AWS where I mean, my God, the amount and variety and complexity of stuff that can be deployed in service of an organization's use case is—it can't be contained in a single brain.And being able to make sense of that, being able to untangle that and figure out, as you say, the kind of the aha moment, the, “Oh, we can take all of this and just reduce it down to nothing,” is hugely, hugely gratifying and valuable to the customer, I'd like to think.Corey: I think you're right. And again, having been doing this in varying capacities for over five years—almost six now; my God—the one thing has been constant throughout all of that is, our number one source for new business has always been word of mouth. And there have been things that obviously contribute to that, and there are other vectors we have as well, but by and large, when someone winds up asking a colleague or a friend or an acquaintance about the problem of their AWS bill, and the response almost universally, is, “Yeah, you should go talk to The Duckbill Group,” that says something that validates that we aren't going too far wrong with what we're approaching. Now that you're back on the freelance data side, I'm looking forward to continuing to work with you, if through no other means and being your customer, just because you solve very interesting and occasionally very specific problems that we periodically see. There's no reason that we can't bring specialists in—and we do from time to time—to look at very specific aspects of a customer problem or a customer constraint, or, in your case for example, a customer data set, which, “Hmm, I have some thoughts on here, but just optimizing what storage class that three petabytes of data lives within seems like it's maybe step two, after figuring what the heck is in it.” Baseline stuff. You know, the place that you live in that I hand-wave over because I'm scared of the complexity.Alex: I am very much looking forward to continuing to work with you on this. There's a whole bunch of really, really exciting opportunities there. And in terms of word of mouth, right, same here. Most of my inbound clientele came to me through word of mouth, especially in the first couple years. And I feel like that's how you know that you're doing it right.If someone hires you, that's one thing, and if someone refers you, to their friends, that's validation that they feel comfortable enough with you and with the work that you can do that they're not going to—you know, they're not going to pass their friends off to someone who's a chump, right? And that makes me feel good. Every time I go, “Oh, I heard from such and such that you're good at this. You want to help me with this?” Like, “Yes, absolutely.”Corey: I've really appreciated the opportunity to work with you and I'm super glad I got the chance to get to know you, including as a person, not just as the person who knows the data, but there's a human being there, too, believe it or not.Alex: Weird. [laugh].Corey: And that's the important part. If people want to learn more about what you're up to, how you think about these things, potentially have you looked at a gnarly data problem they've got, where's the best place to find you now?Alex: So, my business is called Bits on Disk. The website is bitsondisk.com. I do write occasionally there. I'm also on Twitter at @alexras. That's Alex-R-A-S, and I'm on LinkedIn as well. So, if your lovely listeners would like to reach me through any of those means, please don't hesitate to reach out. I would love to talk to them more about the challenges that they're facing in data and how I might be able to help them solve them.Corey: Wonderful. And we will of course, put links to that in the show notes. Thank you again for taking the time to speak with me, spending as much time working here as you did, and honestly, for a lot of the things that you've taught me along the way.Alex: My absolute pleasure. Thank you very much for having me.Corey: Alex Rasmussen, data engineering consultant at Bits on Disk. I'm Cloud Economist Corey Quinn. This is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that is so large it no longer fits in RAM.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

We were fixing servers all night, but at least we have a great story. A special guest joins us to help make a big show announcement. Special Guest: Tim Canham.