POPULARITY
6 episodes with Reproducibility
4 episodes with Reproducibility
6 episodes with Reproducibility
5 episodes with Reproducibility
2 episodes with Reproducibility
2 episodes with Reproducibility
2 episodes with Reproducibility
2 episodes with Reproducibility
3 episodes with Reproducibility
The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/joinMembership will get you access to member only videos, exclusive notes and monthly 1:1 with me. Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA------------------------------------------------------------------------------------------------------------------------------------------------------------------About this episode: ------------------------------------------------------------------------------------------------------------------------------------------------------------------In this conversation, Jacopo and Ciro discuss their journey in building Bauplan, a platform designed to simplify data management and enhance developer experience. They explore the challenges faced in data bottlenecks, the integration of development and production environments, and the unique approach of Bauplan using serverless functions and Git-like versioning for data. The discussion also touches on scalability, handling large data workloads, and the critical aspects of reproducibility and compliance in data management. Chapters:00:00 Introduction03:00 The Data Bottleneck: Challenges in Data Management06:14 Bridging Development and Production: The Need for Integration09:06 Serverless Functions and Git for Data17:03 Developer Experience: Reducing Complexity in Data Management19:45 The Role of Functions in Data Pipelines: A New Paradigm23:40 Building Robust Data Solutions: Versioning and Parameters30:13 Optimizing Data Processing: Bauplan Runtime46:46 Understanding Control Planes and Data Management48:51 Ensuring Robustness in Data Pipelines52:38 Data Quality and Testing Mechanisms54:43 Branching and Collaboration in Data Development57:09 Scalability and Resource Management in Data Functions01:01:13 Handling Large Data Workloads and Use Cases01:09:05 Reproducibility and Compliance in Data Management01:16:46 Future Directions in Data Engineering and Use CasesLinks and References:Bauplan website:https://www.bauplanlabs.com
In today's episode, "Zebrafish Husbandry Reporting & Reproducibility Initiative (Part 1)," we talk with former Zebrafish Husbandry Association President Michelle Altemara about her new initiative to standardize reporting in fish research papers. Science has a reproducibility problem and oftentimes it's because we don't account for all of the variables. Sometimes, “we keep fish on a recirculating rack” is not a good enough way to describe husbandry. If two different facilities are keeping fish at completely different light cycles, the scientific outcomes might be very different! Luckily Michelle and her colleagues are trying to change that by working with journals to advocate for better guidelines on this exact thing. Come listen!This podcast is brought to you by the lungfish. The lungfish quite literally possesses lungs but it also possesses working gills, making it one of the most adaptable species on earth. Other fish might consider them cheaters for having both functioning organs while fish and other animals only possess one. This leads to lungfish often being ostracized at parties and left out of group activities. Cheers to you lungfish. You were born this way and you are special. Thanks for listening to Gettin' Fishy With It! You can find our website with show notes at https://gettingfishypod.substack.com/. You can find us on twitter at @gettinfishypod, and on Instagram @gettingfishypod. You can also find us on Facebook and LinkedIn. If you want to drop us an email, you can send your complaints (or questions!) to gettingfishypod@gmail.com.Our theme music is “Best Time” by FASSOUNDS. Our audio is edited by Amber Park Chiodini. Amber has her own podcast all about movies, called So What Happens Next?We very much appreciate you taking the time to listen to our fiftieth episode! Please help out the podcast by subscribing on your podcast platform of choice. If you could leave us a review, that would be super helpful!If you would like to support the show, you can sign up as a paid member on our Substack, or you can buy us a coffee!Thanks and we'll “sea” you again in two weeks!
Part 2 of this series could have easily been renamed "AI for science: The expert's guide to practical machine learning.” We continue our discussion with Christoph Molnar and Timo Freiesleben to look at how scientists can apply supervised machine learning techniques from the previous episode into their research.Introduction to supervised ML for science (0:00) Welcome back to Christoph Molnar and Timo Freiesleben, co-authors of “Supervised Machine Learning for Science: How to Stop Worrying and Love Your Black Box”The model as the expert? (1:00)Evaluation metrics have profound downstream effects on all modeling decisionsData augmentation offers a simple yet powerful way to incorporate domain knowledgeDomain expertise is often undervalued in data science despite being crucialMeasuring causality: Metrics and blind spots (10:10)Causality approaches in ML range from exploring associations to inferring treatment effectsConnecting models to scientific understanding (18:00)Interpretation methods must stay within realistic data distributions to yield meaningful insightsRobustness across distribution shifts (26:40)Robustness requires understanding what distribution shifts affect your modelPre-trained models and transfer learning provide promising paths to more robust scientific MLReproducibility challenges in ML and science (35:00)Reproducibility challenges differ between traditional science and machine learningGo back to listen to part one of this series for the conceptual foundations that support these practical applications.Check out Christoph and Timo's book “Supervised Machine Learning for Science: How to Stop Worrying and Love Your Black Box” available online now.What did you think? Let us know.Do you have a question or a discussion topic for the AI Fundamentalists? Connect with them to comment on your favorite topics: LinkedIn - Episode summaries, shares of cited articles, and more. YouTube - Was it something that we said? Good. Share your favorite quotes. Visit our page - see past episodes and submit your feedback! It continues to inspire future episodes.
Send us a textShort Summary: A rare, insider's look at the messy realities of scientific research with Stanford's Dr. John Ioannidis. The good, the bad, and the ugly about how scientific research actually works.About the guest: John Ioannidis, MD, PhD is a professor at Stanford University in medicine, epidemiology, population health, and biomedical data science, with an MD from the University of Athens and a PhD from Harvard in biostatistics. He directs the Meta-Research Innovation Center at Stanford (METRICS), focusing on improving research methods and practices. Renowned for his paper “Why Most Published Research Findings Are False,” he's among the most cited scientists globally, tackling biases and reproducibility in science.Note: Podcast episodes are fully available to paid subscribers on the M&M Substack and everyone on YouTube. Partial versions are available elsewhere. Full transcript and other information on Substack.Key Takeaways:Science's “replication crisis” isn't new—it's baked into how tough and bias-prone research is, hitting all fields, not just “soft” ones like psychology.Ioannidis's famous claim, “most published findings are false,” holds up: stats show many “significant” results are flukes due to weak studies or bias.Peer review's a mixed bag—only a third of papers improve, and unpaid, tired reviewers miss a lot, letting shaky stuff slip through.Publishing's a $30 billion game with 50,000+ journals; big players like Elsevier rake in huge profits from subscriptions and fees, often over $10,000 per paper.Researchers game the system—think fake co-authorships or citation cartels—boosting metrics like the H-index, which tracks papers with matching citation counts.Ioannidis's early COVID-19 fatality rate (0.2-0.3%) was spot-on but sparked a firestorm as politics warped science into “clan warfare.”NIH funding's clogged by red tape and favors older researchers, starving young innovators and risky ideas that could shake things up.He's building tools like a public database of scientist stats (4 million downloads!) to spotlight gaming and push for transparent, fair research.*Not medical advice.Support the showAll episodes, show notes, transcripts, etc. at the M&M Substack Affiliates: Lumen device to optimize your metabolism for weight loss or athletic performance. Use code MIND for 10% off. Readwise: Organize and share what you read. Athletic Greens: Comprehensive & convenient daily nutrition. Free 1-year supply of vitamin D with purchase. KetoCitra—Ketone body BHB + potassium, calcium & magnesium, formulated with kidney health in mind. Use code MIND20 for 20% off any subscription. MASA Chips—delicious tortilla chips made from organic corn and grass-fed beef tallow. No seed oils or artificial ingredients. Use code MIND for 20% off. For all the ways you can support my efforts
Today's episode features guest host, Michael Upshall, Community and Outreach Manager at Core, who talks with Dr. Elizabeth Bik, Microbiologist and Science Integrity Consultant. Elisabeth is a prominent microbiologist and renowned investigator into scientific misconduct, particularly the manipulation and falsification of research data. She has uncovered issues in over 7,000 scientific papers, resulting in more than 1,000 retractions. Her work has gained international attention, earning her the 2021 John Maddox Prize. This conversation explores Elizabeth's career trajectory, her work on identifying scientific malpractice, and her thoughts on the systemic issues and potential reforms within the research and publishing ecosystem. View the video of the interview here: https://youtu.be/uEYsqTKHits Social Media: LinkedIn: https://www.linkedin.com/in/mupshall/ https://www.linkedin.com/in/elisabeth-bik-4376782/ Twitter: Keywords: #ScientificIntegrity, #ScientificMisconduct, #research, #ResearchData, #ResearchReliability, #Retractions, #Preprints, #AIInResearch, #Reproducibility, #InformationLiteracy, #InformationScience, #DigitalLibraries, #DigitalTools, #DigitalAge, #PublishingReforms, #InformationPower, #knowledge, #awareness, #efficiency, #innovation, #skills, #career, #partnerships, #collaboration, #scholcomm, #ScholarlyCommunication, #libraries, #librarianship, #LibraryNeeds, #LibraryLove, #ScholarlyPublishing, #AcademicPublishing, #publishing, #LibrariesAndPublishers, #podcasts
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Karen Christman, Sheila Chari, Stella Hurtley, and Robert Stephenson explore academic publishing in stem cell research, focusing on reproducibility, collaboration, and public communication. Editors from top journals discuss curating impactful research, sharing clinical trial data, and addressing challenges in scaling and standardizing therapies. They emphasize bridging silos, advancing precision regenerative medicine, and navigating open access publishing to responsibly propel the field forward. Series: "Stem Cell Channel" [Health and Medicine] [Science] [Show ID: 39940]
Joz Norris and Alexander Bennett join host Alice Fraser for episode 191 of The Gargle - all of the news, and none of the politics.
On this episode of Translating Proteomics, co-hosts Parag Mallick and Andreas Huhmer of Nautilus Biotechnology discuss the reproducibility crisis in biology and specifically focus on how we can enhance reproducibility in computational proteomics. Key topics they cover include:• What the reproducibility crisis is• Factors that make it difficult to replicate multiomics research• Steps we can take to make biology research more reproducibleChapters 00:00 – 01:20 – Introduction01:20– 03:10 – What is reproducibility in research and why is it important?03:10 – 05:42 – Recent work from the Mallick Lab focused on computational proteomics reproducibility05:42 – 09:32 – Ways to help improve reproducibility in computational proteomics – More detailed documentation, moving beyond papers as our main form of documentation, and ensuring computational workflows are available,09:32 – 11:30 – Why Parag got interested reproducibility – Attempts to build AI layers on top of current workflows11:30 – 14:00 – The need to create repositories of analytical workflows codified in a structured way that AI can learn from14:00 – 15:24 – A role for dedicated data curators15:24 – 18:31 – Moving beyond the idea of study endpoints and recognizing data as part of a larger whole18:31 – 21:32 – How does AI fit into the continuous analysis and incorporation of new datasets21:32 – 23:36 – The role of AI in helping researchers design experiments23:36 – 27:25 – Three things we can do today to increase the reproducibility of computational proteomics experiments:· Be clear about the stated hypothesis· Document analyses through workflow engines and containerized workflows· Advocate for support for funding for reproducibility and reproducibility tools27:25 – End – OutroResourcesParag's Gilbert S. Omenn Computational Proteomics Award Lectureo In this lecture, Parag describes his vision for a more reproducible future in proteomicsNature Special on “Challenges in irreproducible research”o A list of articles and perspective pieces discussing the “reproducibility crisis” in researchWhy Most Published Research Findings Are False (Ioannidis 2005)o Article outlining many of the issues that make it difficult to reproduce research findingsReproducibility Project: Cancer Biologyo eLife initiative investigating reproducibility in preclinical cancer researchCenter for Open Science Preregistration Initiativeo Resources for preregistering a hypothesis as part of a studyNational Institute of Standards and Technology (NIST)o US government agency that aims to...
In this episode, we explore the fascinating intersection of artificial intelligence (AI) and academic publishing with Gráinne McNamara, a Research Integrity and Publication Ethics Manager.Gráinne shares her insights on how AI is revolutionising the publishing process, helping to detect plagiarism, image manipulation, and ensuring reproducibility in scientific research. We discuss the ethical challenges of deploying AI in this space, including data privacy concerns and the risk of reinforcing biases in peer review.This conversation dives into the current tools and technologies being used to maintain integrity, the limitations of AI in fraud detection, and the opportunities for AI to improve trust and credibility in academic publishing. Gráinne also provides a glimpse into the future of AI in ensuring methodological and result reproducibility while highlighting the industry's ongoing efforts to stay ahead of emerging trends and challenges.Guest:Gráinne McNamara - Research Integrity and Publication Ethics Manager, specialising in leveraging AI to ensure credibility and ethical standards in scientific publishing.Topics Covered:The Importance of Integrity in Academic PublishingAI Tools for Detecting Plagiarism and Image ManipulationEnhancing Peer Review with AIEthical Concerns and Data Privacy in AI for PublishingChallenges of Bias in AI ModelsThe Future of Reproducibility in Scientific ResearchClick to connect with Dr. Andree Bates for more information in this episode: https://eularis.com/AI For Pharma Growth is the podcast from pioneering Pharma Artificial Intelligence entrepreneur Dr. Andree Bates created to help organisations understand how the use of AI based technologies can easily save them time and grow their brands and business. This show blends deep experience in the sector with demystifying AI for all pharma people, from start up biotech right through to Big Pharma. In this podcast Dr Andree will teach you the tried and true secrets to building a pharma company using AI that anyone can use, at any budget.As the author of many peer-reviewed journals and having addressed over 500 industry conferences across the globe, Dr Andree Bates uses her obsession with all things AI and futuretech to help you to navigate through the, sometimes confusing but, magical world of AI powered tools to grow pharma businesses. This podcast features many experts who have developed powerful AI powered tools that are the secret behind some time saving and supercharged revenue generating business results. Those who share their stories and expertise show how AI can be applied to sales, marketing, production, social media, psychology, customer insights and so much more.
This episode is an interview I did with the new podcast Information Theory. The host of Information Theory is an anonymous technologist trained in physics and machine learning.Information Theory Podcast on YouTube: https://www.youtube.com/@InformationTheoryPodInformation Theory Podcast on Spotify: https://open.spotify.com/show/6PbxeOYInRuH4DBXOAOq5u?si=q90fZh8PRUut5c1XG4K7Sw (00:00) - Introduction to Information Theory podcast (01:19) - The education of a physicist (10:53) - Computational genomics (19:40) - Thinking styles and collaboration in theoretical physics (26:08) - Scientific progress and the Great Stagnation (40:39) - University research administration (45:05) - Reproducibility crisis (57:58) - Impact of basic research (01:03:16) - Critique of NIH and biomedical research (01:06:48) - Personal reflections on Trump's re-election and an inside view of the 47 transition (01:12:37) - Silicon Valley and US politics (01:15:30) - Concerns and hope for America's future Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.--Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (SuperFocus, SafeWeb, Genomic Prediction, Othram) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU. Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on X @hsu_steve.
Send us a textEnglish Edition: Arnaud Legrand, Christophe Pouzat and Konrad Hinsen, three French researchers, who went through the pain of making research data and software reproducible. Out of that pain grew a set of online courses. I met with them to discuss how they developed the courses, the steps they had to go through and what the courses cover. https://www.fun-mooc.fr/en/courses/reproducible-research-methodological-principles-transparent-scie/https://www.fun-mooc.fr/en/courses/reproducible-research-ii-practices-and-tools-for-managing-comput/https://khinsen.nethttps://orgmode.orgSupport the showThank you for listening! Merci de votre écoute! Vielen Dank für´s Zuhören! Contact Details/ Coordonnées / Kontakt: Email mailto:peter@code4thought.org UK RSE Slack (ukrse.slack.com): @code4thought or @piddie US RSE Slack (usrse.slack.com): @Peter Schmidt Mastodon: https://fosstodon.org/@code4thought or @code4thought@fosstodon.org Bluesky: https://bsky.app/profile/code4thought.bsky.social LinkedIn: https://www.linkedin.com/in/pweschmidt/ (personal Profile)LinkedIn: https://www.linkedin.com/company/codeforthought/ (Code for Thought Profile) This podcast is licensed under the Creative Commons Licence: https://creativecommons.org/licenses/by-sa/4.0/
Over the past few decades, a troubling shift has occurred in how we perceive and utilize science, particularly within educational contexts. Traditional science, grounded in facts, evidence, and experimentation, is increasingly being overshadowed by modern social science frameworks that emphasize subjective interpretations over empirical data. This shift has significant implications for the reliability and credibility of scientific knowledge.To receive my weekly posts for FREE in your email account, just submit your email below! The way we know this is a problem is something called the reproducibility crisis. Reproducibility—the ability to replicate the results of a study using the same methods and data—is one of the foundations of science. But studies have shown that this isn't happening. A 2016 survey published in Nature found that over 70% of researchers tried and failed to reproduce another scientist's experiments, and more than half couldn't reproduce their own. If a study can't be replicated, it means the findings aren't proven at all. This issue has eroded public trust in science and made it harder to solve real-world problems.In educational settings, this problem is compounded by the introduction of certain modern social science approaches that prioritize subjective interpretations over objective analysis. Instead of focusing on evidence-based questions like, “How does this process work?” they might ask, “How does this reflect systemic inequalities?” For example, someone might argue that labeling certain smells as “bad” is tied to cultural bias. But how do you test that? How do you prove it's true or false? These claims often rely on interpretations rather than measurable evidence, which makes them less useful for solving real-world problems.Another example is implicit bias testing, which claims to reliably measure unconscious prejudice and predict discriminatory behavior. While the idea has been widely adopted in workplaces and institutions, many researchers have questioned the reliability and validity of these tests. How do you objectively measure or prove the existence of a bias that the person may not be aware of? And can it accurately predict real-world actions? The evidence for these claims is often inconsistent and difficult to replicate.Similarly, in education, the concept of learning styles suggests that students learn best when taught according to their “preferred learning style” (visual, auditory, kinesthetic, etc.). While this sounds intuitive, numerous studies have failed to find consistent evidence supporting the effectiveness of tailoring teaching methods to these styles. How do you test this claim objectively when evidence suggests that all students benefit from well-rounded teaching strategies, regardless of their preferred style?Another common example is cultural appropriation in art and fashion, where critics argue that using certain cultural symbols is inherently exploitative or oppressive. For instance, a designer might be criticized for incorporating a traditional motif from another culture into their work. While these discussions can raise important questions about respect and representation, how do you measure whether such acts cause tangible harm? These claims often rely on subjective perceptions of offense, which are not easily quantified or tested.I am going to attempt to convince you that, if we want to solve real problems and move forward as a society, we need to abandon this way of thinking and return to science rooted in evidence and logic.Let's start with what real science looks like. Science asks clear questions about the world and then tests them. For example, scientists might wonder, “Does this medicine cure disease?” They test it on a large group of people, compare the results, and share their findings so others can confirm their work. If the results hold up, we accept them as truth—at least until new evidence suggests otherwise. This process has given us life-saving breakthroughs like antibiotics, airplanes, and smartphones. Science works because it's based on facts that can be proven and tested repeatedly.This focus on feelings creates something called relativism. Relativism is the idea that all opinions are equally valid, no matter how ridiculous. For example, certain frameworks might argue that math is oppressive because it emphasizes correct answers. But without objective truth, how do we build bridges, design computers, or cure diseases? Imagine an engineer saying, “My truth is that this bridge will hold up,” even if the math proves otherwise. Relativism doesn't lead to progress; it leads to chaos.We're already seeing the damage this mindset has caused. In some workplaces, employees are forced to sit through workshops where they're told that their race or gender determines whether they're an “oppressor” or “oppressed.” These sessions aren't based on evidence but on assumptions. In schools, lessons often focus on how systems are unfair instead of teaching students how to think critically and solve problems. Instead of creating solutions, this approach fosters division and resentment. And since these frameworks reject the idea of objective truth, it's impossible to argue against them—it's like debating with someone who denies the sky is blue.The truth is, these modern approaches don't fix problems; they just find new ones to complain about. Imagine if doctors used these methods. Instead of asking, “What's the best treatment for this disease?” they'd spend all their time arguing about how healthcare systems are unfair. While that might be worth discussing, it doesn't help the patient. Real science, on the other hand, focuses on solutions. It asks testable questions, runs experiments, and uses evidence to make the world better.We need to return to real, objective science. Science works because it relies on evidence, not opinions. It's the reason we have airplanes that don't fall out of the sky and medicines that actually cure diseases. These frameworks might sound sophisticated, but they're a dead end. If we want to move forward, we have to focus on what's real and testable. That's what real science does—and it's the only way to truly solve the problems we face.Thanks for reading Year Of The Opposite - Travis Stoliker's Substack! This post is public so feel free to share it. Get full access to Year Of The Opposite - Travis Stoliker's Substack at www.yearoftheopposite.com/subscribe
Robert (Bob) Wilson is an Associate Professor of Psychology at Georgia Tech. We talk about his tutorial paper (w/ Anne Collins) on computational modelling, and some of his recent work on detecting phishing.BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith.Support the show: https://geni.us/bjks-patreonTimestamps0:00:00: Bob's strange path through computational cognitive neuroscience0:07:37: Phishing: a computational model with real-life applications0:25:46: Start discussing Bob's paper 10 simple rules for computational modeling of behavioral data0:32:15: Rule 0: Why even do computational modelling?0:46:24: Rules 1 & 2: Design a good experiment & Design a good model1:02:51: Rule 3: Simulate!1:05:48: Rules 4 & 5: Parameter estimation and recovery1:18:28: Rule 6: Model recovery1:25:55: Rules 7 & 8: Collect data and validate the model1:33:15: Rule 9: Latent variable analysis1:36:24: Rule 10: Report your results1:37:46: Computational modelling and the open science movement1:40:17: A book or paper more people should read1:43:35: Something Bob wishes he'd learnt sooner1:47:18: Advice for PhD students/postdocsPodcast linksWebsite: https://geni.us/bjks-podTwitter: https://geni.us/bjks-pod-twtRobert's linksWebsite: https://geni.us/wilson-webGoogle Scholar: https://geni.us/wilson-scholarTwitter: https://geni.us/wilson-twtBen's linksWebsite: https://geni.us/bjks-webGoogle Scholar: https://geni.us/bjks-scholarTwitter: https://geni.us/bjks-twtReferencesEpisodes w/ Paul Smaldino: https://geni.us/bjks-smaldinohttps://geni.us/bjks-smaldino_2Bechara, Damasio, Damasio, & Anderson (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition.Feng, Wang, Zarnescu & Wilson (2021). The dynamics of explore–exploit decisions reveal a signal-to-noise mechanism for random exploration. Scientific Reports.Grilli, ... & Wilson (2021). Is this phishing? Older age is associated with greater difficulty discriminating between safe and malicious emails. The Journals of Gerontology: Series B.Hakim, Ebner, ... & Wilson (2021). The Phishing Email Suspicion Test (PEST) a lab-based task for evaluating the cognitive mechanisms of phishing detection. Behavior research methods.Harootonian, Ekstrom & Wilson (2022). Combination and competition between path integration and landmark navigation in the estimation of heading direction. PLoS Computational Biology.Hopfield (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS.MacKay (2003). Information theory, inference and learning algorithms.Miller, Eugene & Pribram (1960). Plans and the Structure of Behaviour.Sweis, Abram, Schmidt, Seeland, MacDonald III, Thomas, & Redish (2018). Sensitivity to “sunk costs” in mice, rats, and humans. Science.Walasek & Stewart (2021). You cannot accurately estimate an individual's loss aversion using an accept–reject task. Decision.Wilson & Collins (2019). Ten simple rules for the computational modeling of behavioral data. Elife.
Today, Dr. Patrick Schloss, Professor in the Department of Microbiology and Immunology in the School of Medicine at the University of Michigan, joins the #QualityQuorum to discuss how the human microbiome is studied, possible pitfalls in such data analysis, and what tools he and his coworkers have developed to lead toward repeatable, hypothesis-driven science. Host: Mark O. Martin Guest: Patrick Schloss Subscribe: Apple Podcasts, Spotify Become a patron of Matters Microbial! Links for this episode An overview of how the gut microbiome is analyzed. One of the articles discussed by Dr. Schloss exploring reproducibility in microbiome studies: “Identifying and Overcoming Threats to Reproducibility, Replicability, Robustness, and Generalizability in Microbiome Research.” Another article discussed by Dr. Schloss, regarding the link between the microbiome and obesity: “Looking for a Signal in the Noise: Revisiting Obesity and the Microbiome.” An article from Dr. Schloss' research team that explores a link between the human microbiome and a type of colorectal cancer. A link to the MOTHUR project, used to analyze microbiome data. A link to a video by Dr. Schloss: “Understanding Disease Through the Lens of the Microbiome.” Dr. Schloss' YouTube channel about data analysis. Dr. Schloss' research group website. Dr. Schloss' faculty website. Intro music is by Reber Clark Send your questions and comments to mattersmicrobial@gmail.com
Subscribe, Rate, & Review Future Fossils on YouTube • Spotify • Apple PodcastsThis week on Future Fossils I welcome back Sara Phinn Huntley (help her fight cancer!), a multimedia artist, writer, and researcher who has spent the last two decades exploring the intersection of psychedelics, technology, and philosophy.An intrepid psychonaut and cartographer of hyperspace, her current focus involves using VR to represent visual/spatial imagination in real-time. Using a multidisciplinary approach, she documents and maps the states revealed by dimethyltriptamime and other psychedelics, cargo culting higher dimensional artifacts through the intersection of chaos mathematics, Islamic geometry, and 3D diagrammatic performance capture. Her work has been published by the Multidisciplinary Association of Psychedelic Studies and featured in Diana Reed Slattery's Xenolinguistics. She is the art director for The Illustrated Field to the DMT Entities with David Jay Brown (forthcoming at Inner Traditions, 2025).✨ Offer Support + Join The Scene• Become a patron on Substack or Patreon• Make a tax-deductible donation to Humans On The Loop• Invite me to work with you as an hourly consultant or advisor on retainer• Join the Holistic Technology & Wise Innovation and Future Fossils Discord servers• Join the Future Fossils Facebook group• Buy the books we discuss from my Bookshop.org reading list• Buy original paintings and prints or commission new work• Tip me with @futurefossils on Venmo, $manfredmacx on CashApp, or @michaelgarfield on PayPal• Buy the show's music on Bandcamp — intro “Olympus Mons” from the Martian Arts EP & outro “Sonnet A” from the Double-Edged Sword EP✨ Main Points + Big Ideas* The Entanglement of Language and Being: DMT entities reveal a profound connection between language and the construction of reality, echoing themes found in esoteric traditions and the emergence of AI.* The Cartography of Hyperspace: The book serves as a guide to the vast and uncharted territory of DMT experiences, highlighting the challenge of classifying subjective encounters and the potential for mapping a multidimensional reality.* The Reproducibility Problem and the Power of Big Data: While acknowledging the inherent challenges of studying subjective experiences, we point to the potential of emerging technologies like AI, blockchain, and large-scale data analysis to offer new insights.* Embodied Bias and the Nature of Evolution: The nonlinear and multidimensional nature of DMT experiences challenges our understanding of time, evolution, and even anatomy, prompting a re-evaluation of our assumptions about reality.* Attention as a Currency: We emphasize the importance of attention in navigating both the DMT space and the rapidly evolving technological landscape, posing critical questions about who or what deserves our focus.* The Question of Human Survival: The episode ends by urging humanity to confront its self-destructive tendencies and leverage its collective wisdom to navigate the challenges and opportunities of the future.✨ ChaptersChapter 1: Sara's Psychedelic Journey and the Genesis of the DMT Entities Field Guide (00:00:00 - 00:10:00)* Sara's fascination with DMT from a young age.* Her exploration of DMT through various artistic media, including performance art and xenolinguistics.* The inception of The Illustrated Field Guide to DMT Entities book, inspired by classic field guides to nature.* The decision to leverage AI in the book's creation due to the vastness of the subject matter.Chapter 2: Language, Being, and the AI Oracle (00:10:00 - 00:20:00)* The role of language in shaping and interpreting DMT entities, drawing parallels to esoteric traditions like the concept of the Logos.* Sara's process of interacting with AI, describing it as "talking to it" to curate the visual representations of DMT entities.* The blurring of categories and the subjective nature of interpreting the raw data of DMT experiences.* The challenge of reconciling diverse and often conflicting perceptions of the same entities.* Language as a compression tool for expressing ineffable experiences.* The increasing relevance of AI in understanding consciousness, particularly with future advancements in brain modeling.Chapter 3: Navigating Ontological Shock and the Nature of DMT Entities (00:20:00 - 00:30:00)* The challenge of reconciling DMT experiences with our "meat space" understanding of reality.* Sara's personal experience of gaining knowledge through DMT, challenging James Kent's view on the limitations of such knowledge.* The neurological basis for some common DMT hallucinations and its implications for understanding the experience.* The interplay of cultural and personal projections in shaping DMT entity encounters.* Exploring the possibility of psychedelics as a way to interact with a simulated reality.* The existence of phenomena that defy current scientific understanding, pointing to the need for open-mindedness.Chapter 4: The Cartography of Hyperspace and the Specter of Evolution (00:30:00 - 00:40:00)* The possibility of DMT entity encounters revealing more about the observer than about independent beings.* The existence of consistent archetypes across different DMT experiences and their overlap with other paranormal phenomena.* The intriguing connection between DMT entities and cross-cultural mythological figures.* Examining the role of genetic lineage and the intergenerational transmission of unusual experiences.* The book as a tool for intellectual curiosity, humility, and exploring the vastness of hyperspace.* The influence of culture in shaping our perceptions of both traditional and modern entities.* Sara's personal stance on the reality of DMT entities - acknowledging their potential existence while remaining open to other interpretations.Chapter 5: The Machine in the Ghost: Folklore, AI, and the Urge to Classify (00:40:00 - 00:50:00)* The blurring lines between insectoid and mechanical entities in both folklore and modern UAP narratives.* The impact of technology and the idea of a simulated reality on our perception of entities.* Sara's view on the potential taxonomic shift in our understanding of entities due to technological advancements.* Exploring the limits of AI in understanding consciousness and the potential for using it as a tool for self-reflection.* The challenge and importance of maintaining a sense of awe and wonder amidst scientific inquiry.Chapter 6: The Problem of Reproducibility and the Potential of Big Data (00:50:00 - 01:00:00)* Acknowledging the inherent limitations of scientific inquiry into subjective experiences.* The promise of machine learning and big data in identifying patterns and correlations across diverse DMT experiences.* The potential for reconstructing visual fields from brain data to gain further insights into the DMT experience.* The potential for utilizing blockchain technology, quadratic voting, and other advanced tools to address researcher bias and context in large-scale data collection.Chapter 7: Embodied Bias and the Non-Linearity of Time (01:00:00 - 01:10:00)* The idea of anatomy as an encoded representation of environmental features and its implications for understanding non-human entities.* Challenging the linear concept of time and evolution in light of the multidimensional experiences offered by DMT.* The vastness and complexity of "meat space" reality and its potential to hold hidden dimensions and Easter eggs.* The potential for AI and advanced computation to unlock deeper understanding of reality in conjunction with psychedelic exploration.Chapter 8: Sara's Breakthrough Experience and the Reverence for Mystery (01:10:00 - 01:20:00)* A detailed description of the experience, including encountering cloaked entities, a 12-dimensional brain diagnostic tool, and a neurosurgeon-like being.* The intensity and reality-shattering nature of the experience, surpassing previous encounters with DMT entities.* Sara's decision to take a break from psychedelics after this experience.* The importance of reverence and respect when engaging with the DMT space and its mysteries.* The continuing potential for breakthroughs and the limitlessness of the DMT rabbit hole.Chapter 9: Attention, AI, and the Question of Human Survival (01:20:00 - 01:30:00)* The book as a shared tapestry of experiences, honoring the work of other artists and researchers.* The importance of acknowledging both shared archetypes and individual variations in DMT experiences.* The potential for AI to evolve beyond human comprehension and the need for humans to adapt.* The question of AI's attention span and its potential implications for human-AI interaction.* The need for humanity to overcome its self-destructive tendencies in order to harness the potential of technology and navigate the future.* Sara's personal mission to inspire progress and wonder through her art.✨ Mentions* David Jay Brown - Author of The Illustrated Field Guide to DMT Entities* Diana Reed Slattery - Author of Xenolinguistics* Ralph Abraham - Chaos theoretician at UCSC who taught Sara about wallpaper groups* James Kent - Author of Alien Information Theory* Aldous Huxley - Author of the essay "Heaven and Hell"* K. Allado-McDowell - Co-director of Google's Artists and Machine Learning program* Roland Fischer - Experimental researcher and pharmacologist* Iain McGilchrist - Psychiatrist and author of The Master and His Emissary* William Irwin Thompson - Historian and poet-philosopher* The Tea Faerie - Psychonaut and harm reduction expert* Terence McKenna - Known for his ideas on the Logos and the psychedelic experience* Andrés Gomez Emilsson - Director of Qualia Research Institute focusing on the mathematics of psychedelic experiences* Chris Bledsoe - Known for his family's experiences with entities in a waking state* Stuart Davis - Host of "Aliens and Artists" and known for his encounters with mantis beings* Graham Hancock - Author who encountered "big-brained robots" during a psychedelic experience* Adam Aronovich - Curator of Healing From Healing* Rodney Ascher - Director of the documentary "A Glitch in the Matrix"* Ian McGilchrist - Author and researcher who studies hemispheric specialization in the brain* René Descartes - Philosopher known for his mind-body dualism and views on animals* Helané Wahbeh - Researcher at the Institute of Noetic Sciences, discussed the reproducibility problem in science This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit michaelgarfield.substack.com/subscribe
Host Katherine Mallett, MD Guest interviewee Charles P. Hannon, MD, MBA, discussing his review article, “American Academy of Orthopaedic Surgeons Clinical Practice Guideline Summary Management of Osteoarthritis of the Hip” from the October, 15, 2024 issue (https://journals.lww.com/Jaaos/toc/2024/10150) Article summarized from the October 1, 2024 issue (https://journals.lww.com/Jaaos/toc/2024/10010) Research article “Standardization and Reproducibility of Dynamic Stress Testing for Occult Pelvic Ring Instability” Follow this link to download these and other articles from the October 1, 2024 issue of JAAOS (https://journals.lww.com/Jaaos/toc/2024/10010)and the October 15, 2024 issue of JAAOS (https://journals.lww.com/Jaaos/toc/2024/10150). The JAAOS Unplugged podcast series is brought to you by the Journal of the American Academy of Orthopaedic Surgeons and the AAOS Resident Assembly. In addition, this podcast is brought to you by our sponsor Avance Solo. To learn more about Avance Solo, visit avancesolo.com. Disclaimer: Neither AAOS nor JAAOS are associated with Avance Solo or any products or services advertised. AAOS does not endorse the advertiser or its products or services
In this episode, Will and Helena are joined by Emmanuel Boakye and Lamis Elkheir to share their experiences as scientists and Open Science advocates in the Global South and how they started the African Reproducibility Network (AREN). African Reproducibility Network Website: https://africanrn.org/ Twitter: https://twitter.com/africanrepro Lamis Elkheir LinkedIn: https://sd.linkedin.com/in/lamis-elkheir-b5844092 Twitter: https://twitter.com/lamiselkheir?lang=en Emmanuel Boakye LinkedIn: https://gh.linkedin.com/in/emmaboakye Twitter: https://twitter.com/thescientistgh
Modern science, especially in the genetic and molecular biology spaces, generate vast amounts of data, and require vast amounts of data to be generated for thorough analysis. For example, finding a rare gene mutation such as BCR-ABL as a biomarker for chronic myeloid leukemia is like searching for a needle in a haystack. For a situation like this, dPCR is an ideal method, but high-throughput automation is also needed.Dr. Clarence Lee, Senior Product Manger at Thermo Fisher Scientific, tells how the QuantStudio™ Absolute Q™ AutoRun dPCR suite helps make the benefits of digital PCR available in an easy-to-use high-throughput system. The conversation covers how automation benefits are provided by MAP16 plates, system software, and the AutoRun plate hotel and loading robot. Clarence also talks about customer applications where he sees automation like this being applied to innovate and drive science forward. In the career corner portion, we learn about Clarence's journey from chemist and biophysicist, to roles in industry and his current role as a product manager. He shares what he loves most about his job and what he's most proud of over his career that has spanned several diverse roles. Visit the Absolute Gene-ius pageto learn more about the guests, the hosts, and the Applied Biosystems QuantStudio Absolute Q Digital PCR System.
Treatment Planning Symposium 16th November Hybrid event: https://www.protrusive.co.uk/rx Are you still afraid of raising the Vertical Dimension? You cannot break free from the shackles of single tooth Dentistry if you don't get comfortable with vertical dimensions changes in Restorative Dentistry. https://youtu.be/Nb-LTyzRKuU Watch PDP197 on Youtube In this episode, Dr. Jaz Gulati and Dr. Mahmoud Ibrahim simplify the complex topic of increasing vertical dimension. What is a safe limit of increasing the vertical dimension? They cover the essentials of joint health, muscle stability, and the importance of centric relation (does it actually matter?) Protrusive Dental Pearl: Use Duralay copings for guide planes to ensure stable dentures with a single path of insertion. While eyeballing the prep can be challenging, he suggests requesting acrylic copings from the lab for precise preparation. He explains that technicians survey models to identify undercuts and determine the path of insertion, and instead of manual prepping, he advises using lab-created reduction copings and acrylic jigs to simplify and accurately guide the preparation process. Highlights of this episode: 02:05 Protrusive Dental Pearl Acyrlic Copings for Guide Planes 03:57 Dr. Mahmoud Ibrahim's Introduction 06:05 Personal Experiences with Vertical Dimension 08:45 Challenges and Techniques in Vertical Dimension 14:17 Clinical Considerations (Restorative Dentistry) and Research 21:15 How to Assess OVD Loss? 24:35 Factors to Consider in Increasing the Vertical Dimension 28:41 Treatment Planning: Orthodontics vs. Restorative Management 32:21 Assessing Cases for Vertical Dimension 34:39 Joint Position and Vertical Dimension 39:47 Occlusal Appliances Prior to Increasing Vertical Dimension 45:26 Joint Relationship 50:49 Reproducibility and Stability in Occlusal Planning 53:00 Summary and Final Thoughts on Vertical Dimension This episode is eligible for 1 CE credit via the quiz on Protrusive Guidance. AGD code: 180 Occlusion (Occlusal therapy) This episode meets GDC Outcomes A and C. Dentists will be able to: 1. Explore key clinical considerations and current research in restorative dentistry related to vertical dimension, enhancing your ability to make informed decisions. 2. Understand the relationship between joint position and vertical dimension, and how to assess and manage this relationship effectively. 3. Recall the guidelines for assessing the vertical dimension and the safe limit for this in dentate patients. If you liked this, you will also like Functionally Generated Path Technique – Conforming to Funky Occlusions – PDP168
We welcome back the ReproducibiliTea Podcast with Will and Helena chatting to Nafisa Jadavji and Nele Haelterman about Reproducibility for Everyone (R4E), a community-led initiative to run reproducibility workshops. Show notes: Repro4Everyone - https://repro4everyone.org
Episode 133I spoke with Peter Lee about:* His early work on compiler generation, metacircularity, and type theory* Paradoxical problems* GPT-4s impact, Microsoft's “Sparks of AGI” paper, and responses and criticismEnjoy—and let me know what you think!Peter is President of Microsoft Research. He leads Microsoft Research and incubates new research-powered products and lines of business in areas such as artificial intelligence, computing foundations, health, and life sciences. Before joining Microsoft in 2010, he was at DARPA, where he established a new technology office that created operational capabilities in machine learning, data science, and computational social science. Prior to that, he was a professor and the head of the computer science department at Carnegie Mellon University. Peter is a member of the National Academy of Medicine and serves on the boards of the Allen Institute for Artificial Intelligence, the Brotman Baty Institute for Precision Medicine, and the Kaiser Permanente Bernard J. Tyson School of Medicine. He served on President Obama's Commission on Enhancing National Cybersecurity. He has testified before both the US House Science and Technology Committee and the US Senate Commerce Committee. With Carey Goldberg and Dr. Isaac Kohane, he is the coauthor of the best-selling book, “The AI Revolution in Medicine: GPT-4 and Beyond.” In 2024, Peter Lee was named by Time magazine as one of the 100 most influential people in health and life sciences.Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions. I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (00:50) Basic vs. applied research* (05:20) Theory and practice in computing* (10:28) Traditional denotational semantics and semantics engineering in modern-day systems* (16:47) Beauty and practicality* (20:40) Metacircularity in the polymorphic lambda calculus: research directions* (24:31) Understanding the nature of difficulties with metacircularity* (26:30) Difficulties with reflection, classic paradoxes* (31:02) Sparks of AGI* (31:41) Reproducibility* (38:04) Confirming and disconfirming theories, foundational work* (42:00) Back and forth between commitments and experimentation* (51:01) Dealing with responsibility* (56:30) Peter's picture of AGI* (1:01:38) OutroLinks:* Peter's Twitter, LinkedIn, and Microsoft Research pages* Papers and references* The automatic generation of realistic compilers from high-level semantic descriptions* Metacircularity in the polymorphic lambda calculus* A Fresh Look at Combinator Graph Reduction* Sparks of AGI* Re-envisioning DARPA* Fundamental Research in Engineering Get full access to The Gradient at thegradientpub.substack.com/subscribe
Sometimes experimental results are serendipitous. Listen as Associate Editor Dr. Crystal Ripplinger (University of California, Davis) talks with authors Dr. Nikki Posnack and Devon Guerrelli (both at Children's National Hospital and The George Washington University School of Engineering and Applied Science), along with expert Dr. Silvia Marchiano (University of Washington), about the new research by Guerrelli et al. published in our Call for Papers on Excitation-Contraction Coupling, Electrophysiology, and Arrhythmias. The Posnack Lab typically investigates environmental chemicals and their impact on cardiac function using microelectrode arrays to record electrical signals from human iPS cells. When performing cardiotoxicity experiments, the authors realized that their baseline measurements varied significantly between their different studies, making it difficult to combine datasets. In doing the legwork to identify potential sources of variability and improve their own internal lab protocols, the authors focused on the reproducibility of their experimental measurements using human iPSCs. Listen as we discuss important recommendations for investigators using these cells to improve their experimental reproducibility. Devon Guerrelli, Jenna Pressman, Shatha Salameh, and Nikki Posnack hiPSC-CM Electrophysiology: Impact of Temporal Changes and Study Parameters on Experimental Reproducibility Am J Physiol Heart Circ Physiol, published June 9, 2024. DOI: 10.1152/ajpheart.00631.2023
In this episode, we continue our discussion of replications. We talk about how to analyze replication studies, which studies are worth replicating, and what is the status of replications in other scientific disciplines. Shownotes Mack, R. W. (1951). The Need for Replication Research in Sociology. American Sociological Review, 16(1), 93–94. https://doi.org/10.2307/2087978 Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774 Sidman, M. (1960). Tactics of Scientific Research: Evaluating Experimental Data in Psychology (New edition). Cambridge Center for Behavioral. Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., … Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687 Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., van 't Veer, A. E., Vranka, M., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438 Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist. https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition/ Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518 Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341 Nosek, B.A., Errington, T.M. (2017) Reproducibility in Cancer Biology: Making sense of replications. eLife 6:e23383. https://doi.org/10.7554/eLife.23383
It's return guest season here at Latent Space! We last talked to Kanjun in October and Jonathan in May (and December post Databricks acquisition): Imbue and Databricks are back for a rare treat: a double-header interview talking about DBRX from Databricks and Imbue 70B, a new internal LLM that “outperforms GPT-4o” zero-shot on a range of reasoning and coding-related benchmarks and datasets, while using 7x less data than Llama 3 70B.While Imbue, being an agents company rather than a model provider, are not releasing their models today, they are releasing almost everything else: * Cleaned-up and extended versions of 11 of the most popular NLP reasoning benchmarks* An entirely new code-focused reasoning benchmark* A fine-tuned 70B model, built with Meta Llama 3, to identify ambiguity* A new dataset of 450,000 human judgments about ambiguity* Infrastructure scripts for bringing a cluster from bare metal to robust, high performance training* Our cost-aware hyperparameter optimizer, CARBS, which automatically and systematically fine-tunes all hyperparameters to derive optimum performance for models of any sizeAs well as EXTREMELY detailed posts on the infrastructure needs, hyperparameter search, and clean versions of the sorry state of industry standard benchmarks. This means for the FIRST TIME (perhaps since Meta's OPT-175B in 2022?) you have this level of educational detail into the hardware and ML nitty gritty of training extremely large LLMs, and if you are in fact training LLMs of this scale you now have evals, optimizers, scripts, and human data/benchmarks you can use to move the industry forward together with Imbue.We are busy running the sold-out AI Engineer World's Fair today, and so are unable to do our usual quality writeup, however, please enjoy our show notes and the excellent conversation! Thanks also to Kanjun, Ashley, Tom and the rest of team Imbue for setting up this interview behind the scenes.Video podTimestamps* [00:00:00] Introduction and catch up with guests* [00:01:55] Databricks' text to image model release* [00:03:46] Details about the DBRX model* [00:05:26] Imbue's infrastructure, evaluation, and hyperparameter optimizer releases* [00:09:18] Challenges of training foundation models and getting infrastructure to work* [00:12:03] Details of Imbue's cluster setup* [00:18:53] Process of bringing machines online and common failures* [00:22:52] Health checks and monitoring for the cluster* [00:25:06] Typical timelines and team composition for setting up a cluster* [00:27:24] Monitoring GPU utilization and performance* [00:29:39] Open source tools and libraries used* [00:32:33] Reproducibility and portability of cluster setup* [00:35:57] Infrastructure changes needed for different model architectures* [00:40:49] Imbue's focus on text-only models for coding and reasoning* [00:42:26] CARBS hyperparameter tuner and cost-aware optimization* [00:51:01] Emergence and CARBS* [00:53:18] Evaluation datasets and reproducing them with high quality* [00:58:40] Challenges of evaluating on more realistic tasks* [01:06:01] Abstract reasoning benchmarks like ARC* [01:10:13] Long context evaluation and needle-in-a-haystack tasks* [01:13:50] Function calling and tool use evaluation* [01:19:19] Imbue's future plans for coding and reasoning applications* [01:20:14] Databricks' future plans for useful applications and upcoming blog postsTranscriptSWYX [00:00:00]: Welcome to the Latent Space Podcast, another super special edition. Today, we have sort of like a two-header. John Frankel from Mosaic Databricks, or Databricks Mosaic, and Josh Albrecht from MBU. Welcome.JOSH [00:00:12]: Hey, glad to be here.SWYX [00:00:14]: Thank you for having us. Hey, so both of you are kind of past guests. Jonathan, you were actually one of the most popular episodes from last year talking about MPT7B. Remember the days when we trained large models and there was 7B?JONATHAN [00:00:30]: Yeah, back when reproducing LLAMA1-7B was considered a huge accomplishment for the field. Those are the good old days. I miss that.SWYX [00:00:38]: As the things have accelerated a lot. Actually, let's do a quick catch up and Josh, you can chime on in as well. So Databricks got acquired. I talked to you at New York.JONATHAN [00:00:45]: Mosaic got acquired, although sometimes it feels like Mosaic acquired Databricks because, you know, we're having a lot of fun being here. But, you know, yeah.SWYX [00:00:52]: Yeah. I mean, you are chief scientist now of Databricks.JONATHAN [00:00:55]: Chief AI scientist. Careful with the title. As much as I would love to understand how Spark works, I'm going to have to defer that to much smarter people than me.SWYX [00:01:03]: Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX. Is that the thing that most people should be aware of?JONATHAN [00:01:13]: Actually, that's no longer the most recent news. Honestly, the most recent news, we announced this, but it was at our Data and AI Summit last week. So it was announced among like 100,000 other things, is that we finally released our text to image model, which has been a year in the making through a collaboration directly with Shutterstock. There was a lot of work put into finding a dataset that we were comfortable with working on and trying to build a model that honestly, I felt like I could trust and that others might be able to trust to put out in the world. So that model was released last week. It's unfortunately just available via API due to the fact that the data is quite sensitive and quite valuable. It's Shutterstock's entire business in a lot of ways, but I'm still really excited that there's now a model that is trained on a dataset where the provenance of every single image is known, and it's a damn good model. So I'm really proud of the team on that.SWYX [00:01:55]: Yeah, amazing. Josh, do you have any thoughts on image model questions?JOSH [00:01:59]: That is not my area of expertise, but I was excited to see the release of it last week as well, and very happy that you guys did a nice job on the data side of everything there. So that was cool to see.SWYX [00:02:09]: I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs. So what is the Shutterstock model? Like, I guess, is this the house model for Shutterstock? Is this Databricks' version of the Shutterstock model? Like, what is this?JONATHAN [00:02:22]: The way that I would think about it is that Shutterstock is doing an amazing business in AI across the board. Their dataset is kind of widely known to be the best stock photos dataset in the world, the most comprehensive, the biggest. When you think about like, what dataset am I going to train a multimodal model on? You call Shutterstock. And I, at least I've heard in the news, like OpenAI, Google, Meta, Apple have all called Shutterstock and made those deals. So a lot of models have had Shutterstock data incorporated into them. But this is the only model I know of so far where it was, you know, exclusively and specifically trained just on the vanilla Shutterstock data. There was nothing else mixed in. We didn't go and scrape the web and find other data or combined datasets or anything like that. And so this is, in some sense, the house blend. But the other piece is that it's just a dataset where the provenance of every image is known in public. Where did the data come from? It is the Shutterstock collection. That's it. You know, nothing less, nothing more. And certainly being at Databricks, if I've learned one thing, I've learned about enterprise customers and what they want out of AI. And one of the things they ask for most is just, what can you tell me about the data the model was trained on? And here, especially for text to image models, where images are just tricky subject matter, there's been a lot of kind of legal conversation about images, especially. It's nice to just have something where I can point to it and say, you know, if you want to know where the images came from, these are what they are and this is how they got there.SWYX [00:03:36]: I will talk a little bit about Databricks because it's relevant to the rest of today's episode. So Databricks, sorry, I keep misspeaking. It's DBRX.JONATHAN [00:03:46]: DBRX, actually, there's been a pronunciation update. It is now D-B-Rex. So we have decided to add a dinosaur mascot because what model doesn't like a mascot? So literally, I wish I could pull it up. There is a little plush dinosaur that we had made. It's like the world's cutest dinosaur, but it is the official mascot of D-B-Rex. And there's a little dinosaur logo that, you know, you'll probably see around a little bit more because DBRX is a mouthful, but D-B-Rex, like, you know, it's just kind of...SWYX [00:04:13]: Rolls off the tongue. I love mascots. Like every company should have a mascot. And I think Hugging Face got it right. You need an emoji mascot because that's the minimal viable image.JONATHAN [00:04:21]: I probably shouldn't talk at all about, you know, Velociraptor, but, you know, that's a, maybe that's something we can talk about later in the summer. I'll just leave it at that.SWYX [00:04:28]: Okay. That's a hint to names. I feel like your names leak a lot of alpha. So just to quickly cover the headline details, DBRX, as Make Sure Experts model, that's fairly big, 132 billion total parameters, so 36 billion active on any input, pre-trained on 12 trillion tokens of text and code, and did really well on evals to the point where you had to dye your hair blue. That's my high level conclusion.JONATHAN [00:04:53]: Never make a bet with your team two weeks out from model launch, even when, you know, human eval is looking quite bad. Because if you set some bar, even if it's arbitrary and you think there's no way in hell they're going to hit it, apparently money doesn't motivate people anymore. Humiliating their boss motivates people. So Josh, you should really take a hint from this. You know, you cannot pay someone enough money to make up for you dyeing your hair blue.JOSH [00:05:15]: I'll keep that in mind for our next model.SWYX [00:05:17]: It works. So speaking of Imbue's next model, perhaps Josh, you want to actually just say hi to the general sort of latent space audience and talk about what we're releasing today. Yeah.JOSH [00:05:26]: I'm Josh, CTO of Imbue, and we're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models. So I think right now, training foundation models from scratch is like a very difficult, time-consuming, expensive, kind of risky endeavor, especially for smaller companies. And the things that we're releasing hopefully make that at least a little bit easier. So the things that we're releasing fall into kind of three different buckets. One is infrastructure and scripts for dealing with the kind of hardware and hardware failures and understanding how well is the actually lowest level of thing actually working so that you can actually do your training at all and at a reasonable speed without having to constantly restart, etc. So infrastructure and training scripts. A second set of things is around the evaluation. So after you've trained it, like how well is this actually working and how do you know how well it's working? We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like pool queue or ANLI, where we've gone through and kind of cleaned up the data as much as possible by looking at all the ones that models get wrong or that are flagged for ambiguity and also our own kind of private reproductions of those where we've done like a kind of clean room black box, like, okay, this is what the data set is supposed to be. Here are some examples. Let's make our own version of this to make sure that there is no data contamination, etc. To make sure that we're actually, you know, not testing on train. And then I think a final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used in the process of cleaning these evaluations and we also hope will be helpful for other people training kind of similar models. And then the third thing is CARBS, our hyperparameter, our cost-aware hyperparameter optimizer, which was especially helpful for being able to experiment at much smaller scales and then scale those experiments up to the much larger scale kind of on the first try without having to retry it. You don't want to be training, you know, 10, 20 different 70B models. You really want to get these larger modelsSWYX [00:07:30]: right on the first try.JOSH [00:07:30]: And so the ability to kind of tune things very precisely and learn scaling laws, not just for, you know, the like data and flops, but also for learning rate and all the other hyperparameters and see like how should you scale these things up was extremely valuable to us as we were training the larger models. Yeah, that's a lot of stuff.SWYX [00:07:49]: Yeah, exactly. So there's a bunch of stuffJOSH [00:07:50]: we'll have to go through all of it.JONATHAN [00:07:52]: Yeah, I just want to throw in how excited I am about this. This is the stuff that nobody ever talks about. That is the difference between success and failure in this stuff. Like, can you get your cluster to run? Can you get software on your cluster? Can you figure out what broke? Because fault tolerance is still not really built into any of the fundamental primitives of training models. And so if something breaks, you have to go figure out what broke, your job stops, you have to restart your job. It is a nightmare just to get to the point where anything can train on the cluster. A basic MPI hello world that has the GPUs talk to each other is hard enough, let alone actually training a model, let alone getting good performance out of the GPUs, let alone actually getting a model that converges to anything interesting. There's so many levels of things you have to accomplish. This is the kind of stuff that matters. I think to a point that Josh made earlier, before we got on here, there are plenty of weights out there. Nobody's released this.JOSH [00:08:46]: Yeah, that was part of the motivation actually is that there are lots of other things that are complimentary, but I have not seen nearly as much discussion about some of these other things that we think are pretty important. I mean, in some sense,SWYX [00:08:56]: I'm very excited to have Jonathan on because this is a little bit, you're a bread and butter with Mosaic. And I think you've released some part with Composer. And I think it's just really interesting to see like a different take, basically a full stack take that's kind of open source today.JONATHAN [00:09:18]: Yeah, it's really kind of, it's been an ordeal to figure this out. And every time something changes, whether it's a new GPU or even a new driver update, you get new creative errors and new things go wrong. And, you know, we've dealt with the weirdest things from, you know, our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center. Like, you know, Porch Pirate basically had stolen our InfiniBand cables back when those were hard to come by. To like, you know, weird recalls of switches to like the strangest stuff has happened. I have my favorite GPU failures I've seen, like ones where the GPU doesn't fail, it has a correctable memory issue and the memory correction causes the GPU to become a straggler and hold up the whole job. Like weird stuff happens and figuring out how to not just identify all of that, but then eventually productize it, is in some sense, the entire story of Mosaic and now Databricks in terms of our ML offering. Really, the thing we offer is we have gone through this suffering and figured out how to even productize that. It has been a pain in the butt.SWYX [00:10:20]: Yeah, it's a lot of work.JOSH [00:10:20]: I think my favorite failure was GPU is just giving wrong math. Like if they give errors, great, because you can see the errors, but if they just give you the wrong math back, not so fun.SWYX [00:10:30]: When did they give you wrong math?JOSH [00:10:32]: Like literally you could just, you know, add two things. For example, the numbers come back. They're not the numbers that they're supposed to be.JONATHAN [00:10:40]: I think it's important to say at this stage, just because like it, I think it goes without saying for Josh and I, but it's worth saying here, this isn't to say that like anything is wrong with us. It's not like NVIDIA did a bad job or, you know, Mellanox did a bad job or the like the server builder, the data center operator, the cloud provider, like the million other parties that are involved in building this. We are running these insane chips that are huge and complicated and built on tiny transistors at insane frequencies with insane heat in data centers that for the most part, were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part. It's fun to joke about all the weird things we see. This is not to say anybody's done anything wrong. This is just kind of part and parcel of working on a massive cluster running at multiple megawatts of power at a time.SWYX [00:11:32]: It's crazy. Yeah.JONATHAN [00:11:33]: So optical cables, like all sorts, like everything.SWYX [00:11:37]: I'll take the opportunity to start going to the sort of infra piece. There's just like a description of the infra just to give people a sense of what we talk about when we talk about massive clusters. So I'm just going to read off the blog post here. This post is about one cluster that has 4,092 H100 GPUs spread across 511 computers. They use unified fabric manager nodes, which manage the infinite band network. And you talk a little bit about your networking. Is there anything unusual about this setup that you'll call out to people?JOSH [00:12:03]: Yeah, actually this particular cluster is a little bit non-standard. The normal, like vanilla setup for these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 1024 GPUs instead of 4,000. Here we have a larger cluster. As you start to get into the larger clusters, the networking becomes a little bit more custom. It's a little bit more, it's a little bit trickier. It's a little bit more difficult to get these things to all be able to talk to each other at the same speed. And so this has, in this particular case, this is a three tier network architecture instead of two tiers, kind of the normal one. So most of the clusters are a little bit smaller. As you get to even larger scales, then this becomes even much more complicated,SWYX [00:12:43]: much more expensive.JOSH [00:12:43]: So we chose this particular scale, kind of knowing our own workloads and kind of what we wanted to do. This was kind of the right size for us. But yeah, I think it's not exactly vanilla already. It's already getting into kind of the custom territory.SWYX [00:12:54]: So my understanding is that there, and is there any part of this that comes with the Voltage Park deal that you guys had? Is that part of the hardware that you got from the deal with them?JOSH [00:13:04]: Yeah, so we worked really closely with Voltage Park to set up all their clusters and infrastructure and everything and kind of decide even like what to order, how should the networking work? Like we were very involved in kind of the construction and bring up of this. And that's what this post is about, is about that process of like bringing up all these, there's like different clusters in different places of different scales. So in this particular post, we're talking about this one 4096 GPU, but there are other clusters that they have as well. And we were very closely involved with figuring out the exact architecture and kind of the trade-offs that go along with picking, you know, those exact components. You really don't want to like place the wrong order because it takes months to get it and it's very expensive. So yeah, we were happy to help out with that.JONATHAN [00:13:43]: And then your bit of good cables get stolen.SWYX [00:13:44]: Yeah, yeah, exactly.JOSH [00:13:47]: We wanted to make sure that we ended up with compute that would work for us and that would also work for their other customers. And so we kind of helped design something so that we would get exactly what we were looking for. We knew that these kinds of details would be super important and that getting down to the level of the hardware and like having these good scripts and everything was going to be a core part of like actually getting this to work. I'm very glad that we did that. I don't think that most companies kind of take that full stack approach, but for us, it certainly paid off.SWYX [00:14:12]: Yeah, it's basically sort of built to spec. It's interesting that relationship because you usually, for the rest of us who don't operate at your scale, we take whatever we can get from cloud providers, but you are basically co-designing from the single machine up. And you described that a little bit. Do you want to take us through the process that you described here?JOSH [00:14:27]: Yeah, so for the actual, like the blog post and kind of bringing these machines online.SWYX [00:14:32]: Yeah.JOSH [00:14:32]: So yeah, I think the process, as we have it broken down in the blog post, there's kind of a few different layers. First is like getting the individual machines to work at all and then getting the machines to actually be able to talk to each other. So getting the InfiniBand networking to work and then getting to a point where, you know, not just the machines are working and they can talk to each other, but everything is actually working correctly. There's a big gap between like it's working at all to it's working perfectly correctly. And then after you have all this stuff working perfectly correctly, nice and healthy, then now you get into kind of the software data, like training issues. And then after that, you're still not done. Like now, even once you're training at full speed, things are going to fail over time. Things are going to change. There's going to be new, you know, firmware updates. Like how do you kind of deal with this change and flux over time without going crazySWYX [00:15:16]: and pulling your hair out,JOSH [00:15:16]: trying to like reproduce things or understand why there were regressions. And so there's a lot of work to kind of automate the infrastructure tooling as well. And kind of the first step, like bringing these things online in the first place, you know, you have hundreds of machines at this point. So you don't necessarily want to be like walking around with like a CD-ROM or a USB drive, like plugging it in with your keyboard, like hitting next, next, next on the OS install. That's not how this works. You do that for one machine. And then you use, we use this thing called Metal as a Service to bring up all the other machines. So it's a kind of server that can kind of install the operating system on these other machines. So most like when you're talking about these machines, like each machine is, you know, on the order of hundreds of thousands of dollars. So they usually come with a kind of out-of-band management interface as well. So they don't, they have their InfiniBand networking. They have their normal 100 gigabit per second Ethernet networking. These are like dual, redundant, et cetera. And then you also have this extra out-of-band management network. So you can log in and you can see like the boot screen or you can see the blue screen of death. You can like get in there and actually see what was wrong, which is pretty fun. And it makes it like possible to automate a lot of this work. So the beginning of that, and the blog post goes into much more detail about like exactly how we set these up and kind of the other errors that we ran into. When you're bringing these online, you'll definitely have failures. Even if they all worked in the factory, they get shipped, some parts come loose, something fails, something goes wrong. So when you're bringing them online, there'll be some that don't quite work for all sorts of reasons. As you start to be working with machines at this scale, like if something happens one in a thousand times, you're like pretty likely to see it. And so you can get pretty rare, weird things, especially since we had fairly early builds and fairly early versions of this hardware. Like these are some of the like first machines that were ever produced, some of the first GPUs. So you've got some extra special things there. We definitely worked with Dell, for example, on making fixes in the firmware level to be like, okay, like this thing is wrong. Like we need to update this at the firmware to like actually fix this particular thing. So we worked pretty closely with Dell and Nvidia. Yeah, that's what I'm saying. Like this stuff gets complicated. And the thing is like, you know, taking a step back, the whole reason we're doing this, right, is that we knew that this was going to be complicated. There would be these kinds of failures. And if we're just using, you know, AWS or some other cloud provider, these errors are still gonna be there and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong. And so we would much rather be able to like call up Dell and say, hey, this isn't working. And they're like, yep, okay, cool. Let's debug it together. Oh, I see. Yeah, cool. We'll ship a firmware update and actually fix this for you. That was a much better experience than like, great, just magically fails. I guess we restart and hope that that machine goes away. Like that's not a very good place to be. So yeah, that's kind of the first place is getting to a place where like GPU training is working on your single node machines. You can observe stuff. We have tons of tooling around like, you know, Prometheus and all sorts of other tools for understanding what's going on in these machines because you don't want to be like logging into each one and looking at the temperature or something you really need to have tooling to collect all these metrics, et cetera. Unfortunately, all of the scripts that we have for this are like for this entire cluster and for all this infrastructure are a little bit like special purpose for our particular thing. So it's not that every script that we have, it's not that you can just like take this and plug this in. Even if we did open source all the tooling that we have, you'd still have to do like a lot of work to open source it. What we are releasing is as many of the things that we can that are going to be useful for other people. You're still going to have to have some way of kind of managing these things, making your own like logging aggregators, et cetera, et cetera. So that's kind of bringing them up to the like, you know, the single nodes that are working. From there, it goes into, I'm happy to keep going if you want. Well, I just want to leave the opportunity for JohnSWYX [00:18:53]: to comment if there's anything that's different from how he runs things.JONATHAN [00:18:57]: Oh, I mean, all I'll say is I'll endorse this and say this s**t is hard. Like this is really, really hard. And, you know, I have a special props to, you know, the folks in Vue because they were building this from the ground up. You know, at Databricks and at Mosaic, we typically work with cloud providers because some of this stuff is just, there's too much to handle. It's complicated. There's a lot to deal with. And this doesn't even get into things like physical security, you know, securing power if you're the data center operator. Like this gets infinitely complicated and you have to abstract somewhere. Like, you know, and then you get to the folks who are literally building their own custom chips and like, good God.SWYX [00:19:36]: Like, oh my God, that's, you know,JONATHAN [00:19:38]: if you're one of those folks, you're having, you know, pour one out for the infra people at some of the AI chip startups who are having a really, really interesting time right now. But this stuff is really hard. And I don't think we talk about it much because there's so many other things that are hard. But the other hard things, I think everybody's becoming pretty familiar with at this point. This is something that I don't think there's ever really been a comprehensive discussion of, at least not that I've seen.SWYX [00:20:00]: Yeah, so my impression is that you guys, Mosaic, have your own software for sort of spinning up and down machines, just like Imbue had to build. But Imbue probably, it sounds like Imbue, you guys went fuller stack. I don't know how to describe it. Like Mosaic is not working with Dell on like their firmware.JONATHAN [00:20:21]: No, no, we're typically working with like, you know, pick your cloud provider on their Dell firmware or what have you. Like, it's kind of, I think one of the things, I don't know, Josh, you can correct me on this. It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens. Like somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provider is not to have to think about power contracts. Somehow I'm still asking them about which InfiniBand provider they used this time to see if this is part of the bad batch of cables I encountered on that cloud provider or what have you. Or like, we're still talking about a firmware update from pick your provider. You can't not do this. It's convenient that they have data center staff who are worrying about what to send back to which provider when, and they have people who can go and wait for the InfiniBand cables so they don't get stolen outside. But, you know, it's kind of, it's impossible not to really go full stack if you're thinking about the infrastructure at all. I don't know, Josh, correct me. No, I think that's right.JOSH [00:21:17]: That's what we expected from the beginning as well, is that we would inevitably have to get into the details here. And I'm glad that we kind of just planned for it. I think it made it a lot easier from our perspective to have direct control over this. Instead of having to go to the cloud provider that goes to the data center, that goes to the supplier, we could just go direct to NVIDIA or DellSWYX [00:21:37]: or the data center,JOSH [00:21:37]: whoever was responsible and be like, hey, this thing needs to change. And they're like, oh, okay. Yeah, that is our responsibility. Great, we can fix that. So it was just a lot easier for us to fix these bugs than if we had to go through an extra layer of email.SWYX [00:21:48]: Something we discussed in the pre-show was that you had a rule of thumb for your cluster of reliability. You say here in the post, by and large, you expect around 3% of your machines to break every week. So you're basically going to turn through all your machines in a year.JOSH [00:22:04]: As it says in the post. So that would be true if it was a uniform failure like that. But as it says in the post, it's usually these kind of problematic nodes. And to be clear, that is the number that we've heard from other people is like they're having about 3%. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig into a large, maybe larger number than we should have of these failures and get to the root cause of it and be like, oh, okay, like that's exactly what's going wrong.SWYX [00:22:33]: How do we fix this?JOSH [00:22:33]: How do we prevent this from happening? How do we make automated checks for this so that if it does happen, it just goes back to whoever owns that particular part of the process and they can fix it immediately.SWYX [00:22:43]: And that's part of what you're also open sourcing, which is the health checks, right? You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is.JOSH [00:22:52]: That one is just a lot of stuff.SWYX [00:22:54]: Yeah.JOSH [00:22:55]: That one is one where we realized that actually like when these machines boot, sometimes they wouldn't actually boot cleanly all the way. Or when they rebooted, they had problems that they didn't have when they were working before, which was kind of frustrating. Like usually if you restart your computer,SWYX [00:23:08]: it gets better.JOSH [00:23:08]: Here you restart. It did not get better.SWYX [00:23:10]: It got worse.JOSH [00:23:10]: That was very frustrating. So this health check looks at every particular line we've ever seen from the boot, like in D message, like every single log line that your computer emitsSWYX [00:23:21]: and says like,JOSH [00:23:21]: have we ever seen this before?SWYX [00:23:23]: Is this expected?JOSH [00:23:23]: Is this in the right order? Or is there something out of place? If there's anything out of place, let me say, okay, great. Like now it goes into this, like longer, more triage list of like, all right, great. Like, is this acceptable?SWYX [00:23:33]: Should we flag this?JOSH [00:23:33]: Like, should someone take a look at this? So we're looking down at a very, very granular detail level, what's happening on these computers to make sure that nothing is out of place. And that's critical because without that, if you're running your training, as Jonathan said, and this thing is slow, like what are you supposed to do? Right?SWYX [00:23:49]: Like you really,JOSH [00:23:49]: you really want to be very certain that like all 4,000 of these GPUs are working like they're supposed to.SWYX [00:23:54]: We know that.JOSH [00:23:54]: And so if it's slow, it's because like we messed up the config or something else and not because of this earlier thing that's like really hard to detect in software later.JONATHAN [00:24:01]: Yeah. I think the, I'm just curious to ask,SWYX [00:24:03]: like, you know,JONATHAN [00:24:03]: suppose you were to set up another, let's say another H100 cluster and it were at a different data center. And instead of the vendor being Dell, it was super micro or what have you. How much of this would be repeatable? And how much of this would you have to redo? I, you know, I genuinely don't know.SWYX [00:24:18]: A decent amount.JOSH [00:24:19]: I think it would go a lot faster the second time. I think there's lots of learnings that we had. And also the blog post,SWYX [00:24:24]: you know, yes,JOSH [00:24:24]: we are releasing the health checks, releasing some scripts, but a lot of the valuable stuff is also in the blog post itself, in the details and kind of the, you know, the learnings that we've had and the sort of errors that we run into. We tried to as much as possible surface those to other peopleSWYX [00:24:36]: could learn from thoseJOSH [00:24:36]: and avoid the same mistakes or failures as well. But I think it would go a lot faster.SWYX [00:24:41]: Although, yes,JOSH [00:24:41]: there would certainly be some things that'd be a little bit different. I mean, there'd probably be different CPUsSWYX [00:24:46]: or whatever,JOSH [00:24:46]: but I think a lot of that stuff is less,SWYX [00:24:49]: it's less,JOSH [00:24:49]: that's the like, that's less variable. I think most of it would apply the second time around. Although I'm sure next timeSWYX [00:24:56]: we're building one,JOSH [00:24:56]: it'll probably be, you know, at a scale that's 10x as big with a different chip or something like this.SWYX [00:25:00]: And then who knows?JOSH [00:25:01]: Yeah, with Kinect X8,JONATHAN [00:25:02]: that will have its own fun behavior and all that good stuff. Yeah.SWYX [00:25:06]: Perhaps there's something that people don't discuss about, and you don't even talk about this in the blog, but I always wonder is what is the timeline that's like kind of reasonable for this amount of work, at least the initial stages? And also what does the team composition look like for setting up a cluster, right? Like what are the mix of skills that you typically would require to get all this going?JOSH [00:25:27]: I'm, I can't really speak to typical. One thing I am very proud of is how much we accomplished with such a ridiculously small team. Like our infrastructure team is like, you know, fluctuates from week to week, depending on like how many things are on fire and how much we need to build. But it's like between like three and six people, like it's small. It's not like some huge team of like tons and tons of engineers. But those people are very, very good at what they do. And so that has allowed us to get a lot of mileage out of out of these things. I think it's not that we're building everything, right? It's not that three to six people build this whole thing. I definitely want to like, you know, say thanks very much to Dell and H5 and NVIDIA and the other people that have done a lot of the work, like to bring up this cluster, you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables. So that's 24,000 things that need to be plugged in. Like that's just a lot of stuff to plug in, right? And you don't want to mess it up. Like each one needs to be done correctly. Like it's a little bit loose. Like it doesn't really work.SWYX [00:26:23]: If you break it,JOSH [00:26:23]: you need to replace it. Like there's a lot of workSWYX [00:26:26]: that goes into this.JOSH [00:26:27]: Yeah.SWYX [00:26:28]: And then, you know,JOSH [00:26:28]: that's just like that's it. That's if you were to do everything right the first time.SWYX [00:26:32]: And if you didn'tJOSH [00:26:32]: have to fix anything. But inevitably, you know, you will have to replace something, which means like taking all the wires out, pulling the thing out, taking all the GPUs out, going and fixing some cable, putting it all back correctly, putting it back in, doing this every time. So there were a lot of people at Dell, NVIDIA and at H5 that all helped a ton with this stuff. I don't know the exact size of the Dell team. It also fluctuated over time.SWYX [00:26:55]: Yeah, excellent. And then, you know, you so you have all the hardware set up and now you're firing it up for a single node. There's a long description that you guys have about just like monitoring the MFU, right? And what each situation might look might be indicative of. One of the most interesting things to me that I saw from here is like, you know, if training immediately starts off at 60 to 80% MFU, something's wrong.SWYX [00:27:24]: But like, you know, like what what are like, you know, some anecdotes or, you know, notable scenarios here that you might you might call out as maybe counterintuitive or super interesting.JOSH [00:27:36]: There's just so many of them. I mean, one of them, which I think is probably pretty common, like common knowledge by this point. But like we did have a sort of likeSWYX [00:27:46]: which one was this exactly?JOSH [00:27:47]: I think for the MFU, like gradually getting worse over time. I think that one, when we saw that the first time we were like, what the heck is going on? Like, why does it get just like a little bit worse? This is so strange. Like, what is it getting lazy or tired or something? Like, is it heat? Like what's going on? And in this particular case, it was memory fragmentation. Because you have hundreds of machines, they're doing garbage collection slightly different times. And then they get slightly further apart and slightly more and more jittered until eventually they're all happening kind of at random times. And just like really messing up each one of your steps. So you just turn off garbage collection and call it a day, basically,SWYX [00:28:20]: to be honest.JOSH [00:28:20]: There's other things you can do if you want to be a little bit more sophisticated about it. But you can also just manuallyJONATHAN [00:28:25]: have it all garbage collect on some interval. Like that's what we've done. We just have a garbage collection callback that just runs. But I've seen the exact same thing.JOSH [00:28:33]: Yeah, yeah, exactly. So I thought that one was kind of funny. And we did trace that one down and look and we did find the actual call. Like, again, this goes to like having good tools. So we had really good tools where we could look at a bunch of like actual traces in C and be like, OK, cool. This is the thing that's taking a lot of time. Or like, you know, this is the thing that doesn't quite line up here. Like, oh, I guess it's garbage collection. OK, cool.SWYX [00:28:52]: Interesting.JOSH [00:28:52]: Yeah, let's just try taking it off.SWYX [00:28:54]: OK, great.JOSH [00:28:54]: That's what it was. Now we can fix it. So for each of them, like basically bugs are not hard if you have good tools. But if you don't have good tools, bugs can be very, very hard. So similarly for like heat, another thing that we saw was like, oh, you know, the CPU is getting throttled. OK, well, it's easy to see if you're monitoring the CPU throttling or monitoring the heat. If you're not monitoring that, it's really hard to know why it's just suddenly one of them is going slower. I noticed also in the pieceSWYX [00:29:17]: that you mentioned FSDP with 0.3. Actually, we met, I went to iClear and Guanhua from the DSP team was there presenting 0++. I was wondering if you want to make any call outs to, you know, particular open source or open library or open whatever implementation teams that were super helpful in your process. I think we ended up actuallyJOSH [00:29:39]: pulling from a whole bunch of different ones to pull things in into our own particular pipeline. So we use things from NVIDIA's, you know, Megatron stuff. We use stuff from probably DeepSpeed. I think we pulled in a bunch of different pieces from a bunch of different places. So it was really nice to see all these working open source like examples. I think I really appreciate all the effort that has gone into actually tuning these things because you can tune them, but it's a lot of work to like tune this stuff and do all this stuff from scratch. It's really nice to have like a working example. I think those are probably the two biggest ones, DeepSpeed and Megatron alone, but there are probably other ones as well.SWYX [00:30:13]: Is there a particular thing in the ecosystem where you would call out as like, you know, there should be something here that is open source, but like it's not really, it's like everyone kind of builds it on their own. I want to say something with the file system because everyone talks about the file system eventually.JOSH [00:30:28]: The file system actually was,SWYX [00:30:30]: I mean, we did somethingJOSH [00:30:31]: kind of dumb there. Like we have our own sort of local mirror so that we can, you know, like a crappy version of S3SWYX [00:30:38]: that's local,JOSH [00:30:38]: but it's just a pretty simple script, right?SWYX [00:30:41]: Like I think we run likeJOSH [00:30:41]: a little web server that just like serves files and then, you know, it can upload themSWYX [00:30:45]: and download them.JOSH [00:30:45]: Okay, great. And part of the reason we did that is that our internet connectionSWYX [00:30:50]: in the beginningJOSH [00:30:50]: was not the like full speedSWYX [00:30:52]: one that we wouldJOSH [00:30:52]: eventually have. And so we are a little bit more kind of bottlenecked in terms of internet bandwidth. And so we had this. I think we looked at a bunch of services out there like Minio and some other ones, but a lot of these like come with a lot of extra overhead and maintenance. And since we already have so much infrastructureSWYX [00:31:09]: to deal with,JOSH [00:31:09]: we kind of didn't want to, you know, bring in a whole other like cloud provider, virtualize something, something.SWYX [00:31:14]: We just wanted something simple.JOSH [00:31:14]: So we went with that, which has been quite helpful. Like our toolsSWYX [00:31:19]: are usually quite simple.JOSH [00:31:19]: It's like Bash and Python and SSH and Docker. Like we'd like to keep things simple so that's easier to debug, like less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes,SWYX [00:31:30]: for example,JOSH [00:31:30]: and we just directly launch these things. And it's just been much easier to debug this way. One tool actually that does come into mind that I will call out is Kraken from Uber. That was great. We love that tool. We were a little bit skeptical. What is it?SWYX [00:31:44]: I'm sorry. Yeah.JOSH [00:31:45]: So Kraken is this, yeah, it's a distributed like Docker registry, basically, that uses BitTorrent to like transfer things between the machines in a sort of nice optimal way. Like in the very beginning, the naive way is like you have this one Docker registry, which was outside of the cluster. So every time we change an image, you know, there's many gigabytes that each of the 500 machines needs to download.SWYX [00:32:07]: So that just takesJOSH [00:32:07]: a really long time. So what this thing does is like just one of them downloads it and then like they all sort of broadcast all the pieces to each other. And it was just like a really nice, fast way of getting these images down. And it was very robust.SWYX [00:32:19]: Like there's a lotJOSH [00:32:19]: going on under the hood, but I think it's a pretty cool tool that we haven't really had any bugs with it at all. Amazing.SWYX [00:32:26]: Yeah. I mean, that's all my questions, I guess, for the info piece. I don't know if, John, you had something that you were sort of burning to ask or.JONATHAN [00:32:33]: No, all I can say is just sameSWYX [00:32:36]: in a lot of places, like, you know, and they're done thatJONATHAN [00:32:38]: seeing this plus one. I think the one big difference, you know, perhaps in philosophies is we've tried to basically standardize on as much commodity stuff as possible, just because, you know, I think the reason I asked about trying to do thisSWYX [00:32:50]: on multiple differentJONATHAN [00:32:50]: pieces of infrastructure is like, I think we're running on like six or seven different clouds right now. And everybody has done something slightly different. And my gosh, the little differences add up as you know, you've seen. And so, you know,SWYX [00:33:04]: our philosophy has been like, whatever the hellJONATHAN [00:33:05]: we can standardize, please let's standardize it. Like vanilla off the shelf FSDB.SWYX [00:33:10]: And like, you know,JONATHAN [00:33:10]: we wrote our own data loader, but we've tried to make that as much of a standard as we can across our infrastructure and in Databricks, because things just start getting really complicatedSWYX [00:33:18]: or like we useJONATHAN [00:33:18]: Kubernetes extensively because it at least gives us a uniform set of APIs. Like that's our hardware abstraction layer to a certain extent for everything else. So it's just, you know, a difference in philosophy there. But otherwise, like, yeah, this stuff is really, really hard. And I feel like we take for granted how much of this, you know, is done for us when you go and you just query chat GPT, for example. Like, oh my God, everything going on underneath that, you know, it's kind of a miracle that the machines boot up, let alone that you can like query a giant language model that's probably doing inference across multiple machines and was trained across thousands of machines. Like, you know, minor miracle.SWYX [00:33:54]: Yeah, it is an awesome amount of power that we invoke with a single API call that we take for granted these days. It's absurd. Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup. We do have our ownJOSH [00:34:19]: sort of replacement. It's just a much simpler version of Kubernetes. Kubernetes is really designed for running services, not for running experiments. Like that's not its like main architecture. And so for us, like we have everything that's like, cool, you're going to run an experiment. So you want it to run to completion, right?SWYX [00:34:34]: OK, great.JOSH [00:34:34]: Like the primitives are sort of built around a slightly different style. And that makes it a lot easier, like just a lot simpler to fit that the nature of like these machines are going to disappear. They will need to be rebooted for infrastructure upgrades. They will like something will happen to the GPUs. Failure is like baked into this as like a core part of our infrastructure. So it's not that we don't have an abstraction. It's that it's a sort of simpler, more tailored abstraction for the particular work that we're doing.JONATHAN [00:34:58]: Yeah, I think it all depends on what your goals are. And like, I think the challenge in a lot of the deep learning stuff right now is that people are trying to like, people often build things that are more complicated than necessary to get the job done. And the complication is the enemy of everything. You know, don't use a fancier parallelism strategy than you have to. Don't use a fancier set of libraries than you have to.SWYX [00:35:18]: Don't do anythingJONATHAN [00:35:18]: that you don't have to do because it's hard enough as it is. Like, don't overcomplicateSWYX [00:35:23]: your own life.JONATHAN [00:35:23]: Don't try to bring in more tools or more fancy architecture tweaks if you absolutely don't have to.SWYX [00:35:29]: Like getting to the minimumJONATHAN [00:35:30]: necessary to get the job done. And it's really tempting to want to try to use everything. So like, I totally understand that one.SWYX [00:35:37]: I think the last piece I'll maybe call out is that I'm just going to weave this in just because I see the opportunity to do it. Are there any infrastructure shifts that need to be, that need to rise because of changing architecture? So I think, for example,SWYX [00:35:57]: you're announcing a dense model, a 70B dense model, whereas John just worked on DBRX and the image-to-text model, which presumably has different bottlenecks.JONATHAN [00:36:10]: That's correct for us. You know, we train both dense and mixture of expert models. The one we happened to, you know, kind of get permission to open source was a mixture of expert model. And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP 03 style, where there's just a lot of parameters getting shuffled back and forth. And your ratio of kind of compute to amount of data that you have to shuffle back and forth becomes a lot worse because you're now, you know, you're only using a fraction of the parameters for every token instead of all the parameters. And so we had to really push the envelope on getting all the stuff to the right places on time. And so actually the networking part of DBRX was the single hardest thing, I think, of the entire process. Just get MOE training, working at scale across a big cluster. We still managed to, I think, do it all with commodity parts, which was very exciting. You know, we were using FSTP and we eventually used HSTP so that we could have HSTP as a version of FSTP where you have multiple smaller replicas and you're doing data parallel within those replicas. And that helped a lot with network latency issues that we were running into just because we were transmitting so much data, you know, for every single part of the process. I think it actually, like, it was instructive for how Google designs their hardware and software together personally. Their training, as far as I understand, using kind of a 03 style of training and have been for a while. They also train mixture of expert models. TPUs have a very different network bandwidth to compute ratio. They have a lot more bandwidth just objectively. And TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory. You know, it's just a different design choice. So the ratio of flops to bandwidth is very different. And that means that it's much easier for Google to be able to pull offSWYX [00:37:54]: some of this stuff.JONATHAN [00:37:54]: They also have interesting, you know, Torus style network architecture or Torus style, like, literal network architectureSWYX [00:38:00]: is not like the model,JONATHAN [00:38:00]: but the network.SWYX [00:38:02]: Is this the sort of block attention? I forgot what you call it. So this is just more or the,JONATHAN [00:38:07]: yeah, this is more, not the ring attention, but these are the ring all reduces. Like you have three different dimensions of rings because they kind of put you in these three dimensional Toruses from what I understand. And so like, you know, Google's infrastructure in some sense is kind of, I wouldn't say built for this, but maybe the way that Google trains models is built for a slightly different bit of infrastructure they have. And it's kind of neat to think about that. You know, as one thing that I think NVIDIA announced for, you know, for, for both the GH200 and the GB200 is this hybrid networking where you'll have blocks of NVLink network chips. I think for the GB200, I think it's like groups of 72 GPUs will all have NVLink to each other. So higher bandwidth, then you'll have normal networking of some kind, InfiniBand or Rocky or what have you between these blocks. And that's kind of a, you know, it's a change due to the fact that, you know, it's hard to build really high bandwidth networks over very large groups, but it is now a blocked networking. And you have to think about how you architect your model and your parallelism differently. You also have to think about fault tolerance differently because it now matters where you lose a GPU, whereas it didn't before. So, you know, it's, it's, it's just all really interesting and really fun speaking personally, but it's going to mean new nightmares when we all move to that generation and have to think about, you know, new versions of these problems.JOSH [00:39:20]: As you go up to larger scales, it gets quite different. Like right now, you know, if you're experiencing, let's say, for example, you experience a GPU failure every day, that's fine.SWYX [00:39:31]: Just restart.JOSH [00:39:31]: If you make your thing 24 times as big, now it's once an hour. Now it stops being quite as easy to just restart, right? So now you have to kind of break, like bake in this sort of redundancy that you didn't have before. So I think as you go up in scale, you end up running into like a lot of really interesting problems that also inform the, the actual like design. Yeah, I mean, as an orchestration guy,SWYX [00:39:52]: this is why I always emphasize like very cheap storage or very fast storage. So you can checkpoint more, but I don't think that's probably not the best solution to for fast, you know, training.JONATHAN [00:40:05]: Which works fine when you're doing language and then you move to vision or video. And then, you know, you have multi petabyte datasetsSWYX [00:40:12]: and getting, you know,JONATHAN [00:40:13]: cheap, fast multi petabyte storage starts to bite. Like I've certainly encountered issues where the literal data center where my GPUs were did not have enough, you know, object store to fit the datasets that people wanted to bring into that data center from whichever users were, were trying to bring them in. And then you get to a wholeSWYX [00:40:31]: different world of hurtJONATHAN [00:40:31]: where you have to keep your data in a different region because the region is just out of storage. So things get fun really fast.SWYX [00:40:39]: Speaking of vision, Josh, actually, you know, Embu is an agents company, but you're only, you're announcing a text-only model. What, where does, where does the vision side come in?JOSH [00:40:49]: I think we've actually done a lot of work in the past and people can see kind of our blog posts about sort of self-supervised learning and some other kind of vision-related stuff in the past as well. So we're very familiar with, with that stuff. But I think our main focus right now is on kind of, as we say, coding and reasoning. And there, there's certainly a visual component to some problems. But, you know, it's not necessarily required for all problems. And actually we found that for most of the kind of like code writing and, and reasoning problems that we care about, the visual part isn't really a huge important part of it. Sometimes if you really need to, you can maybe describeSWYX [00:41:24]: the thing.JOSH [00:41:24]: There are other like, you know, multimodal models that you can use off the shelf to sort of plug in for those particular piecesSWYX [00:41:30]: that you need, right?JOSH [00:41:30]: Like if something is driving a browser or whatever, like you can sometimes get away with not having to have that baked into the original model. So our folk were, you know, in a sense, we kind of do a lot across the stack. We're working on our own infrastructure and pre-training and RL and fine tuning and products and everything. But in another sense, we're very narrowly focused on the application side. So all of the stuff across the stack is kind of going toward a very particular purpose. And so that particular purpose right now doesn't really need vision. So we think that people are going to make all sorts of really cool image modelsSWYX [00:42:00]: like Jonathan, right?JOSH [00:42:00]: And all sorts of interesting multimodal models into the future. We'll let them go do that. That's great. We'll take advantage of that, partner with those people in the future. And right now we're really focused on kind of the core reasoning and coding capabilities and aspects of the model.SWYX [00:42:14]: I wanted to go into carbs since that's kind of the next layer of the stack. We talked about carbs in the first episode with Kanjin because you've actually had a blog post about it like a couple of years ago. Maybe let's introduce it.JONATHAN [00:42:26]: Has that been a couple of years now?JOSH [00:42:28]: No, it must have been at least one year. Hopefully it's not multiple years.SWYX [00:42:32]: Sorry, I'm counting AI time. Yeah, yeah. Yeah, I was going to sayJONATHAN [00:42:35]: you're making me feel really old right now.SWYX [00:42:39]: I count everything before the generally intelligent rename as like, you know, prehistory. Yeah. And now sort of modernity, right? So I actually thought carbs was more about hyperparameter optimization in a sense of like sort of parameters, hyperparameter search. Whereas, you know, when you introduced it, especially in this blog post, it's more about scaling laws and predictability of like, are we sort of in the right ballpark before we scale things up? Maybe sort of recount the history of carbs.JOSH [00:43:10]: Yeah, so it really is a little bit of both. So carbs is, it's maybe a backronym, but it's for cost aware Pareto region Bayesian search. So this is about technically how it works, but carbs is like, you know, we like pastries and stuff.SWYX [00:43:26]: So great, why not? But the point is thatJOSH [00:43:29]: it's a cost aware hyperparameter tuner. So most hyperparameter tuners, you kind of say, OK, here's this objective function. I want you to make this number as big as possible or as small as possible, whichever direction you want to go. So yeah, just go make this number, you know, as small as possible. OK, so it'll try a bunch of differentSWYX [00:43:46]: hyperparameters,JOSH [00:43:46]: a bunch of different configurationsSWYX [00:43:48]: to figure out, like,JOSH [00:43:48]: how do I tweak your network and architecture, et cetera, to get the kind of best performance I possibly can. That's usually saying, like, you know, almost all of these hyperparameter configurations are, let's say they're all going to use the same number of GPUs or the same number of nodes.SWYX [00:44:01]: So it's going to runJOSH [00:44:01]: for the same amount of time.SWYX [00:44:03]: So you can do that.JOSH [00:44:03]: You can get a number out and that's great. But what carbs does is it says,SWYX [00:44:07]: OK, actually,JOSH [00:44:07]: what if we relax that constraint? What if we say each of these different points, we're going to model how expensive it will be to sample this configuration. So if what if we train with just one one hundredth of the data? Like, how well can we do?SWYX [00:44:19]: What if we trainJOSH [00:44:19]: with one tenth of the data? What if we train with all the data? That way you can understand, like, as we get more and more data, as we spend more and more compute,SWYX [00:44:26]: as we make a biggerJOSH [00:44:26]: and bigger network, how does performance change with these things that change? Like how expensive it is to even explore this data point. So by doing that, we can see the scaling laws for not just, you know,SWYX [00:44:36]: the scaling lawsJOSH [00:44:36]: from like the, you know, Chantilla paper, the scaling laws for all parameters. We can see how does how does the number of layers change with this? How does the, you know, the learning rate change? How do the like, you know, various types of regularization change? So you can see these nice scaling laws. And as you're going across costs, like how should this be changing as you're scaling up your model? So that, coupled with the kind of metric that we chose, which is a very precise way of measuring performance, allowed us to really like hone in on parameters that worked really wellSWYX [00:45:05]: and understand, like,JOSH [00:45:05]: how do we want to scale those up, especially as we're changingSWYX [00:45:08]: things about the network?JOSH [00:45:08]: Like one of the things that we did is we used a custom tokenizer. As we change this tokenizer, changes a bunch of other things about the model. So how should we scale up this entirely new tokenizer? Like no one has ever made a model this large with this tokenizer before. And so how do we want toSWYX [00:45:22]: change all these things?JOSH [00:45:22]: Harps kind of shows you, like, look, as you change these parameters, like these other ones are kind of dependent on this.SWYX [00:45:28]: Like this is the, these areJOSH [00:45:28]: the relationships between them. So you can better understand, like, OK, if I'm going to scale this up 10x or 100x, like, where do I want to be? I can only go so far. And so, you know, we did run, like, I think maybe it was like a 14b one or somethingSWYX [00:45:40]: like that to check.JOSH [00:45:41]: But and so we had a bunch of like 1b or 14b and then at 70b. I don't think we had a, I think we just did like one at 14b. So you can, we get to check that like, oh, is this on the curve? Like, is this where we expect? It was like right there. So then great, go on to the next one. Yeah, I mean, that makes a lot of sense.SWYX [00:45:56]: I wonder if, so one of the key questions, and correct me if I'm wrong, but like usually people do search or do their evals just based on loss. But you actually evaluate based on, you know, the sort of end state evals that people might expect, like HellaSwag and Lombata, whatever. What is the norm here? Is there a norm?JOSH [00:46:20]: Yeah, I don't know if there's a hundred percent.SWYX [00:46:21]: I don't know. I only see loss on most people's reports.JOSH [00:46:25]: I think it's easy to, like, loss is very nice because it's very precise. It will tell you, like, very fine grained differences between like really small changes in your hyperparameters or network architecture. Whereas, especially at the smaller scales, if you're looking at like accuracy, it's very noisy. Like it might be zero or a hundred or like, you know, fluctuating by like 10 or 20 percentage points, which makes it really hard to tell, like, did that change actually mean anything? So our loss is sort of a combination of these two. Instead of saying, like, let's just look at perplexity, we say, let's look at perplexity on the tasks that we care about for multiple choice questions effectively.SWYX [00:47:00]: So we're saying like, yes,JOSH [00:47:00]: this is formulated as a multiple choice question, and we're going to look at the, like, you know, the loss of perplexity for this particular answer token. And that ends up being something that's like both targeted to what you actually care about and also very precise. The nice thing about this though is that it's independent of the data that you train on. One thing that's annoying about perplexity or about loss is that as you change your data set, this is really obnoxious because now it fundamentally changes your loss, right? And so you can't tell, like, how do I tweak my data set? But because we have this held out evaluation dat
Our modern lifestyles mean that most of us don't live our lives in sync with our circadian rhythms, which puts our health and well-being at risk. Eating and sleeping at the right time are important tools to help us align our circadian rhythms and reduce our risk of chronic disease. In this episode, circadian rhythm expert Prof. Satchin Panda will tell us how light and food act as master regulators of our body clock, how aligning our lifestyles with our body clock can improve our health, mood and energy levels and how to do this in practice. Satchin is a world-leading expert in the field of circadian rhythm research. He's associate professor at the prestigious SALK institute, he's recipient of the Dana Foundation Award in brain and immune system imaging and he's also the author of two best-selling books, The Circadian Code and The Circadian Diabetes Code.Follow ZOE on InstagramTimecodes:00:00 Introduction01:00 Quickfire questions03:02 What are circadian rhythms?03:48 How do we know about circadian rhythms?04:44 Are all body parts on a 24 hour clock?06:40 How the body enters sleep mode09:25 What happens during sleep?12:08 Why you're not sleeping enough13:30 The surprising impact of daylight savings time17:00 Circadian rhythms aren't just about light19:55 The dangers of shift work21:20 Should you go to bed at sunset?25:40 Why should stop snacking at night26:10 Satchin's famous mice study33:00 The best eating window for health37:27 Does intermittent fasting promote better food choices?40:40 Should you drink black coffee when you wake up? Satchin's books:The Circadian Code The Circadian Diabetes CodeBooks by our ZOE Scientists:Every Body Should Know This by Dr Federica AmatiFood For Life by Prof. Tim SpectorFibre Fuelled by Dr Will Bulsiewicz Studies referenced in today's episode: Effects of 3 months of 10-h per-day time-restricted eating and 3 months of follow-up on bodyweight and cardiometabolic health in Danish individuals at high risk of type 2 diabetes: the RESET single-centre, parallel, superiority, open-label, randomised controlled trial, published in Lancet Healthy LongevityNeuronal reprogramming of mouse and human fibroblasts using transcription factors involved in suprachiasmatic nucleus development, published iScienceLearning from circadian rhythm to transform cancer prevention, prognosis, and survivorship care, published in Trends CancerThe Untapped Potential of Circadian Timing as a Variable for Discoveries and Reproducibility, published in Cell Mol Gastroenterol HepatoHave feedback or a topic you'd like us to cover? Let us know here. Episode transcripts are...
You know how dentists have those intraoral cameras that let patients see what's going on in their mouth? What if SLPs could use those as part of their dysphagia assessment? Or as part of their biofeedback during therapy to check for residue and to see if certain compensatory strategies work? Just imagine quickly peeking into a patient's mouth with an angled intraoral camera and finding bread in the valleculae from the patient's last meal (which was two hours ago)! Turns out we can! James Curtis, PhD, CCC-SLP, and Ann Miles, PhD, are two SLPs and researchers who are exploring this idea and spill all the beans with us in this week's episode of The Swallow Your Pride Podcast! Join James and Anna as they discuss the intraoral camera and… The benefits of adding it to our dysphagia assessments Potential applications in therapy and patient education The pioneering work of Jose Vergara's team in Brazil Technical challenges Patient tolerance Current and future research Access to this instrument Tune in and give a shout-out to your local dentist for inspiring this idea! TIMESTAMPS: Initial exploration of intraoral cameras (00:05:42) Clinical application of intraoral cameras (00:10:44) Advantages and limitations of intraoral cameras (00:12:51) Procedure for using intraoral cameras (00:15:05) Challenges with intraoral cameras (00:19:01) Patient Positioning and Maneuvering (00:20:05) Challenges and Skills of Rigid Exam vs. Flexible Scope (00:20:56) Advantages of Intraoral Cameras (00:21:10) Importance of Post-Swallow Images and Video Clips (00:22:28) Sensitivity and Reliability of Intraoral Cameras (00:24:30) Comparative Research and Reproducibility (00:26:02) Limitations and Need for More Research (00:28:25) Cost and Infection Control Considerations (00:32:15) Integration into Clinical Protocols (00:33:42) Future Research and Implementation Studies (00:37:44) Intraoral Camera Use in New Zealand (00:40:09) Availability and Cost of Intraoral Cameras (00:40:56) Patient Populations for Intraoral Camera Use (00:41:38) Challenges and Benefits of Rigid Endoscopy (00:44:23) The post 333 – Can Intraoral Cameras Improve Dysphagia Management? Let's Zoom In… appeared first on Swallow Your Pride Podcast.
The fields of Cell and gene therapy are booming and poised to change the treatment and prevention of disease. These research areas require the transfer of genetic material to cells, and viral vectors are commonly used here. Specifically, adeno-associated virus (AAV) and lentiviral vectors (LVV) are vectors of choice. We're joined for this episode by MinGin Kim and Kimberly Gomez, both scientists at Thermo Fisher. With backgrounds and expertise in the areas of cell and gene therapy, they help explain what all the excitement is about and how AAV and LVV are used. We hear about some of the challenges associated with viral vector work and get to hear about how digital PCR (dPCR) and good assay design are helping overcome many of these challenges to enable research and the biopharmaceutical industry. As you might expect from Absolute Gene-ius, you also get to hear their respective career path journeys and some really interesting lab stories.Visit the Absolute Gene-ius page to learn more about the guests, the hosts, and the Applied Biosystems QuantStudio Absolute Q Digital PCR System.
Access 2 Perspectives – Conversations. All about Open Science Communication
Maha Said is a post-doctoral researcher on the ERC-funded NanoBubbles project which asks the questions how, when, and why does science fail to correct itself. Originally trained in molecular and cellular biology and working close to science and technology studies especially science integrity, she is currently working in 2 sub-projects, the first being post publication peer review in which articles on the topic of interest are critically analyzed and publicly commented on Pubpeer, and second is the reproducibility project in which research articles that describe the use of nanoparticles for intracellular sensing are examined. On this podcast episode, Jo and Maha engage in a conversation focused on research integrity within the field of nanoparticles. Maha shares her journey into this area, driven by experiences of encountering integrity issues during her PhD research, particularly with antibodies. They discuss the challenges of peer review, replicability, and reproducibility, highlighting Maha's current work on post-publication peer review and replicability initiatives within nanoparticle research, specifically intracellular sensing. They also touch on the complexity of defining and implementing registered reports as a tool for ensuring transparency and accountability in research. Throughout their discussion, they emphasize the importance of critical analysis, collaboration across disciplines, and adaptation to changes in scientific processes. Find more podcast episodes here: https://access2perspectives.pubpub.org/podcast Host: Dr Jo Havemann, ORCID iD 0000-0002-6157-1494 Editing: Ebuka Ezeike Music: Alex Lustig, produced by Kitty Kat License: Attribution 4.0 International (CC BY 4.0) At Access 2 Perspectives, we guide you in your complete research workflow toward state-of-the-art research practices and in full compliance with funding and publishing requirements. Leverage your research projects to higher efficiency and increased collaboration opportunities while fostering your explorative spirit and joy. Website: https://access2perspectives.pubpub.org --- Send in a voice message: https://podcasters.spotify.com/pod/show/access2perspectives/message
This episode originally aired in 2023 and contains advice that has become increasingly relevant. 00:56 - IntroductionBrent Wadas is an Army veteran who has also worked in finance and SaaS. In 2020, he and his two co-founders dove into construction automation with BotBuilt. He joins us today to talk about automated framing. 04:38 - BotBuilt OverviewBrent explains why he sometimes feels like a five-year-old. He and his co-founders found that most automated systems required almost complete uniformity. He explains The Panel Book that contains detailed breakdowns of components, designs, and costs. He talks us through how they use industrial robot arms from eBay to building out wall-panel systems. (Watch a video of the process.) The marked, labeled panels then simply need to be properly placed and nailed together. The contractors working with BotBuilt can complete the framing for a single-family residence in 2.5-5 hours using the panels assembled in their facility. Brent compares BotBuilt's process to that of other automated-construction companies.08:48 - Ikea-style AssemblyEddie asks how BotBuilt lays out instructions for contractors to assemble their plans. Brent explains that the incredibly detailed plans they did for their first run-through ended up in the mud because the builder just wanted the simplest one-page overview plan. Tyler asks for some more details about the computer programming. Brent talks about the math involved and the challenge of regional code variations. Brent emphasizes that they can work up a schedule with just a PDF version of the plans. He talks about high school intern Joseph, whose fast work pace caught Brent off-guard. 18:26 - Growth, the Future, and RecruitingBrent talks about how far things have come in the last couple years and how quickly things are going to change over the next five years. He talks about his company's work with Y Combinator. Eddie asks about the challenges of funding such an ambitious business. Brent talks through the way he and his co-founders process those decisions. This conversation includes some insights gained from buying and renovating fixer-uppers. We find ourselves talking yet again about the challenges of getting the younger generations interested in pursuing construction jobs. Brent ties in some thoughts from his time in the military. 35:10 - What to Consider Before AutomatingTyler asks Brent to outline the things that owners, contractors–anyone–should consider before automating. Here are Brent's thoughts: -Reproducibility is the first thing to consider. If you're looking at a process that repeats the same specs time after time, you might want to consider automation. -Ask yourself, “Is there a problem here?” Don't automate just because you see other people automating. -Consider the personnel demands. Don't set yourself up to pay the same people for the same number of hours just to solve a problem in a more complicated, new way.45:00 - Safety and LegacyWe talk about work site safety, and Brent says that you're 10x more likely to die framing a house than on the battlefield in Afghanistan. He said that 35 service members died every year at the height of the war. 350 people die every year framing houses according to OSHA. He shares how BotBuilt's consistency, standards, and simplicity can make a dent in these numbers. Tyler and Eddie discuss their experiences with safety practices (or the lack thereof) on their early-career worksites. Rarely if ever was there anyone designated to keep an eye on safety. We discuss how messed up that is. This leads us into a discussion about leadership.1:00:33 - Megaphone MessageBrent has a couple megaphone messages. He wants construction workers to keep hope alive, and he wants people from the tech industry as a whole to please come learn the culture of construction. He wants them to discover the wealth and satisfaction that construction has to offer. Find Brent Online: LinkedIn - BotBuilt Check out the partners that make our show possible.Find Us Online: BrosPodcast.com - LinkedIn - Youtube - Instagram - Facebook - TikTok - Eddie's LinkedIn - Tyler's LinkedInIf you enjoy the podcast, please rate us on Apple Podcasts, Spotify, or wherever you listen to us! Thanks for listening
Episode 84. Maryann Martone is an emeritus professor of neuroscience at UC San Diego. She received her BA from Wellesley College in Biological Psychology and Ancient Greek and her PhD in Neuroscience from the UC San Diego
How do you learn about what's going on in the world? Did a news headline grab your attention? Did a news story report on recent research? What do you need to know to be a critical consumer of the news you read? If you are looking to start developing your data self-defense and critical news consumption skills, this book is for you! It reflects a long-term collaboration between a statistician and a journalist to shed light on the statistics behind the stories and the stories behind the statistics. The only prerequisite for enjoying this book is an interest in developing the skills and insights for better understanding news stories that incorporate quantitative information.
Nick Huntington-Klein is an Assistant Professor, Department of Economics, Albers School of Business and Economics, Seattle University. His research focus is econometrics, causal inference, and higher education policy. He's also the author of an introductory causal inference textbook called The Effect and the creator of a number of Stata packages for implementing causal effect estimation procedures. Nick's book, online version: https://theeffectbook.net/ The Paper of How: https://onlinelibrary.wiley.com/share/W2FMEESMMSJMWDEZYY8Y?target=10.1111/obes.12598 Nick's twitter & BlueSky: @nickchk Nick's website: https://nickchk.com Follow along on Twitter: The American Journal of Epidemiology: @AmJEpi Ellie: @EpiEllie Lucy: @LucyStats
In episode 115 of The Gradient Podcast, Daniel Bashir speaks to Ben Wellington.Ben is the Deputy Head of Feature Forecasting at Two Sigma, a financial sciences company. Ben has been at Two Sigma for more than 15 years, and currently leads efforts focused on natural language processing and feature forecasting. He is also the author of data science blog I Quant NY, which has influenced local government policy, including changes in NYC street infrastructure and the design of NYC subway vending machines. Ben is a Visiting Assistant Professor in the Urban and Community Planning program at the Pratt Institute in Brooklyn where he teaches statistics using urban open data. He holds a Ph.D. in Computer Science from New York University.Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at editor@thegradient.pubSubscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (01:30) Ben's background* (04:30) Why Ben was interested in NLP* (05:48) Ben's work on translational equivalence, dominant techniques* (10:14) Scaling, large datasets at Two Sigma* (12:50) Applying ML techniques to quantitative finance, features in financial ML systems* (17:27) Baselines and time-dependence in constructing features, human knowledge* (19:23) Black box models in finance* (24:00) Two Sigma's presence in the AI research community* (26:55) Short- and long-term research initiatives at Two Sigma* (30:42) How ML fits into Two Sigma's investment strategy* (34:05) Alpha and competition in investing* (36:13) Temporality in data* (40:38) Challenges for finance/AI and beating the market* (44:36) Reproducibility* (49:47) I Quant NY and storytelling with data* (56:43) Descriptive statistics and stories* (1:01:05) Benefits of simple methods* (1:07:11) OutroLinks:* Ben's work on translational equivalence and scalable discriminative learning* Two Sigma Insights* Storytelling with data and I Quant NY Get full access to The Gradient at thegradientpub.substack.com/subscribe
What to listen for:Our hosts Robin Greubel and Stacy Barnett dive headfirst into the high-stakes world of canine explosive detection with an esteemed panel of experts, Dr. Lauryn DeGreeff, Dr. Michele Maughan, and Jenna Gadberry. As they dissect the complexities of non-detonable canine training aids, you'll get a rare behind-the-scenes look at the intricate dance of odor chemistry and the safety measures paramount in training with both traditional and peroxide-based explosives. Safety for furry detectives and their handlers leads the charge in the discussion, as they navigate the laboratory and field labyrinth to analyze the effectiveness of different training aids.Ever pondered how a detection dog's nose works like a sophisticated bio-sensor, dissecting the world's odors? Our hosts and guests tackle the unfortunate issue of variability in training aids. With insights from Dr. Nathan Hall, they unravel the scent detection conundrum, especially when dealing with volatile compounds. Drawing from real-world applications and scientific scrutiny, this episode uncovers the essential need for multifaceted exposure and stringent training to enhance the fidelity of our four-legged bomb detectors.Key Topics:Explosives Dogs and Training Aids (0:02:04)Working with Green (naive to odor) Dogs (0:07:40)Dog Training Aids, Inconsistency, and Quality Control (0:12:38)Explosive TTP's (0:16:59)Training Aids for Explosive Detection Dogs (0:19:03)Training Aids and Quality Control (0:26:51)Canine Detection Capabilities and Training Aids (0:33:40)Risk Assessment around Canine Explosives Work (0:38:44)Explosives and Aids Aging (0:49:32)Canine Scent Detection and Research (0:58:22)Resources:Canines As The Original BioSensor & Odor/Scent Chemistry with Dr. Lauryn DeGreeffEvaluation of non-detonable canine training aids for explosives by headspace analysis and canine testingOSAC: Organization of Scientific Area Committees For Forensic ScienceNew Scientific Study About Bomb Dog Training (Podcast Episode)We want to hear from you:Check out the K9 Detection Collaborative FB page and comment on the episode post!K9Sensus Foundation can be found on Facebook and Instagram. We have a Trainer's Group on Facebook!Scentsabilities Nosework is also on Facebook. Here is a Facebook group you should join!Crystal Wing K9 Coach can be found here at CB K9 and here at Evolution Working Dog Club. Also, check out her Functional Obedience Class here.You can follow us for notifications of upcoming episodes, find us at k9detectioncollaborative.com
Deploying Nextcloud the Nix way promises a paradise of reproducibility and simplicity. But is it just a painful trek through configuration hell? We built the dream Nextcloud using Nix and faced reality. Special Guest: Alex Kretzschmar.
ℹ️ IntroductionIn this episode of the Frugalpreneur podcast, host Sarah St. John explores the world of franchising with guest Gregory Mohr, author of the best-selling book, "Real Freedom." Gregory shares his journey into franchising, from his humble beginnings at Taco Bell to becoming a franchise consultant. He delves into the different types of franchises, the financing options available, and the due diligence procedures to consider when selecting a franchise. Listeners are in for a treat as they uncover the wealth of information Gregory provides about the franchising industry and how individuals can make the most informed decisions when considering this path for their entrepreneurial ventures.
KSQD 1-10-2024: All about the new Family Practice Residency program in Santa Cruz and the lack of primary care doctors locally and nationally; The rise in research in medical applications of psychedelic drugs, such as MDMA for PTSD; The lack of reproducibility in medical research; Conflicting meta analysis reports of Vitamin D research; Modern marijuana is more addictive because of higher THC levels
Data in Biotech is a fortnightly podcast exploring how companies leverage data innovation in life sciences. This week, we're delighted to be joined by Harry Rickerby, Co-founder of Briefly Bio, a groundbreaking platform he's developing to streamline experimental design in life sciences using large language models (LLMs). Host Ross Katz speaks with Harry about how LLMs are used to facilitate experiment protocol structuring in the platform, the limitations of LLMs in scientific fields, how we can improve collaboration, consistency, and reproducibility in biotech, and how metadata is generated within the experimental design process while interacting with an LLM. --- If you're a biotech company struggling to unlock a data challenge, CorrDyn can help. Whether you need to supplement existing technology teams with specialist expertise or launch a data program that lays the groundwork for future internal hires, you can partner with Corrdyn to unlock the potential of your business data - today. Visit connect.corrdyn.com/biotech to learn more.
Episode 4 of the online course How To Transform the Sciences: Six Potential Breakthroughshttps://www.sheldrake.org/online-coursesAround 2015, scientists were shocked to find that most papers in high-prestige peer-reviewed scientific journals are not reproducible. In one study of papers in prestigious biomedical journals, 90% could not be replicated, and in experimental psychology more than 60%. This crisis partly arises from systematic biases that Rupert discusses in his chapter on ‘Illusions of Objectivity' in The Science Delusion (2012, new edition 2020; in the US this book is called Science Set Free), including the selective observation and reporting of results, and perverse incentives for scientists and journals to publish striking positive findings. The crisis continues to roll on, as shown, for example, by an editorial in Nature, December 2021, about un-reproducible results in cancer biology.All this is relatively straightforward, but Rupert suggests that some experiments may also involve direct mind-over-matter effects. It has long been known that experimenters can influence their experimental results through their expectations, in so-called ‘experimenter expectancy effects', which is why many clinical trials, psychological and parapsychological experiments are carried out under blind or double-blind conditions.In most other fields of science, experimenter effects are ignored and blind methodologies are rarely employed. Rupert suggests that in addition to the usual sources of bias, experimenters may also influence experiments psychokinetically, through direct mind-over-matter effects. Scientists may be particularly prone to this source of error because most scientists believe psychokinesis is impossible, and hence take no precautions against it. They practise unprotected science. Rupert proposes experiments on experiments to test for the effects of experimenters' hopes and expectations.ReferencesReferences____A Dream, or the Astronomy of the MoonJohann Kepler, published posthumously in 1634 by his sonhttps://sheldrake.org/somnium____Rupert's essay The Replicability Crisis in Sciencehttps://sheldrake.org/replicability____Bad PharmaBen GoldacreFourth Estate, 2012https://sheldrake.org/badpharma____Artifacts in Behavioral ResearchRobert Rosenthal and Ralph L. Rosnow, Oxford University Press, 2009https://sheldrake.org/rosenthal____Over half of psychology studies fail reproducibility testhttps://www.nature.com/articles/nature.2015.18248____Differential indoctrination of examiners and Rorschach responseshttps://psycnet.apa.org/record/1965-12396-001____A longitudinal study of the effects of experimenter bias on the operant learning of laboratory ratshttps://psycnet.apa.org/record/1965-01547-001____Could Experimenter Effects Occur in the Physical and Biological Sciences?Skeptical Inquirer 22(3), 57-58 May / June 1998https://sheldrake.org/skepticalinquirer98____Quantum‐Mechanical Random‐Number Generator https://aip.scitation.org/doi/abs/10.1063/1.1658698------Dr Rupert Sheldrake, PhD, is a biologist and author best known for his hypothesis of morphic resonance. At Cambridge University, as a Fellow of Clare College, he was Director of Studies in biochemistry and cell biology. As the Rosenheim Research Fellow of the Royal Society, he carried out research on the development of plants and the ageing of cells, and together with Philip Rubery discovered the mechanism of polar auxin transport. In India, he was Principal Plant Physiologist at the International Crops Research Institute for the Semi-Arid Tropics, where he helped develop new cropping systems now widely used by farmers. He is the author of more than 100 papers in peer-reviewed journals and his research contributions have been widely recognized by the
00:56 - IntroductionBrent Wadas is an Army veteran who has also worked in finance and SaaS. In 2020, he and his two co-founders dove into construction automation with BotBuilt. He joins us today to talk about automated framing. 04:38 - BotBuilt OverviewBrent explains why he sometimes feels like a five-year-old. He and his co-founders found that most automated systems required almost complete uniformity. He explains The Panel Book that contains detailed breakdowns of components, designs, and costs. He talks us through how they use industrial robot arms from eBay to building out wall-panel systems. (Watch a video of the process.) The marked, labeled panels then simply need to be properly placed and nailed together. The contractors working with BotBuilt can complete the framing for a single-family residence house in 2.5-5 hours using the panels assembled in their facility. Brenth compares BotBuilt's process to that of other automated-construction companies.08:48 - Ikea-style AssemblyEddie asks how BotBuilt lays out instructions for contractors to assemble their plans. Brent explains that the incredibly detailed plans they did for their first run-through ended up in the mud because the builder just wanted the simplest one-page overview plan. Tyler asks for some more details about the computer programming. Brent talks about the math involved and the challenge of regional code variations. Brent emphasizes that they can work up a schedule with just a PDF version of the plans. He talks about high school intern Joseph, whose fast work pace caught Brent off-guard. 18:26 - Growth, the Future, and RecruitingBrent talks about how far things have come in the last couple years and how quickly things are going to change over the next five years. He talks about his company's work with Y Combinator. Eddie asks about the challenges of funding such an ambitious business. Brent talks through the way he and his co-founders process those decisions. This conversation includes some insights gained from buying and renovating fixer-uppers. We find ourselves talking yet again about the challenges of getting the younger generations interested in pursuing construction jobs. Brent ties in some thoughts from his time in the military. 35:10 - What to Consider Before AutomatingTyler asks Brent to outline the things that owners, contractors–anyone–should consider before automating. Here are Brent's thoughts: -Reproducibility is the first thing to consider. If you're looking at a process that repeats the same specs time after time, you might want to consider automation. -Ask yourself, “Is there a problem here?” Don't automate just because you see other people automating. -Consider the personnel demands. Don't set yourself up to pay the same people for the same number of hours just to solve a problem in a more complicated, new way.45:00 - Safety and LegacyWe talk about work site safety, and Brent says that you're 10x more likely to die framing a house than on the battlefield in Afghanistan. He said that 35 service members died every year at the height of the war. 350 people die every year framing houses according to OSHA. He shares how BotBuilt's consistency, standards, and simplicity can make a dent in these numbers. Tyler and Eddie discuss their experiences with safety practices (or the lack thereof) on their early-career worksites. Rarely if ever was there anyone designated to keep an eye on safety. We discuss how messed up that is. This leads us into a discussion about leadership.1:00:33 - Megaphone MessageBrent has a couple megaphone messages. He wants construction workers to keep hope alive, and he wants people from the tech industry as a whole to please come learn the culture of construction. He wants them to discover the wealth and satisfaction that construction has to offer. Find Brent Online: LinkedIn - BotBuilt Check out the partners that make our show possible.Find Us Online: BrosPodcast.com - LinkedIn - Youtube - Instagram - Facebook - TikTok - Eddie's LinkedIn - Tyler's LinkedInIf you enjoy the podcast, please rate us on Apple Podcasts, Spotify, or wherever you listen to us! Thanks for listening
Bob has officially passed the Associate Director torch to JimBo. On this weeks episode of the BootCamp we are discussing some things to consider when developing a discipleship pathway. Things to consider The Goal of Discipleship: Spiritual Transformation In the context of community Confidentiality cultivates a community where we can be honest Through accountability Reproducibility […]
Evan Floden, CEO and Co-founder of Seqera Labs, discusses Nextflow, the push for reproducibility in scientific workflows, and his experience as a scientist with a start-up. The Bioinformatics CRO is a fully distributed contract research company that serves the computational biology needs of biotechnology companies, with a focus on genomics. https://www.bioinformaticscro.com/
The reliability of testing epigenetic DNA methylation using Illumina beadchips is of paramount importance due to the specific intricacies of this technology. Illumina beadchips are widely used platforms for high-throughput epigenetic analysis, employing thousands of probes to measure DNA methylation levels at specific genomic loci. In this week's Everything Epigenetics podcast, Dr. Karen Sugden and I talk about how the reliability of these probes directly impacts the accuracy and validity of the results obtained.Keep in mind that in the context of Illumina beadchips, reliability refers to the consistent and accurate performance of each individual probe across multiple samples and experimental replicates. Each probe is designed to target a specific CpG site, and the methylation signal it generates must be dependable and reproducible.We discuss how reliable probes ensure the accuracy of DNA methylation measurements and how the reliability of probes becomes crucial for reproducibility when conducting large-scale studies using Illumina beadchips, such as epigenome-wide association studies (EWAS).Dr. Sugden and I also discuss how the reliability of probes on Illumina beadchips has implications for cross-study comparisons. For example, if the probes exhibit inconsistent behavior across different experiments or cohorts, it becomes challenging to compare results and draw meaningful insights from combined analyses.Furthermore, we chat about the efficient utilization of resources being linked to probe reliability. Unreliable probes might necessitate repeating experiments or allocating additional resources to validate results, potentially delaying research progress and increasing costs.In the context of epigenetic research, where subtle changes in DNA methylation can hold profound biological significance, the accuracy and consistency of data generated by Illumina beadchips are pivotal. Lastly, we explore Dr. Sugden's current research which includes how epigenetic clocks are associated with cognitive impairment and dementia and marijuana use. In this episode of Everything Epigenetics, you'll learn about: Dr. Karen Sugden's career Reliability and why it mattersHow unreliability arises in epigenetic researchThe process of measuring DNA methylation on Illumina beadchips (or microarrays) Technical errors that could arise when looking at DNA methylationKaren's paper titled “Patterns of Reliability: Assessing the Reproducibility and Integrity of DNA Methylation Measurement”How to untangle data from different beadchips (27K vs. 450K vs. EPIC 850K)What constitutes a reliable probe vs. an unreliable probe How to handle unreliable probesWho is at fault for unreliable probes If reliability is the same for every beadchipHow unreliability impacts epigenetic research How we can deal with unreliabilityThe value of repeated data Creating a “gold standard” work flow for processing epigenetic data How epigenetic clocks associate with cognitive impairment and dementia The connection between epigenetic clocks and marijuanaDr. Sugden's current research investigations Karen Sugden's profile at Duke - https://moffittcaspi.trinity.duke.edu/karen-sugden-0Support the showThank you for joining us at the Everything Epigenetics Podcast and remember you have control over your Epigenetics, so tune in next time to learn more about how.
Nutrition and obesity-related research are scientific topics which should be executed with the same degree of rigor, transparency, and truthful communication as in any other area of science. However, this type of research may be weaker than it should be due to flaws in the types of questions asked, the design of studies, the execution of studies, the analysis of resulting data, the interpretation and communication of studies and results. This weakens the overall quality of the literature and may lead to heightened distrust of nutrition science, which has been shown to be more severe than for other domains of inquiry. Tune into this episode to learn about: · an overview of various aspects of research including selection of questions, design of studies, execution of studies, analysis of data, and interpretation and communication of findings · the quality of existing obesity related research and challenges regarding this type of research in general · examples of where research has gone wrong and suggestions for improvement · what the evidence for obesity treatment and prevention shows and suggestions for prioritizing next steps, future research and treatments · why evidence in the field of nutrition and obesity-related research seems to be more often distorted and distrusted · specific steps to make obesity research more rigorous, probative, valuable, and more transparently and truthfully communicated Full shownotes and resources at: https://soundbitesrd.com/240
We attempt to swap Linux distributions live on our production server, to prove that new tooling makes the Linux distro model obsolete.
© 2020-2025 Ivy Podcast Discovery LLC
