Podcasts about moes

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at teamworkstactical.comWe are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at trainheroic.comWhy Physical Therapists Believe Weird Things — Commander Mark RiebelNuclear submarine officer turned PT for Marine Raiders. This week Drew and Alex sit down with Commander Mark Riebel to talk therapeutic skepticism, why smart people believe dubious things, and what the research actually says about the modalities that dominate clinical practice.What we get into:Confirmation bias in the clinic — why providers remember the wins and discount the losses, and how that quietly keeps bad interventions alive longer than they deserve.The fiduciary vs. the crypto salesman — two models of patient care, and why putting the patient in charge of their own pain is both better medicine and better therapy.Dry needling, cupping, scraping, foam rolling, therapeutic ultrasound, KT tape — what the evidence actually shows, what's placebo, and why that distinction matters more than most providers want to admit.Citation for the discussion of treatment effects vs placebo and other factors: Ezzatvar, Yasmin, et al. "Which portion of physiotherapy treatments' effect is not attributable to the specific effects in people with musculoskeletal pain? A meta-analysis of randomized placebo-controlled trials." journal of orthopaedic & sports physical therapy 54.6 (2024): 391-399.Trigger points, PRI, FMS, pose method — a tour through the tribes of physical therapy and how to think critically about any system that markets itself as the answer.The Future Sailor Preparatory Course — what it looks like, why it matters, and an honest conversation about the physical readiness of the recruiting pool.Weighted pull-ups post bicep repair, rear foot elevated split squats, and John's admirable hamstring appreciation — the after party delivers.Mentioned in this episode:Mark specifically recommended this ESPN video for a discussion of how nocebic language affects healthcare outcomesTherapeutic Skepticism — APTA talk by Mark Riebel and colleaguesCunningham's Law — the best way to get an answer on the internet is not to ask the question, it's to post a wrong answerBarbell Medicine — referenced on pesticide/produce misinformation researchFuture Sailor Preparatory Course — modeled off the Army's Future Soldier Preparatory CourseArmy Baylor — where Mark completed his DPTWest Point Sports Medicine Fellowship — where Mark learned to critically analyze research rather than chase magic tricksCharles Vogel, The Art of Community — former podcast guest, on how social spaces are engineered against genuine connectionLong and Strong — the Mops and Moes training program on TrainHeroic → https://marketplace.trainheroic.com/workout-plan/team/leg-tuck-nation?attrib=565490-web Views expressed are those of the speakers and do not represent any official organization.

We're announcing AIEWF speakers this week! Take the AI Engineering Survey!Today's guest Ethan first joined us for the LS Paper Club as the lead on NVIDIA Cosmos World Model, but then joined xAI and built Grok Imagine in 3 months:He comes back on Latent Space with some nuclear hot takes: that Video Models primarily get their intelligence from LLMs, not from training on video data, and that the next frontier for truly interactive, realtime, long-horizon world models is to work on LLMs (perhaps Interaction Models as well…)Put it this way: In the near term, the next Sora won't be a better video model, but a video agent.Generative Media may more closely follow the evolution of AI coding which went from focusing on one-shot output performance and cost, to multiturn reasoning and planning models for agents and systems that can plan, edit, test, debug, and submit PRs.At a certain point, coding models got so good that the only significant next step to improve performance was handling the orchestration of these models.Now as the performance of video models increases significantly across realism, consistency, & prompt adherence while becoming more cost efficient, the next evolution of video generation may also be systems that can plan, generate, edit, critique, and iterate across an entire creative task. In this episode, Ethan joins swyx and Vibhu to unpack what it actually takes to build frontier image and video systems: data, VAEs, diffusion transformers, audio-video alignment, inference speedups, and the hidden cost of storing and moving massive video datasets. From building NVIDIA's Cosmos world model to joining xAI as Grok Imagine was being built from zero to one, Ethan He has been at the center of some of the most important work in video generation, multimodal models, and real-time world models.We go deep on Grok Imagine, how a small xAI team shipped its first multimodal video model in three months, why iteration speed matters more than almost anything in model development, and why many of the biggest gains come from fixing tiny bugs in data and training pipelines. Flipbook: The future of VideomaxxingVideo agents are almost a sure bet to be the trend in the coming year. We end with a glance at what's beyond video agents:Flipbook caused a minor sensation this year when it was released, but most treat it as a fun demo. Ethan takes it very seriously — with the speed and cost of inference coming down every year, the future of custom video JIT UI is closer than you think. We talked about why videogen models may become the front end of AI, how generative UI could replace traditional HTML/CSS, why world models need to be real-time, interactive, and long-horizon, and why the future of video generation may depend more on language models and agents than on diffusion alone.We discuss:* Why fast iteration mattered more than meetings* Why small training bugs can drive huge model quality gains* Why coding models may make compute the bottleneck again* How image and video models are trained with synthetic captions* The role of VAEs and latent space in frontier video models* Why image models are the foundation for video models* The tradeoff between temporal compression and real-time interactivity* Flipbook, Neural OS, and the future of generative UI* Why future interfaces may go from user intent to pixels* The hidden cost of training video models: storage, egress, and GPU hours* How step distillation and consistency models (like OpenAI sCM) makes video inference orders of magnitude faster* Grok Imagine 0.9 and large-scale audio-video generation* Why audio-video alignment is harder than text-video alignment* Ethan's definition of world models* Reference-to-video, video extension, and long-context video generation* Why xAI's research communication undersells Grok Imagine* How xAI culture shaped the speed of development* AI watermarking, SynthID, and detecting generated media* Why prompt rewriting matters for video models* Grok Imagine Agent and the rise of video agents* Why language models may unlock better video generation* Robotics, physical AI, and embodied world models* Why Ethan left xAI and shifted focus toward LLMs* Self-managed context, memory, and the next frontier for language modelsEthan He* LinkedIn: https://www.linkedin.com/in/ethanhe42* X: https://x.com/EthanHe_42Timestamps00:00:00 Introduction00:01:25 From NVIDIA Cosmos to xAI00:03:24 Building Grok Imagine from Zero to One00:10:07 How Image and Video Models Are Trained00:18:53 Video Compression, VAEs, and Real-Time Tradeoffs00:22:10 Generative UI, Flipbook, and Neural OS00:32:10 The Cost of Training Large Video Models00:37:04 Distillation, GANs, and Fast Video Inference00:41:21 Audio-Video Generation and Grok Imagine 0.900:48:34 What Makes a World Model?00:55:51 Reference Videos, Long Context, and Video Memory01:00:11 xAI Culture, Research, and First-Principles Building01:09:45 AI Safety, Watermarking, and Prompt Rewriting01:13:10 Video Agents and AI-Assisted Creation01:27:32 Why Language Models Unlock Better Video01:31:15 Robotics, Physical AI, and Embodied World Models01:32:38 Why Ethan Left xAI01:34:16 Self-Managed Context and the Future of LLMs01:38:43 Ethan's Career Path and Closing ThoughtsTranscriptIntroduction: Ethan He, Latent Space, and the Path to xAISwyx [00:00:00]: We're here in the studio with Ethan He, most recently of xAI. Welcome.Ethan [00:00:10]: Thank you. Glad being here.Swyx [00:00:11]: We're also here with Vibhu. you were first coming to us or joining the latent space world because you were working on Kosmos at NVIDIA, and you did a paper. We loved it. you presented it as well, so thank you for doing that.Ethan [00:00:23]: I've actually, I also presented the MoEs twice at latent space.Swyx [00:00:29]: How did you actually hear about us? Did we reach out to you? Is that how it worked?Ethan [00:00:33]: No, actually, I-- the community. Like I realized, oh, there is this online community that people talk about AI and also learn from each other through papers every week through the Paperclip. It's very nice.Ethan [00:00:49]: I learned a lot.Swyx [00:00:49]: I think three years stop. We haven't stopped even on Christmas and New Years. many weeks I want to stop but it keeps going.Vibhu [00:00:58]: No, that was good. I think you had posted that you worked on a paper, and I was “Oh, very cool. We have Paperclip. Present then.”Vibhu [00:01:04]: But I might have reached out to you after.Swyx [00:01:05]: you-- because it's an amateur club, right?Swyx [00:01:08]: so it's very unusual and but we have sometimes paper authors come by and actually explain the paper. Today we just did, the poolside paper, which was apparently very good.Vibhu [00:01:18]: Came out yesterday.Vibhu [00:01:19]: pretty interesting, right? Fully open. They talk about everything, systems. So it's a good one. We'll, we'll recommend people to read it.Swyx [00:01:25]: Bring us up to speed on your transition to xAI, ‘cause I actually don't even know when you joined. just like tell the, tell the story about the sort of transition.From NVIDIA Cosmos to xAI: Scaling Video and World ModelsEthan [00:01:34]: Before xAI, I was working on Kosmos world model as in-- at NVIDIA. So Kosmos is, it's a giant video foundation models that can-- that aims to simulate the world and for-- it serves as a foundation of-- for all of the roboticists to build on top of. There, once I built the Kosmos one, I realized as this thing also has a scaling law similar to language model, we need to scale up the video models further. that's, that's why I realized I need to move to somewhere with much more compute resources. That's how ISwyx [00:02:13]: Than NVIDIA?Vibhu [00:02:14]: The GPU rich came themselves.Vibhu [00:02:19]: And timeline-wise, when was Kosmo? It was pretty early, right? It was open world model, open paper, everything.Ethan [00:02:25]: It was end of twenty-four.Vibhu [00:02:28]: End of twenty-four.Ethan [00:02:30]: Then at mid twenty-five, I moved to xAI. At that time-- I joined about the time when xAI was about to build video models and in multi-model models. There were no infra, no data, and no model, and it just-- as a few engineers, we built it in three months and released the first model, Grok Imagine zero point nine.Ethan [00:02:55]: And since then, I keep working on video models and move more from training and to post-training of the video models. For example, like a reference to videos, kind of like the cameo feature and, video extensions. And, before I left, I worked on a world model, leading a small team to focus on the real-time long horizon video generation.Building Grok Imagine From Scratch in Three MonthsSwyx [00:03:24]: Can you give like a rough roadmap of okay, you're on a brand-new team. Grok previously was only text, or they partnered with BFL for their image gen stuff. What do you-- what are the building blocks, right? You have compute, data you can procure somewhere. Like just what are like the sequence of things that people should think about when you're setting up a new team?Vibhu [00:03:43]: actually even deeper, not just data you can procure. You guys had to go through getting the data too, right? So you shipped it pretty fast, but yeahSwyx [00:03:51]: three months is likeVibhu [00:03:52]: From everythingSwyx [00:03:52]: actually like very surprisingly fast.Ethan [00:03:55]: One thing I say like thanks to my experience at NVIDIA, ‘cause first time when we were building Kosmos together, we built it, for about a year. So this is like the second time I do it. Roughly have an idea, what to do. I say the most important thing is the talent. Everyone were very strong and clever, very close with each other towards a common goal. So that speed up things a lot. So you reduce the communication bandwidth among people, and everyone can work towards the same goal. It's, it's like every day there's not that much meetings on the calendar, like maybe like a, like a sync a day, and after that it's, it's just all building. It was pretty fun at that time.Ethan [00:04:47]: And another thing is that xAI has very strong foundations of like data inference, model inference, and the supporting there can help the model develop a lot. When I look at, training models, I don't so actually the top important thing is like how many, how many iterations can you do, per day? and the more iteration can you do, you can, you can train the model much faster. So if you have very strong infra and you have a lot of compute, you can, you can train these models in very short period of time. That can give you a much larger buffer to, for errors, and it also gives you the opportunity to spot more bugs.Iteration Speed, Compute, and Debugging Model PipelinesSwyx [00:05:46]: What is an iteration? Is it like a few hundred steps or what are youEthan [00:05:50]: Let's say just the train-training the model, like from acquire new data and maybe design new algorithms and train a new model, maybe at smaller scale orSwyx [00:06:01]: So cycle time for like any hyperparam that you're searching.Ethan [00:06:04]: Cycle time and tune to like eval this model. Is this model better than my previous iteration?Ethan [00:06:11]: SoSwyx [00:06:11]: So it's like before you, someone had already set this up that you can iterate very quickly.Ethan [00:06:15]: I think the foundation there is extremely good forDeveloping and research models.Ethan [00:06:23]: And often I find is it-- this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the, in the model training pipeline. Those give, those give the biggest boost to the model quality.Vibhu [00:06:46]: It's interesting, right? So you say it's like small team, less communication bandwidth, but also a lot of quality is like find little bugs. It seems counterintuitive, right? You have a lot of people, you can iron out more of those, but it's interesting to see the other side, right?Swyx [00:07:00]: I also wonder, have you-- do you try using LLMs to look for bugs? I don't know.Ethan [00:07:05]: I remember at that time it was mid two thousand and twenty-five, so it's the coding model wasn't quite there yet. I remem- I remember like December two thousand and twenty-five, it was extremely good. Yeah, I've been, I've been using it at that time. It's, it's helpful. sometimes it produce codes that are kind of difficult to maintain, even though like the first time it built something extremely fast. But it gave the, like a spaghetti code, thousands of lines that I couldn't maintain, and the LLM itself couldn't figure out what's, what's wrong and how to improve on top of it. But now I find it much better. Yeah, I want to bring up another point here is now coding models are much more efficient and can help us implement stuff much faster. Compute might become a bottleneck again because previously, like if you want to train a new model, say you want to generate new synthetic data and then or write a new algorithm, it might take a few weeks. And during that period of time, you don't-- you might not have experiments to run. But now you can build that thing within a few hours, then you can immediately train a model.Ethan [00:08:24]: Now you have to have enough compute to try all of the ideas. So compute might be the bottleneck of iterating speed again.Swyx [00:08:36]: yeah, I actually, honestly, I think it's like kind of a stressful job because you're “Well, I should be trying everything, and if I'm not, then I'm not doing my job well.”Vibhu [00:08:48]: there's also the stress of you're eating thousands of GPUs per hour, which is very expensive and, compute can go to other researchers.Swyx [00:08:56]: You got the daddy Elon toVibhu [00:08:57]: You got daddy Elon.Ethan [00:08:59]: It wasVibhu [00:09:00]: But there's still finite amount of compute, like you want to use it, you want to use it well, you want more of it.Ethan [00:09:06]: That was quite stressful indeed. Yeah, I think one thing is the-- with coding models now, like a lot of these jobs can be automated, which is much better. A second, it's a, it's a marathon, so you got to maintain good health and, a regular schedule.Vibhu [00:09:28]: It's, it's hard to hear that when you shift from zero to nothing in two months.Swyx [00:09:32]: and, I think obviously the culture at xAI is very famously, people work very hard. one thing I did want to dive into, in our-- in the notes that you, that you sent ahead of time, you had specific comments about the cost of Video Gen training. presumably this is on the Colossus-1, right? the two hundred megawatt cluster. Any whatever you want to just share on that.Vibhu [00:09:54]: I think there's, there's three things we're talking about, right? So there's Video Gen, there's also the Image Gen model that you put out. Do you want to like complete the, okay, so zero to one, you have a few months. Just what are the stages of create Image Gen model?Swyx [00:10:06]: Oh, yeah, maybe I got distracted.How Image and Video Models Are Trained: Synthetic Captions, Tokenizers, and VAEsVibhu [00:10:07]: Sorry. and then, from there's Video Gen, there's Audio Gen. Would love to get into those next. But what is that first few months like? So small team, a lot of bugs, iterations, but what does it look like? Do we take something off the shelf? Do we just get data compute? What's, what's the few months like? How do you go to state-art Image Gen model? How do you just start?Ethan [00:10:28]: I cannot comment specifically how xAI did, but it's, it's a quite standard process. I can draw some, examples from Cosmos. So mainly it's building a video model, you actually need to build a image model first. And building these two models, the data you need is a hundred percent synthetic pair of language and image or language to video. Because on the, on the internet, actually, the videos don't naturally associate with text. So you can say, oh, like on YouTube, you have the title and you have the description and the commentsSwyx [00:11:11]: TitleEthan [00:11:11]: of a video, but usually they're not relevant to the video itself. And say maybe like the video is a natural scene of mountains or something, and the title is, I'm so happy today.Ethan [00:11:26]: So they have they have no correlation at all. So the first step is to, you have to generate synthetic pair of language with the videos. So you gather videos from the internet, and you use a VLM to caption the videos. So that part, here's a question, like how do you, how do you gather VLM to begin with? So if there's noSwyx [00:11:55]: You, so you fuse the model, right? LikeEthan [00:11:57]: Say if there's no like VLM exists, like how do you generate the text to the beginning, right? It's, it's impossible.Swyx [00:12:04]: I see.Ethan [00:12:05]: In the beginning, it's like you ask human to describe the video as detailed as possible.For example, you ask them to describe everything, like all objects, all characters, and all interaction and dialogues in the, in the videos. So that's in the protocol of Cosmos labeling. We require the objective we give to the labelers was that you have to describe the video as detailed as possible, such that a blind person hears a blob of text can reconstruct what the video is like from their head.Swyx [00:12:43]: Video or image? You're talking about images.Ethan [00:12:44]: Video or image, either one of them.Vibhu [00:12:47]: This was pretty common when we went from clip and DALL-E, right?Vibhu [00:12:51]: It's all training on really detailed captioning of images. So same is applied to video, but insteadEthan [00:12:57]: same appliedVibhu [00:12:57]: of using multimodal model to pass in video images and write rich descriptions, you can alsoSwyx [00:13:04]: I think there's this traditional perspective of supervised, or, very highly human curated thing. I feel like there's a unlock with unsupervised, right? Where like you have enough to bootstrap that you can just throw common corpus on it or, whatever. like unsupervised vision and language pairing, right? Like where you just have, interspersed image and text and it just learns. To me, that is the VLM breakthrough that is different from the clip, different from the LM era.Ethan [00:13:36]: It's interesting to see that you kind of need both data.Ethan [00:13:41]: For example, for theSwyx [00:13:41]: You need it to bootstrap it up. YeahEthan [00:13:43]: for the generative model training, there's also usually like a small percentage of unlabeled data. So the model is instructed to generate a video without any text instruction. That can also help the model generalize. So after this stage of generative synthetic pair, so, one important common step is to train a compressor or a tokenizer of the image or videos. So because, if you train-- If you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it's, it's a lot of tokens. So like one image, it's, a thousand by a thousand, it's like one million tokens, one million pixels. It's impossible to train transformer on that. So it's, you need to train a tokenizer, which can go from image to latent space and latent space back to image.Swyx [00:14:45]: That's why we named the podcast.Swyx [00:14:48]: But, basically, you're talking about vocabulary science.Ethan [00:14:50]: so vocab.Swyx [00:14:51]: And so, what is, what is imp-- like a million is impossible?Ethan [00:14:54]: In generative models, the vocab is continuous. It's a continuous space. We can think about like you map an image to a vector. It's a, it's a fixed length vector. It's sixteen or forty-eight, something like that. And then you map that vector back to the image space. And the mapping is, has-- The mapping is patch-based. So you say you haveEthan [00:15:22]: a sixteen by sixteen patch and you match, you map that patch of pixels into this latent space.Swyx [00:15:29]: We've covered thisVibhu [00:15:30]: This is like the vision transformersSwyx [00:15:32]: VAEs,Ethan [00:15:33]: VAEs.Vibhu [00:15:34]: You basically compress your input, you do your generation, you're reasoning all that generation in smaller dimension, and then you project back out.Swyx [00:15:43]: VAE is a form compression, but I think the for me, the patching thing is from VIT, right?Ethan [00:15:48]: You can make those.Swyx [00:15:49]: Literally the, yeah, the paper is titled like sixteen by sixteen is all you need. something like that. and then I think also, people make a lot of comparisons with this kind of patching with convolutions.Swyx [00:16:02]: Which is you're, you're kind of re- reconstructing the old paradigm with the new.Ethan [00:16:05]: Actually, in VAEs, there are, there are both convolution networks and transformers. You can actually do both.Ethan [00:16:14]: After this VAE, so what you've got is you've got latent space tokens and you've got the language tokens. So now the training of the diffusion transformer, usually generative models use diffusion transformers. It is actually quite standard. It's, it's very similar to how you train a language transformer models. It's not that much difference. It's just the tokens, the visual tokens in, visual tokens out. The only difference is there's a denoising process. So you train the model to unmask some of the noise. So you add, you add random noise to the visual tokens, and then you train the model to remove those noise to generate the clean tokens. Any inference, the model can iteratively remove noise from a hundred percent noise.Swyx [00:17:12]: And then there's also, to speed things along on the tech tree of diffusion, there's CFG, and then there's, there's also, latent diffusion that, there's, there's someone in there. I think, somewhere along the line, obviously, like stability and all these other guys, pioneered a lot of this, architecture. I don't know if you want to get into that or just, or do the video side up to you.Bootstrapping Video from Image Models and Temporal CompressionEthan [00:17:37]: After you train such model, such image model, the reason it's a, it's a foundation for video models is that image models are cheaper to train, and they have much denser connection between language and text. So, sorry, language and images. For example, you train a billion, you train on a billion images, and there's a mapping from the text to the image. And the cost to train the same, like the, a billion, a billion text to a billion videos, that's much more expensive because videosNaturally have more tokens than images. Because the diffusion models, their understanding of, language purely come from this mapping. So if you don't have enough mapping, so if you only train on like a ten million videos or something, there-- you might not see enough language tokens in your training, so your model does not understand human intention enough. So that's why you really-- you train-- you first train this image diffusion models, and then you bootstrap the video model from there.Swyx [00:18:53]: One thing I did want to ask, because I-- actually, I think you're, you're the first per-- video model person I've ever talked to, I think. we've, we've like talked to Luma and all those folks. There's all these tricks in video compression where basically frame by frame there's not that much difference, so actually you don't have to regenerate or save the whole frame, right? but I think MP4 compression or something else like that.Swyx [00:19:16]: is it tempting to use that? Or as far as I can tell, everyone just treats it as, “No, we would just generate every frame.” Is that roughly the state-art?Ethan [00:19:27]: There are a few different approaches. Let's say first, like you want to just directly use MP4 compression and use that as the tokens for the transformers to train, right? So people actually have tried that, but the main challenge is the latent space for the MP4 tokens were not, were not very comprehensible for the models. It's, it's extremely hard to train on that. And there's aEthan [00:20:01]: So that's why they created VAEs, which creates more continuous, latent space, so the models can understand that latent space and learn from it much easier. Even within the VAEs, there are different difficulties of the latent space. So you can imagine something the simplest, the most naive VAE is like you have an image, and you just shuffle all of the images into a, into a vector. So you don't need to train any VAEs, right? But that latent space is extremely hard for models to train on top of. That's why there are some debate on like how do you compress the tokens. So you mentioned like you can compress frame by frame. Also, you can compress, the temporal dimension.Ethan [00:20:52]: The difference is if you compress the temporal dimension, you get a much higher compression rate. Because there's temporal redundancy between frames, because, this frame and the last frame, likely they are mostly similar, so there's only some small difference. for example, I think in 12.1 VAE, they have like a eight by eight by four compression rate. So the four temporal tokens are compressed into one tokens. That can save a lot of, save a lot of the context length. If you do it frame by frame, you have to do maybe like eight by eight by one. Your context length will be four times larger. That being said, the benefit of the frame-- per frame compression, we might come back to this later, is, real-timeness and interactivity. ‘Cause if you, if you strain the output of the model, frame by frame, you can-- the model can respond to any user request immediately. So if you have like a temporal four compression, four times compression, thenSwyx [00:22:06]: It might be laggyEthan [00:22:07]: there's a lag there in nature.Swyx [00:22:10]: So you're very pilled on this. let's just go ahead and bring it up ‘cause we have the visual prepared anyway. There's some frontier applications of real-time video gen. So Flipbook is one of the examples that went viral recently, right? What is Flipbook?Real-Time Generative UI: Flipbook, Neural OS, and Diffusion Front EndsEthan [00:22:23]: Flipbook is kind of like a web brow- web browser. You can see like it has the web bro- browser UI on top. The difference is all of the UIs are generated by generative image model in real time, and anything here are fake. But you can, you can explore inside this wor- this imaginary world. Say like we-- here we have engineering the Great Pyramid. Like the model generates this for us to understand how it works, and if we want to navigate around and understand further, we can click on some of the, some of the description here, and the model will generate a new page, new subpage describing the details we want to know about.Swyx [00:23:14]: So it's basically kind of we're playing a video, but it's pausing for our next interaction, and then it just plays the next thing based on our interaction.Swyx [00:23:23]: Which is kind of cool.Vibhu [00:23:25]: and you kind of decide your story. So this was, how do you make a pyramid? levering technique seemed interesting, right? It shows how do you take Okay, I want to know what is thisSwyx [00:23:35]: The demo, the demo tweet had more animation between frames.Vibhu [00:23:38]: I think it's just skipping,Swyx [00:23:39]: Oh, it's just skipping a lot of frames.Ethan [00:23:40]: they also have a video modeVibhu [00:23:42]: It takes a lot. There's a lot of peopleEthan [00:23:42]: but, a lot of people are using it.Ethan [00:23:45]: So it's not available.Vibhu [00:23:46]: There's a live video stream. We can try,Swyx [00:23:50]: So this is an example of the kind of future that you see at the extreme. We don't-- we're obviously not in it today.Swyx [00:23:56]: But in a world where inference is completely free this is better than generating code and text?Ethan [00:24:02]: So this is, this is a final state of where Viva will be at for word model, I think. Imagine internet doesn't exist, and then you type in google.com. Like what should, what should, what should a model show you?the model can imagine something, and this is what the model imagine. And these web pages, they completely do not exist. So I think as the inference costs come down, we are going to have generative UI for everything. If you think about how the coding model works, so they write code for a web page, and they render the code might be con- converted into binary, and the binary render the pixels on the screen. So we in machine learning, every time we have some breakthrough, obviously it's, it's more intuit. So why don't we have like user instruction to the pixel directly? So the generative UI will be user intention to the pixels directly. And say like even if I want email, let's say everyone have the same interface, but I want, I want it slightly different. I want the email to show to me like a TikTok, so I can swipe left and right for the emails. And or maybe you want something else. We can have completely different things. Or like I have I'm looking at, Instagram stories, and I don't like the Like button. I always may click it. And, generative UI resolved it. So it's going to be a revolutionary replacement of the interface. So in the future, we might have much more powerfulEthan [00:25:50]: LLMs and coding models running behind the scene. And in the, in the front-end, the diffusion model will actually be the front-end to show stuff to you. That's how I imagine it.Swyx [00:26:02]: Diffusion front-end, deterministic back-end.Swyx [00:26:04]: Something like that. I find that very expensive, but,Vibhu [00:26:08]: I find it interesting you called LLMs writing code on the back end deterministic, but okay.Swyx [00:26:14]: you write it onceVibhu [00:26:15]: Compare it toSwyx [00:26:16]: And then you execute.Ethan [00:26:17]: If you think about the cost, say, let's say H100 costs $1 per hour, and if you use this eight hours a day and thirty days, so, every month you're paying this two forty, you'll actually not wanna pay for that. That's even more expensive than Cloud Code Max. But if you think about the compute costs come down like two times every year, and I think the future will likely arrive like within few years.Vibhu [00:26:49]: It's everything, right? compute cost comes down, compute gets faster, model gets smarterEthan [00:26:54]: More efficientVibhu [00:26:54]: model gets smaller.Swyx [00:26:55]: I don't know why you say two times, ‘cause I think it's like 100 times. In language models, it is roughly one hundred to a thousand times every twelve to eighteen months, for the same given level of LMSys, ELO.Vibhu [00:27:08]: That's a net of everything, right? That's model performance alongside compute. So different than just compute costs come down. But, a very interesting future.Swyx [00:27:19]: So the web designers will have to shout out that accessibility is an issue, right? how do you deal with screen readers or whatever. But yes, this is higher bandwidth storytelling than anything you can possibly generate with code, right? So I think that's the rough idea.Ethan [00:27:34]: And I'd like to add a little bit that so human naturally have the maximum bandwidth when we are looking at things, look at videos, and we also have maximum output bandwidth when we are talking. So in the future, it might be something like we talk to AI models, and the AI model responds back with a generative UI. So that would be the maximum input and output bandwidth to interact with AI models before neural link happens.Vibhu [00:28:06]: And it's also very custom, right? Some people are very visual, some people are not as visual, right? They prefer the text. But the best thing about generative UI, right, it can also be text.Swyx [00:28:17]: There's another project that we wanted to highlight, which is the Neural OS. Kinda similar idea, but here you're literally operating, simulating an operating system with a video model.Swyx [00:28:27]: and you can play Doom, you can do Firefox. I find this like mildly less impressive, obviously, because it's an OS that I can run.Swyx [00:28:37]: But here everything is imagined.Vibhu [00:28:40]: I was, used to the Command+W to close the Firefox tab. It didn't crash. That's why I saidSwyx [00:28:45]: It's too immersive.Vibhu [00:28:46]: It's, it's too immersive for me.Swyx [00:28:47]: Too immersive.Vibhu [00:28:48]: I wanted to close the tab.Vibhu [00:28:49]: But yes, I can play generated diffusion.Swyx [00:28:51]: this is shockingly fast.Swyx [00:28:54]: Because I remember there was a demo about like maybe one to two years ago. Someone tried to do the first-person shooter with a image model. There was no consistency. It was very slow. But here it looks like realistically it's-- this is Doom.Vibhu [00:29:07]: I think there's two sides to that, right? There's okay, what is running a game? The heavy part of it is actually the game engine, all the lighting, all that stuff, the graphics. This is just kind of video, right? Like we've solved consistency. This is still, it looks like a few years old image generation. There's some temporal consistency, but it's, it's kind of just images stitched together as frame video. But it's a good visual representation to pi- to picture the future you wanna see, right? that's, that's what I see in these more so.Ethan [00:29:38]: This reminds me of how the video models gets better and better. So Neural OS is kinda if you just look at it feels like it's just a crappy version of the, like the Windows we could have, right? And, but the difference is, so the model, this model is overfitted on the existing operating systems. It can generate nothing different than that. But it's actually also similar to video models. So when we are training these video model, image model, we train them on internet. There's no imaginary supernatural stuff on the internet. But once we train this model, you can prompt the model to generate something supernatural that have never existed in the data set. So if you train your Neural OS or neural computer on the standard screen recordings on the entire internet. The model can imagine completely new interface to interact with the computer.Swyx [00:30:43]: This is one of those things that is magical to me. usually generalizing out of distribution is bad, but somehow we have learned some kind of internal world model that you say, this plus, but it looks like rainbows and butterflies, it'll do it and it will kind of make sense.Swyx [00:31:03]: So yeah, that's kind of cool. Yeah, I don't know if there's any comment more on there. I do, I do wanted to, I did wanted to touch a little bit more on the model architecture stuff, which I think you were getting. It's, really fascinating. We don't get a chance to talk about this enough. So one of the papers that we covered, we've covered every annual, segment anything release. and I don't know if you follow-- you're a computer vision guy, so youEthan [00:31:26]: I knowSwyx [00:31:27]: . So they did memory attention, which is kind of interesting. And I always think, anything where you can, across the temporal dimension, keep some consistency, I think it's, very fascinating, and I don't know if Basically, does that-- the CV side bleeding into video gen side, I think is underexplored, right? we talk about it for labeling, but actually you can borrow the architecture itself.Ethan [00:31:50]: There's, there's also complete different approaches, right? you brought up the term world model, so we went from video model to world model. There is diffusion, but there's also other approaches that people are doing. So maybe we get into those after as well,?Swyx [00:32:03]: He has a whole definition of world models and stuff. I feel like we threw a lot at you. Whatever you want to comment on.Why Video Models Are Expensive: Storage, I/O, and Training ScaleEthan [00:32:10]: I think one thing that we should actually comment back on is okay, so we were talking about the steps to train image gen to video model. One thing we don't see as much of is okay, you brought up the delta in training data, right? SoEthan [00:32:24]: you won't have as much a video model might not generalize, but what is the cost of training a large video model? So we know for LLMs roughly, okay, even like the poolside thing that came out today, right? It's a Gemma level model trained on roughly forty trillion tokens at this many H200s over this much time, right? You can see what is the exact cost of that. So how many GPU hours over how much H200 costs? So how do we do the back-end math of, same thing for video models, image models. How do you, how do you kind of break that down? I can share some back-envelope calculation. So surprisingly, video models is-- the cost is very-- is comparable to language models and obviously the largest scale is language model, maybe like a medium scale to language models. I said just storing the videos alone, it costs a lot. You can, you can maybe look up on AWS or something.Ethan [00:33:20]: You really, say if you have a billion videos and let's say, let's just say like each video, like five megabyte, then you need five petabyte to just store those videos. And also remember we talk about you use a VAE to compress the videos, and you also need to store, typically you need to store those continuous feature, in-- also in your storage. That's also comparable size with the videos themselves. So just storing these videos and the features is tens of petabytes alone. And,Swyx [00:33:58]: I just, I just looked up the calculation. Five petabytes on S3 Standard is one hundred K per month.Ethan [00:34:05]: AndSwyx [00:34:05]: It's comparableEthan [00:34:05]: and you needSwyx [00:34:06]: AndEthan [00:34:06]: And then like tens of petabytes, two hundred K. And even more expensive is you have the ingress and egress.Swyx [00:34:13]: Oh, yeah.Ethan [00:34:14]: Like you-- through the internet. You have to just to download those videos, I believe it's, it's more expensive on AWS than just storing those videos.Swyx [00:34:25]: Storing, yeah.Ethan [00:34:25]: And each training runs, you probably need to pull them once. If you train multiple times, it's, it's even more than that. So it's like just storing the network, those costs is just, it would be a few, a few millions per month to just storing everything, not to mention the GPU cost.Ethan [00:34:45]: AndSwyx [00:34:45]: my side tangent, the compute rental, like GPU rental is very efficient. There's one side, okay, you can be XAI and build your data center. Should we not just build our, storage compute as well? LikeEthan [00:34:57]: Of courseSwyx [00:34:57]: cloud cost compared to just,Ethan [00:34:59]: You save so muchSwyx [00:35:00]: store. Yeah, exactly.Swyx [00:35:01]: Especially with like egress and stuff. So.Ethan [00:35:04]: That's a good idea, but it also comes to-- there are some of its own challenges.Swyx [00:35:09]: Of course, of course.Ethan [00:35:10]: like people who build the GPU data centers, they might not expect this much, storage. And yeah, people build storage, typically they just build it somewhere with just CPUs.Swyx [00:35:23]: I just looked it up. Five-- AWS only charges for egress, not ingress. Tier five for five petabytes is two hundred and thirty K.Ethan [00:35:32]: Even more expensive than the storage.Swyx [00:35:34]: But storing is per month, right? You check in, then you cannot check out. so it's so cool. It's okay. So there's that side.Ethan [00:35:41]: So the TLDR, my backhand mathSwyx [00:35:42]: Data is larger than you think. Yes.Ethan [00:35:44]: my backhand math of GPU hours times GPU cost is also very much, I'm missing some storage.Swyx [00:35:49]: You're also-- you're basically like also more IO bound than normal training.Swyx [00:35:55]: Yes. ‘Cause like data loading, so caching everything, it becomes super important.Ethan [00:36:00]: So in Cosmos, we did a lot of optimizations to make it not IO bound. So, speaking of the training, actually training the model, the GPU cost, if you look up like the open source model, how big these video models are, I think like LTX has nineteen B parameters. That's a dense model. And people are also exploring, MoEs, so it might be twenty B active and, like a hun- hundreds B, total. So that's, that's even-- that's similar size as medium-sized LLM models. And if you, if you look at number of tokens-Uh, we disclose that in Cosmos. It's also like tens of trillions of tokens on the visual tokens. So putting this together, the cost of, training these video models, it's actually comparable with LLMs. Not to mention, the infra is slightly different from LLM, so it might be less efficient to train these models.Inference Speedups: Step Distillation, Consistency Models, and GANsSwyx [00:37:04]: Do you get the benefits of traditional diffusion speed-up? So for, images, there's LCM, LoRAs for, fine-tuning. There's, there's a lot of stuff that's beenEthan [00:37:15]: Flow matching.Swyx [00:37:16]: there's flow matching. There's a lot of stuff that's been done. there's some overlap that applies to diffusion on the inference side and stuff or?Ethan [00:37:23]: so the difference-- the inference side is a completely different story.Ethan [00:37:28]: I think for the training side, it might be a little bit hard to reduce that cost. And for the inference side, the biggest gain is from the distillation of these models. You can-- It's called step distillation, slightly different from knowledge distillation in LLMs. So you-- Typically, for flow matching models, you need like 100 steps or something. Like a distortion model even need even more, like 1,000 steps to generate a good image or video. A step distillation is try to learn to generate fewer step from the model itself. It's kind of like now we-- you use the full model to generate in 100 steps, and then you take a model that only generate 10 steps and let that model to learn from the perfect one.Ethan [00:38:25]: why this workSwyx [00:38:27]: Strong to weak seemingly.Ethan [00:38:28]: It is. It's kind ofSwyx [00:38:29]: DistillationEthan [00:38:29]: kind of like strong to weak. the-- from the modeling perspective, the strong model, the teacher model is trying to model the image and videos of inter-internet, and that distribution is extremely complex. But the step distilled model is just trying to learn from the teacher. The teacher is a model, and the size is fixed, as the distribution is much simpler than the whole internet. That's the intuition I have why step distillation can work. So usually these models serve in productions, they only run in a few steps. In Cosmos, I believe we have, we have like four step and eight steps. If you do some simpler task, image-image translation, it can even run in fewer step, like one step in Cosmos Transfer.Swyx [00:39:22]: I think this is the same intuition that guides a lot of the consistency model work. I sent you a link for, SCM. I don't know if you covered that. To me, that was actually one of, the most impressive papers I've ever seen from OpenAI.Swyx [00:39:34]: That this is the unifying grand concept of consistency models. I don't know if you have any comments on this.Ethan [00:39:41]: So there are, there are a few different approaches,Swyx [00:39:46]: Oh, yeah. Here it is.Swyx [00:39:47]: Two steps versus twenty or 100 steps, whatever. It's already done.Ethan [00:39:52]: So there are, there are a few different approaches, for example, consistency model, and there are also Actually, we shouldn't forget GAN. So GAN, actually, that was, that was the OG ofSwyx [00:40:05]: OGEthan [00:40:05]: step distillation ‘cause it trained just one step to begin with. So actually, a lot of, uh-- For example, there's a distribution matching distillation which use, which uses GAN, as one of the laws for distillation. It-- GAN just tells you, “Hey, generate an image,” and thenEthan [00:40:31]: it has a discriminator to tell, is this image real or not? So the model, the model just need to learn one of the distribution, not the full distribution. Because in training, the model is asked to reconstruct the ground truth image from the internet, which is extremely hard. And in-- When you're training GAN, it's a step process. It's just a, “Hey, you generate image. Does this image look as real as the image from the internet?” Which is a much simpler task. And, yeah, combining a lot of these approaches together, people typically do that, like consistency model and distribution matching and GAN, and we can get these few step models.Audio-Video Generation and Time AlignmentSwyx [00:41:21]: Then there's one step I wanted to add, which is audio and video.Ethan [00:41:26]: So, Grok Imagine zero point nine, I believe it's, it's a first audio video transmodel deployed at a large scale. SoSwyx [00:41:39]: And that was your first model?Ethan [00:41:40]: that was, Grok Imagine's first model. It's, it's audio video, joint generation. I think the hard part is, the modality alignment, ‘cause before this transmodel, we have, we have text to video alignment. We have this, correspondence between text and video. Typically, most of the VLMs, they understand images and videos. Video's very rare, and they don't understand audio mostly. And if you look at the audio generation on the LLM side, you can talk to them perfectly fine, but if you ask them to sing a song or something, it typically is not very good. Also, they don't have, they don't have music either. The hard part is thatUh, actually audio has two component. It has like a discrete component, a continuous component. The discrete component is like the language.Ethan [00:42:44]: So when we speak, it's just, someSwyx [00:42:47]: It's an ASR issue, yeah.Ethan [00:42:49]: It's, it's text token with some characteristics, I would say.Ethan [00:42:54]: But musicSwyx [00:42:56]: I think the speech guys would disagree with this.Swyx [00:42:57]: Like disfluencies and then,Vibhu [00:43:00]: There's tones you can get angry.Ethan [00:43:01]: Well, I say largely.Ethan [00:43:03]: the mu- but the music is completely different. It's, it's very continuous, and you cannot model them like discrete tokens in language models. this is like the hard part for models is, not to mention we have to align text, video, and audio together.Ethan [00:43:26]: SoVibhu [00:43:26]: How?Ethan [00:43:28]: So significant-- some significant challenges are like-- So first, like we talk about as the VLMs, they cannot understand most of them cannot understand audio.Ethan [00:43:39]: So you have to have some way to do the synthetic data generation for audio. You have to caption the model, and that involve, that involve synthetic data and human data effort a lot. And not just surprisingly, most of the LLMs are very bad at recognizing, like the beat, tone, and the details of the of music. They can, they can give some general prediction of which song is this, but it's very hard to describe the details of the music. like we mentioned in image generation, like you have to describe image as detailed as possible so that someone blind can reconstruct that. So here is like someoneVibhu [00:44:32]: DeafEthan [00:44:32]: someone deaf can reconstruct how the music sounds like without actually listening to it. Maybe you can think of it need to have the-- or they call the script.Vibhu [00:44:49]: Subtitles, yeah.Ethan [00:44:49]: You gotta have all the details of the music, and the dialogue.Vibhu [00:44:55]: So is the challenge there typically stuff like music and audio, or is it just Like is there a baseline? Okay, there's enough data where we can understand, narration, conversation, but there's nuances in audio that's where you hit all the data issues or is it just from stage zero, you just do it all right?Ethan [00:45:15]: So one important thing is like the alignment. So the model, the model has to know like the video and audio, the, uh-- it has to have a time-based alignment, like at which time step the video and the audio token correspond to each other. But we actually don't have this kind of alignment for most of the other modalities. If you think about like text and image, text and video, they are loosely aligned. So you can, you can have a description of what's going on in the video, but you don't have to exactly, You typically don't have exact description, oh, at, time step one second like what happened?Vibhu [00:46:02]: It's veryEthan [00:46:03]: At time step two second what happenedVibhu [00:46:03]: coarse. Yeah.Swyx [00:46:05]: So what was the ideal time step? You have to oblate it, and then it's like four seconds or something.Ethan [00:46:09]: So that comes down to how you design the model to, for the model to be aware of as a time, as a time modality. So the model is like a time aware. And that's something pretty unique if you think about LLMs. So if you ask LLM to complete a task, say they, uh-- you ask them and they will say, “Oh, this task will probably take twelve hours to complete,” and they come back in one hour. Say “I've already spent two days on this and I've exhausted everything.”Ethan [00:46:47]: So the LLMs them-themselves, they don't have a sense of time there.Vibhu [00:46:53]: I actually don't think that's just them not having a sense of time. I think it's somewhat based, right?Vibhu [00:46:58]: Like you tell someone, “Okay, go work on this feature. Go implement this,” there's a general understanding you would have of how long that would take without LLMs working at LLM speed, right? So you think back like two years ago, if I tell you to like build me like a new front end for latent space, have a search bar, have all this, you'll estimate that it'll take a few days, right?Vibhu [00:47:19]: So you tell an LLM, “Go build this.” It'll take me a few days. But I think it's somewhat grounded as opposed to them not having the best-- Not saying that they have a great understanding, but I think that example is like you can see where it comes from, right? You're trained on all over the text.Swyx [00:47:35]: They're, they're trying to estimate what a human would say.Vibhu [00:47:37]: because that's what the, that's what the data kind of represents. It's not themEthan [00:47:41]: It came from the corpus on the internet. People have a estimate of how much time.Vibhu [00:47:45]: And not even just in direct like training samples, right? Just your world understanding of tokens of how long stuff takes, right? Go read a book. It'll take you a while, right?Vibhu [00:47:56]: Even if you do nothing but read a book, it takes a few days. So yeah, LLM, I read it took me a few hours.Vibhu [00:48:01]: It'll take me a few hours to go through this research. But this is a tangent.Swyx [00:48:05]: Somewhat, yeah.Swyx [00:48:06]: This is a train of thought I haven't really expressed until now is, which is basically like a full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model. which is like this whole recursive thing down the, down the line. but yes, and that the world model can be wrong and that they need to update it and blah. Yeah. We've, argued this on the, newsletter as well, that there needs to be sort of recursive or adversarial world models.World Models: Real-Time, Long-Horizon, Interactive VideoVibhu [00:48:34]: just, to ask, how do you define world model?Swyx [00:48:38]: Oh, yeah, let's go there.Ethan [00:48:40]: SoVibhu [00:48:40]: So just for context, we talked about, video generation, and then there's a-- if you say there's a distinction between world models, what's your, what's your definition? How do you see the two?Ethan [00:48:53]: So disclaimer, I'm not going to debate, what is world model. Yeah. there are many definitions, so I'll just talk about my definition. Since I came from the multi-model, multi-model domain, so mainly talking from video. So world model is like real-time interactive long horizon videos. So there are three parts. so we-- let's talk about them one by one. So the so interaction, so we just, we just look at Facebook and neural computer. So the interaction part of it, so you, world model can allow you to interact with them through keyboard, mouse, and maybe also voice. So these all is-- all is a modality. You can, you can interact with the model, and the model should respond reasonably. Second part is real time. So once you, once, say, you move your mouse, if, say, the world model generate a game, how fast can the game respond? So if you're like professional CS: GO players- -my say, oh, you have to respond- He's beginner within sub ten milliseconds or- Yeah even less. So that's not most of the- No, sixty FPS. Let's go. Oh, three hundred FPS. Oh, five hundred FPS. Wait. okay, yeah. I didn't do the math, but yeah, okay. Uh- Yeah, three hundred FPS, that's a three millisecond. So you have to respond- Oh, s**t. Okay. YeahEthan [00:50:29]: within a millisecond. Most of the video models cannot do that. Yeah. And, but if you, say, if you have a video model that is, say, like a digital human, the response time might be more generous. Maybe typically, for real-time voice interaction, it's like two hundred millisecond. So that's, that's much more generous. But even two hundred millisecond is pretty, it is pretty tricky, ‘cause remember we mentionedEthan [00:51:01]: you have this, temporal compression coming from the VAE. So if you, if you don't compress the temporal dimension, your sequence length is going to explode. So if you want to have this real-time, real-timeness in your model, you have to do is one context problem. And the third part is long horizon, ‘cause we-- if you're not going to just play with, video games just, a few seconds, most video models only a few seconds. We're going to play with minutes, hours. The model have to be able to generate long-form content.Ethan [00:51:42]: So putting these three together, it's, real-time, long horizon interactive videos. I think the final state will be, for example, like a video, a video version of Playbook, where you can, you can interact with, a neural computer. You move your mouse, and you click on the generative interface, and it will reply to you through pixels- generating in real time. But getting there, it's, it's a very long way to get there. So one of the first step, at Grok Imagine, where I led a small world model team there, was to build video extension. So, video extension- it's the first step of interactivity. Yeah. It's, it's the first step. Yeah. So it's the first step- You have it here, video editing, yeah. Yeah. Yeah. So the first step is because, this unlocks long horizon videos. Typically, for most of the video generation models, you give it a prompt or an image as an initial frame. You generate video, that's it. That's just, one time, done. And some creators would try to, use the last frame as a first frame for the second video. It can-- sometimes it works, but if you do it a few times, it says the quality would decrease. And- It doesn't have that context- Yeah over the full video, so the temporal- Yeah, exactly. Yeah, ‘cause you only gave it the last frame, of course, right? Yeah. Exactly. And- it's actually a pretty fun hack. if you've seen like- Oh, no, he's saying something better. Yeah. And for example, like Vue, I remember Vue 3 has like a second context of the last video. It is slightly better than using the last frame, but it has the same problem-- similar problem that it, the quality would decrease. if you extend a few times to, one minute, the video quality would look much worse than the first video. Second, another problem is that the model doesn't have long-range knowledge of, what's happening before. Say, if they generate some dialogue, some, two people speaking, and their voice might change, over some time, especially if the second conditioning, it does not cover the previous context. So these are the core challenges. So the Grok Imagine video extension, it has historical context of all of the previous generated videos. It can, It has, it has the context of, who is speaking and what objects have appeared and everything, having that to generate the next video. So if we naively do this, you can imagine, just, put all of the previous history video tokens into the context. The context lens will easily explode. Especially for video models, that can be like a few, a few million context, I would imagine- context lens. Yes.Yeah.Swyx [00:54:58]: Let's run with that.Ethan [00:54:59]: for example, like in Cosmos, I think just five seconds of video is like a fifty K or sixty K number of tokens. So like if you do, if you do fifty second, that's a five hundred K tokens. If you do longer than that, easily explode. This long horizon, problem was the first step we're trying to solve world model. It turns out people, yeah, people love video extension. Like a lot, a lot of the creators love using video extension to create longer form videos. This is the part I liked that you have a, you have an intermediate step toward the final goal instead of just a straight shot to the final version very much.Swyx [00:55:48]: But I can see you have a strong vision of where we want to end up.Long Context, Redundancy, and Efficient Interactive VideoVibhu [00:55:51]: Does it seem like it's an efficiency issue? okay, we're at a few million tokens context,. If you draw the parallel to language models, we had very short context, two thousand, eight thousand, then, you scale it up one million, ten million. sure, there's effective context, but at the end of the day, it's just what's it worth? sure, there's a whole training data side. In video, it might be slightly easier ‘cause we have a hundred million token video, right? Just take a movie with the full context there. Like is this efficiency from an inference standpoint that like it's expensive, but we know how to solve it? Or like why is this not the approach? So like my broader point was on your second point of world models, you say it needs to be interactive and live, right? You should be able to play a game and see the interaction live. So one thing I see with research is a lot of what you actually serve is different than what you build, right? So we talked about distillation. You train big model, you distill it, you do quantization, speculative decoding. We do all this stuff to serve it efficiently. Should we not just have a solution, like a world model that can interact well, do inference optimization, serve it, distill it secondary, so make it real time after you solve it? So like a-- another parallel is say, continual learning, right? What we need is someone to solve it and show it works inefficiently. Give it a few years, people will make it efficient. Same thing with regular attention, right? It worked. Over a few years, people have different forms of attention, and we've scaled it to be efficient at log context,? So kind of two things there, right? One is it seems like it works. You've scaled it. Can we not just scale it a lot more efficiently over time? Do we need a separate approach if this works? And same thing with interaction, right? if we can get it done, like if we can solve some way that it works, we can solve making it more efficient from an inference standpoint later.Ethan [00:57:53]: that's actually a very good point. So in videos, there's actually a lot of redundancies. So we solve a lot of the pixel redundancy from VE, but there's more redundancy in long range and long horizon videos. Say, if a character appear in the first clip and then it disappeared, it only reappear at the end of the video, you probably don't need the-- the context, like in the middle of the generation. So you only need that character, where you need. So that's why, I helped build another feature. It's a reference video.Vibhu [00:58:36]: Is it here?Swyx [00:58:36]: is it the same model release or different one?Ethan [00:58:39]: It's a different one.Ethan [00:58:41]: You probably need to search onSwyx [00:58:43]: I'll find itEthan [00:58:43]: X reference to video.Ethan [00:58:46]: So reference video allow you to like upload up to seven images as condition and generate the video. Say, if like I want-- it can, it can be characters or objects or even scenes. Say like I want, I want condition on, Sean's selfie and holding a bladeSwyx [00:59:07]: We have a dogEthan [00:59:08]: or whatever.Swyx [00:59:08]: We put the dog in the thing.Ethan [00:59:09]: you can put them there and the video models will generate the video from and copies the context over. So that can solve a lot of the problems there, like the long context problem. It doesn't need to have a very long context, but it's-- I feel like it's an intermediate solution. The modelSwyx [00:59:29]: It's cheating.Ethan [00:59:30]: the model should be able to like selectively know, where should I draw the references. So say if I want to generate a movie, I generate it autoregressive, like a ten second at a time or something. And now this character appear, I can look back to where it first appear and, bring that back. Yeah, this one, I put the references. Yeah, that's, Optimus, Einstein myself, Annie.Vibhu [01:00:02]: Oddly enough, I used Grok Search to find it, and it pulled your LinkedIn post. But yeah we found it.Ethan [01:00:08]: Interesting.Vibhu [01:00:10]: ButxAI's Underrated Work, Culture, and WatermarkingSwyx [01:00:11]: this is a problem. This is not your fault, but like XAI doesn't communicate all this work that you do very well because they just have the model release and then that's it. But actually, these details are very good.Swyx [01:00:22]: As far as I understand, everything you just described is state-art, like no one else has done it.Vibhu [01:00:30]: A lot of-- yeah, I have a lot moreSwyx [01:00:32]: And then, and then you just put this blog post with the cookies. I'm this is not enough,?Swyx [01:00:37]: but I, obviously this is like the high level numbers that people want to know. But no, okay, soVibhu [01:00:42]: And I wonder, like part of that is also some labs don't share research into what happens. And ifSwyx [01:00:50]: No, but this is literally bragging about how good they are, right?Swyx [01:00:54]: Like, why would you not say that you are capable of extending with full context? this is not a secret sauce. This is like we did the work. yeah, I don't know.Ethan [01:01:02]: different labs have slightly different communication styles.Swyx [01:01:07]: Anyway, if anyone from XAI is listening we are always happy to help you tell your story. Yeah, okay, so you did references, and I think, I think kind of the point you're, you're making is it is sort of like a kludge, right? this is-- you can do seven, but what about 100?Swyx [01:01:23]: Right? Then you need a completely different thing.Ethan [01:01:26]: So I think it's-- this is, a mechanism to, select the context from the history, and you might not put the entire history into the context. for example, there's a paper called Frame Pack, which haveEthan [01:01:41]: a heuristic that the latest history, the last one second, I put the entire history, and the history before that, I would, compress it and makes the video smaller. So they follow this pattern, this build overall pattern that the maximum sequence length is fixed. So the further you are from the current frame, you have a smaller image. So this is just a heuristic. I think it can be more automatic. The model is aware like which history part of it can be select. So this part of the research is actually being actively, worked on by a lot of people. It's also quite interesting. I feel this is actually, this part of long context is a little bit ahead of the LLM part.Ethan [01:02:31]: So for example, like in LLMs, if you-- so contexts keep growing. Let's say if you call tool and the tool call history is extremely long, that's still in context, and keep growing, keep growing. Even if you switch the topic to something else, the whole context was there. There are some agentic harnesses that help you to, say, prune the tool results and, prune Like when you, when you query a file, only show like the top 200 lines or something. Those were very heuristic-driven.Swyx [01:03:08]: For listeners, we did a write-up on the cloud code, leak where there are eight different kinds of pruning, including like you prune the tool results and all that. So you can, you can read up on that kind of thing.Ethan [01:03:17]: I think, one breakthrough in continual learning might be like a way to automatically, manage its own context.Swyx [01:03:27]: These are all heuristics, and they will be replaced by machine learning.Ethan [01:03:30]: InterestinglyVibhu [01:03:32]: TheEthan [01:03:32]: the same thing is being researched in both LLMs and video models.Vibhu [01:03:36]: The interesting thing is also like in the paper you showed, it's actually happening at the model level, right? Compared to like language models, sure, we have base attention, but we'll do our own compression, we'll do our own pruning, which is separate from model error.Vibhu [01:03:49]: Eventually, it all just boils in, hopefully.Swyx [01:03:52]: I think this is a form of like attention, but like also know sort of reasoning attention. I feel like that's different than normal attention.Swyx [01:04:03]: Does that, does that make sense?Ethan [01:04:04]: It's, it's different in the sense that attention, not to mention, set sparse attention aside,

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at teamworkstactical.comWe are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at trainheroic.comThis week Drew and Alex sit down with Libby Alders — chaplain, researcher, library technician, and self-described tri-vocational nerd — to actually figure out what it is, why it matters, and why the military keeps trying to slap a number on something that might not need one.This one goes deep. Grab a coffee.What we get into:What spiritual fitness actually means — Libby breaks it down to four things: knowing what you believe, understanding that beliefs should evolve, being able to coexist with people who believe differently, and being able to recognize harmful or radicalizing ideologies when they show up.The Spiritual Fitness Survey — an 18-question tool with three subscales: horizontal (community and belonging), mixed (purpose and meaning), and vertical (relationship to the transcendent or divine). Moral injury versus PTSD, and why the difference matters for who you call. Libby's shorthand: shame points toward moral injury and the chaplain. Guilt and fear point toward PTSD and psych. Why the research on religion reducing PTSD risk might be missing a confounding variable — moral injury. If the thing that gives your life meaning is also the thing that got violated, you don't have a protective factor. You have an opening.The 724th Special Tactics case study — how Libby and former podcast guest Chris ran focus groups instead of surveys, built a communication tool instead of a formal metric, and ended up with leadership asking to do their own version because the unit couldn't stop talking about it. Capability-based blueprinting — what it is, why more of the military should use it.The interdisciplinary team problem — why nobody knows when to call the chaplain, why over-specialization and over-generalization are both failure modes, and what "informed consumer" training actually looks like in practice.The table theology tangent — why the ritual of eating together is a human performance intervention that no macro calculator captures.Mentioned in this episode:Dr. Harold Koenig, Duke University — geriatric psychiatrist and pioneer in spirituality, religion, and health researchDr. Warren Kinghorn, Duke — another key name at the intersection of mental health and spiritual healthCapability-Based Blueprinting — developed within CHAMP, Dr. Chamberlain's workMatt Larson — former podcast guest, moral injury talk from the H2F Symposium coming soon to the MOPs & MOEs InstagramCharles Vogel, The Art of Community — former podcast guest, Yale Divinity School; the ritual of meals chapter alone is worth the readAllen Frances, Saving Normal — Drew and Alex's white whale guest. Chaired the DSM-IV committee. By DSM-V, had renounced the whole enterprise. If you know him, please help.Rants and Rituals — Libby's upcoming podcast. No one take that name.Views expressed are those of the speakers and do not represent any official organization.

00:00-02:44 Intro 02:45-14:03 The Last 72 14:04-15:14 Gunman Outside White House 15:15-16:02 PFC Mace Veit 16:03-18:25 Marines Using COD Training 18:26-20:47 Welles Crowther 20:48-23:16 RIP Kyle Busch 23:17-01:24:00 Alex Morrow Interview 01:24:01-01:31:19 Post-ShowYou can find every episode of this show on Apple Podcasts, Spotify or YouTube. Prime Members can listen ad-free on Amazon Music. For more, visit barstool.link/ZeroBlog30

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at teamworkstactical.comWe are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at trainheroic.comFit at 50 and Back in the Teams — Jamie Monroe ReturnsJamie Monroe commissioned as a Navy SEAL ensign at 50 years old. That sentence alone is worth an episode. But what Drew and Alex actually get into is bigger than the headline — it's about the lies we tell ourselves about aging, what it really takes to stay ready across decades, and why identity might be the most underrated performance variable in the building.Drew and Alex also open with results from a poll that surprised everyone — including them.What we get into:How a poll asking which soldier is more operationally effective — perfect fitness score with bad sleep and stress, or minimum passing score with great relationships and recovery — came back 90% in favor of option two. And what that says about what the military actually measures versus what it probably should.Jamie's road back in — the heart murmur that got him medically declined years ago, the DCO process, three interviews, a full MEPS physical, the SEAL Physical Screening Test, and finally commissioning in front of 70 friends and family at 50 years old.Why identity is the most underrated longevity tool — Jamie has never called himself old and broken, and he credits that framing as much as any training protocol for why he's still in the game.The simple running framework that actually works — two easy runs, one tempo, one long run, 15 to 20 miles a week. No pose method required. Just run.What fitness culture looks like inside the SEAL teams now versus two decades ago — less about getting jacked, more about the HYROX athlete profile. Strong runners who can also move weight. And pull-ups that actually count.A full breakdown of every major service fitness test — what Jamie likes, what he'd cut, and why the Marine Corps three-mile run might be the most honest single measure of fitness across any branch.The FAT — Drew and Alex's Fitness Aptitude Test — one rep max deadlift, AMRAP pull-ups, five-mile run. Jamie grades it live, makes some edits, and floats a Cooper Test–style 20-minute max distance format that might actually be the move.Old generation versus new generation — who's actually fitter? Jamie gives a straight answer.Mentioned in this episode:ReadyFit — Jamie's AI-powered military fitness testing app using computer vision to automatically score reps, currently in testing with units at Holloman AFBEasy Day Sports — Jamie's event production company, including a recent 5K for the Dallas Cowboys over Draft WeekendThe Red Bull Catcher Race — the only race where the finish line chases youDSI Human Performance & Biosystems Summit — DC, coming up soon. Alex will be there Thursday.Long and Strong — the Mops and Moes training program on TrainHeroic →Want to help get Bryson DeChambeau on the show? Jamie's working on it.Views expressed are those of the speakers and do not represent any official organization.

In deze uitzending van De Nieuwe Wereld gaat Ad Verbrugge in gesprek met voormalig minister van Onderwijs, Cultuur en Wetenschap, Gouke Moes. Samen buigen zij zich over de diepe crisis waarin het Nederlandse onderwijs momenteel verkeert. Van de doorgeslagen verengelsing en de perverse financiële prikkels aan universiteiten, tot de uitholling van het leraarsvak en de verstikkende bureaucratie op scholen. Het resultaat is een vlijmscherpe cultuurfilosofische en politieke analyse van een systeem dat de zorg voor de eigen taal en gemeenschap uit het oog is verloren, en een indringend pleidooi om het tij te keren.Wilt u bij het Symposium van Beter Onderwijs Nederland aanwezig zijn? De bijeenkomst vindt plaats op zaterdag 30 mei, de toegang is volledig gratis en aanmelden kan via: https://www.beteronderwijsnederland.nl/onderwijs-in-beeld/2026/04/symposium-2026/

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at ⁠⁠https://teamworks.com/⁠⁠We are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at ⁠⁠https://account.trainheroic.com/create-account⁠The Father Figure vs. The Dad Bod — How Parenthood Changes Your Relationship With FitnessFor the first time ever, it's just Drew and John. No Alex, no guests — just two dads talking honestly about what happens to training when kids show up and life gets real.This isn't a "here's how to stay jacked after having kids" episode. It's more honest than that. It's about shifting your entire reason for training, giving yourself permission to let go of who you were in the gym before kids, and why the example you set matters more than any number on the bar.Drew is a single dad to a five-year-old girl. John has a three-year-old daughter and an eight-month-old son. Both of them have figured some of this out the hard way.What we get into:How both of their relationships with fitness completely changed after having kids — and why that's actually a good thing.Why Drew stopped caring about PRs and started doing yoga in the garage with his daughter.John's 12-and-a-half-year streak of daily pushups, the Hugh Jackman Wolverine program, and what 11 days of keto on coconut oil actually feels like.The girl dad angle — setting the standard for the type of person your daughter grows up to value, and why that starts now.Why being "there" after a brutal training session isn't the same as being present.Facing your own mortality when you become a parent — and why that's less dark than it sounds.The stroller as a training tool, hiking in dresses, and using your kid as a weight because she thinks it's hilarious.The Open by Andre Agassi, early sports specialization, and why making fitness fun early beats everything else.Mentioned in this episode:Mass Hysteria by Michael Blevins — All In Performance, required reading for girl dadsThe Open by Andre Agassi — John's current read, highly recommendedLong and Strong — the Mops and Moes training program on Train HeroicPhil Collins, Tarzan, Brother Bear, Robin Hood, The Wild Robot — Drew has opinionsWant a program that fits real life — not a perfect schedule that doesn't exist?The Mops and Moes bundle on Train Heroic is built for people with actual constraints. Flexible, auto-regulated, and designed to keep you moving no matter what the week throws at you. → Get access here!Views expressed are those of the speakers and do not represent any official organization.

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at ⁠https://teamworks.com/⁠We are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at ⁠https://account.trainheroic.com/create-account⁠MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Cognitive Performance vs. Mental Skills Training — Are We Getting It Wrong?This week Alex and Drew sit down with Kat, a cognitive performance specialist, to ask a question that sounds simple but isn't: are we actually training cognition — or just calling things cognitive training?The answer, it turns out, is mostly the latter. We're buying expensive tech, running chess drills, and staring at doorknobs. And almost none of it transfers to performance when it actually counts.This one gets into near vs. far transfer, why brain training apps don't work the way we think they do, what orbital warfare has to do with any of this, and why expertise might be the best fatigue management tool we have.If you work in human performance, coach athletes, or just want to understand why the thing you're doing might not be doing what you think it's doing — this episode is for you.Mentioned in this episode:Chase & Simon (1973) — the foundational chess study on expert vs. novice memoryNASA Task Load Index (TLX) — search it, bookmark it, the website is genuinely excellentThe Tyranny of Metrics by Jerry Mueller — referenced again, still relevant, still not on the podcastThink and Fight Drills / Maneuver Chess — US Naval Institute Press, Marine Corps Times, War on the RocksCognitive Performance Training Level One — listed as a resource on the H2F mental domain page, worth reading criticallyNeuroTracker — they're welcome to come on and make their caseReady to train with a program that actually makes sense?The Mops and Moes bundle on Train Heroic — built around real principles, not gimmicks. Get access here!Views expressed are those of the speakers and do not represent any official organization.

MOPs & MOEs is proudly sponsored by Teamworks — the performance operations platform trusted by elite military units and professional sports organizations worldwide. Teamworks brings your scheduling, communications, athlete monitoring, and readiness data into one unified system — so your leaders stay informed, your people stay connected, and your unit stays ready. No more scattered spreadsheets or missed messages. Just one platform built for organizations where performance is the mission. Learn more at https://teamworks.com/We are also supported by TrainHeroic — the coaching and programming platform built for strength and conditioning coaches who train serious athletes. Whether you're programming for a military unit, a tactical team, or individual athletes, TrainHeroic gives you the tools to build and deliver professional training programs, track athlete progress, and communicate directly with your people — all through one app. Your athletes get world-class programming on their phone; you get the visibility to actually coach them. Start your free trial at https://account.trainheroic.com/create-accountMOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.The New Army Combat Field Test — What It Is, What It Isn't, and What We Actually Think About ItThis week Drew and Alex break down the new Combat Field Test — the Army's second mandatory fitness assessment for combat arms soldiers that nobody asked for and everybody has opinions about.Spoiler: the bar might be set so low that it barely changes anything. But the conversation around why that keeps happening is worth having.What we get into:What the CFT actually is — seven events, one running clock, pass or fail. And why if you can run a 10-minute mile you're probably fine.Why the EIB standard is the same test, with body armor and a helmet, three minutes faster — and what that says about how easy the CFT bar actually is.The Pygmalion effect — every time the Army lowered the standard, it told the force exactly what it thought they were capable of.Why one soldier who enlisted in 2019 and served until 2023 only took one PT test in four years of infantry service. Because we kept changing things.The medic problem — combat is literally in their MOS name, and they're not on the list.Why 13 Bravo cannon crew members aren't considered combat arms but Army divers are, and what that says about how this list gets built.The case for publishing MOS-level fitness score averages so soldiers can see where they actually stand relative to their peers.Whether having the Pentagon direct fitness culture is a good thing — and why for some services it might be the only thing that's actually moved the needle.Mentioned in this episode:Secretary Hegseth's September 2025 memo — Military Fitness Standards for the Department of WarACRT — the earlier Army competitor to the ACFT that never made itDA Pam 611-21 — Military Occupational Classification and Structure, Alex knew the number off the top of his headAFT Insight — aftinsight.com, free AFT score interpreter and training programsBUSAR — their proposed fitness test that requires a picnic table and holds up surprisingly wellTyler Vargas Andrews — Whistling Death on Instagram, lost an arm and a leg at Abbey Gate and still goes harder than most people with all four limbsMelissa Stockwell — Paralympic triathlete, showed up to a climbing gym without a prosthetic and just climbed anywayThe Tyranny of Metrics by Jerry Mueller — referenced again, Jerry please come on the podcast

Hey everyone, Alex here

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Drew was recently to challenged to share how the process of creating this podcast has changed his coaching. We've learned a lot along the way, some which has reinforced what we already believed, but we've also changed our minds on plenty of things. On this week's episode we dive into what we've learned from all our guests and conversations. Just like our classic closing question, we frame it in terms of what we've changed our mind on and what we've doubled down on. Whether it's our own training, our approach to coaching others, or even how we view "experts," we take a look at how putting these conversations together have helped us understand this space a little better.Thanks to everyone who has joined us on this journey, especially those of you who have shared your thoughts, insights, recommendations, and more!

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week's episode is a little in house debate. We got the MOPs & MOEs team together to hash out a sticky topic: Is walking exercise?We certainly get into more nuance on than that. Is walking EFFICIENT exercise? WHO is walking appropriate exercise for? What does research say about various walking PROTOCOLS? There are layers to this conversation, and we peel several of those layers back.If you want to dive into some of the research we mention that assessed Interval Walk Training, here are a few of the papers you can look up:Nemoto et al, 2007Nose et al, 2009Morikawa et al, 2011Karstoft et al, 2013Jakobsen et al, 2016Kitajima et al, 2023

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.In this week's episode we deep dive the fitness test used by Backcountry Search and Rescue (BUSAR). They are a non-profit 501(c)(3) organization comprised of members with extensive backcountry and outdoor experience, as well as specialized professional expertise from the NPS, Army, Navy, Marines and Air Force. Members also have experience in aviation, tracking, survival, firefighting, law enforcement, and medical professions.Their unique fitness test consists of the following: Burpee pack pullups - as many reps as you can do in 10 minutes with your SAR pack (~ 20 lbs.) - minimum 50225 pound trap bar deadlift - as many reps as you can do with good form in 1 minute - minimum 153 mile pack test with 45 pounds in under 45 minutes (USFS pack test), followed by additional 30 minutes up and over a picnic table with your SAR Pack (~ 20 lbs.) and a 45# kettlebell or dumbbellAndrew Herrington grew up a free range kid, shaped by Boy Scouts, wilderness, and hard less. A near-fatal rescue at 17 locked his purpose in for good. With nearly two decades in Smokies Search & Resuce, he became a master of survival. His specialties include land navigation, tracking, swiftwater, hunting, trapping, and wildland fire operations. A former backcountry ranger, hog hunter, and wildland firefighter, he blends instinct with experience. As founder and leader of the team, he trains others to think, adapt, and move. Known as an empathetic problem solver, he builds teams under pressure. He doesn't just solve rescues, he forges heroes.Greg Grieco learned early that standing still gets people hurt. As a former wildlife ranger in the Smokies, he tracked and trapped conflict bears relying on his photographic memory and encyclopedic mind to outsmart the beasts before they could cause more problems. A university of Tennessee football alum, Greg brings power, speed, and fearless momentum to every mission. He specializes in land navigation, rapid decision making, and reading terrain like an open book. When hesitation creeps in, he pushes that team forward.

Mistral has been on an absolute tear - with frequent successful model launches it is easy to forget that they raised the largest European AI round in history last year. We were long overdue for a Mistral episode, and we were very fortunate to work with Sophia and Howard to catch up with Pavan (Voxtral lead) and Guillaume (Chief Scientist, Co-founder) on the occasion of this week's Voxtral TTS launch:Mistral can't directly say it, but the benchmarks do imply, that this is basically an open-weights ElevenLabs-level TTS model (Technically, it is a 4B Ministral based multilingual low-latency TTS open weights model that has a 68.4% win rate vs ElevenLabs Flash v2.5). The contributions are not just in the open weights but also in open research: We also spend a decent amount of the pod talking about their architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens (typically only applied in the Image Generation space, as seen in the Flow Matching NeurIPS workshop from the principal authors that we reference in the pod).You can catch up on the paper here and the full episode is live on youtube!Timestamps00:00 Welcome and Guests00:22 Announcing Voxtral TTS01:41 Architecture and Codec02:53 Understanding vs Generation05:39 Flow Matching for Audio07:27 Real Time Voice Agents13:40 Efficiency and Model Strategy14:53 Voice Agents Vision17:56 Enterprise Deployment and Privacy23:39 Fine Tuning and Personalization25:22 Enterprise Voice Personalization26:09 Long-Form Speech Models26:58 Real-Time Encoder Advances27:45 Scaling Context for TTS28:53 What Makes Small Models30:37 Merging Modalities Tradeoffs33:05 Open Source Mission35:51 Lean and Formal Proofs38:40 Reasoning Transfer and Agents40:25 Next Frontiers in Training42:20 Hiring and AI for Science44:19 Forward Deployed Engineering46:22 Customer Feedback Loop48:29 Wrap Up and ThanksTranscriptswyx: Okay, welcome to Latent Space. We're here in the studio with our gues co-host Vibh u. Welcome. Thanks. Excited for this one as well as Guillaume and Pavan from Mistral. Welcome. Excited to be here.Guillaume: Thank you.swyx: Pavan, you are leading audio research at Mistral and Guillaume, you're Chief Scientist,Announcing Voxtral TTSswyxHost(00:05) Okay. (00:05) Welcome to Lean Space. (00:06) We're here in the studio with trustee co-hosts, Vibhu. (00:09) Welcome.VibhuHost(00:11) Very excited for this one.swyxHost(00:12) As well as Guillaume and Pavan from Mistral. (00:15) Welcome. (00:16) Excited to be here. (00:17) Thank you for having us.(00:18) Pavan, you are leading audio research at Mistral and Guillaume, you're a chief scientist. (00:23) What are we announcing today where we're coordinating this release with you guys?GuillaumeGuest(00:26) Yeah, so we are releasing Voxtral TTS. So it's our first audio model that generates speech. It's not our first audio model. We had a couple of releases before.(00:35) We had one in the summer that was Voxtral, our first audio model, but it was like a transcription model, ASR. Like a few months later, we released some update on top of this, supporting more languages. Also a lot of table stack features for our customers, context biasing, precision, timestamping and transcription. We also have some real-time model that can transcribe not just at the end of the level.(00:56) You don't need to fill your entire audio file, but that can also come in real-time. And here, this is a natural extension in the audio, so basically speech generation. So yeah, so we support nine languages, and this is a pretty small model, 3D model, so very fast, and also state of the art. Performed at the same level as the base model, but it's much more efficient in terms of cost, and also much, in terms of cost, it's also much cheaper, only a fraction of the cost of our competitors.(01:22) And we are also releasing the work that this model is running.swyx What's the decision factor?Guillaume It's a good question.swyxThere will be more. Yeah, Pavan, any sort of research notes to add on?Architecture and CodecPavan: But it's a novel architecture that we develop inhouse.We traded on several internal architectures and ended up with a auto aggressive flow matching architecture. And also have a new in-house neural audio codec. Which, converts this audio into all point by herds latent [00:02:00] tokens, semantic and acoustic tokens. And yeah, that's that's their new part about this model and we're pretty excited that it's, it came out with such good quality and Jim was mentioning. Yeah, it's a three B model. It's based off of the TAL model that we actually released just a few months back and insert trunk and mainly meant for like the TTS stuff, but they need text capabilities are also there. Yeah.swyx: So there's a lot to cover.I always I love any, anything to do with novel encodings and all those things because I think that's obviously I creates a lot of efficiency, but also maybe bugs that sometimes happen. You were previously a Gemini and you worked on post training for language models, and maybe a lot of people will have less experience with audio models just in general compared to pure language.What did you find that you have to revisit from scratch as you joined this trial and started doing this? At leastUnderstanding vs GenerationPavan: when it comes to, for, I think the, there are two buckets, I guess the audio understanding and audio [00:03:00] generation. The audio understanding, like the walkthrough models that Kim was mentioning that we released earlier.The walkthrough chat that we released I think July last year, and the follow up transcription only, models family that we released in January, that would be one bucket, and the generation is another bucket. I think. You can also treat them as a unified set of models, but currently the approaches are a little different between these two.To your question on how audio is fed to the model? In the understanding model, it's very similar to actually Pixar models that we also released,swyx: yes.Pavan: That'sswyx: amazing.Pavan: It was pretty, I, that was the first project I worked on after joined Misra. It was pretty, pretty nice. And Wtu was very similar in spirit.I guess So we feed audio through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer decoded transformer model. Yeah. On the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of mode.I [00:04:00] guess the interesting part of what the generation stuff is, the output now has to produce audio and. The approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing attrition and a lot of models which are based off of this kind of approach.And we took a slightly. A different, design decisions around this. But at the end of the day, the neural audio product converts audio into a 12.5 herdz set of latents. And each latent is, has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side.There's several ways to use this at each frame, but we just sum the embedding. So it's like having key different vocabularies. Combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part on the output side, the, it's not the, I don't know if it's the most popular, but one.Popular technique is to have a depth transformer [00:05:00] because you have K tokens at each time step, like with a text, you just have one token at each time step. So you just do predict the token from the vocabulary with, yeah, with just, you get probabilityswyx: This's a very straightforward text. VeryPavan: straightforward.swyx: Yeah.Pavan: But if you have K tokens, then the name thing would be to predict all of them in paddle. That doesn't work. At least that doesn't work that well because audio has more entropy. And the, one of the techniques people use is this depth transformer where you you almost have a small transformer, or it can be L-S-T-M-R in as well, but people use transformers and you predict the K tokens in auto aggressive fashion in that.So you have two auto reive things going on.Flow Matching for AudioPavan: So the thing we did differently is in, instead of having this auto aggressive K step prediction, we have a flow matching model. Instead of modeling this as a discrete token set we trained the codec to be both discrete and continuous to have this flexibility.So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better. So yeah, we took this flow matching, so the, it's a flow [00:06:00] matching head, which takes the latent from the main transformer and like kind in fusion, it's denoising, but in this flow matching itself, velocity estimate.So you go from this noise t all the way to there. Audio latent, which corresponds to the 80 millisecond audio and then, which is sent through the work order to get back the 80 millisecond audio frame.swyx: Yeah. Is this the first application of flow matching in audio? Because usually I come across this in the image.Pavan: Yeah. Actually, in some sense there are models flow matching models in audio, but I think this specific combination I could be wrong. There could be somewhat. No. I haven't seen. I haven't seen much work in this, so I think it's novel and a lot of it's just a way bigger community, so they, I think they pioneer a lot of these diffusion flow matching work, and it's interesting to adopt some of the ideas there into audio and,swyx: yeah.Pavan: Yeah, I'm, personally that's the think part which is trying out about. One of more meta point is unlike text, even in vision, I think this is true, but in [00:07:00] audio step literature that there is no.Winner model, yet there is no, okay, this is the way you do things. It's it's still by, I think people are still iterating and figuring out like what's the best overall recipe. I guess the idea. Pretty sure there are models which are also completely end-to-end, like NATO audio. NATO audio, but it's still not come to a convergence point where this, the right way to think that.That also makes. A space pretty exciting to explore.Real Time Voice AgentsVibhu: What are some of the ways to look at it?Vibhu: There are ways where you can do diffusion for audio generation, but if you want like real time generation, that's a big thing with the approach I'm assuming that you took. Yeah. And also like how do you go about evaluating different axes of what you care about, yeah,Pavan: good point. I think we so you can do just flow matching diffusion for the whole audio. We didn't even go down that path because one of the main applications is voice agents and we want real time streaming, and that's the use case. That's not the only use case, but that's one of the primary use cases we want to get to.So we [00:08:00] picked the auto aggressive approach for that. And within the auto aggressive space, again, you can do chunk by chunk or you can do so we picked the. I think at least personally prefer the operations, which are the simplest, and so we try to see, can we just add audio as just another head to our regular transformer decode model because that kind of makes it easier for eventual end-to-end modeling of audio text native modeling.Yeah. And it works pretty well. So I guess we went with that and we tried a little bit, but the flow matching head itself, like we had a discreet. Diffusion kind of approach, which also works well, but the flow matching work better.swyx: I was just curious about how you also think about this overall direction of research.Do you basically, when you work with the audio team, do you set some high level parameters and then let them explore whatever, or how does it work between you guys?Guillaume: No I think the way it works is that we are the, we are prioritizing together, I think, what are the most important features because there are many things we can do [00:09:00] in audio.Yeah, I think we try to. These are like how we should do things, for instance. Ultimately what we want to do is to build this through duplex model, but we are not going to start this start there directly, I think is. Some of the project people are doing, butswyx: just to confirm, full effects means it can speak while I'm speaking or,Guillaume: yeah.Okay. Audio. Yeah. Yeah. So intimately we're going to get there, but for us it was, we decided to take it like a step by step. So we start with whatever is the most important. I think support customers, which is the transcription is the most popular use case. Then the speech generation, Soviet time, just a bit before that.And then actually to be like more, but try combining everything all together. But but yeah, we thought it was also important to like separate things and optimize each capability one by one before weswyx: measure of that together. And the super omni model. ButGuillaume: very interesting because as Par said, it's when you work on some other domains of this airline and everything, there are many areas where I think it's not as interesting.For instance. Many places, it's essentially just around data or like creating new environments on a lot of kind [00:10:00] of easy things. But things were, I think the research is maybe not as interesting. Were in audio. There are so many ways to actually build this model. So many ways to go around it. That's the sense I think is really interesting.And what we also tried for speed generation is that we tried multiple approaches. What was interesting that even though they were extremely different, they under the big know the particles but the for matching turned out to be quite more natural. So we are happy with this.swyx: Is there intuition why it maybe like flow matching is just models speech better in some natural fundamental, latent dimension?Pavan: No, I think the main thing is e even at a particular time step, there is a distribution of things.swyx: Yes.Pavan: To be predicted like the way you inflate. So you already know the word that you're speaking and Yeah. The intake space, let's say the word maps register a single token for simplicity.In most cases it does. So there is not a lot of so you just pick the word, but with within audio, even the same word could, even with your own voice, could be inflicted in so many different ways. And I think [00:11:00] any approach which like models this distribution and. And flow matching is one, one of the take.It's not the only one at all, but it's a one which works pretty reasonably well. I think that's better. So you have to pick across several different, the intuition I have is it's, there are some, several different clusters each corresponding to some specific way you would inflict, pronounce that thing.And you can't predict the mean of it because that corresponds to some blurred out speech or something like that. But you have to pick one. And then like sharpswyx: conditional inference.Pavan: Yeah, exactly.swyx: Is that all covered under disfluencies, which is I think the normal term of art. Pauses intonations. By the way, I have to thank Sophia for setting all this up, including like some of these really good notes becausePavan: Yeah.swyx: I'm less familiar with the audios for me.Pavan: No. I think dis dismisses are definitely one such Eno defenses is more likeswyx: which is arms are.Pavan: Yeah, arms. And also repeat like you like,swyx: yeah.Pavan: You do this full of words, your thinking, so you repeat the word.swyx: Okay. Whereas intonation is like a diff, it's up up [00:12:00] speak and all this.Okay.Pavan: Yeah. So I think there is a lot of like entropy. And modeling it as a distribution. And a, any technique which helps with it and the depth transformer is a conditional way of modeling this. And Transformers actually really good at it, even though that's a mini transformers. So I think that worked pretty well too for us too.It's just that the main concentration is when you have a depth transformer. If you have K tokens, you need to do K auto steps, right? Even though it's a small thing, it's K steps, which is very vacant, say heavy, but flow matching. We were able to cut it down significantly. So we are able to do the inference in quad steps or 16 steps and it works pretty well.And there are more normal techniques to bring it down even further to like, in extreme case, one step like we're not doing it yet, but it at least the framework, LEDs itself to more efficient and Yes.swyx: And the image guys have done.Pavan: Yeah.swyx: Incredible work guys. Yeah.Pavan: It now you just. Send a prompt and you get an image.swyx: Yeah. Surprisingly not enough. I think image model labs use those techniques in production. I think it's, I feel like it's a lot of research demos, but [00:13:00] nothing I can use on my phone today.Guillaume: The thing, there's a thing that would be interesting here is that since, indeed I've been so much sure that has been done in the vision community compared to radio dys, stomach, I think there are so many long infra Yeah.And there are so many things we can do to actually improve this further. So it's our first version, but we have so many ways to exist, much better and much more efficient, cost efficient, soswyx: yeah.Guillaume: So really it's not a new field at all, of course, but there are still so many things that can be done.Perfect. It'sswyx: nice. I should also mention for those who are newer to flow matching, I think the creator, this guy's name is Alex, he's done I think in Europe's maybe two Europes as ago. There was, there's a very good workshop. There's one hour on like this matching is I would recommend people look that up.That's the other thing, right?Efficiency and Model Strategyswyx: The efficiency wise, like I, I imagine like the reason is open weights the reason you pick 3.6 B backbone it you are 3.4 B you are, try to fit to some kinda hardware constraints. You kinda fits some kinda basic constraints. What are they?Guillaume: Not necessarily, I think something we care about in our model that they're efficient.So we have a [00:14:00] lot of separate model, for instance. So we have this that is very small, very efficient. We also have a small OCR model that is available. Good, highly efficient as well. And I think on a project maybe there, I think companies are going to take is to have a coverage general model that will do a bit of everything.But that is also going to be expensive. On here. What want say is if you care about this specific use case, if you can actually use this model, it just does that. It's extremely good at it. Survey, very efficient. That's why we can actually add. We do, but also OCR that are like really good at that.And that would be much more cost effective factors and the general model that will contain a lot of capabilities you don't really need. So yeah. So we're doing like general model, but also like more customized model. This,Open Weights and BenchmarksVibhu: how does it compare to other TTS models? It's, we are going follow open wave.We're just dropping it. I think it's pretty good.Pavan: Yeah, I think it's pretty good. Like it, it's definitely one of the best. For sure. It's probably I would say it's the best open source model, butVibhu: decipher themselves.swyx: Yeah.Voice Agents VisionVibhu: Why now? How does it fit into broader ral vision? How do you see voice agents?How do you see voice? I think every year I've heard, okay, you're a [00:15:00] voice. You're a voice. There's a lot of architectural stuff. There's a lot of end time that see it, your solving, but where do you see voice setting?Guillaume: We had so many customers asking for voice. That's also why we wanted to build it.What's interesting in this domain is that. In a sense, if you take something simple like transcription it doesn't seem like something that should be very hard to do for a model. It's essentially, it's pattern recognition. It's classification on this. Models are very good at classifying, right?Or nonetheless, when you talk to them it's not there yet, right? It's not, you don't talk to them the same way you talk to a person. On something, maybe people don't realize it. It's in English it's still much better than in any user language, even compared to French instance. If you talk to this million in French, when you see people talking to this they'll talk very slow.They'll articulate as much as they can. So it's not natural, right? We're not yet to this. And I think, yeah, maybe the next generation will not know this, but yeah, I think people that. But our edge will actually always keep this bias speaking very slowly when they talk to this model. Even if maybe, probably in a couple of years, maybe next year it'll not be necessary anymore.But yeah. But what's interesting is to see that yeah, even for like languages [00:16:00] like yeah, French and Spanish Germans that are not no, no resource on religion. You have a lot of audios there on still it's not as good. And I think a consequence. Because then for this, I suppose just is not as much energy, as much effort that has been put done in some other mod that for some vision or like coding.But but yeah, there's still a lot of progress to be done. I think it's just a question of doing the work and it's clear path I think to get there.Pavan: It's a little fascinating because I worked on Google Assistant I think while back at this point, but it's, I think it's, it like when you take a step back, it's fascinating.It's not that long ago. It was like four years ago or five years ago, and it's now it's completely audio in, audio out and the function calling and the whole thing happens completely end to end. And in a very natural,swyx: yeah,Pavan: natural way and still ways to go. Kim was telling, even despite all the previous, it's not like you're speaking to a person.When you talk to any of these agents, bots, or voice mode kind of situation, it's still like a gap. I think that's the great part and I feel like with even the existing [00:17:00] stack, we should be able to get to this very natural speech conversational abilities soon enough I guess.And we'll also hope. I get thatGuillaume: on this kind of the next step, right? Because when you talk to these agents, like usually people are just writing to them and sometimes they'll this very clear, for instance, you are, you want to write code, but you are, you have a very clear idea of how you want the model to implement what you in mind.But so here you are able to spend a lot of time writing. So it's not really efficient on audio is really like a natural interface that is just not there yet, but I think it's just gonna be the place.Vibhu: How's it like building, serving, inferencing, like we see a lot about, it's very easy to take LMS off the shelf, serve them.Fine tuning, deploying. I know you guys have a whole you have Ford, you have a whole stack of customizing, deploying. Is there a lag in getting that. Like distribution channel. Are you helping? There is. So like prompting, lms, you can have them be concise, verbose, all that.They're built on LM backbones, these models. How do you see all that?Enterprise Deployment and PrivacyGuillaume: Yeah, I think this is a lot of what we're doing with our own customers. Very [00:18:00] often they come to us, so it's for different reasons. I think one reason is sometimes they have this lot of privacy concerns.They have this data that it's very sensitive. They don't want data to leave. The companies, they wanted to stay. Inside the company. So we have them deploy model in-house. So either on a, either on premise or on private cloud. So they're not worried that it's given to a third party on the there some leakage.Sometimes they have this kind of many companies have this different, sensitivity of data they have like sometimes channel chat can send it to the cloud has to stay there. So then it creates some kind of heterogeneous workflows where it's annoying. You cannot send some data to the cloud.This one you can, so here, when we actually deploy the model for them, they don't have this consideration. They are like not worried that, this is going to leak. Everything is much easier. So we help them basically do this on the, so it's one of the very proposition. But but the other is very often, when customers use this off the shelf close model, but very sad is that they are not leveraging, these data that have been collecting for four years or something for decades.So much data. Sometimes it's trillions of tokens of [00:19:00] data in a very specific domain. Their domain, which is data that you'll not find in the public, on the public internet. So data on which, like close model, we actually not have access to one, which that's going to be really good. So if they're using like closed source models are basically not benefiting from all these insights.All these data they have collected three years, they can always give it into the context that in France, but is never as good as if you actually train the modern analysis. So yes, that's basically what we help them to do. We actually provide them some purchase, basically what we announced at GTC this week.So we provide them with this, it's basically like a platform with a lot of tools to actually help them process data. Trained on that. Yeah, it's actually the same thing that we're using in the science team. So it's actually very better tested infrastructure, like a lot of efficient training cut base.For a quality pre-training like a fine tuning, even doing S-F-T-I-L. So we help them do this using the same tools as what our science team is building is using. So since it's tools that we've been using for two years now, it's really better tested. It's really sophisticated.So it's the same thing. We are giving to them, giving the company the same thing [00:20:00] that what are same still using internally actually build their own ai and it makes a really big difference. I think sometimes customers. And many in general don't realize how much better the model becomes when you fine tune it on your own data.And you can have a, your model is here. You start from there. You have a cross source model, which is sort here, but if you actually fine tune it can actually really go much further than this. And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything.You don't have to feed like 10 K tokens of contact at every query. So it's it's much easier. It's a bit, I think using a closed source model is really sad because it basically puts. You are not leveraging all this data and you are going to be using the same model as all your old competitors when you're actually using, everything you have been collected for years, which is really valuable.So yeah. So we help basically customers do this. We have a lot of solution I mean deployed for engineers that go in the company that basically look at the problem customers are facing to look at what they're struggling to do what we should do to solve it. So we help them solve them together.So it's I think our approach is a bit different, but here. [00:21:00] Some of their companies and competitors, it's, we don't just release an endpoint on sale, do some stuff on top of that, or we don't just give a checkpoint. We really look very closely with customers. We look at the issues they have, we had them solve them.We really make some tailored solution for the client are facing. Some example are also going to be, sometime we have some customers. They really wanted to have a really good model, really performance on some, like Asian languages on the, if you take some of the shelf models, they can speak it, they can write in this language, but it's not amazing.This language would be like maybe zero 1% of the mixture. So it has been included during training, but very little. So what we did here is upgrade. We trained a new model for them, but so this language was 50% of the mix, so it's much, much stronger. It knows of the dialects, it knows the, so it's yeah.So it's some example of things we can do and it's really arbitrary, custom. I think you had some of their customers, for instance, they wanted some. They wanted some 3D model that can do audio with a very good function cable. So something you wanted to put in the car in particular, they wanted this to be offline because in a car you don't necessarily have access to internet.So [00:22:00] yeah. So here we can actually build the solutions. There is no like model out of the box on this. In the internet you have this very, you have this very general model generalist, like he's strong model. But for things like this, they always want at specific solutions and on some other reasons.Sometimes they come to us is because, like they, they experiment with some closed source model. They get some prototype. They're happy with what they build. They, it works well. They're happy with the performance, and then they want to go to production and then they analyze. But it's extremely expensive.You cannot push this. It's so then they come back to us on this. They can help us build the same thing as this, but using something much cheaper on here. And here we can sometime be something 10 x cheaper by just functioning a model and it'll be better OnPrem on their old server and also much cheaper as well.So yeah,swyx: that's the drop pitch right there. Take all themoney.Vibhu: And outside of that you do, we do put open wave models so people can do this themselves. I feel like not enough people go outta their way.swyx: They're not going to, they're gonna ask them to do it as the expert. IGuillaume: think initially we didn't know, [00:23:00] we wanted completely short at the beginning of the company because, I think our study was not exactly the same as what it is today, but what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge on the, and it was, yeah, on, we were seeing customers struggling with this, but it was even, that was three years ago and no, things are much more complicated because now you don't just have, text on SFT on a simple instruction following.You have reasoning like your agents, you have like tools. You have a multimodal audio, so it's much more complicated than before. And even back then it was hard for customers. So they really need, have some support and this is why actually providing like always some four D position as well. The processFine Tuning and Personalizationswyx: I'm curious is there also voice fine tuning that people do?Pavan: So in this forge we also have a say unified framework. And the hope is like the er speech to text that we released earlier this year. And even the ER chart that we released last year. And I think a big people, I think there's a big, rich ecosystem [00:24:00] of people fine tuning whisper, and people want the same thing with w so it's much stronger than Whisper.And yeah, the the platform offers that kind of fine tuning yeah, which could be any kind of fine tuning. Like for instance, even sometimes people want to support new languages to this, which are tail languages, which we hope to cover. Certain natively, but if there is a language where you data and you want to frank you, I think this is a good use case.Or the other use cases, you, it's the same language, like even English but it's in a very domain specific way.swyx: Yeah. Terminology, jargon, medical stuff.Pavan: Exactly. And also there's specific acoustic conditions like there's a lot of noise or the, and. The model will do decently in most conditions, but you can always make it better.And that those are some of the use cases where you can improve it e even further. And that's one good use case for this and for text to speech. We're just releasing it so we'll have support for that soon too. I think it's similar use case.Voice Personalization Pavan: It's little different the kind of things that you want to extend a [00:25:00] text to speech model to, which could be like voice personalization, voice adaptation for enterprises.Many enterprises need very specific kind of tone, very specific kind of like personality for this kind of voice. And all of those are like good use cases for fine tuning.swyx: This one I was gonna ask you, we never talked about cloning voice clothing here. How important is it, right?Like I can clone a famous person's voice. Okay. ButPavan: the main use case would be like for enterprise personalization, like enterprises need like a lot of customization. You don't want the same. Voice for all the enterprises. Each enterprise want a customized, specialized something which is representative both their brand and also their, I guess safety considerations and the use case I think the kind of thing that you would deploy as a empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot and would be different from like more conversational aspects.I think those are the. [00:26:00] Customizations you would expect from enterprise. And that's the main use case, at least from our side.Vibhu: My, my basic example is you don't want to call to customer services and have the same exact voice. It's just, it's gonna be weird.Long-Form Speech ModelsLong-Form Speech ModelsVibhu: But also on the technical side of this, so there's like a few things in TRO that I thought were pretty interesting.He's a big fan of this paper. Oh, he said very good paper. He said this is the best SR paper he's ever read. Yeah. I've hyped up this voice paper enough. We covered it. Somewhere, but a big thing. So Whisper is known for 32nd generation a 32nd processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done.Even little niches of how the padding is. So it's very much needed. You need to have that padding in there, the synthetic data generation around this. I'm wondering if you can share the same about the new speech to text, right? Text to speech. So how do you. How do you generate long form, coherent?How do you generate, how do you do that? And then any gems? Is there gonna be a paper?Pavan: Yeah. Yeah. They would be a technical report. Okay. Yeah. I think I could have a lot of details.Real-Time Encoder AdvancesPavan: But me I think the [00:27:00] summary of it, actually, some of the considerations in this paper were, because we started with the wipa encoder as the starting point, and now we have in-house encoders, like the bigger time model, for instance, which we released in January.Also release a technical report for that real time model as well, which is this dual stream architecture. It's an interesting architecture. You should check it out. And there we have a causal encoder and I don't think there's any strong, multilingual causal encoder out in the community. So we thought it's a good contribution.So that's one nice encoder there. Other people want to adapt. That's a good end code. And we train it from scratch. I think her. Post stack is now mature enough that we are able to train super strong ENC codes. And some of these considerations, like spatting and stuff, is a function of the Whisper ENC code.And now that we train encoders, inhouse the design concentrations are different.Scaling Context for TTSPavan: And for the question on text to speech, I think that's also leans onto the original auto aggressive decoder backbone. I think, it says very, almost identical considerations. I think the long context in it's not even long con, [00:28:00] so the model processes audio at 12.5 herds, so one second maps to like 12.5 tokens.So I think one minute is like 7.8 tokens. You can get like up to 10 minutes in eight K context window and get half an hour and 30 K context window. So that's and 30 2K context is something that's we are very comfortable training on. We can extend it even much longer. 1 48 K. Okay. You can naturally see how it can extend to even our long generations.Yeah. We need the. Like data recipe and the whole algorithm to work coherently enough through such long context. But the techniques are some way very similar to the text, long context modeling. And the key differences, it's just doing flow matching order regressively instead of a text open prediction.swyx: Okay. I think that was most, most of the sort of voice questions that we had. ButWhat Makes a Model SmallVibhu: I have a big question on Mr. Al, Mr. Small. So what is small? How do we define [00:29:00] small? What is this? What is this? I remember the days of Misal seven B on my laptop. The snuff fitting on my laptop. I could run it on the big laptop, butGuillaume: it's just additional.Question of terminology, like here what we did, baseball is north active parameters, but it's true. Really not give it another name, but yeah, we could have called it medium, but only, I,I suppose it's a model that we released mixture of experts. It's a model that combines different model before which we were doing the same, is that we had one model, general model for Israel. Doing instruction following, were like a separate model that was Devrel trial. So qu coding specify specific to code with another model for Reason Maal.So this were separate artifacts built by different team at trial on what we're doing is basically merging all of this. It was, you had pixel trial was the first vision model. We was like a separate model on the way we do things internally is that we have one team focus on one capability, build one model.On the means mature, mature enough, we decide to merge this into the [00:30:00] matrix. But here it was the first time we basically match all of this into one. But there are some other things we did at first time to merge time, for instance, like more capabilities or function coding I think would be, are, it's going to be much, much better in this trial, small platform.But but yeah, so it's our latest model on the working is,Vibhu: and yeah, key things is it's very sparse. Six, be active pretty efficient to serve. 2 56 K context. Yeah,Merging Capabilities vs Specialistsswyx: I think what's interesting is just this general theory of developing individual capabilities in different teams and then merging them.Where is this going gonna end up?Vibhu: Like we've seen the five things put together in this. Yeah. What are the next five teams?swyx: I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model. This was what they were selling. All modalities and all modalities out.But I feel like you might do it.Guillaume: I think there's some mod where it's not competitive use, for instance for audio. For audio here, if you want to do transcription, I think it makes no sense to use a model. If you just want to trans tech it, it'll be very inefficient. If you want to do audio, you probably just want to be the [00:31:00] one VR 3D model performance essentiallyswyx: the same.It's going to be incredibly cheaper. So here, that's why we wantGuillaume: to have a separate but just does this. Yeah, I think the question is just, yeah. If you are to, to your model. By speech and you asking like a very complex questions on how you do this on the, just to cascade things. Do you want to put a d in a model that has like a one key around it?It's like a, not a competitive discussion, I think unaware if you doing into the direction, but that's possible. Of course. But yeah. But I think for us, the next capabilities we want to try to integrate into these models when we are going to be yes, like marketing or no reasoning better, I think more capabilities that people don't talk too much about, but at high bottom, I think for our customers in our, on different industries, for instance, things are around like a legal computer.I design all these things that is this males out of the box are to put at that. Because people, if you don't prioritize this, there is not like too benchmark on that. Butswyx: this done how toGuillaume: make this good and this just start to do the work. Extracting some that processing it [00:32:00] expression. So yeah.But we are offering the imagine to this.swyx: I think for voice. Yeah. The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right? Which people don't understand spatial audio because like most TTS is just oh, I'm speaking to a microphone in perfect studio quality.But when you have video, like the voice moves around.Pavan: That's true. The constitution was a little different in the sense that there it's like a a standalone artifact where you get the whole thing and you consume it. But in a conversational setting, it's a, you need the extreme low latency.swyx: Yeah,Pavan: streaming would be one of the primary concentrations.swyx: You can build a giant company just doing that, right? So you don't need to do the voice, but I was just know on the theme of merging modalities, that is something I, I am like, wow. Like I didn't, everyone up till, let's say mid last year was just doing these like pipelines of okay, we'll stitch a TTS model with a voice thing and a lip sync [00:33:00] thing and what have you.Nope. Just giant model. Yeah.Open Source MissionVibhu: I have a two part question. So one is, it's still open. It seems like open source is still very core to what you guys do and I just have to plug your paper. Jan 2024. This is the one trial of experts like. Very fundamental research on how to do good.Moes paper comes out very good paper for anyone. That's just side tangent. No.swyx: This thing caused, we bring back, eight by 22 was like the nuclear bomb for open source. I think it takes Shouldn be more seven B more. Yeah. Yeah. But this is a bigger opposite than me.Yeah. Yeah I don't remember this. I remember, I don't think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think. Yeah. The model was did as well.Vibhu: It's just a little update probably.swyx: Yeah. No, but you have a point to make.Vibhu: No, you gotta check that. But then, I just want to hear more broadly on open source for you guys, and when you had asked earlier [00:34:00] about what's next, what are the other, side tapes working on you. You put out Lean straw. This,swyx: it's not necessarily surprise. I was like, I don't, this doesn't fit my mental model or Misra.Guillaume: Yeah. First for open source in general, I think it's really something which looks to the January of the company. I think we started it per once, is we so we have open sourcing with, since the beginning and even before this. So before this, so me and Tim were at Meta, we released LA and I think what was really nice.To see that before this, for most researchers like universities, it was impossible to work on elements. There was no alien outside. And if you look at many of the techniques that were developed after, for instance, was open source all this post-training approaches like even DPOD, like preference optimization, all of this were done by people that had access to this portal.And it'll have been impossible to do without this. So it's really making sense, move faster. So we really want to contribute to this ecosystem. I think like the deep and also like very lot of impact. All these papers that are I think in the open source community are really helping the science community as a whole to move faster.So [00:35:00] we want contribute to this ecosystem. That's why we're releasing very detailed technical reports. So ma trial and our first reason model, and ation, lot of results, things that work, things that did not work as well. Think helpful on the, yeah, so for the audio model also to share a lot of details, share of them for real time model.And the, yeah, so we really want to continue this, basically belong to this community of people who share science. I think we really don't want to be, leading in a world where the smartest model, the best models are only behind, close doors. Only accessible to a shoe companies that we, as a power to decide we can use them on it.I think it's a scary future. We don't want to live in, we really want this model to be accessible to anyone that want. Intelligence to be used unaccessible by anyone who can use it. So yeah, so that's why we are pushing this mission and source model. Yeah. So not, so yeah, no strategy. So it's open source, not the first model, so not the best on the Yeah.Lean and Formal ProofsGuillaume: LIN trial I think is also one step into this direction. So it's yeah, a bit different than what we are usually releasing. But we have a small team internally [00:36:00] working on them. Formal proofing, formal math. So I think a subject we care about in general and we were working on reasoning. I think we started too early before doing reasoning without LMD is very hard, especially when you work with formal systems because the amount of data you have is negligible.It's addressable community of people writing like formal proofs. But the reason why we like it is because I think there is if you look at what people are doing with reasoning, is there, the problems that you can use. Are usually going to be problems where you can verify the output. So for instance, all this ai ME problem where the solution is a number between 100, like a thousand.So you can verify, compare this with a reference or it's an expression. You can actually compare the output expression generic with the reference. But there are many, most of them have problem and most of the reason problem. There is no like way to easily verify the solution. If the question is show that F is continuous, cannot compare in the reference, right?If it's a probe that this is true or probes is properties, there is no way to. You cannot act, simply verify the correctness of your proof. So it's hard to apply the, there is no referable reward here. So [00:37:00] what you could provide is of course, like a judge and judge that will look at your proof. But it's very hard and it's very, you could do certain, some reward hacking happening there.So it's difficult. You could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model says give negative reward because it's a different poop, maybe it was still digit proof, just different. So it's not going to work well. What's nice with lean and with formal probing is that you don't have to worry about this whatsoever.We just,swyx: they're all function is largely compiles in lean is functionally the same. Exactly.Guillaume: It's like a problem if it compiles it's correct. It's very easy. And you can apply this and then you can,swyx: it's just way too small. So no human will actually go and do it.Guillaume: Yeah, that's exactly.It's the only people can do it. It's like a very small committee of people doing a PhD on that. So it's super small. And it's sad because it's actually very useful on not just mat, but also in software verification. So for instance, software verification today. So tiny market. Very few industries work on this and we need that.It's usually going to be like companies like building airplanes, air robotics,swyx: likeGuillaume: things [00:38:00] where they absolutely want to be sure. Life depend on this, but it's very rare that people formally verify the correctness of their software. But I think one of the reasons for this is simply that it's just hard to do.swyx: Are you think of TLA plus? It's the language that some people do for software verification? No. That people use in a ference, but but yeah, it's the reason I think why people don't use it more and why this industry is not as big as could be is because it's very hard. But now with cutting edges that are there, it's going to be very different.Guillaume: We're going to see much more of this. So I think yes, industry there is going to be much larger in the future that we, these models. So yeah. Here also anticipating this a little bit, we wanted to work on that because it's proving like a math theory and like a, essentially the same tools.swyx: Yeah.Reasoning Transfer and Agentsswyx: One of my theories is that because the proofs takes so long, it's actually just a proxy for long horizon reasoning and coherence and planning. Maybe a lot of people will say okay, it's for people who like math. It's for being okay. It's like a niche math language. Who cares? But actually, and you use this as part of your data mixture for [00:39:00] post-training and reasoning, actually, it might spike everywhere else.Yeah. And I think that's un under explored or no one's like really put out a definitive paper on how this generalizes.Guillaume: Yeah, absolutely. AndPavan: I think evenGuillaume: that's what we're seeing already. For instance, you should do some reasoning on math as then the American should do reason even.Yeah. In the early stage. So we, the, there is some transfer, some sort of emergence that happens. And I think some, it's also interesting, it's not just I think the topic in general, but it's, there is a lot of connection with this on including agents because. Sometimes the model can see like a three that it has to prove it's very complex, but then it can take the initiative to say, I'm going to prove this three lr.I'm going to suggest three Rs, and I'm going to in parallel prove each R. So three of them in parallel with sub agents, but I'm also going to prove them in theory and the three tool so you can do this also. Pretty interesting. You can, even if you fail to put one of the LeMar, you can actually, maybe you succeed to put the normal lema too, so you get some possible reward here.So it's a bit less Spartan issue, just get to zero one for the entire thing. [00:40:00] So it's pretty interesting. I think we can actually,Vibhu: yeah, it's also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting yeah, similar score wise, you are, thirty, seventy, a hundred fifty, three hundred bucks.Smaller.swyx: I think cost is a bit unfair, right? ‘cause this one is at like inference cost. It's always there on top with their margins on top of it. But, we don't know anything else, so we gotta figure it out.Vibhu: Okay.Next Frontiers in TrainingVibhu: I did wanna actually push on that more. Not on cost, but you mentioned about, okay, it's a great way to have verifiable long context reasoning.What are other frontiers that, I'm sure you guys are working on internally, there's a lot of push of people pushing back on pre-training. Scaling, RL pushing, compute towards having more than half of your training budget. All on rl. Where are you guys seeing the frontier of research in that?Guillaume: You mean theVibhu: just in foundation model training in the next, one thing that you guys do actually is you do fundamental research from the ground up, right? So you probably have a really good look at where you can [00:41:00] forecast this out.Guillaume: Yeah. I think for us we're still working a lot on the pre-training side.I think we are very far from situational, the pre-training. I think ML four preprinting will be like big step compared to everything we have done before. So we are pretty excited about this. And I think on the other side, I think now we have more and more to think about this algorithm that will actually support this very long trajectories.I think when it was, for instance, GRPO for it doesn't really work this any bit of policy. Which was okay initially because you are solving math problem that can be solved in like a few thousand tokens. So the model can alize them pretty quickly. So when you do your update, the model is never too far off.It's never too far off. But now when you are moving towards this kind of problems where certain takes hours, like six hours to get a reward, then your model is co pick places. So you have bi new infrastructure that supports this, but also new A, so now everything we're doing internally, we're trying to. Build some infra that we actually anticipate is what we have in six months, one now, which is this extremely no scenarios on the, I think when we started Missal, part of me and [00:42:00] we wanted to, is very nice under element where people are there, they can do research, they like with a lot of resources.So it was nice. I think things changed a lot when I think when J Pity came out. I think after that I think was. This one is same again. But but yeah, but it was nice. And I think we also want to work part of this descrip beforeswyx: coming to the end.Hiring and Team Footprintswyx: We're just, obviously, I think you guys are doing incredible work.You've, they are a very impressive vision for open source and for voice. What are you hiring for? What's the what are you looking for that you are trying to join the company?Guillaume: Yeah, so we are hiring a lot of people in our sense team. We're hiring, in all our offices. So we have a, our H two is in France in Paris.We have a small team in London. We like a team in Pato as well. Co we open some offices in in SAU, in Poland. So one in Zurich. We also like some presence in New York as well on Sooner one in San Francisco. So we all bit either way also like hiring remotely. So we're going the team trying to hire like very strong people.I think we want to stay, so the team is not. Instead of fairly small team. [00:43:00] But I think we want to keep it that way. ‘Cause we we find it quite efficient. So like a small team they agile so yeah.swyx: Okay.AI for Science Partnershipsswyx: Let's focus on science and the forward deployed. We actually are strong believers in science.We started the our new science pod that focuses specifically on the air for science. What areas do you think are the most promis.Guillaume: What we're pretty excited about right now, and something we have already started doing or that we'd probably be able to share more about this in a couple of months, is that we are exploring AI for science.And there are a lot of areas where we think that you could get some extremely promising buzz. If you were to apply AI in these domains. There are a lot of long inputs. You just have to find these domains where actually AI has not been yet applied, and it's usually hard to do because the people working in those domains don't necessarily know the capability of these models.They don't know. How I would just have to pair them with Yeah, exactly. Your researcher slashing, which is actually hard to do. But this matching, we're doing it naturally with our customers. So we have some company we are very closely with. So for instance, ISM Andreesen are one of our partners, so we're doing some research with them on their other, like tons of extremely interesting problems.Columns in physics, in [00:44:00] science matter science that they're essentially the only ones to work on. ‘cause they're doing something No, no one else is doing on the, yeah. So there are many domains where AI can actually revolutionize things. Just you have to think about it on you familiar with what can do or to apply it.So yeah, it's something where more modeling with our partners, with our customers sort AI for s, but.swyx: Yeah. Okay.Forward Deployed Skillsswyx: And then for deployed what it makes a good four deployed engineer, what do they need? Where do people fail?Guillaume: I think it's usually you need people that are very familiar with the tech and not necessarily with a lot of research expertise, but that are actually pretty good at using this model that can actually like that know how to do functioning, that know how to like, start some error pipeline.And it's it's not easy. It's something that mucus. Majority of companies will not be able to do this on their own. So here I think we need people that are, that like to solve problems that are accept solving some complex, very concrete problem. It's applied science basically.And yeah, so I think it's not too different. I think from the case you need in research because it's essentially you are trying to find solutions to problems that in [00:45:00] customers have not yet. So sometimes it's easy. Sometimes you're here to do the work. You have to like create synthetic data.Find some edge case. So it can be, yeah. Depends on the problem. But but yeah, you have to, I think it also a bit of patience on the be creative. I think very similar skill is Asian,Pavan: the diversity of the work they do. It always surprises me. It's it's, it goes all the way from the kind of stuff they encounter in industries.It's just very interesting. I think.swyx: Any fun like success anecdotes.Guillaume: Yeah, it can be actually training this small model on edge that just we do one specific thing can be like training some very large model without some specific languages as well. Making models really good at some tube use, like for instance, computer ID design, these kind of things.Is that pairing with vision as well? Yeah,Pavan: and the fact detection for chips or like in, in factories identifying things like it, the. Diversity could be anything where you can deploy these foundation models. So yeah the work to make it work in that specific setting, basically whatever it takes to make it like add value in that, by the way, workflow.Vibhu: Yeah. [00:46:00] And it goes across the stack, right? Like even just pulling up the website like.swyx: It's so broad on compute. It is so broad.Vibhu: We didn't even touch on if you have a coding CLI tool. One thing you guys were actually like, I think the first tool was agents, ral agents. You had the agent builder, you can serve it via API and all that.And I'm guessing forward deploy people.Guillaume: Yeah.Vibhu: Help build that out and stuff.Customer Feedback LoopGuillaume: It is also why we are, so we're doing many things, but I think that's also part of the value proposition that sometime know customers. They're always very. Extremely careful about their data and they don't want to, they don't like, trusting so many partners, trusting one partner for code, giving the data to another third party for like audios and another one.So they don't like this here. What they really like with our approach that we can help them on anything so they don't have to send the data to so many clouds. So yeah,swyx: I think that there can be many orders of magnitude more. F Ds then research scientists and they don't need your full experience, but they're still super variable to customersGuillaume: in practice.These two teams [00:47:00] are still quite intertwine, very often. Yeah. So first of all, they're using the same tools, the same data pipeline and everything on the, it's it's very helpful for the science team to get the feedback and the solution team ‘cause they can. Look at these customers are trying to do this.This is not working. It can really be show in the next version. Yeah. But this is basically a real world eval. Yeah, it's real world eval and it's not something, for instance, if you're just working in the lab, it's just ships model. But you don't do this work of for customers. You have no idea for whether your model is good at this H case.For instance, you even in year found this, right? So yeah, there is a very gap, big gap between the public benchmarks that are very like academic. OnPavan: the rare cases are just very diverse and in the specific concept of a customer, you can fine tune and make it like first evaluate, create a solid eval, benchmark, and then measure in the context of their, the kind of audio.Like for instance, one use case is literally just, there's the word for kids and they have to just say it out. It's a very specific thing. You're just saying one word and then you have to you, you'll grade the kid whether they did it right or not. It's [00:48:00] like R for, but so there're very diverse use cases and the idea is that they, the.Applied scientist engineer will go and make it better. And then from the learnings we incorporate it into the base model itself. So it's it's just better out of the box.Vibhu: Yeah. It's a good full circle system. Like the foundation model evals are all just proxies of what you really, you're never gonna have one that says it, it doesn't make sense for there to be, a one word transcription like that.It's not something you wanna fit on. Perfect.Wrap Up and Thanksswyx: Everyone should go check out everything that Michelle has to offer and try the TTS model, which will link in the show notes. But thank you so much for coming tha thanks. Such a stretch. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.On this episode we're sitting down with three leaders from the Arctic Wolves: 1st Brigade, 11th Airborne Division. They have allowed units across their brigade to conduct "offset PT" which - importantly - is not just "reverse cycle PT" but rather a flexible approach to when physical training is conducted. Their initial pilot showed huge improvements not just in fitness, but across sleep, nutrition, and even social connections. COL Christopher Brawley, the Brigade Commander, had already implemented this model successfully at 1st Ranger Battalion, so he knew the potential it presented to building trust and organizational performance. CSM Jeremiah Waggoner, a multiple time Best Ranger competitor, was focused on the physical fitness expectations for the formation, as well as keeping NCOs in charge of leading training of all types.Dr. Ellie Van Luit, the brigade's H2F Program Director, saw H2F's role as enabling and supporting this effort, but she emphasized that it was a brigade initiative, not an H2F initiative.You can find the full results of the initial pilot on our free downloads page.

In our studies so far in the book of Exodus, we have met the Lord God’s appointed deliverer, Moses, and his many excuses and fears. Having gone to Pharaoh as the Lord God had commanded him with the result being far from what Moes had expected, Moses could only now go back to the Lord […]

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week's episode is a continuation of our conversation with Michael Blevins. In particular we focus on the themes of his article "Bandaids for Bullet Wounds: State Changing to Survive your own Life?" This article has significantly influenced some of our recent discussions around whether we're measuring (and therefore focusing on) the right things in military human performance. We also touch on his experiences training Henry Cavill for several of his movie appearances.Michael's journey began with trying to test his own limits in sports like sailing, rock climbing, and skateboarding. He then transitioned into exploring a range of martial arts disciplines, followed by an evolution into endurance sports, and then the fragility he felt from pushing those limits led him to incorporate weighlifting, crossfit, and strongman style training. He has competed in cycling, triathlon, crossfit, weighlifting, jiujitsu and more.Professionally, he has been a hairdresser, make up artist, photographer, worked in the fashion industry and on the stage... all ultimately developing a skill for building relationships that led him into coaching. He has coached actors preparing for film roles, military service members preparing for selections, and athletes competing at elite levels. Perhaps most notably he coached Henry Cavill leading up to Man of Steel, Batman vs Superman, and Justice League. He also coach both actors and stunt crew for 300: Rise of an Empire, and led a team development camp for the Atlanta Braves. We mentioned the strength manual he published in this conversation, which he's currently rewriting, he's host of the UNFVCKED podcast, and creator of We Are Ollin.In this episode we discussed Ollin's annual event "The Space Race" and you can find the most recent version here.

Pogingen om de rechten van de autochtone Nederlanders te verdedigen, stuiten op onbegrip en zelfs agressie, constateert Wierd Duk in een nieuwe aflevering van de podcast Het Land van Wierd Duk. Hij wijst op de behandeling van oud-minister Gouke Moes in de talkshow van Jeroen Pauw. „Moes vindt dat de Nederlandse cultuur en identiteit worden bedreigd door de massa-immigratie en kreeg een hele tafel tegenover zich met critici die er slechts op uit waren om hem de mond te snoeren.” Ooit keert de wal het schip, vreest Duk. „Je kunt niet de zorgen van zo'n groot deel van de bevolking negeren terwijl je de allochtone minderheid – en vooral de moslims – maar blijft bevoordelen.” Verder in de podcast: DENK eist verdere islamisering van onze instituties. En: hoe nu verder in Iran?See omnystudio.com/listener for privacy information.

Join Kyle, Nader, Vibhu, and swyx live at NVIDIA GTC next week!Now that AIE Europe tix are ~sold out, our attention turns to Miami and World's Fair!The definitive AI Accelerator chip company has more than 10xed this AI Summer:And is now a $4.4 trillion megacorp… that is somehow still moving like a startup. We are blessed to have a unique relationship with our first ever NVIDIA guests: Kyle Kranen who gave a great inference keynote at the first World's Fair and is one of the leading architects of NVIDIA Dynamo (a Datacenter scale inference framework supporting SGLang, TRT-LLM, vLLM), and Nader Khalil, a friend of swyx from our days in Celo in The Arena, who has been drawing developers at GTC since before they were even a glimmer in the eye of NVIDIA:Nader discusses how NVIDIA Brev has drastically reduced the barriers to entry for developers to get a top of the line GPU up and running, and Kyle explains NVIDIA Dynamo as a data center scale inference engine that optimizes serving by scaling out, leveraging techniques like prefill/decode disaggregation, scheduling, and Kubernetes-based orchestration, framed around cost, latency, and quality tradeoffs. We also dive into Jensen's “SOL” (Speed of Light) first-principles urgency concept, long-context limits and model/hardware co-design, internal model APIs (https://build.nvidia.com), and upcoming Dynamo and agent sessions at GTC.Full Video pod on YouTubeTimestamps00:00 Agent Security Basics00:39 Podcast Welcome and Guests07:19 Acquisition and DevEx Shift13:48 SOL Culture and Dynamo Setup27:38 Why Scale Out Wins29:02 Scale Up Limits Explained30:24 From Laptop to Multi Node33:07 Cost Quality Latency Tradeoffs38:42 Disaggregation Prefill vs Decode41:05 Kubernetes Scaling with Grove43:20 Context Length and Co Design57:34 Security Meets Agents58:01 Agent Permissions Model59:10 Build Nvidia Inference Gateway01:01:52 Hackathons And Autonomy Dreams01:10:26 Local GPUs And Scaling Inference01:15:31 Long Running Agents And SF ReflectionsTranscriptAgent Security BasicsNader: Agents can do three things. They can access your files, they can access the internet, and then now they can write custom code and execute it. You literally only let an agent do two of those three things. If you can access your files and you can write custom code, you don't want internet access because that's one to see full vulnerability, right?If you have access to internet and your file system, you should know the full scope of what that agent's capable of doing. Otherwise, now we can get injected or something that can happen. And so that's a lot of what we've been thinking about is like, you know, how do we both enable this because it's clearly the future.But then also, you know, what, what are these enforcement points that we can start to like protect?swyx: All right.Podcast Welcome and Guestsswyx: Welcome to the Lean Space podcast in the Chromo studio. Welcome to all the guests here. Uh, we are back with our guest host Viu. Welcome. Good to have you back. And our friends, uh, Netter and Kyle from Nvidia. Welcome.Kyle: Yeah, thanks for having us.swyx: Yeah, thank you. Actually, I don't even know your titles.Uh, I know you're like architect something of Dynamo.Kyle: Yeah. I, I'm one of the engineering leaders [00:01:00] and a architects of Dynamo.swyx: And you're director of something and developers, developer tech.Nader: Yeah.swyx: You're the developers, developers, developers guy at nvidia,Nader: open source agent marketing, brev,swyx: and likeNader: Devrel tools and stuff.swyx: Yeah. BeenNader: the focus.swyx: And we're, we're kind of recording this ahead of Nvidia, GTC, which is coming to town, uh, again, uh, or taking over town, uh, which, uh, which we'll all be at. Um, and we'll talk a little bit about your sessions and stuff. Yeah.Nader: We're super excited for it.GTC Booth Stunt Storiesswyx: One of my favorite memories for Nader, like you always do like marketing stunts and like while you were at Rev, you like had this surfboard that you like, went down to GTC with and like, NA Nvidia apparently, like did so much that they bought you.Like what, what was that like? What was that?Nader: Yeah. Yeah, we, we, um. Our logo was a chaka. We, we, uh, we were always just kind of like trying to keep true to who we were. I think, you know, some stuff, startups, you're like trying to pretend that you're a bigger, more mature company than you are. And it was actually Evan Conrad from SF Compute who was just like, you guys are like previousswyx: guest.Yeah.Nader: Amazing. Oh, really? Amazing. Yeah. He was just like, guys, you're two dudes in the room. Why are you [00:02:00] pretending that you're not? Uh, and so then we were like, okay, let's make the logo a shaka. We brought surfboards to our booth to GTC and the energy was great. Yeah. Some palm trees too. They,Kyle: they actually poked out over like the, the walls so you could, you could see the bread booth.Oh, that's so funny. AndNader: no one else,Kyle: just from very far away.Nader: Oh, so you remember it backKyle: then? Yeah I remember it pre-acquisition. I was like, oh, those guys look cool,Nader: dude. That makes sense. ‘cause uh, we, so we signed up really last minute, and so we had the last booth. It was all the way in the corner. And so I was, I was worried that no one was gonna come.So that's why we had like the palm trees. We really came in with the surfboards. We even had one of our investors bring her dog and then she was just like walking the dog around to try to like, bring energy towards our booth. Yeah.swyx: Steph.Kyle: Yeah. Yeah, she's the best,swyx: you know, as a conference organizer, I love that.Right? Like, it's like everyone who sponsors a conference comes, does their booth. They're like, we are changing the future of ai or something, some generic b******t and like, no, like actually try to stand out, make it fun, right? And people still remember it after three years.Nader: Yeah. Yeah. You know what's so funny?I'll, I'll send, I'll give you this clip if you wanna, if you wanna add it [00:03:00] in, but, uh, my wife was at the time fiance, she was in medical school and she came to help us. ‘cause it was like a big moment for us. And so we, we bought this cricket, it's like a vinyl, like a vinyl, uh, printer. ‘cause like, how else are we gonna label the surfboard?So, we got a surfboard, luckily was able to purchase that on the company card. We got a cricket and it was just like fine tuning for enterprises or something like that, that we put on the. On the surfboard and it's 1:00 AM the day before we go to GTC. She's helping me put these like vinyl stickers on.And she goes, you son of, she's like, if you pull this off, you son of a b***h. And so, uh, right. Pretty much after the acquisition, I stitched that with the mag music acquisition. I sent it to our family group chat. Ohswyx: Yeah. No, well, she, she made a good choice there. Was that like basically the origin story for Launchable is that we, it was, and maybe we should explain what Brev is andNader: Yeah.Yeah. Uh, I mean, brev is just, it's a developer tool that makes it really easy to get a GPU. So we connect a bunch of different GPU sources. So the basics of it is like, how quickly can we SSH you into a G, into a GPU and whenever we would talk to users, they wanted A GPU. They wanted an A 100. And if you go to like any cloud [00:04:00] provisioning page, usually it's like three pages of forms or in the forms somewhere there's a dropdown.And in the dropdown there's some weird code that you know to translate to an A 100. And I remember just thinking like. Every time someone says they want an A 100, like the piece of text that they're telling me that they want is like, stuffed away in the corner. Yeah. And so we were like, what if the biggest piece of text was what the user's asking for?And so when you go to Brev, it's just big GPU chips with the type that you want withswyx: beautiful animations that you worked on pre, like pre you can, like, now you can just prompt it. But back in the day. Yeah. Yeah. Those were handcraft, handcrafted artisanal code.Nader: Yeah. I was actually really proud of that because, uh, it was an, i I made it in Figma.Yeah. And then I found, I was like really struggling to figure out how to turn it from like Figma to react. So what it actually is, is just an SVG and I, I have all the styles and so when you change the chip, whether it's like active or not it changes the SVG code and that somehow like renders like, looks like it's animating, but it, we just had the transition slow, but it's just like the, a JavaScript function to change the like underlying SVG.Yeah. And that was how I ended up like figuring out how to move it from from Figma. But yeah, that's Art Artisan. [00:05:00]Kyle: Speaking of marketing stunts though, he actually used those SVGs. Or kind of use those SVGs to make these cards.Nader: Oh yeah. LikeKyle: a GPU gift card Yes. That he handed out everywhere. That was actually my first impression of thatNader: one.Yeah,swyx: yeah, yeah.Nader: Yeah.swyx: I think I still have one of them.Nader: They look great.Kyle: Yeah.Nader: I have a ton of them still actually in our garage, which just, they don't have labels. We should honestly like bring, bring them back. But, um, I found this old printing press here, actually just around the corner on Ven ness. And it's a third generation San Francisco shop.And so I come in an excited startup founder trying to like, and they just have this crazy old machinery and I'm in awe. ‘cause the the whole building is so physical. Like you're seeing these machines, they have like pedals to like move these saws and whatever. I don't know what this machinery is, but I saw all three generations.Like there's like the grandpa, the father and the son, and the son was like, around my age. Well,swyx: it's like a holy, holy trinity.Nader: It's funny because we, so I just took the same SVG and we just like printed it and it's foil printing, so they make a a, a mold. That's like an inverse of like the A 100 and then they put the foil on it [00:06:00] and then they press it into the paper.And I remember once we got them, he was like, Hey, don't forget about us. You know, I guess like early Apple and Cisco's first business cards were all made there. And so he was like, yeah, we, we get like the startup businesses but then as they mature, they kind of go somewhere else. And so I actually, I think we were talking with marketing about like using them for some, we should go back and make some cards.swyx: Yeah, yeah, yeah. You know, I remember, you know, as a very, very small breadth investor, I was like, why are we spending time like, doing these like stunts for GPUs? Like, you know, I think like as a, you know, typical like cloud hard hardware person, you go into an AWS you pick like T five X xl, whatever, and it's just like from a list and you look at the specs like, why animate this GP?And, and I, I do think like it just shows the level of care that goes throughout birth and Yeah. And now, and also the, and,Nader: and Nvidia. I think that's what the, the thing that struck me most when we first came in was like the amount of passion that everyone has. Like, I think, um, you know, you talk to, you talk to Kyle, you talk to, like, every VP that I've met at Nvidia goes so close to the metal.Like, I remember it was almost a year ago, and like my VP asked me, he's like, Hey, [00:07:00] what's cursor? And like, are you using it? And if so, why? Surprised at this, and he downloaded Cursor and he was asking me to help him like, use it. And I thought that was, uh, or like, just show him what he, you know, why we were using it.And so, the amount of care that I think everyone has and the passion, appreciate, passion and appreciation for the moment. Right. This is a very unique time. So it's really cool to see everyone really like, uh, appreciate that.swyx: Yeah.Acquisition and DevEx Shiftswyx: One thing I wanted to do before we move over to sort of like research topics and, uh, the, the stuff that Kyle's working on is just tell the story of the acquisition, right?Like, not many people have been, been through an acquisition with Nvidia. What's it like? Uh, what, yeah, just anything you'd like to say.Nader: It's a crazy experience. I think, uh, you know, we were the thing that was the most exciting for us was. Our goal was just to make it easier for developers.We wanted to find access to GPUs, make it easier to do that. And then all, oh, actually your question about launchable. So launchable was just make one click exper, like one click deploys for any software on top of the GPU. Mm-hmm. And so what we really liked about Nvidia was that it felt like we just got a lot more resources to do all of that.I think, uh, you [00:08:00] know, NVIDIA's goal is to make things as easy for developers as possible. So there was a really nice like synergy there. I think that, you know, when it comes to like an acquisition, I think the amount that the soul of the products align, I think is gonna be. Is going speak to the success of the acquisition.Yeah. And so it in many ways feels like we're home. This is a really great outcome for us. Like we you know, I love brev.nvidia.com. Like you should, you should use it's, it's theKyle: front page for GPUs.Nader: Yeah. Yeah. If you want GP views,Kyle: you go there, getswyx: it there, and it's like internally is growing very quickly.I, I don't remember You said some stats there.Nader: Yeah, yeah, yeah. It's, uh, I, I wish I had the exact numbers, but like internally, externally, it's been growing really quickly. We've been working with a bunch of partners with a bunch of different customers and ISVs, if you have a solution that you want someone that runs on the GPU and you want people to use it quickly, we can bundle it up, uh, in a launchable and make it a one click run.If you're doing things and you want just like a sandbox or something to run on, right. Like open claw. Huge moment. Super exciting. Our, uh, and we'll talk into it more, but. You know, internally, people wanna run this, and you, we know we have to be really careful from the security implications. Do we let this run on the corporate network?Security's guidance was, Hey, [00:09:00] run this on breath, it's in, you know, it's, it's, it's a vm, it's sitting in the cloud, it's off the corporate network. It's isolated. And so that's been our stance internally and externally about how to even run something like open call while we figure out how to run these things securely.But yeah,swyx: I think there's also like, you almost like we're the right team at the right time when Nvidia is starting to invest a lot more in developer experience or whatever you call it. Yeah. Uh, UX or I don't know what you call it, like software. Like obviously NVIDIA is always invested in software, but like, there's like, this is like a different audience.Yeah. It's aNader: widerKyle: developer base.swyx: Yeah. Right.Nader: Yeah. Yeah. You know, it's funny, it's like, it's not, uh,swyx: so like, what, what is it called internally? What, what is this that people should be aware that is going on there?Nader: Uh, what, like developer experienceswyx: or, yeah, yeah. Is it's called just developer experience or is there like a broader strategy hereNader: in Nvidia?Um, Nvidia always wants to make a good developer experience. The thing is and a lot of the technology is just really complicated. Like, it's not, it's uh, you know, I think, um. The thing that's been really growing or the AI's growing is having a huge moment, not [00:10:00] because like, let's say data scientists in 2018, were quiet then and are much louder now.The pie is com, right? There's a whole bunch of new audiences. My mom's wondering what she's doing. My sister's learned, like taught herself how to code. Like the, um, you know, I, I actually think just generally AI's a big equalizer and you're seeing a more like technologically literate society, I guess.Like everyone's, everyone's learning how to code. Uh, there isn't really an excuse for that. And so building a good UX means that you really understand who your end user is. And when your end user becomes such a wide, uh, variety of people, then you have to almost like reinvent the practice, right? Yeah. You haveKyle: to, and actually build more developer ux, right?Because the, there are tiers of developer base that were added. You know, the, the hackers that are building on top of open claw, right? For example, have never used gpu. They don't know what kuda is. They, they, they just want to run something.Nader: Yeah.Kyle: You need new UX that is not just. Hey, you know, how do you program something in Cuda and run it?And then, and then we built, you know, like when Deep Learning was getting big, we built, we built Torch and, and, but so recently the amount of like [00:11:00] layers that are added to that developer stack has just exploded because AI has become ubiquitous. Everyone's using it in different ways. Yeah. It'sNader: moving fast in every direction.Vertical, horizontal.Vibhu: Yeah. You guys, you even take it down to hardware, like the DGX Spark, you know, it's, it's basically the same system as just throwing it up on big GPU cluster.Nader: Yeah, yeah, yeah. It's amazing. Blackwell.swyx: Yeah. Uh, we saw the preview at the last year's GTC and that was one of the better performing, uh, videos so far, and video coverage so far.Awesome. This will beat it. Um,Nader: that wasswyx: actually, we have fingersNader: crossed. Yeah.DGX Spark and Remote AccessNader: Even when Grace Blackwell or when, um, uh, DGX Spark was first coming out getting to be involved in that from the beginning of the developer experience. And it just comes back to what youswyx: were involved.Nader: Yeah. St. St.swyx: Mars.Nader: Yeah. Yeah. I mean from, it was just like, I, I got an email, we just got thrown into the loop and suddenly yeah, I, it was actually really funny ‘cause I'm still pretty fresh from the acquisition and I'm, I'm getting an email from a bunch of the engineering VPs about like, the new hardware, GPU chip, like we're, or not chip, but just GPU system that we're putting out.And I'm like, okay, cool. Matters. Now involved with this for the ux, I'm like. What am I gonna do [00:12:00] here? So, I remember the first meeting, I was just like kind of quiet as I was hearing engineering VPs talk about what this box could be, what it could do, how we should use it. And I remember, uh, one of the first ideas that people were idea was like, oh, the first thing that it was like, I think a quote was like, the first thing someone's gonna wanna do with this is get two of them and run a Kubernetes cluster on top of them.And I was like, oh, I think I know why I'm here. I was like, the first thing we're doing is easy. SSH into the machine. And then, and you know, just kind of like scoping it down of like, once you can do that every, you, like the person who wants to run a Kubernetes cluster onto Sparks has a higher propensity for pain, then, then you know someone who buys it and wants to run open Claw right now, right?If you can make sure that that's as effortless as possible, then the rest becomes easy. So there's a tool called Nvidia Sync. It just makes the SSH connection really simple. So, you know, if you think about it like. If you have a Mac, uh, or a PC or whatever, if you have a laptop and you buy this GPU and you want to use it, you should be able to use it like it's A-A-G-P-U in the cloud, right?Um, but there's all this friction of like, how do you actually get into that? That's part of [00:13:00] Revs value proposition is just, you know, there's a CLI that wraps SSH and makes it simple. And so our goal is just get you into that machine really easily. And one thing we just launched at CES, it's in, it's still in like early access.We're ironing out some kinks, but it should be ready by GTC. You can register your spark on Brev. And so now if youswyx: like remote managed yeah, local hardware. Single pane of glass. Yeah. Yeah. Because Brev can already manage other clouds anyway, right?Vibhu: Yeah, yeah. And you use the spark on Brev as well, right?Nader: Yeah. But yeah, exactly. So, so you, you, so you, you set it up at home you can run the command on it, and then it gets it's essentially it'll appear in your Brev account, and then you can take your laptop to a Starbucks or to a cafe, and you'll continue to use your, you can continue use your spark just like any other cloud node on Brev.Yeah. Yeah. And it's just like a pre-provisioned centerswyx: in yourNader: home. Yeah, exactly.swyx: Yeah. Yeah.Vibhu: Tiny little data center.Nader: Tiny little, the size ofVibhu: your phone.SOL Culture and Dynamo Setupswyx: One more thing before we move on to Kyle. Just have so many Jensen stories and I just love, love mining Jensen stories. Uh, my favorite so far is SOL. Uh, what is, yeah, what is S-O-L-S-O-LNader: is actually, i, I think [00:14:00] of all the lessons I've learned, that one's definitely my favorite.Kyle: It'll always stick with you.Nader: Yeah. Yeah. I, you know, in your startup, everything's existential, right? Like we've, we've run out of money. We were like, on the risk of, of losing payroll, we've had to contract our team because we l ran outta money. And so like, um, because of that you're really always forcing yourself to I to like understand the root cause of everything.If you get a date, if you get a timeline, you know exactly why that date or timeline is there. You're, you're pushing every boundary and like, you're not just say, you're not just accepting like a, a no. Just because. And so as you start to introduce more layers, as you start to become a much larger organization, SOL is is essentially like what is the physics, right?The speed of light moves at a certain speed. So if flight's moving some slower, then you know something's in the way. So before trying to like layer reality back in of like, why can't this be delivered at some date? Let's just understand the physics. What is the theoretical limit to like, uh, how fast this can go?And then start to tell me why. ‘cause otherwise people will start telling you why something can't be done. But actually I think any great leader's goal is just to create urgency. Yeah. [00:15:00] There's an infiniteKyle: create compelling events, right?Nader: Yeah.Kyle: Yeah. So l is a term video is used to instigate a compelling event.You say this is done. How do we get there? What is the minimum? As much as necessary, as little as possible thing that it takes for us to get exactly here and. It helps you just break through a bunch of noise.swyx: Yeah.Kyle: Instantly.swyx: One thing I'm unclear about is, can only Jensen use the SOL card? Like, oh, no, no, no.Not everyone get the b******t out because obviously it's Jensen, but like, can someone else be like, no, likeKyle: frontline engineers use it.Nader: Yeah. Every, I think it's not so much about like, get the b******t out. It's like, it's like, give me the root understanding, right? Like, if you tell me something takes three weeks, it like, well, what's the first principles?Yeah, the first principles. It's like, what's the, what? Like why is it three weeks? What is the actual yeah. What's the actual limit of why this is gonna take three weeks? If you're gonna, if you, if let's say you wanted to buy a new computer and someone told you it's gonna be here in five days, what's the SOL?Well, like the SOL is like, I could walk into a Best Buy and pick it up for you. Right? So then anything that's like beyond that is, and is that practical? Is that how we're gonna, you know, let's say give everyone in the [00:16:00] company a laptop, like obviously not. So then like that's the SOL and then it's like, okay, well if we have to get more than 10, suddenly there might be some, right?And so now we can kind of piece the reality back.swyx: So, so this is the. Paul Graham do things that don't scale. Yeah. And this is also the, what people would now call behi agency. Yeah.Kyle: It's actually really interesting because there's a, there's a second hardware angle to SOL that like doesn't come up for all the org sol is used like culturally at aswyx: media for everything.I'm also mining for like, I think that can be annoying sometimes. And like someone keeps going IOO you and you're like, guys, like we have to be stable. We have to, we to f*****g plan. Yeah.Kyle: It's an interesting balance.Nader: Yeah. I encounter that with like, actually just with, with Alec, right? ‘cause we, we have a new conference so we need to launch, we have, we have goals of what we wanna launch by, uh, by the conference and like, yeah.At the end of the day, where isswyx: this GTC?Nader: Um, well this is like, so we, I mean we did it for CES, we did for GT CDC before that we're doing it for GTC San Jose. So I mean, like every, you know, we have a new moment. Um, and we want to launch something. Yeah. And we want to do so at SOL and that does mean that some, there's some level of prioritization that needs [00:17:00] to happen.And so it, it is difficult, right? I think, um, you have to be careful with what you're pushing. You know, stability is important and that should be factored into S-O-L-S-O-L isn't just like, build everything and let it break, you know, that, that's part of the conversation. So as you're laying, layering in all the details, one of them might be, Hey, we could build this, but then it's not gonna be stable for X, y, z reasons.And so that was like, one of our conversations for CES was, you know, hey, like we, we can get this into early access registering your spark with brev. But there are a lot of things that we need to do in order to feel really comfortable from a security perspective, right? There's a lot of networking involved before we deliver that to users.So it's like, okay. Let's get this to a point where we can at least let people experiment with it. We had it in a booth, we had it in Jensen's keynote, and then let's go iron out all the networking kinks. And that's not easy. And so, uh, that can come later. And so that was the way that we layered that back in.Yeah. ButKyle: It's not really about saying like, you don't have to do the, the maintenance or operational work. It's more about saying, you know, it's kind of like [00:18:00] highlights how progress is incremental, right? Like, what is the minimum thing that we can get to. And then there's SOL for like every component after that.But there's the SOL to get you, get you to the, the starting line. And that, that's usually how it's asked. Yeah. On the other side, you know, like SOL came out of like hardware at Nvidia. Right. So SOL is like literally if we ran the accelerator or the GPU with like at basically full speed with like no other constraints, like how FAST would be able to make a program go.swyx: Yeah. Yeah. Right.Kyle: Soswyx: in, in training that like, you know, then you work back to like some percentage of like MFU for example.Kyle: Yeah, that's a, that's a great example. So like, there's an, there's an S-O-L-M-F-U, and then there's like, you know, what's practically achievable.swyx: Cool. Should we move on to sort of, uh, Kyle's side?Uh, Kyle, you're coming more from the data science world. And, uh, I, I mean I always, whenever, whenever I meet someone who's done working in tabular stuff, graph neural networks, time series, these are basically when I go to new reps, I go to ICML, I walk the back halls. There's always like a small group of graph people.Yes. Absolute small group of tabular people. [00:19:00] And like, there's no one there. And like, it's very like, you know what I mean? Like, yeah, no, like it's, it's important interesting work if you care about solving the problems that they solve.Kyle: Yeah.swyx: But everyone else is just LMS all the time.Kyle: Yeah. I mean it's like, it's like the black hole, right?Has the event horizon reached this yet in nerves? Um,swyx: but like, you know, those are, those are transformers too. Yeah. And, and those are also like interesting things. Anyway, uh, I just wanted to spend a little bit of time on, on those, that background before we go into Dynamo, uh, proper.Kyle: Yeah, sure. I took a different path to Nvidia than that, or I joined six years ago, seven, if you count, when I was an intern.So I joined Nvidia, like right outta college. And the first thing I jumped into was not what I'd done in, during internship, which was like, you know, like some stuff for autonomous vehicles, like heavyweight object detection. I jumped into like, you know, something, I'm like, recommenders, this is popular. Andswyx: yeah, he did RexiKyle: as well.Yeah, Rexi. Yeah. I mean that, that was the taboo data at the time, right? You have tables of like, audience qualities and item qualities, and you're trying to figure out like which member of [00:20:00] the audience matches which item or, or more practically which item matches which member of the audience. And at the time, really it was like we were trying to enable.Uh, recommender, which had historically been like a little bit of a CP based workflow into something that like, ran really well in GPUs. And it's since been done. Like there are a bunch of libraries for Axis that run on GPUs. Uh, the common models like Deeplearning recommendation model, which came outta meta and the wide and deep model, which was used or was released by Google were very accelerated by GPUs using, you know, the fast HBM on the chips, especially to do, you know, vector lookups.But it was very interesting at the time and super, super relevant because like we were starting to get like. This explosion of feeds and things that required rec recommenders to just actively be on all the time. And sort of transitioned that a little bit towards graph neural networks when I discovered them because I was like, okay, you can actually use graphical neural networks to represent like, relationships between people, items, concepts, and that, that interested me.So I jumped into that at [00:21:00] Nvidia and, and got really involved for like two-ish years.swyx: Yeah. Uh, and something I learned from Brian Zaro Yeah. Is that you can just kind of choose your own path in Nvidia.Kyle: Oh my God. Yeah.swyx: Which is not a normal big Corp thing. Yeah. Like you, you have a lane, you stay in your lane.Nader: I think probably the reason why I enjoy being in a, a big company, the mission is the boss probably from a startup guy. Yeah. The missionswyx: is the boss.Nader: Yeah. Uh, it feels like a big game of pickup basketball. Like, you know, if you play one, if you wanna play basketball, you just go up to the court and you're like, Hey look, we're gonna play this game and we need three.Yeah. And you just like find your three. That's honestly for every new initiative that's what it feels like. Yeah.Vibhu: It also like shows, right? Like Nvidia. Just releasing state-of-the-art stuff in every domain. Yeah. Like, okay, you expect foundation models with Nemo tron voice just randomly parakeet.Call parakeet just comes out another one, uh, voice. TheKyle: video voice team has always been producing.Vibhu: Yeah. There's always just every other domain of paper that comes out, dataset that comes out. It's like, I mean, it also stems back to what Nvidia has to do, right? You have to make chips years before they're actually produced.Right? So you need to know, you need to really [00:22:00] focus. TheKyle: design process starts likeVibhu: exactlyKyle: three to five years before the chip gets to the market.Vibhu: Yeah. I, I'm curious more about what that's like, right? So like, you have specialist teams. Is it just like, you know, people find an interest, you go in, you go deep on whatever, and that kind of feeds back into, you know, okay, we, we expect predictions.Like the internals at Nvidia must be crazy. Right? You know? Yeah. Yeah. You know, you, you must. Not even without selling to people, you have your own predictions of where things are going. Yeah. And they're very based, very grounded. Right?Kyle: Yeah. It, it, it's really interesting. So there's like two things that I think that Amed does, which are quite interesting.Uh, one is like, we really index into passion. There's a big. Sort of organizational top sound push to like ensure that people are working on the things that they're passionate about. So if someone proposes something that's interesting, many times they can just email someone like way up the chain that they would find this relevant and say like, Hey, can I go work on this?Nader: It's actually like I worked at a, a big company for a couple years before, uh, starting on my startup journey and like, it felt very weird if you were to like email out of chain, if that makes [00:23:00] sense. Yeah. The emails at Nvidia are like mosh pitsswyx: shoot,Nader: and it's just like 60 people, just whatever. And like they're, there's this,swyx: they got messy like, reply all you,Nader: oh, it's in, it's insane.It's insane. They justKyle: help. You know, Maxim,Nader: the context. But, but that's actually like, I've actually, so this is a weird thing where I used to be like, why would we send emails? We have Slack. I am the entire, I'm the exact opposite. I feel so bad for anyone who's like messaging me on Slack ‘cause I'm so unresponsive.swyx: Your emailNader: Maxi, email Maxim. I'm email maxing Now email is a different, email is perfect because man, we can't work together. I'm email is great, right? Because important threads get bumped back up, right? Yeah, yeah. Um, and so Slack doesn't do that. So I just have like this casino going off on the right or on the left and like, I don't know which thread was from where or what, but like the threads get And then also just like the subject, so you can have like working threads.I think what's difficult is like when you're small, if you're just not 40,000 people I think Slack will work fine, but there's, I don't know what the inflection point is. There is gonna be a point where that becomes really messy and you'll actually prefer having email. ‘cause you can have working threads.You can cc more than nine people in a thread.Kyle: You can fork stuff.Nader: You can [00:24:00] fork stuff, which is super nice and just like y Yeah. And so, but that is part of where you can propose a plan. You can also just. Start, honestly, momentum's the only authority, right? So like, if you can just start, start to make a little bit of progress and show someone something, and then they can try it.That's, I think what's been, you know, I think the most effective way to push anything for forward. And that's both at Nvidia and I think just generally.Kyle: Yeah, there's, there's the other concept that like is explored a lot at Nvidia, which is this idea of a zero billion dollar business. Like market creation is a big thing at Nvidia.Like,swyx: oh, you want to go and start a zero billion dollar business?Kyle: Jensen says, we are completely happy investing in zero billion dollar markets. We don't care if this creates revenue. It's important for us to know about this market. We think it will be important in the future. It can be zero billion dollars for a while.I'm probably minging as words here for, but like, you know, like, I'll give an example. NVIDIA's been working on autonomous driving for a a long time,swyx: like an Nvidia car.Kyle: No, they, they'veVibhu: used the Mercedes, right? They're around the HQ and I think it finally just got licensed out. Now they're starting to be used quite a [00:25:00] bit.For 10 years you've been seeing Mercedes with Nvidia logos driving.Kyle: If you're in like the South San Santa Clara, it's, it's actually from South. Yeah. So, um. Zero billion dollar markets are, are a thing like, you know, Jensen,swyx: I mean, okay, look, cars are not a zero billion dollar market. But yeah, that's a bad example.Nader: I think, I think he's, he's messaging, uh, zero today, but, or even like internally, right? Like, like it's like, uh, an org doesn't have to ruthlessly find revenue very quickly to justify their existence. Right. Like a lot of the important research, a lot of the important technology being developed that, that's kind ofKyle: where research, research is very ide ideologically free at Nvidia.Yeah. Like they can pursue things that they wereswyx: Were you research officially?Kyle: I was never in research. Officially. I was always in engineering. Yeah. We in, I'm in an org called Deep Warning Algorithms, which is basically just how do we make things that are relevant to deep warning go fast.swyx: That sounds freaking cool.Vibhu: And I think a lot of that is underappreciated, right? Like time series. This week Google put out time. FF paper. Yeah. A new time series, paper res. Uh, Symantec, ID [00:26:00] started applying Transformers LMS to Yes. Rec system. Yes. And when you think the scale of companies deploying these right. Amazon recommendations, Google web search, it's like, it's huge scale andKyle: Yeah.Vibhu: You want fast?Kyle: Yeah. Yeah. Yeah. Actually it's, it, I, there's a fun moment that brought me like full circle. Like, uh, Amazon Ads recently gave a talk where they talked about using Dynamo for generative recommendation, which was like super, like weirdly cathartic for me. I'm like, oh my God. I've, I've supplanted what I was working on.Like, I, you're using LMS now to do what I was doing five years ago.swyx: Yeah. Amazing. And let's go right into Dynamo. Uh, maybe introduce Yeah, sure. To the top down and Yeah.Kyle: I think at this point a lot of people are familiar with the term of inference. Like funnily enough, like I went from, you know, inference being like a really niche topic to being something that's like discussed on like normal people's Twitter feeds.It's,Nader: it's on billboardsKyle: here now. Yeah. Very, very strange. Driving, driving, seeing just an inference ad on 1 0 1 inference at scale is becoming a lot more important. Uh, we have these moments like, you know, open claw where you have these [00:27:00] agents that take lots and lots of tokens, but produce, incredible results.There are many different aspects of test time scaling so that, you know, you can use more inference to generate a better result than if you were to use like a short amount of inference. There's reasoning, there's quiring, there's, adding agency to the model, allowing it to call tools and use skills.Dyno sort came about at Nvidia. Because myself and a couple others were, were sort of talking about the, these concepts that like, you know, you have inference engines like VLMS, shelan, tenor, TLM and they have like one single copy. They, they, they sort of think about like things as like one single copy, like one replica, right?Why Scale Out WinsKyle: Like one version of the model. But when you're actually serving things at scale, you can't just scale up that replica because you end up with like performance problems. There's a scaling limit to scaling up replicas. So you actually have to scale out to use a, maybe some Kubernetes type terminology.We kind of realized that there was like. A lot of potential optimization that we could do in scaling out and building systems for data [00:28:00] center scale inference. So Dynamo is this data center scale inference engine that sits on top of the frameworks like VLM Shilling and 10 T lm and just makes things go faster because you can leverage the economy of scale.The fact that you have KV cash, which we can define a little bit later, uh, in all these machines that is like unique and you wanna figure out like the ways to maximize your cash hits or you want to employ new techniques in inference like disaggregation, which Dynamo had introduced to the world in, in, in March, not introduced, it was a academic talk, but beforehand.But we are, you know, one of the first frameworks to start, supporting it. And we wanna like, sort of combine all these techniques into sort of a modular framework that allows you to. Accelerate your inference at scale.Nader: By the way, Kyle and I became friends on my first date, Nvidia, and I always loved, ‘cause like he always teaches meswyx: new things.Yeah. By the way, this is why I wanted to put two of you together. I was like, yeah, this is, this is gonna beKyle: good. It's very, it's very different, you know, like we've, we, we've, we've talked to each other a bunch [00:29:00] actually, you asked like, why, why can't we scale up?Nader: Yeah.Scale Up Limits ExplainedNader: model, you said model replicas.Kyle: Yeah. So you, so scale up means assigning moreswyx: heavier?Kyle: Yeah, heavier. Like making things heavier. Yeah, adding more GPUs. Adding more CPUs. Scale out is just like having a barrier saying, I'm gonna duplicate my representation of the model or a representation of this microservice or something, and I'm gonna like, replicate it Many times.Handle, load. And the reason that you can't scale, scale up, uh, past some points is like, you know, there, there, there are sort of hardware bounds and algorithmic bounds on, on that type of scaling. So I'll give you a good example that's like very trivial. Let's say you're on an H 100. The Maxim ENV link domain for H 100, for most Ds H one hundreds is heus, right?So if you scaled up past that, you're gonna have to figure out ways to handle the fact that now for the GPUs to communicate, you have to do it over Infin band, which is still very fast, but is not as fast as ENV link.swyx: Is it like one order of magnitude, like hundreds or,Kyle: it's about an order of magnitude?Yeah. Okay. Um, soswyx: not terrible.Kyle: [00:30:00] Yeah. I, I need to, I need to remember the, the data sheet here, like, I think it's like about 500 gigabytes. Uh, a second unidirectional for ENV link, and about 50 gigabytes a second unidirectional for Infin Band. I, it, it depends on the, the generation.swyx: I just wanna set this up for people who are not familiar with these kinds of like layers and the trash speedVibhu: and all that.Of course.From Laptop to Multi NodeVibhu: Also, maybe even just going like a few steps back before that, like most people are very familiar with. You see a, you know, you can use on your laptop, whatever these steel viol, lm you can just run inference there. All, there's all, you can, youcan run it on thatVibhu: laptop. You can run on laptop.Then you get to, okay, uh, models got pretty big, right? JLM five, they doubled the size, so mm-hmm. Uh, what do you do when you have to go from, okay, I can get 128 gigs of memory. I can run it on a spark. Then you have to go multi GPU. Yeah. Okay. Multi GPU, there's some support there. Now, if I'm a company and I don't have like.I'm not hiring the best researchers for this. Right. But I need to go [00:31:00] multi-node, right? I have a lot of servers. Okay, now there's efficiency problems, right? You can have multiple eight H 100 nodes, but, you know, is that as a, like, how do you do that efficiently?Kyle: Yeah. How do you like represent them? How do you choose how to represent the model?Yeah, exactly right. That's a, that's like a hard question. Everyone asks, how do you size oh, I wanna run GLM five, which just came out new model. There have been like four of them in the past week, by the way, like a bunch of new models.swyx: You know why? Right? Deep seek.Kyle: No comment. Oh. Yeah, but Ggl, LM five, right?We, we have this, new model. It's, it's like a large size, and you have to figure out how to both scale up and scale out, right? Because you have to find the right representation that you care about. Everyone does this differently. Let's be very clear. Everyone figures this out in their own path.Nader: I feel like a lot of AI or ML even is like, is like this. I think people think, you know, I, I was, there was some tweet a few months ago that was like, why hasn't fine tuning as a service taken off? You know, that might be me. It might have been you. Yeah. But people want it to be such an easy recipe to follow.But even like if you look at an ML model and specificKyle: to you Yeah,Nader: yeah.Kyle: And the [00:32:00] model,Nader: the situation, and there's just so much tinkering, right? Like when you see a model that has however many experts in the ME model, it's like, why that many experts? I don't, they, you know, they tried a bunch of things and that one seemed to do better.I think when it comes to how you're serving inference, you know, you have a bunch of decisions to make and there you can always argue that you can take something and make it more optimal. But I think it's this internal calibration and appetite for continued calibration.Vibhu: Yeah. And that doesn't mean like, you know, people aren't taking a shot at this, like tinker from thinking machines, you know?Yeah. RL as a service. Yeah, totally. It's, it also gets even harder when you try to do big model training, right? We're not the best at training Moes, uh, when they're pre-trained. Like we saw this with LAMA three, right? They're trained in such a sparse way that meta knows there's gonna be a bunch of inference done on these, right?They'll open source it, but it's very trained for what meta infrastructure wants, right? They wanna, they wanna inference it a lot. Now the question to basically think about is, okay, say you wanna serve a chat application, a coding copilot, right? You're doing a layer of rl, you're serving a model for X amount of people.Is it a chat model, a coding model? Dynamo, you know, back to that,Kyle: it's [00:33:00] like, yeah, sorry. So you we, we sort of like jumped off of, you know, jumped, uh, on that topic. Everyone has like, their own, own journey.Cost Quality Latency TradeoffsKyle: And I, I like to think of it as defined by like, what is the model you need? What is the accuracy you need?Actually I talked to NA about this earlier. There's three axes you care about. What is the quality that you're able to produce? So like, are you accurate enough or can you complete the task with enough, performance, high enough performance. Yeah, yeah. Uh, there's cost. Can you serve the model or serve your workflow?Because it's not just the model anymore, it's the workflow. It's the multi turn with an agent cheaply enough. And then can you serve it fast enough? And we're seeing all three of these, like, play out, like we saw, we saw new models from OpenAI that you know, are faster. You have like these new fast versions of models.You can change the amount of thinking to change the amount of quality, right? Produce more tokens, but at a higher cost in a, in a higher latency. And really like when you start this journey of like trying to figure out how you wanna host a model, you, you, you think about three things. What is the model I need to serve?How many times do I need to call it? What is the input sequence link was [00:34:00] the, what does the workflow look like on top of it? What is the SLA, what is the latency SLA that I need to achieve? Because there's usually some, this is usually like a constant, you, you know, the SLA that you need to hit and then like you try and find the lowest cost version that hits all of these constraints.Usually, you know, you, you start with those things and you say you, you kind of do like a bit of experimentation across some common configurations. You change the tensor parallel size, which is a form of parallelismVibhu: I take, it goes even deeper first. Gotta think what model.Kyle: Yes, course,ofKyle: course. It's like, it's like a multi-step design process because as you said, you can, you can choose a smaller model and then do more test time scaling and it'll equate the quality of a larger model because you're doing the test time scaling or you're adding a harness or something.So yes, it, it goes way deeper than that. But from the performance perspective, like once you get to the model you need, you need to host, you look at that and you say, Hey. I have this model, I need to serve it at the speed. What is the right configuration for that?Nader: You guys see the recent, uh, there was a paper I just saw like a few days ago that, uh, if you run [00:35:00] the same prompt twice, you're getting like double Just try itagain.Nader: Yeah, exactly.Vibhu: And you get a lot. Yeah. But the, the key thing there is you give the context of the failed try, right? Yeah. So it takes a shot. And this has been like, you know, basic guidance for quite a while. Just try again. ‘cause you know, trying, just try again. Did you try again? All adviceNader: in life.Vibhu: Just, it's a paper from Google, if I'm not mistaken, right?Yeah,Vibhu: yeah. I think it, it's like a seven bas little short paper. Yeah. Yeah. The title's very cute. And it's just like, yeah, just try again. Give it ask context,Kyle: multi-shot. You just like, say like, hey, like, you know, like take, take a little bit more, take a little bit more information, try and fail. Fail.Vibhu: And that basic concept has gone pretty deep.There's like, um, self distillation, rl where you, you do self distillation, you do rl and you have past failure and you know, that gives some signal so people take, try it again. Not strong enough.swyx: Uh, for, for listeners, uh, who listen to here, uh, vivo actually, and I, and we run a second YouTube channel for our paper club where, oh, that's awesome.Vivo just covered this. Yeah. Awesome. Self desolation and all that's, that's why he, to speed [00:36:00] on it.Nader: I'll to check it out.swyx: Yeah. It, it's just a good practice, like everyone needs, like a paper club where like you just read papers together and the social pressure just kind of forces you to just,Nader: we, we,there'sNader: like a big inference.Kyle: ReadingNader: group at a video. I feel so bad every time. I I, he put it on like, on our, he shared it.swyx: One, one ofNader: your guys,swyx: uh, is, is big in that, I forget es han Yeah, yeah,Kyle: es Han's on my team. Actually. Funny. There's a, there's a, there's a employee transfer between us. Han worked for Nater at Brev, and now he, he's on my team.He wasNader: our head of ai. And then, yeah, once we got in, andswyx: because I'm always looking for like, okay, can, can I start at another podcast that only does that thing? Yeah. And, uh, Esan was like, I was trying to like nudge Esan into like, is there something here? I mean, I don't think there's, there's new infant techniques every day.So it's like, it's likeKyle: you would, you would actually be surprised, um, the amount of blog posts you see. And ifswyx: there's a period where it was like, Medusa hydra, what Eagle, like, youKyle: know, now we have new forms of decode, uh, we have new forms of specula, of decoding or new,swyx: what,Kyle: what are youVibhu: excited? And it's exciting when you guys put out something like Tron.‘cause I remember the paper on this Tron three, [00:37:00] uh, the amount of like post train, the on tokens that the GPU rich can just train on. And it, it was a hybrid state space model, right? Yeah.Kyle: It's co-designed for the hardware.Vibhu: Yeah, go design for the hardware. And one of the things was always, you know, the state space models don't scale as well when you do a conversion or whatever the performance.And you guys are like, no, just keep draining. And Nitron shows a lot of that. Yeah.Nader: Also, something cool about Nitron it was released in layers, if you will, very similar to Dynamo. It's, it's, it's essentially it was released as you can, the pre-training, post-training data sets are released. Yeah. The recipes on how to do it are released.The model itself is released. It's full model. You just benefit from us turning on the GPUs. But there are companies like, uh, ServiceNow took the dataset and they trained their own model and we were super excited and like, you know, celebrated that work.ZoomVibhu: different. Zoom is, zoom is CGI, I think, uh, you know, also just to add like a lot of models don't put out based models and if there's that, why is fine tuning not taken off?You know, you can do your own training. Yeah,Kyle: sure.Vibhu: You guys put out based model, I think you put out everything.Nader: I believe I know [00:38:00]swyx: about base. BasicallyVibhu: without baseswyx: basic can be cancelable.Vibhu: Yeah. Base can be cancelable.swyx: Yeah.Vibhu: Safety training.swyx: Did we get a full picture of dymo? I, I don't know if we, what,Nader: what I'd love is you, you mentioned the three axes like break it down of like, you know, what's prefilled decode and like what are the optimizations that we can get with Dynamo?Kyle: Yeah. That, that's, that's, that's a great point. So to summarize on that three axis problem, right, there are three things that determine whether or not something can be done with inference, cost, quality, latency, right? Dynamo is supposed to be there to provide you like the runtime that allows you to pull levers to, you know, mix it up and move around the parade of frontier or the preto surface that determines is this actually possible with inference And AI todayNader: gives you the knobs.Kyle: Yeah, exactly. It gives you the knobs.Disaggregation Prefill vs DecodeKyle: Uh, and one thing that like we, we use a lot in contemporary inference and is, you know, starting to like pick up from, you know, in, in general knowledge is this co concept of disaggregation. So historically. Models would be hosted with a single inference engine. And that inference engine [00:39:00] would ping pong between two phases.There's prefill where you're reading the sequence generating KV cache, which is basically just a set of vectors that represent the sequence. And then using that KV cache to generate new tokens, which is called Decode. And some brilliant researchers across multiple different papers essentially made the realization that if you separate these two phases, you actually gain some benefits.Those benefits are basically a you don't have to worry about step synchronous scheduling. So the way that an inference engine works is you do one step and then you finish it, and then you schedule, you start scheduling the next step there. It's not like fully asynchronous. And the problem with that is you would have, uh, essentially pre-fill and decode are, are actually very different in terms of both their resource requirements and their sometimes their runtime.So you would have like prefill that would like block decode steps because you, you'd still be pre-filing and you couldn't schedule because you know the step has to end. So you remove that scheduling issue and then you also allow you, or you yourself, to like [00:40:00] split the work into two different ki types of pools.So pre-fill typically, and, and this changes as, as model architecture changes. Pre-fill is, right now, compute bound most of the time with the sequence is sufficiently long. It's compute bound. On the decode side because you're doing a full Passover, all the weights and the entire sequence, every time you do a decode step and you're, you don't have the quadratic computation of KV cache, it's usually memory bound because you're retrieving a linear amount of memory and you're doing a linear amount of compute as opposed to prefill where you retrieve a linear amount of memory and then use a quadratic.You know,Nader: it's funny, someone exo Labs did a really cool demo where for the DGX Spark, which has a lot more compute, you can do the pre the compute hungry prefill on a DG X spark and then do the decode on a, on a Mac. Yeah. And soVibhu: that's faster.Nader: Yeah. Yeah.Kyle: So you could, you can do that. You can do machine strat stratification.Nader: Yeah.Kyle: And like with our future generation generations of hardware, we actually announced, like with Reuben, this [00:41:00] new accelerator that is prefilled specific. It's called Reuben, CPX. SoKubernetes Scaling with GroveNader: I have a question when you do the scale out. Yeah. Is scaling out easier with Dynamo? Because when you need a new node, you can dedicate it to either the Prefill or, uh, decode.Kyle: Yeah. So Dynamo actually has like a, a Kubernetes component in it called Grove that allows you to, to do this like crazy scaling specialization. It has like this hot, it's a representation that, I don't wanna go too deep into Kubernetes here, but there was a previous way that you would like launch multi-node work.Uh, it's called Leader Worker Set. It's in the Kubernetes standard, and Leader worker set is great. It served a lot of people super well for a long period of time. But one of the things that it's struggles with is representing a set of cases where you have a multi-node replica that has a pair, right?You know, prefill and decode, or it's not paired, but it has like a second stage that has a ratio that changes over time. And prefill and decode are like two different things as your workload changes, right? The amount of prefill you'll need to do may change. [00:42:00] The amount of decode that you, you'll need to do might change, right?Like, let's say you start getting like insanely long queries, right? That probably means that your prefill scales like harder because you're hitting these, this quadratic scaling growth.swyx: Yeah.And then for listeners, like prefill will be long input. Decode would be long output, for example, right?Kyle: Yeah. So like decode, decode scale. I mean, decode is funny because the amount of tokens that you produce scales with the output length, but the amount of work that you do per step scales with the amount of tokens in the context.swyx: Yes.Kyle: So both scales with the input and the output.swyx: That's true.Kyle: But on the pre-fold view code side, like if.Suddenly, like the amount of work you're doing on the decode side stays about the same or like scales a little bit, and then the prefilled side like jumps up a lot. You actually don't want that ratio to be the same. You want it to change over time. So Dynamo has a set of components that A, tell you how to scale.It tells you how many prefilled workers and decoded workers you, it thinks you should have, and also provides a scheduling API for Kubernetes that allows you to actually represent and affect this scheduling on, on, on your actual [00:43:00] hardware, on your compute infrastructure.Nader: Not gonna lie. I feel a little embarrassed for being proud of my SVG function earlier.swyx: No, itNader: wasreallyKyle: cute. I, Iswyx: likeNader: it's all,swyx: it's all engineering. It's all engineering. Um, that's where I'mKyle: technical.swyx: One thing I'm, I'm kind of just curious about with all with you see at a systems level, everything going on here. Mm-hmm. And we, you know, we're scaling it up in, in multi, in distributed systems.Context Length and Co Designswyx: Um, I think one thing that's like kind of, of the moment right now is people are asking, is there any SOL sort of upper bounds. In terms of like, let's call, just call it context length for one for of a better word, but you can break it down however you like.Nader: Yeah.swyx: I just think like, well, yeah, I mean, like clearly you can engage in hybrid architectures and throw in some state space models in there.All, all you want, but it looks, still looks very attention heavy.Kyle: Yes. Uh, yeah. Long context is attention heavy. I mean, we have these hybrid models, um,swyx: to take and most, most models like cap out at a million contexts and that's it. Yeah. Like for the last two years has been it.Kyle: Yeah. The model hardware context co-design thing that we're seeing these days is actually super [00:44:00] interesting.It's like my, my passion, like my secret side passion. We see models like Kimmy or G-P-T-O-S-S. I'm use these because I, I know specific things about these models. So Kimmy two comes out, right? And it's an interesting model. It's like, like a deep seek style architecture is MLA. It's basically deep seek, scaled like a little bit differently, um, and obviously trained differently as well.But they, they talked about, why they made the design choices for context. Kimmy has more experts, but fewer attention heads, and I believe a slightly smaller attention, uh, like dimension. But I need to remember, I need to check that. Uh, it doesn't matter. But they discussed this actually at length in a blog post on ji, which is like our pu which is like credit puswyx: Yeah.Kyle: Um, in, in China. Chinese red.swyx: Yeah.Kyle: It's, yeah. So it, it's, it's actually an incredible blog post. Uh, like all the mls people in, in, in that, I've seen that on GPU are like very brilliant, but they, they talk about like the creators of Kimi K two [00:45:00] actually like, talked about it on, on, on there in the blog post.And they say, we, we actually did an experiment, right? Attention scales with the number of heads, obviously. Like if you have 64 heads versus 32 heads, you do half the work of attention. You still scale quadratic, but you do half the work. And they made a, a very specific like. Sort of barter in their system, in their architecture, they basically said, Hey, what if we gave it more experts, so we're gonna use more memory capacity.But we keep the amount of activated experts the same. We increase the expert sparsity, so we have fewer experts act. The ratio to of experts activated to number of experts is smaller, and we decrease the number of attention heads.Vibhu: And kind of for context, what the, what we had been seeing was you make models sparser instead.So no one was really touching heads. You're just having, uh,Kyle: well, they, they did, they implicitly made it sparser.Vibhu: Yeah, yeah. For, for Kimmy. They did,Kyle: yes.Vibhu: They also made it sparser. But basically what we were seeing was people were at the level of, okay, there's a sparsity ratio. You want more total parameters, less active, and that's sparsity.[00:46:00]But what you see from papers, like, the labs like moonshot deep seek, they go to the level of, okay, outside of just number of experts, you can also change how many attention heads and less attention layers. More attention. Layers. Layers, yeah. Yes, yes. So, and that's all basically coming back to, just tied together is like hardware model, co-design, which isKyle: hardware model, co model, context, co-design.Vibhu: Yeah.Kyle: Right. Like if you were training a, a model that was like. Really, really short context, uh, or like really is good at super short context tasks. You may like design it in a way such that like you don't care about attention scaling because it hasn't hit that, like the turning point where like the quadratic curve takes over.Nader: How do you consider attention or context as a separate part of the co-design? Like I would imagine hardware or just how I would've thought of it is like hardware model. Co-design would be hardware model context co-designKyle: because the harness and the context that is produced by the harness is a part of the model.Once it's trained in,Vibhu: like even though towards the end you'll do long context, you're not changing architecture through I see. Training. Yeah.Kyle: I mean you can try.swyx: You're saying [00:47:00] everyone's training the harness into the model.Kyle: I would say to some degree, orswyx: there's co-design for harness. I know there's a small amount, but I feel like not everyone has like gone full send on this.Kyle: I think, I think I think it's important to internalize the harness that you think the model will be running. Running into the model.swyx: Yeah. Interesting. Okay. Bash is like the universal harness,Kyle: right? Like I'll, I'll give. An example here, right? I mean, or just like a, like a, it's easy proof, right? If you can train against a harness and you're using that harness for everything, wouldn't you just train with the harness to ensure that you get the best possible quality out of,swyx: Well, the, uh, I, I can provide a counter argument.Yeah, sure. Which is what you wanna provide a generally useful model for other people to plug into their harnesses, right? So if youKyle: Yeah. Harnesses can be open, open source, right?swyx: Yeah. So I mean, that's, that's effectively what's happening with Codex.Kyle: Yeah.swyx: And, but like you may want like a different search tool and then you may have to name it differently or,Nader: I don't know how much people have pushed on this, but can you.Train a model, would it be, have you have people compared training a model for the for the harness versus [00:48:00] like post training forswyx: I think it's the same thing. It's the same thing. It's okay. Just extra post training. INader: see.swyx: And so, I mean, cognition does this course, it does this where you, you just have to like, if your tool is slightly different, um, either force your tool to be like the tool that they train for.Hmm. Or undo their training for their tool and then Oh, that's re retrain. Yeah. It's, it's really annoying and like,Kyle: I would hope that eventually we hit like a certain level of generality with respect to training newswyx: tools. This is not a GI like, it's, this is a really stupid like. Learn my tool b***h.Like, I don't know if, I don't know if I can say that, but like, you know, um, I think what my point kind of is, is that there's, like, I look at slopes of the scaling laws and like, this slope is not working, man. We, we are at a million token con

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.On this week's episode we cover a wide range of topics, from defining "fitness" to the effects of drumming on health parameters, to some very interesting book recommendations. The wide ranging topics reflect the very philosophical approach that our guest, Michael Blevins, brings to fitness and coaching.Michael's journey began with trying to test his own limits in sports like sailing, rock climbing, and skateboarding. He then transitioned into exploring a range of martial arts disciplines, followed by an evolution into endurance sports, and then the fragility he felt from pushing those limits led him to incorporate weighlifting, crossfit, and strongman style training. He has competed in cycling, triathlon, crossfit, weighlifting, jiujitsu and more.Professionally, he has been a hairdresser, make up artist, photographer, worked in the fashion industry and on the stage... all ultimately developing a skill for building relationships that led him into coaching. He has coached actors preparing for film roles, military service members preparing for selections, and athletes competing at elite levels. Perhaps most notably he coached Henry Cavill leading up to Man of Steel, Batman vs Superman, and Justice League. He also coach both actors and stunt crew for 300: Rise of an Empire, and led a team development camp for the Atlanta Braves. We mentioned the strength manual he published in this conversation, which he's currently rewriting, he's host of the UNFVCKED podcast, and creator of We Are Ollin.As a starting point for some of his other content, you can find his article "What Is Fitness?" here.

Die Suid-Afrikaanse musikant Bobby van Jaarsveld het ouers gewaarsku nadat sy seun 90% van sy sig verloor het. Hy het onlangs op sosiale media gewaarsku oor 'n eenvoudige speelding waarmee baie kinders al jare lank speel. In sy Facebook-plasing het die sanger gedeel dat sy seun, Leben, 90% van sy sig verloor het nadat daar 'n laser in sy oog geskyn is. Breakfast with Martin Bester het met Van Jaarsveld oor die insident gesels.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.An Instagram post a few weeks ago about how nicotine reduces bone density and slows healing no matter how it's consumed (smoking, vaping, pouches, etc.) sparked some surprisingly strong reactions. Since neither of us are experts on either nicotine's health effects or bone health in general, we knew we needed to find an expert to fill us in.Dr. Jocelyn Wittstein is a Sports Medicine Orthopaedic Surgeon and Associate Professor of Orthopaedic Surgery at Duke University specializing in the care of adolescent and adult athletes. She cares for soccer, lacrosse, and basketball teams as a team physician and consults with may regional gymnastics facilities for care of high level gymnasts. In Dr. Wittstein's clinical practice, approximately half of her focus is on adolescent and adult knee injuries, with patellofemoral stabilization being a common procedure. In addition to her clinical and research work on the patellofemoral joint, Dr. Wittstein also is a co-investigator on NIH funded studies of biomechanical and biochemical factors contributing to post-traumatic arthritis after ACL reconstruction and meniscus surgery. She is passionate about optimizing patient outcomes and safe return to sport after knee injuries.We talked to her a bit after recording about why different bios of her discuss such different work, and it's because she wears so many hats. Some things that bio missed were her particular emphasis on shoulder instability, work on the unique challenges faced by female athletes across the lifespan, and work on mitigating age related issues... It might not be clear from the broad span of research, but first and foremost she is a Full time surgeon. She was a collegiate gymnast at Cornell University, and she is a mother of five.Dr. Wittstein mentioned the app OSTEO-GAINS which helps with progressive plyometric loading will the goal of increasing bone density.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.In this episode we're returning to one of the "squishiest" topics in military human performance: how to incorporate spirituality into the rest of the human performance domains. Fittingly, we have the chaplain who teamed up with Alex's team, so this is a continuation of many (off air) conversations over the last few years.Chaplain, Captain Conner J. Simms is an Chaplain assigned to the 412 Test Wing, Edwards Air Force Base, CA. He provides spiritual care and ensures the delivery of chaplain support to Airmen, Guardians, and DoD employees across two local area installations. As part of the wing staff at the 412 Test Wing, Chaplain Simms is tasked with advising command regarding the spiritual readiness, morale, ethics, and quality-of-life issues of all Air & Space Forces personnel and authorized DoD personnel.A native of Florida, Chaplain Simms currently resides in Edwards, CA, with his wife and young daughter.  He was commissioned as a Chaplain in April of 2018 and is endorsed by the International Council of Community Churches. Prior to his military service, Chaplain Simms spent over a decade in both local parish ministry and as an ICU/ER chaplain at a level one trauma medical center.He has served as a Traditional Reservist, IMA Reservists, & and now on Active Duty. His time in the ICU at an urban level one trauma hospital as well as two of his deployments (Kuwait – Operation Freedom's Sentinel, JBMDL – Operations Allies Welcome/Refuge) occured during the COVID pandemic. He also served as Lead Chaplain on a joint reserve mission in the Appalachian Mountains providing no-cost healthcare to the community.He is a three time graduate of Joint Special Operations University Chaplaincy programs, and is also a graduate of the Air Force Leader Development Course at Maxwell AFB, a course typically reserved for incoming squadron commanders and senior enlisted leaders. He has provided support to service members across six of the seven geographic combatant commands.One of our primary topics in this episode was the quantification of spirituality through the CHAMP-SOCOM Spiritual Fitness Scale, found here. You can also find a discussion of how to apply it here.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.You asked for it, here it is: a peptides episode. These are the hotness lately, getting tons of business and media attention (and anecdotal reports from athletes). But we constantly hear that there is no human safety or effectiveness data. So what are well intentioned consumers to do? To answer that question we have Dr. Rachele Pojednic back on the pod, and she is uniquely suited to talk about this issue. Rachele is a renowned expert, researcher, international speaker and thought-leader in nutrition and exercise science. Her primary roles are Adjunct Lecturer at Stanford University and Chief Science Officer at RestoreLabs. We don't just look at the current science on peptides, we also dive into what structural challenges have prevented more research in this area. As it turns out, we may be in a moment where regulatory changes may create some big opportunities in the very near future.Rachele mentioned examine.com as a great resource for analysis and summaries on nutrition and supplement research.We also touched on a recent article in Task & Purpose where a Marine Corps lawyer claimed his client unknowingly took prohibited peptides thinking they were approved supplements.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.We were recently invited to give the keynote presentation for the 2026 Fort Benning Human Performance Symposium. In the process of putting our talk together, we solidified our "core fore" concepts that help us filter through everything going on in the military human performance space. This led us to our main argument, which is that we should aim for "data informed" but not "data driven" to avoid falling into some common traps.Several people who couldn't attend the symposium asked if there was a way to listen to the talk, so we thought we'd just publish it as a podcast episode. Key topics we cover include: Nazareth Syndrome, Goodhart's Law, Mcnamara's Fallacy, and Hammond's Corollary (yes it's named after Drew). From there we dive into the challenge of defining "lethality" and what data can and can't do to measure it.Special shout out to SGT Donovan Saulsberry whose incredible voice you'll hear when he introduces us. Apparently he's the unofficial (or maybe official?) voice of Fort Benning. Let us know whether we should hire him to record a new intro for our podcast...

Andrea Speyerbach in gesprek met minister van Onderwijs, Cultuur en Wetenschap Gouke Moes.Wat mensen denken dat een minister doet en wat er werkelijk gebeurt zijn twee totaal verschillende dingen. In deze aflevering wordt afgerekend met de mythe van politieke almacht. Geen Haagse sprookjes, maar de rauwe werkelijkheid van bestuur, media-ophef en beperkte macht.Van academische vrijheid en ideologische druk tot stagevergoedingen, energiebeleid en de rol van ambtenaren. Vanuit persoonlijke ervaring van mbo'er tot minister wordt blootgelegd hoe beleid écht wordt gemaakt. Traag, conflictueus en altijd onder spanning.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.If you've been following any of the recent news around modernization of military dining facilities, there's a good chance Chef Irvine was behind the scenes making it happen. As you'll hear in this conversation he's deeply involved in these efforts, and he's doing it all for free. He started his career as a cook in the British Royal Navy, and after rising to culinary fame, he's giving back to service members in a variety of ways.Chef Robert Irvine is an English-American celebrity chef and talk show host who has appeared on and hosted a variety of Food Network programs including Dinner: Impossible, Worst Cooks in America, Restaurant: Impossible, A Hero's Welcome, Operation Restaurant, All-Star Academy, Guy's Grocery Games, Chopped: Impossible, and Restaurant Express. Irvine currently operates one restaurant, Fresh Kitchen by Robert Irvine, located within The Pentagon. 

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Among other things, this episode discusses the alcohol culture in the military, as well as the role of spirituality in holistic health. Lieutenant Commander (Retired) Drew Sinclair knows a thing or two about both of those, based on his unique and challenging personal journey. From enlisted Rescue Swimmer to OCS to a successful officer career, on the surface it seemed like Drew had it all figured out. But as a high functioning alcoholic, things below the surface weren't so great. After retiring Drew reinvented himself with a focus on a holistic approach to healthier living.Drew Sinclair is a retired U.S. Coast Guard Rescue Swimmer (#700) and Officer who spent more than two decades on the front lines of search and rescue. After overcoming childhood trauma, breaking generational cycles, and walking away from alcohol, Drew completely reinvented his life by rebuilding his identity, deepening his faith, and reshaping his priorities. He now leads a life centered on faith, family, fitness, and freedom.Drew travels full-time across America in an RV with his wife, three kids, two dogs, and a cat, documenting their journey while coaching busy men to reclaim their health through simple, disciplined hybrid training. His grounded, transparent storytelling, from mountain runs to mindset shifts, has inspired countless men to step back into a leadership role in their life and embrace strength and purpose.Drew's message is clear: life does not end at 40. It begins the moment you take ownership, make a decision, and commit to becoming the man you were meant to be. His story is not just about transformation. It is a blueprint for anyone ready to rebuild their life from the inside out.Follow Drew on his Instagram: https://www.instagram.com/thedrewsinclair/Our Drew (Hammond) mentioned discovering our guest Drew (Sinclair) through a facebook post, you can find that here: https://www.facebook.com/andrew.sinclair.1982/videos/when-i-retired-from-the-coast-guard-i-thought-everything-would-be-easyi-thought-/1296994402252868/We also make a couple references to a podcast episode more focused on his rescues, you can find that here: https://www.youtube.com/watch?v=IpwY-tCyqu0

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week we're bringing you an episode of the Captains and Coaches podcast hosted by Tex McQuilkin. Alex met Tex at the 1st Armored Division H2F summit at Ft Bliss, and Tex really appreciated the discussion of Nazareth Syndrome from Alex's presentation. He also gave a hell of a hands on coaching development session.At Captains & Coaches, they believe leadership isn't just taught—it's built through action, resilience, and teamwork. Their mission is to empower athletes, captains, and coaches to excel not just in sports, but in life. They don't just focus on physical training. They deliver a comprehensive approach to developing confidence, leadership, and mental toughness. Whether you're guiding a team, leading on the field, or supporting from the sidelines, their tailored programs are designed to meet your unique needs and drive long-lasting results.Tex McQuilkin brings over 15 years of transformative experience as the Founder and Leadership Strategist of Captains & Coaches. His unique methodology bridges the gap between physical excellence and leadership mastery, empowering athletes to become exceptional performers and influential leaders.From youth athletes to elite collegiate competitors and special operations forces, Tex has guided individuals and teams to breakthrough achievements through his signature blend of disciplined training and empathetic mentorship. As a former four-year starter and three-year team captain for Marymount University Men's Lacrosse, he intimately understands the challenges and opportunities that shape athletic leadership.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.In this week's episode we break down a number of major updates to the Holistic Health and Fitness (H2F) program that were presented by LTG David Francis at a recent AUSA Hot Topic session.Key topics discussed:- H2F Return on Investment (ROI) data- Athletic Trainer contracting- Planned expansion over the next few years- H2F in the Army Reserve and National Guard- Planned facilities, called Soldier Performance Readiness Centers (SPRCs)- Benefits observed from shifting PT time

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.For at least the last four decades every soldier who has taken a PT test has listened to a scripted set of instructions before starting. Those instructions, whether it was an APFT, ACFT, or AFT included one identical sentence:"The results of this test will give you and your commanders an indication of your state of fitness and will act as a guide in determining your physical training needs."But if you ask soldiers whether they've ever been given feedback on what their scores say about their training needs, most will say no. That's why we put together AFT Insight. It includes a calculator that will turn your raw performances into point scores, but it's also much more than a calculator. It will help you determine your "fitness archetype" and explain what that says about how you should train.Try AFT Insight here: https://aftinsight.com/

In this holiday season as we approach the end of the year, we're getting a little reflective. In this episode, we're looking back at the experiences that got us to where we are in terms of how we approach our own health and fitness. Hopefully along the way we highlight some helpful insights.MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This episode actually originated at AUSA during the Sergeant Major of the Army's Professional Development Forum. When CSM Curry submitted a question asking whether physical training during PME would actually be strenuous, I knew I wanted to hear his perspective. And then when I found out he already listened to this podcast, it was a done deal.Command Sergeant Major Shaun D. Curry is a native of Wauwatosa, WI and he enlisted in Army in February 2001. CSM Curry's previous assignments include 2nd Ranger Battalion (JBLM, WA), Human Resources Command / 75th Ranger Regiment Headquarters (Fort Knox, KY), 3rd Ranger Battalion (Fort Benning, GA), 6th Ranger Training Battalion (Eglin AFB, FL), Operations SGM of 2nd Battalion 5th SFAB (JBLM, WA), Battalion CSM of 1st Battalion 21 Infantry Regiment (Schofield Barracks, HI). His most recent assignment was the Brigade CSM for 3rd Infantry Brigade, 25th Infantry Division (Schofield Barracks, HI).CSM Curry completed all levels of the Noncommissioned Officer Professional Military Education up to and including the Sergeants Major Academy (Class 69). His additional military schooling includes Airborne School, the 75th Ranger Regiment Ranger Indoctrination Program (RIP), Static Line Jump Master School, Survival Evasion Resistance and Escape (SERE-C), Reconnaissance Senior Leaders Course (RSLC), Sexual Harassment/Assault Response and Prevention Foundation Course (SHARP), the Common Faculty Development Program Instructor Course (CFDPIC), Master Resilience Training Course (MRT), Combat Advisor Training Course (CAT-C) and Jungle Operations Training Course (JOTC). CSM Curry holds a Master of Science in Organizational Development and Leadership from the University of the Incarnate Word.CSM Curry deployed 14 times for a total of 61 months to both Afghanistan and Iraq in support of the Global War on Terror. His awards and decorations include the Legion of Merit, the Bronze Star Medal with “V” device, Bronze Star Medal (3rd award), Purple Heart, Meritorious Service Medal (5th award), Joint Service Commendation Medal, the Combat Infantryman Badge, Expert Infantryman Badge, the Ranger Tab, Master Parachutists Badge, and he is a member of the Order of Saint Maurice.CSM Curry is married and has three children.https://flowcode.com/p/wR6h4oKsT https://www.armyupress.army.mil/Journals/NCO-Journal/Muddy-Boots/Warfighting-Readiness/

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.It's always a great episode when you get to bring a personal friend on the pod! This week's guest was a very physically fit infantry officer when he and Alex were in the same battalion. He went on to join Special Forces, but ended up being medically retired. Holistic approaches to health and fitness have featured prominently in his transition and recovery journey, and we had him on to learn what that has looked likeWill Webb is a former U.S. Army Special Forces officer who now supports Veterans through holistic wellness and mental health work. Growing up in a military family and later graduating from West Point, Will spent a decade in service until a career-ending injury forced him to slow down and confront how deeply the mind, body, and spirit are connected.That transition led him on a personal healing journey, including a life-changing ayahuasca retreat in Peru that helped him reconnect with his spirituality, face his shadow, and find a new path aligned with service and purpose.Will now works with Heroic Hearts Project and the Truxtun Foundation, two nonprofits offering holistic options for Veterans. He's also earning his Master's in Clinical Psychology at Antioch University, specializing in Spiritual and Depth Psychology.Based in Venice, California, Will shares the practices that supported his own recovery - surfing, yoga, sound healing, and transformational life coaching - helping others regulate their nervous systems, cultivate resilience, and step into alignment with their truth as they navigate the next chapter in their lives.elevatedventuring.comOr email him at: will.webb@heroicheartsproject.orgHeroic Hearts Project and Truxtun Foundationhttps://warriorside.org/ Learn about the use of MDMA with soldiers in Ukraine here: https://www.lucid.news/maps-mdma-assisted-therapy-ukraine-war/ Some examples of Icaros can be found here: https://open.spotify.com/playlist/2ZyDBxPNqRvAUJQEk2hFVC?si=e51a305efd38464f

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This episode covers two primary topics: First, we chat about wearable technology, its evolving accuracy, and how to implement it to change health and fitness behaviors at scale. Second, we discuss the challenges of doing business with the military, particularly in the technology space where innovation is rapid and constant.Geoff Wylde leads ŌURA's Human Performance team, delivering technology solutions that help tactical and critical workforce communities optimize physical and mental readiness. Beginning with an illness‑monitoring initiative for the U.S. Air Force, his team has expanded Oura's programs for first responders, active‑duty military, and veterans to include athletic performance, fatigue risk management, chronic stress management, and holistic health and well‑being. Before ŌURA, Geoff led programs on technology policy and industrial strategy at the World Economic Forum and spent a decade in strategy consulting at PricewaterhouseCoopers.Outside of work, Geoff volunteers and serves on the board of Healing Waters, a nonprofit that brings chronically ill and at‑risk communities into nature through camping and whitewater rafting in Northern California.Here's a peer reviewed validation study that compares the sleep/HRV accuracy of Garmin Fenix 6, Oura Generation 3, Oura Generation 4, Polar Grit X Pro, and Whoop 4.0.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Dr. Rich Willy is a new Associate Professor in the PhD program in the School of Health and Rehabilitation Sciences at The Ohio State University. He holds a PhD in Biomechanics and Movement Science from the University of Delaware and a Master's of Physical Therapy from Ohio University. He is a licensed physical therapist with over two decades of clinical and academic experience. His research focuses on the biomechanics of running-related injuries, bone stress injuries, and rehabilitation strategies for tactical and athletic populations.Dr. Willy has authored more than 80 peer-reviewed publications and book chapters, and his work has been featured in high-impact journals such as British Journal of Sports Medicine, Journal of Orthopaedic & Sports Physical Therapy, and American Journal of Sports Medicine. Dr. Willy contributes to clinical practice guidelines for patellofemoral pain and running injuries. He is a frequently invited speaker at national and international conferences, including symposia for the US and International Olympic Committees, NBA teams, and sports medicine meetings.His research has been supported by the Department of Defense and APTA Orthopaedics, among others. Current projects include optimizing load carriage biomechanics, developing sex-specific training interventions, and advancing wearable technologies for injury prevention and rehabilitation.He and his wife also run Montana Running Lab, a hugely valuable resource curating the best clinical evidence for athletes and rehab professionals. We highly recommend their instagram as an evidence based source of information. We'll talk a bit about some of the resources available there at the end of this episode.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week we're back for round 2 with Brian Carroll. You should definitely go back and listen to our first episode with him before you dive into this one. We brought Brian back to ask a variety of follow up questions, some our own and some provided by you guys in your responses to the initial episode. The most frequent piece of feedback we got was about his interpretation on MRIs. We get to that a little later in the episode, so stick with it.Other topics for this discussion include the psychology of recovering from injury, the complexity of the relationship between pain and injury, and for our video viewers he even breaks out some spine models to demonstrate a few of the concepts he discusses.He mentions Michael Shacklock and neurodynamics a few times, if you want to learn more about that check out their page. He also mentions a few videos they're making about MRI interpretation, here are links to the first two:"MRI Case Study: Why They Matter and Why They Don't Tell the Whole Story""Does Your MRI Tell the Whole Story?"Brian also asked that we include the following clarification in response to his exchange with Alex about Elon Musk's role in SpaceX engineering (this is copied directly from Brian's email): "Elon Musk actually does design rockets and create technology for various aspects of rocket science and aerospace advancement. He oversees engineering and development projects for SpaceX yet only holds bachelor degrees in both economics and engineering from the University of Pennsylvania. He began (but did not complete) a PhD program at Stanford before launching PayPal etc, he lasted 2 days in the program. He credits mentors and reading many books and studies, as have I with MRi's. "

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠⁠get your first 7 days of training with us FREE by clicking here.⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week's episode is a crossover with Ray and Travis over at Coaches' AAR. If you're in the tactical human performance space, they're much more focused on the practical aspects of coaching. Where we get into strategic, policy, and cultural issues, they keep it focused on the practitioner's perspective.Go follow Coaches' AAR on the following platforms!Coaches' AAR on InstagramCoaches' AAR on Spotify

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. ⁠Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic. ⁠ MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠⁠⁠TrainHeroic and you can ⁠get your first 7 days of training with us FREE by clicking here.⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.On September 30th, Secretary of War Pete Hegseth spoke to an audience of the US military's most senior leaders, announcing a sweeping set of policy reforms. These included a heavy emphasis on fitness, both force-wide and for particular education and training courses. In this episode we review many of those changes. We start with a rapid overview of the "non-fitness" changes, like grooming standards and adjustments to investigation processes. After than we move into a more focused discussion of the fitness-specific changes and the ramifications these will have.Drew mentioned our "Fitness Aptitude Test" (FAT), if you want to check that out and try it for yourself, here's a blog post about it.

MOPs & MOEs is powered by TrainHeroic, the best coaching app on the planet. Click here to get 14 days FREE and a consult with the coaches at TrainHeroic to help you get your coaching business rolling on TrainHeroic.  MOPs & MOEs delivers our training through ⁠⁠⁠⁠⁠⁠TrainHeroic and you can get your first 7 days of training with us FREE by clicking here.To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.On this week's episode we're breaking down two recently released memos that establish guidelines for medical oversight of the Army's H2F teams. Across the military embedded human performance is becoming a bigger and bigger part of how we train and care for service members, but having so many medical providers operating outside the clinic creates some policy challenges. Whether it's "integrated operational support," H2F, or something else, policies like these are popping up across multiple services. The first memo is "Credentialing Policy for Certified Athletic Trainers" and it clears up some grey area about how ATs fit into the Army's medical system.The second is "U.S. Army Medical Command Responsibility for Clinical Quality Management of Holistic Health and Fitness Program" and it applies to all medical providers on H2F teams (not just ATs). It also has important consequences for facility standards in the spaces that H2F teams provide medical care.

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.If you enjoy the discussion on stoicism, pre-order our guest's book now! On this week's episode we're diving deep (get it?) on an extremely unique corner of the military human performance world. Submariners operate in an environment unlike any other, and trying to maintain health and fitness under those conditions involves a variety of challenges. Our guest is uniquely qualified to provide insights about life in this world.Commander William Spears is a submarine warfare officer in the U.S. Navy. A native of Pineville, Louisiana, William enlisted in the U.S. Navy's nuclear propulsion program after high school. Upon completing technical training, he was admitted to the United States Naval Academy, graduating in 2008 with a degree in Mechanical Engineering and commissioning as an officer. Today, he also holds defense-related master's degrees from the Naval Postgraduate School, the Air Command and Staff College, and the Eisenhower School for National Security and Resource Strategy.William has served in nuclear-powered submarines across a variety of classes and mission profiles, including duty as the Weapons Officer of a fast-attack submarine and the Executive Officer of a Trident missile submarine. Ashore, he has served as a tactical evaluator on an inspection team responsible to assess the combat readiness of U.S. submarines, and he currently works in the Office of the Secretary of Defense (OSD CAPE) in the Pentagon. He will return to sea duty in the summer of 2026.He has his own passion for exercise, which is not  always a clean fit with the culture of the submarine community, as you'll hear in this episode.He also writes on leadership, ethics, and military topics. His book Stoicism as a Warrior Philosophy releases in the US in November.

MOPs & MOEs is ⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Our guest for this episode was brought low by persistent back pain despite being one of the strongest people on the planet. But through his journey to find a solution for that pain, he found a new mission to help others avoid the mistakes he made in the past. His book, "The Gift of Injury: The Strength Athlete's Guide To Recovering From Back Injury To Winning Again" co-written with Dr. Stuart McGill is highly praised by athletes and rehab professionals alike.Since entering his first powerlifting meet in 1999, Brian Carroll has since risen to the absolute pinnacle of the sport. Brian totaled 2730 at 275 and 2651 at 242 with more than ten times his body weight in three different classes (220, 242, 275), and both bench pressed and deadlifted over 800 pounds in two other weight classes. He's totaled 2600 over 20 times in 2 different weight classes in his career. He has squatted over 1000lbs more than 60 times and was the first man to squat of 1300lbs.After ten years of high-level powerlifting competition and an all-time World Record squat at 220 with 1030, in 2009, Brian was competing for a Police academy scholarship. On a hot and humid July morning, Brian, hurdling over a barricade at 275lbs, landed on, fell, and hurt his back. The resulting back pain has been a huge part of his professional journey, leading him to seek out Stu McGill in 2013 who he latered co-authored The Gift of Injury with in 2017. It's worth nothing that his 1306lb squat came AFTER all of the back back and working with Dr. McGill.He owns his own coaching and consulting business Power Rack Strength where he works with a wide range of athletes and frequently delivers professional education for rehab professionals.Abnormal magnetic-resonance scans of the lumbar spine in asymptomatic subjects. A prospective investigation.""Magnetic resonance imaging of the lumbar spine in people without back pain""Association of Lumbar MRI Findings with Current and Future Back Pain in a Population-based Cohort Study""Systematic Literature Review of Imaging Features of Spinal Degeneration in Asymptomatic Populations""Does magnetic resonance imaging predict future low back pain? A systematic review"

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Drew is on a side quest to dive into niche areas of military human performance, and this episode is extremely niche. A few episodes ago we mentioned the fitness test for the Marine Corps Body Bearers (which includes bench press, squat, barbell curls, and behind the neck overhead presses). We got some of the details wrong, and thanks to the our audience we were put in touch with a former bearer to set the record straight.Billy Lashley served as a United States Marine and World Famous Body Bearer from 2019 to 2023. In that time, he performed 625 funerals — including high-profile state and joint services — and took part in Friday night parades and wreath-laying ceremonies at Marine Barracks Washington. His roles within the section included recruiter and instructor, giving him a front-row seat to both the weight of the mission and the responsibility of preparing others for it.This is small and extremely unique community that upholds some elite performance standards, and our conversation spans recruiting/testing standards, training protocols, and how leaders in the organization maintain the culture.Billy has much longer hair now, and is even more jacked. Follow him on Instagram at @blashley96.Here's some official Marine Corps media diving into the organization if you want more after listening to the episode:Marine Corps Body Bearers Part IMarine Corps Body Bearers Part II

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.In this week's episode we're breaking down some recently published research. Specifically, "Personalized, Evidence-Informed Training Plans and Exercise Prescriptions for Performance, Fitness and Health" by Henning Wackerhage and Brad Schoenfeld. Up front, the article itself is an opinion piece, but it's based on an extensive review of the literature, and provides thorough citations. It's a useful article specifically because it synthesizes so much evidence into some practical guidelines for coaches.The authors advocate for an athlete, client and patient-centered approach whereby an individual's needs and abilities are the main consideration behind all decision-making. They also lay out a subjective, pragmatic six-step approach that details how to write a training plan or exercise prescription that is partially based on scientific evidence.

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Find ⁠Tactical Alpinism on Instagram here⁠You can find Mauri's podcast on ⁠Spotify⁠ or on ⁠Apple PodcastsWe recently had Mauri on to discuss his research on lactate threshold based training, but after he joined the conversation on our Discord we found out we missed an even more important topic. Fitness plays a huge role in ROTC cadets' ranking, and those rankings determine their choices of component and branch. As an instructor, Mauri's human performance focused approach dramatically enhanced his school's outcomes, so in the conversation we explore what worked.We discussed news of a cadet's death at Advanced Camp, you can find that story here.You can find coverage of the ROTC "rebalancing and optimization" (downgrading programs) here.

Miss Jill Moes from Divich Elementary just won a $50 Amazon Gift Card courtesy of Best Mattress to help clear her Amazon wish list!

On July 31st President Trump signed an executive order re-establishing the President's Council on Sports, Fitness, and Nutrition and directing the new council to develop a proposal on bringing back the Presidential Fitness Test. This test figures prominently in the childhood memories of many Americans, with pride for some and trauma for others. In this episode we break down the latest news within the decades of historical context that got us here. You can read "The Soft American" here (we consider it mandatory reading for MOPs & MOEs followers)For background on our mention of physical education in Europe (especially the Turnverein movement) check out our episode History of Army Fitness with Dr. EastFor some context on the President's Council on Sports, Fitness and Nutrition, check out our episode with former council member Rob WilkinsWe mentioned Maintenance Phase's episode on the PFT and you can find that hereWe also mentioned a similar perspective on the test presented in this article on VoxDrew referenced the official history of the council provided on the HHS websiteAlex referenced the FitnessGram teacher training which provides an overview of the program This article highlights the lack of academic scrutiny focused on physical education, including FitnessGramThe source of the claim that the average school budget for physical education is $764 annually is this article from TimeYou can read the La Sierra High School Physical Education handbook here, including the basic philosophies as well as the specific events and standards

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This episode includes a reference to this Defense Health Agency report that found that the more H2F resources provided for Reserve soldiers, the better results they saw.If you follow the MOPs & MOEs blog, you already know this week's guests from things like his 5 part series "The Other 28 Days" on how to implement human performance for part time service members or his "Maximizing Fitness Efficiency" piece on minimal effective dose training. Members of our discord server know he's always bringing research citations to the conversations happening there.Mark Christiani is an Army Veteran who served in Ranger Regiment before transitioning into the human performance space. He currently works with O2X as an On-Site Human Performance Specialist at the 81st Readiness Division of the Army Reserve. Mark served as the Brigade Lead Strength and Conditioning Coach for GAP Solutions for not just any brigade, but 44th Medical Brigade where Drew works. He holds a Master of Science in Sports Medicine from Georgia Southern University and is a Certified Strength and Conditioning Specialist (CSCS) and Registered Strength and Conditioning Coach (RSCC). 

MOPs & MOEs is ⁠⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.Find Tactical Alpinism on Instagram hereYou can find Mauri's podcast on Spotify or on Apple PodcastsYou can download a copy of Mauri's thesis, “Relationship Between the Lactate Thresholds and Endurance Performance in Trained Runners” hereIn this episode we're returning to the topic of how mountain athletes and tactical athletes have similar fitness demands. Mauri is particularly qualified to discuss this topic since he is a bit of both. His perspectives include being an infantry officer, alpinist, coach to endurance athletes, certified mountain guide, and more.Mauri served in a multitude of leadership roles as an infantry officer in the US Army. During that service he applied many advanced planning and navigation techniques to make mountain missions successful. He adapted the military's operational planning process for use in the mountains by combining military planning and navigation techniques with mountain objectives. He now leads Tactical Alpinism, where he provides training and education for both tactical professionals and civilians pursuing high levels of performance in the mountains. This includes both physical training and technical mountain navigation.

MOPs & MOEs is ⁠⁠⁠⁠⁠powered by TrainHeroic!⁠⁠⁠⁠⁠To continue the conversation, ⁠⁠⁠⁠⁠join our Discord!⁠⁠⁠⁠⁠ We have experts standing by to answer your questions.This week's episode is just the two of us, and we're discussing a topic that we've referenced a few times on social media: Nazareth Syndrome. One of the simplest ways to explain this phenomenon is "nobody trusts the hometown kid." The origins of this idea are biblical (Jesus was rejected by his own community because to them he was just the carpenter they knew), but the applications are very practical. Have you ever seen a leader latch onto an idea from a guest speaker or outside consultant that their subordinates have been trying to explain for ages? That's because human nature makes us more receptive to these messages from outsides than from people we're too familiar.In this conversation we break down how this affects the military, and specifically how it plays out in human performance settings (both within teams, and between the teams and the units they support).