Podcast appearances and mentions of Brian Levin

For all those who missed out on London, see you in Miami next week!Notion, the knowledge work decacorn, has been building AI tooling since before ChatGPT, with many hits from Q&A in 2023 and unified AI in 2024 and Meeting Notes in 2025. At the end of their last Make user conference, Ryan Nystrom teased Notion 3.0's Custom Agents - and they are finally embracing the Agent Lab playbook!Sarah Sachs and Simon Last of Notion join us for a deep dive into how Notion built Custom Agents, why it took years and multiple rebuilds to get right, and what it means to turn a productivity tool into an agent-native system of record for enterprise work.We go inside the product, engineering, evals, pricing, and org design decisions behind one of the most ambitious AI product efforts in software today — from early failed tool-calling experiments in 2022 to agent harnesses, progressive tool disclosure, meeting notes as data capture, and the long-term vision for software factories and agentic work.We discuss:* Sarah and Simon's path to launching Notion Custom Agents, and why the feature was rebuilt four or five times before it was ready for production* Why early agent attempts failed: no tool-calling standard, short context windows, unreliable models, and too much complexity exposed to the model* The “Agent Lab” thesis: not just wrapping a model, but understanding how people collaborate and building the right product system around frontier capabilities* How Notion thinks about roadmap timing: not swimming upstream against model limitations, but also building early enough that the product is ready when the models are* Why coding agents feel like the kernel of AGI, and how Notion is thinking about “software factories” made up of agents that spec, code, test, debug, review, and maintain codebases together* How Sarah runs AI engineering at Notion (“notes from Token Town”): objective-setting over idea ownership, low-ego teams comfortable deleting their own work, and a culture designed to swarm around fast-changing opportunities* The “Simon Vortex,” company hackathons, and why security gets pulled in early rather than late* How Notion organizes AI: core AI capabilities and infrastructure, product packaging teams, and a broader company mandate that every product surface must increasingly work for both humans and agents* Why prototypes have become much easier to build internally, and how “demos over memos” changes product development inside a tool the whole company already uses every day* Notion's eval philosophy: regression tests, launch-quality evals, and “frontier/headroom” evals that intentionally only pass ~30% of the time so the company can see where model capabilities are going* What a “Model Behavior Engineer” is, and why Notion treats eval writing, failure analysis, and model understanding as a distinct function rather than just software engineering* The changing role of software engineers in the age of coding agents, and why the new job looks less like typing code and more like supervising a rigorous outer system of agents, PRs, and verification loops* How the “software factory” should work: specs, self-verification, bug flows, subagents, and minimizing human intervention while preserving the invariants that matter* A live walkthrough of a Notion Custom Agent handling coworking space tenant applications by triaging email, enriching applicants with web search, and writing structured data into a Notion database* How agents compose inside Notion: shared databases as primitives, agents invoking other agents, “manager agents” supervising dozens of specialized agents, and memory implemented simply as pages and databases* Notion's take on MCP vs CLI: why Simon is bullish on CLI's self-debugging nature, where MCP still makes sense, and how Sarah thinks about capability, determinism, permissioning, and pricing alignment* The evolution of Notion's internal agent harness: from early JavaScript coding agents, to custom XML, to Markdown and SQL-like abstractions, to tool definitions, progressive disclosure, and a much shorter system prompt* Why Notion cares about teaching “the top of the class,” building for sophisticated operators rather than abstracting away too much capability for everyone* How agent setup works today: agents that can configure themselves, inspect their own failures, and edit their own instructions — with guardrails around permissions* How Notion prices Custom Agents: credits as an abstraction over tokens, model type, serving tier, web search, and future sandbox costs; why usage-based pricing was necessary; and how “auto” tries to match the right model to the right task* Why Notion is not eager to train a foundation model, where they do fine-tune and optimize today, and why retrieval/ranking is one of the most important investment areas as more searches come from agents rather than humans* Why Meeting Notes became one of Notion's strongest growth loops: not just as transcription, but as high-signal data capture that powers search, custom agents, follow-up workflows, and the broader system of record for company collaboration* Why Notion is more interested in being the place where collaboration data lives than in building hardware themselves — and how wearables or other capture devices may eventually feed into that systemSarah SachsLinkedIn: https://www.linkedin.com/in/sarahmsachsX: https://x.com/sarahmsachsSimon LastLinkedIn: https://www.linkedin.com/in/simon-last-41404140X: https://x.com/simonlastFull Video EpisodeTimestamps* 00:00:00 Introduction and launching Notion Custom Agents* 00:01:17 Why Notion rebuilt agents four or five times* 00:03:35 Building for where models are going, not just where they are* 00:05:32 The Agent Lab thesis, wrappers, and product intuition* 00:08:07 User journeys, leadership, and low-ego AI teams* 00:13:16 The Simon Vortex, hackathons, and bringing security in early* 00:16:39 Team structure, demos over memos, and building for agents* 00:20:25 Evals, Notion's Last Exam, and the Model Behavior Engineer role* 00:27:37 Evals as an agent harness and the changing role of software engineers* 00:30:42 The software factory: specs, verification, and agent workflows* 00:32:18 Live demo: a custom agent for coworking space applications* 00:35:08 Composing agents, manager agents, and memory as pages* 00:38:15 Notion Mail, Gmail, native integrations, and tools* 00:39:43 MCP vs CLI and the cost of capability* 00:44:13 When Notion uses MCP vs building its own integrations* 00:47:43 The history of Notion's agent harness rebuilds* 00:55:35 Power users, public tools, and the setup agent* 00:58:01 Self-fixing agents, permissions, and “flippy”* 01:01:13 Pricing, credits, and choosing the right model automatically* 01:09:01 Why Notion isn't training its own frontier model* 01:14:07 Retrieval, ranking, and search built for agents* 01:17:27 Meeting Notes as data capture and workflow automation* 01:21:18 Wearables, hardware, and Notion as the system of record* 01:23:45 OutroTranscript[00:00:00] Alessio: Hey everyone. Welcome to the Latent Space podcast. This is Alessio founder of Kernel Labs and I'm joined by swyx, editor of the Latent Space.[00:00:11] swyx: Hello. Hello. We're back in the beautiful studio that, uh, Alessio has set up for us with Simon and Sarah from Notion. Welcome.[00:00:18] Sarah Sachs: Thanks for having us.[00:00:19] Alessio: Thanks for having us. Yeah.[00:00:20] swyx: Congrats on the launch recently the custom agents, finally it's here. How's it feel?[00:00:26] Sarah Sachs: We ship things slowly. So it had been in Alpha for a little bit and at the point at which is it's an alpha, um, there's a group of people that are making sure it's ready for prod, and then there's a group of people working on the next thing.So sometimes some of these launches are a bit delayed satisfaction, so it's quite nice to remind yourself all the work you did because we do have a habit of like. Being two or three milestones ahead. Uh, just ‘cause you have to be, you know, you can't get complacent. Um, but it's been great that people understood how this is helpful.And I think that's just easier in general building AI tools today than it was two, three years ago. People kind of get it and so that user education, um, there's just, it was our most successful launch in terms of free trials and converting people and things like that. It was really successful, so yeah.But there's a lot to build.[00:01:12] swyx: Making it free for three months helps.[00:01:16] Sarah Sachs: Yep.[00:01:17] Simon Last: It was definitely super exciting for me because it's probably the fourth or fifth time that we rebuilt that.[00:01:22] swyx: Yes.[00:01:23] Simon Last: And I mean,[00:01:24] swyx: you've been building this since like 20, 22.[00:01:26] Simon Last: Yeah, I mean, like, it was even right when we got access to like GPT four in late 20 22, 1 of the first ideas we had is like, oh, okay, let's make an agent that I, we used the word assistant at the time, there wasn't really the word, the word agent yet, but, oh, we'll give an access to all the tools the notion can do, and then it, we run in the background like, like do work for us.And then we just tried that many times and it just. Was too early. Um,[00:01:48] swyx: I need to force you to like double click on that. What is too early? What didn't work?[00:01:52] Sarah Sachs: We were fine to, like, before function calling came out. We were trying to fine tune with the Frontier Labs and with fireworks, like a function calling model on notion functions.This is right when I joined. I joined because, um, we needed a manager as Simon was needed to be able to go on vacation. So, uh, that's, that's around when I joined, so you can speak much more to it.[00:02:11] Simon Last: Yeah, we did partnerships with both philanthropic and open AI at different times, uh, to try to, at the time the, I mean, when we first tried, there wasn't even a constant of like tools yet.We, we sort of designed our own like, like tool calling framework and then we tried to fine tune the models to, uh, to use it over multiple turns. Um, and because it, it didn't work well out the box, I think. Yeah. The models are just too dumb and the context thing was also way too short.[00:02:37] Alsesio: Yeah.[00:02:37] Simon Last: Um, and yeah, we just kind of banged our head against it for a long time.Uh, unfortunately it was always like, there was always like sort of. Glimmers that it was working, but um, it never felt quite robust enough to be like a useful, delightful thing. Um, until I would say, uh, the big unlock was probably like Sonic 3.6 or seven, uh, early last year. And that's when we started working on our agent, which we shipped last year.Um, and then, and then uh, uh, custom agents, kinda a similar capability and that, that one just took longer because we, we just wanted to get the reliability up a lot higher. ‘cause it's actually running in the background.[00:03:14] Sarah Sachs: And the product interface of like permissions and understanding, you know, this custom agent is shared in a Slack channel with X group of people and has access to documents that are surfaced to Y group of people.And the intersect experts, Y might not be whole. And so how do you build the product around making sure administrators understand that permissioning took multiple swings.[00:03:35] Alsesio: Everything is hard back at the end of the day. Yeah. I'm curious, like when the models are not working, how do you inform the product roadmap of like, okay, we should probably build, expecting the models to be better at some reasonable pace, but at the same time we need to, you know, you had a lot of customers in 2022.It's not like you were a new company or like no user base.[00:03:54] Simon Last: Yeah, I mean I think there's always the balance of, you know, like you want to be a GI pilled and thinking ahead and building for where things are going. Uh, but also you wanna be like shipping useful things. And so we always try to like, like keep a balance there.You know, we. We try to take clear, like a portfolio approach. You know, we're always working on multiple projects and, and we're always trying to work on, you know, maintaining things where that have already shipped, like, like shipping new things that are like eminently working well and make them really good.And, and then we wanna always have a few projects that are a little bit crazy. Um,[00:04:23] Alsesio: and what are the a GI peel projects that you have today? I'm curious about, uh, you don't have to share exactly what you're working on, but I'm curious what are things today that maybe in 18 months people will be like, oh, obviously this was gonna work[00:04:35] Sarah Sachs: 18 months.[00:04:37] Alsesio: Yeah, 18 months is, you know,[00:04:37] Sarah Sachs: it's a long time and Yeah. Yeah.[00:04:39] Simon Last: I mean, there's a number of things happening. I think one thing that's becoming more clear is I think like, like, uh, coding agents are the kernel of EGI, sort of, everything is a coding agent. Mm-hmm. I think that's, that's sort of one, one direction.Um, and then, yeah, the exciting thing about that is sort of your agent can sort of bootstrap its own software and capabilities and actually debug and maintain them. And so yeah, we're, we're, we're thinking a lot about that. And then, yeah, like, like another category of things that I'm, I'm really excited about is like, uh, we call the software factory also.People are using this, uh, this, this sort of word. Um, basically it just means can you create sort of like a, as automated as possible, a workflow for developing debugging. Mm-hmm. Merging, reviewing, and maintaining a code base and a service where there's a bunch of agents working together inside, and like, like how does that work?[00:05:28] Sarah Sachs: If you think back to your initial question, like, why did this take so long? I think something,[00:05:32] swyx: I didn't say that, but Yes. Okay. Go ahead.[00:05:34] Sarah Sachs: Why, what, what changed over the three and half years of trying[00:05:37] swyx: it? Exactly. Right. Because most people always say like, it didn't work yet. Then reasoning models came, then it worked.I was like, okay, let's go a little[00:05:43] Sarah Sachs: bit. That's, I mean, that's part of it, but I think the other part of it that I actually think is really what will set notion apart for every new capability is we have like. Two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream.So like quickly realizing if you're just pressing against model capabilities versus not exposing the model to the right information, not having the right infrastructure set up. That and of itself is the skill of intuition. And the second is to see, okay, you're not swimming upstream. Which direction is the river flowing and what is like, how do we think ahead about the product and start building it even if it's not great yet, so that when it is there, we're ready for it.Right? And like those can sometimes feel like counterintuitive things. Like we can be trying to fine tune a tool calling model when they don't exist yet. And that the trick is to not do that for too long, but realize that there was something there. And we've had a lot of things which like, um, we're just like not swimming in the right direction with the streams.I think we had multiple versions of transcription before we got meeting notes, right? Oh, I gotta talk[00:06:39] swyx: about that. Yeah.[00:06:40] Sarah Sachs: Yeah. Um, and so. I, I, I think that like we, we really closely partner with the Frontier Labs on capabilities and we also have to have strong conviction on, as those capabilities move.Notion is about being the best place for you to collaborate and do your work. And how does that narrative change if the way that we work changes?Yeah.[00:06:58] swyx: Yeah. You told me you were a fan of the Agent Lab thesis, and this is, this is kind of it, right?[00:07:02] Sarah Sachs: Right. I show that thesis to so many candidates. Like I have it as like micro chrome autofill.Um, at this point, like it's one of my most visitations[00:07:10] swyx: because like, is this the, here's why you should work in notion and not open, open eye. I, it's like,[00:07:14] Sarah Sachs: here's, here's what's different about it.[00:07:16] swyx: Yeah.[00:07:16] Sarah Sachs: And here's why. It's not just a rapper. I actually think more and more people understand it's not just a wrapper.[00:07:21] swyx: Yeah.[00:07:22] Sarah Sachs: Um, and by the way, like in the beginning, parts of what we build are wrappers on functionality. That works well, of course, but that's not really the most, um. I would say that's not the product that, that drives revenue. And that's not necessarily always what users need.[00:07:35] swyx: I mean, you know, notion is the AWS wrapper, but like the, the wrapper is very beautiful and like very, very well polished.So[00:07:40] Sarah Sachs: like the analogy,[00:07:41] swyx: like[00:07:42] Sarah Sachs: the analogy that I've been coming back to his Datadog in AWS[00:07:45] swyx: Yeah.[00:07:46] Sarah Sachs: So, uh, Datadog could not exist with, without cloud storage. Right. That it's kind of fundamental that that works. Um, and AWS has like a CloudWatch product, but Datadog is an expert on understanding how people want observability on the products they launch.And we're experts in understanding how people wanna collaborate, and that's really where our expertise lies.[00:08:04] swyx: Totally.[00:08:04] Sarah Sachs: Um, regardless of the tools that we use,[00:08:07] Alsesio: I'm kind of curious how you think about implicit versus explicit expertise. I feel like Datadog is half and half implicit and explicit. It's like they understand across markets and industries what engineering teams usually look for.With notion, it's almost like more of the expertise is at the edge because you as a platform, you're like so horizontal that the end user is not really the same. Mm-hmm. Like with Datadog, the end user is always like, yeah, an engineering lead, a kinda like SRE related person with notion. It can be anything.So I'm curious how you put that expertise into a product versus, you know, obviously it, WS cannot build notion. It's, that doesn't quite work in this case, but[00:08:44] Simon Last: it's, it's a little bit differently shaped. I think, you know, a classic vertical SaaS, like the data is kind of like that. They understand their individual customer very deeply.It's kinda a narrow slice, um, notion has always been super horizontal. And our, our task has always been to sort of balance these two somewhat opposing forces of like, we're listening to our customers and what they want us to build. It's a broad slice. And then also we're thinking about like, okay, how do we decompose what they want into, uh, nice primitives that are, that are really nice to use and we'll, we'll get us like as much bang for the buck as possible.And then, you know. Maintain the whole system, make it all like, like super clean and nice to use.[00:09:22] Sarah Sachs: We still have user journeys. I mean, we still focus on like core. I actually think the failure of our team is when we focus too much on what are cools that are, what are tools that are[00:09:31] Simon Last: mm-hmm.[00:09:31] Sarah Sachs: Cool tools. I actually think that's when we make have the least velocity because you still need some sort of focus on a user journey.So like for instance, we'll all sit down every Friday and look at the P 99 of like the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of tasks. Like we still focus on like, this has, like this should work. Email triaging should work. Mm-hmm. Right. And similarly, like when we're talking about before building, um, chatting, um, before we started filming about, okay, how can I do PDF export?Well that's functionality that then merits. Maybe we should build a tool that has access to a computer sandbox in a file system and the ability to write code. Right? Right. Um, but it's because we're thinking about the fact that our users to do their, to do their daily work, need to export PDFs, not because we're like, Hmm, I think a computer tool could be cool.Like, let's just see what happens. Mm-hmm. Like we, we have to focus on some user journeys, otherwise we just don't have like, enough strategy to, to prioritize.[00:10:29] swyx: I think there's a lot of like really strong opinions that you've had. Do you have like sort of like a towel of Sarah Sachs? Like, you know, like what, how do you run your team?Like I feel like you just have accumulated all these strong opinions. Obviously part, part of this is your, your token town thing.[00:10:43] Sarah Sachs: I think the TAs working with Service X is, um, you'd have to, it depends who you ask. Um, I think it depends if you're on my team or a partner Right. Or a vendor.[00:10:54] swyx: Yeah. There other people want to run their teams the way that you're Yeah.You're like bringing these things. And then also similarly, uh, Simon, when you did the custom agents demo, you had like, well, we've been using custom agents and here's the super long list of everything that we do. No humans ever read it. Right? That's what you said. I was like,[00:11:07] Sarah Sachs: yeah. So I think for, for me, um, something that I learned very quickly and became very comfortable with was that my job was not to be the ideas per person or the technical expert.My job was to make it so that everybody understood the objective, had a resource to help prioritize what they should work on, and had an avenue to prioritize what they thought was important. And I think that's true with all, all leadership, but I think especially on the AI team. Almost all of our best ideas come from prototypes, from people that have a cool idea because they saw a user problem, and it's a huge disservice if all of those ideas have to pass, like the sniff test of what me and a product partner or Simon and Ivan decided were the direction, right?Because a lot of what we're doing is leaning into capabilities, so. I think that's the first thing is like, I don't really view like the role of engineering leadership as like, uh, hierarchical, nor has it ever been, but especially now, like very willing to change direction based on, um, like proof is in the pudding.Yeah. And like, and I think we have rebuilt our harness three or four times. And when you do that, then the second rule of engineering leadership is like you need to build a team that's comfortable deleting their own code and is very low ego and is driven by what's best for the company. And, um, doesn't write design docs because they think it's their promotion packet.Right. And that's a culture that notion had long before I joined, but like our willingness to just swarm on different problems and um, redo things that we've built before because something has changed. Like, there's a lot of friction that can happen at companies when you do that. And it doesn't happen at Notion.And because it doesn't happen when new people join. Like they don't wanna be the ones that are saying, we shouldn't do this. I wrote that code. So then it's, you know, you, you create a culture that everyone thoughts and that culture comes directly, I think from Simon and Ivan though, um, because they're very open-minded.[00:12:50] swyx: Anything that you,[00:12:50] Simon Last: you'd add? I'm not a manager, like, like, like Sarah is. Um, a lot of my role is really to try to think a little bit ahead, make sure that we're, we're building on the right capabilities and then like the prototyping stuff. And yeah, it's really, really critical to always just be starting again.It's like, okay, this is new thing. What does this mean? What if we just rethought everything or wrote everything? And so I, I'm, I'm basically just doing that in a loop every six months.[00:13:16] swyx: Yeah. Do you believe in internal hackathons for this stuff?[00:13:19] Sarah Sachs: I think there's like two different versions. So one is like, we just have a, a, a solid bench of senior engineers that come and go on what we call the Simon Vortex and Productionizing what we built, right?Because when you're in the Simon Vortex, the velocity is super high. The direction changes daily, and it's meant to be like the equivalent of a SC Works lab. We don't need to do hackathons for that. We need to have senior engineers that we trust to come in and out of those projects. For instance, like management boundaries are really loose.Like you report to him, but you work for her right now. Yeah. That's something that when we hire managers, it's important they don't care about because we tend to form more structures. Yeah. Don't be too[00:13:54] swyx: territorial.[00:13:55] Sarah Sachs: We form more. It's after we ship things, not not before, just historically. Um, the second thing is we do have companywide hackathons.Actually we just had our demos day for the hackathon we had last week this morning. That's more for people that aren't directly working on the project, feeling like they have the time to pause and learn how to make themselves more productive or how they would use notion custom agents to build something.Or part of the hackathon was actually encouraging everyone across the company to build their own agentic tool loop, calling from scratch. Follow like an every blog post on how to do what I think because we want[00:14:26] swyx: just with the compound engineering one. Yeah.[00:14:28] Sarah Sachs: We want everyone to use cloud code in the company or whatever the coding agent they please and understand that fundamental.So we set aside a day and a half. We're all leadership, encourage everyone on their teams across the company to do it. So we have hackathons like that. I would say like kind of facetiously, like everything we build is a little bit like a hackathon until it graduates and puts on big boy pants and as a product ops rollout leader and has a assigned data scientists and stuff like that,[00:14:54] swyx: security review enterprise stuff,[00:14:56] Sarah Sachs: actually security reviews one of the things that we bring in first because it just slows us down way more and, um, causes a lot of tension and they build better product if they're involved early.So, um, that is probably the first person to get involved in something that's the[00:15:09] swyx: right PR approved answer.[00:15:10] Sarah Sachs: No, but it's not just PR approved. It like, um, um, it's[00:15:13] swyx: actually real. It's actually real. It's like, um, I'm just saying scar[00:15:15] Sarah Sachs: tissue.[00:15:15] swyx: Yeah,[00:15:16] Sarah Sachs: because like, you know, my background's also, I worked at Robinhood for a number of years.Yes. So like, uh, compliance and things like that, um, are a little bit more, you learn the hard way when it doesn't come naturally.[00:15:26] Simon Last: Yeah. I think the. The hackathon is really important for uplifting the general population, but like, if that's the only way you can build new things, you're kind of toast. I mean, it, it has to be like the daily processes, like, you know, building these new things.Um, and it has to be about, I think like, I think in the AI era a lot more leverage accumulates to the most curious and excited people. And so it's like we're all about just like activating that energy. You know, like if someone's protesting something on the weekend that they're excited about and it's important, that should be the main thing that we're doing.Yeah. Um, it's not a hackathon that we schedule once a quarter, it's just like, yeah. Daily process. Part of the culture.[00:16:02] Sarah Sachs: I mean, that's how we shift image generation and notion now. It was always this thing that would be kind of nice to have, but it wasn't really clear where that was necessarily aligned in product priorities.It'd be a lot of work. And we had someone on the database collections team, Jimmy, who was like. I really wanna do image generation for cover photos and inside notion. And we're like, if you wanna build it, like it's, do it please. Like we encourage you. We gave ‘em all the resources of working directly with Gemini and being able to like track the token usage and it working through endpoints.We gave them eval, support, everything, and then became a, a full project.[00:16:34] Alsesio: Yeah.[00:16:35] Sarah Sachs: That's why you can't have like ego as a, a leader. Like that's, that's how we work.[00:16:39] Alsesio: What's the size of the team today, both engineering and overall?[00:16:43] Sarah Sachs: I manage, uh, the team. That's what we'll call it. Core AI capabilities and infrastructure.That's about 50 people. But then we have per i partner teams that do packaging. So how it shows up in the corner chat versus custom agents versus meeting notes, that's another 30, 40 people. And, and then every team that has a product service at Notion that a user can interface with owns the tool that the agent interfaces with the editor team.The team that did CRDT for offline mode is the same team that handles how two agents, um, edit competing blocks. Mm-hmm. Right? It's the same problem. The team that built the underlying SQL engine is the same team that owns how the agent asks it to run a SQL query, and it does it performantly. And so from that regard, anyone working on product engineering is tasked with making them work for customers that are humans and agents because over time the majority of our traffic will be coming from agencies using in our interface, not humans.And so. Our objective is to make it so that the whole product org is building for agents.[00:17:40] Alsesio: Yeah. How has it changed internally? The activation bar is kind of lowered a lot. Like anybody can kind of create a prototype very, somewhat easily, especially if you're like an existing code base. Have you raised the bar on like what type of prototype people need to bring forward to gonna be taken?Not like seriously, but like, you know what I[00:17:58] Simon Last: mean? Yeah. I think the bar is lowered in many ways. Be like, one thing our, uh, our team built that is really cool is our, uh, our, our design team made a whole separate GitHub repo, uh, called the, the design Playground. And it's basically just to create a bunch of like, like helper components and you, uh, for, for quickly a throwing together UIs.And it's become like actually quite sophisticated. Like it has like an agent in there and like, uh, that's pretty fun. So like, we pretty much, like, they don't do mocks, they just make like, like full, full prototypes.[00:18:27] swyx: Here it is. It works.[00:18:28] Simon Last: They give you like a u rl. They're like, okay, all right. So we have to make the, like the real production version of that.Um, and then for engineers. A prototype looks like just making it a feature flag that actually works. Like that's sort of the bar.[00:18:39] Sarah Sachs: Something to understand that's really unique about notion. One of the reasons I joined we're super lucky is no one uses Notion in their job as much as people that work at Notion.[00:18:46] Simon Last: Of course.[00:18:47] Sarah Sachs: So I think there's very few companies, maybe if you worked on Chrome I guess, but like everything that we ship, we ship internally first and get a lot of really quick feedback. And also sometimes our dev instance is totally borked and you have to change a bunch of flags to get things done. And that's kind of like, but everyone, so people that do it ticketing, people that do supply chain procurement, recruiting, everyone is using the same instance of notion with like a lot of flags on for these prototypes people build.Um, and so we have this, Brian Levin, one of the designers on our team, I think evangelize this concept of demos over memos.[00:19:18] swyx: Ooh, too[00:19:20] Sarah Sachs: good. Um, which has been, uh, very good for building demos, and I think it's put a big pressure point on us to have really strong product conviction, because if anything can be demoed, you really need a strong filter of making sure that if you know, you're doing X amount of work, you're making the, you're, you're focusing on one tower, you're not just building a really flat hill.Right. That's actually where I think there has to be more conviction from our PMs, um, and our designers and, and well, the company really to have conviction of what journey we're going on.[00:19:52] Simon Last: But overall, I feel like it works pretty well. Like people, almost all the engineers have good enough taste to realize that like, this prototype doesn't actually make sense in the product, or, or it does.So it's not that common that I would see a prototype. It's like, oh, this makes no sense. Mm-hmm. It's like, you know, people are doing reasonable things and, and, and then it's just a matter of. Which things we build first and then often just, just figuring out how to turn it on and off. There's our, in the, in our like experimental chat ui, there's this, there's probably like, like a hundred check boxes in there.[00:20:22] Sarah Sachs: Kills me[00:20:23] Simon Last: the things you could turn on and off.[00:20:25] Sarah Sachs: Uh, but I think that, okay, so that is kind of true, Simon, but like being the person that manages the evals team, like there is a level of intensity that it adds to the platform team. So, you know, if we're gonna do image generation and notion, all of a sudden the way that we do attachments and the way that we, um, our LLM completion like cortex talks and expects tokens back and now it's getting images back.Like there's a lot of platform work that we do need to, like solidify a little bit. So sometimes it'll be in dev for a couple weeks before it makes it to prod just because we still have to like, make it robust, make it HIPAA compliant, ZDR compliant, figure out the right contracting with the vendor, whatever it is.And we need to eval it because we want the team. To still maintain what they build. That's the one thing is like if we have a bunch of prototypes, it can't just be like a small group of people that then maintain whatever end prototypes. So we have invested a lot of people in an eval and model behavior understanding teams that, we call it agent dev velocity.So your dev velocity building agents can be faster if we invest in that platform. And so we have a whole org dedicated to Asian, um, platform velocity so that you can build your own eval and then maintain it once you ship it. So if a new model release comes out and we, every[00:21:38] swyx: team maintains their own eval,[00:21:40] Sarah Sachs: we maintain the eval framework.Every team owns their own evals and a lot of them we've integrated to Optin, to ci, or we run them nightly and we have a team, uh, a custom agent that triggers to a team to look at the major failures. That's really critical because if we have like all these different surfaces now, a lot of it's on the same agent harness, so it's easier to maintain.It's just packaging of different agent harnesses, but new functionality of the agent. Let's say that like we wanna update like. Uh, you know, they deprecated, sonnet, um, four or whatever it is and we need to auto update. Are[00:22:11] swyx: they already? That's so, okay. Yeah. Actually wasn't that long ago.[00:22:14] Alsesio: Theywere[00:22:14] Alsesio: just 3.5.[00:22:15] Sarah Sachs: 3.537. Just got deprecated.[00:22:18] swyx: 3 7, 5 0.2 or, yeah. No,[00:22:20] Sarah Sachs: it's not. 5.2 is five point. Five point no. Yeah, five four is 40% more expensive than five two. So if they deprecated five two, you would hear they can, you would hear from me about that one. Um, but, uh, another conversation to have.[00:22:35] swyx: I have a cheeky evals question for you.Have you noticed any secret degradation from any of the major model providers?[00:22:40] Sarah Sachs: Secret degradation,[00:22:42] swyx: like. During the War Bay, when it's high traffic, it suddenly gets dumber.[00:22:47] Sarah Sachs: Yeah. I mean, not just between the, I mean, we definitely notice flakiness, we've definitely noticed, particularly for some providers, that things are slower during working hours and[00:22:57] swyx: there's a latency argument.Yes. Not a quality argument.[00:22:59] Sarah Sachs: No. I think the quality difference that's interesting is, um, even though companies that say they're selling the same, a, it's really into like quanti quantization, but like companies that say they're selling the same model through different vendors, whether it be through first party or Bedrock, Azure, et cetera.We do see different qualities sometimes, and that's not necessarily what's advertised.[00:23:21] swyx: Yeah. Kidney went to the point of like, if we, they shipped like this, like eval across all the providers and it was like very obvious we were secret equalizing and it was very,[00:23:28] Sarah Sachs: yeah. But[00:23:29] swyx: that's very embarrassing.[00:23:30] Sarah Sachs: You know, um, we hire Subprocess to figure that out for us.So we just wanna understand where it's regressing or where it's optimized. And sometimes we're okay with regressions that optimize latency if they're the appropriate regressions. Our job is to make sure we have the evals to understand the changes that are important to us. And even like when we're partnering with labs on pre-releasees of models, they'll send us multiple snapshots.And this is less about quantization, but more just regressions. Like they have shipped models that were not the snapshots that we wanted, and they have changed the snapshots that they shipped based on the feedback that we give. Because our feedback tends to be more enterprise work focused and not coding agent focused.And definitely those can be bummers, like, you know, uh, we know that this wasn't the version you wanted, but we'll help you make it work. I mean, we always make it work, but that definitely happens.[00:24:16] Alsesio: Yeah. Do you have, um, failing evals that you're just hoping, oh, that will have success eventually when a good model comes out?[00:24:23] Sarah Sachs: Uh, I mean, yeah. So I think. I mean, I could talk about this for 60 minutes, so I will limit myself. I think it's a real issue when people say evals and it's just like, that's quality, that's like unit, I mean, it's like saying testing. It's not just unit tests, right? So. We have the equivalent of unit test.Regression test. Those live in ci, those have to pass a certain percent, you know, within some stochastic error rate. Then we have, as you're building a product, evals of these aren't passing right now, and this is launch quality. So we have a report card and we need to, on these categories, you know, be it 80 or 90% of all of these user journeys to launch, and then what we have what we call frontier or headroom evals, where we actively wanna be at 30% pass rate.And that's actually been a effort that we took in partnership with philanthropic and OpenAI in the past maybe two or three months, because we actually hit a point where our evals were saturated and we weren't able to really give insightful feedback other than it wasn't worse. And not only is that not helpful for our partners, it's not helpful for us to understand where the stream is going.You know, going back to that analogy. And so we spent a lot of time thinking about. What notions last exam looks like, right? Mm-hmm. Not just humanities, last exam. Ooh, notions last exam. Mm-hmm. And, um, there's a lot of, you know, dreams about what that would look like. I know we've talked a lot about benchmarking, um, swix, but, uh, yeah.Notions last exam is a big thing inside the company and we have people, full-time staff to it exclusively. Mm. We have a data scientist, a model behavior engineer, and an full-time, um, evals engineer just dedicated to the evals that we pass 30% of the time.[00:25:56] swyx: What you're hiring for[00:25:57] Sarah Sachs: MBEs? I am hiring[00:25:58] swyx: What is an MBEA[00:25:59] Sarah Sachs: model?Behavior Engineer Model. Behavior engineers started with a title data specialist before I joined when they were working with Simon on like, uh, Google Sheets and like Simon just needed someone to look through Google Sheets and say, yes, no, this looks bad. This looks good. Right? And so we hired people with kind of diverse linguistics background.We had like a linguistics PhD dropout. Mm-hmm. And a Stanford ate new grad. And they're amazing. And they formed a new function basically. And over time we've built a whole team, um, with a manager who's now kind of reinventing what that role is with coding agents. So they used to be kind of manually inspecting code.Now they're primarily building agents that can write evals for themselves or LLM judges. There's a really funny day I can send you the picture where Simon, about a year and a half ago, was teaching them how to use GitHub. Um, and they're on the whiteboard and it was like, okay, I think it would be so much faster if our data specialists learned how to use GitHub and like learned how to commit these things in Dakota.And, and that was then and now I think, you know, coding has been a lot more accessible. Um, but moving forward it's this mix of like data scientist PM and prompt engineer because there's craft in understanding like even like what models can and can't do things. How do we define like that headroom? How do we define like what a good journey is?Um, is this model better or not? Why is this failing? There's some qualitative work, but then there's also like a lot of instinct and taste to it, and that's not necessarily software engineering. And so we have like very firm conviction and we have had for a number of years now that that is its own career path and we have always welcomed the misfits, so to speak.So we really firmly believe that you don't need an engineering background to be the best at this job. And that's what's quite unique about this particular role.[00:27:37] Simon Last: Yeah, this is something that I've been pretty excited about recently is we made an effort basically to treat the eval system as like an agent harness.So if you think about it, like, you know, you should be able to have an agent end-to-end, download a dataset, run an eval, iterate on a failure, debug, and, and then implement a fix. And ultimately you should be able to, you know, drive the full time process with a human sort of observing the, you know, the outer uh, system.So yeah, we went, went pretty hard on that. And that's, that's worked extremely well so far. It's like basically just to turn it into a coding agent, uh, uh, problem.[00:28:11] swyx: Your coding agent or just whatever[00:28:13] Simon Last: harness No coding agent. Yeah, code, cloud code. It should be totally general. Yeah. I think if it would be a mistake to like, like fix it on any, any particular coding agent.At the end of the day, it's just like CLI tools.[00:28:21] Sarah Sachs: It's like the same way that you would've a coding agent write the unit test. You should have a coding agent write the eval.[00:28:26] swyx: Yeah.[00:28:26] Sarah Sachs: But there's a lot of supervision in that still. We just don't believe that supervision has to come from software engineers because a lot of it is like, um, kind of you XREE and whatever, and these are the people that also triage failures and tell us where we should be investing next.[00:28:40] swyx: Yeah. I'm gonna go ahead and ask a spicy question. Is there a data, there are no software engineers at Notion.[00:28:46] Simon Last: Um,[00:28:46] Sarah Sachs: what does it mean to be a software engineer?[00:28:47] swyx: Exactly.[00:28:48] Simon Last: I mean, I think the way things are going is like we're on some continuum where. If, if you look back three years ago, humans were typing all the code and then we had auto complete, you're typing list of the code.Then we had sort of like filling agents, filling lines, and now we're getting into like agents doing longer range tasks where you can debug and implement a fix and then verify it works and you know, get your, get your PR even like, like Merion deployed. I think we're sort of just moving up the abstraction ladder and then the human role becomes more about observing and maintaining the outer system.There's a string of agents flowing through, like me prs what's going off the rails. Like what do I need to approve? Is there like a learning or memory mechanism that that works? So it's kind of a hard engineering problem. There's a, you know, there's, there's a lot to do there. I think we're just sort of moving up stack[00:29:34] Sarah Sachs: the same transition machine learning engineers have made, right?Like I haven't looked at a PR curve in a while.[00:29:39] swyx: Yeah. You used to do this stuff and now, um, auto research can do it,[00:29:42] Sarah Sachs: right? Like I think it depends on what you define as a software engineer.[00:29:46] swyx: Yes. It's, that's changing for sure.[00:29:49] Sarah Sachs: I think every software engineer in notion this summer went through like this, um, sheer, um, one of our engineering leads of the company called it, like every software engineer is going through the, the, uh, identity crisis that every manager goes through, where all of a sudden they realize their ability to write code is less important than their ability to delegate in context switch.And I think that is a transition out of being a software engineer. But[00:30:12] Simon Last: yeah. Yeah, there's a critical difference to being a manager, which is that like, it is actually very deeply technical. The problem, you know, humans are very like, like, like fuzzy and you can't like treat a team of humans like a, like a rigorous system where like, you know, prs like, like flow through and can be in like a block status and then what happens when they're blocked, right.With a set of agents, you actually can do that. And, and, and I think it's actually, there's a lot of interesting technical rigor that that goes into that it's like it's a technical design problem. Ultimately.[00:30:42] Alsesio: What is the design of the software factory that you're building?[00:30:46] Simon Last: Yeah, I mean, I think we're. Trying a lot of different things.I mean, ultimately you want to design a system that requires as little human intervention as possible, but like still maintaining the in variance that, that you care about. So yeah, we're exploring a lot different ideas there. I mean, I think I could talk about a few things I think are important there.Like, one thing I think is really important is, um, having some kind of like specification layer you can just commit marked on files. Mm-hmm. That works pretty well, but[00:31:15] swyx: it's nice to be notion man. I'm just saying like the spec, like Yeah. The natural home for specs is notion.[00:31:21] Simon Last: Yeah. Right. It can be a database of pages.Yeah. I mean, it needs to be something that is, you know, human readable and I viewable and I think that's pretty key. Another really key component is like the, the self verification loop. Yes. You need really, really good testing layers, basically. And that's a really deep, uh, uh, problem. But by getting that right, you know, and then, and then it's kinda like the workflow of like.What happens when there's a bug? How does it flow into the system? Like, is it like a subagent working on it? How does it make a PR and how does that get reviewed? And me, and then, you know, so there's like the, the flow or process.[00:31:56] swyx: Yeah. Cool. Uh, you know, one thing we did work out before you guys came in was this demo or this[00:32:01] Simon Last: agents[00:32:02] swyx: agent demo.Uh,[00:32:03] Simon Last: so every,[00:32:04] Alsesio: every time we do an episode, we try the product. Right. I don't think there's ever been an episode that I haven't tried. Yeah. Um,[00:32:11] swyx: and we, we try, try is a, a big word. Like since day one lane space has been on Notion, but this is the, this is the net new thing. Yes.[00:32:18] Alsesio: So this is for Nel Labs, which is the space we're in.So next week we're opening applications for tenants. So there's a web form, let me, we got this form done here. Uh, so, uh, before. Uh, the workflow would be I get an email, then I look at the person. It was like, should I spend time talking to this person? Then I respond, they respond back. So I build this. So the name it came up for on its own.Can you maybe h how do, how does it come up with its own name?[00:32:43] Simon Last: Yeah, that's a pretty app name. It's, it, it is just a random, it's a random, a name generator.[00:32:47] Alsesio: Oh, that's funny. It just came,[00:32:49] Simon Last: the fact that it picked that is, is kind of hilarious. I'm pretty sure it's just determined,[00:32:54] Sarah Sachs: resilient collector. I, I think I've never looked at the code for that.I've never second guessed it. I think it's kind of like a madlib situation.[00:33:00] Simon Last: Yeah, I think you're right. Yeah. It's, it's totally a, a deterministic. Oh, I thought it was great. Yes. Although, although when the, if you use the AI to set itself up, it can update its own name, so. Okay. Um,[00:33:11] Sarah Sachs: how did you create it? It, did you just do[00:33:12] Alsesio: classroom?I,[00:33:13] Sarah Sachs: okay.[00:33:13] Alsesio: I did, yeah. I'll say just check my inbox for applications for a coworking space. Keep a people, so it created the database for me. Which I have here. And I guess database is like an notion table because everything is notion. Um, and then whenever um, an email comes in, like here, it just creates a new role for the person.Mm-hmm. And then it uses web search to enrich the mm-hmm. The profile. So it kind of like searches the web and it's like, this is who this person is, this is when they say they wanna move in and kind of updates everything else. This is, I mean, it's not a GI, but to me, I don't wanna do this work. So it feels like, I mean, it took me maybe like 15 minutes to set up the whole thing.Um, and I really like that most of the information should live here. You know, it is not like some other tool asking me[00:34:01] Sarah Sachs: Yeah.[00:34:01] Alsesio: To like, bring my stuff there. It's like I would've probably already created an ocean thing.[00:34:06] Sarah Sachs: Mm-hmm.[00:34:06] Alsesio: So[00:34:07] Sarah Sachs: most of our biggest use cases and gains are from. That extra layer of human involvement in the process to make it so right.And so like one of our biggest use cases is bug triaging. So if someone posts something in Slack, can you just have a custom agent that lives there that has its own routing constitution of what team this belongs to, creates a task in your task database and then posts in that Slack channel, right? Like that's like one of the first things that we built internally, I think.And it's completely changed the way that notion functions as a company. Nothing falls through, well, most things don't fall through the crack. We don't know what we don't know. But it's not replacing people, it's replacing processes.[00:34:44] Alsesio: Yeah.[00:34:44] Sarah Sachs: Right.[00:34:45] Alsesio: And I'm curious how you think about composability of these things.So the other one I was working on is like a. These filler. So whenever somebody signs up as a tenant, kind of he'll sell the lease for them. There should probably some agent that is like office manager agent mm-hmm. That can handle the request, make the lease, and then, uh, give them a ADA access to the office and all of that.How do you think about that feature?[00:35:08] Simon Last: Yeah, so I mean, there's, there's two ways you can compose. One way is by using like the data primitives. So you can, you know, you, you could give, you have one agent, uh, be writing to the database and there's another agent that's walked in the database. So that's, that's one way that they, they can coordinate that's like a little bit more decoupled and mm-hmm.Works really well. Or you, you can couple them. So I, I think it's actually not released yet. Releasing it like next week is, uh, in the settings for an agent, you can give access to invoke any other agent.[00:35:34] swyx: Hmm.[00:35:34] Simon Last: So you can have them just. Just, uh, uh, talk directly. So[00:35:37] swyx: you, was there a limit on like, number of recursions or just,[00:35:40] Simon Last: um, probably,[00:35:42] swyx: you know what I mean?Like, you can just get an infinite loop that way there's[00:35:45] Simon Last: some kind of Yeah,[00:35:46] Sarah Sachs: I think it's, there is actually a number somewhere.[00:35:49] swyx: I believe I'm just, you know, like, you're, you're, someone's gonna screw up. You[00:35:51] Simon Last: should you try to see[00:35:53] swyx: Yeah. I mean, everything's gonna be paperclips.[00:35:55] Simon Last: Oh, yeah. Yeah. But, uh, but, but that's really useful.Yeah. So we, you know, like I just, I, I helped, uh, someone internally the other day, they had, they had built like over 30 custom agents for, uh, for our go to market team doing all kinds of different things. You know, for example, like researching, you know, like, like filling information about, about a customer or like, like triaging customer feedback or like, uh, something like that.Literally over 30 of them. And, and then he, and then he even made like a database of all the agents and then he is like, okay, and, and now I'm getting 70, over 70 notifications per day with just the agents are blocked on various things. Uh, and then I was like, oh, okay, cool. You know, the obvious thing to do there is to make a manager agent,[00:36:32] Sarah Sachs: right?[00:36:33] Simon Last: That's gonna sort of blocks be another abstraction layer in between your, your, uh, uh, 30 agents. Uh, so yeah, we, we send out with like a manager agent and then has access to invoke all the other agents and it's sort of like, like watching and observing them and then it sort of, it just creates a layer of abstraction.So instead of 70 notifications per day, it's like, like five. And then, and then the manager agent can help like, uh, debug and fix any problems with the,[00:36:54] swyx: does this is a concept of like an inbox or something like piece, you're basically saying that they can message each other?[00:37:00] Simon Last: Yeah.[00:37:01] Sarah Sachs: Well[00:37:01] swyx: they use the system of record, which, which is[00:37:02] Sarah Sachs: notion, so we[00:37:03] Simon Last: actually, yeah, we didn't make any special concepts at all.[00:37:06] swyx: They're interested to the motion notifications that I would've got,[00:37:09] Sarah Sachs: they can just like write a task to a database that the other agent's task to listening to, or they can actually call a web book to the agent, like they can just add the agent. Okay.[00:37:17] Simon Last: Yeah, I mean, this is something that, that we're still working on.I, I think we, you know, like, like generally, generally the way we do these things is, you know, you first make it possible, maybe like a sort of janky way. So I, I, I think the way I set ‘em up is like, you know, we created like a new database that was sort of like issues mm-hmm. That the custom agents were, were experiencing, and then gave them all access to file an issue and then the manager has access to, to read the issues.Um, and that works pretty well, essentially like, like give it its own like internal issue tracker just for the agents. And then, you know, if that becomes a, a concept that seems useful, generally maybe we will think of how to package it in. But I mean, generally we try to just keep it to composing the primitive if we can.You know, another example of this is we have no built-in memory concept. Memory is, is just pages and databases. And so if you wanna give a memory, just give it a page and give it. Edit access to that page and the[00:38:03] swyx: human can edit it. Agent can edit[00:38:04] Simon Last: it. Yeah. And so that works, that pattern works extremely well on it.And you know, depending this case, you can have it be just a page or it could be an entire database with, you know, or, you know, I can have sub pages is is pretty on what you can do with that.[00:38:15] Alsesio: So when I was setting this up, uh, I connected my inbox and it was like, do you wanna use Gmail or Notion Mail? And I'm like, I don't wanna use Eater, I just want you to do it.I'm curious how you think about, you know, notion, mail, notion, calendar, all of these kind of ui ux interfaces, full stack[00:38:29] Simon Last: notion.[00:38:30] Alsesio: Yeah. When like at the same time you have the agents abstracting them away from you in a way, you know, how do you spend like the product calories so to speak?[00:38:37] Simon Last: Yeah, I mean, I think it's pretty important that you don't have to use, not your mail to connect to the mail capability.So we can just connect to Gmail or, or whatever you want, uh, to use. And we're thinking of the mail service as being really great to the extent that it's really agent built, right? So maybe the mail app is just sort of a prepackaged agent that helps you automate your, your inbox.[00:39:00] Alsesio: Yeah, the auto labeling is great.Think[00:39:03] Sarah Sachs: the, when we, um, integrate with Gmail for instance, we have a series of tools available that are available via MCP or API to Gmail. When we integrate with Notion Mail, we have the Notion Mail engineering team to build us the, um, exact right tools that optimize latency, optimize performance and quality.They own that quality. Um, there's product leads there. They're directly thinking about the user problems that happen in mail. So it tends to be when we build integrations and connections, we build natively first. Um, and then think about, um, extending them generally just because it's also easier. Mm-hmm. Um, um, to build natively first.Um, so that tends to be how we phase things out.[00:39:43] swyx: Talking about integrations, you prompted me, so I gotta ask. M-C-P-C-L-I. What's going on? What's the[00:39:48] Simon Last: Yeah. Opinion. I think, I mean, I'm, I'm definitely bullish and excited about cli. I think there's a few really cool things about cli. So one really cool thing is like, um, is that it's in the terminal environment, so it gets a bunch of extra power.So it, you know, for example, it can like, like paginating and cursor through like long outputs. Um, and it has a progressive disclosure inherently. Uh, so, you know, you don't see all the tools at once. It's just, you see the CLI wrapper and you can like use the, the help commands and, and, and read files. And then I think the most important thing that's, that's super cool is that there, it's also inherently a, a bootstrapped.So if there's an issue, uh, the agent can debug and fix itself within the same environment that it uses the tool.[00:40:30] swyx: Mm.[00:40:30] Simon Last: Right. Like, you know, I think I saw a tweet this morning. Someone said, you know, my agent didn't have a browser, so I asked it to make all a browser tool and within a hundred lines of code, it gave itself a little browser, like, like wrapping the, the, the chromium API, um.That's pretty incredible. And then if there was a bug, it would just immediately try to fix it. Mm-hmm. Right. On the other hand, if you use an, you know, if you use like of, of the Chrome dev tools, MCP, I've had this issue where like, like sometimes the transport gets like messed up. If it gets messed up, the agent has no way to fix itself.It, it no longer has a browser, it's, it's not broken. Right. I think that's, that's pretty fundamental, but I would say like a lot of the, the bad things about it can be fixed. Uh, so I think like, as a progressive disclosure, that can be fixed with, with right harness. Like, it, it obviously doesn't make sense to show it all the tools all the time.That's not really inherent to the MCP protocol. It's just like how you wrap it and use it.[00:41:16] swyx: There's many poorly built MCPs because we didn't know.[00:41:19] Simon Last: Yeah, yeah. I mean it was just early, like, like the obvious thing is, uh, you know, to start with is, is to just show it all the tools and it's like, okay, now we have a hundred tools.Yeah. And like the tool calling actually works. So let's of[00:41:28] swyx: your success[00:41:29] Simon Last: give it a way to like, like filter to source the tools. So yeah, I would say like broadly speaking, I'm really bullish on cli. I'm still bullish on CPS and in a certain environment. I think in, in particular, CP is really great for when you want sort of like a narrow, lightweight agent.I think there's, there's definitely a lot of use cases where, where you don't want like a full coding agent with a compute run time. And also you want it to be like more tightly permissioned. MCP inherently has a really strong permission model, like all you can do is call the tools. A CLI is a little bit murkier.It's like, can I access the, if PI token are you, like, properly sort of like re-encrypt the token so it can't like exfiltrate it, it introduce a lot of like, like new issues, which are. Real and hard to solve. And MCP is just like the dumb simple thing that works and it that it's pretty good.[00:42:12] Sarah Sachs: I'll add two more perspectives, not from it working well for Notion, but how notion like commits to both platforms.Notion is dedicated to being the best system of record for where people do their enterprise work. So we will always support our MCP and so far as other people are using cps, right? So regardless of our perspective, we've put a lot of effort into our MCP and we have a fantastic team that we're building, um, to do more there.And the second thing I'll say, I think, um, we all think a lot, but lately I've been thinking a lot about making sure there's a value alignment and pricing, um, with capability.[00:42:43] swyx: Literally our next question[00:42:44] Sarah Sachs: and. Needing language to execute deterministic tasks feels wasteful and requiring on a language model to interface with third party providers seems wasteful for tasks that don't require it.And particularly because our custom agents are using usage-based pricing. We think of pricing as like the barrier of entry for use of our product, and we're quite committed to making sure that it's not wasteful. Um, not just because it's a bad deal for our customers, but it's also bad business. We wanna have as many buyers, like there's a, there's an elasticity of demand and so if we can have our agents properly execute code that calls on CLI deterministically, it's a one-time cost, right?Versus constantly having a language model integrate with an MCP over and over and over and paying those like repeated token fees and it's happening outside the cash window, then you're paying for it over and over and over and it's just kind of unnecessary and less deterministic when it doesn't have to be.[00:43:36] Alessio: Yeah, the open-endedness I think is like, the main thing is like, well, if I go write code to just call an API, I would never use an MCP. But then you need an NCP sometimes when you know what to call, but you don't want it to restart versus like, I think the it built a browser from scratch is like, it's great when you're doing it on your own, but like if your customers were having your AI write a browser from scratch every time and you had to pay the token cost of that, yeah.You'd be like, no, no. The Chrome dev tools CP is actually pretty great. Just use that. I'm curious, how do you make that decision? Like should it be. Just straight API call very narrow. Should it be an MCP? Should it be super open-ended?[00:44:10] Sarah Sachs: Do you mean for when we ship notion capabilities or when we add capabilities to[00:44:13] Alessio: notion[00:44:14] Sarah Sachs: AI or,[00:44:14] Alessio: I mean, you might have a capability that the only way to do is an open-ended agent, like an agent with a coding sandbox.[00:44:21] Sarah Sachs: Yeah. In Notion ai they're not explicit, not We also ship an MCP.[00:44:24] Alsesio: Yeah. Yeah. In B,[00:44:25] Sarah Sachs: yeah.[00:44:26] Alsesio: Internally. Okay. Like is there ever a discussion of like, we're not gonna ship it because we're not able to tie it down? Or are you happy to just like,[00:44:33] Sarah Sachs: um, no. I mean, there are a lot of things where we choose not to use MCP because we wanna add more high touch to quality.I think search an agent to find is like the largest instance of that, where we have. Um, slack and linear and Jira search and notion that is not using necessarily the search MCP functionality that is provided by those companies. And that's because it's quite critical we think, to how our agent trajectories work is for us to have a little bit more control on the functionality of the search journey.And so it usually comes from quality and there's a long tail of things and that's why we built an MCP client or an MCP server, excuse me, so that people can connect whatever they want. There's that long tail, right. But we, for search particularly, I would say that's like the primary entry point, but there are other connections as well that it's a little bit of secret sauce a

Aaron Kaufman and Brian Levin share an unfiltered look at how independent filmmakers can build real momentum by creating instead of waiting. Drawing from experiences ranging from YouTube comedy to studio features, they explain the power of the “Rodriguez List”—writing scripts based on the assets you already have. Their approach emphasizes volume, consistency, and learning through execution rather than chasing perfect conditions or external validation.The episode also explores the realities of producing indie films in a crowded marketplace, from the challenges first-time directors face to the importance of protecting a story's emotional core. Through lessons learned on Flock of Dudes and beyond, Aaron and Brian make it clear that modern filmmaking success comes from action, adaptability, and the courage to put work into the world—long before anyone gives you permission.Become a supporter of this podcast: https://www.spreaker.com/podcast/bulletproof-screenwriting-podcast--2881148/support.

Aaron Kaufman and Brian Levin share an unfiltered look at how independent filmmakers can build real momentum by creating instead of waiting. Drawing from experiences ranging from YouTube comedy to studio features, they explain the power of the “Rodriguez List”—writing scripts based on the assets you already have. Their approach emphasizes volume, consistency, and learning through execution rather than chasing perfect conditions or external validation.The episode also explores the realities of producing indie films in a crowded marketplace, from the challenges first-time directors face to the importance of protecting a story's emotional core. Through lessons learned on Flock of Dudes and beyond, Aaron and Brian make it clear that modern filmmaking success comes from action, adaptability, and the courage to put work into the world—long before anyone gives you permission.Become a supporter of this podcast: https://www.spreaker.com/podcast/indie-film-hustle-a-filmmaking-podcast--2664729/support.

The assassination of right-wing activist Charlie Kirk is testing political leaders, as some meet the moment with calls for unity and others call for a fight. Scott talks with Brian Levin, the founder of the Center for the Study of Hate and Extremism at CSU San Bernardino, about the rise of political violence and where we go from here. Check out Political Breakdown's weekly newsletter, delivered straight to your inbox. Learn more about your ad choices. Visit megaphone.fm/adchoices

With Trump and Hegseth's Bombastic Claims of "Total Obliteration" Blowing Up, Will Iran Now Sprint to a Bomb? | While Talking Peace, This is Trump's War and It's Only Just Beginning | Trump and Miller's ICE Stormtroopers' Reckless and Brutal Roundups of Non-Criminals on the Streets of LA backgroundbriefing.org/donate twitter.com/ianmastersmedia bsky.app/profile/ianmastersmedia.bsky.social facebook.com/ianmastersmedia

The House Speaker Uses an Antisemitic Trope Blaming George Soros as he Launches Hearings on Anti Semitism on Campuses | The Role of Right Wing, Christian Zionist and MAGA Provocateurs in Campus Violence | Models of Dialogue and Deescalation as Outside Opportunists Stoke Violence on Campuses backgroundbriefing.org/donate twitter.com/ianmastersmedia facebook.com/ianmastersmedia

MSNBC's Katie Phang hosts “The Beat” on Monday, August 27, and reports on the latest in the Israel-Hamas war and Donald Trump. Brian Levin, Micahel Beschloss, Neal Katyal, Melissa Murray, David Rhode and Elise Labbott join.

Turi Ryder is filling in for Santita Jackson Show Today's Guests Are: Atty. Chiquita Hall-Jackson, Debra Kerr, and Brian Levin

Series: Informational - In this episode of Pipe and Tamper Mike and Brian chat about the upcoming 2023 Las Vegas International Pipe Show. We first do our Show N Tell where Brian shows everything he picked at the Chicago show. Afterwards, we spend the rest of our time talking about the upcoming Vegas show including what to expect, what's changing, and everything else in between.

White supremacist groups are apparently recruiting hispanics & blacks to their cause now, since any crime is now blamed on whites - Brian Levin of Cal State calls it Hispanic white supremacy; this demonstrates how such labels are defenses against opposition to Democrat policies and are a bigger joke that Dave Chappelle's ‘black white supremacist Clayton Bigsby. In a racist screed worthy of John C. Calhoun, a real black-white-supremacist Democrat, Congressman Hank Johnson of Georgia, said earlier this week that immigrants are inherently waiters and construction workers, and that we need them to bring us food and clean up our hospitals. As Title 42 ended on Thursday, it's business as usual at the southern US-border. Decent people mixed with human smugglers, drug cartels, terrorists, Chinese nationals, etc. Children kidnapped and drugged and women from South America in particular are raped up to 8/10 (https://www.hhri.org/publication/i-didnt-have-anywhere-to-run-migrant-women-are-facing-a-rape-epidemic/). Hundreds of thousands waited to pour across the border on Thursay as a Federal Judge in Florida put a stop to the ‘parole' plan of the White House, which continues to violate Article 4, Section 4 of the U.S. Constitution, ensuring states protection from “invasion.” A few word redefinitions fixes that problem though.According to the IOM on United Nations Migration, the “highest ever reported” number of displacements and movements occurred around the world in 2022 - 61 million. (https://www.iom.int/news/record-number-609-million-internal-displacements-2022-idmc-report) This record number coincides with the United Nations assessment from 2000 called “Replacement Migration.” That report outlined how population decline and population aging were leading to drops in overall population numbers from the US to Italy to Japan. By 2050 it is excepted these countries lose tens of millions in domestic population. The UN reports that in the “absence of migration, the declines in population size will be even greater than those projected…” Thus, “the levels of migration needed to offset population aging are extremely large, and in all cases entail vastly more immigration than occurred in the past.” (https://www.un.org/development/desa/pd/sites/www.un.org.development.desa.pd/files/unpd-egm_200010_un_2001_replacementmigration.pdf) Such mass migrations are akin to the forced migrations of Mao; just like his environmental polices destroyed the land and water, and his idea of equality meant equal poverty. All of this sounds familiar.This show is part of the Spreaker Prime Network, if you are interested in advertising on this podcast, contact us at https://www.spreaker.com/show/5328407/advertisement

In his interview on The Truth in this Art Podcast, Brian Levin shares his insights and experiences as a filmmaker with a strong background in television and film. As the CEO of both Groovy Monsters and Red Tower, Levin offers a unique perspective on the industry and the creative process. Listeners are sure to gain valuable insights into the world of filmmaking and the challenges and rewards of running a successful production company. Whether you're a seasoned professional or simply a fan of the art of film, Brian Levin's interview is sure to be both informative and entertaining.Creators & Guests Rob Lee - Host Brian Levin - Guest The Truth in This Art podcast's current season is sponsored by The Gutierrez Memorial Fund and The Robert W. Deutsch Foundation, both of which are dedicated to enhancing the quality of life in Baltimore and beyond. The Gutierrez Memorial Fund focuses on supporting artists and art organizations that serve Maryland communities, while The Robert W. Deutsch Foundation invests in innovative individuals, projects, and ideas. We appreciate their support in making this season happen. ★ Support this podcast ★

Attacks on electrical systems were up by nearly 80% in 2022 compared to the previous year. Monday, the FBI said it foiled an alleged neo-Nazi plot to attack multiple substations and transformers around Baltimore. It follows recent physical attacks on electrical stations in North Carolina, Oregon and Washington. William Brangham spoke with Brian Levin about the increasing attacks by extremists. PBS NewsHour is supported by - https://www.pbs.org/newshour/about/funders

Brian Levin - Criminologist, civil rights attorney and professor of criminal justice and director of the Center for the Study of Hate & Extremism at California State University, San Bernardino. Election deniers will be on the ballot in 48 states in the midterms. He joins Tavis to unpack how this data is connected to far-right extremism in the U.S., and to discuss the impact extremism has had on our recent socio-political landscape.

Western Europe Braces For Putin's Permanent Cutoff of Russian Gas | The Political Left is More Focused on Biden's Shortcomings Than a Looming Fascist Takeover | An Assessment of the Upcoming Hearing on Trump's Connections to the Militias backgroundbriefing.org/donate twitter.com/ianmastersmedia facebook.com/ianmastersmedia

As the House Committee on Jan. 6 wraps up its fifth public hearing, we're joined by member Adam Schiff of Burbank. The Democrat breaks down what the goals of the committee are and how they hope to get there. Plus, Brian Levin, director of the Center for the Study of Hate and Extremism at Cal State University, San Bernardino, discusses the role of groups like the Proud Boys and the Oathkeepers and why their rhetoric resonates so strongly in places like Southern California. Get in Touch Want to ask Alex a question? Visit the SoCal in 17 page On Twitter using hashtag #SOCALIN17 or her handle @alexcoheninla In this Episode Find the January 6th Committee on Twitter at: @January6thCmte Find Rep. Adam Schiff on Twitter at: @RepAdamSchiff Find Brian Levin on Twitter at: @proflevin

Brian Levin, criminologist, civil rights attorney and professor of criminal justice and director of the Center for the Study of Hate & Extremism at California State University, San Bernardino, talks about the hate crime investigation, and the motivation of the suspect who shot 13 people in a Buffalo supermarket this weekend.

How does political rhetoric contribute to violent bigotry and hate crimes, and how does that violence impact communities, beyond the direct victims? On Today's Show:Brian Levin, criminologist, civil rights attorney and professor of criminal justice and director of the Center for the Study of Hate & Extremism at California State University, San Bernardino, talks about the hate crime investigation, and the motivation of the suspect who shot 13 people in a Buffalo supermarket this weekend.

Korea24 – 2022.03.22. (Tuesday) News Briefing: The presidential office says it is not opposed to the incoming government’s plan to relocate the top office to Yongsan, but it does have concerns about national security. (Eunice Kim) In-Depth News Analysis: Last Wednesday marked the one-year anniversary of the spa shootings in Atlanta, U.S, that claimed eight lives, six of whom were Asian women. The shootings sent shock waves through the country and brought national attention to anti-Asian hate. However, a year later, violence against Asians continues to plague the US. Anti-Asian hate crimes jumped 339% nationwide last year, according to a study by the Center for the Study of Hate and Extremism. The director of that center, Brian Levin, who is also a Professor of Criminal Justice at California State University, San Bernardino, joins us on the line to tell us more. Korea Trending with Titi Aiyanyo: 1. First Lady Kim Jung-sook sent handmade dried persimmon gift sets to the staff of the presidential office to thank them as the President's term comes to an end. (김정숙 여사, 靑비서관 51명에 손수 말린 곶감 선물해) 2. One of the most popular cherry blossom roads in Yeouido will be partially opened later this month for the first time in 3 years. (여의도 벚꽃길, 3년 만에 걷는다···축제 없이 부분개방) 3. Children's book illustrator Suzy Lee has become the first Korean to win a prestigious Hans Christian Andersen Award. (이수지 작가, 韓 최초 '아동문학 노벨상' 안데르센상 수상) Touch Base In Seoul: This week, we meet Yang Jae-min, the first Korean basketball player to play in the Japanese B.League. The 22-year-old comes from a basketball family, and his dream to play the game at the highest level has already taken him all over the world, including Spain and US college basketball. He joined the Shinshu Brave Warriors in Japan in 2020, and he joins us via video call to tell us his story. Morning Edition Preview with Richard Larkin: - In tomorrow’s Korea Times, Kwak Yeon-soo reports that trot singer Song Ga-in has been named the honorary hanbok ambassador of 2022. - Tomorrow’s Korea Herald features a report by Sanjay Kumar on how Poland has been helping refugees from Ukraine since Russia’s invasion, according to Poland’s Ambassador to Korea Piotr Ostaszewski.

Brian Levin, director of the Center for the Study of Hate and Extremism at California State University, San Bernardino, discusses the rise of anti-Black and anti-Asian hate crimes around the country and in New York City.

There's been a well-documented surge in hate crimes against Asian Americans here in the Bay Area and across the country, but a new study says race-based crime is spiking among many other ethnic groups as well. The report finds hate crimes jumping in many cities to a level not seen in 50 years, with Black, LGBTQ, and Jewish people also among the most targeted. For more, KCBS Radio news anchors Jeff Bell and Patti Reising, along with KCBS Radio political analyst and host of this segment Doug Sovern, spoke with Brian Levin, the longtime executive director of the Center for the Study of Hate and Extremism at Cal State San Bernardino, which conducted this study.

This Week on Inside Southern Miss Athletics, John Cox, The Voice of the Golden Eagles, spends some time with Director of Athletics, Jeremy McClain, and discusses a number of topics, including the excitement of the start of spring sports. Women's softball coach Brian Levin talks of the start of his team's 2022 season and women's tennis coach Steve Schram expounds on his team's season as well.See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

  It's been a year and then some since marauders, egged on by that rotten deviled egg Donald Trump, attacked the US Capital.  There have charges and arrests and investigations on the law side, but Brian Levin of Cal State San Bernardino says hate and sedition on the lawless side have only grown. 

One year ago today, the country watched in shock as rioters flooded through the doors of the United States Capitol. KCSB's Alexandra Goldberg welcomes back Criminologist and civil rights attorney Brian Levin, the Director for the Center for Study of Hate and Extremism, to discuss his reflections on the Jan. 6 insurrection. For more information about Professor Levin, head to csusb.edu. Sound byte credit: CSPAN

This week on Inside Southern Miss Athletics, The Voice of the Golden Eagles, John Cox, spends some time with long time assistant athletic director for communications Jack Duggan about his career covering the Golden Eagles and some of his fondest memories and sits down with head women's softball coach Brian Levin and talks about his career and what to expect from the 2022 Lady Eagles. Director of Athletics Jeremy McClain stops my to talk about a myriad of things happening this time of year in college athletics.See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

How to Screen Out Racist Cops Who Join Anti-Government Militias | With a Leader of the Oath Keepers Cooperating With Investigators, We Should Learn More About Who Planned the Assault on the Capitol | Paralysis at the U.N. Caused by China and Russia Allows the Slaughter of Ethiopian Civilians to Continue backgroundbriefing.org/donate twitter.com/ianmastersmedia facebook.com/ianmastersmedia

America's slowest campers on the racist right aren't afraid of being eaten by a bear, but being replaced by people of color.  Brian Levin, director of the Center for the Study of Hate & Extremism at California State University, San Bernardino on our national outburst of hate crimes.  A recent study shows where, geographically the insurrectionists who attacked the Capitol on January 6 came from, and suggests where, ideologically, they're coming from.

Abbiamo provato a esplorare il rapporto tra vaccini, politica internazionale e media con Joy Y. Zhang, studiosa del rapporto tra media e scienza, Visiting Fellow, Global Biopolitics Research Centre, Kings College London. La dr.ssa Zhang è ospite in questi giorni dell'Aspen Global Congress on Scientific Thinking and Action, organizzato da The Aspen Institute. Subito dopo abbiamo viaggiato tra Stati Uniti e Russia, per esplorare il botta e risposta tra Joe Biden e Vladimir Putin e le sue ripercussioni diplomatiche: ne abbiamo parlato con Mario Del Pero (docente di Storia della politica estera americana a SciencePo), con Brian Katulis (Senior Fellow al Center for American Progress, ex funzionario del Dipartimento di Stato e del Dipartimento della Difesa sotto la presidenza di Bill Clinton) e con Andrey Kortunov (Direttore Generale del Russian International Affairs Council). Infine siamo atterrati negli Stati Uniti per indagare le ragioni della sparatoria di Atlanta, nella quale ieri sono rimaste uccise otto persone: ne abbiamo parlato con Brian Levin (docente di diritto penale e direttore del Center for the Study of Hate and Extremism dell Università di San Bernardino, California).

On this week's episode of The PineBeltSPORTS Podcast, Southern Miss softball coach Brian Levin joins the show to discuss the Golden Eagles' upcoming season. The episode also includes Southern Miss baseball Scott Berry's press conference from media day.

Brian Levin has worked as a criminologist and civil rights attorney.  Now he’s a Professor at California State University at San Bernardino specializing in analysis of hate crime, terrorism and the law. We talk about the backstory behind the mob attack on Congress and strategies for future civility.  Where the rioters come from and how they can be brought back to “normal.” 

What we need to know about the threat of far right extremist violence. Plus Bill Press with Susan Rice, incoming Direction of the Domestic Policy Council   An extended conversation with Brian Levin, director of the Center for the Study of Hate.  Plus, Bill Press with an archive interview from 2019 with Susan Rice on the publication of her memoir.   Brian Levin - Pt. 1 In the lead up to the November election, there was significant concern about right wing inspired political violence, encouraged by Trump's incendiary rhetoric. As we approach the inauguration, those concerns remain as Trump continues to make unprecedented attacks on  the electoral process. Just before the election, we spoke with one of the most trusted voices on hate and extremism in America about how the landscape of political violence has changed under the Trump administration, and why a major election creates a moment of even greater risk.   Brian Levin - Pt. 2 We’re speaking with Brian Levin, director of the Center for the Study of Hate and Extremism at California State University San Bernardino.   Susan Rice Bill Press with Susan Rice,  President elect Biden’s choice to lead the White House Domestic Policy Council, and author of Tough Love: My Story of Things Worth Fighting For. If you'd like to hear the entire interview, visit BillPresspods.com.   Jim Hightower Is more technology the answer to too much technology?

A perfect storm. What we need to know about the threat of far right extremist violence as election day draws near. Plus Bill Press on the latest presidential polls, the state of the race, and what’s still at play.   An extended conversation with Brian Levin, director of the Center for the Study of Hate and Extremism at California State University San Bernardino. And Bill Press with Larry Sabato, founder and director of the University of Virginia Center for Politics and Sabato’s Crystal Ball.   Brian Levin - Pt 1 In the wake of the reported plot to kidnap Michigan Democratic Governor Gretchen Whitmer, attention has turned to what we know about the threat of violent extremism in our current political climate. Before that news broke, we spoke with one of the most trusted voices on hate and extremism in America about the nature of that threat, the role of white supremacy, and why a major election creates a moment of even greater risk.   Brian Levin - Pt 2 We’re speaking with Brian Levin, director of the Center for the Study of Hate and Extremism at California State University San Bernardino. And we turn the conversation to how to confront hate and extremism.   Larry Sabato Larry Sabato, founder and director of the University of Virginia Center for Politics and Sabato’s Crystal Ball, tells Bill Press what he expects out of this unprecedented campaign season. If you'd like to hear the entire interview, visit Bill Presspods.com.   Jim Hightower Meet a race-baiting ventriloquist

Brian Levin, director del Centro para el Estudio del Odio y del Extremismo de la Universidad Estatal de California, habló en Mañanas BLU sobre las protestas que sacuden a varias ciudades de Estados Unidos por el asesinato de George Floyd a manos de un policía en Minneapolis.See omnystudio.com/listener for privacy information.

How Rioting and Looting Will Help Trump Cast Himself for Reelection as the "Law and Order" President | The Know-Nothings of the 1850's Have Come Back as Donald Trump's GOP of Today | Trump's Bluster Against China Backfires backgroundbriefing.org/donate twitter.com/ianmastersmedia facebook.com/ianmastersmedia

With a theme of cinematic atmosphere and a meld of past and present, this week on BEHIND THE LENS we take a look at two films – A NUN'S CURSE and UNION BRIDGE – and two exciting filmmakers – TOMMY FAIRCLOTH and BRIAN LEVIN. First up, take a listen to our prerecorded exclusive interview with writer/director/editor/cinematographer TOMMY FAIRCLOTH going in-depth with A NUN'S CURSE. Shot on location in a Civil War era prison, the story melds a present day road trip of four friends (one of whom is obsessed with all things nun-related, especially the legendary Sister Monday) with the prison's history, and Sister Monday's murderous legend, as the group is trapped inside the prison for the night. Stunning, rich visuals mesmerize while strong performances and story provide intrigue. Listen as Tommy digs deep into the production design and location, challenges and benefits of shooting within the walls of the prison itself, casting, cinematography and the use of light and focal length, and the film's meticulous sound design. Then writer/director BRIAN LEVIN joins us live talking about his new film UNION BRIDGE where a family's secret dating back to the Civil War may now destroy it if the truth of what actually happened during the war ever comes to light. And thanks to visions of the past now haunting this generation's family members, the truth seems destined to rear its head. Another film with strong ambient cinematography with its own southern roots shooting near the Mason-Dixon Line in another historic area. Listen as Ben talks about the idea for the film, jumping from a career in comedy writing to heavy drama with an historical and supernatural edge, meeting the challenges of being a first-time feature director, casting, dealing with the weather, Sebastian Slayter's cinematographic design, the benefit of shooting in a location rich with history, working within a minimal budget, and staying true to his own vision for the film are just some of the things we discuss. http://behindthelensonline.net http://eliasentertainmentnetwork.com

Writer director Brian Levin and Alex Breaux of the upcoming Union Bridge! Union Bridge

Writer director Brian Levin and Alex Breaux of the upcoming Union Bridge! Tracks for this podcast - Original theme by The Tomb of Nick Cage and "Beyond Imagination" and "A Case For Murder" by Music of the Month Blackfrog --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app --- Send in a voice message: https://anchor.fm/withoutyourhead/message

On this week's episode of the PineBeltSPORTS Podcast, Andrew recaps and previews the high school basketball quarterfinals (2:56). From there, a deep dive into every area team's start to the baseball season (16:36). Andrew then talks about Southern Miss softball's elite offense (32:26) which includes interviews with head coach Brian Levin (36:28) and leading home run hitter Amber Pack (39:15) from Wednesday's doubleheader against Alcorn State. The show closes out with a recap and preview of USM baseball's weekend (41:25).

Bob & Kelly talk to new Lady Eagle softball coach Brian Levin & Al Holder.

In recent years, Europe has been rocked by reports of increasing anti-Semitism. But what is happening within our own borders? Brian Levin, Director of California State University’s Center for the Study of Hate and Extremism, joins us to share his latest findings, analyze the trends in hate crimes in the United States, and explore effective responses. Then, with the national election in Israel less than a week away, Haaretz journalist Allison Kaplan Sommer joins us on "The Battle for Balfour” to discuss the social media bots scandal and share her prediction of who will be Israel’s next prime minister.

Brian Levin is a Baltimore based filmmaker who recently returned from a decade in Los Angeles. He has created content for Disney/Time Warner/Lionsgate/Comedy Central and many other film/tv companies.  He has just wrapped up his directing debut on his new film, which was shot in Maryland and had many cast and crew from the area.  The film will now be submitted to festivals around the world as it begins its journey to theaters.

The murders of 11 worshipers at a Pittsburgh synagogue is the most deadly antisemitic attack in USA history. Nancy Grace visits with Jon Pushinski, a member of Dor Hadash congregation that was meeting in the Tree of Life synagogue when Robert Bowers allegedly launch his shooting rampage. Others on the panel include Brian Levin, director of the Center for the Study of Hate & Extremism, forensics expert Joseph Scott Morgan, psycho analyst Dr. Bethany Marshall, terror expert Bruce Alexander, and reporter Art Harris.

We assembled a group of religious/community leaders and the founders of the NoH8 campaign to discuss the state of hate in 2018. With the help of Cal State San Bernardino hate-crime law professor Brian Levin we explored where we are and what we need to do. To watch a video interview of these guests, visit the In Depth page on FOX 11: http://www.foxla.com/entertainment/features/fox-11-news-in-depth-the-state-of-hate-in-2018 Hit the subscribe button on this podcast, and connect with Hal on social: Facebook.com/HalEisner Instagram.com/HalEisner Twitter.com/HalEisner You can watch FOX 11 News In Depth Sunday mornings at 9am PT, or watch the segments online at foxla.com/indepth. Thanks for listening! Hal Eisner

Recorded from our live #CiscoChat on 5/31/18. Wireless network complexity is on the rise, increasing with each new device that’s added. IT teams need solutions that streamline their maintenance and troubleshooting processes. Cisco’s DNA Center Assurance and the Aironet 4800 provide an easier approach to troubleshooting and issue resolution that maintains the user experience while ensuring reliability. And by adding 360-degree contextual insights across the network, users, devices, and apps, potential issues can be detected before they even become problems. Andrew Caine is joined by Senior Product Managers Taranpreet “TP” Kohli and Brian Levin, to share their insights and expertise. For more on DNA Center Assurance and the Aironet 4800, visit http://cs.co/Aironet4800

The FBI acknowledges it failed to tracked down school shooter Nikolas Cruz after being tipped off to a threatening YouTube post, but that missed opportunity was not the only red flag missed. Nancy Grace discusses the investigation into the massacre at Marjory Stoneman Douglas High School with experts, including hate crime expert Brian Levin, security analyst Rob Hessell, former DEA agent David Katz, forensics expert Joseph Scott Morgan, child welfare law specialist Ashley Willcott, psychologist Caryn Stark, and reporter John Lemley.

Many questions about rules and regulations for home schools, after 13 siblings are found malnourished and living in deplorable conditions at their home in Perris, where they were home-schooled.  Their parents have been arrested.  President Trump vs. Soviet dictator Josef Stalin--the comparison has been drawn...on the floor of the Senate?!?  A gay college student is stabbed to death in Orange County while on break for the holidays.  A transgender woman is stabbed to death in LA, her body found in a burned home.  We talk to Brian Levin, director of the Center for the Study of Hate and Extremism at Cal State San Bernardino. See omnystudio.com/policies/listener for privacy information. Learn more about your ad choices. Visit podcastchoices.com/adchoices

This week on Flyover, a conversation about the division between Americans who count gun ownership as a fundamental freedom and those who feel that there should be more rules around who has the right to exercise that freedom. Our guests were Brian Levin, director of California State University-San Bernardino's Center for the Study of Hate & Extremism; and Jaclyn Schildkraut, assistant professor of public justice at the State University of New York at Oswego.

Stephen Paddock, the 64-year-old millionaire blamed for killing 59 people and wounding 527 others in Las Vegas, doesn't fit the profile of a mass shooter. The retired accountant fired high-powered rifles from the windows of a suite on the 32nd floor of a hotel Sunday night. Nancy Grace digs into the questions surrounding the massacre with Brian Levin, director of the Center for the Study of Hate & Extremism, psycho analyst Dr. Bethany Marshall, forensics expert Joseph Scott Morgan and reporter Cheryl White. Nancy also looks at a disturbing case in Florida, where a teacher and 2 school administrators face charges after an investigation into the treatment of a boy with non-verbal autism. Northwest Florida Daily News reporters Heather Osbourne and Tom McLaughlin and the child's father Eddie Perillo discuss the case with Grace.

Periodically in recent years, there has appeared to be a resurgence of activity in the U.S. by white supremacist groups, the Ku Klux Klan and neo Nazis, marching openly in US cities often defending their right to assemble and spout hate speech as constitutionally protected freedom of speech. As the marches and gatherings occasionally become more frequent, tensions have risen. Counter protesters appear on the streets to stand against the divisive ideologies. Some of them are provoked to act aggressively. And skirmishes have resulted. Fractions of both groups choose violence. One of the high profile confrontations was in Charlottesville, Virginia in the summer of 2017…and there have been others. It’s all sparked a debate about appropriate and effective ways to demonstrate peacefully to counter hate speech and extreme ideologies. One core riddle seems to have been, whether people of good conscience and compassion should NOT show up to white supremacist rallies and just ignore the hate speech, whether they should show up and engage in comedic put downs of the extremist groups, or whether to show up and actively out-shout and out number them, or should they be even more aggressive in confronting the hate speech purveyors, as the groups that came to be known as ANTI-FA for anti-fascist, tried in chasing and harassing the white power groups. We’re going to offer our own conversation on the topic today… Our guests are social activist and author John Dear, Jesuit priest who’s written extensively on Gandhi as well as books like Living Peace, Peace Behind Bars, and A Persistent Peace. Tonya Covington is Director of Conflict Resolution at Outcomes, Inc. in Albuquerque, New Mexico. Tonya has been a trained mediator and teacher of mediation for 27 years. Tonya has expertise in Workplace and Cross-cultural conflict. Brian Levin is a professor of criminal justice and director of the Center for the Study of Hate and Extremism at California State University at San Bernadino. He is a court certified expert on extremism in both the US and the UK and has testified before both houses of Congress. He has also consulted for the FBI and Department of Homeland Security, as well as for universities and civil rights groups. We’ve also heard on tape from Frank Meeink – a former skinhead who now lectures on tolerance and compassion after long ago quitting is association with neo-Nazi groups.

Periodically in recent years, there has appeared to be a resurgence of activity in the U.S. by white supremacist groups, the Ku Klux Klan and neo Nazis, marching openly in US cities often defending their right to assemble and spout hate speech as constitutionally protected freedom of speech. As the marches and gatherings occasionally become more frequent, tensions have risen. Counter protesters appear on the streets to stand against the divisive ideologies. Some of them are provoked to act aggressively. And skirmishes have resulted. Fractions of both groups choose violence. One of the high profile confrontations was in Charlottesville, Virginia in the summer of 2017…and there have been others. It's all sparked a debate about appropriate and effective ways to demonstrate peacefully to counter hate speech and extreme ideologies. One core riddle seems to have been, whether people of good conscience and compassion should NOT show up to white supremacist rallies and just ignore the hate speech, whether they should show up and engage in comedic put downs of the extremist groups, or whether to show up and actively out-shout and out number them, or should they be even more aggressive in confronting the hate speech purveyors, as the groups that came to be known as ANTI-FA for anti-fascist, tried in chasing and harassing the white power groups. We're going to offer our own conversation on the topic today… Our guests are social activist and author John Dear, Jesuit priest who's written extensively on Gandhi as well as books like Living Peace, Peace Behind Bars, and A Persistent Peace. Tonya Covington is Director of Conflict Resolution at Outcomes, Inc. in Albuquerque, New Mexico. Tonya has been a trained mediator and teacher of mediation for 27 years. Tonya has expertise in Workplace and Cross-cultural conflict. Brian Levin is a professor of criminal justice and director of the Center for the Study of Hate and Extremism at California State University at San Bernadino. He is a court certified expert on extremism in both the US and the UK and has testified before both houses of Congress. He has also consulted for the FBI and Department of Homeland Security, as well as for universities and civil rights groups. We've also heard on tape from Frank Meeink – a former skinhead who now lectures on tolerance and compassion after long ago quitting is association with neo-Nazi groups.

Om Charlottesville, den omedvetna rasismen och frågan ifall rasistiska idéer anno 2017 kan vara en framgångsrik politisk strategi? Dagens Konflikt ska med avstamp i det som hänt och händer i USA försöka förstå RASISMEN som politisk kraft. Det har ju länge funnits ett antagande att med historiens erfarenheter i ryggen och med vetenskapens framsteg som grund - så är rasismen en kraft som i politisk mening för en tynande tillvaro. Men stämmer det? Eller kan Donald Trumps förskjutning av dom politiska gränserna vara en framgångsrik strategi? Vi ska börja med att betrakta den här frågan med psykologins glasögon, för att förstå vem som är och kan bli rasist. Den psykologiska forskningen, inte minst i USA, har nämligen under dom senaste decennierna börjat tala om begreppet implicit bias, undermedveten fördomsfullhet. Vårt psyke styr oss automatiskt i riktningar som gör oss mer fördomsfulla. Det här tyder alltså på att det finns en liten rasist inom oss alla. Hör Marie Nilsson Boijs reportage där du bland annat får stifta bekantskap med självhjälpsgruppen Rasists Anonymous. Övriga medverkande: Jens Agerström, docent i psykologi vid Linnéuniversitetet i Växjö, Nazar Akrami, docent i psykologi vid Uppsala Universitet och Reverend Nathan King, pastor i Trinity united church of christ i Concord North Carolina. Stephen Rappaport hade läst in 12-punktsprogrammet för Anonyma Rasister. Det brukar ju ofta sägas att rasismen är en kraft på dekis, på väg ut, otidsenlig. Stämmer det, i ett amerikanskt sammanhang? Stämmer det som den förra presidenten Barack Obama brukade säga, att historiens långa kurva sluttar mot rättvisan - och, med andra ord, bort från rasismen? Ivar Ekman ringde upp historikern Ibram Kendi, som är professor i både historia och internationell politik på American University i Washington DC och författare till den uppmärksammade och prisbelönta boken Stamped from the beginning- a definitive history of racist ideas in America. Och Ibram Kendi menar att tanken om framsteg i rasrelationerna i USA stämmer - men att det bara är halva bilden. Hur ser det egentligen ut för rasismens konkreta manifestationer på marken i USA? Hur går det för vit maktrörelsen och högerextremismen? Konflikts reporter Simon Moser, ringde upp Brian Levin vid Center for the Study of Hate and Extremism, vid California State University i San Bernadino, vars forskning visar att den stärker sina positioner. De senaste två åren har det sammanlagt varit fler vit makt-möten än under de tidigare 20 åren. Medverkar gör även Ruth Ben-Ghiat, professor i historia vid New York University, som menar att tonen i det offentliga samtalet har blivit råare. Gäster i programmet: Anderas Johansson Heinö, statsvetare som är förlagschef på den liberala, marknadsvänliga tankesmedjan Timbro och Christer Mattsson, biträdande föreståndare för Segerstedtinstitutet vid Göteborgs Universitet, ett kunskapscentrum för förebyggandet av våldsutövande extremism. Kan Sverige dra några lärdomar från det amerikanska exemplet? Ivar Ekman träffade en av dom ledande aktivisterna i Black Lives Matter-rörelsen som var på tillfälligt besök till Sverige, juridikprofessorn Justin Hansford och resonerade med honom om rasismens allmängiltighet. Programledare: Ivar Ekman Producent: Marie Nilsson Boij

The 22-year-old man was on the radar of anti-terror cops, but he was still able to kill 22 people at an Ariana Grande concert in England with a suicide bomb. In this episode, Nancy Grace talks with Brian Levin, Director of Center for the Study of Hate and Extremism, about the terror attack and how it could happen again in the United States.

Aaron Kaufman is a producer, writer, and director best known for producing Machete Kills, and Sin City 2 for Robert Rodriguez. He also directed the film, Urge, starring Pierce Bronson. Brian Levin a producer and writer known for Playing with Guns and Boys in Blue. They've just produced the new movie, Flock of Dudes, which is out September 30th, 2016. Show Notes-- Stu Maschwitz's book, 'Rebel's DV Guide - A guide to making an action movie on the cheapContactAaron Kaufman -- TwitterBrian Levin-- Official SiteDave Bullis-- Official Site-- Twitter-- Instagram-- Facebook-- Youtube-- BlabSupport the Podcast1. Sign Up for Dave's email list2. Rate the Podcast on iTunes3. Buy on Amazon.com using my affiliate link4. Buy Final Draft screenwriting software using my affiliate link5. Buy Jason Brubaker's, 'How to Sell Your Movie' course via my affiliate link6. Buy Jason's Brubaker, '101 Short Film Ideas' book for only $7!7. My Podcast Amazon wish list 8. Buy a shirt in the Zazzle store9. The Dave Bullis Podcast on PatreonSubscribe to the Podcast-- Podbean -- iTunes -- Stitcher-- Google Play Podcasts

Welcome to the 2nd episode of the 2 part series on the new Cisco 3800 & 2800 access points. On todays show we are talking again with Mark Denny & Brian Levin but this time we add in Jim Florwick. Jim is one of the most entertaining speakers from Cisco ...

Welcome to this first part of a two part series on the new Cisco 3800 and 2800 series access points. Today we talk with Mark Denny and Brian Levin, product managers for the two new access points. This first part of the series is an introduction to the new APs ...

Wednesday, June 4, 2014 - Join Host Barb Jordan and guests Head Coach Shane Winkler of the Chicago Bandits, Head Coach Brian Levin of the Akron Racers and USSSA Pride catcher Kristyn Sandberg.Listeners can call in to ask guests questions or can ask by tweeing host @BarbJordan18.Tonight's show is presented by Boombah - www.boombah.com.Purchase Championship Series tickets at npfchampionship.com.

Wednesday, June 4, 2014 - Join Host Barb Jordan and guests Head Coach Shane Winkler of the Chicago Bandits, Head Coach Brian Levin of the Akron Racers and USSSA Pride catcher Kristyn Sandberg.Listeners can call in to ask guests questions or can ask by tweeing host @BarbJordan18.Tonight's show is presented by Boombah - www.boombah.com.Purchase Championship Series tickets at npfchampionship.com.

This week: My thoughts on the Boston marathon bombing and a clip of Bill Maher and Brian Levin discussing Islam. As always...thanks for listening!

Today we finally meet the man behind the Perky Jerky empire, Brian Levin and Helen reviews the Denny Emerson book How Good Riders Get Good. Listen in.