Podcasts about red teamer

On this episode of The Cybersecurity Defenders Podcast we talk with John Vaina, AI Researcher and Red Teamer, about AI risk and safety.John is an expert in AI risk, safety, and security. John currently works as an AI red team operator, tackling some of the most complex challenges in the field. His work spans traditional cybersecurity concerns, such as identifying vulnerabilities in AI systems, to cutting-edge tasks like testing for emergent behaviors and conducting AI alignment and safety audits.John's expertise includes evaluating ethical and bias risks, ensuring model robustness, and running adversarial attack simulations to uncover potential weaknesses. Beyond these technical aspects, he also addresses broader safety issues, including scenarios involving CBRNE threats and other high-stakes risks.John's unique combination of technical skills, strategic thinking, and a focus on ethical considerations makes him a leading voice in ensuring that AI technologies are safe, secure, and aligned with human values.

Adam used to break into companies for a living—legally. As a red teamer, he watched the attack surface shift from networks to endpoints to something new: identity. The Snowflake breach proved it—attackers aren't breaking in anymore, they're logging in. Adam saw it coming, founded Push Security to stop it, and now he's here to break it all down. They're our new sponsor, so if that's not your thing, no worries—catch you in the next one. But his story? Fascinating. Hacked is brought to you by Push Security—helping companies stop identity attacks before they happen. Phishing, credential stuffing, session hijacking—Push tackles it right where it starts: in the browser. Smart, seamless, and built for how people actually work. Check them out at pushsecurity.com. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Apologies for lower audio quality; we lost recordings and had to use backup tracks. Our guests today are Anastasios Angelopoulos and Wei-Lin Chiang, leads of Chatbot Arena, fka LMSYS, the crowdsourced AI evaluation platform developed by the LMSys student club at Berkeley, which became the de facto standard for comparing language models. Arena ELO is often more cited than MMLU scores to many folks, and they have attracted >1,000,000 people to cast votes since its launch, leading top model trainers to cite them over their own formal academic benchmarks:The Limits of Static BenchmarksWe've done two benchmarks episodes: Benchmarks 101 and Benchmarks 201. One issue we've always brought up with static benchmarks is that 1) many are getting saturated, with models scoring almost perfectly on them 2) they often don't reflect production use cases, making it hard for developers and users to use them as guidance. The fundamental challenge in AI evaluation isn't technical - it's philosophical. How do you measure something that increasingly resembles human intelligence? Rather than trying to define intelligence upfront, Arena let users interact naturally with models and collect comparative feedback. It's messy and subjective, but that's precisely the point - it captures the full spectrum of what people actually care about when using AI.The Pareto Frontier of Cost vs IntelligenceBecause the Elo scores are remarkably stable over time, we can put all the chat models on a map against their respective cost to gain a view of at least 3 orders of magnitude of model sizes/costs and observe the remarkable shift in intelligence per dollar over the past year:This frontier stood remarkably firm through the recent releases of o1-preview and price cuts of Gemini 1.5:The Statistics of SubjectivityIn our Benchmarks 201 episode, Clémentine Fourrier from HuggingFace thought this design choice was one of shortcomings of arenas: they aren't reproducible. You don't know who ranked what and what exactly the outcome was at the time of ranking. That same person might rank the same pair of outputs differently on a different day, or might ask harder questions to better models compared to smaller ones, making it imbalanced. Another argument that people have brought up is confirmation bias. We know humans prefer longer responses and are swayed by formatting - Rob Mulla from Dreadnode had found some interesting data on this in May:The approach LMArena is taking is to use logistic regression to decompose human preferences into constituent factors. As Anastasios explains: "We can say what components of style contribute to human preference and how they contribute." By adding these style components as parameters, they can mathematically "suck out" their influence and isolate the core model capabilities.This extends beyond just style - they can control for any measurable factor: "What if I want to look at the cost adjusted performance? Parameter count? We can ex post facto measure that." This is one of the most interesting things about Arena: You have a data generation engine which you can clean and turn into leaderboards later. If you wanted to create a leaderboard for poetry writing, you could get existing data from Arena, normalize it by identifying these style components. Whether or not it's possible to really understand WHAT bias the voters have, that's a different question.Private EvalsOne of the most delicate challenges LMSYS faces is maintaining trust while collaborating with AI labs. The concern is that labs could game the system by testing multiple variants privately and only releasing the best performer. This was brought up when 4o-mini released and it ranked as the second best model on the leaderboard:But this fear misunderstands how Arena works. Unlike static benchmarks where selection bias is a major issue, Arena's live nature means any initial bias gets washed out by ongoing evaluation. As Anastasios explains: "In the long run, there's way more fresh data than there is data that was used to compare these five models." The other big question is WHAT model is actually being tested; as people often talk about on X / Discord, the same endpoint will randomly feel “nerfed” like it happened for “Claude European summer” and corresponding conspiracy theories:It's hard to keep track of these performance changes in Arena as these changes (if real…?) are not observable.The Future of EvaluationThe team's latest work on RouteLLM points to an interesting future where evaluation becomes more granular and task-specific. But they maintain that even simple routing strategies can be powerful - like directing complex queries to larger models while handling simple tasks with smaller ones.Arena is now going to expand beyond text into multimodal evaluation and specialized domains like code execution and red teaming. But their core insight remains: the best way to evaluate intelligence isn't to simplify it into metrics, but to embrace its complexity and find rigorous ways to analyze it. To go after this vision, they are spinning out Arena from LMSys, which will stay as an academia-driven group at Berkeley.Full Video PodcastChapters* 00:00:00 - Introductions* 00:01:16 - Origin and development of Chatbot Arena* 00:05:41 - Static benchmarks vs. Arenas* 00:09:03 - Community building* 00:13:32 - Biases in human preference evaluation* 00:18:27 - Style Control and Model Categories* 00:26:06 - Impact of o1* 00:29:15 - Collaborating with AI labs* 00:34:51 - RouteLLM and router models* 00:38:09 - Future of LMSys / ArenaShow Notes* Anastasios Angelopoulos* Anastasios' NeurIPS Paper Conformal Risk Control* Wei-Lin Chiang* Chatbot Arena* LMSys* MTBench* ShareGPT dataset* Stanford's Alpaca project* LLMRouter* E2B* DreadnodeTranscriptAlessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, Partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.Swyx [00:00:14]: Hey, and today we're very happy and excited to welcome Anastasios and Wei Lin from LMSys. Welcome guys.Wei Lin [00:00:21]: Hey, how's it going? Nice to see you.Anastasios [00:00:23]: Thanks for having us.Swyx [00:00:24]: Anastasios, I actually saw you, I think at last year's NeurIPS. You were presenting a paper, which I don't really super understand, but it was some theory paper about how your method was very dominating over other sort of search methods. I don't remember what it was, but I remember that you were a very confident speaker.Anastasios [00:00:40]: Oh, I totally remember you. Didn't ever connect that, but yes, that's definitely true. Yeah. Nice to see you again.Swyx [00:00:46]: Yeah. I was frantically looking for the name of your paper and I couldn't find it. Basically I had to cut it because I didn't understand it.Anastasios [00:00:51]: Is this conformal PID control or was this the online control?Wei Lin [00:00:55]: Blast from the past, man.Swyx [00:00:57]: Blast from the past. It's always interesting how NeurIPS and all these academic conferences are sort of six months behind what people are actually doing, but conformal risk control, I would recommend people check it out. I have the recording. I just never published it just because I was like, I don't understand this enough to explain it.Anastasios [00:01:14]: People won't be interested.Wei Lin [00:01:15]: It's all good.Swyx [00:01:16]: But ELO scores, ELO scores are very easy to understand. You guys are responsible for the biggest revolution in language model benchmarking in the last few years. Maybe you guys want to introduce yourselves and maybe tell a little bit of the brief history of LMSysWei Lin [00:01:32]: Hey, I'm Wei Lin. I'm a fifth year PhD student at UC Berkeley, working on Chatbot Arena these days, doing crowdsourcing AI benchmarking.Anastasios [00:01:43]: I'm Anastasios. I'm a sixth year PhD student here at Berkeley. I did most of my PhD on like theoretical statistics and sort of foundations of model evaluation and testing. And now I'm working 150% on this Chatbot Arena stuff. It's great.Alessio [00:02:00]: And what was the origin of it? How did you come up with the idea? How did you get people to buy in? And then maybe what were one or two of the pivotal moments early on that kind of made it the standard for these things?Wei Lin [00:02:12]: Yeah, yeah. Chatbot Arena project was started last year in April, May, around that. Before that, we were basically experimenting in a lab how to fine tune a chatbot open source based on the Llama 1 model that I released. At that time, Lama 1 was like a base model and people didn't really know how to fine tune it. So we were doing some explorations. We were inspired by Stanford's Alpaca project. So we basically, yeah, grow a data set from the internet, which is called ShareGPT data set, which is like a dialogue data set between user and chat GPT conversation. It turns out to be like pretty high quality data, dialogue data. So we fine tune on it and then we train it and release the model called V2. And people were very excited about it because it kind of like demonstrate open way model can reach this conversation capability similar to chat GPT. And then we basically release the model with and also build a demo website for the model. People were very excited about it. But during the development, the biggest challenge to us at the time was like, how do we even evaluate it? How do we even argue this model we trained is better than others? And then what's the gap between this open source model that other proprietary offering? At that time, it was like GPT-4 was just announced and it's like Cloud One. What's the difference between them? And then after that, like every week, there's a new model being fine tuned, released. So even until still now, right? And then we have that demo website for V2 now. And then we thought like, okay, maybe we can add a few more of the model as well, like API model as well. And then we quickly realized that people need a tool to compare between different models. So we have like a side by side UI implemented on the website to that people choose, you know, compare. And we quickly realized that maybe we can do something like, like a battle on top of ECLMs, like just anonymize it, anonymize the identity, and that people vote which one is better. So the community decides which one is better, not us, not us arguing, you know, our model is better or what. And that turns out to be like, people are very excited about this idea. And then we tweet, we launch, and that's, yeah, that's April, May. And then it was like first two, three weeks, like just a few hundred thousand views tweet on our launch tweets. And then we have regularly double update weekly, beginning at a time, adding new model GPT-4 as well. So it was like, that was the, you know, the initial.Anastasios [00:04:58]: Another pivotal moment, just to jump in, would be private models, like the GPT, I'm a little,Wei Lin [00:05:04]: I'm a little chatty. That was this year. That was this year.Anastasios [00:05:07]: Huge.Wei Lin [00:05:08]: That was also huge.Alessio [00:05:09]: In the beginning, I saw the initial release was May 3rd of the beta board. On April 6, we did a benchmarks 101 episode for a podcast, just kind of talking about, you know, how so much of the data is like in the pre-training corpus and blah, blah, blah. And like the benchmarks are really not what we need to evaluate whether or not a model is good. Why did you not make a benchmark? Maybe at the time, you know, it was just like, Hey, let's just put together a whole bunch of data again, run a, make a score that seems much easier than coming out with a whole website where like users need to vote. Any thoughts behind that?Wei Lin [00:05:41]: I think it's more like fundamentally, we don't know how to automate this kind of benchmarks when it's more like, you know, conversational, multi-turn, and more open-ended task that may not come with a ground truth. So let's say if you ask a model to help you write an email for you for whatever purpose, there's no ground truth. How do you score them? Or write a story or a creative story or many other things like how we use ChatterBee these days. It's more open-ended. You know, we need human in the loop to give us feedback, which one is better. And I think nuance here is like, sometimes it's also hard for human to give the absolute rating. So that's why we have this kind of pairwise comparison, easier for people to choose which one is better. So from that, we use these pairwise comparison, those to calculate the leaderboard. Yeah. You can add more about this methodology.Anastasios [00:06:40]: Yeah. I think the point is that, and you guys probably also talked about this at some point, but static benchmarks are intrinsically, to some extent, unable to measure generative model performance. And the reason is because you cannot pre-annotate all the outputs of a generative model. You change the model, it's like the distribution of your data is changing. New labels to deal with that. New labels are great automated labeling, right? Which is why people are pursuing both. And yeah, static benchmarks, they allow you to zoom in to particular types of information like factuality, historical facts. We can build the best benchmark of historical facts, and we will then know that the model is great at historical facts. But ultimately, that's not the only axis, right? And we can build 50 of them, and we can evaluate 50 axes. But it's just so, the problem of generative model evaluation is just so expansive, and it's so subjective, that it's just maybe non-intrinsically impossible, but at least we don't see a way. We didn't see a way of encoding that into a fixed benchmark.Wei Lin [00:07:47]: But on the other hand, I think there's a challenge where this kind of online dynamic benchmark is more expensive than static benchmark, offline benchmark, where people still need it. Like when they build models, they need static benchmark to track where they are.Anastasios [00:08:03]: It's not like our benchmark is uniformly better than all other benchmarks, right? It just measures a different kind of performance that has proved to be useful.Swyx [00:08:14]: You guys also published MTBench as well, which is a static version, let's say, of Chatbot Arena, right? That people can actually use in their development of models.Wei Lin [00:08:25]: Right. I think one of the reasons we still do this static benchmark, we still wanted to explore, experiment whether we can automate this, because people, eventually, model developers need it to fast iterate their model. So that's why we explored LM as a judge, and ArenaHard, trying to filter, select high-quality data we collected from Chatbot Arena, the high-quality subset, and use that as a question and then automate the judge pipeline, so that people can quickly get high-quality signal, benchmark signals, using this online benchmark.Swyx [00:09:03]: As a community builder, I'm curious about just the initial early days. Obviously when you offer effectively free A-B testing inference for people, people will come and use your arena. What do you think were the key unlocks for you? Was it funding for this arena? Was it marketing? When people came in, do you see a noticeable skew in the data? Which obviously now you have enough data sets, you can separate things out, like coding and hard prompts, but in the early days, it was just all sorts of things.Anastasios [00:09:31]: Yeah, maybe one thing to establish at first is that our philosophy has always been to maximize organic use. I think that really does speak to your point, which is, yeah, why do people come? They came to use free LLM inference, right? And also, a lot of users just come to the website to use direct chat, because you can chat with the model for free. And then you could think about it like, hey, let's just be kind of like more on the selfish or conservative or protectionist side and say, no, we're only giving credits for people that battle or so on and so forth. Strategy wouldn't work, right? Because what we're trying to build is like a big funnel, a big funnel that can direct people. And some people are passionate and interested and they battle. And yes, the distribution of the people that do that is different. It's like, as you're pointing out, it's like, that's not as they're enthusiastic.Wei Lin [00:10:24]: They're early adopters of this technology.Anastasios [00:10:27]: Or they like games, you know, people like this. And we've run a couple of surveys that indicate this as well, of our user base.Wei Lin [00:10:36]: We do see a lot of developers come to the site asking polling questions, 20-30%. Yeah, 20-30%.Anastasios [00:10:42]: It's obviously not reflective of the general population, but it's reflective of some corner of the world of people that really care. And to some extent, maybe that's all right, because those are like the power users. And you know, we're not trying to claim that we represent the world, right? We represent the people that come and vote.Swyx [00:11:02]: Did you have to do anything marketing-wise? Was anything effective? Did you struggle at all? Was it success from day one?Wei Lin [00:11:09]: At some point, almost done. Okay. Because as you can imagine, this leaderboard depends on community engagement participation. If no one comes to vote tomorrow, then no leaderboard.Anastasios [00:11:23]: So we had some period of time when the number of users was just, after the initial launch, it went lower. Yeah. And, you know, at some point, it did not look promising. Actually, I joined the project a couple months in to do the statistical aspects, right? As you can imagine, that's how it kind of hooked into my previous work. At that time, it wasn't like, you know, it definitely wasn't clear that this was like going to be the eval or something. It was just like, oh, this is a cool project. Like Wayland seems awesome, you know, and that's it.Wei Lin [00:11:56]: Definitely. There's in the beginning, because people don't know us, people don't know what this is for. So we had a hard time. But I think we were lucky enough that we have some initial momentum. And as well as the competition between model providers just becoming, you know, became very intense. Intense. And then that makes the eval onto us, right? Because always number one is number one.Anastasios [00:12:23]: There's also an element of trust. Our main priority in everything we do is trust. We want to make sure we're doing everything like all the I's are dotted and the T's are crossed and nobody gets unfair treatment and people can see from our profiles and from our previous work and from whatever, you know, we're trustworthy people. We're not like trying to make a buck and we're not trying to become famous off of this or that. It's just, we're trying to provide a great public leaderboard community venture project.Wei Lin [00:12:51]: Yeah.Swyx [00:12:52]: Yes. I mean, you are kind of famous now, you know, that's fine. Just to dive in more into biases and, you know, some of this is like statistical control. The classic one for human preference evaluation is humans demonstrably prefer longer contexts or longer outputs, which is actually something that we don't necessarily want. You guys, I think maybe two months ago put out some length control studies. Apart from that, there are just other documented biases. Like, I'd just be interested in your review of what you've learned about biases and maybe a little bit about how you've controlled for them.Anastasios [00:13:32]: At a very high level, yeah. Humans are biased. Totally agree. Like in various ways. It's not clear whether that's good or bad, you know, we try not to make value judgments about these things. We just try to describe them as they are. And our approach is always as follows. We collect organic data and then we take that data and we mine it to get whatever insights we can get. And, you know, we have many millions of data points that we can now use to extract insights from. Now, one of those insights is to ask the question, what is the effect of style, right? You have a bunch of data, you have votes, people are voting either which way. We have all the conversations. We can say what components of style contribute to human preference and how do they contribute? Now, that's an important question. Why is that an important question? It's important because some people want to see which model would be better if the lengths of the responses were the same, were to be the same, right? People want to see the causal effect of the model's identity controlled for length or controlled for markdown, number of headers, bulleted lists, is the text bold? Some people don't, they just don't care about that. The idea is not to impose the judgment that this is not important, but rather to say ex post facto, can we analyze our data in a way that decouples all the different factors that go into human preference? Now, the way we do this is via statistical regression. That is to say the arena score that we show on our leaderboard is a particular type of linear model, right? It's a linear model that takes, it's a logistic regression that takes model identities and fits them against human preference, right? So it regresses human preference against model identity. What you get at the end of that logistic regression is a parameter vector of coefficients. And when the coefficient is large, it tells you that GPT 4.0 or whatever, very large coefficient, that means it's strong. And that's exactly what we report in the table. It's just the predictive effect of the model identity on the vote. The other thing that you can do is you can take that vector, let's say we have M models, that is an M dimensional vector of coefficients. What you can do is you say, hey, I also want to understand what the effect of length is. So I'll add another entry to that vector, which is trying to predict the vote, right? That tells me the difference in length between two model responses. So we have that for all of our data. We can compute it ex post facto. We added it into the regression and we look at that predictive effect. And then the idea, and this is formally true under certain conditions, not always verifiable ones, but the idea is that adding that extra coefficient to this vector will kind of suck out the predictive power of length and put it into that M plus first coefficient and quote, unquote, de-bias the rest so that the effect of length is not included. And that's what we do in style control. Now we don't just do it for M plus one. We have, you know, five, six different style components that have to do with markdown headers and bulleted lists and so on that we add here. Now, where is this going? You guys see the idea. It's a general methodology. If you have something that's sort of like a nuisance parameter, something that exists and provides predictive value, but you really don't want to estimate that. You want to remove its effect. In causal inference, these things are called like confounders often. What you can do is you can model the effect. You can put them into your model and try to adjust for them. So another one of those things might be cost. You know, what if I want to look at the cost adjusted performance of my model, which models are punching above their weight, parameter count, which models are punching above their weight in terms of parameter count, we can ex post facto measure that. We can do it without introducing anything that compromises the organic nature of theWei Lin [00:17:17]: data that we collect.Anastasios [00:17:18]: Hopefully that answers the question.Wei Lin [00:17:20]: It does.Swyx [00:17:21]: So I guess with a background in econometrics, this is super familiar.Anastasios [00:17:25]: You're probably better at this than me for sure.Swyx [00:17:27]: Well, I mean, so I used to be, you know, a quantitative trader and so, you know, controlling for multiple effects on stock price is effectively the job. So it's interesting. Obviously the problem is proving causation, which is hard, but you don't have to do that.Anastasios [00:17:45]: Yes. Yes, that's right. And causal inference is a hard problem and it goes beyond statistics, right? It's like you have to build the right causal model and so on and so forth. But we think that this is a good first step and we're sort of looking forward to learning from more people. You know, there's some good people at Berkeley that work on causal inference for the learning from them on like, what are the really most contemporary techniques that we can use in order to estimate true causal effects if possible.Swyx [00:18:10]: Maybe we could take a step through the other categories. So style control is a category. It is not a default. I have thought that when you wrote that blog post, actually, I thought it would be the new default because it seems like the most obvious thing to control for. But you also have other categories, you have coding, you have hard prompts. We consider that.Anastasios [00:18:27]: We're still actively considering it. It's just, you know, once you make that step, once you take that step, you're introducing your opinion and I'm not, you know, why should our opinion be the one? That's kind of a community choice. We could put it to a vote.Wei Lin [00:18:39]: We could pass.Anastasios [00:18:40]: Yeah, maybe do a poll. Maybe do a poll.Swyx [00:18:42]: I don't know. No opinion is an opinion.Wei Lin [00:18:44]: You know what I mean?Swyx [00:18:45]: Yeah.Wei Lin [00:18:46]: There's no neutral choice here.Swyx [00:18:47]: Yeah. You have all these others. You have instruction following too. What are your favorite categories that you like to talk about? Maybe you tell a little bit of the stories, tell a little bit of like the hard choices that you had to make.Wei Lin [00:18:57]: Yeah. Yeah. Yeah. I think the, uh, initially the reason why we want to add these new categories is essentially to answer some of the questions from our community, which is we won't have a single leaderboard for everything. So these models behave very differently in different domains. Let's say this model is trend for coding, this model trend for more technical questions and so on. On the other hand, to answer people's question about like, okay, what if all these low quality, you know, because we crowdsource data from the internet, there will be noise. So how do we de-noise? How do we filter out these low quality data effectively? So that was like, you know, some questions we want to answer. So basically we spent a few months, like really diving into these questions to understand how do we filter all these data because these are like medias of data points. And then if you want to re-label yourself, it's possible, but we need to kind of like to automate this kind of data classification pipeline for us to effectively categorize them to different categories, say coding, math, structure, and also harder problems. So that was like, the hope is when we slice the data into these meaningful categories to give people more like better signals, more direct signals, and that's also to clarify what we are actually measuring for, because I think that's the core part of the benchmark. That was the initial motivation. Does that make sense?Anastasios [00:20:27]: Yeah. Also, I'll just say, this does like get back to the point that the philosophy is to like mine organic, to take organic data and then mine it x plus factor.Alessio [00:20:35]: Is the data cage-free too, or just organic?Anastasios [00:20:39]: It's cage-free.Wei Lin [00:20:40]: No GMO. Yeah. And all of these efforts are like open source, like we open source all of the data cleaning pipeline, filtering pipeline. Yeah.Swyx [00:20:50]: I love the notebooks you guys publish. Actually really good just for learning statistics.Wei Lin [00:20:54]: Yeah. I'll share this insights with everyone.Alessio [00:20:59]: I agree on the initial premise of, Hey, writing an email, writing a story, there's like no ground truth. But I think as you move into like coding and like red teaming, some of these things, there's like kind of like skill levels. So I'm curious how you think about the distribution of skill of the users. Like maybe the top 1% of red teamers is just not participating in the arena. So how do you guys think about adjusting for it? And like feels like this where there's kind of like big differences between the average and the top. Yeah.Anastasios [00:21:29]: Red teaming, of course, red teaming is quite challenging. So, okay. Moving back. There's definitely like some tasks that are not as subjective that like pairwise human preference feedback is not the only signal that you would want to measure. And to some extent, maybe it's useful, but it may be more useful if you give people better tools. For example, it'd be great if we could execute code with an arena, be fantastic.Wei Lin [00:21:52]: We want to do it.Anastasios [00:21:53]: There's also this idea of constructing a user leaderboard. What does that mean? That means some users are better than others. And how do we measure that? How do we quantify that? Hard in chatbot arena, but where it is easier is in red teaming, because in red teaming, there's an explicit game. You're trying to break the model, you either win or you lose. So what you can do is you can say, Hey, what's really happening here is that the models and humans are playing a game against one another. And then you can use the same sort of Bradley Terry methodology with some, some extensions that we came up with in one of you can read one of our recent blog posts for, for the sort of theoretical extensions. You can attribute like strength back to individual players and jointly attribute strength to like the models that are in this jailbreaking game, along with the target tasks, like what types of jailbreaks you want.Wei Lin [00:22:44]: So yeah.Anastasios [00:22:45]: And I think that this is, this is a hugely important and interesting avenue that we want to continue researching. We have some initial ideas, but you know, all thoughts are welcome.Wei Lin [00:22:54]: Yeah.Alessio [00:22:55]: So first of all, on the code execution, the E2B guys, I'm sure they'll be happy to helpWei Lin [00:22:59]: you.Alessio [00:23:00]: I'll please set that up. They're big fans. We're investors in a company called Dreadnought, which we do a lot in AI red teaming. I think to me, the most interesting thing has been, how do you do sure? Like the model jailbreak is one side. We also had Nicola Scarlini from DeepMind on the podcast, and he was talking about, for example, like, you know, context stealing and like a weight stealing. So there's kind of like a lot more that goes around it. I'm curious just how you think about the model and then maybe like the broader system, even with Red Team Arena, you're just focused on like jailbreaking of the model, right? You're not doing kind of like any testing on the more system level thing of the model where like, maybe you can get the training data back, you're going to exfiltrate some of the layers and the weights and things like that.Wei Lin [00:23:43]: So right now, as you can see, the Red Team Arena is at a very early stage and we are still exploring what could be the potential new games we can introduce to the platform. So the idea is still the same, right? And we build a community driven project platform for people. They can have fun with this website, for sure. That's one thing, and then help everyone to test these models. So one of the aspects you mentioned is stealing secrets, stealing training sets. That could be one, you know, it could be designed as a game. Say, can you still use their credential, you know, we hide, maybe we can hide the credential into system prompts and so on. So there are like a few potential ideas we want to explore for sure. Do you want to add more?Anastasios [00:24:28]: I think that this is great. This idea is a great one. There's a lot of great ideas in the Red Teaming space. You know, I'm not personally like a Red Teamer. I don't like go around and Red Team models, but there are people that do that and they're awesome. They're super skilled. When I think about the Red Team arena, I think those are really the people that we're building it for. Like, we want to make them excited and happy, build tools that they like. And just like chatbot arena, we'll trust that this will end up being useful for the world. And all these people are, you know, I won't say all these people in this community are actually good hearted, right? They're not doing it because they want to like see the world burn. They're doing it because they like, think it's fun and cool. And yeah. Okay. Maybe they want to see, maybe they want a little bit.Wei Lin [00:25:13]: I don't know. Majority.Anastasios [00:25:15]: Yeah.Wei Lin [00:25:16]: You know what I'm saying.Anastasios [00:25:17]: So, you know, trying to figure out how to serve them best, I think, I don't know where that fits. I just, I'm not expressing. And give them credits, right?Wei Lin [00:25:24]: And give them credit.Anastasios [00:25:25]: Yeah. Yeah. So I'm not trying to express any particular value judgment here as to whether that's the right next step. It's just, that's sort of the way that I think we would think about it.Swyx [00:25:35]: Yeah. We also talked to Sander Schulhoff of the HackerPrompt competition, and he's pretty interested in Red Teaming at scale. Let's just call it that. You guys maybe want to talk with him.Wei Lin [00:25:45]: Oh, nice.Swyx [00:25:46]: We wanted to cover a little, a few topical things and then go into the other stuff that your group is doing. You know, you're not just running Chatbot Arena. We can also talk about the new website and your future plans, but I just wanted to briefly focus on O1. It is the hottest, latest model. Obviously, you guys already have it on the leaderboard. What is the impact of O1 on your evals?Wei Lin [00:26:06]: Made our interface slower.Anastasios [00:26:07]: It made it slower.Swyx [00:26:08]: Yeah.Wei Lin [00:26:10]: Because it needs like 30, 60 seconds, sometimes even more to, the latency is like higher. So that's one. Sure. But I think we observe very interesting things from this model as well. Like we observe like significant improvement in certain categories, like more technical or math. Yeah.Anastasios [00:26:32]: I think actually like one takeaway that was encouraging is that I think a lot of people before the O1 release were thinking, oh, like this benchmark is saturated. And why were they thinking that? They were thinking that because there was a bunch of models that were kind of at the same level. They were just kind of like incrementally competing and it sort of wasn't immediately obvious that any of them were any better. Nobody, including any individual person, it's hard to tell. But what O1 did is it was, it's clearly a better model for certain tasks. I mean, I used it for like proving some theorems and you know, there's some theorems that like only I know because I still do a little bit of theory. Right. So it's like, I can go in there and ask like, oh, how would you prove this exact thing? Which I can tell you has never been in the public domain. It'll do it. It's like, what?Wei Lin [00:27:19]: Okay.Anastasios [00:27:20]: So there's this model and it crushed the benchmark. You know, it's just like really like a big gap. And what that's telling us is that it's not saturated yet. It's still measuring some signal. That was encouraging. The point, the takeaway is that the benchmark is comparative. There's no absolute number. There's no maximum ELO. It's just like, if you're better than the rest, then you win. I think that was actually quite helpful to us.Swyx [00:27:46]: I think people were criticizing, I saw some of the academics criticizing it as not apples to apples. Right. Like, because it can take more time to reason, it's basically doing some search, doing some chain of thought that if you actually let the other models do that same thing, they might do better.Wei Lin [00:28:03]: Absolutely.Anastasios [00:28:04]: To be clear, none of the leaderboard currently is apples to apples because you have like Gemini Flash, you have, you know, all sorts of tiny models like Lama 8B, like 8B and 405B are not apples to apples.Wei Lin [00:28:19]: Totally agree. They have different latencies.Anastasios [00:28:21]: Different latencies.Wei Lin [00:28:22]: Control for latency. Yeah.Anastasios [00:28:24]: Latency control. That's another thing. We can do style control, but latency control. You know, things like this are important if you want to understand the trade-offs involved in using AI.Swyx [00:28:34]: O1 is a developing story. We still haven't seen the full model yet, but it's definitely a very exciting new paradigm. I think one community controversy I just wanted to give you guys space to address is the collaboration between you and the large model labs. People have been suspicious, let's just say, about how they choose to A-B test on you. I'll state the argument and let you respond, which is basically they run like five anonymous models and basically argmax their Elo on LMSYS or chatbot arena, and they release the best one. Right? What has been your end of the controversy? How have you decided to clarify your policy going forward?Wei Lin [00:29:15]: On a high level, I think our goal here is to build a fast eval for everyone, and including everyone in the community can see the data board and understand, compare the models. More importantly, I think we want to build the best eval also for model builders, like all these frontier labs building models. They're also internally facing a challenge, which is how do they eval the model? That's the reason why we want to partner with all the frontier lab people, and then to help them testing. That's one of the... We want to solve this technical challenge, which is eval. Yeah.Anastasios [00:29:54]: I mean, ideally, it benefits everyone, right?Wei Lin [00:29:56]: Yeah.Anastasios [00:29:57]: And people also are interested in seeing the leading edge of the models. People in the community seem to like that. Oh, there's a new model up. Is this strawberry? People are excited. People are interested. Yeah. And then there's this question that you bring up of, is it actually causing harm?Wei Lin [00:30:15]: Right?Anastasios [00:30:16]: Is it causing harm to the benchmark that we are allowing this private testing to happen? Maybe stepping back, why do you have that instinct? The reason why you and others in the community have that instinct is because when you look at something like a benchmark, like an image net, a static benchmark, what happens is that if I give you a million different models that are all slightly different, and I pick the best one, there's something called selection bias that plays in, which is that the performance of the winning model is overstated. This is also sometimes called the winner's curse. And that's because statistical fluctuations in the evaluation, they're driving which model gets selected as the top. So this selection bias can be a problem. Now there's a couple of things that make this benchmark slightly different. So first of all, the selection bias that you include when you're only testing five models is normally empirically small.Wei Lin [00:31:12]: And that's why we have these confidence intervals constructed.Anastasios [00:31:16]: That's right. Yeah. Our confidence intervals are actually not multiplicity adjusted. One thing that we could do immediately tomorrow in order to address this concern is if a model provider is testing five models and they want to release one, and we're constructing the models at level one minus alpha, we can just construct the intervals instead at level one minus alpha divided by five. That's called Bonferroni correction. What that'll tell you is that the final performance of the model, the interval that gets constructed, is actually formally correct. We don't do that right now, partially because we know from simulations that the amount of selection bias you incur with these five things is just not huge. It's not huge in comparison to the variability that you get from just regular human voters. So that's one thing. But then the second thing is the benchmark is live, right? So what ends up happening is it'll be a small magnitude, but even if you suffer from the winner's curse after testing these five models, what'll happen is that over time, because we're getting new data, it'll get adjusted down. So if there's any bias that gets introduced at that stage, in the long run, it actually doesn't matter. Because asymptotically, basically in the long run, there's way more fresh data than there is data that was used to compare these five models against these private models.Swyx [00:32:35]: The announcement effect is only just the first phase and it has a long tail.Anastasios [00:32:39]: Yeah, that's right. And it sort of like automatically corrects itself for this selection adjustment.Swyx [00:32:45]: Every month, I do a little chart of Ellim's ELO versus cost, just to track the price per dollar, the amount of like, how much money do I have to pay for one incremental point in ELO? And so I actually observe an interesting stability in most of the ELO numbers, except for some of them. For example, GPT-4-O August has fallen from 12.90

Jeremiah Roe has held many roles in cybersecurity:  Field CISO, Red Teamer, Advisor, Consultant, Etc.  He currently advises for OffSec, who provide quality cybersecurity training.  Drew Simonis and Allan Alford determined that Jeremiah would be a great guest for launching a 3-part mini series - each of the three shows exploring People, Process and Technology respectively. The three cover the following topics in a lively conversation that journeys into several aspects of People as they relate to cybersecurity: People, Process, and Technology - Which is most important? If they knew what we knew about cybersecurity, would they behave differently? How to leverage training budges for a win-win-win. People gonna peop, businesses gonna biz. Incentivization, Positive Reinforcement and Deputization Enabling camaraderie - not just good culture Groupthink and Tribalism Join the three as they ride the cyber trails of "People" in the PPT triad! Y'all be good now!

In this new episode of the Stories in AI podcast, I sit down with the Architect of Ethical AI Practice at Salesforce, Yoav Schlesinger. Yoav gives us a great look at how large industry players like Salesforce are implementing AI, looking ahead at future AI developments, and managing all the risks that come with managing large swaths of data. About Yoav: Yoav Schlesinger is an Architect of Ethical AI Practice at Salesforce, helping the company embed and instantiate ethical product practices to maximize the societal benefits of AI. Prior to coming to Salesforce, Yoav was a founding member of the Tech and Society Solutions Lab at Omidyar Network, where he launched the Responsible Computer Science Challenge and helped develop EthicalOS, a risk mitigation toolkit for product managers. Find him at: LinkedIn: https://www.linkedin.com/in/yschlesinger/

In this episode, hosts Nathan Labenz and Erik Torenberg delve into the exciting and concerning aspects of GPT4, the latest large multimodal model from OpenAI. Nathan, who was a Red Teamer for GPT4, shares his experience working with the model, using it for real-life scenarios, and how he found it to be human-level rather than human-like intelligence. Nathan also highlights some of its fundamental weaknesses and limitations and shares the most concerning elements of GPT4. They end the episode pondering the arrival of GPT5 and whether we are in AI's "goldilocks moment." RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix's culture deck Patty McCord. https://link.chtbl.com/hrheretics Timestamps for E11: Nathan Labenz and Erik Torenberg of The Cognitive Revolution Podcast (0:00) Preview of Nathan on this episode (1:13) Upstream: Erik's new interview show (1:41) Intro (5:40) Nathan's experience as a GPT4 Red Teamer (11:22) Catching the AI wave (14:30) Using GPT4 for real-life scenarios (17:33) Sponsor: Omneky (21:00) Human-level, not human-like intelligence (26:00) GPT4 weaknesses (28:14) More real-life use cases for GPT4 (35:25) Teaching AI to communicate within itself (40:25) GPT4's limitations (44:32) Nathan's learnings from using GPT4 (46:56) Nathan joining the Red Team (47:29) The most concerning thing about GPT4 (1:05:42) GPT5 (1:06:42) Open AI's regulatory breadcrumbs (1:13:52) AI's goldilocks moment Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Twitter: @CogRev_Podcast @labenz (Nathan) @eriktorenberg (Erik) Join 1000's of subscribers of our Substack: https://cognitiverevolution.substack

Cybercriminals are motivated by one simple incentive - money. Their favorite tools are bots to leverage sophistication, scalability, and ease of use. The effect is the creation of the underground bot ecosystem. This community allows threat actors to work together and continually improve their tactics. They sell bypasses for rule-based anti-bot solutions to other less technical fraudsters. In this episode of CyberWire-X, the CyberWire's CSO, Chief Analyst, and Senior Fellow, Rick Howard, is joined in the first half by Hash Table member Etay Maor. Cato Networks' Senior Director Security Strategy. They discuss this reality that has put defenders at a serious disadvantage and the mitigation steps to consider for future attacks.. In the second half of the show, CyberWire podcast host Dave Bittner talks with our episode sponsor Kasada's founder Sam Crowther talking about what he saw first-hand as a red teamer at a major Australian bank and what inspired him to reimagine bot mitigation with the founding principle of undermining the attacker's ROI.

We have a special defcon edition of the show. Special guest Alh4zr3d, a red teamer and twitch streamer is here to talk a bit about what he does, how he got into it, and why it's technically not a cult.  Alh4zr3d twitch: https://www.twitch.tv/alh4zr3d Alh4zr3d discord: https://discord.gg/AFKDE6Ad3G More What the Shell? https://whattheshellpod.com 

What is in your Hackel Box, In this story we talk about being a Red Teamer, and the tools I use for network testing an penetration.

Today we have Covert Entry expert Deviant Ollam to talk about physical security, how he got into the industry, his stories from in the field as a Red Teamer, and how he is looking to change a phrase that many have adopted in the industry. https://deviating.net/ YouTube:https://www.youtube.com/user/DeviantOllam  Twitter:https://twitter.com/deviantollam Instagram:https://instagram.com/deviantollam GitHub:https://github.com/deviantollam Trainings:https://www.redteamalliance.com/RTCG.html

Gabe's curiosity for how things work as a child is later used as a red teamer, but not leveraged until years later.Gabe shares his story and how his approach to learning and his career reflects how he prepares for 100 mile runs._______________________GuestGabe ThompsonOn Linkedin | https://www.linkedin.com/in/gabethompson1811/On Twitter | https://twitter.com/grnbeltwarrior______________________HostPhillip WylieOn ITSPmagazine  

Gabe's curiosity for how things work as a child is later used as a red teamer, but not leveraged until years later.Gabe shares his story and how his approach to learning and his career reflects how he prepares for 100 mile runs._______________________GuestGabe ThompsonOn Linkedin | https://www.linkedin.com/in/gabethompson1811/On Twitter | https://twitter.com/grnbeltwarrior______________________HostPhillip WylieOn ITSPmagazine  

Joe Vest started his journey as an aspiring herbalist, then his focus turned to IT working as a sysadmin.Joe's knack for technology as a sysadmin led him to cybersecurity and then red teaming when the discipline was not very known or practiced._______________________GuestJoe VestOn Linkedin | https://www.linkedin.com/in/joe-vest/On Twitter | https://twitter.com/joevest______________________HostPhillip WylieOn ITSPmagazine  

Joe Vest started his journey as an aspiring herbalist, then his focus turned to IT working as a sysadmin.Joe's knack for technology as a sysadmin led him to cybersecurity and then red teaming when the discipline was not very known or practiced._______________________GuestJoe VestOn Linkedin | https://www.linkedin.com/in/joe-vest/On Twitter | https://twitter.com/joevest______________________HostPhillip WylieOn ITSPmagazine  

In this episode of Hack Chat, we dive into the mindset of learning new tools of the trade and discuss how Mubix continues to elevate his skills. Mubix is a Red Teamer who is always looking for a challenge and willing to learn anything to overcome it. Mubix continues to push the limits of his skills, and is one of the most influential people on Twitter discussing everything Red Teaming.Learn more about the Hack Chat series here: https://www.sentinelone.com/lp/hackchat/ Learn more about SentinelOne here: https://www.sentinelone.com

A "Suplex" is an offensive move used in wrestling by which an attacker uses his weight to throw a defender.Phillip Wylie’s unusual journey into the field of cybersecurity is preceded by his career as a powerlifter and pro wrestler for the WCW. He has since taken full control in the industry as an offensive security professional. He is an established Red Teamer, Pentester, Ethical Hacking Instructor, and founder of the PWN School Project - an education focused cybersecurity organization that offers free pentesting and ethical hacking education to the public. Phillip and I overtake the bar to discuss his exhilarating path into cybersecurity, bug bounties, ethical hacking learning resources, and more!Tony the Bartender pins a Negroni.Support the show (https://paypal.me/thebarcodepodcast)

Episode 6 of the podcast covers insights into the impacts of threat intelligence from the perspective of the ultimate red-teamer, Tyler Robinson of Nisos, Inc. Intro (00:27) Question 1 (01:29) – As a red-teamer, when you're doing your job, do you feel threatened by threat feeds, or things like the MITRE Attack framework? Question 2 (03:37) – So you're a sophisticated adversary – do you actually take these things into account in preparing your attack? IE pulling in threat feeds, and matching your approach against frameworks that the target may use. Question 3 (07:00) – So you've done a lot of this over the years, what keeps you up at night, and I'd guess it's probably similar for the black hats out there, when it comes to your success in compromising a target's infrastructure? Question 4 (09:49) – What are the gaps that you see when it comes to modern threat intelligence? Question 5 (11:13) – Where do you see the industry moving in the next year or two? Recap & Take-aways (14:54)

Hva driver Oddvar med som gjør at han risikerer at Politiet eller FBI plutselig står på døren? Og hvordan forhindre han det skjer. Det høres kanskje ut som en fortelling fra underverden, men det er bare en vanlig dag på jobb for Oddvar som jobber som Red teamer i TrustedSec.Finn ut hva Sikkerhet kan ha tilfelles med Ikea møbler, hva betyr de forskjellige fargerike teamene når vi snakker om sikkerhet og mye annen nyttig informasjonFølg oddvar på Twitter @Oddvarmoe eller bloggen https://oddvar.moe/

In this episode, veteran hacker and red teamer Tom Van de Wiele answers questions from our listeners. Tom covers the ethics of ethical hacking, how to prioritize solving the myriad of security issues companies face, why he includes a banana in his hacking kit, the importance of communication skills in his job, and much more. A great listen both for those already in the industry and those wanting to break in. Links: Episode 29 transcript Episode 2 - Breaking into Infosec: Advice from an Ethical Hacker 21 tips for getting started in ethical hacking

This week, we welcome Tony Punturiero, Community Manager at Offensive Security, to talk about the journey of turning from a Blue Teamer to a Red Teamer, and kick starting an InfoSec community! In the Security News, BlackHat USA 2019 breaks records once again, new flaws in Qualcomm Chips expose Android devices to hacking, DEFCON 27 badge hacking for beginners, the CapitalOne hacker may have stolen from more than 30 companies, and a new data breach that exposed millions of fingerprint and facial recognition records! In our final segment, we air three Pre-Recorded interviews from the SE Village and BT Village from DEFCON 27, with O'Shea Bowens, Tyler Robinson, and Aaran Leyland!   Full Show Notes: https://wiki.securityweekly.com/Episode616 Visit https://www.securityweekly.com/psw for all the latest episodes!   Visit https://securityweekly.com/acm to sign up for a demo or buy our AI Hunter! Follow us on Twitter: https://www.twitter.com/securityweekly Like us on Facebook: https://www.facebook.com/secweekly

This week, we welcome Tony Punturiero, Community Manager at Offensive Security, to talk about the journey of turning from a Blue Teamer to a Red Teamer, and kick starting an InfoSec community! In the Security News, BlackHat USA 2019 breaks records once again, new flaws in Qualcomm Chips expose Android devices to hacking, DEFCON 27 badge hacking for beginners, the CapitalOne hacker may have stolen from more than 30 companies, and a new data breach that exposed millions of fingerprint and facial recognition records! In our final segment, we air three Pre-Recorded interviews from the SE Village and BT Village from DEFCON 27, with O'Shea Bowens, Tyler Robinson, and Aaran Leyland!   Full Show Notes: https://wiki.securityweekly.com/Episode616 Visit https://www.securityweekly.com/psw for all the latest episodes!   Visit https://securityweekly.com/acm to sign up for a demo or buy our AI Hunter! Follow us on Twitter: https://www.twitter.com/securityweekly Like us on Facebook: https://www.facebook.com/secweekly

In this episode I catch up with an old coworker who I worked with on vulnerability scanning software and has since become a red-teamer or “adversarial simulation”. He started as a gamer, and we talk about how he learned the foundation of his hacking skills from his days of gaming. Chris opens up on the … Continue reading "Episode 21: Chris “Lopi” Spehn, Game Hacker to Red Teamer"

We talk about the need for skillset, why constantly learning new ones make sense. The conversation cernts around the need to be a generalist as a Red Teamer and security professional in general.

Permalink and Transcript: http://gettingintoinfosec.com/dan In this first episode, I chat with Dan Borges, a professional red teamer, blogger, and security tool developer. Dan discusses his early experiences using and exploiting computer systems, how InfoSec experts work with companies, and a new tools he and other created and released this year! Episode Highlights: Dan explains how he became involved in information security, including his introduction to programming through a Lego robotics program. His early experiences as a pen-tester—i.e. a penetration tester, who looks for system security weaknesses—and why it’s difficult to get hands-on experience in that field. The benefits of becoming an Offensive Security Certified Professional (OSCP). What does a red team do in an organization, and how is it different from pen-testing? Dan describes the day-to-day life of a pen-tester and the kind of conflicts they can run into. A few war stories from the trenches of InfoSec, as well as some of the tools pen-testers use. How being grounded led to Dan’s earliest hacking experiences, and the ways his parents fostered his interests and mentality. What conferences should InfoSec beginners check out? Fun and beneficial ways you can “hack” reading. Dan’s tips for those starting off or looking to transition into Infosec. An in-depth look at one of the newer tools Dan uses for his work. The rules and intricacies of InfoSec competitions. Quotes: “It’s such a catch-22 to get practical, hands-on experience to go to these jobs because, y’know, hacking’s illegal, right?” “We don’t just go in and blow the brakes off people, we’re trying to measurably improve security.” “It was a constant escalation war, cat-and-mouse like that. They’d take something away and I’d figure out how to use the computer with that limitation.” Links: Dan Borges’ personal blog: http://lockboxx.blogspot.com/ Dan’s LinkedIn: https://www.linkedin.com/in/borges1337/ Dan on Twitter: https://twitter.com/1njection Dan and Alex's DEFCON Talk on Gscript: https://www.youtube.com/watch?v=8yjMlMf8NpQ Gscript: Genesis Scripting Engine: https://github.com/gen0cide/gscript NationalCPTC (Collegiate Penetration Testing Competition): https://nationalcptc.org/ Outro Music: Missing You by Trash80: https://trash80.bandcamp.com/track/missing-you Getting Into Infosec: Twitter: https://twitter.com/coffeewithayman YouTube: https://www.youtube.com/channel/UCg6gV_gdfc188HZdN8LUx4A Book: https://www.amazon.com/Breaking-Step-Step-Starting-Information-ebook/dp/B07N15GTPC/

Is Nerdcore the future of Infosec and Hip Hop? OHM-I shows the way. At your wit’s end because of APT attacks? Tried everything but the bad guys keep finding creative ways in? Have you considered listening to Nerdcore hip hop for inspiration? No? You’re missing out. Yo, you need fortify heavily I see your database services and version 1 SMB You know what this nerd gon’ do, payloads I’mma turn on you That quad 4 make you sad forever, that’s ETERNALBLUE That’s just a taste of the knowledge OHM-I has in store for you. In this episode of InSecurity, Matt Stephenson sits down with Nerdcore superstar OHM-I to find out what is happening in the world of Nerdcore Hip Hop and where it intersects with the world of cybersecurity. OHM-I has spent over a decade in both games and has plenty to say about his time in each. Did legends like Grandmaster Flash and Rick Rubin hack music in the formative days of Hip Hop? What role can tech companies play in bringing young black men into STEM programs at an early age? Is there such a thing as Nerdcore Country music?  About OHM-I, AKA Leron Gray Leron Gray is the secret identity of Nerdcore superstar MC and producer and OHM-I. Leron spent 10 years in the Navy including a recent post as a Cryptologic Technician. With a history of network vulnerability assessments, incident response and network reconstruction, he now enters the private sector as a pen-tester and Red-Teamer. With multiple LPs, EPs and singles to his credit, OHM-I is a stalwart on the Nerdcore scene. Look for him at events like Magfest, SXSW, Nerdapalooza and other Nerd-friendly venues. Dig into his virtual crates on Bandcamp and Soundcloud to hear more!   About Matt Stephenson Insecurity Podcast host Matt Stephenson (@packmatt73) leads the Security Technology team at Cylance, which puts him in front of crowds, cameras, and microphones all over the world. He is the regular host of the InSecurity podcast and host of CylanceTV Twenty years of work with the world’s largest security, storage, and recovery companies has introduced Stephenson to some of the most fascinating people in the industry. He wants to get those stories told so that others can learn from what has come before. Every week on the InSecurity Podcast, Matt interviews leading authorities in the security industry to gain an expert perspective on topics including risk management, security control friction, compliance issues, and building a culture of security. Each episode provides relevant insights for security practitioners and business leaders working to improve their organization’s security posture and bottom line. To hear more, visit: ThreatVector InSecurity Podcasts: https://threatvector.cylance.com/en_us/category/podcasts.html iTunes/Apple Podcasts link: https://itunes.apple.com/us/podcast/insecurity/id1260714697?mt=2 GooglePlay Music link: https://play.google.com/music/listen#/ps/Ipudd6ommmgdsboen7rjd2lvste Look for the InSecurity Podcast with Matt Stephenson on Spotify and Soundcloud!

Zach Schlumpf is the Recruiting Coordinator for IOActive. An Army Veteran, former Red Teamer, and Seattle Locksport volunteer, Zach joins us to discuss recruiting, social engineering, and the balance between technical and soft skills! Full Show Notes: https://wiki.securityweekly.com/SSWEpisode64 Visit http://securityweekly.com/category/ssw for all the latest episodes!

Zach Schlumpf is the Recruiting Coordinator for IOActive. An Army Veteran, former Red Teamer, and Seattle Locksport volunteer, Zach joins us to discuss recruiting, social engineering, and the balance between technical and soft skills! Full Show Notes: https://wiki.securityweekly.com/SSWEpisode64 Visit http://securityweekly.com/category/ssw for all the latest episodes!

Red Teams.  For some, it's the "frenemy".  For others, it's the greener grass on the other side of the defence wall.  In this episode I spend some time speaking with security consultant Mark Kikta about Red Teaming.  Mark has been a Red Teamer for a while and has a lot of experience to share.  We talk about a number of different things, share some laughs and try to shed some light on an often misunderstood group. Mark has also graciously offered to hang out in our Slack channel!  Just message @mark to get in touch with him if you have questions or just want to say "hey". Some links of interest: CircleCityCon - Seeing Purple Hybrid Security Teams for the Enterprise Time Based Security Slack Sign-Up Link: https://signup.purplesquadsec.com Want to reach out to the show?  There's a few ways to get in touch! Show Twitter: @PurpleSquadSec John's Twitter: @JohnsNotHere Website: purplesquadsec.com Slack Sign-Up Link: https://signup.purplesquadsec.com John's Peerlyst Profile: https://www.peerlyst.com/users/john-svazic Thanks for listening, and I will talk with you again next time!Find out more at http://purplesquadsec.com