Podcasts about Syslog

Libfuzzer goes into maintenance-only mode and syslog vulnerabilities plague some vendors in this week's episode. Links and vulnerability summaries for this episode are available at: https://dayzerosec.com/podcast/240.html [00:00:00] Introduction [00:00:20] LibFuzzer in Maintainence-only Mode [00:11:41] Heap-based buffer overflow in the glibc's syslog() [CVE-2023-6246] [00:26:33] Hunting for ~~Un~~authenticated n-days in Asus Routers [00:34:44] Inside the LogoFAIL PoC: From Integer Overflow to Arbitrary Code Execution [00:35:51] Chaos Communication Congress (37C3) recap [00:36:51] GitHub - google/oss-fuzz-gen: LLM powered fuzzing via OSS-Fuzz. The DAY[0] Podcast episodes are streamed live on Twitch twice a week: -- Mondays at 3:00pm Eastern (Boston) we focus on web and more bug bounty style vulnerabilities -- Tuesdays at 7:00pm Eastern (Boston) we focus on lower-level vulnerabilities and exploits. We are also available on the usual podcast platforms: -- Apple Podcasts: https://podcasts.apple.com/us/podcast/id1484046063 -- Spotify: https://open.spotify.com/show/4NKCxk8aPEuEFuHsEQ9Tdt -- Google Podcasts: https://www.google.com/podcasts?feed=aHR0cHM6Ly9hbmNob3IuZm0vcy9hMTIxYTI0L3BvZGNhc3QvcnNz -- Other audio platforms can be found at https://anchor.fm/dayzerosec You can also join our discord: https://discord.gg/daTxTK9

In this livestream, Ahmed Kira and I provided more details about the Cribl Stream Reference Architecture, which is designed to help observability admins achieve faster and more valuable stream deployment. We explained the guidelines for deploying the comprehensive reference architecture to meet the needs of large customers with diverse, high-volume data flows. Then, we shared different use cases and discussed their pros and cons.  Cribl's Reference Architectures provide a way for admins to get 70% of the way towards deploying Cribl Stream. The sample environment below is a template for sending data to many destinations while minimizing data egress costs. It incorporates solutions to some of the challenges typical larger organizations might face.  MS Azure Worker Group In this sample environment, the leader is up in Cribl Cloud and managed by Cribl. On the right-hand side, you'll see an Azure worker group. There are two reasons to consider putting a worker group in a different cloud provider. The first is to be as close to the data you're collecting as possible. By keeping the data close, you can minimize the amount of processing necessary and cut egress costs. With this setup, you're also reducing the risks of having competing workloads. Failing small is much better than failing big. Additionally, when establishing a security or observability data lake, you don't need to put all that data in the same data lake, S3 bucket, or blob storage. With Cribl, you can have them in different places and still be able to replay against all of that data. We often see customers with Azure and AWS workers using Cribl-to-Cribl connectivity between the two clouds to exchange data. This way, they can avoid building custom code or dealing with the vagaries of exchanging data between clouds. On-Prem General-Purpose Worker Group The next worker group in our sample architecture above is an on-prem, general-purpose worker group. With this worker group, you can combine most of your data sources and have them go to one worker group in your data center. This is especially useful if you have a lot of Splunk universal forwarders, Cribl Edge agents, and Filebeat agents — you'll want to send those to a dedicated worker group so you're not competing for different workloads. Another big reason for this approach is segmentation. For example, if you need to separate your PCI or PHI workflow, you can use this setup to break up your data or meet compliance requirements. If you need to upload that data to an Elastic or Splunk cloud, having the Cribl Stream worker group allows you to stage your data, manage it, and get it to those destinations. Syslog Worker Group Another architectural consideration worth looking into is having one Syslog worker group. This allows you to do your commit-and-deploys once instead of one region at a time. A lot of organizations struggle with the contention that high-volume Syslog causes. Adding an agent workload can make the situation worse, so having separate worker groups allows you to scale. The difference between this worker group and others is Syslog groups have load balancers that will send data to the local workers in that data center. In Cribl Stream, there will still be one logical Syslog worker group to manage, reducing administrative burden and the maintenance required. If you take one thing away from reading this post or watching the live stream, please DO NOT send your data to a single Syslog destination port! You'll get the best results by getting as many workers involved as possible — do everything you can to avoid being pinned to a single core. Cribl Cloud Worker Group With Cribl Cloud, you will also get at least one worker group by default that you can allocate to all your AWS data sources — like in the sample architecture. But you can also send all of your cloud, on-prem, and other non-AWS data sources there. Either way, you won't have to manage as much infrastructure. Instead, you can leverage the Cribl Cloud worker group and the Cribl Cloud leader if your use case allows for it. This is especially important for threat surface reduction. Taking data in from multiple SaaS platforms means opening up your perimeter to everything that Cloudflare could produce, which is probably half the entire internet. Cribl Cloud can handle all of those threats and keep you secure. Replay Worker Group The last worker group in this reference architecture that people don't typically consider is the Replay worker group. It's a great practice to allocate your replays to a separate worker group, where the workload can be spun up and spun down — instead of on your production worker groups where you're processing real-time streaming data. Using your production worker group for replay can suddenly add terabytes of data to your existing live data flows and slow everything down. A minimal-cost, ephemeral replay worker group lets you scale up to meet your needs without interrupting your production workloads. A recent customer took advantage of this by deploying their replay worker group in AWS ECS. As more data gets requested and downloaded, ECS spins up additional instances. The worker group scales larger as more data is retrieved and then scales down if there's nothing to do. Choice and Control Over All of Your Data When you have multiple worker groups, you don't have to worry about going to different places to manage them — it can all still be done by one Cribl leader. You can also have multiple data lakes and replay from all of them via one central location within Cribl. This flexibility gives you complete control to make the best choices for you. So, if your security team wants to use Azure for its data lake and your operations team wants to use AWS, it's no problem. Or, if you want to use one S3 bucket for forensics and another for yearly retention, you have that option available. The best part is that all the data in your data lake is vendor-neutral. You can return that data to Cribl Stream using replay and send it to any tool you want. Check out the full live stream for insights on integrating Cribl Stream into any environment, enabling faster value realization with minimal effort. Our goal is to assist SecOps and Observability data admins in spending less time figuring out how to use Cribl Stream and more time getting value. Don't miss out on this opportunity to enhance your observability administration skills. More Videos in our Cribl Reference Architecture Series Introduction to the Cribl Stream Reference Architecture How the All in One Worker Group Fits Into the Cribl Stream Reference Architecture Scaling Syslog Scaling Effectively for a High Volume of Agents How SpyCloud Architected its Cribl Stream Deployment  

In this live stream, Cribl's Ed Bailey and Ahmed Kira go into more detail about the Cribl Stream Reference Architecture, with a focus on scaling syslog. They share a few use cases, some guidelines for handling high-volume UDP and TCP syslog traffic, and talk about the pros and cons of some of the different approaches to tackling this challenge.  

Avi Freedman, CEO at Kentik, joins Corey on Screaming in the Cloud to discuss the fun of solving for observability. Corey and Avi discuss how great simplicity can be deceiving, and Avi points out that with great simplicity comes great complexity. Avi discusses examples of this that he sees in Kentik customer environments, as well as the differences he sees in cloud environments from traditional data center environments. Avi also reveals his predictions for the future and how enterprise M&A will affect the way companies view data centers and VPCs. About AviAvi Freedman is the co-founder and CEO of network observability company Kentik. He has decades of experience as a networking technologist and executive. As a network pioneer in 1992, Freedman started Philadelphia's first ISP, known as netaxs. He went on to run network operations at Akamai for over a decade as VP of network infrastructure and then as chief network scientist. He also ran the network at AboveNet and was the CTO of ServerCentral.Links Referenced: Kentik: https://kentik.com Email: avi@kentik.com Twitter: https://twitter.com/avifreedman LinkedIn: https://www.linkedin.com/in/avifreedman TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Most Companies find out way too late that they've been breached. Thinkst Canary changes this. Deploy Canaries and Canarytokens in minutes and then forget about them. Attackers tip their hand by touching 'em giving you the one alert, when it matters. With 0 admin overhead and almost no false-positives, Canaries are deployed (and loved) on all 7 continents. Check out what people are saying at canary.love today!Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. This promoted guest episode is brought to us by our friends at Kentik. And into my social grist mill, they have thrown Avi Freedman, their CEO. Avi, thank you for joining me.Avi: Thank you for having me, Corey. I've been a big fan for some time, I have never actually fallen off my seat laughing, but I've come close a couple times on some of your threads.Corey: You must have a great chair.Avi: I should probably upgrade it [laugh].Corey: [laugh]. I have been looking forward to this conversation for a while because you are one of those rare creatures who comes from a similar world to what I did where we were grumpy and old before our time because we worked on physical infrastructure in data centers, we basically wrangled servers into doing the things that we wanted them to do when hardware reliability was an aspiration rather than a reality. And we also moved on from that, in many ways. We are not blind to the modern order of how computers work. But you still run a lot of what you do in data centers, but many of your customers are in cloud. You speak both languages very fluently because of the unifying thread between all of this, which is, of course, the network. How did you wind up in, I guess we'll call it network hell.Avi: [laugh]. I mean, network hell was truly… in the '90s, when the internet was—I mean, the internet is sort of like the human body: the more you study it, the more amazing it is that it ever worked in the first place, not that it breaks sometimes—was the bugs, and trying to put together the technology back then, you know, that we had the life is pretty good nowadays, other than the [laugh] immense complexity that has been unleashed on us by everyone taking the same technology and then writing it in their own software and giving it their own marketing names. And thus, you have multi-cloud networking. So, got into it because it's a problem that needs to be solved, right? There's no ESP that connects the applications together; the network still needs to make it work. And now people own some of it, and then more of it, they don't own, but they're still responsible for it. So, it's a fun problem to solve.Corey: The timing of this episode is apt because I've used Kentik myself for a few things over the years. And to be fair, using it for any of my personal networking problems is a bit like noticing, “Oh, I have a loose thread here on my shirt. Pass me the chainsaw.” It's, my environment is tiny and it's over-scoped. But I just earlier this week wound up having to analyze a day's worth of Flow Logs from one of my clients, and to do this, I had to spin up an EC2 instance with 128 gigs of RAM and then load the Flow Logs for that day into RAM, and then—not kidding—I ran into OOM Killer because I ran out of RAM on this thing.Avi: [laugh].Corey: It is, like, yeah, that's right. The network is chatty, the logs are immense, and it's easy to forget. Because the reason I was doing this was just to figure out what are the things that are talking to each other in this environment to drive up some aspects of data transfer costs. But that is an esoteric use case for this; it's not why most people tend to think about network observability. So, I'm going to ask you the blunt question up front here because it might be a really short episode. Do we have to care about networking in the least now that cloud is the default in most locations? It is just an API call away, isn't it?Avi: With great simplicity comes great complexity. So, to the people running infrastructure, to developers or architects, turning it all on, it looks like just API calls. But did you set the policies right? Can the things talk to each other? Are they talking in patterns that are causing you wild data transfer costs?All these things ultimately come back to some team that actually has to make it go. And can be pretty hard to figure that out, right, because it's not just the VPC Flow Logs. It's, what's the policy? It's, what are they talking to that maybe isn't in that cloud, that's maybe in another cloud? So, how do you bring it all together? Like, you could have—and maybe you should have—used Athena, right? You can put VPC Flow Logs in S3 buckets and use Athena and run SQL queries if all you want is your top talker.Corey: Oh, I did. That's how I started, but Athena is, uh… it has some challenges. Let's just put it that way and leave it there. DuckDB is what I was using and I'm much happier with it for a variety of excellent reasons.Avi: Okay. Well, I'll tease you another time about, you know—I lost this battle at Kentik. We actually don't use swap, but I'm a big fan of having swap and monitoring it so the OOM Killer only does what you want or doesn't fire at all. But that's a separate religious debate.Corey: There's a counterargument of running an in-memory data store. And then oh, we're going to use it as swap though, so it's like, hang on, this just feels like running a normal database with extra steps.Avi: Computers allow you to do amazing things and only occasionally slap you nowadays with it. It's pretty amazing. But back to the question. APIs make it easy to turn on, but not so easy to run. The observability that you get within a given cloud is typically very limited.Google actually has the best. They show some topology and other things. I mean, a lot of what we do involves scraping API calls in the cloud to figure out what does this all mean, then convolving it with the VPC Flow Logs and making it look like a network, and what are the gateways, and what are the rules being applied and what can't talk to itself? If you just look at VPC Flow Logs like it's Syslog, good luck trying to figure out what VPCs are talking to each other. It's exactly the problem that you were describing.So, the ease of turning it on is exactly inversely proportional to the ease of running it. And, you know, as a vendor, we think it's an awesome [laugh] problem, but we feel for our customers. And you know, occasionally it's a pain to get the IAM roles set up to scrape things and help them, but that's you know, that's just part of the job.Corey: It's fascinating to me, just looking from an AWS perspective, just how much work clearly has to be done to translate their Byzantine and very strange networking environment and concepts into things that customers see. Because in many cases, the things that the virtual machines that we've run on top of EC2, let alone anything higher level, is being lied to the entire time about what the actual topology of the environment is. It was most notable, for me at least, at re:Invent 2022, the most recent one, where they announced they have a TCP replacement, scalable, reliable data grammar SRD. It's a new protocol entirely. It's, “Oh, wow, can we use it?” “No.” “Okay.” Like, I get that it's a lot of work, I get you're excited about it. Are you going to talk to us about how it actually works? “Oh, absolutely not.” So… okay, good for you, I guess.Avi: Doesn't Amazon have to write a press release before they build anything, and doesn't the press release have to say, like, why people give a shit, why people care?Corey: Yep. And their story on this was oh, it enables us to be a lot faster at letting EBS volumes talk to some of our beefier instances.Avi: [laugh].Corey: And that's all well and good, don't get me wrong, but it's also, “Yay, it's more reliable,” is a difficult message to send. I mean, it's hard enough when—and it's necessary because you've got to tacitly admit that reliability and performance haven't been all they could be. But when it's no longer an issue for most folks, now you're making them wonder, like, wait, how bad was it? It's just a strange message.Avi: Yeah. One of my projects for this weekend is, I actually got a gaming PC and I'm going to try compression offload to the CUDA cores because right now, we do compress and decompress with Intel cores. And like, if I'm successful there and we can get 30% faster subqueries—which doesn't really matter, you know, on the kind of massive queries we run—and 20% more use out of the computers that we actually run, I'm probably not going to do a press release about it. But good to see the pattern.But you know, what you said is pretty interesting. As people like Kentik, we have to put together, well, on Azure, you can have VPCs that cross regions, right? And in other places, you can't. And in Google, you have performance metrics that come out and you can get it very frequently, and in Amazon and Azure, you can't. Like, how do you take these kinds of telemetry that are all the same stuff underneath, but packaged up differently in different quantos and different things and make it all look the same is actually pretty fun and interesting.And it's pretty—you know, if you give some cloud engineers who focus on the infrastructure layer enough beers or alcohol or just room to talk, you can hear some funny stories. And it all made sense to somebody in the first place, but unpacking it and actually running it as a common infrastructure can be quite fun.Corey: One of the things that I have found notable about your perspective, as particularly, you're running all of the network ingest, to my understanding, in your data center environment. Because we talked about this when you were kind enough to invite me to your company all-hands offsite, presumably I assume when people do that, it's so they can beat me up in the alley, but that only happened twice. I was very pleasantly surprised.Avi: [And you 00:09:23] made fun of us only three times, so you know, you beat us—Corey: Exactly.Avi: —but it was all enjoyed.Corey: But always with love. Now, what I found fascinating was you and I sat down for a while and you talked about your data center architecture. And you asked me—since I don't have anything to sell you—is there an economical way that I could see running your environment on top of AWS? And the answer was sure, if by economical you mean an absolute minimum of six times what you're currently paying a year, sure you can get there. But it just does not make sense for any realistic approach to doing this.And the reason I bring this up is that you're in a data center not because of religious beliefs, “Of, well, this is good enough for my grandpappy, so it's good enough for me.” It's because it solves the problem you have in a way that the cloud providers clearly cannot. But you also are not anti-cloud. So, many folks who are all-in on data centers seem to be doing it out of pure self-interest where, well, if everyone goes all-in on cloud, then we have nothing left to sell them. I've used AWS VPC Flow Logs. They have nothing that could even remotely be termed network observability. Your future is assured as long as people understand what it is that you're providing them and what are you that adds. So yeah, people keep going in a cloud direction, you're happy as houses.Avi: We'll use the best tools for building our infrastructure that we can, right? We use cloud. In fact, we're just buying some reserved instances, which always, you know, I give it the hairy eyeball, but you know, we're probably always going to have our CI/CD bursty stuff in the cloud. We have performance testing regions on all the major clouds so that we can tell people what performance is to and from cloud. Like, that we have to use cloud for.And if there's an always-on model, which starts making sense in the cloud, then I try not to be the first to use anything, but [laugh] we'll be one of the first to use it. But every year, we talk to, you know, the major clouds because we're customers of all them, for as I said, our testing infrastructure if nothing else, and you know, some of them for some other parts, you know, for example, proxying VPC Flow Logs, we run infrastructure on Kubernetes in all—in the three biggest to proxy VPC Flow Logs, you know, and so that's part of our bill. But if something's always on, you know, one of our storage servers, it's a $15,000 machine that, you know, realistically runs five years, but even if you assume it runs three years, we get financing for it, cost a couple $100 a month to host, and that's inclusive of our ops team that runs, sort of, everything, you just do the math. That same machine would be, you know, even not including data transfer would be maybe 3500 a month on cloud. The economics just don't quite make sense.For burst, for things like CI/CD, test, seasonality, I think it's great. And if we have patterns like that, you know, we're the first to use it. So, it's just a question of using what's best. And a lot of our customers are in that realm, too. I would say some of them are a little over-rotated, you know, they've had big mandates to go one way or the other and don't have the right, you know, sort of nuanced view, but I think over time, that's going to fix itself. And yeah, as you were saying, like, the more people use cloud, the better we do, so it's just really a question of what's the best for us at our infrastructure and at any given time.Corey: I think that that is something that is not fully appreciated or well understood is that I work with cloud technologies because for what I do, it makes an awful lot of sense. But I've been lately doing a significant build-out in my home network on the perspective of yeah, this makes sense for what I do. And I now have increased number of workloads that I'm running here and I got to say, it feels a little strange, on some level, not to be paying AWS on something metered by the second whenever I'm running a job here. That always feels a little on the weird side. But I'm not suggesting I filled my house with servers either.Avi: [unintelligible 00:13:18] going to report you to the House on Cloudian Activities Committee [laugh] for—Corey: [laugh].Avi: To straighten you out about your infrastructure use and beliefs. I do have to ask you, and I do have some foreknowledge of this, where is the controller for your network running? Is it running in your house or—Corey: Oh, the WiFi controller lives in Ohio with all the other unpleasant things. I mean, even data transfer between Ohio and Virginia—if you're on AWS—is half-price because data wants to get out of Ohio just as much as the people do. And that's fine, but it can also fail out of band. I can chill that thing for a while and I'm not able to provision new equipment, I can't spin up new SSIDs, but—Avi: Right. It's the same as [kale scale 00:14:00], which is, like, sufficiently indistinguishable from magic, but it's nice there's [head scale 00:14:05] in case something happened to them. But yeah, you know, you just can't set up new stuff without your SSHing old way while it's down. So.Corey: And worst case, it goes away irretrievably, I can spin a new one up, I can pair everything locally, do it by repointing DNS locally, and life will go on. It's one of those areas where, like, I would not have this in Ohio if latency was a concern if it was routing every packet out halfway across the country before it hit the general internet. That would be a challenge for me. But that's not what I'm doing.Avi: Yeah, yeah. No, that makes sense. And I think also—Corey: And I certainly pay AWS by the second for that thing. That's—I have a three-year savings plan for that thing, and if nothing else, it was useful for me just to figure out what the living hell was going on with the savings plan purchase project one year. That was just, it was challenged to get that straightened out in some ways. Turns out that the high watermark of the console is a hundred-and-some-odd-thirty-million dollars you can add to cart and click the buy button. Have fun.Avi: My goodness. Okay, well.Corey: The API goes up to $26.2 billion. Try that in a free tier account, preferably someone else's.Avi: I would love to have such problems. Right now, that is not one of them. We don't spend that much on infrastructure.Corey: Oh, that is more than Amazon's—AWS's at least—quarterly revenue. So, if you wind up doing a $26.2 billion, it's like—it's that old saw. You owe Amazon a million dollars, you have a problem. If you owe Amazon $26 billion, Amazon has a problem. Yeah, that's when Andy Jassy calls you 20 minutes after you make that purchase, and at least to me, he yells at me with a, “Listen here, asshole,” and it sort of devolves from there.Avi: Well, I do live in Seattle, so you know, they send the posse out, I'm pretty sure.Corey: [laugh] I will be keynoting DevOpsDays Seattle on August 1st with a talk that might very well resonate with your perspective, “The Modern Devops: A Million Ways to Die in Production.”Avi: That is very cool. I mean, ultimately, I think that's what cloud comes back to. When cloud was being formed, it's just other people's computers, storage, and network. I don't know if you'd argue that there's a politics, control plane, or a—Corey: Oh, I would say, “Cloud? There's no cloud; just someone else's cost center.”Avi: Exactly. And so, how do you configure it? And back to the question of, should everything be on-prem or does cloud abstract at all, it's all the same stuff that we've been doing for decades and decades, just with other people's software and names, which you help decode. And then it's the question we've always had: what's the best thing to do? Do you like [Wellfleet 00:16:33] or [Protion 00:16:35]? Now, do you like Azure [laugh] or Google or Amazon or somebody else or running your own?Corey: It's almost this generation's equivalent of Vi versus Emacs.Avi: Yes. I guess there could be a crowd equivalent. I use VI, but only because I'm a lisp addict and I don't want to get stuck refining Eliza macros and connecting to the ChatGPT in Emacs. So, you know. Someone just did a Emacs as PID 0. So basically, no init, just, you know, the kernel boots into Emacs, and then someone of course had to do a VI as PID 0. And I have to admit, Emacs would be a lot more useful as a PID 0, even though I use VI.Corey: I would say that—I mean, you wind up in writing in Emacs and writing lisp in it, then I've got to say every third thing you say becomes a parenthetical.Avi: Exactly. Ha.Corey: But I want to say that there's also a definite moving of data going on there that I think is a scale that, for those of us working mostly in home labs and whatnot, can be hard to imagine. And I see that just in terms of the volume of Flow Logs, which to be clear, are smaller than the data transfer they are representing in almost every case.Avi: Almost every.Corey: You see so much of the telemetry that comes out of there and what customers are seeing and what their problems are, in different ways. It's not just Flow Logs, you ingest a whole bunch of different telemetry through a variety of modern and ancient and everything in between variety of protocols to support, you know, the horror that is network equipment interoperability. And just, I can't—I feel like I can't do a terrific job of doing justice to describing just how comprehensive Kentik is, once you get it set up as a product. What is on the wire has always been for me the arbiter of truth because computers will lie to you, but it's very tricky to get them to lie and get the network story to cover for it.Avi: Right. I mean, ultimately, that's one of the sources of truth. There's routing, there's performance testing, there's a whole lot of different things, and as you were saying, in any one of these slices of your, let's just pick the network. There's many different things that all mean the same, but look different that you need to put together. You could—the nerd term would be, you know, normalizing. You need to take all this stuff and normalize it.But traffic, we agree, that's where we started with. We call it the what if what is. What's actually happening on the infrastructure and that's the ancient stuff like IPFIX and NetFlow and sFlow. Some people that would argue that, you know, the [IATF 00:19:04] would say, “Oh, we're still innovating and it's still current,” but you know, it's certainly on-prem only. The major cloud vendors would say, “Oh, well, you can run the router—cloud routers—or you could run cloud versions of the big routers,” but we don't really see that as a super common pattern today.But what's really the difference between NetFlow and the VPC Flow Log? Well, some VPC Flow Logs have permit deny because they're really firewall logs, but ultimately, it's something went from here to there. There might not be a TCP flag, but there might be something else in cloud. And, you know, maybe there's rum data, which is also another kind of traffic. And ultimately, all together, we try to take that and then the business metadata to say, whether it's NetBox in the old world or Kubernetes in the new world, or some other [unintelligible 00:19:49], what application is this? What user is this?So, you can ask questions about why am I blowing up between these cloud regions? What applications are doing it, right? VPC Flow Logs by themselves don't know that, so you need to add that kind of metadata in. And then there's performance testing, which is sort of the what is. Something we do, Thousand Eyes does, some other people do.It's not the actual source of truth, but for example, if you're having a performance problem getting between, you know, us-east and Azure in the east, well, there's three other ways you can get there. If your actual traffic isn't getting there that way, then how do you know which one to use? Well, let's fire up some tests. There's all the metrics on what all of the devices are reporting, just like you get metrics from your machines and from your applications, and then there's stuff even up at the routing layer, which God help you, hopefully you don't need to actually get in and debug, but sometimes you do. And sometimes, you know, your neighbor tells the mailman that that mail is for me and not for you and they believe them and then you have a big problem when your bills don't get paid.The same thing happens in the cloud, the same thing happens on the internet [unintelligible 00:20:52] at the routing. So, the goal is, take all the different sources of it, make it the same within each type, and then pull it all together so you can look at a single place, you can look at a map, you can look at everything, whether it's the cloud, whether it's your own data centers, your own WAN, into the internet and in between in a coherent way that understands your application. So, it's a small task that we've bit off, but you know, we have fun solving it.Corey: Do you find that when you look at customer environments, that they are, and I don't mean to be disparaging here, truly I don't, but if you were to ask me to design something today, I would probably not even be using VPCs if I'm doing this completely greenfield. I would be a lot more cloud-first, et cetera, et cetera. Whereas in many cases, that is not the right path, especially if you know, customers have the temerity to not be founded within the last 18 months before AWS existed in some ways. Do you find that the majority of what they're doing looks like they're treating the cloud like data centers or do you find that they are leveraging cloud in ways that surprise you and would not be possible in traditional data centers? Because I can't shake the feeling that the network has a source of truth for figuring out what's really going on [is very hard to beat 00:22:05].Avi: Yes, for the most part, to both your assertion at the end and sort of the question. So, in terms of the question, for the most part, people think of VPCs as… you know, they could just equivalent be VLANs and [unintelligible 00:22:21], right? I've got policies, and I have these things that are talking to each other, and everything else is not local. And I've got—you know, it's not a perfect mapping to physical interfaces in VLANs but it's the equivalent of that.And that is sort of how people think about it. In the data center, you'd call it micro-segmentation, in the cloud, you call it clouding, but you know, just applying all the same policies and saying this stuff can talk to each other and not. Which is always sort of interesting, if you don't actually know what is talking [laugh] to each other to apply those policies. Which is a lot of what you know, Kentik gets brought in for first. I think where we see the cloud-native thinking, which is overlaid on top of that—you could call it overlay, I guess—which is service mesh.Now, putting aside the question of what's going to be a service mesh, what's going to be a network mesh, where there's something like [unintelligible 00:23:13] sit, the idea that there's a way that you look at traffic above the packets at, you know, layers three to more layer seven, that can do things like load balancing, do things like telemetry, do things like policy enforcement, that is a layer that we see very commonly that a lot of the old school folks have—you know, they want their lsu F5s and they want their F5 script. And they're like, “Why can't I have this in the cloud?”—which I guess you could buy it from F5 if you really want—but that's pretty common. Now, not everything's a sidecar anymore and there's still debates about what's going on there, but that's pretty common, even where the underlying cloud just looks like it could just be a data center.And that seems to be state of the art, I would say, our traditional enterprise customers, for sure. Our web company customers, and you know, service providers use cloud more for their OTT and some other things. As we work with them, they're a little bit more likely to be on-prem, you know, historic. But remember, in the enterprise, there's still a lot of M&A going on, I think that's even going to pick up in the next couple of years and a lot of what they're doing is lift-and-shift of [laugh] actual data centers. And my theory is, it's got to be easier to just make it look like VPCs than completely redo it.Corey: I'd say that there's reasons that things are the way that they are. Like, ignoring that this is the better approach from a technical perspective entirely because that's often not the only answer, it's we have assurances we made as part of audit compliance regimes, of our SOC 2, of how we handle certain things and what those controls are. And yeah, it's not hard for even a junior employee, most of the time, to design a reasonable architecture on a whiteboard. The problem is, how do you take something pre-existing and get it to a state that closely resembles that while not turning it off for a long time?Avi: Right. And I think we're starting to see some things that probably shouldn't exist, like, people trying to do VXLAN as overlays into and between VPCs because that was how their data s—you know, they're more modern on the data center side and trying to do that. But generally, I think people have an understanding they need to be designing architecture for greenfield things that aren't too far bleeding edge, unless it's like a pure developer shop, and also can map to the least common denominator kinds of infrastructure that people have. Now, sometimes that may be serverless, which means, you know, more CDN use and abstracted layers in front, but for, you know, running your own components, we see a lot of differences but also a lot of commonality. It's differences at the micro but commonality the macro. And I don't know what you see in your practice. So.Corey: I will say that what I see in practice is that there's a dichotomy where you have born-in-the-cloud companies where 80% of their spend is on a single workload and you can do a whole bunch of deep optimizations. And then you see the conglomerate approach where it's giant spend, but it's all very diffuse across 1500 different applications. And different philosophies, different processes, different cultures give rise to a lot of these things. I will say that if I had a magic wand, I would—and again, the fact that you sponsor and promote this episode is deeply appreciated. Thank you—Avi: You're welcome.Corey: —but it does not mean that you get to compromise my authenticity and objectivity. You can't buy my opinion, just my attention. But I will say this, that I would love it if my customers used Kentik because it is one of the best things I've ever seen to describe what is talking to what that scale and in volume without going super deep into the weeds. Now, obviously, I'm not allowed to start rolling out random things into customer environments. That's how I get sued to death. But, ugh, I wish it was there.Avi: You probably shouldn't set up IAM rules without asking them, yes. That wouldn't be bad.Corey: There's a reason that the only writable stuff that I have access to is generating reports in Cost Explorer.Avi: [laugh]. Okay.Corey: Everything else is read-only. All we do is to have conversations with folks. It sets context for those conversations. I used to think that we'd be doing this as a software offering. I no longer believe that actually solves the in-depth problems that people have.Avi: Well, I appreciate the praise. I even take some of the backhanded praise slash critique at the beginning because we think a lot about, you know, we did design for these complex and often hybrid infrastructures and it's true, we didn't design it for the two or four router, you know, infrastructure. If we had bootstrapped longer, if we'd done some other things, we might have done it that way. We don't want to be exclusionary. It's just sort of how we focus.But in the kind of customers that you have, these are things that we're thinking about what can we do to make it easier to onboard because people have these massive challenges seeing the traffic and understanding it and the cost and security and the performance, but to do more with the VPC Flow Logs, we need to get some of those metrics. We think about should we make an open-source thing. I don't know how much you've seen the concern that people have universally across cloud providers that they turn on something like Kentik, and they're going to hit their API rate limiter. Which is like, really, you can't build a cache for that at the scale that these guys run at, the large cloud providers. I don't really understand that. But it is what it is.We spent a lot of time thinking about that because of security policy, and getting the kind of metrics that we need. You know, if we open-source some of that, would it make it easier, plug it into people's observability infrastructure, we'd like to get that onboarding time down, even for those more complex infrastructures. But you know, the payoff is there, you know? It only takes a day of elapsed time and one hour or so. It's just you got to get a lot of approvals to get the kind of telemetry that you need to make sense of this in some environments.Corey: Oh, yes. And that's part of the problem, too, is like, you could talk about one of those big environments where you have 1500 apps all talking to each other. You can't make sense of any of it without talking to people and having contacts and occasionally get a little bit of [unintelligible 00:29:07] just what these things are named. But at that point, you're just speculating wildly. And, you know, it's an engineering trap, where I'm just going to guess rather than asking someone who knows the answer because I don't want to look foolish. It's… you just three weeks chasing your own tail. Who's the foolish one?Avi: We're not in a competitive business to yours—Corey: [laugh].Avi: But I do often ask when we're starting off, “So, can you point us at the source of truth that describes what all your applications are?” And usually, they're, like, “[laugh]. No.” But you know, at the same time to make sense of this stuff, you also need that metadata and that's something that we designed to be able to take.Now, Kubernetes has some of that. You may have some of it in ServiceNow, a lot of people use. You may have it in your own text file, CSV somewhere. It may be in NetBox, which we've seen people actually use for the cloud, more on the web company and service provider side, but even some traditional enterprise is starting to use it. So, a lot of what we have to do as a vendor is put all that together because yeah, when you're running across multiple environments and thousands of applications, ultimately scrying at IP addresses and VPC IDs is not going to be sufficient.So, the good news is, almost everybody has those sources and we just tried to drag it out of them and pull it back together. And for now, we refuse to actually try to get into that business because it's not a—seems sort of like, you know, SAP where you're going to be sending consultants forever, and not as interesting as the problems we're trying to solve.Corey: I really want to thank you, not just for supporting the show of course, but also for coming here to suffer my slings and arrows. If people want to learn more, where's the best place for them to find you? And please don't respond with an IP address.Avi: 127.0.0.1. You're welcome at my home at any time.Corey: There's no place like localhost.Avi: There's no place like localhost. Indeed. So, the company is kentik.com, K-E-N-T-I-K. I am avi@kentik.com. I am@avifriedman on Twitter and LinkedIn and some other things. And happy to chat with nerds, infrastructure nerds, cloud nerds, network nerds, software nerds, debate, maybe not VI versus Emacs, but should you swap space or not, and what should your cloud architecture look like?Corey: And we will, of course, put links to that in the [show notes 00:31:20].Avi: Thank you.Corey: Thank you so much for being so generous with your time. I really appreciate it.Avi: Thank you for having this forum. And I will let you know when I am down in San Francisco with some time.Corey: I would be offended if you didn't take the time to at least say hello. Avi Friedman, CEO at Kentik. I'm Cloud Economist Corey Quinn, and this has been a promoted guest episode of Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a all five-star review on your podcast platform of choice, along with an angry comment saying how everything, start to finish, is somehow because of the network.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

In this live stream discussion, Eugene Katz and I explain the importance of a quality reference architecture in successful software deployment and guide viewers on how to begin with the Cribl Stream Reference Architecture. They help users establish end-state goals, share different use cases, and help data administrators identify which parts of the reference architecture apply to their specific situation. It's also available on our podcast feed if you want to listen on the go. If you want to automatically get every episode of the Stream Life podcast, you can subscribe on your favorite podcast app. The Cribl Stream Reference Architecture serves as a starting point for incorporating our vendor-agnostic observability pipeline into your existing IT and Security architecture. We know firsthand how difficult it can be to onboard and deploy new tools — mistakes were certainly made when we launched back— so we designed this information to help you get 70-80% of the way to a scalable deployment of our flagship product, Cribl Stream. It's impossible to account for all the variability in IT, but this framework should be a useful tool in helping set up your particular environment and avoiding a lot of pain points as you grow. Keepeep in mind that applying the considerations here within the context of your network and security architecture is just as important as any of the technical guidance. Establish Your End State Goal First The most important thing you can do with any new deployment or takeover of existing deployment is to define your end state at the beginning. For something mission critical — like your logging, telemetry, or especially security logging — you have to decide on your business objective before anything else. Let's say you want a scalable platform that can survive failure to a certain level — what is that level? It's good to know the average amount of data that gets processed on a good day, but what happens on a bad day? This is a very important discussion to have with your business leaders because it's essential for your telemetry and security to work when everything's going badly. You have to be able to reverse engineer how many cores, systems load balancers, etc you'll need to have in place — otherwise, you're just picking a number out of thin air and rolling the dice. You could also miss out on an opportunity to align with your capacity team on the amount of hardware you'll need. General Sizing Considerations and Planning for Failure CPU We generally recommend allocating one physical core for each 400GB/day of IN+OUT throughput. For virtual cores, you'll need 200 GB/day, but it'll still be the same number of worker processes. There are more details in our Sizing and Scaling documentation for Graviton vs Intel-based work processes, as well as recommendations for which VMs to choose for AWS or Azure deployments. As far as headroom for handling data spikes goes — that's where distributed deployment comes in. You'll distribute not only across the different worker processes and individual worker nodes, but you'll also have multiple worker nodes and scale out horizontally. With Stream, you can not only pass all of your data through it, but you can also process your data along the way. You can account for more regex or turning Windows XML into JSON by using the pipeline profiling feature to run a sample and see how long the expression might be taking — just note that variations will depend on each user's specific situation. Memory Big aggregations or large lookups get loaded into memory for each worker process and take up space, and each worker process gets about 2GB of memory by default. We learned about this the hard way — when we started loading in those giant lookups we suddenly started eating a whole lot more memory. JSON is more CPU-bound than a memory-hungry application, but as you expand your use cases, you've got to be ready to add more memory and resources as appropriate. Disk Size, Speed & Persistent Cues Stream offers two different options for writing to disk if you have a situation where one of your destinations is experiencing an outage or slowdown. Instead of losing that data or stopping its flow altogether, you can set up a source-persistent or destination-persistent queue as a temporary solution, and once the destination is ready it will start sending those persistent events in. Once the destination is restored, the data in a source-persistent queue will go through your whole pipeline, so it will take up a lot of resources as it flows all the way through to the destination. On the other hand, a destination-persistent queue will require less resources, because that data has already gone through the whole pipeline. Destination queues are a great way to have a buffer in situations where you're gathering the data in a data center in another country and passing it into your security data lake before it's processed. This leaves you with options in the case of failure. This is an area where your original business objectives come in — how will you size your persistent queue? Will you have an hour-long buffer, or maybe a 24-hour buffer? Be sure to think through these situations before they arise. Connection Management Managing connections is tough, especially when you're working with thousands of data sources, universal forwarders, and pieces of network gear that need to be configured. We recommend always having load balancers available if you're going to be working with agentless protocols like Syslog, TCP Syslog, UDP Syslog, HEC, and HTTP — but make sure you manage that connection overhead and don't point everything at one server, or you'll find yourself in a world of trouble. Once you're done balancing the load across the different workers, you have to account for the total number of connections — 400 per CPU core is manageable, but it will depend on your EPS. If you have more than 250 connections per core, then you need to start thinking about testing what's optimal for your architecture. What is your EPS and how sustained is it? How many forwarders do you have? How fast are they writing? Do you have big senders? Single Worker Groups vs. Multiple Worker Groups A single, or all-in-one, worker group is appropriate for small-medium sized enterprises working with less than or near 1T of data per day. If your sources are small enough to handle spikes or are unlikely to reach capacity, then this type of architecture may be appropriate. A setup involving multiple worker groups is necessary for larger organizations or if you have sensitive or complex data to process. The first thing that customers will do is split up pull and push worker groups. Push worker groups like data from Syslog in universal forwarders are usually consistent, but the pull side of things can be a different story. Mixing the data you're pulling down from CrowdStrike, which has a series of huge spikes followed by no data flow, might be problematic. Your pull sources will also be managed by the leader in terms of scheduling, so you want to make sure that you have those sources fairly close to the leader to avoid running into network latency, and potentially having skipped pulls. These are just some of the things to consider in the design of your enterprise's architecture. Watch the live stream on Introducing the Cribl Stream Reference Architecture to get more detail and insights on integrating Cribl Stream into any environment, enabling faster value realization with minimal effort. This is the first of many discussions on the Cribl Stream Reference Architecture, tailored to SecOps and Observability data admins. Take advantage of this opportunity to empower your observability administration skills, and stay tuned for future conversations that will dive deeper into each of the topics discussed here.

Finding Gaps in Syslog https://isc.sans.edu/diary/Finding%20Gaps%20in%20Syslog%20-%20How%20to%20find%20when%20nothing%20happened/29314 Internet Explorer Vulnerabilty used in Malicious Word Document https://blog.google/threat-analysis-group/internet-explorer-0-day-exploited-by-north-korean-actor-apt37/ Zombinder Obfuscation Service used by Ermac https://www.threatfabric.com/blogs/zombinder-ermac-and-desktop-stealers.html Cisco IP Phone Vulnerability CVE-2022-20968 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ipp-oobwrite-8cMF5r7U daloRADIUS Vulnerablity CVE-2022-23475 https://securityonline.info/cve-2022-23475-account-take-over-flaw-in-open-source-radius-web-management-app/ SANS Holiday Hack Challenge https://www.sans.org/mlp/holiday-hack-challenge/

Finding Gaps in Syslog https://isc.sans.edu/diary/Finding%20Gaps%20in%20Syslog%20-%20How%20to%20find%20when%20nothing%20happened/29314 Internet Explorer Vulnerabilty used in Malicious Word Document https://blog.google/threat-analysis-group/internet-explorer-0-day-exploited-by-north-korean-actor-apt37/ Zombinder Obfuscation Service used by Ermac https://www.threatfabric.com/blogs/zombinder-ermac-and-desktop-stealers.html Cisco IP Phone Vulnerability CVE-2022-20968 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ipp-oobwrite-8cMF5r7U daloRADIUS Vulnerablity CVE-2022-23475 https://securityonline.info/cve-2022-23475-account-take-over-flaw-in-open-source-radius-web-management-app/ SANS Holiday Hack Challenge https://www.sans.org/mlp/holiday-hack-challenge/

About IanIan Smith is Field CTO at Chronosphere where he works across sales, marketing, engineering and product to deliver better insights and outcomes to observability teams supporting high-scale cloud-native environments. Previously, he worked with observability teams across the software industry in pre-sales roles at New Relic, Wavefront, PagerDuty and Lightstep.Links Referenced: Chronosphere: https://chronosphere.io Last Tweet in AWS: lasttweetinaws.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I find that something I'm working on aligns perfectly with a person that I wind up basically convincing to appear on this show. Today's promoted guest is Ian Smith, who's Field CTO at Chronosphere. Ian, thank you for joining me.Ian: Thanks, Corey. Great to be here.Corey: So, the coincidental aspect of what I'm referring to is that Chronosphere is, despite the name, not something that works on bending time, but rather an observability company. Is that directionally accurate?Ian: That's true. Although you could argue it probably bend a little bit of engineering time. But we can talk about that later.Corey: [laugh]. So, observability is one of those areas that I think is suffering from too many definitions, if that makes sense. And at first, I couldn't make sense of what it was that people actually meant when they said observability, this sort of clarified to me at least when I realized that there were an awful lot of, well, let's be direct and call them ‘legacy monitoring companies' that just chose to take what they were already doing and define that as, “Oh, this is observability.” I don't know that I necessarily agree with that. I know a lot of folks in the industry vehemently disagree.You've been in a lot of places that have positioned you reasonably well to have opinions on this sort of question. To my understanding, you were at interesting places, such as LightStep, New Relic, Wavefront, and PagerDuty, which I guess technically might count as observability in a very strange way. How do you view observability and what it is?Ian: Yeah. Well, a lot of definitions, as you said, common ones, they talk about the three pillars, they talk really about data types. For me, it's about outcomes. I think observability is really this transition from the yesteryear of monitoring where things were much simpler and you, sort of, knew all of the questions, you were able to define your dashboards, you were able to define your alerts and that was really the gist of it. And going into this brave new world where there's a lot of unknown things, you're having to ask a lot of sort of unique questions, particularly during a particular instance, and so being able to ask those questions in an ad hoc fashion layers on top of what we've traditionally done with monitoring. So, observability is sort of that more flexible, more dynamic kind of environment that you have to deal with.Corey: This has always been something that, for me, has been relatively academic. Back when I was running production environments, things tended to be a lot more static, where, “Oh, there's a problem with the database. I will SSH into the database server.” Or, “Hmm, we're having a weird problem with the web tier. Well, there are ten or 20 or 200 web servers. Great, I can aggregate all of their logs to Syslog, and worst case, I can log in and poke around.”Now, with a more ephemeral style of environment where you have Kubernetes or whatnot scheduling containers into place that have problems you can't attach to a running container very easily, and by the time you see an error, that container hasn't existed for three hours. And that becomes a problem. Then you've got the Lambda universe, which is a whole ‘nother world pain, where it becomes very challenging, at least for me, in order to reason using the old style approaches about what's actually going on in your environment.Ian: Yeah, I think there's that and there's also the added complexity of oftentimes you'll see performance or behavioral changes based on even more narrow pathways, right? One particular user is having a problem and the traffic is spread across many containers. Is it making all of these containers perform badly? Not necessarily, but their user experience is being affected. It's very common in say, like, B2B scenarios for you to want to understand the experience of one particular user or the aggregate experience of users at a particular company, particular customer, for example.There's just more complexity. There's more complexity of the infrastructure and just the technical layer that you're talking about, but there's also more complexity in just the way that we're handling use cases and trying to provide value with all of this software to the myriad of customers in different industries that software now serves.Corey: For where I sit, I tend to have a little bit of trouble disambiguating, I guess, the three baseline data types that I see talked about again and again in observability. You have logs, which I think I've mostly I can wrap my head around. That seems to be the baseline story of, “Oh, great. Your application puts out logs. Of course, it's in its own unique, beautiful format. Why wouldn't it be?” In an ideal scenario, they're structured. Things are never ideal, so great. You're basically tailing log files in some cases. Great. I can reason about those.Metrics always seem to be a little bit of a step beyond that. It's okay, I have a whole bunch of log lines that are spitting out every 500 error that my app is throwing—and given my terrible code, it throws a lot—but I can then ideally count the number of times that appears and then that winds up incrementing counter, similar to the way that we used to see with StatsD, for example, and Collectd. Is that directionally correct? As far as the way I reason about, well so far, logs and metrics?Ian: I think at a really basic level, yes. I think that, as we've been talking about, sort of greater complexity starts coming in when you have—particularly metrics in today's world of containers—Prometheus—you mentioned StatsD—Prometheus has become sort of like the standard for expressing those things, so you get situations where you have incredibly high cardinality, so cardinality being the interplay between all the different dimensions. So, you might have, my container is a label, but also the type of endpoint is running on that container as a label, then maybe I want to track my customer organizations and maybe I have 5000 of those. I have 3000 containers, and so on and so forth. And you get this massive explosion, almost multiplicatively.For those in the audience who really live and read cardinality, there's probably someone screaming about well, it's not truly multiplicative in every sense of the word, but, you know, it's close enough from an approximation standpoint. As you get this massive explosion of data, which obviously has a cost implication but also has, I think, a really big implication on the core reason why you have metrics in the first place you alluded to, which is, so a human being can reason about it, right? You don't want to go and look at 5000 log lines; you want to know, out of those 5000 log lines of 4000 errors and I have 1000, OKs. It's very easy for human beings to reason about that from a numbers perspective. When your metrics start to re-explode out into thousands, millions of data points, and unique sort of time series more numbers for you to track, then you're sort of losing that original goal of metrics.Corey: I think I mostly have wrapped my head around the concept. But then that brings us to traces, and that tends to be I think one of the hardest things for me to grasp, just because most of the apps I build, for obvious reasons—namely, I'm bad at programming and most of these are proof of concept type of things rather than anything that's large scale running in production—the difference between a trace and logs tends to get very muddled for me. But the idea being that as you have a customer session or a request that talks to different microservices, how do you collate across different systems all of the outputs of that request into a single place so you can see timing information, understand the flow that user took through your application? Is that again, directionally correct? Have I completely missed the plot here? Which is again, eminently possible. You are the expert.Ian: No, I think that's sort of the fundamental premise or expected value of tracing, for sure. We have something that's akin to a set of logs; they have a common identifier, a trace ID, that tells us that all of these logs essentially belong to the same request. But importantly, there's relationship information. And this is the difference between just having traces—sorry, logs—with just a trace ID attached to them. So, for example, if you have Service A calling Service B and Service C, the relatively simple thing, you could use time to try to figure this out.But what if there are things happening in Service B at the same time there are things happening in Service C and D, and so on and so forth? So, one of the things that tracing brings to the table is it tells you what is currently happening, what called that. So oh, I know that I'm Service D. I was actually called by Service B and I'm not just relying on timestamps to try and figure out that connection. So, you have that information and ultimately, the data model allows you to fully sort of reflect what's happening with the request, particularly in complex environments.And I think this is where, you know, tracing needs to be sort of looked at as not a tool for—just because I'm operating in a modern environment, I'm using some Kubernetes, or I'm using Lambda, is it needs to be used in a scenario where you really have troubles grasping, from a conceptual standpoint, what is happening with the request because you need to actually fully document it. As opposed to, I have a few—let's say three Lambda functions. I maybe have some key metrics about them; I have a little bit of logging. You probably do not need to use tracing to solve, sort of, basic performance problems with those. So, you can get yourself into a place where you're over-engineering, you're spending a lot of time with tracing instrumentation and tracing tooling, and I think that's the core of observability is, like, using the right tool, the right data for the job.But that's also what makes it really difficult because you essentially need to have this, you know, huge set of experience or knowledge about the different data, the different tooling, and what influential architecture and the data you have available to be able to reason about that and make confident decisions, particularly when you're under a time crunch which everyone is familiar with a, sort of like, you know, PagerDuty-style experience of my phone is going off and I have a customer-facing incident. Where is my problem? What do I need to do? Which dashboard do I need to look at? Which tool do I need to investigate? And that's where I think the observability industry has become not serving the outcomes of the customers.Corey: I had a, well, I wouldn't say it's a genius plan, but it was a passing fancy that I've built this online, freely available Twitter client for authoring Twitter threads—because that's what I do is that of having a social life—and it's available at lasttweetinaws.com. I've used that as a testbed for a few things. It's now deployed to roughly 20 AWS regions simultaneously, and this means that I have a bit of a problem as far as how to figure out not even what's wrong or what's broken with this, but who's even using it?Because I know people are. I see invocations all over the planet that are not me. And sometimes it appears to just be random things crawling the internet—fine, whatever—but then I see people logging in and doing stuff with it. I'd kind of like to log and see who's using it just so I can get information like, is there anyone I should talk to about what it could be doing differently? I love getting user experience reports on this stuff.And I figured, ah, this is a perfect little toy application. It runs in a single Lambda function so it's not that complicated. I could instrument this with OpenTelemetry, which then, at least according to the instructions on the tin, I could then send different types of data to different observability tools without having to re-instrument this thing every time I want to kick the tires on something else. That was the promise.And this led to three weeks of pain because it appears that for all of the promise that it has, OpenTelemetry, particularly in a Lambda environment, is nowhere near ready for being able to carry a workload like this. Am I just foolish on this? Am I stating an unfortunate reality that you've noticed in the OpenTelemetry space? Or, let's be clear here, you do work for a company with opinions on these things. Is OpenTelemetry the wrong approach?Ian: I think OpenTelemetry is absolutely the right approach. To me, the promise of OpenTelemetry for the individual is, “Hey, I can go and instrument this thing, as you said and I can go and send the data, wherever I want.” The sort of larger view of that is, “Well, I'm no longer beholden to a vendor,”—including the ones that I've worked for, including the one that I work for now—“For the definition of the data. I am able to control that, I'm able to choose that, I'm able to enhance that, and any effort I put into it, it's mine. I own that.”Whereas previously, if you picked, say, for example, an APM vendor, you said, “Oh, I want to have some additional aspects of my information provider, I want to track my customer, or I want to track a particular new metric of how much dollars am I transacting,” that effort really going to support the value of that individual solution, it's not going to support your outcomes. Which is I want to be able to use this data wherever I want, wherever it's most valuable. So, the core premise of OpenTelemetry, I think, is great. I think it's a massive undertaking to be able to do this for at least three different data types, right? Defining an API across a whole bunch of different languages, across three different data types, and then creating implementations for those.Because the implementations are the thing that people want, right? You are hoping for the ability to, say, drop in something. Maybe one line of code or preferably just, like, attach a dependency, let's say in Java-land at runtime, and be able to have the information flow through and have it complete. And this is the premise of, you know, vendors I've worked with in the past, like New Relic. That was what New Relic built on: the ability to drop in an agent and get visibility immediately.So, having that out-of-the-box visibility is obviously a goal of OpenTelemetry where it makes sense—Go, it's very difficult to attach things at runtime, for example—but then saying, well, whatever is provided—let's say your gRPC connections, database, all these things—well, now I want to go and instrument; I want to add some additional value. As you said, maybe you want to track something like I want to have in my traces the email address of whoever it is or the Twitter handle of whoever is so I can then go and analyze that stuff later. You want to be able to inject that piece of information or that instrumentation and then decide, well, where is the best utilized? Is it best utilized in some tooling from AWS? Is it best utilized in something that you've built yourself? Is it best of utilized an open-source project? Is it best utilized in one of the many observability vendors, or is even becoming more common, I want to shove everything in a data lake and run, sort of, analysis asynchronously, overlay observability data for essentially business purposes.All of those things are served by having a very robust, open-source standard, and simple-to-implement way of collecting a really good baseline of data and then make it easy for you to then enhance that while still owning—essentially, it's your IP right? It's like, the instrumentation is your IP, whereas in the old world of proprietary agents, proprietary APIs, that IP was basically building it, but it was tied to that other vendor that you were investing in.Corey: One thing that I was consistently annoyed by in my days of running production infrastructures at places, like, you know, large banks, for example, one of the problems I kept running into is that this, there's this idea that, “Oh, you want to use our tool. Just instrument your applications with our libraries or our instrumentation standards.” And it felt like I was constantly doing and redoing a lot of instrumentation for different aspects. It's not that we were replacing one vendor with another; it's that in an observability, toolchain, there are remarkably few, one-size-fits-all stories. It feels increasingly like everyone's trying to sell me a multifunction printer, which does one thing well, and a few other things just well enough to technically say they do them, but badly enough that I get irritated every single time.And having 15 different instrumentation packages in an application, that's either got security ramifications, for one, see large bank, and for another it became this increasingly irritating and obnoxious process where it felt like I was spending more time seeing the care and feeding of the instrumentation then I was the application itself. That's the gold—that's I guess the ideal light at the end of the tunnel for me in what OpenTelemetry is promising. Instrument once, and then you're just adjusting configuration as far as where to send it.Ian: That's correct. The organization's, and you know, I keep in touch with a lot of companies that I've worked with, companies that have in the last two years really invested heavily in OpenTelemetry, they're definitely getting to the point now where they're generating the data once, they're using, say, pieces of the OpenTelemetry pipeline, they're extending it themselves, and then they're able to shove that data in a bunch of different places. Maybe they're putting in a data lake for, as I said, business analysis purposes or forecasting. They may be putting the data into two different systems, even for incident and analysis purposes, but you're not having that duplication effort. Also, potentially that performance impact, right, of having two different instrumentation packages lined up with each other.Corey: There is a recurring theme that I've noticed in the observability space that annoys me to no end. And that is—I don't know if it's coming from investor pressure, from folks never being satisfied with what they have, or what it is, but there are so many startups that I have seen and worked with in varying aspects of the observability space that I think, “This is awesome. I love the thing that they do.” And invariably, every time they start getting more and more features bolted onto them, where, hey, you love this whole thing that winds up just basically doing a tail-F on a log file, so it just streams your logs in the application and you can look for certain patterns. I love this thing. It's great.Oh, what's this? Now, it's trying to also be the thing that alerts me and wakes me up in the middle of the night. No. That's what PagerDuty does. I want PagerDuty to do that thing, and I want other things—I want you just to be the log analysis thing and the way that I contextualize logs. And it feels like they keep bolting things on and bolting things on, where everything is more or less trying to evolve into becoming its own version of Datadog. What's up with that?Ian: Yeah, the sort of, dreaded platform play. I—[laugh] I was at New Relic when there were essentially two products that they sold. And then by the time I left, I think there was seven different products that were being sold, which is kind of a crazy, crazy thing when you think about it. And I think Datadog has definitely exceeded that now. And I definitely see many, many vendors in the market—and even open-source solutions—sort of presenting themselves as, like, this integrated experience.But to your point, even before about your experience of these banks it oftentimes become sort of a tick-a-box feature approach of, “Hey, I can do this thing, so buy more. And here's a shared navigation panel.” But are they really integrated? Like, are you getting real value out of it? One of the things that I do in my role is I get to work with our internal product teams very closely, particularly around new initiatives like tracing functionality, and the constant sort of conversation is like, “What is the outcome? What is the value?”It's not about the feature; it's not about having a list of 19 different features. It's like, “What is the user able to do with this?” And so, for example, there are lots of platforms that have metrics, logs, and tracing. The new one-upmanship is saying, “Well, we have events as well. And we have incident response. And we have security. And all these things sort of tie together, so it's one invoice.”And constantly I talk to customers, and I ask them, like, “Hey, what are the outcomes that you're getting when you've invested so heavily in one vendor?” And oftentimes, the response is, “Well, I only need to deal with one vendor.” Okay, but that's not an outcome. [laugh]. And it's like the business having a single invoice.Corey: Yeah, that is something that's already attainable today. If you want to just have one vendor with a whole bunch of crappy offerings, that's what AWS is for. They have AmazonBasics versions of everything you might want to use in production. Oh, you want to go ahead and use MongoDB? Well, use AmazonBasics MongoDB, but they call it DocumentDB because of course they do. And so, on and so forth.There are a bunch of examples of this, but those companies are still in business and doing very well because people often want the genuine article. If everyone was trying to do just everything to check a box for procurement, great. AWS has already beaten you at that game, it seems.Ian: I do think that, you know, people are hoping for that greater value and those greater outcomes, so being able to actually provide differentiation in that market I don't think is terribly difficult, right? There are still huge gaps in let's say, root cause analysis during an investigation time. There are huge issues with vendors who don't think beyond sort of just the one individual who's looking at a particular dashboard or looking at whatever analysis tool there is. So, getting those things actually tied together, it's not just, “Oh, we have metrics, and logs, and traces together,” but even if you say we have metrics and tracing, how do you move between metrics and tracing? One of the goals in the way that we're developing product at Chronosphere is that if you are alerted to an incident—you as an engineer; doesn't matter whether you are massively sophisticated, you're a lead architect who has been with the company forever and you know everything or you're someone who's just come out of onboarding and is your first time on call—you should not have to think, “Is this a tracing problem, or a metrics problem, or a logging problem?”And this is one of those things that I mentioned before of requiring that really heavy level of knowledge and understanding about the observability space and your data and your architecture to be effective. And so, with the, you know, particularly observability teams and all of the engineers that I speak with on a regular basis, you get this sort of circumstance where well, I guess, let's talk about a real outcome and a real pain point because people are like, okay, yeah, this is all fine; it's all coming from a vendor who has a particular agenda, but the thing that constantly resonates is for large organizations that are moving fast, you know, big startups, unicorns, or even more traditional enterprises that are trying to undergo, like, a rapid transformation and go really cloud-native and make sure their engineers are moving quickly, a common question I will talk about with them is, who are the three people in your organization who always get escalated to? And it's usually, you know, between two and five people—Corey: And you can almost pick those perso—you say that and you can—at least anyone who's worked in environments or through incidents like this more than a few times, already have thought of specific people in specific companies. And they almost always fall into some very predictable archetypes. But please, continue.Ian: Yeah. And people think about these people, they always jump to mind. And one of the things I asked about is, “Okay, so when you did your last innovation around observably”—it's not necessarily buying a new thing, but it maybe it was like introducing a new data type or it was you're doing some big investment in improving instrumentation—“What changed about their experience?” And oftentimes, the most that can come out is, “Oh, they have access to more data.” Okay, that's not great.It's like, “What changed about their experience? Are they still getting woken up at 3 am? Are they constantly getting pinged all the time?” One of the vendors that I worked at, when they would go down, there were three engineers in the company who were capable of generating list of customers who are actually impacted by damage. And so, every single incident, one of those three engineers got paged into the incident.And it became borderline intolerable for them because nothing changed. And it got worse, you know? The platform got bigger and more complicated, and so there were more incidents and they were the ones having to generate that. But from a business level, from an observability outcomes perspective, if you zoom all the way up, it's like, “Oh, were we able to generate the list of customers?” “Yes.”And this is where I think the observability industry has sort of gotten stuck—you know, at least one of the ways—is that, “Oh, can you do it?” “Yes.” “But is it effective?” “No.” And by effective, I mean those three engineers become the focal point for an organization.And when I say three—you know, two to five—it doesn't matter whether you're talking about a team of a hundred or you're talking about a team of a thousand. It's always the same number of people. And as you get bigger and bigger, it becomes more and more of a problem. So, does the tooling actually make a difference to them? And you might ask, “Well, what do you expect from the tooling? What do you expect to do for them?” Is it you give them deeper analysis tools? Is it, you know, you do AI Ops? No.The answer is, how do you take the capabilities that those people have and how do you spread it across a larger population of engineers? And that, I think, is one of those key outcomes of observability that no one, whether it be in open-source or the vendor side is really paying a lot of attention to. It's always about, like, “Oh, we can just shove more data in. By the way, we've got petabyte scale and we can deal with, you know, 2 billion active time series, and all these other sorts of vanity measures.” But we've gotten really far away from the outcomes. It's like, “Am I getting return on investment of my observability tooling?”And I think tracing is this—as you've said, it can be difficult to reason about right? And people are not sure. They're feeling, “Well, I'm in a microservices environment; I'm in cloud-native; I need tracing because my older APM tools appear to be failing me. I'm just going to go and wriggle my way through implementing OpenTelemetry.” Which has significant engineering costs. I'm not saying it's not worth it, but there is a significant engineering cost—and then I don't know what to expect, so I'm going to go on through my data somewhere and see whether we can achieve those outcomes.And I do a pilot and my most sophisticated engineers are in the pilot. And they're able to solve the problems. Okay, I'm going to go buy that thing. But I've just transferred my problems. My engineers have gone from solving problems in maybe logs and grepping through petabytes worth of logs to using some sort of complex proprietary query language to go through your tens of petabytes of trace data but actually haven't solved any problem. I've just moved it around and probably just cost myself a lot, both in terms of engineering time and real dollars spent as well.Corey: One of the challenges that I'm seeing across the board is that observability, for certain use cases, once you start to see what it is and its potential for certain applications—certainly not all; I want to hedge that a little bit—but it's clear that there is definite and distinct value versus other ways of doing things. The problem is, is that value often becomes apparent only after you've already done it and can see what that other side looks like. But let's be honest here. Instrumenting an application is going to take some significant level of investment, in many cases. How do you wind up viewing any return on investment that it takes for the very real cost, if only in people's time, to go ahead instrumenting for observability in complex environments?Ian: So, I think that you have to look at the fundamentals, right? You have to look at—pretend we knew nothing about tracing. Pretend that we had just invented logging, and you needed to start small. It's like, I'm not going to go and log everything about every application that I've had forever. What I need to do is I need to find the points where that logging is going to be the most useful, most impactful, across the broadest audience possible.And one of the useful things about tracing is because it's built in distributed environments, primarily for distributed environments, you can look at, for example, the biggest intersection of requests. A lot of people have things like API Gateways, or they have parts of a monolith which is still handling a lot of requests routing; those tend to be areas to start digging into. And I would say that, just like for anyone who's used Prometheus or decided to move away from Prometheus, no one's ever gone and evaluated Prometheus solution without having some sort of Prometheus data, right? You don't go, “Hey, I'm going to evaluate a replacement for Prometheus or my StatsD without having any data, and I'm simultaneously going to generate my data and evaluate the solution at the same time.” It doesn't make any sense.With tracing, you have decent open-source projects out there that allow you to visualize individual traces and understand sort of the basic value you should be getting out of this data. So, it's a good starting point to go, “Okay, can I reason about a single request? Can I go and look at my request end-to-end, even in a relatively small slice of my environment, and can I see the potential for this? And can I think about the things that I need to be able to solve with many traces?” Once you start developing these ideas, then you can have a better idea of, “Well, where do I go and invest more in instrumentation? Look, databases never appear to be a problem, so I'm not going to focus on database instrumentation. What's the real problem is my external dependencies. Facebook API is the one that everyone loves to use. I need to go instrument that.”And then you start to get more clarity. Tracing has this interesting network effect. You can basically just follow the breadcrumbs. Where is my biggest problem here? Where are my errors coming from? Is there anything else further down the call chain? And you can sort of take that exploratory approach rather than doing everything up front.But it is important to do something before you start trying to evaluate what is my end state. End state obviously being sort of nebulous term in today's world, but where do I want to be in two years' time? I would like to have a solution. Maybe it's open-source solution, maybe it's a vendor solution, maybe it's one of those platform solutions we talked about, but how do I get there? It's really going to be I need to take an iterative approach and I need to be very clear about the value and outcomes.There's no point in doing a whole bunch of instrumentation effort in things that are just working fine, right? You want to go and focus your time and attention on that. And also you don't want to go and burn just singular engineers. The observability team's purpose in life is probably not to just write instrumentation or just deploy OpenTelemetry. Because then we get back into the land where engineers themselves know nothing about the monitoring or observability they're doing and it just becomes a checkbox of, “I dropped in an agent. Oh, when it comes time for me to actually deal with an incident, I don't know anything about the data and the data is insufficient.”So, a level of ownership supported by the observability team is really important. On that return on investment, sort of, though it's not just the instrumentation effort. There's product training and there are some very hard costs. People think oftentimes, “Well, I have the ability to pay a vendor; that's really the only cost that I have.” There's things like egress costs, particularly volumes of data. There's the infrastructure costs. A lot of the times there will be elements you need to run in your own environment; those can be very costly as well, and ultimately, they're sort of icebergs in this overall ROI conversation.The other side of it—you know, return and investment—return, there's a lot of difficulty in reasoning about, as you said, what is the value of this going to be if I go through all this effort? Everyone knows a sort of, you know, meme or archetype of, “Hey, here are three options; pick two because there's always going to be a trade off.” Particularly for observability, it's become an element of, I need to pick between performance, data fidelity, or cost. Pick two. And when data fidelity—particularly in tracing—I'm talking about the ability to not sample, right?If you have edge cases, if you have narrow use cases and ways you need to look at your data, if you heavily sample, you lose data fidelity. But oftentimes, cost is a reason why you do that. And then obviously, performance as you start to get bigger and bigger datasets. So, there's a lot of different things you need to balance on that return. As you said, oftentimes you don't get to understand the magnitude of those until you've got the full data set in and you're trying to do this, sort of, for real. But being prepared and iterative as you go through this effort and not saying, “Okay, well, I'm just going to buy everything from one vendor because I'm going to assume that's going to solve my problem,” is probably that undercurrent there.Corey: As I take a look across the entire ecosystem, I can't shake the feeling—and my apologies in advance if this is an observation, I guess, that winds up throwing a stone directly at you folks—Ian: Oh, please.Corey: But I see that there's a strong observability community out there that is absolutely aligned with the things I care about and things I want to do, and then there's a bunch of SaaS vendors, where it seems that they are, in many cases, yes, advancing the state of the art, I am not suggesting for a second that money is making observability worse. But I do think that when the tool you sell is a hammer, then every problem starts to look like a nail—or in my case, like my thumb. Do you think that there's a chance that SaaS vendors are in some ways making this entire space worse?Ian: As we've sort of gone into more cloud-native scenarios and people are building things specifically to take advantage of cloud from a complexity standpoint, from a scaling standpoint, you start to get, like, vertical issues happening. So, you have things like we're going to charge on a per-container basis; we're going to charge on a per-host basis; we're going to charge based off the amount of gigabytes that you send us. These are sort of like more horizontal pricing models, and the way the SaaS vendors have delivered this is they've made it pretty opaque, right? Everyone has experiences, or has jerks about overages from observability vendors' massive spikes. I've worked with customers who have used—accidentally used some features and they've been billed a quarter million dollars on a monthly basis for accidental overages from a SaaS vendor.And these are all terrible things. Like, but we've gotten used to this. Like, we've just accepted it, right, because everyone is operating this way. And I really do believe that the move to SaaS was one of those things. Like, “Oh, well, you're throwing us more data, and we're charging you more for it.” As a vendor—Corey: Which sort of erodes your own value proposition that you're bringing to the table. I mean, I don't mean to be sitting over here shaking my fist yelling, “Oh, I could build a better version in a weekend,” except that I absolutely know how to build a highly available Rsyslog cluster. I've done it a handful of times already and the technology is still there. Compare and contrast that with, at scale, the fact that I'm paying 50 cents per gigabyte ingested to CloudWatch logs, or a multiple of that for a lot of other vendors, it's not that much harder for me to scale that fleet out and pay a much smaller marginal cost.Ian: And so, I think the reaction that we're seeing in the market and we're starting to see—we're starting to see the rise of, sort of, a secondary class of vendor. And by secondary, I don't mean that they're lesser; I mean that they're, sort of like, specifically trying to address problems of the primary vendors, right? Everyone's aware of vendors who are attempting to reduce—well, let's take the example you gave on logs, right? There are vendors out there whose express purpose is to reduce the cost of your logging observability. They just sit in the middle; they are a middleman, right?Essentially, hey, use our tool and even though you're going to pay us a whole bunch of money, it's going to generate an overall return that is greater than if you had just continued pumping all of your logs over to your existing vendor. So, that's great. What we think really needs to happen, and one of the things we're doing at Chronosphere—unfortunate plug—is we're actually building those capabilities into the solution so it's actually end-to-end. And by end-to-end, I mean, a solution where I can ingest my data, I can preprocess my data, I can store it, query it, visualize it, all those things, aligned with open-source standards, but I have control over that data, and I understand what's going on with particularly my cost and my usage. I don't just get a bill at the end of the month going, “Hey, guess what? You've spent an additional $200,000.”Instead, I can know in real time, well, what is happening with my usage. And I can attribute it. It's this team over here. And it's because they added this particular label. And here's a way for you, right now, to address that and cap it so it doesn't cost you anything and it doesn't have a blast radius of, you know, maybe degraded performance or degraded fidelity of the data.That though is diametrically opposed to the way that most vendors are set up. And unfortunately, the open-source projects tend to take a lot of their cues, at least recently, from what's happening in the vendor space. One of the ways that you can think about it is a sort of like a speed of light problem. Everyone knows that, you know, there's basic fundamental latency; everyone knows how fast disk is; everyone knows the, sort of like, you can't just make your computations happen magically, there's a cost of running things horizontally. But a lot of the way that the vendors have presented efficiency to the market is, “Oh, we're just going to incrementally get faster as AWS gets faster. We're going to incrementally get better as compression gets better.”And of course, you can't go and fit a petabyte worth of data into a kilobyte, unless you're really just doing some sort of weird dictionary stuff, so you feel—you're dealing with some fundamental constraints. And the vendors just go, “I'm sorry, you know, we can't violate the speed of light.” But what you can do is you can start taking a look at, well, how is the data valuable, and start giving the people controls on how to make it more valuable. So, one of the things that we do with Chronosphere is we allow you to reshape Prometheus metrics, right? You go and express Prometheus metrics—let's say it's a business metric about how many transactions you're doing as a business—you don't need that on a per-container basis, particularly if you're running 100,000 containers globally.When you go and take a look at that number on a dashboard, or you alert on it, what is it? It's one number, one time series. Maybe you break it out per region. You have five regions, you don't need 100,000 data points every minute behind that. It's very expensive, it's not very performant, and as we talked about earlier, it's very hard to reason about as a human being.So, giving the tools to be able to go and condense that data down and make it more actionable and more valuable, you get performance, you get cost reduction, and you get the value that you ultimately need out of the data. And it's one of the reasons why, I guess, I work at Chronosphere. Which I'm hoping is the last observability [laugh] venture I ever work for.Corey: Yeah, for me a lot of the data that I see in my logs, which is where a lot of this stuff starts and how I still contextualize these things, is nonsense that I don't care about and will never care about. I don't care about load balance or health checks. I don't particularly care about 200 results for the favicon when people visit the site. I care about other things, but just weed out the crap, especially when I'm paying by the pound—or at least by the gigabyte—in order to get that data into something. Yeah. It becomes obnoxious and difficult to filter out.Ian: Yeah. And the vendors just haven't done any of that because why would they, right? If you went and reduced the amount of log—Corey: Put engineering effort into something that reduces how much I can charge you? That sounds like lunacy. Yeah.Ian: Exactly. They're business models entirely based off it. So, if you went and reduced every one's logging bill by 30%, or everyone's logging volume by 30% and reduced the bills by 30%, it's not going to be a great time if you're a publicly traded company who has built your entire business model on essentially a very SaaS volume-driven—and in my eyes—relatively exploitative pricing and billing model.Corey: Ian, I want to thank you for taking so much time out of your day to talk to me about this. If people want to learn more, where can they find you? I mean, you are a Field CTO, so clearly you're outstanding in your field. But if, assuming that people don't want to go to farm country, where's the best place to find you?Ian: Yeah. Well, it'll be a bunch of different conferences. I'll be at KubeCon this year. But chronosphere.io is the company website. I've had the opportunity to talk to a lot of different customers, not from a hard sell perspective, but you know, conversations like this about what are the real problems you're having and what are the things that you sort of wish that you could do?One of the favorite things that I get to ask people is, “If you could wave a magic wand, what would you love to be able to do with your observability solution?” That's, A, a really great part, but oftentimes be being able to say, “Well, actually, that thing you want to do, I think I have a way to accomplish that,” is a really rewarding part of this particular role.Corey: And we will, of course, put links to that in the show notes. Thank you so much for being so generous with your time. I appreciate it.Ian: Thanks, Corey. It's great to be here.Corey: Ian Smith, Field CTO at Chronosphere on this promoted guest episode. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, which going to be super easy in your case, because it's just one of the things that the omnibus observability platform that your company sells offers as part of its full suite of things you've never used.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode we are going to look at Syslog.We will be discussing Introduction to Syslog, Syslog Operation, Syslog Message Format, Syslog Facilities, and finally Configure Syslog Timestamp.Thank you so much for listening to this episode of my series on Enterprise Networking, Security, and Automation for the Cisco Certified Network Associate (CCNA).Once again, I'm Kevin and this is KevTechify. Let's get this adventure started.All my details and contact information can be found on my website, https://KevTechify.com-------------------------------------------------------Cisco Certified Network Associate (CCNA)Enterprise Networking, Security, and Automation v3 (ENSA)Episode 10 - Network ManagementPart E - SyslogPodcast Number: 52-------------------------------------------------------Equipment I like.Home Lab ►► https://kit.co/KevTechify/home-labNetworking Tools ►► https://kit.co/KevTechify/networking-toolsStudio Equipment ►► https://kit.co/KevTechify/studio-equipment 

About ClintClint is the CEO and a co-founder at Cribl, a company focused on making observability viable for any organization, giving customers visibility and control over their data while maximizing value from existing tools.Prior to co-founding Cribl, Clint spent two decades leading product management and IT operations at technology and software companies, including Splunk and Cricket Communications. As a former practitioner, he has deep expertise in network issues, database administration, and security operations.Links: Cribl: https://cribl.io/ Cribl.io: https://cribl.io Docs.cribl.io: https://docs.cribl.io Sandbox.cribl.io: https://sandbox.cribl.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I have a repeat guest joining me on this promoted episode. Clint Sharp is the CEO and co-founder of Cribl. Clint, thanks for joining me.Clint: Hey, Corey, nice to be back.Corey: I was super excited when you gave me the premise for this recording because you said you had some news to talk about, and I was really excited that oh, great, they're finally going to buy a vowel so that people look at their name and understand how to pronounce it. And no, that's nowhere near forward-looking enough. It's instead it's some, I guess, I don't know, some product announcement or something. But you know, hope springs eternal. What have you got for us today?Clint: Well, one of the reasons I love talking to your audiences because product announcements actually matter to this audience. It's super interesting, as you get into starting a company, you're such, like, a product person, you're like, “Oh, I have this new set of things that's really going to make your life better.” And then you go out to, like, the general media, and you're like, “Hey, I have this product.” And they're like, “I don't care. What product? Do you have a funding announcement? Do you have something big in the market that—you know, do you have a new executive? Do you”—it's like, “No, but, like, these features, like these things, that we—the way we make our lives better for our customers. Isn't that interesting?” “No.”Corey: Real depressing once you—“Do you have a security breach to announce?” It's, “No. God no. Why would I wind up being that excited about it?” “Well, I don't know. I'd be that excited about it.” And yeah, the stuff that mainstream media wants to write about in the context of tech companies is exactly the sort of thing that tech companies absolutely do not want to be written about for. But fortunately, that is neither here nor there.Clint: Yeah, they want the thing that gets the clicks.Corey: Exactly. You built a product that absolutely resonates in its target market and outside of that market. It's one of those, what is that thing, again? If you could give us a light refresher on what Cribl is and does, you'll probably do a better job of it than I will. We hope.Clint: We'd love to. Yeah, so we are an observability company, fundamentally. I think one of the interesting things to talk about when it comes to observability is that observability and security are merging. And so I like to say observability and include security people. If you're a security person, and you don't feel included by the word observability, sorry.We also include you; you're under our tent here. So, we sell to technology professionals, we help make their lives better. And we do that today through a flagship product called LogStream—which is part of this announcement, we're actually renaming to Stream. In some ways, we're dropping logs—and we are a pipeline company. So, we help you take all of your existing agents, all of your existing data that's moving, and we help you process that data in the stream to control costs and to send it multiple places.And it sounds kind of silly, but one of the biggest problems that we end up solving for a lot of our enterprises is, “Hey, I've got, like, this old Syslog feed coming off of my firewalls”—like, you remember those things, right? Palo Alto firewalls, ASA firewalls—“I actually get that thing to multiple places because, hey, I want to get that data into another security solution. I want to get that data into a data lake. How do I do that?” Well, in today's world, that actually turns out is sort of a neglected set of features, like, the vendors who provide you logging solutions, being able to reshape that data, filter that data, control costs, wasn't necessarily at the top of their priority list.It wasn't nefarious. It wasn't like people are like, “Oh, I'm going to make sure that they can't process this data before it comes into my solution.” It's more just, like, “I'll get around to it eventually.” And the eventually never actually comes. And so our streaming product helps people do that today.And the big announcement that we're making this week is that we're extending that same processing technology down to the endpoint with a new product we're calling Cribl Edge. And so we're taking our existing best-in-class management technology, and we're turning it into an agent. And that seems kind of interesting because… I think everybody sort of assumed that the agent is dead. Okay, well, we've been building agents for a decade or two decades. Isn't everything exactly the same as it was before?But we really saw kind of a dearth of innovation in that area in terms of being able to manage your agents, being able to understand what data is available to be collected, being able to auto-discover the data that needs to be able to be collected, turning those agents into interactive troubleshooting experiences so that we can, kind of, replicate the ability to zoom into a remote endpoint and replicate that Linux command line experience that we're not supposed to be getting anymore because we're not supposed to SSH into boxes anymore. Well, how do I replicate that? How do I see how much disk is on this given endpoint if I can't SSH into that box? And so Cribl Edge is a rethink about making this rich, interactive experience on top of all of these agents that become this really massive distributed system that we can process data all the way out at where the data is being emitted.And so that means that now we don't nec—if you want to process that data in the stream, okay, great, but if you want to process that data at its origination point, we can actually provide you cheaper cost because now you're using a lot of that capacity that's sitting out there on your endpoints that isn't really being used today anyway—the average utilization of a Kubernetes cluster is like 30%—Corey: It's that high. I'm sort of surprised.Clint: Right? I know. So, Datadog puts out the survey every year, which I think is really interesting, and that's a number that always surprised me is just that people are already paying for this capacity, right? It's sitting there, it's on their AWS bill already, and with that average utilization, a lot of the stuff that we're doing in other clusters, or while we're moving that data can actually just be done right there where the data is being emitted. And also, if we're doing things like filtering, we can lower egress charges, there's lots of really, really good goodness that we can do by pushing that processing further closer to its origination point.Corey: You know, the timing of this episode is somewhat apt because as of the time that we're recording this, I spent most of yesterday troubleshooting and fixing my home wireless network, which is a whole Ubiquity-managed thing. And the controller was one of their all-in-one box things that kept more or less power cycling for no apparent reason. How do I figure out why it's doing that? Well, I'm used to, these days, doing everything in a cloud environment where you can instrument things pretty easily, where things start and where things stop is well understood. Finally, I just gave up and used a controller that's sitting on an EC2 instance somewhere, and now great, now I can get useful telemetry out of it because now it's stuff I know how to deal with.It also, turns out that surprise, my EC2 instance is not magically restarting itself due to heat issues. What a concept. So, I have a newfound appreciation for the fact that oh, yeah, not everything lives in a cloud provider's regions. Who knew? This is a revelation that I think is going to be somewhat surprising for folks who've been building startups and believe that anything that's older than 18 months doesn't exist.But there's a lot of data centers out there, there are a lot of agents living all kinds of different places. And workloads continue to surprise me even now, just looking at my own client base. It's a very diverse world when we're talking about whether things are on-prem or whether they're in cloud environments.Clint: Well, also, there's a lot of agents on every endpoint period, just due to the fact that security guys want an agent, the observability guys want an agent, the logging people want an agent. And then suddenly, I'm, you know, I'm looking at every endpoint—cloud, on-prem, whatever—and there's 8, 10 agents sitting there. And so I think a lot of the opportunity that we saw was, we can unify the data collection for metric type of data. So, we have some really cool defaults. [unintelligible 00:07:30] this is one of the things where I think people don't focus much on, kind of, the end-user experience. Like, let's have reasonable defaults.Let's have the thing turn on, and actually, most people's needs are set without tweaking any knobs or buttons, and no diving into YAML files and looking at documentation and trying to figure out exactly the way I need to configure this thing. Let's collect metric data, let's collect log data, let's do it all from one central place with one agent that can send that data to multiple places. And I can send it to Grafana Cloud, if I want to; I can send it to Logz.io, I can send it to Splunk, I can send it to Elasticsearch, I can send it to AWS's new Elasticsearch-y the thing that we don't know what they're going to call it yet after the lawsuit. Any of those can be done right from the endpoint from, like, a rich graphical experience where I think that there's a really a desire now for people to kind of jump into these configuration files where really a lot of these users, this is a part-time job, and so hey, if I need to go set up data collection, do I want to learn about this detailed YAML file configuration that I'm only going to do once or twice, or should I be able to do it in an easy, intuitive way, where I can just sit down in front of the product, get my job done and move on without having to go learn some sort of new configuration language?Corey: Once upon a time, I saw an early circa 2012, 2013 talk from Jordan Sissel, who is the creator of Logstash, and he talked a lot about how challenging it was to wind up parsing all of the variety of log files out there. Even something is relatively straightforward—wink, wink, nudge, nudge—as timestamps was an absolute monstrosity. And a lot of people have been talking in recent years about OpenTelemetry being the lingua franca that everything speaks so that is the wave of the future, but I've got a level with you, looking around, it feels like these people are living in a very different reality than the one that I appear to have stumbled into because the conversations people are having about how great it is sound amazing, but nothing that I'm looking at—granted from a very particular point of view—seems to be embracing it or supporting it. Is that just because I'm hanging out in the wrong places, or is it still a great idea whose time has yet to come, or something else?Clint: So, I think a couple things. One is every conversation I have about OpenTelemetry is always, “Will be.” It's always in the future. And there's certainly a lot of interest. We see this from customer after customer, they're very interested in OpenTelemetry and what the OpenTelemetry strategy is, but as an example OpenTelemetry logging is not yet finalized specification; they believe that they're still six months to a year out. It seems to be perpetually six months to a year out there.They are finalized for metrics and they are finalized for tracing. Where we see OpenTelemetry tends to be with companies like Honeycomb, companies like Datadog with their tracing product, or Lightstep. So, for tracing, we see OpenTelemetry adoption. But tracing adoption is also not that high either, relative to just general metrics of logs.Corey: Yeah, the tracing implementations that I've seen, for example, Epsagon did this super well, where it would take a look at your Lambdas Function built into an application, and ah, we're going to go ahead and instrument this automatically using layers or extensions for you. And life was good because suddenly you got very detailed breakdowns of exactly how data was flowing in the course of a transaction through 15 Lambdas Function. Great. With everything else I've seen, it's, “Oh, you have to instrument all these things by hand.” Let me shortcut that for you: That means no one's going to do it. They never are.It's anytime you have to do that undifferentiated heavy lifting of making sure that you put the finicky code just so into your application's logic, it's a shorthand for it's only going to happen when you have no other choice. And I think that trying to surface that burden to the developer, instead of building it into the platform so they don't have to think about it is inherently the wrong move.Clint: I think there's a strong belief in Silicon Valley that—similar to, like, Hollywood—that the biggest export Silicon Valley is going to have is culture. And so that's going to be this culture of, like, developer supporting their stuff in production. I'm telling you, I sell to banks and governments and telcos and I don't see that culture prevailing. I see a application developed by Accenture that's operated by Tata. That's a lot of inertia to overcome and a lot of regulation to overcome as well, and so, like, we can say that, hey, separation of duties isn't really a thing and developers should be able to support all their own stuff in production.I don't see that happening. It may happen. It'll certainly happen more than zero. And tracing is predicated on the whole idea that the developer is scratching their own itch. Like that I am in production and troubleshooting this and so I need this high-fidelity trace-level information to understand what's going on with this one user's experience, but that doesn't tend to be in the enterprise, how things are actually troubleshot.And so I think that more than anything is the headwind that slowing down distributed tracing adoption. It's because you're putting the onus on solving the problem on a developer who never ends up using the distributed tracing solution to begin with because there's another operations department over there that's actually operating the thing on a day-to-day basis.Corey: Having come from one of those operations departments myself, the way that I would always fix things was—you know, in the era that I was operating it made sense—you'd SSH into a box and kick the tires, poke around, see what's going on, look at the logs locally, look at the behaviors, the way you'd expect it to these days, that is considered a screamingly bad anti-pattern and it's something that companies try their damnedest to avoid doing at all. When did that change? And what is the replacement for that? Because every time I asked people for the sorts of data that I would get from that sort of exploration when they're trying to track something down, I'm more or less met with blank stares.Clint: Yeah. Well, I think that's a huge hole and one of the things that we're actually trying to do with our new product. And I think the… how do I replicate that Linux command line experience? So, for example, something as simple, like, we'd like to think that these nodes are all ephemeral, but there's still a disk, whether it's virtual or not; that thing sometimes fills up, so how do I even do the simple thing like df -kh and see how much disk is there if I don't already have all the metrics collected that I needed, or I need to go dive deep into an application and understand what that application is doing or seeing, what files it's opening, or what log files it's writing even?Let's give some good examples. Like, how do I even know what files an application is running? Actually, all that information is all there; we can go discover that. And so some of the things that we're doing with Edge is trying to make this rich, interactive experience where you can actually teleport into the end node and see all the processes that are running and get a view that looks like top and be able to see how much disk is there and how much disk is being consumed. And really kind of replicating that whole troubleshooting experience that we used to get from the Linux command line, but now instead, it's a tightly controlled experience where you're not actually getting an arbitrary shell, where I could do anything that could give me root level access, or exploit holes in various pieces of software, but really trying to replicate getting you that high fidelity information because you don't need any of that information until you need it.And I think that's part of the problem that's hard with shipping all this data to some centralized platform and getting every metric and every log and moving all that data is the data is worthless until it isn't worthless anymore. And so why do we even move it? Why don't we provide a better experience for getting at the data at the time that we need to be able to get at the data. Or the other thing that we get to change fundamentally is if we have the edge available to us, we have way more capacity. I can store a lot of information in a few kilobytes of RAM on every node, but if I bring thousands of nodes into one central place, now I need a massive amount of RAM and a massive amount of cardinality when really what I need is the ability to actually go interrogate what's running out there.Corey: The thing that frustrates me the most is the way that I go back and find my old debug statements, which is, you know, I print out whatever it is that the current status is and so I can figure out where something's breaking.Clint: [Got here 00:15:08].Corey: Yeah. I do it within AWS Lambda functions, and that's great. And I go back and I remove them later when I notice how expensive CloudWatch logs are getting because at 50 cents per gigabyte of ingest on those things, and you have that Lambda function firing off a fair bit, that starts to add up when you've been excessively wordy with your print statements. It sounds ridiculous, but okay, then you're storing it somewhere. If I want to take that log data and have something else consume it, that's nine cents a gigabyte to get it out of AWS and then you're going to want to move it again from wherever it is over there—potentially to a third system, because why not?—and it seems like the entire purpose of this log data is to sit there and be moved around because every time it gets moved, it winds up somehow costing me yet more money. Why do we do this?Clint: I mean, it's a great question because one of the things that I think we decided 15 years ago was that the reason to move this data was because that data may go poof. So, it was on a, you know, back in my day, it was an HP DL360 1U rackmount server that I threw in there, and it had raid zero discs and so if that thing went dead, well, we didn't care, we'd replace it with another one. But if we wanted to find out why it went dead, we wanted to make sure that the data had moved before the thing went dead. But now that DL360 is a VM.Corey: Yeah, or a container that is going to be gone in 20 minutes. So yeah, you don't want to store it locally on that container. But discs are also a fair bit more durable than they once were, as well. And S3 talks about its 11 nines of durability. That's great and all but most of my application logs don't need that. So, I'm still trying to figure out where we went wrong.Clint: Well, I think it was right for the time. And I think now that we have durable storage at the edge where that blob storage has already replicated three times and we can reattach—if that box crashes, we can reattach new compute to that same block storage. Actually, AWS has some cool features now, you can actually attach multiple VMs to the same block store. So, we could actually even have logs being written by one VM, but processed by another VM. And so there are new primitives available to us in the cloud, which we should be going back and re-questioning all of the things that we did ten to 15 years ago and all the practices that we had because they may not be relevant anymore, but we just never stopped to ask why.Corey: Yeah, multi-attach was rolled out with their IO2 volumes, which are spendy but great. And they do warn you that you need a file system that actively supports that and applications that are aware of it. But cool, they have specific use cases that they're clearly imagining this for. But ten years ago, we were building things out, and, “Ooh, EBS, how do I wind up attaching that from multiple instances?” The answer was, “Ohh, don't do that.”And that shaped all of our perspectives on these things. Now suddenly, you can. Is that, “Ohh don't do that,” gut visceral reaction still valid? People don't tend to go back and re-examine the why behind certain best practices until long after those best practices are now actively harmful.Clint: And that's really what we're trying to do is to say, hey, should we move log data anymore if it's at a durable place at the edge? Should we move metric data at all? Like, hey, we have these big TSDBs that have huge cardinality challenges, but if I just had all that information sitting in RAM at the original endpoint, I can store a lot of information and barely even touch the free RAM that's already sitting out there at that endpoint. So, how to get out that data? Like, how to make that a rich user experience so that we can query it?We have to build some software to do this, but we can start to question from first principles, hey, things are different now. Maybe we can actually revisit a lot of these architectural assumptions, drive cost down, give more capability than we actually had before for fundamentally cheaper. And that's kind of what Cribl does is we're looking at software is to say, “Man, like, let's question everything and let's go back to first principles.” “Why do we want this information?” “Well, I need to troubleshoot stuff.” “Okay, well, if I need to troubleshoot stuff, well, how do I do that?” “Well, today we move it, but do we have to? Do we have to move that data?” “No, we could probably give you an experience where you can dive right into that endpoint and get really, really high fidelity data without having to pay to move that and store it forever.” Because also, like, telemetry information, it's basically worthless after 24 hours, like, if I'm moving that and paying to store it, then now I'm paying for something I'm never going to read back.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats V-U-L-T-R.com slash screaming.Corey: And worse, you wind up figuring out, okay, I'm going to store all that data going back to 2012, and it's petabytes upon petabytes. And great, how do I actually search for a thing? Well, I have to use some other expensive thing of compute that's going to start diving through all of that because the way I set up my partitioning, it isn't aligned with anything looking at, like, recency or based upon time period, so right every time I want to look at what happened 20 minutes ago, I'm looking at what happened 20 years ago. And that just gets incredibly expensive, not just to maintain but to query and the rest. Now, to be clear, yes, this is an anti-pattern. It isn't how things should be set up. But how should they be set up? And it is the collective the answer to that right now actually what's best, or is it still harkening back to old patterns that no longer apply?Clint: Well, the future is here, it's just unevenly distributed. So there's, you know, I think an important point about us or how we think about building software is with this customer is first attitude and fundamentally bringing them choice. Because the reality is that doing things the old way may be the right decision for you. You may have compliance requirements to say—there's a lot of financial services institutions, for example, like, they have to keep every byte of data written on any endpoint for seven years. And so we have to accommodate their requirements.Like, is that the right requirement? Well, I don't know. The regulator wrote it that way, so therefore, I have to do it. Whether it's the right thing or the wrong thing for the business, I have no choice. And their decisions are just as right as the person who says this data is worthless and should all just be thrown away.We really want to be able to go and say, like, hey, what decision is right? We're going to give you the option to do it this way, we're going to give you the option to do it this way. Now, the hard part—and that when it comes down to, like, marketing, it's like you want to have this really simple message, like, “This is the one true path.” And a lot of vendors are this way, “There's this new wonderful, right, true path that we are going to take you on, and follow along behind me.” But the reality is, enterprise worlds are gritty and ugly, and they're full of old technology and new technology.And they need to be able to support getting data off the mainframe the same way as they're doing a brand new containerized microservices application. In fact, that brand new containerized microservices application is probably talking to the mainframe through some API. And so all of that has to work at once.Corey: Oh, yeah. And it's all of our payment data is in our PCI environment that PCI needs to have every byte logged. Great. Why is three-quarters of your infrastructure considered the PCI environment? Maybe you can constrain that at some point and suddenly save a whole bunch of effort, time, money, and regulatory drag on this.But as you go through that journey, you need to not only have a tool that will work when you get there but a tool that will work where you are today. And a lot of companies miss that mark, too. It's, “Oh, once you modernize and become the serverless success story of the decade, then our product is going to be right for you.” “Great. We'll send you a postcard if we ever get there and then you can follow up with us.”Alternately, it's well, “Yeah, we're this is how we are today, but we have a visions of a brighter tomorrow.” You've got to be able to meet people where they are at any point of that journey. One of the things I've always respected about Cribl has been the way that you very fluidly tell both sides of that story.Clint: And it's not their fault.Corey: Yeah.Clint: Most of the people who pick a job, they pick the job because, like—look, I live in Kansas City, Missouri, and there's this data processing company that works primarily on mainframes, it's right down the road. And they gave me a job and it pays me $150,000 a year, and I got a big house and things are great. And I'm a sysadmin sitting there. I don't get to play with the new technology. Like, that customer is just as an applicable customer, we want to help them exactly the same as the new Silicon Valley hip kid who's working at you know, a venture-backed startup, they're doing everything natively in the cloud. Those are all right decisions, depending on where you happen to find yourself, and we want to support you with our products, no matter where you find yourself on the technology spectrum.Corey: Speaking of old and new, and the trends of the industry, when you first set up this recording, you mentioned, “Oh, yeah, we should make it a point to maybe talk about the acquisition,” at which point I sprayed coffee across my iMac. Thanks for that. Turns out it wasn't your acquisition we were talking about so much as it is the—at the time we record this—-the yet-to-close rumored acquisition of Splunk by Cisco.Clint: I think it's both interesting and positive for some people, and sad for others. I think Cisco is obviously a phenomenal company. They run the networking world. The fact that they've been moving into observability—they bought companies like AppDynamics, and we were talking about Epsagon before the show, they bought—ServiceNow, just bought Lightstep recently. There's a lot of acquisitions in this space.I think that when it comes to something like Splunk, Splunk is a fast-growing company by compared to Cisco. And so for them, this is something that they think that they can put into their distribution channel, and what Cisco knows how to do is to sell things like they're very good at putting things through their existing sales force and really amplifying the sales of that particular thing that they have just acquired. That being said, I think for a company that was as innovative as Splunk, I do find it a bit sad with the idea that it's going to become part of this much larger behemoth and not really probably driving the observability and security industry forward anymore because I don't think anybody really looks at Cisco as a company that's driving things—not to slam them or anything, but I don't really see them as driving the industry forward.Corey: Somewhere along the way, they got stuck and I don't know how to reconcile that because they were a phenomenally fast-paced innovative company, briefly the most valuable company in the world during the dotcom bubble. And then they just sort of stalled out somewhere and, on some level, not to talk smack about it, but it feels like the level of innovation we've seen from Splunk has curtailed over the past half-decade or so. And selling to Cisco feels almost like a tacit admission that they are effectively out of ideas. And maybe that's unfair.Clint: I mean, we can look at the track record of what's been shipped over the last five years from Splunk. And again they're a partner, their customers are great, I think they still have the best log indexing engine on the market. That was their core product and what has made them the majority of their money. But there's not been a lot new. And I think objectively we can look at that without throwing stones and say like, “Well, what net-new? You bought SignalFX. Like, good for you guys like that seems to be going well. You've launched your observability suite based off of these acquisitions.” But organic product-wise, there's not a lot coming out of the factory.Corey: I'll take it a bit further-slash-sadder, we take a look at some great companies that were acquired—OpenDNS, Duo Security, SignalFX, as you mentioned, Epsagon, ThousandEyes—and once they've gotten acquired by Cisco, they all more or less seem to be frozen in time, like they're trapped in amber, which leads us up to the natural dinosaur analogy that I'll probably make in a less formal setting. It just feels like once a company is bought by Cisco, their velocity peters out, a lot of their staff leaves, and what you see is what you get. And I don't know if that's accurate, I'm just not looking in the right places, but every time I talk to folks in the industry about this, I get a lot of knowing nods that are tied to it. So, whether or not that's true or not, that is very clearly, at least in some corners of the market, the active perception.Clint: There's a very real fact that if you look even at very large companies, innovation is driven from a core set of a handful of people. And when those people start to leave, the innovation really stops. It's those people who think about things back from first principles—like why are we doing things? What different can we do?—and they're the type of drivers that drive change.So, Frank Slootman wrote a book recently called Amp it Up that I've been reading over the last weekend, and he talks—has this article that was on LinkedIn a while back called “Drivers vs. Passengers” and he's always looking for drivers. And those drivers tend to not find themselves as happy in bigger companies and they tend to head for the exits. And so then you end up with the people who are a lot of the passenger type of people, the people who are like—they'll carry it forward, they'll continue to scale it, the business will continue to grow at whatever rate it's going to grow, but you're probably not going to see a lot of the net-new stuff. And I'll put it in comparison to a company like Datadog who I have a vast amount of respect for I think they're incredibly innovative company, and I think they continue to innovate.Still driven by the founders, the people who created the original product are still there driving the vision, driving forward innovation. And that's what tends to move the envelope is the people who have the moral authority inside of an even larger organization to say, “Get behind me. We're going in this direction. We're going to go take that hill. We're going to go make things better for our customers.” And when you start to lose those handful of really critical contributors, that's where you start to see the innovation dry up.Corey: Where do you see the acquisitions coming from? Is it just at some point people shove money at these companies that got acquired that is beyond the wildest dreams of avarice? Is it that they believe that they'll be able to execute better on their mission and they were independently? These are still smart, driven, people who have built something and I don't know that they necessarily see an acquisition as, “Well, time to give up and coast for a while and then I'll leave.” But maybe it is. I've never found myself in that situation, so I can't speak for sure.Clint: You kind of I think, have to look at the business and then whoever's running the business at that time—and I sit in the CEO chair—so you have to look at the business and say, “What do we have inside the house here?” Like, “What more can we do?” If we think that there's the next billion-dollar, multi-billion-dollar product sitting here, even just in our heads, but maybe in the factory and being worked on, then we should absolutely not sell because the value is still there and we're going to grow the company much faster as an independent entity than we would you know, inside of a larger organization. But if you're the board of directors and you're looking around and saying like, hey look, like, I don't see another billion-dollar line of bus—at this scale, right, if your Splunk scale, right? I don't see another billion-dollar line of business sitting here, we could probably go acquire it, we could try to add it in, but you know, in the case of something like a Splunk, I think part of—you know, they're looking for a new CEO right now, so now they have to go find a new leader who's going to come in, re-energize and, kind of, reboot that.But that's the options that they're considering, right? They're like, “Do I find a new CEO who's going to reinvigorate things and be able to attract the type of talent that's going to lead us to the next billion-dollar line of business that we can either build inside or we can acquire and bring in-house? Or is the right path for me just to say, ‘Okay, well, you know, somebody like Cisco's interested?'” or the other path that you may see them go down to something like Silver Lake, so Silver Lake put a billion dollars into the company last year. And so they may be looking at and say, “Okay, well, we really need to do some restructuring here and we want to do it outside the eyes of the public market. We want to be able to change pricing model, we want to be able to really do this without having to worry about the stock price's massive volatility because we're making big changes.”And so I would say there's probably two big options there considering. Like, do we sell to Cisco, do we sell to Silver Lake, or do we really take another run at this? And those are difficult decisions for the stewards of the business and I think it's a different decision if you're the steward of the business that created the business versus the steward of the business for whom this is—the I've been here for five years and I may be here for five years more. For somebody like me, a company like Cribl is literally the thing I plan to leave on this earth.Corey: Yeah. Do you have that sense of personal attachment to it? On some level, The Duckbill Group, that's exactly what I'm staring at where it's great. Someone wants to buy the Last Week in AWS media side of the house.Great. Okay. What is that really, beyond me? Because so much of it's been shaped by my personality. There's an audience, sure, but it's a skeptical audience, one that doesn't generally tend to respond well to mass market, generic advertisements, so monetizing that is not going to go super well.“All right, we're going to start doing data mining on people.” Well, that's explicitly against the terms of service people signed up for, so good luck with that. So, much starts becoming bizarre and strange when you start looking at building something with the idea of, oh, in three years, I'm going to unload this puppy and make it someone else's problem. The argument is that by building something with an eye toward selling it, you build a better-structured business, but it also means you potentially make trade-offs that are best not made. I'm not sure there's a right answer here.Clint: In my spare time, I do some investments, angel investments, and that sort of thing, and that's always a red flag for me when I meet a founder who's like, “In three to five years, I plan to sell it to these people.” If you don't have a vision for how you're fundamentally going to alter the marketplace and our perception of everything else, you're not dreaming big enough. And that to me doesn't look like a great investment. It doesn't look like the—how do you attract employees in that way? Like, “Okay, our goal is to work really hard for the next three years so that we will be attractive to this other bigger thing.” They may be thinking it on the inside as an available option, but if you think that's your default option when starting a company, I don't think you're going to end up with the outcome is truly what you're hoping for.Corey: Oh, yeah. In my case, the only acquisition story I see is some large company buying us just largely to shut me up. But—Clint: [laugh].Corey: —that turns out to be kind of expensive, so all right. I also don't think it serve any of them nearly as well as they think it would.Clint: Well, you'll just become somebody else on Twitter. [laugh].Corey: Yeah, “Time to change my name again. Here we go.” So, if people want to go and learn more about a Cribl Edge, where can they do that?Clint: Yeah, cribl.io. And then if you're more of a technical person, and you'd like to understand the specifics, docs.cribl.io. That's where I always go when I'm checking out a vendor; just skip past the main page and go straight to the docs. So, check that out.And then also, if you're wanting to play with the product, we make online available education called Sandboxes, at sandbox.cribl.io, where you can go spin up your own version of the product, walk through some interactive tutorials, and get a view on how it might work for you.Corey: Such a great pattern, at least for the way that I think about these things. You can have flashy videos, you can have great screenshots, you can have documentation that is the finest thing on this earth, but let me play with it; let me kick the tires on it, even with a sample data set. Because until I can do that, I'm not really going to understand where the product starts and where it stops. That is the right answer from where I sit. Again, I understand that everyone's different, not everyone thinks like I do—thankfully—but for me, that's the best way I've ever learned something.Clint: I love to get my hands on the product, and in fact, I'm always a little bit suspicious of any company when I go to their webpage and I can't either sign up for the product or I can't get to the documentation, and I have to talk to somebody in order to learn. That's pretty much I'm immediately going to the next person in that market to go look for somebody who will let me.Corey: [laugh]. Thank you again for taking so much time to speak with me. I appreciate it. As always, it's a pleasure.Clint: Thanks, Corey. Always enjoy talking to you.Corey: Clint Sharp, CEO and co-founder of Cribl. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment. And when you hit submit, be sure to follow it up with exactly how many distinct and disparate logging systems that obnoxious comment had to pass through on your end of things.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode we are going to look at SNMP Configuration.We will be discussing Introduction to SNMP, SNMP Operation, Management Information Base (MIB), SNMP Versions, SNMP Vulnerabilities, SNMPv3, SNMPv3 Security Configuration, and Configure Cisco Devices for Syslog, NTP, and SSH Operations.Thank you so much for listening to this episode of my series on Network Security.Once again, I'm Kevin and this is KevTechify. Let's get this adventure started.http://KevTechify.com***********************************Network Security v1Episode 6 - Device Monitoring and ManagementPart G- SNMP ConfigurationPodcast Number: 25

In this episode we are going to look at Network Security Using Syslog.We will be discussing Introduction to Syslog, Syslog Operation, Syslog Message Format, Syslog Facilities, Configure Syslog Timestamps, Syslog Systems, and Syslog Configuration.Thank you so much for listening to this episode of my series on Network Security.Once again, I'm Kevin and this is KevTechify. Let's get this adventure started.http://KevTechify.com***********************************Network Security v1Episode 6 - Device Monitoring and ManagementPart E- Network Security Using SyslogPodcast Number: 23

The best-informed business insights that support better decision-making begin with data collection, ahead of data processing and analytics. Enterprises nowadays are engulfed by data floods, with data sources ranging from cloud services, applications, to thousands of internal servers. The massive volume of data that organizations must process presents data ingestion challenges for many large companies. In this episode, data security engineer, Vitalli Rudenskyi, discusses the decision to replace a vendor security information and event management (SIEM) system by developing a custom solution with Apache Kafka® and Kafka Connect for a better data collection strategy.Having a data collection infrastructure layer is mission critical for Vitalii and the team in helping enterprises protect data and detect security events. Building on the base of Kafka, their custom SIEM infrastructure is configurable and designed to be able to ingest and analyze huge amounts of data, including personally identifiable information (PII) and healthcare data. When it comes to collecting data, there are two fundamental choices: push or pull. But how about both? Vitalii shares that Kafka Connect API extensions are integral to data ingestion in Kafka. The three key components to allow their SIEM system to collect and record daily by pushing and pulling: NettySource Connector: A connector developed to receive data from different network devices to Apache Kafka. It helps receive data using both the TCP and UDP transport protocols and can be adopted to receive any data from Syslog to SNMP and NetFlow.PollableAPI Connector: A connector made to receive data from remote systems, pulling data from different remote APIs and services.Transformations Library: These are useful extensions to the existing out-of-the-box transformations. Approach with “tag and apply” transformations that transform data into the right place in the right format after collecting data.Listen to learn more as Vitalii shares the importance of data collection and the building of a custom solution to address multi-source data management requirements. EPISODE LINKSFeed Your SIEM Smart with Kafka ConnectTo Pull or to Push Your Data with Kafka Connect? That Is the Question.Free Kafka Connect 101 CourseSyslog Source Connector for Confluent PlatformJoin the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

On this Azure Centric Podcast, we are talking about the newest Azure features announced during this week. Marcos Nogueira and Andrew Lowes bring their point of view on these new Azure features: • General availability: ExpressRoute Global Reach Pricing Reduction • Azure Backup: Upgrade to TLS 1.2 or above for secure MARS agent backups by September 1, 2021 • General availability: Update in Policy Compliance for Resource Type Policies • Azure Migrate private endpoint support available in public preview • General availability: Enterprise-scale landing zone reference implementation for AKS • Public preview: Alerts based smart detection for Application Insights • Public preview: Syslog event collection from Azure Monitor Agent for Linux distros • General availability: Confidential Computing price reduction on DCsv2 virtual machines • Azure IoT Edge integration with Azure Monitor is now in public preview • Five more free services available with an Azure free account You will find videos about Microsoft Azure Technologies, Microsoft Certifications and Technology in general on this channel. The Podcast series is a very informal conversation with different guests. The Azure Concept series is where we bring the real-life experience of using Azure Technologies on the field. Don't forget to subscribe and make sure to hit the like button and share this video if you enjoyed it. Facebook page - https://www.facebook.com/azurecentric Twitter - https://twitter.com/azurecentric Instagram - https://www.instagram.com/azurecentric LinkedIn - https://www.linkedin.com/company/azurecentric SoundCloud - https://bit.ly/acsoundcloudpodcast Google - https://bit.ly/acgooglepodcast Apple - https://bit.ly/acapplepodcast Spotify - https://bit.ly/acspotifypodcast YouTube - https://bit.ly/azurecentricyoutube #AzurePodcast #AzureCentric #AzureUpdates

Jamie author of Software Telemetry book discusses Software Telemetry, why telemetry data is so important and the discipline of tracing, logging, and monitoring infrastructure.

Show Notes and Links Michael Roitzsch returns to Syslog and we talk about scheduling in operating systems. Michael has focussed his research on soft real-time systems and explored, among other things, how good scheduling can help with smooth video display. In this episode, we introduce basic scheduling concepts, what is happening in scheduling today and how the increasingly multi-core world will shape schedulers in the future. Michael is the acting head of the TU Dresden Operating Systems group and researcher at the Barkhausen Institute. Additional sound effects from https://www.zapsplat.com. Discuss the episode in Matrix room #ukvly:matrix.org or on Freenode IRC #ukvly. Send feedback to podcast@ukvly.org or via Twitter. Resources GWU Lecture: Linux CFS vs Real-Time Scheduling TU Dresden: Operating Systems Group Syslog: Contact Tracing Apps Wikipedia: Real-time Computing ATLAS Arachne Grand Central Dispatch (GCD)

FreeBSD Remote Process Plugin Final Milestone achieved, Tailscale for OpenBSD, macOS to FreeBSD migration, monitoring of our OpenBSD machines, OPNsense 20.7.6 released, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD Remote Process Plugin: Final Milestone Achieved (https://www.moritz.systems/blog/freebsd-remote-plugin-final-milestone-achieved/) Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are working on a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one. Tailscale on OpenBSD (https://rakhesh.com/linux-bsd/tailscale-on-openbsd/) I spent some time setting this up today evening and thought I’d post the steps here. Nothing fancy, just putting together various pieces actually. I assume you know what Tailscale is; if not check out their website. Basically it is a mesh network built on top of Wireguard. Using it you can have all your devices both within your LAN(s) and outside be on one overlay network as if they are all on the same LAN and can talk to each other. It’s my new favourite thing! News Roundup macOS to FreeBSD migration a.k.a why I left macOS (https://antranigv.am/weblog_en/posts/macos_to_freebsd/) This is not a technical documentation for how I migrated from macOS to FreeBSD. This is a high-level for why I migrated from macOS to FreeBSD. Not so long ago, I was using macOS as my daily driver. The main reason why I got a macbook was the underlying BSD Unix and the nice graphics it provides. Also, I have an iPhone. But they were also the same reasons for why I left macOS. Our monitoring of our OpenBSD machines, such as it is (as of November 2020 (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/OurOpenBSDMonitoring) We have a number of OpenBSD firewalls in service (along with some other OpenBSD servers for things like VPN endpoints), and I was recently asked how we monitor PF and overall network traffic on them. I had to disappoint the person who asked with my answer, because right now we mostly don't (although this is starting to change). OPNsense 20.7.6 released (https://opnsense.org/opnsense-20-7-6-released/) This update brings the usual mix of reliability fixes, plugin and third party software updates: FreeBSD, HardenedBSD, PHP, OpenSSH, StrongSwan, Suricata and Syslog-ng amongst others. Please note that Let's Encrypt users need to reissue their certificates manually after upgrading to this version to fix the embedded certificate chain issue with the current signing CA switch going on. NYC Bug Jan 2021 with Michael W. Lucas (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/nycbug) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions cy - .so files (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/cy%20-%20.so%20files) ben - mixer volume (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/ben%20-%20mixer%20volume) probono - live cds (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/probono%20-%20live%20cds) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***

FreeBSD 12.2 is available, ZFS Webinar, Enhancing Syzkaller support for NetBSD, how the OpenBSD -stable packages are built, OPNsense 20.7.4 released, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD 12.2 Release (https://www.freebsd.org/releases/12.2R/relnotes.html) The release notes for FreeBSD 12.2-RELEASE contain a summary of the changes made to the FreeBSD base system on the 12-STABLE development line. This document lists applicable security advisories that were issued since the last release, as well as significant changes to the FreeBSD kernel and userland. Some brief remarks on upgrading are also presented. ZFS Webinar: November 18th (https://klarasystems.com/learning/best-practices-for-optimizing-zfs1/) Join us on November 18th for a live discussion with Allan Jude (VP of Engineering at Klara Inc) in this webinar centred on “best practices of ZFS” Building Your Storage Array – Everything from picking the best hardware to RAID-Z and using mirrors. Keeping up with Data Growth – Expanding and growing your pool, and of course, shrinking with device evacuation. Datasets and Properties – Controlling settings with properties and many other tricks! News Roundup Google Summer of Code 2020: [Final Report] Enhancing Syzkaller support for NetBSD (https://blog.netbsd.org/tnf/entry/google_summer_of_code_20202) Sys2syz would give an extra edge to Syzkaller for NetBSD. It has a potential of efficiently automating the conversion of syscall definitions to syzkaller’s grammar. This can aid in increasing the number of syscalls covered by Syzkaller significantly with the minimum possibility of manual errors. Let’s delve into its internals. How the OpenBSD -stable packages are built (https://dataswamp.org/~solene/2020-10-29-official-openbsd-stable-architecture.html) In this long blog post, I will write about the technical details of the OpenBSD stable packages building infrastructure. I have setup the infrastructure with the help of Theo De Raadt who provided me the hardware in summer 2019, since then, OpenBSD users can upgrade their packages using pkg_add -u for critical updates that has been backported by the contributors. Many thanks to them, without their work there would be no packages to build. Thanks to pea@ who is my backup for operating this infrastructure in case something happens to me. OPNsense 20.7.4 released (https://opnsense.org/opnsense-20-7-4-released/) This release finally wraps up the recent Netmap kernel changes and tests. The Realtek vendor driver was updated as well as third party software cURL, libxml2, OpenSSL, PHP, Suricata, Syslog-ng and Unbound just to name a couple of them. Beastie Bits Binutils and linker changes (https://www.dragonflydigest.com/2020/11/03/25120.html) 28 Years of NetBSD contributions (https://github.com/NetBSD/src/graphs/contributors) Bluetooth Audio on OpenBSD (https://ifconfig.se/bluetooth-audio-openbsd.html) K8s Bhyve (https://k8s-bhyve.convectix.com) *** Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Sean - C Flags (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/376/feedback/Sean%20-%20C%20Flags.md) Thierry - RPI ZFS question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/376/feedback/Thierry%20-%20RPI%20ZFS%20question.md) Thierry's script (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/376/feedback/script.md) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***

Learn to Run Your I.T. Business Hosted by Paco Lebron from ProdigyTeks John Dubinsky from Maven Group Billy Austin from Rocket Cyber MSP Unplugged Video Live Show and Chat every Sunday at 7:30pm EST Email: Jeff@MSPUnplugged.com Support This Show Patreon.com/MSPUnplugged BuyMeACoffee.com/MSPUnplugged PayPal.Me/MSPUnplugged   Main Topic: Managed SOC Platform Built for MSPs delivering cyber security to SMBs ENDPOINT | NETWORK | CLOUD   Links: Call to Action: Our friends over at Syncro are hosting a giveaway for a few weeks. Those who sign up for a free trial of Syncro and attend a product tour will receive a nifty Syncro mug!  Visit the URL to book your demo today: https://bit.ly/mugpl   Paco Tips Smart Calling on Sales In this new day and age you need to re-adjust your sales process and comfort. My good friend blah blah blah   John Tips Microsoft Windows File Recovery (Windows 20-04 only) Accidentally deleted an important file? Wiped clean your hard drive? Unsure of what to do with corrupted data? Windows File Recovery can help recover your personal data. ** Do not ignore the Store for Microsoft tools that might help. - Windows Scan, Microsoft Whiteboard, Wireless Display, etc.   whatsmyip.org  WhatsMyIP.org started in April of 2001 when I bought this domain name on a whim. This site was originally hosted on an Apple Mac Quadra 700, connected to a residential internet connection, and the IP Addresses were retrieved via AppleScript. We've come a very long way. If you want to see how long, check us out on Archive.org! ** Port Scanners, WhoIs & DNS, Website Rankings, Password Generator, MAC Address Lookup, etc., etc., etc.   VOIP Tips (rant)  If you are going to be a VOIP provider you ARE going to be a network engineer. If your vendor cannot provide you recommendations on the following - FIND A NEW VENDOR. Your vendor vetting is EXTREMELY important.  Your vendor will easily be 70% of a successful installation.  95% of VOIP sales vendors will ask nothing about the current infrastructure or supply necessary requirements.  Understand that your customers really have no idea what VOIP is, they simply want phone service. There is nothing VOIP cannot do versus an on-site PBX with the correct vendor and configuration. You need to have commercial grade equipment for a successful installation that supports the settings requirements of VOIP (RTP and SIP traffic (QoS) and firewalls that are SIP/VoIP friendly). Know and understand UDP.  You will need to configure or accommodate this for BLF (Busy Lamp Field).    Know and understand QOS (quality of service) so you can prioritize traffic. Large VOIP installations are going to most likely need a separate VLAN for voice. Correctly configure your firewall to bypass trusted VOIP traffic without inspection. You will need both LAN and WAN bandwidth. Example: Take the amount of phone connections times 100kpbs (check with your VOIP vendor) for a good approximation of just your voice needs. Do not forget about VOIP security.  Make sure your vendor considers compliance and feature needs (spam calls, HIPAA, training, etc.).  Make sure you 911 is correctly configured.   Billy Tip Have insight and visibility into customer account authentications.   Guest:  Billy Austin - President & Co-founder at RocketCyber Billy Austin LinkedIn Profile RocketCyber: FAQ’s BLOG How to Videos   RocketCyber's threat detection apps provide solutions for many cybersecurity use cases. Each use case facilitates cyber monitoring opportunities for the managed service provider. Login to your account to turn on preferred RocketApps, no separate installation required.   Each App is purposely built to detect malicious/suspicious activity spanning endpoint, network, and cloud attack pillars. When threats are detected, RocketCyber provides the MSP operator reporting, a triage view, and the ability to generate tickets to your PSA.   Cybersecurity Framework (NIST)   Pax8 NIST Cybersecurity Framework   Windows Defender Cylance Webroot Sentinel One   Untangle Firewall What is a firewall Syslog - firewall log monitoring acting as a syslog collector. Messages are parsed, analyzed, and enriched with threat intel for potential threat indicators. When a threat or security event is detected, message details show up in the console. What is a Syslog? DISCLAIMER: This description contains affiliate links, which means that if you click on one of the product links, we’ll receive a small commission.   Music By Jim Holley

Encrypted Crash Dumps in FreeBSD, Time on Unix, Improve ZVOL sync write performance with a taskq, central log host with syslog-ng, NetBSD Entropy overhaul, Setting Up NetBSD Kernel Dev Environment, and more. Headlines EKCD - Encrypted Crash Dumps in FreeBSD (https://oshogbo.vexillium.org/blog/74/) Some time ago, I was describing how to configure networking crash dumps. In that post, I mentioned that there is also the possibility to encrypt crash dumps. Today we will look into this functionality. Initially, it was implemented during Google Summer of Code 2013 by my friend Konrad Witaszczyk, who made it available in FreeBSD 12. If you can understand Polish, you can also look into his presentation on BSD-PL on which he gave a comprehensive review of all kernel crash dumps features. The main issue with crash dumps is that they may include sensitive information available in memory during a crash. They will contain all the data from the kernel and the userland, like passwords, private keys, etc. While dumping them, they are written to unencrypted storage, so if somebody took out the hard drive, they could access sensitive data. If you are sending a crash dump through the network, it may be captured by third parties. Locally the data are written directly to a dump device, skipping the GEOM subsystem. The purpose of that is to allow a kernel to write a crash dump even in case a panic occurs in the GEOM subsystem. It means that a crash dump cannot be automatically encrypted with GELI. Time on Unix (https://venam.nixers.net/blog/unix/2020/05/02/time-on-unix.html) Time, a word that is entangled in everything in our lives, something we’re intimately familiar with. Keeping track of it is important for many activities we do. Over millennia we’ve developed different ways to calculate it. Most prominently, we’ve relied on the position the sun appears to be at in the sky, what is called apparent solar time. We’ve decided to split it as seasons pass, counting one full cycle of the 4 seasons as a year, a full rotation around the sun. We’ve also divided the passing of light to the lack thereof as days, a rotation of the earth on itself. Moving on to more precise clock divisions such as seconds, minutes, and hours, units that meant different things at different points in history. Ultimately, as travel got faster, the different ways of counting time that evolved in multiple places had to converge. People had to agree on what it all meant. See the article for more News Roundup Improve ZVOL sync write performance by using a taskq (https://github.com/openzfs/zfs/commit/0929c4de398606f8305057ca540cf577e6771c30) A central log host with syslog-ng on FreeBSD - Part 1 (https://blog.socruel.nu/freebsd/a-central-log-host-with-syslog-ng-on-freebsd.html) syslog-ng is the Swiss army knife of log management. You can collect logs from any source, process them in real time and deliver them to wide range of destinations. It allows you to flexibly collect, parse, classify, rewrite and correlate logs from across your infrastructure. This is why syslog-ng is the perfect solution for the central log host of my (mainly) FreeBSD based infrastructure. HEADS UP: NetBSD Entropy Overhaul (https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html) This week I committed an overhaul of the kernel entropy system. Please let me know if you observe any snags! For the technical background, see the thread on tech-kern a few months ago: https://mail-index.NetBSD.org/tech-kern/2019/12/21/msg025876.html. Setting Up NetBSD Kernel Dev Environment (https://adityapadala.com/2020/04/20/Setting-Up-NetBSD-Kernel-Dev-Environment/) I used T_PAGEFLT’s blog post as a reference for setting my NetBSD kernel development environment since his website is down I’m putting down the steps here so it would be helpful for starters. Beastie Bits You can now use ccache to speed up dsynth even more. (https://www.dragonflydigest.com/2020/05/04/24480.html) Improving libossaudio, and the future of OSS in NetBSD (http://blog.netbsd.org/tnf/entry/improving_libossaudio_and_the_future) DragonFlyBSD DHCPCD Import dhcpcd-9.0.2 with the following changes (http://lists.dragonflybsd.org/pipermail/commits/2020-April/769021.html) Reminder: watch this space for upcoming FreeBSD Office Hours, next is May 13th at 2pm Eastern, 18:00 UTC (https://wiki.freebsd.org/OfficeHours) Feedback/Questions Ghislain - ZFS Question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/349/feedback/Ghislain%20-%20ZFS%20Question.md) Jake - Paypal Donations (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/349/feedback/Jake%20-%20Paypal%20Donations.md) Oswin - Hammer tutorial (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/349/feedback/Oswin%20-%20Hammer%20tutorial.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.

Are you frustrated with the task of configuring syslog servers yourself to properly ingest data into Splunk? Take control of the syslog beast once and for all and point your "514" traffic to the new Splunk Connect for Syslog! This new Splunk-supported connector makes quick work of past struggles with syslog servers, sourcetyping, data enrichment, and scale. In this session we will dive into the configuration of the Splunk Connect for Syslog to properly filter, sourcetype, and format your data. We will demonstrate several out-of-the-box examples, highlighting new functionality such as HEC and Kafka transport for resiliency and scale, simple extensions for new device types, and data enrichment that extends far beyond simple sourcetyping of the raw message. Lastly, we will look forward to the integration of syslog with Splunk's new Data Stream Processor, and highlight appropriate use cases for each solution. By the time we wrap up, you will know how to tame the syslog beast! Speaker(s) Ryan Faircloth, Security Product Manager, Splunk Mark Bonsack, Staff Sales Engineer, Splunk Slides PDF link - https://conf.splunk.com/files/2019/slides/FN1651.pdf?podcast=1577146202 Product: Splunk Enterprise, Splunk Cloud, Splunk Data Fabric Search and Data Stream Processor Track: Foundations/Platform Level: Good for all skill levels

Are you frustrated with the task of configuring syslog servers yourself to properly ingest data into Splunk? Take control of the syslog beast once and for all and point your "514" traffic to the new Splunk Connect for Syslog! This new Splunk-supported connector makes quick work of past struggles with syslog servers, sourcetyping, data enrichment, and scale. In this session we will dive into the configuration of the Splunk Connect for Syslog to properly filter, sourcetype, and format your data. We will demonstrate several out-of-the-box examples, highlighting new functionality such as HEC and Kafka transport for resiliency and scale, simple extensions for new device types, and data enrichment that extends far beyond simple sourcetyping of the raw message. Lastly, we will look forward to the integration of syslog with Splunk's new Data Stream Processor, and highlight appropriate use cases for each solution. By the time we wrap up, you will know how to tame the syslog beast! Speaker(s) Ryan Faircloth, Security Product Manager, Splunk Mark Bonsack, Staff Sales Engineer, Splunk Slides PDF link - https://conf.splunk.com/files/2019/slides/FN1651.pdf?podcast=1577146230 Product: Splunk Enterprise, Splunk Cloud, Splunk Data Fabric Search and Data Stream Processor Track: Foundations/Platform Level: Good for all skill levels

Are you frustrated with the task of configuring syslog servers yourself to properly ingest data into Splunk? Take control of the syslog beast once and for all and point your "514" traffic to the new Splunk Connect for Syslog! This new Splunk-supported connector makes quick work of past struggles with syslog servers, sourcetyping, data enrichment, and scale. In this session we will dive into the configuration of the Splunk Connect for Syslog to properly filter, sourcetype, and format your data. We will demonstrate several out-of-the-box examples, highlighting new functionality such as HEC and Kafka transport for resiliency and scale, simple extensions for new device types, and data enrichment that extends far beyond simple sourcetyping of the raw message. Lastly, we will look forward to the integration of syslog with Splunk's new Data Stream Processor, and highlight appropriate use cases for each solution. By the time we wrap up, you will know how to tame the syslog beast! Speaker(s) Ryan Faircloth, Security Product Manager, Splunk Mark Bonsack, Staff Sales Engineer, Splunk Slides PDF link - https://conf.splunk.com/files/2019/slides/FN1651.pdf?podcast=1577146253 Product: Splunk Enterprise, Splunk Cloud, Splunk Data Fabric Search and Data Stream Processor Track: Foundations/Platform Level: Good for all skill levels

Are you frustrated with the task of configuring syslog servers yourself to properly ingest data into Splunk? Take control of the syslog beast once and for all and point your "514" traffic to the new Splunk Connect for Syslog! This new Splunk-supported connector makes quick work of past struggles with syslog servers, sourcetyping, data enrichment, and scale. In this session we will dive into the configuration of the Splunk Connect for Syslog to properly filter, sourcetype, and format your data. We will demonstrate several out-of-the-box examples, highlighting new functionality such as HEC and Kafka transport for resiliency and scale, simple extensions for new device types, and data enrichment that extends far beyond simple sourcetyping of the raw message. Lastly, we will look forward to the integration of syslog with Splunk's new Data Stream Processor, and highlight appropriate use cases for each solution. By the time we wrap up, you will know how to tame the syslog beast! Speaker(s) Ryan Faircloth, Security Product Manager, Splunk Mark Bonsack, Staff Sales Engineer, Splunk Slides PDF link - https://conf.splunk.com/files/2019/slides/FN1651.pdf?podcast=1577146268 Product: Splunk Enterprise, Splunk Cloud, Splunk Data Fabric Search and Data Stream Processor Track: Foundations/Platform Level: Good for all skill levels

Are you frustrated with the task of configuring syslog servers yourself to properly ingest data into Splunk? Take control of the syslog beast once and for all and point your "514" traffic to the new Splunk Connect for Syslog! This new Splunk-supported connector makes quick work of past struggles with syslog servers, sourcetyping, data enrichment, and scale. In this session we will dive into the configuration of the Splunk Connect for Syslog to properly filter, sourcetype, and format your data. We will demonstrate several out-of-the-box examples, highlighting new functionality such as HEC and Kafka transport for resiliency and scale, simple extensions for new device types, and data enrichment that extends far beyond simple sourcetyping of the raw message. Lastly, we will look forward to the integration of syslog with Splunk's new Data Stream Processor, and highlight appropriate use cases for each solution. By the time we wrap up, you will know how to tame the syslog beast! Speaker(s) Ryan Faircloth, Security Product Manager, Splunk Mark Bonsack, Staff Sales Engineer, Splunk Slides PDF link - https://conf.splunk.com/files/2019/slides/FN1651.pdf?podcast=1577146225 Product: Splunk Enterprise, Splunk Cloud, Splunk Data Fabric Search and Data Stream Processor Track: Foundations/Platform Level: Good for all skill levels

Unix is 50, Hunting down Ken's PDP-7, OpenBSD and OPNSense have new releases, Clarification on what GhostBSD is, sshuttle - VPN over SSH, and more. Headlines Unix is 50 (https://www.bell-labs.com/unix50/) In the summer of 1969 computer scientists Ken Thompson and Dennis Ritchie created the first implementation of Unix with the goal of designing an elegant and economical operating system for a little-used PDP-7 minicomputer at Bell Labs. That modest project, however, would have a far-reaching legacy. Unix made large-scale networking of diverse computing systems — and the Internet — practical. The Unix team went on to develop the C language, which brought an unprecedented combination of efficiency and expressiveness to programming. Both made computing more "portable". Today, Linux, the most popular descendent of Unix, powers the vast majority of servers, and elements of Unix and Linux are found in most mobile devices. Meanwhile C++ remains one of the most widely used programming languages today. Unix may be a half-century old but its influence is only growing. Hunting down Ken's PDP-7: video footage found (https://bsdimp.blogspot.com/2019/10/video-footage-of-first-pdp-7-to-run-unix.html) In my prior blog post, I traced Ken's scrounged PDP-7 to SN 34. In this post I'll show that we have actual video footage of that PDP-7 due to an old film from Bell Labs. this gives us almost a minute of footage of the PDP-7 Ken later used to create Unix. News Roundup OpenBSD 6.6 Released (https://openbsd.org/66.html) Announce: https://marc.info/?l=openbsd-tech&m=157132024225971&w=2 Upgrade Guide: https://openbsd.org/faq/upgrade66.html Changelog: https://openbsd.org/plus66.html OPNsense 19.7.5 released (https://opnsense.org/opnsense-19-7-5-released/) Hello friends and followers, Lots of plugin and ports updates this time with a few minor improvements in all core areas. Behind the scenes we are starting to migrate the base system to version 12.1 which is supposed to hit the next 20.1 release. Stay tuned for more infos in the next month or so. Here are the full patch notes: + system: show all swap partitions in system information widget + system: flatten services_get() in preparation for removal + system: pin Syslog-ng version to specific package name + system: fix LDAP/StartTLS with user import page + system: fix a PHP warning on authentication server page + system: replace most subprocess.call use + interfaces: fix devd handling of carp devices (contributed by stumbaumr) + firewall: improve firewall rules inline toggles + firewall: only allow TCP flags on TCP protocol + firewall: simplify help text for direction setting + firewall: make protocol log summary case insensitive + reporting: ignore malformed flow records + captive portal: fix type mismatch for timeout read + dhcp: add note for static lease limitation with lease registration (contributed by Northguy) + ipsec: add margintime and rekeyfuzz options + ipsec: clear $dpdline correctly if not set + ui: fix tokenizer reorder on multiple saves + plugins: os-acme-client 1.26[1] + plugins: os-bind will reload bind on record change (contributed by blablup) + plugins: os-etpro-telemetry minor subprocess.call replacement + plugins: os-freeradius 1.9.4[2] + plugins: os-frr 1.12[3] + plugins: os-haproxy 2.19[4] + plugins: os-mailtrail 1.2[5] + plugins: os-postfix 1.11[6] + plugins: os-rspamd 1.8[7] + plugins: os-sunnyvalley LibreSSL support (contributed by Sunny Valley Networks) + plugins: os-telegraf 1.7.6[8] + plugins: os-theme-cicada 1.21 (contributed by Team Rebellion) + plugins: os-theme-tukan 1.21 (contributed by Team Rebellion) + plugins: os-tinc minor subprocess.call replacement + plugins: os-tor 1.8 adds dormant mode disable option (contributed by Fabian Franz) + plugins: os-virtualbox 1.0 (contributed by andrewhotlab) Dealing with the misunderstandings of what is GhostBSD (http://ghostbsd.org/node/194) Since the release of 19.09, I have seen a lot of misunderstandings on what is GhostBSD and the future of GhostBSD. GhostBSD is based on TrueOS with FreeBSD 12 STABLE with our twist to it. We are still continuing to use TrueOS for OpenRC, and the new package's system for the base system that is built from ports. GhostBSD is becoming a slow-moving rolling release base on the latest TrueOS with FreeBSD 12 STABLE. When FreeBSD 13 STABLE gets released, GhostBSD will be upgraded to TrueOS with FreeBSD 13 STABLE. Our official desktop is MATE, which means that the leading developer of GhostBSD does not officially support XFCE. Community releases are maintained by the community and for the community. GhostBSD project will provide help to build and to host the community release. If anyone wants to have a particular desktop supported, it is up to the community. Sure I will help where I can, answer questions and guide new community members that contribute to community release. There is some effort going on for Plasma5 desktop. If anyone is interested in helping with XFCE and Plasma5 or in creating another community release, you are well come to contribute. Also, Contribution to the GhostBSD base system, to ports and new ports, and in house software are welcome. We are mostly active on Telegram https://t.me/ghostbsd, but you can also reach us on the forum. SHUTTLE – VPN over SSH | VPN Alternative (https://www.terminalbytes.com/sshuttle-vpn-over-ssh-vpn-alternative/) Looking for a lightweight VPN client, but are not ready to spend a monthly recurring amount on a VPN? VPNs can be expensive depending upon the quality of service and amount of privacy you want. A good VPN plan can easily set you back by 10$ a month and even that doesn’t guarantee your privacy. There is no way to be sure whether the VPN is storing your confidential information and traffic logs or not. sshuttle is the answer to your problem it provides VPN over ssh and in this article we’re going to explore this cheap yet powerful alternative to the expensive VPNs. By using open source tools you can control your own privacy. VPN over SSH – sshuttle sshuttle is an awesome program that allows you to create a VPN connection from your local machine to any remote server that you have ssh access on. The tunnel established over the ssh connection can then be used to route all your traffic from client machine through the remote machine including all the dns traffic. In the bare bones sshuttle is just a proxy server which runs on the client machine and forwards all the traffic to a ssh tunnel. Since its open source it holds quite a lot of major advantages over traditional VPN. OpenSSH 8.1 Released (http://www.openssh.com/txt/release-8.1) Security ssh(1), sshd(8), ssh-add(1), ssh-keygen(1): an exploitable integer overflow bug was found in the private key parsing code for the XMSS key type. This key type is still experimental and support for it is not compiled by default. No user-facing autoconf option exists in portable OpenSSH to enable it. This bug was found by Adam Zabrocki and reported via SecuriTeam's SSD program. ssh(1), sshd(8), ssh-agent(1): add protection for private keys at rest in RAM against speculation and memory side-channel attacks like Spectre, Meltdown and Rambleed. This release encrypts private keys when they are not in use with a symmetric key that is derived from a relatively large "prekey" consisting of random data (currently 16KB). This release includes a number of changes that may affect existing configurations: ssh-keygen(1): when acting as a CA and signing certificates with an RSA key, default to using the rsa-sha2-512 signature algorithm. Certificates signed by RSA keys will therefore be incompatible with OpenSSH versions prior to 7.2 unless the default is overridden (using "ssh-keygen -t ssh-rsa -s ..."). New Features ssh(1): Allow %n to be expanded in ProxyCommand strings ssh(1), sshd(8): Allow prepending a list of algorithms to the default set by starting the list with the '^' character, E.g. "HostKeyAlgorithms ^ssh-ed25519" ssh-keygen(1): add an experimental lightweight signature and verification ability. Signatures may be made using regular ssh keys held on disk or stored in a ssh-agent and verified against an authorized_keys-like list of allowed keys. Signatures embed a namespace that prevents confusion and attacks between different usage domains (e.g. files vs email). ssh-keygen(1): print key comment when extracting public key from a private key. ssh-keygen(1): accept the verbose flag when searching for host keys in known hosts (i.e. "ssh-keygen -vF host") to print the matching host's random-art signature too. All: support PKCS8 as an optional format for storage of private keys to disk. The OpenSSH native key format remains the default, but PKCS8 is a superior format to PEM if interoperability with non-OpenSSH software is required, as it may use a less insecure key derivation function than PEM's. Beastie Bits Say goodbye to the 32 CPU limit in NetBSD/aarch64 (https://twitter.com/jmcwhatever/status/1185584719183962112) vBSDcon 2019 videos (https://www.youtube.com/channel/UCvcdrOSlYOSzOzLjv_n1_GQ/videos) Browse the web in the terminal - W3M (https://www.youtube.com/watch?v=3Hfda0Tjqsg&feature=youtu.be) NetBSD 9 and GSoC (http://netbsd.org/~kamil/GSoC2019.html#slide1) BSDCan 2019 Videos (https://www.youtube.com/playlist?list=PLeF8ZihVdpFegPoAKppaDSoYmsBvpnSZv) NYC*BUG Install Fest: Nov 6th 18:45 @ Suspenders (https://www.nycbug.org/index?action=view&id=10673) FreeBSD Miniconf at linux.conf.au 2020 Call for Sessions Now Open (https://www.freebsdfoundation.org/blog/freebsd-miniconf-at-linux-conf-au-2020-call-for-sessions-now-open/) FOSDEM 2020 - BSD Devroom Call for Participation (https://people.freebsd.org/~rodrigo/fosdem20/) University of Cambridge looking for Research Assistants/Associates (https://twitter.com/ed_maste/status/1184865668317007874) Feedback/Questions Trenton - Beeping Thinkpad (http://dpaste.com/0ZEXNM6#wrap) Alex - Per user ZFS Datasets (http://dpaste.com/1K31A65#wrap) Allan’s old patch from 2015 (https://reviews.freebsd.org/D2272) Javier - FBSD 12.0 + ZFS + encryption (http://dpaste.com/1XX4NNA#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.

OPNsense 19.7.1 is out, ZFS on Linux still has annoying issues with ARC size, Hammer2 is now default, NetBSD audio – an application perspective, new FreeNAS Mini, and more. Headlines OPNsense 19.7.1 (https://opnsense.org/opnsense-19-7-1-released/) We do not wish to keep you from enjoying your summer time, but this is a recommended security update enriched with reliability fixes for the new 19.7 series. Of special note are performance improvements as well as a fix for a longstanding NAT before IPsec limitation. Full patch notes: system: do not create automatic copies of existing gateways system: do not translate empty tunables descriptions system: remove unwanted form action tags system: do not include Syslog-ng in rc.freebsd handler system: fix manual system log stop/start/restart system: scoped IPv6 "%" could confuse mwexecf(), use plain mwexec() instead system: allow curl-based downloads to use both trusted and local authorities system: fix group privilege print and correctly redirect after edit system: use cached address list in referrer check system: fix Syslog-ng search stats firewall: HTML-escape dynamic entries to display aliases firewall: display correct IP version in automatic rules firewall: fix a warning while reading empty outbound rules configuration firewall: skip illegal log lines in live log interfaces: performance improvements for configurations with hundreds of interfaces reporting: performance improvements for Python 3 NetFlow aggregator rewrite dhcp: move advanced router advertisement options to correct config section ipsec: replace global array access with function to ensure side-effect free boot ipsec: change DPD action on start to "dpdaction = restart" ipsec: remove already default "dpdaction = none" if not set ipsec: use interface IP address in local ID when doing NAT before IPsec web proxy: fix database reset for Squid 4 by replacing use of sslcrtd with securityfile_certgen plugins: os-acme-client 1.24[1] plugins: os-bind 1.6[2] plugins: os-dnscrypt-proxy 1.5[3] plugins: os-frr now restricts characters BGP prefix-list and route-maps[4] plugins: os-google-cloud-sdk 1.0[5] ports: curl 7.65.3[6] ports: monit 5.26.0[7] ports: openssh 8.0p1[8] ports: php 7.2.20[9] ports: python 3.7.4[10] ports: sqlite 3.29.0[11] ports: squid 4.8[12] Stay safe and hydrated, Your OPNsense team ZFS on Linux still has annoying issues with ARC size (https://utcc.utoronto.ca/~cks/space/blog/linux/ZFSOnLinuxARCShrinkage) One of the frustrating things about operating ZFS on Linux is that the ARC size is critical but ZFS's auto-tuning of it is opaque and apparently prone to malfunctions, where your ARC will mysteriously shrink drastically and then stick there. Linux's regular filesystem disk cache is very predictable; if you do disk IO, the cache will relentlessly grow to use all of your free memory. This sometimes disconcerts people when free reports that there's very little memory actually free, but at least you're getting value from your RAM. This is so reliable and regular that we generally don't think about 'is my system going to use all of my RAM as a disk cache', because the answer is always 'yes'. (The general filesystem cache is also called the page cache.) This is unfortunately not the case with the ZFS ARC in ZFS on Linux (and it wasn't necessarily the case even on Solaris). ZFS has both a current size and a 'target size' for the ARC (called 'c' in ZFS statistics). When your system boots this target size starts out as the maximum allowed size for the ARC, but various events afterward can cause it to be reduced (which obviously limits the size of your ARC, since that's its purpose). In practice, this reduction in the target size is both pretty sticky and rather mysterious (as ZFS on Linux doesn't currently expose enough statistics to tell why your ARC target size shrunk in any particular case). The net effect is that the ZFS ARC is not infrequently quite shy and hesitant about using memory, in stark contrast to Linux's normal filesystem cache. The default maximum ARC size starts out as only half of your RAM (unlike the regular filesystem cache, which will use all of it), and then it shrinks from there, sometimes very significantly, and once shrunk it only recovers slowly (if at all). News Roundup Hammer2 is now default (http://lists.dragonflybsd.org/pipermail/commits/2019-June/718989.html) ``` commit a49112761c919d42d405ec10252eb0553662c824 Author: Matthew Dillon Date: Mon Jun 10 17:53:46 2019 -0700 installer - Default to HAMMER2 * Change the installer default from HAMMER1 to HAMMER2. * Adjust the nrelease build to print the location of the image files when it finishes. Summary of changes: nrelease/Makefile | 2 +- usr.sbin/installer/dfuibe_installer/flow.c | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/a49112761c919d42d405ec10252eb0553662c824 ``` NetBSD audio – an application perspective (https://netbsd.org/gallery/presentations/nia/netbsd-audio/) NetBSD audio – an application perspective ... or, "doing it natively, because we can" audio options for NetBSD in pkgsrc Use NetBSD native audio (sun audio/audioio.h) Or OSS emulation layer: Basically a wrapper around sun audio in the kernel. Incomplete and old version, but works for simple stuff Many many abstraction layers available: OpenAL-Soft alsa-lib (config file required) libao, GStreamer (plugins!) PortAudio, SDL PulseAudio, JACK ... lots more!? some obsolete stuff (esd, nas?) Advantages of using NetBSD audio directly Low latency, low CPU usage: Abstraction layers differ in latency (SDL2 vs ALSA/OpenAL) Query device information: Is /dev/audio1 a USB microphone or another sound card? Avoid bugs from excessive layering Nice API, well documented: [nia note: I had no idea how to write audio code. I read a man page and now I do.] Your code might work on illumos too [nia note: SDL2 seems very sensitive to the blk_ms sysctl being high or low, with other implementations there seems to be a less noticable difference. I don't know why.] New FreeNAS Mini (https://www.ixsystems.com/blog/new-freenas-mini-models-release-pr/) Two new FreeNAS Mini systems join the very popular FreeNAS Mini and Mini XL: FreeNAS Mini XL+: This powerful 10 Bay platform (8x 3.5” and 1x 2.5” hot-swap, 1x 2.5” internal) includes the latest, compact server technology and provides dual 10GbE ports, 8 CPU cores and 32 GB RAM for high performance workgroups. The Mini XL+ scales beyond 100TB and is ideal for very demanding applications, including hosting virtual machines and multimedia editing. Starting at $1499, the Mini XL+ configured with cache SSD and 80 TB capacity is $4299, and consumes about 100 Watts. FreeNAS Mini E: This cost-effective 4 Bay platform provides the resources required for SOHO use with quad GbE ports and 8 GB of RAM. The Mini E is ideal for file sharing, streaming and transcoding video at 1080p. Starting at $749, the Mini E configured with 8 TB capacity is $999, and consumes about 36 Watts. Beastie Bits Welcome to NetBSD 9.99.1! (https://mail-index.netbsd.org/source-changes/2019/07/30/msg107671.html) Berkeley smorgasbord — part II (http://blog.snailtext.com/posts/berkeley-smorgasbord-part-2.html) dtracing postgres (https://www.youtube.com/watch?v=Brt41xnMZqo&list=PLuJmmKtsV1dOTmlImlD9U5j1P1rLxS2V8&index=20&t=0s) Project Trident 19.07-U1 now available (https://project-trident.org/post/2019-07-30_19.07-u1_available/) Need a Secure Operating System? Take a Look at OpenBSD (https://www.devprojournal.com/technology-trends/operating-systems/need-a-secure-operating-system-take-a-look-at-openbsd/) Feedback/Questions Jeff - OpenZFS Port Testing Feedback (http://dpaste.com/2AT7JGP#wrap) Malcolm - Best Practices for Custom Ports (http://dpaste.com/1R170D7) Michael - Little Correction (http://dpaste.com/0CERP6R) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.

Det är nu dags att avsluta vår miniserie i tre delar och ta allt vi har lärt oss och avsluta med att "bygga en SOC (Security Operations Center)" för att kunna upptäcka och spåra hot. Dessutom tar vi upp hur hösten kommer se ut på IT-säkerhetspodden. Detta avsnitt finns som video med presentation här: https://www.youtube.com/watch?v=pTguo_kKpTk

I det andra av tre avsnitt om loggning går vi in på fem omutliga krav för att bygga en säker loggstruktur. Dessa krav är grunden för att överhuvudtaget kunna få någon form av säkerhet för en sådan lösning. Erik Zalitis förklarar också vilket som är det magiska ordet för att få allting du någonsin önskar. Detta avsnitt finns som video med presentation här: https://www.youtube.com/watch?v=yy2M7Z06Sd8

Microsoft Azure Sentinel is a new Cloud native SIEM service with built-in AI for analytics that removes the cost and complexity of achieving a central and focused near real-time view of the active threats in your environment. Koby Koren from the Azure Sentinel engineering team walks through the entire solution with an end-to-end demonstration from how to set it up, perform queries, investigations and more. Azure Sentinel is in preview today. Follow the link to try for yourself https://aka.ms/AzureSentinel

GSoC 2018 Projects announced, tutorial FreeBSD jails with iocage, new Code of Conduct for FreeBSD, libhijack, and fancy monitoring for OpenSMTPD This episode was brought to you by Headlines Google Summer of Code 2018 (https://summerofcode.withgoogle.com/organizations/?sp-page=5) FreeBSD (https://www.freebsd.org/projects/summerofcode.html) FreeBSD Google Summer oF Code Ideas (https://wiki.freebsd.org/SummerOfCodeIdeas) You can join #freebsd-soc on the efnet IRC network to chat with FreeBSD developers interested in mentoring student proposals and projects, past FreeBSD/GSoC students, and other students applying to FreeBSD/GSoC this year. NetBSD (https://mail-index.netbsd.org/netbsd-advocacy/2018/02/12/msg000765.html) You can get a stipend (paid for by Google) and spend a few months getting to know and improving the insides of NetBSD or pkgsrc. ``` The schedule is: 12-27 March Applying 23 April Find out if you were accepted 14 May - 22 August Do the project! We have some suggestions for suitable projects: - ARM EFI bootloader - Using libFuzzer on base tools - Refactoring ALTQ (QoS implementation) and integrating with NPF - Testsuite for libcurses - Improve pkgin Other suggestions and details are at: https://wiki.netbsd.org/projects/gsoc/ ``` These projects are suggestions; you can come up with your own. Suggestions for other suitable projects are welcome. Feel free to contact, or chat around on IRC: irc.freenode.org #netbsd #netbsd-code #pkgsrc Haiku (https://summerofcode.withgoogle.com/organizations/4821756754264064/) Students: How to Apply for a Haiku Idea (https://www.haiku-os.org/community/gsoc/2018/students) Project Ideas (https://www.haiku-os.org/community/gsoc/2018/ideas) > If you have questions you can contact the devs on IRC: irc.freenode.org #haiku FreeBSD Jails with iocage (http://norrist.devio.us/iocage_freebsd.html) Introduction FreeBSD jails allow users to run multiple, isolated instances of FreeBSD on a single server. Iocage simplifies the management of FreeBSD Jails. Following this tutorial, the jails will be configured to bind to an IP address on the jail host's internal network, and the host OS will pass traffic from the external network to the jail. The jails will be managed with Iocage. Iocage uses ZFS properties to store configuration data for each jail, so a ZFS file system is required. Network setup These steps will: Set up the internal network. Enable the pf packet filter Configure pf pass internet traffic to and from the jail. PF is full featured firewall, and can do more than just pass traffic to an internal network. Refer to the PF documentation for additional configuration options. Run the following to configure the internal network and enable pf. sysrc cloned_interfaces+="lo1" sysrc ifconfig_lo1="inet 192.0.2.1/24" sysrc pf_enable="YES" Put the following in /etc/pf.conf ``` Variables ext_if should be set to the hosts external NIC extif = "vtnet0" jailif = "lo1" jailnet = $jailif:network NAT allows the jails to access the external network nat on $extif from $jailnet to any -> ($ext_if) Redirect traffic on port 80 to the web server jail Add similar rules for additional jails rdr pass on $ext_if inet proto tcp to port 80 -> 192.0.2.10 ``` Reboot to activate the network changes ZFS The best way to use ZFS on a VPS is to attach block storage as a new disk. If block storage is not available, you can optionally use a file as the ZFS device. Enable and start ZFS. sysrc zfs_enable="YES" service zfs start ZFS using Block storage List the available disks. If you are using a VPS, the block store will probably be the second disk. geom disk list Create a ZFS pool named jailstore. zpool create jailstore /dev/vtbd1 ZFS using a file Create the ZFS file. dd if=/dev/zero of=/zfsfile bs=1M count=4096 Create a ZFS pool named jailstore. zpool create jailstore /zfsfile Install iocage the easy way pkg install py36-iocage Skip to "Using iocage" Install iocage the hard way Swap file Smaller servers may not have enough RAM to build iocage. If needed, create a swap file and reboot. dd if=/dev/zero of=/swapfile bs=1M count=1024 echo 'swapfile="/swapfile"' >> /etc/rc.conf reboot Install some build dependencies pkg install subversion python36 git-lite libgit2 py36-pip Building iocage requires the FreeBSD source. svn checkout https://svn.freebsd.org/base/releng/11.1 /usr/src Get the latest FreeBSD ports tree. ``` portsnap fetch portsnap extract ``` + build iocage. cd /usr/ports/sysutils/iocage/ make install Using iocage ``` iocage activate jailstore iocage fetch iocage create -n www ip4_addr="lo1|192.0.2.10/24" -r 11.1-RELEASE iocage start www iocage console www ``` Once you have a shell inside the jail, install and start Apache. pkg install apache24 sysrc apache24_enable="yes" service apache24 start Port 80 on the jail will now be accessible on the hosts IP address. Multiple jails. Additional jails can be installed using the example above. Install the new jail with the iocage create command , but use a different IP address Expose the new jail to the network by adding additional rules to pf.conf. iXsystems SNIA Persistent Memory Summit 2018 Report (https://www.ixsystems.com/blog/snia-report-2018/) New FreeBSD Code of Conduct (https://www.freebsd.org/internal/code-of-conduct.html) The FreeBSD Project is inclusive. We want the FreeBSD Project to be a venue where people of all backgrounds can work together to make the best operating system, built by a strong community. These values extend beyond just development to all aspects of the Project. All those given recognition as members of the Project in whatever form are seen as ambassadors of the Project. Diversity is a huge strength and is critical to the long term success of the Project. To that end we have a few ground rules that we ask people to adhere to. This code applies equally to everyone representing the FreeBSD Project in any way, from new members, to committers, to the core team itself. These rules are intended to ensure a safe, harassment-free environment for all and to ensure that everyone feels welcome both working within, and interacting with, the Project. This document is not an exhaustive list of things that you should not do. Rather, consider it a guide to make it easier to enrich all of us and the technical communities in which we participate. This code of conduct applies to all spaces used by the FreeBSD Project, including our mailing lists, IRC channels, and social media, both online and off. Anyone who is found to violate this code of conduct may be sanctioned or expelled from FreeBSD Project controlled spaces at the discretion of the FreeBSD Code of Conduct Committee. Some FreeBSD Project spaces may have additional rules in place, which will be made clearly available to participants. Participants are responsible for knowing and abiding by these rules. Harassment includes but is not limited to: + Comments that reinforce systemic oppression related to gender, gender identity and expression, sexual orientation, disability, mental illness, neurodiversity, physical appearance, body size, age, race, or religion. + Unwelcome comments regarding a person's lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment. + Deliberate misgendering. + Deliberate use of "dead" or rejected names. + Gratuitous or off-topic sexual images or behaviour in spaces where they're not appropriate. + Physical contact and simulated physical contact (e.g., textual descriptions like "hug" or "backrub") without consent or after a request to stop. + Threats of violence. + Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm. + Deliberate intimidation. + Stalking or following. + Harassing photography or recording, including logging online activity for harassment purposes. + Sustained disruption of discussion. + Unwelcome sexual attention. + Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others. + Continued one-on-one communication after requests to cease. + Deliberate "outing" of any private aspect of a person's identity without their consent except as necessary to protect vulnerable people from intentional abuse. + Publication of non-harassing private communication without consent. + Publication of non-harassing private communication with consent but in a way that intentionally misrepresents the communication (e.g., removes context that changes the meaning). + Knowingly making harmful false claims about a person. Interview - Benno Rice - benno@freebsd.org (mailto:benno@freebsd.org) / @jeamland (https://twitter.com/jeamland) News Roundup libhijack in PoC||GTFO 0x17! (https://www.soldierx.com/news/libhijack-PoCGTFO-0x17) Hijacking Your Free Beasties In the land of red devils known as Beasties exists a system devoid of meaningful exploit mitigations. As we explore this vast land of opportunity, we will meet our ELFish friends, [p]tracing their very moves in order to hijack them. Since unprivileged process debugging is enabled by default on FreeBSD, we can abuse PTrace to create anonymous memory mappings, inject code into them, and overwrite PLT/GOT entries. We will revive a tool called libhijack to make our nefarious activities of hijacking ELFs via PTrace relatively easy. Nothing presented here is technically new. However, this type of work has not been documented in this much detail, tying it all into one cohesive work. In Phrack 56, Silvio Cesare taught us ELF research enthusiasts how to hook the PLT/GOT. The Phrack 59 article on Runtime Process Infection briefly introduces the concept of injecting shared objects by injecting shellcode via PTrace that calls dlopen(). No other piece of research, however, has discovered the joys of forcing the application to create anonymous memory mappings in which to inject Code. This is only part one of a series of planned articles that will follow libhijack's development. The end goal is to be able to anonymously inject shared objects. The libhijack project is maintained by the SoldierX community. Previous Research All prior work injects code into the stack, the heap, or existing executable code. All three methods create issues on today's systems. On amd64 and arm64, the two architectures libhijack cares about, the stack is non-executable by default. jemalloc, the heap implementation on FreeBSD, creates non-executable mappings. Obviously overwriting existing executable code destroys a part of the executable image. The Role of ELF > FreeBSD provides a nifty API for inspecting the entire virtual memory space of an application. The results returned from the API tells us the protection flags (readable, writable, executable) of each mapping. If FreeBSD provides such a rich API, why would we need to parse the ELF headers? PLT/GOT hijacking requires parsing ELF headers. One would not be able to find the PLT/GOT without iterating through the Process Headers to find the Dynamic Headers, eventually ending up with the DT_PLTGOT entry. With FreeBSD's libprocstat API, we don't have a need for parsing ELF headers until we get to the PLT/GOT stage, but doing so early makes it easier for the attacker using libhijack The Future of libhijack Writing devious code in assembly is cumbersome. Assembly doesn't scale well to multiple architectures. Instead, we would like to write our devious code in C, compiling to a shared object that gets injected anonymously. This requires writing a remote RTLD within libhijack and is in progress. Writing a remote RTLD will take a while as doing so is not an easy task. Additionally, creation of a general-purpose helper library that gets injected would be helpful. It could aid in PLT/GOT redirection attacks, possibly storing the addresses of functions we've previously hijacked. This work is dependent on the remote RTLD. libhijack currently lacks documentation. Once the ABI and API stabilize, formal documentation will be written. Conclusion Using libhijack, we can easily create anonymous memory mappings, inject into them arbitrary code, and hijack the PLT/GOT on FreeBSD. On HardenedBSD, a hardened derivative of FreeBSD, libhijack is fully mitigated through PaX NOEXEC. We've demonstrated that wrapper-style Capsicum is ineffective on FreeBSD. Through the use of libhijack, we emulate a control flow hijack in which the application is forced to call sandbox_open and fdlopen on the resulting file descriptor. Further work to support anonymous injection of full shared objects, along with their dependencies, will be supported in the future. Imagine injecting libpcap into Apache to sniff traffic whenever "GET /pcap" is sent. In order to prevent abuse of PTrace, FreeBSD should set the security.bsd.unprivilegedprocdebug to 0 by default. In order to prevent process manipulation, FreeBSD should implement PaX NOEXEC. libhijack can be found at https://github.com/SoldierX/libhijack Introduction to POSIX shell (https://sircmpwn.github.io/2018/02/05/Introduction-to-POSIX-shell.html) What the heck is the POSIX shell anyway? Well, the POSIX (the Portable Operating System Interface) shell is the standard Unix shell - standard meaning it was formally defined and shipped in a published standard. This makes shell scripts written for it portable, something no other shell can lay claim to. The POSIX shell is basically a formalized version of the venerable Bourne shell, and on your system it lives at /bin/sh, unless you're one of the unlucky masses for whom this is a symlink to bash. Why use POSIX shell? The “Bourne Again shell”, aka bash, is not standardized. Its grammar, features, and behavior aren't formally written up anywhere, and only one implementation of bash exists. Without a standard, bash is defined by its implementation. POSIX shell, on the other hand, has many competing implementations on many different operating systems - all of which are compatible with each other because they conform to the standard. Any shell that utilizes features specific to Bash are not portable, which means you cannot take them with you to any other system. Many Linux-based systems do not use Bash or GNU coreutils. Outside of Linux, pretty much everyone but Hurd does not ship GNU tools, including bash1. On any of these systems, scripts using “bashisms” will not work. This is bad if your users wish to utilize your software anywhere other than GNU/Linux. If your build tooling utilizes bashisms, your software will not build on anything but GNU/Linux. If you ship runtime scripts that use bashisms, your software will not run on anything but GNU/Linux. The case for sticking to POSIX shell in shipping software is compelling, but I argue that you should stick to POSIX shell for your personal scripts, too. You might not care now, but when you feel like flirting with other Unicies you'll thank me when all of your scripts work. One place where POSIX shell does not shine is for interactive use - a place where I think bash sucks, too. Any shell you want to use for your day-to-day command line work is okay in my book. I use fish. Use whatever you like interactively, but stick to POSIX sh for your scripts. How do I use POSIX shell? At the top of your scripts, put #!/bin/sh. You don't have to worry about using env here like you might have been trained to do with bash: /bin/sh is the standardized location for the POSIX shell, and any standards-conforming system will either put it there or make your script work anyway. The next step is to avoid bashisms. There are many, but here are a few that might trip you up: [[ condition ]] does not work; use [ condition ] Arrays do not work; use IFS Local variables do not work; use a subshell The easiest way to learn about POSIX shell is to read the standard - it's not too dry and shorter than you think. Using standard coreutils The last step to writing portable scripts is to use portable tools. Your system may have GNU coreutils installed, which provides tools like grep and cut. Unfortunately, GNU has extended these tools with its own non-portable flags and tools. It's important that you avoid these. One dead giveaway of a non-portable flag is long flags, e.g. grep --file=FILE as opposed to grep -f. The POSIX standard only defines the getopt function - not the proprietary GNU getopt_long function that's used to interpret long options. As a result, no long flags are standardized. You might worry that this will make your scripts difficult to understand, but I think that on the whole it will not. Shell scripts are already pretty alien and require some knowledge to understand. Is knowledge of what the magic word grep means much different from knowledge of what grep -E means? I also like that short flags allow you to make more concise command lines. Which is better: ps --all --format=user --without-tty, or ps -aux? If you are inclined to think the former, do you also prefer function(a, b, c) { return a + b + c; } over (a, b, c) => a + b + c? Conciseness matters, and POSIX shell supports comments if necessary! Some tips for using short flags: They can be collapsed: cmd -a -b -c is equivalent to cmd -abc If they take additional arguments, either a space or no separation is acceptable: cmd -f"hello world" or cmd -f "hello world" A good reference for learning about standardized commands is, once again, the standard. From this page, search for the command you want, or navigate through “Shell & Utilities” -> “Utilities” for a list. If you have man-pages installed, you will also find POSIX man pages installed on your system with the p postfix, such as man 1p grep. Note: at the time of writing, the POSIX man pages do not use dashes if your locale is UTF-8, which makes searching for flags with / difficult. Use env LC_ALL=POSIX man 1p grep if you need to search for flags, and I'll speak to the maintainer of man-pages about this. FreeBSD Broadcom Wi-Fi Improvements (http://landonf.org/code/freebsd/Broadcom_WiFi_Improvements.20180122.html) Introduction Since 2015, I've been working on improving FreeBSD support for Broadcom Wi-Fi devices and SoCs, including authoring the bhnd(4) driver family, which provides a unified bus and driver programming interface for these devices. First committed in early 2016, bhnd(4) allowed us to quickly bring up FreeBSD/MIPS on Broadcom SoCs, but it has taken much longer to implement the full set of features required to support modern Broadcom SoftMAC Wi-Fi hardware. Thanks to the generosity of the FreeBSD Foundation, I've recently finished implementing the necessary improvements to the bhnd(4) driver family. With these changes in place, I was finally able to port the existing bwn(4) Broadcom SoftMAC Wi-Fi driver to the bhnd(4) bus, and implement initial support for the BCM43224 and BCM43225 chipsets, with additional hardware support to be forthcoming. Now that my efforts on FreeBSD/Broadcom Wi-Fi support have progressed far enough to be generally useful, I wanted to take some time to provide a brief overview of Broadcom's Wi-Fi hardware, and explain how my work provides a foundation for further FreeBSD Broadcom Wi-Fi/SoC improvements. A Brief Background on Broadcom Wi-Fi Hardware Broadcom's Wi-Fi devices are members of the Broadcom Home Networking Division (BHND) device family; other BHND devices include MIPS/ARM SoCs (including Wi-Fi SoCs commonly found in consumer access points), as well as a large variety of related networking hardware. BHND devices utilize a common set of Broadcom IP cores (or "functional blocks") connected via one of two on-chip bus architectures: Hardware designed prior to 2009 used Broadcom's “SSB” backplane architecture, based on Sonics Silicon's interconnect IP. Subsequent hardware adopted Broadcom's “BCMA” backplane, based on ARM's AMBA IP. The IP cores used in earlier SSB-based devices were adapted for compatibility with the new backplane. When BHND hardware is used in a PCI Wi-Fi card, or a SDIO Wi-Fi module, the device's dual-mode peripheral controller is configured to operate as an endpoint device on the host's peripheral bus, bridging access to the SoC hardware: Host access to SoC address space is provided via a set of register windows (e.g., a set of configurable windows into SoC address space mapped via PCI BARs) DMA is supported by the bridge core's sparse mapping of host address space into the backplane address space. These address regions may be used as a target for the on-chip DMA engines. Any backplane interrupt vectors routed to the bridge core may be mapped by the bridge to host interrupts (e.g., PCI INTx/MSI/MSI-X). The host is generally expected to provide drivers for the IP cores found on the SoC backplane; since these cores are found in both BHND SoCs and BHND Wi-Fi devices, it is advantageous to share driver and platform code between the two targets. Modernizing FreeBSD's Broadcom SoftMAC Wi-Fi Support FreeBSD support for Broadcom SoftMAC Wi-Fi adapters is provided by two partially overlapping PCI/CardBus drivers: Legacy Wi-Fi adapters are supported by bwi(4). This driver remains in-tree to support devices incompatible with v4 or later firmware (e.g. BCM4301, BCM4302, BCM4306 rev 1-2), all of which were released prior to December 2002. Modern Wi-Fi adapters are supported by bwn(4), with access to on-chip cores mediated by bhnd(4). Prior to my work porting bwn(4) to bhnd(4), access to on-chip cores was mediated by sibabwn, a PCI/WiFi-specific derivative of the legacy siba(4) SSB bus driver. There were two major limitations to sibabwn that have long blocked adding support for newer SoftMAC Wi-Fi chipsets: the newer BCMA interconnect found in post-2009 hardware was not supported by siba(4), and siba_bwn assumed a PCI/PCIe bridge, preventing its use on FreeBSD/MIPS Broadcom SoCs with interconnect-attached D11 cores. The new bhnd(4) driver family, written as a replacement for siba(4) and siba_bwn, provides: A unified bus driver interface for both SSB and BCMA on-chip interconnects A generic BHND bridge driver framework for host-connected BHND devices (e.g. Wi-Fi adapters, etc) A PCI/PCIe bridge core driver, for PCI-attached BHND devices. An abstract BHND NVRAM API, with support for the varied NVRAM formats found in BHND Wi-Fi adapters and SoCs. Drivers for common BHND platform peripherals (UARTs, SPROM/flash, PMUs, etc) By porting bwn(4) to bhnd(4), we are now able to support existing BCMA devices with MAC/PHY/Radio combinations readily supported by bwn(4), as was the case with the BCM43224 and BCM43225 chipsets. This also opens the door to porting additional PHY support from Broadcom's ISC-licensed Linux drivers, and will allow us to bring up bwn(4) on Broadcom WiSoCs supported by FreeBSD/MIPS. Monitor OpenSMTPD using Logstash and Grafana (https://www.tumfatig.net/20180129/monitor-opensmtpd-using-logstash-grafana/) Logs are usefull. Graphs are sexy. Here's a way to get a view on what happens to your OpenSMTPD traffic, using Web v2.0 tools ; namely Logstash & Grafana. For those who would not be aware of those tools, logstash is some kind of log-parser that can eat syslog formatted logs and write them into elasticsearch ; in “document” format. Grafana is a Web frontend that can dig into various databases and render graphics from requests. I won't go into the whole “how to install” process here. Installation is quite straight forward and online documentation is quite clear. What you need OpenSMTPD deals with emails and logs its activity via Syslog. Syslog is configured to send the logs to Logstash. Logstash has a set of rules configured to transform the text-oriented information into searchable document-oriented data. The transformed data is stored into Elasticsearch. Elasticsearch provides Web API to search and find stuff. Grafana connects to ELS to get data and draw the graphs. Beastie Bits CharmBUG Presentation - Writing FreeBSD Malware (https://www.meetup.com/CharmBUG/events/247995596/) March London *BSD meeting 13/03/18 (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2018-February/014180.html) FreBSD Ports Workshop (https://wiki.freebsd.org/MateuszPiotrowski/Ports/Workshop) The history of NetBSD/atari and support for ATARI compatible Milan / OSC2018Osaka (https://speakerdeck.com/tsutsui/osc2018osaka) SSH Mastery, 2nd Edition (https://www.tiltedwindmillpress.com/?product=ssh-mastery-2nd-edition) *** Feedback/Questions Stephen - Viewer Interview Question (http://dpaste.com/06WTRB9#wrap) pb - trust expanding your 280TB pool (http://dpaste.com/0TZV6CM#wrap) Tim - ZFS questions for the ZFS Man (http://dpaste.com/0759X1E#wrap) Daniel - ZFS full backup question (http://dpaste.com/1SJXSBQ#wrap) ***

Jocke fyller år och podden fyller jämnt antal avsnitt. Vi firar genom att prata Firewatch, nyare svenska rollspel och sedan ordentligt med DATA: syslogservrar, stapelbara datorer och routerbyten. Det blir även leksaker, pulvermat, shakerdesign och helt färska nyheter om Datormagazin retro! h2>Länkar Bahco 59/S13B Vår vän Kristian i Jönköping Firewatch Kult – det gamla rollspelet och det nya Chock – gammalt skräckrollspel Äventyrsspel översatte Kult – samlarkortspelet Tales from the loop – Simon Stålenhag-rollspelet Oktoberlandet Rasputin Boken om Äventyrsspel Nils Gulliksson-boken Syslog Docker Centos Att sätta upp en syslogserver på Centos Macminicolo Rambox – tack Markus Borgelin för tipset! Doublespace Discord – appen vi pratar via Trillian Litestep Clavister CCR1016–12G Queal – trevlig pulvermat med en förvånansvärt underlägsen shaker. (De bytte precis design, men Fredrik har den nya) Sony MDR–100ABN Bose quiet comfort Suido Regent Sony MDR–1000X Philips Hue-lampor Webhallen Homekit GU10-sockel för lampor Retrogathering AMD-kontra-Xeon-diskussioner, del ett och del två Hyper-threading Två nördar - en podcast. Fredrik Björeman och Joacim Melin diskuterar allt som gör livet värt att leva. Fullständig avsnittsinformation finns här: https://www.bjoremanmelin.se/podcast/avsnitt-64-ar-det-doublespace-fast-for-mac.html.

This week on BSDNow, we'll be talking to Diane Bruce about using it for Ham Radio Enthusiasts, the RPi3 and much more! That plus all the latest news from the week, This episode was brought to you by Headlines PC-BSD is now TrueOS (https://www.trueos.org/2016/09/01/pc-bsd-evolves-into-trueos/) If you've been watching this show the past few months, I've been dropping little hints about the upcoming rename of PC-BSD -> TrueOS. We've made that more official finally, and are asking folks to test out the software before a wider announcement this fall. For those wondering about the name change, it's been something discussed over the past few years at different times. With us beginning to move more aggressively with changes for 11.0 (and eventually 12-CURRENT), the time seemed right to have a fresh start, using it as a spring-board to introduce all the changes in both software, and development / release model. I'll be discussing more about this shift in a talk at MeetBSD2016 (Another reason for you to go), but here's some of the highlights. No longer tied to specific FreeBSD point-releases, TrueOS will instead follow a rolling-release model based upon FreeBSD -CURRENT. Special tooling and features (Such as boot-environments) make this a feasible option that we didn't have as easily in the early days of PC-BSD. In addition, TrueOS builds some things different from vanilla FreeBSD. Specifically Matt Macy's DRM and Linux Compat work, LibreSSL directly in base, built from External Toolchain (No clang in base system package) and much more. New tools have have replaced, and are in the process of replacing the legacy PC-BSD control panel as well, which allows remote operation, either via Qt GUI, or WebSockets / REST API's. I'll be talking about more as things unfold, but for now please feel free to test and let us have feedback while we push towards a more stable release. *** The Voicemail Scammers Never Got Past Our OpenBSD Greylisting (http://bsdly.blogspot.com/2016/08/the-voicemail-scammers-never-got-past.html) Peter Hansteen (That grumpy BSD guy) gives us an interesting look at how their OpenBSD grey-listing prevented spam from ever making it to their inbox. Specifically it looks like it occurred during Aug 23rd and 24th, with a particularly nasty ransomware payload destined to play havoc with Windows systems. Peter then walks us through their three-server mail setup, and how spamd is run in greylisting mode on each. The results? Nothing short of perfection: > “From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois). The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes. There were none. The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0. Zero. Not a single one made it through even to content filtering.” Not bad at all! Looks like spam-trap addresses + grey-listing is the way to go for stopping this kind of foolishness. Checkout Peter's blog post for more details, but perhaps this will encourage you to setup a similar-type system for your business. *** FreeBSD on a tiny system; what's missing (http://adrianchadd.blogspot.com/2016/08/freebsd-on-tiny-system-whats-missing.html) Adrian Chadd talks about some of the bits that are missing to make FreeBSD truly useful on small embedded devices Some of this stuff can be done now, but requires more work than it should “The first is a lack of real service management. FreeBSD doesn't have a service management daemon - the framework assumes that daemons implement their own background and monitoring. It would be much nicer if init or something similar to init could manage services and start/restart them where appropriate.” Of course, on a system with 32mb of memory, such a service manager would need to be very light weight “maybe I want to only start telnetd or dropbear/sshd whenever a connection comes in. But I'd also like to be able to add services for monitoring, such as dnsmasq and hostapd.” telnetd and sshd can be run from inetd, but often depend on special support from the daemon “The next is a lack of suitable syslog daemon. Yes, I'd like to be able to log some messages locally - even if it's only a couple hundred kilobytes of messages. I'd also like to be able to push messages to a remote service. Unfortunately the FreeBSD syslog daemon doesn't do log rotation or maximum log file sizes itself - it's done by "newsyslog" which runs out of cron. This isn't any good for real embedded systems with limited storage.” Syslog leaves much to be desired, especially in its configuration syntax, and filtering capabilities. Having it be able to detect with log files have grown beyond a reasonable size and fire off newsyslog would be very interesting “Then yes, there's a lack of a cron service. It'd be nice to have that integrated into the service management framework so things could be easily added/removed. I may just use cron, but that means cron is also always running which adds memory footprint (~1.3 megabytes) for something that is almost never actually active. When you have 32MB of RAM, that's quite a bit of wasted memory.” Systems have come back full circle, to where 32MB and 64MB are amounts of memory people expect to work with, while other people still want the system to perform well with 32 or 64 GB of memory It will be interesting to see how this balancing act plays out, trying to make the same codebase useful for extremely small and extremely large systems at the same time, while also running it on your middle of the road laptop. *** So I lost my OpenBSD FDE password (https://blog.filippo.io/so-i-lost-my-openbsd-fde-password/) “The other day I set up a new OpenBSD instance with a nice RAID array, encrypted with Full Disk Encryption. And promptly proceeded to forget part of the passphrase.” So they started a little project Goal: “We need to extract enough info from the encrypted disk and rebuild enough of the decryption algorithm to be able to rapidly try many passphrases.” The post walks through how they reverse engineered the encryption system from the source code and a hexdump of a small encrypted memory disk “Now that we know how to extract the data and how to try passphrases against it, it will be trivial to write a bruteforce tool to recover the part of passphrase I forgot.” So, rather than having to try every possible passphrase, they only had to try fuzzing around the known keyword that was their passphrase. “UPDATE: I found it! After fixing a bug or two in the brute force tool and almost losing hope, it found the right combination of forgotten word and (Italian) misspelling.” This work lead to the author recommending that OpenBSD consider strengthening the key derivation algorithm (http://marc.info/?l=openbsd-tech&m=147316661717410&w=2) used in its FDE. Rather than using a fixed number of rounds (8000 currently), do a small benchmark and determine how much work can be done in a reasonable amount of time This is what FreeBSD's GELI FDE does, targeting ‘over 2 million microseconds' of work. On my desktop i5-3570 this results in 974842 rounds. The number will likely not be the same twice because of minor variations in how long it will take in microseconds. *** Interview - Diane Bruce - db@freebsd.org (mailto:db@freebsd.org) / @Dianora_1 (https://twitter.com/Dianora_1) Ham Radio, RPi3 and more! News Roundup See Me (Michael W. Lucas) in 2016 (http://blather.michaelwlucas.com/archives/2739) Looking for a chance to interact with author Michael W Lucas in meat-space? (That sounds wrong) If so, he has posted a list of the up-coming conferences he'll be speaking at, starting with Ohio LinuxFest Oct 7-8, where he'll be giving an introduction to ZFS talk. Nov 8th, he'll also be at MUG (Michigan User Group) giving a PAM talk. Sadly, no MeetBSD for Michael this year [moment of silence], but if you are able to make it to one of the aforementioned gatherings, be sure to bring your books for autographs. We promise he doesn't bite. Much. *** It's hard work printing nothing (http://www.tedunangst.com/flak/post/its-hard-work-printing-nothing) “It all starts with a bug report to LibreSSL that the openssl tool crashes when it tries to print NULL. This bug doesn't manifest on OpenBSD because libc will convert NULL strings to ”(null)” when printing. However, this behavior is not required, and as observed, it's not universal. When snprintf silently accepts NULL, that simply leads to propagating the error.” “There's an argument to be made that silly error messages are better than crashing browsers, but stacking layers of sand seems like a poor means of building robust software in the long term.” “As soon as development for the next release of OpenBSD restarted, some developers began testing a patch that would remove this crutch from printf.” If you'd like to help with this work, see our call for volunteers from 2 weeks ago: opportunity to help: %s audit in mandoc (https://marc.info/?l=openbsd-misc&m=147059272201219&w=2) Of course, immediately things started to complain. The configure script for talloc does a number of checks (check out the additional interesting observations by TedU here) “The test checking that our snprintf function conforms to the C99 standard actually contains, at a minimum, 3 deviations from the standard. It should say “Checking for non-conformant vsnprintf”.” “Of course, we're dealing with NULL pointers, so all bets are off, but I wonder what people who expect printf NULL to work expect out of strlen? Does it return 0? Does it crash?” So, talloc decides that the system printf is no good, and it should use its own bundled implementation “After all the configure testing, eventually the build will fail, because somebody forgot to actually add the replacement object file to the Makefile.” “If the replacement function has never been used, that's hardly reassuring that it is actually better tested than the version we have in libc.” *** Revisiting W^X with OpenBSD 6.0 (http://blog.acumensecurity.net/revisiting-wx-with-openbsd-6-0/) OpenBSD 6.0 includes enforcement of W^X in user-land This prevents an application from being able to map a page of memory with both Write and Execute permissions (protecting mmap(2)) Once mapped a page of memory should not be able to have permissions escalated (protecting mprotect(2)) OpenBSD 6.0 enforces the strict W^X definition, and not the PaX/grsec “once write never execute” type of policy *** OpenBSD imports a letsencrypt client into the base system (http://undeadly.org/cgi?action=article&sid=20160901060733) We've mentioned letskencrypt before (A native C version of the letsencrypt client, developed by Kristaps). Looks like it's undergoing a name-change to “acme-client” and has made it's way into OpenBSD's base system! This should ensure first-class support for management of Let's Encrypt certificates, here's hoping the portable version continues to thrive as well. Congrats to Kristaps! *** Beastie Bits OpenBSD: Release Songs 6.0: "Goodbye" -- no more CD releases (https://www.openbsd.org/lyrics.html#60f) FreeBSD 101 Hacks (https://nanxiao.gitbooks.io/freebsd-101-hacks/content/) LibreSSL enabled by default in HardenedBSD (https://hardenedbsd.org/article/shawn-webb/2016-08-20/libressl-enabled-default) DragonflyBSD removes last bits of 32-bit Linux emulation and has no plans to implement 64-bit linux emulation (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624241.html) OpenBSD has sent 32bit sparc to the great bitbucket in the sky (https://twitter.com/phessler/status/771277693090467840) Front Range BSD User Group September Meeting (http://slexy.org/view/s2hm4HBkb2) KnoxBug TrueOS Wrap-up (http://knoxbug.org/content/going-with-the-flow) Feedback/Questions Cody - TrueOS Questions (http://pastebin.com/mVK8G1Vr) John - FreeNAS Backups (http://pastebin.com/xsUNUfCS) Herminio - PowerPC + OpenBSD (http://pastebin.com/nHkWuNkm) Dennis - pmake vs bmake (http://pastebin.com/NAh7r6Ed) Al - Upgrade conflicts (http://pastebin.com/8HaK7yJ6) ***

News:DesktopBSD 1.6 and FreeBSD 6.3 released.Setting up a central syslog server.If you are concerned about the security of your logs, use a dedicated machine and lock it down.Keep clocks in sync.You may need to change log rotation schedule in /etc/newsyslog.conf. You can rotate based in size and/or time. This can be as much a policy decision as a hardware decision.On central log host, change syslogd flags to listen to network. Each BSD does this differently, so check the man pages. Also, check out the -n flag for busy environments.Make sure host firewall allows syslog traffic through.Be careful to limit syslog traffic to just the trusted network or hosts. FreeBSD man page refers to syslogd as a "remote disk filling service".For heavy logging environments, it is important to have a dedicated network. A down syslogd server can create a lot of "ARP who-has" broadcasts.Most network devices such as printers and commercial firewalls support sending to a central syslog server. Take a look at "Snare" for Windows hosts.To send messages from a Unix host, specify the host name prepended with @ instead of a file for logging in /etc/syslog.conf. For example, change /var/log/xferlog to @loghost.mydomain.biz. You can also copy and edit the line to have it log to both a local file and a remote host.File Info: 7Min, 3MBOgg Link:https://archive.org/download/bsdtalk138/bsdtalk138.ogg

This episode focuses on some of Cisco's Virtual Security Appliances, the ASA1000v, the Virtual Security Gateway (VSG) and the Virtual Network Management Center (VNMC). Rama Darbha and Michael Robertson discuss how administrators can use these products in their virtual environments, as well as the packet forwarding path and troubleshooting techniques for these products.

The panel discusses best practices for configuring devices to generate syslogs, and how the TAC investigates syslogs provided by customers. Tips and tricks for parsing through large syslog files, as well as techniques and tools for finding useful information are discussed.

Syslog is the primary source of information about intrusion-related activity on a Unix system. Searching for known messages and patterns in syslog data is easy to do, and many tools are available for doing so. However, information and patterns that are not already "known" -- those that have not been seen or derived already, may provide even more information about attacks and intrusions. Data mining techniques can help us discover and analyze that information, but, the general lack of structure in syslog data makes it impossible to apply these techniques directly to the data. To address the problem, we are researching methods of generating patterns from an archive of system logs which can uniquely identify syslog messages by the variant and invariant elements of the messages. Once syslog messages can be uniquely identified, data mining techniques for use in intrusion detection or forensic analysis will be far more useful. About the speaker: Abe Singer is a Computer Security Researcher with the Security Technologies Group at the San Diego Supercomputer Center. Involved with both operational security and research, his work involves growing SDSC logging infrastructure and analysis capabilities, participating in incident response and investigation, and working with the Teragrid Security Working Group. Mr. Singer's current research is in analysis of syslog data and data mining of logs for security. In addition to his work at SDSC, Mr. Singer is an occasional consultant and expert witness, and runs the San Diego Regional Information Watch (www.sdriw.org).

Syslog is the primary source of information about intrusion-related activity on a Unix system. Searching for known messages and patterns in syslog data is easy to do, and many tools are available for doing so. However, information and patterns that are not already "known" -- those that have not been seen or derived already, may provide even more information about attacks and intrusions. Data mining techniques can help us discover and analyze that information, but, the general lack of structure in syslog data makes it impossible to apply these techniques directly to the data. To address the problem, we are researching methods of generating patterns from an archive of system logs which can uniquely identify syslog messages by the variant and invariant elements of the messages. Once syslog messages can be uniquely identified, data mining techniques for use in intrusion detection or forensic analysis will be far more useful.