Podcasts about sunos

We reveal how we turned our humble LAN into a public server farm, all while keeping our IP address under wraps and our ISP blissfully unaware.Sponsored By:Core Contributor Membership: Take $1 a month of your membership for a lifetime!Tailscale: Tailscale is a programmable networking software that is private and secure by default - get it free on up to 100 devices! 1Password Extended Access Management: 1Password Extended Access Management is a device trust solution for companies with Okta, and they ensure that if a device isn't trusted and secure, it can't log into your cloud apps. Support LINUX UnpluggedLinks:

An airhacks.fm conversation with Jose Paumard (@JosePaumard) about: TI-57 was stateless Oric 1, BigDecimal use cases, the travelling salesman algorithm, the Cray, working with Sun SPARC machines, CM5 and NeXTcube, the conference in generate code, star recognition, working at research Lab in Paris, enjoying emacs, emacs vs. vim, writing documentation in LatEx working on SunOS then Solaris, HPUX and CDE, 512 MB RAM of the price of a flat in Paris, processing large images and recognising building in real time, wavelet and cosine transforms, starting as professor in 1994 , JDBC war leased in 1997 with Java 1.1., working as devrel at Oracle three years again, running AI models, project Panama is the bridge, Java innovation, pattern matching in Java, String Templates, Java 21 LTS, youbube.com/java Jose Paumard on twitter: @JosePaumard

Bryan and Adam and the Oxide Friends take on GPT and its implications for software engineering. Many aspiring programmers are concerned that the future of the profession is in jeopardy. Spoiler: the Oxide Friends see a bright future for human/GPT collaboration in software engineering.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 20th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers on MM DD included Josh Clulow, Keith Adams, Ashley Williams, and others. (Did we miss your name and/or get it wrong? Drop a PR!)Live chat from the show (lightly edited): ahl: John Carmack's tweet ahl: ...and the discussion Wizord: https://twitter.com/balajis/status/1636797265317867520 (the $1M bet on BTC, I take) dataphract: "prompt engineering" as in "social engineering" rather than "civil engineering" Grevian: I was surprised at how challenging getting good prompts could be, even if I wouldn't quite label it engineering TronDD: https://www.aiweirdness.com/search-or-fabrication/ MattCampbell: I tested ChatGPT in an area where I have domain expertise, and it got it very wrong. TronDD: Also interesting https://www.youtube.com/watch?v=jPhJbKBuNnA Wizord: the question is, when will it be in competition with people? Wizord: copilot also can review code and find bugs if you ask it in a right way ag_dubs: i suspect that a new job will be building tools that help make training sets better and i strongly suspect that will be a programming job. ai will need tools and data and content and there's just a whole bunch of jobs to build tools for AI instead of people Wizord: re "reading manual and writing DTrace scripts" I think it's possible, if done with a large enough token window. Wizord: (there are already examples of GPT debugging code, although trivial ones) flaviusb: The chat here is really interesting to me, as it seems to miss the point of the thing. ChatGPT does not and can not ever 'actually work' - and whether it works is kind of irrelevant. Like, the Jaquard Looms and Numerical Control for machining did not 'work', but that didn't stop the roll out. Columbus: Maybe it has read the dtrace manual

Xe Iaso is the Archmage of Infrastructure at Tailscale and previously worked at Heroku.This episode originally aired on Software Engineering Radio but includes some additional discussion about their blog near the end of the episode.Topics covered: Use cases for VPNs Simplifying service authentication by identifying users via IP Peer-to-peer vs centralized "Virtual Pain Networks" Tailscale's tech stack and why they forked the go compiler DERP relay servers Struggling with the iOS network extension size limit The surprisingly small amount of infrastructure required to run a VPN Running your company on your own product Working at Heroku vs Tailscale Using the socratic style of debate in technical blog posts Related Links @theprincessxena Xe's Blog ACL samples Go links origin story How Tailscale works Tailscale SSH How Tailscale assigns IP addresses Hey linker, can you spare a meg? My Blog is Hilariously Overengineered to the Point People Think it's a Static Site The Sheer Terror of PAM Transcript[00:00:00] Jeremy: Today I'm talking to Xe Iaso, they're the archmage of infrastructure at tailscale, and they also have a great blog everyone should check out. Xe, welcome to software engineering radio.[00:00:12] Xe: Thanks. It's great to be here. [00:00:14] Jeremy: I think the first thing we should start with, is what's a, a VPN, because I think some people they may have used it to remote into their workplace or something like that. But I think the, the scope of what it's good for and what it does is a lot broader than that. So maybe you could talk a little bit about that first.[00:00:31] Xe: Okay. a VPN is short for virtual private network. It's basically a fake network that's overlaid on top of existing networks. And then you can use that network to do whatever you would with a normal computer network. this term has been co-opted by companies that are attempting to get into the, like hide my ass style market, where, you know, you encrypt your internet information and keep it safe from hackers.But, uh, so it makes it really annoying and hard to talk about what a VPN actually is. Because tailscale, uh, the company I work for is closer to like the actual intent of a VPN and not just, you know, like hide your internet traffic. That's already encrypted anyway with another level of encryption and just make a great access point for, uh, three letter agencies.But are there, use cases, past that, like when you're developing a piece of software, why would you decide to use a VPN outside of just because I want my, you know, my workers to be able to get access to this stuff.[00:01:42] Xe: So something that's come up, uh, when I've been working at tailscale is that sometimes we'll make changes to something. And it'll be changes to like the user experience of something on the admin panel or something. So in a lot of other places I've worked in order to have other people test that, you know, you'd have to push it to the cloud.It would have to spin up a review app in Heroku or some terrifying terraform of abomination would have to put it out onto like an actual cluster or something. But with tail scale, you know, if your app is running locally, you just give like the name of your computer and the port number. And you know, other people are able to just see it and poke it and experience it.And that basically turns the, uh, feedback cycle from, you know, like having to wait for like the state of the world to converge, to, you know, make a change, press F five, give the URL to a coworker and be like, Hey, is this Gucci?they can connect to your app as if you were both connected to the same switch.[00:02:52] Jeremy: You don't have to worry about, pushing to a cloud service or opening ports, things like that.[00:02:57] Xe: Yep. It will act like it's in the same room, even when they're not it'll even work. if you're at both at Starbucks and the Starbucks has reasonable policies, like holy crap, don't allow devices to connect to each other directly. so you know, you're working on. Like your screenplay app at your Starbucks or something, and you have a coworker there and you're like, Hey, uh, check this out and, uh, give them the link.And then, you know, they're also seeing the screenplay editor.[00:03:27] Jeremy: in terms of security and things like that. I mean, I'm picturing it kind of like we were sitting in the same room and there's a switch and we both plugged in. Normally when you do something like that, you kind of have, full access to whatever else is on the switch. Uh, you know, provided that's not being blocked by a, a firewall.is there like a layer of security on top of that, that a VPN service like tailscale would provide.[00:03:53] Xe: Yes. Um, there are these things called access control lists, which are kind of like firewall rules, except you don't have to deal with like the nightmare of writing an IP tables rule that also works in windows firewall and whatever they use in Mac OS. The ACL rules are applied at the tailnet level for every device in the tailnet.So if you have like developer machines, you can put people into groups as things like developers and say that developer machines can talk to production, but not people in QA. They can only talk to testing and people on SRE have, you know, permissions to go everywhere and people within their own teams can connect to each other. you can make more complicated policies like that fairly easily.[00:04:44] Jeremy: And when we think about infrastructure for, for companies, you were talking about how there could be development, infrastructure, production, infrastructure, and you kind of separate it all out. when you're working with cloud infrastructure. A lot of times, there's the, I always forget what it stands for, but there's like IAM.There's like policies that you can set up with the cloud provider that says these users can access this, or these machines can access this. And, and I wonder from your perspective, when you would choose to use that versus use something at the, the network or the, the VPN level.[00:05:20] Xe: The way I think about it is that things like IAM enforce, permissions for like more granularly scoped things like can create EC2 instances or can delete EC2 instances or something like that. And that's just kind of a different level of thing. uh, tailscale, ACLs are more, you know, X is allowed to connect to Y or with tailscale, SSH X is allowed to connect as user Y.and that's really different than like arbitrary capability things like IAM offers.you could think about it as an IAM system, but the main permissions that it's exposing are can X connect to Y on Zed port.[00:06:05] Jeremy: What are some other use cases where if you weren't using a VPN, you'd have to do a lot more work or there's a lot more complexity, kind of what are some cases where it's like, okay, using a VPN here makes a lot of sense.(The quick and simple guide to go links https://www.trot.to/go-links) [00:06:18] Xe: There is a service internal to tailscale called go, which is a, clone of Google's so-called go links where it's basically a URL shortener that lives at http://go. And, you know, you have go/something to get to some internal admin service or another thing to get to like, you know, the company directory and notion or something, and this kind of thing you could do with a normal setup, you know, you could set it up and have to do OAuth challenges everywhere and, you know, have to put and make sure that everyone has the right DNS configuration so that, it shows up in the right place.And then you have to deal with HTTPS um, because OAuth requires HTTPS for understandable and kind of important reasons. And it's just a mess. Like there's so many layers of stuff like the, the barrier to get, you know, like just a darn URL, shortener up turns from 20 minutes into three days of effort trying to, you know, understand how these various arcane things work together.You need to have state for your OAuth implementation. You need to worry about what the hell a a JWT is (sigh) . It's it it's just bad. And I really think that something like tailscale with everybody has an IP address. In order to get into the network, you have to sign in with your, auth provider, your, a provider tells tailscale who you are.So transitively every IP address is tied to an owner, which means that you can enforce access permission based on the IP address and the metadata about it that you grab from the tailscale. daemon, it's just so much simpler. Like you don't have to think about, oh, how do I set up OAuth this time? What the hell is an oauth proxy?Um, what is a Kubernetes? That sort of thing you just think about like doing the thing and you just do it. And then everything else gets taken care of it. It's like kind of the ultimate network infrastructure, because it's both omnipresent and something you don't have to think about. And I think that's really the power of tailscale.[00:08:39] Jeremy: typically when you would spin up a, a service that you want your developers or your system admins, to be able to log into, you would have to have some way of authenticating and authorizing that user. And so you were talking about bringing in OAuth and having your, your service understand that.But I, I guess what you're saying is that when you have something like tailscale, that's kind of front loaded, I guess you, you authenticate with tail scale, you get onto the network, you get your IP. And then from that point on you can access all these different services that know like, Hey, because you're on the network, we know you're authenticated and those services can just maybe map that IP that's not gonna change to like users in some kind of table. Um, and not have to worry about figuring out how do I authenticate this user.[00:09:34] Xe: I would personally more suggest that you use the, uh, whois, uh, look up route in the tailscale daemon's local API, but basically, yeah, you don't really have to worry too much about like the authentication layer because the authentication layer has already been done. You know, you've already done your two factor with Gmail or whatever, and then you can just transitively push that property onto your other machines.[00:10:01] Jeremy: So when you talk about this, this whois daemon, can you give an example of I'm in the network now I'm gonna make a service call to an application. what, what am I doing with this? This whois daemon?[00:10:14] Xe: It's more of like a internal API call that we expose via tailscaled's, uh, Unix, socket. but basically you give it an IP address and a port, and it tells you who the person is. It's kind of like the Unix ident protocol in a way, except completely not. And at a high level, you know, if you have something like a proxy for Grafana, you have that proxy for Grafana, make a call to the local tailscale daemon, and be like, Hey, who was this person?And the tailscale, daemon will spit back at JSON object. Like, oh, it's this person on this device and there you can do additional logic like maybe you shouldn't be allowed to delete things from an iOS device, you know, crazy ideas like that. there's not really support for like arbitrary capabilities and tailscaled at the time of recording, but we've had some thoughts would be cool.[00:11:17] Jeremy: would that also include things like having roles, for example, even if it's just strings, um, that you get back so that your application would know, okay. This person, is supposed to have admin access to this service based on what I got back from, this, this service.[00:11:35] Xe: Not currently, uh, you can probably do it via convention or something, but what's currently implemented in the actual, like, source code and user experience that they, you can't do that right now. Um, it is something that I've been, trying to think about different ways to solve, but it's also a problem.That's a bit big for me personally, to tackle.[00:11:59] Jeremy: there's, there's so many, I guess, different ways of doing it. That it's kind of interesting to think of a solution that's kind of built into the, the network. Yeah.[00:12:10] Xe: Yeah. and when I describe that authentication thing to some people, it makes them recoil in shock because there's kind of a Stockholm syndrome type effect with security, for a lot of things where, the easy way to do something and the secure way to do something are, you know, like completely opposite and directly conflicting with each other in almost every way.And over time, people have come to associate security or like corporate VPNs as annoying, complicated, and difficult. And the idea of something that isn't annoying, complicated or difficult will make people reject it, like just on principle, because you know, they've been trained that, you know, VPN equals virtual pain network and it, it's hard to get that association outta people's heads because you know, a lot of VPNs are virtual pain networks.Like. I used to work for Salesforce and Salesforce had this corporate VPN where no matter what you did, all of your traffic would go out to the internet from their data center. I think it was in San Francisco or something. And I was in the Seattle area. So whenever I had the VPN on my latency to Google shot up by like eight times and being a software person, you know, I use Google the same way that others breathe and it, it was just not fun.And I only had the VPN on for the bare minimum of when I needed it. And, oh God, it was so bad.[00:13:50] Jeremy: like some people, when they picture a VPN, they picture exactly what you're describing, where all of my traffic is gonna get routed to some central point. It's gonna go connect to the thing for me and then send the result back. so maybe you could talk a little bit about why that's, that's maybe a wrong assumption, I guess, in the case of tailscale, or maybe in the case of just more modern VPN solutions.[00:14:13] Xe: Yeah. So the thing that I was describing is what I've been lovingly calling the, uh, single point of failure as a service type model of VPN, where, you know, you have like the big server somewhere, it concentrates all the connections and, you know, like does things to make the computer feel like they've teleported over there, but overall it's a single point of failure.And if that falls over, you know, like goodbye, VPN. everybody's just totally screwed. And in contrast, tailscale does a more peer-to-peer thing so that everyone is basically on equal footing. Everyone can send traffic directly to each other, and if it can't get directly to there, it'll use a network of, uh, relay servers, uh, lovingly called Derp and you don't have to worry about, your single point of failure in your cluster, because there's just no single point of failure.Everything will directly communicate as much as possible. And if it can't, it'll still communicate anyway.[00:15:18] Jeremy: let's say I start up my computer and I wanna connect to a server in a data center somewhere at the very beginning, am I connecting to some server hosted at tailscale? And then. There's some kind of negotiation process where after that I connect directly or do I just connect directly straight away?[00:15:39] Xe: If you just turn on your laptop and log in, you know, to it signs into tailscale and gets you on the tailnet and whatnot, then it will actually start all connections via Derp just so that it can negotiate the, uh, direct connection. And in case it can't, you know, it's already connected via Derp so it just continues the connection with Derp and this creates a kind of seamless magic type experience where doing things over Derp is slower.Yes, it is measurably slower because you know, like you're not going directly, you're doing TCP inside of TCP. And you know, that comes with a average minefield of lasers or whatever you call it. And it does work though. It's not ideal if you wanna do things like copy large amounts of data, but if you want just want ssh into prod and see the logs for what the heck is going on and why you're getting paged at 3:00 AM. it's pretty great.[00:16:40] Jeremy: What you, you were calling Derp is it where you have servers kind of all over the world and somehow it determines which one's, I guess, is it which one's closest to your destination or which one's closest to you. I'm kind of[00:16:54] Xe: It's really interesting. It's one of the most weird distributed systems, uh, type things that I've ever seen. It's the kind of thing that could only come outta the mind of an X Googler, but basically every tailscale, every tailscale node has a connection to all of the Derp servers and through process of, you know, latency testing.It figures out which connection is the fastest and the lowest latency. And it calls that it's home Derp but because it's connected to everything is connected to every Derp you can have two people with different home Derps getting their packets relayed too other clients from different Derps.So, you know, if you have a laptop in Ottawa and a laptop in San Francisco, the laptop in San Francisco will probably use the, uh, Derp that's closest to it. But the laptop in Ottawa will also use the Derp that's closest to it. So you get this sort of like asynchronous thing, and it actually works out a lot better in practice, than you're probably imagining.[00:17:52] Jeremy: And then these servers, what was the, the technical term for them? Are they like relays or what's[00:17:58] Xe: They're relays. Uh, they only really deal with encrypted wire guard packets, and there's, no way for us at tailscale, to see the contents of Derp messages, it is literally just a forwarder. It, it literally just forwards things based on the key ID.[00:18:17] Jeremy: I guess if tail scale isn't able to decrypt the traffic, is, is that because the, the keys are only on the user's devices, like it's on their laptop and on the server they're trying to reach, or[00:18:31] Xe: Yeah. The private keys are live and die with those devices or the devices they were minted on. And the public keys are given to the coordination server and the coordination server spreads those around to every device in your tailnet. It does some limiting so that like, if you don't have ACL access to something, you don't get the private key, you don't get the, uh, public key for it.The public key, not the private key, the public key, not the private key. And yeah. Then, you know, you just go that way and it'll just figure it out. It's pretty nice.[00:19:03] Jeremy: When we're kind of talking about situations where it can't connect directly, that's where you would use the relay. what are kind of the typical cases where that happens, where you, you aren't able to just connect directly?[00:19:17] Xe: Hotel, wifi and paranoid network security setups, hotel wifi is the most notorious one because you know, you have like an overpriced wifi connection. And if you bring, like, I don't know like, You you're recording a bunch of footage on your iPhone. And because in, 2022. The iPhone has the USB2 connection on it.And you know, you wanna copy that. You wanna use the network, but you can't. So you could just let it upload through iCloud or something, or, you know, do the bare minimum. You need to get the, to get the data off with Derp it wouldn't be ideal, but it would work. And ironically enough, that entire complexity involved with, you know, doing TCP inside of TCP to copy a video file over to your laptop might actually be faster than USB2, which is something that I did the math for a while ago.And I just started laughing.[00:20:21] Jeremy: Yeah, that that is pretty, pretty ridiculous [00:20:23] Xe: welcome to the future, man (laughs) .[00:20:27] Jeremy: in terms of connecting directly, usually when you have a computer on the internet, you don't have all your ports open, you don't necessarily allow, just anybody to send you traffic over UDP and so forth. let's say I wanna send, UDP data to a, a server on my network, but, you know, maybe it has some TCP ports open. I I'm assuming once I connect into the network via the VPN, I'm able to use other protocols and ports that weren't necessarily exposed. Is that correct?[00:21:01] Xe: Yeah, you can use UDP. you can do basically anything you would do on a normal network except multicast um, because multicast is weird.I mean, there's thoughts on how to handle multicast, but the main problem is that like wireguard, which is what is tail tailscale is built on top of, is, so called OSI model layer three network, where it's at like, you know, the IP address level and multicast is a layer two or data link layer type thing.And, those are different numbers and, you can't really easily put, you know, like broadcast packets into IP, uh, IPV4 thinks otherwise, but, uh, in practice, no people don't actually use the broadcast address.[00:21:48] Jeremy: so for someone who's, they, they have a project or their company wants to get started. I mean, what does onboarding look like? What, what do they have to do to get all these devices talking to one another?[00:22:02] Xe: basically you, install tail scale, you log in with a little GUI thing or on a Linux server, you run tailscale up, and then you all log to the, to a, like a G suite account with the same domain name. So, you know, if your domain is like example.com, then everybody logs in with their example.com G suite account.And, there is no step three, everything is allowed and everything can just connect and you can change the permissions from there. By default, the ACLs are set to a, you know, very permissive allow everyone to talk to everyone on any port. Uh, just so that people can verify that it's working, you know, you can ping to your heart's content.You can play Minecraft with others. You can, you know, host an HTTP server. You can SSH into your development box and and write blog post with emacs, whatever you want.[00:22:58] Jeremy: okay, you install the, the software on your servers, your workstations, your laptops, and so on. And then at, after that there's some kind of webpage or dashboard you would go in and say, I want these people to be able to access these things and [00:23:14] Xe: Mm-hmm [00:23:15] Jeremy: these ports and so on.[00:23:17] Xe: you, uh, can customize the access control rules with something that looks like JSON, but with trailing commas and comments allowed, and you can go from there to customize basically anything to your heart's content. you can set rules so that people on the DevOps team can access everything, but you know, maybe marketing doesn't need access to the production database.So you don't have to worry about that as much.[00:23:45] Jeremy: there's, there's kind of different options for VPNs. CloudFlare access, zero tier, there's, there's some kind of, I think it's Nebula from slack or something like that. so I was kind of curious from your perspective, what's the, difference between those kinds of services and, and tailscale.[00:24:04] Xe: I'm gonna lead this out by saying that I don't totally understand the differences between a lot of them, because I've only really worked with tailscale. I know things about the other options, but, uh, I have the most experience with tailscale but from what I've been able to tell, there are things that tailscale offers that others don't like reverse mapping of IP addresses to people, or, there's this other feature that we've been working on, where you can embed tail scale as a library inside your go application, and then write a internal admin service that isn't exposed to the internet, but it's only exposed over tailscale.And I haven't seen a way to do those things with those others, but again, I haven't done much research. Um, I understand that zero tier has some layer, two capabilities, but I've, I don't have enough time in the day to look into.[00:25:01] Jeremy: There's been different, I guess you would call them VPN protocols. I mean, there's people have probably worked with IP sec in some situations they may have heard of OpenVPN, wireguard. in the case of tailscale, I believe you chose to build it on top of wireguard.So I wonder if you could talk a little bit about why, you chose wireguard and, and maybe what makes it unique.[00:25:27] Xe: I wasn't on the team that initially wrote like the core of tailscale itself. But from what I understand, wire guard was chosen because, what overhead, uh, it's literally, you just encrypt the packets, you send it to the other server, the other server decrypts them. And you know, you're done. it's also based purely on the public key. Um, the key pairs involved. And from what I understand, like at the wireguard protocol level, there's no reason why you, why you would need an IP address at all in theory, but in practice, you kind of need an IP address because you know, everything sucks. But also wire guard is like UDP only, which I think it at it's like core implementation, which is a step up from like AnyConnect and OpenVPN where they have TCP modes.So you can experience the, uh, glorious, trash fire of TCP in TCP. And from what I understand with wireguard, you don't need to set up a certificate authority or figure out how the heck to revoke certificates. Uh, you just have key pairs and if a node needs to be removed, you delete the key pair and you're done.And I think that really matches up with a lot of the philosophy behind how tailscale networks work a lot better. You know, you have a list of keys and if the network changes the list of keys changes, that's, that's the end of the story.So maybe one of the big selling points was just What has the least amount of things I guess, to deal with, or what's the, the simplest, when you're using a component that you want to put into your own product, you kind of want the least amount of things that could go wrong, I guess.[00:27:14] Xe: Yeah. It's more like simple, but not like limiting. Like, for example, a set of tinker toys is simple in that, you know, you can build things that you don't have to worry too much about the material science, but a set of tinker toys is also limiting because you know, like they're little wooden, dowels and little circles made out of wind that you stick the dowels into, you know, you can only do so much with it.And I think that in comparison, wireguard is simple. You know, there's just key pairs. They're just encryption. And it's simple in it's like overall theory and it's implementation, but it's not limiting. Like you can do pretty much anything you want with it.inherently whenever we build something, that's what we want, but that's a, that's an interesting way of putting it. Yeah.[00:28:05] Xe: Yeah. It. It can be kind of annoyingly hard to figure out how to make things as simple as they need to be, but still allow for complexity to occur. So you don't have to like set up a keyboard macro to write if error not equals nil over and over.[00:28:21] Jeremy: I guess the next thing I'd like to talk a little bit about is. We we've covered it a little bit, but at a high level, I understand that that tailscale uses wireguard, which is the open source, VPN protocol, I guess you could call it. And then there's the client software. You're saying you need to install on each of the servers and workstations.But there's also a, a control plane. and I wonder if you could kind of talk a little bit about I guess at a high level, what are all the different components of, of tailscale?[00:28:54] Xe: There's the agent that you install in your devices. The agent is basically the same between all the devices. It's all written in go, and it turns out that go can actually cross compile fairly well. So you have. Your, you know, your implementation in go, that is basically the, the same code, more or less running on windows, MacOS, freeBSD, Android, ChromeOS, iOS, Linux.I think I just listed all the platforms. I'm not sure, but you have that. And then there's the sort of control plane on tailscale's side, the control plane is basically like control, uh, which is, uh, I think a get smart reference. and that is basically a key dropbox. So, you know, you You authenticate through there. That's where the admin panel's hosted. And that's what tells the different tailscale nodes uh, the keys of all the other machines on the tailnet. And also on tailscale side there's, uh, Derp which is a fleet of a bunch of different VPSs in various clouds, all over the world, both to try to minimize cost and to, uh, have resiliency because if both digital ocean and Vultr go down globally, we probably have bigger problems.[00:30:15] Jeremy: I believe you mentioned that the, the clients were written in go, are the control plane and the relay, the Derp portion. Are those also written in go or are they[00:30:27] Xe: They're all written and go, yeah,go as much as possible. Yeah.It's kind of what happens when you have some ex go team members is the core people involved in tail scale, like. There's a go compiler fork that has some additional patches that go upstream either can't accept, uh, won't accept or hasn't yet accepted, for a while. It was how we did things like trying to shave off by bites from binary size to attempt to fit it into the iOS network extension limit.Because for some reason they only allowed you to have 15 megabytes of Ram for both like your application and working Ram. And it turns out that 15 megabytes of Ram is way more than enough to do something like OpenVPN. But you know, when you have a peer-to-peer VPN engine, it doesn't really work that well.So, you know, that's a lot of interesting engineering challenge.[00:31:28] Jeremy: That was specifically for iOS. So to run it on an iPhone.[00:31:32] Xe: Yeah. Um, and amazingly after the person who did all of the optimization to the linker, trying to get the binary size down as much as possible, like replacing Unicode packages was something that's more coefficient, you know, like basically all but compressing parts of the binary to try to save space. Then the iOS, I think 15 beta dropped and we found out that they increased the network extension Ram limit to 50 megabytes and the look of defeat on that poor person's face. I feel very bad for him.[00:32:09] Jeremy: you got what you wanted, but you're sad about it,[00:32:12] Xe: Yeah.[00:32:14] Jeremy: so that's interesting too. you were using a fork of the go compiler [00:32:19] Xe: Basically everything that is built is built using, uh, the tailscale fork, of the go compiler.[00:32:27] Jeremy: Going forward is the sort of assumption is that's what you'll do, or is it you're, you're hoping you can get this stuff upstreamed and then eventually move off of it.[00:32:36] Xe: I'm pretty sure that, I, I don't know if I can really make a forward looking statement like that, but, I've come to accept the fact that there's a fork of the go compiler. And as a result, it allows a lot more experimentation and a bit more of control, a bit more control over what's going on. like I'm, I'm not like the most happy with it, but I've, I understand why it exists and I'm, I've made my peace with it.[00:33:07] Jeremy: And I suppose it, it helps somewhat that the people who are working on it actually originally worked on the, go compiler at Google. Is that right?[00:33:16] Xe: Oh yeah. If, uh, there weren't ex go team people working on that, then I would definitely feel way less comfortable about it. But I trust that the people that are working on it, know what they're doing at least enough.[00:33:30] Jeremy: I, I feel like, that's, that's kind of the position we put ourselves in with software in general, right? Is like, do we trust our ourselves enough to do this thing we're doing?[00:33:39] Xe: Yeah. And trust is a bitch.[00:33:44] Jeremy: um, I think one of the things that's interesting about tail scale is that it's a product that's kind of it's like network infrastructure, right? It's to connect you to your other devices. And that's a little different than somebody running a software as a service. And so. how do you test something that's like built to support a network and, and how is that different than just making a web app or something like that.[00:34:11] Xe: Um, well, it's a lot more complicated for one, especially when you have to have multiple devices in the mix with multiple different operating systems. And I was working on some integration tests, doing stuff for a while, and it was really complicated. You have to spin up virtual machines, you know, you have to like make sure the virtual machines are attempting to download the version of the tailscale client you wanna test and. It's it's quite a lot in practice.[00:34:42] Jeremy: I mean, do you have a, a lab, you know, with Android phones and iPhones and laptops and all this sort of stuff, and you have some kind of automated test suite to see like, Hey, if these machines are in Ottawa and, my servers in San Francisco, like you're mentioning before that I can get from my iPhone to this server and the data center over here, that kind of thing.[00:35:06] Xe: What's the right way to phrase this without making things look bad. Um, it's a work in progress. It it's, it's really a hard problem to solve, uh, especially when the company is fully remote and, uh, like. Address that's listed on the business records is literally one of the founders condos because you know, the company has no office.So that makes the logistics for a lot of this. Even more fun.[00:35:37] Jeremy: Probably any company that's in an early stage feels the same way where it's like, everything's a work in progress and we're just gonna, we're gonna keep going and we're gonna get there. And as long as everything keeps running, we're good.[00:35:50] Xe: Yeah. I, I don't like thinking about it in that way, because it kind of sounds like pessimistic or defeatist, but at some level it's, it, it really is a work in progress because it's, it's a hard problem and hard problems take a lot of time to solve, especially if you want a solution that you're happy with.[00:36:10] Jeremy: And, and I think it's kind of a unique case too, where it's not like if it goes down, it's like people can't do their job. Right. So it's yeah.[00:36:21] Xe: Actually, if tail scales like control plane goes down, I don't think people would notice until they tried to like boot up a, a reboot, a laptop, or connect a new device to their tailnet. Because once, once all the tailscale agents have all of the information they need from the control plate, you know, they just, they just continue on independently and don't have to care.Derp is also fairly independent of the, like the key dropbox component. And, you know, if that, if that goes down Derp doesn't care at all,[00:37:00] Jeremy: Oh, okay. So if the control plane is down, as long as you had authenticated earlier in the day, you can still, I don't know if it's cached or something, but you can still continue to reach the relay servers, the Derp servers or your, [00:37:15] Xe: other nodes. Yeah. I, I'm pretty sure that in most cases, the control plane could be down for several hours a day and nobody would notice unless they're trying to deal with the admin panel.[00:37:28] Jeremy: Got it. that's a little bit of a relief, I suppose, for, for all of you running it,[00:37:33] Xe: Yeah. Um, it's also kind of hard to sell people on the idea of here is a VPN thing. You don't need to self host it and they're like, what? Why? And yeah, it can be fun.[00:37:49] Jeremy: though, I mean, I feel like anybody who has, self-hosted a VPN, they probably like don't really wanna do it. I don't know. Maybe I'm wrong.[00:38:00] Xe: well, so a lot of the idea of wanting to self host it is, uh, I think it's more of like trying to be self-sufficient and not have to rely on other companies, failures dictating your company's downtime. And, you know, like from some level that's very understandable. And, you know, if, you know, like tail scale were to get bought out and the new owners would, you know, like basically kill the product, they'd still have something that would work for them.I don't know if like such a defeatist attitude is like productive. But it is certainly the opinion that I have received when I have asked people why they wanna self-host. other people, don't want to deal with identity providers or the, like, they wanna just use their, they wanna use their own identity provider.And what was hilarious was there was one, there was one thing where they were like our old VPN server died once and we got locked out of our network. So therefore we wanna, we wanna self-host tailscale in the future so that this won't happen again.And I'm like, buddy, let's, let's just, let's just take a moment and retrace our steps here. CAuse I don't think you mean what you think you mean.[00:39:17] Jeremy: yeah, yeah. [00:39:19] Xe: In general, like I suggest people that, you know, even if they're like way deep into the tailscale, Kool-Aid they still have at least one other method of getting into their servers. Ideally, two. I, I admit that I'm, I come from an SRE style background and I am way more paranoid than most, but it, I usually like having, uh, a backup just in case.[00:39:44] Jeremy: So I, I suppose, on, on that note, let's, let's talk a little bit about your role at tailscale. the title of the archmage of infrastructure is one of the, the coolest titles I've, uh, I've seen. So maybe you can go a little bit into what that entails at, at tailscale.[00:40:02] Xe: I started that title as a joke that kind of stuck, uh, my intent, my initial intent was that every time someone asked, I'd say, I'd have a different, you know, like mystic sounding title, but, uh, archmage of infrastructure kind of stuck. And since then, I've actually been pivoting more into developer relations stuff rather than pure software engineering.And, from the feedback that I've gotten at the various conferences I've spoken at, they like that title, even though it doesn't really fit with developer relations work at all, it it's like it fits because it doesn't. You know, that kind of coney kind of way.[00:40:40] Jeremy: I guess this would go more into the, the infrastructure side, but. What does the, the scale of your infrastructure look like? I mean, I, I think that you touched a little bit on the fact that you have relay servers all over the place and you've got this control plane, but I wonder if you could give people a little bit of perspective of what kind of undertaking this is.[00:41:04] Xe: I am pretty sure at this point we have more developer laptops and the like, than we do production servers. Um, I'm pretty sure that the scale of the production of production servers are in the tens, at most. Um, it turns out that computers are pretty darn and efficient and, uh, you don't really need like a lot of computers to do something amazing.[00:41:27] Jeremy: the part that I guess surprises me a little bit is, is the relay servers, I suppose, because, I would imagine there's a lot of traffic that goes through those. are you finding that just most of the time they just aren't needed and usually you can make a direct connection and that's why you don't need too many of these.[00:41:45] Xe: From what I understand. I don't know if we actually have a way to tell, like what percentage of data is going over the relays versus not. And I think that was an intentional decision, um, that may have been revisited I'm operating based off of like six to 12 month old information right now. But in general, like the only state that the relay servers has is in Ram.And whenever the relay, whenever you disconnect the server, the state is dropped.[00:42:18] Jeremy: Okay.[00:42:19] Xe: and even then that state is like, you know, this key is listening. It is, uh, connected, uh, in case you wanna send packets over here, I guess. it's a bit less bandwidth than you're probably thinking it's not like enough to max it out 24/7, but it is, you know, measurable and there are some, you know, costs associated with it. This is also why it's on digital ocean and vulture and not AWS. but in general, it's a lot less than you'd think. I'm pretty sure that like, if I had to give a baseless assumption, I'd say that probably about like 85% of traffic goes directly.And the remaining is like the few cases in the whole punching engine that we haven't figured out yet. Like Palo Alto fire walls. Oh God. Those things are a nightmare.[00:43:13] Jeremy: I see. So it's most of the traffic actually ends up. Being straight peer to peer. Doesn't have to go through your infrastructure. And, and therefore it's like, you don't need too many machines, uh, to, to make this whole thing work.[00:43:28] Xe: Yeah. it turns out that computers are pretty darn fast and that copying data is something that computers are really good at doing. Um, so if you have, you know, some pretty darn fast computers, basically just sitting there and copying data back and forth all day, like it, you can do a lot with shockingly little.Um, when I first started, I believe that the Derp VMs were using like sometimes as little as one core and 512 megabytes of Ram as like a primary Derp. And, you know, we only noticed when, there were some weird connection issues for people that were only on Derp because there were enough users that the machine had ran out of memory.So we just, you know, upped the, uh, virtual machine size and called it a day. But it's, it's truly remarkable how mu how far you can get with very little[00:44:23] Jeremy: And you mentioned the relay servers, the, the Derp servers were on services like digital ocean and Vultr. I'm assuming because of the, the bandwidth cost, for the control plane, is, is that on AWS or some other big cloud provider?[00:44:39] Xe: it's on AWS. I believe it's in EU central 1.[00:44:44] Jeremy: You're helping people connect from device to device and in a situation like that. what does monitoring look like in, in incidents? Like what are you looking for to determine like, Hey, something's not working.[00:44:59] Xe: there's monitoring with, you know, Prometheus, Grafana, all of that stuff. there are some external probing things. there's also some continuous functional testing for trying to connect to tailscale and like log in as an account. And if that fails like twice in a row, then, you know, something's very wrong and, you know, raise the alarm.But in general. A lot of our monitoring is kind of hard at some level because you know, we're tailscale at a tailscale can't always benefit from tailscale to help operate tail scale because you know, it's tailscale. Um, so it, it still trying to figure out how to detangle the chicken and egg situation.It's really annoying.there's the, the term dog fooding, right? Where they're saying like, oh, we, we run, um, our own development on our own platform or our own software. but I could see when your product is network infrastructure, VPNs, where that could be a little, little dicey.[00:46:06] Xe: Yeah, it is very annoying. But I I'm pretty sure we'll figure something out. It is just a matter of when, another thing that's come up is we've kind of wanted to use tailscale's SSH features, where you specify ACLs in your, you specify ACL rules to allow people to SSH, to other nodes as various users.but if that becomes your main access to production, then you know, like if tailscale is down and you're tailscale, like how do you get in, uh, then there's been various philosophical discussions about this. it's also slightly worse if you use what's called check mode in SSH, where, uh, tail scale, SSH without check mode, you know, you just, it, the, the server checks against the policy rules and the ACL and if it. if it's okay, it lets you in. And if not, it says no, but with check mode, there's also this like eight hour, there's this like eight hour quote unquote lifetime for you to have like sudo mode on GitHub, where you do an auth an auth challenge with your auth aprovider. And then, you know, you're given a, uh, Hey, this person has done this thing type verification.And if that's down and that goes through the control plane, and if the control plane is down and you're tailscale, trying to debug the control plane, and in order to get into the control plane over tailscale, you need to use the, uh, control plane. It, you know, that's like chicken and egg problem level 78,which is a mythical level of chicken egg problem that, uh, has only been foretold in the legends of yore or something.[00:47:52] Jeremy: at that point, it sounds like somebody just needs to, to drive to the data center and plug into the switch.[00:47:59] Xe: I mean, It's not, it's not going to, it probably wouldn't be like, you know, we need to get a person with an angle grinder off of Craigslist type bad. Like it was with the Facebook BGP outage, but it it's definitely a chicken and egg problem in its own right.it makes you do a lot of lateral thinking too, which is also kind of interesting.[00:48:20] Jeremy: When, when you say lateral thinking, I'm just kind of curious, um, if you have an example of what you mean.[00:48:27] Xe: I don't know of any example that isn't NDAed. Um, but basically, you know, tail scale is getting to the, to the point where tailscale is relying on tailscale to make tailscale function and you know, yeah. This is classic oroboros style problem.I've heard a, uh, a wise friend of mine said that that is an ideal problem to have, which sounds weird at face value. But if you're getting to that point, that means that you're successful enough that, you know, you're having that problem, which is in itself a good thing, paradoxically.[00:49:07] Jeremy: better to have that problem than to have nobody care about the product. Right.[00:49:12] Xe: Yeah.[00:49:13] Jeremy: kind of on that, that note, um, you mentioned you worked at, at Salesforce, uh, I believe that was working on Heroku. I wonder if you could talk a little about your experience working at, you know, tailscale, which is kind of more of a, you know, early startup versus, uh, an established company like Salesforce.[00:49:36] Xe: So at the time I was working at Heroku, it definitely didn't feel like I was working at Salesforce for the majority of it. It felt like I was working, you know, at Heroku, like on my resume, I listed as Heroku. When I talked about it to people, I said, I worked at Heroku and that sales force was this, you know, mythical, Ohana thing that I didn't have to deal with unless I absolutely had to.By the end of the time I was working at Heroku, uh, the salesforce, uh, sort of started to creep in and, you know, we moved from tracking issues in GitHub issues. Like we were used to, to using their, oh, what's the polite way to say this, their creation, which is, which was like the moral equivalent of JIRA implemented on top of Salesforce.You had to be behind the VPN for it. And, you know, every ticket had 20 fields and, uh, there were no templates. And in comparison with tail scale, you know, we just use GitHub issues, maybe some like things in notion for doing like longer term tracking or Kanban stuff, but it's nice to not have. you know, all of the pomp and ceremony of filling out 20 fields in a ticket for like two sentences of this thing is obviously wrong and it's causing X to happen.Please fix.[00:51:08] Jeremy: I, I like that, that phrase, the, the creation, that's a very, very diplomatic term.[00:51:14] Xe: I mean, I can think of other ways to describe it, but I'm pretty sure those ways wouldn't be allowed on the podcast. So[00:51:25] Jeremy: Um, but, but yeah, I, I know what you mean for sure where, it, it feels like there's this movement from, Hey, let's just do what we need. Like let's fill in the information that's actually relevant and don't do anything else to a shift to, we need to fill in these 10 fields because that's the thing we do.Yeah.[00:51:48] Xe: Yeah. and in the time I've been working for tail scale, I'm like employee ID 12. And, uh, tail scale has gone from a company where I literally know everyone to just recently to the point where I don't know everyone anymore. And it's a really weird feeling. I've never been in a, like a small stage startup that's gotten to this size before, and I've described some of my feelings to other people who have been there and they're like, yeah, welcome to the club. So I figure a lot of it is normal. from what I understand, though, there's a lot of intentionality to try to prevent tail skill from becoming, you know, like Google style, complexity, organizational complexity, unless that is absolutely necessary to do something.[00:52:36] Jeremy: it's a function of size, right? Like as you have more people, more teams, then more process comes in. that's a really tricky balance to, to grow and still keep that feeling of, I'm just doing the thing, I'm doing the work rather than all this other process stuff.[00:52:57] Xe: Yeah, but it, I've also kind of managed to pigeonhole myself off into a corner with devrel stuff. And that's been nice. I've been working a bunch with, uh, like marketing people and, uh, helping out with support occasionally and doing a, like a godawful amount of writing.[00:53:17] Jeremy: the, the writing, for our audience's benefit, I, I think they should, they should really check out your blog because I think that the way you write your, your articles is very thoughtful in terms of the balance of the actual example code or example scripts and the descriptions and, and some there's a little bit of a narrative sometimes too.So, [00:53:40] Xe: Um, I'm actually more of a prose writer just by like how I naturally write things. And a lot of the style of how I write things is, I will take elements from, uh, the Socratic style of dialogue where, you know, you have the student and the teacher. And, you know, sometimes the student will ask questions that the teacher will answer.And I found that that's a particularly useful way to help model understanding or, you know, like put side concepts off into their own little blurbs or other things like that. I also started doing those conversation things with, uh, furry art, specifically to dunk on a homophobe that was getting very angry at furry art being in, uh, another person's blog.And that's it, it's occasionally fun to go into the, uh, orange website of bad takes and see the comments when people complain about it. oh gosh, the bad takes are hilariously good. Sometimes.[00:54:45] Jeremy: it's good that you have like a, a positive, mindset around that. I know some people can read, uh, that sort of stuff and go, you know, just get really bummed out. [00:54:54] Xe: One of the ways I see it is that a lot of the time algorithms are based on like sheer numbers. So if you like get something that makes people argue in the comments, that number will go up and because there's more comments on it, it makes more people more likely to, to read the article and click on it.So, sometimes I have been known to sprinkle, what's the polite way to say this. I've been known to sprinkle like intentionally kind of things that will, uh, get people and make them want to argue about it in the comments. Purely to make the engagement numbers rise up, which makes more people likely to read the article.And, it's kind of a dirty practice, but you know, it makes more people read the article and more people benefit. So, you know, like it's kind of morally neutral, I guess.[00:55:52] Jeremy: usually that, that seems like, a sketchy thing. But I feel like if it's in service to, uh, like a technical blog post, I mean, why not? Right.[00:56:04] Xe: And a lot of the times I'll usually have the like, uh, kind of bad take, be in a little conversation blurb thing so that people will additionally argue about the characterization of, you know, the imaginary cartoon shark or whatever.[00:56:20] Jeremy: That's good. It's the, uh, it's the Xe Xe universe that they're, they're stepping into.[00:56:27] Xe: I've heard people describe it, uh, lovingly as the xeiaso.net cinematic universe.I've had some ideas on how to expand it in the future with more characters that have more different kind of diverse backgrounds. But, uh, it turns out that writing this stuff is hard. Like actually very hard because you have to get this right.You have to get the right balance of like snark satire, uh, like enlightenment. Andit's, it's surprisingly harder than you'd think. Um, but after a while, I've just sort of managed to like figure out as I'm writing where the side tangents come off and which ones I should keep and which ones I should, uh, prune and which ones can also help, Gain deeper understanding with a little like Socratic dialogue to start with a Mo like an incomplete assumption, like an incomplete picture.And then, you know, a question of, wait, what about this thing? Doesn't that conflict with that? And like, well, yes. technically it does, but realistically we don't have to worry about that as much. So we can think about it just in terms of this bigger model and, uh, that's okay. Like, uh, I mentioned the OSI model earlier, you know, like the seven layer OSI model it's, you know, genuinely overkill for basically everything, except it's a really great conceptual model for figuring out the difference between, you know, like an ethernet cable, an ethernet, like the ethernet card, the IP stack TCP and, you know, TLS or whatever.I have a couple talks that are gonna be up by the time this is published. Uh, one of them is my, uh, rustconf talk on my, or what was it called? I think it was called the surreal horrors of PAM or something where I discussed my experience, trying to bug a PAM module in rust, uh, for work. And, uh, it's the kind of story where, you know, it's bad when you have a break point on dlopen.[00:58:31] Jeremy: That sounds like a nightmare.[00:58:32] Xe: Oh yeah. Like part of the attempting to fix that process involved, going very deep. We're talking like an HTML frame set in the internet archive for sunOS documentation that was written around the time that PAM was used. Like it's things that are bad enough were like everything in the frame set, but the contents had eroded away through bit rot and you know, you're very lucky just to have what you do.[00:59:02] Jeremy: well, I'm, I'm glad it was. It was you and not me. we'll get to, to hear about it and, and not have to go through the, the suffering ourselves.[00:59:11] Xe: yeah. One of the things I've been telling people is that I'm not like a brilliant programmer. Like I know a bunch of people who are definitely way smarter than me, but what I am is determined and, uh, determination is a bit stronger of a force than you'd think.[00:59:27] Jeremy: Yeah. I mean, without it, nothing gets done. Right.[00:59:30] Xe: Yeah.[00:59:31] Jeremy: as we wrap up, is there anything we missed or anything else you wanna mention? [00:59:36] Xe: if you wanna look at my blog, it's on xeiaso.net. That's X, E I a S o.net. Um, that's where I post things. You can see, like the 280 something articles at time of recording. It's probably gonna get to 300 at some point, oh God, it's gonna get to 300 at some point. Um, and yeah, from, I try to post articles about weekly, uh, depending on facts and circumstances, I have a bunch of talks coming up, like one about the hilarious over engineering I did in my blog.And maybe some more. If I get back positive responses from calls for paper submissions,[01:00:21] Jeremy: Very cool. Well, Xe thank you so much for, for coming on software engineering radio.[01:00:27] Xe: Yeah. Thank you for having me. I hope you have a good day and, uh, try out tailscale, uh, note my bias, but I think it's great.

An airhacks.fm conversation with Stuart Marks (@stuartmarks) about: Wang 2200 Laboratories computer with 10 years, David Ahl 101 Basic Computer Games, Basic without "else", GOTO and GOSUB, Pascal Records and Java, conditional evaluation in Pascal, the criticism on Pascal, Bill Joy added the socket interface to BSD 4.2, replacing VMS with BSD, the Bill Joy long weekend, starting at Sun Microsystems, working with James Gosling on the NeWS windows system, Postcript based windows system, NeWS ran on SunOS, SunOS 5 became Solaris, the unpleasant UNIX wars with AT&T, HP and IBM, X-Window vs. NeWS, shared state and NeWS, display postscript became the NeXT system, the X-NeWS merge OS, Open Look and Motif, OSF-opensource foundation, Motif became the dominant OS, creating a eCommerce system with Java at Sun, working with James Gosling at NeWS, project Oak and Project Green, Star Seven, licensing WebLogic and Tengah, personal Java and the Java Ring, Java on Sharp Zaurus and on Palm, working on J2ME, working with JavaFX, Chris Oliver started JavaFX, F3 and Forms Follow Function, Java FX Script was an own language, Richard Bair was the JavaFX architect, Jasper Potts was was the Java FX UI designer, JavaFX is based on final classes, the fragile base class / brittle base class problem, the general subclassing problem, implementing a 2d traversial algorithm for Java FX, Sun was shrinking, Java FX was growing, Brian Goetz worked to improve the Java FX internals, RIAs - Rich Internet Applications, Silverlight, Flash, Flex and JavaFX, JavaFX supported CSS, the compiler bug war story, binding propagates side effects, Robert Field is working on jshell, Stuart Marks on twitter: @stuartmarks, Stuart Marks blog: stuartmarks.wordpress.com

Oxide and Friends Twitter Space: October 4th, 2021Economics and Open SourceWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 4th, 2021.In addition to Bryan Cantrill and Adam Leventhal, speakers on October 4th included Edwin Peer, James Todd, Peter Corless, Matt Campbell, jasonbking, Simeon Miteff, Josh Clulow, Ian, Joe Thompson, Dan Cross, Tom Lyon, Tim Burnham, and vint serp. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: Mark Jones Lorenzo (2017) Endless Loop: The History of the BASIC Programming Language bookJohn Kemeny wiki [@3:11](https://youtu.be/JDd8xGSP9DA?t=191) Tim's excellent tweet William Gibson wiki John Browne (1996) The Bug Count Also Rises short story [@5:38](https://youtu.be/JDd8xGSP9DA?t=338) Growing up with BASIC [@8:03](https://youtu.be/JDd8xGSP9DA?t=483) Braille 'n Speak PDA (intro video), BASIC programming TI-BASIC language [@10:39](https://youtu.be/JDd8xGSP9DA?t=639) Speaking program reading off system calls in real time  snoop could output to /dev/audio [@13:39](https://youtu.be/JDd8xGSP9DA?t=819) Joel Spolsky (2002) Strategy Letter V blog Bryan's (2004) The Economics of Software blog Software “maintenance” [@20:02](https://youtu.be/JDd8xGSP9DA?t=1202) Cathedral and the Bazaar, wiki“Forkophilic” development model and the Alan Cox -ac Linux tree [@26:07](https://youtu.be/JDd8xGSP9DA?t=1567) Open source as something in the commercial best interest of a business  SCO v IBM wiki Halloween documents wiki Steve Ballmer's “Linux is a cancer” quote in the Chicago Sun-Times OpenOffice.org wiki (open sourced from StarOffice) [@30:29](https://youtu.be/JDd8xGSP9DA?t=1829) Document editing as a service. Services and open source Richard Stallman on SaaS [@33:34](https://youtu.be/JDd8xGSP9DA?t=2014) The Joel Test link Joel's (2007) Strategy Letter VI blog “Everybody wants to be a platform” [@38:58](https://youtu.be/JDd8xGSP9DA?t=2338) Joel's take on Sun  Making the pie larger. Porting NFS to rival platforms The Sun Network Filesystem: Design, Implementation and Experience has a section on porting experiences. Monetizing software - “Sun could never monetize software, only hardware” [@44:44](https://youtu.be/JDd8xGSP9DA?t=2684) Window toolkits, “cross platform”, write once run anywhere “Write once, debug everywhere” What's the directory separator on MVS? or Stratos VOS? [@51:40](https://youtu.be/JDd8xGSP9DA?t=3100) James' experience working on Tomcat  Joel's (2002) Lord Palmerston on Programming blog Graphics toolkits, Electron/Web vs Native [@1:05:21](https://youtu.be/JDd8xGSP9DA?t=3921) “OpenSolaris downloads are potential buyers for the ZFS appliance” [@1:06:17](https://youtu.be/JDd8xGSP9DA?t=3977) Jason Hoffman “The Sun does not shine on me”  Strategy cannot make up for poor execution Sun CEO Jonathan Schwartz didn't travel to meet customers Demoing to a hostile audience “Asteroid named Linux on a collision course” tweet [@1:13:20](https://youtu.be/JDd8xGSP9DA?t=4400) Open-core, AWS services, monetizing open source  “People will pay for a service” Could Apple open source? [@1:18:43](https://youtu.be/JDd8xGSP9DA?t=4723) Packaged solutions; giving mom a linux box. Free software: free for whom? Support relationships. People want support [@1:22:05](https://youtu.be/JDd8xGSP9DA?t=4925) Why didn't Sun embrace Linux?  ZFS on Linux, Ubuntu The Sourceware Operating System Proposal – Larry McVoy's open source SunOS 4 proposal. Sun bought Cobalt wiki [@1:25:33](https://youtu.be/JDd8xGSP9DA?t=5133) “The writing was on the wall for Sun..”  x86 price-performance “Couldn't you buy like 100 x86 computers for that price?” RISC machine in-fighting, while Intel undercuts the market [@1:31:01](https://youtu.be/JDd8xGSP9DA?t=5461) Josh's work on frustrating hardware configuration [@1:33:25](https://youtu.be/JDd8xGSP9DA?t=5605) Peter's experience as a Sun customer  Vertical scaling, but not so much horizontal scaling Clusters of cheap commodity hardware outperforming big multiway boxes Importance of open source for big internet companies Traders used Sun workstations, for fast trading [@1:38:39](https://youtu.be/JDd8xGSP9DA?t=5919) Oxide is bringing up their first server boards! If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Luke Leighton Luke Leighton is the Lead Dev and Project Coordinator for Libre-SOC LinkedIn: https://www.linkedin.com/in/lkclnet/ Twitter: https://twitter.com/lkcl LibreSOC: https://libre-soc.org Notes: https://www.crnhq.org/12-skill-summary/ https://aldacenter.org Credits: Music by ikson: https://www.iksonmusic.com Special Guest: Luke Kenneth Casson Leighton.

Steve Jobs had an infamous split with the board of directors of Apple and left the company shortly after the release of the original Mac. He was an innovator who at 21 years old had started Apple in the garage with Steve Wozniak and at 30 years old while already plenty wealthy felt he still had more to give and do. We can say a lot of things about him but he was arguably one of the best product managers ever.  He told Apple he'd be taking some “low-level staffers” and ended up taking Rich Page, Bud Tribble, Dan'l Lewin, George Crow, and Susan Barnes to be the CFO. They also took Susan Kare and Joanna Hoffman. had their eyes on a computer that was specifically targeting higher education. They wanted to build computers for researchers and universities.  Companies like CDC and Data General had done well in Universities. The team knew there was a niche that could be carved out there. There were some gaps with the Mac that made it a hard sell in research environments. Computer scientists needed object-oriented programming and protected memory. Having seen the work at PARC on object-oriented languages, Jobs knew the power and future-proof approach.  Unix System V had branched a number of times and it was a bit more of a red ocean than I think they realized. But Jobs put up $7 million of his own money to found NeXT Computer. He'd add another $5 million and Ross Perot would add another $20 million. The pay bands were one of the most straight-forward of any startup ever founded. The senior staff made $75,000 and everyone else got $50,000. Simple.  Ironically, so soon after the 1984 Super Bowl ad where Jobs based IBM, they hired the man who designed the IBM logo, Paul Rand, to design a logo for NeXT. They paid him $100,000 flat. Imagine the phone call when Jobs called IBM to get them to release Rand from a conflict of interest in working with them.  They released the first computer in 1988. The NeXT Computer, as it was called, was expensive for the day, coming in at $6,500. It sported a Motorola 68030 CPU and clocked in at a whopping 25 MHz. And it came with a special operating system called NeXTSTEP. NeXTSTEP was based on the Mach kernel with some of the source code coming from BSD. If we go back a little, Unix was started at Bell Labs in 1969 and by the late 70s had been forked from Unix System V to BSD, Unix version 7, and PWB - with each of those resulting in other forks that would eventually become OpenBSD, SunOS, NetBSD, Solaris, HP-UX, Linux, AIX, and countless others.  Mach was developed at Carnegie Mellon University and is one of the earliest microkernels. For Mach, Richard Rashid (who would later found Microsoft Research) and Avie Tevanian, were looking specifically to distributed computing. And the Mach project was kicked off in 1985, the same year Jobs left Apple.  Mach was backwards-compatible to BSD 4.2 and so could run a pretty wide variety of software. It allowed for threads, or units of execution and tasks or objects that enabled threads. It provided support for messages, which for object oriented languages are typed data objects that fall outside the scope of tasks and threads and then a protected message queue, to manage the messages between tasks and rights of access. They stood it up on a DEC VAX and released it publicly in 1987. Here's the thing, Unix licensing from Bell Labs was causing problems. So it was important to everyone that the license be open. And this would be important to NeXT as well. NeXT needed a next-generation operating system and so Avi Tevanian was recruited to join NeXT as the Vice President of Software Engineering. There, he designed NeXTSTEP with a handful of engineers. The computers had custom boards and were fast. And they were a sleek black like nothing I'd seen before. But Bill Gates was not impressed claiming that “If you want black, I'll get you a can of paint.” But some people loved the machines and especially some of the tools NeXT developed for programmers. They got a factory to produce the machines and it only needed to crank out 100 a month as opposed to the thousands it was built to produce. In other words, the price tag was keeping universities from buying the machines. So they pivoted a little. They went up-market with the NeXTcube in 1990, which ran NeXTSTEP, OPENSTEP, or NetBSD and came with the Motorola 68040 CPU. This came machine in at $8,000 to almost $16,000. It came with a hard drive. For the lower end of the market they also released the NeXTstation in 1990, which shipped for just shy of $5,000. The new models helped but by 1991 they had to lay off 5 percent of the company and another 280 by 1993. That's when the hardware side got sold to Canon so NeXT could focus exclusively on NeXTSTEP.  That is, until they got acquired by Apple in 1997. By the end, they'd sold around 50,000 computers. Apple bought NeXT for $429 million and 1.5 million shares of Apple stock, trading at 22 cents at the time, which was trading at $17 a share so worth another $25 and a half million dollars. That makes the deal worth $454 million or $9,080 per machine NeXT had ever built. But it wasn't about the computer business, which had already been spun down. It was about Jobs and getting a multi-tasking, object-oriented, powerhouse of an operating system, the grandparent of OS X - and the derivative macOS, iOS, iPadOS, watchOS, and tvOS forks. The work done at NeXT has had a long-term impact on the computer industry as a whole. For one, the spinning pinwheel on a Mac. And the Dock. And the App Store. And Objective-C. But also Interface Builder as an IDE was revolutionary. Today we use Xcode. But many of the components go back all the way. And so much more.  After the acquisition, NeXT became Mac OS X Server in 1999 and by 2001 was Mac OS X. The rest there is history. But the legacy of the platform is considerable. Just on NeXTSTEP we had a few pretty massive successes. Tim Berners-Lee developed the first web browser WorldWideWeb on NeXTSTEP for a NeXT . Other browsers for other platforms would come but his work became the web as we know it today. The machine he developed the web on is now on display at the National Museum of Science and Media in the UK. We also got games like Quake, Heretic, Stife, and Doom from Interface Builder. And webobjects. And the people.  Tevanian came with NeXT to Apple as the Senior Vice President of Software Engineering. Jobs became an advisor, then CEO. Craig Federighi came with the acquisition as well - now Apple's VP of software engineering. And I know dozens of others who came in from NeXT and helped reshape the culture at Apple. Next.com still redirects to Apple.com. It took three years to ship that first computer at NeXT. It took 2 1/2 years to develop the iPhone. The Apple II, iPod, iPad, and first iMac were much less. Nearly 5 years for the original Mac. Some things take a little more time to flush out than others. Some need the price of components or new components to show up before you know it can be insanely great. Some need false starts like the Steve Jobs Steve Jobs famously said Apple wanted to create a computer in a book in 1983. That finally came out with the release of the iPad in 2010, 27 years later.  And so the final component of the Apple acquisition of NeXT to mention is Steve Jobs himself. He didn't initially come in. He'd just become a billionaire off Pixar and was doing pretty darn well. His arrival back at Apple signified the end of a long draught for the company and all those products we mentioned and the iTunes music store and the App Store (both initially built on WebObjects) would change the way we consume content forever. His impact was substantial. For one, after factoring stock splits, the company might still be trading at .22 cents a share, which is what it would be today with all that. Instead they're the most highly valued company in the world. But that pales in comparison to the way he and his teams and that relentless eye to product and design has actually changed the world. And the way his perspectives on privacy help protect us today, long after he passed.  The heroes journey (as described is a storytelling template that follows a hero from disgrace, to learn the mistakes of their past and reinvent themselves amidst a crisis throughout a grand adventure, and return home transformed. NeXT and Pixar represent part of that journey here. Which makes me wonder: what is my own Monomyth? Where will I return to? What is or was my abyss? These can be large or small. And while very few people in the world will have one like Steve Jobs did, we should all reflect on ours and learn from them. And yes that was plural because life is not so simple that there is one. The past, and our understanding of it, predicts the future. Good luck on your journey. 

5x if_bridge Performance Improvement, How Unix Won, Understanding VLAN Configuration on FreeBSD, Using bhyve PCI passthrough on OmniOS, TrueNAS 11.3-U2 Available, and more. Headlines 5x if_bridge Performance Improvement (https://www.freebsdfoundation.org/blog/500-if_bridge-performance-improvement/) With FreeBSD Foundation grant, Kristof Provost harnesses new parallel techniques to uncork performance bottleneck + Kristof also streamed some of his work, providing an interesting insight into how such development work happens + > https://www.twitch.tv/provostk/videos How Unix Won (https://blog.vivekhaldar.com/post/617189040564928512/how-unix-won) +> Unix has won in every conceivable way. And in true mythic style, it contains the seeds of its own eclipse. This is my subjective historical narrative of how that happened. I’m using the name “Unix” to include the entire family of operating systems descended from it, or that have been heavily influenced by it. That includes Linux, SunOS, Solaris, BSD, Mac OS X, and many, many others. Both major mobile OSs, Android and iOS, have Unix roots. Their billions of users dwarf those using clunky things like laptops and desktops, but even there, Windows is only the non-Unix viable OS. Almost everything running server-side in giant datacenters is Linux. How did Unix win? News Roundup Check logs of central syslog-ng log host on FreeBSD (https://blog.socruel.nu/freebsd/check-logs-of-syslog-ng-log-host-on-freebsd.html) This blog post continues where the blog post A central log host with syslog-ng on FreeBSD left off. Open source solutions to check syslog log messages exist, such as Logcheck or Logwatch. Although these are not to difficult to implement and maintain, I still found these to much. So I went for my own home grown solution to check the syslog messages of the SoCruel.NU central log host. And the solution presented in this blog post works pretty well for me! Understanding VLAN Configuration on FreeBSD (https://genneko.github.io/playing-with-bsd/networking/freebsd-vlan/) Until recently, I’ve never had a chance to use VLANs on FreeBSD hosts, though I sometimes configure them on ethernet switches. But when I was playing with vnet jails, I suddenly got interested in VLAN configuration on FreeBSD and experimented with it for some time. I wrote this short article to summarize my current understanding of how to configure VLANs on FreeBSD. Using bhyve PCI passthrough on OmniOS (https://www.cyber-tec.org/2019/05/29/using-bhyve-pci-passthrough-on-omnios/) Some hardware is not supported in illumos yet, but luckily there is bhyve which supports pci passthrough to any guest operating system. To continue with my OmniOS desktop on "modern" hardware I would love wifi support, so why not using a bhyve guest as router zone which provide the required drivers? TrueNAS 11.3-U2 is Generally Available (https://www.ixsystems.com/blog/truenas-11-3-u2-is-available/) TrueNAS 11.3-U2.1 is generally available as of 4/22/2020. This update is based on FreeNAS 11.3-U2 which has had over 50k deployments and received excellent community and third party reviews. The Release Notes are available on the iXsystems.com website. Beastie Bits HardenedBSD April 2020 Status Report (https://hardenedbsd.org/article/shawn-webb/2020-04-24/hardenedbsd-april-2020-status-report) NYC Bug’s Mailing List - Listing of open Dev Jobs (http://lists.nycbug.org/pipermail/jobs/2020-April/000553.html) Feedback/Questions Greg - Lenovo (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/350/feedback/Greg%20-%20Lenovos.md) Matt - BSD Packaging (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/350/feedback/Matt%20-%20BSD%20Packaging.md) Morgan - Performance (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/350/feedback/Morgan%20-%20Performance.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)

A PI-powered Plan 9 cluster, an SSH tarpit, rdist for when Ansible is too much, falling in love with OpenBSD again, how I created my first FreeBSD port, the Tilde Institute of OpenBSD education and more. Headlines A Pi-Powered Plan 9 Cluster Plan 9 from Bell Labs comes from the same stable as the UNIX operating system, which of course Linux was designed after, and Apple’s OS X runs on top of a certified UNIX operating system. Just like UNIX, Plan 9 was developed as a research O/S — a vehicle for trying out new concepts — with it building on key UNIX principles and taking the idea of devices are just files even further. In this post, we take a quick look at the Plan 9 O/S and some of the notable features, before moving on to the construction of a self-contained 4-node Raspberry Pi cluster that will provide a compact platform for experimentation. Endlessh: an SSH Tarpit I’m a big fan of tarpits: a network service that intentionally inserts delays in its protocol, slowing down clients by forcing them to wait. This arrests the speed at which a bad actor can attack or probe the host system, and it ties up some of the attacker’s resources that might otherwise be spent attacking another host. When done well, a tarpit imposes more cost on the attacker than the defender. The Internet is a very hostile place, and anyone who’s ever stood up an Internet-facing IPv4 host has witnessed the immediate and continuous attacks against their server. I’ve maintained such a server for nearly six years now, and more than 99% of my incoming traffic has ill intent. One part of my defenses has been tarpits in various forms. News Roundup rdist(1) – when Ansible is too much The post written about rdist(1) on johan.huldtgren.com sparked us to write one as well. It's a great, underappreciated, tool. And we wanted to show how we wrapped doas(1) around it. There are two services in our infrastructure for which we were looking to keep the configuration in sync and to reload the process when the configuration had indeed changed. There is a pair of nsd(8)/unbound(8) hosts and a pair of hosts running relayd(8)/httpd(8) with carp(4) between them. We didn't have a requirement to go full configuration management with tools like Ansible or Salt Stack. And there wasn't any interest in building additional logic on top of rsync or repositories. > Enter rdist(1), rdist is a program to maintain identical copies of files over multiple hosts. It preserves the owner, group, mode, and mtime of files if possible and can update programs that are executing. Falling in love with OpenBSD again I was checking the other day and was appalled at how long it has been since I posted here. I had been working a job during 2018 that had me traveling 3,600 miles by air every week so that is at least a viable excuse. So what is my latest project? I wanted to get something better than the clunky old T500 “freedom laptop” that I could use as my daily driver. Some background here. My first paid gig as a programmer was on SunOS 4 (predecessor to Solaris) and Ultrix (on a DEC MicroVAX). I went from there to a Commodore Amiga (preemptive multitasking in 1985!). I went from there to OS/2 (I know, patron saint of lost causes) and then finally decided to “sell out” and move to Windows as the path of least resistance in the mid 90’s. My wife bought me an iPod literally just as they started working with computers other than Macs and I watched with fascination as Apple made the big gamble and moved away from PowerPC chips to Intel. That was the beginning of the Apple Fan Boi years for me. My gateway drug was a G4 MacMini and I managed somehow to get in on the pre-production, developer build of an Intel-based Mac. I was quite happy on the platform until about three years ago. How I Created My First FreeBSD Port I created my first FreeBSD port recently. I found that FreeBSD didn't have a port for GoCD, which is a continuous integration and continuous deployment (CI/CD) system. This was a great opportunity to learn how to build a FreeBSD port while also contributing back to the community The Tilde Institute of OpenBSD Education Welcome to tilde.institute! This is an OpenBSD machine whose purpose is to provide a space in the tildeverse for experimentation with and education of the OpenBSD operating system. A variety of editors, shells, and compilers are installed to allow for development in a native OpenBSD environment. OpenBSD's httpd(8) is configured with slowcgi(8) as the fastcgi provider and sqlite3 available. This allows users to experiment with web development using compiled CGI in C, aka the BCHS Stack. In addition to php7.0 and mysql (mariadb) by request, this provides an environment where the development of complex web apps is possible. Beastie Bits SoloBSD 19.03-STABLE WireGuard for NetBSD [NetBSD - Removing PF](https://mail-index.netbsd.org/tech-kern/2019/03/29/msg024883.html ) What does the N in nmake stand for? A Map of the Internet from May 1973 NSA-B-Gone : A sketchy hardware security device for your x220 Feedback/Questions Jake - A single jail as a VPN client Matt - Surprising BSD Features cia - Routing and ZFS Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.

We explain the physics behind ZFS, DTrace switching to the GPL, Emacs debugging, syncookies coming to PF & FreeBSD's history on EC2. This episode was brought to you by Headlines 128 bit storage: Are you high? (https://blogs.oracle.com/bonwick/128-bit-storage:-are-you-high) For people who have heard about ZFS boiling oceans and wonder where that is coming from, we dug out this old piece from 2004 on the blog of ZFS co-creator Jeff Bonwick, originally from the Sun website. 64 bits would have been plenty ... but then you can't talk out of your ass about boiling oceans then, can you? Well, it's a fair question. Why did we make ZFS a 128-bit storage system? What on earth made us think it's necessary? And how do we know it's sufficient? Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 2^64 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. If 64 bits isn't enough, the next logical step is 128 bits. That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. But it does raise the question: what are the theoretical limits to storage capacity? Although we'd all like Moore's Law to continue forever, quantum mechanics imposes some fundamental limits on the computation rate and information capacity of any physical device. In particular, it has been shown that 1 kilogram of matter confined to 1 liter of space can perform at most 10^51 operations per second on at most 10^31 bits of information [see Seth Lloyd, "Ultimate physical limits to computation." Nature 406, 1047-1054 (2000)]. A fully-populated 128-bit storage pool would contain 2^128 blocks = 2^137 bytes = 2^140 bits; therefore the minimum mass required to hold the bits would be (2^140 bits) / (10^31 bits/kg) = 136 billion kg. That's a lot of gear. To operate at the 1031 bits/kg limit, however, the entire mass of the computer must be in the form of pure energy. By E=mc^2, the rest energy of 136 billion kg is 1.2x1028 J. The mass of the oceans is about 1.4x1021 kg. It takes about 4,000 J to raise the temperature of 1 kg of water by 1 degree Celcius, and thus about 400,000 J to heat 1 kg of water from freezing to boiling. The latent heat of vaporization adds another 2 million J/kg. Thus the energy required to boil the oceans is about 2.4x106 J/kg * 1.4x1021 kg = 3.4x1027 J. Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. Best part of all: you don't have to understand any of this to use ZFS. Rest assured that you won't hit any limits with that filesystem for a long time. You still have to buy bigger disks over time, though... *** dtrace for Linux, Oracle relicenses dtrace (https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-linux-oracle-does-the-right-thing/) At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom. Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port: ``` commit e1744f50ee9bc1978d41db7cc93bcf30687853e6 Author: Tomas Jedlicka tomas.jedlicka@oracle.com Date: Tue Aug 1 09:15:44 2017 -0400 dtrace: Integrate DTrace Modules into kernel proper This changeset integrates DTrace module sources into the main kernel source tree under the GPLv2 license. Sources have been moved to appropriate locations in the kernel tree. ``` That is right, dtrace dropped the CDDL and switched to the GPL! The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out). The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory. The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL. Thank you Oracle for making everyone's life easier by waving your magic relicensing wand! Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest. Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don't always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all. Debugging Emacs or: How I Learned to Stop Worrying and Love DTrace (http://nullprogram.com/blog/2018/01/17/) For some time Elfeed was experiencing a strange, spurious failure. Every so often users were seeing an error (spoiler warning) when updating feeds: “error in process sentinel: Search failed.” If you use Elfeed, you might have even seen this yourself. From the surface it appeared that curl, tasked with the responsibility for downloading feed data, was producing incomplete output despite reporting a successful run. Since the run was successful, Elfeed assumed certain data was in curl's output buffer, but, since it wasn't, it failed hard. Unfortunately this issue was not reproducible. Manually running curl outside of Emacs never revealed any issues. Asking Elfeed to retry fetching the feeds would work fine. The issue would only randomly rear its head when Elfeed was fetching many feeds in parallel, under stress. By the time the error was discovered, the curl process had exited and vital debugging information was lost. Considering that this was likely to be a bug in Emacs itself, there really wasn't a reliable way to capture the necessary debugging information from within Emacs Lisp. And, indeed, this later proved to be the case. A quick-and-dirty work around is to use condition-case to catch and swallow the error. When the bizarre issue shows up, rather than fail badly in front of the user, Elfeed could attempt to swallow the error — assuming it can be reliably detected — and treat the fetch as simply a failure. That didn't sit comfortably with me. Elfeed had done its due diligence checking for errors already. Someone was lying to Elfeed, and I intended to catch them with their pants on fire. Someday. I'd just need to witness the bug on one of my own machines. Elfeed is part of my daily routine, so surely I'd have to experience this issue myself someday. My plan was, should that day come, to run a modified Elfeed, instrumented to capture extra data. I would have also routinely run Emacs under GDB so that I could inspect the failure more deeply. For now I just had to wait to hunt that zebra. Bryan Cantrill, DTrace, and FreeBSD Over the holidays I re-discovered Bryan Cantrill, a systems software engineer who worked for Sun between 1996 and 2010, and is most well known for DTrace. My first exposure to him was in a BSD Now interview in 2015. I had re-watched that interview and decided there was a lot more I had to learn from him. He's become a personal hero to me. So I scoured the internet for more of his writing and talks. Some interesting operating system technology came out of Sun during its final 15 or so years — most notably DTrace and ZFS — and Bryan speaks about it passionately. Almost as a matter of luck, most of it survived the Oracle acquisition thanks to Sun releasing it as open source in just the nick of time. Otherwise it would have been lost forever. The scattered ex-Sun employees, still passionate about their prior work at Sun, along with some of their old customers have since picked up the pieces and kept going as a community under the name illumos. It's like an open source flotilla. Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn't have these Sun technologies available. The main reason is license incompatibility. Sun released its code under the CDDL, which is incompatible with the GPL. Ubuntu does infamously include ZFS, but other distributions are unwilling to take that risk. Porting DTrace is a serious undertaking since it's got its fingers throughout the kernel, which also makes the licensing issues even more complicated. Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that. Rather than adopt ZFS and DTrace, they've been reinvented from scratch: btrfs instead of ZFS, and a slew of partial options instead of DTrace. Normally I'm most interested in system call tracing, and my go to is strace, though it certainly has its limitations — including this situation of debugging curl under Emacs. Another famous example of NIH is Linux's epoll(2), which is a broken version of BSD kqueue(2). So, if I want to try these for myself, I'll need to install a different operating system. I've dabbled with OmniOS, an OS built on illumos, in virtual machines, using it as an alien environment to test some of my software (e.g. enchive). OmniOS has a philosophy called Keep Your Software To Yourself (KYSTY), which is really just code for “we don't do packaging.” Honestly, you can't blame them since they're a tiny community. The best solution to this is probably pkgsrc, which is essentially a universal packaging system. Otherwise you're on your own. There's also openindiana, which is a more friendly desktop-oriented illumos distribution. Still, the short of it is that you're very much on your own when things don't work. The situation is like running Linux a couple decades ago, when it was still difficult to do. If you're interested in trying DTrace, the easiest option these days is probably FreeBSD. It's got a big, active community, thorough documentation, and a huge selection of packages. Its license (the BSD license, duh) is compatible with the CDDL, so both ZFS and DTrace have been ported to FreeBSD. What is DTrace? I've done all this talking but haven't yet described what DTrace really is. I won't pretend to write my own tutorial, but I'll provide enough information to follow along. DTrace is a tracing framework for debugging production systems in real time, both for the kernel and for applications. The “production systems” part means it's stable and safe — using DTrace won't put your system at risk of crashing or damaging data. The “real time” part means it has little impact on performance. You can use DTrace on live, active systems with little impact. Both of these core design principles are vital for troubleshooting those really tricky bugs that only show up in production. There are DTrace probes scattered all throughout the system: on system calls, scheduler events, networking events, process events, signals, virtual memory events, etc. Using a specialized language called D (unrelated to the general purpose programming language D), you can dynamically add behavior at these instrumentation points. Generally the behavior is to capture information, but it can also manipulate the event being traced. Each probe is fully identified by a 4-tuple delimited by colons: provider, module, function, and probe name. An empty element denotes a sort of wildcard. For example, syscall::open:entry is a probe at the beginning (i.e. “entry”) of open(2). syscall:::entry matches all system call entry probes. Unlike strace on Linux which monitors a specific process, DTrace applies to the entire system when active. To run curl under strace from Emacs, I'd have to modify Emacs' behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That's a big deal. So, when it comes to this Elfeed issue, FreeBSD is much better poised for debugging the problem. All I have to do is catch it in the act. However, it's been months since that bug report and I'm not really making this connection yet. I'm just hoping I eventually find an interesting problem where I can apply DTrace. Bryan Cantrill: Talks I have given (http://dtrace.org/blogs/bmc/2018/02/03/talks/) *** News Roundup a2k18 Hackathon preview: Syncookies coming to PF (https://undeadly.org/cgi?action=article;sid=20180207090000) As you may have heard, the a2k18 hackathon is in progress. As can be seen from the commit messages, several items of goodness are being worked on. One eagerly anticipated item is the arrival of TCP syncookies (read: another important tool in your anti-DDoS toolset) in PF. Henning Brauer (henning@) added the code in a series of commits on February 6th, 2018, with this one containing the explanation: ``` syncookies for pf. when syncookies are on, pf will blindly answer each and every SYN with a syncookie-SYNACK. Upon reception of the ACK completing the 3WHS, pf will reconstruct the original SYN, shove it through pf_test, where state will be created if the ruleset permits it. Then massage the freshly created state (we won't see the SYNACK), set up the sequence number modulator, and call into the existing synproxy code to start the 3WHS with the backend host. Add an - somewhat basic for now - adaptive mode where syncookies get enabled if a certain percentage of the state table is filled up with half-open tcp connections. This makes pf firewalls resilient against large synflood attacks. syncookies are off by default until we gained more experience, considered experimental for now. see http://bulabula.org/papers/2017/bsdcan/ for more details. joint work with sashan@, widely discussed and with lots of input by many ``` The first release to have this feature available will probably be the upcoming OpenBSD 6.3 if a sufficient number of people test this in their setups (hint, hint). More info is likely to emerge soon in post-hackathon writeups, so watch this space! [Pale Moon] A Perfect example of how not to approach OS developers/packagers Removed from OpenBSD Ports due to Licensing Issues (https://github.com/jasperla/openbsd-wip/issues/86) FreeBSD Palemoon branding violation (https://lists.freebsd.org/pipermail/freebsd-ports/2018-February/112455.html) Mightnight BSD's response (https://twitter.com/midnightbsd/status/961232422091280386) *** FreeBSD EC2 History (http://www.daemonology.net/blog/2018-02-12-FreeBSD-EC2-history.html) A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years. Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments. August 25, 2006: Amazon EC2 launches. It supports a single version of Ubuntu Linux; FreeBSD is not available. December 13, 2010: I manage to get FreeBSD running on EC2 t1.micro instances. March 22, 2011: I manage to get FreeBSD running on EC2 "cluster compute" instances. July 8, 2011: I get FreeBSD 8.2 running on all 64-bit EC2 instance types, by marking it as "Windows" in order to get access to Xen/HVM virtualization. (Unfortunately this meant that users had to pay the higher "Windows" hourly pricing.) January 16, 2012: I get FreeBSD 9.0 running on 32-bit EC2 instances via the same "defenestration" trick. (Again, paying the "Windows" prices.) August 16, 2012: I move the FreeBSD rc.d scripts which handle "EC2" functionality (e.g., logging SSH host keys to the console) into the FreeBSD ports tree. October 7, 2012: I rework the build process for FreeBSD 9.1-RC1 and later to use "world" bits extracted from the release ISOs; only the kernel is custom-built. Also, the default SSH user changes from "root" to "ec2-user". October 31, 2012: Amazon launches the "M3" family of instances, which support Xen/HVM without FreeBSD needing to pay the "Windows" tax. November 21, 2012: I get FreeBSD added to the AWS Marketplace. October 2, 2013: I finish merging kernel patches into the FreeBSD base system, and rework the AMI build (again) so that FreeBSD 10.0-ALPHA4 and later use bits extracted from the release ISOs for the entire system (world + kernel). FreeBSD Update can now be used for updating everything (because now FreeBSD/EC2 uses a GENERIC kernel). October 27, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA2 and later AMIs will run FreeBSD Update when they first boot in order to download and install any critical updates. December 1, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA4 and later AMIs bootstrap the pkg tool and install packages at boot time (by default, the "awscli" package). December 9, 2013: I add configinit to FreeBSD 10.0-RC1 and later to allow systems to be easily configured via EC2 user-data. July 1, 2014: Amazon launches the "T2" family of instances; now the most modern family for every type of EC2 instance (regular, high-memory, high-CPU, high-I/O, burstable) supports HVM and there should no longer be any need for FreeBSD users to pay the "Windows tax". November 24, 2014: I add code to FreeBSD 10.2 and later to automatically resize their root filesystems when they first boot; this means that a larger root disk can be specified at instance launch time and everything will work as expected. April 1, 2015: I integrate the FreeBSD/EC2 build process into the FreeBSD release building process; FreeBSD 10.2-BETA1 and later AMIs are built by the FreeBSD release engineering team. January 12, 2016: I enable Intel 82599-based "first generation EC2 Enhanced Networking" in FreeBSD 11.0 and later. June 9, 2016: I enable the new EC2 VGA console functionality in FreeBSD 11.0 and later. (The old serial console also continues to work.) June 24, 2016: Intel 82599-based Enhanced Networking works reliably in FreeBSD 11.0 and later thanks to discovering and working around a Xen bug. June 29, 2016: I improve throughput on Xen blkfront devices (/dev/xbd*) by enabling indirect segment I/Os in FreeBSD 10.4 and later. (I wrote this functionality in July 2015, but left it disabled by default a first because a bug in EC2 caused it to hurt performance on some instances.) July 7, 2016: I fix a bug in FreeBSD's virtual memory initialization in order to allow it to support boot with 128 CPUs; aka. FreeBSD 11.0 and later support the EC2 x1.32xlarge instance type. January 26, 2017: I change the default configuration in FreeBSD 11.1 and later to support EC2's IPv6 networking setup out of the box (once you flip all of the necessary switches to enable IPv6 in EC2 itself). May 20, 2017: In collaboration with Rick Macklem, I make FreeBSD 11.1 and later compatible with the Amazon "Elastic File System" (aka. NFSv4-as-a-service) via the newly added "oneopenown" mount option (and lots of bug fixes). May 25, 2017: I enable support for the Amazon "Elastic Network Adapter" in FreeBSD 11.1 and later. (The vast majority of the work — porting the driver code — was done by Semihalf with sponsorship from Amazon.) December 5, 2017: I change the default configuration in FreeBSD 11.2 and later to make use of the Amazon Time Sync Service (aka. NTP-as-a-service). The current status The upcoming FreeBSD release (11.2) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial VGA), and every EC2 instance type (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances). Colin's Patreon' page if you'd like to support him (https://www.patreon.com/cperciva) X network transparency X's network transparency has wound up mostly being a failure (https://utcc.utoronto.ca/~cks/space/blog/unix/XNetworkTransparencyFailure) I was recently reading Mark Dominus's entry about some X keyboard problems, in which he said in passing (quoting himself): I have been wondering for years if X's vaunted network transparency was as big a failure as it seemed: an interesting idea, worth trying out, but one that eventually turned out to be more trouble than it was worth. [...] My first reaction was to bristle, because I use X's network transparency all of the time at work. I have several programs to make it work very smoothly, and some core portions of my environment would be basically impossible without it. But there's a big qualification on my use of X's network transparency, namely that it's essentially all for text. When I occasionally go outside of this all-text environment of xterms and emacs and so on, it doesn't go as well. X's network transparency was not designed as 'it will run xterm well'; originally it was to be something that should let you run almost everything remotely, providing a full environment. Even apart from the practical issues covered in Daniel Stone's slide presentation, it's clear that it's been years since X could deliver a real first class environment over the network. You cannot operate with X over the network in the same way that you do locally. Trying to do so is painful and involves many things that either don't work at all or perform so badly that you don't want to use them. In my view, there are two things that did in general X network transparency. The first is that networks turned out to not be fast enough even for ordinary things that people wanted to do, at least not the way that X used them. The obvious case is web browsers; once the web moved to lots of images and worse, video, that was pretty much it, especially with 24-bit colour. (It's obviously not impossible to deliver video across the network with good performance, since YouTube and everyone else does it. But their video is highly encoded in specialized formats, not handled by any sort of general 'send successive images to the display' system.) The second is that the communication facilities that X provided were too narrow and limited. This forced people to go outside of them in order to do all sorts of things, starting with audio and moving on to things like DBus and other ways of coordinating environments, handling sophisticated configuration systems, modern fonts, and so on. When people designed these additional communication protocols, the result generally wasn't something that could be used over the network (especially not without a bunch of setup work that you had to do in addition to remote X). Basic X clients that use X properties for everything may be genuinely network transparent, but there are very few of those left these days. (Not even xterm is any more, at least if you use XFT fonts. XFT fonts are rendered in the client, and so different hosts may have different renderings of the same thing, cf.) < What remains of X's network transparency is still useful to some of us, but it's only a shadow of what the original design aimed for. I don't think it was a mistake for X to specifically design it in (to the extent that they did, which is less than you might think), and it did help X out pragmatically in the days of X terminals, but that's mostly it. (I continue to think that remote display protocols are useful in general, but I'm in an usual situation. Most people only ever interact with remote machines with either text mode SSH or a browser talking to a web server on the remote machine.) PS: The X protocol issues with synchronous requests that Daniel Stone talks about don't help the situation, but I think that even with those edges sanded off X's network transparency wouldn't be a success. Arguably X's protocol model committed a lesser version of part of the NeWS mistake. X's network transparency was basically free at the time (https://utcc.utoronto.ca/~cks/space/blog/unix/XFreeNetworkTransparency) I recently wrote an entry about how X's network transparency has wound up mostly being a failure for various reasons. However, there is an important flipside to the story of X's network transparency, and that is that X's network transparency was almost free at the time and in the context it was created. Unlike the situation today, in the beginning X did not have to give up lots of performance or other things in order to get network transparency. X originated in the mid 1980s and it was explicitly created to be portable across various Unixes, especially BSD-derived ones (because those were what universities were mostly using at that time). In the mid to late 1980s, Unix had very few IPC methods, especially portable ones. In particular, BSD systems did not have shared memory (it was called 'System V IPC' for the obvious reasons). BSD had TCP and Unix sockets, some System V machines had TCP (and you could likely assume that more would get it), and in general your safest bet was to assume some sort of abstract stream protocol and then allow for switchable concrete backends. Unsurprisingly, this is exactly what X did; the core protocol is defined as a bidirectional stream of bytes over an abstracted channel. (And the concrete implementation of $DISPLAY has always let you specify the transport mechanism, as well as allowing your local system to pick the best mechanism it has.) Once you've decided that your protocol has to run over abstracted streams, it's not that much more work to make it network transparent (TCP provides streams, after all). X could have refused to make the byte order of the stream clear or required the server and the client to have access to some shared files (eg for fonts), but I don't think either would have been a particularly big win. I'm sure that it took some extra effort and care to make X work across TCP from a different machine, but I don't think it took very much. (At the same time, my explanation here is probably a bit ahistorical. X's initial development seems relatively strongly tied to sometimes having clients on different machines than the display, which is not unreasonable for the era. But it doesn't hurt to get a feature that you want anyway for a low cost.) I believe it's important here that X was intended to be portable across different Unixes. If you don't care about portability and can get changes made to your Unix, you can do better (for example, you can add some sort of shared memory or process to process virtual memory transfer). I'm not sure how the 1980s versions of SunView worked, but I believe they were very SunOS dependent. Wikipedia says SunView was partly implemented in the kernel, which is certainly one way to both share memory and speed things up. PS: Sharing memory through mmap() and friends was years in the future at this point and required significant changes when it arrived. Beastie Bits Grace Hopper Celebration 2018 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/grace-hopper-celebration-2018-call-for-participation/) Google Summer of Code: Call for Project Ideas (https://www.freebsdfoundation.org/blog/google-summer-of-code-call-for-project-ideas/) The OpenBSD Foundation 2018 Fundraising Campaign (https://undeadly.org/cgi?action=article;sid=20180129190641) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) AsiaBSDcon 2018 Registration is open (https://2018.asiabsdcon.org/) Tarsnap support for Bitcoin ending April 1st; and a Chrome bug (http://mail.tarsnap.com/tarsnap-announce/msg00042.html) Feedback/Questions Todd - Couple Questions (http://dpaste.com/195HGHY#wrap) Seth - Tar Snap (http://dpaste.com/1N7NQVQ#wrap) Alex - sudo question (http://dpaste.com/3D9P1DW#wrap) Thomas - FreeBSD on ARM? (http://dpaste.com/24NMG47#wrap) Albert - Austria BSD User Group (http://dpaste.com/373CRX7#wrap)

We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)

We read the FreeBSD Q3 status report, explore good and bad syscalls, list GOG Games for OpenBSD, and show you what devmatch can do. This episode was brought to you by Headlines FreeBSD Q3 Status Report 2017 (https://lists.freebsd.org/pipermail/freebsd-announce/2017-December/001818.html) FreeBSD Team Reports FreeBSD Release Engineering Team Ports Collection The FreeBSD Core Team The FreeBSD Foundation Projects FreeBSD CI Kernel Intel 10G iflib Driver Update Intel iWARP Support pNFS Server Plan B Architectures AMD Zen (family 17h) support Userland Programs Updates to GDB Ports FreeBSDDesktop OpenJFX 8 Puppet Documentation Absolute FreeBSD, 3rd Edition Manual Pages Third-Party Projects The nosh Project ####FreeBSD Foundation Q4 Update (https://www.freebsdfoundation.org/wp-content/uploads/2017/12/FreeBSD-Foundation-Q4-Update.pdf) *** ###11 syscalls that rock the world (https://www.cloudatomiclab.com/prosyscall/) 0. read > You cannot go wrong with a read. You can barely EFAULT it! On Linux amd64 it is syscall zero. If all its arguments are zero it returns zero. Cool! 1. pipe > The society for the preservation of historic calling conventions is very fond of pipe, as in many operating systems and architectures it preserves the fun feature of returning both of the file descriptors as return values. At least Linux MIPS does, and NetBSD does even on x86 and amd64. Multiple return values are making a comeback in languages like Lua and Go, but C has always had a bit of a funny thing about them, but they have long been supported in many calling conventions, so let us use them in syscalls! Well, one syscall. 2. kqueue > When the world went all C10K on our ass, and scaleable polling was a thing, Linux went epoll, the BSDs went kqueue and Solaris went /dev/poll. The nicest interface was kqueue, while epoll is some mix of edge and level triggered semantics and design errors so bugs are still being found. 3. unshare > Sounds like a selfish syscall, but this generous syscall call is the basis of Linux namespaces, allowing a process to isolate its resources. Containers are built from unshares. 4. setns > If you liked unshare, its younger but cooler friend takes file descriptors for namespaces. Pass it down a unix socket to another process, or stash it for later, and do that namespace switching. All the best system calls take file descriptors. 5. execveat > Despite its somewhat confusing name (FreeBSD has the saner fexecve, but other BSDs do not have support last time I checked), this syscall finally lets you execute a program just given a file descriptor for the file. I say finally, as Linux only implemented this in 3.19, which means it is hard to rely on it (yeah, stop using those stupid old kernels folks). Before that Glibc had a terrible userspace implementation that is basically useless. Perfect for creating sandboxes, as you can sandbox a program into a filesystem with nothing at all in, or with a totally controlled tree, by opening the file to execute before chroot or changing the namespace. 6. pdfork > Too cool for Linux, you have to head out to FreeBSD for this one. Like fork, but you get a file descriptor for the process not a pid. Then you can throw it in the kqueue or send it to another process. Once you have tried process descriptors you will never go back. 7. signalfd > You might detect a theme here, but if you have ever written traditional 1980s style signal handlers you know how much they suck. How about turning your signals into messages that you can read on, you guessed it, file descriptors. Like, usable. 8. wstat > This one is from Plan 9. It does the opposite of stat and writes the same structure. Simples. Avoids having chmod, chown, rename, utime and so on, by the simple expedient of making the syscall symmetric. Why not? 9. clonefile > The only cool syscall on OSX, and only supported on the new APFS filesystem. Copies whole files or directories on a single syscall using copy on write for all the data. Look on my works, copyfilerange and despair. 10. pledge > The little sandbox that worked. OpenBSD only here, they managed to make a simple sandbox that was practical for real programs, like the base OpenBSD system. Capsicum form FreeBSD (and promised for Linux for years but no sign) is a lovely design, and gave us pdfork, but its still kind of difficult and intrusive to implement. Linux has, well, seccomp, LSMs, and still nothing that usable for the average program. ###Eleven syscalls that suck (https://www.cloudatomiclab.com/antisyscall/) 0. ioctl > It can‘t decide if it‘s arguments are integers, strings, or some struct that is lost in the midst of time. Make up your mind! Plan 9 was invented to get rid of this. 1. fcntl > Just like ioctl but for some different miscellaneous operations, because one miscelleny is not enough. 2. tuxcall > Linux put a web server in the kernel! To win a benchmark contest with Microsoft! It had it‘s own syscall! My enum tux_reactions are YUK! Don‘t worry though, it was a distro patch (thanks Red Hat!) and never made it upstream, so only the man page and reserved number survive to taunt you and remind you that the path of the righteous is beset by premature optmization! 3. iosetup > The Linux asynchronous IO syscalls are almost entirely useless! Almost nothing works! You have to use ODIRECT for a start. And then they still barely work! They have one use, benchmarking SSDs, to show what speed you could get if only there was a usable API. Want async IO in kernel? Use Windows! 4. stat, and its friends and relatives > Yes this one is useful, but can you find the data structure it uses? We have oldstat, oldfstat, ustat, oldlstat, statfs, fstatfs, stat, lstat, fstat, stat64, lstat64, fstat64, statfs64, fstatfs64, fstatat64 for stating files and links and filesystems in Linux. A new bunch will be along soon for Y2038. Simplify your life, use a BSD, where they cleaned up the mess as they did the cooking! Linux on 32 bit platforms is just sucky in comparison, and will get worse. And don't even look at MIPS, where the padding is wrong. 5. Linux on MIPS > Not a syscall, a whole implemntation of the Linux ABI. Unlike the lovely clean BSDs, Linux is different on each architecture, system calls randomly take arguments in different orders, and constants have different values, and there are special syscalls. But MIPS takes the biscuit, the whole packet of biscuits. It was made to be binary compatible with old SGI machines that don't even exist, and has more syscall ABIs than I have had hot dinners. Clean it up! Make a new sane MIPS ABI and deprecate the old ones, nothing like adding another variant. So annoying I think I threw out all my MIPS machines, each different. 6. inotify, fanotify and friends > Linux has no fewer than three file system change notification protocols. The first, dnotify hopped on ioctl‘s sidekick fcntl, while the two later ones, inotify and fanotify added a bunch more syscalls. You can use any of them, and they still will not provide the notification API you want for most applications. Most people use the second one, inotify and curse it. Did you know kqueue can do this on the BSDs? 7. personality > Oozing in personality, but we just don't get along. Basically obsolete, as the kernel can decide what kind of system emulation to do from binaries directly, it stays around with some use cases in persuading ./configure it is running on a 32 bit system. But it can turn off ASLR, and let the CVEs right into your system. We need less persoanlity! 8. gettimeofday > Still has an obsolete timezone value from an old times when people thought timezones should go all the way to the kernel. Now we know that your computer should not know. Set its clock to UTC. Do the timezones in the UI based on where the user is, not the computer. You should use clock_gettime now. Don't even talk to me about locales. This syscall is fast though, don't use it for benchmarking, its in the VDSO. 9. splice and tee > These, back in 2005 were a quite nice idea, although Linux said then “it is incomplete, the interfaces are ugly, and it will oops the system if anything goes wrong”. It won't oops your system now, but usage has not taken off. The nice idea from Linus was that a pipe is just a ring buffer in the kernel, that can have a more general API and use cases for performant code, but a decade on it hasn't really worked out. It was also supposed to be a more general sendfile, which in many ways was the successor of that Tux web server, but I think sendfile is still more widely used. 10. userfaultfd > Yes, I like file descriptors. Yes CRIU is kind of cool. But userspace handling page faults? Is nothing sacred? I get that you can do this badly with a SIGSEGV handler, but talk about lipstick on a pig. *** ###OpenBSD 6.0 on an iMac G3 from 1999 (http://www.increasinglyadequate.com/macppc.html) > A while ago I spent $50 for an iMac G3 (aka the iMac,1). This iconic model restored Apple's fortunes in the late '90s. Since the iMac G3 can still boot Mac OSes 8 and 9, I mostly use the machine to indulge a nostalgia for childhood schooldays spent poking at the operating system and playing Escape Velocity. But before I got around to that, I decided to try out the software that the previous owner had left on the machine. The antiquated OSX 10.2 install and 12 year old versions of Safari and Internet Explorer were too slow and old to use for anything. Updating to newer software was almost impossible; a later OSX is required to run the little PowerPC-compatible software still languishing in forgotten corners of the Internet. This got me thinking: could this machine be used, really used, nowadays? Lacking a newer OSX disc, I decided to try the most recent OpenBSD release. (And, since then, to re-try with each new OpenBSD release.) Below are the results of this experiment (plus a working xorg.conf file) and a few background notes. Background > This iMac is a Revision D iMac G3 in grape. It's part of the iMac,1 family of computers. This family includes all tray-loading iMac G3s. (Later iMac G3s had a slot-loading CD drive and different components.) Save for a slightly faster processor, a dedicated graphics card, and cosmetic tweaks to the case, my iMac is identical to the prior year's line-launching Bondi Blue iMac. My machine has had its memory upgraded from 32 MB to 320 MB. Thank Goodness. > The Revision D iMac G3 shipped with Mac OS 8.5. It can run up to Mac OS 9.2.2 or OSX 10.3.9. Other operating systems that tout support for the iMac,1 include NetBSD, OpenBSD, and a shrinking number of Linux distributions. > OpenBSD is simple (by design) and well-maintained. In contrast, NetBSD seems rather more complex and featureful, and I have heard grumbling that despite its reputation for portability, NetBSD really only works well on amd64. I'd test that assertion if OpenBSD's macppc installation instructions didn't seem much simpler than NetBSD's. Linux is even more complicated, although most distros are put together in a way that you can mostly ignore that complexity (until you can't). In the end I went with OpenBSD because I am familiar with it and because I like it. Installing OpenBSD on the iMac,1 > Installing OpenBSD on this iMac was simple. It's the same procedure as installing OpenBSD on an amd64 rig. You put in the installation disc; you tell the machine to boot from it; and then you answer a few prompts, most of which simply ask you to press enter. In this case, OpenBSD recognizes all machine's hardware just fine, including sound and networking, though I had a little trouble with video. > The OpenBSD documentation says video should just work and that an xorg.conf file isn't necessary. As such, it no longer ships with an xorg.conf file. Though that's never posed a problem on my other OpenBSD machines, it does here. Video doesn't work out of the box on my iMac,1. startx just blanks the screen. Fortunately, because the BSDs use a centralized development model where each operating system is stored in one repository, OpenBSD's website provides a web interface to the source code going back to the early days. I was able to find the last version of the sample xorg.conf that used to ship on macppc. With a little tweaking, I transformed that file into this one (https://www.increasinglyadequate.com/files/xorg.conf), with which video works just fine. Just drop it into your iMac's /etc/X11 directory. You'll also need to remember to set the machdep.allowaperture sysctl to 2 (e.g., as root run sysctl machdep.allowaperture=2), although the installer will do that automatically if you answer yes to the question about whether you plan to run X. > All that being said, video performance is pretty poor. I am either doing something wrong, or OpenBSD doesn't have accelerated video for this iMac, or this machine is just really old! I will discuss performance below. Running OpenBSD on the iMac,1 > The machine performs okay under OpenBSD. You can expect to ably run minimalistic software under minimalistic window managers. I tried dillo, mrxvt, and cmus under cwm and fvwm. Performance here was just fine. I also tried Firefox 26, 33, and 34 under fvwm and cwm. Firefox ran, but "modern," Javascript-heavy sites were an exercise in frustration; the 2015 version of CNN.com basically froze Firefox for 30 seconds or more. A lighter browser like dillo is doable. > You'll notice that I used the past-tense to talk about Firefox. Firefox currently doesn't build on PowerPC on OpenBSD. Neither does Chromium. Neither do a fair number of applications. But whatever -- there's still a lot of lighter applications available, and it's these you'll use day-to-day on a decades-old machine. > Lightweight window managers work okay, as you'd expect. You can even run heavier desktop environments, such as xfce, though you'll give up a lot of performance. > I ran the Ubench benchmark on this iMac and two more modern machines also running OpenBSD. The benchmark seems like an old one; I don't know how (if at all) it accounts for hardware changes in the past 13 years. That is, I don't know if the difference in score accurately measures the difference in real-world performance. Here are the results anyway: Conclusion > Except for when I check to see if OpenBSD still works, I run Mac OS9 on this rig. I have faster and better machines for running OpenBSD. If I didn't -- if this rig were, improbably, all I had left, and I was waiting on the rush delivery of something modern -- then I would use OpenBSD on my iMac,1. I'd have to stick to lightweight applications, but at least they'd be up-to-date and running on a simple, stable, OS. *** ##News Roundup ###34th Chaos Communication Congress Schedule (https://events.ccc.de/congress/2017/Fahrplan/index.html) Many talks are streamed live (http://streaming.media.ccc.de/34c3), a good mixture of english and german talks May contain DTraces of FreeBSD (https://events.ccc.de/congress/2017/Fahrplan/events/9196.html) Are all BSDs created equally? (https://events.ccc.de/congress/2017/Fahrplan/events/8968.html) library operating systems (https://events.ccc.de/congress/2017/Fahrplan/events/8949.html) Hardening Open Source Development (https://events.ccc.de/congress/2017/Fahrplan/events/9249.html) *** ###OpenBSD 6.2 + CDE (https://jamesdeagle.blogspot.co.uk/2017/12/openbsd-62-cde.html) > If you've noticed a disruption in the time-space continuum recently, it is likely because I have finally been able to compile and install the Common Desktop Environment (CDE) in a current and actively-developed operating system (OpenBSD 6.2 in this case). > This comes after so many attempts (across multiple platforms) that ended up with the build process prematurely stopping itself in its own tracks for a variety of infinitesimal reasons that were beyond my comprehension as a non-programmer, or when there was success it was not without some broken parts. As for the latter, I've been able to build CDE on OpenIndiana Hipster, but with an end product where I'm unable to change the color scheme in dtstyle (because "useColorObj" is set to "False"), with a default color scheme that is low-res and unpleasant. As for changing "useColorObj" to "True", I tried every recommended trick I could find online, but nothing worked. > My recent attempts at installing CDE on OpenBSD (version 6.1) saw the process stop due to a number of errors that are pure gibberish to these naive eyes. While disappointing, it was par for the course within my miserable experience with trying to build this particular desktop environment. As I wrote in this space in November 2015, in the course of explaining part of my imperitive for installing Solaris 10: > And so I have come to think of building the recently open-sourced CDE as being akin to a coffee mug I saw many years ago. One side of the mug read "Turn the mug to see how to keep an idiot busy." On the other side, it read "Turn the mug to see how to keep an idiot busy." I'm through feeling like an idiot, which is partially why I'm on this one-week journey with Solaris 10. > While I thoroughly enjoyed running Solaris 10 on my ThinkPad T61p, and felt a devilish thrill at using it out in the open at my local MacBook- and iPhone-infested Starbucks and causing general befuddlement and consternation among the occasional prying yoga mom, I never felt like I could do much with it beyond explore the SunOS 5.10 command line and watch YouTube videos. While still supported by its current corporate owner (whose name I don't even want to type), it is no longer actively developed and is thus little more than a retro toy. I hated the idea of installing anything else over it, but productivity beckoned and it was time to tearfully and reluctantly drag myself off the dance floor. > In any case, just last week I noticed that the Sourceforge page for the OpenBSD build had some 6.2-specific notes by way of a series of four patches, and so I decided 'what the heck, let's give this puppy another whirl'. After an initial abortive attempt at a build, I surmised that I hadn't applied the four patches correctly. A day or two later, I took a deep breath and tried again, this time resolving to not proceed with the time make World build command until I could see some sign of a successful patch process. (This time around, I downloaded the patches and moved them into the directory containing the CDE makefiles, and issued each patch command as patch Once I had the thing up and running, and with a mind bursting with fruit flavor, I started messing about. The first order of business was to create a custom color scheme modelled after the default color scheme in UnixWare. (Despite any baggage that system carries from its previous ownership under SCO, I adored the aesthetics of UnixWare 7.1.4 two years ago when I installed the free one month trial version on my ThinkPad. For reasons that escape me now, I named my newly-created color scheme in honor of UnixWare 7.1.3.) > Like a proud papa, I immediately tweeted the above screenshot and risked irritating a Linux kid or two in the process, given SCO's anti-climatic anti-Linux patent trolling from way back when. (I'm not out to irritate penguinistas, I just sure like this color scheme.) Final Thoughts > It may look a little clunky at first, and may be a little bling-challenged, but the more I use CDE and adapt to it, the more it feels like an extension of my brain. Perhaps this is because it has a lot zip and behaves in a consistent and coherent manner. (I don't want to go too much further down that road here, as OSnews's Thom Holwerda already gave a good rundown about ten years ago.) > Now that I have succesfully paired my absolute favorite operating system with a desktop environment that has exerted an intense gravitational hold on me for many, many years, I don't anticipate distrohopping any time soon. And as I attain a more advanced knowledge of CDE, I'll be chronicling any new discoveries here for the sake of anyone following me from behind as I feel my way around this darkened room. *** ###devmatch(8) added to FreeBSD HEAD (https://www.mail-archive.com/svn-src-all@freebsd.org/msg154719.html) ``` Log: Match unattached devices on the system to potential kernel modules. devmatch(8) matchs up devices in the system device tree with drivers that may match them. For each unattached device in the system, it tries to find matching PNP info in the linker hints and prints modules to load to claim the devices. In --unbound mode, devmatch can look for drivers that have attached to devices in the device tree and have plug and play information, but for which no PNP info exists. This helps find drivers that haven't been converted yet that are in use on this system. In addition, the ability to dump out linker.hints is provided. Future commits will add hooks to devd.conf and rc.d to fully automate using this information. Added: head/usr.sbin/devmatch/ head/usr.sbin/devmatch/Makefile (contents, props changed) head/usr.sbin/devmatch/devmatch.8 (contents, props changed) head/usr.sbin/devmatch/devmatch.c (contents, props changed) Modified: head/usr.sbin/Makefile Modified: head/usr.sbin/Makefile ``` + Oh, you naughty committers: :-) https://www.mail-archive.com/svn-src-all@freebsd.org/msg154720.html Beastie Bits New FreeBSD Journal issue: Monitoring and Metrics (https://www.freebsdfoundation.org/journal/) OpenBSD Engine Mix available on GOG.com (https://www.gog.com/mix/openbsd_engine_available) OpenBSD Foundation reached their 2017 fundraising goal (http://www.openbsdfoundation.org/campaign2017.html) TrueOS 17.12 Review – An Easy BSD (https://www.youtube.com/watch?v=nKr1GCsV-gA) LibreSSL 2.6.4 Released (https://bsdsec.net/articles/libressl-2-6-4-released-fixed) *** ##Feedback/Questions Mike - BSD 217 & Winning over Linux Users (http://dpaste.com/3AB7J4P#wrap) JLR - Boot Environments Broken? (http://dpaste.com/2K0ZDH9#wrap) Kevr - ZFS question and suggestion (http://dpaste.com/04MXA5P#wrap) Ivan - FreeBSD read cache - ZFS (http://dpaste.com/1P9ETGQ#wrap) ***

We recap our devsummit experiences at BSDCambridge, share why memcmp is more complicated than expected, explore Docker on FreeBSD, and we look at a retro terminal. This episode was brought to you by Headlines BSDCam recap (https://wiki.freebsd.org/DevSummit/201708) The 2017 Cambridge DevSummit took place from 2-4 August 2017. The event took place over three days including a formal dinner at St John's College, and was attended by 55 registered developers and guests. Prior to the start of the conference, we had a doc hacking lounge, the computer lab provided a room where we could meet and try to spend some time on documentation. Sevan walked two interested people through the process of creating a documentation patch and submitting it for the first time. In the process, found ways to improve the documentation on how to write documentation. The event is run "un-conference style" in that we brainstorm the actual session schedule on the first morning, with a focus on interactive topics that reflect the interests and exploit the knowledge of the attendees. The idea is to maximize the amount of discussion and decisions that can be made while we are all in the same room The first morning, we all gather in the slightly too small, and even more slightly under air conditioned FW11 classroom. We go around the room introducing ourselves, and listing a few topics we would be interested in discussing. Eventually the whiteboard is full of topics, with various numbers of ticks beside them to indicate the number of interested people There are breakout rooms of all sizes, so even topics with only a small group of interested folks can get a lot accomplished The most difficult is trying to schedule the sessions, as there is much overlap and people usually want to be in concurrent sessions, or someone's schedule means they won't be available that day, etc. This years working groups: Toolchain (Compilers, Linkers, External Toolchain, Static analysis and sanitizers) Virtualization (bhyve, xen, jails, docker) Transport (TCP) and Network Performance Security and mitigations (W^X, noexec stack, CFI, ASLR, KASLR, Safe Stack, etc) Testing (Status, What to test, How to test, QA for releases) Capsicum (Automation with LLVM etc, Casper, Namespacing, “Services”, capsh) Desktop / WiFi (drm-next, drivers, resume, power, installer, desktop, OOB Experience) Tracing (Blackbox, DTrace, KTR, ptrace, truss, hardware tracing) Packaging and Packaged Base (Sets, Kernels, Ports & flavours, sub-packages, privlib) Architectural Security Features (CPU Features: SGX, PXN/PAN, Pointer Authentication, AMD Memory Encryption, Libcrunch, RISC-V, CheriABI) Architectures and Embedded systems (RISC-V, ARM, ARM64, MIPS(64), SPARC64) Teaching (Audiences, Objectives, Targets, Material, future directions) Provisioning and Management Tools (CfgMgmt tools, Image building, VM/bhyve orchestration, Preconfigured VMs for testing, Wishlist) Storage (ZFS status update, ZFS encryption infrastructure, ZFS Zero Copy / Sendfile, Acceleration of checksums and raidz parity calculations, sesutil, mpsutil) And that wasn't everything. We then had a series of short talklets: Enhancing and replacing mmap() SDIO support eBPF support for FreeBSD Tracing + Virtualization Practical DMA Attack Protection On Thursday night there was a special dinner at St John's College Overall it was a great DevSummit, and I even managed to get some of the work assigned to me finished. Shortly I will commit an update to the boot loader menu that will automatically populate the kernel selection menu with the automatically detected list of installed kernels. The list is also properly refreshed when you switch boot environments. *** Hosts/BSD – for when you need to run your BSD inside a penguin (https://wiki.qemu.org/index.php/Hosts/BSD) This wiki provides details on how to run each of the various BSDs under QEMU The target audience is Linux developers looking to test their apps etc under BSD The wiki is in need of some love, there are some option questions, and it lacks some polish There are instructions on building qemu from source, but it should likely mention the qemu-devel port There should probably also be instructions on using other architectures, like ARM/MIPS etc If you have used QEMU, or would like to spend the time to learn how, please help update this wiki *** memcmp -- more complicated than you might expect (http://trust-in-soft.com/memcmp-requires-pointers-to-fully-valid-buffers/) “A suspicious pattern in open-source software” One bug recently found by John using tis-interpreter on a widely used open-source library involved the comparison of strings with memcmp. The unexpected condition was that memcmp was, in one case, called with a pointer to a buffer shorter than the length passed as third argument, breaking one of the two symmetrical pre-conditions in the function's ACSL contract A reason that may have made this use of memcmp look okay to the developer is that the buffers being passed to it always differed before the end of the buffers were reached. a memcmp implementation based on stopping as soon as a difference is found, would not have caused any out-of-bounds read access The first question raised was whether the pattern memcmp("a", "bc", 3) was problematic according to the letter of the C standard. If it was, the second question was whether the busy maintainer of one of the Open Source packages that make the Internet tick should be bothered with a bug report. I would like to be able to say that memcmp's ACSL contract was the product of careful deliberation, but unfortunately this is not the case: many standard function contracts were written quickly in order to get most of the standard library covered, and have not been tested by time. Anyway, upon proofreading the relevant clause in the C11 standard, my feeling was that the ACSL formalization was, in this particular case, right, and that it was undefined behavior to pass as memcmp argument a buffer that wasn't fully valid, even if the implementation sort-of needs to read the buffer's characters in order for the purpose of finding the first mismatch. The post then goes on to look at the memcmp code in glibc There are two distinct optimizations for long buffers, one that applies when both buffers start at the same offset modulo the word size, memcmpcommonalignment, and one that applies when they don't, memcmpnotcommonalignment. The function memcmpcommonalignment is relatively well-behaved: it reads from the two buffers aligned word by aligned word, and thus reads the entire words that contain differing bytes. If the caller passed buffers that aren't valid after the differing byte, this amounts to reading out of bounds, but this sort of out-of-bounds access is not detected by the typical MMU, which works at the scale of the page. The “notcommon_alignment” case, however, tells a different story. When passed the carefully (mis-)aligned buffers t1 and (char*)t2+1, although these buffers differ in the 8th byte, Glibc's implementation of memcmp reads 8 bytes beyond the end of t1. By making the 16th byte differ instead of the 8th one, it is also possible to make Glibc's implementation of memcmp read 16 bytes beyond the end of t1. In conclusion, yes, some implementations of memcmp will crash when invoked with buffers that aren't valid for the full length, even if they differ early. The circumstances are rare (probably the reason this bug was still there to be found in a library that had already been tested with all the available techniques) but outside the programmer's control. The pattern described in this post should be reported as a bug when found. It is interesting to read the detailed analysis of a bug in such a basic libc feature *** News Roundup Docker on FreeBSD (http://daemon-notes.com/articles/network/docker) There are two approaches to running Docker on FreeBSD. First one was created back in 2015 and it was a native port of Docker engine to FreeBSD. It was an ambitious project but nobody stepped forward to continuously port the never-ending flow of upstream code to FreeBSD. So the port still exists (sysutils/docker-freebsd) but it wasn't updated since 2015 and it is Docker v1 (it is v17 as of 2017). The other approach is to use official way of running Docker on platforms other than Linux. Well, somewhat official as Docker still does not support FreeBSD as a host officially. This is docker-machine tool which in turn will use VirtualBox to run a virtual machine with Linux and Docker engine. docker utility on the host will communicate with the engine inside VB where all the work will be done. This article describes what needs to be done to start using it. Before we begin you need VirtualBox installed. Do not skip adding /boot/loader.conf and /etc/rc.conf lines mentioned on that page. You won't need user inteface or anything, docker-machine will do all the work, just make sure VirtualBox is present and ready to be used. `pkg install docker docker-machine docker-compose' Docker will store its stuff in ~/.docker. You might not want the virtual machine image files to live in your home, in this case just create a symlink: mkdir ~/.docker ln -s /storage/docker ~/.docker/machine docker-machine create --driver virtualbox --virtualbox-memory 2048 --virtualbox-cpu-count 2 --virtualbox-disk-size 102400 --virtualbox-hostonly-cidr "10.2.1.1/24" docker1 Here's the example. We are creating machine named docker1. It is using VirtualBox driver, the vm has 2G of memory, 2 cores and 100G of disk space. docker-machine setups VirtualBox to use host-only network adapter (it will create vboxnet0 interface on the host automatically) and we are instructing it to use 10.2.1.1/24 as the address of this adapter — change it to what suits your needs or omit this flag (default is 192.168.99.1/24). And basically that is all. Check if it is running: docker-machine ls If you do open VirtualBox interface you will find a virtual machine named docker1 running. You can start/stop/whatever your machine using docker-machine utility. Here's how you can connect to the machine: docker utility by default tries to talk to Docker engine running on the same host. However with specific environment variables you can instruct it to talk to other host. docker-machine can export these variables for you. eval docker-machine env docker1 docker run hello-world There was quite a bit of discussion about docker at the FreeBSD developers summit in Cambridge during the first week of August. Two docker developers who had worked on the Mac OS X port, one of whom is an OpenBSD advocate, explained how docker has evolved, and the linux-isms have been abstracted away such that a truly native docker solution for FreeBSD can be built and maintained with a lot less headache than before I look forward to seeing if we can't make that happen *** The POSIX Shell And Utilities (http://shellhaters.org/) The POSIX Shell And Utilities Compiled for The Shell Hater's Handbook *** PostgreSQL – logging to a file (http://dan.langille.org/2017/07/31/postgresql-logging-to-a-file/) These steps were carried out on FreeBSD 11.0 with PostgreSQL 9.6 (two of my favorite tools). I like logging. I like logging PostgreSQL. With logs, you can see what happened. Without, you can only guess. Setting up logging for PostgreSQL involves several parts, each of which must be completed or else I don't get what I want. This is not a criticism of PostgreSQL. It's a feature. I am documenting this because each time I configure a new PostgreSQL instance, it takes me more than one iteration to get it working. The goal: this post lets both you and me get it right the first time. The parts include: + Telling PostgreSQL to log via syslog + Telling FreeBSD to local postgres to /var/log/postgres.log (my preference). + Telling PostgreSQL the things you want logged. + Changes to postgresql.conf The file location varies with the version installed. For PostgreSQL 9.6 on FreeBSD, the file is /var/db/postgres/data96/postgresql.conf (adjust 96 according to the version installed). I made these changes to that file. log_destination = 'syslog' log_min_messages = notice log_min_error_statement = notice log_checkpoints = on log_lock_waits = on log_timezone = 'UTC' By default, PostgreSQL logs to the local0 facility and is controlled by the syslog_facility in postgresql.conf. This will be used in syslog.conf (see the next section of this post). The above mentioned changes require a reload: service postgresql reload Changes to /etc/syslog.conf Now that we have PostgreSQL logging to syslog, we want to tell syslog where to put those messages. I changed this line in /etc/syslog.conf:*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages With .notice pulling in some local0 messages, adding local0.none to the line will free the messages up for later use in the configuration file. Otherwise, the PostgreSQL messages will be in /var/log/messages. The changed line is: `.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local0.none /var/log/messages Then, to get the messages into my preferred location, I added this to the file: local0.* /var/log/postgresql.log` Log file rotation For rotating my log file, I added a new file: /usr/local/etc/newsyslog.conf.d/postgresql96 /var/log/postgresql.log pgsql:wheel 640 7 * $D0 GB /var/db/postgres/data96/postmaster.pid 30 Before restarting syslog, I did this, so the destination file existed. This isn't always/strictly necessary, but because the ownership is not chown root:wheel, I do it to get that part set. touch /var/log/postgresql.log chown pgsql:wheel Restarting syslog: sudo kill -HUP `sudo cat /var/run/syslog.pid ` That's it Now you should see PostgreSQL logging in /var/log/postgresql.log. mandoc-1.14.2 released (http://undeadly.org/cgi?action=article&sid=20170729122350) i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ (http://mandoc.bsd.lv/). ```From: Ingo Schwarze schwarze@usta.de Date: Fri, 28 Jul 2017 20:12:44 +0200 To: discuss@mandoc.bsd.lv Subject: mandoc-1.14.2 released Hi, i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ . All downstream maintainers are encouraged to update their ports and packages from 1.14.1 to 1.14.2. Mandoc 1.14.2 is a feature release introducing: a new -Tmarkdown output mode anchors for deep linking into -Thtml manual pages a superset of the functionality of the former mdoclint(1) utility a new -Wstyle message level with several new messages automatic line breaking inside individual tbl(7) cells a rewrite of the eqn(7) lexer, and some eqn(7) rendering improvements support for many additional low-level roff(7) features and various smaller features and bug fixes. For more details, see: http://mandoc.bsd.lv/NEWS With the improved mandoc features, only twenty-five out of the ten thousand software packages in the OpenBSD ports tree still need groff to format their manual pages. Since the project has been called "mandoc" rather than "mdocml" for several years now, the website, the distribution tarball, and the source extraction directory are now also called "mandoc" rather than "mdocml". The release was tested on the following systems: + OpenBSD-current and OpenBSD-stable + NetBSD-current + illumos + Debian Linux + Void Linux x86_64 glibc and musl + Crux Linux + SunOS 5.11.2, 5.10, and 5.9 As before, catman(8) and the regression suite cannot be used on SunOS 5.10 and SunOS 5.9. A big thanks to everybody who provided patches, bug reports, feature suggestions, advice, and help with testing! Yours, Ingo``` Beastie Bits A good looking terminal emulator which mimics the old cathode display. Available in x11/cool-retro-terminal (https://github.com/Swordfish90/cool-retro-term) Milestone Complete! OpenRC conversion (https://www.trueos.org/blog/milestone-complete-openrc-conversion/) Healthy developer interaction between FreeBSD and IllumOS re: mdb (https://illumos.topicbox.com/groups/developer/discussions/T5eae6079331c4df4) Large Batch of Kernel Errata Patches Released (http://undeadly.org/cgi?action=article&sid=20170804053102) opnsense 17.7 released (https://opnsense.org/opnsense-17-7-released/) Twitter Co-Founder and CEO states “FreeBSD rules them all” (https://twitter.com/jack/status/892605692317650944) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Dominik - Monitoring Software (http://dpaste.com/08971FQ) Darren - Wonderful Awk (http://dpaste.com/0YCS4DN) Andrew - Thanks (http://dpaste.com/0ZREKTV) Jens - Migration Questions (http://dpaste.com/1GVZNWN) ***

In this episode, we clear up the myth about scrub of death, look at Wayland and Weston on FreeBSD, Intel QuickAssist is here, and we check out OpenSMTP on OpenBSD. This episode was brought to you by Headlines Matt Ahrens answers questions about the “Scrub of Death” In working on the breakdown of that ZFS article last week, Matt Ahrens contacted me and provided some answers he has given to questions in the past, allowing me to answer them using HIS exact words. “ZFS has an operation, called SCRUB, that is used to check all data in the pool and recover any data that is incorrect. However, if a bug which make errors on the pool persist (for example, a system with bad non-ecc RAM) then SCRUB can cause damage to a pool instead of recover it. I heard it called the “SCRUB of death” somewhere. Therefore, as far as I understand, using SCRUB without ECC memory is dangerous.” > I don't believe that is accurate. What is the proposed mechanism by which scrub can corrupt a lot of data, with non-ECC memory? > ZFS repairs bad data by writing known good data to the bad location on disk. The checksum of the data has to verify correctly for it to be considered "good". An undetected memory error could change the in-memory checksum or data, causing ZFS to incorrectly think that the data on disk doesn't match the checksum. In that case, ZFS would attempt to repair the data by first re-reading the same offset on disk, and then reading from any other available copies of the data (e.g. mirrors, ditto blocks, or RAIDZ reconstruction). If any of these attempts results in data that matches the checksum, then the data will be written on top of the (supposed) bad data. If the data was actually good, then overwriting it with the same good data doesn't hurt anything. > Let's look at what will happen with 3 types of errors with non-ECC memory: > 1. Rare, random errors (e.g. particle strikes - say, less than one error per GB per second). If ZFS finds data that matches the checksum, then we know that we have the correct data (at least at that point in time, with probability 1-1/2^256). If there are a lot of memory errors happening at a high rate, or if the in-memory checksum was corrupt, then ZFS won't be able to find a good copy of the data , so it won't do a repair write. It's possible that the correctly-checksummed data is later corrupted in memory, before the repair write. However, the window of vulnerability is very very small - on the order of milliseconds between when the checksum is verified, and when the write to disk completes. It is implausible that this tiny window of memory vulnerability would be hit repeatedly. > 2. Memory that pretty much never does the right thing. (e.g. huge rate of particle strikes, all memory always reads 0, etc). In this case, critical parts of kernel memory (e.g. instructions) will be immediately corrupted, causing the system to panic and not be able to boot again. > 3. One or a few memory locations have "stuck bits", which always read 0 (or always read 1). This is the scenario discussed in the message which (I believe) originally started the "Scrub of Death" myth: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ This assumes that we read in some data from disk to a memory location with a stuck bit, "correct" that same bad memory location by overwriting the memory with the correct data, and then we write the bad memory location to disk. However, ZFS doesn't do that. (It seems the author thinks that ZFS uses parity, which it only does when using RAID-Z. Even with RAID-Z, we also verify the checksum, and we don't overwrite the bad memory location.) > Here's what ZFS will actually do in this scenario: If ZFS reads data from disk into a memory location with a stuck bit, it will detect a checksum mismatch and try to find a good copy of the data to repair the "bad" disk. ZFS will allocate a new, different memory location to read a 2nd copy of the data, e.g. from the other side of a mirror (this happens near the end of dslscanscrub_cb()). If the new memory location also has a stuck bit, then its checksum will also fail, so we won't use it to repair the "bad" disk. If the checksum of the 2nd copy of the data is correct, then we will write it to the "bad" disk. This write is unnecessary, because the "bad" disk is not really bad, but it is overwriting the good data with the same good data. > I believe that this misunderstanding stems from the idea that ZFS fixes bad data by overwriting it in place with good data. In reality, ZFS overwrites the location on disk, using a different memory location for each read from disk. The "Scrub of Death" myth assumes that ZFS overwrites the location in memory, which it doesn't do. > In summary, there's no plausible scenario where ZFS would amplify a small number of memory errors, causing a "scrub of death". Additionally, compared to other filesystems, ZFS checksums provide some additional protection against bad memory. “Is it true that ZFS verifies the checksum of every block on every read from disk?” > Yes “And if that block is incorrect, that ZFS will repair it?” > Yes “If yes, is it possible set options or flag for change that behavior? For example, I would like for ZFS to verify checksums during any read, but not change anything and only report about issues if it appears. Is it possible?” > There isn't any built-in flag for doing that. It wouldn't be hard to add one though. If you just wanted to verify data, without attempting to correct it, you could read or scan the data with the pool was imported read-only “If using a mirror, when a file is read, is it fully read and verified from both sides of the mirror?” > No, for performance purposes, each block is read from only one side of the mirror (assuming there is no checksum error). “What is the difference between a scrub and copying every file to /dev/null?” > That won't check all copies of the file (e.g. it won't check both sides of the mirror). *** Wayland, and Weston, and FreeBSD - Oh My! (https://euroquis.nl/bobulate/?p=1617) KDE's CI system for FreeBSD (that is, what upstream runs to continuously test KDE git code on the FreeBSD platform) is missing some bits and failing some tests because of Wayland. Or rather, because FreeBSD now has Wayland, but not Qt5-Wayland, and no Weston either (the reference implementation of a Wayland compositor). Today I went hunting for the bits and pieces needed to make that happen. Fortunately, all the heavy lifting has already been done: there is a Weston port prepared and there was a Qt5-Wayland port well-hidden in the Area51 plasma5/ branch. I have taken the liberty of pulling them into the Area51 repository as branch qtwayland. That way we can nudge Weston forward, and/or push Qt5-Wayland in separately. Nicest from a testing perspective is probably doing both at the same time. I picked a random “Hello World” Wayland tutorial and also built a minimal Qt program (using QMessageBox::question, my favorite function to hate right now, because of its i18n characteristics). Then, setting XDGRUNTIMEDIR to /tmp/xdg, I could start Weston (as an X11 client), wayland-hello (as a Wayland client, displaying in Weston) and qt-hello (as either an X11 client, or as a Wayland client). So this gives users of Area51 (while shuffling branches, granted) a modern desktop and modern display capabilities. Oh my! It will take a few days for this to trickle up and/or down so that the CI can benefit and we can make sure that KWin's tests all work on FreeBSD, but it's another good step towards tight CI and another small step towards KDE Plasma 5 on the desktop on FreeBSD. pkgsrcCon 2017 report (https://blog.netbsd.org/tnf/entry/pkgsrccon_2017_report) This years pkgsrcCon returned to London once again. It was last held in London back in 2014. The 2014 con was the first pkgsrcCon I attended, I had been working on Darwin/PowerPC fixes for some months and presented on the progress I'd made with a 12" G4 PowerBook. I took away a G4 Mac Mini that day to help spare the PowerBook for use and dedicate a machine for build and testing. The offer of PowerPC hardware donations was repeated at this years con, thanks to jperkin@ who showed up with a backpack full of Mac Minis (more on that later). Since 2014 we have held cons in Berlin (2015) & Krakow (2016). In Krakow we had talks about a wide range of projects over 2 days, from Haiku Ports to Common Lisp to midipix (building native PE binaries for Windows) and back to the BSDs. I was very pleased to continue the theme of a diverse program this year. Aside from pkgsrc and NetBSD, we had talks about FreeBSD, OpenBSD, Slackware Linux, and Plan 9. Things began with a pub gathering on the Friday for the pre-con social, we hung out and chatted till almost midnight on a wide range of topics, such as supporting a system using NFS on MS-DOS, the origins of pdksh, corporate IT, culture and many other topics. On parting I was asked about the starting time on Saturday as there was some conflicting information. I learnt that the registration email had stated a later start than I had scheduled for & advertised on the website, by 30 minutes. Lesson learnt: register for your own event! Not a problem, I still needed to setup a webpage for the live video stream, I could do both when I got back. With some trimming here and there I had a new schedule, I posted that to the pkgsrcCon website and moved to trying to setup a basic web page which contained a snippet of javascript to play a live video stream from Scale Engine. 2+ hours later, it was pointed out that the XSS protection headers on pkgsrc.org breaks the functionality. Thanks to jmcneill@ for debugging and providing a working page. Saturday started off with Giovanni Bechis speaking about pledge in OpenBSD and adding support to various packages in their ports tree, alnsn@ then spoke about installing packages from a repo hosted on the Tor network. After a quick coffee break we were back to hear Charles Forsyth speak about how Plan 9 and Inferno dealt with portability, building software and the problem which are avoided by the environment there. This was followed by a very energetic rant by David Spencer from the Slackbuilds project on packaging 3rd party software. Slackbuilds is a packaging system for Slackware Linux, which was inspired by FreeBSD ports. For the first slot after lunch, agc@ gave a talk on the early history of pkgsrc followed by Thomas Merkel on using vagrant to test pkgsrc changes with ease, locally, using vagrant. khorben@ covered his work on adding security to pkgsrc and bsiegert@ covered the benefits of performing our bulk builds in the cloud and the challenges we currently face. My talk was about some topics and ideas which had inspired me or caught my attention, and how it could maybe apply to my work.The title of the talk was taken from the name of Andrew Weatherall's Saint Etienne remix, possibly referring to two different styles of track (dub & vocal) merged into one or something else. I meant it in terms of applicability of thoughts and ideas. After me, agc@ gave a second talk on the evolution of the Netflix Open Connect appliance which runs FreeBSD and Vsevolod Stakhov wrapped up the day with a talk about the technical implementation details of the successor to pkgtools in FreeBSD, called pkg, and how it could be of benefit for pkgsrc. For day 2 we gathered for a hack day at the London Hack Space. I had burn't some some CD of the most recent macppc builds of NetBSD 8.0BETA and -current to install and upgrade Mac Minis. I setup the donated G4 minis for everyone in a dual-boot configuration and moved on to taking apart my MacBook Air to inspect the wifi adapter as I wanted to replace it with something which works on FreeBSD. It was not clear from the ifixit teardown photos of cards size, it seemed like a normal mini-PCIe card but it turned out to be far smaller. Thomas had also had the same card in his and we are not alone. Thomas has started putting together a driver for the Broadcom card, the project is still in its early days and lacks support for encrypted networks but hopefully it will appear on review.freebsd.org in the future. weidi@ worked on fixing SunOS bugs in various packages and later in the night we setup a NetBSD/macppc bulk build environment together on his Mac Mini. Thomas setup an OpenGrock instance to index the source code of all the software available for packaging in pkgsrc. This helps make the evaluation of changes easier and the scope of impact a little quicker without having to run through a potentially lengthy bulk build with a change in mind to realise the impact. bsiegert@ cleared his ticket and email backlog for pkgsrc and alnsn@ got NetBSD/evbmips64-eb booting on his EdgeRouter Lite. On Monday we reconvened at the Hack Space again and worked some more. I started putting together the talks page with the details from Saturday and the the slides which I had received, in preparation for the videos which would come later in the week. By 3pm pkgsrcCon was over. I was pretty exhausted but really pleased to have had a few days of techie fun. Many thanks to The NetBSD Foundation for purchasing a camera to use for streaming the event and a speedy response all round by the board. The Open Source Specialist Group at BCS, The Chartered Institute for IT and the London Hack Space for hosting us. Scale Engine for providing streaming facility. weidi@ for hosting the recorded videos. Allan Jude for pointers, Jared McNeill for debugging, NYCBUG and Patrick McEvoy for tips on streaming, the attendees and speakers. This year we had speakers from USA, Italy, Germany and London E2. Looking forward to pkgsrcCon 2018! The videos and slides are available here (http://www.pkgsrc.org/pkgsrcCon/2017/talks.html) and the Internet Archive (http://archive.org/details/pkgsrcCon-2017). News Roundup QuickAssist Driver for FreeBSD is here and pfSense Support Coming (https://www.servethehome.com/quickassist-driver-freebsd-pfsupport-coming/) This week we have something that STH readers will be excited about. Before I started writing for STH, I was a reader and had been longing for QuickAssist support ever since STH's first Rangeley article over three and a half years ago. It was clear from the get-go that Rangeley was going to be the preeminent firewall appliance platform of its day. The scope of products that were impacted by the Intel Atom C2000 series bug showed us it was indeed. For my personal firewalls, I use pfSense on that Rangeley platform so I have been waiting to use QuickAssist with my hardware for almost an entire product generation. + New Hardware and QuickAssist Incoming to pfSense (Finally) pfSense (and a few other firewalls) are based on FreeBSD. FreeBSD tends to lag driver support behind mainstream Linux but it is popular for embedded security appliances. While STH is the only site to have done QuickAssist benchmarks for OpenSSL and IPSec VPNs pre-Skylake, we expect more platforms to use it now that the new Intel Xeon Scalable Processor Family is out. With the Xeon Scalable platforms, the “Lewisburg” PCH has QuickAssist options of up to 100Gbps, or 2.5x faster than the previous generation add-in cards we tested (40Gbps.) We now have more and better hardware for QAT, but we were still devoid of a viable FreeBSD QAT driver from Intel. That has changed. Our Intel Xeon Scalable Processor Family (Skylake-SP) Launch Coverage Central has been the focus of the STH team's attention this week. There was another important update from Intel that got buried, a publicly available Intel QuickAssist driver for FreeBSD. You can find the driver on 01.org here dated July 12, 2017. Drivers are great, but we still need support to be enabled in the OS and at the application layer. Patrick forwarded me this tweet from Jim Thompson (lead at Netgate the company behind pfSense): The Netgate team has been a key company pushing QuickAssist appliances in the market, usually based on Linux. To see that QAT is coming to FreeBSD and that they were working to integrate into “pfSense soon” is more than welcome. For STH readers, get ready. It appears to be actually and finally happening. QuickAssist on FreeBSD and pfSense OpenBSD on the Huawei MateBook X (https://jcs.org/2017/07/14/matebook) The Huawei MateBook X is a high-quality 13" ultra-thin laptop with a fanless Core i5 processor. It is obviously biting the design of the Apple 12" MacBook, but it does have some notable improvements such as a slightly larger screen, a more usable keyboard with adequate key travel, and 2 USB-C ports. It also uses more standard PC components than the MacBook, such as a PS/2-connected keyboard, removable m.2 WiFi card, etc., so its OpenBSD compatibility is quite good. In contrast to the Xiaomi Mi Air, the MateBook is actually sold (2) in the US and comes with a full warranty and much higher build quality (though at twice the price). It is offered in the US in a "space gray" color for the Core i5 model and a gold color for the Core i7. The fanless Core i5 processor feels snappy and doesn't get warm during normal usage on OpenBSD. Doing a make -j4 build at full CPU speed does cause the laptop to get warm, though the palmrest maintains a usable temperature. The chassis is all aluminum and has excellent rigidity in the keyboard area. The 13.0" 2160x1440 glossy IPS "Gorilla glass" screen has a very small bezel and its hinge is properly weighted to allow opening the lid with one hand. There is no wobble in the screen when open, even when jostling the desk that the laptop sits on. It has a reported brightness of 350 nits. I did not experience any of the UEFI boot variable problems that I did with the Xiaomi, and the MateBook booted quickly into OpenBSD after re-initializing the GPT table during installation. OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot (https://blog.cagedmonster.net/opensmtpd-under-openbsd-with-ssl-virtualusers-dovecot/) During the 2013 AsiaBSDCon, the team of OpenBSD presented its mail solution named OpenSMTPD. Developed by the OpenBSD team, we find the so much appreciated philosophy of its developers : security, simplicity / clarity and advanced features. Basic configuration : OpenSMTPD is installed by default, we can immediately start with a simple configuration. > We listen on our interfaces, we specify the path of our aliases file so we can manage redirections. > Mails will be delivered for the domain cagedmonster.net to mbox (the local users mailbox), same for the aliases. > Finally, we accept to relay local mails exclusively. > We can now enable smtpd at system startup and start the daemon. Advanced configuration including TLS : You can use SSL with : A self-signed certificate (which will not be trusted) or a certificate generated by a trusted authority. LetsEncrypt uses Certbot to generated your certificate. You can check this page for further informations. Let's focus on the first. Generation of the certificate : We fix the permissions : We edit the config file : > We have a mail server with SSL, it's time to configure our IMAP server, Dovecot, and manage the creation of virtual users. Dovecot setup, and creation of Virtual Users : We will use the package system of OpenBSD, so please check the configuration of your /etc/pkg.conf file. Enable the service at system startup : Setup the Virtual Users structure : Adding the passwd table for smtpd : Modification of the OpenSMTPD configuration : We declare the files used for our Virtual Accounts, we include SSL, and we configure mails delivery via the Dovecot lmtp socket. We'll create our user lina@cagedmonster.net and set its password. Configure SSL Configure dovecot.conf Configure mail.con Configure login.conf : Make sure that the value of openfiles-cur in /etc/login.conf is equal or superior of 1000 ! Starting Dovecot *** OpenSMTPD and Dovecot under OpenBSD with MySQL support and SPAMD (https://blog.cagedmonster.net/opensmtpd-and-dovecot-under-openbsd-with-mysql-support-and-spamd/) This article is the continuation of my previous tutorial OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot. We'll use the same configuration and add some features so we can : Use our domains, aliases, virtual users with a MySQL database (MariaDB under OpenBSD). Deploy SPAMD with OpenSMTPD for a strong antispam solution. + Setup of the MySQL support for OpenSMTPD & Dovecot + We create our SQL database named « smtpd » + We create our SQL user « opensmtpd » we give him the privileges on our SQL database and we set its password + We create the structure of our SQL database + We generate our password with Blowfish (remember it's OpenBSD !) for our users + We create our tables and we include our datas + We push everything to our database + Time to configure OpenSMTPD + We create our mysql.conf file and configure it + Configuration of Dovecot.conf + Configuration of auth-sql.conf.ext + Configuration of dovecot-sql.conf.ext + Restart our services OpenSMTPD & SPAMD : SPAMD is a service simulating a fake SMTP server and relying on strict compliance with RFC to determine whether the server delivering a mail is a spammer or not. + Configuration of SPAMD : + Enable SPAMD & SPAMLOGD at system startup : + Configuration of SPAMD flags + Configuration of PacketFilter + Configuration of SPAMD + Start SPAMD & SPAMLOGD Running a TOR relay on FreeBSD (https://networkingbsdblog.wordpress.com/2017/07/14/freebsd-tor-relay-using-priveledge-seperation/) There are 2 main steps to getting a TOR relay working on FreeBSD: Installing and configuring Tor Using an edge router to do port translation In my case I wanted TOR to run it's services on ports 80 and 443 but any port under 1024 requires root access in UNIX systems. +So I used port mapping on my router to map the ports. +Begin by installing TOR and ARM from: /usr/ports/security/tor/ /usr/ports/security/arm/ Arm is the Anonymizing Relay Monitor: https://www.torproject.org/projects/arm.html.en It provides useful monitoring graph and can be used to configure the torrc file. Next step edit the torrc file (see Blog article for the edit) It is handy to add the following lines to /etc/services so you can more easily modify your pf configuration. torproxy 9050/tcp #torsocks torOR 9090/tcp #torOR torDIR 9099/tcp #torDIR To allow TOR services my pf.conf has the following lines: # interfaces lan_if=”re0″ wifi_if=”wlan0″ interfaces=”{wlan0,re0}” tcp_services = “{ ssh torproxy torOR torDIR }” # options set block-policy drop set loginterface $lan_if # pass on lo set skip on lo scrub in on $lan_if all fragment reassemble # NAT nat on $lan_if from $wifi_if:network to !($lan_if) -> ($lan_if) block all antispoof for $interfaces #In NAT pass in log on $wifi_if inet pass out all keep state #ICMP pass out log inet proto icmp from any to any keep state pass in log quick inet proto icmp from any to any keep state #SSH pass in inet proto tcp to $lan_if port ssh pass in inet proto tcp to $wifi_if port ssh #TCP Services on Server pass in inet proto tcp to $interfaces port $tcp_services keep state The finally part is mapping the ports as follows: TOR directory port: LANIP:9099 —> WANIP:80 TOR router port: LANIP:9090 —-> WANIP:443 Now enable TOR: $ sudo echo “tor_enable=YES” >> /etc/rc.conf Start TOR: $ sudo service tor start *** Beastie Bits OpenBSD as a “Desktop” (Laptop) (http://unixseclab.com/index.php/2017/06/12/openbsd-as-a-desktop-laptop/) Sascha Wildner has updated ACPICA in DragonFly to Intel's version 20170629 (http://lists.dragonflybsd.org/pipermail/commits/2017-July/625997.html) Dport, Rust, and updates for DragonFlyBSD (https://www.dragonflydigest.com/2017/07/18/19991.html) OPNsense 17.7 RC1 released (https://opnsense.org/opnsense-17-7-rc1/) Unix's mysterious && and || (http://www.networkworld.com/article/3205148/linux/unix-s-mysterious-andand-and.html#tk.rss_unixasasecondlanguage) The Commute Deck : A Homebrew Unix terminal for tight places (http://boingboing.net/2017/06/16/cyberspace-is-everting.html) FreeBSD 11.1-RC3 now available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087407.html) Installing DragonFlyBSD with ORCA when you're totally blind (http://lists.dragonflybsd.org/pipermail/users/2017-July/313528.html) Who says FreeBSD can't look good (http://imgur.com/gallery/dc1pu) Pratik Vyas adds the ability to do paused VM migrations for VMM (http://undeadly.org/cgi?action=article&sid=20170716160129) Feedback/Questions Hrvoje - OpenBSD MP Networking (http://dpaste.com/0EXV173#wrap) Goran - debuggers (http://dpaste.com/1N853NG#wrap) Abhinav - man-k (http://dpaste.com/1JXQY5E#wrap) Liam - university setup (http://dpaste.com/01ERMEQ#wrap)

I afton: BBS! Med anledning av en ny svensk BBS-dokumentär blir det en liten dykning i svensk BBS-historia med fokus på mjukvara för systemen. Kan du hjälpa Jocke att hitta fler svenska KOM-system? Säg till! Och finns det någon svensk eller europeisk motsvarighet till archive.org? 38:24 Fredrik blev sugen på typsnitt 43:00 Ny sommartröja! Macpro-tröjor, och Macpro-loggan genom tiderna 52:37 Jocke mot nya utmaningar 56:29 Fredrik har begått en premiär 59:41 WWDC 2017 - en vecka senare: är det innovation eller komma-i-kapp som Apple sysslar med? Jocke har funderat och svalnat i en vecka och undrar hur nya och spännande WWDC-nyheterna egentligen är och var. Vore Apple idag ännu bättre med mer blåslampa och konkurrens? Länkar Senaste Roderick on the line Ny BBS-dokumentär av Simon Klose The pirate bay: AFK Jockes artikel om BBS:er Nikom KOM Svenska Wikipedia fattar ingenting GCC Sklaffkom VAX Sunos Skom SOAP är något få vill pyssla med numera Python-modul för LysKOM Macpro magazine msvcrt.dll DNIX Fidonet Prompt Operator Sublime text IA Ny sommartröja! Panics terminalinställningar Macpro t-shirt Fredriks krönika om iPad och iOS 11 WWDC 2017 - en vecka senare: är det innovation eller komma-i-kapp som Apple sysslar med? Diamond Rio PMP300 Longhorn Ctrl-Walt-delete Samsung galaxy S8 John Grubers text om iPad pro 10.5 Geekbench Emacs lisp CVS FOA - Försvarets forskningsanstalt s://boingboing.net/”>Boingboing Fullständig avsnittsinformation finns här: https://www.bjoremanmelin.se/podcast/avsnitt-81-connect-28800-arq-v34-lapm-v42bis.html.

This week on the show, Allan and I are going to be showing you a very interesting interview we did talking about using FreeBSD to drive This episode was brought to you by Headlines FreeBSD Core Team Election (https://www.freebsd.org/administration.html#t-core) Core.9 has been elected, and will officially take over from Core.8 on Wednesday, 6 July 2016 Many thanks to the outgoing members of the core team for their service over the last 2 years 214 out of 325 eligible voters (65.8%) cast their votes in an election counting 14 candidates. The top nine candidates are, in descending order of votes received: 180 84.1% Ed Maste (incumbent) 176 82.2% George V. Neville-Neil (incumbent) 171 79.9% Baptiste Daroussin (incumbent) 168 78.5% John Baldwin 166 77.6% Hiroki Sato (incumbent) 147 68.7% Allan Jude 132 61.7% Kris Moore 121 56.5% Benedict Reuschling 108 50.5% Benno Rice There was no tie for ninth. BSDNow and the entire community would also like to extend their thanks to all those who stood for election to the core team Next week's core meeting will encompass the members of Core.8 and Core.9, as responsibility for any outstanding items will be passed from outgoing members of core to the new incoming members *** Why I run OpenBSD (http://deftly.net/posts/2016-05-31-why-i-run-openbsd.html) This week we have a good article / blog post talking about why the posted has moved to OpenBSD from Linux. “One thing I learned during my travels between OSs: consistency is everything. Most operating systems seem to, at least, keep a consistent interface between themselves and binaries / applications. They do this by keeping consistent APIs (Application Programming Interfaces) and ABIs (Application Binary Interfaces). If you take a binary from a really old version of Linux and run or build it on a brand-spanking new install of Linux, it will likely Just Work™. This is great for applications and developers of applications. Vendors can build binaries for distribution and worry less about their product working when it gets out in the wild (sure this binary built in 2016 will run on RedHat AS2.1!!).“ The author then goes through another important part of the consistency argument, with what he calls “UPI” or “User Program Interfaces”. In other words, while the ABI may be stable, what about the end-user tooling that the user directly has to interact with on a daily basis? “This inconsistency seems to have come to be when Linux started getting wireless support. For some reason someone (vendors, maybe?) decided that ifconfig wasn't a good place to let users interact with their wireless device. Maybe they felt their device was special? Maybe there were technical reasons? The bottom line is, someone decided to create a new utility to manage a wireless device… and then another one came along… pretty soon there was iwconfig(8), iw(8), ifconfig(8), some funky thing that let windows drivers interface with Linux.. and one called ip(8) I am sure there are others I am forgetting, but I prefer to forget. I have moved onto greener pastures and the knowledge of these programs no longer serves me.” The article then goes through the rundown of how he evaluated the various BSD's and ultimately settled on OpenBSD: “OpenBSD won the showdown. It was the most complete, simple, and coherent system. The documentation was thorough, the code was easy to follow and understand. It had one command to configure all of the network interfaces! I didn't have wireless, but I was able to find a cheap USB adapter that worked by simply running man -k wireless and reading about the USB entries. It didn't have some of the applications I use regularly, so I started reading about ports (intuitively, via man ports!).” The ultimate NetBSD Router (http://blog.tbrodel.me/2016/#netbsd-router) “So yesterday I spent the day setting up a new firewall at home here, based off of this BSD Now tutorial. Having set up a couple of OpenBSD routers before, either based on old laptops, bulky old power-sucking desktops or completely over-specced machines like the Intel NUC, I wanted to get some kind of BSD onto a low-powered ARM board and use that instead.” “I've had a couple of Cubietrucks lying around for a while now, I've used them in a couple of art installations, running Debian and Pure Data, but over all they've been a bit disappointing. It's more the manufacturer's fault but they require blobs for the graphics and audio, which Debian won't allow, so as a multimedia board they're dud for video, and only passable for audio work with a usb sound card. So they've been collecting dust.” “Only thing missing is a second NIC, luckily I had an Apple USB->Ethernet dongle lying around, which when I bought it was the cheapest thing I could find on eBay that OpenBSD definitely supported. There, and on NetBSD, it's supported by the axe(4) driver. USB 2.0 works fine for me as I live in Australia and my ISP can only give me 30Mbps, so this should do for the forseeable future.” + The article then walks through installing and configuring NetBSD + Configuration includes: pf, unbound, and dhcpd “This project has been really fun, I started with basically no experience with NetBSD and have finished with a really useful, low-powered and robust appliance. It's a testament to the simplicity of the NetBSD system, and the BSD design principles in general, that such a novice as myself could figure this out. The NetBSD project has easily the most polished experience on Allwinner ARM boards, even Debian doesn't make it this easy. It's been a joy running the system, it has the bits I love from OpenBSD; ksh(1), tmux(1), an http daemon in base and of course, pf(4). This is mixed with some of the pragmatism I see in FreeBSD; a willingness to accept blobs if that really is the only way to boot, or get audio, or a video console.” bhyve-Bootable Boot Environments (http://callfortesting.org/bhyve-boot-environments/) We have a lengthy article also today from our friend Michael Dexter, who asks the basic question “What if multibooting and OS upgrades weren't horrible?” No doubt if you've been a frequent listener to this show, you've heard Allan or Myself talking about ZFS Boot Environments, and how they can “change your life”. Well today Michael goes further into detail on how the BE's work, and how they can be leveraged to do neat things, like installing other versions of an operating system from the original running system. “If you are reading this, you have probably used a personal computer with a BSD or GNU/Linux operating system and at some point attempted to multiboot between multiple operating systems on the same computer. This goal is typically attempted with complex disk partitioning and a BSD or GNU/Linux boot loader like LILO or GRUB, plus several hours of frustrating experimentation and perhaps data loss. While exotic OS experimentation has driven my virtualization work since the late 1990s, there are very pragmatic reasons for multibooting the same OS on the same hardware, notable for updates and failback to "known good" versions. To its credit, FreeBSD has long had various strategies including the NanoBSD embedded system framework with primary and secondary root partitions, plus the nextboot(8) utility for selecting the "next" kernel with various boot parameters. Get everything set correctly and you can multiboot "with impunity". “That's a good start, and over time we have seen ZFS "boot environments" be used by PC-BSD and FreeNAS to allow for system updates that allow one to fall back to previous versions should something go wrong. Hats off to these efforts but they exist in essentially purpose-built appliance environments. I have long sensed that there is more fun to be had here and a wonderful thing happened with FreeBSD 10.3 and 11.0: Allan Jude added a boot environment menu to the FreeBSD loader” From here Michael takes us through the mechanical bits of actually creating a new ZFS dataset (BE) and performing a fresh FreeBSD 10.3 installation into this new boot-environment. The twist comes at the end, where he next sets up the BE to be a root NFS for booting in bhyve! This is interesting and gives you a way to test booting into your new environment via a VM, before rebooting the host directly into it. *** Interview - Edicarla Andrade & Vinícius Zavam - @egypcio (https://twitter.com/egypcio) BSD-Powered Robots News Roundup Tomohiro Kasumi explains what “@@” means, in the context of the Hammer filesystem (http://lists.dragonflybsd.org/pipermail/users/2016-June/249717.html) A post from the Dragonfly users' mailing list about what the @@ construct means in the Hammer filesystem “@@ represents the existence of a PFS which is logically separated pseudo filesystem space within HAMMER's B-Tree” “HAMMER only has 1 large B-Tree per filesystem (not per PFS), so all the PFS exist within that single B-Tree. PFS are separated by localization parameter which is one of the B-Tree keys used to lookup the tree.” Each substring in "@@-1:00001" means: "@@" means it's a PFS or snapshot. "-1" means it's a master. ":" is just a separator. "00001" means it's PFS#1, where PFS#0 is the default PFS created on newfs. There is no "00000" because that's what's mounted on /HAMMER. PFS# is used for localization parameter. “Localization parameter has the highest priority when inserting or looking up B-Tree elements, so fs elements that belong to the same PFS# tend to be localized (clustered) within the B-Tree” There is also a note about how snapshots are named: "@@0x00..." A user points out that having : in the path can confuse some applications, such as in the case of adding the current directory or a relative path to the $PATH environment variable, which is a colon delimited list of paths This seems quite a bit more confusing that the datasets created by ZFS, but they might have other useful properties *** FreeBSD 11.0 nearing RC1 (https://www.freebsd.org/releases/11.0R/schedule.html) We've all been eagerly awaiting the pending release of FreeBSD 11.0, and the schedule has now been updated! The first release candidate is slated for July 29th! If all goes well (and we stick to schedule) there will be another RC2 and possible RC3 release, before 11.0 officially drops near the end of August. Start playing with those builds folks, be sure to send your feedback to the team to make this the best .0 release ever! *** TensorFlow on FreeBSD (http://ecc-comp.blogspot.com/2016/06/tensorflow-on-freebsd.html) Next we have a blog post about the experience of a “new” FreeBSD user trying to deploy some non-ported software to his new system. Specifically he was interested in running TensorFlow, but not doing a port himself, because in his words: “First, I apologize for not supplying a port archive myself. After reading the FreeBSD handbook for creating a port, it's too complex of a task for me right now. I've only been using FreeBSD for two weeks. I would also not like to waste anyone's time giving them a terrible port archive and mess up their system.” First of all, good ports are often born out of bad ports! Don't let the porting framework daunt you, give it a go, since that's the only way you are going to learn how to write “good” ports over time. The porters-handbook is a good first place to start, plus the community usually is very helpful in providing feedback. He then walks us through the changes made to the TensorFlow code (starting with the assumption that OSX was a good “flavor” to begin porting from) and ultimately compiling. This ends up with the creation of a pip package which works! A good tutorial, and also very similar to what goes on in the porting process. With this write-up perhaps somebody will take up creating a port of it… hint hint! *** NetBSD: A New Beginning? (http://jamesdeagle.blogspot.ca/2016/06/netbsd-new-beginning.html) We don't get enough NetBSD news at times, but this post by James Deagle talks about his adventure with NetBSD 7.0 and making it his “new beginning” “After a few months of traipsing around the worlds of SunOS and Linux, I'm back to NetBSD for what I hope will be a lengthy return engagement. And while I'm enamored of NetBSD for all the previously-mentioned reasons, I'm already thinking ahead to some problems to solve, some of which have also been mentioned before.” He then goes through and lists some of the small nits he's still running into during the daily workflow YouTube audio - Specifically he mentions that no audio is playing, but wonders if Flash plays some part. (Ideally you're not using Flash though, in which case you need to check the audio backend FF is using. Try PulseAudio since it seems the best supported. If pulse is already enabled, install ‘pavucontrol' to make sure audio is playing to the correct sound device) Slow gaming performance (TuxKart and Celestia) - Check DRI / Xorg? Or is it CPU bound? Lastly some unspecified Wireless issues, which typically end up being driver related. (Or use another chipset) Beastie Bits Reproducible NetBSD? 77.7% of the way there (https://reproducible.debian.net/netbsd/netbsd.html) Create FreeBSD virtual machine using qemu. Run the VM using xhyve. (https://gist.github.com/zg/38a3afa112ddf7de4912aafc249ec82f) FreeBSD PowerPC 32bit pkg repository (unofficial). ~19,500 packages, more to come (https://joshcummings.net/pub/FreeBSD) NetBSD machines at Open Source Conference 2016 Gunma (http://mail-index.netbsd.org/netbsd-advocacy/2016/05/16/msg000706.html) Adam Leventhal (of ZFS and DTrace) does an analysis of APFS (http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/) SemiBug June meeting summary (http://lists.nycbug.org/pipermail/semibug/2016-June/000106.html) KnoxBug Meeting (http://knoxbug.org/content/2016-07-26) Feedback/Questions Andrew - iocage (http://pastebin.com/nuYTzaG6) Florian - Arm + GitHub (http://pastebin.com/PzY68hNS) Clint - Synth (http://pastebin.com/JESGZjLu) Leonardo - Translations (http://pastebin.com/b4LAiPs4) Zachary - Moving things to VMs (http://pastebin.com/VRc8fvBk) ***

A brief description of playing around with SunOS 4.1.4, which was the last version of SunOS to be based on BSD.File Info: 17Min, 8MbOgg Link: https://archive.org/download/bsdtalk265/bsdtalk265.ogg

This week on BSDNow, Allan is back from the Storage Summit in Silicon Valley! We are going to get his thoughts on how the conference went, plus bring you the latest ZFS info discussed. That plus the usual BSD news is This episode was brought to you by Headlines OpenBSD website operators urged to fix mind-alteringly bad bug (http://www.theregister.co.uk/2016/02/21/openbsd_website_operator_patch_now_for_the_sake_of_your_sanity/?mt=1456206806399) We start off a bit light-hearted this week, with the important, breaking news that finally a long-standing OpenBSD bug has been addressed for the HTTP daemon. Specifically? It changes the default 404 page fonts away from Comic Sans, to a bit more crowd-pleasing alternative: “For some reason the httpd status pages (e.g. 404) use the Comic Sans typeface. This patch removes comic sans and sets the typeface to the default sans-serif typeface of the client. “This lowers the number of people contacting website maintainers with typeface complaints bordering on harassment”. Operators running HTTPD are highly encouraged to update their systems to the latest code, right now……... No seriously, we are waiting for you. Get it done now and then we'll continue with the show. Registration for AsiaBSDCon 2016 is now open + Talk Schedule (https://2016.asiabsdcon.org/registration/?lang=en) After a few delays, the registration for AsiaBSDCon has now opened! The conference starts in less than two weeks! now, so be sure to get signed up ASAP. In addition the schedule has been posted, and here's some of the highlights of this year's conference. In addition to FreeBSD and NetBSD dev summits on the first two days, we have some excellent tutorials being given this year by Kirk, Gnn, Dru and more! (https://2016.asiabsdcon.org/program.html.en) The regular paper talks also have lots of good ones this year, including this crazy encrypted boot loader one given by our very own Allan Jude! *** OPENBSD ON AWS : AN UNEXPECTED JOURNEY (http://blog.d2-si.fr/2016/02/15/openbsd-on-aws/?hn) We have a blog post from Antoine Jacoutot, talking about the process of getting OpenBSD up and running in AWS It starts with his process of creating an AMI from scratch, which ended up not being that bad: create and loopback-mount a raw image containing a UFS filesystem extract the OpenBSD base sets (which are just regular tarballs) and kernel enable console output (so that one could “aws ec2 get-console-output”) install the boot loader on the image then use the ec2 tools to import the RAW image to S3, convert it into a volume (ec2-import-volume) which we can snapshot (ec2-create-snapshot) and create an AMI from (ec2-register) The blog post also has a link to a script which automates this process, so don't be daunted if you didn't quite follow all of that. Thanks to the recently landed DomU support, the final pieces of the puzzle fell into place, allowing OpenBSD to function as a proper guest (with networking!) Next it details the process of injecting a public SSH key into the instances for instant remote access. An ec2-init.sh script was created (also on github) which does the following: setting the hostname installing the provided SSH public key to /root/.ssh/authorized_keys executing user-data (if it starts with a shebang) displaying the host SSH fingerprints on the console (to match cloud-init) With that done, OpenBSD is pretty much AWS ready! He then gives a brief walkthrough of setting up nginx for new users, but if you've already done this before then the instance is ready for you to hacking on. Start thinking of ideas for things with FreeBSD for Google's 2016 Summer of Code (https://wiki.freebsd.org/SummerOfCodeIdeas) Students and Developers, listen up! It's time to start thinking about GSoC again, and FreeBSD is looking to update its project ideas page. There's some good ones on the list, plus ones that should be pruned (such as GELI boot), but now is the time to start adding new ones before we get too deep into the process. This goes for the other BSD's as well, start thinking about your proposals, or if you are developer, which projects would be a good fit for mentoring. (Improving the Linux Compat layer is one I think should be done!) Guide to getting started with kernel hacking (https://wiki.freebsd.org/Graphics/Getting%20started%20with%20kernel%20projects) One of the things that's been asked frequently is how to contribute towards the efforts to bring updated DRM / X drivers to the FreeBSD kernel. Jean-Sébastien Pédron has started a great guide on the Wiki which details how to get started with the porting effort, and that developers need not be afraid of helping. *** Storage Summit Roundup Earlier this week a number of developers from FreeBSD, as well as various vendors that use FreeBSD, or provide products used with FreeBSD met for a Storage Summit (https://wiki.freebsd.org/201602StorageSummit), to discuss the future of these technologies The summit was co-located with the USENIX FAST (Filesystems And Storage Technologies) conference The summit was sponsored by the FreeBSD Foundation and FlightAware After a short introduction, the event opened with a Networking Synergy panel The focus of this panel was to see if there were techniques and lessons learned in improving the networking stack over the last 10 years that could be applied to improving the storage stack A lot of time was spent discussing issues like multi-queue support, CPU scheduling, and ways to modernize the stack CAM Scheduling & Locking Revamp (https://wiki.freebsd.org/201602StorageSummit/CAM) No notes posted User Space Storage Stack (https://wiki.freebsd.org/201602StorageSummit/UserSpace) One of the user space storage stacks discussed was Diskmap Like netmap, but for disks (diskmap) Kernel bypass for accessing disks Ilias Marinos, who is working on diskmap at Cambridge University, described diskmap to the group A design discussion then followed in which the memory management was covered as that's an issue for any sort of "IO" map system Action Items: Discuss with Luigi the idea of code merges Need a reset path API Kernel buffer mapping for reliability Support for other interfaces (SATA/SCSI) GEOM layer adaptation Adapting to New Storage Technologies (https://wiki.freebsd.org/201602StorageSummit/NewStorageTechnologies) This working group was led by Adrian Palmer, from Seagate SMR Persistent Memory Session 1: Device Identification and the structural requirements Agenda: We'll look over the Identification nuances and what needs to change to support the structure. Support for IO order guarantees, forward-write only requirements, new commands and topology. Dig into CAM and GEOM layers. Solutions should be fast and have as few code paths as possible Results: Small audience. We talked about zoned characteristics, and how it can be used in various workloads, projected to be implemented in years Session 2: Information dissemination and consumption Agenda: Where and how will information from the report_zones command be gathered, stored, combined and used. This will include userspace storage and multi-volume management. Will CAM store this data, or will GEOM? How frequently will this need to be queried/updated/verified from the drive? Results: Merged with ZFS working group to discuss SMR. Came up with idea that could be implemented as circular buffer zone type. Began to discuss solutions among developers ZFS (https://wiki.freebsd.org/201602StorageSummit/ZFS) During the first session we discussed how to improve dedup support + A dedup throttle or cap was discussed. When the size of the DDT grows beyond this size, new entries would not be deduped. An alternative to this was also discussed, where when the DDT reached the cap size, it would remove a random entry with only a single reference from the DDT to make room for the new entry. When a block is going to be freed, if it is not found in the DDT, it is assumed to have only 1 reference, and removed. There was also discussion of replacing the DDT with an in-memory hash table and a “log” of increment/decrement operations, that is periodically compacted. The hash table is recreated from the log at pool import time. This would reduce the in-memory footprint of the DDT, as well as speed up all write operations as adding an entry to the dedup log will be less expensive than updating the DDT. There was also discussion of using dedicated device(s) for the DDT, either using the DDT on SSD work by Nexenta, or the Metadata Classes work by Intel The first session also discussed Secure Delete and related things The desire for an implementation of TRIM that uses the “secure erase” functionality provided by some disks was expressed Overwriting sectors with patterns of garbage may be insufficient because SSDs may internally remap where a specific LBA physically resides The possibility of using something like the “eager zero” feature to periodically write zeros over all free blocks in the pool to erase any lingering data fragments Problems with the FreeBSD TRIM implementation were discussed, as well as looking at ways to implement the new ZFS TRIM implementation on FreeBSD ABD (ARC Buf Data) was discussed, a new design that lessens the requirement for contiguous memory. Only a small area of contiguous blocks is reserved at boot, and compressed ARC blocks are constructed of scatter-gather lists of individual pages The second session combined with the SMR group and talked about SMR support in ZFS Later in the second session ZFS Encryption was also discussed, mostly with a focus on what the use cases are The third session combined all of the groups for an overview of upcoming ZFS features including device removal and channel programs There was also a request for code review, for mostly finished projects like Persistent L2ARC, Writeback cache, and Large dnode support Hallway Track ZFS / VFS Interaction Adrian Palmer has been a FreeBSD hobbyist since FreeBSD 7, and I think I managed to convince him to start contributing *** News Roundup One Week with NetBSD 7.0: Back to Unix basics (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-back-to-unix.html) The author of this blog series is sending a week using NetBSD 7.0, following a previous series on Solaris 10 “This is actually familiar territory, as I've been using BSD variants almost exclusively since 2006. My recent SunOS explorations were triggered last summer by OpenBSD having choked on my current laptop's NVIDIA card, and from what I could see at the time, FreeBSD had the same problem, although I now know NVIDIA drivers exist for that system. The thing that keeps me from going all-in with FreeBSD 10.x, however, is the fact that Firefox crashes and leaves "core dump" messages in its wake, and I'm just not a Chrome kinda guy.” “For those with a catholic taste in Unix, NetBSD is a keg party at the Vatican. If you're an absolute Unix beginner, or have been living on Ubuntu-based Linux distros for too long, then you may feel stranded at first by NetBSD's sparseness. You'll find yourself staring into the abyss and seeing only a blinking cursor staring back. If you have the presence of mind to type startx, you'll be greeted by twm, a window manager offering little more than an xterm window with the same blinking cursor until you learn how to configure the .twmrc file to include whatever applications you want or need in the right-click menu.” “As for NetBSD itself, I can't think of any major productivity applications that can't be installed, and most multimedia stuff works fine.” Issues the author hopes to sort out in later posts: Audio playback (youtube videos in Firefox) Wireless Flash Digital Camera SD Card readability, video playback Audacity A “fancy” desktop like Gnome 2, KDE, or xfce In a follow-up post (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-libreoffice.html), the author got LibreOffice installed and sorted out the audio issues they were having In a later follow-up (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-mixed-review-of.html) XFCE is up and running as well *** ZFS is for Containers in Ubuntu 16.04 (http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html) As you may have heard, Ubuntu 16.04 will include ZFS -- baked directly into Ubuntu -- supported by Canonical “ZFS one of the most beloved features of Solaris, universally coveted by every Linux sysadmin with a Solaris background. To our delight, we're happy to make to OpenZFS available on every Ubuntu system.” What does “supported by Canonical” mean? “You'll find zfs.ko automatically built and installed on your Ubuntu systems. No more DKMS-built modules” “The user space zfsutils-linux package will be included in Ubuntu Main, with security updates provided by Canonical” The article then provides a quick tutorial for setting up Linux Containers (LXC) backed by ZFS In the example, ZFS is backed by a file on the existing disk, not by a real disk, and with no redundancy However, the setup script seems to support using real block devices The Software Freedom Conservancy (https://sfconservancy.org/) is expected to issue a statement detailing their opinion on the legalities and licensing issues of bundling ZFS with Linux. *** Polling is a Hack: Server Sent Events (EventSource) with gevent, Flask, nginx, and FreeBSD (http://hypatia.software/2016/01/29/polling-is-a-hack-server-sent-events-eventsource-with-gevent-flask-nginx-and-freebsd/) A tutorial on setting up ‘Server-Sent Events', also know as EventSource in javascript, to notify website clients of new data, rather than having the javascript constantly poll for new data. The setup uses FreeBSD, nginx, gevent, Python, and the Flask framework The tutorial walks through setting a basic Python application using the Flask framework Then setting up the client side in Javascript Then for the server side setup, it covers installing and configuring nginx, and py-supervisor on FreeBSD The tutorial also includes links to additional resources and examples, including how to rate limit the Flash application *** Why FreeBSD? (http://www.aikchar.me/blog/why-freebsd.html) An excellent article written by Hamza Sheikh, discussing why FreeBSD is now his clear choice for learning UNIX. The article is pretty well written and lengthy, but has some great parts which we wanted to share with you: There were many rough edges in the Linux world and some of them exist even today. Choosing the right distribution (distro) for the task at hand is always the first and most difficult decision to make. While this is a strength of the Linux community it is also its weakness. This is exacerbated with the toxic infighting within the community in the last few years. A herd of voices believes it is their right to bring down a distro community because it is not like their distro of choice. Forking upstream projects has somehow become taboo. Hurling abuse in mailing lists is acceptable. Helping new users is limited to lambasting their distro of choice. Creating conspiracy theories over software decisions is the way to go. Copyleft zealots roam social media declaring non-copyleft free software heretic abominations. It all boils down to an ecosystem soured by the presence of maniacs who have the loudest voices and they seem to be everywhere you turn. Where is the engineering among all this noise? Btrfs - baking for a long time - is still nowhere near ZFS in stability or feature parity. systemd is an insatiable entity that feeds on every idea in sight and just devours indiscriminately. Wayland was promised years ago and its time has yet to arrive. Containers are represented by Docker that neither securely contains applications nor makes them easy to manage in production. Firewalling is dithering between firewalld, nftables, etc. SystemTap cannot match DTrace. In the same time span what do various BSDs offer? pf, CARP, ZFS, Hammer, OpenSSH, jails, pkgsrc, (software) ports, DTrace, hardware portability; just to name a few. Few would deny that BSDs have delivered great engineering with free software licenses to the entire world. To me they appear to be better flag bearers of free software with engineering to back it. He then goes through some of the various BSD's and the specifics on why FreeBSD was the logical choice for his situation. But at the end has a great summary on the community as a whole: Finally - and maybe repeating myself here - I have nothing but praise for the community. Be it BSD Now, mailing lists, Reddit, Twitter, LFNW, or SeaGL, people have encouraged me, answered my questions, and filed bugs for me. I have been welcomed and made a part of the community with open arms. These reasons are (good) enough for me to use FreeBSD and contribute to it. BeastieBits OPNsense 16.1.3 released (https://opnsense.org/opnsense-16-1-3-released/) Copies of "FreeBSD Mastery: Specialty Filesystems" seen in the wild (https://twitter.com/Savagedlight/status/700001944547491842) pfsense training available in Europe (http://www.netgate.com/training/) LiteBSD now has 50 ports in its ports tree (https://github.com/ibara/LiteBSD-Ports) Ports tree locked for OpenBSD 5.9 (http://marc.info/?l=openbsd-ports&m=145615281431064&w=2) “FreeBSD Filesystem Fun” at March semibug (http://blather.michaelwlucas.com/archives/2556) Event #46 — Embedded Platforms (BSD, OpenWRT, Plan 9 & Inferno) (http://oshug.org/event/46) Feedback/Questions Frank - ZFS RAM? (http://slexy.org/view/s21lcCKrSB) David - ARM Porting (http://slexy.org/view/s204lxjvlq) Johnny - Lumina Default? (http://slexy.org/view/s2xMiSNLYn) Adam - PC-BSD Install and Q's (http://slexy.org/view/s214gJbLwD) Jeremy - Video Card Q (http://slexy.org/view/s20UNyzEeh) ***

Coming up on BSDNow - We know init systems have been all the rage This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** Headlines Interview with Renato Westphal (http://undeadly.org/cgi?action=article&sid=20151123113224&mode=expanded) An interview with Brazilian OpenBSD developer Renato Westphal He describes how he first got into OpenBSD, working on a University-Industry partnership program and looking to deploy LDP (Label Distribution Protocol) for MPLS. He ported OpenBSDs ldpd(8) to Linux, but then contributed his bug fixes and improvements back to OpenBSD When asked if he was motivated to replace closed-source router implementations with OpenBSD: “Well, I don't administer any network, I work full time as a programmer. I have some friends however that succeeded replacing closed vendor solutions with OpenBSD boxes and that for sure motivates me to keep doing what I'm doing. My biggest motivation, however, is the challenge of resolving complex problems writing trivially simple code that is both secure and efficient.” They also go on to discuss some of the interesting features of EIGRP, and developing eigrpd(8) What do you think is missing from routing in OpenBSD: “Implementing new features and protocols while they are in their draft stage in IETF. I'd like to see OpenBSD as the reference platform for the development of new routing and networking technologies in general” *** Let's Encrypt on a FreeBSD NGINX reverse proxy (http://savagedlight.me/2015/11/24/lets-encrypt-on-a-freebsd-nginx-reverse-proxy/) We have a neat guide/story today on how to setup the “Let's Encrypt” certificates on a FreeBSD / nginx reverse proxy Backstory: For those who don't know, “Let's Encrypt” (https://letsencrypt.org) is a new Certificate Authority, which will allow you to create free and automated certificates. They have been in closed beta for several months now, and will be opening to a public beta Dec 3rd (tomorrow) This guide is particularly timely, since by the time most of you are watching this episode, the public beta will be up and running. Most of the instructions are fairly straight-forward. She starts by installing the lets-encrypt package from ports/pkg and modifying her nginx with a ‘catch-all' vhost that re-directs traffic to the https versions of a site. With that done, the certificate creation is just a few commands to get started, in which she shows creating a cert for multiple domains As a bonus! She includes a nice renewal script which can be run from cron. It will monitor the certs daily, and renew it when it's 14 days from expiring, or throw an error for somebody to look at. *** Mike Larkins OpenBSD vmm subsystem now in tree (http://marc.info/?l=openbsd-tech&m=144822644214614&w=2) An openBSD native hypervisor has taken another step closer to reality, with Mike Larkin pushing the initial bits of “vmm” into the base kernel/world He mentions in the commit message that it still needs a lot of work, and as such is disabled by default. However for the adventurous among you, it can be turned on and tested Right now there is no BIOS, and as such it can only be used to boot other OpenBSD instances, although he mentions other BSD's could be supported fairly quickly (He did a 1 hour port to bootstrap NetBSD) No big documentation expected for this release, since there is so much ongoing churn. Take a look at the man page for details on getting started. *** The story of how Yahoo switched to FreeBSD (http://zer0.org/daemons/yahoobsd.html) Yahoo originally started running on SunOS, but quickly found it not able to cope with the high frequency of HTTP requests “Having spend many frustrating hours trying to install other PC OS's, I was a bit skeptical. I had no intention of spending three days trying to install yet another one. To my surprise I went to the FreeBSD Web site, downloaded the floppy boot image, booted a PC with the created floppy, answered a few install questions, and a few minutes later FreeBSD was installing over the Net. The real surprise was when I came back later to a fully configured system that actually worked.” “If anything had gone wrong with that install it would likely been the end of that trial. Luckily for us that it was the easiest and most painless OS installs I had ever experienced.” Just that easily, Yahoo might never have ended up on FreeBSD “A couple of days later we added a FreeBSD box to our cluster of Web servers. Not only did it out-perform the rest of our machines, but it was more stable.” From my understanding of stories told over dinner, Yahoo had a few very important perl scripts, and they tended to crash on Linux, but kept running without issue on FreeBSD Related hackernews thread (https://news.ycombinator.com/item?id=10558288) *** iXsystems iXsystem's recap of LISA 2015 (https://www.ixsystems.com/whats-new/lisa-2015/) *** Interview - Mark Heily - mark@heily.com (mailto:mark@heily.com) / @MarkHeily (https://twitter.com/MarkHeily) relaunchd (https://github.com/mheily/relaunchd) *** News Roundup Inline Intrusion Prevision System is an upcoming OPNSense Feature (https://opnsense.org/inline-intrusion-prevention/) The next OPNSense release, 16.1 is around the corner and today we have a sneak peek at their new Inline Intrusion Prevention system Suricata working with Netmap 2.1 enabled version, which allows Deep Packet Inspection of traffic. Such as looking at each packet individually and only blocking specific ones. They use the example of blocking Warcraft (oh noes!) Enabling this feature is just a simple mouse-click away, and various default rules are included as part of the Emerging Threats Community rules. *** Matthew Dillion working on Hardlinks in Hammer2 (http://lists.dragonflybsd.org/pipermail/commits/2015-November/458763.html) We have an interesting commit from Matthew Dillon for Hammer2, specifically targeted at hard-links The backstory he gives us: “The H2 design has had a long-standing problem of losing track of hardlinks when intermediate directories are renamed, breaking the common-parent-directory design for the inode target.” The implemented fix was one which instead places the hardlink target in the first common parent directory, which is marked with “xlink” via chflag If no parent directory is marked “xlink”, it will fall-through instead to the root of the mount They also modified their installworld to set “/” /usr/,/var/,/home/ as “xlink” flagged This prevents moving hard-links across these directories, but is similar to dealing with multiple partitions / datasets already. *** Japan's NetBSD User Group showed off some NetBSD machines at the 2015 Tokushima Open Source Conference (http://lists.nycbug.org/pipermail/talk/2015-November/016403.html) It's been a little while since we've shown off a bunch of odd devices running NetBSD, but we have an update from the 2015 Tokushima Open Source Conference. This time around, we have pictures of the booth, as well as a variety of oddities such as: ODroid-C1 / Sharp X68030 Sharp NetWalker Sharp WZero3 (Cell phone) Give them a look, this time around they have nice cards pictured which details the hardware being used (in english none the less!) *** One of the three OpenBSD users Blog Post by Adam Wolk (http://blog.tintagel.pl/2015/11/22/one-of-the-three-openbsd-users.html) An OpenBSD user comments on a recent interaction with the syncthing project (a dropbox like alternative) The application has an auto-update feature (which doesn't mix well with package systems in the first place), but it doesn't work on OpenBSD because there is no /proc/curproc/file to determine the filename of the executable. This is a trivially easy task, but when the bug was reported, syncthings response was “Maybe one of the three (https://data.syncthing.net/#metrics) OpenBSD users feel strongly enough about this to propose a patch. :D” Part of the issue is that many users (especially the type that would run OpenBSD) opt out of reporting metrics, so OpenBSD is under-represented in the metrics the project developers are basing their decisions on Maybe someone can post a patch to solve the problem. While FreeBSD can provide a linux procfs, it would be better to use a more portable way to get the location of the process binary *** BeastieBits DragonFly BSD 4.4 RC branch created (http://lists.dragonflybsd.org/pipermail/commits/2015-November/458818.html) HOWTO: NFS booting bhyve (http://oshogbo.vexillium.org/blog/39/) DragonFly BSD is looking for a 4.4 RC image by the end of November (http://lists.dragonflybsd.org/pipermail/kernel/2015-November/175040.html) Support for Atheros QCA953x "Honeybee" has been added to FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=290910) Top updated in DragonflyBSD to allow the 'c' command (http://lists.dragonflybsd.org/pipermail/commits/2015-November/458692.html) FreeBSD textbook makes appearance on the 6pm news in the Netherlands 12:49 (http://www.npo.nl/nos-journaal/30-11-2015/POW_00941854) SemiBug gives a recap of its Inaugural meeting and its plans for future meetups (http://blather.michaelwlucas.com/archives/2495) *** Feedback/Questions Adam - GELI on USB (http://slexy.org/view/s204HRCPdR) Noble - Radius on FreeBSD (http://slexy.org/view/s21q2WWisr) Jim - Backporting Wifi Code (http://slexy.org/view/s21L59OGyF) Mohammad - Zombies! (http://slexy.org/view/s20nWwzTGS) Miguel - ScaleEngine BTS (http://slexy.org/view/s201Kpd4GX) ***

Statistics: mp3 requests the the last 20 days of March by operating system.2543 Linux, 1 Irix, 21 SunOS, 2847 BSD, 3279 Macintosh, 11340 Windows.Interview with Peter Hofer from DesktopBSD

Das Betriebssystem mit dem X verwirrt und entzückt die Gemüter. Während Journalisten auch im Jahre 30 seit seiner Erschaffung das System immer noch nicht so richtig zu begreifen scheinen, erfreuen sich die Ideen der UNIX-Philospohie größter Beliebtheit beim avarage hacker wie beim newbie. Die Verwirrung hat Methode. Seitdem "UNICS" aus der Taufe gehoben wurde entwickelte sich ein Stammbaum, der unüberschaubarer nicht sein könnte. Namen wie UNIX V6, V7, System III, System V, BSD, 2BSD, 4BSD, SunOS, XENIX u.v.a.m. stehen für eine jahrelange Weiterentwicklung des Kernels, der Programmierumgebung und des sogenannten User Interfaces. In diesem Wettlauf der Konzepte blieben nur wenige Modelle am Leben, trotzdem gibt es auch heute noch nicht _das_ UNIX. UNIX ist also Philosphie und inspiriert heute Systeme wie Linux, GNU, OSF/1, Solaris und - man staune - MacOS X. Chaosradio liefert eine Übersicht über die Entwicklungsgeschichte von UNIX und zeigt die wesentliche Stärken des Systems auf, die auch am Übergang zur seiner vierten Dekade erkennen lassen: UNIX is here to stay. Happy birthday UNIX!