POPULARITY
37 episodes with bsds
2 episodes with bsds
2 episodes with bsds
2 episodes with bsds
2 episodes with bsds
2 episodes with bsds
2 episodes with bsds
FreeBSD 13.5-RELEASE Now Available, From Chaos to Clarity: How We Tackled FreeBSD's 7,000 Bug Backlog, zfs-2.3.1, Complications of funding an open source operating system, Why Choose to Use the BSDs in 2025, First Use on GhostBSD, Better Shell History Search, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD 13.5-RELEASE Now Available (https://lists.freebsd.org/archives/freebsd-announce/2025-March/000181.html) From Chaos to Clarity: How We Tackled FreeBSD's 7,000 Bug Backlog (https://freebsdfoundation.org/blog/from-chaos-to-clarity-how-we-tackled-freebsds-7000-bug-backlog/) News Roundup zfs-2.3.1 (https://github.com/openzfs/zfs/releases/tag/zfs-2.3.1) Complications of funding an open source operating system (https://posixcafe.org/blogs/2025/03/11/0/) Why Choose to Use the BSDs in 2025 (https://it-notes.dragas.net/2025/03/23/osday-2025-why-choose-bsd-in-2025/) First Use on GhostBSD (https://technophobeconfessions.wordpress.com/2025/03/18/first-use-on-ghostbsd/) Better Shell History Search (https://tratt.net/laurie/blog/2025/better_shell_history_search.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Russell - Questions (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/606/feedback/russell%20-%20questions.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
I Tried FreeBSD as a Desktop in 2025. Here's How It Went, Cray 1 Supercomputer Performance Comparisons With Home Computers Phones and Tablets, The first perfect computer, Find Name Wildcard Gotcha, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines I Tried FreeBSD as a Desktop in 2025. Here's How It Went (https://www.howtogeek.com/i-tried-freebsd-as-a-desktop-heres-how-it-went/) Cray 1 Supercomputer Performance Comparisons With Home Computers Phones and Tablets (http://www.roylongbottom.org.uk/Cray%201%20Supercomputer%20Performance%20Comparisons%20With%20Home%20Computers%20Phones%20and%20Tablets.htm) News Roundup State of virtualizing the BSDs on Apple Silicon (https://briancallahan.net/blog/20250222.html) The first perfect computer (https://celso.io/posts/2025/01/26/the-first-perfect-computer/) Find Name Wildcard Gotcha (https://utcc.utoronto.ca/~cks/space/blog/unix/FindNameWildcardGotcha) New Patreon Levels Level 1 - user memory (Tip Jar) @ $1 / month Show your support for the show Level 2 - virtual memory (Ad-Free Episodes) @ $5 / month Ad-free episodes Level 3 - kmem (VIP Patron) @ $10 / month Everything in higher memory levels & Your feedback and questions jump the queue and go in the next episode. Personal shout outs (with your consent) for recommending articles we cover. Level 4 - physical memory @ $20 / month What's included: Everything in higher memory levels & You can send in audio/video questions and we'll air your audio in the show feedback section (if the quality of your recording is decent) Behind-the-scenes content - Raw Video from Recording sessions with intro/outro discussion not included in the show Additional Content when we all make it Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
Debunking Common Myths About FreeBSD - Part 2, FreeBSD 13.4-RELEASE Announcement, OpenBSD -current has moved to version 7.6, acpidumping,Install snac2 on FreeBSD – An ActivityPub Instance for the Fediverse, Managing dotfiles with chezmoi, Podman testing on FreeBSD, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Debunking Common Myths About FreeBSD - Part 2 (https://klarasystems.com/articles/debunking-common-myths-about-freebsd-2/) FreeBSD 13.4-RELEASE Announcement (https://www.freebsd.org/releases/13.4R/announce/) FreeBSD 14.0 end-of-life (https://bsdsec.net/articles/freebsd-14-0-end-of-life) - You should have upgraded to 14.1 by now OpenBSD -current has moved to version 7.6 (https://www.undeadly.org/cgi?action=article;sid=20240918052239) News Roundup acpidumping (https://adventurist.me/posts/00325) Install snac2 on FreeBSD – An ActivityPub Instance for the Fediverse (https://gyptazy.com/install-snac2-on-freebsd-an-activitypub-instance-for-the-fediverse/) Installing Uptime-Kuma on a FreeBSD Jail (https://it-notes.dragas.net/2024/07/22/install-uptime-kuma-freebsd-jail/) Managing dotfiles with chezmoi (https://stoddart.github.io/2024/09/08/managing-dotfiles-with-chezmoi.html) Podman testing on FreeBSD (https://github.com/oci-playground/freebsd-podman-testing) Undeadly Bits OpenSSH 9.9 released! (https://undeadly.org/cgi?action=article;sid=20240921181110) OpenBSD now enforcing no invalid NUL characters in shell scripts (http://undeadly.org/cgi?action=article;sid=20240924105732) EuroBSDCon 2024 presentations are now up (http://undeadly.org/cgi?action=article;sid=20240924092154) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions rel4x - Secure by default (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/581/feedback/rel4x%20-%20Secure%20by%20default.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
The XZ Backdoor, NetBSD 10.0, iX announces that they will put out a release of TrueNAS 13.3, State of the Terminal, LibreSSL 3.8.4 and 3.9.1 released and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines People have no doubt heard of this by now, but are not aware of the BSD side of things since its mostly been Linux getting all the news. It'd be nice if we could give a summary of the issue and then address how it does/doesn't affect the BSDs. The XZ Backdoor (https://boehs.org/node/everything-i-know-about-the-xz-backdoor) NetBSD's statement (https://blog.netbsd.org/tnf/entry/statement_on_backdoor_in_xz) FreeBSD's statement (https://lists.freebsd.org/archives/freebsd-security/2024-March/000248.html) OpenBSD? NetBSD 10.0 (https://netbsd.org/releases/formal-10/NetBSD-10.0.html) News Roundup iX announces that they will put out a release of TrueNAS 13.3 (https://www.truenas.com/blog/truenas-core-13-3-plans/) A community fork has been announced (https://www.zvault.io) State of the Terminal (https://gpanders.com/blog/state-of-the-terminal/) LibreSSL 3.8.4 and 3.9.1 released (https://undeadly.org/cgi?action=article;sid=20240328181819) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Derek via feedback has asked for some discussion around this NetBSD security advisory (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/554/feedback/Derek%20-%20NetBSD%20Security%20Advisory.md) -- Advisory Link (https://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2024-001.txt.asc) Ben - Nextcloud Installation (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/554/feedback/Ben%20-%20nexcloud%20installation.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
This week we're joined by FreeBSD & OpenZFS developer, Allan Jude, to learn all about FreeBSD. Allan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there & answers every curious question we have about this powerful (yet underrepresented) Unix-based operating system.
This week we're joined by FreeBSD & OpenZFS developer, Allan Jude, to learn all about FreeBSD. Allan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there & answers every curious question we have about this powerful (yet underrepresented) Unix-based operating system.
The Mood Stabilizing Boys of Comedy™ take the Bipolar Spectrum Diagnostic Scale. This week we chat about mania, hypomania, depression, and mental health. Does your mood and energy fluctuate wildly at times? Do you cycle between periods of hyperactivity and extreme depression? Do you ever go long periods without sleeping? Listen to the episode and take the questionnaire along with The MSBC™ to see if you can out bipolar Shane and win the worst/best prize (depending on the day): http://www.thecarlatcmeinstitute.com/complete/html/assess/BSDS.pdf You can always listen to the show wherever you podcast or go to Spotify or YouTube for video options. The Mind Under Matter Campout Festival in Raleigh is right around the corner! Get tickets for Friday Sep 9 thru Sunday Sep 11 and join us for music, science talks, art, comedy, wellness and surprises! https://mindunderpod.com/pages/campout
Differences between base and ports LLVM in OpenBSD, Netgraph for FreeBSD's bhyve Networking, Audio on FreeBSD – Quick Guide, FreeBSD's Legend starts at 1.0, Hacker News running by FreeBSD, TrueNAS 13, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Differences between base and ports LLVM in OpenBSD (https://www.cambus.net/differences-between-base-and-ports-llvm-in-openbsd/) Using Netgraph for FreeBSD's bhyve Networking (https://klarasystems.com/articles/using-netgraph-for-freebsds-bhyve-networking/?utm_source=bsdweekly) News Roundup Audio on FreeBSD – Quick Guide (https://freebsdfoundation.org/freebsd-project/resources/audio-on-freebsd/) [Legends start at 1.0! – FreeBSD in 1993] Part 1 (https://eerielinux.wordpress.com/2022/06/18/legends-start-at-1-0-freebsd-in-1993-pt-1/) Part 2 (https://eerielinux.wordpress.com/2022/06/19/legends-start-at-1-0-freebsd-in-1993-pt-2/) *** ### Hacker News running by FreeBSD. Take that, Linux! (https://news.ycombinator.com/item?id=16076041) *** ### TrueNAS 13 (https://www.theregister.com/2022/05/11/truenas_13_released/) *** Beastie Bits Notable OpenBSD news you may have missed, 2022-06-28 edition (http://undeadly.org/cgi?action=article;sid=20220628135253) rEFInd design for all the BSDs (https://github.com/indgy/refind-bsd-black) OpenBGPD 7.4 released (https://undeadly.org/cgi?action=article;sid=20220619185920) Hotfix GhostBSD 22.06.18 ISO is now available (http://ghostbsd.org/22.06.18_iso_is_now_available) *** ###Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Brad - Jails Question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/463/feedback/Brad%20-%20Jails%20Question.md) Freezr - A few questions (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/463/feedback/Freezr%20-%20A%20few%20questions.md) A different Brad - Drive question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/463/feedback/A%20different%20Brad%20-%20Drive%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Episode 2 of Something To Talk About was a very open and honest conversation with Julie Bishop. The former Minister for Foreign Affairs and deputy leader of the Liberal Party was very frank about the events that had made headlines from Canberra, #metoo, she gave her thoughts on the need for independent counselling at Parliament House and responded to the rumours that her leadership bid was blocked by the "BSDs." See omnystudio.com/listener for privacy information.
In this very timely episode former Minister for Foreign Affairs and deputy leader of the Liberal Party Julie Bishop is Sam's guest. In the first half of this episode Sam and Julie discuss the recent events that have made headlines from Canberra, #metoo, Julie's thoughts on the need for independent counselling at Parliament House, Julie responds to the rumours that her leadership bid was blocked by the "BSDs," her work for justice for the families of MH17 and her time spent with Andrew Chan and Myuran Sukumaran before their execution. Sam and Julie also talk about some light subjects, including Julie's thoughts on how her first day as Chancellor of the ANU was greeted by the apocalypse, who she thought was the coolest world leader and Julie tells an unbelievable story about a dry dinner attended by key world leaders....and one of the world leaders had snuck in a hip flask of whiskey! Episode notes - For more information on Julie Bishop click here or copy this into your browser: https://www.aph.gov.au/Senators_and_Members/Parliamentarian?MPID=83P and this: www.juliebishopandpartners.com.au Julie's twitter is @honjuliebishop Here is the link for International Women's Day: https://www.internationalwomensday.com/ This episode of Something To Talk About was recorded in Sydney on the morning of Thursday March 4th 2021 You can find Stellar magazine in print every Sunday, pick up the latest copy inside The Sunday Telegraph (NSW), the Sunday Herald Sun (Victoria), The Sunday Mail (Queensland), Sunday Mail (SA) and Sunday Tasmanian (Tasmania). See omnystudio.com/listener for privacy information.
Looking at Lumina Desktop 2.0, 2 months of KPTI development in SmartOS, OpenBSD email service, an interview with Ryan Zezeski, NomadBSD released, and John Carmack's programming retreat with OpenBSD. This episode was brought to you by Headlines Looking at Lumina Desktop 2.0 (https://www.trueos.org/blog/looking-lumina-desktop-2-0/) A few weeks ago I sat down with Lead Developer Ken Moore of the TrueOS Project to get answers to some of the most frequently asked questions about Lumina Desktop from the open source community. Here is what he said on Lumina Desktop 2.0. Do you have a question for Ken and the rest of the team over at the TrueOS Project? Make sure to read the interview and comment below. We are glad to answer your questions! Ken: Lumina Desktop 2.0 is a significant overhaul compared to Lumina 1.x. Almost every single subsystem of the desktop has been streamlined, resulting in a nearly-total conversion in many important areas. With Lumina Desktop 2.0 we will finally achieve our long-term goal of turning Lumina into a complete, end-to-end management system for the graphical session and removing all the current runtime dependencies from Lumina 1.x (Fluxbox, xscreensaver, compton/xcompmgr). The functionality from those utilities is now provided by Lumina Desktop itself. Going along with the session management changes, we have compressed the entire desktop into a single, multi-threaded binary. This means that if any rogue script or tool starts trying to muck about with the memory used by the desktop (probably even more relevant now than when we started working on this), the entire desktop session will close/crash rather than allowing targeted application crashes to bypass the session security mechanisms. By the same token, this also prevents “man-in-the-middle” type of attacks because the desktop does not use any sort of external messaging system to communicate (looking at you dbus). This also gives a large performance boost to Lumina Desktop The entire system for how a user's settings get saved and loaded has been completely redone, making it a “layered” settings system which allows the default settings (Lumina) to get transparently replaced by system settings (OS/Distributor/SysAdmin) which can get replaced by individual user settings. This results in the actual changes in the user setting files to be kept to a minimum and allows for a smooth transition between updates to the OS or Desktop. This also provides the ability to “restrict” a user's desktop session (based on a system config file) to the default system settings and read-only user sessions for certain business applications. The entire graphical interface has been written in QML in order to fully-utilize hardware-based GPU acceleration with OpenGL while the backend logic and management systems are still written entirely in C++. This results in blazing fast performance on the backend systems (myriad multi-threaded C++ objects) as well as a smooth and responsive graphical interface with all the bells and whistles (drag and drop, compositing, shading, etc). Q: Are there future plans to implement something like Lumina in a MAC Jail? While I have never tried out Lumina in a MAC jail, I do not see anything on that page which should stop it from running in one right now. Lumina is already designed to be run as an unpriviledged user and is very smart about probing the system to find out what is/not available before showing anything to the user. The only thing that comes to mind is that you might need to open up some other system devices so that X11 itself can draw to the display (graphical environment setup is a bit different than CLI environment). Q: I look forward to these changes. I know the last time I used it when I would scroll I would get flashes like the refresh rate was not high enough. It will be nice to have a fast system as well as I know with the more changes Linux is becoming slower. Not once it has loaded but in the loading process. I will do another download when these changes come out and install again and maybe stay this time. If I recall correctly, one of the very first versions of Lumina (pre-1.0) would occasionally flicker. If that is still happening, you might want to verify that you are using the proper video driver for your hardware and/or enable the compositor within the Lumina settings. Q: Why was enlightenment project not considered for TrueOS? It is BSD licensed and is written in C. This was a common question about 4(?) years ago with the first release of the Lumina desktop and it basically boiled down to long-term support and reliability of the underlying toolkit. Some of the things we had to consider were: cross-platform/cross-architecture support, dependency reliability and support framework (Qt5 > EFL), and runtime requirements and dependency tracking (Qt5 is lighter than the EFL). That plus the fact that the EFL specifically states that it is linux-focused and the BSD's are just an afterthought (especially at the time we were doing the evaluation). Q: I have two questions. 1) The default layout of Unity(menu bar with actual menu entries on top and icon dock on the side) is one of the few things I liked about my first voyage into non-Windows systems, and have been missing since moving on to other distros(and now also other non-Linux systems). However in 1.4.0 screenshots on Lumina's site, the OSX-like layout has the menu attached to the window. Will 2.0 be able to have the menus on the bar? 2) Is there any timeline for a public release, or are you taking a “when it's ready” approach? In Lumina you can already put panels on the left/right side of the screen and give you something like the layout of the Unity desktop. The embedded menu system is not available in Lumina because that is not a specification supported by X11 and the window manager standards at the present time. The way that functionality is currently run on Linux is a hacky-bypass of the display system which only really works with the GTK3 and Qt5 toolkits, resulting in very odd overall desktop behavior in mixed environments where some apps use other graphical toolkits. We are targetting the 18.06 STABLE release of TrueOS for Lumina 2, but that is just a guideline and if necessary we will push back the release date to allow for additional testing/fixing as needed. A long two months (https://blog.cooperi.net/a-long-two-months) IllumOS/SmartOS developer Alex Wilson describes the journey of developing KPTI for IllumOS > On Monday (January 1st) I had the day off work for New Year's day, as is usual in most of the western world, so I slept in late. Lou and her friend decided to go to the wax museum and see several tourist attractions around SF, and I decided to pass the day at home reading. That afternoon, work chat started talking about a Tumblr post by pythonsweetness about an Intel hardware security bug. At the time I definitely did not suspect that this was going to occupy most of my working life for the next (almost) two months. Like many people who work on system security, I had read Anders Fogh's post about a "Negative Result" in speculative execution research in July of 2017. At the time I thought it was an interesting writeup and I remember being glad that researchers were looking into this area. I sent the post to Bryan and asked him about his thoughts on it at the time, to which he replied saying that "it would be shocking if they left a way to directly leak out memory in the speculative execution". None of us seriously thought that there would be low-hanging fruit down that research path, but we also felt it was important that there was someone doing work in the area who was committed to public disclosure. At first, after reading the blog post on Monday, we thought (or hoped) that the bug might "just" be a KASLR bypass and wouldn't require a lot of urgency. We tried to reach out to Intel at work to get more information but were met with silence. (We wouldn't hear back from them until after the disclosure was already made public.) The speculation on Tuesday intensified, until finally on Wednesday morning I arrived at the office to find links to late Tuesday night tweets revealing exploits that allowed arbitrary kernel memory reads. Wednesday was not a happy day. Intel finally responded to our emails -- after they had already initiated public disclosure. We all spent a lot of time reading. An arbitrary kernel memory read (an info leak) is not that uncommon as far as bugs go, but for the most part they tend to be fairly easy to fix. The thing that makes the Meltdown and Spectre bugs particularly notable is that in order to mitigate them, a large amount of change is required in very deep low-level parts of the kernel. The kind of deep parts of the kernel where there are 20-year old errata workarounds that were single-line changes that you have to be very careful to not accidentally undo; the kind of parts where, as they say, mortals fear to tread. On Friday we saw the patches Matthew Dillon put together for DragonFlyBSD for the first time. These were the first patches for KPTI that were very straightforward to read and understand, and applied to a BSD-derived kernel that was similar to those I'm accustomed to working on. To mitigate Meltdown (and partially one of the Spectre variants), you have to make sure that speculative execution cannot reach any sensitive data from a user context. This basically means that the pages the kernel uses for anything potentially sensitive have to be unmapped when we are running user code. Traditionally, CPUs that were built to run a multi-user, UNIX-like OS did this by default (SPARC is an example of such a CPU which has completely separate address spaces for the kernel and userland). However, x86 descends from a single-address-space microcontroller that has grown up avoiding backwards-incompatible changes, and has never really introduced a clean notion of multiple address spaces (segmentation is the closest feature really, and it was thrown out for 64-bit AMD64). Instead, operating systems for x86 have generally wound up (at least in the post-AMD64 era) with flat address space models where the kernel text and data is always present in the page table no matter whether you're in user or kernel mode. The kernel mappings simply have the "supervisor" bit set on them so that user code can't directly access them. The mitigation is basically to stop doing this: to stop mapping the kernel text, data and other memory into the page table while we're running in userland. Unfortunately, the x86 design does not make this easy. In order to be able to take interrupts or traps, the CPU has to have a number of structures mapped in the current page table at all times. There is also no ability to tell an x86 CPU that you want it to switch page tables when an interrupt occurs. So, the code that we jump to when we take an interrupt, as well as space for a stack to push context onto have to be available in both page tables. And finally, of course, we need to be able to figure out somehow what the other page table we should switch to is when we enter the kernel. When we looked at the patches for Linux (and also the DragonFlyBSD patches at the time) on Friday and started asking questions, it became pretty evident that the initial work done by both was done under time constraints. Both had left the full kernel text mapped in both page tables, and the Linux trampoline design seemed over-complex. I started talking over some ideas with Robert Mustacchi about ways to fix these and who we should talk to, and reached out to some of my old workmates from the University of Queensland who were involved with OpenBSD. It seemed to me that the OpenBSD developers would care about these issues even more than we did, and would want to work out how to do the mitigation right. I ended up sending an email to Philip Guenther on Friday afternoon, and on Saturday morning I drove an hour or so to meet up with him for coffee to talk page tables and interrupt trampolines. We wound up spending a good 6 hours at the coffee shop, and I came back with several pages of notes and a half-decent idea of the shape of the work to come. One detail we missed that day was the interaction of per-CPU structures with per-process page tables. Much of the interrupt trampoline work is most easily done by using per-CPU structures in memory (and you definitely want a per-CPU stack!). If you combine that with per-process page tables, however, you have a problem: if you leave all the per-CPU areas mapped in all the processes, you will leak information (via Meltdown) about the state of one process to a different one when taking interrupts. In particular, you will leak things like %rip, which ruins all the work being done with PIE and ASLR pretty quickly. So, there are two options: you can either allocate the per-CPU structures per-process (so you end up with $NCPUS * $NPROCS of them); or you can make the page tables per-CPU. OpenBSD, like Linux and the other implementations so far, decided to go down the road of per-CPU per-process pages to solve this issue. For illumos, we took the other route. In illumos, it turned out that we already had per-CPU page tables. Robert and I re-discovered this on the Sunday of that week. We use them for 32-bit processes due to having full P>V PAE support in our kernel (which is, as it turns out, relatively uncommon amongst open-source OS). The logic to deal with creating and managing them and updating them was all already written, and after reading the code we concluded we could basically make a few small changes and re-use all of it. So we did. By the end of that second week, we had a prototype that could get to userland. But, when working on this kind of kernel change we have a rule of thumb we use: after the first 70% of the patch is done and we can boot again, now it's time for the second 70%. In fact it turned out to be more like the second 200% for us -- a tedious long tail of bugs to solve that ended up necessitating some changes in the design as well. At first we borrowed the method that Matt Dillon used for DragonFlyBSD, by putting the temporary "stack" space and state data for the interrupt trampolines into an extra page tacked onto the end of *%gs (in illumos the structure that lives there is the cpu_t). If you read the existing logic in interrupt handlers for dealing with %gs though, you will quickly notice that the corner cases start to build up. There are a bunch of situations where the kernel temporarily alters %gs, and some of the ways to mess it up have security consequences that end up being worse than the bug we're trying to fix. As it turns out, there are no less than 3 different ways that ISRs use to try to get to having the right cpu_t in %gs on illumos, as it turns out, and they are all subtly different. Trying to tell which you should use when requires a bunch of test logic that in turn requires branches and changes to the CPU state, which is difficult to do in a trampoline where you're trying to avoid altering that state as much as possible until you've got the real stack online to push things into. I kept in touch with Philip Guenther and Mike Larkin from the OpenBSD project throughout the weeks that followed. In one of the discussions we had, we talked about the NMI/MCE handlers and the fact that their handling currently on OpenBSD neglected some nasty corner-cases around interrupting an existing trap handler. A big part of the solution to those issues was to use a feature called IST, which allows you to unconditionally change stacks when you take an interrupt. Traditionally, x86 only changes the stack pointer (%rsp on AMD64) while taking an interrupt when there is a privilege level change. If you take an interrupt while already in the kernel, the CPU does not change the stack pointer, and simply pushes the interrupt stack frame onto the stack you're already using. IST makes the change of stack pointer unconditional. If used unwisely, this is a bad idea: if you stay on that stack and turn interrupts back on, you could take another interrupt and clobber the frame you're already in. However, in it I saw a possible way to simplify the KPTI trampoline logic and avoid having to deal with %gs. A few weeks into the project, John Levon joined us at work. He had previously worked on a bunch of Xen-related stuff as well as other parts of the kernel very close to where we were, so he quickly got up to speed with the KPTI work as well. He and I drafted out a "crazy idea" on the whiteboard one afternoon where we would use IST for all interrupts on the system, and put the "stack" they used in the KPTI page on the end of the cpu_t. Then, they could easily use stack-relative addresses to get the page table to change to, then pivot their stack to the real kernel stack memory, and throw away (almost) all the conditional logic. A few days later, we had convinced each other that this was the way to go. Two of the most annoying x86 issues we had to work around were related to the SYSENTER instruction. This instruction is used to make "fast" system calls in 32-bit userland. It has a couple of unfortunate properties: firstly, it doesn't save or restore RFLAGS, so the kernel code has to take care of this (and be very careful not to clobber any of it before saving or after restoring it). Secondly, if you execute SYSENTER with the TF ("trap"/single-step flag) set by a debugger, the resulting debug trap's frame points at kernel code instead of the user code where it actually happened. The first one requires some careful gymnastics on the entry and return trampolines specifically for SYSENTER, while the second is a nasty case that is incidentally made easier by using IST. With IST, we can simply make the debug trap trampoline check for whether we took the trap in another trampoline's code, and reset %cr3 and the destination stack. This works for single-stepping into any of the handlers, not just the one for SYSENTER. To make debugging easier, we decided that traps like the debug/single-step trap (as well as faults like page faults, #GP, etc.) would push their interrupt frame in a different part of the KPTI state page to normal interrupts. We applied this change to all the traps that can interrupt another trampoline (based on the instructions we used). These "paranoid" traps also set a flag in the KPTI struct to mark it busy (and jump to the double-fault handler if it is), to work around some bugs where double-faults are not correctly generated. It's been a long and busy two months, with lots of time spent building, testing, and validating the code. We've run it on as many kinds of machines as we could get our hands on, to try to make sure we catch issues. The time we've spent on this has been validated several times in the process by finding bugs that could have been nasty in production. One great example: our patches on Westmere-EP Xeons were causing busy machines to throw a lot of L0 I-cache parity errors. This seemed very mysterious at first, and it took us a few times seeing it to believe that it was actually our fault. This was actually caused by the accidental activation of a CPU errata for Westmere (B52, "Memory Aliasing of Code Pages May Cause Unpredictable System Behaviour") -- it turned out we had made a typo and put the "cacheable" flag into a variable named flags instead of attrs where it belonged when setting up the page tables. This was causing performance degradation on other machines, but on Westmere it causes cache parity errors as well. This is a great example of the surprising consequences that small mistakes in this kind of code can end up having. In the end, I'm glad that that erratum existed, otherwise it may have been a long time before we caught that bug. As of this week, Mike and Philip have committed the OpenBSD patches for KPTI to their repository, and the patches for illumos are out for review. It's a nice kind of symmetry that the two projects who started on the work together after the public disclosure at the same time are both almost ready to ship at the same time at the other end. I'm feeling hopeful, and looking forward to further future collaborations like this with our cousins, the BSDs. The IllumOS work has since landed, on March 12th (https://github.com/joyent/illumos-joyent/commit/d85fbfe15cf9925f83722b6d62da49d549af615c) *** OpenBSD Email Service (https://github.com/vedetta-com/caesonia) Features Efficient: configured to run on min. 512MB RAM and 20GB SSD, a KVM (cloud) VPS for around $2.50/mo 15GB+ uncompressed Maildir, rivals top free-email providers (grow by upgrading SSD) Email messages are gzip compressed, at least 1/3 more space with level 6 default Server side full text search (headers and body) can be enabled (to use the extra space) Mobile data friendly: IMAPS connections are compressed Subaddress (+tag) support, to filter and monitor email addresses Virtual domains, aliases, and credentials in files, Berkeley DB, or SQLite3 Naive Bayes rspamd filtering with supervised learning: the lowest false positive spam detection rates Carefree automated Spam/ and Trash/ cleaning service (default: older than 30 days) Automated quota management, gently assists when over quota Easy backup MX setup: using the same configuration, install in minutes on a different host Worry-free automated master/master replication with backup MX, prevents accidental loss of email messages Resilient: the backup MX can be used as primary, even when the primary is not down, both perfect replicas Flexible: switching roles is easy, making the process of changing VPS hosts a breeze (no downtime) DMARC (with DKIM and SPF) email-validation system, to detect and prevent email spoofing Daily (spartan) stats, to keep track of things Your sieve scripts and managesieve configuration, let's get started Considerations By design, email message headers need to be public, for exchanges to happen. The body of the message can be encrypted by the user, if desired. Moreover, there is no way to prevent the host from having access to the virtual machine. Therefore, full disk encryption (at rest) may not be necessary. Given our low memory requirements, and the single-purpose concept of email service, Roundcube or other web-based IMAP email clients should be on a different VPS. Antivirus software users (usually) have the service running on their devices. ClamAV can easily be incorporated into this configuration, if affected by the types of malware it protects against, but will require around 1GB additional RAM (or another VPS). Every email message is important, if properly delivered, for Bayes classification. At least 200 ham and 200 spam messages are required to learn what one considers junk. By default (change to use case), a rspamd score above 50% will send the message to Spam/. Moving messages in and out of Spam/ changes this score. After 95%, the message is flagged as "seen" and can be safely ignored. Spamd is effective at greylisting and stopping high volume spam, if it becomes a problem. It will be an option when IPv6 is supported, along with bgp-spamd. System mail is delivered to an alias mapped to a virtual user served by the service. This way, messages are guaranteed to be delivered via encrypted connection. It is not possible for real users to alias, nor mail an external mail address with the default configuration. e.g. puffy@mercury.example.com is wheel, with an alias mapped to (virtual) puffy@example.com, and user (puffy) can be different for each. Interview - Ryan Zezeski - rpz@joyent.com (mailto:rpz@joyent.com) / @rzezeski (https://twitter.com/rzezeski) News Roundup John Carmack's programming retreat to hermit coding with OpenBSD (https://www.facebook.com/permalink.php?story_fbid=2110408722526967&id=100006735798590) After a several year gap, I finally took another week-long programming retreat, where I could work in hermit mode, away from the normal press of work. My wife has been generously offering it to me the last few years, but I'm generally bad at taking vacations from work. As a change of pace from my current Oculus work, I wanted to write some from-scratch-in-C++ neural network implementations, and I wanted to do it with a strictly base OpenBSD system. Someone remarked that is a pretty random pairing, but it worked out ok. Despite not having actually used it, I have always been fond of the idea of OpenBSD — a relatively minimal and opinionated system with a cohesive vision and an emphasis on quality and craftsmanship. Linux is a lot of things, but cohesive isn't one of them. I'm not a Unix geek. I get around ok, but I am most comfortable developing in Visual Studio on Windows. I thought a week of full immersion work in the old school Unix style would be interesting, even if it meant working at a slower pace. It was sort of an adventure in retro computing — this was fvwm and vi. Not vim, actual BSD vi. In the end, I didn't really explore the system all that much, with 95% of my time in just the basic vi / make / gdb operations. I appreciated the good man pages, as I tried to do everything within the self contained system, without resorting to internet searches. Seeing references to 30+ year old things like Tektronix terminals was amusing. I was a little surprised that the C++ support wasn't very good. G++ didn't support C++11, and LLVM C++ didn't play nicely with gdb. Gdb crashed on me a lot as well, I suspect due to C++ issues. I know you can get more recent versions through ports, but I stuck with using the base system. In hindsight, I should have just gone full retro and done everything in ANSI C. I do have plenty of days where, like many older programmers, I think “Maybe C++ isn't as much of a net positive as we assume...”. There is still much that I like, but it isn't a hardship for me to build small projects in plain C. Maybe next time I do this I will try to go full emacs, another major culture that I don't have much exposure to. I have a decent overview understanding of most machine learning algorithms, and I have done some linear classifier and decision tree work, but for some reason I have avoided neural networks. On some level, I suspect that Deep Learning being so trendy tweaked a little bit of contrarian in me, and I still have a little bit of a reflexive bias against “throw everything at the NN and let it sort it out!” In the spirit of my retro theme, I had printed out several of Yann LeCun's old papers and was considering doing everything completely off line, as if I was actually in a mountain cabin somewhere, but I wound up watching a lot of the Stanford CS231N lectures on YouTube, and found them really valuable. Watching lecture videos is something that I very rarely do — it is normally hard for me to feel the time is justified, but on retreat it was great! I don't think I have anything particularly insightful to add about neural networks, but it was a very productive week for me, solidifying “book knowledge” into real experience. I used a common pattern for me: get first results with hacky code, then write a brand new and clean implementation with the lessons learned, so they both exist and can be cross checked. I initially got backprop wrong both times, comparison with numerical differentiation was critical! It is interesting that things still train even when various parts are pretty wrong — as long as the sign is right most of the time, progress is often made. I was pretty happy with my multi-layer neural net code; it wound up in a form that I can just drop it into future efforts. Yes, for anything serious I should use an established library, but there are a lot of times when just having a single .cpp and .h file that you wrote ever line of is convenient. My conv net code just got to the hacky but working phase, I could have used another day or two to make a clean and flexible implementation. One thing I found interesting was that when testing on MNIST with my initial NN before adding any convolutions, I was getting significantly better results than the non-convolutional NN reported for comparison in LeCun ‘98 — right around 2% error on the test set with a single 100 node hidden layer, versus 3% for both wider and deeper nets back then. I attribute this to the modern best practices —ReLU, Softmax, and better initialization. This is one of the most fascinating things about NN work — it is all so simple, and the breakthrough advances are often things that can be expressed with just a few lines of code. It feels like there are some similarities with ray tracing in the graphics world, where you can implement a physically based light transport ray tracer quite quickly, and produce state of the art images if you have the data and enough runtime patience. I got a much better gut-level understanding of overtraining / generalization / regularization by exploring a bunch of training parameters. On the last night before I had to head home, I froze the architecture and just played with hyperparameters. “Training!” Is definitely worse than “Compiling!” for staying focused. Now I get to keep my eyes open for a work opportunity to use the new skills! I am dreading what my email and workspace are going to look like when I get into the office tomorrow. Stack-register Checking (https://undeadly.org/cgi?action=article;sid=20180310000858) Recently, Theo de Raadt (deraadt@) described a new type of mitigation he has been working on together with Stefan Kempf (stefan@): How about we add another new permission! This is not a hardware permission, but a software permission. It is opportunistically enforced by the kernel. The permission is MAP_STACK. If you want to use memory as a stack, you must mmap it with that flag bit. The kernel does so automatically for the stack region of a process's stack. Two other types of stack occur: thread stacks, and alternate signal stacks. Those are handled in clever ways. When a system call happens, we check if the stack-pointer register points to such a page. If it doesn't, the program is killed. We have tightened the ABI. You may no longer point your stack register at non-stack memory. You'll be killed. This checking code is MI, so it works for all platforms. For more detail, see Theo's original message (https://marc.info/?l=openbsd-tech&m=152035796722258&w=2). This is now available in snapshots, and people are finding the first problems in the ports tree already. So far, few issues have been uncovered, but as Theo points out, more testing is necessary: Fairly good results. A total of 4 problems have been found so far. go, SBCL, and two cases in src/regress which failed the new page-alignment requirement. The SBCL and go ones were found at buildtime, since they use themselves to complete build. But more page-alignment violations may be found in ports at runtime. This is something I worry about a bit. So please everyone out there can help: Use snapshots which contain the stack-check diff, update to new packages, and test all possible packages. Really need a lot of testing for this, so please help out. So, everybody, install the latest snapshot and try all your favorite ports. This is the time to report issues you find, so there is a good chance this additional security feature is present in 6.3 (and works with third party software from packages). NomadBSD 1.0 has been released (https://freeshell.de/~mk/projects/nomadbsd.html) NomadBSD is a live system for flash drives, based on FreeBSD® 11.1 (amd64) Change Log The setup process has been improved. Support for optional geli encryption of the home partition has been added Auto-detection of NVIDIA graphics cards and their corresponding driver has been added. (Thanks to holgerw and lme from BSDForen.de) An rc script to start the GEOM disk scheduler on the root device has been added. More software has been added: accessibility/redshift (starts automatically) audio/cantata audio/musicpd audio/ncmpc ftp/filezilla games/bsdtris mail/neomutt math/galculator net-p2p/transmission-qt5 security/fpm2 sysutils/bsdstats x11/metalock x11/xbindkeys Several smaller improvements and bugfixes. Screenshots https://freeshell.de/~mk/projects/nomadbsd-ss1.png https://freeshell.de/~mk/projects/nomadbsd-ss2.png https://freeshell.de/~mk/projects/nomadbsd-ss3.png https://freeshell.de/~mk/projects/nomadbsd-ss4.png https://freeshell.de/~mk/projects/nomadbsd-ss5.png https://freeshell.de/~mk/projects/nomadbsd-ss6.png Beastie Bits KnoxBug - Nagios (http://knoxbug.org/2018-03-27) vBSDcon videos landing (https://www.youtube.com/playlist?list=PLfJr0tWo35bc9FG_reSki2S5S0G8imqB4) AsiaBSDCon 2017 videos (https://www.youtube.com/playlist?list=PLnTFqpZk5ebBTyXedudGm6CwedJGsE2Py) DragonFlyBSD Adds New "Ptr_Restrict" Security Option (https://www.phoronix.com/scan.php?page=news_item&px=DragonFlyBSD-Ptr-Restrict) A Dexter needs your help (https://twitter.com/michaeldexter/status/975603855407788032) Mike Larkin at bhyvecon 2018: OpenBSD vmm(4) update (https://undeadly.org/cgi?action=article;sid=20180309064801) [HEADS UP] - OFED/RDMA stack update (https://lists.freebsd.org/pipermail/freebsd-arch/2018-March/018900.html) *** Feedback/Questions Ron - Interview someone using DragonflyBSD (http://dpaste.com/3BM6GSW#wrap) Brad - Gaming and all (http://dpaste.com/3X4ZZK2#wrap) Mohammad - Sockets vs TCP (http://dpaste.com/0PJMKRD#wrap) Paul - All or at least most of Bryan Cantrill's Talks (http://dpaste.com/2WXVR1X#wrap) ***
AsiaBSDcon review, Meltdown and Spectre Patches in FreeBSD stable, Interview with MidnightBSD founder, 8 months with TrueOS, mysteries of GNU and BSD split This episode was brought to you by Headlines AsiaBSDCon 2018 has concluded (https://2018.asiabsdcon.org/) We have just returned from AsiaBSDCon in Tokyo, Japan last weekend Please excuse our jetlag The conference consisted two days of meeting followed by 2 days of paper presentations We arrived a few days early to see some sights and take a few extra delicious meals in Tokyo The first day of meetings was a FreeBSD developer summit (while Benedict was teaching his two tutorials) where we discussed the FreeBSD release cycle and our thoughts on improving it, the new Casper capsicum helper service, and developments in SDIO which will eventually enable WiFi and SD card readers on more embedded devices The second day of meetings consisted of bhyvecon, a miniconf that covered development in all hypervisors on all BSDs. It also included presentations on the porting of bhyve to IllumOS. Then the conference started There were a number of great presentations, plus an amazing hallway track as usual It was great to see many old friends and to spend time discussing the latest happenings in BSD. A couple of people came by and asked to take a picture with us and we were happy to do that. *** FreeBSD releases Spectre and Meltdown mitigations for 11.1 (https://www.freebsd.org/security/advisories/FreeBSD-SA-18:03.speculative_execution.asc) Speculative execution vulnerability mitigation is a work in progress. This advisory addresses the most significant issues for FreeBSD 11.1 on amd64 CPUs. We expect to update this advisory to include 10.x for amd64 CPUs. Future FreeBSD releases will address this issue on i386 and other CPUs. freebsd-update will include changes on i386 as part of this update due to common code changes shared between amd64 and i386, however it contains no functional changes for i386 (in particular, it does not mitigate the issue on i386). Many modern processors have implementation issues that allow unprivileged attackers to bypass user-kernel or inter-process memory access restrictions by exploiting speculative execution and shared resources (for example, caches). An attacker may be able to read secret data from the kernel or from a process when executing untrusted code (for example, in a web browser). + Meltdown: The mitigation is known as Page Table Isolation (PTI). PTI largely separates kernel and user mode page tables, so that even during speculative execution most of the kernel's data is unmapped and not accessible. A demonstration of the Meltdown vulnerability is available at https://github.com/dag-erling/meltdown. A positive result is definitive (that is, the vulnerability exists with certainty). A negative result indicates either that the CPU is not affected, or that the test is not capable of demonstrating the issue on the CPU (and may need to be modified). A patched kernel will automatically enable PTI on Intel CPUs. The status can be checked via the vm.pmap.pti sysctl PTI introduces a performance regression. The observed performance loss is significant in microbenchmarks of system call overhead, but is much smaller for many real workloads. + Spectre V2: There are two common mitigations for Spectre V2. This patch includes a mitigation using Indirect Branch Restricted Speculation, a feature available via a microcode update from processor manufacturers. The alternate mitigation, Retpoline, is a feature available in newer compilers. The feasibility of applying Retpoline to stable branches and/or releases is under investigation. The patch includes the IBRS mitigation for Spectre V2. To use the mitigation the system must have an updated microcode; with older microcode a patched kernel will function without the mitigation. IBRS can be disabled via the hw.ibrsdisable sysctl (and tunable), and the status can be checked via the hw.ibrsactive sysctl. IBRS may be enabled or disabled at runtime. Additional detail on microcode updates will follow. + Wiki tracking the vulnerabilities and mitigations on different platforms (https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities) Interview with MidnightBSD Founder and Lead Dev Lucas Holt (https://itsfoss.com/midnightbsd-founder-lucas-holt/) Recently, I have taken a little dip into the world of BSD. As part of my attempt to understand the BSD world a little better, I connected with Lucas Holt (MidnightBSD founder and lead developer) to ask him a few questions about his project. Here are his answers. It's FOSS: Please explain MidnightBSD in a nutshell. How is it different than other BSDs? Lucas Holt: MidnightBSD is a desktop focused operating system. When it's considered stable, it will provide a full desktop experience. This differs from other efforts such as TrueOS or GhostBSD in that it's not a distro of FreeBSD, but rather a fork. MidnightBSD has its own package manager, mport as well as unique package cluster software and several features built into user land such as mDNSresponder, libdispatch, and customizations throughout the system. It's FOSS: Who is MidnightBSD aimed at? Lucas Holt: The goal with MidnightBSD has always been to provide a desktop OS that's usable for everyday tasks and that even somewhat non technical people can use. Early versions of Mac OS X were certainly an inspiration. In practice, we're rather far from that goal at this point, but it's been an excellent learning opportunity. It's FOSS: What is your background in computers? Lucas Holt: I started in technical support at a small ISP and moved into web design and system administration. While there, I learned BSDi, Solaris and Linux. I also started tinkering with programming web apps in ASP and a little perl CGI. I then did a mix of programming and system administration jobs through college and graduated with a bachelors in C.S. from Eastern Michigan University. During that time, I learned NetBSD and FreeBSD. I started working on several projects such as porting Apple's HFS+ code to FreeBSD 6 and working on getting the nforce2 chipset SATA controller working with FreeBSD 6, with the latter getting committed. I got a real taste for BSD and after seeing the lack of interest in the community for desktop BSDs, I started MidnightBSD. I began work on it in late 2005. Currently, I'm a Senior Software Engineer focusing on backend rest services by day and a part-time graduate student at the University of Michigan Flint. It's FOSS: I recently installed TrueOS. I was disappointed that a couple of the programs I wanted were not available. The FreeBSD port system looked mildly complicated for beginners. I'm used to using pacman to get the job done quickly. How does MidnightBSD deal with ports? Lucas Holt: MidnightBSD has it's own port system, mports, which shared similarities with FreeBSD ports as well as some ideas from OpenBSD. We decided early on that decent package management was essential for regular users. Power users will still use ports for certain software, but it's just so time consuming to build everything. We started work on our own package manager, mport. Every package is a tar lzma archive with a sqlite3 manifest file as well as a sqlite 3 index that's downloaded from our server. This allows users to query and customize the package system with standard SQL queries. We're also building more user friendly graphical tools. Package availability is another issue that most BSDs have. Software tends to be written for one or two operating systems and many projects are reluctant to support other systems, particularly smaller projects like MidnightBSD. There are certainly gaps. All of the BSD projects need more volunteers to help with porting software and keeping it up to date. It's FOSS: During your June 2015 interview on BSDNow, you mentioned that even though you support both i386 and amd64, that you recommend people choose amd64. Do you have any plans to drop i386 support in the future, like many have done? Lucas Holt: Yes, we do plan to drop i386 support, mostly because of the extra work needed to build and maintain packages. I've held off on this so far because I had a lot of feedback from users in South America that they still needed it. For now, the plan is to keep i386 support through 1.0 release. That's probably a year or two out. It's FOSS: What desktop environments does MidnightBSD support? Lucas Holt: The original plan was to use Etoile as a desktop environment, but that project changed focus. We currently support Xfce, Gnome 3, WindowMaker + GNUstep + Gworkspace as primary choices. We also have several other window managers and desktop environments available such as Enlightenment, rat poison, afterstep, etc. Early versions offered KDE 3.x but we had some issues with KDE 4. We may revisit that with newer versions. It's FOSS: What is MidnightBSD's default filesystem? Do you support DragonflyBSD's HAMMER filesystem? What other filesystems? Lucas Holt: Boot volumes are UFS2. We also support ZFS for additional storage. We have read support for ExFat, NTFS, ext2, CD9660. NFS v3 and v4 are also supported for network file systems. We do not support HAMMER, although it was considered. I would love to see HAMMER2 get added to MidnightBSD eventually. It's FOSS: Is MidnightBSD affected by the recent Spectre and Meltdown issues? Lucas Holt: Yes. Most operating systems were affected by these issues. We were not informed of the issue until the general public became aware. Work is ongoing to come up with appropriate mitigations. Unfortunately, we do not have a patch yet. It's FOSS: The Raspberry Pi and its many clones have made the ARM platform very popular. Are there any plans to make MidnightBSD available on that platform? Lucas Holt: No immediate plans. ARM is an interesting architecture, but by the very nature of SoC designs, takes a lot of work to support a broad number of devices. It might be possible when we stop supporting i386 or if someone volunteers to work on the ARM port. Eventually, I think most hobby systems will need to run ARM chips. Intel's planning on locking down hardware with UEFI 3 and this may make it difficult to run on commodity hardware in the future not only for MidnightBSD but other systems as well. At one point, MidinightBSD ran on sparc64. When workstations were killed off, we dropped support. A desktop OS on a server platform makes little sense. It's FOSS: Does MidnightBSD offer support for Linux applications? Lucas Holt: Yes, we offer Linux emulation. It's emulating a 2.6.16 kernel currently and that needs to be updated so support newer apps. It's possible to run semi-recent versions of Firefox, Thunderbird, Java, and OpenOffice on it though. I've also used it to host game servers in the past and play older games such as Quake 3, enemy territory, etc. It's FOSS: Could you comment on the recent dust-up between the Pale Moon browser developers and the team behind the OpenBSD ports system? [Author's Note: For those who haven't heard about this, let me summarize. Last month, someone from the OpenBSD team added the Pale Moon browser to their ports collection. A Pale Moon developer demanded that they include Pale Moon's libraries instead of using system libraries. As the conversation continued, it got more hostile, especially on the Pale Moon side. The net result is that Pale Moon will not be available on OpenBSD, MidnightBSD, or FreeBSD.] Lucas Holt: I found this discussion frustrating. Many of the BSD projects hear a lot of complaints about browser availability and compatibility. With Firefox moving to Rust, it makes it even more difficult. Then you get into branding issues. Like Firefox, the Pale Moon developers have decided to protect their brand at the cost of users. Unlike the Firefox devs, they've made even stranger requirements for branding. It is not possible to use a system library version of anything with Pale Moon and keep their branding requirements. As such, we cannot offer Pale Moon in MidnightBSD. The reason this is an issue for an open source project is that many third party libraries are used in something as complex as a web browser. For instance, Gecko-based browsers use several multimedia libraries, sqlite3 (for bookmarks), audio and video codecs, etc. Trying to maintain upstream patches for each of these items is difficult. That's why the BSDs have ports collections to begin with. It allows us to track and manage custom patches to make all these libraries work. We go through a lot of effort in keeping these up to date. Sometimes upstream patches don't get included. That means our versions are the only working copies. With pale moon's policy, we'd need to submit separate patches to their customized versions of all these libraries too and any new release of the browser would not be available as changes occur. It might not even be possible to compile pale moon without a patch locally. With regard to Rust, it requires porting the language, as well as an appropriate version of LLVM before you can even start on the browser. It's FOSS: If someone wanted to contribute to your project, both financial and technical, how can they do that? Lucas Holt: Financial assistance for the project can be submitted online. We have a page outlining how to make donations with Patreon, Paypal or via bitcoin. Donations are not tax deductible. You can learn more at http://www.midnightbsd.org/donate/ We also need assistance with translations, porting applications, and working on the actual OS. Interested parties can contact us on the mailing list or through IRC on freenode #midnightbsd We also could use assistance with mirroring ISOs and packages. I would like to thank Lucas for taking the time to reply to my many questions. For more information about MidnightBSD or to download it, please visit their website. The most recent version of MidnightBSD is 0.8.6. News Roundup 8 months with TrueOS (https://inflo.ws/blog/post/2018-03-03-trueos-8th-month-review/) Purpose of this review - what it is and what it is not. I vowed to write down what I felt about TrueOS if I ever got to the six month mark of usage. This is just that. This is neither a tutorial, nor a piece of evangelism dedicated towards it. This is also not a review of specific parts of TrueOS such as Lumina or AppCafe, since I don't use them at all. In the spirit of presenting a screen shot, here is my i3wm displaying 4 windows in one screen - a configuration that I never use. https://inflo.ws/blog/images/trues-screenshot.png The primary tasks I get done with my computer. I need a tiling wm with multi-desktop capability. As regards what I do with a computer, it is fairly straightforward to describe if I just list down my most frequently used applications. xterm (CLI) Emacs (General editing and org mode) Intellij IDEA (Java, Kotlin, SQL) Firefox (Main web browser, with Multi-Account Containers) Thunderbird (Work e-mail) Notmuchmail (Personal e-mail) Chromium/Iridium (Dumb web browser) Telegram Desktop weechat (with wee-slack) cmus (Music player) mpv (Video player) mps-youtube (Youtube client) transmission-gtk Postgresql10 (daemon) Rabbitmq (daemon) Seafile (file sync) Shotwell (manage pictures) GIMP (Edit pictures) Calibre (Manage e-books) VirtualBox All of these are available as binary packages from the repository. Since I use Intellij Ultimate edition, I decided to download the no-jdk linux version from the website rather than install it. This would make sure that it gets updated regularly. Why did I pick TrueOS ? I ran various Linux distributions from 2001 all the way till 2009, till I discovered Arch, and continued with it till 2017. I tried out Void for two months before I switched to TrueOS. Over the last few years, I started feeling like no matter which Linux distribution I touched, they all just stopped making a lot of sense. Generally in the way things were organised, and particularly in terms of software like systemd, which just got pushed down my throat. I couldn't wrap my head around half the things going on in my computer. Mostly I found that Linux distributions stopped becoming a collection of applications that got developed together to something more coupled by software mechanisms like systemd - and that process was more and more opaque. I don't want to talk about the merits and de-merits of systemd, lets just say that I found it of no use and an unnecessary hassle. In February, I found myself in charge of the entire technology stack of a company, and I was free to make choices. A friend who was a long time FreeBSD user convinced me to try it on the servers. My requirement then was to run Postgres, Rabbitmq, Nginx and a couple of JVM processes. The setup was zero hassle and it hasn't changed much in a year. About three months of running FreeBSD-11.x on servers was enough for me to consider it for my laptop. I was very apprehensive of hardware support, but luckily my computer is a Thinkpad, and Thinkpads sort of work out of the box with various BSDs. My general requirements were: Must run Intellij IDEA. Must have proper graphics and sound driver support. Must be able to run VirtualBox. I had to pick from FreeBSD, NetBSD and OpenBSD, since these were the major BSDs that I was familiar with. One of my requirements was that I needed to be able to run VMs just in case I needed to test something on Windows/Linux. This ruled out OpenBSD. Then I was left with NetBSD and FreeBSD. NetBSD's driver support for newer Intel chip-sets were questionable, and FreeBSD was the only choice then. When I was digging through FreeBSD forums, I found out that running the 11.x RELEASE on my laptop was out of the question since it didn't have proper drivers for my chip-set either. A few more hours of digging led me to GhostBSD and TrueOS. I picked TrueOS straightaway because - well because TrueOS came from the old PC-BSD and it was built off FreeBSD-12-CURRENT with the latest drivers integrated. I downloaded the UNSTABLE version available in June 2017, backed up ALL my data and home directory, and then installed it. There were no glitches during installation - I simply followed the installation as described in the handbook and everything was fine. My entire switch from Arch/Void to TrueOS took about an hour, discounting the time it took to backup my data to an external hard disk. It was that easy. Everything I wanted to work just worked, everything was available in the repo. Tweaks from cooltrainer.org : I discovered this excellent tutorial that describes setting up a FreeBSD 11 desktop. It documents several useful tweaks, some of which I applied. A few examples - Fonts, VirtualBox, Firewall, UTF-8 sections. TrueOS (and FreeBSD) specific things I liked Open-rc The open-rc init system is familiar and is well documented. TrueOS specific parts are described here. When I installed postgresql10-server, there was no open-rc script for it, but I could cobble one together in two hours with zero prior experience writing init scripts. Later on I figured out that the init script for postgresql9 would work for 10 as well, and used that. Boot Environments This was an alien concept to me, but the first time I did an update without waiting for a CDN sync to finish, my computer booted into the shell and remained there. The friendly people at TrueOS discourse asked me to roll back to an older BE and wait for sync to finish. I dug through the forums and found "ZFS / Snapshots basics & How-To's for those new to TrueOS". This describes ZFS and BEs, and is well worth reading. ZFS My experience with boot environments was enough to convince me about the utility of ZFS. I am still reading about it and trying things out, and whatever I read just convinces me more about why it is good. File-system layout Coming from the Linux world, how the FreeBSD file-system is laid out seemed odd at first. Then I realised that it was the Linux distros that were doing the odd thing. e.g : The whole OS is split into base system and applications. All the non base system configurations and apps go into /usr/local. That made a lot of sense. The entire OS is developed along with its applications as a single coherent entity, and that shows. Documentation The handbooks for both TrueOS and FreeBSD are really really good. For e.g, I kept some files in an LUKS encrypted drive (when I used Arch Linux). To find an equivalent, all I had to do was read the handbook and look at the GELI section. It is actually nice being able to go to a source like Handbook and things from there just work. Arch Linux and Gentoo has excellent documentation as well, if anyone is wondering about Linux distros. Community The TrueOS community on both Telegram as well as on Discourse are very friendly and patient. They help out a lot and do not get upset when I pose really stupid questions. TrueOS core developers hangout in the Telegram chat-room too, and it is nice being able to talk to them directly about things. What did not work in TrueOS ? The following things that worked during my Linux tenure doesn't work in TrueOS. Netflix Google Hangouts Electron based applications (Slack, Skype) These are not major concerns for the kind of work I do, so it doesn't bother me much. I run a WinXP VM to play some old games, and a Bunsenlabs installation for Linux things like Hangouts/Netflix. I don't have a video calling system setup in TrueOS because I use my phone for both voice and video calls exclusively. Why am I staying on TrueOS ? Great community - whether on Discourse or on the telegram channel, the people make you feel welcome. If things go unanswered, someone will promise to work on it/file a bug/suggest work-arounds. Switching to TrueOS was philosophical as well - I thought a lot more about licenses, and I have arrived at the conclusion that I like BSD more than GPL. I believe it is a more practical license. I believe TrueOS is improving continuously, and is a great desktop UNIX if you put some time into it. AsiaBSDCon 2016 videos now available (https://www.youtube.com/playlist?list=PLnTFqpZk5ebD-FfVScL-x6ZnZSecMA1jI) The videos from AsiaBSDCon 2016 have been posted to youtube, 30 videos in all We'll cover the videos from 2017 next week The videos from 2018 should be posted in 4-6 weeks I are working on a new version of https://papers.freebsd.org/ that will make it easier to find the papers, slides, and videos of all talks related to FreeBSD *** syspatches will be provided for both supported releases (https://undeadly.org/cgi?action=article;sid=20180307234243) Good news for people doing upgrades only once per year: syspatches will be provided for both supported releases. The commit from T.J. Townsend (tj@) speaks for itself: ``` Subject: CVS: cvs.openbsd.org: www From: T.J. Townsend Date: 2018-03-06 22:09:12 CVSROOT: /cvs Module name: www Changes by: tj@cvs.openbsd.org 2018/03/06 15:09:12 Modified files: . : errata61.html stable.html faq : faq10.html Log message: syspatches will now be provided for both supported releases. ``` Thanks to all the developers involved in providing these! Update: An official announcement has been released: ``` I'm happy to announce that we are now able to provide two releases worth of syspatches on the amd64 and i386 platforms. The binary patches for 6.1 will hit the mirrors shortly, so you will be able to catch up with the errata on https://www.openbsd.org/errata61.html using the syspatch utility. People running amd64 will thus get the meltdown workaround. This means in particular that 6.2 will remain supported by syspatch when 6.3 comes out. Thanks to robert and ajacoutot for their amazing work on syspatch and for all their help. Thanks also to tj and the volunteers from #openbsd for their timely tests and of course to Theo for overseeing it all. ``` Exploring permutations and a mystery with BSD and GNU split filenames (https://www.lorainekv.com/permutations_split_and_gsplit/) Recently, I was playing around with the split command-line tool on Mac OS X, and I decided to chop a 4000-line file into 4000 separate single-line files. However, when I attempted to run split -l1, I ran into a funny error: split: too many files Curious to see if any splitting had occurred, I ran ls and sure enough, a huge list of filenames appeared, such as: xaa xab ... xzy xzz Now I could see why you'd run out of unique filenames - there are only 26 letters in the alphabet and these filenames were only three letters long. Also, they all seemed to begin with the letter "x". BSD split's filename defaults I checked the manual for split's defaults and confirmed what I was seeing: each file into which the file is split is named by the prefix followed by a lexically ordered suffix using suffix_length characters in the range 'a-z'. If -a is not specified, two letters are used as the suffix....with the prefix 'x' and with suffixes as above. Got it, so running split with the defaults for prefix name and suffix length will give me filenames that always start with the letter "x" followed by two-letter alphabetical permutations composed of a-z letters, with repeats allowed. I say "repeats allowed" because I noticed filenames such as xaa and xbb in the output. Side node: The reason why I say "permutations" rather than "combinations" is because letter order matters. For example, xab and xba are two distinct and legitimate filenames. Here's a nice explanation about the difference between permutations and combinations. Some permutation math So how many filenames can you get from the BSD split tool using the defaults? There are permutation formulas out there for repeating values and non-repeating values. Based on split's behavior, I wanted to use the repeating values formula: n^r where n equals the number of possible values (26 for a-z) and r equals the number of values (2, since there are only 2 letters after "x" in the filename). 26^2 = 676 So the total number of filename permutations allowed with BSD split's defaults should be 676. To double check, I ran ls | wc -l to get the total number of files in my split_test directory. The output was 677. If you subtract my original input file, input.txt, then you have 676, or the number of permutations split would allow before running out of filenames! Neat. But I still wanted my 4000 files. Moar permutations pls While 26^2 permutations doesn't support 4000 different filenames, I wondered if I could increase r to 3. Then, I'd have 17,576 different filename permutations to play with - more than enough. Earlier, I remembered the manual mentioning suffix length: -a suffixlength Use suffixlength letters to form the suffix of the file name. So I passed 3 in with the -a flag and guess what? I got my 4000 files! split -l1 -a3 input.txt ls | wc -l 4001 But that was a lot of work. It would be great if split would just handle these permutations and suffix lengths by default! In fact, I vaguely remember splitting large files into smaller ones with numerical filenames, which I prefer. I also remember not having to worry about suffixes in the past. But numerical filenames didn't seem to be an option with split installed on Mac OS X - there was no mention of it in the manual. Turns out that I was remembering GNU split from using the Debian OS two years ago, a different flavor of the split tool with different defaults and behaviors. Beastie Bits Michael Lucas is speaking at mug.org 10 April 2018 (https://blather.michaelwlucas.com/archives/3121) PkgsrcCon 2018 July 7+8 Berlin (http://pkgsrc.org/pkgsrcCon/2018/) Tint2 rocks (http://www.vincentdelft.be/post/post_20180310) Open Source Summit Europe 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/open-source-summit-europe-2018-call-for-proposals/) Travel Grants for BSDCan 2018 (https://www.freebsdfoundation.org/blog/bsdcan-2018-travel-grant-application-now-open/) BSDCan 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/bsdcan-2018-freebsd-developers-summit-call-for-proposals/) OpenBSD vmm(4) update, by Mike Larkin (https://www.openbsd.org/papers/asiabsdcon2018-vmm-slides.pdf) Feedback/Questions Morgan ZFS Install Question (http://dpaste.com/3NZN49P#wrap) Andre - Splitting ZFS Array, or not (http://dpaste.com/3V09BZ5#wrap) Jake - Python Projects (http://dpaste.com/2CY5MRE#wrap) Dave - Screen Sharing & Video Conference (http://dpaste.com/257WGCB#wrap) James - ZFS disk id switching (http://dpaste.com/3HAPZ90#wrap)
We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)
We review Meltdown and Spectre responses from various BSD projects, show you how to run CentOS with bhyve, GhostBSD 11.1 is out, and we look at the case against the fork syscall. This episode was brought to you by Headlines More Meltdown Much has been happened this week, but before we get into a status update of the various mitigations on the other BSDs, some important updates: Intel has recalled the microcode update they issued on January 8th. It turns out this update can cause Haswell and Broadwell based systems to randomly reboot, with some frequency. (https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/) AMD has confirmed that its processors are vulnerable to both variants of Spectre, and the the fix for variant #2 will require a forthcoming microcode update, in addition to OS level mitigations (https://www.amd.com/en/corporate/speculative-execution) Fujitsu has provided a status report for most of its products, including SPARC hardware (https://sp.ts.fujitsu.com/dmsp/Publications/public/Intel-Side-Channel-Analysis-Method-Security-Review-CVE2017-5715-vulnerability-Fujitsu-products.pdf) The Register of course has some commentary (https://www.theregister.co.uk/2018/01/12/intel_warns_meltdown_spectre_fixes_make_broadwells_haswells_unstable/) If new code is needed, Intel will need to get it right: the company already faces numerous class action lawsuits. Data centre operators already scrambling to conduct unplanned maintenance will not be happy about the fix reducing stability. AMD has said that operating system patches alone will address the Spectre bounds check bypass bug. Fixing Spectre's branch target injection flaw will require firmware fixes that AMD has said will start to arrive for Ryzen and EPYC CPUs this week. The Register has also asked other server vendors how they're addressing the bugs. Oracle has patched its Linux, but has told us it has “No comment/statement on this as of now” in response to our query about its x86 systems, x86 cloud, Linux and Solaris on x86. The no comment regarding Linux is odd as fixes for Oracle Linux landed here (https://linux.oracle.com/errata/ELSA-2018-4006.html) on January 9th. SPARC-using Fujitsu, meanwhile, has published advice (PDF) revealing how it will address the twin bugs in its servers and PCs, and also saying its SPARC systems are “under investigation”. Response from OpenBSD: (https://undeadly.org/cgi?action=article;sid=20180106082238) 'Meltdown, aka "Dear Intel, you suck"' (https://marc.info/?t=151521438600001&r=1&w=2) Theo de Raadt's response to Meltdown (https://www.itwire.com/security/81338-handling-of-cpu-bug-disclosure-incredibly-bad-openbsd-s-de-raadt.html) That time in 2007 when Theo talked about how Intel x86 had major design problems in their chips (https://marc.info/?l=openbsd-misc&m=118296441702631&w=2) OpenBSD gets a Microcode updater (https://marc.info/?l=openbsd-cvs&m=151570987406841&w=2) Response from Dragonfly BSD: (http://lists.dragonflybsd.org/pipermail/users/2018-January/313758.html) The longer response in four commits One (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627151.html) Two (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627152.html) Three (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627153.html) Four (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627154.html) Even more Meltdown (https://www.dragonflydigest.com/2018/01/10/20718.html) DragonflyBSD master now has full IBRS and IBPB support (http://lists.dragonflybsd.org/pipermail/users/2018-January/335643.html) IBRS (Indirect Branch Restricted Speculation): The x86 IBRS feature requires corresponding microcode support. It mitigates the variant 2 vulnerability. If IBRS is set, near returns and near indirect jumps/calls will not allow their predicted target address to be controlled by code that executed in a less privileged prediction mode before the IBRS mode was last written with a value of 1 or on another logical processor so long as all RSB entries from the previous less privileged prediction mode are overwritten. Speculation on Skylake and later requires these patches ("dynamic IBRS") be used instead of retpoline. If you are very paranoid or you run on a CPU where IBRS=1 is cheaper, you may also want to run in "IBRS always" mode. IBPB (Indirect Branch Prediction Barrier): Setting of IBPB ensures that earlier code's behavior does not control later indirect branch predictions. It is used when context switching to new untrusted address space. Unlike IBRS, IBPB is a command MSR and does not retain its state. DragonFlyBSD's Meltdown Fix Causing More Slowdowns Than Linux (https://www.phoronix.com/scan.php?page=article&item=dragonfly-bsd-meltdown&num=1) NetBSD HOTPATCH() (http://mail-index.netbsd.org/source-changes/2018/01/07/msg090945.html) NetBSD SVS (Separate Virtual Space) (http://mail-index.netbsd.org/source-changes/2018/01/07/msg090952.html) Running CentOS with Bhyve (https://www.daemon-security.com/2018/01/bhyve-centos-0110.html) With the addition of UEFI in FreeBSD (since version 11), users of bhyve can use the UEFI boot loader instead of the grub2-bhyve port for booting operating systems such as Microsoft Windows, Linux and OpenBSD. The following page provides information necessary for setting up bhyve with UEFI boot loader support: https://wiki.freebsd.org/bhyve/UEFI Features have been added to vmrun.sh to make it easier to setup the UEFI boot loader, but the following is required to install the UEFI firmware pkg: # pkg install -y uefi-edk2-bhyve With graphical support, you can use a vnc client like tigervnc, which can be installed with the following command: # pkg install -y tigervnc In the case of most corporate or government environments, the Linux of choice is RHEL, or CentOS. Utilizing bhyve, you can test and install CentOS in a bhyve VM the same way you would deploy a Linux VM in production. The first step is to download the CentOS iso (for this tutorial I used the CentOS minimal ISO): http://isoredirect.centos.org/centos/7/isos/x8664/CentOS-7-x8664-Minimal-1708.iso I normally use a ZFS Volume (zvol) when running bhyve VMs. Run the following commands to create a zvol (ensure you have enough disk space to perform these operations): # zfs create -V20G -o volmode=dev zroot/centos0 (zroot in this case is the zpool I am using) Similar to my previous post about vmrun.sh, you need certain items to be configured on FreeBSD in order to use bhyve. The following commands are necessary to get things running: ``` echo "vfs.zfs.vol.mode=2" >> /boot/loader.conf kldload vmm ifconfig tap0 create sysctl net.link.tap.uponopen=1 net.link.tap.uponopen: 0 -> 1 ifconfig bridge0 create ifconfig bridge0 addm em0 addm tap0 ifconfig bridge0 up ``` (replace em0 with whatever your physical interface is). There are a number of utilities that can be used to manage bhyve VMs, and I am sure there is a way to use vmrun.sh to run Linux VMs, but since all of the HowTos for running Linux use the bhyve command line, the following script is what I use for running CentOS with bhyve. ``` !/bin/sh General bhyve install/run script for CentOS Based on scripts from pr1ntf and lattera HOST="127.0.0.1" PORT="5901" ISO="/tmp/centos.iso" VMNAME="centos" ZVOL="centos0" SERIAL="nmda0A" TAP="tap1" CPU="1" RAM="1024M" HEIGHT="800" WIDTH="600" if [ "$1" == "install" ]; then Kill it before starting it bhyvectl --destroy --vm=$VMNAME bhyve -c $CPU -m $RAM -H -P -A -s 0,hostbridge -s 2,virtio-net,$TAP -s 3,ahci-cd,$ISO -s 4,virtio-blk,/dev/zvol/zroot/$ZVOL -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,/dev/$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd $VMNAME kill it after bhyvectl --destroy --vm=$VMNAME elif [ "$1" == "run" ]; then Kill it before starting it bhyvectl --destroy --vm=centos bhyve -c $CPU -m $RAM -w -H -s 0,hostbridge -s 2,virtio-net,$TAP -s 4,virtio-blk,/dev/zvol/zroot/$ZVOL -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,/dev/$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd $VMNAME & else echo "Please type install or run"; fi ``` The variables at the top of the script can be adjusted to fit your own needs. With the addition of the graphics output protocol in UEFI (or UEFI-GOP), a VNC console is launched and hosted with the HOST and PORT setting. There is a password option available for the VNC service, but the connection should be treated as insecure. It is advised to only listen on localhost with the VNC console and tunnel into the host of the bhyve VM. Now with the ISO copied to /tmp/centos.iso, and the script saved as centos.sh you can run the following command to start the install: # ./centos.sh install At this point, using vncviewer (on the local machine, or over an SSH tunnel), you should be able to bring up the console and run the CentOS installer as normal. The absolutely most critical item is to resolve an issue with the booting of UEFI after the installation has completed. Because of the path used in bhyve, you need to run the following to be able to boot CentOS after the installation: # cp -f /mnt/sysimage/boot/efi/EFI/centos/grubx64.efi /mnt/sysimage/boot/efi/EFI/BOOT With this setting changed, the same script can be used to launch your CentOS VM as needed: # ./centos.sh run If you are interested in a better solution for managing your Linux VM, take a look at the various bhyve management ports in the FreeBSD ports tree. Interview - newnix architect - @newnix (https://bsd.network/@newnix) News Roundup GhostBSD 11.1 - FreeBSD for the desktop (https://distrowatch.com/weekly.php?issue=20180108#ghostbsd) GhostBSD is a desktop oriented operating system which is based on FreeBSD. The project takes the FreeBSD operating system and adds a desktop environment, some popular applications, a graphical package manager and Linux binary compatibility. GhostBSD is available in two flavours, MATE and Xfce, and is currently available for 64-bit x86 computers exclusively. I downloaded the MATE edition which is available as a 2.3GB ISO file. Installing GhostBSD's system installer is a graphical application which begins by asking us for our preferred language, which we can select from a list. We can then select our keyboard's layout and our time zone. When it comes to partitioning we have three main options: let GhostBSD take over the entire disk using UFS as the file system, create a custom UFS layout or take over the entire disk using ZFS as the file system. UFS is a classic file system and quite popular, it is more or less FreeBSD's equivalent to Linux's ext4. ZFS is a more advanced file system with snapshots, multi-disk volumes and optional deduplication of data. I decided to try the ZFS option. Once I selected ZFS I didn't have many more options to go through. I was given the chance to set the size of my swap space and choose whether to set up ZFS as a plain volume, with a mirrored disk for backup or in a RAID arrangement with multiple disks. I stayed with the plain, single disk arrangement. We are then asked to create a password for the root account and create a username and password for a regular user account. The installer lets us pick our account's shell with the default being fish, which seemed unusual. Other shells, including bash, csh, tcsh, ksh and zsh are available. The installer goes to work copying files and offers to reboot our computer when it is done. Early impressions The newly installed copy of GhostBSD boots to a graphical login screen where we can sign into the account we created during the install process. Signing into our account loads the MATE 1.18 desktop environment. I found MATE to be responsive and applications were quick to open. Early on I noticed odd window behaviour where windows would continue to slide around after I moved them with the mouse, as if the windows were skidding on ice. Turning off compositing in the MATE settings panel corrected this behaviour. I also found the desktop's default font (Montserrat Alternates) to be hard on my eyes as the font is thin and, for lack of a better term, bubbly. Fonts can be easily adjusted in the settings panel. A few minutes after I signed into my account, a notification appeared in the system tray letting me know software updates were available. Clicking the update icon brings up a small window showing us a list of package updates and, if any are available, updates to the base operating system. FreeBSD, and therefore GhostBSD, both separate the core operating system from the applications (packages) which run on the operating system. This means we can update the core of the system separately from the applications. GhostBSD's core remains relatively static and minimal while applications are updated using a semi-rolling schedule. When we are updating the core operating system, the update manager will give us the option of rebooting the system to finish the process. We can dismiss this prompt to continue working, but the wording of the prompt may be confusing. When asked if we want to reboot to continue the update process, the options presented to us are "Continue" or "Restart". The Continue option closes the update manager and returns us to the MATE desktop. The update manager worked well for me and the only issue I ran into was when I dismissed the update manager and then wanted to install updates later. There are two launchers for the update manager, one in MATE's System menu and one in the settings panel. Clicking either of these launchers didn't accomplish anything. Running the update manager from the command line simply caused the process to lock up until killed. I found if I had dismissed the update manager once, I'd have to wait until I logged in again to use it. Alternatively, I could use a command line tool or use the OctoPkg package manager to install package updates. Conclusions Most of my time with GhostBSD, I was impressed and happy with the operating system. GhostBSD builds on a solid, stable FreeBSD core. We benefit from FreeBSD's performance and its large collection of open source software packages. The MATE desktop was very responsive in my trial and the system is relatively light on memory, even when run on ZFS which has a reputation for taking up more memory than other file systems. FreeBSD Looks At Making Wayland Support Available By Default (https://www.phoronix.com/scan.php?page=news_item&px=FreeBSD-Wayland-Availability) There's an active discussion this week about making Wayland support available by default on FreeBSD. FreeBSD has working Wayland support -- well, assuming you have working Intel / Radeon graphics -- and do have Weston and some other Wayland components available via FreeBSD Ports. FreeBSD has offered working Wayland support that is "quite usable" for more than one year. But, it's not too easy to get going with Wayland on FreeBSD. Right now those FreeBSD desktop users wanting to use/develop with Wayland currently need to rebuild the GTK3 tool-kit, Mesa, and other packages with Wayland support enabled. This call for action now is about allowing the wayland=on to be made the default. This move would then allow these dependencies to be built with Wayland support by default, but for the foreseeable future FreeBSD will continue defaulting to X.Org-based sessions. The FreeBSD developers mostly acknowledge that Wayland is the future and the cost of enabling Wayland support by default is just slightly larger packages, but that weight is still leaner than the size of the X.Org code-base and its dependencies. FreeBSD vote thread (https://lists.freebsd.org/pipermail/freebsd-ports/2017-December/111906.html) TrueOS Fliped the switch already (https://github.com/trueos/trueos-core/commit/f48dba9d4e8cefc45d6f72336e7a0b5f42a2f6f1) fork is not my favorite syscall (https://sircmpwn.github.io/2018/01/02/The-case-against-fork.html) This article has been on my to-write list for a while now. In my opinion, fork is one of the most questionable design choices of Unix. I don't understand the circumstances that led to its creation, and I grieve over the legacy rationale that keeps it alive to this day. Let's set the scene. It's 1971 and you're a fly on the wall in Bell Labs, watching the first edition of Unix being designed for the PDP-11/20. This machine has a 16-bit address space with no more than 248 kilobytes of memory. They're discussing how they're going to support programs that spawn new programs, and someone has a brilliant idea. “What if we copied the entire address space of the program into a new process running from the same spot, then let them overwrite themselves with the new program?” This got a rousing laugh out of everyone present, then they moved on to a better design which would become immortalized in the most popular and influential operating system of all time. At least, that's the story I'd like to have been told. In actual fact, the laughter becomes consensus. There's an obvious problem with this approach: every time you want to execute a new program, the entire process space is copied and promptly discarded when the new program begins. Usually when I complain about fork, this the point when its supporters play the virtual memory card, pointing out that modern operating systems don't actually have to copy the whole address space. We'll get to that, but first — First Edition Unix does copy the whole process space, so this excuse wouldn't have held up at the time. By Fourth Edition Unix (the next one for which kernel sources survived), they had wisened up a bit, and started only copying segments when they faulted. This model leads to a number of problems. One is that the new process inherits all of the parent's process descriptors, so you have to close them all before you exec another process. However, unless you're manually keeping tabs on your open file descriptors, there is no way to know what file handles you must close! The hack that solves this is CLOEXEC, the first of many hacks that deal with fork's poor design choices. This file descriptors problem balloons a bit - consider for example if you want to set up a pipe. You have to establish a piped pair of file descriptors in the parent, then close every fd but the pipe in the child, then dup2 the pipe file descriptor over the (now recently closed) file descriptor 1. By this point you've probably had to do several non-trivial operations and utilize a handful of variables from the parent process space, which hopefully were on the stack so that we don't end up copying segments into the new process space anyway. These problems, however, pale in comparison to my number one complaint with the fork model. Fork is the direct cause of the stupidest component I've ever heard of in an operating system: the out-of-memory (aka OOM) killer. Say you have a process which is using half of the physical memory on your system, and wants to spawn a tiny program. Since fork “copies” the entire process, you might be inclined to think that this would make fork fail. But, on Linux and many other operating systems since, it does not fail! They agree that it's stupid to copy the entire process just to exec something else, but because fork is Important for Backwards Compatibility, they just fake it and reuse the same memory map (except read-only), then trap the faults and actually copy later. The hope is that the child will get on with it and exec before this happens. However, nothing prevents the child from doing something other than exec - it's free to use the memory space however it desires! This approach now leads to memory overcommittment - Linux has promised memory it does not have. As a result, when it really does run out of physical memory, Linux will just kill off processes until it has some memory back. Linux makes an awfully big fuss about “never breaking userspace” for a kernel that will lie about memory it doesn't have, then kill programs that try to use the back-alley memory they were given. That this nearly 50 year old crappy design choice has come to this astonishes me. Alas, I cannot rant forever without discussing the alternatives. There are better process models that have been developed since Unix! The first attempt I know of is BSD's vfork syscall, which is, in a nutshell, the same as fork but with severe limitations on what you do in the child process (i.e. nothing other than calling exec straight away). There are loads of problems with vfork. It only handles the most basic of use cases: you cannot set up a pipe, cannot set up a pty, and can't even close open file descriptors you inherited from the parent. Also, you couldn't really be sure of what variables you were and weren't editing or allowed to edit, considering the limitations of the C specification. Overall this syscall ended up being pretty useless. Another model is posixspawn, which is a hell of an interface. It's far too complicated for me to detail here, and in my opinion far too complicated to ever consider using in practice. Even if it could be understood by mortals, it's a really bad implementation of the spawn paradigm — it basically operates like fork backwards, and inherits many of the same flaws. You still have to deal with children inheriting your file descriptors, for example, only now you do it in the parent process. It's also straight-up impossible to make a genuine pipe with posixspawn. (Note: a reader corrected me - this is indeed possible via posixspawnfileactionsadddup2.) Let's talk about the good models - rfork and spawn (at least, if spawn is done right). rfork originated from plan9 and is a beautiful little coconut of a syscall, much like the rest of plan9. They also implement fork, but it's a special case of rfork. plan9 does not distinguish between processes and threads - all threads are processes and vice versa. However, new processes in plan9 are not the everything-must-go fuckfest of your typical fork call. Instead, you specify exactly what the child should get from you. You can choose to include (or not include) your memory space, file descriptors, environment, or a number of other things specific to plan9. There's a cool flag that makes it so you don't have to reap the process, too, which is nice because reaping children is another really stupid idea. It still has some problems, mainly around creating pipes without tremendous file descriptor fuckery, but it's basically as good as the fork model gets. Note: Linux offers this via the clone syscall now, but everyone just fork+execs anyway. The other model is the spawn model, which I prefer. This is the approach I took in my own kernel for KnightOS, and I think it's also used in NT (Microsoft's kernel). I don't really know much about NT, but I can tell you how it works in KnightOS. Basically, when you create a new process, it is kept in limbo until the parent consents to begin. You are given a handle with which you can configure the process - you can change its environment, load it up with file descriptors to your liking, and so on. When you're ready for it to begin, you give the go-ahead and it's off to the races. The spawn model has none of the flaws of fork. Both fork and exec can be useful at times, but spawning is much better for 90% of their use-cases. If I were to write a new kernel today, I'd probably take a leaf from plan9's book and find a happy medium between rfork and spawn, so you could use spawn to start new threads in your process space as well. To the brave OS designers of the future, ready to shrug off the weight of legacy: please reconsider fork. Enable ld.lld as bootstrap linker by default on amd64 (https://svnweb.freebsd.org/changeset/base/327783) Enable ld.lld as bootstrap linker by default on amd64 For some time we have been planning to migrate to LLVM's lld linker. Having a man page was the last blocking issue for using ld.lld to link the base system kernel + userland, now addressed by r327770. Link the kernel and userland libraries and binaries with ld.lld by default, for additional test coverage. This has been a long time in the making. On 2013-04-13 I submitted an upstream tracking issue in LLVM PR 23214: [META] Using LLD as FreeBSD's system linker. Since then 85 individual issues were identified, and submitted as dependencies. These have been addressed along with two and a half years of other lld development and improvement. I'd like to express deep gratitude to upstream lld developers Rui Ueyama, Rafael Espindola, George Rimar and Davide Italiano. They put in substantial effort in addressing the issues we found affecting FreeBSD/amd64. To revert to using ld.bfd as the bootstrap linker, in /etc/src.conf set WITHOUTLLDBOOTSTRAP=yes If you need to set this, please follow up with a PR or post to the freebsd-toolchain mailing list explaining how default WITHLLDBOOTSTRAP failed for your use case. Note that GNU ld.bfd is still installed as /usr/bin/ld, and will still be used for linking ports. ld.lld can be installed as /usr/bin/ld by setting in /etc/src.conf WITH_LLD_IS_LLD=yes A followup commit will set WITHLLDIS_LD by default, possibly after Clang/LLVM/lld 6.0 is merged to FreeBSD. Release notes: Yes Sponsored by: The FreeBSD Foundation Followup: https://www.mail-archive.com/svn-src-all@freebsd.org/msg155493.html *** Beastie Bits BSDCAN2017 Interview with Peter Hessler, Reyk Floeter, and Henning Brauer (https://undeadly.org/cgi?action=article;sid=20171229080944) video (https://www.youtube.com/watch?v=e-Xim3_rJns) DSBMD (https://freeshell.de/~mk/projects/dsbmd.html) ccc34 talk - May contain DTraces of FreeBSD (https://media.ccc.de/v/34c3-9196-may_contain_dtraces_of_freebsd) [scripts to run an OpenBSD mirror, rsync and verify])(https://github.com/bluhm/mirror-openbsd) Old School PC Fonts (https://int10h.org/oldschool-pc-fonts/readme/) Feedback/Questions David - Approach and Tools for Snapshots and Remote Replication (http://dpaste.com/33HKKEM#wrap) Brian - Help getting my FreeBSD systems talking across the city (http://dpaste.com/3QWFEYR#wrap) Malcolm - First BSD Meetup in Stockholm happened and it was great (http://dpaste.com/1Z9Y8H1) Brad - Update on TrueOS system (http://dpaste.com/3EC9RGG#wrap) ***
We read the FreeBSD Q3 status report, explore good and bad syscalls, list GOG Games for OpenBSD, and show you what devmatch can do. This episode was brought to you by Headlines FreeBSD Q3 Status Report 2017 (https://lists.freebsd.org/pipermail/freebsd-announce/2017-December/001818.html) FreeBSD Team Reports FreeBSD Release Engineering Team Ports Collection The FreeBSD Core Team The FreeBSD Foundation Projects FreeBSD CI Kernel Intel 10G iflib Driver Update Intel iWARP Support pNFS Server Plan B Architectures AMD Zen (family 17h) support Userland Programs Updates to GDB Ports FreeBSDDesktop OpenJFX 8 Puppet Documentation Absolute FreeBSD, 3rd Edition Manual Pages Third-Party Projects The nosh Project ####FreeBSD Foundation Q4 Update (https://www.freebsdfoundation.org/wp-content/uploads/2017/12/FreeBSD-Foundation-Q4-Update.pdf) *** ###11 syscalls that rock the world (https://www.cloudatomiclab.com/prosyscall/) 0. read > You cannot go wrong with a read. You can barely EFAULT it! On Linux amd64 it is syscall zero. If all its arguments are zero it returns zero. Cool! 1. pipe > The society for the preservation of historic calling conventions is very fond of pipe, as in many operating systems and architectures it preserves the fun feature of returning both of the file descriptors as return values. At least Linux MIPS does, and NetBSD does even on x86 and amd64. Multiple return values are making a comeback in languages like Lua and Go, but C has always had a bit of a funny thing about them, but they have long been supported in many calling conventions, so let us use them in syscalls! Well, one syscall. 2. kqueue > When the world went all C10K on our ass, and scaleable polling was a thing, Linux went epoll, the BSDs went kqueue and Solaris went /dev/poll. The nicest interface was kqueue, while epoll is some mix of edge and level triggered semantics and design errors so bugs are still being found. 3. unshare > Sounds like a selfish syscall, but this generous syscall call is the basis of Linux namespaces, allowing a process to isolate its resources. Containers are built from unshares. 4. setns > If you liked unshare, its younger but cooler friend takes file descriptors for namespaces. Pass it down a unix socket to another process, or stash it for later, and do that namespace switching. All the best system calls take file descriptors. 5. execveat > Despite its somewhat confusing name (FreeBSD has the saner fexecve, but other BSDs do not have support last time I checked), this syscall finally lets you execute a program just given a file descriptor for the file. I say finally, as Linux only implemented this in 3.19, which means it is hard to rely on it (yeah, stop using those stupid old kernels folks). Before that Glibc had a terrible userspace implementation that is basically useless. Perfect for creating sandboxes, as you can sandbox a program into a filesystem with nothing at all in, or with a totally controlled tree, by opening the file to execute before chroot or changing the namespace. 6. pdfork > Too cool for Linux, you have to head out to FreeBSD for this one. Like fork, but you get a file descriptor for the process not a pid. Then you can throw it in the kqueue or send it to another process. Once you have tried process descriptors you will never go back. 7. signalfd > You might detect a theme here, but if you have ever written traditional 1980s style signal handlers you know how much they suck. How about turning your signals into messages that you can read on, you guessed it, file descriptors. Like, usable. 8. wstat > This one is from Plan 9. It does the opposite of stat and writes the same structure. Simples. Avoids having chmod, chown, rename, utime and so on, by the simple expedient of making the syscall symmetric. Why not? 9. clonefile > The only cool syscall on OSX, and only supported on the new APFS filesystem. Copies whole files or directories on a single syscall using copy on write for all the data. Look on my works, copyfilerange and despair. 10. pledge > The little sandbox that worked. OpenBSD only here, they managed to make a simple sandbox that was practical for real programs, like the base OpenBSD system. Capsicum form FreeBSD (and promised for Linux for years but no sign) is a lovely design, and gave us pdfork, but its still kind of difficult and intrusive to implement. Linux has, well, seccomp, LSMs, and still nothing that usable for the average program. ###Eleven syscalls that suck (https://www.cloudatomiclab.com/antisyscall/) 0. ioctl > It can‘t decide if it‘s arguments are integers, strings, or some struct that is lost in the midst of time. Make up your mind! Plan 9 was invented to get rid of this. 1. fcntl > Just like ioctl but for some different miscellaneous operations, because one miscelleny is not enough. 2. tuxcall > Linux put a web server in the kernel! To win a benchmark contest with Microsoft! It had it‘s own syscall! My enum tux_reactions are YUK! Don‘t worry though, it was a distro patch (thanks Red Hat!) and never made it upstream, so only the man page and reserved number survive to taunt you and remind you that the path of the righteous is beset by premature optmization! 3. iosetup > The Linux asynchronous IO syscalls are almost entirely useless! Almost nothing works! You have to use ODIRECT for a start. And then they still barely work! They have one use, benchmarking SSDs, to show what speed you could get if only there was a usable API. Want async IO in kernel? Use Windows! 4. stat, and its friends and relatives > Yes this one is useful, but can you find the data structure it uses? We have oldstat, oldfstat, ustat, oldlstat, statfs, fstatfs, stat, lstat, fstat, stat64, lstat64, fstat64, statfs64, fstatfs64, fstatat64 for stating files and links and filesystems in Linux. A new bunch will be along soon for Y2038. Simplify your life, use a BSD, where they cleaned up the mess as they did the cooking! Linux on 32 bit platforms is just sucky in comparison, and will get worse. And don't even look at MIPS, where the padding is wrong. 5. Linux on MIPS > Not a syscall, a whole implemntation of the Linux ABI. Unlike the lovely clean BSDs, Linux is different on each architecture, system calls randomly take arguments in different orders, and constants have different values, and there are special syscalls. But MIPS takes the biscuit, the whole packet of biscuits. It was made to be binary compatible with old SGI machines that don't even exist, and has more syscall ABIs than I have had hot dinners. Clean it up! Make a new sane MIPS ABI and deprecate the old ones, nothing like adding another variant. So annoying I think I threw out all my MIPS machines, each different. 6. inotify, fanotify and friends > Linux has no fewer than three file system change notification protocols. The first, dnotify hopped on ioctl‘s sidekick fcntl, while the two later ones, inotify and fanotify added a bunch more syscalls. You can use any of them, and they still will not provide the notification API you want for most applications. Most people use the second one, inotify and curse it. Did you know kqueue can do this on the BSDs? 7. personality > Oozing in personality, but we just don't get along. Basically obsolete, as the kernel can decide what kind of system emulation to do from binaries directly, it stays around with some use cases in persuading ./configure it is running on a 32 bit system. But it can turn off ASLR, and let the CVEs right into your system. We need less persoanlity! 8. gettimeofday > Still has an obsolete timezone value from an old times when people thought timezones should go all the way to the kernel. Now we know that your computer should not know. Set its clock to UTC. Do the timezones in the UI based on where the user is, not the computer. You should use clock_gettime now. Don't even talk to me about locales. This syscall is fast though, don't use it for benchmarking, its in the VDSO. 9. splice and tee > These, back in 2005 were a quite nice idea, although Linux said then “it is incomplete, the interfaces are ugly, and it will oops the system if anything goes wrong”. It won't oops your system now, but usage has not taken off. The nice idea from Linus was that a pipe is just a ring buffer in the kernel, that can have a more general API and use cases for performant code, but a decade on it hasn't really worked out. It was also supposed to be a more general sendfile, which in many ways was the successor of that Tux web server, but I think sendfile is still more widely used. 10. userfaultfd > Yes, I like file descriptors. Yes CRIU is kind of cool. But userspace handling page faults? Is nothing sacred? I get that you can do this badly with a SIGSEGV handler, but talk about lipstick on a pig. *** ###OpenBSD 6.0 on an iMac G3 from 1999 (http://www.increasinglyadequate.com/macppc.html) > A while ago I spent $50 for an iMac G3 (aka the iMac,1). This iconic model restored Apple's fortunes in the late '90s. Since the iMac G3 can still boot Mac OSes 8 and 9, I mostly use the machine to indulge a nostalgia for childhood schooldays spent poking at the operating system and playing Escape Velocity. But before I got around to that, I decided to try out the software that the previous owner had left on the machine. The antiquated OSX 10.2 install and 12 year old versions of Safari and Internet Explorer were too slow and old to use for anything. Updating to newer software was almost impossible; a later OSX is required to run the little PowerPC-compatible software still languishing in forgotten corners of the Internet. This got me thinking: could this machine be used, really used, nowadays? Lacking a newer OSX disc, I decided to try the most recent OpenBSD release. (And, since then, to re-try with each new OpenBSD release.) Below are the results of this experiment (plus a working xorg.conf file) and a few background notes. Background > This iMac is a Revision D iMac G3 in grape. It's part of the iMac,1 family of computers. This family includes all tray-loading iMac G3s. (Later iMac G3s had a slot-loading CD drive and different components.) Save for a slightly faster processor, a dedicated graphics card, and cosmetic tweaks to the case, my iMac is identical to the prior year's line-launching Bondi Blue iMac. My machine has had its memory upgraded from 32 MB to 320 MB. Thank Goodness. > The Revision D iMac G3 shipped with Mac OS 8.5. It can run up to Mac OS 9.2.2 or OSX 10.3.9. Other operating systems that tout support for the iMac,1 include NetBSD, OpenBSD, and a shrinking number of Linux distributions. > OpenBSD is simple (by design) and well-maintained. In contrast, NetBSD seems rather more complex and featureful, and I have heard grumbling that despite its reputation for portability, NetBSD really only works well on amd64. I'd test that assertion if OpenBSD's macppc installation instructions didn't seem much simpler than NetBSD's. Linux is even more complicated, although most distros are put together in a way that you can mostly ignore that complexity (until you can't). In the end I went with OpenBSD because I am familiar with it and because I like it. Installing OpenBSD on the iMac,1 > Installing OpenBSD on this iMac was simple. It's the same procedure as installing OpenBSD on an amd64 rig. You put in the installation disc; you tell the machine to boot from it; and then you answer a few prompts, most of which simply ask you to press enter. In this case, OpenBSD recognizes all machine's hardware just fine, including sound and networking, though I had a little trouble with video. > The OpenBSD documentation says video should just work and that an xorg.conf file isn't necessary. As such, it no longer ships with an xorg.conf file. Though that's never posed a problem on my other OpenBSD machines, it does here. Video doesn't work out of the box on my iMac,1. startx just blanks the screen. Fortunately, because the BSDs use a centralized development model where each operating system is stored in one repository, OpenBSD's website provides a web interface to the source code going back to the early days. I was able to find the last version of the sample xorg.conf that used to ship on macppc. With a little tweaking, I transformed that file into this one (https://www.increasinglyadequate.com/files/xorg.conf), with which video works just fine. Just drop it into your iMac's /etc/X11 directory. You'll also need to remember to set the machdep.allowaperture sysctl to 2 (e.g., as root run sysctl machdep.allowaperture=2), although the installer will do that automatically if you answer yes to the question about whether you plan to run X. > All that being said, video performance is pretty poor. I am either doing something wrong, or OpenBSD doesn't have accelerated video for this iMac, or this machine is just really old! I will discuss performance below. Running OpenBSD on the iMac,1 > The machine performs okay under OpenBSD. You can expect to ably run minimalistic software under minimalistic window managers. I tried dillo, mrxvt, and cmus under cwm and fvwm. Performance here was just fine. I also tried Firefox 26, 33, and 34 under fvwm and cwm. Firefox ran, but "modern," Javascript-heavy sites were an exercise in frustration; the 2015 version of CNN.com basically froze Firefox for 30 seconds or more. A lighter browser like dillo is doable. > You'll notice that I used the past-tense to talk about Firefox. Firefox currently doesn't build on PowerPC on OpenBSD. Neither does Chromium. Neither do a fair number of applications. But whatever -- there's still a lot of lighter applications available, and it's these you'll use day-to-day on a decades-old machine. > Lightweight window managers work okay, as you'd expect. You can even run heavier desktop environments, such as xfce, though you'll give up a lot of performance. > I ran the Ubench benchmark on this iMac and two more modern machines also running OpenBSD. The benchmark seems like an old one; I don't know how (if at all) it accounts for hardware changes in the past 13 years. That is, I don't know if the difference in score accurately measures the difference in real-world performance. Here are the results anyway: Conclusion > Except for when I check to see if OpenBSD still works, I run Mac OS9 on this rig. I have faster and better machines for running OpenBSD. If I didn't -- if this rig were, improbably, all I had left, and I was waiting on the rush delivery of something modern -- then I would use OpenBSD on my iMac,1. I'd have to stick to lightweight applications, but at least they'd be up-to-date and running on a simple, stable, OS. *** ##News Roundup ###34th Chaos Communication Congress Schedule (https://events.ccc.de/congress/2017/Fahrplan/index.html) Many talks are streamed live (http://streaming.media.ccc.de/34c3), a good mixture of english and german talks May contain DTraces of FreeBSD (https://events.ccc.de/congress/2017/Fahrplan/events/9196.html) Are all BSDs created equally? (https://events.ccc.de/congress/2017/Fahrplan/events/8968.html) library operating systems (https://events.ccc.de/congress/2017/Fahrplan/events/8949.html) Hardening Open Source Development (https://events.ccc.de/congress/2017/Fahrplan/events/9249.html) *** ###OpenBSD 6.2 + CDE (https://jamesdeagle.blogspot.co.uk/2017/12/openbsd-62-cde.html) > If you've noticed a disruption in the time-space continuum recently, it is likely because I have finally been able to compile and install the Common Desktop Environment (CDE) in a current and actively-developed operating system (OpenBSD 6.2 in this case). > This comes after so many attempts (across multiple platforms) that ended up with the build process prematurely stopping itself in its own tracks for a variety of infinitesimal reasons that were beyond my comprehension as a non-programmer, or when there was success it was not without some broken parts. As for the latter, I've been able to build CDE on OpenIndiana Hipster, but with an end product where I'm unable to change the color scheme in dtstyle (because "useColorObj" is set to "False"), with a default color scheme that is low-res and unpleasant. As for changing "useColorObj" to "True", I tried every recommended trick I could find online, but nothing worked. > My recent attempts at installing CDE on OpenBSD (version 6.1) saw the process stop due to a number of errors that are pure gibberish to these naive eyes. While disappointing, it was par for the course within my miserable experience with trying to build this particular desktop environment. As I wrote in this space in November 2015, in the course of explaining part of my imperitive for installing Solaris 10: > And so I have come to think of building the recently open-sourced CDE as being akin to a coffee mug I saw many years ago. One side of the mug read "Turn the mug to see how to keep an idiot busy." On the other side, it read "Turn the mug to see how to keep an idiot busy." I'm through feeling like an idiot, which is partially why I'm on this one-week journey with Solaris 10. > While I thoroughly enjoyed running Solaris 10 on my ThinkPad T61p, and felt a devilish thrill at using it out in the open at my local MacBook- and iPhone-infested Starbucks and causing general befuddlement and consternation among the occasional prying yoga mom, I never felt like I could do much with it beyond explore the SunOS 5.10 command line and watch YouTube videos. While still supported by its current corporate owner (whose name I don't even want to type), it is no longer actively developed and is thus little more than a retro toy. I hated the idea of installing anything else over it, but productivity beckoned and it was time to tearfully and reluctantly drag myself off the dance floor. > In any case, just last week I noticed that the Sourceforge page for the OpenBSD build had some 6.2-specific notes by way of a series of four patches, and so I decided 'what the heck, let's give this puppy another whirl'. After an initial abortive attempt at a build, I surmised that I hadn't applied the four patches correctly. A day or two later, I took a deep breath and tried again, this time resolving to not proceed with the time make World build command until I could see some sign of a successful patch process. (This time around, I downloaded the patches and moved them into the directory containing the CDE makefiles, and issued each patch command as patch Once I had the thing up and running, and with a mind bursting with fruit flavor, I started messing about. The first order of business was to create a custom color scheme modelled after the default color scheme in UnixWare. (Despite any baggage that system carries from its previous ownership under SCO, I adored the aesthetics of UnixWare 7.1.4 two years ago when I installed the free one month trial version on my ThinkPad. For reasons that escape me now, I named my newly-created color scheme in honor of UnixWare 7.1.3.) > Like a proud papa, I immediately tweeted the above screenshot and risked irritating a Linux kid or two in the process, given SCO's anti-climatic anti-Linux patent trolling from way back when. (I'm not out to irritate penguinistas, I just sure like this color scheme.) Final Thoughts > It may look a little clunky at first, and may be a little bling-challenged, but the more I use CDE and adapt to it, the more it feels like an extension of my brain. Perhaps this is because it has a lot zip and behaves in a consistent and coherent manner. (I don't want to go too much further down that road here, as OSnews's Thom Holwerda already gave a good rundown about ten years ago.) > Now that I have succesfully paired my absolute favorite operating system with a desktop environment that has exerted an intense gravitational hold on me for many, many years, I don't anticipate distrohopping any time soon. And as I attain a more advanced knowledge of CDE, I'll be chronicling any new discoveries here for the sake of anyone following me from behind as I feel my way around this darkened room. *** ###devmatch(8) added to FreeBSD HEAD (https://www.mail-archive.com/svn-src-all@freebsd.org/msg154719.html) ``` Log: Match unattached devices on the system to potential kernel modules. devmatch(8) matchs up devices in the system device tree with drivers that may match them. For each unattached device in the system, it tries to find matching PNP info in the linker hints and prints modules to load to claim the devices. In --unbound mode, devmatch can look for drivers that have attached to devices in the device tree and have plug and play information, but for which no PNP info exists. This helps find drivers that haven't been converted yet that are in use on this system. In addition, the ability to dump out linker.hints is provided. Future commits will add hooks to devd.conf and rc.d to fully automate using this information. Added: head/usr.sbin/devmatch/ head/usr.sbin/devmatch/Makefile (contents, props changed) head/usr.sbin/devmatch/devmatch.8 (contents, props changed) head/usr.sbin/devmatch/devmatch.c (contents, props changed) Modified: head/usr.sbin/Makefile Modified: head/usr.sbin/Makefile ``` + Oh, you naughty committers: :-) https://www.mail-archive.com/svn-src-all@freebsd.org/msg154720.html Beastie Bits New FreeBSD Journal issue: Monitoring and Metrics (https://www.freebsdfoundation.org/journal/) OpenBSD Engine Mix available on GOG.com (https://www.gog.com/mix/openbsd_engine_available) OpenBSD Foundation reached their 2017 fundraising goal (http://www.openbsdfoundation.org/campaign2017.html) TrueOS 17.12 Review – An Easy BSD (https://www.youtube.com/watch?v=nKr1GCsV-gA) LibreSSL 2.6.4 Released (https://bsdsec.net/articles/libressl-2-6-4-released-fixed) *** ##Feedback/Questions Mike - BSD 217 & Winning over Linux Users (http://dpaste.com/3AB7J4P#wrap) JLR - Boot Environments Broken? (http://dpaste.com/2K0ZDH9#wrap) Kevr - ZFS question and suggestion (http://dpaste.com/04MXA5P#wrap) Ivan - FreeBSD read cache - ZFS (http://dpaste.com/1P9ETGQ#wrap) ***
We take a look at two-faced Oracle, cover a FAMP installation, how Netflix works the complex stuff, and show you who the patron of yak shaving is. This episode was brought to you by Headlines Why is Oracle so two-faced over open source? (https://www.theregister.co.uk/2017/10/12/oracle_must_grow_up_on_open_source/) Oracle loves open source. Except when the database giant hates open source. Which, according to its recent lobbying of the US federal government, seems to be "most of the time". Yes, Oracle has recently joined the Cloud Native Computing Foundation (CNCF) to up its support for open-source Kubernetes and, yes, it has long supported (and contributed to) Linux. And, yes, Oracle has even gone so far as to (finally) open up Java development by putting it under a foundation's stewardship. Yet this same, seemingly open Oracle has actively hammered the US government to consider that "there is no math that can justify open source from a cost perspective as the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." That punch to the face was delivered in a letter to Christopher Liddell, a former Microsoft CFO and now director of Trump's American Technology Council, by Kenneth Glueck, Oracle senior vice president. The US government had courted input on its IT modernisation programme. Others writing back to Liddell included AT&T, Cisco, Microsoft and VMware. In other words, based on its letter, what Oracle wants us to believe is that open source leads to greater costs and poorly secured, limply featured software. Nor is Oracle content to leave it there, also arguing that open source is exactly how the private sector does not function, seemingly forgetting that most of the leading infrastructure, big data, and mobile software today is open source. Details! Rather than take this counterproductive detour into self-serving silliness, Oracle would do better to follow Microsoft's path. Microsoft, too, used to Janus-face its way through open source, simultaneously supporting and bashing it. Only under chief executive Satya Nadella's reign did Microsoft realise it's OK to fully embrace open source, and its financial results have loved the commitment. Oracle has much to learn, and emulate, in Microsoft's approach. I love you, you're perfect. Now change Oracle has never been particularly warm and fuzzy about open source. As founder Larry Ellison might put it, Oracle is a profit-seeking corporation, not a peace-loving charity. To the extent that Oracle embraces open source, therefore it does so for financial reward, just like every other corporation. Few, however, are as blunt as Oracle about this fact of corporate open-source life. As Ellison told the Financial Times back in 2006: "If an open-source product gets good enough, we'll simply take it. So the great thing about open source is nobody owns it – a company like Oracle is free to take it for nothing, include it in our products and charge for support, and that's what we'll do. "So it is not disruptive at all – you have to find places to add value. Once open source gets good enough, competing with it would be insane... We don't have to fight open source, we have to exploit open source." "Exploit" sounds about right. While Oracle doesn't crack the top-10 corporate contributors to the Linux kernel, it does register a respectable number 12, which helps it influence the platform enough to feel comfortable building its IaaS offering on Linux (and Xen for virtualisation). Oracle has also managed to continue growing MySQL's clout in the industry while improving it as a product and business. As for Kubernetes, Oracle's decision to join the CNCF also came with P&L strings attached. "CNCF technologies such as Kubernetes, Prometheus, gRPC and OpenTracing are critical parts of both our own and our customers' development toolchains," said Mark Cavage, vice president of software development at Oracle. One can argue that Oracle has figured out the exploitation angle reasonably well. This, however, refers to the right kind of exploitation, the kind that even free software activist Richard Stallman can love (or, at least, tolerate). But when it comes to government lobbying, Oracle looks a lot more like Mr Hyde than Dr Jekyll. Lies, damned lies, and Oracle lobbying The current US president has many problems (OK, many, many problems), but his decision to follow the Obama administration's support for IT modernisation is commendable. Most recently, the Trump White House asked for feedback on how best to continue improving government IT. Oracle's response is high comedy in many respects. As TechDirt's Mike Masnick summarises, Oracle's "latest crusade is against open-source technology being used by the federal government – and against the government hiring people out of Silicon Valley to help create more modern systems. Instead, Oracle would apparently prefer the government just give it lots of money." Oracle is very good at making lots of money. As such, its request for even more isn't too surprising. What is surprising is the brazenness of its position. As Masnick opines: "The sheer contempt found in Oracle's submission on IT modernization is pretty stunning." Why? Because Oracle contradicts much that it publicly states in other forums about open source and innovation. More than this, Oracle contradicts much of what we now know is essential to competitive differentiation in an increasingly software and data-driven world. Take, for example, Oracle's contention that "significant IT development expertise is not... central to successful modernization efforts". What? In our "software is eating the world" existence Oracle clearly believes that CIOs are buyers, not doers: "The most important skill set of CIOs today is to critically compete and evaluate commercial alternatives to capture the benefits of innovation conducted at scale, and then to manage the implementation of those technologies efficiently." While there is some truth to Oracle's claim – every project shouldn't be a custom one-off that must be supported forever – it's crazy to think that a CIO – government or otherwise – is doing their job effectively by simply shovelling cash into vendors' bank accounts. Indeed, as Masnick points out: "If it weren't for Oracle's failures, there might not even be a USDS [the US Digital Service created in 2014 to modernise federal IT]. USDS really grew out of the emergency hiring of some top-notch internet engineers in response to the Healthcare.gov rollout debacle. And if you don't recall, a big part of that debacle was blamed on Oracle's technology." In short, blindly giving money to Oracle and other big vendors is the opposite of IT modernisation. In its letter to Liddell, Oracle proceeded to make the fantastic (by which I mean "silly and false") claim that "the fact is that the use of open-source software has been declining rapidly in the private sector". What?!? This is so incredibly untrue that Oracle should score points for being willing to say it out loud. Take a stroll through the most prominent software in big data (Hadoop, Spark, Kafka, etc.), mobile (Android), application development (Kubernetes, Docker), machine learning/AI (TensorFlow, MxNet), and compare it to Oracle's statement. One conclusion must be that Oracle believes its CIO audience is incredibly stupid. Oracle then tells a half-truth by declaring: "There is no math that can justify open source from a cost perspective." How so? Because "the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." Which I guess is why Oracle doesn't use any open source like Linux, Kubernetes, etc. in its services. Oops. The Vendor Formerly Known As Satan The thing is, Oracle doesn't need to do this and, for its own good, shouldn't do this. After all, we already know how this plays out. We need only look at what happened with Microsoft. Remember when Microsoft wanted us to "get the facts" about Linux? Now it's a big-time contributor to Linux. Remember when it told us open source was anti-American and a cancer? Now it aggressively contributes to a huge variety of open-source projects, some of them homegrown in Redmond, and tells the world that "Microsoft loves open source." Of course, Microsoft loves open source for the same reason any corporation does: it drives revenue as developers look to build applications filled with open-source components on Azure. There's nothing wrong with that. Would Microsoft prefer government IT to purchase SQL Server instead of open-source-licensed PostgreSQL? Sure. But look for a single line in its response to the Trump executive order that signals "open source is bad". You won't find it. Why? Because Microsoft understands that open source is a friend, not foe, and has learned how to monetise it. Microsoft, in short, is no longer conflicted about open source. It can compete at the product level while embracing open source at the project level, which helps fuel its overall product and business strategy. Oracle isn't there yet, and is still stuck where Microsoft was a decade ago. It's time to grow up, Oracle. For a company that builds great software and understands that it increasingly needs to depend on open source to build that software, it's disingenuous at best to lobby the US government to put the freeze on open source. Oracle needs to learn from Microsoft, stop worrying and love the open-source bomb. It was a key ingredient in Microsoft's resurgence. Maybe it could help Oracle get a cloud clue, too. Install FAMP on FreeBSD (https://www.linuxsecrets.com/home/3164-install-famp-on-freebsd) The acronym FAMP refers to a set of free open source applications which are commonly used in Web server environments called Apache, MySQL and PHP on the FreeBSD operating system, which provides a server stack that provides web services, database and PHP. Prerequisites sudo Installed and working - Please read Apache PHP5 or PHP7 MySQL or MariaDB Install your favorite editor, ours is vi Note: You don't need to upgrade FreeBSD but make sure all patches have been installed and your port tree is up-2-date if you plan to update by ports. Install Ports portsnap fetch You must use sudo for each indivdual command during installations. Please see link above for installing sudo. Searching Available Apache Versions to Install pkg search apache Install Apache To install Apache 2.4 using pkg. The apache 2.4 user account managing Apache is www in FreeBSD. pkg install apache24 Confirmation yes prompt and hit y for yes to install Apache 2.4 This installs Apache and its dependencies. Enable Apache use sysrc to update services to be started at boot time, Command below adds "apache24enable="YES" to the /etc/rc.conf file. For sysrc commands please read ```sysrc apache24enable=yes Start Apache service apache24 start``` Visit web address by accessing your server's public IP address in your web browser How To find Your Server's Public IP Address If you do not know what your server's public IP address is, there are a number of ways that you can find it. Usually, this is the address you use to connect to your server through SSH. ifconfig vtnet0 | grep "inet " | awk '{ print $2 }' Now that you have the public IP address, you may use it in your web browser's address bar to access your web server. Install MySQL Now that we have our web server up and running, it is time to install MySQL, the relational database management system. The MySQL server will organize and provide access to databases where our server can store information. Install MySQL 5.7 using pkg by typing pkg install mysql57-server Enter y at the confirmation prompt. This installs the MySQL server and client packages. To enable MySQL server as a service, add mysqlenable="YES" to the /etc/rc.conf file. This sysrc command will do just that ```sysrc mysqlenable=yes Now start the MySQL server service mysql-server start Now run the security script that will remove some dangerous defaults and slightly restrict access to your database system. mysqlsecureinstallation``` Answer all questions to secure your newly installed MySQL database. Enter current password for root (enter for none): [RETURN] Your database system is now set up and we can move on. Install PHP5 or PHP70 pkg search php70 Install PHP70 you would do the following by typing pkg install php70-mysqli mod_php70 Note: In these instructions we are using php5.7 not php7.0. We will be coming out with php7.0 instructions with FPM. PHP is the component of our setup that will process code to display dynamic content. It can run scripts, connect to MySQL databases to get information, and hand the processed content over to the web server to display. We're going to install the modphp, php-mysql, and php-mysqli packages. To install PHP 5.7 with pkg, run this command ```pkg install modphp56 php56-mysql php56-mysqli Copy sample PHP configuration file into place. cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini Regenerate the system's cached information about your installed executable files rehash``` Before using PHP, you must configure it to work with Apache. Install PHP Modules (Optional) To enhance the functionality of PHP, we can optionally install some additional modules. To see the available options for PHP 5.6 modules and libraries, you can type this into your system pkg search php56 Get more information about each module you can look at the long description of the package by typing pkg search -f apache24 Optional Install Example pkg install php56-calendar Configure Apache to Use PHP Module Open the Apache configuration file vim /usr/local/etc/apache24/Includes/php.conf DirectoryIndex index.php index.html Next, we will configure Apache to process requested PHP files with the PHP processor. Add these lines to the end of the file: SetHandler application/x-httpd-php SetHandler application/x-httpd-php-source Now restart Apache to put the changes into effect service apache24 restart Test PHP Processing By default, the DocumentRoot is set to /usr/local/www/apache24/data. We can create the info.php file under that location by typing vim /usr/local/www/apache24/data/info.php Add following line to info.php and save it. Details on info.php info.php file gives you information about your server from the perspective of PHP. It' useful for debugging and to ensure that your settings are being applied correctly. If this was successful, then your PHP is working as expected. You probably want to remove info.php after testing because it could actually give information about your server to unauthorized users. Remove file by typing rm /usr/local/www/apache24/data/info.php Note: Make sure Apache / meaning the root of Apache is owned by user which should have been created during the Apache install is the owner of the /usr/local/www structure. That explains FAMP on FreeBSD. IXsystems IXsystems TrueNAS X10 Torture Test & Fail Over Systems In Action with the ZFS File System (https://www.youtube.com/watch?v=GG_NvKuh530) How Netflix works: what happens every time you hit Play (https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b) Not long ago, House of Cards came back for the fifth season, finally ending a long wait for binge watchers across the world who are interested in an American politician's ruthless ascendance to presidency. For them, kicking off a marathon is as simple as reaching out for your device or remote, opening the Netflix app and hitting Play. Simple, fast and instantly gratifying. What isn't as simple is what goes into running Netflix, a service that streams around 250 million hours of video per day to around 98 million paying subscribers in 190 countries. At this scale, providing quality entertainment in a matter of a few seconds to every user is no joke. And as much as it means building top-notch infrastructure at a scale no other Internet service has done before, it also means that a lot of participants in the experience have to be negotiated with and kept satiated?—?from production companies supplying the content, to internet providers dealing with the network traffic Netflix brings upon them. This is, in short and in the most layman terms, how Netflix works. Let us just try to understand how Netflix is structured on the technological side with a simple example. Netflix literally ushered in a revolution around ten years ago by rewriting the applications that run the entire service to fit into a microservices architecture?—?which means that each application, or microservice's code and resources are its very own. It will not share any of it with any other app by nature. And when two applications do need to talk to each other, they use an application programming interface (API)?—?a tightly-controlled set of rules that both programs can handle. Developers can now make many changes, small or huge, to each application as long as they ensure that it plays well with the API. And since the one program knows the other's API properly, no change will break the exchange of information. Netflix estimates that it uses around 700 microservices to control each of the many parts of what makes up the entire Netflix service: one microservice stores what all shows you watched, one deducts the monthly fee from your credit card, one provides your device with the correct video files that it can play, one takes a look at your watching history and uses algorithms to guess a list of movies that you will like, and one will provide the names and images of these movies to be shown in a list on the main menu. And that's the tip of the iceberg. Netflix engineers can make changes to any part of the application and can introduce new changes rapidly while ensuring that nothing else in the entire service breaks down. They made a courageous decision to get rid of maintaining their own servers and move all of their stuff to the cloud?—?i.e. run everything on the servers of someone else who dealt with maintaining the hardware while Netflix engineers wrote hundreds of programs and deployed it on the servers rapidly. The someone else they chose for their cloud-based infrastructure is Amazon Web Services (AWS). Netflix works on thousands of devices, and each of them play a different format of video and sound files. Another set of AWS servers take this original film file, and convert it into hundreds of files, each meant to play the entire show or film on a particular type of device and a particular screen size or video quality. One file will work exclusively on the iPad, one on a full HD Android phone, one on a Sony TV that can play 4K video and Dolby sound, one on a Windows computer, and so on. Even more of these files can be made with varying video qualities so that they are easier to load on a poor network connection. This is a process known as transcoding. A special piece of code is also added to these files to lock them with what is called digital rights management or DRM?—?a technological measure which prevents piracy of films. The Netflix app or website determines what particular device you are using to watch, and fetches the exact file for that show meant to specially play on your particular device, with a particular video quality based on how fast your internet is at that moment. Here, instead of relying on AWS servers, they install their very own around the world. But it has only one purpose?—?to store content smartly and deliver it to users. Netflix strikes deals with internet service providers and provides them the red box you saw above at no cost. ISPs install these along with their servers. These Open Connect boxes download the Netflix library for their region from the main servers in the US?—?if there are multiple of them, each will rather store content that is more popular with Netflix users in a region to prioritise speed. So a rarely watched film might take time to load more than a Stranger Things episode. Now, when you will connect to Netflix, the closest Open Connect box to you will deliver the content you need, thus videos load faster than if your Netflix app tried to load it from the main servers in the US. In a nutshell… This is what happens when you hit that Play button: Hundreds of microservices, or tiny independent programs, work together to make one large Netflix service. Content legally acquired or licensed is converted into a size that fits your screen, and protected from being copied. Servers across the world make a copy of it and store it so that the closest one to you delivers it at max quality and speed. When you select a show, your Netflix app cherry picks which of these servers will it load the video from> You are now gripped by Frank Underwood's chilling tactics, given depression by BoJack Horseman's rollercoaster life, tickled by Dev in Master of None and made phobic to the future of technology by the stories in Black Mirror. And your lifespan decreases as your binge watching turns you into a couch potato. It looked so simple before, right? News Roundup Moving FreshPorts (http://dan.langille.org/2017/11/15/moving-freshports/) Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice. In preparation for this move, I have: DNS TTL reduced to 60s Posted to Twitter Updated the status page Put the website put in offline mode: What was missed I turned off commit processing on the new server, but I did not do this on the old server. I should have: sudo svc -d /var/service/freshports That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless. The offline code Here is the basic stuff I used to put the website into offline mode. The main points are: header(“HTTP/1.1 503 Service Unavailable”); ErrorDocument 404 /index.php I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code. The dump The database dump just started (Sun Nov 5 17:07:22 UTC 2017). root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump That should take about 30 minutes. I have set a timer to remind me. Total time was: 1464.82 real 1324.96 user 37.22 sys The MD5 is: MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5 It is now: Sun Nov 5 17:34:07 UTC 2017 The rsync The rsync should take about 10-20 minutes. I have already done an rsync of yesterday's dump file. The rsync today should copy over only the deltas (i.e. differences). The rsync started at about Sun Nov 5 17:36:05 UTC 2017 That took 2m9.091s The MD5 matches. The restore The restore should take about 30 minutes. I ran this test yesterday. It is now Sun Nov 5 17:40:03 UTC 2017. $ createdb -T template0 -E SQL_ASCII freshports.testing $ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump Done. real 25m21.108s user 1m57.508s sys 0m15.172s It is now Sun Nov 5 18:06:22 UTC 2017. Insert break here About here, I took a 30 minute break to run an errand. It was worth it. Changing DNS I'm ready to change DNS now. It is Sun Nov 5 19:49:20 EST 2017 Done. And nearly immediately, traffic started. How many misses? During this process, XXXXX requests were declined: $ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log XXXXX That's it, we're done Total elapsed time: 1 hour 48 minutes. There are still a number of things to follow up on, but that was the transfers. The new FreshPorts Server (http://dan.langille.org/2017/11/17/x8dtu-3/) *** Using bhyve on top of CEPH (https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005876.html) Hi, Just an info point. I'm preparing for a lecture tomorrow, and thought why not do an actual demo.... Like to be friends with Murphy :) So after I started the cluster: 5 jails with 7 OSDs This what I manually needed to do to boot a memory stick Start een Bhyve instance rbd --dest-pool rbddata --no-progress import memstick.img memstick rbd-ggate map rbddata/memstick ggate-devvice is available on /dev/ggate1 kldload vmm kldload nmdm kldload iftap kldload ifbridge kldload cpuctl sysctl net.link.tap.uponopen=1 ifconfig bridge0 create ifconfig bridge0 addm em0 up ifconfig ifconfig tap11 create ifconfig bridge0 addm tap11 ifconfig tap11 up load the GGate disk in bhyve bhyveload -c /dev/nmdm11A -m 2G -d /dev/ggate1 FB11 and boot a single from it. bhyve -H -P -A -c 1 -m 2G -l com1,/dev/nmdm11A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap11 -s 4,ahci-hd,/dev/ggate1 FB11 & bhyvectl --vm=FB11 --get-stats Connect to the VM cu -l /dev/nmdm11B And that'll give you a bhyve VM running on an RBD image over ggate. In the installer I tested reading from the bootdisk: root@:/ # dd if=/dev/ada0 of=/dev/null bs=32M 21+1 records in 21+1 records out 734077952 bytes transferred in 5.306260 secs (138341865 bytes/sec) which is a nice 138Mb/sec. Hope the demonstration does work out tomorrow. --WjW *** Donald Knuth - The Patron Saint of Yak Shaves (http://yakshav.es/the-patron-saint-of-yakshaves/) Excerpts: In 2015, I gave a talk in which I called Donald Knuth the Patron Saint of Yak Shaves. The reason is that Donald Knuth achieved the most perfect and long-running yak shave: TeX. I figured this is worth repeating. How to achieve the ultimate Yak Shave The ultimate yak shave is the combination of improbable circumstance, the privilege to be able to shave at your hearts will and the will to follow things through to the end. Here's the way it was achieved with TeX. The recount is purely mine, inaccurate and obviously there for fun. I'll avoid the most boring facts that everyone always tells, such as why Knuth's checks have their own Wikipedia page. Community Shaving is Best Shaving Since the release of TeX, the community has been busy working on using it as a platform. If you ever downloaded the full TeX distribution, please bear in mind that you are downloading the amassed work of over 40 years, to make sure that each and every TeX document ever written builds. We're talking about documents here. But mostly, two big projects sprung out of that. The first is LaTeX by Leslie Lamport. Lamport is a very productive researcher, famous for research in formal methods through TLA+ and also known laying groundwork for many distributed algorithms. LaTeX is based on the idea of separating presentation and content. It is based around the idea of document classes, which then describe the way a certain document is laid out. Think Markdown, just much more complex. The second is ConTeXt, which is far more focused on fine grained layout control. The Moral of the Story Whenever you feel like “can't we just replace this whole thing, it can't be so hard” when handling TeX, don't forget how many years of work and especially knowledge were poured into that system. Typesetting isn't the most popular knowledge around programmers. Especially see it in the context of the space it is in: they can't remove legacy. Ever. That would break documents. TeX is also not a programming language. It might resemble one, but mostly, it should be approached as a typesetting system first. A lot of it's confusing lingo gets much better then. It's not programming lingo. By approaching TeX with an understanding for its history, a lot of things can be learned from it. And yes, a replacement would be great, but it would take ages. In any case, I hope I thoroughly convinced you why Donald Knuth is the Patron Saint of Yak Shaves. Extra Credits This comes out of a enjoyable discussion with [Arne from Lambda Island](https://lambdaisland.com/https://lambdaisland.com/, who listened and said “you should totally turn this into a talk”. Vincent's trip to EuroBSDCon 2017 (http://www.vincentdelft.be/post/post_20171016) My euroBSDCon 2017 Posted on 2017-10-16 09:43:00 from Vincent in Open Bsd Let me just share my feedback on those 2 days spent in Paris for the EuroBSDCon. My 1st BSDCon. I'm not a developer, contributor, ... Do not expect to improve your skills with OpenBSD with this text :-) I know, we are on October 16th, and the EuroBSDCon of Paris was 3 weeks ago :( I'm not quick !!! Sorry for that Arrival at 10h, I'm too late for the start of the key note. The few persons behind a desk welcome me by talking in Dutch, mainly because of my name. Indeed, Delft is a city in Netherlands, but also a well known university. I inform them that I'm from Belgium, and the discussion moves to the fact the Fosdem is located in Brussels. I receive my nice T-shirt white and blue, a bit like the marine T-shirts, but with the nice EuroBSDCon logo. I'm asking where are the different rooms reserved for the BSD event. We have 1 big on the 1st floor, 1 medium 1 level below, and 2 smalls 1 level above. All are really easy to access. In this entrance we have 4 or 5 tables with some persons representing their company. Those are mainly the big sponsors of the event providing details about their activity and business. I discuss a little bit with StormShield and Gandi. On other tables people are selling BSD t-shirts, and they will quickly be sold. "Is it done yet ?" The never ending story of pkg tools In the last Fosdem, I've already hear Antoine and Baptiste presenting the OpenBSD and FreeBSD battle, I decide to listen Marc Espie in the medium room called Karnak. Marc explains that he has rewritten completely the pkg_add command. He explains that, at contrario with other elements of OpenBSD, the packages tools must be backward compatible and stable on a longer period than 12 months (the support period for OpenBSD). On the funny side, he explains that he has his best idea inside his bath. Hackathons are also used to validate some ideas with other OpenBSD developers. All in all, he explains that the most time consuming part is to imagine a good solution. Coding it is quite straightforward. He adds that better an idea is, shorter the implementation will be. A Tale of six motherboards, three BSDs and coreboot After the lunch I decide to listen the talk about Coreboot. Indeed, 1 or 2 years ago I had listened the Libreboot project at Fosdem. Since they did several references to Coreboot, it's a perfect occasion to listen more carefully to this project. Piotr and Katazyba Kubaj explains us how to boot a machine without the native Bios. Indeed Coreboot can replace the bios, and de facto avoid several binaries imposed by the vendor. They explain that some motherboards are supporting their code. But they also show how difficult it is to flash a Bios and replace it by Coreboot. They even have destroyed a motherboard during the installation. Apparently because the power supply they were using was not stable enough with the 3v. It's really amazing to see that open source developers can go, by themselves, to such deep technical level. State of the DragonFly's graphics stack After this Coreboot talk, I decide to stay in the room to follow the presentation of Fran?ois Tigeot. Fran?ois is now one of the core developer of DrangonflyBSD, an amazing BSD system having his own filesystem called Hammer. Hammer offers several amazing features like snapshots, checksum data integrity, deduplication, ... Francois has spent his last years to integrate the video drivers developed for Linux inside DrangonflyBSD. He explains that instead of adapting this code for the video card to the kernel API of DrangonflyBSD, he has "simply" build an intermediate layer between the kernel of DragonflyBSD and the video drivers. This is not said in the talk, but this effort is very impressive. Indeed, this is more or less a linux emulator inside DragonflyBSD. Francois explains that he has started with Intel video driver (drm/i915), but now he is able to run drm/radeon quite well, but also drm/amdgpu and drm/nouveau. Discovering OpenBSD on AWS Then I move to the small room at the upper level to follow a presentation made by Laurent Bernaille on OpenBSD and AWS. First Laurent explains that he is re-using the work done by Antoine Jacoutot concerning the integration of OpenBSD inside AWS. But on top of that he has integrated several other Open Source solutions allowing him to build OpenBSD machines very quickly with one command. Moreover those machines will have the network config, the required packages, ... On top of the slides presented, he shows us, in a real demo, how this system works. Amazing presentation which shows that, by putting the correct tools together, a machine builds and configure other machines in one go. OpenBSD Testing Infrastructure Behind bluhm.genua.de Here Jan Klemkow explains us that he has setup a lab where he is able to run different OpenBSD architectures. The system has been designed to be able to install, on demand, a certain version of OpenBSD on the different available machines. On top of that a regression test script can be triggered. This provides reports showing what is working and what is not more working on the different machines. If I've well understood, Jan is willing to provide such lab to the core developers of OpenBSD in order to allow them to validate easily and quickly their code. Some more effort is needed to reach this goal, but with what exists today, Jan and his colleague are quite close. Since his company is using OpenBSD business, to his eyes this system is a "tit for tat" to the OpenBSD community. French story on cybercrime Then comes the second keynote of the day in the big auditorium. This talk is performed by the colonel of french gendarmerie. Mr Freyssinet, who is head of the Cyber crimes unit inside the Gendarmerie. Mr Freyssinet explains that the "bad guys" are more and more volatile across countries, and more and more organized. The small hacker in his room, alone, is no more the reality. As a consequence the different national police investigators are collaborating more inside an organization called Interpol. What is amazing in his talk is that Mr Freyssinet talks about "Crime as a service". Indeed, more and more hackers are selling their services to some "bad and temporary organizations". Social event It's now time for the famous social event on the river: la Seine. The organizers ask us to go, by small groups, to a station. There is a walk of 15 minutes inside Paris. Hopefully the weather is perfect. To identify them clearly several organizers takes a "beastie fork" in their hands and walk on the sidewalk generating some amazing reactions from some citizens and toursits. Some of them recognize the Freebsd logo and ask us some details. Amazing :-) We walk on small and big sidewalks until a small stair going under the street. There, we have a train station a bit like a metro station. 3 stations later they ask us to go out. We walk few minutes and come in front of a boat having a double deck: one inside, with nice tables and chairs and one on the roof. But the crew ask us to go up, on the second deck. There, we are welcome with a glass of wine. The tour Eiffel is just at few 100 meters from us. Every hour the Eiffel tower is blinking for 5 minutes with thousands of small lights. Brilliant :-) We see also the "statue de la libertee" (the small one) which is on a small island in the middle of the river. During the whole night the bar will be open with drinks and some appetizers, snacks, ... Such walking diner is perfect to talk with many different persons. I've discussed with several persons just using BSD, they are not, like me, deep and specialized developers. One was from Switzerland, another one from Austria, and another one from Netherlands. But I've also followed a discussion with Theo de Raadt, several persons of the FreeBSD foundation. Some are very technical guys, other just users, like me. But all with the same passion for one of the BSD system. Amazing evening. OpenBSD's small steps towards DTrace (a tale about DDB and CTF) On the second day, I decide to sleep enough in order to have enough resources to drive back to my home (3 hours by car). So I miss the 1st presentations, and arrive at the event around 10h30. Lot of persons are already present. Some faces are less "fresh" than others. I decide to listen to Dtrace in OpenBSD. After 10 minutes I am so lost into those too technical explainations, that I decide to open and look at my PC. My OpenBSD laptop is rarely leaving my home, so I've never had the need to have a screen locking system. In a crowded environment, this is better. So I was looking for a simple solution. I've looked at how to use xlock. I've combined it with the /ets/apm/suspend script, ... Always very easy to use OpenBSD :-) The OpenBSD web stack Then I decide to follow the presentation of Michael W Lucas. Well know person for his different books about "Absolute OpenBSD", Relayd", ... Michael talks about the httpd daemon inside OpenBSD. But he also present his integration with Carp, Relayd, PF, FastCGI, the rules based on LUA regexp (opposed to perl regexp), ... For sure he emphasis on the security aspect of those tools: privilege separation, chroot, ... OpenSMTPD, current state of affairs Then I follow the presentation of Gilles Chehade about the OpenSMTPD project. Amazing presentation that, on top of the technical challenges, shows how to manage such project across the years. Gilles is working on OpenSMTPD since 2007, thus 10 years !!!. He explains the different decisions they took to make the software as simple as possible to use, but as secure as possible, too: privilege separation, chroot, pledge, random malloc, ? . The development starts on BSD systems, but once quite well known they received lot of contributions from Linux developers. Hoisting: lessons learned integrating pledge into 500 programs After a small break, I decide to listen to Theo de Raadt, the founder of OpenBSD. In his own style, with trekking boots, shorts, backpack. Theo starts by saying that Pledge is the outcome of nightmares. Theo explains that the book called "Hacking blind" presenting the BROP has worried him since few years. That's why he developed Pledge as a tool killing a process as soon as possible when there is an unforeseen behavior of this program. For example, with Pledge a program which can only write to disk will be immediately killed if he tries to reach network. By implementing Pledge in the +-500 programs present in the "base", OpenBSD is becoming more secured and more robust. Conclusion My first EuroBSDCon was a great, interesting and cool event. I've discussed with several BSD enthusiasts. I'm using OpenBSD since 2010, but I'm not a developer, so I was worried to be "lost" in the middle of experts. In fact it was not the case. At EuroBSDCon you have many different type of enthusiasts BSD's users. What is nice with the EuroBSDCon is that the organizers foresee everything for you. You just have to sit and listen. They foresee even how to spend, in a funny and very cool attitude, the evening of Saturday. > The small draw back is that all of this has a cost. In my case the whole weekend cost me a bit more than 500euro. Based on what I've learned, what I've saw this is very acceptable price. Nearly all presentations I saw give me a valuable input for my daily job. For sure, the total price is also linked to my personal choice: hotel, parking. And I'm surely biased because I'm used to go to the Fosdem in Brussels which cost nothing (entrance) and is approximately 45 minutes of my home. But Fosdem is not the same atmosphere and presentations are less linked to my daily job. I do not regret my trip to EuroBSDCon and will surely plan other ones. Beastie Bits Important munitions lawyering (https://www.jwz.org/blog/2017/10/important-munitions-lawyering/) AsiaBSDCon 2018 CFP is now open, until December 15th (https://2018.asiabsdcon.org/) ZSTD Compression for ZFS by Allan Jude (https://www.youtube.com/watch?v=hWnWEitDPlM&feature=share) NetBSD on Allwinner SoCs Update (https://blog.netbsd.org/tnf/entry/netbsd_on_allwinner_socs_update) *** Feedback/Questions Tim - Creating Multi Boot USB sticks (http://dpaste.com/0FKTJK3#wrap) Nomen - ZFS Questions (http://dpaste.com/1HY5MFB) JJ - Questions (http://dpaste.com/3ZGNSK9#wrap) Lars - Hardening Diffie-Hellman (http://dpaste.com/3TRXXN4) ***
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
In this episode, we take a look at the reimplementation of NetBSD using a Microkernel, check out what makes DHCP faster, and see what high-process count support for DragonflyBSD has to offer, and we answer the questions you've always wanted to ask us. This episode was brought to you by Headlines A Reimplementation Of Netbsd Using a Microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Minix author Andy Tanenbaum writes in Part 1 of a-reimplementation-of-netbsd-using-a-microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project. Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running. The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this. The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project. Part 2 (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-2-of-2/) is also available. *** Rapid DHCP: Or, how do Macs get on the network so fast? (https://cafbit.com/post/rapid_dhcp_or_how_do/) One of life's minor annoyances is having to wait on my devices to connect to the network after I wake them from sleep. All too often, I'll open the lid on my EeePC netbook, enter a web address, and get the dreaded "This webpage is not available" message because the machine is still working on connecting to my Wi-Fi network. On some occasions, I have to twiddle my thumbs for as long as 10-15 seconds before the network is ready to be used. The frustrating thing is that I know it doesn't have to be this way. I know this because I have a Mac. When I open the lid of my MacBook Pro, it connects to the network nearly instantaneously. In fact, no matter how fast I am, the network comes up before I can even try to load a web page. My curiosity got the better of me, and I set out to investigate how Macs are able to connect to the network so quickly, and how the network connect time in other operating systems could be improved. I figure there are three main categories of time-consuming activities that occur during network initialization: Link establishment. This is the activity of establishing communication with the network's link layer. In the case of Wi-Fi, the radio must be powered on, the access point detected, and the optional encryption layer (e.g. WPA) established. After link establishment, the device is able to send and receive Ethernet frames on the network. Dynamic Host Configuration Protocol (DHCP). Through DHCP handshaking, the device negotiates an IP address for its use on the local IP network. A DHCP server is responsible for managing the IP addresses available for use on the network. Miscellaneous overhead. The operating system may perform any number of mundane tasks during the process of network initialization, including running scripts, looking up preconfigured network settings in a local database, launching programs, etc. My investigation thus far is primarily concerned with the DHCP phase, although the other two categories would be interesting to study in the future. I set up a packet capture environment with a spare wireless access point, and observed the network activity of a number of devices as they initialized their network connection. For a worst-case scenario, let's look at the network activity captured while an Android tablet is connecting: This tablet, presumably in the interest of "optimization", is initially skipping the DHCP discovery phase and immediately requesting its previous IP address. The only problem is this is a different network, so the DHCP server ignores these requests. After about 4.5 seconds, the tablet stubbornly tries again to request its old IP address. After another 4.5 seconds, it resigns itself to starting from scratch, and performs the DHCP discovery needed to obtain an IP address on the new network. In all fairness, this delay wouldn't be so bad if the device was connecting to the same network as it was previously using. However, notice that the tablet waits a full 1.13 seconds after link establishment to even think about starting the DHCP process. Engineering snappiness usually means finding lots of small opportunities to save a few milliseconds here and there, and someone definitely dropped the ball here. In contrast, let's look at the packet dump from the machine with the lightning-fast network initialization, and see if we can uncover the magic that is happening under the hood: The key to understanding the magic is the first three unicast ARP requests. It looks like Mac OS remembers certain information about not only the last connected network, but the last several networks. In particular, it must at least persist the following tuple for each of these networks: > 1. The Ethernet address of the DHCP server > 2. The IP address of the DHCP server > 3. Its own IP address, as assigned by the DHCP server During network initialization, the Mac transmits carefully crafted unicast ARP requests with this stored information. For each network in its memory, it attempts to send a request to the specific Ethernet address of the DHCP server for that network, in which it asks about the server's IP address, and requests that the server reply to the IP address which the Mac was formerly using on that network. Unless network hosts have been radically shuffled around, at most only one of these ARP requests will result in a response—the request corresponding to the current network, if the current network happens to be one of the remembered networks. This network recognition technique allows the Mac to very rapidly discover if it is connected to a known network. If the network is recognized (and presumably if the Mac knows that the DHCP lease is still active), it immediately and presumptuously configures its IP interface with the address it knows is good for this network. (Well, it does perform a self-ARP for good measure, but doesn't seem to wait more than 13ms for a response.) The DHCP handshaking process begins in the background by sending a DHCP request for its assumed IP address, but the network interface is available for use during the handshaking process. If the network was not recognized, I assume the Mac would know to begin the DHCP discovery phase, instead of sending blind requests for a former IP address as the Galaxy Tab does. The Mac's rapid network initialization can be credited to more than just the network recognition scheme. Judging by the use of ARP (which can be problematic to deal with in user-space) and the unusually regular transmission intervals (a reliable 1.0ms delay between each packet sent), I'm guessing that the Mac's DHCP client system is entirely implemented as tight kernel-mode code. The Mac began the IP interface initialization process a mere 10ms after link establishment, which is far faster than any other device I tested. Android devices such as the Galaxy Tab rely on the user-mode dhclient system (part of the dhcpcd package) dhcpcd program, which no doubt brings a lot of additional overhead such as loading the program, context switching, and perhaps even running scripts. The next step for some daring kernel hacker is to implement a similarly aggressive DHCP client system in the Linux kernel, so that I can enjoy fast sign-on speeds on my Android tablet, Android phone, and Ubuntu netbook. There already exists a minimal DHCP client implementation in the Linux kernel, but it lacks certain features such as configuring the DNS nameservers. Perhaps it wouldn't be too much work to extend this code to support network recognition and interface with a user-mode daemon to handle such auxillary configuration information received via DHCP. If I ever get a few spare cycles, maybe I'll even take a stab at it. You can also find other ways of optimizing the dhclient program and how it works in the dhclient tutorial on Calomel.org (https://calomel.org/dhclient.html). *** BSDCam Trip Report (https://www.freebsdfoundation.org/blog/bsdcam-2017-trip-report-michael-lucas/) Over the decades, FreeBSD development and coordination has shifted from being purely on-line to involving more and more in-person coordination and cooperation. The FreeBSD Foundation sponsors a devsummit right before BSDCan, EuroBSDCon, and AsiaBSDCon, so that developers traveling to the con can leverage their airfare and hammer out some problems. Yes, the Internet is great for coordination, but nothing beats a group of developers spending ten minutes together to sketch on a whiteboard and figuring out exactly how to make something bulletproof. In addition to the coordination efforts, though, conference devsummits are hierarchical. There's a rigid schedule, with topics decided in advance. Someone leads the session. Sessions can be highly informative, passionate arguments, or anything in between. BSDCam is… a little different. It's an invaluable part of the FreeBSD ecosystem. However, it's something that I wouldn't normally attend. But right now, is not normal. I'm writing a new edition of Absolute FreeBSD. To my astonishment, people have come to rely on this book when planning their deployments and operations. While I find this satisfying, it also increases the pressure on me to get things correct. When I wrote my first FreeBSD book back in 2000, a dozen mailing lists provided authoritative information on FreeBSD development. One person could read every one of those lists. Today, that's not possible—and the mailing lists are only one narrow aspect of the FreeBSD social system. Don't get me wrong—it's pretty easy to find out what people are doing and how the system works. But it's not that easy to find out what people will be doing and how the system will work. If this book is going to be future-proof, I needed to leave my cozy nest and venture into the wilds of Cambridge, England. Sadly, the BSDCam chair agreed with my logic, so I boarded an aluminum deathtrap—sorry, a “commercial airliner”—and found myself hurtled from Detroit to Heathrow. And one Wednesday morning, I made it to the William Gates building of Cambridge University, consciousness nailed to my body by a thankfully infinite stream of proper British tea. BSDCam attendance is invitation only, and the facilities can only handle fifty folks or so. You need to be actively working on FreeBSD to wrangle an invite. Developers attend from all over the world. Yet, there's no agenda. Robert Watson is the chair, but he doesn't decide on the conference topics. He goes around the room and asks everyone to introduce themselves, say what they're working on, and declare what they want to discuss during the conference. The topics of interest are tallied. The most popular topics get assigned time slots and one of the two big rooms. Folks interested in less popular topics are invited to claim one of the small breakout rooms. Then the real fun begins. I started by eavesdropping in the virtualization workshop. For two hours, people discussed FreeBSD's virtualization needs, strengths, and weaknesses. What needs help? What should this interface look like? What compatibility is important, and what isn't? By the end of the session, the couple dozen people had developed a reasonable consensus and, most importantly, some folks had added items to their to-do lists. Repeat for a dozen more topics. I got a good grip on what's really happening with security mitigation techniques, FreeBSD's cloud support, TCP/IP improvements, advances in teaching FreeBSD, and more. A BSDCan devsummit presentation on packaging the base system is informative, but eavesdropping on two dozen highly educated engineers arguing about how to nail down the final tidbits needed to make that a real thing is far more educational. To my surprise, I was able to provide useful feedback for some sessions. I speak at a lot of events outside of the FreeBSD world, and was able to share much of what I hear at Linux conferences. A tool that works well for an experienced developer doesn't necessarily work well for everyone. Every year, I leave BSDCan tired. I left BSDCam entirely exhausted. These intense, focused discussions stretched my brain. But, I have a really good idea where key parts of FreeBSD development are actually headed. This should help future-proof the new Absolute FreeBSD, as much as any computer book can be future-proof. Plus, BSDCam throws the most glorious conference dinner I've ever seen. I want to thank Robert Watson for his kind invitation, and the FreeBSD Foundation for helping defray the cost of this trip Interview - The BSDNow Crew As a kid, what did you dream of to become as an adult? JT: An Astronaut BR: I wanted to be a private detective, because of all the crime novels that I read back then. I didn't get far with it. However, I think the structured analysis skills (who did what, when, and such) help me in debugging and sysadmin work. AJ: Didn't think about it much How do you manage to stay organized day to day with so much things you're actively doing each day? (Day job, wife/girlfriend, conferences, hobbies, friends, etc.) JT: Who said I was organized? BR: A lot of stuff in my calendar as reminders, open browser tabs as “to read later” list. A few things like task switching when getting stuck helps. Also, focus on a single goal for the day, even though there will be distractions. Slowly, but steadily chip away at the things you're working on. Rather than to procrastinate and put things back to review later, get started early with easy things for a big task and then tackle the hard part. Often, things look totally chaotic and unmanageable, until you start working on them. AJ: I barely manage. Lots of Google Calendar reminders, and the entire wall of my office is covered in whiteboard sheet todo lists. I use pinboard.in to deal with finding and organizing bookmarks. Write things down, don't trust your memory. What hobbies outside of IT do you have? JT: I love photography, but I do that Professional part time, so I'm not sure if that counts as a hobby anymore. I guess it'd have to be working in the garage on my cars. BR: I do Tai Chi to relax once a week in a group, but can also do it alone, pretty much everywhere. Way too much Youtube watching and browsing the web. I did play some games before studying at the university and I'm still proud that I could control it to the bare minimum not to impact my studies. A few “lapses” from time to time, revisiting the old classics since the newer stuff won't run on my machines anyway. Holiday time is pretty much spent for BSD conferences and events, this is where I can relax and talk with like-minded people from around the world, which is fascinating. Plus, it gets me to various places and countries I never would have dared to visit on my own. AJ: I play a few video games, and I like to ski, although I don't go very often as most of my vacation time is spent hanging out with my BSD friends at various conferences How do you relax? JT: What is this word ‘relax' and what does it mean? BR: My Tai Chi plays a big part in it I guess. I really calms you and the constant stream of thoughts for a while. It also gives you better clarity of what's important in life. Watching movies, sleeping long. AJ: Usually watching TV or Movies. Although I have taken to doing most of my TV watching on my exercise bike now, but it is still mentally relaxing If FreeBSD didn't exist, which BSD flavour would you use? Why? JT: I use TrueOS, but if FreeBSD didn't exist, that project might not either… so… My other choice would be HardenedBSD, but since it's also based on FreeBSD I'm in the same dillema. BR: I once installed NetBSD to see what It can do. If FreeBSD wouldn't exist, I would probably try my luck with it. OpenBSD is also appealing, but I've never installed it. AJ: When I started using FreeBSD in 2000, the only other BSD I had heard of at the time was OpenBSD. If FreeBSD wasn't around, I don't think the world would look like it does, so it is hard to speculate. If any of the BSD's weren't around and you had to use Linux, which camp would belong to? (Redhat, SUSE, Debian, Ubuntu, Gentoo?) JT: I learned Linux in the mid 90s using Slackware, which I used consistently up until the mid 2000s, when I joined the PuppyLinux community and eventually became a developer (FYI, Puppy was/is/can be based on Slackware -- its complicated). So I'd go back to using either Slackware or PuppyLinux. BR: I tried various Linux distributions until I landed at Debian. I used is pretty extensively as my desktop OS at home, building custom kernels and packages to install them until I discovered FreeBSD. I ran both side by side for a few months for learning until one day I figured out that I had not booted Debian in a while, so I switched completely. AJ: The first Linux I played with was Slackware, and it is the most BSD like, but the bits of Linux I learned in school were Redhat and so I can somewhat wrap my head around it, although now that they are changing everything to systemd, all of that old knowledge is more harmful than useful. Are you still finding yourself in need to use Windows/Mac OS? Why? JT: I work part time as a professional Photographer, so I do use Windows for my photography work. While I can do everything I need to do in Linux, it comes down to being pragmatic about my time. What takes me several hours to accomplish in Linux I can accomplish in 20 minutes on Windows. BR: I was a long time Windows-only user before my Unix days. But back when Vista was about to come out and I needed a new laptop, my choice was basically learning to cope with Vistas awful features or learn MacOS X. I did the latter, it increased my productivity since it's really a good Unix desktop experience (at least, back then). I only have to use Windows at work from time to time as I manage our Windows Terminal server, which keeps the exposure low enough and I only connect to it to use a certain app not available for the Mac or the BSDs. AJ: I still use Windows to play games, for a lot of video conferencing, and to produce BSD Now. Some of it could be done on BSD but not as easily. I have promised myself that I will switch to 100% BSD rather than upgrade to Windows 10, so we'll see how that goes. Please describe your home networking setup. Router type, router OS, router hardware, network segmentation, wifi apparatus(es), other devices connected, and anything else that might be interesting about your home network. BR: Very simple and boring: Apple Airport Express base station and an AVM FritzBox for DNS, DHCP, and the link to my provider. A long network cable to my desktop machine. That I use less and less often. I just bought an RPI 3 for some home use in the future to replace it. Mostly my brother's and my Macbook Pro's are connected, our phones and the iPad of my mother. AJ: I have a E3-1220 v3 (dual 3.1ghz + HT) with 8 GB of ram, and 4x Intel gigabit server NICs as my router, and it runs vanilla FreeBSD (usually some snapshot of -current). I have 4 different VLANs, Home, Office, DMZ, and Guest WiFi. WiFi is served via a tiny USB powered device I bought in Tokyo years ago, it serves 3 different SSIDs, one for each VLAN except the DMZ. There are ethernet jacks in every room wired for 10 gigabit, although the only machines with 10 gigabit are my main workstation, file server, and some machines in the server rack. There are 3 switches, one for the house (in the laundry room), one for the rack, and one for 10gig stuff. There is a rack in the basement spare bedroom, it has 7 servers in it, mostly storage for live replicas of customer data for my company. How do guys manage to get your work done on FreeBSD desktops? What do you do when you need to a Linux or Windows app that isn't ported, or working? I've made several attempts to switch to FreeBSD, but each attempt failed because of tools not being available (e.g. Zoom, Dropbox, TeamViewer, Crashplan) or broken (e.g. VirtualBox). BR: I use VIrtualBox for everything that is not natively available or Windows-only. Unfortunately, that means no modern games. I mostly do work in the shell when I'm on FreeBSD and when it has to be a graphical application, then I use Fluxbox as the DE. I want to get work done, not look at fancy eye-candy that get's boring after a while. Deactivated the same stuff on my mac due to the same reason. I look for alternative software online, but my needs are relatively easy to satisfy as I'm not doing video editing/rendering and such. AJ: I generally find that I don't need these apps. I use Firefox, Thunderbird, OpenSSH, Quassel, KomodoEdit, and a few other apps, so my needs are not very demanding. It is annoying when packages are broken, but I usually work around this with boot environments, and being able to just roll back to a version that worked for a few days until the problem is solved. I do still have access to a windows machine for the odd time I need specific VPN software or access to Dell/HP etc out-of-band management tools. Which desktop environments are your favorite, and why? For example, I like i3, Xfce, and I'm drawn to Lumina's ethos, but so far always seem to end up back on Xfc because of its ease of use, flexibility, and dashing good looks. JT: As a Lumina Desktop developer, I think my preference is obvious. ;) I am also a long timeOpenBox user, so I have a soft place in my heart for that as well. BR: I use Fluxbox when I need to work with a lot of windows or an application demands X11. KDE and others are too memory heavy for me and I rarely use even 20% of the features they provide. AJ: I was a long time KDE user, but I have adopted Lumina. I find it fast, and that it gets out of my way and lets me do what I want. It had some annoyances early on, but I've nagged the developers into making it work for me. Which command-line shells do you prefer, why, and how (if at all) have you customised the environment or prompt? BR: I use zsh, but without all the fancy stuff you can find online. It might make you more productive, yes. But again, I try to keep things simple. I'm slowly learning tmux and want to work more in it in the future. I sometimes look at other BSD people's laptops and am amazed at what they do with window-management in tmux. My prompt looks like this: bcr@Voyager:~> 20:20 17-08-17 Put this in your .zshrc to get the same result: PROMPT='%n@%m:%~>' RPROMPT='%T %D' AJ: I started using tcsh early on, because it was the shell on the first box I had access to, and because one of the first things I read in “BSD Hacks” was how to enable ‘typo correction”, which made my life a lot better especially on dial up in the early days. My shell prompt looks like this: allan@CA-TOR1-02:/usr/home/allan% What is one thing (or more) missing in FreeBSD you would import from another project or community? Could be tech, process, etc. JT: AUFS from Linux BR: Nohup from Illumos where you can detach an already running process and put it in the background. I often forget that and I'm not in tmux when that happens, so I can see myself use that feature a lot. AJ: Zones (more complete Jails) from IllumOS how do you manage your time to learn about and work on FreeBSD? Does your work/employment enable what you do, or are your contributions mainly done in private time? JT: These days I'm mostly learning things I need for work, so it just falls into something I'm doing while working on work projects. BR: We have a lot of time during the semester holidays to learn on our own, it's part of the idea of being in a university to keep yourself updated, at least for me. Especially in the fast moving world of IT. I also read a lot in my free time. My interests can shift sometimes, but then I devour everything I can find on the topic. Can be a bit excessive, but has gotten me where I am now and I still need a lot to learn (and want to). Since I work with FreeBSD at work (my owndoing), I can try out many things there. AJ: My work means a spend a lot of time working with FreeBSD, but not that much time working ON it. My contributions are mostly done outside of work, but as I own the company I do get more flexibility to take time off for conferences and other FreeBSD related stuff. we know we can bribe Michael W Lucas with gelato (good gelato that is), but what can we use to bribe you guys? Like when I want to have Allan to work on fixing a bug which prevents me from running ZFS on this fancy rock64 board? BR: Desserts of various kinds. AJ: I am probably not the right person to look at your rock64 board. Most people in the project have taken to bribing me with chocolate. In general, my todo list is so long, the best way is a trade, you take this task and I'll take that task. Is your daily mobile device iOS, Android, Windows Mobile, or other? Why? JT: These days I'm using Android on my Blackberry Priv, but until recently I was still a heavy user of Sailfish OS. I would use SailfishOS everyday, if I could find a phone with a keyboard that I could run it on. BR: iOS on the iPhone 7 currently. Never used an Android phone, saw it on other people's devices and what they can do with it (much more). But the infrequent security updates (if any at all) keep me away from it. AJ: I have a Google Nexus 6 (Android 7.1). I wanted the ‘pure' Android experience, and I had been happy with my previous Nexus S. I don't run a custom OS/ROM or anything because I use the phone to verify that video streams work on an ‘average users device'. I am displeased that support for my device will end soon. I am not sure what device I will get next, but it definitely won't be an iPhone. News Roundup Beta Update - Request for (more) Testing (http://undeadly.org/cgi?action=article&sid=20170808065718&mode=flat&count=30) https://beta.undeadly.org/ has received an update. The most significant changes include: The site has been given a less antiquated "look". (As the topic icons have been eliminated, we are no longer seeking help with those graphics.) The site now uses a moderate amount of semantic HTML5. Several bugs in the HTML fragment validator (used for submissions and comments) have been fixed. To avoid generating invalid HTML, submission content which fails validation is no longer displayed in submission/comment previews. Plain text submissions are converted to HTML in a more useful fashion. (Instead of just converting each EOL to , the converter now generates proper paragraphs and interprets two or more consecutive EOLs as indicating a paragraph break.) The redevelopment remains a work-in-progress. Many thanks to those who have contributed! As before, constructive feedback would be appreciated. Of particular interest are reports of bugs in behaviour (for example, in the HTML validator or in authentication) that would preclude the adoption of the current code for the main site. High-process-count support added to master (http://lists.dragonflybsd.org/pipermail/users/2017-August/313552.html) We've fixed a number of bottlenecks that can develop when the number of user processes runs into the tens of thousands or higher. One thing led to another and I said to myself, "gee, we have a 6-digit PID, might as well make it work to a million!". With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. And, in fact, as today's machines start to ratchet up there in both memory capacity and core count, with fast storage (NVMe) and fast networking (10GigE and higher), even in consumer boxes, this is actually something that one might want to do. With AMD's threadripper and EPYC chips now out, the IntelAMD cpu wars are back on! Boasting up to 32 cores (64 threads) per socket and two sockets on EPYC, terabytes of ram, and motherboards with dual 10GigE built-in, the reality is that these numbers are already achievable in a useful manner. In anycase, I've tested these changes on a dual-socket xeon. I can in-fact start 900,000 processes. They don't get a whole lot of cpu and running 'ps' would be painful, but it works and the system is still responsive from the shell with all of that going on. xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 In fact, judging from the memory use, these minimal test processes only eat around 60KB each. 900,000 of them ate only 55GB on a 128GB machine. So even a million processes is not out of the question, depending on the cpu requirements for those processes. Today's modern machines can be stuffed with enormous amounts of memory. Of course, our PIDs are currently limited to 6 digits, so a million is kinda the upper limit in terms of discrete user processes (verses pthreads which are less restricted). I'd rather not go to 7 digits (yet). CFT: Driver for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens (https://lists.freebsd.org/pipermail/freebsd-current/2017-August/066783.html) Following patch [1] adds support for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens via evdev protocol. It is intended to be a native replacement of hid-multitouch.c driver found in Linux distributions and multimedia/webcamd port. Patch is made for 12-CURRENT and most probably can be applied to recent 11-STABLE and 11.1-RELEASE (not tested) How to test" 1. Apply patch [1] 2. To compile this driver into the kernel, place the following lines into your kernel configuration file: device wmt device usb device evdev Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5): wmt_load="YES" 3. Install x11-drivers/xf86-input-evdev or x11-drivers/xf86-input-libinput port 4. Tell XOrg to use evdev or libinput driver for the device: ``` Section "ServerLayout" InputDevice "TouchScreen0" "SendCoreEvents" EndSection Section "InputDevice" Identifier "TouchScreen0" Driver "evdev" # Driver "libinput" Option "Device" "/dev/input/eventXXX" EndSection ``` Exact value of "/dev/input/eventXXX" can be obtained with evemu-record utility from devel/evemu. Note1: Currently, driver does not support pens or touchpads. Note2: wmt.ko should be kld-loaded before uhid driver to take precedence over it! Otherwise uhid can be kld-unloaded after loading of wmt. wmt review: https://reviews.freebsd.org/D12017 Raw diff: https://reviews.freebsd.org/D12017.diff *** Beastie Bits BSDMag Programing Languages Infographic (https://bsdmag.org/programm_history/) t2k17 Hackathon Report: Bob Beck on buffer cache tweaks, libressl and pledge progress (http://undeadly.org/cgi?action=article&sid=20170815171854) New FreeBSD Journal (https://www.freebsdfoundation.org/past-issues/resource-control/) NetBSD machines at Open Source Conference 2017 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2017/08/10/msg000744.html) *** Feedback/Questions Dan - HDD question (http://dpaste.com/3H6TDJV) Benjamin - scrub of death (http://dpaste.com/10F086V) Jason - Router Opinion (http://dpaste.com/2D9102K) Sohrab - Thanks (http://dpaste.com/1XYYTWF) ***
DragonflyBSD 4.8.1 has been released, we explore how the X11 clipboard works, and look at OpenBSD gaming resources. This episode was brought to you by Headlines LLVM, Clang and compiler-rt support enhancements (https://blog.netbsd.org/tnf/entry/llvm_clang_and_compiler_rt) In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero. LLVM In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests. There were four threads within LLVM: Broken std::callonce with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::callonce. I've discovered that the llvm::callonce workaround isn't sufficient for the whole LLVM functionality, as std::callonce can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker. Unportable shell construct hardcoded in tests ">&". This has been fixed upstream. LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X. Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt. The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests. Clang I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt. compiler-rt The current compiler-rt tasks can be divided into: upstream sanitizer common code shared with POSIX platforms upstream sanitizer common code shared with Linux and FreeBSD upstream sanitizer common code shared with FreeBSD upstream sanitizer common code specific to NetBSD build, execute and pass tests for sanitizer common code in check-santizer This means that ubsan, asan and the rest of the specific sanitizers wait in queue. All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one. The last point with check-sanitizer unveiled so far two generic bugs on NetBSD: Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls. Honor PTHREADDESTRUCTORITERATIONS in libpthread. These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems. The fixes were introduced by Christos Zoulas and backported to NetBSD-8. Plan for the next milestone I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future. LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0). This work was sponsored by The NetBSD Foundation. The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: http://netbsd.org/donations/#how-to-donate *** DragonFly BSD 4.8.1 released (http://lists.dragonflybsd.org/pipermail/commits/2017-August/626150.html) +Updates by dev: + Antonio Huete Jimenez (1): + libc/gmon: Replace sbrk() with mmap() + Francois Tigeot (3): + drm: bring in Linux compability changes from master + drm/linux: make flushwork() more robust + drm/i915: Update to Linux 4.7.10 + Imre Vadász (4): + drm - Fix hrtimer, don't reset timer->function to NULL in timeout handler. + sound - Delete devfs clone handler for /dev/dsp and /dev/mixer on unload. + ifvtnet - Allocate struct vtnettxheader entries from a queue. + Make sure that cam(4)'s dashutdown handler runs before DEVICESHUTDOWN(). + Matthew Dillon (24): + kernel - MFC b48dd28447fc (sigtramp workaround) + kernel - Fix deadlock in sound system + kernel - Fix broken wakeup in crypto code + kernel - Add KERNPROCSIGTRAMP + gcc - Adjust the unwind code to use the new sigtramp probe sysctl + kernel - Implement NX + kernel - Implement NX (2) + kernel - Implement machdep.pmapnxenable TUNABLE + kernel - Implement NX (3) - cleanup + kernel - Temporarily set the default machdep.pmapnxenable to 0 + param - Change _DragonFlyversion to 400801 + kernel - Fix i915 deadlock + pthreads - Change PTHREADSTACKMIN + libc - Fix bug in rcmdsh() + ppp - Fix minor overflow in protocol search + libtelnet - Fix improper statement construction (not a bug in the binary) + libdevstat - Limit sscanf field, fix redundant condition + openssh - Fix a broken assignment + window - Fix Graphics capability enable test + kernel - Fix event preset + mfiutil - Fix static buffer overflow + mixer - Fix sscanf() overflow + gcore - fix overflow in sscanf + kernel - Fix improper parens + Sascha Wildner (17): + libkvm: Fix char pointer dereference. + Fix some cases where an index was used before its limits check. + Really ensure that our world/kernel are built under POSIX locale ("C"). + zoneinfo: Create a /usr/share/zoneinfo/UTC link. + kernel/cam: Add CAMSCSIITNEXUSLOST (in preparation for virtioscsi(4)). + kernel: Add FreeBSD's virtioscsi(4) driver. + ccdconfig(8): Add missing free(). + libpuffs: Fix two asserts. + kernel/acpi: Untangle the wakecode generation during buildkernel. + kernel/acpica: Better check AcpiOsPredefinedOverride()'s InitVal argument + kernel/acpica: ACPITHREADID is unsigned. + kernel/acpica: Return curthread as thread id from AcpiOsGetThreadId(). + kernel/acpica: Remove no longer needed #include. + kernel/acpi: Call AcpiInitializeSubsystem() before AcpiInitializeTables(). + kernel/urtwn: Add missing braces. + kernel/ieee80211: Add missing braces. + libthreadxu: Fix checking of pthreadbarrierinit()'s count argument. + Sepherosa Ziehau (7): + sound/hda: Sync device ID table with FreeBSD + inet6: Restore mbuf hash after defragmentation. + pf: Normalized, i.e. defragged, packets requiring rehash. + em: Enable MSI by default on devices has PCI advanced features capability. + sched: Change CPU_SETSIZE to signed int, same as FreeBSD/Linux. + usched: Allow process to change self cpu affinity + ix: Fixup TX/RX ring settings for X550, which supports 64/64 TX/RX rings. + zrj (1): + Revert "Always use unix line endings" Porting Unix to the 386: A Practical Approach (http://www.informatica.co.cr/unix-source-code/research/1991/0101.html) The University of California's Berkeley Software Distribution (BSD) has been the catalyst for much of the innovative work done with the UNIX operating system in both the research and commercial sectors. Encompassing over 150 Mbytes (and growing) of cutting-edge operating systems, networking, and applications software, BSD is a fully functional and nonproprietary complete operating systems software distribution (see Figure 1). In fact, every version of UNIX available from every vendor contains at least some Berkeley UNIX code, particularly in the areas of filesystems and networking technologies. However, unless one could pay the high cost of site licenses and equipment, access to this software was simply not within the means of most individual programmers and smaller research groups. The 386BSD project was established in the summer of 1989 for the specific purpose of porting BSD to the Intel 80386 microprocessor platform so that the tools this software offers can be made available to any programmer or research group with a 386 PC. In coordination with the Computer Systems Research Group (CSRG) at the University of California at Berkeley, we successively ported a basic research system to a common AT class machine (see, Figure 2), with the result that approximately 65 percent of all 32-bit systems could immediately make use of this new definition of UNIX. We have been refining and improving this base port ever since. By providing the base 386BSD port to CSRG, our hope is to foster new interest in Berkeley UNIX technology and to speed its acceptance and use worldwide. We hope to see those interested in this technology build on it in both commercial and noncommercial ventures. In this and following articles, we will examine the key aspects of software, strategy, and experience that encompassed a project of this magnitude. We intend to explore the process of the 386BSD port, while learning to effectively exploit features of the 386 architecture for use with an advanced operating system. We also intend to outline some of the tradeoffs in implementation goals which must be periodically reexamined. Finally, we will highlight extensions which remain for future work, perhaps to be done by some of you reading this article today. Note that we are assuming familiarity with UNIX, its concepts and structures, and the basic functions of the 386, so we will not present exhaustive coverage of these areas. In this installment, we discuss the beginning of our project and the initial framework that guided our efforts, in particular, the development of the 386BSD specification. Future articles will address specific topics of interest and actual nonproprietary code fragments used in 386BSD. Among the future areas to be covered are: 386BSD process context switching Executing the first 386BSD process on the PC 386BSD kernel interrupt and exception handling 386BSD INTERNET networking ISA device drivers and system support 386BSD bootstrap process *** X11: How does “the” clipboard work (https://www.uninformativ.de/blog/postings/2017-04-02/0/POSTING-en.html) > If you have used another operating system before you switched to something that runs X11, you will have noticed that there is more than one clipboard: > Sometimes, you can use the mouse to select some text, switch to another window, and then hit the middle mouse button to paste text. > Sometimes, you can select text, then hit some hotkey, e.g. Ctrl+C, switch to another window, hit another hotkey, e.g. Ctrl+V, and paste said text. > Sometimes, you can do both. > Selections as a form of IPC First things first, in X11 land, “clipboards” are called “selections”. Yes, there is more than one selection and they all work independently. In fact, you can use as many selections as you wish. In theory, that is. When using selections, you make different clients communicate with each other. This means that those clients have to agree on which selections to use. You can't just invent your own selection and then expect Firefox to be compatible with it. How are selections identified? There are three “standard” selection names: PRIMARY: The “middle mouse clipboard” SECONDARY: Virtually unused these days CLIPBOARD: The “Ctrl+C clipboard” Program 1: Query selection owners Content type and conversion Program 2: Get clipboard as UTF-8 Program 3: Owning a selection Program 4: Content type TARGETS Handling binary data using xclip Large amounts of data Clipboard managers Summary News Roundup TrueOS Documentation: A great way to give back! (https://www.trueos.org/blog/trueos-documentation-great-way-give-back/) The TrueOS project is always looking for community contribution. Documentation changes are a great way for users to not only make a solid contribution to the project, but learn more about it too! Over the last few months, many users have asked for both simple and detailed instructions on making documentation changes. These are now added to the TrueOS handbook in the Contributing to TrueOS section. If interested in making a small alteration to the TrueOS handbook, here are some instructions for submitting a patch through the GitHub website. These instructions are also applicable to the Lumina and SysAdm handbooks. Lumina documentation is in the the lumina-docs repository, and SysAdm guides are in sysadm-docs. Make a Doc change! A GitHub account is required to submit patches to the TrueOS docs. Open a web browser and sign in to GitHub or make a new account. When making a new account, be sure to use an often checked email address, as all communication regarding patches and pull requests are sent to this address. Navigate to the trueos-docs GitHub repository. Click on the trueos-handbook directory to view all the documentation files. Open the .rst file corresponding to the chapter needing an update. The chapter names are reflected in the title of the .rst files. For example, open install.rst to fix an error spotted in handbook chapter 3: “Install”. This first image shows the trueos-docs repository and the contents of the trueos-handbook directory Open the desired chapter file by clicking its entry in the list. The trueos.rst file is an index file and should be ignored. Begin editing the file by clicking the Pencil icon in the upper right corner above the file's text. The file moves to edit mode, where it is now possible to make changes, as the next image shows. Editing install.rst with GitHub When making a simple change, it is recommended to avoid adjusting the specific formatting elements and instead work within or around them. Once satisfied, scroll to the bottom of the page and write a detailed commit summary of the new changes. Click Propose file change (green button), then Create pull request to submit the changes to the project. GitHub then does an automated merge check. Click Create pull request again to submit the change to the repository. In the final step, a developer or project committer reviews the changes, merging them into the project or asking for more changes as necessary. Learn more about TrueOS documentation To learn more about the underlying structure of TrueOS documentation like the Sphinx Documentation Generator and reStructuredText markup, browse the Advanced Documentation Changes section of the TrueOS handbook. This section also contains instructions for forking the repository and configuring a local clone, build testing, updating the translation files, and other useful information. The Sphinx website is also a valuable resource. libHijack Revival (https://www.soldierx.com/news/Hijack-Revival) Over a decade ago, while standing naked and vulnerable in the comfort of my steaming hot shower, I gathered my thoughts as humans typically attempt to do in the wee hours of the morning. Thoughts of a post-exploitation exercise raced in my mind, the same thoughts that made sleeping the night before difficult. If only I could inject into Apache some code that would allow me to hook into its parsing engine without requiring persistance. Putting a file-backed entry into /proc/pid/maps would tip off the security team to a compromise. The end-goal was to be able to send Apache a special string and have Apache perform a unique action based on the special string. FelineMenace's Binary Protection Schemes whitepaper provided inspiration. Silvio Cesare paved the way into PLT/GOT redirection attacks. Various Phrack articles selflessly contributed to the direction I was to head. Alas, in the aforementioned shower, an epiphany struck me. I jumped as an awkward stereotypical geek does: like an elaborate Elaine Benes dance rehearsal in the air. If I used PTrace, ELF, and the PLT/GOT to my advantage, I could cause the victim application to allocate anonymous memory mappings arbitrarily. In the newly-created memory mapping, I could inject arbitrary code. Since a typical operating system treats debuggers as God-like applications, the memory mapping could be mapped without write access, but as read and execute only. Thus enabling the stealth that I sought. The project took a few years to develop in my spare time. I ended up creating several iterations, taking a rough draft/Proof-of-Concept style code and rewriting it to be more efficient and effective. I had toyed with FreeBSD off-and-on for over a decade by this point, but by-and-large I was still mostly using Linux. FreeBSD gained DTrace and ZFS support, winning me over from the Linux camp. I ported libhijack to FreeBSD, giving it support for both Linux and FreeBSD simultaneously. In 2013, I started work on helping Oliver Pinter with his ASLR implementation, which was originally destined to be upstreamed to FreeBSD. It took a lot of work, and my interest in libhijack faded. As a natural consequence, I handed libhijack over to SoldierX, asking the community to take it and enhance it. Over four years went by without a single commit. The project was essentially abandoned. My little baby was dead. This past week, I wondered if libhijack could even compile on FreeBSD anymore. Given that four years have passed by and major changes have happened in those four years, I thought libhijack would need a major overhaul just to compile, let alone function. Imagine my surprise when libhijack needed only a few fixups to account for changes in FreeBSD's RTLD. Today, I'm announcing the revival of libhijack. No longer is it dead, but very much alive. In order to develop the project faster, I've decided to remove support for Linux, focusing instead on FreeBSD. I've removed hundreds of lines of code over the past few days. Supporting both FreeBSD and Linux meant some code had to be ugly. Now the beautification process has begun. I'm announcing the availability of libhijack 0.7.0 today. The ABI and API should be considered unstable as they may change without notice. Note that HardenedBSD fully mitigates libhijack from working with two security features: setting security.bsd.unprivilegedprocdebug to 0 by default and the implementation of PaX NOEXEC. The security.bsd.unprivilegedprocdebug sysctl node prevents PTrace access for applications the debugger itself did not fork+execve for unprivileged (non-root) users. Privileged users (the root account) can use PTrace to its fullest extent. HardenedBSD's implementation of PaX NOEXEC prevents the creation of memory mappings that are both writable and executable. It also prevents using mprotect to toggle between writable and executable. In libhijack's case, FreeBSD grants libhijack the ability to write to memory mappings that are not marked writable. Debuggers do this to set breakpoints. HardenedBSD behaves differently due to PaX NOEXEC. Each memory mapping has a notion of a maximum protection level. When a memory mapping is created, if the write bit is set, then HardenedBSD drops the execute bit from the maximum protection level. When the execute bit is set at memory mapping creation time, then the write bit is dropped from the maximum protection level. If both the write and execute bits are set, then the execute bit is silently dropped from both the mapping creation request and the maximum protection level. The maximum protection level is always obeyed, even for debuggers. Thus we see that PaX NOEXEC is 100% effective in preventing libhijack from injecting code into a process. Here is a screenshot showing PaX NOEXEC preventing libhijack from injecting shellcode into a newly-created memory mapping. What's next for libhijack? Here's what we have planned, in no particular order: Python bindings Port to arm64 This requires logic for handling machine-dependent code. High priority. Finish anonymous shared object injection. This requires implementing a custom RTLD from within libhijack. More cleanups. Adhere to style(9). libhijack can be found on GitHub @ https://github.com/SoldierX/libhijack *** Contributing to FreeBSD (https://blather.michaelwlucas.com/archives/2988) I've talked to a whole bunch of folks who say things like “I'm a junior programmer. I'm looking for a way to help. I have no specific expertise, but I'm willing to learn.” Today, I present such junior programmers with an opportunity. An opportunity for you to learn skills that will be incredibly valuable to your career, and will simultaneously expand your career opportunities. For decades, FreeBSD has relied on its users for testing. They expect users to install pre-release versions of the OS and exercise them to identify regressions. That's necessary, but it's nowhere near enough. The FreeBSD Testing Project is building an automated test suite for the entire operating system. They have a whole mess of work to do. There's only four people on the team, so each additional person that contributes can have a serious impact. They have tutorials on how to write tests, and sample tests. There's a whole bunch of tests left to be written. You have an almost open field. They need tests for everything from ls(1) to bhyve. (Yes, ls(1) broke at one point in the last few years.) Everything needs testing. Learning to write, submit, and commit small tests is valuable experience for developing the big tests. What's more, learning to write tests for a system means learning the system. Developing tests will transform you into a FreeBSD expert. Once you've demonstrated your competence, worth, and ability to work within the project, other FreeBSD teams will solicit your help and advice. The Project will suck you in. Testing is perhaps the most valuable contribution anyone can make to an open source project. And this door into the FreeBSD Project is standing wide, wide open. OpenBSD Gaming Resource (https://mrsatterly.com/openbsd_games.html) > What isn't there to love about playing video games on your favorite operating system? OpenBSD and video games feels like a natural combination to me. My resource has software lists, links to free games not in ports, lists of nonfree games, and recommendations. The Table of Contents has these high-level items for you: > General Resources > OpenBSD Exclusive > Ports > Network Clients > Browser Games > Game Engines > Multiple Game Engines > Multiple System Emulation > Computer Emulation > Game Console Emulation > Live Media Emulation > Operating System Emulation > Games in Other Software Have fun with these games! *** Beastie Bits Dragonfly introduces kcollect(8) (https://www.dragonflydigest.com/2017/08/07/20061.html) The Faces of Open Source (http://facesofopensource.com/unix/) Edgemesh CEO, Jake Loveless and Joyent CTO, Bryan Cantrill join together for a fireside chat to discuss distributed caching at scale, Docker, Node.js, Mystery Science Theater 3000, and more! (https://www.joyent.com/blog/joyent-edgemesh-cache-me-if-you-can) UFS: Place the information needed to find alternate superblocks to the end of the area reserved for the boot block (https://svnweb.freebsd.org/base?view=revision&revision=322297) Let ‘localhost' be localhost (https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-04) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Morgan - btrfs deprecated (http://dpaste.com/0JEYE1K) Ben - UEFI, GELI, BEADM, and more (http://dpaste.com/2TP90HD) Brad - Hostname Clarification (http://dpaste.com/1MQH1BD) M Rod - BSD Laptop (http://dpaste.com/39C6PGN) Jeremy - Contributing to BSDs (http://dpaste.com/3SVP5SF) ***
We recap our devsummit experiences at BSDCambridge, share why memcmp is more complicated than expected, explore Docker on FreeBSD, and we look at a retro terminal. This episode was brought to you by Headlines BSDCam recap (https://wiki.freebsd.org/DevSummit/201708) The 2017 Cambridge DevSummit took place from 2-4 August 2017. The event took place over three days including a formal dinner at St John's College, and was attended by 55 registered developers and guests. Prior to the start of the conference, we had a doc hacking lounge, the computer lab provided a room where we could meet and try to spend some time on documentation. Sevan walked two interested people through the process of creating a documentation patch and submitting it for the first time. In the process, found ways to improve the documentation on how to write documentation. The event is run "un-conference style" in that we brainstorm the actual session schedule on the first morning, with a focus on interactive topics that reflect the interests and exploit the knowledge of the attendees. The idea is to maximize the amount of discussion and decisions that can be made while we are all in the same room The first morning, we all gather in the slightly too small, and even more slightly under air conditioned FW11 classroom. We go around the room introducing ourselves, and listing a few topics we would be interested in discussing. Eventually the whiteboard is full of topics, with various numbers of ticks beside them to indicate the number of interested people There are breakout rooms of all sizes, so even topics with only a small group of interested folks can get a lot accomplished The most difficult is trying to schedule the sessions, as there is much overlap and people usually want to be in concurrent sessions, or someone's schedule means they won't be available that day, etc. This years working groups: Toolchain (Compilers, Linkers, External Toolchain, Static analysis and sanitizers) Virtualization (bhyve, xen, jails, docker) Transport (TCP) and Network Performance Security and mitigations (W^X, noexec stack, CFI, ASLR, KASLR, Safe Stack, etc) Testing (Status, What to test, How to test, QA for releases) Capsicum (Automation with LLVM etc, Casper, Namespacing, “Services”, capsh) Desktop / WiFi (drm-next, drivers, resume, power, installer, desktop, OOB Experience) Tracing (Blackbox, DTrace, KTR, ptrace, truss, hardware tracing) Packaging and Packaged Base (Sets, Kernels, Ports & flavours, sub-packages, privlib) Architectural Security Features (CPU Features: SGX, PXN/PAN, Pointer Authentication, AMD Memory Encryption, Libcrunch, RISC-V, CheriABI) Architectures and Embedded systems (RISC-V, ARM, ARM64, MIPS(64), SPARC64) Teaching (Audiences, Objectives, Targets, Material, future directions) Provisioning and Management Tools (CfgMgmt tools, Image building, VM/bhyve orchestration, Preconfigured VMs for testing, Wishlist) Storage (ZFS status update, ZFS encryption infrastructure, ZFS Zero Copy / Sendfile, Acceleration of checksums and raidz parity calculations, sesutil, mpsutil) And that wasn't everything. We then had a series of short talklets: Enhancing and replacing mmap() SDIO support eBPF support for FreeBSD Tracing + Virtualization Practical DMA Attack Protection On Thursday night there was a special dinner at St John's College Overall it was a great DevSummit, and I even managed to get some of the work assigned to me finished. Shortly I will commit an update to the boot loader menu that will automatically populate the kernel selection menu with the automatically detected list of installed kernels. The list is also properly refreshed when you switch boot environments. *** Hosts/BSD – for when you need to run your BSD inside a penguin (https://wiki.qemu.org/index.php/Hosts/BSD) This wiki provides details on how to run each of the various BSDs under QEMU The target audience is Linux developers looking to test their apps etc under BSD The wiki is in need of some love, there are some option questions, and it lacks some polish There are instructions on building qemu from source, but it should likely mention the qemu-devel port There should probably also be instructions on using other architectures, like ARM/MIPS etc If you have used QEMU, or would like to spend the time to learn how, please help update this wiki *** memcmp -- more complicated than you might expect (http://trust-in-soft.com/memcmp-requires-pointers-to-fully-valid-buffers/) “A suspicious pattern in open-source software” One bug recently found by John using tis-interpreter on a widely used open-source library involved the comparison of strings with memcmp. The unexpected condition was that memcmp was, in one case, called with a pointer to a buffer shorter than the length passed as third argument, breaking one of the two symmetrical pre-conditions in the function's ACSL contract A reason that may have made this use of memcmp look okay to the developer is that the buffers being passed to it always differed before the end of the buffers were reached. a memcmp implementation based on stopping as soon as a difference is found, would not have caused any out-of-bounds read access The first question raised was whether the pattern memcmp("a", "bc", 3) was problematic according to the letter of the C standard. If it was, the second question was whether the busy maintainer of one of the Open Source packages that make the Internet tick should be bothered with a bug report. I would like to be able to say that memcmp's ACSL contract was the product of careful deliberation, but unfortunately this is not the case: many standard function contracts were written quickly in order to get most of the standard library covered, and have not been tested by time. Anyway, upon proofreading the relevant clause in the C11 standard, my feeling was that the ACSL formalization was, in this particular case, right, and that it was undefined behavior to pass as memcmp argument a buffer that wasn't fully valid, even if the implementation sort-of needs to read the buffer's characters in order for the purpose of finding the first mismatch. The post then goes on to look at the memcmp code in glibc There are two distinct optimizations for long buffers, one that applies when both buffers start at the same offset modulo the word size, memcmpcommonalignment, and one that applies when they don't, memcmpnotcommonalignment. The function memcmpcommonalignment is relatively well-behaved: it reads from the two buffers aligned word by aligned word, and thus reads the entire words that contain differing bytes. If the caller passed buffers that aren't valid after the differing byte, this amounts to reading out of bounds, but this sort of out-of-bounds access is not detected by the typical MMU, which works at the scale of the page. The “notcommon_alignment” case, however, tells a different story. When passed the carefully (mis-)aligned buffers t1 and (char*)t2+1, although these buffers differ in the 8th byte, Glibc's implementation of memcmp reads 8 bytes beyond the end of t1. By making the 16th byte differ instead of the 8th one, it is also possible to make Glibc's implementation of memcmp read 16 bytes beyond the end of t1. In conclusion, yes, some implementations of memcmp will crash when invoked with buffers that aren't valid for the full length, even if they differ early. The circumstances are rare (probably the reason this bug was still there to be found in a library that had already been tested with all the available techniques) but outside the programmer's control. The pattern described in this post should be reported as a bug when found. It is interesting to read the detailed analysis of a bug in such a basic libc feature *** News Roundup Docker on FreeBSD (http://daemon-notes.com/articles/network/docker) There are two approaches to running Docker on FreeBSD. First one was created back in 2015 and it was a native port of Docker engine to FreeBSD. It was an ambitious project but nobody stepped forward to continuously port the never-ending flow of upstream code to FreeBSD. So the port still exists (sysutils/docker-freebsd) but it wasn't updated since 2015 and it is Docker v1 (it is v17 as of 2017). The other approach is to use official way of running Docker on platforms other than Linux. Well, somewhat official as Docker still does not support FreeBSD as a host officially. This is docker-machine tool which in turn will use VirtualBox to run a virtual machine with Linux and Docker engine. docker utility on the host will communicate with the engine inside VB where all the work will be done. This article describes what needs to be done to start using it. Before we begin you need VirtualBox installed. Do not skip adding /boot/loader.conf and /etc/rc.conf lines mentioned on that page. You won't need user inteface or anything, docker-machine will do all the work, just make sure VirtualBox is present and ready to be used. `pkg install docker docker-machine docker-compose' Docker will store its stuff in ~/.docker. You might not want the virtual machine image files to live in your home, in this case just create a symlink: mkdir ~/.docker ln -s /storage/docker ~/.docker/machine docker-machine create --driver virtualbox --virtualbox-memory 2048 --virtualbox-cpu-count 2 --virtualbox-disk-size 102400 --virtualbox-hostonly-cidr "10.2.1.1/24" docker1 Here's the example. We are creating machine named docker1. It is using VirtualBox driver, the vm has 2G of memory, 2 cores and 100G of disk space. docker-machine setups VirtualBox to use host-only network adapter (it will create vboxnet0 interface on the host automatically) and we are instructing it to use 10.2.1.1/24 as the address of this adapter — change it to what suits your needs or omit this flag (default is 192.168.99.1/24). And basically that is all. Check if it is running: docker-machine ls If you do open VirtualBox interface you will find a virtual machine named docker1 running. You can start/stop/whatever your machine using docker-machine utility. Here's how you can connect to the machine: docker utility by default tries to talk to Docker engine running on the same host. However with specific environment variables you can instruct it to talk to other host. docker-machine can export these variables for you. eval docker-machine env docker1 docker run hello-world There was quite a bit of discussion about docker at the FreeBSD developers summit in Cambridge during the first week of August. Two docker developers who had worked on the Mac OS X port, one of whom is an OpenBSD advocate, explained how docker has evolved, and the linux-isms have been abstracted away such that a truly native docker solution for FreeBSD can be built and maintained with a lot less headache than before I look forward to seeing if we can't make that happen *** The POSIX Shell And Utilities (http://shellhaters.org/) The POSIX Shell And Utilities Compiled for The Shell Hater's Handbook *** PostgreSQL – logging to a file (http://dan.langille.org/2017/07/31/postgresql-logging-to-a-file/) These steps were carried out on FreeBSD 11.0 with PostgreSQL 9.6 (two of my favorite tools). I like logging. I like logging PostgreSQL. With logs, you can see what happened. Without, you can only guess. Setting up logging for PostgreSQL involves several parts, each of which must be completed or else I don't get what I want. This is not a criticism of PostgreSQL. It's a feature. I am documenting this because each time I configure a new PostgreSQL instance, it takes me more than one iteration to get it working. The goal: this post lets both you and me get it right the first time. The parts include: + Telling PostgreSQL to log via syslog + Telling FreeBSD to local postgres to /var/log/postgres.log (my preference). + Telling PostgreSQL the things you want logged. + Changes to postgresql.conf The file location varies with the version installed. For PostgreSQL 9.6 on FreeBSD, the file is /var/db/postgres/data96/postgresql.conf (adjust 96 according to the version installed). I made these changes to that file. log_destination = 'syslog' log_min_messages = notice log_min_error_statement = notice log_checkpoints = on log_lock_waits = on log_timezone = 'UTC' By default, PostgreSQL logs to the local0 facility and is controlled by the syslog_facility in postgresql.conf. This will be used in syslog.conf (see the next section of this post). The above mentioned changes require a reload: service postgresql reload Changes to /etc/syslog.conf Now that we have PostgreSQL logging to syslog, we want to tell syslog where to put those messages. I changed this line in /etc/syslog.conf:*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages With .notice pulling in some local0 messages, adding local0.none to the line will free the messages up for later use in the configuration file. Otherwise, the PostgreSQL messages will be in /var/log/messages. The changed line is: `.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local0.none /var/log/messages Then, to get the messages into my preferred location, I added this to the file: local0.* /var/log/postgresql.log` Log file rotation For rotating my log file, I added a new file: /usr/local/etc/newsyslog.conf.d/postgresql96 /var/log/postgresql.log pgsql:wheel 640 7 * $D0 GB /var/db/postgres/data96/postmaster.pid 30 Before restarting syslog, I did this, so the destination file existed. This isn't always/strictly necessary, but because the ownership is not chown root:wheel, I do it to get that part set. touch /var/log/postgresql.log chown pgsql:wheel Restarting syslog: sudo kill -HUP `sudo cat /var/run/syslog.pid ` That's it Now you should see PostgreSQL logging in /var/log/postgresql.log. mandoc-1.14.2 released (http://undeadly.org/cgi?action=article&sid=20170729122350) i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ (http://mandoc.bsd.lv/). ```From: Ingo Schwarze schwarze@usta.de Date: Fri, 28 Jul 2017 20:12:44 +0200 To: discuss@mandoc.bsd.lv Subject: mandoc-1.14.2 released Hi, i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ . All downstream maintainers are encouraged to update their ports and packages from 1.14.1 to 1.14.2. Mandoc 1.14.2 is a feature release introducing: a new -Tmarkdown output mode anchors for deep linking into -Thtml manual pages a superset of the functionality of the former mdoclint(1) utility a new -Wstyle message level with several new messages automatic line breaking inside individual tbl(7) cells a rewrite of the eqn(7) lexer, and some eqn(7) rendering improvements support for many additional low-level roff(7) features and various smaller features and bug fixes. For more details, see: http://mandoc.bsd.lv/NEWS With the improved mandoc features, only twenty-five out of the ten thousand software packages in the OpenBSD ports tree still need groff to format their manual pages. Since the project has been called "mandoc" rather than "mdocml" for several years now, the website, the distribution tarball, and the source extraction directory are now also called "mandoc" rather than "mdocml". The release was tested on the following systems: + OpenBSD-current and OpenBSD-stable + NetBSD-current + illumos + Debian Linux + Void Linux x86_64 glibc and musl + Crux Linux + SunOS 5.11.2, 5.10, and 5.9 As before, catman(8) and the regression suite cannot be used on SunOS 5.10 and SunOS 5.9. A big thanks to everybody who provided patches, bug reports, feature suggestions, advice, and help with testing! Yours, Ingo``` Beastie Bits A good looking terminal emulator which mimics the old cathode display. Available in x11/cool-retro-terminal (https://github.com/Swordfish90/cool-retro-term) Milestone Complete! OpenRC conversion (https://www.trueos.org/blog/milestone-complete-openrc-conversion/) Healthy developer interaction between FreeBSD and IllumOS re: mdb (https://illumos.topicbox.com/groups/developer/discussions/T5eae6079331c4df4) Large Batch of Kernel Errata Patches Released (http://undeadly.org/cgi?action=article&sid=20170804053102) opnsense 17.7 released (https://opnsense.org/opnsense-17-7-released/) Twitter Co-Founder and CEO states “FreeBSD rules them all” (https://twitter.com/jack/status/892605692317650944) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Dominik - Monitoring Software (http://dpaste.com/08971FQ) Darren - Wonderful Awk (http://dpaste.com/0YCS4DN) Andrew - Thanks (http://dpaste.com/0ZREKTV) Jens - Migration Questions (http://dpaste.com/1GVZNWN) ***
In this episode, we clear up the myth about scrub of death, look at Wayland and Weston on FreeBSD, Intel QuickAssist is here, and we check out OpenSMTP on OpenBSD. This episode was brought to you by Headlines Matt Ahrens answers questions about the “Scrub of Death” In working on the breakdown of that ZFS article last week, Matt Ahrens contacted me and provided some answers he has given to questions in the past, allowing me to answer them using HIS exact words. “ZFS has an operation, called SCRUB, that is used to check all data in the pool and recover any data that is incorrect. However, if a bug which make errors on the pool persist (for example, a system with bad non-ecc RAM) then SCRUB can cause damage to a pool instead of recover it. I heard it called the “SCRUB of death” somewhere. Therefore, as far as I understand, using SCRUB without ECC memory is dangerous.” > I don't believe that is accurate. What is the proposed mechanism by which scrub can corrupt a lot of data, with non-ECC memory? > ZFS repairs bad data by writing known good data to the bad location on disk. The checksum of the data has to verify correctly for it to be considered "good". An undetected memory error could change the in-memory checksum or data, causing ZFS to incorrectly think that the data on disk doesn't match the checksum. In that case, ZFS would attempt to repair the data by first re-reading the same offset on disk, and then reading from any other available copies of the data (e.g. mirrors, ditto blocks, or RAIDZ reconstruction). If any of these attempts results in data that matches the checksum, then the data will be written on top of the (supposed) bad data. If the data was actually good, then overwriting it with the same good data doesn't hurt anything. > Let's look at what will happen with 3 types of errors with non-ECC memory: > 1. Rare, random errors (e.g. particle strikes - say, less than one error per GB per second). If ZFS finds data that matches the checksum, then we know that we have the correct data (at least at that point in time, with probability 1-1/2^256). If there are a lot of memory errors happening at a high rate, or if the in-memory checksum was corrupt, then ZFS won't be able to find a good copy of the data , so it won't do a repair write. It's possible that the correctly-checksummed data is later corrupted in memory, before the repair write. However, the window of vulnerability is very very small - on the order of milliseconds between when the checksum is verified, and when the write to disk completes. It is implausible that this tiny window of memory vulnerability would be hit repeatedly. > 2. Memory that pretty much never does the right thing. (e.g. huge rate of particle strikes, all memory always reads 0, etc). In this case, critical parts of kernel memory (e.g. instructions) will be immediately corrupted, causing the system to panic and not be able to boot again. > 3. One or a few memory locations have "stuck bits", which always read 0 (or always read 1). This is the scenario discussed in the message which (I believe) originally started the "Scrub of Death" myth: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ This assumes that we read in some data from disk to a memory location with a stuck bit, "correct" that same bad memory location by overwriting the memory with the correct data, and then we write the bad memory location to disk. However, ZFS doesn't do that. (It seems the author thinks that ZFS uses parity, which it only does when using RAID-Z. Even with RAID-Z, we also verify the checksum, and we don't overwrite the bad memory location.) > Here's what ZFS will actually do in this scenario: If ZFS reads data from disk into a memory location with a stuck bit, it will detect a checksum mismatch and try to find a good copy of the data to repair the "bad" disk. ZFS will allocate a new, different memory location to read a 2nd copy of the data, e.g. from the other side of a mirror (this happens near the end of dslscanscrub_cb()). If the new memory location also has a stuck bit, then its checksum will also fail, so we won't use it to repair the "bad" disk. If the checksum of the 2nd copy of the data is correct, then we will write it to the "bad" disk. This write is unnecessary, because the "bad" disk is not really bad, but it is overwriting the good data with the same good data. > I believe that this misunderstanding stems from the idea that ZFS fixes bad data by overwriting it in place with good data. In reality, ZFS overwrites the location on disk, using a different memory location for each read from disk. The "Scrub of Death" myth assumes that ZFS overwrites the location in memory, which it doesn't do. > In summary, there's no plausible scenario where ZFS would amplify a small number of memory errors, causing a "scrub of death". Additionally, compared to other filesystems, ZFS checksums provide some additional protection against bad memory. “Is it true that ZFS verifies the checksum of every block on every read from disk?” > Yes “And if that block is incorrect, that ZFS will repair it?” > Yes “If yes, is it possible set options or flag for change that behavior? For example, I would like for ZFS to verify checksums during any read, but not change anything and only report about issues if it appears. Is it possible?” > There isn't any built-in flag for doing that. It wouldn't be hard to add one though. If you just wanted to verify data, without attempting to correct it, you could read or scan the data with the pool was imported read-only “If using a mirror, when a file is read, is it fully read and verified from both sides of the mirror?” > No, for performance purposes, each block is read from only one side of the mirror (assuming there is no checksum error). “What is the difference between a scrub and copying every file to /dev/null?” > That won't check all copies of the file (e.g. it won't check both sides of the mirror). *** Wayland, and Weston, and FreeBSD - Oh My! (https://euroquis.nl/bobulate/?p=1617) KDE's CI system for FreeBSD (that is, what upstream runs to continuously test KDE git code on the FreeBSD platform) is missing some bits and failing some tests because of Wayland. Or rather, because FreeBSD now has Wayland, but not Qt5-Wayland, and no Weston either (the reference implementation of a Wayland compositor). Today I went hunting for the bits and pieces needed to make that happen. Fortunately, all the heavy lifting has already been done: there is a Weston port prepared and there was a Qt5-Wayland port well-hidden in the Area51 plasma5/ branch. I have taken the liberty of pulling them into the Area51 repository as branch qtwayland. That way we can nudge Weston forward, and/or push Qt5-Wayland in separately. Nicest from a testing perspective is probably doing both at the same time. I picked a random “Hello World” Wayland tutorial and also built a minimal Qt program (using QMessageBox::question, my favorite function to hate right now, because of its i18n characteristics). Then, setting XDGRUNTIMEDIR to /tmp/xdg, I could start Weston (as an X11 client), wayland-hello (as a Wayland client, displaying in Weston) and qt-hello (as either an X11 client, or as a Wayland client). So this gives users of Area51 (while shuffling branches, granted) a modern desktop and modern display capabilities. Oh my! It will take a few days for this to trickle up and/or down so that the CI can benefit and we can make sure that KWin's tests all work on FreeBSD, but it's another good step towards tight CI and another small step towards KDE Plasma 5 on the desktop on FreeBSD. pkgsrcCon 2017 report (https://blog.netbsd.org/tnf/entry/pkgsrccon_2017_report) This years pkgsrcCon returned to London once again. It was last held in London back in 2014. The 2014 con was the first pkgsrcCon I attended, I had been working on Darwin/PowerPC fixes for some months and presented on the progress I'd made with a 12" G4 PowerBook. I took away a G4 Mac Mini that day to help spare the PowerBook for use and dedicate a machine for build and testing. The offer of PowerPC hardware donations was repeated at this years con, thanks to jperkin@ who showed up with a backpack full of Mac Minis (more on that later). Since 2014 we have held cons in Berlin (2015) & Krakow (2016). In Krakow we had talks about a wide range of projects over 2 days, from Haiku Ports to Common Lisp to midipix (building native PE binaries for Windows) and back to the BSDs. I was very pleased to continue the theme of a diverse program this year. Aside from pkgsrc and NetBSD, we had talks about FreeBSD, OpenBSD, Slackware Linux, and Plan 9. Things began with a pub gathering on the Friday for the pre-con social, we hung out and chatted till almost midnight on a wide range of topics, such as supporting a system using NFS on MS-DOS, the origins of pdksh, corporate IT, culture and many other topics. On parting I was asked about the starting time on Saturday as there was some conflicting information. I learnt that the registration email had stated a later start than I had scheduled for & advertised on the website, by 30 minutes. Lesson learnt: register for your own event! Not a problem, I still needed to setup a webpage for the live video stream, I could do both when I got back. With some trimming here and there I had a new schedule, I posted that to the pkgsrcCon website and moved to trying to setup a basic web page which contained a snippet of javascript to play a live video stream from Scale Engine. 2+ hours later, it was pointed out that the XSS protection headers on pkgsrc.org breaks the functionality. Thanks to jmcneill@ for debugging and providing a working page. Saturday started off with Giovanni Bechis speaking about pledge in OpenBSD and adding support to various packages in their ports tree, alnsn@ then spoke about installing packages from a repo hosted on the Tor network. After a quick coffee break we were back to hear Charles Forsyth speak about how Plan 9 and Inferno dealt with portability, building software and the problem which are avoided by the environment there. This was followed by a very energetic rant by David Spencer from the Slackbuilds project on packaging 3rd party software. Slackbuilds is a packaging system for Slackware Linux, which was inspired by FreeBSD ports. For the first slot after lunch, agc@ gave a talk on the early history of pkgsrc followed by Thomas Merkel on using vagrant to test pkgsrc changes with ease, locally, using vagrant. khorben@ covered his work on adding security to pkgsrc and bsiegert@ covered the benefits of performing our bulk builds in the cloud and the challenges we currently face. My talk was about some topics and ideas which had inspired me or caught my attention, and how it could maybe apply to my work.The title of the talk was taken from the name of Andrew Weatherall's Saint Etienne remix, possibly referring to two different styles of track (dub & vocal) merged into one or something else. I meant it in terms of applicability of thoughts and ideas. After me, agc@ gave a second talk on the evolution of the Netflix Open Connect appliance which runs FreeBSD and Vsevolod Stakhov wrapped up the day with a talk about the technical implementation details of the successor to pkgtools in FreeBSD, called pkg, and how it could be of benefit for pkgsrc. For day 2 we gathered for a hack day at the London Hack Space. I had burn't some some CD of the most recent macppc builds of NetBSD 8.0BETA and -current to install and upgrade Mac Minis. I setup the donated G4 minis for everyone in a dual-boot configuration and moved on to taking apart my MacBook Air to inspect the wifi adapter as I wanted to replace it with something which works on FreeBSD. It was not clear from the ifixit teardown photos of cards size, it seemed like a normal mini-PCIe card but it turned out to be far smaller. Thomas had also had the same card in his and we are not alone. Thomas has started putting together a driver for the Broadcom card, the project is still in its early days and lacks support for encrypted networks but hopefully it will appear on review.freebsd.org in the future. weidi@ worked on fixing SunOS bugs in various packages and later in the night we setup a NetBSD/macppc bulk build environment together on his Mac Mini. Thomas setup an OpenGrock instance to index the source code of all the software available for packaging in pkgsrc. This helps make the evaluation of changes easier and the scope of impact a little quicker without having to run through a potentially lengthy bulk build with a change in mind to realise the impact. bsiegert@ cleared his ticket and email backlog for pkgsrc and alnsn@ got NetBSD/evbmips64-eb booting on his EdgeRouter Lite. On Monday we reconvened at the Hack Space again and worked some more. I started putting together the talks page with the details from Saturday and the the slides which I had received, in preparation for the videos which would come later in the week. By 3pm pkgsrcCon was over. I was pretty exhausted but really pleased to have had a few days of techie fun. Many thanks to The NetBSD Foundation for purchasing a camera to use for streaming the event and a speedy response all round by the board. The Open Source Specialist Group at BCS, The Chartered Institute for IT and the London Hack Space for hosting us. Scale Engine for providing streaming facility. weidi@ for hosting the recorded videos. Allan Jude for pointers, Jared McNeill for debugging, NYCBUG and Patrick McEvoy for tips on streaming, the attendees and speakers. This year we had speakers from USA, Italy, Germany and London E2. Looking forward to pkgsrcCon 2018! The videos and slides are available here (http://www.pkgsrc.org/pkgsrcCon/2017/talks.html) and the Internet Archive (http://archive.org/details/pkgsrcCon-2017). News Roundup QuickAssist Driver for FreeBSD is here and pfSense Support Coming (https://www.servethehome.com/quickassist-driver-freebsd-pfsupport-coming/) This week we have something that STH readers will be excited about. Before I started writing for STH, I was a reader and had been longing for QuickAssist support ever since STH's first Rangeley article over three and a half years ago. It was clear from the get-go that Rangeley was going to be the preeminent firewall appliance platform of its day. The scope of products that were impacted by the Intel Atom C2000 series bug showed us it was indeed. For my personal firewalls, I use pfSense on that Rangeley platform so I have been waiting to use QuickAssist with my hardware for almost an entire product generation. + New Hardware and QuickAssist Incoming to pfSense (Finally) pfSense (and a few other firewalls) are based on FreeBSD. FreeBSD tends to lag driver support behind mainstream Linux but it is popular for embedded security appliances. While STH is the only site to have done QuickAssist benchmarks for OpenSSL and IPSec VPNs pre-Skylake, we expect more platforms to use it now that the new Intel Xeon Scalable Processor Family is out. With the Xeon Scalable platforms, the “Lewisburg” PCH has QuickAssist options of up to 100Gbps, or 2.5x faster than the previous generation add-in cards we tested (40Gbps.) We now have more and better hardware for QAT, but we were still devoid of a viable FreeBSD QAT driver from Intel. That has changed. Our Intel Xeon Scalable Processor Family (Skylake-SP) Launch Coverage Central has been the focus of the STH team's attention this week. There was another important update from Intel that got buried, a publicly available Intel QuickAssist driver for FreeBSD. You can find the driver on 01.org here dated July 12, 2017. Drivers are great, but we still need support to be enabled in the OS and at the application layer. Patrick forwarded me this tweet from Jim Thompson (lead at Netgate the company behind pfSense): The Netgate team has been a key company pushing QuickAssist appliances in the market, usually based on Linux. To see that QAT is coming to FreeBSD and that they were working to integrate into “pfSense soon” is more than welcome. For STH readers, get ready. It appears to be actually and finally happening. QuickAssist on FreeBSD and pfSense OpenBSD on the Huawei MateBook X (https://jcs.org/2017/07/14/matebook) The Huawei MateBook X is a high-quality 13" ultra-thin laptop with a fanless Core i5 processor. It is obviously biting the design of the Apple 12" MacBook, but it does have some notable improvements such as a slightly larger screen, a more usable keyboard with adequate key travel, and 2 USB-C ports. It also uses more standard PC components than the MacBook, such as a PS/2-connected keyboard, removable m.2 WiFi card, etc., so its OpenBSD compatibility is quite good. In contrast to the Xiaomi Mi Air, the MateBook is actually sold (2) in the US and comes with a full warranty and much higher build quality (though at twice the price). It is offered in the US in a "space gray" color for the Core i5 model and a gold color for the Core i7. The fanless Core i5 processor feels snappy and doesn't get warm during normal usage on OpenBSD. Doing a make -j4 build at full CPU speed does cause the laptop to get warm, though the palmrest maintains a usable temperature. The chassis is all aluminum and has excellent rigidity in the keyboard area. The 13.0" 2160x1440 glossy IPS "Gorilla glass" screen has a very small bezel and its hinge is properly weighted to allow opening the lid with one hand. There is no wobble in the screen when open, even when jostling the desk that the laptop sits on. It has a reported brightness of 350 nits. I did not experience any of the UEFI boot variable problems that I did with the Xiaomi, and the MateBook booted quickly into OpenBSD after re-initializing the GPT table during installation. OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot (https://blog.cagedmonster.net/opensmtpd-under-openbsd-with-ssl-virtualusers-dovecot/) During the 2013 AsiaBSDCon, the team of OpenBSD presented its mail solution named OpenSMTPD. Developed by the OpenBSD team, we find the so much appreciated philosophy of its developers : security, simplicity / clarity and advanced features. Basic configuration : OpenSMTPD is installed by default, we can immediately start with a simple configuration. > We listen on our interfaces, we specify the path of our aliases file so we can manage redirections. > Mails will be delivered for the domain cagedmonster.net to mbox (the local users mailbox), same for the aliases. > Finally, we accept to relay local mails exclusively. > We can now enable smtpd at system startup and start the daemon. Advanced configuration including TLS : You can use SSL with : A self-signed certificate (which will not be trusted) or a certificate generated by a trusted authority. LetsEncrypt uses Certbot to generated your certificate. You can check this page for further informations. Let's focus on the first. Generation of the certificate : We fix the permissions : We edit the config file : > We have a mail server with SSL, it's time to configure our IMAP server, Dovecot, and manage the creation of virtual users. Dovecot setup, and creation of Virtual Users : We will use the package system of OpenBSD, so please check the configuration of your /etc/pkg.conf file. Enable the service at system startup : Setup the Virtual Users structure : Adding the passwd table for smtpd : Modification of the OpenSMTPD configuration : We declare the files used for our Virtual Accounts, we include SSL, and we configure mails delivery via the Dovecot lmtp socket. We'll create our user lina@cagedmonster.net and set its password. Configure SSL Configure dovecot.conf Configure mail.con Configure login.conf : Make sure that the value of openfiles-cur in /etc/login.conf is equal or superior of 1000 ! Starting Dovecot *** OpenSMTPD and Dovecot under OpenBSD with MySQL support and SPAMD (https://blog.cagedmonster.net/opensmtpd-and-dovecot-under-openbsd-with-mysql-support-and-spamd/) This article is the continuation of my previous tutorial OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot. We'll use the same configuration and add some features so we can : Use our domains, aliases, virtual users with a MySQL database (MariaDB under OpenBSD). Deploy SPAMD with OpenSMTPD for a strong antispam solution. + Setup of the MySQL support for OpenSMTPD & Dovecot + We create our SQL database named « smtpd » + We create our SQL user « opensmtpd » we give him the privileges on our SQL database and we set its password + We create the structure of our SQL database + We generate our password with Blowfish (remember it's OpenBSD !) for our users + We create our tables and we include our datas + We push everything to our database + Time to configure OpenSMTPD + We create our mysql.conf file and configure it + Configuration of Dovecot.conf + Configuration of auth-sql.conf.ext + Configuration of dovecot-sql.conf.ext + Restart our services OpenSMTPD & SPAMD : SPAMD is a service simulating a fake SMTP server and relying on strict compliance with RFC to determine whether the server delivering a mail is a spammer or not. + Configuration of SPAMD : + Enable SPAMD & SPAMLOGD at system startup : + Configuration of SPAMD flags + Configuration of PacketFilter + Configuration of SPAMD + Start SPAMD & SPAMLOGD Running a TOR relay on FreeBSD (https://networkingbsdblog.wordpress.com/2017/07/14/freebsd-tor-relay-using-priveledge-seperation/) There are 2 main steps to getting a TOR relay working on FreeBSD: Installing and configuring Tor Using an edge router to do port translation In my case I wanted TOR to run it's services on ports 80 and 443 but any port under 1024 requires root access in UNIX systems. +So I used port mapping on my router to map the ports. +Begin by installing TOR and ARM from: /usr/ports/security/tor/ /usr/ports/security/arm/ Arm is the Anonymizing Relay Monitor: https://www.torproject.org/projects/arm.html.en It provides useful monitoring graph and can be used to configure the torrc file. Next step edit the torrc file (see Blog article for the edit) It is handy to add the following lines to /etc/services so you can more easily modify your pf configuration. torproxy 9050/tcp #torsocks torOR 9090/tcp #torOR torDIR 9099/tcp #torDIR To allow TOR services my pf.conf has the following lines: # interfaces lan_if=”re0″ wifi_if=”wlan0″ interfaces=”{wlan0,re0}” tcp_services = “{ ssh torproxy torOR torDIR }” # options set block-policy drop set loginterface $lan_if # pass on lo set skip on lo scrub in on $lan_if all fragment reassemble # NAT nat on $lan_if from $wifi_if:network to !($lan_if) -> ($lan_if) block all antispoof for $interfaces #In NAT pass in log on $wifi_if inet pass out all keep state #ICMP pass out log inet proto icmp from any to any keep state pass in log quick inet proto icmp from any to any keep state #SSH pass in inet proto tcp to $lan_if port ssh pass in inet proto tcp to $wifi_if port ssh #TCP Services on Server pass in inet proto tcp to $interfaces port $tcp_services keep state The finally part is mapping the ports as follows: TOR directory port: LANIP:9099 —> WANIP:80 TOR router port: LANIP:9090 —-> WANIP:443 Now enable TOR: $ sudo echo “tor_enable=YES” >> /etc/rc.conf Start TOR: $ sudo service tor start *** Beastie Bits OpenBSD as a “Desktop” (Laptop) (http://unixseclab.com/index.php/2017/06/12/openbsd-as-a-desktop-laptop/) Sascha Wildner has updated ACPICA in DragonFly to Intel's version 20170629 (http://lists.dragonflybsd.org/pipermail/commits/2017-July/625997.html) Dport, Rust, and updates for DragonFlyBSD (https://www.dragonflydigest.com/2017/07/18/19991.html) OPNsense 17.7 RC1 released (https://opnsense.org/opnsense-17-7-rc1/) Unix's mysterious && and || (http://www.networkworld.com/article/3205148/linux/unix-s-mysterious-andand-and.html#tk.rss_unixasasecondlanguage) The Commute Deck : A Homebrew Unix terminal for tight places (http://boingboing.net/2017/06/16/cyberspace-is-everting.html) FreeBSD 11.1-RC3 now available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087407.html) Installing DragonFlyBSD with ORCA when you're totally blind (http://lists.dragonflybsd.org/pipermail/users/2017-July/313528.html) Who says FreeBSD can't look good (http://imgur.com/gallery/dc1pu) Pratik Vyas adds the ability to do paused VM migrations for VMM (http://undeadly.org/cgi?action=article&sid=20170716160129) Feedback/Questions Hrvoje - OpenBSD MP Networking (http://dpaste.com/0EXV173#wrap) Goran - debuggers (http://dpaste.com/1N853NG#wrap) Abhinav - man-k (http://dpaste.com/1JXQY5E#wrap) Liam - university setup (http://dpaste.com/01ERMEQ#wrap)
This week on BSD Now, we clear up some ZFS FUD, show you how to write a NetBSD kernel module, and cover DragonflyBSD on the desktop. This episode was brought to you by Headlines ZFS is the best file system (for now) (http://blog.fosketts.net/2017/07/10/zfs-best-filesystem-now/) In my ongoing effort to fight misinformation and FUD about ZFS, I would like to go through this post in detail and share my thoughts on the current state and future of OpenZFS. The post starts with: ZFS should have been great, but I kind of hate it: ZFS seems to be trapped in the past, before it was sidelined it as the cool storage project of choice; it's inflexible; it lacks modern flash integration; and it's not directly supported by most operating systems. But I put all my valuable data on ZFS because it simply offers the best level of data protection in a small office/home office (SOHO) environment. Here's why. When ZFS first appeared in 2005, it was absolutely with the times, but it's remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10 The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. The lawsuit was resolved, and it didn't really impede ZFS. Some say it is the reason that Apple didn't go with ZFS, but there are other theories too. By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. Yes, Oracle taking over Sun and closing the source for ZFS definitely seemed like a setback at the time, but the OpenZFS project was started and active development has continued as an ever increasing pace. As of today, more than half of the code in OpenZFS has been written since the fork from the last open version of Oracle ZFS. the CDDL license Sun applied to the ZFS code was https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ (judged incompatible) with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world's server operating system. That hasn't stopped the ZFS-on-Linux project, or Ubuntu… Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and http://blog.fosketts.net/2008/09/15/greenbytes-embraces-extends-zfs/ (GreenBytes) helped push ZFS forward in the enterprise, but Oracle's lackluster commitment to Sun in the datacenter started having an impact. Lots of companies have adopted OpenZFS for their products. Before OpenZFS, there were very few non-Sun appliances that used ZFS, now there are plenty. OpenZFS Wiki: Companies with products based on OpenZFS (http://open-zfs.org/wiki/Companies) OpenZFS remains little-changed from what we had a decade ago. Other than the fact that half of the current code did not exist a decade ago… Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. This is one of the weaker points in ZFS. As it turns out, the demand for deduplication is actually not that strong. Most of the win can be had with transparent compression. However, there are a number of suggested designs to work around the dedup problems: Dedup Ceiling: Set a limit on the side of the DDT and just stop deduping new unique blocks when this limit is reached. Allocation Classes: A feature being developed by Intel for a supercomputer, will allow different types of data to be classified, and dedicated vdevs (or even metaslabs within a vdev), to be dedicated to that class of data. This could be extended to having the DDT live on a fast device like an PCIe NVMe, combined with the Dedup Ceiling when the device is full. DDT Pruning: Matt Ahrens described a design where items in the DDT with only a single reference, would be expired in an LRU type fashion, to allow newer blocks to live in the DDT in hopes that they would end up with more than a single reference. This doesn't cause bookkeeping problems since when a block is about to be freed, if it is NOT listed in the DDT, ZFS knows it was never deduplicated, so the current block must be the only reference, and it can safely be freed. This provides a best case scenario compared to Dedup Ceiling, since blocks that will deduplicate well, are likely to be written relatively close together, whereas the chance to a dedup match on a very old block is much lower. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today's prices. As we talked about a few weeks ago, ECC is best, but it is not required. If you want your server to stay up for a long time, to be highly available, you'll put ECC in it. Don't let a lack of ECC stop you from using ZFS, you are just putting your data at more risk. The scrub of death is a myth. ZFS does not ‘require' lots of ram. Your NAS will work happily with 8 GB instead of 32 GB of RAM. Its cache hit ratio will be much lower, so performance will be worse. It won't be able to buffer as many writes, so performance will be worse. Copy-on-Write has some drawbacks, data tends to get scattered and fragmented across the drives when it is written gradually. The ARC (RAM Cache) lessens the pain of this, and allows ZFS to batch incoming writes up into nice contiguous writes. ZFS purposely alternates between reading and writing, since both are faster when the other is not happening. So writes are batched up until there is too much dirty data, or the timeout expires. Then reads are held off while the bulk linear write finishes as quickly as possible, and reads are resumed. Obviously all of this works better and more efficiently in larger batches, which you can do if you have more RAM. ZFS can be tuned to use less RAM, and if you do not have a lot of RAM, or you have a lot of other demand on your RAM, you should do that tuning. And ZFS never really adapted to today's world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It's laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it's everywhere in performance PC's. Make up your mind, is 32GB of ram too expensive or not… the L2ARC exists specifically for the case where it is not possible to just install more RAM. Be it because there are no more slots, of limits of the processor, or limits of your budget. The SLOG is optional, but it never needs to be very big. A number of GBs of SLC flash is all you need, it is only holding writes that have not been flushed to the regular storage devices yet. The reason the documentation talks about SLC specifically is because your SLOG needs a very high write endurance, something never the newest NVMe devices cannot yet provide. Of course you can use NVMe devices with ZFS, lots of people do. All flash ZFS arrays are for sale right now. Other than maybe a little tuning of the device queue depths, ZFS just works and there is nothing to think about. However, to say there is nothing happening in this space is woefully inaccurate. The previously mentioned allocation classes code can be used to allocate metadata (4 KB blocks) on SSD or NVMe, while allocating bulk storage data (up to 16 MB blocks) on spinning disks. Extended a bit beyond what Intel is building for their super computer, this will basically create hybrid storage for ZFS. With the metaslab classes feature, it will even be possible to mix classes on the same device, grouping small allocations and large allocations in different areas, decreasing fragmentation. Then there's the question of flexibility, or lack thereof. Once you build a ZFS volume, it's pretty much fixed for life. There are only three ways to expand a storage pool: Replace each and every drive in the pool with a larger one (which is great but limiting and expensive) It depends on your pool layout. If you design with this in mind using ZFS Mirrors, it can be quite useful Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff) The unbalanced LUNs performance issues were sorted out in 2013-2016. 2014: OpenZFS Allocation Performance (http://open-zfs.org/w/images/3/31/Performance-George_Wilson.pdf) 2016: OpenZFS space allocation: doubling performance on large and fragmented pools (http://www.bsdcan.org/2016/schedule/events/710.en.html) These also mostly solved the performance issues when a pool gets full, you can run a lot closer to the edge now Build a new pool and “zfs send” your datasets to it (which is what I do, even though it's kind of tricky) This is one way to do it, yes. There is another way coming, but I can't talk about it just yet. Look for big news later this year. Apart from option 3 above, you can't shrink a ZFS pool. Device removal is arriving now. It will not work for RAIDZ*, but for Mirrors and Stripes you will be able to remove a device. I've probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it's startlingly limiting and out of touch with the present solid-state-dominated storage world. I don't feel like ZFS is out of touch with solid state. Lots of people are running SSD only pools. I will admit the tiered storage options in ZFS are a bit limited still, but there is a lot of work being done to overcome this. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It's going to be a long time before I trust anything other than ZFS! + I agree. + ZFS has a great track record of doing its most important job, keeping your data safe. + Work is ongoing to make ZFS more performance, and more flexible. The import thing is that this work is never allowed to compromise job #1, keeping your data safe. + Hybrid/tiered storage features, re-RAID-ing, are coming + There is a lot going on with OpenZFS, check out the notes from the last two OpenZFS Developer Summits just to get an idea of what some of those things are: 2015 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2015) & 2016 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2016) Some highlights: Compressed ARC Compressed send/recv ABD (arc buf scatter/gather) ZFS Native Encryption (scrub/resilver, send/recv, etc without encryption keys loaded) Channel Programs (do many administrative operations as one atomic transaction) Device Removal Redacted send/recv ZStandard Compression TRIM Support (FreeBSD has its own, but this will be more performant and universal) Faster Scrub/Resilver (https://youtu.be/SZFwv8BdBj4) Declustered RAID (https://youtu.be/MxKohtFSB4M) Allocation Classes (https://youtu.be/28fKiTWb2oM) Multi-mount protection (for Active/Passive failover) Zpool Checkpoint (undo almost anything) Even more Improved Allocator Performance vdev spacemap log ZIL performance improvements (w/ or w/o SLOG) Persistent L2ARC What I don't think the author of this article understands is how far behind every other filesystem is. 100s of Engineer years have gone into OpenZFS, and the pace is accelerating. I don't see how BtrFS can ever catch up, without a huge cash infusion. Writing a NetBSD kernel module (https://saurvs.github.io/post/writing-netbsd-kern-mod/) Kernel modules are object files used to extend an operating system's kernel functionality at run time. In this post, we'll look at implementing a simple character device driver as a kernel module in NetBSD. Once it is loaded, userspace processes will be able to write an arbitrary byte string to the device, and on every successive read expect a cryptographically-secure pseudorandom permutation of the original byte string. You will need the NetBSD Source Code. This doc (https://www.netbsd.org/docs/guide/en/chap-fetch.html) will explain how you can get it. The article gives an easy line by line walkthrough which is easy to follow and understand. The driver implements the bare minimum: open, close, read, and write, plus the module initialization function It explains the differences in how memory is allocated and freed in the kernel It also describes the process of using UIO to copy data back and forth between userspace and the kernel Create a Makefile, and compile the kernel module Then, create a simple userspace program to use the character device that the kernel module creates All the code is available here (https://github.com/saurvs/rperm-netbsd) *** DragonFlyBSD Desktop! (https://functionallyparanoid.com/2017/07/11/dragonflybsd-desktop/) If you read my last post (https://functionallyparanoid.com/2017/06/30/boot-all-the-things/), you know that I set up a machine (Thinkpad x230) with UEFI and four operating systems on it. One, I had no experience with – DragonFlyBSD (other than using Matthew Dillon's C compiler for the Amiga back in the day!) and so it was uncharted territory for me. After getting the install working, I started playing around inside of DragonFlyBSD and discovered to my delight that it was a great operating system with some really unique features – all with that BSD commitment to good documentation and a solid coupling of kernel and userland that doesn't exist (by design) in Linux. So my goal for my DragonFlyBSD desktop experience was to be as BSD as I possibly could. Given that (and since I'm the maintainer of the port on OpenBSD ), I went with Lumina as the desktop environment and XDM as the graphical login manager. I have to confess that I really like the xfce terminal application so I wanted to make sure I had that as well. Toss in Firefox, libreOffice and ownCloud sync client and I'm good to go! OK. So where to start. First, we need to get WiFi and wired networking happening for the console at login. To do that, I added the following to /etc/rc.conf: wlans_iwn0=”wlan0″ ifconfig_wlan0=”WPA DHCP” ifconfig_em0=”DHCP” I then edited /etc/wpa_supplicant.conf to put in the details of my WiFi network: network={ ssid=”MY-NETWORK-NAME” psk=”my-super-secret-password” } A quick reboot showed that both wired and wireless networking were functional and automatically were assigned IP addresses via DHCP. Next up is to try getting into X with whatever DragonFlyBSD uses for its default window manager. A straight up “startx” met with, shall we say, less than stellar results. Therefore, I used the following command to generate a simple /etc/X11/xorg.conf file: # Xorg -configure # cp /root/xorg.conf.new /etc/X11/xorg.conf With that file in place, I could get into the default window manager, but I had no mouse. After some searching and pinging folks on the mailing list, I was able to figure out what I needed to do. I added the following to my /etc/rc.conf file: moused_enable=”YES” moused_type=”auto” moused_port=”/dev/psm0″ I rebooted (I'm sure there is an easier way to get the changes but I don't know it… yet) and was able to get into a basic X session and have a functional mouse. Next up, installing and configuring Lumina! To do that, I went through the incredibly torturous process of installing Lumina: # pkg install lumina Wow! That was really, really hard. I might need to pause here to catch my breath.
In which we interview a unicorn, FreeNAS 11.0 is out, show you how to run Nextcloud in a FreeBSD jail, and talk about the connection between oil changes and software patches. This episode was brought to you by Headlines FreeNAS 11.0 is Now Here (http://www.freenas.org/blog/freenas-11-0/) The FreeNAS blog informs us: After several FreeNAS Release Candidates, FreeNAS 11.0 was released today. This version brings new virtualization and object storage features to the World's Most Popular Open Source Storage Operating System. FreeNAS 11.0 adds bhyve virtual machines to its popular SAN/NAS, jails, and plugins, letting you use host web-scale VMs on your FreeNAS box. It also gives users S3-compatible object storage services, which turns your FreeNAS box into an S3-compatible server, letting you avoid reliance on the cloud. FreeNAS 11.0 also introduces the beta version of a new administration GUI. The new GUI is based on the popular Angular framework and the FreeNAS team expects the GUI to be themeable and feature complete by 11.1. The new GUI follows the same flow as the existing GUI, but looks better. For now, the FreeNAS team has released it in beta form to get input from the FreeNAS community. The new GUI, as well as the classic GUI, are selectable from the login screen. Also new in FreeNAS 11 is an Alert Service page which configures the system to send critical alerts from FreeNAS to other applications and services such as Slack, PagerDuty, AWS, Hipchat, InfluxDB, Mattermost, OpsGenie, and VictorOps. FreeNAS 11.0 has an improved Services menu that adds the ability to manage which services and applications are started at boot. The FreeNAS community is large and vibrant. We invite you to join us on the FreeNAS forum (https://forums.freenas.org/index.php) and the #freenas IRC channel on Freenode. To download FreeNAS and sign-up for the FreeNAS Newsletter, visit freenas.org/download (http://www.freenas.org/download/). Building an IPsec Gateway With OpenBSD (https://www.exoscale.ch/syslog/2017/06/26/building-an-ipsec-gateway-with-openbsd/) Pierre-Yves Ritschard wrote the following blog article: With private networks just released on Exoscale, there are now more options to implement secure access to Exoscale cloud infrastructure. While we still recommend the bastion approach, as detailed in this article (https://www.exoscale.ch/syslog/2016/01/15/secure-your-cloud-computing-architecture-with-a-bastion/), there are applications or systems which do not lend themselves well to working this way. In these cases, the next best thing is building IPsec gateways. IPsec is a protocol which works directly at layer 3. It uses its configuration to determine which network flows should be sent encrypted on the wire. Once IPsec is correctly configured, selected network flows are transparently encrypted and applications do not need to modify anything to benefit from secured traffic. In addition to encryption, IPSec also authenticates the end points, so you can be sure you are exchanging packets with a trusted host For the purposes of this article we will work under the following assumptions: We want a host to network setup, providing access to cloud-hosted infrastructure from a desktop environment. Only stock tooling should be used on desktop environment, no additional VPN client should be needed. In this case, to ensure no additional software is needed on the client, we will configure an L2TP/IPsec gateway. This article will use OpenBSD as the operating system to implement the gateway. While this choice may sound surprising, OpenBSD excels at building gateways of all sorts thanks to its simple configuration formats and inclusion of all necessary software and documentation to do so in the base system. The tutorial assumes you have setup a local network between the hosts in the cloud, and walks through the configuration of an OpenBSD host as a IPsec gateway On the OpenBSD host, all necessary software is already installed. We will configure the system, as well as pf, npppd, and ipsec + Configure L2TP + Configure IPsec + Configure NAT + Enabled services: ipsec isakmpd npppd The tutorial then walks through configuring a OS X client, but other desktops will be very similar *** Running Nextcloud in a jail on FreeBSD (https://ramsdenj.com/2017/06/05/nextcloud-in-a-jail-on-freebsd.html) I recently setup Nextcloud 12 inside a FreeBSD jail in order to allow me access to files i might need while at University. I figured this would be a optimal solution for files that I might need access to unexpectedly, on computers where I am not in complete control. My Nextcloud instance is externally accessible, and yet if someone were to get inside my Jail, I could rest easy knowing they still didn't have access to the rest of my host server. I chronicled the setup process including jail setup using iocage, https with Lets Encrypt, and full setup of the web stack. Nextcloud has a variety of features such as calendar synchronization, email, collaborative editing, and even video conferencing. I haven't had time to play with all these different offerings and have only utilized the file synchronization, but even if file sync is not needed, Nextcloud has many offerings that make it worth setting up. MariaDB, PHP 7.0, and Apache 2.4 To manage my jails I'm using iocage. In terms of jail managers it's a fairly new player in the game of jail management and is being very actively developed. It just had a full rewrite in Python, and while the code in the background might be different, the actual user interface has stayed the same. Iocage makes use of ZFS clones in order to create “base jails”, which allow for sharing of one set of system packages between multiple jails, reducing the amount of resources necessary. Alternatively, jails can be completely independent from each other; however, using a base jail makes it easier to update multiple jails as well. + pkg install iocage + sysrc iocageenable=YES + iocage fetch -r 11.0-RELEASE + iocage create tag="stratus" jailzfs=on vnet=off boot=on ip4_addr="sge0|172.20.0.100/32" -r 11.0-RELEASE + iocage start stratus + iocage console stratus I have chosen to provide storage to the Nextcloud Jail by mounting a dataset over NFS on my host box. This means my server can focus on serving Nextcloud and my storage box can focus on housing the data. The Nextcloud Jail is not even aware of this since the NFS Mount is simply mounted by the host server into the jail. The other benefit of this is the Nextcloud jail doesn't need to be able to see my storage server, nor the ability to mount the NFS share itself. Using a separate server for storage isn't necessary and if the storage for my Nextcloud server was being stored on the same server I would have created a ZFS dataset on the host and mounted it into the jail. Next I set up a dataset for the database and delegated it into the jail. Using a separate dataset allows me to specify certain properties that are better for a database, it also makes migration easier in case I ever need to move or backup the database. With most of the requirements in place it was time to start setting up Nextcloud. The requirements for Nextcloud include your basic web stack of a web server, database, and PHP. Also covers the setup of acme.sh for LetsEncrypt. This is now available as a package, and doesn't need to be manually fetched Install a few more packages, and do a bit of configuration, and you have a NextCloud server *** Historical: My first OpenBSD Hackathon (http://bad.network/historical-my-first-openbsd-hackathon.html) This is a blog post by our friend, and OpenBSD developer: Peter Hessler This is a story about encouragement. Every time I use the word "I", you should think "I as in me, not I as in the author". In 2003, I was invited to my first OpenBSD Hackathon. Way before I was into networking, I was porting software to my favourite OS. Specifically, I was porting games. On the first night most of the hackathon attendees end up at the bar for food and beer, and I'm sitting next to Theo de Raadt, the founder of OpenBSD. At some point during the evening, he's telling me about all of these "crazy" ideas he has about randomizing libraries, and protections that can be done in ld.so. (ld.so is the part of the OS that loads the libraries your program needs. It's, uh, kinda important.) Theo is encouraging me to help implement some of these ideas! At some point I tell Theo "I'm just a porter, I don't know C." Theo responds with "It isn't hard, I'll have Dale (Rahn) show you how ld.so works, and you can do it." I was hoping that all of this would be forgotten by the next day, but sure enough Dale comes by. "Hey, are you Peter? Theo wanted me to show you how ld.so works" Dale spends an hour or two showing me how it works, the code structure, and how to recover in case of failure. At first I had lots of failures. Then more failures. And even more failures. Once, I broke my machine so badly I had to reinstall it. I learned a lot about how an OS works during this. But, I eventually started doing changes without it breaking. And some even did what I wanted! By the end of the hackathon I had came up with a useful patch, that was committed as part of a larger change. I was a nobody. With some encouragement, enough liquid courage to override my imposter syndrome, and a few hours of mentoring, I'm now doing big projects. The next time you're sitting at a table with someone new to your field, ask yourself: how can you encourage them? You just might make the world better. Thank you Dale. And thank you Theo. Everyone has to start somewhere. One of the things that sets the BSDs apart from certain other open source operating systems, is the welcoming community, and the tradition of mentorship. Sure, someone else in the OpenBSD project could have done the bits that Peter did, likely a lot more quickly, but then OpenBSD wouldn't have gained a new committer. So, if you are interested in working on one of the BSDs, reach out, and we'll try to help you find a mentor. What part of the system do you want to work on? *** Interview - Dan McDonald - allcoms@gmail.com (mailto:allcoms@gmail.com) (danboid) News Roundup FreeBSD 11.1-RC1 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087340.html) 11.1-RC1 Installation images are available for: amd64, i386 powerpc, powerpc64 sparc64 armv6 BANANAPI, BEAGLEBONE, CUBIEBOARD, CUBIEBOARD2, CUBOX-HUMMINGBOARD, GUMSTIX, RPI-B, RPI2, PANDABOARD, WANDBOARD aarch64 (aka arm64), including the RPI3, Pine64, OverDrive 1000, and Cavium Server A summary of changes since BETA3 includes: Several build toolchain related fixes. A use-after-free in RPC client code has been corrected. The ntpd(8) leap-seconds file has been updated. Various VM subsystem fixes. The '_' character is now allowed in newfs(8) labels. A potential sleep while holding a mutex has been corrected in the sa(4) driver. A memory leak in an ioctl handler has been fixed in the ses(4) driver. Virtual Machine Disk Images are available for the amd64 and i386 architectures. Amazon EC2 AMI Images of FreeBSD/amd64 EC2 AMIs are available The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases. Systems running earlier FreeBSD releases can upgrade as follows: freebsd-update upgrade -r 11.1-RC1 During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly. freebsd-update install The system must be rebooted with the newly installed kernel before continuing. shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: freebsd-update install It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 10.x. Alternatively, the user can install misc/compat10x and other compatibility libraries, afterwards the system must be rebooted into the new userland: shutdown -r now Finally, after rebooting, freebsd-update needs to be run again to remove stale files: freebsd-update install Oil changes, safety recalls, and software patches (http://www.daemonology.net/blog/2017-06-14-oil-changes-safety-recalls-software-patches.html) Every few months I get an email from my local mechanic reminding me that it's time to get my car's oil changed. I generally ignore these emails; it costs time and money to get this done (I'm sure I could do it myself, but the time it would cost is worth more than the money it would save) and I drive little enough — about 2000 km/year — that I'm not too worried about the consequences of going for a bit longer than nominally advised between oil changes. I do get oil changes done... but typically once every 8-12 months, rather than the recommended 4-6 months. From what I've seen, I don't think I'm alone in taking a somewhat lackadaisical approach to routine oil changes. On the other hand, there's another type of notification which elicits more prompt attention: Safety recalls. There are two good reasons for this: First, whether for vehicles, food, or other products, the risk of ignoring a safety recall is not merely that the product will break, but rather that the product will be actively unsafe; and second, when there's a safety recall you don't have to pay for the replacement or fix — the cost is covered by the manufacturer. I started thinking about this distinction — and more specifically the difference in user behaviour — in the aftermath of the "WannaCry" malware. While WannaCry attracted widespread attention for its "ransomware" nature, the more concerning aspect of this incident is how it propagated: By exploiting a vulnerability in SMB for which Microsoft issued patches two months earlier. As someone who works in computer security, I find this horrifying — and I was particularly concerned when I heard that the NHS was postponing surgeries because they couldn't access patient records. Think about it: If the NHS couldn't access patient records due to WannaCry, it suggests WannaCry infiltrated systems used to access patient records — meaning that someone else exploiting the same vulnerabilities could have accessed those records. The SMB subsystem in Windows was not merely broken; until patches were applied, it was actively unsafe. I imagine that most people in my industry would agree that security patches should be treated in the same vein as safety recalls — unless you're certain that you're not affected, take care of them as a matter of urgency — but it seems that far more users instead treat security patches more like oil changes: something to be taken care of when convenient... or not at all, if not convenient. It's easy to say that such users are wrong; but as an industry it's time that we think about why they are wrong rather than merely blaming them for their problems. There are a few factors which I think are major contributors to this problem. First, the number of updates: When critical patches occur frequently enough to become routine, alarm fatigue sets in and people cease to give the attention updates deserve, even if on a conscious level they still recognize the importance of applying updates. Colin also talks about his time as the FreeBSD Security Officer, and the problems in ensuring the patches are correct and do not break the system when installed He also points out the problem of systems like Windows Update, the combines optional updates, and things like its license checking tool, in the same interface that delivers important updates. Or my recent machines, that gets constant popups about how some security updates will not be delivered because my processor is too new. My bank sends me special offers in the mail but phones if my credit card usage trips fraud alarms; this is the sort of distinction in intrusiveness we should see for different types of software updates Finally, I think there is a problem with the mental model most people have of computer security. Movies portray attackers as geniuses who can break into any system in minutes; journalists routinely warn people that "nobody is safe"; and insurance companies offer insurance against "cyberattacks" in much the same way as they offer insurance against tornados. Faced with this wall of misinformation, it's not surprising that people get confused between 400 pound hackers sitting on beds and actual advanced persistent threats. Yes, if the NSA wants to break into your computer, they can probably do it — but most attackers are not the NSA, just like most burglars are not Ethan Hunt. You lock your front door, not because you think it will protect you from the most determined thieves, but because it's an easy step which dramatically reduces your risk from opportunistic attack; but users don't see applying security updates as the equivalent of locking their front door when they leave home. SKIP grep, use AWK (http://blog.jpalardy.com/posts/skip-grep-use-awk/) This is a tip from Jonathan Palardy in a series of blog posts about awk. It is especially helpful for people who write a lot of shell scripts or are using a lot of pipes with awk and grep. Over the years, I've seen many people use this pattern (filter-map): $ [data is generated] | grep something | awk '{print $2}' but it can be shortened to: $ [data is generated] | awk '/something/ {print $2}' AWK can take a regular expression (the part between the slashes) and matches that to the input. Anything that matches is being passed to the print $2 action (to print the second column). Why would I do this? I can think of 4 reasons: *it's shorter to type *it spawns one less process *awk uses modern (read “Perl”) regular expressions, by default – like grep -E *it's ready to “augment” with more awk How about matching the inverse (search for patterns that do NOT match)? But “grep -v” is OK… Many people have pointed out that “grep -v” can be done more concisely with: $ [data is generated] | awk '! /something/' See if you have such combinations of grep piped to awk and fix those in your shell scripts. It saves you one process and makes your scripts much more readable. Also, check out the other intro links on the blog if you are new to awk. *** vim Adventures (https://vim-adventures.com) This website, created by Doron Linder, will playfully teach you how to use vim. Hit any key to get started and follow the instructions on the playing field by moving the cursor around. There is also a menu in the bottom left corner to save your game. Try it out, increase your vim-fu, and learn how to use a powerful text editor more efficiently. *** Beastie Bits Slides from PkgSrcCon (http://pkgsrc.org/pkgsrcCon/2017/talks.html) OpenBSD's doas adds systemd compat shim (http://marc.info/?l=openbsd-tech&m=149902196520920&w=2) Deadlock Empire -- “Each challenge below is a computer program of two or more threads. You take the role of the Scheduler - and a cunning one! Your objective is to exploit flaws in the programs to make them crash or otherwise malfunction.” (https://deadlockempire.github.io/) EuroBSDcon 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/eurobsdcon-2017-travel-grant-application-now-open/) Registration for vBSDCon is open (http://www.vbsdcon.com/) - Registration is only $100 if you register before July 31. Discount hotel rooms arranged at the Hyatt for only $100/night while supplies last. BSD Taiwan call for papers opens, closes July 31st (https://bsdtw.org/)Windows Application Versand *** Feedback/Questions Joseph - Server Monitoring (http://dpaste.com/2AM6C2H#wrap) Paulo - Updating Jails (http://dpaste.com/1Z4FBE2#wrap) Kevin - openvpn server (http://dpaste.com/2MNM9GJ#wrap) Todd - several questions (http://dpaste.com/17BVBJ3#wrap) ***
The NetBSD 8.0 release process is underway, we try to measure the weight of an electron, and look at stack clashing. This episode was brought to you by Headlines NetBSD 8.0 release process underway (https://mail-index.netbsd.org/netbsd-announce/2017/06/06/msg000267.html) Soren Jacobsen writes on NetBSD-announce: If you've been reading source-changes@, you likely noticed the recent creation of the netbsd-8 branch. If you haven't been reading source-changes@, here's some news: the netbsd-8 branch has been created, signaling the beginning of the release process for NetBSD 8.0. We don't have a strict timeline for the 8.0 release, but things are looking pretty good at the moment, and we expect this release to happen in a shorter amount of time than the last couple major releases did. At this point, we would love for folks to test out netbsd-8 and let us know how it goes. A couple of major improvements since 7.0 are the addition of USB 3 support and an overhaul of the audio subsystem, including an in-kernel mixer. Feedback about these areas is particularly desired. To download the latest binaries built from the netbsd-8 branch, head to [http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/(]http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/) Thanks in advance for helping make NetBSD 8.0 a stellar release! OpenIndiana Hipster 2017.04 is here (https://www.openindiana.org/2017/05/03/openindiana-hipster-2017-04-is-here/) Desktop software and libraries Xorg was updated to 1.18.4, xorg libraries and drivers were updated. Mate was updated to 1.16 Intel video driver was updated, the list of supported hardware has significantly extended (https://wiki.openindiana.org/oi/Intel+KMS+driver) libsmb was updated to 4.4.6 gvfs was updated to 1.26.0 gtk3 was updated to 3.18.9 Major text editors were updated (we ship vim 8.0.104, joe 4.4, emacs 25.2, nano 2.7.5 pulseaudio was updated to 10.0 firefox was updated to 45.9.0 thunderbird was updated to 45.8.0 critical issue in enlightenment was fixed, now it's operational again privoxy was updated to 3.0.26 Mesa was updated to 13.0.6 Nvidia driver was updated to 340.102 Development tools and libraries GCC 6 was added. Patches necessary to compile illumos-gate with GCC 6 were added (note, compiling illumos-gate with version other than illumos-gcc-4.4.4 is not supported) GCC 7.1 added to Hipster (https://www.openindiana.org/2017/05/05/gcc-7-1-added-the-hipster-and-rolling-forward/) Bison was updated to 3.0.4 Groovy 2.4 was added Ruby 1.9 was removed, Ruby 2.3 is the default Ruby now Perl 5.16 was removed. 64-bit Perl 5.24 is shipped. 64-bit OpenJDK 8 is the default OpenJDK version now. Mercurial was updated to 4.1.3 Git was updated to 2.12.2 ccache was updated to 3.3.3 QT 5.8.0 was added Valgrind was updated to 3.12.0 Server software PostgreSQL 9.6 was added, PostgreSQL 9.3-9.5 were updated to latest minor versions MongoDB 3.4 was added MariaDB 10.1 was added NodeJS 7 was added Percona Server 5.5/5.6/5.7 and MariaDB 5.5 were updated to latest minor versions OpenVPN was updated to 2.4.1 ISC Bind was updated to 9.10.4-P8 Squid was updated to 3.5.25 Nginx was updated to 1.12.0 Apache 2.4 was updated to 2.4.25. Apache 2.4 is the default Apache server now. Apache 2.2 will be removed before the next snapshot. ISC ntpd was updated to 4.2.8p10 OpenSSH was updated to 7.4p1 Samba was updated to 4.4.12 Tcpdump was updated to 4.9.0 Snort was updated to 2.9.9.0 Puppet was updated to 3.8.6 A lot of other bug fixes and minor software updates included. *** PKGSRC at The University of Wisconsin–Milwaukee (https://uwm.edu/hpc/software-management/) This piece is from the University of Wisconsin, Milwaukee Why Use Package Managers? Why Pkgsrc? Portability Flexibility Modernity Quality and Security Collaboration Convenience Growth Binary Packages for Research Computing The University of Wisconsin — Milwaukee provides binary pkgsrc packages for selected operating systems as a service to the research computing community. Unlike most package repositories, which have a fixed prefix and frequently upgraded packages, these packages are available for multiple prefixes and remain unchanged for a given prefix. Additional packages may be added and existing packages may be patched to fix bugs or security issues, but the software versions will not be changed. This allows researchers to keep older software in-place indefinitely for long-term studies while deploying newer software in later snapshots. Contributing to Pkgsrc Building Your Own Binary Packages Check out the full article and consider using pkgsrc for your own research purposes. PKGSrc Con is this weekend! (http://www.pkgsrc.org/pkgsrcCon/2017/) *** Measuring the weight of an electron (https://deftly.net/posts/2017-06-01-measuring-the-weight-of-an-electron.html) An interesting story of the struggles of one person, aided only by their pet Canary, porting Electron to OpenBSD. This is a long rant. A rant intended to document lunacy, hopefully aid others in the future and make myself feel better about something I think is crazy. It may seem like I am making an enemy of electron, but keep in mind that isn't my intention! The enemy here, is complexity! My friend Henry, a canary, is coming along for the ride! Getting the tools At first glance Electron seems like a pretty solid app, it has decent docs, it's consolidated in a single repository, has a lot of visibility, porting it shouldn't be a big deal, right? After cloning the repo, trouble starts: Reading through the doc, right off the bat there are a few interesting things: At least 25GB disk space. Huh, OK, some how this ~47M repository is going to blow up to 25G? Continuing along with the build, I know I have two versions of clang installed on OpenBSD, one from ports and one in base. Hopefully I will be able to tell the build to use one of these versions. Next, it's time to tell the bootstrap that OpenBSD exists as a platform. After that is fixed, the build-script runs. Even though cloning another git repo fails, the build happily continues. Wait. Another repository failed to clone? At least this time the build failed after trying to clone boto.. again. I am guessing it tried twice because something might have changed between now and the last clone? Off in the distance we catch a familiar tune, it almost sounds like Gnarls Barkley's song Crazy, can't tell for sure. As it turns out, if you are using git-fsck, you are unable to clone boto and requests. Obviously the proper fix for his is to not care about the validity of the git objects! So we die a little inside and comment out fsckobjects in our ~/.gitconfig. Next up, chromium-58 is downloaded… Out of curiosity we look at vendor/libchromiumcontent/script/update, it seems its purpose is to download / extract chromium clang and node, good thing we already specified --clang_dir or it might try to build clang again! 544 dots and 45 minutes later, we have an error! The chromium-58.0.3029.110.tar.xz file is mysteriously not there anymore.. Interesting. Wut. “Updating Clang…”. Didn't I explicitly say not to build clang? At this point we have to shift projects, no longer are we working on Electron.. It's libchromiumcontent that needs our attention. Fixing sub-tools Ahh, our old friends the dots! This is the second time waiting 45+ minutes for a 500+ MB file to download. We are fairly confident it will fail, delete the file out from under itself and hinder the process even further, so we add an explicit exit to the update script. This way we can copy the file somewhere safe! Another 45 minute chrome build and saving the downloaded executable to a save space seems in order. Fixing another 50 occurrences of error conditions let's the build continue - to another clang build. We remove the call to update_clang, because.. well.. we have two copies of it already and the Electron doc said everything would be fine if we had >= clang 3.4! More re-builds and updates of clang and chromium are being commented out, just to get somewhere close to the actual electron build. Fixing sub-sub-tools Ninja needs to be build and the script for that needs to be told to ignore this “unsupported OS” to continue. No luck. At this point we are faced with a complex web of python scripts that execute gn on GN files to produce ninja files… which then build the various components and somewhere in that cluster, something doesn't know about OpenBSD… I look at Henry, he is looking a photo of his wife and kids. They are sitting on a telephone wire, the morning sun illuminating their beautiful faces. Henry looks back at me and says “It's not worth it.” We slam the laptop shut and go outside. Interview - Dan McDonald - allcoms@gmail.com (mailto:allcoms@gmail.com) (danboid) News Roundup g4u 2.6 (ghosting for unix) released 18th birthday (https://mail-index.netbsd.org/netbsd-users/2017/06/08/msg019625.html) Hubert Feyrer writes in his mail to netbsd-users: After a five-year period for beta-testing and updating, I have finally released g4u 2.6. With its origins in 1999, I'd like to say: Happy 18th Birthday, g4u! About g4u: g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported. The past: When I started g4u, I had the task to install a number of lab machines with a dual-boot of Windows NT and NetBSD. The hype was about Microsoft's "Zero Administration Kit" (ZAK) then, but that did barely work for the Windows part - file transfers were slow, depended on the clients' hardware a lot (requiring fiddling with MS DOS network driver disks), and on the ZAK server the files for installing happened do disappear for no good reason every now and then. Not working well, and leaving out NetBSD (and everything else), I created g4u. This gave me the (relative) pain of getting things working once, but with the option to easily add network drivers as they appeared in NetBSD (and oh they did!), plus allowed me to install any operating system. The present: We've used g4u successfully in our labs then, booting from CDROM. I also got many donations from public and private institutions plus companies from many sectors, indicating that g4u does make a difference. In the meantime, the world has changed, and CDROMs aren't used that much any more. Network boot and USB sticks are today's devices of choice, cloning of a full disk without knowing its structure has both advantages but also disadvantages, and g4u's user interface is still command-line based with not much space for automation. For storage, FTP servers are nice and fast, but alternatives like SSH/SFTP, NFS, iSCSI and SMB for remote storage plus local storage (back to fun with filesystems, anyone? avoiding this was why g4u was created in the first place!) should be considered these days. Further aspects include integrity (checksums), confidentiality (encryption). This leaves a number of open points to address either by future releases, or by other products. The future: At this point, my time budget for g4u is very limited. I welcome people to contribute to g4u - g4u is Open Source for a reason. Feel free to get back to me for any changes that you want to contribute! The changes: Major changes in g4u 2.6 include: Make this build with NetBSD-current sources as of 2017-04-17 (shortly before netbsd-8 release branch), binaries were cross-compiled from Mac OS X 10.10 Many new drivers, bugfixes and improvements from NetBSD-current (see beta1 and beta2 announcements) Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!) Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz In addition to MD5, add SHA512 checksums Congratulation, g4u. Check out the g4u website (http://fehu.org/~feyrer/g4u/) and support the project if you are using it. *** Fixing FreeBSD Networking on Digital Ocean (https://wycd.net/posts/2017-05-19-fixing-freebsd-networking-on-digital-ocean.html) Most cloud/VPS providers use some form of semi-automated address assignment, rather than just regular static address configuration, so that newly created virtual machines can configure themselves. Sometimes, especially during the upgrade process, this can break. This is the story of one such user: I decided it was time to update my FreeBSD Digital Ocean droplet from the end-of-life version 10.1 (shame on me) to the modern version 10.3 (good until April 2018), and maybe even version 11 (good until 2021). There were no sensitive files on the VM, so I had put it off. Additionally, cloud providers tend to have shoddy support for BSDs, so breakages after messing with the kernel or init system are rampant, and I had been skirting that risk. The last straw for me was a broken pkg: /usr/local/lib/libpkg.so.3: Undefined symbol "openat" So the user fires up freebsd-update and upgrades to FreeBSD 10.3 I rebooted, and of course, it happened: no ssh access after 30 seconds, 1 minute, 2 minutes…I logged into my Digital Ocean account and saw green status lights for the instance, but something was definitely wrong. Fortunately, Digital Ocean provides console access (albeit slow, buggy, and crashes my browser every time I run ping). ifconfig revealed that the interfaces vtnet0 (public) and vtnet1 (private) haven't been configured with IP addresses. Combing through files in /etc/rc.*, I found a file called /etc/rc.digitalocean.d/${DROPLETID}.conf containing static network settings for this droplet (${DROPLETID} was something like 1234567). It seemed that FreeBSD wasn't picking up the Digital Ocean network settings config file. The quick and dirty way would have been to messily append the contents of this file to /etc/rc.conf, but I wanted a nicer way. Reading the script in /etc/rc.d/digitalocean told me that /etc/rc.digitalocean.d/${DROPLET_ID}.conf was supposed to have a symlink at /etc/rc.digitalocean.d/droplet.conf. It was broken and pointed to /etc/rc.digitalocean.d/.conf, which could happen when the curl command in /etc/rc.d/digitalocean fails Maybe the curl binary was also in need for an upgrade so failed to fetch the droplet ID Using grep to fish for files containing droplet.conf, I discovered that it was hacked into the init system via loadrcconfig() in /etc/rc.subr I would prefer if Digital Ocean had not customized the version of FreeBSD they ship quite so much I could fix that symlink and restart the services: set DROPLET_ID=$(curl -s http://169.254.169.254/metadata/v1/id) ln -s -f /etc/rc.digitalocean.d/${DROPLET_ID}.conf /etc/rc.digitalocean.d/droplet.conf /etc/rc.d/netif restart /etc/rc.d/routing restart Networking was working again, and I could then ssh into my server and run the following to finish the upgrade: freebsd-update install At this point, I decided that I didn't want to deal with this mess again until at least 2021, so I decided to go for 11.0-RELEASE freebsd-update -r 11.0-RELEASE update freebsd-update install reboot freebsd-update install pkg-static install -f pkg pkg update pkg upgrade uname -a FreeBSD hostname 11.0-RELEASE-p9 FreeBSD 11.0-RELEASE-p9 pkg -v 1.10.1 The problem was solved correctly, and my /etc/rc.conf remains free of generated cruft. The Digital Ocean team can make our lives easier by having their init scripts do more thorough system checking, e.g., catching broken symlinks and bad network addresses. I'm hopeful that collaboration of the FreeBSD team and cloud providers will one day result in automatic fixing of these situations, or at least a correct status indicator. The Digital Ocean team didn't really know many FreeBSD people when they made the first 10.1 images, they have improved a lot, but they of course could always use more feedback from BSD users ** Stack Clash (https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt) A 12-year-old question: "If the heap grows up, and the stack grows down, what happens when they clash? Is it exploitable? How? In 2005, Gael Delalleau presented "Large memory management vulnerabilities" and the first stack-clash exploit in user-space (against mod_php 4.3.0 on Apache 2.0.53) (http://cansecwest.com/core05/memory_vulns_delalleau.pdf) In 2010, Rafal Wojtczuk published "Exploiting large memory management vulnerabilities in Xorg server running on Linux", the second stack-clash exploit in user-space (CVE-2010-2240) (http://www.invisiblethingslab.com/resources/misc-2010/xorg-large-memory-attacks.pdf) Since 2010, security researchers have exploited several stack-clashes in the kernel-space, In user-space, however, this problem has been greatly underestimated; the only public exploits are Gael Delalleau's and Rafal Wojtczuk's, and they were written before Linux introduced a protection against stack-clashes (a "guard-page" mapped below the stack) (https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2010-2240) In this advisory, we show that stack-clashes are widespread in user-space, and exploitable despite the stack guard-page; we discovered multiple vulnerabilities in guard-page implementations, and devised general methods for: "Clashing" the stack with another memory region: we allocate memory until the stack reaches another memory region, or until another memory region reaches the stack; "Jumping" over the stack guard-page: we move the stack-pointer from the stack and into the other memory region, without accessing the stack guard-page; "Smashing" the stack, or the other memory region: we overwrite the stack with the other memory region, or the other memory region with the stack. So this advisory itself, is not a security vulnerability. It is novel research showing ways to work around the mitigations against generic vulnerability types that are implemented on various operating systems. While this issue with the mitigation feature has been fixed, even without the fix, successful exploitation requires another application with its own vulnerability in order to be exploited. Those vulnerabilities outside of the OS need to be fixed on their own. FreeBSD-Security post (https://lists.freebsd.org/pipermail/freebsd-security/2017-June/009335.html) The issue under discussion is a limitation in a vulnerability mitigation technique. Changes to improve the way FreeBSD manages stack growth, and mitigate the issue demonstrated by Qualys' proof-of-concept code, are in progress by FreeBSD developers knowledgeable in the VM subsystem. FreeBSD address space guards (https://svnweb.freebsd.org/base?view=revision&revision=320317) HardenedBSD Proof of Concept for FreeBSD (https://github.com/lattera/exploits/blob/master/FreeBSD/StackClash/001-stackclash.c) HardenedBSD implementation: https://github.com/HardenedBSD/hardenedBSD/compare/de8124d3bf83d774b66f62d11aee0162d0cd1031...91104ed152d57cde0292b2dc09489fd1f69ea77c & https://github.com/HardenedBSD/hardenedBSD/commit/00ad1fb6b53f63d6e9ba539b8f251b5cf4d40261 Qualys PoC: freebsd_cve-2017-fgpu.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-fgpu.c) Qualys PoC: freebsd_cve-2017-fgpe.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-fgpe.c) Qualys PoC: freebsd_cve-2017-1085.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-1085.c) Qualys PoC: OpenBSD (https://www.qualys.com/2017/06/19/stack-clash/openbsd_at.c) Qualys PoC: NetBSD (https://www.qualys.com/2017/06/19/stack-clash/netbsd_cve-2017-1000375.c) *** Will ZFS and non-ECC RAM kill your data? (http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/) TL;DR: ECC is good, but even without, having ZFS is better than not having ZFS. What's ECC RAM? Is it a good idea? What's ZFS? Is it a good idea? Is ZFS and non-ECC worse than not-ZFS and non-ECC? What about the Scrub of Death? The article walks through ZFS folk lore, and talks about what can really go wrong, and what is just the over-active imagination of people on the FreeNAS forums But would using any other filesystem that isn't ZFS have protected that data? ‘Cause remember, nobody's arguing that you can lose data to evil RAM – the argument is about whether evil RAM is more dangerous with ZFS than it would be without it. I really, really want to use the Scrub Of Death in a movie or TV show. How can I make it happen? I don't care about your logic! I wish to appeal to authority! OK. “Authority” in this case doesn't get much better than Matthew Ahrens, one of the cofounders of ZFS at Sun Microsystems and current ZFS developer at Delphix. In the comments to one of my filesystem articles on Ars Technica, Matthew said “There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” Beastie Bits EuroBSDcon 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/eurobsdcon-2017-travel-grant-application-now-open/) FreeBSD 11.1-BETA3 is out, please give it a test (https://lists.freebsd.org/pipermail/freebsd-stable/2017-June/087303.html) Allan and Lacey let us know the video to the Postgresql/ZFS talk is online (http://dpaste.com/1FE80FJ) Trapsleds (https://marc.info/?l=openbsd-tech&m=149792179514439&w=2) BSD User group in North Rhine-Westphalia, Germany (https://bsd.nrw/) *** Feedback/Questions Joe - Home Server Suggestions (http://dpaste.com/2Z5BJCR#wrap) Stephen - general BSD (http://dpaste.com/1VRQYAM#wrap) Eduardo - ZFS Encryption (http://dpaste.com/2TWADQ8#wrap) Joseph - BGP Kernel Error (http://dpaste.com/0SC0GAC#wrap) ***
This week on BSD Now we cover the latest FreeBSD Status Report, a plan for Open Source software development, centrally managing bhyve with Ansible, libvirt, and pkg-ssh, and a whole lot more. This episode was brought to you by Headlines FreeBSD Project Status Report (January to March 2017) (https://www.freebsd.org/news/status/report-2017-01-2017-03.html) While a few of these projects indicate they are a "plan B" or an "attempt III", many are still hewing to their original plans, and all have produced impressive results. Please enjoy this vibrant collection of reports, covering the first quarter of 2017. The quarterly report opens with notes from Core, The FreeBSD Foundation, the Ports team, and Release Engineering On the project front, the Ceph on FreeBSD project had made considerable advances, and is now usable as the net/ceph-devel port via the ceph-fuse module. Eventually they hope to have a kernel RADOS block device driver, so fuse is not required CloudABI update, including news that the Bitcoin reference implementation is working on a port to CloudABI eMMC Flash and SD card updates, allowing higher speeds (max speed changes from ~40 to ~80 MB/sec). As well, the MMC Stack can now also be backed by the CAM framework. Improvements to the Linuxulator More detail on the pNFS Server plan B that we discussed in a previous week Snow B.V. is sponsoring a dutch translation of the FreeBSD Handbook using the new .po system *** A plan for open source software maintainers (http://www.daemonology.net/blog/2017-05-11-plan-for-foss-maintainers.html) Colin Percival describes in his blog “a plan for open source software maintainers”: I've been writing open source software for about 15 years now; while I'm still wet behind the ears compared to FreeBSD greybeards like Kirk McKusick and Poul-Henning Kamp, I've been around for long enough to start noticing some patterns. In particular: Free software is expensive. Software is expensive to begin with; but good quality open source software tends to be written by people who are recognized as experts in their fields (partly thanks to that very software) and can demand commensurate salaries. While that expensive developer time is donated (either by the developers themselves or by their employers), this influences what their time is used for: Individual developers like doing things which are fun or high-status, while companies usually pay developers to work specifically on the features those companies need. Maintaining existing code is important, but it is neither fun nor high-status; and it tends to get underweighted by companies as well, since maintenance is inherently unlikely to be the most urgent issue at any given time. Open source software is largely a "throw code over the fence and walk away" exercise. Over the past 15 years I've written freebsd-update, bsdiff, portsnap, scrypt, spiped, and kivaloo, and done a lot of work on the FreeBSD/EC2 platform. Of these, I know bsdiff and scrypt are very widely used and I suspect that kivaloo is not; but beyond that I have very little knowledge of how widely or where my work is being used. Anecdotally it seems that other developers are in similar positions: At conferences I've heard variations on "you're using my code? Wow, that's awesome; I had no idea" many times. I have even less knowledge of what people are doing with my work or what problems or limitations they're running into. Occasionally I get bug reports or feature requests; but I know I only hear from a very small proportion of the users of my work. I have a long list of feature ideas which are sitting in limbo simply because I don't know if anyone would ever use them — I suspect the answer is yes, but I'm not going to spend time implementing these until I have some confirmation of that. A lot of mid-size companies would like to be able to pay for support for the software they're using, but can't find anyone to provide it. For larger companies, it's often easier — they can simply hire the author of the software (and many developers who do ongoing maintenance work on open source software were in fact hired for this sort of "in-house expertise" role) — but there's very little available for a company which needs a few minutes per month of expertise. In many cases, the best support they can find is sending an email to the developer of the software they're using and not paying anything at all — we've all received "can you help me figure out how to use this" emails, and most of us are happy to help when we have time — but relying on developer generosity is not a good long-term solution. Every few months, I receive email from people asking if there's any way for them to support my open source software contributions. (Usually I encourage them to donate to the FreeBSD Foundation.) Conversely, there are developers whose work I would like to support (e.g., people working on FreeBSD wifi and video drivers), but there isn't any straightforward way to do this. Patreon has demonstrated that there are a lot of people willing to pay to support what they see as worthwhile work, even if they don't get anything directly in exchange for their patronage. It seems to me that this is a case where problems are in fact solutions to other problems. To wit: Users of open source software want to be able to get help with their use cases; developers of open source software want to know how people are using their code. Users of open source software want to support the the work they use; developers of open source software want to know which projects users care about. Users of open source software want specific improvements; developers of open source software may be interested in making those specific changes, but don't want to spend the time until they know someone would use them. Users of open source software have money; developers of open source software get day jobs writing other code because nobody is paying them to maintain their open source software. I'd like to see this situation get fixed. As I envision it, a solution would look something like a cross between Patreon and Bugzilla: Users would be able sign up to "support" projects of their choosing, with a number of dollars per month (possibly arbitrary amounts, possibly specified tiers; maybe including $0/month), and would be able to open issues. These could be private (e.g., for "technical support" requests) or public (e.g., for bugs and feature requests); users would be able to indicate their interest in public issues created by other users. Developers would get to see the open issues, along with a nominal "value" computed based on allocating the incoming dollars of "support contracts" across the issues each user has expressed an interest in, allowing them to focus on issues with higher impact. He poses three questions to users about whether or not people (users and software developers alike) would be interested in this and whether payment (giving and receiving, respectively) is interesting Check out the comments (and those on https://news.ycombinator.com/item?id=14313804 (reddit.com)) as well for some suggestions and discussion on the topic *** OpenBSD vmm hypervisor: Part 2 (http://www.h-i-r.net/2017/04/openbsd-vmm-hypervisor-part-2.html) We asked for people to write up their experience using OpenBSD's VMM. This blog post is just that This is going to be a (likely long-running, infrequently-appended) series of posts as I poke around in vmm. A few months ago, I demonstrated some basic use of the vmm hypervisor as it existed in OpenBSD 6.0-CURRENT around late October, 2016. We'll call that video Part 1. Quite a bit of development was done on vmm before 6.1-RELEASE, and it's worth noting that some new features made their way in. Work continues, of course, and I can only imagine the hypervisor technology will mature plenty for the next release. As it stands, this is the first release of OpenBSD with a native hypervisor shipped in the base install, and that's exciting news in and of itself To get our virtual machines onto the network, we have to spend some time setting up a virtual ethernet interface. We'll run a DHCP server on that, and it'll be the default route for our virtual machines. We'll keep all the VMs on a private network segment, and use NAT to allow them to get to the network. There is a way to directly bridge VMs to the network in some situations, but I won't be covering that today. Create an empty disk image for your new VM. I'd recommend 1.5GB to play with at first. You can do this without doas or root if you want your user account to be able to start the VM later. I made a "vmm" directory inside my home directory to store VM disk images in. You might have a different partition you wish to store these large files in. Boot up a brand new vm instance. You'll have to do this as root or with doas. You can download a -CURRENT install kernel/ramdisk (bsd.rd) from an OpenBSD mirror, or you can simply use the one that's on your existing system (/bsd.rd) like I'll do here. The command will start a VM named "test.vm", display the console at startup, use /bsd.rd (from our host environment) as the boot image, allocate 256MB of memory, attach the first network interface to the switch called "local" we defined earlier in /etc/vm.conf, and use the test image we just created as the first disk drive. Now that the VM disk image file has a full installation of OpenBSD on it, build a VM configuration around it by adding the below block of configuration (with modifications as needed for owner, path and lladdr) to /etc/vm.conf I've noticed that VMs with much less than 256MB of RAM allocated tend to be a little unstable for me. You'll also note that in the "interface" clause, I hard-coded the lladdr that was generated for it earlier. By specifying "disable" in vm.conf, the VM will show up in a stopped state that the owner of the VM (that's you!) can manually start without root access. Let us know how VMM works for you *** News Roundup openbsd changes of note 621 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-621) More stuff, more fun. Fix script to not perform tty operations on things that aren't ttys. Detected by pledge. Merge libdrm 2.4.79. After a forced unmount, also unmount any filesystems below that mount point. Flip previously warm pages in the buffer cache to memory above the DMA region if uvm tells us it is available. Pages are not automatically promoted to upper memory. Instead it's used as additional memory only for what the cache considers long term buffers. I/O still requires DMA memory, so writing to a buffer will pull it back down. Makefile support for systems with both gcc and clang. Make i386 and amd64 so. Take a more radical approach to disabling colours in clang. When the data buffered for write in tmux exceeds a limit, discard it and redraw. Helps when a fast process is running inside tmux running inside a slow terminal. Add a port of witness(4) lock validation tool from FreeBSD. Use it with mplock, rwlock, and mutex in the kernel. Properly save and restore FPU context in vmm. Remove KGDB. It neither compiles nor works. Add a constant time AES implementation, from BearSSL. Remove SSHv1 from ssh. and more... *** Digging into BSD's choice of Unix group for new directories and files (https://utcc.utoronto.ca/~cks/space/blog/unix/BSDDirectoryGroupChoice) I have to eat some humble pie here. In comments on my entry on an interesting chmod failure, Greg A. Woods pointed out that FreeBSD's behavior of creating everything inside a directory with the group of the directory is actually traditional BSD behavior (it dates all the way back to the 1980s), not some odd new invention by FreeBSD. As traditional behavior it makes sense that it's explicitly allowed by the standards, but I've also come to think that it makes sense in context and in general. To see this, we need some background about the problem facing BSD. In the beginning, two things were true in Unix: there was no mkdir() system call, and processes could only be in one group at a time. With processes being in only one group, the choice of the group for a newly created filesystem object was easy; it was your current group. This was felt to be sufficiently obvious behavior that the V7 creat(2) manpage doesn't even mention it. Now things get interesting. 4.1c BSD seems to be where mkdir(2) is introduced and where creat() stops being a system call and becomes an option to open(2). It's also where processes can be in multiple groups for the first time. The 4.1c BSD open(2) manpage is silent about the group of newly created files, while the mkdir(2) manpage specifically claims that new directories will have your effective group (ie, the V7 behavior). This is actually wrong. In both mkdir() in sysdirectory.c and maknode() in ufssyscalls.c, the group of the newly created object is set to the group of the parent directory. Then finally in the 4.2 BSD mkdir(2) manpage the group of the new directory is correctly documented (the 4.2 BSD open(2) manpage continues to say nothing about this). So BSD's traditional behavior was introduced at the same time as processes being in multiple groups, and we can guess that it was introduced as part of that change. When your process can only be in a single group, as in V7, it makes perfect sense to create new filesystem objects with that as their group. It's basically the same case as making new filesystem objects be owned by you; just as they get your UID, they also get your GID. When your process can be in multiple groups, things get less clear. A filesystem object can only be in one group, so which of your several groups should a new filesystem object be owned by, and how can you most conveniently change that choice? One option is to have some notion of a 'primary group' and then provide ways to shuffle around which of your groups is the primary group. Another option is the BSD choice of inheriting the group from context. By far the most common case is that you want your new files and directories to be created in the 'context', ie the group, of the surrounding directory. If you fully embrace the idea of Unix processes being in multiple groups, not just having one primary group and then some number of secondary groups, then the BSD choice makes a lot of sense. And for all of its faults, BSD tended to relatively fully embrace its changes While it leads to some odd issues, such as the one I ran into, pretty much any choice here is going to have some oddities. Centrally managed Bhyve infrastructure with Ansible, libvirt and pkg-ssh (http://www.shellguardians.com/2017/05/centrally-managed-bhyve-infrastructure.html) At work we've been using Bhyve for a while to run non-critical systems. It is a really nice and stable hypervisor even though we are using an earlier version available on FreeBSD 10.3. This means we lack Windows and VNC support among other things, but it is not a big deal. After some iterations in our internal tools, we realised that the installation process was too slow and we always repeated the same steps. Of course, any good sysadmin will scream "AUTOMATION!" and so did we. Therefore, we started looking for different ways to improve our deployments. We had a look at existing frameworks that manage Bhyve, but none of them had a feature that we find really important: having a centralized repository of VM images. For instance, SmartOS applies this method successfully by having a backend server that stores a catalog of VMs and Zones, meaning that new instances can be deployed in a minute at most. This is a game changer if you are really busy in your day-to-day operations. The following building blocks are used: The ZFS snapshot of an existing VM. This will be our VM template. A modified version of oneoff-pkg-create to package the ZFS snapshots. pkg-ssh and pkg-repo to host a local FreeBSD repo in a FreeBSD jail. libvirt to manage our Bhyve VMs. The ansible modules virt, virtnet and virtpool. Once automated, the installation process needs 2 minutes at most, compared with the 30 minutes needed to manually install VM plus allowing us to deploy many guests in parallel. NetBSD maintainer in the QEMU project (https://blog.netbsd.org/tnf/entry/netbsd_maintainer_in_the_qemu) QEMU - the FAST! processor emulator - is a generic, Open Source, machine emulator and virtualizer. It defines state of the art in modern virtualization. This software has been developed for multiplatform environments with support for NetBSD since virtually forever. It's the primary tool used by the NetBSD developers and release engineering team. It is run with continuous integration tests for daily commits and execute regression tests through the Automatic Test Framework (ATF). The QEMU developers warned the Open Source community - with version 2.9 of the emulator - that they will eventually drop support for suboptimally supported hosts if nobody will step in and take the maintainership to refresh the support. This warning was directed to major BSDs, Solaris, AIX and Haiku. Thankfully the NetBSD position has been filled - making NetBSD to restore official maintenance. Beastie Bits OpenBSD Community Goes Gold (http://undeadly.org/cgi?action=article&sid=20170510012526&mode=flat&count=0) CharmBUG's Tor Hack-a-thon has been pushed back to July due to scheduling difficulties (https://www.meetup.com/CharmBUG/events/238218840/) Direct Rendering Manager (DRM) Driver for i915, from the Linux kernel to Haiku with the help of DragonflyBSD's Linux Compatibility layer (https://www.haiku-os.org/blog/vivek/2017-05-05_[gsoc_2017]_3d_hardware_acceleration_in_haiku/) TomTom lists OpenBSD in license (https://twitter.com/bsdlme/status/863488045449977864) London Net BSD Meetup on May 22nd (https://mail-index.netbsd.org/regional-london/2017/05/02/msg000571.html) KnoxBUG meeting May 30th, 2017 - Introduction to FreeNAS (http://knoxbug.org/2017-05-30) *** Feedback/Questions Felix - Home Firewall (http://dpaste.com/35EWVGZ#wrap) David - Docker Recipes for Jails (http://dpaste.com/0H51NX2#wrap) Don - GoLang & Rust (http://dpaste.com/2VZ7S8K#wrap) George - OGG feed (http://dpaste.com/2A1FZF3#wrap) Roller - BSDCan Tips (http://dpaste.com/3D2B6J3#wrap) ***
Today on the show, we've got a look at running OpenBSD on a APU, some BSD in your Android, managing your own FreeBSD cloud service with ansible and much more. Keep it turned on your place to B...SD! This episode was brought to you by Headlines OpenBSD on PC Engines APU2 (https://github.com/elad/openbsd-apu2) A detailed walkthrough of building an OpenBSD firewall on a PC Engines APU2 It starts with a breakdown of the parts that were purchases, totally around $200 Then the reader is walked through configuring the serial console, flashing the ROM, and updating the BIOS The next step is actually creating a custom OpenBSD install image, and pre-configuring its serial console. Starting with OpenBSD 6.0, this step is done automatically by the installer Installation: Power off the APU2 Insert the bootable OpenBSD installer USB flash drive to one of the USB slots on the APU2 Power on the APU2, press F10 to get to the boot menu, and choose to boot from USB (usually option number 1) At the boot> prompt, remember the serial console settings (see above) Also at the boot> prompt, press Enter to start the installer Follow the installation instructions The driver used for wireless networking is athn(4). It might not work properly out of the box. Once OpenBSD is installed, run fw_update with no arguments. It will figure out which firmware updates are required and will download and install them. When it finishes, reboot. Where the rubber meets the road… (part one) (https://functionallyparanoid.com/2016/11/29/where-the-rubber-meets-the-road-part-one/) A user describes their adventures installing OpenBSD and Arch Linux on a new Lenovo X1 Carbon (4th gen, skylake) They also detail why they moved away from their beloved Macbook, which while long, does describe a journey away from Apple that we've heard elsewhere. The journey begins with getting a new Windows laptop, shrinking the partition and creating space for a triple-boot install, of Windows / Arch / OpenBSD Brian then details how he setup the partitioning and performed the initial Arch installation, getting it tuned to his specifications. Next up was OpenBSD though, and that went sideways initially due to a new NVMe drive that wasn't fully supported (yet) The article is split into two parts (we will bring you the next installment at a future date), but he leaves us with the plan of attack to build a custom OpenBSD kernel with corrected PCI device identifiers. We wish Brian luck, and look forward to the “rest of the story” soon. *** Howto setup a FreeBSD jail server using iocage and ansible. (https://github.com/JoergFiedler/freebsd-ansible-demo) Setting up a FreeBSD jail server can be a daunting task. However when a guide comes along which shows you how to do that, including not exposing a single (non-jailed) port to the outside world, you know we had a take a closer look. This guide comes to us from GitHub, courtesy of Joerg Fielder. The project goals seem notable: Ansible playbook that creates a FreeBSD server which hosts multiple jails. Travis is used to run/test the playbook. No service on the host is exposed externally. All external connections terminate within a jail. Roles can be reused using Ansible Galaxy. Combine any of those roles to create FreeBSD server, which perfectly suits you. To get started, you'll need a machine with Ansible, Vagrant and VirtualBox, and your credentials to AWS if you want it to automatically create / destroy EC2 instances. There's already an impressive list of Anisible roles created for you to start with: freebsd-build-server - Creates a FreeBSD poudriere build server freebsd-jail-host - FreeBSD Jail host freebsd-jailed - Provides a jail freebsd-jailed-nginx - Provides a jailed nginx server freebsd-jailed-php-fpm - Creates a php-fpm pool and a ZFS dataset which is used as web root by php-fpm freebsd-jailed-sftp - Installs a SFTP server freebsd-jailed-sshd - Provides a jailed sshd server. freebsd-jailed-syslogd - Provides a jailed syslogd freebsd-jailed-btsync - Provides a jailed btsync instance server freebsd-jailed-joomla - Installs Joomla freebsd-jailed-mariadb - Provides a jailed MariaDB server freebsd-jailed-wordpress - Provides a jailed Wordpress server. Since the machines have to be customized before starting, he mentions that cloud-init is used to do the following: activate pf firewall add a pass all keep state rule to pf to keep track of connection states, which in turn allows you to reload the pf service without losing the connection install the following packages: sudo bash python27 allow passwordless sudo for user ec2-user “ From there it is pretty straight-forward, just a couple commands to spin up the VM's either locally on your VirtualBox host, or in the cloud with AWS. Internally the VM's are auto-configured with iocage to create jails, where all your actual services run. A neat project, check it out today if you want a shake-n-bake type cloud + jail solution. Colin Percival's bsdiff helps reduce Android apk bandwidth usage by 6 petabytes per day (http://android-developers.blogspot.ca/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html) A post on the official Android-Developers blog, talks about how they used bsdiff (and bspatch) to reduce the size of Android application updates by 65% bsdiff was developed by FreeBSD's Colin Percival Earlier this year, we announced that we started using the bsdiff algorithm (by Colin Percival). Using bsdiff, we were able to reduce the size of app updates on average by 47% compared to the full APK size. This post is actually about the second generation of the code. Today, we're excited to share a new approach that goes further — File-by-File patching. App Updates using File-by-File patching are, on average, 65% smaller than the full app, and in some cases more than 90% smaller. Android apps are packaged as APKs, which are ZIP files with special conventions. Most of the content within the ZIP files (and APKs) is compressed using a technology called Deflate. Deflate is really good at compressing data but it has a drawback: it makes identifying changes in the original (uncompressed) content really hard. Even a tiny change to the original content (like changing one word in a book) can make the compressed output of deflate look completely different. Describing the differences between the original content is easy, but describing the differences between the compressed content is so hard that it leads to inefficient patches. So in the second generation of the code, they use bsdiff on each individual file, then package that, rather than diffing the original and new archives bsdiff is used in a great many other places, including shrinking the updates for the Firefox and Chrome browsers You can find out more about bsdiff here: http://www.daemonology.net/bsdiff/ A far more sophisticated algorithm, which typically provides roughly 20% smaller patches, is described in my doctoral thesis (http://www.daemonology.net/papers/thesis.pdf). Considering the gains, it is interesting that no one has implemented Colin's more sophisticated algorithm Colin had an interesting observation (https://twitter.com/cperciva/status/806426180379230208) last night: “I just realized that bandwidth savings due to bsdiff are now roughly equal to what the total internet traffic was when I wrote it in 2003.” *** News Roundup Distrowatch does an in-depth review of NAS4Free (https://distrowatch.com/weekly.php?issue=20161114#nas4free) Jesse Smith over at DistroWatch has done a pretty in-depth review of Nas4Free. The review starts with mentioning that NAS4Free works on 3 platforms, ARM/i386/AMD64 and for the purposes of this review he would be using AMD64 builds. After going through the initial install (doing typical disk management operations, such as GPT/MBR, etc) he was ready to begin using the product. One concern originally observed was that the initial boot seemed rather slow. Investigation revealed this was due to it loading the entire OS image into memory, and the first (long) disk read did take some time, but once loaded was super responsive. The next steps involved doing the initial configuration, which meant creating a new ZFS storage pool. After this process was done, he did find one puzzling UI option called “VM” which indicated it can be linked to VirtualBox in some way, but the Docs didn't reveal its secrets of usage. Additionally covered were some of the various “Access” methods, including traditional UNIX permissions, AD and LDAP, and then various Sharing services which are typical to a NAS, Such as NFS / Samba and others. One neat feature was the built-in file-browser via the web-interface, which allows you another method of getting at your data when sometimes NFS / Samba or WebDav aren't enough. Jesse gives us a nice round-up conclusion as well Most of the NAS operating systems I have used in the past were built around useful features. Some focused on making storage easy to set up and manage, others focused on services, such as making files available over multiple protocols or managing torrents. Some strive to be very easy to set up. NAS4Free does pretty well in each of the above categories. It may not be the easiest platform to set up, but it's probably a close second. It may not have the prettiest interface for managing settings, but it is quite easy to navigate. NAS4Free may not have the most add-on services and access protocols, but I suspect there are more than enough of both for most people. Where NAS4Free does better than most other solutions I have looked at is security. I don't think the project's website or documentation particularly focuses on security as a feature, but there are plenty of little security features that I liked. NAS4Free makes it very easy to lock the text console, which is good because we do not all keep our NAS boxes behind locked doors. The system is fairly easy to upgrade and appears to publish regular security updates in the form of new firmware. NAS4Free makes it fairly easy to set up user accounts, handle permissions and manage home directories. It's also pretty straight forward to switch from HTTP to HTTPS and to block people not on the local network from accessing the NAS's web interface. All in all, I like NAS4Free. It's a good, general purpose NAS operating system. While I did not feel the project did anything really amazing in any one category, nor did I run into any serious issues. The NAS ran as expected, was fairly straight forward to set up and easy to manage. This strikes me as an especially good platform for home or small business users who want an easy set up, some basic security and a solid collection of features. Browsix: Unix in the browser tab (https://browsix.org/) Browsix is a research project from the PLASMA lab at the University of Massachusetts, Amherst. The goal: Run C, C++, Go and Node.js programs as processes in browsers, including LaTeX, GNU Make, Go HTTP servers, and POSIX shell scripts. “Processes are built on top of Web Workers, letting applications run in parallel and spawn subprocesses. System calls include fork, spawn, exec, and wait.” Pipes are supported with pipe(2) enabling developers to compose processes into pipelines. Sockets include support for TCP socket servers and clients, making it possible to run applications like databases and HTTP servers together with their clients in the browser. Browsix comprises two core parts: A kernel written in TypeScript that makes core Unix features (including pipes, concurrent processes, signals, sockets, and a shared file system) available to web applications. Extended JavaScript runtimes for C, C++, Go, and Node.js that support running programs written in these languages as processes in the browser. This seems like an interesting project, although I am not sure how it would be used as more than a toy *** Book Review: PAM Mastery (https://www.cyberciti.biz/reviews/book-review-pam-mastery/) nixCraft does a book review of Michael W. Lucas' “Pam Mastery” Linux, FreeBSD, and Unix-like systems are multi-user and need some way of authenticating individual users. Back in the old days, this was done in different ways. You need to change each Unix application to use different authentication scheme. Before PAM, if you wanted to use an SQL database to authenticate users, you had to write specific support for that into each of your applications. Same for LDAP, etc. So Open Group lead to the development of PAM for the Unix-like system. Today Linux, FreeBSD, MacOS X and many other Unix-like systems are configured to use a centralized authentication mechanism called Pluggable Authentication Modules (PAM). The book “PAM Mastery” deals with the black magic of PAM. Of course, each OS chose to implement PAM a little bit differently The book starts with the basic concepts about PAM and authentication. You learn about Multi-Factor Authentication and why use PAM instead of changing each program to authenticate the user. The author went into great details about why PAM is useful for developers and sysadmin for several reasons. The examples cover CentOS Linux (RHEL and clones), Debian Linux, and FreeBSD Unix system. I like the way the author described PAM Configuration Files and Common Modules that covers everyday scenarios for the sysadmin. PAM configuration file format and PAM Module Interfaces are discussed in easy to understand language. Control flags in PAM can be very confusing for new sysadmins. Modules can be stacked in a particular order, and the control flags determine how important the success or failure of a particular module. There is also a chapter about using one-time passwords (Google Authenticator) for your application. The final chapter is all about enforcing good password policies for users and apps using PAM. The sysadmin would find this book useful as it covers a common authentication scheme that can be used with a wide variety of applications on Unix. You will master PAM topics and take control over authentication for your organization IT infrastructure. If you are Linux or Unix sysadmin, I would highly recommend this book. Once again Michael W Lucas nailed it. The only book you may need for PAM deployment. get “PAM Mastery” (https://www.michaelwlucas.com/tools/pam) *** Reflections on Trusting Trust - Ken Thompson, co-author of UNIX (http://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html) Ken Thompson's "cc hack" - Presented in the journal, Communication of the ACM, Vol. 27, No. 8, August 1984, in a paper entitled "Reflections on Trusting Trust", Ken Thompson, co-author of UNIX, recounted a story of how he created a version of the C compiler that, when presented with the source code for the "login" program, would automatically compile in a backdoor to allow him entry to the system. This is only half the story, though. In order to hide this trojan horse, Ken also added to this version of "cc" the ability to recognize if it was recompiling itself to make sure that the newly compiled C compiler contained both the "login" backdoor, and the code to insert both trojans into a newly compiled C compiler. In this way, the source code for the C compiler would never show that these trojans existed. The article starts off by talking about a content to write a program that produces its own source code as output. Or rather, a C program, that writes a C program, that produces its own source code as output. The C compiler is written in C. What I am about to describe is one of many "chicken and egg" problems that arise when compilers are written in their own language. In this case, I will use a specific example from the C compiler. Suppose we wish to alter the C compiler to include the sequence "v" to represent the vertical tab character. The extension to Figure 2 is obvious and is presented in Figure 3. We then recompile the C compiler, but we get a diagnostic. Obviously, since the binary version of the compiler does not know about "v," the source is not legal C. We must "train" the compiler. After it "knows" what "v" means, then our new change will become legal C. We look up on an ASCII chart that a vertical tab is decimal 11. We alter our source to look like Figure 4. Now the old compiler accepts the new source. We install the resulting binary as the new official C compiler and now we can write the portable version the way we had it in Figure 3. The actual bug I planted in the compiler would match code in the UNIX "login" command. The replacement code would miscompile the login command so that it would accept either the intended encrypted password or a particular known password. Thus if this code were installed in binary and the binary were used to compile the login command, I could log into that system as any user. Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions. Next “simply add a second Trojan horse to the one that already exists. The second pattern is aimed at the C compiler. The replacement code is a Stage I self-reproducing program that inserts both Trojan horses into the compiler. This requires a learning phase as in the Stage II example. First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere. So now there is a trojan'd version of cc. If you compile a clean version of cc, using the bad cc, you will get a bad cc. If you use the bad cc to compile the login program, it will have a backdoor. The source code for both backdoors no longer exists on the system. You can audit the source code of cc and login all you want, they are trustworthy. The compiler you use to compile your new compiler, is the untrustworthy bit, but you have no way to know it is untrustworthy, and no way to make a new compiler, without using the bad compiler. The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect. Acknowledgment: I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics. I can- not find a more specific reference to this document. I would appreciate it if anyone who can supply this reference would let me know. Beastie Bits Custom made Beastie Stockings (https://www.etsy.com/listing/496638945/freebsd-beastie-christmas-stocking) Migrating ZFS from mirrored pool to raidz1 pool (http://ximalas.info/2016/12/06/migrating-zfs-from-mirrored-pool-to-raidz1-pool/) OpenBSD and you (https://home.nuug.no/~peter/blug2016/) Watson.org FreeBSD and Linux cross reference (http://fxr.watson.org/) OpenGrok (http://bxr.su/) FreeBSD SA-16:37: libc (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:37.libc.asc) -- A 26+ year old bug found in BSD's libc, all BSDs likely affected -- A specially crafted argument can trigger a static buffer overflow in the library, with possibility to rewrite following static buffers that belong to other library functions. HardenedBSD issues correction for libc patch (https://github.com/HardenedBSD/hardenedBSD/commit/fb823297fbced336b6beeeb624e2dc65b67aa0eb) -- original patch improperly calculates how many bytes are remaining in the buffer. From December the 27th until the 30th there the 33rd Chaos Communication Congress[0] is going to take place in Hamburg, Germany. Think of it as the yearly gathering of the european hackerscene and their overseas friends. I am one of the persons organizing the "BSD assembly (https://events.ccc.de/congress/2016/wiki/Assembly:BSD)" as a gathering place for BSD enthusiasts and waving the flag amidst the all the other projects / communities. Feedback/Questions Chris - IPFW + Wifi (http://pastebin.com/WRiuW6nn) Jason - bhyve pci (http://pastebin.com/JgerqZZP) Al - pf errors (http://pastebin.com/3XY5MVca) Zach - Xorg settings (http://pastebin.com/Kty0qYXM) Bart - Wireless Support (http://pastebin.com/m3D81GBW) ***
This week on BSDNow, we've got voting news for you (No not that election), a closer look at This episode was brought to you by Headlines ARIN 38 involvement, vote! (http://lists.nycbug.org/pipermail/talk/2016-October/016878.html) Isaac (.Ike) Levy, one of our interview guests from earlier this year, is running for a seat on the 15 person ARIN Advisory Council His goal is to represent the entire *BSD community at this important body that makes decisions about how IP addresses are allocated and managed Biographies and statements for all of the candidates are available here (https://www.arin.net/participate/elections/candidate_bios.pdf) The election ends Friday October 28th If elected, Ike will be looking for input from the community *** LibreSSL not just available but default (DragonFlyBSD) (https://www.dragonflydigest.com/2016/10/19/18794.html) DragonFly has become the latest BSD to join the growing LibreSSL family. As mentioned a few weeks back, they were in the process of wiring it up as a replacement for OpenSSL. With this latest commit, you can now build the entire base and OpenSSL isn't built at all. Congrats, and hopefully more BSDs (and Linux) jump on the bandwagon Compat_43 is gone (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624734.html) RiP 4.3 Compat support.. Well for DragonFly anyway. This commit finally puts out to pasture the 4.3 support, which has been disabled by default in DragonFly for almost 5 years now. This is a nice cleanup of their tree, removing more than a thousand lines of code and some of the old cruft still lingering from 4.3. *** Create your first FreeBSD kernel module (http://meltmes.kiloreux.me/create-your-first-freebsd-kernel-module/) This is an interesting tutorial from Abdelhadi Khiati, who is currently a master's student in AI and robotics I have been lucky enough to participate in Google Summer of Code with the FreeBSD foundation. I was amazed by the community surrounding it which was noob friendly and very helpful (Thank you FreeBSD We will run two storage controllers (ctrl-a, ctrl-b) and a host (cln-1). A virtual SAS drive (da0) of 256 MB is configured as “shareable” in Virtual Media Manager and simultaneously connected with both storage controllers The basic settings are applied to both controllers One interesting setting is: kern.cam.ctl.harole – configures default role for the node. So ctrl-a is set as 0 (primary node), ctrl-b – 1 (secondary node). The role also can be specified on per-LUN basis which allows to distribute LUNs over both controllers evenly. Note, kern.cam.ctl.haid and kern.cam.ctl.ha_mode are read-only parameters and must be set only via the /boot/loader.conf file. Once kern.cam.ctl.ha_peer is set, and the peers connect to each other, the log messages should reflect this: CTL: HA link status changed from 0 to 1 CTL: HA link status changed from 1 to 2 The link states can be: 0 – not configured, 1 – configured but not established and 2 – established Then ctld is configured to export /dev/da0 on each of the controllers Then the client is booted, and uses iscsid to connect to each of the exposed targets sysctl kern.iscsi.failondisconnection=1 on the client is needed to drop connection with one of the controllers in case of its failure As we know that da0 and da1 on the client are the same drive, we can put them under multipathing control: gmultipath create -A HA /dev/da0 /dev/da1 The document them shows a file being copied continuously to simulate load. Because the multipath is configured in ‘active/active' mode, the traffic is split between the two controllers Then the secondary controller is turned off, and iscsi disconnects that path, and gmultipath adapts and sends all of the traffic over the primary path. When the secondary node is brought back up, but the primary is taken down, traffic stops The console on the client is filled with errors: “Logical unit not accessible, asymmetric access state transition” The ctl(4) man page explains: > If there is no primary node (both nodes are secondary, or secondary node has no connection to primary one), secondary node(s) report Transitioning state. > Therefore, it looks like a “normal” behavior of CTL HA cluster in a case of disaster and loss of the primary node. It also means that a very lucky administrator can restore the failed primary controller before timeouts are elapsed. If the primary is down, the secondary needs to be promoted by some other process (CARP maybe?): sysctl kern.cam.ctl.ha_role=0 Then traffic follows again This is a very interesting look at this new feature, and I hope to see more about it in the future *** Is SPF Simply Too Hard for Application Developers? (http://bsdly.blogspot.com/2016/10/is-spf-simply-too-hard-for-application.html) Peter Hansteen asks an interesting question: The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers? He tells a story about trying to file his Norwegian taxes, and running into a bug Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site. The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics. The next time I checked my mail at bsdly.net, I found this bounce: support@altinn.no: SMTP error from remote mail server after RCPT TO:: host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed which means that somebody, somewhere tried to send a message to support@altinn.no, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain. What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails. After having a bunch of other problems, he finally gets a message back from the tax authority support staff: It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported. The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem. Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification? Beastie Bits Looking for a very part-time SysAdmin (https://lists.freebsd.org/pipermail/freebsd-jobs/2016-October/000930.html) If anyone wants to build the latest nodejs on OpenBSD... (https://twitter.com/qb1t/status/789610796380598272) IBM considers donating Power8 servers to OpenBSD (https://marc.info/?l=openbsd-misc&m=147680858507662&w=2) Install and configure DNS server in FreeBSD (https://galaxy.ansible.com/vbotka/freebsd-dns/) bhyve vulnerability in FreeBSD 11.0 (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:32.bhyve.asc) Feedback/Questions Larry - Pkg Issue (http://pastebin.com/8hwDVQjL) Larry - Followup (http://pastebin.com/3nswwk90) Jason - TrueOS (http://pastebin.com/pjfYWdXs) Matias - ZFS HALP! (http://pastebin.com/2tAmR5Wz) Robroy - User/Group (http://pastebin.com/7vWvUr8K) ***
This week on BSDNow, we're going to be hearing about Allan's trip to EuroBSDCon, plus an Interview about “Bro on BSD”! Stay tuned, for your place to This episode was brought to you by Headlines EuroBSDCon 2016 Wrapup Ollivier Robert's Photos from EuroBSDCon (https://assets.keltia.net/photos/EuroBSDCon-2016/) Get your BSDNow die-cut stickers (http://www.jupiterbroadcasting.com/stickers/) NetBSD for newbies - Develop your own Power PC (http://discusscomputerx.blogspot.com/2016/09/netbsd-for-noobies-your-power-laptop.html) We don't get to feature too many stories on NetBSD being deployed as a Power PC (Not PowerPC, you know, a Powerful “PC”), so we jumped at this one. Specifically it starts off with some of the pre-req's that you'll need to get started, such as NetBSD 7.0.1 / amd64, along with some information about which wireless nics you may be using. (NetBSD like other BSD's will give a driver based device name for network interfaces) From there, instructions on how to write your WPA_supplicant config are provided, in order for us to fetch the NetBSD sources and convert to their -STABLE branch. After doing a CVS checkout of the sources, he then provides a walkthrough of doing a kernel compile / install, however it mentions changing the config, but doesn't provide an example of what options were changed. Perhaps to remove drivers we don't need? At this point the rest of the “desktop” setup is pretty straight forward. Some packages are added such as openbox, lxappearance, firefox, etc. To get working sound, firefox requires pulseaudio, which in turn needs dbus, so instructions on getting that service up and running are provided as well. When it's all said and done, you'll end up with your shiny new NetBSD -STABLE desktop (or laptop), bragging rights achieved! *** More about OpenSMTPD 6.0.0 (https://www.poolp.org/tech/posts/2016/09/12/opensmtpd-6-0-0-released/) OpenSMTPd 6.0.0 has just been released “and it's quite different from former releases.” “Unlike most of our releases, it comes out with almost no new feature.”, “Turns out most of the changes are not visible.” Changelog: new fork+reexec model so each process has its own randomized memory space logging format has been reworked a "multi-line response" bug in the LMTP delivery backend has been fixed connections concurrency limits have been bumped artificial delaying in remote sessions have been reduced dhparams option has been removed dhe option has been added, supporting auto and legacy modes smtp engine has been simplified various cosmetic changes, code cleanup and documentation improvement “The OpenSMTPD bootstrap process was quite simple: Upon executation, the parent process would read configuration, build a memory representation of it and would then create a bunch of socketpair() before fork()-ing all of its child processes.” The problem is that this does not take advantage of the new address randomization feature. Each child will have the same memory layout, copied from the parent process “So deraadt@ suggested that if OpenSMTPD would not just fork() children but instead fork() them and reexecute the smtpd binary, then each of the children would have its own randomized memory space.” “The idea itself is neat, however not so trivial to implement because when we reexec the whole "inherit configuration and descriptors" part goes away. It's not just fork and exec, it's fork and exec and figure a way for the parent to pass back all the information and descriptors back to the new post-fork instance so it is the new instance that allocates memory and decides where the information goes.” *** Upgrade a FreeBSD 10.3 Installation with ZFS on Root and Full Disk Encryption to 11.0 (http://ftfl.ca/blog/2016-09-17-zfs-fde-one-pool-conversion.html) While FreeBSD 11.0 is not out yet, Joseph Mingrone has helped me work out and test the instructions for upgrading a FreeBSD 10.3 ZFS on full disk encryption setup (bootpool + zpool) to the new GELIBoot feature, which does not require any unencrypted partitions, just the 128kb bootcode Note: Do not upgrade to FreeBSD 11.0 yet. While some images have landed on the FTP server, they do not contain the final openssl fix and are going to be recreated. Currently, GELIBoot does not support key files, so the first step is to reencrypt the master key with only a passphrase. Next, to avoid GELIBoot picking up encrypted partitions that it does not support, or partitions you do not want decrypted at boot, only partitions with the GELIBoot flag are decrypted, so set the flag on your root partition Then, move the loader, kernel, and other files into /boot on the root filesystem, instead of them living on the bootpool. This allows the kernel to be versioned with boot environments, and is the main purpose of this work Then, install the newer gptzfsboot, as this is required to support GELIBoot The old 2gb bootpool partition is then purposely mislabeled as freebsd-vinum, so it is not picked up by the boot blocks. Later, if the upgrade is successful, this partition can be deleted, and used as addition swap or something In order to boot correctly, you want all boot environments to have the ‘canmount' ZFS property set to ‘noauto' Thank you to Joseph for taking the time to prod me for the information required to write this up, and for testing it and finding all of the issues *** Interview - Michael Shirk - mshirk@daemon-security.com (mailto:mshirk@daemon-security.com) / @shirkdog (https://twitter.com/shirkdog) Running Bro on BSD *** News Roundup FreeBSD based distro for virtual hosting platform and appliance (https://clonos.tekroutine.com/) An interesting new FreeBSD-based project as shown up online, called “ClonOS”, which bills itself as a “free open-source FreeBSD-based platform for virtual environments creation and management” It looks to be leveraging an impressive list of technologies, including Bhyve, Xen, Jails and CBSD / Puppet for management tasks. Among its list of features: ZFS features support; VM cloning, export, import Ethernet SoftSwitch for separated networking jails for lightweight container VNC terminal for VM/containers Templates for VM/containers Configuration management/helpers Multi-node operation Multi-Node? Color me intrigued! Right now it appears to be under heavy development, but we'll reach out to the developer to see if we can get an interview lined up at some point! The Raspberry PI Platform and The Challenges of Developing FreeBSD (https://bsdmag.org/oleksandr_rybalko/) BSDMag recently did an interview with FreeBSD developer Olesandr Rybalko! Oleksandr lives in the Ukraine, and while you may not have heard of him, he has worked on some cool projects for FreeBSD including the new “vt” console driver (Which a lot of people are using now), and ARM/MIPS support. The interview covers some of the work he's done to get the PI support working with FreeBSD: I think, my main help here was a USB OTG driver, which I wrote before for another device (Ralink RT3052), then port it to R-Pi. But it was rewritten by Hans Peter Selasky. I do not know so much about USB as Hans knows. Another useful part of my help is Xorg support. I did a simple Xorg video driver which uses framebuffer exported by virtual terminal subsystem. That is help to many guys to start use RPi as a simple desktop system. He was also asked the question “Why would FreeBSD be good fit for ARM?” FreeBSD is very powerful as a network server. All modern network features in one box, with very fast processing. Another good side of FreeBSD is modularity. It is not required to write code to use some driver that was already written for another system, you can just define it in configuration files (kernel config, kernel hints, FDT). So if you want build a nice, R-Pi based, home server – use FreeBSD. If you want to play with devices attached to R-Pi's GPIO – use FreeBSD. He also discusses his work on the ZRouter project, which is a very light-weight platform for tiny routers / embedded devices. But lastly the RPI comes up again, specifically asking him how interested individuals can get involved. Specifically the wiki.freebsd.org is a great reference point for those intested in getting started with FreeBSD on embedded. The warm community is also a plus! Trying out the FreeBSD powered TrueOS (http://www.phoronix.com/scan.php?page=news_item&px=TrueOS-First-Spin) The folks over at Phoronix have done an early look at the new TrueOS desktop images and given some of their thoughts. First up he gives props to the installer, noting that: The TrueOS desktop installer is basically the same as from the PC-BSD days, just re-branded. Still one of the easiest BSD graphical installers I've dealt with and makes it a breeze for setting up a FreeBSD-on-ZFS system by default. After that they took it for a minimal spin, and thing mostly seem to be working. He mentions some of the default apps (Such as qupzilla and trojita) aren't their favorite, but Lumina has come quite a ways for 1.0, despite a few rough edges still. (We are in the process of changing those default e-mail / browser apps) Lastly the article mentions that it's time to do a more full BSD round-up to see the state of installation of them, which we happen to have next! Trying out 8 BSDs on a modern PC (http://www.phoronix.com/scan.php?page=article&item=trying-8-bsds&num=1) First up was TrueOS again, which no major changes there, easy install and done. From there he tries out DragonFlyBSD, which he mentions that while the installer isn't as easy, it is still one of his favorite BSD's, working with all the hardware they've thrown at it. Next up was GhostBSD, which also has an Easy-To-Use graphical installer similar to TrueOS that made it quick to get loaded and up to the Mate desktop. Also tested was FreeBSD 11.0-RC2, which he mentions was easy to installed, and once done then ‘pkg' could be used to easily get the setup he wanted setup. Turning over to page two we get to the naughty list of BSD's he had troubles with. First up was OpenBSD which he tried 6.0. After installation and first boot, the display kept ‘disappearing' which meant he couldn't get IP information to try SSH'ing into the box. Perhaps a display driver error? NetBSD 7 was up next, where the installer couldn't get past a root device prompt. Most likely trouble finding the install media, which was the same story with MightnightBSD as well. Also tested was “PacBSD” (Formerly ArchBSD) which he did manage to get installed, but not after major fighting with the process. After the process he ran into some issues getting packages up and running, but mentions it may have been bad timing due to them moving to a new server at the time. *** IllumOS imports a modified FreeBSD boot loader to replace grub 0.97 (https://www.listbox.com/member/archive/182181/2016/09/sort/time_rev/page/1/entry/0:1/20160923124232:B7978ED4-81AC-11E6-A6DA-02E3F010038B/) Toomas Soome's work to port the FreeBSD boot loader to IllumOS has been merged into illumos-gate, the upstream repository for all IllumOS distributions Toomas' work has also resulted in a number of commits to FreeBSD, and code sharing in both directions Toomas helped me a lot with the building of the ZFS boot environment listing menu, even though on IllumOS they use a configuration file to list the BEs, rather than interrogating the live zpool like we do in FreeBSD Toomas' work to improve msdosfs and the block cache to speed up booting IllumOS also greatly helped FreeBSD This work means IllumOS can now boot from a RAID-Z (the old grub they used could not), and if the work Toomas has done on FreeBSD is any indication, support for almost all other zpool features is also on the way This work also sets IllumOS on a path to eventually having UEFI boot as well It is good to see this work happening, FreeBSD technology being reused elsewhere, but also the improvements being made for IllumOS are coming back to FreeBSD, often landing upstream first, to make merging them into IllumOS easier. The mailing list post describes how to convert existing systems away from grub, as well as how to opt to remain on grub for a while longer. Grub 0.97 is expected to be removed from IllumOS within a year. *** Beastie Bits A demo of booting CentOS and Windows 10 in FreeBSD Bhyve through VNC headless (https://www.youtube.com/watch?v=8YQQfXqtyaA) This year's anemic output (http://blather.michaelwlucas.com/archives/2762) “PAM Mastery” ebook now out (http://blather.michaelwlucas.com/archives/2771) How-to Install OpenBSD 6.0 plus XFCE desktop and basic applications (https://www.youtube.com/watch?v=oC5D9fenQBs) *** Feedback/Questions Piotr - LibreBoot (http://pastebin.com/yniniNpV) Alan - FreeBSD and PC-BSD (http://pastebin.com/dCNX0yF7) Eduardo - Newcomers (http://pastebin.com/LndNeAYb) Greg - ZFS ACL's (http://pastebin.com/F0y6L6NK) Brian - Laptop Recs (http://pastebin.com/sqMPJGMM) ***
This week on BSDNow, Allan is currently at EuroBSDCon! However due to the magic of video (or time travel), you still get a new episode. (You're Welcome!). Stay tuned This episode was brought to you by Headlines Performance Improvements for FreeBSD Kernel Debugging (http://backtrace.io/blog/blog/2016/08/25/improving-freebsd-kernel-debugging/) “We previously explored FreeBSD userspace coredumps (http://backtrace.io/blog/blog/2015/10/03/whats-a-coredump). Backtrace's debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them.” “A kernel core is typically only generated in exceptional circumstances. Unlike userspace processes, kernel routines cannot fault without sacrificing the machine's availability. This means things like page faults and illegal instructions inside the kernel stop the machine, instead of just one process. At that point, in most cases, it is only usable enough to inspect its state in a debugger, or to generate a core file.” No one likes it when this happens. This is why backtrace.io is focused on being able to figure out why it is happening “A FreeBSD kernel core file can be formatted in several different ways. This depends on which type of dump was performed. Full core dumps are ELF files, similar in structure to userspace core files. However, as RAM size grew, this became more difficult to manage. In 2006, FreeBSD introduced minidumps, which are much smaller without making the core file useless. This has been the default dump type since FreeBSD 6.0.” The article goes into detail on the minidump format, and some basic debugging techniques “Libkvm will first determine whether the virtual address lies within the kernel or direct maps. If it lies in the kernel map, libkvm will consult the page table pages to discover the corresponding physical address. If it lies in the direct map, it can simply mask off the direct map base address. If neither of these applies, the address is illegal. This process is encapsulated by vatopa, or “virtual address to physical address”. Once the physical address is determined, libkvm consults the core file's bitmap to figure out where in the core file it is located.” “minidumps include a sparse bitmap indicating the pages that are included. These pages are dumped sequentially in the last section. Because they are sparse in a not entirely predictable way, figuring the offset into the dump for a particular physical address cannot be reduced to a trivial formula.” The article goes into detail about how lookups against this map are slow, and how they were improved “For typical manual debugger use, the impact of this change isn't noticeable, which is probably why the hash table implementation has been in use for 10 years. However, for any automated debugging process, the extra latency adds up quickly.” “On a sample 8GB kernel core file (generated on a 128GB server), crashinfo improves from 44 seconds to 9 seconds, and uses 30% less memory” “Backtrace began shipping a version of this performance improvement in ptrace in February 2016. This enables us to also offer significantly faster tracing of FreeBSD kernel cores to customers running current and older releases of FreeBSD. On July 17, 2016, our work improving libkvm scaling was committed to FreeBSD/head. It will ship with FreeBSD 12.0.” *** OpenBSD gunzip pipeline tightening (https://www.mail-archive.com/tech@openbsd.org/msg34035.html) OpenBSD has rethought the way they handle package signing Changing from: 1/ fetch data -> 2/ uncompress it -> 3/ check signature -> 4/ process data To: 1/ fetch data -> 2/ check signature -> 3/ uncompress -> 4/ process data “The solution is to move the signature outside of the gzip header” “Now, Since step 1/ is privsep, as long as step 2 is airtight, 3/ and 4/are no longer vulnerable” Guidelines: small, self-contained code to parse simple gzip headers signify-style signature in the gzip comment. Contains checksums of 64K blocks of the compressed archive don't even think about passing the original gzip header through use as a pipeline step: does not need to download full archive to use it, and never ever pass any data to the gunzip part before it's been verified. “Note that afaik we haven't had any hole in our gunzipping process. Well… waiting for an accident to happen is not how we do things. Hopefully, this should prevent future mishaps.” *** OpenVPN On FreeBSD 10.3 (http://ramsdenj.com/2016/07/25/openvpn-on-freebsd-10_3.html) “While trying to setup OpenVPN, I noticed there was no up-to-date information with correct instructions. OpenVPN uses EasyRSA to setup keys, it has recently been changed in version 3. As a result of this, the old steps to configure OpenVPN are no longer correct. I went through the process of setting up a VPN using OpenVPN on FreeBSD 10.3.” I know FreeBSD developer Adrian Chadd complained about this exact problem when he was trying to setup a VPN before attending DEFCON The tutorial walks through the basic steps: Install the needed software Configure EasyRSA Create a CA Generate keys and DH params OpenVPN Server Config OpenVPN Client Config Starting the daemon It even finishes off with bonus instructions on Port Forwarding, Firewalls, and Dynamic DNS *** lsop (https://github.com/606u/lsop) LSOP is the tool a bunch of users have been asking for “a FreeBSD utility to list all processes running with outdated binaries or shared libraries” How does it work? “lsop iterates over all running processes and looks through memory-mapped files with read + execute access; then it checks if those files are still available or have been modified/deleted.” How would you use it? After installing an system update (that doesn't require a reboot to update the kernel), or upgrade your packages, you still need to know which daemons need to be restarted to use the patched libraries and binaries This tool gives you that list Thanks to Bogdan Boyadzhiev for writing this much needed tool *** News Roundup OpenBSD 2016 Fundraising Campaign (http://www.openbsdfoundation.org/campaign2016.html) The OpenBSD fund-raising campaign has given us a status update on the state of 2016. They start by giving us a re-cap of previous years: “2015 was a good year for the foundation financially, with one platinum, one gold, four silver and 3 bronze donors providing half of our total donations. 680 individuals making smaller contributions provided the other half. While the total was down significantly after 2014's blockbuster year, we again exceeded our goal.” As of Sept 5th, they were at approx $115k out of a total goal of 250k. If you are an OpenBSD user, remember to contribute before the end of the year. Small amounts help, and the money of course goes to great causes such as hackathons and running the OpenBSD infrastructure. Update firewall Bad Countries (https://github.com/KaiLoi/update-fw-BC) Network and Systems admins know, sometimes when all else fails you need to break out the HUGE ban-hammer. In this case sometimes entire countries get put on the excrement list until the attacks stop. We have a handy GitHub project today, which will assist you in doing exactly that, enter update-fw-BC. (Update firewall by country) This perl script may be your savior when dealing with instances that require major brute force. It specifically works with IPFW, PF and IPTABLES, which will allow it to run across a variety of BSD's or even Linux. It will ingest a list of IP's that you feed it (perhaps from another tool such as sshguard) and determine what block the IP belongs to, and match according to country. Detailed setup instructions for the various firewalls are included, and some instructions for FreeBSD, although using it on OpenBSD or other $BSD should also be easy to adapt. *** More utilities via moreutils (https://distrowatch.com/weekly.php?issue=20160822#tips) In most BSDs, the “core” set of utilities and commands are just part of the base system, but on Linux, they are usually provided by the “coreutils” package. However, on Linux and now FreeBSD, there is a “moreutils” package, that provides a number of interesting additional basic utilities, including: chronic: Run a task via crontab, and only generate output if the task fails combine: binary AND two text files together, only displaying lines that are in both files errno: look up the text description of a specific error number ifdata: parse out specific information from ifconfig ifne: if-not-empty, only run a command if the output of the pipe is not blank isutf8: determine if a file or stdin contains utf8 lckdo: execute a command with a lock held, to prevent a second copy from spawning mispipe: return the exit code of the first command in a pipe chain, rather than the last parallel: run multiple jobs at once pee: tee standard input to multiple pipes sponge: write standard input to a file, allows you to overwrite a file in place: sort file | sponge file ts: add a timestamp to each line of standard input vidir: edit a directory in vi, great for bulk renames vipe: insert vi into a pipe, edit the content before it is passed to the next command zrun: uncompress the arguments before passing them. Like gzless and friends, but for any command Just goes to show the power of the original UNIX philosophy, chaining together a bunch of small useful tools to do really powerful things *** OpenBSD: SNI support added to libtls, httpd in –current (http://undeadly.org/cgi?action=article&sid=20160823100144) libtls, LibreSSL's improved API to replace the OpenSSL standard, now has a set of functions to implement SNI (Server Name Indication) Until a few years ago, each different SSL/TLS enabled website required a unique IP address, because typical HTTP Virtual Hosting (differentiating which content to serve based on the Host header in the HTTP request), didn't work because the request was encrypted. Finally the TLS standard was updated to include the hostname of the site the user is requesting in the TLS handshake, so the server can return the corresponding certificate, and multiple TLS enabled websites can be hosted on a single IP address The new API includes the ability to provide additional keypairs (via tlsconfigaddkeypair{file,mem}()) And allow the server to determine what servername the client requested viatlsconnservername() This is much easier to use, and therefore safer and less error prone, than the OpenSSL API The libtls API is used in a number of OpenBSD tools, including the httpd *** Beastie Bits Shawn Webb of HardenedBSD joins the OPNSense Core Team (https://opnsense.org/new-core-team-member/) How to install 2.11 BSD on a (simulated) PDP11 (http://vak.ru/doku.php/proj/pdp11/211bsd) OpenBSD Puffy needlepoint pixelart (https://nemessica.tintagel.pl/blog/OpenBSD-Puffy/) PulseAudio has been removed from dports (DragonFly BSD) (http://lists.dragonflybsd.org/pipermail/users/2016-August/313010.html) pfSense 2.4 pre-alpha available for testing, based on FreeBSD 11.0 (https://blog.pfsense.org/?p=2118) Call for Testing - Bhyve HDA Sound Emulation (https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-September/004700.html) *** Feedback/Questions Matthew - ZFS Hole Birth (http://pastebin.com/CrZiDAF0) Hunter - systemd-mount (http://pastebin.com/GztjY4wz) Anonymous - Cool'n'quiet (http://pastebin.com/gG4j4RCi) Nathan - Datacenter (http://pastebin.com/9XgPzMM9) Chuck - OpenBSD w/DO (http://pastebin.com/FM2xYcxh) ***
This week on BSDNow, we have all sorts of interesting news, including a Kernel Fuzzing audit done for OpenBSD, a much improved This episode was brought to you by Headlines Multiple Bugs in OpenBSD Kernel (http://marc.info/?l=oss-security&m=146853062403622&w=2) Its patch Wednesday! (OR last Thursday if you were watching the mailing lists) Jesse Hertz and Tim Newsham (part of the NCC Group calling themselves project Triforce) have been working with the OpenBSD team to fix some newly discovered bugs in the kernel using fuzzing. Specifically they were able to track down several potential methods to corrupt memory or panic the kernel: mmap_panic: Malicious calls to mmap() can trigger an allocation panic or trigger memory corruption. kevent_panic: Any user can panic the kernel with the kevent system call. thrsleeppanic: Any user can panic the kernel with the _thrsleep system Call. thrsigdivertpanic: Any user can panic the kernel with the _thrsigdivert system call. ufsgetdentspanic: Any user can panic the kernel with the getdents system call. mount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when mounting a tmpfs filesystem. unmount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when unmounting a filesystem. tmpfsmknodpanic: Root can panic kernel with mknod on a tmpfs filesystem. This was a great find, and we have a link to more of the results, if you would like to explore them in more detail. NCC Group OpenBSD Kernel fuzzing results (http://www.openwall.com/lists/oss-security/2016/07/14/5) Would like to see more work like this done in all of the BSDs *** Running CockroachDB in a FreeBSD Jail (https://www.cockroachlabs.com/blog/critters-in-a-jar-running-cockroachdb-in-a-freebsd-jail/) The developers behind CockroachDB have written up a nice walkthrough of getting their software to run inside FreeBSD jails. “Manually encapsulating CockroachDB using Linux cgroups is no easy task, which is why tools like Docker exist in the first place. By comparison, running server processes natively in FreeBSD jails is straightforward and robust.” The walkthrough begins with compiling CockroachDB straight from source (A port is pending), which is pretty easy relying upon bash / git / gmake and GO. With the compile finished, the next step will be mounting linprocfs, although that may be going away in the future: “(Note: Linux compatibility files / packages / libraries are not needed further. CockroachDB uses Linux's procfs to inspect system properties via gosigar. If/when gosigar evolves to read FreeBSD properties natively, CockroachDB will not need linprocfs any more.)” With the initial setup complete, the walkthrough then takes us through the process of creating the rc.d script (Which should be included with the port) and ultimately setting up ezjail and deploying CockroachDB within. With the word getting out about jails and their functionality, we hope to see more projects also provide walkthroughs and FreeBSD support natively. Kudos to the CockroachDB team! *** Usermount bugs (https://marc.info/?l=openbsd-announce&m=146854517406640&w=2) kern.usermount, (vfs.usermount on FreeBSD) is a sysctl that can be enabled to allow an unprivileged user to mount filesystems. It is very useful for allowing non-root users to mount a USB stick or other external media. It is not without its dangers though: > “kern.usermount=1 is unsafe for everyone, since it allows any non-pledged program to call the mount/umount system calls. There is no way any user can be expected to keep their system safe / reliable with this feature. Ignore setting to =1, and after release we'll delete the sysctl entirely.” In OpenBSD 6.0 and forward, the setting will no longer work, and root privileges will be required to mount a filesystem If there is a bug in the filesystem driver, the user could potentially exploit that and root the system > “In addition to the patched bugs, several panics were discovered by NCC that can be triggered by root or users with the usermount option set. These bugs are not getting patched because we believe they are only the tip of the iceberg. The mount system call exposes too much code to userland to be considered secure” This is a very pragmatic way of dealing with these issues, as it is not really possible to be sure that EVERY bug has been fixed, and that this feature is no longer an exploit vector usermount being removed from OpenBSD (http://undeadly.org/cgi?action=article&sid=20160715125022) I use this facility in FreeBSD extensively, combined with ZFS permission delegation, to allow non-root users to create and mount new ZFS datasets, and to do replication without requiring any root access There are some safety belts, for instance: the user must own the directory that the new filesystem will be mounted to, so they can't mount to /etc and replace the password file with their own *** Let's Encrypt client from BSD in C (https://kristaps.bsd.lv/letskencrypt/) File this one under the category of “It's about time!”, but Kristaps (Who we've interviewed in the past) has released some new software for interacting with letsencrypt. The header for the project site sums it up nicely: “Be up-front about security: OpenSSL is known to have issues, you can't trust what comes down the pipe, and your private key's integrity is a hard requirement. Not a situation where you can be careless. letskencrypt is a client for Let's Encrypt users, but one designed for security. No Python. No Ruby. No Bash.A straightforward, open source implementation in C that isolates each step of the sequence.” What specifically does it isolate you ask? Right now it is broken down into 6 steps: read and parse an account and domain private key authenticate with the Let's Encrypt server authorise each domain listed for the certificate submit the X509 request receive and serialise the signed X509 certificate request, receive, and serialise the certificate chain from the issuer I don't know about all of you, but I'm going to be switching over one of my systems this weekend. *** News Roundup Videos from the FOSDEM BSD Dev room are now online (https://video.fosdem.org/2016/k4601/) The videos from the BSD Dev room at FOSDEM have been stealthily posted online at some point since last I checked The videos are individually linked from the talks on the Schedule (https://archive.fosdem.org/2016/schedule/track/bsd/) The talk pages also include the slides, which can help you to follow along *** FreeBSD on Jetson TK1 (http://kernelnomicon.org/?p=628) The nVidia Jetson TK1 is a medium sized ARM device that is a big more than your standard Raspberry Pi The device has: NVIDIA 4-Plus-1™ Quad-Core ARM® Cortex™-A15 CPU (2.3 GHz) NVIDIA Kepler GPU with 192 CUDA Cores 2 GB DDR3L x16 Memory with 64-bit Width 16 GB 4.51 eMMC Memory 1 Half Mini-PCIE Slot 1 Full-Size SD/MMC Connector 1 Full-Size HDMI Port 1 USB 2.0 Port, Micro AB 1 USB 3.0 Port, A 1 RS232 Serial Port 1 ALC5639 Realtek Audio Codec with Mic In and Line Out 1 RTL8111GS Realtek GigE LAN 1 SATA Data Port SPI 4 MByte Boot Flash The following signals are available through an expansion port: DP/LVDS Touch SPI 1x4 + 1x1 CSI-2 GPIOs UART HSIC i2c The device costs $192 USD from nVidia or Amazon Oleksandr Tymoshenko (gonzo@freebsd.org) has a post describing what it takes to get FreeBSD running on the Jetson TK1 > “First of all – my TK1 didn't have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader.” They tried using the provided tool, compiled on FreeBSD since it uses libusb, but it gave an error. Falling back to trying from Ubuntu, they got the same error. They then flashed the TK1 with newer firmware, and suddenly, uboot is available. The post then walks through pxe booting FreeBSD on the TK1 The guide then walks through replacing the UBoot with a version compatible with UBLDR, for more features We'll have to wait for another post to get FreeBSD burned onto the device, but at this point, you can reliably boot it without any user interaction I have one of these devices, so I am very interested in this work *** Why we use OpenBSD at VidiGaurd (https://blog.vidiguard.com/why-we-use-openbsd-at-vidiguard-4521f217b2b7#.9r86v742v) VidiGuard (Which makes autonomous drone solutions for security monitoring) has posted an interesting write-up on why they use OpenBSD. Specifically they start by mentioning while they are in business to provide physical security, they just as equally value their data security, especially their customer data. They name 4 specific features that matter to them, starting with Uncompromising Quality and Security: “Over the past 20 years, OpenBSD's focus on uncompromising quality and code correctness has yielded an operating system second-to-none. Code auditing and review is core to the project's development process. The team's focus on security includes integrated cryptography, new security mitigation techniques, and an optional-security-is-no-security stance, making it arguably the most secure operating system available today. This approach pays off in the form of only a few security updates for a given release, compared to other operating systems that might release a handful of updates every week.” High praise indeed! They also mention the sane-defaults, documentation and last but not least, the license as also winning factors in making OpenBSD their operating system of choice. Thanks to VidiGuard for publically detailing the use of BSD, and we hope to see other business follow suit! *** "You can (and should) slow down and learn how things work" – Interview with Dru Lavigne (https://bsdmag.org/dru_lavigne/) If you've been around the BSD community for any length of time, you no doubt have heard of Dru Lavigne (Or perhaps own one of her books!) She was recently interviewed by Luca Ferrari for BSD Magazine and you may find it a fascinating read. The 2nd question asked sounded a lot like our opener to an interview (How did you get into BSD) “ In the mid 90s, I went back to school to learn network and system administration. As graduation grew near and I started looking for a work, I noticed that all the interesting jobs wanted Unix skills. Wanting to increase my skills, and not having any money, I did an Internet search for “Free Unix”. The first hit was freebsd.org. I went to the website and started reading the Handbook and thought “I can do this”. Since I only had access to one computer and wanted to ramp up my skills quickly, I printed out the installation and networking chapters of the Handbook. I replaced the current operating system with FreeBSD and forced myself to learn how to do everything I needed to do on that computer in FreeBSD. It was a painful (and scary) few weeks as I figured out how to transition the family's workflow to FreeBSD, but it was also exhilarating to learn that “yes, I can do this!. Since then, I've had the opportunity to try out or administer the other BSDs, several Linux distros, SCO, and Solaris. I found that the layout, logic, and release engineering process of the BSDs makes the most sense to me and I'm happiest when on a BSD system.” When asked, Dru also had a good response to what challenges potential new UNIX or BSD users may face: “Students who haven't been exposed to open source before are used to thinking of technology in terms of a purchasable brand consisting of “black boxes” that are supposed to “just work”, without having to think about how they work. You can (and should) slow down and learn how things work. It can be a mind shift to learn that the freedom to use and change how something works does exist, and isn't considered stealing. And that learning how something works, while hard, can be fun. BSD culture, in particular, is well suited for those who have the time and temperament to dive into how things work. With over 40 years of freely available source and commit messages, you can dive as deep as you want into learning how things came to be, how they evolved over the years, how they work now, and how they can be improved. There is a diverse range of stuff to choose from: from user tools to networking to memory management to hardware drivers to security mechanisms and so on. There is also a culture of sharing and learning and encouragement for users who demonstrate that they have done their homework and have their own ideas to contribute.” The interview is quite long, and Dru provides fantastic insights into more aspects of BSD in general. Well worth your time to read! *** Beastie Bits: Ed Maste is seeking testing 'withoutgpldtc' (https://twitter.com/ed_maste/status/755474764479672321) “PAM Mastery” tech reviewers wanted (http://blather.michaelwlucas.com/archives/2717) OPNsense 16.7 RC2 (https://opnsense.org/opnsense-16-7-rc2-released/) Jupyter Notebook for bootstrapping Arduino on FreeBSD (https://nbviewer.jupyter.org/github/DadAtH-me/Projects/blob/master/arduino-on-nix.ipynb) The Design and Implementation of the Anykernel and Rump Kernels (second edition) (http://www.fixup.fi/misc/rumpkernel-book/) Complete desktop synchronisation with Unison and FreeBSD jails (xjails) (https://github.com/kbs1/freebsd-synced-xjails) Feedback/Questions Eric - List most popular files (http://pastebin.com/S7u0VeVi) Robroy - ZFS Write Cache (http://pastebin.com/81Zmj0cX) Luis - FreeNAS HW Setup (http://pastebin.com/SfeKR7v2) Emett - Python Followup (http://pastebin.com/wy4ar0YH) Peter - Multicast + Jails (http://pastebin.com/zd2QAu25) ***
Today on the show, we are going to be talking to Jim Brown (of BSD Cert Fame) about his home-brew sprinkler system… Wait for it… This episode was brought to you by Headlines Distrowatch reviews OpenBSD and PCBSD's live upgrade method (http://distrowatch.com/weekly.php?issue=20160620#upgrade) Upgrading… The bane of any sysadmin! Distrowatch has recently done a write-up on the in-place upgrading of various distros / BSDs including PC-BSD and OpenBSD. Lets look first at the PC-BSD attempt, which was done going from 9.2 -> 10. “I soon found trying to upgrade either the base system or pkg would fail. The update manager did not provide details as to what had gone wrong and so I decided to attempt a manual upgrade by following the FreeBSD Handbook as I had when performing a live upgrade of FreeBSD back in May. At first the manual process seemed to work, downloading the necessary patches for FreeBSD 10 and getting me to resolve conflicts between my existing configuration files and the new versions. Part way through, we are asked to reboot and then continue the upgrade process using the freebsd-update command utility. PC-BSD failed to reboot and, in fact, the boot loader no longer found any operating systems to run.” Ouch! I'm not sure on the particular commands used, but to lose the boot-loader indicates something went horribly wrong. There is good news in this though. After the pain experienced in the 9.X upgrade process, 11.0 has been vastly improved to help fix this going forward. The updater is also self-updating, which means future changes to tools such as package can be accounted for in previously released versions. Moving on to OpenBSD, Jesse had much better luck: > “The documentation provided explains how to upgrade OpenBSD 5.8 to version 5.9 step-by-step and the instructions worked exactly as laid out. Upgrading requires two reboots, one to initiate the upgrade process and one to boot into the new version of OpenBSD. Upgrading the base operating system took approximately ten minutes, including the two reboots. Upgrading the third-party packages took another minute or two. The only quirk I ran into was that I had to manually update my repository mirror information to gain access to the new packages available for OpenBSD 5.9. If this step is not done, then the pkg_add package manager will continue to pull in packages from the old repository we set up for OpenBSD 5.8. “ A good read, and they covered some Linux distros such as Mint and OpenMandriva as well, if you want to find out how they fared. *** A curated list of awesome DTrace books, articles, videos, tools and resources (http://awesome-dtrace.com/) The website awesome-dtrace.com compiles a list of resources, including books, articles, videos, tools, and other resources, to help you get the most out of DTrace The list of books includes 2 open source books that are available on the web, and of course Brendan Gregg's official DTrace book There are also cheat sheets, one-liner collections, and a set of DTrace war stories A breakdown of different PID providers and the userspace statically defined tracepoints The videos from DTrace.conf 2008, 2012, and soon 2016 And links to the tools to start using DTrace with your favourite programming language, including Erlang, Node.JS, Perl, PHP, Python, or Ruby There are also DTrace setups for MySQL/MariaDB, and PostreSQL Joyent has even written a mod_usdt DTrace module for the Apache web server This seems like a really good resource, and with the efforts of the new OpenDTrace project, to modernize the dtracetoolkit and make it more useful across the different supported operating systems, there has never been a better time to start learning DTrace *** Installing OpenBSD using a serial console with no external monitor (http://unix.stackexchange.com/questions/292891/how-can-i-install-openbsd-using-the-serial-console-without-external-monitor-wi) Have you found yourself needing to install OpenBSD from USB, but with a twist, as in no external monitor? Well somebody has and asked the question on stackexchange. The answer provided is quite well explained, but in a nut-shell the process involves downloading the USB image and making some tweaks before copying it to the physical media. Specifically with a couple of well-placed echo's into boot.conf, the serial-port can be enabled and ready for use: echo "stty com0 115200" > /mnt/etc/boot.conf echo "set tty com0" >> /mnt/etc/boot.conf + After that, simply boot the box and you are ready to access the serial console and drive the installation as normal! #bsdhacks GSoC 2016 Reports: Split debug symbols for pkgsrc builds (https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug) The NetBSD blog provides a status report on one of the GSoC projects that is nearing its midterm evaluation The project to split debugging data into separate pkgsrc packages, so that users can install the debugging symbols if they need them to debug a failing application The report is very detailed, and includes “A quick introduction to ELF and how debug information are stored/stripped off” It walks through the process of writing a simple example application, compiling it, and dealing with the debug data It includes a number of very useful diagrams, and a summary of what changes needed to be make to the pkgsrc makefile infrastructure With this as a recipe, someone should be able to do something quite similar for FreeBSD's ports tree *** iXsystems iXsystems' TrueNAS Firmware Update Delivers Compelling Performance, Replication, and Graphing Improvements (https://www.ixsystems.com/blog/ixsystems-truenas-firmware-update-delivers-compelling-performance-replication-graphing-improvements/) *** Interview - Jim Brown - jpb@jimby.name (mailto:jpb@jimby.name) FreeBSD+BBB Sprinkler System News Roundup From the past : A Research Unix Reader (http://www.cs.dartmouth.edu/~doug/reader.pdf) A paper by by Douglas McIlroy “Selected pages from the nine research editions of the UNIX® Programmer's Manual illustrate the development of the system” “Accompanying commentary recounts some of the needs, events, and individual contributions that shaped this evolution.” Interesting insight into the evolution of the origin UNIX operating system *** Evolution of C programming practices – Unix 1973–2015 (http://kristerw.blogspot.com/2016/06/evolution-of-c-programming-practices.html) From the author of the recent post we covered, “20 years of NetBSD code bloat”, comes a new post “I found a recent paper that also looks at how the BSD code base has evolved, but from a very different perspective compared to my code-size investigation.” The paper "The Evolution of C Programming Practices: A Study of the Unix Operating System 1973–2015" investigates coding style, and tests seven hypotheses by looking at metrics (line length, number of volatile in the source code, etc.) in 66 releases of Unix from 1973 to 2014. The hypotheses are: > + Programming practices reflect technology affordances (e.g. developers may be more liberal with screen space when using high resolution displays) > + Modularity increases with code size > + New language features are increasingly used to saturation point > + Programmers trust the compiler for register allocation > + Code formatting practices converge to a common standard > + Software complexity evolution follows self correction feedback mechanisms > + Code readability increases and the result is that they seem to be true, as interpreted through the metrics. > “The data points for the releases have somewhat random dates. One issue is that the paper use each release's mean file date (the average of the files' last modification time) instead of the release date (that is why the graphs stop at November 2010, even though FreeBSD 10 was released in 2014). The idea is that this better reflects the age of the code base, but this has the effect of compressing some of the data points (especially the clustering around 1993-1994), and it makes the spline fitting even more suspect.” > “One other problem is that the original data used by the researchers seems to have incorrect timestamps. For example, 4.3BSD Net/1 was released in 1989, but is listed as 1993-12-25 in the paper. The same is true for at least the Net/2 release too, which was released in 1991, but the paper list it as 1993-07-02.” *** [old release pictures] openbsd 2.1 - 5.9, straight from theo's bookshelf. (https://twitter.com/blakkheim/status/747540167112671232) Speaking of old releases, our Producer JT picked up this gem at Southeast Linuxfest this year (https://twitter.com/q5sys/status/748003859012984837) Noah Axon shares a scan of his NetBSD 1.4 disc (https://plus.google.com/+NoahAxon/posts/VsiQhUn3tHb) Jan van den broek shares a pic of his FreeBSD 2.2.5 set (https://plus.google.com/101232368324501316985/posts/4QsaJE2KxXh) *** FreeBSD: Just in Time (https://bsdmag.org/just_in_time/) Another BSDMag goodie this week, we have a small article written by Jonathan Garrido which details their experience switching to FreeBSD for a NTP server. The article is short, but a good read: > “A Few years ago we had a time problem. Suddenly our linux NTP server, for a reason that I still do not know, started to fail giving us a lot of issues within all the equipment and services within our network. After a quick and brief meeting with management, I found out that there was not sufficient budget left for a fancy and well-suited appliance. So, with no time (literally) and no money to spend, I decided to give it a try and utilized a homemade open source solution, and the operating system of choice was FreeBSD 10.0.” “Now, let's pause for a second. You may be thinking, why in the world is this guy doing this, when he has never installed a BSD machine in his life? The answer is very simple; here, in the Dominican Republic, in the heart of the Caribbean, FreeBSD has a very good reputation when it comes to reliability and security. In fact, there is some collective thought within the sysadmin community that says something like: “If you want to deal only once with a service, install it over FreeBSD.”” Jonathan then goes through some of the steps taken to initial deploy NTP services, but with that out of the way, he has a great summary: > “Fascinated with the whole experience, we migrate one of our internal dns servers to a second FreeBSD machine and at the moment of this writing we are testing haproxy, an open source load-balancing proxy into a another server with the same OS. > After all this, no time issues have been reported in the past 2 years, so at least for my environment, FreeBSD came just in time.“ *** Beastie Bits MiniBSD laptop computer (https://hackaday.io/project/643-minibsd-laptop-computer) The state of LibreSSL in FreeBSD (https://attilagyorffy.com/2016/07/02/the-state-of-libressl-in-freebsd/) Justin Sherrill is looking for someone willing to run a Go builder with DragonflyBSD (https://www.dragonflydigest.com/2016/07/01/18372.html) Tiny Unix tools for Windows (https://tinyapps.org/blog/windows/201606040700_tiny_unix_tools_windows.html) OpenBSD's doas added to the FreeBSD Ports Tree (http://www.freshports.org/security/doas/) ubuntuBSD 16.04 to feature a combo of BusyBox and OpenRC, no systemd (http://linux.softpedia.com/blog/ubuntubsd-16-04-will-feature-a-combination-of-busybox-and-openrc-but-no-systemd-505463.shtml) Syncast Podcast 4 : Curl, libcurl and the future of the web, with Daniel Stenberg (http://podcast.sysca.st/podcast/4-curl-libcurl-future-web-daniel-stenberg/) Feedback/Questions Harri - Using beadm / zfssnap (http://pastebin.com/qKeCd63F) Jonathan - bhyve vs Proxmox (http://pastebin.com/EhXDwbWQ) Mohammad - Bhyve gfx passthrough (http://pastebin.com/ZCNk4Bga) Jeremy - Shapshots and more Snapshots (http://pastebin.com/xp7nzEYa) Ron - Microphone (http://pastebin.com/H2xr53CR) ***
On this episode of BSDNow, we will be talking to Glen Barber and Peter Wemm of the FreeBSD RE and Cluster Admin teams! That plus our This episode was brought to you by Headlines 2016 FreeBSD Community Survey (https://www.surveymonkey.com/r/freebsd2016) We often get comments from our listeners, “I'm not a developer, how can I help out”? Well today is your chance to do something. The FreeBSD Foundation has its 2016 Community Survey online, where they are asking for feedback from you! I just did the survey, it'll take you about 5 minutes, but gives you a chance to provide valuable feedback to the foundation about things that are important to you. Be sure to answer in as much detail as possible and the foundation will review and use this feedback for its operations going forward. *** ART, OpenBSDs new routing table, single thread performances (http://www.grenadille.net/post/2016/06/17/ART-single-thread-performances) OpenBSD has changed the way routes are looked up in the kernel as part of their path to an SMP networking stack The “Allotment Routing Table” (ART) is a performance tradeoff, where more memory is used to store the routing table, in exchange for faster lookups With this new arrangement, a full BGP routing table will grow from 130MB to 180MB of memory “ART is a free multibit trie based routing table. To keep it simple, it can be seen as using more memory for fewer CPU cycles. In other words, we get a faster lookup by wasting memory. The original paper (http://www.hariguchi.org/art/art.pdf) presents some performance comparisons between two ART configurations and the BSD Radix. But how does this apply to OpenBSD?” “I asked Hrvoje Popovski to run his packet forwarding test on his Xeon box (E5-2620 v2 @ 2.10GHz, 2400.34 MHz) with ix(4) (82599) interfaces. The test setup consist of three machines with the OpenBSD box in the middle” “The simulations have been performed with an OpenBSD -current from June 9th. The machine is configured with pf(4) disabled in order to force a single route lookup for every IPv4 packet. Based on the result of the lookup the kernel decide if it should forward, deliver or drop the packet” *** BSDCan 2016 Playlist (https://www.youtube.com/playlist?list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC) The complete set of videos from BSDCan is online and ready to be consumed Remember the good-ole days where we would wait months (or years) to get videos posted from conferences? Well, who are we kidding, some conferences STILL do that, but we can't count BSDCan among them. Only two weeks out from this years exciting BSDCan, and all the videos have now landed on YouTube. Granted, this is no substitute for actually being at the conference, but even if you attended you probably missed quite a few of the talks. There are no videos of the hallway track, which is the best part of the conference Except the dinner discussion of course. and don't forget the hacker lounge *** Should you be scared of Unix signals? (http://jvns.ca/blog/2016/06/13/should-you-be-scared-of-signals/) Do you know much about UNIX Signals? Are you afraid of their complexity? Do you know there are signals other than SIGKILL? This article talks about the practical implications of signals from a programming perspective The things you need to consider when dealing with signals Basically, you register a “signal handler”, the function that will be run when a signal arrives As you program is running, if a signal arrives, your program will be interrupted. Its current state will be saved and any system calls in progress will return EINTR (Error, Interrupted), then your signal handler will be run. Once the signal handler is complete, the state of your application will be restored, and execution will resume As long as your program properly handles this interruption, and errors that might result from it (getting EINTR from a read() call, instead of the data you expected), then everything should be fine. Of course, you need to be careful what you do inside your signal handler, as if you modify any variables or state in your application, it might be very confused when it resumes. *** Interview - Glen and Peter- News Roundup Unik - The Unikernel Compilation and Deployment Platform (uses NetBSD's Rump) (https://github.com/emc-advanced-dev/unik) We've talked a bit about NetBSD's RUMP (unikernel) in the past, including articles on how to deploy services using it. Now we have an interesting project which makes the process super-easy, and dare-we-say almost “Docker-Like?” The Unik project has a fairly complete walkthrough right on their GitHub project page, including details on installation and creating your own unikernel containers. In addition, it provides instructions on boot-strapping your own Go/Node.js/Python/Java applications, and supports out of Box VCenter / AWS / Qemu / VirtualBox providers. *** PkgSrc 50th Release Highlights () pkgsrc is celebrating its 50th release, and to highlight this, they have posted a series of interviews from people who have been active in the project pkgsrc 50th release interviews - Jonathan Perkin (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_jonathan) pkgsrc 50th release interviews - Ryo ONODERA (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_ryo) pkgsrc 50th release interviews - Joerg Sonnenberg (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interview_with) pkgsrc 50th release interviews - Sevan Janiyan (https://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_sevan) *** Migrating to FreeBSD from Solaris 11 (http://justinholcomb.me/blog/2016/02/28/migration-to-freebsd-part1.html) Part 2 (http://justinholcomb.me/blog/2016/03/12/migration-to-freebsd-part2.html) Part 3 (http://justinholcomb.me/blog/2016/03/19/migration-to-freebsd-part3.html) Part 4 (http://justinholcomb.me/blog/2016/03/26/migration-to-freebsd-part4.html) Part 5 (http://justinholcomb.me/blog/2016/04/03/migration-to-freebsd-part5.html) *** How to chroot www/firefox on NetBSD (https://github.com/alnsn/localpkgsrc/tree/master/firefox-chroot) Looking for a jail-like method of running FireFox on NetBSD? (Or possibly other BSDs?) We have a github repo with details on how to setup and run FireFox using a chroot using a “webuser” account for safety. Think of this as a jail alternative, may be useful on systems with no jail support. Of interest is the method used to do X forwarding. It uses Xorg TCP listen option (which is often off by default for security reasons). Perhaps SSH X forwarding would be a better alternative. (Or nullfs mounts of /tmp) *** Beastie Bits Tredly - V1 Release Candidate (https://github.com/tredly/tredly/releases/tag/v1.0.0-rc.1) Call for Testing - ypldap testing against OpenLDAP and Microsoft Active Directory (http://lists.freebsd.org/pipermail/freebsd-current/2016-June/061775.html) BSD Magazine, June 2016 Out Now (https://bsdmag.org/) Hammer2 - Add xxhash to H2 and throw in debug stuff for performance testing (http://lists.dragonflybsd.org/pipermail/commits/2016-June/500610.html) chyves pre-announcement (http://justinholcomb.me/blog/2016/06/14/chyves-project-preannouncement.html) *** Feedback/Questions Michael - Versioning (http://pastebin.com/1hpGrmuL) Michael - Removing Encryption (http://pastebin.com/2PkrMGGx) Bostjan - PC-BSD Questions (http://pastebin.com/q5VdmNxG) Fong - ZFS Rollback (http://pastebin.com/2aedLV7d) Jochen - Docker on FBSD (http://pastebin.com/dneVZkXc) ***
This week on the show, Allan and I have gotten a bit more sleep since AsiaBSDCon, which is excellent since there is a LOT of news to cover. That plus our interview with Ports SecTeam member Mark Felder. So keep it This episode was brought to you by Headlines FreeNAS 9.10 Released (http://lists.freenas.org/pipermail/freenas-announce/2016-March/000028.html) OS: The base OS version for FreeNAS 9.10 is now FreeBSD 10.3-RC3, bringing in a huge number of OS-related bug fixes, performance improvements and new features. +Directory Services: You can now connect to large AD domains with cache disabled. +Reporting: Add the ability to send collectd data to a remote graphite server. +Hardware Support: Added Support for Intel I219-V & I219-LM Gigabit Ethernet Chipset Added Support for Intel Skylake architecture Improved support for USB devices (like network adapters) USB 3.0 devices now supported. +Filesharing: Samba (SMB filesharing) updated from version 4.1 to 4.3.4 Added GUI feature to allow nfsv3-like ownership when using nfsv4 Various bug fixes related to FreeBSD 10. +Ports: FreeBSD ports updated to follow the FreeBSD 2016Q1 branch. +Jails: FreeBSD Jails now default to a FreeBSD 10.3-RC2 based template. Old jails, or systems on which jails have been installed, will still default to the previous FreeBSD 9.3 based template. Only those machinesusing jails for the first time (or deleting and recreating their jails dataset) will use the new template. +bhyve: ++In the upcoming 10 release, the CLI will offer full support for managing virtual machines and containers. Until then, the iohyve command is bundled as a stop-gap solution to provide basic VM management support - *** Ubuntu BSD's first Beta Release (https://sourceforge.net/projects/ubuntubsd/) Under the category of “Where did this come from?”, we have a first beta release of Ubuntu BSD. Specifically it is Ubuntu, respun to use the FreeBSD kernel and ZFS natively. From looking at the minimal information up on sourceforge, we gather that is has a nice text-based installer, which supports ZFS configuration and iSCSI volume creation setups. Aside from that, it includes the XFCE desktop out of box, but claims to be suitable for both desktops and servers alike right now. We will keep an eye on this, if anybody listening has already tested it out, maybe drop us a line on your thoughts of how this mash-up works out. *** FreeBSD - a lesson in poor defaults (http://vez.mrsk.me/freebsd-defaults.txt) Former BSD producer, and now OpenBSD developer, TJ, writes a post detailing the defaults he changes in a fresh FreeBSD installation Maybe some of these should be the defaults While others are definitely a personal preference, or are not as security related as they seem A few of these, while valid criticisms, but some are done for a reason Specifically, the OpenSSH changes. So, you're a user, you install FreeBSD 10.0, and it comes with OpenSSH version X, which has some specific defaults As guaranteed by the FreeBSD Project, you will have a nice smooth upgrade path to any version in the 10.x branch Just because OpenSSH has released version Y, doesn't mean that the upgrade can suddenly remove support for DSA keys, or re-adding support for AES-CBC (which is not really weak, and which can be hardware accelerated, unlikely most of the replacements) “FreeBSD is the team trying to increase the risk.” Is incorrect, they are trying to reduce the impact on the end user Specifically, a user upgrading from 10.x to 10.3, should not end up locked out of their SSH server, or otherwise confronted by unexpected errors or slowdowns because of upstream changes I will note again, (and again), that the NONE cipher can NOT allow a user to “shoot themselves in the foot”, encryption is still used during the login phase, it is just disabled for the file transfer phase. The NONE cipher will refuse to work for an interactive session. While the post states that the NONE cipher doesn't improve performance that much, it infact does In my own testing, chacha20-poly1305 1.3 gbps, aes128-gcm (fastest) 5.0 gbps, NONE cipher 6.3 gbps That means that the NONE cipher is an hour faster to transfer 10 TB over the LAN. The article suggests just removing sendmail with no replacement. Not sure how they expect users to deliver mail, or the daily/weekly reports Ports can be compiled as a regular user. Only the install phase requires root for ntpd, it is not clear that there is an acceptable replacement yet, but I will not that it is off by default In the sysctl section, I am not sure I see how enabling tcp blackhole actually increases security at all I am not sure that linking to every security advisory in openssl since 2001 is actually useful Encrypted swap is an option in bsdinstall now, but I am not sure it is really that important FreeBSD now uses the Fortuna PRNG, upgraded to replace the older Yarrow, not vanilla RC4. “The resistance from the security team to phase out legacy options makes mewonder if they should be called a compatibility team instead.” I do not think this is the choice of the security team, it is the ABI guarantee that the project makes. The stable/10 branch will always have the same ABI, and a program or driver compiled against it will work with any version on that branch The security team doesn't really have a choice in the matter. Switching the version of OpenSSL used in FreeBSD 9.x would likely break a large number of applications the user has installed Something may need to be done differently, since it doesn't look like any version of OpenSSL, (or OpenSSH), will be supported for 5 years ever again *** ZFS Raidz Performance, Capacity and Integrity (https://calomel.org/zfs_raid_speed_capacity.html) An updated version of an article comparing the performance of various ZFS vdev configurations The settings users in the test may not reflect your workload If you are benchmarking ZFS, consider using multiple files across different datasets, and not making all of the writes synchronous Also, it is advisable to run more than 3 runs of each test Comparing the numbers from the 12 and 24 disk tests, it is surprising to see that the 12 mirror sets did not outperform the other configurations. In the 12 drive tests, the 6 mirror sets had about the same read performance as the other configurations, it is not clear why the performance with more disks is worse, or why it is no longer in line with the other configurations More investigation of this would be required There are obviously so other bottlenecks, as 5x SSDs in RAID-Z1 performed the same as 17x SSDs in RAID-Z1 Interesting results none the less *** iXSystems FreeNAS Mini Review (http://www.nasanda.com/2016/03/ixsystems-freenas-mini-nas-device-reviewed/) Interview - Mark Felder - feld@freebsd.org (mailto:feld@freebsd.org) / @feldpos (https://twitter.com/feldpos) Ports, Ports and more Ports DigitalOcean Digital Ocean's guide to setting up an OpenVPN server (https://www.digitalocean.com/community/tutorials/how-to-configure-and-connect-to-a-private-openvpn-server-on-freebsd-10-1) News Roundup AsiaBSDCon OpenBSD Papers (http://undeadly.org/cgi?action=article&sid=20160316153158&mode=flat&count=0) + Undeadly.org has compiled a handy list of the various OpenBSD talks / papers that were offered a few weeks ago at AsiaBSDCon 2016. Antoine Jacoutot (ajacoutot@) - OpenBSD rc.d(8) (slides | paper) Henning Brauer (henning@) - Running an ISP on OpenBSD (slides) Mike Belopuhov (mikeb@) - Implementation of Xen PVHVM drivers in OpenBSD (slides | paper) Mike Belopuhov (mikeb@) - OpenBSD project status update (slides) Mike Larkin (mlarkin@) - OpenBSD vmm Update (slides) Reyk Floeter (reyk@) - OpenBSD vmd Update (slides) Each talk provides slides, and some the papers as well. Also included is the update to ‘vmm' discussed at bhyveCon, which will be of interest to virtualization enthusiasts. *** Bitcoin Devs could learn a lot from BSD (http://bitcoinist.net/bitcoin-devs-could-learn-a-lot-from-bsd/) An interesting article this week, comparing two projects that at first glance may not be entirely related, namely BitCoin and BSD. The article first details some of the woes currently plaguing the BitCoin development community, such as toxic community feedback to changes and stakeholders with vested financial interests being unable to work towards a common development purpose. This leads into the crux or the article, about what BitCoin devs could learn from BSD: First and foremost, the way code is developed needs change to stop the current negative trend in Bitcoin. The FreeBSD project has a rigid internal hierarchy of people with write access to their codebase, which the various Bitcoin implementations also have, but BSD does this in a way that is very open to fresh eyes on their code, allowing parallel problem solving without the petty infighting we see in Bitcoin. Anyone can propose a commit publicly to the code, make it publicly available, and democratically decide which change ends up in the codebase. FreeBSD has a tiny number of core developers compared to the size of their codebase, but at any point, they have a huge community advancing their project without hard forks popping up at every small disagreement. Brian Armstrong commented recently on this flaw with Bitcoin development, particularly with the Core Devs: “Being high IQ is not enough for a team to succeed. You need to make reasonable tradeoffs, collaborate, be welcoming, communicate, and be easy to work with. Any team that doesn't have this will be unable to attract top talent and will struggle long term. In my opinion, perhaps the biggest risk in Bitcoin right now is, ironically, one of the things which has helped it the most in the past: the Bitcoin Core developers.” A good summary of the culture that could be adopted is summed up as follows: The other thing Bitcoin devs could learn from is the BSD community's adoption of the Unix Design philosophy. Primarily “Worse is Better,” The rule of Diversity, and Do One Thing and Do It Well. “Worse is Better” emphasizes using extant functional solutions rather than making more complex ones, even if they would be more robust. The Rule of Diversity stresses flexibility of the program being developed, allowing for modification and different implementations without breaking. Do one Thing and Do it well is a mantra of the BSD and Unix Communities that stresses modularity and progress over “perfect” solutions. Each of these elements help to make BSD a wildly successful open source project with a healthy development community and lots of inter-cooperation between the different BSD systems. While this is the opposite of what we see with Bitcoin at present, the situation is salvageable provided changes like this are made, especially by Core Developers. All in all, a well written and interesting take on the FreeBSD/BSD project. We hope the BitCoin devs can take something useful from it down the road. *** FreeBSD cross-compiling with gcc and poudriere (http://ben.eficium.net/2016/03/freebsd-cross-compiling-with-gcc.html) Cross-Compiling, always a challenge, has gotten easier using poudriere and qemu in recent years. However this blog post details some of the particular issues still being face when trying to compile some certain ports for ARM (I.E. rPi) that don't play nicely with FreeBSD's default CLANG compiler. The writer (Ben Slack) takes us through some of the work-arounds he uses to build some troublesome ports, namely lsof and libatomic_ops. Note this is not just an issue with cross compile, the above mentioned ports also don't build with clang on the Pi directly. After doing the initial poudriere/qemu cross-compile setup, he then shows us the minor tweaks to adjust which compiler builds specific ports, and how he triggers the builds using poudriere. With the actual Makefile adjustment being so minor, one wonders if this shouldn't just be committed upstream, with some if (ARM) - USE_GCC=yes type conditional. *** Nvidia releases new Beta graphics driver for FreeBSD (https://devtalk.nvidia.com/default/topic/925607/unix-graphics-announcements-and-news/linux-solaris-and-freebsd-driver-364-12-beta-/) Added support for the following GPUs: GeForce 920MX & GeForce 930MX Added support for the Vulkan API version 1.0. Fixed a bug that could cause incorrect frame rate reporting on Quadro Sync configurations with multiple GPUs. Added a new RandR property, CscMatrix, which specifies a 3x4 color-space conversion matrix. Improved handling of the X gamma ramp on GF119 and newer GPUs. On these GPUs, the RandR gamma ramp is always 1024 entries and now applies to the cursor and VDPAU or workstation overlays in addition to the X root window. Fixes for bugs and added several other EGL extensions *** Beastie Bits New TN Bug started (http://knoxbug.org/) DragonFlyBSD Network/TCP Performance's gets a bump (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/4a43469a10cef8c17553c342aab9d73611ea7bc8?utm_source=anzwix) FreeBSD Foundation introduces a new website and logo (https://www.freebsdfoundation.org/blog/introducing-a-new-look-for-the-foundation/) Our producer made these based on the new logo: http://q5sys.sh/2016/03/a-new-freebsd-foundation-logo-means-its-time-for-some-new-wallpapers/ http://q5sys.sh/2016/03/pc-bsd-and-lumina-desktop-wallpapers/ https://github.com/pcbsd/lumina/commit/60314f46247b7ad6e877af503b3814b0be170da8 IPv6 errata for 5.7/5.8, pledge errata for 5.9 (http://undeadly.org/cgi?action=article&sid=20160316190937&mode=flat) Sponsoring “PAM Mastery” (http://blather.michaelwlucas.com/archives/2577) A visualization of FreeBSD commits on GitHub for 2015 (https://rocketgraph.com/s/v89jBkKN4e-) The VAX platform is no more (http://undeadly.org/cgi?action=article&sid=20160309192510) Feedback/Questions Hunter - Utils for Blind (http://slexy.org/view/s20KPYDOsq) Chris - ZFS Quotas (http://slexy.org/view/s2EHdI3z3L) Anonymous - Tun, Tap and Me! (http://slexy.org/view/s21Nx1VSiU) Andrew - Navigating the BSDs (http://slexy.org/view/s2ZKK2DZTL) Brent - Wifi on BSD (http://slexy.org/view/s20duO29mN) ***
This week on BSDNow, Allan is back from the Storage Summit in Silicon Valley! We are going to get his thoughts on how the conference went, plus bring you the latest ZFS info discussed. That plus the usual BSD news is This episode was brought to you by Headlines OpenBSD website operators urged to fix mind-alteringly bad bug (http://www.theregister.co.uk/2016/02/21/openbsd_website_operator_patch_now_for_the_sake_of_your_sanity/?mt=1456206806399) We start off a bit light-hearted this week, with the important, breaking news that finally a long-standing OpenBSD bug has been addressed for the HTTP daemon. Specifically? It changes the default 404 page fonts away from Comic Sans, to a bit more crowd-pleasing alternative: “For some reason the httpd status pages (e.g. 404) use the Comic Sans typeface. This patch removes comic sans and sets the typeface to the default sans-serif typeface of the client. “This lowers the number of people contacting website maintainers with typeface complaints bordering on harassment”. Operators running HTTPD are highly encouraged to update their systems to the latest code, right now……... No seriously, we are waiting for you. Get it done now and then we'll continue with the show. Registration for AsiaBSDCon 2016 is now open + Talk Schedule (https://2016.asiabsdcon.org/registration/?lang=en) After a few delays, the registration for AsiaBSDCon has now opened! The conference starts in less than two weeks! now, so be sure to get signed up ASAP. In addition the schedule has been posted, and here's some of the highlights of this year's conference. In addition to FreeBSD and NetBSD dev summits on the first two days, we have some excellent tutorials being given this year by Kirk, Gnn, Dru and more! (https://2016.asiabsdcon.org/program.html.en) The regular paper talks also have lots of good ones this year, including this crazy encrypted boot loader one given by our very own Allan Jude! *** OPENBSD ON AWS : AN UNEXPECTED JOURNEY (http://blog.d2-si.fr/2016/02/15/openbsd-on-aws/?hn) We have a blog post from Antoine Jacoutot, talking about the process of getting OpenBSD up and running in AWS It starts with his process of creating an AMI from scratch, which ended up not being that bad: create and loopback-mount a raw image containing a UFS filesystem extract the OpenBSD base sets (which are just regular tarballs) and kernel enable console output (so that one could “aws ec2 get-console-output”) install the boot loader on the image then use the ec2 tools to import the RAW image to S3, convert it into a volume (ec2-import-volume) which we can snapshot (ec2-create-snapshot) and create an AMI from (ec2-register) The blog post also has a link to a script which automates this process, so don't be daunted if you didn't quite follow all of that. Thanks to the recently landed DomU support, the final pieces of the puzzle fell into place, allowing OpenBSD to function as a proper guest (with networking!) Next it details the process of injecting a public SSH key into the instances for instant remote access. An ec2-init.sh script was created (also on github) which does the following: setting the hostname installing the provided SSH public key to /root/.ssh/authorized_keys executing user-data (if it starts with a shebang) displaying the host SSH fingerprints on the console (to match cloud-init) With that done, OpenBSD is pretty much AWS ready! He then gives a brief walkthrough of setting up nginx for new users, but if you've already done this before then the instance is ready for you to hacking on. Start thinking of ideas for things with FreeBSD for Google's 2016 Summer of Code (https://wiki.freebsd.org/SummerOfCodeIdeas) Students and Developers, listen up! It's time to start thinking about GSoC again, and FreeBSD is looking to update its project ideas page. There's some good ones on the list, plus ones that should be pruned (such as GELI boot), but now is the time to start adding new ones before we get too deep into the process. This goes for the other BSD's as well, start thinking about your proposals, or if you are developer, which projects would be a good fit for mentoring. (Improving the Linux Compat layer is one I think should be done!) Guide to getting started with kernel hacking (https://wiki.freebsd.org/Graphics/Getting%20started%20with%20kernel%20projects) One of the things that's been asked frequently is how to contribute towards the efforts to bring updated DRM / X drivers to the FreeBSD kernel. Jean-Sébastien Pédron has started a great guide on the Wiki which details how to get started with the porting effort, and that developers need not be afraid of helping. *** Storage Summit Roundup Earlier this week a number of developers from FreeBSD, as well as various vendors that use FreeBSD, or provide products used with FreeBSD met for a Storage Summit (https://wiki.freebsd.org/201602StorageSummit), to discuss the future of these technologies The summit was co-located with the USENIX FAST (Filesystems And Storage Technologies) conference The summit was sponsored by the FreeBSD Foundation and FlightAware After a short introduction, the event opened with a Networking Synergy panel The focus of this panel was to see if there were techniques and lessons learned in improving the networking stack over the last 10 years that could be applied to improving the storage stack A lot of time was spent discussing issues like multi-queue support, CPU scheduling, and ways to modernize the stack CAM Scheduling & Locking Revamp (https://wiki.freebsd.org/201602StorageSummit/CAM) No notes posted User Space Storage Stack (https://wiki.freebsd.org/201602StorageSummit/UserSpace) One of the user space storage stacks discussed was Diskmap Like netmap, but for disks (diskmap) Kernel bypass for accessing disks Ilias Marinos, who is working on diskmap at Cambridge University, described diskmap to the group A design discussion then followed in which the memory management was covered as that's an issue for any sort of "IO" map system Action Items: Discuss with Luigi the idea of code merges Need a reset path API Kernel buffer mapping for reliability Support for other interfaces (SATA/SCSI) GEOM layer adaptation Adapting to New Storage Technologies (https://wiki.freebsd.org/201602StorageSummit/NewStorageTechnologies) This working group was led by Adrian Palmer, from Seagate SMR Persistent Memory Session 1: Device Identification and the structural requirements Agenda: We'll look over the Identification nuances and what needs to change to support the structure. Support for IO order guarantees, forward-write only requirements, new commands and topology. Dig into CAM and GEOM layers. Solutions should be fast and have as few code paths as possible Results: Small audience. We talked about zoned characteristics, and how it can be used in various workloads, projected to be implemented in years Session 2: Information dissemination and consumption Agenda: Where and how will information from the report_zones command be gathered, stored, combined and used. This will include userspace storage and multi-volume management. Will CAM store this data, or will GEOM? How frequently will this need to be queried/updated/verified from the drive? Results: Merged with ZFS working group to discuss SMR. Came up with idea that could be implemented as circular buffer zone type. Began to discuss solutions among developers ZFS (https://wiki.freebsd.org/201602StorageSummit/ZFS) During the first session we discussed how to improve dedup support + A dedup throttle or cap was discussed. When the size of the DDT grows beyond this size, new entries would not be deduped. An alternative to this was also discussed, where when the DDT reached the cap size, it would remove a random entry with only a single reference from the DDT to make room for the new entry. When a block is going to be freed, if it is not found in the DDT, it is assumed to have only 1 reference, and removed. There was also discussion of replacing the DDT with an in-memory hash table and a “log” of increment/decrement operations, that is periodically compacted. The hash table is recreated from the log at pool import time. This would reduce the in-memory footprint of the DDT, as well as speed up all write operations as adding an entry to the dedup log will be less expensive than updating the DDT. There was also discussion of using dedicated device(s) for the DDT, either using the DDT on SSD work by Nexenta, or the Metadata Classes work by Intel The first session also discussed Secure Delete and related things The desire for an implementation of TRIM that uses the “secure erase” functionality provided by some disks was expressed Overwriting sectors with patterns of garbage may be insufficient because SSDs may internally remap where a specific LBA physically resides The possibility of using something like the “eager zero” feature to periodically write zeros over all free blocks in the pool to erase any lingering data fragments Problems with the FreeBSD TRIM implementation were discussed, as well as looking at ways to implement the new ZFS TRIM implementation on FreeBSD ABD (ARC Buf Data) was discussed, a new design that lessens the requirement for contiguous memory. Only a small area of contiguous blocks is reserved at boot, and compressed ARC blocks are constructed of scatter-gather lists of individual pages The second session combined with the SMR group and talked about SMR support in ZFS Later in the second session ZFS Encryption was also discussed, mostly with a focus on what the use cases are The third session combined all of the groups for an overview of upcoming ZFS features including device removal and channel programs There was also a request for code review, for mostly finished projects like Persistent L2ARC, Writeback cache, and Large dnode support Hallway Track ZFS / VFS Interaction Adrian Palmer has been a FreeBSD hobbyist since FreeBSD 7, and I think I managed to convince him to start contributing *** News Roundup One Week with NetBSD 7.0: Back to Unix basics (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-back-to-unix.html) The author of this blog series is sending a week using NetBSD 7.0, following a previous series on Solaris 10 “This is actually familiar territory, as I've been using BSD variants almost exclusively since 2006. My recent SunOS explorations were triggered last summer by OpenBSD having choked on my current laptop's NVIDIA card, and from what I could see at the time, FreeBSD had the same problem, although I now know NVIDIA drivers exist for that system. The thing that keeps me from going all-in with FreeBSD 10.x, however, is the fact that Firefox crashes and leaves "core dump" messages in its wake, and I'm just not a Chrome kinda guy.” “For those with a catholic taste in Unix, NetBSD is a keg party at the Vatican. If you're an absolute Unix beginner, or have been living on Ubuntu-based Linux distros for too long, then you may feel stranded at first by NetBSD's sparseness. You'll find yourself staring into the abyss and seeing only a blinking cursor staring back. If you have the presence of mind to type startx, you'll be greeted by twm, a window manager offering little more than an xterm window with the same blinking cursor until you learn how to configure the .twmrc file to include whatever applications you want or need in the right-click menu.” “As for NetBSD itself, I can't think of any major productivity applications that can't be installed, and most multimedia stuff works fine.” Issues the author hopes to sort out in later posts: Audio playback (youtube videos in Firefox) Wireless Flash Digital Camera SD Card readability, video playback Audacity A “fancy” desktop like Gnome 2, KDE, or xfce In a follow-up post (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-libreoffice.html), the author got LibreOffice installed and sorted out the audio issues they were having In a later follow-up (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-mixed-review-of.html) XFCE is up and running as well *** ZFS is for Containers in Ubuntu 16.04 (http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html) As you may have heard, Ubuntu 16.04 will include ZFS -- baked directly into Ubuntu -- supported by Canonical “ZFS one of the most beloved features of Solaris, universally coveted by every Linux sysadmin with a Solaris background. To our delight, we're happy to make to OpenZFS available on every Ubuntu system.” What does “supported by Canonical” mean? “You'll find zfs.ko automatically built and installed on your Ubuntu systems. No more DKMS-built modules” “The user space zfsutils-linux package will be included in Ubuntu Main, with security updates provided by Canonical” The article then provides a quick tutorial for setting up Linux Containers (LXC) backed by ZFS In the example, ZFS is backed by a file on the existing disk, not by a real disk, and with no redundancy However, the setup script seems to support using real block devices The Software Freedom Conservancy (https://sfconservancy.org/) is expected to issue a statement detailing their opinion on the legalities and licensing issues of bundling ZFS with Linux. *** Polling is a Hack: Server Sent Events (EventSource) with gevent, Flask, nginx, and FreeBSD (http://hypatia.software/2016/01/29/polling-is-a-hack-server-sent-events-eventsource-with-gevent-flask-nginx-and-freebsd/) A tutorial on setting up ‘Server-Sent Events', also know as EventSource in javascript, to notify website clients of new data, rather than having the javascript constantly poll for new data. The setup uses FreeBSD, nginx, gevent, Python, and the Flask framework The tutorial walks through setting a basic Python application using the Flask framework Then setting up the client side in Javascript Then for the server side setup, it covers installing and configuring nginx, and py-supervisor on FreeBSD The tutorial also includes links to additional resources and examples, including how to rate limit the Flash application *** Why FreeBSD? (http://www.aikchar.me/blog/why-freebsd.html) An excellent article written by Hamza Sheikh, discussing why FreeBSD is now his clear choice for learning UNIX. The article is pretty well written and lengthy, but has some great parts which we wanted to share with you: There were many rough edges in the Linux world and some of them exist even today. Choosing the right distribution (distro) for the task at hand is always the first and most difficult decision to make. While this is a strength of the Linux community it is also its weakness. This is exacerbated with the toxic infighting within the community in the last few years. A herd of voices believes it is their right to bring down a distro community because it is not like their distro of choice. Forking upstream projects has somehow become taboo. Hurling abuse in mailing lists is acceptable. Helping new users is limited to lambasting their distro of choice. Creating conspiracy theories over software decisions is the way to go. Copyleft zealots roam social media declaring non-copyleft free software heretic abominations. It all boils down to an ecosystem soured by the presence of maniacs who have the loudest voices and they seem to be everywhere you turn. Where is the engineering among all this noise? Btrfs - baking for a long time - is still nowhere near ZFS in stability or feature parity. systemd is an insatiable entity that feeds on every idea in sight and just devours indiscriminately. Wayland was promised years ago and its time has yet to arrive. Containers are represented by Docker that neither securely contains applications nor makes them easy to manage in production. Firewalling is dithering between firewalld, nftables, etc. SystemTap cannot match DTrace. In the same time span what do various BSDs offer? pf, CARP, ZFS, Hammer, OpenSSH, jails, pkgsrc, (software) ports, DTrace, hardware portability; just to name a few. Few would deny that BSDs have delivered great engineering with free software licenses to the entire world. To me they appear to be better flag bearers of free software with engineering to back it. He then goes through some of the various BSD's and the specifics on why FreeBSD was the logical choice for his situation. But at the end has a great summary on the community as a whole: Finally - and maybe repeating myself here - I have nothing but praise for the community. Be it BSD Now, mailing lists, Reddit, Twitter, LFNW, or SeaGL, people have encouraged me, answered my questions, and filed bugs for me. I have been welcomed and made a part of the community with open arms. These reasons are (good) enough for me to use FreeBSD and contribute to it. BeastieBits OPNsense 16.1.3 released (https://opnsense.org/opnsense-16-1-3-released/) Copies of "FreeBSD Mastery: Specialty Filesystems" seen in the wild (https://twitter.com/Savagedlight/status/700001944547491842) pfsense training available in Europe (http://www.netgate.com/training/) LiteBSD now has 50 ports in its ports tree (https://github.com/ibara/LiteBSD-Ports) Ports tree locked for OpenBSD 5.9 (http://marc.info/?l=openbsd-ports&m=145615281431064&w=2) “FreeBSD Filesystem Fun” at March semibug (http://blather.michaelwlucas.com/archives/2556) Event #46 — Embedded Platforms (BSD, OpenWRT, Plan 9 & Inferno) (http://oshug.org/event/46) Feedback/Questions Frank - ZFS RAM? (http://slexy.org/view/s21lcCKrSB) David - ARM Porting (http://slexy.org/view/s204lxjvlq) Johnny - Lumina Default? (http://slexy.org/view/s2xMiSNLYn) Adam - PC-BSD Install and Q's (http://slexy.org/view/s214gJbLwD) Jeremy - Video Card Q (http://slexy.org/view/s20UNyzEeh) ***
Coming up this week, we will be talking to John Marino about his work on the ports-mgmt utility “Synth” and the cross-pollination between DragonFly and FreeBSD. That plus the latest news and your email here on This episode was brought to you by Headlines glibc and the BSDs (https://blog.des.no/2016/02/freebsd-and-cve-2015-7547/) You have likely already heard about CVE-2015-7547 (https://access.redhat.com/security/cve/cve-2015-7547) “A stack-based buffer overflow was found in the way the libresolv library performed dual A/AAAA DNS queries. A remote attacker could create a specially crafted DNS response which could cause libresolv to crash or, potentially, execute code with the permissions of the user running the library.” “Note: this issue is only exposed when libresolv is called from the nss_dns NSS service module.” More details from Google's Online Security team blog (https://googleonlinesecurity.blogspot.ca/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html) “Naturally, people have started asking whether FreeBSD is affected. The FreeBSD Security Officer has not yet released an official statement, but in the meantime, here is a brief look at the issue as far as FreeBSD is concerned.” “First of all: neither FreeBSD itself nor native FreeBSD applications are affected. While the resolver in FreeBSD's libc and GNU libc share a common parentage, the bug was introduced when the latter was rewritten to send A and AAAA queries in parallel rather than sequentially when the application requests both.” The same most likely applies to the other BSDs “However, Linux applications running under emulation on a FreeBSD system use the GNU libc and are therefore vulnerable unless patched.” A patch to update emulation/linux_base-c6 has been prepared and should be committed soon Running ‘pkg audit' will list any known vulnerable packages installed on your system “The issue can be mitigated by only using resolvers you trust, and configuring them to avoid sending responses which can trigger the bug.” “If you already have your own resolvers, you can configure them to avoid sending UDP responses larger than 2048 bytes. If the response does not fit in 2048 bytes, the server will send a truncated response, and the client should retry using TCP. While a similar bug exists in the code path for TCP requests, I believe that it can only be exploited by a malicious resolver, and interposing your own resolver will protect affected Linux systems and applications.” Dag-Erling's blog post also includes instructions and configuration examples for locking down your resolver, or setting up your own resolver if you don't have one already *** OpenBSD Foundation - 2016 Fundraising Campaign (http://www.openbsdfoundation.org/campaign2016.html) The OpenBSD foundation has announced their 2016 fundraising campaign, and set the goal of raising $250k for the year. While they mention that fundraising for 2015 didn't hit 2014's blockbuster numbers, it still exceeded the goal set, with an almost equal mix of corporate and community donors. ‘Our goal for 2016 is to increase the amount of support we offer for development, without compromising our regular support for the projects. We would like to: Plan and support more developer events (hackathons), and allow for more developers to attend these events. Continue to improve the project infrastructure. Fund more dedicated developer time for targeted development of specific projects.‘ To give you an idea of how much OpenBSD technology is used around the world, they broke it down this way: If $10 were given for every installation of OpenBSD in the last year from the master site (ignoring the mirrors) we would be at our goal. If $2 were given for every download of the OpenSSH source code in the last year from the master site (ignoring the mirrors) we would be at our goal. If a penny was donated for every pf or OpenSSH installed with a mainstream operating system or phone in the last year we would be at our goal. Getting Started with ION-DTN 3.4.0 on FreeBSD (https://sgeos.github.io/freebsd/ion/dtn/2016/02/07/getting-started-with-ion-dtn-3-4-0-on-freebsd.html) “The Interplanetary Overlay Network (ION) software distribution is an implementation of Delay-Tolerant Networking (DTN) architecture as described in Internet RFC 4838, suitable for use in spacecraft” This tutorial covers setting up ION 3.4.0 on FreeBSD The tutorial starts by downloading the ION software, and installing the relevant build tools The instructions allow ION to be installed system-wide, or for a specific user The each host is configured Then pings are traded between the hosts to ensure everything works Then a web page is served over the interplanetary network Sadly I don't have any hosts on other planets to test with. The tutorial also includes a troubleshooting guide *** Open Storage Issue – New BSD Mag is Out! (https://bsdmag.org/download/open_storage/) The next issue of BSDMag (The Open Storage Issue) just landed which features an interview with Matt Olander of iXsystems. During the interview, Matt talks about the culture of support for open-source down at iX, not only FreeNAS and PC-BSD, but the FreeBSD foundation, Slackware and more. He also gets to extol the virtues of the open-source development model itself, why it tends to lead to better code overall. In addition to the lead interview with Matt, this issue also features some other great interviews with Open Source storage vendors, and even some ZFS howto's about setting up your ZIL devive *** Interview - John Marino - marino@freebsd.org (mailto:marino@freebsd.org) FreeNAS with FreeBSD as its base helped save taxpayers $36,000 for a small public school district (https://www.ixsystems.com/whats-new/2016/02/11/january-missioncomplete-best-story/) News Roundup Getting Started With Tor Hidden Services on FreeBSD (https://sgeos.github.io/tor/freebsd/nc/curl/2016/02/06/getting-started-with-tor-hidden-services-on-freebsd.html) Ever wondered how to setup and use a Tor hidden service? We have a walkthrough posted over on github.io which details how to do that on a FreeBSD -CURRENT system. The basics are pretty simple, installing security/tor is the first step (although, he is using portmaster, you may wish to just ‘pkg install security/tor') The walkthrough provides an example server hosting just the date/time on port 8080, which you can use as an example and to verify it works, before serving anything real. Once a local server is ready to serve something, the Tor setup is pretty quick, basically just two lines of config in torrc: HiddenServiceDir /usr/home/tor/hidden_service/ HiddenServicePort 80 127.0.0.1:8080 After starting the service, the walkthrough will show you how to get the new hostname for this hidden service and verify its functionality. ZFS Remote Mirrors for Home Use (https://github.com/hughobrien/zfs-remote-mirror) A recently updated tutorial on remotely mirroring your ZFS files Using a spare old computer, or a SBC like a Raspberry Pi, and an (external) hard drive It covers installing and configuring FreeBSD for both sides of the remote replication The new appendix covers the creation of a Raspberry Pi image, although a prebuilt one is also provided The setup uses GELI to ensure the data is encrypted at-rest Updating and maintaining both systems is covered in detail The article is very detailed, and covers pretty much every aspect of the setup, including suggestions on where to physically locate the remote system, and configuration tips to reduce the chance that local intervention will be required Most importantly, it covers the disaster recovery steps. How to get your files back when bad things happen *** Lumina Desktop 0.8.8 Released (http://lumina-desktop.org/lumina-desktop-0-8-8-released/) PC-BSD's very own Lumina desktop has issued a new release, 0.8.8 Notable in this release is support for NetBSD out of box, improvements to the start menu, and ability to change monitor resolutions in the X configuration tool. (Also the desktop font colors look better!) 0.8.8 is now available in PC-BSD via pkg, and FreeBSD ports/pkg system as well. Lumina Desktop aims for v1.0 in July 2016 (http://fossforce.com/2016/02/lumina-desktop-getting-ready-freebsd-11-0/) We also have a blog post from Larry over at FossForce, highlighting that 1.0 of Lumina is still targeted for July(ish) *** NetBSD on Google's Compute Engine (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20160213_1951.html) A NetBSD developer has gotten NetBSD running on Google Compute Engine, a service somewhat similar to Amazon's EC2, and Microsoft's Azure Support is still being worked on, but I imagine it will land in NetBSD before too long NetBSD on GCE dmesg (http://dmesgd.nycbug.org/index.cgi?action=dmesgd&do=view&id=2900) OpenBSD on GCE (http://marc.info/?l=openbsd-misc&m=138610199311393&w=2) FreeBSD on GCE (https://github.com/swills/FreeBSD-gcloud) *** BeastieBits htop 2.0 released - an interactive process viewer for Unix (including FreeBSD and OpenBSD) (http://hisham.hm/htop/) Full set of binary packages for 7.0 released for ARM v6 and v7 (hf) (http://mail-index.netbsd.org/port-arm/2016/01/31/msg003648.html) DragonFly 4.4.2 released (https://www.dragonflybsd.org/release44/) LibertyBSD 5.8 has been released (http://libertybsd.net/) Broadwell systems may want to take advantage of the patch by Imre Vadasz (http://lists.dragonflybsd.org/pipermail/commits/2016-January/459239.html) Finding the hard-to-spot bugs in FreeBSD (http://www.viva64.com/en/b/0377/) Feedback/Questions Johnny - The Daily Show (http://slexy.org/view/s21dwzoXRn) Randy - Let it BSD (http://slexy.org/view/s2Hmmu5pUr) Miguel - NullFS (http://slexy.org/view/s20tOLsHHj) Jaek - PC-BSD Hardware (http://slexy.org/view/s2N9wQ1n5X) ***
This week on the show, we will be talking to FreeBSD developer and former core-team member John Baldwin about a variety of topics, including running a DevSummit, everything you needed or wanted to know. Coming up right now on BSDNow, the place to B...SD. This episode was brought to you by Headlines FreeBSD server retired after almost 19 years (http://www.theregister.co.uk/2016/01/14/server_retired_after_18_years_and_ten_months_beat_that_readers/) We've heard stories about this kind of thing before, that box that often sits under-appreciated, but refuses to die. Well the UK register has picked up on a story of a FreeBSD server finally being retired after almost 19 years of dedicated service. “In its day, it was a reasonable machine - 200MHz Pentium, 32MB RAM, 4GB SCSI-2 drive,” Ross writes. “And up until recently, it was doing its job fine.” Of late, however the “hard drive finally started throwing errors, it was time to retire it before it gave up the ghost!” The drive's a Seagate, for those of you looking to avoid drives that can't deliver more than 19 years of error-free operations. This system in particular had been running FreeBSD 2.2.1 over the years. Why not upgrade you ask? Ross has an answer for that: “It was heavily firewalled and only very specific services were visible to anyone, and most only visible to our directly connected customers,” Ross told Vulture South. “By the time it was probably due for a review, things had moved so far that all the original code was so tightly bound to the operating system itself, that later versions of the OS would have (and ultimately, did) require substantial rework. While it was running and not showing any signs of stress, it was simply expedient to leave sleeping dogs lie.” All in all, an amazing story of the longevity of a system and its operating system. Do you have a server with a similar or even greater uptime? Let us know so we can try and top this story. *** Roundup of all the BSDs (https://www.linuxvoice.com/group-test-bsd-distros/) The magazine LinuxVoice recently did a group test of a variety of “BSD Distros”. Included in their review were Free/Open/Net/Dragon/Ghost/PC It starts with a pretty good overview of BSD in general, its starts and the various projects / forks that spawned from it, such as FreeNAS / Junos / Playstation / PFSense / etc The review starts with a look at OpenBSD, and the consensus reached is that it is good, but does require a bit more manual work to run as a desktop. (Most of the review focuses on desktop usage). It ends up with a solid ⅘ stars though. Next it moves into GhostBSD, discusses it being a “Live” distro, which can optionally be installed to disk. It loses a few points for lacking a graphical package management utility, and some bugs during the installation, but still earns a respectable ⅗ stars. Dragonfly gets the next spin and gets praise for its very-up to date video driver support and availability of the HAMMER filesystem. It also lands at ⅗ stars, partly due to the reviewer having to use the command-line for management. (Notice a trend here?) NetBSD is up next, and gets special mention for being one of the only “distros” that doesn't do frequent releases. However that doesn't mean you can't have updated packages, since the review mentions pkgsrc and pkg as both available to customize your desktop. The reviewer was slightly haunted by having to edit files in /etc by hand to do wireless, but still gives NetBSD a ⅗ overall. Last up are FreeBSD and PC-BSD, which get a different sort of head-to-head review. FreeBSD goes first, with mention that the text-install is fairly straight-forward and most configuration will require being done by hand. However the reviewer must be getting use to the command-line at this point, because he mentions: “This might sound cumbersome, but is actually pretty straightforward and at the end produces a finely tuned aerodynamic system that does exactly what you want it to do and nothing else.” He does mention that FreeBSD is the ultimate DIY system, even to the point of not having the package management tools provided out of box. PC-BSD ultimately gets a lot of love in this review, again with it being focused on desktop usage this follows. Particularly popular are all the various tools written to make PC-BSD easier to use, such as Life-Preserver, Warden, the graphical installer and more. (slight mistake though, Life-Preserver does not use rsync to backup to FreeNAS, it does ZFS replication) In the end he rates FreeBSD ⅘ and PC-BSD a whopping 5/5 for this roundup. While reviews may be subjective to the particular use-case being evaluated for, it is still nice to see BSD getting some press and more interest from the Linux community in general. *** OpenBSD Laptops (http://www.tedunangst.com/flak/post/openbsd-laptops) Our buddy Ted Unangst has posted a nice “planning ahead” guide for those thinking of new laptops for 2016 and the upcoming OpenBSD 5.9 He starts by giving us a status update on several of the key driver components that will be in 5.9 release“5.9 will be the first release to support the graphics on Broadwell CPUs. This is anything that looks like i5-5xxx. There are a few minor quirks, but generally it works well. There's no support for the new Skylake models, however. They'll probably work with the VESA driver but minus suspend/resume/acceleration (just as 5.8 did with Broadwell).” He then goes on to mention that the IWM driver works well with most of the revisions (7260, 7265, and 3160) that ship with broadwell based laptops, however the newer skylake series ships with the 8260, which is NOT yet supported. He then goes on to list some of the more common makes and models to look for, starting with the broadwell based X1 carbons which work really well (Kris gives +++), but make sure its not the newer skylake model just yet. The macbook gets a mention, but probably should be avoided due to broadcom wifi The Dell XPS he mentions as a good choice for a powerful (portable) desktops *** Significant changes from NetBSD 7.0 to 8.0 (https://www.netbsd.org/changes/changes-8.0.html) Updated to GCC 4.8.5 Imported dhcpcd and replaced rtsol and rtsold gpt(8) utility gained the ability to resize partitions and disks, as well as change the type of a partition OpenSSH 7.1 and OpenSSL 1.0.1q FTP client got support for SNI for https Imported dtrace from FreeBSD Add syscall support Add lockstat support *** Interview - John Baldwin - jhb@freebsd.org (mailto:jhb@freebsd.org) / @BSDHokie (https://twitter.com/BSDHokie) FreeBSD Kernel Debugging News Roundup Dragonfly Mail Agent spreads to FreeBSD and NetBSD (https://www.dragonflydigest.com/2016/01/18/17508.html) DMA, the Dragonfly Mail Agent is now available not only in Dragonfly's dports, but also FreeBSD ports, and NetBSD pkgsrc “dma is a small Mail Transport Agent (MTA), designed for home and office use. It accepts mails from locally installed Mail User Agents (MUA) and delivers the mails either locally or to a remote destination. Remote delivery includes several features like TLS/SSL support and SMTP authentication. dma is not intended as a replacement for real, big MTAs like sendmail(8) or postfix(1). Consequently, dma does not listen on port 25 for incoming connections.” There was a project looking at importing DMA into the FreeBSD base system to replace sendmail, I wonder of the port signals that some of the blockers have been fixed *** ZFS UEFI Support has landed! (https://svnweb.freebsd.org/base?view=revision&revision=294068) Originally started by Eric McCorkle Picked up by Steven Hartland Including modularizing the existing UFS boot code, and adding ZFS boot code General improvements to the EFI loader including using more of libstand instead of containing its own implementations of many common functions Thanks to work by Toomas Soome, there is now a Beastie Menu as part of the EFI loader, similar to the regular loader As soon as this was committed, I added a few lines to it to connect the ZFS BE Menu to it, thanks to all of the above, without whom my work wouldn't be usable It should be relatively easy to hook my GELI boot stuff in as a module, and possibly just stack the UFS and ZFS modules on top of it I might try to redesign the non-EFI boot code to use a similar design instead of what I have now *** How three BSD OSes compare to ten Linux Distros (http://www.phoronix.com/scan.php?page=article&item=3bsd-10linux) After benchmarking 10 of the latest Linux distros, Phoronix took to benchmarking 3 of the big BSDs DragonFlyBSD 4.4.1 - The latest DragonFly release with GCC 5.2.1 and the HAMMER file-system. OpenBSD 5.8 - OpenBSD 5.8 with GCC 4.2.1 as the default compiler and FFS file-system. PC-BSD 10.2 - Derived off FreeBSD 10.2, the defaults were the Clang 3.4.1 compiler and ZFS file-system. In the SQLite test, PCBSD+ZFS won out over all of the Linux distros, including those that were also using ZFS In the first compile benchmark, PCBSD came second only to Intel's Linux distro, Clear Linux. OpenBSD can last, although it is not clear if the benchmark was just comparing the system compiler, which would be unfair to OpenBSD In Disk transaction performance, against ZFS won the day, with PCBSD edging out the Linux distros. OpenBSD's older ffs was hurt by the lack of soft updates, and DragonFly's Hammer did not perform well. Although in an fsync() heavy test, safety is more important that speed As with all benchmarks, these obviously need to be taken with a grain of salt In some of them you can clearly see that the ‘winner' has a much higher standard error, suggesting that the numbers are quite variable *** OPNSense 15.7.24 Released (https://opnsense.org/opnsense-15-7-24-released/) We are just barely into the new year and OPNSense has dropped a new release on us to play with. This new version, 15.7.24 brings a bunch of notable changes, which includes improvements to the firewall UI and a plugin management section of the firmware page. Additionally better signature verification using PKG's internal verification mechanisms was added for kernel and world updates. The announcement contains the full rundown of changes, including the suricata, openvpn and ntp got package bumps as well. *** Beastie Bits A FreeBSD 10 Desktop How-to (https://cooltrainer.org/a-freebsd-desktop-howto/) (A bit old, but still one of the most complete walkthroughs of a desktop FreeBSD setup from scratch) BSD and Scale 14 (http://fossforce.com/2016/01/bsd-ready-scale-14x/) Xen support enabled in OpenBSD -current (http://undeadly.org/cgi?action=article&sid=20160114113445&mode=expanded) Feedback/Questions Matt - Zil Sizes (http://slexy.org/view/s20a0mLaAv) Drin - IPSEC (http://slexy.org/view/s21qpiTF8h) John - ZFS + UEFI (http://slexy.org/view/s2HCq0r0aD) Jake - ZFS Cluster SAN (http://slexy.org/view/s2VORfyqlS) Phillip - Media Server (http://slexy.org/view/s20ycRhUkM) ***
This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! Headlines How to create new binary packages in the Ports system on OpenBSD (http://functionallyparanoid.com/2015/11/06/where-do-binary-packages-come-from/) Creating a port is often a great first step you can take to get involved in your favorite BSD of choice, and (often) doesn't require any actual programming to do so. In this article we have a great walkthrough for users on creating a new ported application, and eventually binary package, on OpenBSD As mentioned in the tutorial, a good starting place is always an existing port, which can you use as a template for your new creation. Tip: Try to pick something similar, I.E. python for a python app, Qt for Qt, etc. This tutorial will first walk you through the process of creating your Makefile and related description about the new port. Once you've created the initial Makefile, there are a bunch of new “make” targets you can begin to run to try building your port, everything from “make fetch” to “make makesum” and “make package”. Using these tests you can verify that your port is correct and results in the installable package/app you wanted. *** Status update on pledge(2) (http://undeadly.org/cgi?action=article&sid=20151116152318) OpenBSD has been working very aggressively to convert much of their base system applications to using pledge(2) “Formerly Tame(2)) Theo has provided a great status update on where that stands as of right now and the numbers look like the following: Out of 600 ELF binaries, 368 of them have been updated to utilize pledge(2) in some manner This is quite a few, and includes everything from openssl, ping, sftp, grep, gzip and much more There are still a number of “pledge-able” commands waiting for conversion, such as login, sysctl, nfsd, ssh and others. He also mentions that there does exist some subset of commands which aren't viable pledge(2) candidates, such as simple things like “true”, or commands like reboot/mount or even perl itself. *** FreeBSD booting on the Onion Omega (https://onion.io/omega/) Tiny $19 MIPS SoC ($25 with dock that provides built in mini-USB Serial interface, power supply, LED lights, GPIO expansion, USB port, etc) A number of pluggable ‘expansions' are available, including: Arduino Dock (connect the Omega device to your existing Arduino components) Blue Tooth Lower Energy 10/100 Ethernet Port Relay expansion (2 relays each, can stack up to 8 expansions to control 16 relays) Servo expansion (control up to 16 PWM servos, like robotic arms or camera mounts) OLED expansion (1" monochrome 128x64 OLED display) Thermal Printer Kit (includes all wiring and other components) The device is the product of a successful Kick Starter campaign (https://www.kickstarter.com/projects/onion/onion-omega-invention-platform-for-the-internet-of/description) from March of this year Specs: Atheros AR9330 rev1 400MHZ MIPS 24K 64MB DDR2 400MHz 16MB Flash 802.11b/g/n 150Mbps Atheros Wifi + 100mbps Atheros Wired Ethernet 18 GPIO Pins USB Controller Using the freebsd-wifi-build (https://github.com/freebsd/freebsd-wifi-build/wiki) tool, I was able to build a new firmware for the device based on a profile for a similar device based on the same Atheros chip. I hope to have time to validate some of the settings and get them posted up into the wiki and get the kernel configuration committed to FreeBSD in the next week or two It is an interesting device compared to the TP-Link WDR3600's we did at BSDCan, as it has twice as much flash, leaving more room for the system image, but only half as much ram, and a slower CPU *** SSH Performance testing (https://wiki.freebsd.org/SSHPerf) There has been a discussion (https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058244.html) about the value of upkeeping the HPN (High Performance Networking) patch to OpenSSH in the base system of FreeBSD As part of this, I did some fresh benchmarks on my pair of new high end servers The remaining part to be done is testing different levels of latency By tweaking the socket buffer sizes, I was able to saturate the full 10 gigabit with netcat, iperf, etc From the tests that have been done so far, it doesn't look like even the NONE cipher can reach that level of performance because of the MAC (Message Authentication Code) It does appear that some of the auto-tuning in HPN is not worked as expected Explicitly setting -oTcpRcvBuf=7168 (KB) is enough to saturate a gigabit with 50ms RTT (round trip time) *** iXsystems iX gives an overview of FreeBSD at SeaGl 2015 (https://www.ixsystems.com/whats-new/seagl-2015/) On the FreeNAS Blog, Michael Dexter explains the ZFS Intent Log and SLOG (http://www.freenas.org/whats-new/2015/11/zfs-zil-and-slog-demystified.html) Interview - George Wilson - wilzun@gmail.com (mailto:wilzun@gmail.com) / @zfsdude (https://twitter.com/zfsdude) OpenZFS and Delphix *** News Roundup Nicholas Marriott has replaced the aging version of less(1) in OpenBSD (http://undeadly.org/cgi?action=article&sid=20151105223808) Sometimes less isn't more, it's just less In this story, we have news that the old version of less(1) in OpenBSD has now been ripped out in favor of the more modern fork from illumos founder Garrett D'Amore. In addition to being a “more” modern version, it also includes far “less” of the portability code, uses terminfo, replacing termcap and is more POSIX compliant. *** FreeBSD gets initial support for advanced SMR drives (https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058522.html) Kenneth D. Merry ken@freebsd.org has developed initial support for Host Managed, and Host Aware Shingled Magnetic Recording drives in FreeBSD, available as a patch against both -current and 10-stable “This includes support for Host Managed, Host Aware and Drive Managed SMRdrives that are either SCSI (ZBC) or ATA (ZAC) attached via a SAScontroller. This does not include support for SMR ATA drives attached viaan ATA controller. Also, I have not yet figured out how to properly detecta Host Managed ATA drive, so this code won't do that.” SMR drives have overlapping tracks, because the read head can be much smaller than the write head The drawback to this approach is that writes to the disk must take place in 256 MB “zones” that must be written from the beginning New features in the patch: A new 'camcontrol zone' command that allows displaying and managing drive zones via SCSI/ATA passthrough. A new zonectl(8) utility that uses the new DIOCZONECMD ioctl to display and manage zones via the da(4) (and later ada(4)) driver. Changes to diskinfo -v to display the zone mode of a drive. A new disk zone API, sys/sys/disk_zone.h. A new bio type, BIO_ZONE, and modifications to GEOM to support it. This new bio will allow filesystems to query zone support in a drive and manage zoned drives. Extensive modifications to the da(4) driver to handle probing SCSI and SATA behind SAS SMR drives. Additional CAM CDB building functions for zone commands. “We (Spectra Logic) are working on ZFS changes that will use this CAM and GEOM infrastructure to make ZFS play well with SMR drives. Those changes aren't yet done.” It is good to see active development in this area, especially from experts in archival storage A second patch (https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058521.html) is also offered, that improves the pass(4) passthrough interface for disks, and introduces a new camdd(8) command, a version of dd that uses the pass(4) interface, kqueue, and separate reader/writer threads for improved performance He also presents a feature wishlist that includes some interesting benchmarking features, including a ‘sink' mode, where reads from the device are just thrown away, rather than having to write then to /dev/null *** Initial implemtnation of 802.11n now in iwm(4) (http://undeadly.org/cgi?action=article&sid=20151112212739) OpenBSD laptop users rejoice! 802.11n has landed! Initially only for the iwm(4) driver, support is planned for other devices in the future Includes support for all the required (non-optional) bits to make 802.11N functional Adds a new 11n mode to ifmedia, and MCS (modulation coding scheme) that sits alongside the ieee80211_rateset structure. No support for MIMO / SGI (Short Guard Interval) or 40 MHz wide-channels, but perhaps we will see those in a future update. They are asking users for testing against a wide variety of any/all APs! *** Freebsd adds support for Bluetooth LE Security Management (https://svnweb.freebsd.org/base?view=revision&revision=290038) FreeBSD + BlueTooth, not something we discuss a lot about, but it is still under active development. The most recently added features come from Takanori Watanabe, and adds new LE Security Management. Specifically, it enables support for BLE Security Manager Protocol(SMP), and enables a userland tool to wait for the underlying HCI connection to be encrypted. *** Building OpnSense on HardenedBSD (http://0xfeedface.org/2015/11/07/hbsd-opnsense.html) Looking for a way to further Harden your router? We have a tutorial from the HardenedBSD developer, Shawn Webb, about how to build OpnSense on HBSD 10-STABLE. You'll need to first be running HBSD 10-STABLE somewhere, in this article he is using bhyve for the builder VM. The build process itself is mostly pretty straight-forward, but there are a number of different repos that all have to be checked out, so pay attention to which goes where. +In this example he does a targeted build for a Netgate RCC-VE-4860, but you can pick your particular build. *** Beastie Bits 1 BTC bounty for chromium bug! (https://github.com/gliaskos/freebsd-chromium/issues/40) DesktopBSD 2.0 M1 released (http://www.desktopbsd.net/forums/threads/desktopbsd-2-0-m1-released.806/) By implementing asynchronous pru_attach for UDP, Sepherosa Ziehau has increased connect rate by around 15K connections per second (http://lists.dragonflybsd.org/pipermail/commits/2015-October/458500.html) Stephen Bourne, known for the Bourne Shell, will be giving a talk at NYCBUG this week (http://lists.nycbug.org/pipermail/talk/2015-October/016384.html) Tor Browser 5.0.3 for OpenBSD released (http://lists.nycbug.org/pipermail/talk/2015-October/016390.html) The Tor BSD Diversity Project (https://torbsd.github.io/) aim to Increase the number of Tor relays running BSDs. We envision this happening by increasing the total number of relays, with the addition of more BSD users running relays; Make the Tor Browser available under BSD operating systems using native packaging mechanisms. Our first target is OpenBSD; Engage the broader BSD community about the Tor anonymity network and the place that BSD Unix should occupy in the privacy community at large. Screenshots from Unix People circa 2002 (https://anders.unix.se/2015/10/28/screenshots-from-developers--unix-people-2002/) Feedback/Questions Dominik - Bhyve Setup (http://slexy.org/view/s21xTyirkO) John - beadm + GELI (http://slexy.org/view/s2YVi7ULlJ) Darrall - ZFS + RAID = Problems (http://slexy.org/view/s20lRTaZSy) Hamza - Which shell? (http://slexy.org/view/s2omNWdTBU) Amenia - FreeBSD routing (http://slexy.org/view/s21Y8bPbnm) ***
This week, Allan is out of town at another Developer Summit, but we have a great episode coming This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** Headlines WhatsApp founder, on how it got so HUGE (http://www.wired.com/2015/10/whatsapps-co-founder-on-how-the-iconoclastic-app-got-huge/) Wired has interviewed WhatsApp co-founder Brian Acton, about the infrastructure behind WhatsApp WhatsApp manages 900 million users with a team of 50, while Twitter needs around 4,000 employees to manage 300 million users. “FreeBSD has a nicely tuned network stack and extremely good reliability. We find managing FreeBSD installations to be quite straightforward.” “Linux is a beast of complexity. FreeBSD has the advantage of being a single distribution with an extraordinarily good ports collection.” “To us, it has been an advantage as we have had very few problems that have occurred at the OS level. With Linux, you tend to have to wrangle more and you want to avoid that if you can.” “FreeBSD happened because both Jan and I have experience with FreeBSD from Yahoo!.” Additional Coverage (http://uk.businessinsider.com/whatsapp-built-using-erlang-and-freebsd-2015-10) *** User feedback in the SystemD vs BSD init (https://www.textplain.net/blog/2015/problems-with-systemd-and-why-i-like-bsd-init/) We have a very detailed blog post this week from Randy Westlund, about his experiences on Linux and BSD, contrasting the init systems. What he finds is that while, it does make some things easier, such as writing a service file once, and having it run everywhere, the tradeoff comes in the complexity and lack of transparency. Another area of concern was the reproducibility of boots, how in his examples on servers, there can often be times when services start in different orders, to save a few moments of boot-time. His take on the simplicity of BSD's startup scripts is that they are very easy to hack on and monitor, while not introducing the feature creep we have seen in sysd. It will be interesting to see NextBSD / LaunchD and how it compares in the future! *** Learn to embrace open source, or get buried (http://opensource.com/business/15/10/ato-interview-jim-salter) At the recent “All Things Open” conference, opensource.com interviewed Jim Salter He describes how he first got started using FreeBSD to host his personal website He then goes on to talk about starting FreeBSDWiki.net and what its goals were The interview then talks about using Open Source at solve customers' problems at his consulting firm Finally, the talks about his presentation at AllThingsOpen: Move Over, Rsync (http://allthingsopen.org/talks/move-over-rsync/) about switching to ZFS replication *** HP's CTO Urges businesses to avoid permissive licenses (http://lwn.net/Articles/660428/) Martin Fink went on a rant about the negative effects of license proliferation While I agree that having too many new licenses is confusing and adds difficulty, I didn't agree with his closing point “He then ended the session with an extended appeal to move the open-source software industry away from permissive licenses like Apache 2.0 and toward copyleft licenses like the GPL” “The Apache 2.0 license is currently the most widely used "permissive" license. But the thing that developers overlook when adopting it, he said, is that by using Apache they are also making a choice about how much work they will have to put into building any sort of community around the project. If you look at Apache-licensed projects, he noted, "you'll find that they are very top-heavy with 'governance' structures." Technical committees, working groups, and various boards, he said, are needed to make such projects function. But if you look at copyleft projects, he added, you find that those structures simply are not needed.” There are plenty of smaller permissively licensed projects that do not have this sort of structure, infact, most of this structure comes from being an Apache run project, rather than from using the Apache or any other permissive license Luckily, he goes on to state that the “OpenSwitch code is released under the Apache 2.0 license, he said, because the other partner companies viewed that as a requirement.” “HP wanted to get networking companies and hardware suppliers on board. In order to get all of the legal departments at all of the partners to sign on to the project, he said, HP was forced to go with a permissive license” Hopefully the trend towards permissive licenses continues Additionally, in a separate LWN post: RMS Says: “I am not saying that competitors to a GNU package are unjust or bad -- that isn't necessarily so. The pertinent point is that they are competitors. The goal of the GNU Project is for GNU to win the competition. Each GNU package is a part of the GNU system, and should contribute to the success of the GNU Project. Thus, each GNU package should encourage people to run other GNU packages rather than their competitors -- even competitors which are free software.” (http://lwn.net/Articles/659757/) Never thought I'd see RMS espousing vendor lock-in *** Interview - Brian Callahan - bcallah@devio.us (mailto:bcallah@devio.us) / @twitter (https://twitter.com/__briancallahan) The BSDs in Education *** News Roundup Digital Libraries in Africa making use of DragonflyBSD and HAMMER (http://lists.dragonflybsd.org/pipermail/users/2015-October/228403.html) In the international development context, we have an interesting post from Michael Wilson of the PeerCorps Trust Fund. They are using DragonFlyBSD and FreeBSD to support the Tanzanian Digital Library Initiative in very resource-limited settings. They cite among the most important reasons for using BSD as the availability and quality of the documentation, as well as the robustness of the filesystems, both ZFS and HAMMER. Their website is now online over at (http://www.tandli.com/) , check it out to see exactly how BSD is being used in the field *** netflix hits > 65gbps from a single freebsd box (https://twitter.com/ed_maste/status/655120086248763396) A single socket server, with a high end Xeon E5 processor and a dual ported Chelsio T580 (2x 40 Gbps ports) set a netflix record pushing over 65 Gbps of traffic from a single machine The videos were being pushed from SSDs and some new high end NVMe devices The previous record at Netflix was 52 Gbps from a single machine, but only with very experimental settings. The current work is under much more typical settings By the end of that night, traffic surged to over 70 Gbps Only about 10-15% of that traffic was encrypted with the in-kernel TLS engine that Netflix has been working on with John-Mark Gurney It was reported that the machine was only using about 65% cpu, and had plenty of head room If I remember the discussion correctly, there were about 60,000 streams running off the machine *** Lumina Desktop 0.8.7 has been released (http://lumina-desktop.org/lumina-desktop-0-8-7-released/) A very large update has landed for PC-BSD's Lumina desktop A brand new “Start” menu has been added, which enables quick launch of favorite apps, pinning to desktop / favorites and more. Desktop icons have been overhauled, with better font support, and a new Grid system for placement of icons. Support for other BSD's such as DragonFly has been improved, along with TONS of internal changes to functionality and backends. Almost too many things to list here, but the link above will have full details, along with screenshots. *** A LiveUSB for NetBSD has been released by Jibbed (http://www.jibbed.org/) After a three year absence, the Jibbed project has come back with a Live USB image for NetBSD! The image contains NetBSD 7.0, and is fully R/W, allowing you to run the entire system from a single USB drive. Images are available for 8Gb and 4Gb sticks (64bit and 32bit respectively), along with VirtualBox images as well For those wanting X, it includes both X and TWM, although ‘pkgin' is available, so you can quickly add other desktops to the image *** Beastie Bits After recent discussions of revisiting W^X support in Mozilla Firefox, David Coppa has flipped the switch to enable it for OpenBSD users running -current. (http://undeadly.org/cgi?action=article&sid=20151021191401&mode=expanded) Using the vt(4) driver to change console resolution (http://lme.postach.io/post/changing-console-resolution-in-freebsd-10-with-vt-4) The FreeBSD Foundation gives a great final overview of the Grace Hopper Conference (http://freebsdfoundation.blogspot.com/2015/10/conference-recap-grace-hopper.html) A dialog about Compilers in the (BSD) base system (https://medium.com/@jmmv/compilers-in-the-bsd-base-system-1c4515a18c49) One upping their 48-core work from July, The Semihalf team shows off their the 96-core SMP support for FreeBSD on Cavium ThunderX (ARMv8 architecture (https://www.youtube.com/watch?v=1q5aDEt18mw) NYC Bug's November meeting will be featuring a talk by Stephen R. Bourne (http://lists.nycbug.org/pipermail/talk/2015-October/016384.html) New not-just-BSD postcast, hosted by two OpenBSD devs Brandon Mercer and Joshua Stein (http://garbage.fm/) Feedback/Questions Stefan (http://slexy.org/view/s21wjbhCJ4) Zach (http://slexy.org/view/s21TbKS5t0) Jake (http://slexy.org/view/s20AkO1i1R) Corey (http://slexy.org/view/s2nrUMatU5) Robroy (http://slexy.org/view/s2pZsC7arX) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
Coming up this week on the show, we'll be talking with Damien Miller of the OpenSSH team. Their 7.0 release has some major changes, including phasing out older crypto and changing one of the defaults that might surprise you. This episode was brought to you by Headlines EdgeRouter Lite, meet OpenBSD (http://www.tedunangst.com/flak/post/OpenBSD-on-ERL) The ERL, much like the Raspberry Pi and a bunch of other cheap boards, is getting more and more popular as more things get ported to run on it We've covered installing NetBSD and FreeBSD on them before, but OpenBSD has gotten a lot better support for them as well now (including the onboard storage in 5.8) Ted Unangst got a hold of one recently and kindly wrote up some notes about installing and using OpenBSD on it He covers doing a network install, getting the (slightly strange) bootloader working with u-boot and some final notes about the hardware More discussion can be found on Hacker News (https://news.ycombinator.com/item?id=10079210) and various (https://www.reddit.com/r/openbsd/comments/3hgf2c) other (https://www.marc.info/?t=143974140500001&r=1&w=2) places (https://lobste.rs/s/acz9bu/openbsd_on_edgerouter_lite) One thing to note (https://www.marc.info/?l=openbsd-misc&m=143991822827285&w=2) about these devices: because of their MIPS64 processor, they'll have weaker ASLR than X86 CPUs (and no W^X at all) *** Design and Implementation of the FreeBSD Operating System interview (http://www.infoq.com/articles/freebsd-design-implementation-review) For those who don't know, the "Design and Implementation of the FreeBSD Operating System" is a semi-recently-revived technical reference book for FreeBSD development InfoQ has a review of the book up for anyone who might be interested, but they also have an interview the authors "The book takes an approach to FreeBSD from inside out, starting with kernel services, then moving to process and memory management, I/O and devices, filesystems, IPC and network protocols, and finally system startup and shutdown. The book provides dense, technical information in a clear way, with lots of pseudo-code, diagrams, and tables to illustrate the main points." Aside from detailing a few of the chapters, the interview covers who the book's target audience is, some history of the project, long-term support, some of the newer features and some general OS development topics *** Path list parameter in OpenBSD tame (https://www.marc.info/?l=openbsd-cvs&m=144027474117290&w=2) We've mentioned OpenBSD's relatively new "tame (https://marc.info/?l=openbsd-tech&m=143725996614627&w=2)" subsystem a couple times before: it's an easy-to-implement "self-containment" framework, allowing programs to have a reduced feature set mode with even less privileges One of the early concerns from users of other process containment tools was that tame was too broad in the way it separated disk access - you could either read/write files or not, nothing in between Now there's the option to create a whitelist of specific files and directories that your binary is allowed to access, giving a much finer-grained set of controls to developers The next step is to add tame restraints to the OpenBSD userland utilities, which should probably be done by 5.9 More discussion can be found on Reddit (https://www.reddit.com/r/openbsd/comments/3i2lk7) and Hacker News (https://news.ycombinator.com/item?id=10104886) *** FreeBSD & PC-BSD 10.2-RELEASE (https://www.freebsd.org/releases/10.2R/announce.html) The FreeBSD team has released the second minor version bump to the 10.x branch, including all the fixes from 10-STABLE since 10.1 came out The Linux compatibility layer has been updated to support CentOS 6, rather than the much older Fedora Core base used previously, and the DRM graphics code has been updated to match Linux 3.8.13 New installations (and newly-upgraded systems) will use the quarterly binary package set, rather than the rolling release model that most people are used to A VXLAN driver was added, allowing you to create virtual LANs by encapsulating the ethernet frame in a UDP packet The bhyve codebase is much newer, enabling support for AMD CPUs with SVM and AMD-V extensions ARM and ARM64 code saw some fixes and improvements, including SMP support on a few specific boards and support for a few new boards The bootloader now supports entering your GELI passphrase before loading the kernel in full disk encryption setups In addition to assorted userland fixes and driver improvements, various third party tools in the base system were updated: resolvconf, ISC NTPd, netcat, file, unbound, OpenSSL, sendmail Check the full release notes (https://www.freebsd.org/releases/10.2R/relnotes.html) for the rest of the details and changes PC-BSD also followed with their 10.2-RELEASE (http://blog.pcbsd.org/2015/08/pc-bsd-10-2-release-now-available), sporting a few more additional features *** Interview - Damien Miller - djm@openbsd.org (mailto:djm@openbsd.org) / @damienmiller (https://twitter.com/damienmiller) OpenSSH: phasing out broken crypto, default cipher changes News Roundup NetBSD at Open Source Conference Shimane (https://mail-index.netbsd.org/netbsd-advocacy/2015/08/22/msg000692.html) We weren't the only ones away at conferences last week - the Japanese NetBSD guys are always raiding one event or another This time they had NetBSD running on some Sony NWS devices (MIPS-based) JavaStations (https://en.wikipedia.org/wiki/JavaStation) were also on display - something we haven't ever seen before (made between 1996-2000) *** BAFUG videos (https://www.youtube.com/watch?v=-XF20nitI90) The Bay Area FreeBSD users group has been uploading some videos of their recent meetings Devin Teske hosts the first one, discussing adding GELI support to the bootloader, including some video demonstrations of how it works Shortly after beginning, Adrian Chadd takes over the conversation and they discuss various problems (and solutions) related to the bootloader - for example, how can we type encryption passwords with non-US keyboard layouts In a second video (https://www.youtube.com/watch?v=49sPYHh473U), Jordan Hubbard and Kip Macy introduce "NeXTBSD aka FreeBSD X" In it, they discuss their ideas of merging more Mac OS X features into FreeBSD (launchd to replace the init system, some APIs, etc) People should record presentations at their BSD users groups and send them to us *** L2TP over IPSEC on OpenBSD (http://frankgroeneveld.nl/2015/08/16/configuring-l2tp-over-ipsec-on-openbsd-for-mac-os-x-clients) If you've got an OpenBSD box and some Mac OS X clients that need secure communications, surprise: they can work together pretty well Using only the base tools in both operating systems, you can build a nice IPSEC setup for tunneling all your traffic This guide specifically covers L2TP, using npppd and pre-shared keys Server setup, client setup, firewall configuration and routing-related settings are all covered in detail *** Reliable bare metal with TrueOS (http://www.tubsta.com/2015/08/reliable-bare-metal-server-using-trueosfreebsd) Imagine a server version of PC-BSD with some useful utilities preinstalled - that's basically TrueOS This article walks you through setting up a FreeBSD -CURRENT server (using TrueOS) to create a pretty solid backup solution Most importantly, he also covers how to keep everything redundant and deal with hard drives failing The author chose to go with the -CURRENT branch because of the delay between regular releases, and newer features not making their way to users as fast as he'd like Another factor is that there are no binary snapshots of FreeBSD -CURRENT that can be easily used for in-place upgrades, but with TrueOS (and some other BSDs) there are *** Kernel W^X on i386 (https://www.marc.info/?l=openbsd-cvs&m=144047868127049&w=2) We mentioned some big W^X kernel changes in OpenBSD a while back (https://www.marc.info/?l=openbsd-tech&m=142120787308107&w=2), but the work was mainly for x86_64 CPU architecture (which makes sense; that's what most people run now) Mike Larkin is back again, and isn't leaving the people with older hardware out, committing similar kernel work into the i386 platform now as well Check out our interview with Mike (http://www.bsdnow.tv/episodes/2015_05_13-exclusive_disjunction) for some more background info on memory protections like W^X *** Feedback/Questions Markus writes in (http://slexy.org/view/s2iGoeYMyb) Sean writes in (http://slexy.org/view/s21bIFfmUS) Theo writes in (http://slexy.org/view/s21Hjm8Tsa) ***
This week on the show, we'll be talking with Peter Toth. He's got a jail management system called "iocage" that's been getting pretty popular recently. Have we finally found a replacement for ezjail? We'll see how it stacks up. This episode was brought to you by Headlines FreeBSD on Olimex RT5350F-OLinuXino (https://www.bidouilliste.com/blog/2015/07/22/FreeBSD-on-Olimex-RT5350F-OLinuXino) If you haven't heard of the RT5350F-OLinuXino-EVB, you're not alone (actually, we probably couldn't even remember the name if we did know about it) It's a small board with a MIPS CPU, two ethernet ports, wireless support and... 32MB of RAM This blog series documents installing FreeBSD on the device, but it is quite a DIY setup at the moment In part two of the series (https://www.bidouilliste.com/blog/2015/07/24/FreeBSD-on-Olimex-RT5350F-OLinuXino-Part-2), he talks about the GPIO and how you can configure it Part three is still in the works, so check the site later on for further progress and info *** The modern OpenBSD home router (https://www.azabani.com/2015/08/06/modern-openbsd-home-router.html) In a new series of blog posts, one guy takes you through the process of building an OpenBSD-based gateway (http://www.bsdnow.tv/tutorials/openbsd-router) for his home network "It's no secret that most consumer routers ship with software that's flaky at best, and prohibitively insecure at worst" Armed with a 600MHz Pentium III CPU, he shows the process of setting up basic NAT, firewalling and even getting hostap mode working for wireless This guide also covers PPP and IPv6, in case you have those requirements In a similar but unrelated series (http://jaytongarnett.blogspot.com/2015/07/openbsd-router-bt-home-hub-5-replacement.html), another user does a similar thing - his post also includes details on reusing your consumer router as a wireless bridge He also has a separate post (http://jaytongarnett.blogspot.com/2015/08/openbsd-l2tpipsec-vpn-works-with.html) for setting up an IPSEC VPN on the router *** NetBSD at Open Source Conference 2015 Kansai (https://mail-index.netbsd.org/netbsd-advocacy/2015/08/10/msg000691.html) The Japanese NetBSD users group has teamed up with the Kansai BSD users group and Nagoya BSD users group to invade another conference They had NetBSD running on all the usual (unusual?) devices, but some of the other BSDs also got a chance to shine at the event Last time they mostly had ARM devices, but this time the centerpiece was an OMRON LUNA88k They had at least one FreeBSD and OpenBSD device, and at least one NetBSD device even had Adobe Flash running on it And what conference would be complete without an LED-powered towel *** OpenSSH 7.0 released (https://lists.mindrot.org/pipermail/openssh-unix-dev/2015-August/034289.html) The OpenSSH team has just finished up the 7.0 release, and the focus this time is deprecating legacy code SSHv1 support is disabled, 1024 bit diffie-hellman-group1-sha1 KEX is disabled and the v00 cert format authentication is disabled The syntax for permitting root logins has been changed, and is now called "prohibit-password" instead of "without-password" (this makes it so root can login, but only with keys) - all interactive authentication methods for root are also disabled by default now If you're using an older configuration file, the "without-password" option still works, so no change is required You can now control which public key types are available for authentication, as well as control which public key types are offered for host authentications Various bug fixes and documentation improvements are also included Aside from the keyboard-interactive and PAM-related bugs, this release includes one minor security fix: TTY permissions were too open, so users could write messages to other logged in users In the next release, even more deprecation is planned: RSA keys will be refused if they're under 1024 bits, CBC-based ciphers will be disabled and the MD5 HMAC will also be disabled *** Interview - Peter Toth - peter.toth198@gmail.com (mailto:peter.toth198@gmail.com) / @pannonp (https://twitter.com/pannonp) Containment with iocage (https://github.com/iocage/iocage) News Roundup More c2k15 reports (http://undeadly.org/cgi?action=article&sid=20150809105132) A few more hackathon reports from c2k15 in Calgary are still slowly trickling in Alexander Bluhm's up first, and he continued improving OpenBSD's regression test suite (this ensures that no changes accidentally break existing things) He also worked on syslogd, completing the TCP input code - the syslogd in 5.8 will have TLS support for secure remote logging Renato Westphal sent in a report (http://undeadly.org/cgi?action=article&sid=20150811171006) of his very first hackathon He finished up the VPLS implementation and worked on EIGRP (which is explained in the report) - the end result is that OpenBSD will be more easily deployable in a Cisco-heavy network Philip Guenther also wrote in (http://undeadly.org/cgi?action=article&sid=20150809165912), getting some very technical and low-level stuff done at the hackathon His report opens with "First came a diff to move the grabbing of the kernel lock for soft-interrupts from the ASM stubs to the C routine so that mere mortals can actually push it around further to reduce locking." - not exactly beginner stuff There were also some C-state, suspend/resume and general ACPI improvements committed, and he gives a long list of random other bits he worked on as well *** FreeBSD jails, the hard way (https://clinta.github.io/freebsd-jails-the-hard-way) As you learned from our interview this week, there's quite a selection of tools available to manage your jails This article takes the opposite approach, using only the tools in the base system: ZFS, nullfs and jail.conf Unlike with iocage, ZFS isn't actually a requirement for this method If you are using it, though, you can make use of snapshots for making template jails *** OpenSSH hardware tokens (http://www.tancsa.com/mdtblog/?p=73) We've talked about a number of ways to do two-factor authentication with SSH, but what if you want it on both the client and server? This blog post will show you how to use a hardware token as a second authentication factor, for the "something you know, something you have" security model It takes you through from start to finish: formatting the token, generating keys, getting it integrated with sshd Most of this will apply to any OS that can run ssh, and the token used in the example can be found online for pretty cheap too *** LibreSSL 2.2.2 released (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.2.2-relnotes.txt) The LibreSSL team has released version 2.2.2, which signals the end of the 5.8 development cycle and includes many fixes At the c2k15 hackathon, developers uncovered dozens of problems in the OpenSSL codebase with the Coverity code scanner, and this release incorporates all those: dead code, memory leaks, logic errors (which, by the way, you really don't want in a crypto tool...) and much more SSLv3 support was removed from the "openssl" command, and only a few other SSLv3 bits remain - once workarounds are found for ports that specifically depend on it, it'll be removed completely Various other small improvements were made: DH params are now 2048 bits by default, more old workarounds removed, cmake support added, etc It'll be in 5.8 (due out earlier than usual) and it's in the FreeBSD ports tree as well *** Feedback/Questions James writes in (http://slexy.org/view/s216lrsVVd) Stuart writes in (http://slexy.org/view/s20uGUHWLr) ***
We've finally reached a hundred episodes, and this week we'll be talking to Sebastian Wiedenroth about pkgsrc. Though originally a NetBSD project, now it runs pretty much everywhere, and he even runs a conference about it! This episode was brought to you by Headlines Remote DoS in the TCP stack (https://blog.team-cymru.org/2015/07/another-day-another-patch/) A pretty devious bug in the BSD network stack has been making its rounds for a while now, allowing remote attackers to exhaust the resources of a system with nothing more than TCP connections While in the LAST_ACK state, which is one of the final stages of a connection's lifetime, the connection can get stuck and hang there indefinitely This problem has a slightly confusing history that involves different fixes at different points in time from different people Juniper originally discovered the bug and announced a fix (https://kb.juniper.net/InfoCenter/index?page=content&id=JSA10686) for their proprietary networking gear on June 8th On June 29th, FreeBSD caught wind of it and fixed the bug in their -current branch (https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?view=patch&r1=284941&r2=284940&pathrev=284941), but did not issue a security notice or MFC the fix back to the -stable branches On July 13th, two weeks later, OpenBSD fixed the issue (https://www.marc.info/?l=openbsd-cvs&m=143682919807388&w=2) in their -current branch with a slightly different patch, citing the FreeBSD revision from which the problem was found Immediately afterwards, they merged it back to -stable and issued an errata notice (http://ftp.openbsd.org/pub/OpenBSD/patches/5.7/common/010_tcp_persist.patch.sig) for 5.7 and 5.6 On July 21st, three weeks after their original fix, FreeBSD committed yet another slightly different fix (https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?view=patch&r1=285777&r2=285776&pathrev=285777) and issued a security notice (https://lists.freebsd.org/pipermail/freebsd-announce/2015-July/001655.html) for the problem (which didn't include the first fix) After the second fix from FreeBSD, OpenBSD gave them both another look and found their single fix to be sufficient, covering the timer issue in a more general way NetBSD confirmed they were vulnerable too, and applied another completely different fix (http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/netinet/tcp_output.c.diff?r1=1.183&r2=1.184&only_with_tag=MAIN) to -current on July 24th, but haven't released a security notice yet DragonFly is also investigating the issue now to see if they're affected as well *** c2k15 hackathon reports (http://undeadly.org/cgi?action=article&sid=20150721180312&mode=flat) Reports from OpenBSD's latest hackathon (http://www.openbsd.org/hackathons.html), held in Calgary this time, are starting to roll in (there were over 40 devs there, so we might see a lot more of these) The first one, from Ingo Schwarze, talks about some of the mandoc work he did at the event He writes, "Did you ever look at a huge page in man, wanted to jump to the definition of a specific term - say, in ksh, to the definition of the "command" built-in command - and had to step through dozens of false positives with the less '/' and 'n' search keys before you finally found the actual definition?" With mandoc's new internal jump targets, this is a problem of the past now Jasper also sent in a report (http://undeadly.org/cgi?action=article&sid=20150723124332&mode=flat), doing his usual work with Puppet (and specifically "Facter," a tool used by Puppet to gather various bits of system information) Aside from that and various ports-related work, Jasper worked on adding tame support to some userland tools, fixing some Octeon stuff and introduced something that OpenBSD has oddly lacked until now: an "-i" flag for sed (hooray!) Antoine Jacoutot gave a report (http://undeadly.org/cgi?action=article&sid=20150722205349&mode=flat) on what he did at the hackathon as well, including improvements to the rcctl tool (for configuring startup services) It now has an "ls" subcommand with status parsing, allowing you to list running services, stopped services or even ones that failed to start or are supposed to be running (he calls this "the poor man's service monitoring tool") He also reworked some of the rc.d system to allow smoother operation of multiple instances of the same daemon to run (using tor with different config files as an example) His list also included updating ports, updating ports documentation, updating the hotplug daemon and laying out some plans for automatic sysmerge for future upgrades Foundation director Ken Westerback was also there (http://undeadly.org/cgi?action=article&sid=20150722105658&mode=flat), getting some disk-related and laptop work done He cleaned up and committed the 4k sector softraid code that he'd been working on, as well as fixing some trackpad issues Stefan Sperling, OpenBSD's token "wireless guy," had a lot to say (http://undeadly.org/cgi?action=article&sid=20150722182236&mode=flat) about the hackathon and what he did there (and even sent in his write-up before he got home) He taught tcpdump about some new things, including 802.11n metadata beacons (there's a lot more specific detail about this one in the report) Bringing a bag full of USB wireless devices with him, he set out to get the unsupported ones working, as well as fix some driver bugs in the ones that already did work One quote from Stefan's report that a lot of people seem to be talking about: "Partway through the hackathon tedu proposed an old diff of his to make our base ls utility display multi-byte characters. This led to a long discussion about how to expand UTF-8 support in base. The conclusion so far indicates that single-byte locales (such as ISO-8859-1 and KOI-8) will be removed from the base OS after the 5.8 release is cut. This simplifies things because the whole system only has to care about a single character encoding. We'll then have a full release cycle to bring UTF-8 support to more base system utilities such as vi, ksh, and mg. To help with this plan, I started organizing a UTF-8-focused hackathon for some time later this year." Jeremy Evans wrote in (http://undeadly.org/cgi?action=article&sid=20150725180527&mode=flat) to talk about updating lots of ports, moving the ruby ports up to the latest version and also creating perl and ruby wrappers for the new tame subsystem While he's mainly a ports guy, he got to commit fixes to ports, the base system and even the kernel during the hackathon Rafael Zalamena, who got commit access at the event, gives his very first report (http://undeadly.org/cgi?action=article&sid=20150725183439&mode=flat) on his networking-related hackathon activities With Rafael's diffs and help from a couple other developers, OpenBSD now has support for VPLS (https://en.wikipedia.org/wiki/Virtual_Private_LAN_Service) Jonathan Gray got a lot done (http://undeadly.org/cgi?action=article&sid=20150728184743&mode=flat) in the area of graphics, working on OpenGL and Mesa, updating libdrm and even working with upstream projects to remove some GNU-specific code As he's become somewhat known for, Jonathan was also busy running three things in the background: clang's fuzzer, cppcheck and AFL (looking for any potential crashes to fix) Martin Pieuchot gave an write-up (http://undeadly.org/cgi?action=article&sid=20150724183210&mode=flat) on his experience: "I always though that hackathons were the best place to write code, but what's even more important is that they are the best (well actually only) moment where one can discuss and coordinate projects with other developers IRL. And that's what I did." He laid out some plans for the wireless stack, discussed future plans for PF, made some routing table improvements and did various other bits to the network stack Unfortunately, most of Martin's secret plans seem to have been left intentionally vague, and will start to take form in the next release cycle We're still eagerly awaiting a report from one of OpenBSD's newest developers (https://twitter.com/phessler/status/623291827878137856), Alexandr Nedvedicky (the Oracle guy who's working on SMP PF and some other PF fixes) OpenBSD 5.8's "beta" status was recently reverted, with the message "take that as a hint (https://www.marc.info/?l=openbsd-cvs&m=143766883514831&w=2)," so that may mean more big changes are still to come... *** FreeBSD quarterly status report (https://www.freebsd.org/news/status/report-2015-04-2015-06.html) FreeBSD has published their quarterly status report for the months of April to June, citing it to be the largest one so far It's broken down into a number of sections: team reports, projects, kernel, architectures, userland programs, ports, documentation, Google Summer of Code and miscellaneous others Starting off with the cluster admin, some machines were moved to the datacenter at New York Internet, email services are now more resilient to failure, the svn mirrors (now just "svn.freebsd.org") are now using GeoGNS with official SSL certs and general redundancy was increased In the release engineering space, ARM and ARM64 work continues to improve on the Cavium ThunderX, more focus is being put into cloud platforms and the 10.2-RELEASE cycle is reaching its final stages The core team has been working on phabricator, the fancy review system, and is considering to integrate oauth support soon Work also continues on bhyve, and more operating systems are slowly gaining support (including the much-rumored Windows Server 2012) The report also covers recent developments in the Linux emulation layer, and encourages people using 11-CURRENT to help test out the 64bit support Multipath TCP was also a hot topic, and there's a brief summary of the current status on that patch (it will be available publicly soon) ZFSguru, a project we haven't talked about a lot, also gets some attention in the report - version 0.3 is set to be completed in early August PCIe hotplug support is also mentioned, though it's still in the development stages (basic hot-swap functions are working though) The official binary packages are now built more frequently than before with the help of additional hardware, so AMD64 and i386 users will have fresher ports without the need for compiling Various other small updates on specific areas of ports (KDE, XFCE, X11...) are also included in the report Documentation is a strong focus as always, a number of new documentation committers were added and some of the translations have been improved a lot Many other topics were covered, including foundation updates, conference plans, pkgsrc support in pkgng, ZFS support for UEFI boot and much more *** The OpenSSH bug that wasn't (http://bsdly.blogspot.com/2015/07/the-openssh-bug-that-wasnt.html) There's been a lot of discussion (https://www.marc.info/?t=143766048000005&r=1&w=2) about a supposed flaw (https://kingcope.wordpress.com/2015/07/16/openssh-keyboard-interactive-authentication-brute-force-vulnerability-maxauthtries-bypass/) in OpenSSH, allowing attackers to substantially amplify the number of password attempts they can try per session (without leaving any abnormal log traces, even) There's no actual exploit to speak of; this bug would only help someone get more bruteforce tries in with a fewer number of connections (https://lists.mindrot.org/pipermail/openssh-unix-dev/2015-July/034209.html) FreeBSD in its default configuration, with PAM (https://en.wikipedia.org/wiki/Pluggable_authentication_module) and ChallengeResponseAuthentication enabled, was the only one vulnerable to the problem - not upstream OpenSSH (https://www.marc.info/?l=openbsd-misc&m=143767296016252&w=2), nor any of the other BSDs, and not even the majority of Linux distros If you disable all forms of authentication except public keys, like you're supposed to (https://stribika.github.io/2015/01/04/secure-secure-shell.html), then this is also not a big deal for FreeBSD systems Realistically speaking, it's more of a PAM bug (https://www.marc.info/?l=openbsd-misc&m=143782167322500&w=2) than anything else OpenSSH added an additional check (https://anongit.mindrot.org/openssh.git/patch/?id=5b64f85bb811246c59ebab) for this type of setup that will be in 7.0, but simply changing your sshd_config is enough to mitigate the issue for now on FreeBSD (or you can run freebsd-update (https://lists.freebsd.org/pipermail/freebsd-security-notifications/2015-July/000248.html)) *** Interview - Sebastian Wiedenroth - wiedi@netbsd.org (mailto:wiedi@netbsd.org) / @wied0r (https://twitter.com/wied0r) pkgsrc (https://en.wikipedia.org/wiki/Pkgsrc) and pkgsrcCon (http://pkgsrc.org/pkgsrcCon/) News Roundup Now served by OpenBSD (https://tribaal.io/this-now-served-by-openbsd.html) We've mentioned that you can also install OpenBSD on DO droplets, and this blog post is about someone who actually did it The use case for the author was for a webserver, so he decided to try out the httpd in base Configuration is ridiculously simple, and the config file in his example provides an HTTPS-only webserver, with plaintext requests automatically redirecting TLS 1.2 by default, strong ciphers with LibreSSL and HSTS (https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) combined give you a pretty secure web server *** FreeBSD laptop playbooks (https://github.com/sean-/freebsd-laptops) A new project has started up on Github for configuring FreeBSD on various laptops, unsurprisingly named "freebsd-laptops" It's based on ansible, and uses the playbook format for automatic set up and configuration Right now, it's only working on a single Lenovo laptop, but the plan is to add instructions for many more models Check the Github page for instructions on how to get started, and maybe get involved if you're running FreeBSD on a laptop *** NetBSD on the NVIDIA Jetson TK1 (https://blog.netbsd.org/tnf/entry/netbsd_on_the_nvidia_jetson) If you've never heard of the Jetson TK1 (https://developer.nvidia.com/jetson-tk1), we can go ahead and spoil the secret here: NetBSD runs on it As for the specs, it has a quad-core ARMv7 CPU at 2.3GHz, 2 gigs of RAM, gigabit ethernet, SATA, HDMI and mini-PCIE This blog post shows which parts of the board are working with NetBSD -current (which seems to be almost everything) You can even run X11 on it, pretty sweet *** DragonFly power mangement options (http://lists.dragonflybsd.org/pipermail/users/2015-July/207911.html) DragonFly developer Sepherosa, who we've had on the show, has been doing some ACPI work over there In this email, he presents some of DragonFly's different power management options: ACPI P-states, C-states, mwait C-states and some Intel-specific bits as well He also did some testing with each of them and gave his findings about power saving If you've been thinking about running DragonFly on a laptop, this would be a good one to read *** OpenBSD router under FreeBSD bhyve (https://www.quernus.co.uk/2015/07/27/openbsd-as-freebsd-router/) If one BSD just isn't enough for you, and you've only got one machine, why not run two at once This article talks about taking a FreeBSD server running bhyve and making a virtualized OpenBSD router with it If you've been considering switching over your router at home or the office, doing it in a virtual machine is a good way to test the waters before committing to real hardware The author also includes a little bit of history on how he got into both operating systems There are lots of mixed opinions about virtualizing core network components, so we'll leave it up to you to do your research Of course, the next logical step is to put that bhyve host under Xen on NetBSD... *** Feedback/Questions Kevin writes in (http://slexy.org/view/s2yPVV5Wyp) Logan writes in (http://slexy.org/view/s21zcz9rut) Peter writes in (http://slexy.org/view/s21CRmiPwK) Randy writes in (http://slexy.org/view/s211zfIXff) ***
This week we'll be talking with Ryan Lortie and Baptiste Daroussin about GNOME on BSD. Upstream development is finally treating the BSDs as a first class citizen, so we'll hear about how the recent porting efforts have been since. This episode was brought to you by Headlines OpenBSD presents tame (https://www.marc.info/?l=openbsd-tech&m=143725996614627&w=2) Theo de Raadt sent out an email detailing OpenBSD's new "tame" subsystem, written by Nicholas Marriott and himself, for restricting what processes can and can't do When using tame, programs will switch to a "restricted-service operating mode," limiting them to only the things they actually need to do As for the background: "Generally there are two models of operation. The first model requires a major rewrite of application software for effective use (ie. capsicum). The other model in common use lacks granularity, and allows or denies an operation throughout the entire lifetime of a process. As a result, they lack differentiation between program 'initialization' versus 'main servicing loop.' systrace had the same problem. My observation is that programs need a large variety of calls during initialization, but few in their main loops." Some initial categories of operation include: computation, memory management, read-write operations on file descriptors, opening of files and, of course, networking Restrictions can also be stacked further into the lifespan of the process, but removed abilities can never be regained (obviously) Anything that tries to access resources outside of its in-place limits gets terminated with a SIGKILL or, optionally, a SIGABRT (which can produce useful core dumps for investigation) Also included are 29 examples of userland programs that get additional protection with very minimal changes to the source - only 2 or 3 lines needing changed in the case of binaries like cat, ps, dmesg, etc. This is an initial work-in-progress version of tame, so there may be more improvements or further (https://www.marc.info/?l=openbsd-tech&m=143740834710502&w=2) control (https://www.marc.info/?l=openbsd-tech&m=143741052411159&w=2) options added before it hits a release (very specific access policies can sometimes backfire (https://forums.grsecurity.net/viewtopic.php?f=7&t=2522), however) The man page, also included in the mail, provides some specifics about how to integrate tame properly into your code (which, by design, was made very easy to do - making it simple means third party programs are more likely to actually use it) Kernel bits are in the tree now (https://www.marc.info/?l=openbsd-cvs&m=143727335416513&w=2), with userland changes starting to trickle in too Combined with a myriad of memory protections (http://www.bsdnow.tv/episodes/2015_05_13-exclusive_disjunction), tight privilege separation and (above all else (https://en.wikipedia.org/wiki/OpenBSD_security_features)) good coding practices, tame should further harden the OpenBSD security fortress Further discussion (https://news.ycombinator.com/item?id=9928221) can (https://www.reddit.com/r/programming/comments/3dsr0t) be (http://undeadly.org/cgi?action=article&sid=20150719000800&mode=flat) found (https://news.ycombinator.com/item?id=9909429) in (https://www.reddit.com/r/linux/comments/3ds66o) the (https://lobste.rs/s/tbbtfs) usual (https://www.reddit.com/r/openbsd/comments/3ds64c) places (https://www.reddit.com/r/BSD/comments/3ds681) you'd expect *** Using Docker on FreeBSD (https://wiki.freebsd.org/Docker) With the experimental Docker port landing in FreeBSD a few weeks ago, some initial docs are starting to show up This docker is "the real thing," and isn't using a virtual machine as the backend - as such, it has some limitations The FreeBSD wiki has a page detailing how it works in general, as well as more info about those limitations When running Linux containers, it will only work as well as the Linux ABI compat layer for your version of FreeBSD (11.0, or -CURRENT when we're recording this, is where all the action is for 64bit support) For users on 10.X, there's also a FreeBSD container available, which allows you to use Docker as a fancy jail manager (it uses the jail subsystem internally) Give it a try, let us know how you find it to be compared to other solutions *** OpenBSD imports doas, removes sudo (http://www.tedunangst.com/flak/post/doas) OpenBSD has included the ubiquitous "sudo" utility for many years now, and the current maintainer of sudo (Todd C. Miller) is also a long-time OpenBSD dev The version included in the base system was much smaller than the latest current version used elsewhere, but was based on older code Some internal discussion lead to the decision that sudo should probably be moved to ports now, where it can be updated easily and offer all the extra features that were missing in base (LDAP and whatnot) Ted Unangst conjured up with a rewritten utility to replace it in the base system, dubbed "do as," with the aim of being more simple and compact There were concerns that sudo was too big and too complicated, and a quick 'n' dirty check reveals that doas is around 350 lines of code, while sudo is around 10,000 - which would you rather have as a setuid root binary? After the initial import, a number of developers began reviewing and improving various bits here and there You can check out the code (http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/doas/) now if you're interested Command usage (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/doas.1) and config syntax (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/doas.conf.5) seem pretty straightforward More discussion (https://news.ycombinator.com/item?id=9914693) on HN *** What would you like to see in FreeBSD (https://www.reddit.com/r/freebsd/comments/3d80vt/what_would_you_like_to_see_in_freebsd/) Adrian Chadd started a reddit thread about areas in which FreeBSD could be improved, asking the community what they'd like to see There are over 200 comments that span a wide range of topics, so we'll just cover a few of the more popular requests - check the very long thread if you're interested in more The top comment says things don't "just work," citing failover link aggregation of LACP laggs, PPPoE issues, disorganized jail configuration options, unclear CARP configuration and userland dtrace being unstable Another common one was that there are three firewalls in the base system, with ipfilter and pf being kinda dead now - should they be removed, and more focus put into ipfw? Video drivers also came up frequently, with users hoping for better OpenGL support and support for newer graphics cards from Intel and AMD - similar comments were made about wireless chipsets as well Some other replies included more clarity with pkgng output, paying more attention to security issues, updating PF to match the one in OpenBSD, improved laptop support, a graphical installer, LibreSSL in base, more focus on embedded MIPS devices, binary packages with different config options, steam support and lots more At least one user suggested better "marketing" for FreeBSD, with more advocacy and (hopefully) more business adoption That one really applies to all the BSDs, and regular users (that's you listening to this) can help make it happen for whichever ones you use right now Maybe Adrian can singlehandedly do all the work and make all the users happy *** Interview - Ryan Lortie & Baptiste Daroussin Porting the latest GNOME code to FreeBSD News Roundup Introducing resflash (http://stable.rcesoftware.com/resflash/) If you haven't heard of resflash before, it's "a tool for building OpenBSD images for embedded and cloud environments in a programmatic, reproducible way" One of the major benefits to images like this is the read-only filesystem, so there's no possibility of filesystem corruption if power is lost There's an optional read-write partition as well, used for any persistent changes you want to make You can check out the source code on Github (https://github.com/bconway/resflash) or read the main site for more info *** Jails with iocage (http://pid1.com/posts/post10.html) There are a growing number of FreeBSD jail management utilities: ezjail, cbsd, warden and a few others After looking at all the different choices, the author of this blog post eventually settled on iocage (https://github.com/iocage/iocage) for the job The post walks you through the basic configuration and usage of iocage for creating managing jails If you've been unhappy with ezjail or some of the others, iocage might be worth giving a try instead (it also has really good ZFS integration) *** DragonFly GPU improvements (http://lists.dragonflybsd.org/pipermail/users/2015-July/207892.html) DragonFlyBSD continues to up their graphics game, this time with Intel's ValleyView series of CPUs These GPUs are primarily used in the newer Atom CPUs and offer much better performance than the older ones A git branch was created to hold the fixes for now while the last remaining bugs get fixed Fully-accelerated Broadwell support and an update to newer DRM code are also available in the git branch, and will be merged to the main tree after some testing *** Branchless development (http://www.tedunangst.com/flak/post/branchless-development) Ted Unangst has a new blog post up, talking about software branches and the effects of having (or not having) them He covers integrating and merging code, and the versioning problems that can happen with multiple people contributing at once "For an open source project, branching is counter intuitively antisocial. For instance, I usually tell people I'm running OpenBSD, but that's kind of a lie. I'm actually running teduBSD, which is like OpenBSD but has some changes to make it even better. Of course, you can't have teduBSD because I'm selfish. I'm also lazy, and only inclined to make my changes work for me, not everyone else." The solution, according to him, is bringing all the code the developers are using closer together One big benefit is that WIP code gets tested much faster (and bugs get fixed early on) *** Feedback/Questions Matthew writes in (http://slexy.org/view/s21yQtBCCK) Chris writes in (http://slexy.org/view/s21oFA80kY) Anonymous writes in (http://slexy.org/view/s2JYvTlJlm) Bill writes in (http://slexy.org/view/s21LXvk53z) ***
Coming up this time on the show, we'll be talking with the CTO of Xinuos, David Meyer, about their adoption of FreeBSD. We also discuss the BSD license model for businesses and the benefits of contributing changes back. This episode was brought to you by Headlines Enabling FreeBSD on AArch64 (https://community.arm.com/groups/processors/blog/2015/07/07/enabling-freebsd-on-aarch64) One of the things the FreeBSD foundation has been dumping money into lately is ARM64 support, but we haven't heard too much about it - this article should change that Since it's on a mainstream ARM site, the article begins with a bit of FreeBSD history, leading up to the current work on ARM64 There's also a summary of some of the ARM work done at this year's BSDCan, including details about running it on the Cavium ThunderX platform (which has 48 cores) As of just a couple months ago, dtrace is even working on this new architecture Come 11.0-RELEASE, the plan is for ARM64 to get the same "tier 1" treatment as X86, which would imply binary updates for base and ports - something Raspberry Pi users often complain about not having *** OpenBSD's tcpdump detailed (https://www.youtube.com/watch?v=8kR-tW1kyDc#t=8) Most people are probably familiar with tcpdump (https://en.wikipedia.org/wiki/Tcpdump), a very useful packet sniffing and capturing utility that's included in all the main BSD base systems This video guide is specifically about the version in OpenBSD, which has gone through some major changes (it's pretty much a fork with no version number anymore) Unlike on the other platforms, OpenBSD's tcpdump will always run in a chroot as an unprivileged user - this has saved it from a number of high-profile exploits It also has support for the "pf.os" system, allowing you to filter out operating system fingerprints in the packet captures There's also PF (and pflog) integration, letting you see which line in your ruleset triggered a specific match Being able to run tcpdump directly on your router (http://www.bsdnow.tv/tutorials/openbsd-router) is pretty awesome for troubleshooting *** More FreeBSD foundation at BSDCan (http://freebsdfoundation.blogspot.com/2015/07/bsdcan-2015-trip-report-kamil-czekirda.html) The FreeBSD foundation has another round of trip reports from this year's BSDCan First up is Kamil Czekirda, who gives a good summary of some of the devsummit, FreeBSD-related presentations, some tutorials, getting freebsd-update bugs fixed and of course eating cake A second post (http://freebsdfoundation.blogspot.com/2015/07/bsdcan-2015-trip-report-christian.html) from Christian Brueffer, who cleverly planned ahead to avoid jetlag, details how he got some things done during the FreeBSD devsummit Their third report (http://freebsdfoundation.blogspot.com/2015/07/bsdcan-2015-trip-report-warren-block.html) is from our buddy Warren Block, who (unsurprisingly) worked on a lot of documentation-related things, including getting more people involved with writing them In true doc team style, his report is the most well-written of the bunch, including lots of links and a clear separation of topics (doc lounge, contributing to the wiki, presentations...) Finally, the fourth one (http://freebsdfoundation.blogspot.com/2015/07/bsdcan-2015-trip-report-shonali.html) comes to us from Shonali Balakrishna, who also gives an outline of some of the talks "Not only does a BSD conference have way too many very smart people in one room, but also some of the nicest." *** DragonFly on the Chromebook C720 (https://www.dragonflydigest.com/2015/07/08/16391.html) If you've got one of the Chromebook laptops and weren't happy with the included OS, DragonFlyBSD might be worth a go This article is a "mini-report" on how DragonFly functions on the device as a desktop, and While the 2GB of RAM proved to be a bit limiting, most of the hardware is well-supported DragonFly's wiki has a full guide (http://www.dragonflybsd.org/docs/newhandbook/ConfigChromebook/) on getting set up on one of these devices as well *** Interview - David Meyer - info@xinuos.com (mailto:info@xinuos.com) / @xinuos (https://twitter.com/xinuos) Xinuos, BSD license model vs. others, community interaction News Roundup Introducing LiteBSD (https://github.com/sergev/LiteBSD) We definitely don't talk about 4.4BSD a lot on the show LiteBSD is "a variant of [the] 4.4BSD operating system adapted for microcontrollers" If you've got really, really old hardware (or are working in the embedded space) then this might be an interesting hobby project to look info *** HardenedBSD announces ASLR completion (http://hardenedbsd.org/article/shawn-webb/2015-07-06/announcing-aslr-completion) HardenedBSD, now officially a full-on fork of FreeBSD (http://hardenedbsd.org/content/about), has declared their ASLR patchset to be complete The latest and last addition to the work was VDSO (Virtual Dynamic Shared Object) randomization, which is now configurable with a sysctl This post gives a summary of the six main features they've added since the beginning (http://www.bsdnow.tv/episodes/2014_08_27-reverse_takeover) Only a few small things are left to do - man page cleanups, possibly shared object load order improvements *** Unlock the reaper (https://www.marc.info/?l=openbsd-tech&m=143636371501474&w=2) In the ongoing quest to make more of OpenBSD SMP-friendly, a new patch was posted that unlocks the reaper in the kernel When there's a zombie process (https://en.wikipedia.org/wiki/Zombie_process) causing a resource leak, it's the reaper's job (https://en.wikipedia.org/wiki/Wait_%28system_call%29) to deallocate their resources (and yes we're still talking about computers, not horror movies) Initial testing has yielded positive (https://www.marc.info/?l=openbsd-tech&m=143642748717836&w=2) results (https://www.marc.info/?l=openbsd-tech&m=143639356810690&w=2) and no regressions (https://www.marc.info/?l=openbsd-tech&m=143638955809675&w=2) They're looking for testers, so you can install a -current snapshot and get it automatically An updated version of the patch is coming soon (https://www.marc.info/?l=openbsd-tech&m=143643025118637&w=2) too A hackathon (http://www.openbsd.org/images/hackathons/c2k15-s.gif) is going on right now, so you can expect more SMP improvements in the near future *** The importance of mentoring (http://adrianchadd.blogspot.com/2015/07/the-importance-of-mentoring-or-how-i.html) Adrian Chadd has a blog post up about mentoring new users, and it tells the story of how he originally got into FreeBSD He tells the story of, at age 11, meeting someone else who knew about making crystal sets that became his role model Eventually we get to his first FreeBSD 1.1 installation (which he temporarily abandoned for Linux, since it didn't have a color "ls" command) and how he started using the OS Nowadays, there's a formal mentoring system in FreeBSD While he talks about FreeBSD in the post, a lot of the concepts apply to all the BSDs (or even just life in general) *** Feedback/Questions Sean writes in (http://slexy.org/view/s29LpvIxDD) Herminio writes in (http://slexy.org/view/s21I1MZsDl) Stuart writes in (http://slexy.org/view/s20kk3ilM6) Richard writes in (http://slexy.org/view/s2pL5xA80B) ***
Coming up this time on the show, we'll be chatting with Lee Sharp. He's recently revived the m0n0wall codebase, now known as SmallWall, and we'll find out what the future holds for this new addition to the BSD family. Answers to your emails and all this week's news, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines BSDCan and pkgsrcCon videos (https://www.youtube.com/channel/UCAEx6zhR2sD2pAGKezasAjA/videos) Even more BSDCan 2015 videos are slowly but surely making their way to the internet Nigel Williams, Multipath TCP for FreeBSD (https://www.youtube.com/watch?v=P3vB_FWtyIs) Stephen Bourne, Early days of Unix and design of sh (https://www.youtube.com/watch?v=2kEJoWfobpA) John Criswell, Protecting FreeBSD with Secure Virtual Architecture (https://www.youtube.com/watch?v=hRIC_aF_u24) Shany Michaely, Expanding RDMA capability over Ethernet in FreeBSD (https://www.youtube.com/watch?v=stsaeKvF3no) John-Mark Gurney, Adding AES-ICM and AES-GCM to OpenCrypto (https://www.youtube.com/watch?v=JaufZ7yCrLU) Sevan Janiyan, Adventures in building (https://www.youtube.com/watch?v=-HMXyzybgdM) open source software (https://www.youtube.com/watch?v=Xof-uKnQ6cY) And finally, the BSDCan 2015 closing (https://www.youtube.com/watch?v=Ynm0bGnYdfY) Some videos (https://vimeo.com/channels/pkgsrccon/videos) from this year's pkgsrcCon (http://pkgsrc.org/pkgsrcCon/2015/) are also starting to appear online Sevan Janiyan, A year of pkgsrc 2014 - 2015 (https://vimeo.com/channels/pkgsrccon/132767946) Pierre Pronchery, pkgsrc meets pkg-ng (https://vimeo.com/channels/pkgsrccon/132766052) Jonathan Perkin, pkgsrc at Joyent (https://vimeo.com/channels/pkgsrccon/132760863) Jörg Sonnenberger, pkg_install script framework (https://vimeo.com/channels/pkgsrccon/132757658) Benny Siegert, New Features in BulkTracker (https://vimeo.com/channels/pkgsrccon/132751897) This is the first time we've ever seen recordings from the conference - hopefully they continue this trend *** OPNsense 15.7 released (https://forum.opnsense.org/index.php?topic=839.0) The OPNsense team has released version 15.7, almost exactly six months after their initial debut (http://www.bsdnow.tv/episodes/2015_01_14-common_sense_approach) In addition to pulling in the latest security fixes from upstream FreeBSD, 15.7 also includes new integration of an intrusion detection system (and new GUI for it) as well as new blacklisting options for the proxy server Taking a note from upstream PF's playbook, ALTQ traffic shaping support has finally been retired as of this release (it was deprecated from OpenBSD a few years ago, and the code was completely removed (http://undeadly.org/cgi?action=article&sid=20140419151959) just over a year ago) The LibreSSL flavor has been promoted to production-ready, and users can easily migrate over from OpenSSL via the GUI - switching between the two is simple; no commitment needed Various third party ports have also been bumped up to their latest versions to keep things fresh, and there's the usual round of bug fixes included Shortly afterwards, 15.7.1 (https://forum.opnsense.org/index.php?topic=915.0) was released with a few more small fixes *** NetBSD at Open Source Conference 2015 Okinawa (https://mail-index.netbsd.org/netbsd-advocacy/2015/07/04/msg000688.html) If you liked last week's episode (http://www.bsdnow.tv/episodes/2015_07_01-lost_technology) then you'll probably know what to expect with this one The NetBSD users group of Japan hit another open source conference, this time in Okinawa This time, they had a few interesting NetBSD machines on display that we didn't get to see in the interview last week We'd love to see something like this in North America or Europe too - anyone up for installing BSD on some interesting devices and showing them off at a Linux con? *** OpenBSD BGP and VRFs (http://firstyear.id.au/entry/21) "VRFs (https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding), or in OpenBSD rdomains, are a simple, yet powerful (and sometimes confusing) topic" This article aims to explain both BGP and rdomains, using network diagrams, for some network isolation goodness With multiple rdomains, it's also possible to have two upstream internet connections, but lock different groups of your internal network to just one of them The idea of a "guest network" can greatly benefit from this separation as well, even allowing for the same IP ranges to be used without issues Combining rdomains with the BGP protocol allows for some very selective and precise blocking/passing of traffic between networks, which is also covered in detail here The BSDCan talk on rdomains (https://www.youtube.com/watch?v=BizrC8Zr-YY) expands on the subject a bit more if you haven't seen it, as well as a few related (https://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/) posts (http://cybermashup.com/2013/05/21/complex-routing-with-openbsd/) *** Interview - Lee Sharp - lee@smallwall.org (mailto:lee@smallwall.org) SmallWall (http://smallwall.org), a continuation of m0n0wall News Roundup Solaris adopts more BSD goodies (https://blogs.oracle.com/solarisfw/entry/pf_for_solaris) We mentioned a while back that Oracle developers have begun porting a current version of OpenBSD's PF firewall to their next version, even contributing back patches for SMP and other bug fixes They recently published an article about PF, talking about what's different about it on their platform compared to others - not especially useful for BSD users, but interesting to read if you like firewalls Darren Moffat, who was part of originally getting an SSH implementation into Solaris, has a second blog post (https://blogs.oracle.com/darren/entry/openssh_in_solaris_11_3) up about their "SunSSH" fork Going forward, their next version is going to offer a completely vanilla OpenSSH option as well, with the plan being to phase out SunSSH after that The article talks a bit about the history of getting SSH into the OS, forking the code and also lists some of the differences between the two In a third blog post (https://blogs.oracle.com/darren/entry/solaris_new_system_calls_getentropy), they talk about a new system call they're borrowing from OpenBSD, getentropy(2) (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/getentropy.2), as well as the addition of arc4random (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man3/arc4random.3) to their libc With an up-to-date and SMP-capable PF, ZFS with native encryption, jail-like Zones, unaltered OpenSSH and secure entropy calls… is Solaris becoming better than us? Look forward to the upcoming "Solaris Now" podcast (not really) *** EuroBSDCon 2015 talks and tutorials (https://2015.eurobsdcon.org/talks/) This year's EuroBSDCon is set to be held in Sweden at the beginning of October, and the preliminary list of accepted presentations has been published The list looks pretty well-balanced between the different BSDs, something Paul would be happy to see if he was still with us It even includes an interesting DragonFly talk and a couple talks from NetBSD developers, in addition to plenty of FreeBSD and OpenBSD of course There are also a few tutorials (https://2015.eurobsdcon.org/tutorials/) planned for the event, some you've probably seen already and some you haven't Registration for the event will be opening very soon (likely this week or next) *** Using ZFS replication to improve offsite backups (https://www.iceflatline.com/2015/07/using-zfs-replication-features-in-freebsd-to-improve-my-offsite-backups/) If you take backups seriously, you're probably using ZFS and probably keeping an offsite copy of the data This article covers doing just that, but with a focus on making use of the replication capability It'll walk you through taking a snapshot of your pool and then replicating it to another remote system, using "zfs send" and SSH - this has the benefit of only transferring the files that have changed since the last time you did it Steps are also taken to allow a regular user to take and manage snapshots, so you don't need to be root for the SSH transfer Data integrity is a long process - filesystem-level checksums, resistance to hardware failure, ECC memory, multiple copies in different locations... they all play a role in keeping your files secure; don't skip out on any of them One thing the author didn't mention in his post: having an offline copy of the data, ideally sealed in a safe place, is also important *** Block encryption in OpenBSD (http://anadoxin.org/blog/blog/20150705/block-encryption-in-openbsd/) We've covered (http://www.bsdnow.tv/tutorials/fde) ways to do fully-encrypted installations of OpenBSD (and FreeBSD) before, but that requires dedicating a whole drive or partition to the sensitive data This blog post takes you through the process of creating encrypted containers in OpenBSD, à la TrueCrypt - that is, a file-backed virtual device with an encrypted filesystem It goes through creating a file that looks like random data, pointing vnconfig at it, setting up the crypto and finally using it as a fake storage device The encrypted container method offers the advantage of being a bit more portable across installations than other ways *** Docker hits FreeBSD ports (https://svnweb.freebsd.org/ports?view=revision&revision=391421) The inevitable has happened, and an early FreeBSD port of docker is finally here Some details and directions (https://github.com/kvasdopil/docker/blob/freebsd-compat/FREEBSD-PORTING.md) are available to read if you'd like to give it a try, as well as a list of which features work and which don't There was also some Hacker News discussion (https://news.ycombinator.com/item?id=9840025) on the topic *** Microsoft donates to OpenSSH (http://undeadly.org/cgi?action=article&sid=20150708134520&mode=flat) We've talked about big businesses using BSD and contributing back before, even mentioning a few other large public donations - now it's Microsoft's turn With their recent decision to integrate OpenSSH into an upcoming Windows release, Microsoft has donated a large sum of money to the OpenBSD foundation, making them a gold-level sponsor They've also posted some contract work offers on the OpenSSH mailing list, and say that their changes will be upstreamed if appropriate - we're always glad to see this *** Feedback/Questions Joe writes in (http://slexy.org/view/s2NqbhwOoH) Mike writes in (http://slexy.org/view/s2T3NEia98) Randy writes in (http://slexy.org/view/s20RlTK6Ha) Tony writes in (http://slexy.org/view/s2rjCd0bGX) Kevin writes in (http://slexy.org/view/s21PfSIyG5) ***
Coming up this week, we'll be talking with Jun Ebihara about some lesser-known CPU architectures in NetBSD. He'll tell us what makes these old (and often forgotten) machines so interesting. As usual, we've also got answers to your emails and all this week's news on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Out with the old, in with the less (http://www.tedunangst.com/flak/post/out-with-the-old-in-with-the-less) Our friend Ted Unangst has a new article up, talking about "various OpenBSD replacements and reductions" "Instead of trying to fix known bugs, we're trying to fix unknown bugs. It's not based on the current buggy state of the code, but the anticipated future buggy state of the code. Past bugs are a bigger factor than current bugs." In the post, he goes through some of the bigger (and smaller) examples of OpenBSD rewriting tools to be simpler and more secure It starts off with a lesser-known SCSI driver that "tried to do too much" being replaced with three separate drivers "Each driver can now be modified in isolation without unintentional side effects on other hardware, or the need to consider if and where further special cases need to be added. Despite the fact that these three drivers duplicate all the common boilerplate code, combined they only amount to about half as much code as the old driver." In contrast to that example, he goes on to cite mandoc as taking a very non "unixy" direction, but at the same time being smaller and simpler than all the tools it replaced The next case is the new http daemon, and he talks a bit about the recently-added rewrite support being done in a simple and secure way (as opposed to regex and its craziness) He also talks about the rewritten "file" utility: "Almost by definition, its sole input will be untrusted input. Perversely, people will then trust what file tells them and then go about using that input, as if file somehow sanitized it." Finally, sudo in OpenBSD's base system is moving to ports soon, and the article briefly describes a new tool that may or may not replace it (https://marc.info/?l=openbsd-ports&m=143481227122523&w=2), called "doas" There's also a nice wrap-up of all the examples at the end, and the "Pruning and Polishing (http://www.openbsd.org/papers/pruning.html)" talk is good complementary reading material *** More OpenZFS and BSDCan videos (https://www.youtube.com/channel/UC0IK6Y4Go2KtRueHDiQcxow/videos) We mentioned last week (http://www.bsdnow.tv/episodes/2015_06_24-bitrot_group_therapy) that some of the videos from the second OpenZFS conference in Europe were being uploaded - here's some more Matt Ahrens did a Q&A session (https://www.youtube.com/watch?v=I6fXZ_6OT5c) and talked about ZFS send and receive (https://www.youtube.com/watch?v=iY44jPMvxog), as well as giving an overview of OpenZFS (https://www.youtube.com/watch?v=RQlMDmnty80) George Wilson talked about a performance retrospective (https://www.youtube.com/watch?v=KBI6rRGUv4E) Toshiba (https://www.youtube.com/watch?v=sSi47-k78IM), Syneto (https://www.youtube.com/watch?v=Hhje5KEF5cE) and HGST (https://www.youtube.com/watch?v=aKgxXipss8k) also gave some talks about their companies and how they're using ZFS As for BSDCan, more of their BSD presentations have been uploaded too... Ryan Stone, PCI SR-IOV on FreeBSD (https://www.youtube.com/watch?v=INeMd-i5jzM) George Neville-Neil, Measure Twice, Code Once (https://www.youtube.com/watch?v=LE4wMsP7zeA) Kris Moore, Unifying jail and package management for PC-BSD, FreeNAS and FreeBSD (https://www.youtube.com/watch?v=qNYXqpJiFN0) Warner Losh, I/O Scheduling in CAM (https://www.youtube.com/watch?v=3WqOLolj5EU) Kirk McKusick, An Introduction to the Implementation of ZFS (https://www.youtube.com/watch?v=l-RCLgLxuSc) Midori Kato, Extensions to FreeBSD Datacenter TCP for Incremental Deployment Support (https://www.youtube.com/watch?v=zZXvjhWcg_4) Baptiste Daroussin, Packaging FreeBSD's (https://www.youtube.com/watch?v=Br6izhH5P1I) base system (https://www.youtube.com/watch?v=v7px6ktoDAI) Matt Ahrens, New OpenZFS features supporting remote replication (https://www.youtube.com/watch?v=UOX7WDAjqso) Ed Schouten, CloudABI Cloud computing meets fine-grained capabilities (https://www.youtube.com/watch?v=SVdF84x1EdA) The audio of Ingo Schwarze's talk "mandoc: becoming the main BSD manual toolbox" got messed up, but there's an alternate recording here (http://www.bsdcan.org/2015/audio/mandoc.mp3), and the slides are here (http://www.openbsd.org/papers/bsdcan15-mandoc.pdf) *** SMP steroids for PF (https://www.marc.info/?l=openbsd-tech&m=143526329006942&w=2) An Oracle employee that's been porting OpenBSD's PF to an upcoming Solaris release has sent in an interesting patch for review Attached to the mail was what may be the beginnings of making native PF SMP-aware Before you start partying, the road to SMP (specifically, giant lock removal) is a long and very complicated one, requiring every relevant bit of the stack to be written with it in mind - this is just one piece of the puzzle The initial response (https://www.marc.info/?l=openbsd-tech&m=143532243322281&w=2) has been quite positive though, with some back and forth (https://www.marc.info/?l=openbsd-tech&m=143532963824548&w=2) between developers and the submitter For now, let's be patient and see what happens *** DragonFly 4.2.0 released (http://www.dragonflybsd.org/release42/) DragonFlyBSD has released the next big update of their 4.x branch, complete with a decent amount of new features and fixes i915 and Radeon graphics have been updated, and DragonFly can claim the title of first BSD with Broadwell support in a release Sendmail in the base system has been replaced with their homegrown DragonFly Mail Agent, and there's a wiki page (http://www.dragonflybsd.com/docs/docs/newhandbook/mta/) about configuring it They've also switched the default compiler to GCC 5, though why they've gone in that direction instead of embracing Clang is a mystery The announcement page also contains a list of kernel changes, details on the audio and graphics updates, removal of the SCTP protocol, improvements to the temperature sensors, various userland utility fixes and a list of updates to third party tools Work is continuing on the second generation HAMMER filesystem, and Matt Dillon provides a status update in the release announcement There was also some hacker news discussion (https://news.ycombinator.com/item?id=9797932) you can check out, as well as upgrade instructions (http://lists.dragonflybsd.org/pipermail/users/2015-June/207801.html) *** OpenSMTPD 5.7.1 released (https://opensmtpd.org/announces/release-5.7.1.txt) The OpenSMTPD guys have just released version 5.7.1, a major milestone version that we mentioned recently Crypto-related bits have been vastly improved: the RSA engine is now privilege-separated, TLS errors are handled more gracefully, ciphers and curve preferences can now be specified, the PKI interface has been reworked to allow custom CAs, SNI and certificate verification have been simplified and the DH parameters are now 2048 bit by default The long-awaited filter API is now enabled by default, though still considered slightly experimental Documentation has been improved quite a bit, with more examples and common use cases (as well as exotic ones) Many more small additions and bugfixes were made, so check the changelog for the full list Starting with 5.7.1, releases are now cryptographically (https://twitter.com/OpenSMTPD/status/613257722574839808) signed (https://www.opensmtpd.org/archives/opensmtpd-5.7.1.sum.sig) to ensure integrity This release has gone through some major stress testing to ensure stability - Gilles regularly asks their Twitter followers to flood a test server (https://twitter.com/OpenSMTPD/status/608399272447471616) with thousands of emails per second, even offering prizes (https://twitter.com/OpenSMTPD/status/608235180839567360) to whoever can DDoS them the hardest OpenSMTPD runs on all the BSDs of course, and seems to be getting pretty popular lately Let's all encourage (mailto:feedback@bsdnow.tv) Kris to stop procrastinating on switching from Postfix *** Interview - Jun Ebihara (蛯原純) - jun@netbsd.org (mailto:jun@netbsd.org) / @ebijun (https://twitter.com/ebijun) Lesser-known CPU architectures, embedded NetBSD devices News Roundup FreeBSD foundation at BSDCan (http://freebsdfoundation.blogspot.com/2015/06/bsdcan-2015-trip-report-steven-douglas.html) The FreeBSD foundation has posted a few BSDCan summaries on their blog The first, from Steven Douglas, begins with a sentiment a lot of us can probably identify with: "Where I live, there are only a handful of people that even know what BSD is, let alone can talk at a high level about it. That was one of my favorite things, being around like minded people." He got to meet a lot of the people working on big-name projects, and enjoyed being able to ask them questions so easily Their second (http://freebsdfoundation.blogspot.com/2015/06/bsdcan-2015-trip-report-ahmed-kamal.html) trip report is from Ahmed Kamal, who flew in all the way from Egypt A bit starstruck, he seems to have enjoyed all the talks, particularly Andrew Tanenbaum's about MINIX and NetBSD There are also two more wrap-ups from Zbigniew Bodek (http://freebsdfoundation.blogspot.com/2015/06/bsdcan-2015-trip-report-zbigniew-bodek.html) and Vsevolod Stakhov (http://freebsdfoundation.blogspot.com/2015/06/bsdcan-2015-trip-report-vsevolod-stakhov.html), so you've got plenty to read *** OpenBSD from a veteran Linux user perspective (http://cfenollosa.com/blog/openbsd-from-a-veteran-linux-user-perspective.html) In a new series of blog posts, a self-proclaimed veteran Linux user is giving OpenBSD a try for the first time "For the first time I installed a BSD box on a machine I control. The experience has been eye-opening, especially since I consider myself an 'old-school' Linux admin, and I've felt out of place with the latest changes on the system administration." The post is a collection of his thoughts about what's different between Linux and BSD, what surprised him as a beginner - admittedly, a lot of his knowledge carried over, and there were just minor differences in command flags One of the things that surprised him (in a positive way) was the documentation: "OpenBSD's man pages are so nice that RTFMing somebody on the internet is not condescending but selfless." He also goes through some of the basics, installing and updating software, following different branches It concludes with "If you like UNIX, it will open your eyes to the fact that there is more than one way to do things, and that system administration can still be simple while modern." *** FreeBSD on the desktop, am I crazy (http://sysconfig.org.uk/freebsd-on-the-desktop-am-i-crazy.html) Similar to the previous article, the guy that wrote the SSH two factor authentication post we covered last week has another new article up - this time about FreeBSD on the desktop He begins with a bit of forewarning for potential Linux switchers: "It certainly wasn't an easy journey, and I'm tempted to say do not try this at home to anybody who isn't going to leverage any of FreeBSD's strong points. Definitely don't try FreeBSD on the desktop if you haven't used it on servers or virtual machines before. It's got less in common with Linux than you might think." With that out of the way, the list of positives is pretty large: a tidy base system, separation between base and ports, having the option to choose binary packages or ports, ZFS, jails, licensing and of course the lack of systemd The rest of the post talks about some of the hurdles he had to overcome, namely with graphics and the infamous Adobe Flash Also worth noting is that he found jails to be not only good for isolating daemons on a server, but pretty useful for desktop applications as well In the end, he says it was worth all the trouble, and is even planning on converting his laptop to FreeBSD soon too *** OpenIKED and Cisco CSR 1000v IPSEC (https://www.netflask.net/ipsec-ikev2-cisco-csr1000v-openiked/) This article covers setting up a site-to-site IPSEC tunnel between a Cisco CSR 1000v router and an OpenBSD gateway running OpenIKED What kind of networking blog post would be complete without a diagram where the internet is represented by a big cloud There are lots of details (and example configuration files) for using IKEv2 and OpenBSD's built-in IKE daemon It also goes to show that the BSDs generally play well with existing network infrastructure, so if you were a business that's afraid to try them… don't be *** HardenedBSD improves stack randomization (https://github.com/HardenedBSD/hardenedBSD/commit/bd5cecb4dc7947a5e214fc100834399b4bffdee8) The HardenedBSD guys have improved their FreeBSD ASLR patchset, specifically in the stack randomization area In their initial implementation, the stack randomization was a random gap - this update makes the base address randomized as well They're now stacking the new on top of the old as well, with the goal being even more entropy This change triggered an ABI and API incompatibility, so their major version has been bumped *** OpenSSH 6.9 released (https://lists.mindrot.org/pipermail/openssh-unix-announce/2015-July/000121.html) The OpenSSH team has announced the release of a new version which, following their tick/tock major/minor release cycle, is focused mainly on bug fixes There are a couple new things though - the "AuthorizedKeysCommand" config option now takes custom arguments One very notable change is that the default cipher has changed as of this release The traditional pairing of AES128 in counter mode with MD5 HMAC has been replaced by the ever-trendy ChaCha20-Poly1305 combo Their next release, 7.0, is set to get rid a number of legacy items: PermitRootLogin will be switched to "no" by default, SSHv1 support will be totally disabled, the 1024bit diffie-hellman-group1-sha1 KEX will be disabled, old ssh-dss and v00 certs will be removed, a number of weak ciphers will be disabled by default (including all CBC ones) and RSA keys will be refused if they're under 1024 bits Many small bugs fixes and improvements were also made, so check the announcement for everything else The native version is in OpenBSD -current, and an update to the portable version should be hitting a ports or pkgsrc tree near you soon *** Feedback/Questions Brad writes in (http://slexy.org/view/s2Ws6Y2rZy) Mason writes in (http://slexy.org/view/s21GvZ5xbs) Jochen writes in (http://slexy.org/view/s209TrPK4e) Simon writes in (http://slexy.org/view/s21TQjUjxv) ***
This week on the show, we'll be chatting with Marc Espie. He's recently added some additional security measures to dpb, OpenBSD's package building tool, and we'll find out why they're so important. We've also got all this week's news, answers to your emails and even a BSDCan wrap-up, coming up on BSD Now - the place to B.. SD. This episode was brought to you by Headlines BSDCan 2015 videos (https://www.bsdcan.org/2015/schedule/) BSDCan just ended last week, but some of the BSD-related presentation videos are already online Allan Jude, UCL for FreeBSD (https://www.youtube.com/watch?v=8l6bhKIDecg) Andrew Cagney, What happens when a dwarf and a daemon start dancing by the light of the silvery moon? (https://www.youtube.com/watch?v=XDIcD4LR5HE) Andy Tanenbaum, A reimplementation of NetBSD (https://www.youtube.com/watch?v=0pebP891V0c) using a MicroKernel (https://www.youtube.com/watch?v=Bu1JuwVfYTc) Brooks Davis, CheriBSD: A research fork of FreeBSD (https://www.youtube.com/watch?v=DwCg-51vFAs) Giuseppe Lettieri, Even faster VM networking with virtual passthrough (https://www.youtube.com/watch?v=Lo6wDCapo4k) Joseph Mingrone, Molecular Evolution, Genomic Analysis and FreeBSD (https://www.youtube.com/watch?v=K2pnf1YcMTY) Olivier Cochard-Labbe, Large-scale plug&play x86 network appliance deployment over Internet (https://www.youtube.com/watch?v=6jhSvdnu4k0) Peter Hessler, Using routing domains / routing tables in a production network (https://www.youtube.com/watch?v=BizrC8Zr-YY) Ryan Lortie, a stitch in time: jhbuild (https://www.youtube.com/watch?v=YSVFnM3_2Ik) Ted Unangst, signify: Securing OpenBSD From Us To You (https://www.youtube.com/watch?v=9R5s3l-0wh0) Many more still to come... *** Documenting my BSD experience (http://pid1.com/posts/post1.html) Increasingly common scenario: a long-time Linux user (since the mid-90s) decides it's finally time to give BSD a try "That night I came home, I had been trying to find out everything I could about BSD and I watched many videos, read forums, etc. One of the shows I found was BSD Now. I saw that they helped people and answered questions, so I decided to write in." In this ongoing series of blog posts, a user named Michael writes about his initial experiences with trying different BSDs for some different tasks The first post covers ZFS on FreeBSD, used to build a file server for his house (and of course he lists the hardware, if you're into that) You get a glimpse of a brand new user trying things out, learning how great ZFS-based RAID arrays are and even some of the initial hurdles someone could run into He's also looking to venture into the realm of replacing some of his VMs with jails and bhyve soon His second post (http://pid1.com/posts/post2.html) explores replacing the firewall on his self-described "over complicated home network" with an OpenBSD box After going from ipfwadmin to ipchains to iptables, not even making it to nftables, he found the simple PF syntax to be really refreshing All the tools for his networking needs, the majority of which are in the base system, worked quickly and were easy to understand Getting to hear experiences like this are very important - they show areas where all the BSD developers' hard work has paid off, but can also let us know where we need to improve *** PC-BSD tries HardenedBSD builds (https://github.com/pcbsd/hardenedBSD-stable) The PC-BSD team has created a new branch of their git repo with the HardenedBSD ASLR patches integrated They're not the first major FreeBSD-based project to offer an alternate build - OPNsense did that (https://hardenedbsd.org/article/shawn-webb/2015-05-08/hardenedbsd-teams-opnsense) a few weeks ago - but this might open the door for more projects to give it a try as well With Personacrypt, OpenNTPD, LibreSSL and recent Tor integration through the tools, these additional memory protections will offer PC-BSD users even more security that a default FreeBSD install won't have Time will tell if more projects and products like FreeNAS might be interested too *** C-states in OpenBSD (https://www.marc.info/?l=openbsd-cvs&m=143423172522625&w=2) People who run BSD on their notebooks, you'll want to pay attention to this one OpenBSD has recently committed some ACPI improvements for deep C-states (http://www.hardwaresecrets.com/article/Everything-You-Need-to-Know-About-the-CPU-C-States-Power-Saving-Modes/611), enabling the processor to enter a low-power mode According (https://twitter.com/StevenUniq/status/610586711358316545) to a (https://www.marc.info/?l=openbsd-misc&m=143430996602802&w=2) few users (https://www.marc.info/?l=openbsd-misc&m=143429914700826&w=2) so far (https://www.marc.info/?l=openbsd-misc&m=143425943026225&w=2), the change has resulted in dramatically lower CPU temperatures on their laptops, as well as much better battery life If you're running OpenBSD -current on a laptop, try out the latest snapshot and report back (https://www.marc.info/?l=openbsd-misc&m=143423391222952&w=2) with your findings *** NetBSD at Open Source Conference 2015 Hokkaido (https://mail-index.netbsd.org/netbsd-advocacy/2015/06/13/msg000687.html) The Japanese NetBSD users group never sleeps, and they've hit yet another open source conference As is usually the case, lots of strange machines on display were running none other than NetBSD (though it was mostly ARM this time) We'll be having one of these guys on the show next week to discuss some of the lesser-known NetBSD platforms *** Interview - Marc Espie - espie@openbsd.org (mailto:espie@openbsd.org) / @espie_openbsd (https://twitter.com/espie_openbsd) Recent (https://www.marc.info/?l=openbsd-ports&m=143051151521627&w=2) improvements (https://www.marc.info/?l=openbsd-ports&m=143151777209226&w=2) to OpenBSD's dpb (http://www.bsdnow.tv/tutorials/dpb) tool News Roundup Introducing xhyve, bhyve on OS X (https://github.com/mist64/xhyve/blob/master/README.md) We've talked about FreeBSD's "bhyve" hypervisor a lot on the show, and now it's been ported to another OS As the name "xhyve" might imply, it's a port of bhyve to Mac OS X Currently it only has support for virtualizing a few Linux distributions, but more guest systems can be added in the future It runs entirely in userspace, and has no extra requirements beyond OS X 10.10 or newer There are also a few examples (http://www.pagetable.com/?p=831) on how to use it *** 4K displays on DragonFlyBSD (http://www.dragonflybsd.org/docs/newhandbook/docs/newhandbook/4KDisplays/) If you've been using DragonFly as a desktop, maybe with those nice Broadwell graphics, you'll be pleased to know that 4K displays work just fine Matthew Dillon wrote up a wiki page about some of the specifics, including a couple gotchas Some GUI applications might look weird on such a huge resolution, HDMI ports are mostly limited to a 30Hz refresh rate, and there are slightly steeper hardware requirements for a smooth experience *** Sandboxing port daemons on OpenBSD (http://coderinaworldofcode.blogspot.com/2015/06/chrooting-mumble-server-on-openbsd.html) We talked about different containment methods last week, and mentioned that a lot of the daemons in OpenBSD's base as chrooted by default - things from ports or packages don't always get the same treatment This blog post uses a mumble server as an example, but you can apply it to any service from ports that doesn't chroot by default It goes through the process of manually building a sandbox with all the libraries you'll need to run the daemon, and this setup will even wipe and refresh the chroot every time you restart it With a few small changes, similar tricks could be done on the other BSDs as well - everybody has chroots *** SmallWall 1.8.2 released (http://smallwall.freeforums.net/thread/44/version-1-8-2-released) SmallWall is a relatively new BSD-based project that we've never covered before It's an attempt to keep the old m0n0wall codebase going, and appears to have started around the time m0n0wall called it quits They've just released the first official version (http://www.smallwall.org/download.html), so you can give it a try now If you're interested in learning more about SmallWall, the lead developer just might be on the show in a few weeks... *** Feedback/Questions David writes in (http://slexy.org/view/s21gRTNnk7) Brian writes in (http://slexy.org/view/s2DdiMvELg) Dan writes in (http://slexy.org/view/s2h4ZS6SMd) Joel writes in (http://slexy.org/view/s20kA1jeXY) Steve writes in (http://slexy.org/view/s2wJ9HP1bs) ***
Coming up this week, we'll be chatting with Lucas Holt, founder of MidnightBSD. It's a slightly lesser-known fork of FreeBSD, with a focus on easy desktop use. We'll find out what's different about it and why it was created. Answers to your emails and all this week's news, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Zocker, it's like docker on FreeBSD (http://toni.yweb.fi/2015/05/zocker-diy-docker-on-freebsd.html) Containment is always a hot topic, and docker has gotten a lot of hype in Linux land in the last couple years - they're working on native FreeBSD support at the moment This blog post is about a docker-like script, mainly for ease-of-use, that uses only jails and ZFS in the base system In total, it's 1,500 lines of shell script (https://github.com/toddnni/zocker) The post goes through the process of using the tool, showing off all the subcommands and explaining the configuration In contrast to something like ezjail, Zocker utilizes the jail.conf system in the 10.x branch *** Patrol Read in OpenBSD (https://www.marc.info/?l=openbsd-cvs&m=143285964216970&w=4) OpenBSD has recently imported some new code to support the Patrol Read (http://www.intel.com/support/motherboards/server/sb/CS-028742.htm) function of some RAID controllers In a nutshell, Patrol Read is a function that lets you check the health of your drives in the background, similar to a zpool "scrub" operation The goal is to protect file integrity by detecting drive failures before they can damage your data It detects bad blocks and prevents silent data corruption, while marking any bad sectors it finds *** HAMMER 2 improvements (http://lists.dragonflybsd.org/pipermail/commits/2015-May/418653.html) DragonFly BSD has been working on the second generation HAMMER FS It now uses LZ4 compression by default, which we've been big fans of in ZFS They've also switched to a faster CRC (http://lists.dragonflybsd.org/pipermail/commits/2015-May/418652.html) algorithm, further improving HAMMER's performance, especially (http://lists.dragonflybsd.org/pipermail/commits/2015-May/418651.html) when using iSCSI *** FreeBSD foundation May update (https://www.freebsdfoundation.org/press/2015mayupdate.pdf) The FreeBSD foundation has published another update newsletter, detailing some of the things they've been up to lately In it, you'll find some development status updates: notably more ARM64 work and the addition of 64 bit Linux emulation Some improvements were also made to FreeBSD's release building process for non-X86 architectures There's also an AsiaBSDCon recap that covers some of the presentations and the dev events They also have an accompanying blog post (http://freebsdfoundation.blogspot.com/2015/05/another-data-center-site-visit-nyi.html) where Glen Barber talks about more sysadmin and clusteradm work at NYI *** Interview - Lucas Holt - questions@midnightbsd.org (mailto:questions@midnightbsd.org) / @midnightbsd (https://twitter.com/midnightbsd) MidnightBSD News Roundup The launchd on train is never coming (http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/launchd-on-bsd.html) Replacement of init systems has been quite controversial in the last few years Fortunately, the BSDs have avoided most of that conflict thus far, but there have been a few efforts made to port launchd from OS X (https://en.wikipedia.org/wiki/Launchd) This blog post details the author's opinion on why he thinks we're never going to have launchd in any of the BSDs Email us your thoughts on the matter *** Native SSH comes to… Windows (http://blogs.msdn.com/b/looking_forward_microsoft__support_for_secure_shell_ssh1/archive/2015/06/02/managing-looking-forward-microsoft-support-for-secure-shell-ssh.aspx) In what may be the first (and last) mention of Microsoft on BSD Now... They've just recently announced that PowerShell will get native SSH support in the near future It's not based on the commercial SSH either, it's the same one from OpenBSD that we already use everywhere Up until now, interacting between BSD and Windows has required something like PuTTY, WinSCP, FileZilla or Cygwin - most of which are based on really outdated versions The announcement also promises that they'll be working with the OpenSSH community, so we'll see how many Microsoft-submitted patches make it upstream (or how many donations (http://www.openbsdfoundation.org/index.html) they make) *** Moving to FreeBSD (http://www.textplain.net/blog/2015/moving-to-freebsd/) This blog post describes a long-time Linux user's first BSD switching experience The author first talks about his Linux journey, eventually coming to love the more customization-friendly systems, but the journey ended with systemd After doing a bit of research, he gave FreeBSD a try and ended up liking it - the rest of the post mostly covers why that is He also plans to write about his experience with other BSDs, and is writing some tutorials too - we'll check in with him again later on *** Feedback/Questions Adam writes in (http://slexy.org/view/s29hS2cI05) Dan writes in (http://slexy.org/view/s20VRZYBsw) Ivan writes in (http://slexy.org/view/s20bumJ5u9) Josh writes in (http://slexy.org/view/s21BU6Pnka) ***
This week on the show, we've got something pretty different. We went to a Linux convention and asked various people if they've ever tried BSD and what they know about it. Stay tuned for that, all this week's news and, of course, answers to your emails, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines LUKS in OpenBSD (https://www.marc.info/?l=openbsd-tech&m=143247114716771&w=2) Last week, we were surprised to find out that DragonFlyBSD has support (http://leaf.dragonflybsd.org/cgi/web-man?command=cryptsetup§ion=8) for dm-crypt (https://en.wikipedia.org/wiki/Dm-crypt), sometimes referred to as LUKS (Linux Unified Key Setup (https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup)) It looks like they might not be the only BSD with support for it for much longer, as OpenBSD is currently reviewing a patch for it as well LUKS would presumably be an additional option in OpenBSD's softraid (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/softraid.4) system, which already provides native disk encryption Support hasn't been officially committed yet, it's still going through testing, but the code is there if you want to try it out and report your findings If enabled, this might pave the way for the first (semi-)cross platform encryption scheme since the demise of TrueCrypt (and maybe other BSDs will get it too in time) *** FreeBSD gets 64bit Linux emulation (https://lists.freebsd.org/pipermail/svn-src-head/2015-May/072255.html) For those who might be unfamiliar, FreeBSD has an emulation layer (https://www.freebsd.org/doc/handbook/linuxemu.html) to run Linux-only binaries (as rare as they may be) The most common use case is for desktop users, enabling them to run proprietary applications like Adobe Flash or Skype Similar systems can also be found in NetBSD (https://www.netbsd.org/docs/guide/en/chap-linux.html) and OpenBSD (http://www.openbsd.org/faq/faq9.html#Interact) (though disabled by default on the latter) However, until now, it's only supported binaries compiled for the i386 architecture This new update, already committed to -CURRENT, will open some new possibilities that weren't previously possible Meanwhile, HardenedBSD considers removing the emulation layer (https://hardenedbsd.org/content/poll-linuxulator-removal) entirely *** BSD at Open Source Conference 2015 Nagoya (https://mail-index.netbsd.org/netbsd-advocacy/2015/05/23/msg000686.html) We've covered the Japanese NetBSD users group setting up lots of machines at various conferences in the past, but now they're expanding Their latest report includes many of the NetBSD things you'd expect, but also a couple OpenBSD machines Some of the NetBSD ones included a Power Mac G4, SHARP NetWalker, Cubieboard2 and the not-so-foreign Raspberry Pi One new addition of interest is the OMRON LUNA88k, running the luna88k (http://www.openbsd.org/luna88k.html) port of OpenBSD There was even an old cell phone running Windows games (https://twitter.com/tsutsuii/status/601458973338775553) on NetBSD Check the mailing list post for some (https://pbs.twimg.com/media/CFrSmztWEAAS2uE.jpg) links (http://image.movapic.com/pic/m_201505230541335560130d49213.jpeg) to (http://image.movapic.com/pic/m_2015052305145455600ccea723a.jpeg) all (https://pbs.twimg.com/media/CFjPv9_UEAA8iEx.jpg:large) of (https://pbs.twimg.com/media/CD4k6ZUUMAA0tEM.jpg) the (https://pbs.twimg.com/media/CFqn1GXUsAAFuro.jpg) nice (https://pbs.twimg.com/media/CFdIS2IUkAAZvjc.jpg) pictures (https://pbs.twimg.com/media/CFf5mToUIAAFrRU.jpg) *** LLVM introduces OpenMP support (http://blog.llvm.org/2015/05/openmp-support_22.html) One of the things that has kept some people in the GCC camp is the lack of OpenMP (https://en.wikipedia.org/wiki/OpenMP) support in LLVM According to the blog post, it "enables Clang users to harness full power of modern multi-core processors with vector units" With Clang being the default in FreeBSD, Bitrig and OS X, and with some other BSDs exploring the option of switching, the need for this potential speed boost was definitely there This could also open some doors for more BSD in the area of high performance computing, putting an end to the current Linux monopoly *** Interview - Eric, FSF, John, Jose, Kris and Stewart Various "man on the street" style mini-interviews News Roundup BSD-licensed gettext replacement (https://gitlab.com/worr/libintl/blob/master/src/usr.bin/gettext/gettext.c) If you've ever installed ports on any of the BSDs, you've probably had GNU's gettext pulled in as a dependency Wikipedia says "gettext is an internationalization and localization (i18n) system commonly used for writing multilingual programs on Unix-like computer operating systems" A new BSD-licensed rewrite has begun, with the initial version being for NetBSD (but it's likely to be portable) If you've got some coding skills, get involved with the project - the more freely-licensed replacements, the better *** Unix history git repo (https://github.com/dspinellis/unix-history-repo) A git repository was recently created to show off some Unix source code history The repository contains 659 thousand commits and 2306 merges You can see early 386BSD commits all the way up to some of the more modern FreeBSD code If you want to browse through the giant codebase, it can be a great history lesson *** PCBSD 10.1.2 and Lumina updates (http://blog.pcbsd.org/2015/05/hotfix-release-to-10-1-2-now-available/) We mentioned 10.1.1 being released last week (and all the cool features a couple weeks before) but now 10.1.2 is out This minor update contained a few hotfixes: RAID-Z installation, cache and log devices and the text-only installer in UEFI mode There's also a new post (http://blog.pcbsd.org/2015/05/lumina-desktop-status-updatefaq/) on the PCBSD blog about Lumina, answering some frequently asked questions and giving a general status update *** Feedback/Questions Jake writes in (http://slexy.org/view/s25h4Biwzq) Van writes in (http://slexy.org/view/s2AF0bGmL6) Anonymous writes in (http://slexy.org/view/s20Ie1USFD) Dominik writes in (http://slexy.org/view/s20vBtoKqL) (text answer (http://slexy.org/view/s20RjbIT5v)) Chris writes in (http://slexy.org/view/s20USR3WzT) *** Mailing List Gold Death by chocolate (https://lists.mindrot.org/pipermail/openssh-unix-dev/2015-May/033945.html) ***
This time on the show, we'll be chatting with Jed Reynolds about ZFS. He's been using it extensively on a certain other OS, and we can both learn a bit about the other side's implementation. Answers to your questions and all this week's news, coming up on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Playing with sandboxing (http://blog.conviso.com.br/2015/05/playing-with-sandbox-analysis-of_13.html) Sandboxing and privilege separation are popular topics these days - they're the goal of the new "shill" scripting language, they're used heavily throughout OpenBSD, and they're gaining traction with the capsicum framework This blog post explores capsicum in FreeBSD, some of its history and where it's used in the base system They also include some code samples so you can verify that capsicum is actually denying the program access to certain system calls Check our interview about capsicum (http://www.bsdnow.tv/episodes/2014_05_28-the_friendly_sandbox) from a while back if you haven't seen it already *** OpenNTPD on by default (https://www.marc.info/?l=openbsd-cvs&m=143195693612629&w=4) OpenBSD has enabled ntpd (http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change) by default in the installer, rather than prompting the user if they want to turn it on In nearly every case, you're going to want to have your clock synced via NTP With the HTTPS constraints feature also enabled by default, this should keep the time checked and accurate, even against spoofing attacks Lots of problems can be traced back to the time on one system or another being wrong, so this will also eliminate some of those cases For those who might be curious (http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/etc/ntpd.conf), they're using the "pool.ntp.org (http://www.pool.ntp.org/en/)" cluster of addresses and google for HTTPS constraints (but these can be easily changed (http://www.bsdnow.tv/tutorials/ntpd)) *** FreeBSD workshop in Landshut (https://www.banym.de/freebsd/review-first-freebsd-workshop-in-landshut-on-15-may-2015) We mentioned a BSD installfest happening in Germany a few weeks back, and the organizer wrote in with a review of the event The installfest instead became a "FreeBSD workshop" session, introducing curious new users to some of the flagship features of the OS They covered when to use UFS or ZFS, firewall options, the release/stable/current branches and finally how to automate installations with Ansible If you're in south Germany and want to give similar introduction talks or Q&A sessions about the other BSDs, get in touch We'll hear more from him about how it went in the feedback section today *** Swap encryption in DragonFly (http://lists.dragonflybsd.org/pipermail/users/2015-May/207690.html) Doing full disk encryption (http://www.bsdnow.tv/tutorials/fde) is very important, but something that people sometimes overlook is encrypting their swap This can actually be more important than the contents of your disks, especially if an unencrypted password or key hits your swap (as it can be recovered quite easily) DragonFlyBSD has added a new experimental option to automatically encrypt your swap partition in fstab There was another way (http://lists.dragonflybsd.org/pipermail/users/2015-May/207691.html) to do it previously, but this is a lot easier You can achieve similar results in FreeBSD by adding ".eli" to the end of the swap device in fstab, there are a few steps (https://www.netbsd.org/docs/misc/#cgd-swap) to do it in NetBSD and swap in OpenBSD is encrypted by default A one-time key will be created and then destroyed in each case, making recovery of the plaintext nearly impossible *** Interview - Jed Reynolds - jed@bitratchet.com (mailto:jed@bitratchet.com) / @jed_reynolds (https://twitter.com/jed_reynolds) Comparing ZFS on Linux and FreeBSD News Roundup USB thermometer on OpenBSD (http://www.cambus.net/rding-temper-gold-usb-thermometer-on-openbsd/) So maybe you've got BSD on your server or router, maybe NetBSD on a toaster, but have you ever used a thermometer with one? This blog post introduces the RDing TEMPer Gold USB thermometer, a small device that can tell the room temperature, and how to get it working on OpenBSD Wouldn't you know it, OpenBSD has a native "ugold (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/ugold.4)" driver to support it with the sensors framework How useful such a device would be is another story though *** NAS4Free now on ARM (http://sourceforge.net/projects/nas4free/files/NAS4Free-ARM/10.1.0.2.1511/) We talk a lot about hardware for network-attached storage devices on the show, but ARM doesn't come up a lot That might be changing soon, as NAS4Free has just released some ARM builds These new (somewhat experimental) images are based on FreeBSD 11-CURRENT Included in the announcement is a list of fully-supported and partially-supported hardware that they've tested it with If anyone has experience with running a NAS on slightly exotic hardware, write in to us *** pkgsrcCon 2015 CFP and info (http://pkgsrc.pub/pkgsrcCon/2015/) This year's pkgsrcCon will be in Berlin, Germany on July 4th and 5th (https://mail-index.netbsd.org/pkgsrc-users/2015/05/16/msg021560.html) They're looking for talk proposals and ideas for things you'd like to see If you or your company uses pkgsrc, or if you're just interested in NetBSD in general, it would be a good event to check out *** BSDTalk episode 253 (http://bsdtalk.blogspot.com/2015/05/bsdtalk253-george-neville-neil.html) BSDTalk has released another new episode In it, he interviews George Neville-Neil about the 2nd edition of "The Design and Implementation of the FreeBSD Operating System" They discuss what's new since the last edition, who the book's target audience is and a lot more We're up to 90 episodes now, slowly catching up to Will... *** Feedback/Questions Dominik writes in (http://slexy.org/view/s2SWlyuOeb) Brad writes in (http://slexy.org/view/s216z44lDU) Corvin writes in (http://slexy.org/view/s2djtX0dSE) James writes in (http://slexy.org/view/s21XM4hPRh) ***
This time on the show, we'll be talking with Ed Schouten about CloudABI. It's a new application binary interface with a strong focus on isolation and restricted capabilities. As always, all this week's BSD news and answers to your emails, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines FreeBSD quarterly status report (https://www.freebsd.org/news/status/report-2015-01-2015-03.html) The FreeBSD team has posted a report of the activities that went on between January and March of this year As usual, it's broken down into separate reports from the various teams in the project (ports, kernel, virtualization, etc) The ports team continuing battling the flood of PRs, closing quite a lot of them and boasting nearly 7,000 commits this quarter The core team and cluster admins dealt with the accidental deletion of the Bugzilla database, and are making plans for an improved backup strategy within the project going forward FreeBSD's future release support model was also finalized and published in February, which should be a big improvement for both users and the release team Some topics are still being discussed internally, mainly MFCing ZFS ARC responsiveness patches to the 10 branch and deciding whether to maintain or abandon C89 support in the kernel code Lots of activity is happening in bhyve, some of which we've covered recently (http://www.bsdnow.tv/episodes/2015_04_29-on_the_list), and a number of improvements were made this quarter Clang, LLVM and LLDB have been updated to the 3.6.0 branch in -CURRENT Work to get FreeBSD booting natively on the POWER8 CPU architecture is also still in progress, but it does boot in KVM for the time being The project to replace forth in the bootloader with lua is in its final stages, and can be used on x86 already ASLR work (http://www.bsdnow.tv/episodes/2014_08_27-reverse_takeover) is still being done by the HardenedBSD guys, and their next aim is position-independent executable The report also touches on multipath TCP support, the new automounter, opaque ifnet, pkgng updates, secureboot (which should be in 10.2-RELEASE), GNOME and KDE on FreeBSD, PCIe hotplugging, nested kernel support and more Also of note: work is going on to make ARM a Tier 1 platform in the upcoming 11.0-RELEASE (and support for more ARM boards is still being added, including ARM64) *** OpenBSD 5.7 released (http://www.openbsd.org/57.html) OpenBSD has formally released another new version, complete with the giant changelog we've come to expect In the hardware department, 5.7 features many driver improvements and fixes, as well as support for some new things: USB 3.0 controllers, newer Intel and Atheros wireless cards and some additional 10gbit NICs If you're using one of the Soekris boards, there's even a new driver (http://bodgitandscarper.co.uk/openbsd/further-soekris-net6501-improvements-for-openbsd/) to manipulate the GPIO and LEDs on them - this has some fun possibilities Some new security improvements include: SipHash (https://en.wikipedia.org/wiki/SipHash) being sprinkled in some areas to protect hashing functions, big W^X improvements (https://www.marc.info/?l=openbsd-tech&m=142120787308107&w=2) in the kernel space, static PIE (http://www.bsdnow.tv/episodes/2015_04_15-pie_in_the_sky) on all architectures, deterministic "random" functions being replaced (https://www.marc.info/?l=openbsd-tech&m=141807224826859&w=2) with strong randomness, and support for remote logging over TLS The entire source tree has also been audited to use reallocarray (http://lteo.net/blog/2014/10/28/reallocarray-in-openbsd-integer-overflow-detection-for-free/), which unintentionally saved (https://splone.com/blog/2015/3/11/integer-overflow-prevention-in-c) OpenBSD's libc from being vulnerable to earlier attacks (https://guidovranken.wordpress.com/2015/02/04/full-disclosure-heap-overflow-in-h-spencers-regex-library-on-32-bit-systems/) affecting other BSDs' implementations Being that it's OpenBSD, a number of things have also been removed from the base system: procfs, sendmail, SSLv3 support and loadable kernel modules are all gone now (not to mention the continuing massacre of dead code in LibreSSL) Some people seem to be surprised about the removal of loadable modules, but almost nothing utilized them in OpenBSD, so it was really just removing old code that no one used anymore - very different from FreeBSD or Linux in this regard, where kernel modules are used pretty heavily BIND and nginx have been taken out, so you'll need to either use the versions in ports or switch to Unbound and the in-base HTTP daemon Speaking of httpd, it's gotten a number of new (http://www.openbsd.org/papers/httpd-slides-asiabsdcon2015.pdf) features (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/httpd.conf.5), and has had time to grow and mature since its initial debut - if you've been considering trying it out, now would be a great time to do so This release also includes the latest OpenSSH (with stronger fingerprint types and host key rotation), OpenNTPD (with the HTTPS constraints feature), OpenSMTPD, LibreSSL and mandoc (http://www.bsdnow.tv/episodes/2014_11_12-a_mans_man) Check the errata page (http://www.openbsd.org/errata57.html) for any post-release fixes, and the upgrade guide (http://www.openbsd.org/faq/upgrade57.html) for specific instructions on updating from 5.6 Groundwork has also been laid for some major SMP scalability improvements - look forward to those in future releases There's a song and artwork (http://www.openbsd.org/lyrics.html#57) to go along with the release as always, and CDs should be arriving within a few days - we'll show some pictures next week Consider picking one up (https://www.openbsdstore.com) to support the project (and it's the only way to get puffy stickers) For those of you paying close attention, the banner image (http://www.openbsd.org/images/puffy57.gif) for this release just might remind you of a certain special episode (http://www.bsdnow.tv/episodes/2014_09_03-its_hammer_time) of BSD Now... *** Tor-BSD diversity project (https://torbsd.github.io/) We've talked about Tor on the show a few times, and specifically about getting more of the network on BSD (Linux has an overwhelming majority right now) A new initiative has started to do just that, called the Tor-BSD diversity project "Monocultures in nature are dangerous, as vulnerabilities are held in common across a broad spectrum. Diversity means single vulnerabilities are less likely to harm the entire ecosystem. [...] A single kernel vulnerability in GNU/Linux that impacting Tor relays could be devastating. We want to see a stronger Tor network, and we believe one critical ingredient for that is operating system diversity." In addition to encouraging people to put up more relays, they're also continuing work on porting the Tor Browser Bundle to BSD, so more desktop users can have easy access to online privacy There's an additional progress report (http://trac.haqistan.net/blog/tor-browser-ports-progress) for that part specifically, and it looks like most of the work is done now Engaging the broader BSD community about Tor and fixing up the official documentation are also both on their todo list If you've been considering running a node to help out, there's always our handy tutorial (http://www.bsdnow.tv/tutorials/tor) on getting set up *** PC-BSD 10.1.2-RC1 released (http://blog.pcbsd.org/2015/05/pc-bsd-10-1-2-rc1-now-available/) If you want a sneak peek at the upcoming PC-BSD 10.1.2, the first release candidate is now available to grab This quarterly update includes a number of new features, improvements and even some additional utilities PersonaCrypt is one of them - it's a new tool for easily migrating encrypted home directories between systems A new "stealth mode" option allows for a one-time login, using a blank home directory that gets wiped after use Similarly, a new "Tor mode" allows for easy tunneling of all your traffic through the Tor network IPFW is now the default firewall, offering improved VIMAGE capabilities The life preserver backup tool now allows for bare-metal restores via the install CD ISC's NTP daemon has been replaced with OpenNTPD (http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change), and OpenSSL has been replaced with LibreSSL (http://www.bsdnow.tv/episodes/2015_03_25-ssl_in_the_wild) It also includes the latest Lumina (http://www.bsdnow.tv/episodes/2014_09_10-luminary_environment) desktop, and there's another post dedicated to that (http://blog.pcbsd.org/2015/05/pc-bsd-10-1-2-rc1-lumina-desktop-0-8-4-released/) Binary packages have also been updated to fresh versions from the ports tree More details, including upgrade instructions, can be found in the linked blog post *** Interview - Ed Schouten - ed@freebsd.org (mailto:ed@freebsd.org) / @edschouten (https://twitter.com/edschouten) CloudABI (https://www.bsdcan.org/2015/schedule/track/Security/524.en.html) News Roundup Open Household Router Contraption (http://code.saghul.net/index.php/2015/05/01/announcing-the-open-household-router-contraption/) This article introduces OpenHRC, the "Open Household Router Contraption" In short, it's a set of bootstrapping scripts to turn a vanilla OpenBSD install into a feature-rich gateway device It also makes use of Ansible playbooks for configuration, allowing for a more "mass deployment" type of setup Everything is configured via a simple text file, and you end up with a local NTP server, DHCP server, firewall (obviously) and local caching DNS resolver - it even does DNSSEC validation All the code is open source and on Github (https://github.com/ioc32/openhrc), so you can read through what's actually being changed and put in place There's also a video guide (https://www.youtube.com/watch?v=LZeKDM5jc90) to the entire process, if you're more of a visual person *** OPNsense 15.1.10 released (https://forum.opnsense.org/index.php?topic=365.0) Speaking of BSD routers, if you're looking for a "prebuilt and ready to go" option, OPNsense has just released a new version 15.1.10 drops some of the legacy patches they inherited from pfSense, aiming to stay closer to the mainline FreeBSD source code Going along with this theme, they've redone how they do ports, and are now kept totally in sync with the regular ports tree Their binary packages are now signed using the fingerprint-style method, various GUI menus have been rewritten and a number of other bugs were fixed NanoBSD-based images are also available now, so you can try it out on hardware with constrained resources as well Version 15.1.10.1 (https://twitter.com/opnsense/status/596009164746432512) was released shortly thereafter, including a hotfix for VLANs *** IBM Workpad Z50 and NetBSD (https://www.ibm.com/developerworks/community/blogs/hpcgoulash/entry/ibm_workpad_z50_netbsd_an_interesting_combination1?lang=en) Before the infamous netbook fad came and went, IBM had a handheld PDA device that looked pretty much the same Back in 1999, they released the Workpad Z50 (http://www.hpcfactor.com/reviews/hardware/ibm/workpad-z50/) with Windows CE, sporting a 131MHz MIPS CPU, 16MB of RAM and a 640x480 display You can probably tell where this is going... the article is about installing NetBSD it "What prevents me from taking my pristine Workpad z50 to the local electronics recycling facility is NetBSD. With a little effort it is possible to install recent versions of NetBSD on the Workpad z50 and even have XWindows running" The author got pkgsrc up and running on it too, and cleverly used distcc to offload the compiling jobs to something a bit more modern He's also got a couple (https://www.youtube.com/watch?v=hSLVnSZKB9I) videos (https://www.youtube.com/watch?v=mIA-NWEHLM4) of the bootup process and running Xorg (neither of which we'd call "speedy" by any stretch of the imagination) *** FreeBSD from the trenches (http://freebsdfoundation.blogspot.com/2015/04/from-trenches-tips-tricks-edition.html) The FreeBSD foundation has a new blog post up in their "from the trenches" series, detailing FreeBSD in some real-world use cases In this installment, Glen Barber talks about how he sets up all his laptops with ZFS and GELI While the installer allows for an automatic ZFS layout, Glen notes that it's not a one-size-fits-all thing, and goes through doing everything manually Each command is explained, and he walks you through the process of doing an encrypted installation (http://www.bsdnow.tv/tutorials/fde) on your root zpool *** Broadwell in DragonFly (http://lists.dragonflybsd.org/pipermail/users/2015-May/207671.html) DragonFlyBSD has officially won the race to get an Intel Broadwell graphics driver Their i915 driver has been brought up to speed with Linux 3.14's, adding not only Broadwell support, but many other bugfixes for other cards too It's planned for commit to the main tree very soon, but you can test it out with a git branch for the time being *** Feedback/Questions Bostjan writes in (http://slexy.org/view/s216QQcHyX) Hunter writes in (http://slexy.org/view/s21hGSk3c0) Hrishi writes in (http://slexy.org/view/s20JwPw9Je) Clint writes in (http://slexy.org/view/s2x1GYr7y6) Sergei writes in (http://slexy.org/view/s2swXxr2PX) *** Mailing List Gold How did you guess (https://lists.freebsd.org/pipermail/freebsd-advocacy/2015-May/004541.html) ***
© 2020-2025 Ivy Podcast Discovery LLC
