Podcast appearances and mentions of kris moore

This week Adam talks with Kris Moore, Senior Vice President of Engineering at iXsystems, about all things TrueNAS. They discuss the history of TrueNAS starting from its origins as a FreeBSD project, TrueNAS Core being in maintenance mode, the momentum and innovation happening in TrueNAS Scale, the evolution of the TrueNAS user interface, managing ZFS compatibility in TrueNAS, the business model of iXsystems and their commitment to the open-source community, and of course what's to come in the upcoming Dragonfish release of TrueNAS Scale.

This week Adam talks with Kris Moore, Senior Vice President of Engineering at iXsystems, about all things TrueNAS. They discuss the history of TrueNAS starting from its origins as a FreeBSD project, TrueNAS Core being in maintenance mode, the momentum and innovation happening in TrueNAS Scale, the evolution of the TrueNAS user interface, managing ZFS compatibility in TrueNAS, the business model of iXsystems and their commitment to the open-source community, and of course what's to come in the upcoming Dragonfish release of TrueNAS Scale.

Kris Moore is the Vice President of Engineering for iXsystems. Kris is an experienced executive with a demonstrated history of working in the computer software industry and is closely involved in the TrueNAS Open Storage community. He holds strong information technology skills and expertise in Shell scripting, C++, operating systems, FreeBSD, and system administration. He started learning about open source in the 90's where he was involved in developing web services using a very early version of FreeBSD. From that experience, he learned the basics of shell interface and how to check his e-mail without a GUI by using the “pine” command, which according to him, “was very uber-l33t” at the time. During his college years, Kris played around with other Unix and Linux platforms (caldera Linux, SuSE), however when he needed to do serious work, such as setting up a web or file server, he always fell back to his FreeBSD roots. To this day, he still prefers FreeBSD because it offers a greater degree of stability than other operating systems. You can find Kris Moore on the following sites: Twitter LinkedIn Here are some links provided by Kris Moore: iXsystems TrueNAS PLEASE SUBSCRIBE TO THE PODCAST Spotify: ⁠http://isaacl.dev/podcast-spotify⁠ Apple Podcasts: ⁠http://isaacl.dev/podcast-apple⁠ Google Podcasts: ⁠http://isaacl.dev/podcast-google⁠ RSS: ⁠http://isaacl.dev/podcast-rss⁠ You can check out more episodes of Coffee and Open Source on ⁠https://www.coffeeandopensource.com⁠ Coffee and Open Source is hosted by Isaac Levin (⁠https://twitter.com/isaacrlevin⁠) --- Support this podcast: https://podcasters.spotify.com/pod/show/coffeandopensource/support

Here are this week's special guests and energy experts on The Crude Life Week In Review Episode 488. Hunter Berg has a fantastic job these days. Finding students who want a “free education” at Williston State College. According to Berg, executive director, Williston State College Foundation, the university has been [...]

Kris Moore LinkedIn: https://www.linkedin.com/in/kris-moore-24b9612/ Notes: Credits: Music by ikson: https://www.iksonmusic.com Special Guest: Kris Moore.

Your hosts Donovon Moore and Ayush Ajmani invite a special guest on the show to discuss the history of the Ethiopian province and how the current instability in the Tigray region could cause another civil war.

Time Codes: 00:00:27 - Introduction 00:03:03 - Fare-thee-well to Paul! 00:05:51 - The Cardboard Kingdom 00:28:41 - All Summer Long 00:44:11 -Be Prepared 01:06:46 - Wrap up 01:08:09 - Contact us On this episode of the Comics Alternative's Young Readers show, Gwen and Derek discuss summer 2018 new releases, all geared to middle-grade readers. The first text, edited and illustrated by Chad Sell, is The Cardboard Kingdom, released by Random House Graphic. Readers learn about the lives and dreams of a group of neighborhood kids in short stories written by Jay Fuller, David DeMeo, Katie Schenkel, Kris Moore, Molly Muldoon, Vid Alliger, Manuel Betancourt, Michael Cole, Cloud Jacobs, and Barbara Perez Marquez. The second text they discuss is Hope Larson's All Summer Long from Farrar Straus Giroux. And finally, Gwen and Derek review Vera Brosgol's long-awaited memoir, Be Prepared, released by First Second Books. They start by reviewing a comic that demonstrates the experimentation that is currently taking place in the young reader category: Chad Sell's edited collection of connected short stories, The Cardboard Kingdom. Set in a suburban neighborhood and featuring a truly diverse and engaging group of young kids, these stories show how imagination can function as a coping device. Young reader short story collections are not entirely new. Random House has also supported the Comics Squadseries, edited by Jennifer Holm and a rotating cast of co-editors, that have included themed volumes on "Recess," "Lunch," and "Detention." However, this short story collection boast only one artist, Chad Sell, and is presented as a cohesive narrative, with authors identified only at the end of the text. As such, the comic has a more cohesive feel, rather than a collection of fundamentally different stories that are linked only by theme. After that, Gwen and Derek check out Hope Larson's All Summer Long. This is the story of a 13-year-old, Bina, and her attempts to find meaning over the course of a summer. Growing up, she had been used to spending the summer with her best friend Austin, but during this particular summer, Austin goes off to soccer camp, leaving Bina to fend for herself. And part of this fending includes Austin's aggressive older sister, Charlie. This is a book all about growing up and finding your way over the course of significant life changes. Finally, Gwen and Derek discuss Vera Brosgol's BePrepared. This memoir focuses on the summer between 4th and 5th grades, when she attends Russian summer camp in the hope of finding friends with whom she will have something in common. The opening vignette in the novel focuses on young Vera's sense of cultural and economic isolation: as an immigrant from Russia and the daughter of a single mom living in a prosperous east coast suburb, she is often slow to pick up on the latest trends -- such as American Girl dolls -- and unable to approximate the lavish birthday parties that her classmates' parents are able to throw for their children. At the end of the school year, Vera listens to the plans that her friends are making, attending girl scout camp, taking vacations to faraway destinations…and she feels left out again. However, at the Russian Orthodox church that she attends, Vera learns from Ksenya, a Sunday school friend, about Orra, a Russian heritage camp, and she is certain that it will not only be fun, but will give her something to talk about with her school friends in autumn. While the experience is certainly life changing, it nonetheless becomes something quite different than what Vera had expected.

We cover an interview about Unix Architecture Evolution, another vBSDcon trip report, how to teach an old Unix about backspace, new NUMA support coming to FreeBSD, and stack pointer checking in OpenBSD. This episode was brought to you by Headlines Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD (https://fosdem.org/2018/interviews/diomidis-spinellis/) Q: Could you briefly introduce yourself? I'm a professor of software engineering, a programmer at heart, and a technology author. Currently I'm also the editor in chief of the IEEE Software magazine. I recently published the book Effective Debugging, where I detail 66 ways to debug software and systems. Q: What will your talk be about, exactly? I will describe how the architecture of the Unix operating system evolved over the past half century, starting from an unnamed system written in PDP-7 assembly language and ending with a modern FreeBSD system. My talk is based, first, on a GitHub repository where I tried to record the system's history from 1970 until today and, second, on the evolution of documented facilities (user commands, system calls, library functions) across revisions. I will thus present the early system's defining architectural features (layering, system calls, devices as files, an interpreter, and process management) and the important ones that followed in subsequent releases: the tree directory structure, user contributed code, I/O redirection, the shell as a user program, groups, pipes, scripting, and little languages. Q: Why this topic? Unix stands out as a major engineering breakthrough due to its exemplary design, its numerous technical contributions, its impact, its development model, and its widespread use. Furthermore, the design of the Unix programming environment has been characterized as one offering unusual simplicity, power, and elegance. Consequently, there are many lessons that we can learn by studying the evolution of the Unix architecture, which we can apply to the design of new systems. I often see modern systems that suffer from a bloat of architectural features and a lack of clear form on which functionality can be built. I believe that many of the modern Unix architecture defining features are excellent examples of what we should strive toward as system architects. Q: What do you hope to accomplish by giving this talk? What do you expect? I'd like FOSDEM attendees to leave the talk with their mind full with architectural features of timeless quality. I want them to realize that architectural elegance isn't derived by piling design patterns and does not need to be expensive in terms of resources. Rather, beautiful architecture can be achieved on an extremely modest scale. Furthermore, I want attendees to appreciate the importance of adopting flexible conventions rather than rigid enforcement mechanisms. Finally, I want to demonstrate through examples that the open source culture was part of Unix from its earliest days. Q: What are the most significant milestones in the development of Unix? The architectural development of Unix follows a path of continuous evolution, albeit at a slowing pace, so I don't see here the most important milestones. I would however define as significant milestones two key changes in the way Unix was developed. The first occurred in the late 1970s when significant activity shifted from a closely-knit team of researchers at the AT&T Bell Labs to the Computer Science Research Group in the University of California at Berkeley. This opened the system to academic contributions and growth through competitive research funding. The second took place in the late 1980s and the 1990s when Berkeley open-sourced the the code it had developed (by that time a large percentage of the system) and enthusiasts built on it to create complete open source operating system distributions: 386BSD, and then FreeBSD, NetBSD, OpenBSD, and others. Q: In which areas has the development of Unix stalled? The data I will show demonstrate that there were in the past some long periods where the number of C library functions and system calls remained mostly stable. Nowadays there is significant growth in the number of all documented facilities with the exception of file formats. I'm looking forward to a discussion regarding the meaning of these growth patterns in the Q&A session after the talk. Q: What are the core features that still link the 1970 PDP-7 system to the latest FreeBSD 11.1 release, almost half a century apart? Over the past half-century the Unix system has grown by four orders of magnitude from a few thousand lines of code to many millions. Nevertheless, looking at a 1970s architecture diagram and a current one reveals that the initial architectural blocks are still with us today. Furthermore, most system calls, user programs, and C library functions of that era have survived until today with essentially similar functionality. I've even found in modern FreeBSD some lines of code that have survived unchanged for 40 years. Q: Can we still add innovative changes to operating systems like FreeBSD without breaking the ‘Unix philosophy'? Will there be a moment where FreeBSD isn't recognizable anymore as a descendant of the 1970 PDP-7 system? There's a saying that “form liberates”. So having available a time-tested form for developing operating system functionality allows you to innovate in areas that matter rather than reinventing the wheel. Such concepts include having commands act as a filter, providing manual pages with a consistent structure, supplying build information in the form of a Makefile, installing files in a well-defined directory hierarchy, implementing filesystems with an standardized object-oriented interface, and packaging reusable functions as a library. Within this framework there's ample space for both incremental additions (think of jq, the JSON query command) and radical innovations (consider the Solaris-derived ZFS and dtrace functionality). For this reason I think that BSD and Linux systems will always be recognizable as direct or intellectual descendants of the 1970s Research Unix editions. Q: Have you enjoyed previous FOSDEM editions? Immensely! As an academic I need to attend many scientific conferences and meetings in order to present research results and interact with colleagues. This means too much time spent traveling and away from home, and a limited number of conferences I'm in the end able to attend. Nevertheless, attending FOSDEM is an easy decision due to the world-changing nature of its theme, the breadth of the topics presented, the participants' enthusiasm and energy, as well as the exemplary, very efficient conference organization. Another vBSDCon trip report we just found (https://www.weaponizedawesome.com/blog/?cat=53) We just got tipped about another trip report from vBSDCon, this time from one of the first time speakers: W. Dean Freeman Recently I had the honor of co-presenting on the internals of FreeBSD's Kernel RNG with John-Mark Gurney at the 3rd biennial vBSDCon, hosted in Reston, VA hosted by Verisign. I've been in and out of the FreeBSD community for about 20 years. As I've mentioned on here before, my first Unix encounter was FreeBSD 2.2.8 when I was in the 7th or 8th grade. However, for all that time I've never managed to get out to any of the cons. I've been to one or two BUG meetings and I've met some folks from IRC before, but nothing like this. A BSD conference is a very different experience than anything else out there. You have to try it, it is the only way to truly understand it. I'd also not had to do a stand-up presentation really since college before this. So, my first BSD con and my first time presenting rolled into one made for an interesting experience. See, he didn't say terrifying. It went very well. You should totally submit a talk for the next conference, even if it is your first. That said, it was amazing and invigorating experience. I got to meet a few big names in the FreeBSD community, discuss projects, ideas for FreeBSD, etc. I did seem to spend an unusual amount of time talking about FIPS and Common Criteria with folks, but to me that's a good sign and indicative that there is interest in working to close gaps between FreeBSD and the current requirements so that we can start getting FreeBSD and more BSD-based products into the government and start whittling away the domination of Linux (especially since Oracle has cut Solaris, SPARC and the ZFS storage appliance business units). There is nothing that can match the high bandwidth interchange of ideas in person. The internet has made all kinds of communication possible, and we use it all the time, but every once in a while, getting together in person is hugely valuable. Dean then went on to list some of the talks he found most valuable, including DTrace, Capsicum, bhyve, *BSD security tools, and Paul Vixie's talk about gets() I think the talk that really had the biggest impact on me, however, was Kyle Kneisl's talk on BSD community dynamics. One of the key points he asked was whether the things that drew us to the BSD community in the first place would be able to happen today. Obviously, I'm not a 12 or 13 year old kid anymore, but it really got me thinking. That, combined with getting face time with people I'd previously only known as screen names has recently drawn me back into participating in IRC and rejoining mailing lists (wdf on freenode. be on the lookout!) Then Dean covered some thoughts on his own talk: JMG and my talk seems to have been well received, with people paying lots of attention. I don't know what a typical number of questions is for one of these things, but on day one there weren't that many questions. We got about 5 during our question time and spent most of the rest of the day fielding questions from interested attendees. Getting a “great talk!” from GNN after coming down from the stage was probably one of the major highlights for me. I remember my first solo talk, and GNN asking the right question in the middle to get me to explain a part of it I had missed. It was very helpful. I think key to the interest in our presentation was that JMG did a good job framing a very complicated topic's importance in terms everyone could understand. It also helped that we got to drop some serious truth bombs. Final Thoughts: I met a lot of folks in person for the first time, and met some people I'd never known online before. It was a great community and I'm glad I got a chance to expand my network. Verisign were excellent hosts and they took good care of both speakers (covering airfare, rooms, etc.) and also conference attendees at large. The dinners that they hosted were quite good as well. I'm definitely interested in attending vBSDCon again and now that I've had a taste of meeting IRL with the community on scale of more than a handful, I have every intention of finally making it to BSDCan next year (I'd said it in 2017, but then moved to Texas for a new job and it wasn't going to be practical). This year for sure, though! Teaching an Almost 40-year Old UNIX about Backspace (https://virtuallyfun.com/2018/01/17/teaching_an_almost_40-year_old_unix_about_backspace/) Introduction I have been messing with the UNIX® operating system, Seventh Edition (commonly known as UNIX V7 or just V7) for a while now. V7 dates from 1979, so it's about 40 years old at this point. The last post was on V7/x86, but since I've run into various issues with it, I moved on to a proper installation of V7 on SIMH. The Internet has some really good resources on installing V7 in SIMH. Thus, I set out on my own journey on installing and using V7 a while ago, but that was remarkably uneventful. One convenience that I have been dearly missing since the switch from V7/x86 is a functioning backspace key. There seem to be multiple different definitions of backspace: BS, as in ASCII character 8 (010, 0x08, also represented as ^H), and DEL, as in ASCII character 127 (0177, 0x7F, also represented as ^?). V7 does not accept either for input by default. Instead, # is used as the erase character and @ is used as the kill character. These defaults have been there since UNIX V1. In fact, they have been “there” since Multics, where they got chosen seemingly arbitrarily. The erase character erases the character before it. The kill character kills (deletes) the whole line. For example, “ba##gooo#d” would be interpreted as “good” and “bad line@good line” would be interpreted as “good line”. There is some debate on whether BS or DEL is the correct character for terminals to send when the user presses the backspace key. However, most programs have settled on DEL today. tmux forces DEL, even if the terminal emulator sends BS, so simply changing my terminal to send BS was not an option. The change from the defaults outlined here to today's modern-day defaults occurred between 4.1BSD and 4.2BSD. enf on Hacker News has written a nice overview of the various conventions Getting the Diff For future generations as well as myself when I inevitably majorly break this installation of V7, I wanted to make a diff. However, my V7 is installed in SIMH. I am not a very intelligent man, I didn't keep backup copies of the files I'd changed. Getting data out of this emulated machine is an exercise in frustration. In the end, I printed everything on screen using cat(1) and copied that out. Then I performed a manual diff against the original source code tree because tabs got converted to spaces in the process. Then I applied the changes to clean copies that did have the tabs. And finally, I actually invoked diff(1). Closing Thoughts Figuring all this out took me a few days. Penetrating how the system is put together was surprisingly fairly hard at first, but then the difficulty curve eased up. It was an interesting exercise in some kind of “reverse engineering” and I definitely learned something about tty handling. I was, however, not pleased with using ed(1), even if I do know the basics. vi(1) is a blessing that I did not appreciate enough until recently. Had I also been unable to access recursive grep(1) on my host and scroll through the code, I would've probably given up. Writing UNIX under those kinds of editing conditions is an amazing feat. I have nothing but the greatest respect for software developers of those days. News Roundup New NUMA support coming to FreeBSD CURRENT (https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068145.html) Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem, and vmpage*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VMNUMAALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VMNUMAALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff Stack pointer checking – OpenBSD (https://marc.info/?l=openbsd-tech&m=151572838911297&w=2) Stefan (stefan@) and I have been working for a few months on this diff, with help from a few others. At every trap and system call, it checks if the stack-pointer is on a page that is marked MAPSTACK. execve() is changed to create such mappings for the process stack. Also, libpthread is taught the new MAPSTACK flag to use with mmap(). There is no corresponding system call which can set MAP_FLAG on an existing page, you can only set the flag by mapping new memory into place. That is a piece of the security model. The purpose of this change is to twart stack pivots, which apparently have gained some popularity in JIT ROP attacks. It makes it difficult to place the ROP stack in regular data memory, and then perform a system call from it. Workarounds are cumbersome, increasing the need for far more gadgetry. But also the trap case -- if any memory experiences a demand page fault, the same check will occur and potentially also kill the process. We have experimented a little with performing this check during device interrupts, but there are some locking concerns and performance may then become a concern. It'll be best to gain experience from handle of syncronous trap cases first. chrome and other applications I use run fine! I'm asking for some feedback to discover what ports this breaks, we'd like to know. Those would be ports which try to (unconventionally) create their stacks in malloc()'d memory or inside another Data structure. Most of them are probably easily fixed ... Qt 5.9 on FreeBSD (https://euroquis.nl/bobulate/?p=1768) Tobias and Raphael have spent the past month or so hammering on the Qt 5.9 branch, which has (finally!) landed in the official FreeBSD ports tree. This brings FreeBSD back up-to-date with current Qt releases and, more importantly, up-to-date with the Qt release KDE software is increasingly expecting. With Qt 5.9, the Elisa music player works, for instance (where it has run-time errors with Qt 5.7, even if it compiles). The KDE-FreeBSD CI system has had Qt 5.9 for some time already, but that was hand-compiled and jimmied into the system, rather than being a “proper” ports build. The new Qt version uses a new build system, which is one of the things that really slowed us down from a packaging perspective. Some modules have been reshuffled in the process. Some applications depending on Qt internal-private headers have been fixed along the way. The Telegram desktop client continues to be a pain in the butt that way. Following on from Qt 5.9 there has been some work in getting ready for Clang 6 support; in general the KDE and Qt stack is clean and modern C++, so it's more infrastructural tweaks than fixing code. Outside of our silo, I still see lots of wonky C++ code being fixed and plenty of confusion between pointers and integers and strings and chars and .. ugh. Speaking of ugh, I'm still planning to clean up Qt4 on ARM aarch64 for FreeBSD; this boils down to stealing suitable qatomic implementations from Arch Linux. For regular users of Qt applications on FreeBSD, there should be few to no changes required outside the regular upgrade cycle. For KDE Plasma users, note that development of the ports has changed branches; as we get closer to actually landing modern KDE bits, things have been renamed and reshuffled and mulled over so often that the old plasma5 branch wasn't really right anymore. The kde5-import branch is where it's at nowadays, and the instructions are the same: the x11/kde5 metaport will give you all the KDE Frameworks 5, KDE Plasma Desktop and modern KDE Applications you need. Adding IPv6 to an Nginx website on FreeBSD / FreshPorts (https://dan.langille.org/2018/01/13/adding-ipv6-to-an-nginx-website-on-freebsd-freshports/) FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized. There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation. All of this took about 3 hours. We are using: FreeBSD 11.1 Bind 9.9.11 nginx 1.12.2 Fallout I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses. ZFS on TrueOS: Why We Love OpenZFS (https://www.trueos.org/blog/zfs-trueos-love-openzfs/) TrueOS was the first desktop operating system to fully implement the OpenZFS (Zettabyte File System or ZFS for short) enterprise file system in a stable production environment. To fully understand why we love ZFS, we will look back to the early days of TrueOS (formerly PC-BSD). The development team had been using the UFS file system in TrueOS because of its solid track record with FreeBSD-based computer systems and its ability to check file consistency with the built-in check utility fsck. However, as computing demands increased, problems began to surface. Slow fsck file verification on large file systems, slow replication speeds, and inconsistency in data integrity while using UFS logging / journaling began to hinder users. It quickly became apparent that TrueOS users would need a file system that scales with evolving enterprise storage needs, offers the best data protection, and works just as well on a hobbyist system or desktop computer. Kris Moore, the founder of the TrueOS project, first heard about OpenZFS in 2007 from chatter on the FreeBSD mailing lists. In 2008, the TrueOS development team was thrilled to learn that the FreeBSD Project had ported ZFS. At the time, ZFS was still unproven as a graphical desktop solution, but Kris saw a perfect opportunity to offer ZFS as a cutting-edge file system option in the TrueOS installer, allowing the TrueOS project to act as an indicator of how OpenZFS would fair in real-world production use. The team was blown away by the reception and quality of OpenZFS on FreeBSD-based systems. By its nature, ZFS is a copy-on-write (CoW) file system that won't move a block of data until it both writes the data and verifies its integrity. This is very different from most other file systems in use today. ZFS is able to assure that data stays consistent between writes by automatically comparing write checksums, which mitigates bit rot. ZFS also comes with native RaidZ functionality that allows for enterprise data management and redundancy without the need for expensive traditional RAID cards. ZFS snapshots allow for system configuration backups in a split-second. You read that right. TrueOS can backup or restore snapshots in less than a second using the ZFS file system. Given these advantages, the TrueOS team decided to use ZFS as its exclusive file system starting in 2013, and we haven't looked back since. ZFS offers TrueOS users the stable workstation experience they want, while simultaneously scaling to meet the increasing demands of the enterprise storage market. TrueOS users are frequently commenting on how easy it is to use ZFS snapshots with our built-in snapshot utility. This allows users the freedom to experiment with their system knowing they can restore it in seconds if anything goes wrong. If you haven't had a chance to try ZFS with TrueOS, browse to our download page and make sure to grab a copy of TrueOS. You'll be blown away by the ease of use, data protection functionality, and incredible flexibility of RaidZ. Beastie Bits Source Code Podcast Interview with Michael W Lucas (https://blather.michaelwlucas.com/archives/3099) Operating System of the Year 2017: NetBSD Third place (https://w3techs.com/blog/entry/web_technologies_of_the_year_2017) OPNsense 18.1-RC1 released (https://opnsense.org/opnsense-18-1-rc1-released/) Personal OpenBSD Wiki Notes (https://balu-wiki.readthedocs.io/en/latest/security/openbsd.html) BSD section can use some contribution (https://guide.freecodecamp.org/bsd-os/) The Third Research Edition Unix Programmer's Manual (now available in PDF) (https://github.com/dspinellis/unix-v3man) Feedback/Questions Alex - my first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) Brian - Brian - openbsd 6.2 and enlightenment .17 (http://dpaste.com/24DYF1J#wrap) ***

We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)

OpenBSD 6.2 is here, style arguments, a second round of viewer interview questions, how to set CPU affinity for FreeBSD jails, containers on FreeNAS & more! Headlines OpenBSD 6.2 Released (https://www.openbsd.org/62.html) OpenBSD continues their six month release cadence with the release of 6.2, the 44th release On a disappointing note, the song for 6.2 will not be released until December Highlights: Improved hardware support on modern platforms including ARM64/ARMv7 and octeon, while amd64 users will appreciate additional support for the Intel Kaby Lake video cards. Network stack improvements include extensive SMPization improvements and a new FQ-CoDel queueing discipline, as well as enhanced WiFi support in general and improvements to iwn(4), iwm(4) and anthn(4) drivers. Improvements in vmm(4)/vmd include VM migration, as well as various compatibility and performance improvements. Security enhancements including a new freezero(3) function, further pledge(2)ing of base system programs and conversion of several daemons to the fork+exec model. Trapsleds, KARL, and random linking for libcrypto and ld.so, dramatically increase security by making it harder to find helpful ROP gadgets, and by creating a unique order of objects per-boot. A unique kernel is now created by the installer to boot from after install/upgrade. The base system compiler on the amd64 and i386 platforms has switched to clang(1). New versions of OpenSSH, OpenSMTPd, LibreSSL and mandoc are also included. The kernel no longer handles IPv6 Stateless Address Autoconfiguration (RFC 4862), allowing cleanup and simplification of the IPv6 network stack. Improved IPv6 checks for IPsec policies and made them consistent with IPv4. Enabled the use of per-CPU caches in the network packet allocators. Improved UTF-8 line editing support for ksh(1) Emacs and Vi input mode. breaking change for nvme(4) users with GPT: If you are booting from an nvme(4) drive with a GPT disk layout, you are affected by an off-by-one in the driver with the consequence that the sector count in your partition table may be incorrect. The only way to fix this is to re-initialize the partition table. Backup your data to another disk before you upgrade. In the new bsd.rd, drop to a shell and re-initialize the GPT: fdisk -iy -g -b 960 sdN Why we argue: style (https://www.sandimetz.com/blog/2017/6/1/why-we-argue-style) I've been thinking about why we argue about code, and how we might transform vehement differences of opinion into active forces for good. My thoughts spring from a very specific context. Ten or twelve times a year I go to an arbitrary business and spend three or more days teaching a course in object-oriented design. I'm an outsider, but for a few days these business let me in on their secrets. Here's what I've noticed. In some places, folks are generally happy. Programmers get along. They feel as if they are all "in this together." At businesses like this I spend most of my time actually teaching object-oriented design. Other places, folks are surprisingly miserable. There's a lot of discord, and the programmers have devolved into competing "camps." In these situations the course rapidly morphs away from OO Design and into wide-ranging group discussions about how to resolve deeply embedded conflicts. Tolstoy famously said that "Happy families are all alike; every unhappy family is unhappy in its own way." This is known as the Anna Karenina Principle, and describes situations in which success depends on meeting all of a number of criteria. The only way to be happy is to succeed at every one of them. Unhappiness, unfortunately, can be achieved by any combination of failure. Thus, all happy businesses are similar, but unhappy ones appear unique in their misery. Today I'm interested in choices of syntax, i.e whether or not your shop has agreed upon and follows a style guide. If you're surprised that I'm starting with this apparently mundane issue, consider yourself lucky in your choice of workplace. If you're shaking your head in rueful agreement about the importance of this topic, I feel your pain. I firmly believe that all of the code that I personally have to examine should come to me in a consistent format. Code is read many more times than it is written, which means that the ultimate cost of code is in its reading. It therefore follows that code should be optimized for readability, which in turn dictates that an application's code should all follow the same style. This is why FreeBSD, and most other open source projects, have a preferred style. Some projects are less specific and less strict about it. Most programmers agree with the prior paragraph, but here's where things begin to break down. As far as I'm concerned, my personal formatting style is clearly the best. However, I'm quite sure that you feel the same. It's easy for a group of programmers to agree that all code should follow a common style, but surprisingly difficult to get them to agree on just what that common style should be. Avoid appointing a human "style cop", which just forces someone to be an increasingly ill-tempered nag. Instead, supply programmers with the information they need to remedy their own transgressions. By the time a pull request is submitted, mis-stylings should long since have been put right. Pull request conversations ought to be about what code does rather than how code looks. What about old code? Ignore it. You don't have to re-style all existing code, just do better from this day forward. Defer updating old code until you touch it for other reasons. Following this strategy means that the code you most often work on will gradually take on a common style. It also means that some of your existing code might never get updated, but if you never look at it, who cares? If you choose to re-style code that you otherwise have no need to touch, you're declaring that changing the look of this old code has more value to your business than delivering the next item on the backlog. The opportunity cost of making a purely aesthetic change includes losing the benefit of what you could have done instead. The rule-of-thumb is: Don't bother updating the styling of stable, existing code unless not doing so costs you money. Most open source projects also avoid reformatting code just to change the style, because of the merge conflicts this will cause for downstream consumers If you disagree with the style guide upon which your team agrees, you have only two honorable options: First, you can obey the guide despite your aversion. As with me in the Elm story above, this act is likely to change your thinking so that over time you come to prefer the new style. It's possible that if you follow the guide you'll begin to like it. Alternatively, you can decide you will not obey the style guide. Making this decision demands that you leave your current project and find some other project whose guide matches your preferred style. Go there and follow that one. Notice that both of these choices have you following a guide. This part is not optional. The moral of this story? It's more important for all code to be formatted the same than it is for any one of us to get our own way. Commit to agreeing upon and following a style guide. And if you find that your team cannot come to an agreement, step away from this problem and start a discussion about power. There have been many arguments about style, and it can often be one of the first complaints of people new to any open source project This article covers it fairly well from both sides, a) you should follow the style guide of the project you are contributing to, b) the project should review your actual code, then comment on the style after, and provide gentle guidance towards the right style, and avoid being “style cops” *** Interview - The BSDNow Crew, Part II News Roundup Building FreeBSD for the Onion Omega 2 (https://github.com/sysadminmike/freebsd-onion-omega2-build) I got my Onion Omega 2 devices in the mail quite a while ago, but I had never gotten around to trying to install FreeBSD on them. They are a different MIPS SoC than the Onion Omega 1, so it would not work out of the box at the time. Now, the SoC is supported! This guide provides the steps to build an image for the Omega 2 using the freebsd-wifi-build infrastructure First some config files are modified to make the image small enough for the Omega 2's flash chip The DTS (Device Tree Source) files are not yet included in FreeBSD, so they are fetched from github Then the build for the ralink SoC is run, with the provided DTS file and the MT7628_FDT kernel config Once the build is complete, you'll have a tftp image file. Then that image is compressed, and bundled into a uboot image Write the files to a USB stick, and plug it into the Omega's dock Turn it on while holding the reset button with console open Press 1 to get into the command line. You will need to reset the usb: usb reset Then load the kernel boot image: fatload usb 0:1 0x80800000 kernel.MT7628_FDT.lzma.uImage And boot it: bootm 0x80800000 At this point FreeBSD should boot Mount a userland, and you should end up in multi-user mode Hopefully this will get even easier in the next few weeks, and we'll end up with a more streamlined process to tftp boot the device, then write FreeBSD into the onboard flash so it boots automatically. *** Setting the CPU Affinity on FreeBSD Jails with ezjail (https://www.neelc.org/setting-the-cpu-affinity-on-freebsd-jails-with-ezjail/) While there are more advanced resource controls available for FreeBSD jails, one of the most basic ways to control CPU usage is to limit the subset of CPUs that each jail can use. This can make sure that every jail has access to some dedicated resources, while at the same time doesn't have the ability to entirely dominate the machine I just got a new home server: a HP ProLiant ML110 G6. Being a FreeBSD person myself, it was natural that I used it on my server instead of Linux I chose to use ezjail to manage the jails on my ProLiant, with the initial one being a Tor middle node. Despite the fact that where my ML110 is, the upstream is only 35mbps (which is pretty good for cable), I did not want to give my Tor jail access to all four cores. Setting the CPU Affinity would let you choose a specific CPU core (or a range of cores) you want to use. However, it does not just let you pick the number of CPU cores you want and make FreeBSD choose the core running your jail. Going forward, I assumed that you have already created a jail using ezjail-admin. I also do not cover limiting a jail to a certain percentage of CPU usage. ezjail-admin config -c [CORENUMBERFIRST]-[CORENUMBERLAST] [JAIL_NAME] or ezjail-admin config -c [CORENUMBERFIRST],[CORENUMBERSECOND],...,[CORENUMBERN] [JAILNAME] And hopefully, you should have your ezjail-managed FreeBSD jail limited to the CPU cores you want. While I did not cover a CPU percentage or RAM usage, this can be done with rctl I'll admit: it doesn't really matter which CPU a jail runs on, but it might matter if you don't want a jail to have access to all the CPU cores available and only want [JAILNAME] to use one core. Since it's not really possible just specify the number of CPU cores with ezjail (or even iocell), a fallback would be to use CPU affinity, and that requires you to specify an exact CPU core. I know it's not the best solution (it would be better if we could let the scheduler choose provided a jail only runs on one core), but it's what works. We use this at work on high core count machines. When we have multiple databases colocated on the same machine, we make sure each one has a few cores to itself, while it shares other cores with the rest of the machine. We often reserve a core or two for the base system as well. *** A practical guide to containers on FreeNAS for a depraved psychopath. (https://medium.com/@andoriyu/a-practical-guide-to-containers-on-freenas-for-a-depraved-psychopath-c212203c0394) If you are interested in playing with Docker, this guide sets up a Linux VM running on FreeBSD or FreeNAS under bhyve, then runs linux docker containers on top of it You know that jails are dope and I know that jails are dope, yet no one else knows it. So here we are stuck with docker. Two years ago I would be the last person to recommend using docker, but a whole lot of things has changes past years… This tutorial uses iohyve to manage the VMs on the FreeBSD or FreeNAS There are many Linux variants you can choose from — RancherOS, CoreOS are the most popular for docker-only hosts. We going to use RancherOS because it's more lightweight out of the box. Navigate to RancherOS website and grab link to latest version sudo iohyve setup pool=zpool kmod=1 net=em0 sudo iohyve fetch https://releases.rancher.com/os/latest/rancheros.iso sudo iohyve renameiso rancheros.iso rancheros-v1.0.4.iso sudo pkg install grub2-bhyve sudo iohyve create rancher 32G sudo iohyve set rancher loader=grub-bhyve ram=8G cpu=8 con=nmdm0 os=debian sudo iohyve install rancher rancheros-v1.0.4.iso sudo iohyve console rancher Then the tutorial does some basic configuration of RancherOS, and some house keeping in iohyve to make RancherOS come up unattended at boot The whole point of this guide is to reduce pain, and using the docker CLI is still painful. There are a lot of Web UIs to control docker. Most of them include a lot of orchestrating services, so it's just overkill. Portainer is very lightweight and can be run even on Raspberry Pi Create a config file as described After reboot you will be able to access WebUI on 9000 port. Setup is very easy, so I won't go over it The docker tools for FreeBSD are still being worked on. Eventually you will be able to host native FreeBSD docker containers on FreeBSD jails, but we are not quite there yet In the meantime, you can install sysutils/docker and use it to manage the docker instances running on a remote machine, or in this case, the RancherOS VM running in bhyve *** Beastie Bits The Ghost of Invention: A Visit to Bell Labs, excerpt from the forthcoming book: “Kitten Clone: Inside Alcatel-Lucent” (https://www.wired.com/2014/09/coupland-bell-labs/) OpenBSD Cookbook (set of Ansible playbooks) (https://github.com/ligurio/openbsd-cookbooks) 15 useful sockstat commands to find open ports on FreeBSD (https://www.tecmint.com/sockstat-command-examples-to-find-open-ports-in-freebsd/) A prehistory of Slashdot (https://medium.freecodecamp.org/a-pre-history-of-slashdot-6403341dabae) Using ed, the unix line editor (https://medium.com/@claudio.santos.ribeiro/using-ed-the-unix-line-editor-557ed6466660) *** Feedback/Questions Malcolm - ZFS snapshots (http://dpaste.com/16EB3ZA#wrap) Darryn - Zones (http://dpaste.com/1DGHQJP#wrap) Mohammad - SSH Keys (http://dpaste.com/08G3VTB#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)

We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***

This episode gives you the full dose of BSDCan 2017 recap as well as a blog post on conference speaking advice. Headlines Pre-conference activities: Goat BoF, FreeBSD Foundation Board Meeting, and FreeBSD Journal Editorial Board Meeting The FreeBSD Foundation has a new President as Justin Gibbs is busy this year with building a house, so George Neville-Neil took up the task to serve as President, with Justin Gibbs as Secretary. Take a look at the updated Board of Directors (https://www.freebsdfoundation.org/about/board-of-directors/). We also have a new staff member (https://www.freebsdfoundation.org/about/staff/): Scott Lamons joined the Foundation team as senior program manager. Scott's work for the Foundation will focus on managing and evangelizing programs for advanced technologies in FreeBSD including preparing project plans, coordinating resources, and facilitating interactions between commercial vendors, the Foundation, and the FreeBSD community. The Foundation also planned various future activities, visits of upcoming conferences, and finding new ways to support and engage the community. The Foundation now has interns in the form of co-op students from the University of Waterloo, Canada. This is described further in the May 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/may-2017-development-projects-update/). Both students (Siva and Charlie) were also the conference, helping out at the Foundation table, demonstrating the tinderbox dashboard. Follow the detailed instructions (https://www.freebsdfoundation.org/news-and-events/blog/blog-post/building-a-physical-freebsd-build-status-dashboard/) to build one of your own. The Foundation put out a call for Project Proposal Solicitation for 2017 (https://www.freebsdfoundation.org/blog/freebsd-foundation-2017-project-proposal-solicitation/). If you think you have a good proposal for work relating to any of the major subsystems or infrastructure for FreeBSD, we'd be happy to review it. Don't miss the deadlines for travel grants to some of the upcoming conferences. You can find the necessary forms and deadlines at the Travel Grant page (https://www.freebsdfoundation.org/what-we-do/travel-grants/travel-grants/) on the Foundation website. Pictures from the Goat BoF can be found on Keltia.net (https://assets.keltia.net/photos/BSDCan-2017/Royal%20Oak/index.html) Overlapping with the GoatBoF, members of the FreeBSD Journal editorial board met in a conference room in the Novotel to plan the upcoming issues. Topics were found, authors identified, and new content was discussed to appeal to even more readers. Check out the FreeBSD Journal website (https://www.freebsdfoundation.org/journal/) and subscribe if you like to support the Foundation in that way. FreeBSD Devsummit Day 1 & 2 (https://wiki.freebsd.org/DevSummit/201706) The first day of the Devsummit began with introductory slides by Gordon Tetlow, who organized the devsummit very well. Benno Rice of the FreeBSD core team presented the work done on the new Code of Conduct, which will become effective soon. A round of Q&A followed, with positive feedback from the other devsummit attendees supporting the new CoC. After that, Allan Jude joined to talk about the new FreeBSD Community Proposal (FCP) (https://github.com/freebsd/fcp) process. Modelled after IETF RFCs, Joyent RFDs, and Python PEP, it is a new way for the project to reach consensus on the design or implementation of new features or processes. The FCP repo contains FCP#0 that describes the process, and a template for writing a proposal. Then, the entire core team (except John Baldwin, who could not make it this year) and core secretary held a core Q&A session, Answering questions, gathering feedback and suggestions. After the coffee break, we had a presentation about Intel's QAT integration in FreeBSD. When the lunch was over, people spread out into working groups about BearSSL, Transport (TCP/IP), and OpenZFS. OpenZFS working group (https://pbs.twimg.com/media/DBu_IMsWAAId2sN.jpg:large): Matt Ahrens lead the group, and spent most of the first session providing a status update about what features have been recently committed, are out for review, on the horizon, or in the design phase. Existing Features Compressed ARC Compressed Send/Recv Recently Upstreamed A recent commit improved RAID-Z write speeds by declaring writes to padding blocks to be optional, and to always write them if they can be aggregated with the next write. Mostly impacts large record sizes. ABD (ARC buffer scatter/gather) Upstreaming In Progress Native Encryption Channel Programs Device Removal (Mirrors and Stripes) Redacted Send/recv Native TRIM Support (FreeBSD has its own, but this is better and applies to all ZFS implementations) Faster (mostly sequential) scrub/resilver DRAID (A great deal of time was spent explaining how this works, with diagrams on the chalk board) vdev metadata classes (store metadata on SSDs with data is on HDDs, or similar setups. Could also be modified to do dedup to SSD) Multi-mount protection (“safe import”, for dual-headed storage shelves) zpool checkpoint (rollback an entire pool, including zfs rename and zfs destroy) Further Out Import improvements Import with missing top-level vdevs (some blocks unreadable, but might let you get some data) Improved allocator performance -- vdev spacemap log ZIL performance Persistent L2ARC ZSTD Compression Day 2 Day two started with the Have/Want/Need session for FreeBSD 12.0. A number of features that various people have or are in the process of building, were discussed with an eye towards upstreaming them. Features we want to have in time for 12.0 (early 2019) were also discussed. After the break was the Vendor summit, which continued the discussion of how FreeBSD and its vendors can work together to make a better operating system, and better products based on it After lunch, the group broke up into various working groups: Testing/CI, Containers, Hardening UFS, and GELI Improvements Allan lead the GELI Improvements session. The main thrust of the discussions was fixing an outstanding bug in GELI when using both key slots with passphrases. To solve this, and make GELI more extensible, the metadata format will be extended to allow it to store more than 512 bytes of data (currently 511 bytes are used). The new format will allow arbitrarily large metadata, defined at creation time by selecting the number of user key slots desired. The new extended metadata format will contain mostly the same fields, except the userkey will no longer be a byte array of IV-key, Data-key, HMAC, but a struct that will contain all data about that key This new format will store the number of pkcs5v2 iterations per key, instead of only having a single location to store this number for all keys (the source of the original bug) A new set of flags per key, to control some aspects of the key (does it require a keyfile, etc), as well as possibly the role of the key. An auxdata field related to the flags, this would allow a specific key with a specific flag set, to boot a different partition, rather than decrypt the main partition. A URI to external key material is also stored per key, allowing GELI to uniquely identify the correct data to load to be able to use a specific decryption key And the three original parts of the key are stored in separate fields now. The HMAC also has a type field, allowing for a different HMAC algorithm to be used in the future. The main metadata is also extended to include a field to store the number of user keys, and to provide an overall HMAC of the metadata, so that it can be verified using the master key (provide any of the user keys) Other topics discussed: Ken Merry presented sedutil, a tool for managing Self Encrypting Drives, as may be required by certain governments and other specific use cases. Creating a deniable version of GELI, where the metadata is also encrypted The work to implemented GELI in the UEFI loader was discussed, and a number of developers volunteered to review and test the code Following the end of the Dev Summit, the “Newcomers orientation and mentorship” session was run by Michael W. Lucas, which attempts to pair up first time attendees with oldtimers, to make sure they always know a few people they can ask if they have questions, or if they need help getting introduced to the right people. News Roundup Conference Day 1 (http://www.bsdcan.org/2017/schedule/day_2017-06-09.en.html) The conference opened with some short remarks from Dan Langille, and then the opening keynote by Dr Michael Geist, a law professor at the University of Ottawa where he holds the Canada Research Chair in Internet and E-commerce Law. The keynote focused on what some of the currently issues are, and how the technical community needs to get involved at all levels. In Canada especially, contacting your representatives is quite effective, and when it does not happen, they only hear the other side of the story, and often end up spouting talking points from lobbyists as if they were facts. The question period for the keynote ran well overtime because of the number of good questions the discussion raised, including how do we fight back against large telcos with teams of lawyers and piles of money. Then the four tracks of talks started up for the day The day wrapped up with the Work In Progress (WIP) session. Allan Jude presented work on ZSTD compression in ZFS Drew Gallatin presented about work at Netflix on larger mbufs, to avoid the need for chaining and to allow more data to be pushed at once. Results in an 8% CPU time reduction when pushing 90 gbps of TLS encrypted traffic Dan Langille presented about letsencrypt (the acme.sh tool specifically), and bacula Samy Al Bahra presented about Concurrency Kit *** Conference Day 2 (http://www.bsdcan.org/2017/schedule/day_2017-06-10.en.html) Because Dan is a merciful soul, BSDCan starts an hour later on the second day Another great round of talks and BoF sessions over lunch The hallway track was great as always, and I spent most of the afternoon just talking with people Then the final set of talks started, and I was torn between all four of them Then there was the auction, and the closing party *** BSDCan 2017 Auction Swag (https://blather.michaelwlucas.com/archives/2962) Groff Fundraiser Pins: During the conference, You could get a unique Groff pin, by donating more than the last person to either the FreeBSD or OpenBSD foundation Michael W. Lucas and his wife Liz donated some interesting home made and local items to the infamous Charity Auction I donated the last remaining copy of the “Canadian Edition” of “FreeBSD Mastery: Advanced ZedFS”, and a Pentium G4400 (Skylake) CPU (Supports ECC or non-ECC) Peter Hessler donated his pen (Have you read “Git Commit Murder” yet?) Theo De Raadt donated his autographed conference badge David Maxwell donated a large print of the group photo from last years FreeBSD Developers Summit, which was purchased by Allan There was also a FreeBSD Dev Summit T-Shirt (with the Slogan: What is Core doing about it?) autographed by all of the attending members of core, with a forged jhb@ signature. Lastly, someone wrote “I

A pledge of love to OpenBSD, combating ransomware like WannaCry with OpenZFS, and using PFsense to maximize your non-gigabit Internet connection This episode was brought to you by Headlines ino64 project committed to FreeBSD 12-CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=318736) The ino64 project has been completed and merged into FreeBSD 12-CURRENT Extend the inot, devt, nlinkt types to 64-bit ints. Modify struct dirent layout to add doff, increase the size of dfileno to 64-bits, increase the size of dnamlen to 16-bits, and change the required alignment. Increase struct statfs fmntfromname[] and fmntonname[] array length MNAMELEN to 1024 This means the length of a mount point (MNAMELEN) has been increased from 88 byte to 1024 bytes. This allows longer ZFS dataset names and more nesting, and generally improves the usefulness of nested jails It also allow more than 4 billion files to be stored in a single file system (both UFS and ZFS). It also deals with a number of NFS problems, such as Amazon's EFS (cloud NFS), which uses 64 bit IDs even with small numbers of files. ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and by employing other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat around are broken in backward and forward incompatible ways. A bug in poudriere that may cause some packages to not rebuild is being fixed. Many packages like perl will need to be rebuilt after this change Update note: strictly follow the instructions in UPDATING. Build and install the new kernel with COMPAT_FREEBSD11 option enabled, then reboot, and only then install new world. So you need the new GENERIC kernel with the COMPAT_FREEBSD11 option, so that your old userland will work with the new kernel, and you need to build, install, and reboot onto the new kernel before attempting to install world. The usual process of installing both and then rebooting will NOT WORK Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kmoore) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib). Sponsored by: The FreeBSD Foundation (emaste, kib) Why I love OpenBSD (https://medium.com/@h3artbl33d/why-i-love-openbsd-ca760cf53941) Jeroen Janssen writes: I do love open source software. Oh boy, I really do love open source software. It's extendable, auditable, and customizable. What's not to love? I'm astonished by the idea that tens, hundreds, and sometimes even thousands of enthusiastic, passionate developers collaborate on an idea. Together, they make the world a better place, bit by bit. And this leads me to one of my favorite open source projects: the 22-year-old OpenBSD operating system. The origins of my love affair with OpenBSD From Linux to *BSD The advantages of OpenBSD It's extremely secure It's well documented It's open source > It's neat and clean My take on OpenBSD ** DO ** Combating WannaCry and Other Ransomware with OpenZFS Snapshots (https://www.ixsystems.com/blog/combating-ransomware/) Ransomware attacks that hold your data hostage using unauthorized data encryption are spreading rapidly and are particularly nefarious because they do not require any special access privileges to your data. A ransomware attack may be launched via a sophisticated software exploit as was the case with the recent “WannaCry” ransomware, but there is nothing stopping you from downloading and executing a malicious program that encrypts every file you have access to. If you fail to pay the ransom, the result will be indistinguishable from your simply deleting every file on your system. To make matters worse, ransomware authors are expanding their attacks to include just about any storage you have access to. The list is long, but includes network shares, Cloud services like DropBox, and even “shadow copies” of data that allow you to open previous versions of files. To make matters even worse, there is little that your operating system can do to prevent you or a program you run from encrypting files with ransomware just as it can't prevent you from deleting the files you own. Frequent backups are touted as one of the few effective strategies for recovering from ransomware attacks but it is critical that any backup be isolated from the attack to be immune from the same attack. Simply copying your files to a mounted disk on your computer or in the Cloud makes the backup vulnerable to infection by virtue of the fact that you are backing up using your regular permissions. If you can write to it, the ransomware can encrypt it. Like medical workers wearing hazmat suits for isolation when combating an epidemic, you need to isolate your backups from ransomware. OpenZFS snapshots to the rescue OpenZFS is the powerful file system at the heart of every storage system that iXsystems sells and of its many features, snapshots can provide fast and effective recovery from ransomware attacks at both the individual user and enterprise level as I talked about in 2015. As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks. OpenZFS snapshots in practice While OpenZFS is available on a number of desktop operating systems such as TrueOS and macOS, the most effective way to bring the benefits of OpenZFS snapshots to the largest number of users is with a network of iXsystems TrueNAS, FreeNAS Certified and FreeNAS Mini unified NAS and SAN storage systems. All of these can provide OpenZFS-backed SMB, NFS, AFP, and iSCSI file and block storage to the smallest workgroups up through the largest enterprises and TrueNAS offers available Fibre Channel for enterprise deployments. By sharing your data to your users using these file and block protocols, you can provide them with a storage infrastructure that can quickly recover from any ransomware attack thrown at it. To mitigate ransomware attacks against individual workstations, TrueNAS and FreeNAS can provide snapshotted storage to your VDI or virtualization solution of choice. Best of all, every iXsystems TrueNAS, FreeNAS Certified, and FreeNAS Mini system includes a consistent user interface and the ability to replicate between one another. This means that any topology of individual offices and campuses can exchange backup data to quickly mitigate ransomware attacks on your organization at all levels. Join us for a free webinar (http://www.onlinemeetingnow.com/register/?id=uegudsbc75) with iXsystems Co-Founder Matt Olander and learn more about why businesses everywhere are replacing their proprietary storage platforms with TrueNAS then email us at info@ixsystems.com or call 1-855-GREP-4-IX (1-855-473-7449), or 1-408-493-4100 (outside the US) to discuss your storage needs with one of our solutions architects. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @twitter (https://twitter.com/mwlauthor) Books, conferences, and how these two combine + BR: Welcome back. Tell us what you've been up to since the last time we interviewed you regarding books and such. + AJ: Tell us a little bit about relayd and what it can do. + BR: What other books do you have in the pipeline? + AJ: What are your criteria that qualifies a topic for a mastery book? + BR: Can you tell us a little bit about these writing workshops that you attend and what happens there? + AJ: Without spoiling too much: How did you come up with the idea for git commit murder? + BR: Speaking of BSDCan, can you tell the first timers about what to expect in the http://www.bsdcan.org/2017/schedule/events/890.en.html (Newcomers orientation and mentorship) session on Thursday? + AJ: Tell us about the new WIP session at BSDCan. Who had the idea and how much input did you get thus far? + BR: Have you ever thought about branching off into a new genre like children's books or medieval fantasy novels? + AJ: Is there anything else before we let you go? News Roundup Using LLDP on FreeBSD (https://tetragir.com/freebsd/networking/using-lldp-on-freebsd.html) LLDP, or Link Layer Discovery Protocol allows system administrators to easily map the network, eliminating the need to physically run the cables in a rack. LLDP is a protocol used to send and receive information about a neighboring device connected directly to a networking interface. It is similar to Cisco's CDP, Foundry's FDP, Nortel's SONMP, etc. It is a stateless protocol, meaning that an LLDP-enabled device sends advertisements even if the other side cannot do anything with it. In this guide the installation and configuration of the LLDP daemon on FreeBSD as well as on a Cisco switch will be introduced. If you are already familiar with Cisco's CDP, LLDP won't surprise you. It is built for the same purpose: to exchange device information between peers on a network. While CDP is a proprietary solution and can be used only on Cisco devices, LLDP is a standard: IEEE 802.3AB. Therefore it is implemented on many types of devices, such as switches, routers, various desktop operating systems, etc. LLDP helps a great deal in mapping the network topology, without spending hours in cabling cabinets to figure out which device is connected with which switchport. If LLDP is running on both the networking device and the server, it can show which port is connected where. Besides physical interfaces, LLDP can be used to exchange a lot more information, such as IP Address, hostname, etc. In order to use LLDP on FreeBSD, net-mgmt/lldpd has to be installed. It can be installed from ports using portmaster: #portmaster net-mgmt/lldpd Or from packages: #pkg install net-mgmt/lldpd By default lldpd sends and receives all the information it can gather , so it is advisable to limit what we will communicate with the neighboring device. The configuration file for lldpd is basically a list of commands as it is passed to lldpcli. Create a file named lldpd.conf under /usr/local/etc/ The following configuration gives an example of how lldpd can be configured. For a full list of options, see %man lldpcli To check what is configured locally, run #lldpcli show chassis detail To see the neighbors run #lldpcli show neighbors details Check out the rest of the article about enabling LLDP on a Cisco switch experiments with prepledge (http://www.tedunangst.com/flak/post/experiments-with-prepledge) Ted Unangst takes a crack at a system similar to the one being designed for Capsicum, Oblivious Sandboxing (See the presentation at BSDCan), where the application doesn't even know it is in the sandbox MP3 is officially dead, so I figure I should listen to my collection one last time before it vanishes entirely. The provenance of some of these files is a little suspect however, and since I know one shouldn't open files from strangers, I'd like to take some precautions against malicious malarkey. This would be a good use for pledge, perhaps, if we can get it working. At the same time, an occasional feature request for pledge is the ability to specify restrictions before running a program. Given some untrusted program, wrap its execution in a pledge like environment. There are other system call sandbox mechanisms that can do this (systrace was one), but pledge is quite deliberately designed not to support this. But maybe we can bend it to our will. Our pledge wrapper can't be an external program. This leaves us with the option of injecting the wrapper into the target program via LD_PRELOAD. Before main even runs, we'll initialize what needs initializing, then lock things down with a tight pledge set. Our eventual target will be ffplay, but hopefully the design will permit some flexibility and reuse. So the new code is injected to override the open syscall, and reads a list of files from an environment variable. Those files are opened and the path and file descriptor are put into a linked list, and then pledge is used to restrict further access to the file system. The replacement open call now searches just that linked list, returning the already opened file descriptors. So as long as your application only tries to open files that you have preopened, it can function without modification within the sandbox. Or at least that is the goal... ffplay tries to dlopen() some things, and because of the way dlopen() works, it doesn't go via the libc open() wrapper, so it doesn't get overridden ffplay also tries to call a few ioctl's, not allowed After stubbing both of those out, it still doesn't work and it is just getting worse Ted switches to a new strategy, using ffmpeg to convert the .mp3 to a .wav file and then just cat it to /dev/audio A few more stubs for ffmpeg, including access(), and adding tty access to the list of pledges, and it finally works This point has been made from the early days, but I think this exercise reinforces it, that pledge works best with programs where you understand what the program is doing. A generic pledge wrapper isn't of much use because the program is going to do something unexpected and you're going to have a hard time wrangling it into submission. Software is too complex. What in the world is ffplay doing? Even if I were working with the source, how long would it take to rearrange the program into something that could be pledged? One can try using another program, but I would wager that as far as multiformat media players go, ffplay is actually on the lower end of the complexity spectrum. Most of the trouble comes from using SDL as an abstraction layer, which performs a bunch of console operations. On the flip side, all of this early init code is probably the right design. Once SDL finally gets its screen handle setup, we could apply pledge and sandbox the actual media decoder. That would be the right way to things. Is pledge too limiting? Perhaps, but that's what I want. I could have just kept adding permissions until ffplay had full access to my X socket, but what kind of sandbox is that? I don't want naughty MP3s scraping my screen and spying on my keystrokes. The sandbox I created had all the capabilities one needs to convert an MP3 to audible sound, but the tool I wanted to use wasn't designed to work in that environment. And in its defense, these were new post hoc requirements. Other programs, even sed, suffer from less than ideal pledge sets as well. The best summary might be to say that pledge is designed for tomorrow's programs, not yesterday's (and vice versa). There were a few things I could have done better. In particular, I gave up getting audio to work, even though there's a nice description of how to work with pledge in the sio_open manual. Alas, even going back and with a bit more effort I still haven't succeeded. The requirements to use libsndio are more permissive than I might prefer. How I Maximized the Speed of My Non-Gigabit Internet Connection (https://medium.com/speedtest-by-ookla/engineer-maximizes-internet-speed-story-c3ec0e86f37a) We have a new post from Brennen Smith, who is the Lead Systems Engineer at Ookla, the company that runs Speedtest.net, explaining how he used pfSense to maximize his internet connection I spend my time wrangling servers and internet infrastructure. My daily goals range from designing high performance applications supporting millions of users and testing the fastest internet connections in the world, to squeezing microseconds from our stack —so at home, I strive to make sure that my personal internet performance is running as fast as possible. I live in an area with a DOCSIS ISP that does not provide symmetrical gigabit internet — my download and upload speeds are not equal. Instead, I have an asymmetrical plan with 200 Mbps download and 10 Mbps upload — this nuance considerably impacted my network design because asymmetrical service can more easily lead to bufferbloat. We will cover bufferbloat in a later article, but in a nutshell, it's an issue that arises when an upstream network device's buffers are saturated during an upload. This causes immense network congestion, latency to rise above 2,000 ms., and overall poor quality of internet. The solution is to shape the outbound traffic to a speed just under the sending maximum of the upstream device, so that its buffers don't fill up. My ISP is notorious for having bufferbloat issues due to the low upload performance, and it's an issue prevalent even on their provided routers. They walk through a list of router devices you might consider, and what speeds they are capable of handling, but ultimately ended up using a generic low power x86 machine running pfSense 2.3 In my research and testing, I also evaluated IPCop, VyOS, OPNSense, Sophos UTM, RouterOS, OpenWRT x86, and Alpine Linux to serve as the base operating system, but none were as well supported and full featured as PFSense. The main setting to look at is the traffic shaping of uploads, to keep the pipe from getting saturated and having a large buffer build up in the modem and further upstream. This build up is what increases the latency of the connection As with any experiment, any conclusions need to be backed with data. To validate the network was performing smoothly under heavy load, I performed the following experiment: + Ran a ping6 against speedtest.net to measure latency. + Turned off QoS to simulate a “normal router”. + Started multiple simultaneous outbound TCP and UDP streams to saturate my outbound link. + Turned on QoS to the above settings and repeated steps 2 and 3. As you can see from the plot below, without QoS, my connection latency increased by ~1,235%. However with QoS enabled, the connection stayed stable during the upload and I wasn't able to determine a statistically significant delta. That's how I maximized the speed on my non-gigabit internet connection. What have you done with your network? FreeBSD on 11″ MacBook Air (https://www.geeklan.co.uk/?p=2214) Sevan Janiyan writes in his tech blog about his experiences running FreeBSD on an 11'' MacBook Air This tiny machine has been with me for a few years now, It has mostly run OS X though I have tried OpenBSD on it (https://www.geeklan.co.uk/?p=1283). Besides the screen resolution I'm still really happy with it, hardware wise. Software wise, not so much. I use an external disk containing a zpool with my data on it. Among this data are several source trees. CVS on a ZFS filesystem on OS X is painfully slow. I dislike that builds running inside Terminal.app are slow at the expense of a responsive UI. The system seems fragile, at the slightest push the machine will either hang or become unresponsive. Buggy serial drivers which do not implement the break signal and cause instability are frustrating. Last week whilst working on Rump kernel (http://rumpkernel.org/) builds I introduced some new build issues in the process of fixing others, I needed to pick up new changes from CVS by updating my copy of the source tree and run builds to test if issues were still present. I was let down on both counts, it took ages to update source and in the process of cross compiling a NetBSD/evbmips64-el release, the system locked hard. That was it, time to look what was possible elsewhere. While I have been using OS X for many years, I'm not tied to anything exclusive on it, maybe tweetbot, perhaps, but that's it. On the BSDnow podcast they've been covering changes coming in to TrueOS (formerly PC-BSD – a desktop focused distro based on FreeBSD), their experiments seemed interesting, the project now tracks FreeBSD-CURRENT, they've replaced rcng with OpenRC as the init system and it comes with a pre-configured desktop environment, using their own window manager (Lumina). Booting the USB flash image it made it to X11 without any issue. The dock has a widget which states the detected features, no wifi (Broadcom), sound card detected and screen resolution set to 1366×768. I planned to give it a try on the weekend. Friday, I made backups and wiped the system. TrueOS installed without issue, after a short while I had a working desktop, resuming from sleep worked out of the box. I didn't spend long testing TrueOS, switching out NetBSD-HEAD only to realise that I really need ZFS so while I was testing things out, might as well give stock FreeBSD 11-STABLE a try (TrueOS was based on -CURRENT). Turns out sleep doesn't work yet but sound does work out of the box and with a few invocations of pkg(8) I had xorg, dwm, firefox, CVS and virtuabox-ose installed from binary packages. VirtualBox seems to cause the system to panic (bug 219276) but I should be able to survive without my virtual machines over the next few days as I settle in. I'm considering ditching VirtualBox and converting the vdi files to raw images so that they can be written to a new zvol for use with bhyve. As my default keyboard layout is Dvorak, OS X set the EFI settings to this layout. The first time I installed FreeBSD 11-STABLE, I opted for full disk encryption but ran into this odd issue where on boot the keyboard layout was Dvorak and password was accepted, the system would boot and as it went to mount the various filesystems it would switch back to QWERTY. I tried entering my password with both layout but wasn't able to progress any further, no bug report yet as I haven't ruled myself out as the problem. Thunderbolt gigabit adapter –bge(4) (https://www.freebsd.org/cgi/man.cgi?query=bge) and DVI adapter both worked on FreeBSD though the gigabit adapter needs to be plugged in at boot to be detected. The trackpad bind to wsp(4) (https://www.freebsd.org/cgi/man.cgi?query=wsp), left, right and middle clicks are available through single, double and tripple finger tap. Sound card binds to snd_hda(4) (https://www.freebsd.org/cgi/man.cgi?query=snd_hda) and works out of the box. For wifi I'm using a urtw(4) (https://www.freebsd.org/cgi/man.cgi?query=urtw) Alfa adapter which is a bit on the large side but works very reliably. A copy of the dmesg (https://www.geeklan.co.uk/files/macbookair/freebsd-dmesg.txt) is here. Beastie Bits OPNsense - call-for-testing for SafeStack (https://forum.opnsense.org/index.php?topic=5200.0) BSD 4.4: cat (https://www.rewritinghistorycasts.com/screencasts/bsd-4.4:-cat) Continuous Unix commit history from 1970 until today (https://github.com/dspinellis/unix-history-repo) Update on Unix Architecture Evolution Diagrams (https://www.spinellis.gr/blog/20170510/) “Relayd and Httpd Mastery” is out! (https://blather.michaelwlucas.com/archives/2951) Triangle BSD User Group Meeting -- libxo (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) *** Feedback/Questions Carlos - ASUS Tinkerboard (http://dpaste.com/1GJHPNY#wrap) James - Firewall question (http://dpaste.com/0QCW933#wrap) Adam - ZFS books (http://dpaste.com/0GMG5M2#wrap) David - Managing zvols (http://dpaste.com/2GP8H1E#wrap) ***

This week on BSDNow, reports from AsiaBSDcon, TrueOS and FreeBSD news, Optimizing IllumOS Kernel, your questions and more. This episode was brought to you by Headlines AsiaBSDcon Reports and Reviews () AsiaBSDcon schedule (https://2017.asiabsdcon.org/program.html.en) Schedule and slides from the 4th bhyvecon (http://bhyvecon.org/) Michael Dexter's trip report on the iXsystems blog (https://www.ixsystems.com/blog/ixsystems-attends-asiabsdcon-2017) NetBSD AsiaBSDcon booth report (http://mail-index.netbsd.org/netbsd-advocacy/2017/03/13/msg000729.html) *** TrueOS Community Guidelines are here! (https://www.trueos.org/blog/trueos-community-guidelines/) TrueOS has published its new Community Guidelines The TrueOS Project has existed for over ten years. Until now, there was no formally defined process for interested individuals in the TrueOS community to earn contributor status as an active committer to this long-standing project. The current core TrueOS developers (Kris Moore, Ken Moore, and Joe Maloney) want to provide the community more opportunities to directly impact the TrueOS Project, and wish to formalize the process for interested people to gain full commit access to the TrueOS repositories. These describe what is expected of community members and committers They also describe the process of getting commit access to the TrueOS repo: Previously, Kris directly handed out commit bits. Now, the Core developers have provided a small list of requirements for gaining a TrueOS commit bit: Create five or more pull requests in a TrueOS Project repository within a single six month period. Stay active in the TrueOS community through at least one of the available community channels (Gitter, Discourse, IRC, etc.). Request commit access from the core developers via core@trueos.org OR Core developers contact you concerning commit access. Pull requests can be any contribution to the project, from minor documentation tweaks to creating full utilities. At the end of every month, the core developers review the commit logs, removing elements that break the Project or deviate too far from its intended purpose. Additionally, outstanding pull requests with no active dissension are immediately merged, if possible. For example, a user submits a pull request which adds a little-used OpenRC script. No one from the community comments on the request or otherwise argues against its inclusion, resulting in an automatic merge at the end of the month. In this manner, solid contributions are routinely added to the project and never left in a state of “limbo”. The page also describes the perks of being a TrueOS committer: Contributors to the TrueOS Project enjoy a number of benefits, including: A personal TrueOS email alias: @trueos.org Full access for managing TrueOS issues on GitHub. Regular meetings with the core developers and other contributors. Access to private chat channels with the core developers. Recognition as part of an online Who's Who of TrueOS developers. The eternal gratitude of the core developers of TrueOS. A warm, fuzzy feeling. Intel Donates 250.000 $ to the FreeBSD Foundation (https://www.freebsdfoundation.org/news-and-events/latest-news/new-uranium-level-donation-and-collaborative-partnership-with-intel/) More details about the deal: Systems Thinking: Intel and the FreeBSD Project (https://www.freebsdfoundation.org/blog/systems-thinking-intel-and-the-freebsd-project/) Intel will be more actively engaging with the FreeBSD Foundation and the FreeBSD Project to deliver more timely support for Intel products and technologies in FreeBSD. Intel has contributed code to FreeBSD for individual device drivers (i.e. NICs) in the past, but is now seeking a more holistic “systems thinking” approach. Intel Blog Post (https://01.org/blogs/imad/2017/intel-increases-support-freebsd-project) We will work closely with the FreeBSD Foundation to ensure the drivers, tools, and applications needed on Intel® SSD-based storage appliances are available to the community. This collaboration will also provide timely support for future Intel® 3D XPoint™ products. Thank you very much, Intel! *** Applied FreeBSD: Basic iSCSI (https://globalengineer.wordpress.com/2017/03/05/applied-freebsd-basic-iscsi/) iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered. The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it. Then, enable ctld and start it: sysrc ctld_enable=”YES” service ctld start You can use the ctladm command to see what is going on: root@bsdtarget:/dev # ctladm lunlist (7:0:0/0): Fixed Direct Access SPC-4 SCSI device (7:0:1/1): Fixed Direct Access SPC-4 SCSI device root@bsdtarget:/dev # ctladm devlist LUN Backend Size (Blocks) BS Serial Number Device ID 0 block 10485760 512 MYSERIAL 0 MYDEVID 0 1 block 10485760 512 MYSERIAL 1 MYDEVID 1 Now, let's configure the client side: In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. sysrc iscsid_enable=”YES” service iscsid start Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session: + iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget You should now have a new device (check dmesg), in this case, da1 The guide them walks through partitioning the disk, and laying down a UFS file system, and mounting it This it walks through how to disconnect iscsi, incase you don't want it anymore This all looked nice and easy, and it works very well. Now lets see what happens when you try to mount the iSCSI from Windows Ok, that wasn't so bad. Now, instead of sharing an entire space disk on the host via iSCSI, share a zvol. Now your windows machine can be backed by ZFS. All of your problems are solved. Interview - Philipp Buehler - pbuehler@sysfive.com (mailto:pbuehler@sysfive.com) Technical Lead at SysFive, and Former OpenBSD Committer News Roundup Half a dozen new features in mandoc -T html (http://undeadly.org/cgi?action=article&sid=20170316080827) mandoc (http://man.openbsd.org/mandoc.1)'s HTML output mode got some new features Even though mdoc(7) is a semantic markup language, traditionally none of the semantic annotations were communicated to the reader. [...] Now, at least in -T html output mode, you can see the semantic function of marked-up words by hovering your mouse over them. In terminal output modes, we have the ctags(1)-like internal search facility built around the less(1) tag jump (:t) feature for quite some time now. We now have a similar feature in -T html output mode. To jump to (almost) the same places in the text, go to the address bar of the browser, type a hash mark ('#') after the URI, then the name of the option, command, variable, error code etc. you want to jump to, and hit enter. Check out the full report by Ingo Schwarze (schwarze@) and try out these new features *** Optimizing IllumOS Kernel Crypto (http://zfs-create.blogspot.com/2014/05/optimizing-illumos-kernel-crypto.html) Sašo Kiselkov, of ZFS fame, looked into the performance of the OpenSolaris kernel crypto framework and found it lacking. The article also spends a few minutes on the different modes and how they work. Recently I've had some motivation to look into the KCF on Illumos and discovered that, unbeknownst to me, we already had an AES-NI implementation that was automatically enabled when running on Intel and AMD CPUs with AES-NI support. This work was done back in 2010 by Dan Anderson.This was great news, so I set out to test the performance in Illumos in a VM on my Mac with a Core i5 3210M (2.5GHz normal, 3.1GHz turbo). The initial tests of “what the hardware can do” were done in OpenSSL So now comes the test for the KCF. I wrote a quick'n'dirty crypto test module that just performed a bunch of encryption operations and timed the results. KCF got around 100 MB/s for each algorithm, except half that for AES-GCM OpenSSL had done over 3000 MB/s for CTR mode, 500 MB/s for CBC, and 1000 MB/s for GCM What the hell is that?! This is just plain unacceptable. Obviously we must have hit some nasty performance snag somewhere, because this is comical. And sure enough, we did. When looking around in the AES-NI implementation I came across this bit in aes_intel.s that performed the CLTS instruction. This is a problem: 3.1.2 Instructions That Cause VM Exits ConditionallyCLTS. The CLTS instruction causes a VM exit if the bits in position 3 (corresponding to CR0.TS) are set in both the CR0 guest/host mask and the CR0 read shadow. The CLTS instruction signals to the CPU that we're about to use FPU registers (which is needed for AES-NI), which in VMware causes an exit into the hypervisor. And we've been doing it for every single AES block! Needless to say, performing the equivalent of a very expensive context switch every 16 bytes is going to hurt encryption performance a bit. The reason why the kernel is issuing CLTS is because for performance reasons, the kernel doesn't save and restore FPU register state on kernel thread context switches. So whenever we need to use FPU registers inside the kernel, we must disable kernel thread preemption via a call to kpreemptdisable() and kpreemptenable() and save and restore FPU register state manually. During this time, we cannot be descheduled (because if we were, some other thread might clobber our FPU registers), so if a thread does this for too long, it can lead to unexpected latency bubbles The solution was to restructure the AES and KCF block crypto implementations in such a way that we execute encryption in meaningfully small chunks. I opted for 32k bytes, for reasons which I'll explain below. Unfortunately, doing this restructuring work was a bit more complicated than one would imagine, since in the KCF the implementation of the AES encryption algorithm and the block cipher modes is separated into two separate modules that interact through an internal API, which wasn't really conducive to high performance (we'll get to that later). Anyway, having fixed the issue here and running the code at near native speed, this is what I get: AES-128/CTR: 439 MB/s AES-128/CBC: 483 MB/s AES-128/GCM: 252 MB/s Not disastrous anymore, but still, very, very bad. Of course, you've got keep in mind, the thing we're comparing it to, OpenSSL, is no slouch. It's got hand-written highly optimized inline assembly implementations of most of these encryption functions and their specific modes, for lots of platforms. That's a ton of code to maintain and optimize, but I'll be damned if I let this kind of performance gap persist. Fixing this, however, is not so trivial anymore. It pertains to how the KCF's block cipher mode API interacts with the cipher algorithms. It is beautifully designed and implemented in a fashion that creates minimum code duplication, but this also means that it's inherently inefficient. ECB, CBC and CTR gained the ability to pass an algorithm-specific "fastpath" implementation of the block cipher mode, because these functions benefit greatly from pipelining multiple cipher calls into a single place. ECB, CTR and CBC decryption benefit enormously from being able to exploit the wide XMM register file on Intel to perform encryption/decryption operations on 8 blocks at the same time in a non-interlocking manner. The performance gains here are on the order of 5-8x.CBC encryption benefits from not having to copy the previously encrypted ciphertext blocks into memory and back into registers to XOR them with the subsequent plaintext blocks, though here the gains are more modest, around 1.3-1.5x. After all of this work, this is how the results now look on Illumos, even inside of a VM: Algorithm/Mode 128k ops AES-128/CTR: 3121 MB/s AES-128/CBC: 691 MB/s AES-128/GCM: 1053 MB/s So the CTR and GCM speeds have actually caught up to OpenSSL, and CBC is actually faster than OpenSSL. On the decryption side of things, CBC decryption also jumped from 627 MB/s to 3011 MB/s. Seeing these performance numbers, you can see why I chose 32k for the operation size in between kernel preemption barriers. Even on the slowest hardware with AES-NI, we can expect at least 300-400 MB/s/core of throughput, so even in the worst case, we'll be hogging the CPU for at most ~0.1ms per run. Overall, we're even a little bit faster than OpenSSL in some tests, though that's probably down to us encrypting 128k blocks vs 8k in the "openssl speed" utility. Anyway, having fixed this monstrous atrocity of a performance bug, I can now finally get some sleep. To made these tests repeatable, and to ensure that the changes didn't break the crypto algorithms, Saso created a crypto_test kernel module. I have recently created a FreeBSD version of crypto_test.ko, for much the same purposes Initial performance on FreeBSD is not as bad, if you have the aesni.ko module loaded, but it is not up to speed with OpenSSL. You cannot directly compare to the benchmarks Saso did, because the CPUs are vastly different. Performance results (https://wiki.freebsd.org/OpenCryptoPerformance) I hope to do some more tests on a range of different sized CPUs in order to determine how the algorithms scale across different clock speeds. I also want to look at, or get help and have someone else look at, implementing some of the same optimizations that Saso did. It currently seems like there isn't a way to perform addition crypto operations in the same session without regenerating the key table. Processing additional buffers in an existing session might offer a number of optimizations for bulk operations, although in many cases, each block is encrypted with a different key and/or IV, so it might not be very useful. *** Brendan Gregg's special freeware tools for sysadmins (http://www.brendangregg.com/specials.html) These tools need to be in every (not so) serious sysadmins toolbox. Triple ROT13 encryption algorithm (beware: export restrictions may apply) /usr/bin/maybe, in case true and false don't provide too little choice... The bottom command lists you all the processes using the least CPU cycles. Check out the rest of the tools. You wrote similar tools and want us to cover them in the show? Send us an email to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** A look at 2038 (http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/) I remember the Y2K problem quite vividly. The world was going crazy for years, paying insane amounts of money to experts to fix critical legacy systems, and there was a neverending stream of predictions from the media on how it's all going to fail. Most didn't even understand what the problem was, and I remember one magazine writing something like the following: Most systems store the current year as a two-digit value to save space. When the value rolls over on New Year's Eve 1999, those two digits will be “00”, and “00” means “halt operation” in the machine language of many central processing units. If you're in an elevator at this time, it will stop working and you may fall to your death. I still don't know why they thought a computer would suddenly interpret data as code, but people believed them. We could see a nearby hydropower plant from my parents' house, and we expected it to go up in flames as soon as the clock passed midnight, while at least two airplanes crashed in our garden at the same time. Then nothing happened. I think one of the most “severe” problems was the police not being able to open their car garages the next day because their RFID tokens had both a start and end date for validity, and the system clock had actually rolled over to 1900, so the tokens were “not yet valid”. That was 17 years ago. One of the reasons why Y2K wasn't as bad as it could have been is that many systems had never used the “two-digit-year” representation internally, but use some form of “timestamp” relative to a fixed date (the “epoch”). The actual problem with time and dates rolling over is that systems calculate timestamp differences all day. Since a timestamp derived from the system clock seemingly only increases with each query, it is very common to just calculate diff = now - before and never care about the fact that now could suddenly be lower than before because the system clock has rolled over. In this case diff is suddenly negative, and if other parts of the code make further use of the suddenly negative value, things can go horribly wrong. A good example was a bug in the generator control units (GCUs) aboard Boeing 787 “Dreamliner” aircrafts, discovered in 2015. An internal timestamp counter would overflow roughly 248 days after the system had been powered on, triggering a shut down to “safe mode”. The aircraft has four generator units, but if all were powered up at the same time, they would all fail at the same time. This sounds like an overflow caused by a signed 32-bit counter counting the number of centiseconds since boot, overflowing after 248.55 days, and luckily no airline had been using their Boing 787 models for such a long time between maintenance intervals. The “obvious” solution is to simply switch to 64-Bit values and call it day, which would push overflow dates far into the future (as long as you don't do it like the IBM S/370 mentioned before). But as we've learned from the Y2K problem, you have to assume that computer systems, computer software and stored data (which often contains timestamps in some form) will stay with us for much longer than we might think. The years 2036 and 2038 might be far in the future, but we have to assume that many of the things we make and sell today are going to be used and supported for more than just 19 years. Also many systems have to store dates which are far in the future. A 30 year mortgage taken out in 2008 could have already triggered the bug, and for some banks it supposedly did. sysgettimeofday() is one of the most used system calls on a generic Linux system and returns the current time in form of an UNIX timestamp (timet data type) plus fraction (susecondst data type). Many applications have to know the current time and date to do things, e.g. displaying it, using it in game timing loops, invalidating caches after their lifetime ends, perform an action after a specific moment has passed, etc. In a 32-Bit UNIX system, timet is usually defined as a signed 32-Bit Integer. When kernel, libraries and applications are compiled, the compiler will turn this assumption machine code and all components later have to match each other. So a 32-Bit Linux application or library still expects the kernel to return a 32-Bit value even if the kernel is running on a 64-Bit architecture and has 32-Bit compatibility. The same holds true for applications calling into libraries. This is a major problem, because there will be a lot of legacy software running in 2038. Systems which used an unsigned 32-Bit Integer for timet push the problem back to 2106, but I don't know about many of those. The developers of the GNU C library (glibc), the default standard C library for many GNU/Linux systems, have come up with a design for year 2038 proofness for their library. Besides the timet data type itself, a number of other data structures have fields based on timet or the combined struct timespec and struct timeval types. Many methods beside those intended for setting and querying the current time use timestamps 32-Bit Windows applications, or Windows applications defining _USE32BITTIMET, can be hit by the year 2038 problem too if they use the timet data type. The _time64t data type had been available since Visual C 7.1, but only Visual C 8 (default with Visual Studio 2015) expanded timet to 64 bits by default. The change will only be effective after a recompilation, legacy applications will continue to be affected. If you live in a 64-Bit world and use a 64-Bit kernel with 64-Bit only applications, you might think you can just ignore the problem. In such a constellation all instances of the standard time_t data type for system calls, libraries and applications are signed 64-Bit Integers which will overflow in around 292 billion years. But many data formats, file systems and network protocols still specify 32-Bit time fields, and you might have to read/write this data or talk to legacy systems after 2038. So solving the problem on your side alone is not enough. Then the article goes on to describe how all of this will break your file systems. Not to mention your databases and other file formats. Also see Theo De Raadt's EuroBSDCon 2013 Presentation (https://www.openbsd.org/papers/eurobsdcon_2013_time_t/mgp00001.html) *** Beastie Bits Michael Lucas: Get your name in “Absolute FreeBSD 3rd Edition” (https://blather.michaelwlucas.com/archives/2895) ZFS compressed ARC stats to top (https://svnweb.freebsd.org/base?view=revision&revision=r315435) Matthew Dillon discovered HAMMER was repeating itself when writing to disk. Fixing that issue doubled write speeds (https://www.dragonflydigest.com/2017/03/14/19452.html) TedU on Meaningful Short Names (http://www.tedunangst.com/flak/post/shrt-nms-fr-clrty) vBSDcon and EuroBSDcon Call for Papers are open (https://www.freebsdfoundation.org/blog/submit-your-work-vbsdcon-and-eurobsdcon-cfps-now-open/) Feedback/Questions Craig asks about BSD server management (http://pastebin.com/NMshpZ7n) Michael asks about jails as a router between networks (http://pastebin.com/UqRwMcRk) Todd asks about connecting jails (http://pastebin.com/i1ZD6eXN) Dave writes in with an interesting link (http://pastebin.com/QzW5c9wV) > applications crash more often due to errors than corruptions. In the case of corruption, a few applications (e.g., Log-Cabin, ZooKeeper) can use checksums and redundancy to recover, leading to a correct behavior; however, when the corruption is transformed into an error, these applications crash, resulting in reduced availability. ***

This is a very special BSD Now! New exciting changes are coming to the show and we're gonna cover them, so stick around or you'll miss it! Interview – Kris Moore – kris@trueos.org / @pcbsdKrisTrueOS founder, FreeNAS developer, BSD Now co-hostBenedict Reuschling – bcr@freebsd.org / @bsdbcrFreeBSD commiter & FreeBSD Foundation Vice President, BSD Now co-host Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***

This week on BSDNow, Allan and I are in Tokyo for AsiaBSDCon, but not to worry, we have a full episode lined up and ready to go. Hackathon reports This episode was brought to you by Headlines OpenBSD A2k17 hackathon reports a2k17 hackathon report: Patrick Wildt on the arm64 port (http://undeadly.org/cgi?action=article&sid=20170131101827) a2k17 hackathon report: Antoine Jacoutot on syspatch, rc.d improvements and more (http://undeadly.org/cgi?action=article&sid=20170203232049) a2k17 hackathon report: Martin Pieuchot on NET_LOCK and much more (http://undeadly.org/cgi?action=article&sid=20170127154356) a2k17 hackathon report: Kenneth Westerback on the hidden wonders of the build system, the network stack and more (http://undeadly.org/cgi?action=article&sid=20170127031836) a2k17 hackathon report: Bob Beck on LibreSSL progress and more (http://undeadly.org/cgi?action=article&sid=20170125225403) *** NetBSD is now reproducible (https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds) Christos Zoulas posts to the NetBSD blog that he has completed his project to make fully reproducible NetBSD builds for amd64 and sparc64 I have been working on and off for almost a year trying to get reproducible builds (the same source tree always builds an identical cdrom) on NetBSD. I did not think at the time it would take as long or be so difficult, so I did not keep a log of all the changes I needed to make. I was also not the only one working on this. Other NetBSD developers have been making improvements for the past 6 years. I would like to acknowledge the NetBSD build system (aka build.sh) which is a fully portable cross-build system. This build system has given us a head-start in the reproducible builds work. I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file. Finally other developers who have started, motivated and did a lot of work getting us here like Joerg Sonnenberger and Thomas Klausner for their work on reproducible builds, and Todd Vierling and Luke Mewburn for their work on build.sh. Some of the stumbling blocks that were overcome: Timestamps Date/time/author embedded in source files Timezone sensitive code Directory order / build order Non-sanitized data stored in files Symbolic links / paths General tool inconsistencies: including gcc profiling, the fact that GPT partition tables, are by definition, globally unique each time they are created, and the iso9660 standard calls for a timestamp with a timezone. Toolchain Build information / tunables / environment. NetBSD now has a knob ‘MKREPRO', if set to YES it sets a long list of variables to a consistent set of a values. The post walks through how these problems where solves Future Work: Vary more parameters and find more inconsistencies Verify that cross-building is reproducible Verify that unprivileged builds are reproducible Test on other platforms *** Features are faults redux (http://www.tedunangst.com/flak/post/features-are-faults-redux) From Ted Unangst Last week I gave a talk for the security class at Notre Dame based on features are faults but with some various commentary added. It was an exciting trip, with the opportunity to meet and talk with the computer vision group as well. Some other highlights include the Indiana skillet I had for breakfast, which came with pickles and was amazing, and explaining the many wonders of cvs to the Linux users group over lunch. After that came the talk, which went a little something like this. I got started with OpenBSD back about the same time I started college, although I had a slightly different perspective then. I was using OpenBSD because it included so many security features, therefore it must be the most secure system, right? For example, at some point I acquired a second computer. What's the first thing anybody does when they get a second computer? That's right, set up a kerberos domain. The idea that more is better was everywhere. This was also around the time that ipsec was getting its final touches, and everybody knew ipsec was going to be the most secure protocol ever because it had more options than any other secure transport. We'll revisit this in a bit. There's been a partial attitude adjustment since then, with more people recognizing that layering complexity doesn't result in more security. It's not an additive process. There's a whole talk there, about the perfect security that people can't or won't use. OpenBSD has definitely switched directions, including less code, not more. All the kerberos code was deleted a few years ago. Let's assume about one bug per 100 lines of code. That's probably on the low end. Now say your operating system has 100 million lines of code. If I've done the math correctly, that's literally a million bugs. So that's one reason to avoid adding features. But that's a solveable problem. If we pick the right language and the right compiler and the right tooling and with enough eyeballs and effort, we can fix all the bugs. We know how to build mostly correct software, we just don't care. As we add features to software, increasing its complexity, new unexpected behaviors start to emerge. What are the bounds? How many features can you add before craziness is inevitable? We can make some guesses. Less than a thousand for sure. Probably less than a hundred? Ten maybe? I'll argue the answer is quite possibly two. Interesting corollary is that it's impossible to have a program with exactly two features. Any program with two features has at least a third, but you don't know what it is My first example is a bug in the NetBSD ftp client. We had one feature, we added a second feature, and just like that we got a third misfeature (http://marc.info/?l=oss-security&m=141451507810253&w=2) Our story begins long ago. The origins of this bug are probably older than I am. In the dark times before the web, FTP sites used to be a pretty popular way of publishing files. You run an ftp client, connect to a remote site, and then you can browse the remote server somewhat like a local filesystem. List files, change directories, get files. Typically there would be a README file telling you what's what, but you don't need to download a copy to keep. Instead we can pipe the output to a program like more. Right there in the ftp client. No need to disconnect. Fast forward a few decades, and http is the new protocol of choice. http is a much less interactive protocol, but the ftp client has some handy features for batch downloads like progress bars, etc. So let's add http support to ftp. This works pretty well. Lots of code reused. http has one quirk however that ftp doesn't have. Redirects. The server can redirect the client to a different file. So now you're thinking, what happens if I download http://somefile and the server sends back 302 http://|reboot. ftp reconnects to the server, gets the 200, starts downloading and saves it to a file called |reboot. Except it doesn't. The function that saves files looks at the first character of the name and if it's a pipe, runs that command instead. And now you just rebooted your computer. Or worse. It's pretty obvious this is not the desired behavior, but where exactly did things go wrong? Arguably, all the pieces were working according to spec. In order to see this bug coming, you needed to know how the save function worked, you needed to know about redirects, and you needed to put all the implications together. The post then goes into a lot more detail about other issues. We just don't have time to cover it all today, but you should go read it, it is very enlightening What do we do about this? That's a tough question. It's much easier to poke fun at all the people who got things wrong. But we can try. My attitudes are shaped by experiences with the OpenBSD project, and I think we are doing a decent job of containing the complexity. Keep paring away at dependencies and reducing interactions. As a developer, saying “no” to all feature requests is actually very productive. It's so much faster than implementing the feature. Sometimes users complain, but I've often received later feedback from users that they'd come to appreciate the simplicity. There was a question about which of these vulnerabilities were found by researchers, as opposed to troublemakers. The answer was most, if not all of them, but it made me realize one additional point I hadn't mentioned. Unlike the prototypical buffer overflow vulnerability, exploiting features is very reliable. Exploiting something like shellshock or imagetragick requires no customized assembly and is independent of CPU, OS, version, stack alignment, malloc implementation, etc. Within about 24 hours of the initial release of shellshock, I had logs of people trying to exploit it. So unless you're on about a 12 hour patch cycle, you're going to have a bad time. reimplement zfsctl (.zfs) support (https://svnweb.freebsd.org/changeset/base/314048) avg@ (Andriy Gapon) has rewritten the .zfs support in FreeBSD The current code is written on top of GFS, a library with the generic support for writing filesystems, which was ported from Illumos. Because of significant differences between illumos VFS and FreeBSD VFS models, both the GFS and zfsctl code were heavily modified to work on FreeBSD. Nonetheless, they still contain quite a few ugly hacks and bugs. This is a reimplementation of the zfsctl code where the VFS-specific bits are written from scratch and only the code that interacts with the rest of ZFS is reused. Some ideas are picked from an independent work by Will (wca@) This work improves the overall quality of the ZFS port to FreeBSD The code that provides support for ZFS .zfs/ directory functionality has been reimplemented. It is no longer possible to create a snapshot by mkdir under .zfs/snapshot/. That should be the only user visible change. TIL: On IllumOS, you can create, rename, and destroy snapshots, by manipulating the virtual directories in the .zfs/snapshots directory. If enough people would find this feature useful, maybe it could be implemented (rm and rename have never existed on FreeBSD). At the same time, it seems like rather a lot of work, when the ZFS command line tools work so well. Although wca@ pointed out on IRC, it can be useful to be able to create a snapshot over NFS, or SMB. Interview - Konrad Witaszczyk - def@freebsd.org (mailto:def@freebsd.org) Encrypted Kernel Crash Dumps *** News Roundup PBKDF2 Performance improvements on FreeBSD (https://svnweb.freebsd.org/changeset/base/313962) Joe Pixton did some research (https://jbp.io/2015/08/11/pbkdf2-performance-matters/) and found that, because of the way the spec is written, most PBKDF2 implementations are 2x slower than they need to be. Since the PBKDF is used to derive a key, used for encryption, this poses a problem. The attacker can derive a key twice as fast as you can. On FreeBSD the PBKDF2 was configured to derive a SHA512-HMAC key that would take approximately 2 seconds to calculate. That is 2 seconds on one core. So an attacker can calculate the same key in 1 second, and use many cores. Luckily, 1 second is still a long time for each brute force guess. On modern CPUs with the fast algorithm, you can do about 500,000 iterations of PBKDF per second (per core). Until a recent change, OpenBSD used only 8192 iterations. It now uses a similar benchmark of ~2 seconds, and uses bcrypt instead of a SHA1-HMAC. Joe's research showed that the majority of implementations were done the ‘slow' way. Calculating the initial part of the outer round each iteration, instead of reusing the initial calculation over and over for each round. Joe submitted a match to FreeBSD to solve this problem. That patch was improved, and a test of tests were added by jmg@, but then work stalled I picked up the work, and fixed some merge conflicts in the patch that had cropped up based on work I had done that moved the HMAC code to a separate file. This work is now committed. With this change, all newly generated GELI keys will be approximately 2x as strong. Previously generated keys will take half as long to calculate, resulting in faster mounting of encrypted volumes. Users may choose to rekey, to generate a new key with the larger default number of iterations using the geli(8) setkey command. Security of existing data is not compromised, as ~1 second per brute force attempt is still a very high threshold. If you are interested in the topic, I recommend the video of Joe's presentation from the Passwords15 conference in Las Vegas *** Quick How-To: Updating a screenshot in the TrueOS Handbook (https://www.trueos.org/blog/quick-updating-screenshot-trueos-handbook/) Docs writers, might be time to pay attention. This week we have a good walk-through of adding / updating new screenshots to the TrueOS Sphinx Documentation. For those who have not looked in the past, TrueOS and FreeNAS both have fantastic docs by the team over at iXsystems using Sphinx as their doc engine. Often we get questions from users asking what “they can do to help” but don't necessarily have programming skills to apply. The good news is that using Sphinx is relatively easy, and after learning some minio rst syntax you can easily help fix, or even contribute to new sections of the TrueOS (Or FreeNAS) documentation. In this example, Tim takes us through the process of replacing an old out of date screenshot in the handbook with the latest hotness. Starting with a .png file, he then locates the old screenshot name and adds the updated version “lumina-e.png” to “lumina-f.png”. With the file added to the tree, the relevant section of .rst code can be adjusted and the sphinx build run to verify the output HTML looks correct. Using this method you can easily start to get involved with other aspects of documentation and next thing you know you'll be writing boot-loaders like Allan! *** Learn C Programming With 9 Excellent Open Source Books (https://www.ossblog.org/learn-c-programming-with-9-excellent-open-source-books/) Now that you've easily mastered all your documentation skills, you may be ready to take on a new challenge. (Come on, that boot-loader isn't going to write itself!) We wanted to point out some excellent resources to get you started on your journey into writing C. Before you think, “oh, more books to purchase”, wait there's good news. These are the top-9 open-source books that you can download in digital form free of charge. Now I bet we got your attention. We start the rundown with “The C Book”, by Mike Banahan, Declan Brady and Mark Doran, which will lay the groundwork with your introduction into the C language and concepts. Next up, if you are going to do anything, do it with style, so take a read through the “C Elements of Style” which will make you popular at all the parties. (We can't vouch for that statement) From here we have a book on using C to build your own minimal “lisp” interpreter, reference guides on GNU C and some other excellent introduction / mastery books to help round-out your programming skill set. Your C adventure awaits, hopefully these books can not only teach you good C, but also make you feel confident when looking at bits of the FreeBSD world or kernel with a proper foundation to back it up. *** Running a Linux VM on OpenBSD (http://eradman.com/posts/linuxvm-on-openbsd.html) Over the past few years we've talked a lot about Virtualization, Bhyve or OpenBSD's ‘vmm', but qemu hasn't gotten much attention. Today we have a blog post with details on how to deploy qemu to run Linux on top of an OpenBSD host system. The starts by showing us how to first provision the storage for qemu, using the handy ‘qemu-img' command, which in this example only creates a 4GB disk, you'll probably want more for real-world usage though. Next up the qemu command will be run, pay attention to the particular flags for network and memory setup. You'll probably want to bump it up past the recommended 256M of memory. Networking is always the fun part, as the author describes his intended setup I want OpenBSD and Debian to be able to obtain an IP via DHCP on their wired interfaces and I don't want external networking required for an NFS share to the VM. To accomplish this I need two interfaces since dhclient will erase any other IPv4 addresses already assigned. We can't assign an address directly to the bridge, but we can configure a virtual Ethernet device and add it. The setup for this portion involves touching a few more files, but isn't that painless. Some “pf” rules to enable NAT for and dhcpd setup to assign a “fixed” IP to the vm will get us going, along with some additional details on how to configure the networking for inside the debian VM. Once those steps are completed you should be able to mount NFS and share data from the host to the VM painlessly. Beastie Bits MacObserver: Interview with Open Source Developer & Former Apple Manager Jordan Hubbard (https://www.macobserver.com/podcasts/background-mode-jordan-hubbard/) 2016 Google Summer of Code Mentor Summit and MeetBSD Trip Report: Gavin Atkinson (https://www.freebsdfoundation.org/blog/2016-google-summer-of-code-mentor-summit-and-meetbsd-trip-report-gavin-atkinson/) Feedback/Questions Joe - BGP / Vultr Followup (http://pastebin.com/TNyHBYwT) Ryan Moreno asks about Laptops (http://pastebin.com/s4Ypezsz) ***

This week on BSDNow, we have “Weird Unix Things”, “Is it getting Steamy in here?” and an Interview about BSD Sockets API. (Those This episode was brought to you by Headlines playonbsd with TrueOS: It's Getting Steamy in Here and I've Had Too Much Wine (https://www.trueos.org/blog/playonbsd-trueos-getting-steamy-ive-much-wine/) We've done a couple of tutorials in the past on using Steam and Wine with PC-BSD, but now with the addition of playonbsd to the AppCafe library, you have more options than ever before to game on your TrueOS system. We're going to have a look today at playonbsd, how it works with TrueOS, and what you can expect if you want to give it a try on your own system. Let's dive right in! Once playonbsd is installed, go back to your blank desktop, right-click on the wallpaper, and select terminal. Playonbsd does almost all the configuring for you, but there are still a couple of simple options you'll want to configure to give yourself the best experience. In your open terminal, type: playonbsd. You can also find playonbsd by doing a fast search using Lumina's built-in search function in the start menu after it's been installed. Once opened, a graphical interface greets us with easy to navigate menus and even does most of the work for you. A nice graphical UI that hides the complexity of setting up WINE and Steam, and lets you pick select the game you want, and get it setup Start gaming quicker, without the headache If you're a PC gamer, you should definitely give playonbsd a try! You may be surprised at how well it works. If you want to know ahead of time if your games are well supported or not, head on over to WineHQ and do a search. Many people have tested and provided feedback and even solutions for potential problems with a large variety of video games. This is a great resource if you run into a glitch or other problem. Weird Unix thing: 'cd //' (https://jvns.ca/blog/2017/02/08/weird-unix-things-cd/) So why can you do ‘cd //tmp', and it isn't the same as ‘cd /tmp'? The spec says: An implementation may further simplify curpath by removing any trailing characters that are not also leading characters, replacing multiple non-leading consecutive characters with a single , and replacing three or more leading characters with a single . If, as a result of this canonicalization, the curpath variable is null, no further steps shall be taken. “So! We can replace “three or more leading / characters with a single slash”. That does not say anything about what to do when there are 2 / characters though, which presumably is why cd //tmp leaves you at //tmp.” A pathname that begins with two successive slashes may be interpreted in an implementation-defined manner So what is it for? Well, the blog did a bit of digging and came up with this stackoverflow answer (http://unix.stackexchange.com/questions/256497/on-what-systems-is-foo-bar-different-from-foo-bar/256569#256569) In cygwin and some other systems // is treated as a unix-ified version of , to access UNC windows file sharing paths like servershare Perforce, the vcs, uses // to denote a path relative to the depot It seems to have been used in the path for a bunch of different network file systems, but also for myriad other things Testing out snapshots in Apple's next-generation APFS file system (https://arstechnica.com/apple/2017/02/testing-out-snapshots-in-apples-next-generation-apfs-file-system/) Adam Leventhal takes his DTrace hammer to Apple's new file system to see what is going on Back in June, Apple announced its new upcoming file system: APFS, or Apple File System. There was no mention of it in the WWDC keynote, but devotees needed no encouragement. They picked over every scintilla of data from the documentation on Apple's developer site, extrapolating, interpolating, eager for whatever was about to come. In the WWDC session hall, the crowd buzzed with a nervous energy, eager for the grand unveiling of APFS. I myself badge-swapped my way into the conference just to get that first glimpse of Apple's first original filesystem in the 30+ years since HFS Apple's presentation didn't disappoint the hungry crowd. We hoped for a modern filesystem, optimized for next generation hardware, rich with features that have become the norm for data centers and professionals. With APFS, Apple showed a path to meeting those expectations. Dominic Giampaolo and Eric Tamura, leaders of the APFS team, shared performance optimizations, data integrity design, volume management, efficient storage of copied data, and snapshots—arguably the feature of APFS most directly in the user's control. It's 2017, and Apple already appears to be making good on its promise with the revelation that the forthcoming iOS 10.3 will use APFS. The number of APFS tinkerers using it for their personal data has instantly gone from a few hundred to a few million. Beta users of iOS 10.3 have already made the switch apparently without incident. They have even ascribed unscientifically-significant performance improvements to APFS. Previously Adam had used DTrace to find a new syscall introduced in OS X, fs_snapshot, but he had not dug into how to use it. Now it seems, the time has come Learning from XNU and making some educated guesses, I wrote my first C program to create an APFS snapshot. This section has a bit of code, which you can find in this Github repo (https://github.com/ahl/apfs) That just returned “fs_snapshot: Operation not permitted” So, being Adam, he used DTrace to figure out what the problem was Running this DTrace script in one terminal while running the snapshot program in another shows the code flow through the kernel as the program executes In the code flow, the privcheckcred() function jumps out as a good place to continue because of its name, the fact that fs_snapshot calls it directly, and the fact that it returns 1 which corresponds with EPERM, the error we were getting. Turns out, it just requires some sudo With a little more testing I wrote my own version of Apple's unreleased snapUtil command from the WWDC demo We figured out the proper use of the fssnapshot system call and reconstructed the WWDC snapUtil. But all this time an equivalent utility has been lurking on macOS Sierra. If you look in /System/Library/Filesystems/apfs.fs/Contents/Resources/, Apple has included a number of APFS-related utilities, including apfssnapshot (and, tantalizingly, a tool called hfs_convert). Snapshots let you preserve state to later peruse; we can also revert an APFS volume to a previous state to restore its contents. The current APFS semantics around rollback are a little odd. The revert operation succeeds, but it doesn't take effect until the APFS volume is next mounted Another reason Apple may not have wanted people messing around with snapshots is that the feature appears to be incomplete. Winding yourself into a state where only a reboot can clear a mounted snapshot is easy, and using snapshots seems to break some of the diskutil APFS output It is interesting to see what you can do with DTrace, as well as to see what a DTrace and ZFS developer things of APFS *** Interview - Tom Jones - tj@enoti.me (mailto:tj@enoti.me) Replacing the BSD Sockets API *** News Roundup FreeBSD rc.d script to map ethernet device names by MAC address (https://github.com/eborisch/ethname) Self-contained FreeBSD rc.d script for re-naming devices based on their MAC address. I needed it due to USB Ethernet devices coming up in different orders across OS upgrades. Copy ethname into /usr/local/etc/rc.d/ Add the following to rc.conf: ethnameenable="YES" ethnamedevices="em0 ue0 ue1" # Replace with desired devices to rename Create /usr/local/etc/ifmap in the following format: 01:23:45:67:89:ab eth0 01:23:45:67:89:ac eth1 That's it. Use ifconfig_="" settings in rc.conf with the new names. I know MFSBSD has something like this, but a polished up hybrid of the two should likely be part of the base system if something is not already available This would be a great “Junior Job”, if say, a viewer wanted to get started with their first FreeBSD patch *** Mog: A different take on the Unix tool cat (https://github.com/witchard/mog) Do you abuse cat to view files? Did you know cat is meant for concatenating files, meaning: cat part1 part2 part3 > wholething.txt mog is a tool for actually viewing files, and it adds quite a few nice features Syntax highlight scripts Print a hex dump of binary files Show details of image files Perform objdump on executables List a directory mog reads the $HOME/.mogrc config file which describes a series of operations it can do in an ordered manner. Each operation has a match command and an action command. For each file you give to mog it will test each match command in turn, when one matches it will perform the action. A reasonably useful config file is generated when you first run it. How Unix erases things when you type a backspace while entering text (https://utcc.utoronto.ca/~cks/space/blog/unix/HowUnixBackspaces) Yesterday I mentioned in passing that printing a DEL character doesn't actually erase anything. This raises an interesting question, because when you're typing something into a Unix system and hit your backspace key, Unix sure erases the last character that you entered. So how is it doing that? The answer turns out to be basically what you'd expect, although the actual implementation rapidly gets complex. When you hit backspace, the kernel tty line discipline rubs out your previous character by printing (in the simple case) Ctrl-H, a space, and then another Ctrl-H. Of course just backing up one character is not always the correct way of erasing input, and that's when it gets complicated for the kernel. To start with we have tabs, because when you (the user) backspace over a tab you want the cursor to jump all the way back, not just move back one space. The kernel has a certain amount of code to work out what column it thinks you're on and then back up an appropriate number of spaces with Ctrl-Hs. Then we have the case when you quoted a control character while entering it, eg by typing Ctrl-V Ctrl-H; this causes the kernel to print the Ctrl-H instead of acting on it, and it prints it as the two character sequence ^H. When you hit backspace to erase that, of course you want both (printed) characters to be rubbed out, not just the 'H'. So the kernel needs to keep track of that and rub out two characters instead of just one. Chris then provides an example, from IllumOS, of the kernel trying to deal with multibyte characters FreeBSD also handles backspacing a space specially, because you don't need to actually rub that out with a 'b b' sequence; you can just print a plain b. Other kernels don't seem to bother with this optimization. The FreeBSD code for this is in sys/kern/ttyttydisc.c in the ttydiscrubchar function PS: If you want to see the kernel's handling of backspace in action, you usually can't test it at your shell prompt, because you're almost certainly using a shell that supports command line editing and readline and so on. Command line editing requires taking over input processing from the kernel, and so such shells are handling everything themselves. My usual way to see what the kernel is doing is to run 'cat >/dev/null' and then type away. And you thought the backspace key would be simple... *** FreeBSD ports now have Wayland (http://www.freshports.org/graphics/wayland/) We've discussed the pending Wayland work, but we wanted to point you today to the ports which are in mainline FreeBSD ports tree now. First of all, (And I was wondering how they would deal with this) it has landed in the “graphics” category, since Wayland is the Anti-X11, putting it in x11/ didn't make a lot of sense. Couple of notes before you start installing new packages and expecting wayland to “just work” First, this does require that you have working DRM from the kernel side. You'll want to grab TrueOS or build from Matt Macy's FreeBSD branches on GitHub before testing on any kind of modern Intel GPU. Nvidia with modesetting should be supported. Next, not all desktops will “just work”. You may need to grab experimental Weston for compositor. KDE / Gnome (And Lumina) and friends will grow Wayland support in the future, so don't expect to just fire up $whatever and have it all work out of box. Feedback is needed! This is brand new functionality for FreeBSD, and the maintainers will want to hear your results. For us on the TrueOS side we are interested as well, since we want to port Lumina over to Wayland soon(ish) Happy Experimenting! *** Beastie Bits Faces of FreeBSD 2017: Joseph Kong (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2017-joseph-kong/) OPNsense 17.1 “Eclectic Eagle”, based on FreeBSD 11 Released (https://opnsense.org/opnsense-17-1-released/) Why you should start programming on UNIX (http://www.koszek.com/blog/2017/01/28/why-you-should-start-programming-on-unix/) OpenSMTPD Mail Filtering (http://eradman.com/posts/opensmtpd-filtering.html) Feedback/Questions Zane - Databases and Jails (http://pastebin.com/89AyGe5F) Mohammad - USB Install (http://pastebin.com/Te8sz9id) Chuck - Updating Jails (http://pastebin.com/G2SzahWL) David - Lumina / LXQt (http://pastebin.com/71ExJLpL) ***

This week on the show, we've got FreeBSD quarterly Status reports to discuss, OpenBSD changes to the installer, EC2 and IPv6 and more. Stay This episode was brought to you by Headlines OpenBSD changes of note 6 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-6) OpenBSD can now be cross built with clang. Work on this continues Build ld.so with -fno-builtin because otherwise clang would optimize the local versions of functions like dlmemset into a call to memset, which doesn't exist. Add connection timeout for ftp (http). Mostly for the installer so it can error out and try something else. Complete https support for the installer. I wonder how they handle certificate verification. I need to look into this as I'd like to switch the FreeBSD installer to this as well New ocspcheck utility to validate a certificate against its ocsp responder. net lock here, net lock there, net lock not quite everywhere but more than before. More per cpu counters in networking code as well. Disable and lock Silicon Debug feature on modern Intel CPUs. Prevent wireless frame injection attack described at 33C3 in the talk titled “Predicting and Abusing WPA2/802.11 Group Keys” by Mathy Vanhoef. Add support for multiple transmit ifqueues per network interface. Supported drivers include bge, bnx, em, myx, ix, hvn, xnf. pledge now tracks when a file as opened and uses this to permit or deny ioctl. Reimplement httpd's support for byte ranges. Fixes a memory DOS. FreeBSD 2016Q4 Status Report (https://www.freebsd.org/news/status/report-2016-10-2016-12.html) An overview of some of the work that happened in October - December 2016 The ports tree saw many updates and surpassed 27,000 ports The core team was busy as usual, and the foundation attended and/or sponsored a record 24 events in 2016. CEPH on FreeBSD seems to be coming along nicely. For those that do not know, CEPH is a distributed filesystem that can sit on top of another filesystem. That is, you can use it to create a clustered filesystem out of a bunch of ZFS servers. Would love to have some viewers give it a try and report back. OpenBSM, the FreeBSD audit framework, got some updates Ed Schouten committed a front end to export sysctl data in a format usable by Prometheus, the open source monitoring system. This is useful for other monitoring software too. Lots of updates for various ARM boards There is an update on Reproducible Builds in FreeBSD, “ It is now possible to build the FreeBSD base system (kernel and userland) completely reproducibly, although it currently requires a few non-default settings”, and the ports tree is at 80% reproducible Lots of toolchain updates (gcc, lld, gdb) Various updates from major ports teams *** Amazon rolls out IPv6 support on EC2 (http://www.daemonology.net/blog/2017-01-26-IPv6-on-FreeBSD-EC2.html) A few hours ago Amazon announced that they had rolled out IPv6 support in EC2 to 15 regions — everywhere except the Beijing region, apparently. This seems as good a time as any to write about using IPv6 in EC2 on FreeBSD instances. First, the good news: Future FreeBSD releases will support IPv6 "out of the box" on EC2. I committed changes to HEAD last week, and merged them to the stable/11 branch moments ago, to have FreeBSD automatically use whatever IPv6 addresses EC2 makes available to it. Next, the annoying news: To get IPv6 support in EC2 from existing FreeBSD releases (10.3, 11.0) you'll need to run a few simple commands. I consider this unfortunate but inevitable: While Amazon has been unusually helpful recently, there's nothing they could have done to get support for their IPv6 networking configuration into FreeBSD a year before they launched it. You need the dual-dhclient port: pkg install dual-dhclient And the following lines in your /etc/rc.conf: ifconfigDEFAULT="SYNCDHCP acceptrtadv" ipv6activateallinterfaces="YES" dhclientprogram="/usr/local/sbin/dual-dhclient" + It is good to see FreeBSD being ready to use this feature on day 0, not something we would have had in the past Finally, one important caveat: While EC2 is clearly the most important place to have IPv6 support, and one which many of us have been waiting a long time to get, this is not the only service where IPv6 support is important. Of particular concern to me, Application Load Balancer support for IPv6 is still missing in many regions, and Elastic Load Balancers in VPC don't support IPv6 at all — which matters to those of us who run non-HTTP services. Make sure that IPv6 support has been rolled out for all the services you need before you start migrating. Colin's blog also has the details on how to actually activate IPv6 from the Amazon side, if only it was as easy as configuring it on the FreeBSD side *** FreeBSD's George Neville-Neil tries valiantly for over an hour to convince a Linux fan of the error of their ways (https://www.youtube.com/watch?v=cofKxtIO3Is) In today's episode of the Lunduke Hour I talk to George Neville-Neil -- author and FreeBSD advocate. He tries to convince me, a Linux user, that FreeBSD is better. + They cover quite a few topics, including: + licensing, and the motivations behind it + vendor relations + community + development model + drivers and hardware support + George also talks about his work with the FreeBSD Foundation, and the book he co-authored, “The Design and Implementation of the FreeBSD Operating System, 2nd Edition” News Roundup An interactive script that makes it easy to install 50+ desktop environments following a base install of FreeBSD 11 (https://github.com/rosedovell/unixdesktops) And I thought I was doing good when I wrote a patch for the installer that enables your choice of 3 desktop environments... This is a collection of scripts meant to install desktop environments on unix-like operating systems following a base install. I call one of these 'complete' when it meets the following requirements: + A graphical logon manager is presented without user intervention after powering on the machine + Logging into that graphical logon manager takes the user into the specified desktop environment + The user can open a terminal emulator I need to revive my patch, and add Lumina to it *** Firefox 51 on sparc64 - we did not hit the wall yet (https://blog.netbsd.org/tnf/entry/firefox_51_on_sparc64_we) A NetBSD developers tells the story of getting Firefox 51 running on their sparc64 machine It turns out the bug impacted amd64 as well, so it was quickly fixed They are a bit less hopeful about the future, since Firefox will soon require rust to compile, and rust is not working on sparc64 yet Although there has been some activity on the rust on sparc64 front, so maybe there is hope The post also look at a few alternative browsers, but it not hopeful *** Introducing Bloaty McBloatface: a size profiler for binaries (http://blog.reverberate.org/2016/11/07/introducing-bloaty-mcbloatface.html) I'm very excited to announce that today I'm open-sourcing a tool I've been working on for several months at Google. It's called Bloaty McBloatface, and it lets you explore what's taking up space in your .o, .a, .so, and executable binary files. Bloaty is available under the Apache 2 license. All of the code is available on GitHub: github.com/google/bloaty. It is quick and easy to build, though it does require a somewhat recent compiler since it uses C++11 extensively. Bloaty primarily supports ELF files (Linux, BSD, etc) but there is some support for Mach-O files on OS X too. I'm interested in expanding Bloaty's capabilities to more platforms if there is interest! I need to try this one some of the boot code files, to see if there are places we can trim some fat We've been using Bloaty a lot on the Protocol Buffers team at Google to evaluate the binary size impacts of our changes. If a change causes a size increase, where did it come from? What sections/symbols grew, and why? Bloaty has a diff mode for understanding changes in binary size The diff mode looks especially interesting. It might be worth setting up some kind of CI testing that alerts if a change results in a significant size increase in a binary or library *** A BSD licensed mdns responder (https://github.com/kristapsdz/mdnsd) One of the things we just have to deal with in the modern world is service and system discovery. Many of us have fiddled with avahi or mdnsd and related “mdns” services. For various reasons those often haven't been the best-fit on BSD systems. Today we have a github project to point you at, which while a bit older, has recently been updated with pledge() support for OpenBSD. First of all, why do we need an alternative? They list their reasons: This is an attempt to bring native mdns/dns-sd to OpenBSD. Mainly cause all the other options suck and proper network browsing is a nice feature these days. Why not Apple's mdnsd ? 1 - It sucks big time. 2 - No BSD License (Apache-2). 3 - Overcomplex API. 4 - Not OpenBSD-like. Why not Avahi ? 1 - No BSD License (LGPL). 2 - Overcomplex API. 3 - Not OpenBSD-like 4 - DBUS and lots of dependencies. Those already sound like pretty compelling reasons. What makes this “new” information again is the pledge support, and perhaps it's time for more BSD's to start considering importing something like mdnsd into their base system to make system discovery more “automatic” *** Beastie Bits Benno Rice at Linux.Conf.Au: The Trouble with FreeBSD (https://www.youtube.com/watch?v=Ib7tFvw34DM) State of the Port of VMS to x86 (http://vmssoftware.com/pdfs/State_of_Port_20170105.pdf) Microsoft Azure now offers Patent Troll Protection (https://thestack.com/cloud/2017/02/08/microsoft-azure-now-offers-patent-troll-ip-protection/) FreeBSD Storage Summit 2017 (https://www.freebsdfoundation.org/news-and-events/event-calendar/freebsd-storage-summit-2017/) If you are going to be in Tokyo, make sure you come to (http://bhyvecon.org/) Feedback/Questions Farhan - Laptops (http://pastebin.com/bVqsvM3r) Hjalti - rclone (http://pastebin.com/7KWYX2Mg) Ivan - Jails (http://pastebin.com/U5XyzMDR) Jungle - Traffic Control (http://pastebin.com/sK7uEDpn) ***

This week on BSDNow we have a cantrill special to bring you! All three interviews back to back in their original glory, you won't want to miss This episode was brought to you by – Show Notes: –FOSDEM 2017 BSD Dev Room Videos Ubuntu Slaughters Kittens | BSD Now 103The Cantrill Strikes Back | BSD Now 117Return of the Cantrill | BSD Now 163

This week on BSDNow, I'm out of town but we have a great interview with Ken Moore (My brother) about the latest in BSD desktop computing and This episode was brought to you by Interview - Ken Moore - ken@trueos.org (mailto:ken@trueos.org) TrueOS, Lumina, Sys Admin, The BSD Desktop Ecosystem + KM: Thank you for joining us again, can you believe it has been an entire year? + AJ: Let's start by getting an update on Lumina, what has happened in the last year? + KM: What is the change you are most proud of in that time? + AJ: What do you think of the recent introduction of Wayland to the ports tree? Do you think this will impact Lumina? Do you have any plans? + KM: + AJ: What has changed with SysAdm after a year of development? + KM: What plans do you have for the future of SysAdm? + AJ: How has it been working with the drm-next branch? Does it feel like that is progressing? + KM: Can you tell us about some of the other TrueOS work you have been doing? + AJ: What are your thoughts on how the BSD Desktop Ecosystem has changed over the last year? Do you think the future looks better or worse now? + KM: Do you think systemd is going to continue to make things work? Or does it seem like there is enough resistance to it that fewer projects are going to throw out support for anything not-systemd + AJ: Anything else you want to add?

This week on BSDNow, we're going to be leading off with the latest news about Wayland and Xorg support on FreeBSD, then a look at OpenBSD ARM64 This episode was brought to you by Headlines Wayland is now in the FreeBSD Ports tree (https://svnweb.freebsd.org/ports?view=revision&revision=432406) This commit brings Wayland, the new windowing system, into the FreeBSD ports tree “This port was first created by Koop Mast (kwm@) then updated and improved by Johannes Lundberg” “Wayland is intended as a simpler replacement for X, easier to develop and maintain. GNOME and KDE are expected to be ported to it.” Wayland is designed for desktop and laptop use, rather than X, which was designed for use over the network, where clients were not powerful enough to run the applications locally. “Wayland is a protocol for a compositor to talk to its clients as well as a C library implementation of that protocol. The compositor can be a standalone display server running on Linux kernel modesetting and evdev input devices, an X application, or a wayland client itself. The clients can be traditional applications, X servers (rootless or fullscreen) or other display servers.” “Please report bugs to the FreeBSD bugtracker!” It is good to see this project progressing, as it seems in a few generations, high performance graphics drivers may only be actively developed for Wayland. *** Call For Testing: xorg 1.18.4 and newer intel/ati DDX (https://lists.freebsd.org/pipermail/freebsd-x11/2017-January/018738.html) Baptiste Daroussin, and the FreeBSD X11 team, have issued a call for testing for the upgrade to Xorg 1.18.4 Along with it comes newer ATI/AMD and Intel drivers “Note that you will need to rebuild all the xf86-* packages to work with thatnewer xorg (hence the bump of the revision)” “Do not expect newer gpu supported as this is not the kernel part”, it only provides the newer Xorg driver, not the kernel mode setting driver (this is a separate project) “If you experience any issue with intel or radeon driver please try to use the new modesetting driver provided by xorg directly (note that fedora and debian recommend the use of the new driver instead of the ati/intel one)” *** Error handling in C (http://www.tedunangst.com/flak/post/to-errno-or-to-error) “Unlike other languages which have one preferred means of signalling an error, C is a multi error paradigm language. Error handling styles in C can be organized into one of several distinct styles, such as popular or correct. Some examples of each.” “One very popular option is the classic unix style. -1 is returned to indicate an error.” “Another option seen in the standard C library is NULL for errors.” “The latter has the advantage that NULL is a false value, which makes it easier to write logical conditions. File descriptor 0 is valid (stdin) but false, while -1 is invalid but true.” “And of course, there's the worst of both worlds approach requiring a special sentinel that you'll probably forget to use” “Other unix functions, those that don't need to return a file descriptor, stick to just 0 and -1” “Of course, none of these functions reveal anything about the nature of the error. For that, you need to consult the errno on the side” The article goes on to describe different ways of dealing with the issue, and return values. There is also coverage of more complex examples and involve a context that might contain the error message It is really interesting to see the differences, and the pitfalls of each approach *** Fixing POSIX Filenames (http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html) “Traditionally, Unix/Linux/POSIX pathnames and filenames can be almost any sequence of bytes. A pathname lets you select a particular file, and may include zero or more “/” characters. Each pathname component (separated by “/”) is a filename; filenames cannot contain “/”. Neither filenames nor pathnames can contain the ASCII NUL character (), because that is the terminator.” “This lack of limitations is flexible, but it also creates a legion of unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws). It also makes it impossible to consistently and accurately display filenames, causes portability problems, and confuses users.” “This article will try to convince you that adding some tiny limitations on legal Unix/Linux/POSIX filenames would be an improvement. Many programs already presume these limitations, the POSIX standard already permits such limitations, and many Unix/Linux filesystems already embed such limitations — so it'd be better to make these (reasonable) assumptions true in the first place. This article will discuss, in particular, the three biggest problems: control characters in filenames (including newline, tab, and escape), leading dashes in filenames, and the lack of a standard character encoding scheme (instead of using UTF-8). These three problems impact programs written in any language on Unix/Linux/POSIX system. There are other problems, of course. Spaces in filenames can cause problems; it's probably hopeless to ban them outright, but resolving some of the other issues will simplify handling spaces in filenames. For example, when using a Bourne shell, you can use an IFS trick (using IFS=printf 'nt') to eliminate some problems with spaces. Similarly, special metacharacters in filenames cause some problems; I suspect few if any metacharacters could be forbidden on all POSIX systems, but it'd be great if administrators could locally configure systems so that they could prevent or escape such filenames when they want to. I then discuss some other tricks that can help.” “After limiting filenames slightly, creating completely-correct programs is much easier, and some vulnerabilities in existing programs disappear. This article then notes some others' opinions; I knew that some people wouldn't agree with me, but I'm heartened that many do agree that something should be done. Finally, I briefly discuss some methods for solving this long-term; these include forbidding creation of such names (hiding them if they already exist on the underlying filesystem), implementing escaping mechanisms, or changing how tools work so that these are no longer problems (e.g., when globbing/scanning, have the libraries prefix “./” to any filename beginning with “-”). Solving this is not easy, and I suspect that several solutions will be needed. In fact, this paper became long over time because I kept finding new problems that needed explaining (new “worms under the rocks”). If I've convinced you that this needs improving, I'd like your help in figuring out how to best do it!” “Filename problems affect programs written in any programming language. However, they can be especially tricky to deal with when using Bourne shells (including bash and dash). If you just want to write shell programs that can handle filenames correctly, you should see the short companion article Filenames and Pathnames in Shell: How to do it correctly (http://www.dwheeler.com/essays/filenames-in-shell.html).” Imagine that you don't know Unix/Linux/POSIX (I presume you really do), and that you're trying to do some simple tasks. For our purposes we will create simple scripts on the command line (using a Bourne shell) for these tasks, though many of the underlying problems affect any program. For example, let's try to print out the contents of all files in the current directory, putting the contents into a file in the parent directory: cat * > ../collection # WRONG cat ./* > ../collection # CORRECT cat find . -type f > ../collection # WRONG ( set -f ; for file in find . -type f ; do # WRONG cat "$file" done ) > ../collection ( find . -type f | xargs cat ) > ../collection # WRONG, WAY WRONG Just think about trying to remove a file named: -rf / *** News Roundup OpenBSD ARM64 (https://www.openbsd.org/arm64.html) A new page has appeared on the OpenBSD website, offering images for ARM64 “The current target platforms are the Pine64 and the Raspberry Pi 3.” “OpenBSD/arm64 bundles various platforms sharing the 64-bit ARM architecture. Due to the fact that there are many System on a Chips (SoC) around, OpenBSD/arm64 differentiates between various SoCs and may have a different level of support between them” The page contains a list of the devices that are supported, and which components have working drivers At the time of recording, the link to download the snapshots did not work yet, but by time this airs a week from now, it should be working. *** The design of Chacha20 (http://loup-vaillant.fr/tutorials/chacha20-design) Seems like every few episodes we end up discussing Ciphers (With their o-so amusing naming) and today is no exception. We have a great writeup on the D & I of the ‘chacha20' cipher written by “Loup Vaillant” First of all, is this story for you? Maybe the summary will help make that call: “Quick summary: Chacha20 is ARX-based hash function, keyed, running in counter mode. It embodies the idea that one can use a hash function to encrypt data.” If your eyes didn't glaze over, then you are cleared to proceed. Chacha20 is built around stream ciphers: While Chacha20 is mainly used for encryption, its core is a pseudo-random number generator. The cipher text is obtained by XOR'ing the plain text with a pseudo-random stream: ciphertext = plaintext XOR chacha_stream(key, nonce) Provided you never use the same nonce with the same key twice, you can treat that stream as a one time pad. This makes it very simple: unlike block ciphers, you don't have to worry about padding, and decryption is the same operation as encryption: plaintext = ciphertext XOR chacha_stream(key, nonce) Now we just have to get that stream. The idea that the streams can mimic the concept of a one-time pad does make chacha20 very attractive, even to a non-crypto guy such as myself. From here the article goes into depth on how the cipher scrambles 512bit blocks using the quarter-round method (A forth of a block or 4 32bit numbers) Some ascii art is used here to help visualize how this done, in the quarter round-phase, then to the complete block as the 4 quarters are run in parallel over the entire 512 bit block. From here the article goes more into depth, looking at the complete chacha block, and the importance of a seemingly unnecessary 32byte constant (Hint: it's really important) If crypto is something you find fascinating, you'll want to make sure you give this one a full read-through. *** CyberChef - Coming to a FreeBSD Ports tree near you (https://twitter.com/DLangille/status/823915729430913025) Dan Langille tweets that he will be creating a port of GCHQ's CyberChef tool “CyberChef is a simple, intuitive web app for carrying out all manner of "cyber" operations within a web browser. These operations include creating hexdumps, simple encoding like XOR or Base64, more complex encryption like AES, DES and Blowfish, data compression and decompression, calculating hashes and checksums, IPv6 and X.509 parsing, and much more.” “The tool is designed to enable both technical and non-technical analysts to manipulate data in complex ways without having to deal with complex tools or algorithms. It was conceived, designed, built and incrementally improved by an analyst in their 10% innovation time over several years. Every effort has been made to structure the code in a readable and extendable format, however it should be noted that the analyst is not a professional developer and the code has not been peer-reviewed for compliance with a formal specification.” Some handy functions, beyond stuff like base64 encoding: Network Enumeration (CIDR to list of IPS) (https://gchq.github.io/CyberChef/?recipe=%5B%7B%22op%22%3A%22Parse%20IP%20range%22%2C%22args%22%3A%5Btrue%2Ctrue%2Cfalse%5D%7D%5D&input=MTcyLjIxLjAuMzIvMjcK) Browser User Agent Parser (what browser is that, based on your HTTP logs) XOR Brute Force: enter some XOR'd text, and try every possible key to find plaintext. Optionally give it a regex of known plaintext to find the right key. Calculate the “Shannon Entropy” of the input (how random is this data) It also has a number of built in regular expressions for common things, very useful The project is up on github if you want to play with the code *** Building Electron and VSCode in FreeBSD11 (https://gist.github.com/prash-wghats/89be1ee069d2acf23c289e9c606616e1) A patch and set of instructions for building Electron and VSCode on FreeBSD “Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring. It is also customizable, so users can change the editor's theme, keyboard shortcuts, and preferences. It is free and open-source, although the official download is under a proprietary license.” “Visual Studio Code is based on Electron, a framework which is used to deploy Node.js applications for the desktop running on the Blink layout engine. Although it uses the Electron framework, the software is not a fork of Atom, it is actually based on Visual Studio Online's editor (codename "Monaco")” It would be interesting to see official support for VSCode on FreeBSD Has anyone tried VSCode on the FreeBSD Code base? *** Beastie Bits Soft Label Keys (http://roy.marples.name/blog/blog/soft-label-keys) WPA1 (TKIP) disabled by default (OpenBSD) (https://www.mail-archive.com/source-changes@openbsd.org/msg84599.html) Cool but obscure unix tools (https://kkovacs.eu/cool-but-obscure-unix-tools) KDE Frameworks and Plasma on FreeBSD (http://euroquis.nl/bobulate/?p=1521) Initiative to migrate OpenBSD mirrors to HTTPS (https://www.mail-archive.com/source-changes@openbsd.org/msg84904.html) That moment you realize FreeBSD has got some Star Wars fans (http://i.imgur.com/dC7c1y4.png) Pagelink (https://wiki.freebsd.org/PortsSubversionPrimer)

This week on BSD Now, we will be discussing a wide variety of topics including Routers, Run-Controls, the “Rule” of silence and some This episode was brought to you by Headlines Ports no longer build on EOL FreeBSD versions (https://www.reddit.com/r/freebsd/comments/5ouvmp/ports_no_longer_build_on_eol_freebsd_versions/) The FreeBSD ports tree has been updated to automatically fail if you try to compile ports on EOL versions of FreeBSD (any version of 9.x or earlier, 10.0 - 10.2, or 11 from before 11.0) This is to prevent shooting yourself in the food, as the compatibility code for those older OSes has been removed now that they are no longer supported. If you use pkg, you will also run into problems on old releases. Packages are always built on the oldest supported release in a branch. Until recently, this meant packages for 10.1, 10.2, and 10.3 were compiled on 10.1. Now that 10.1 and 10.2 are EOL, packages for 10.x are compiled on 10.3. This matters because 10.3 supports the new openat() and various other *at() functions used by capsicum. Now that pkg and packages are built on a version that supports this new feature, they will not run on systems that do not support it. So pkg will exit with an error as soon as it tries to open a file. You can work around this temporarily by using the pkg-static command, but you should upgrade to a supported release immediately. *** Improving TrueOS: OpenRC (https://www.trueos.org/blog/improving-trueos-openrc/) With TrueOS moving to a rolling-release model, we've decided to be a bit more proactive in sharing news about new features that are landing. This week we've posted an article talking about the transition to OpenRC In past episodes you've heard me mention OpenRC, but hopefully today we can help answer any of those lingering questions you may still have about it The first thing always asked, is “What is OpenRC?” OpenRC is a dependency-based init system working with the system provided init program. It is used with several Linux distributions, including Gentoo and Alpine Linux. However, OpenRC was created by the NetBSD developer Roy Marples in one of those interesting intersections of Linux and BSD development. OpenRC's development history, portability, and 2-clause BSD license make its integration into TrueOS an easy decision. Now that we know a bit about what it is, how does it behave differently than traditional RC? TrueOS now uses OpenRC to manage all system services, as opposed to FreeBSD's RC. Instead of using rc.d for base system rc scripts, OpenRC uses init.d. Also, every service in OpenRC has its own user configuration file, located in /etc/conf.d/ for the base system and /usr/local/etc.conf.d/ for ports. Finally, OpenRC uses runlevels, as opposed to the FreeBSD single- or multi- user modes. You can view the services and their runlevels by typing $ rc-update show -v in a CLI. Also, TrueOS integrates OpenRC service management into SysAdm with the Service Manager tool One of the prime benefits of OpenRC is much faster boot-times, which is important in a portable world of laptops (and desktops as well). But service monitoring and crash detection are also important parts of what make OpenRC a substantial upgrade for TrueOS. Lastly people have asked us about migration, what is done, what isn't? As of now almost all FreeBSD base system services have been migrated over. In addition most desktop-facing services required to run Lumina and the like are also ported. We are still going through the ports tree and converting legacy rc.d scripts to init.d, but the process takes time. Several new folks have begun contributing OpenRC scripts and we hope to have all the roughly 1k ports converted over this year. BSDRP Releases 1.70 (https://sourceforge.net/projects/bsdrp/files/BSD_Router_Project/1.70/) A new release of the BSD Router Project This distro is designed to replace high end routers, like those from Cisco and Juniper, with FreeBSD running on regular off-the-shelf server. Highlights: Upgraded to FreeBSD 11.0-STABLE r312663 (skip 11.0 for massive performance improvement) Re-Added: netmap-fwd (https://github.com/Netgate/netmap-fwd) Add FIBsync patch to netmap-fwd from Zollner Robert netmap pkt-gen supports IPv6, thanks to Andrey V. Elsukov (ae@freebsd.org) bird 1.6.3 (add BGP Large communities support) OpenVPN 2.4.0 (adds the high speed AEAD GCM cipher) All of the other packages have also been upgraded A lot of great work has been done on BSDRP, and it has also generated a lot of great benchmarks and testing that have resulted in performance increases and improved understanding of how FreeBSD networking scales across different CPU types and speeds *** DragonFlyBSD gets UEFI support (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/7b1aa074fcd99442a1345fb8a695b62d01d9c7fd) This commit adds support for UEFI to the Dragonfly Installer, allowing new systems to be installed to boot from UEFI This script (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/9d53bd00e9be53d6b893afd79111370ee0c053b0) provides a way to build a HAMMER filesystem that works with UEFI There is also a UEFI man page (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/d195d5099328849c500d4a1b94d6915d3c72c71e) The install media (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/5fa778d7b36ab0981ff9dcbd96c71ebf653a6a19) has also been updated to support booting from either UEFI or MBR, in the same way that the FreeBSD images work *** News Roundup The Rule of Silence (http://www.linfo.org/rule_of_silence.html) “The rule of silence, also referred to as the silence is golden rule, is an important part of the Unix philosophy that states that when a program has nothing surprising, interesting or useful to say, it should say nothing. It means that well-behaved programs should treat their users' attention and concentration as being valuable and thus perform their tasks as unobtrusively as possible. That is, silence in itself is a virtue.” This doesn't mean a program cannot be verbose, it just means you have to ask it for the additional output, rather than having it by default “There is no single, standardized statement of the Unix philosophy, but perhaps the simplest description would be: "Write programs that are small, simple and transparent. Write them so that they do only one thing, but do it well and can work together with other programs." That is, the philosophy centers around the concepts of smallness, simplicity, modularity, craftsmanship, transparency, economy, diversity, portability, flexibility and extensibility.” “This philosophy has been fundamental to the the fact that Unix-like operating systems have been thriving for more than three decades, far longer than any other family of operating systems, and can be expected to see continued expansion of use in the years to come” “The rule of silence is one of the oldest and most persistent design rules of such operating systems. As intuitive as this rule might seem to experienced users of such systems, it is frequently ignored by the developers of other types of operating systems and application programs for them. The result is often distraction, annoyance and frustration for users.” “There are several very good reasons for the rule of silence: (1) One is to avoid cluttering the user's mind with information that might not be necessary or might not even be desired. That is, unnecessary information can be a distraction. Moreover, unnecessary messages generated by some operating systems and application programs are sometimes poorly worded, and can cause confusion or needless worry on the part of users.” No news is good news. When there is bad news, error messages should be descriptive, and ideally tell the user what they might do about the error. “A third reason is that command line programs (i.e., all-text mode programs) on Unix-like operating systems are designed to work together with pipes, i.e., the output from one program becomes the input of another program. This is a major feature of such systems, and it accounts for much of their power and flexibility. Consequently, it is important to have only the truly important information included in the output of each program, and thus in the input of the next program.” Have you ever had to try to strip out useless output so you could feed that data into another program? “The rule of silence originally applied to command line programs, because all programs were originally command line programs. However, it is just as applicable to GUI (graphical user interfaces) programs. That is, unnecessary and annoying information should be avoided regardless of the type of user interface.” “A example is the useless and annoying dialog boxes (i.e., small windows) that pop up on the display screen with with surprising frequency on some operating systems and programs. These dialog boxes contain some obvious, cryptic or unnecessary message and require the user to click on them in order to close them and proceed with work. This is an interruption of concentration and a waste of time for most users. Such dialog boxes should be employed only in situations in which some unexpected result might occur or to protect important data.” It goes on to make an analogy about Public Address systems. If too many unimportant messages, like advertisements, are sent over the PA system, people will start to ignore them, and miss the important announcements. *** The Tao of tmux (https://leanpub.com/the-tao-of-tmux/read) An interesting article floated across my news feed a few weeks back. It's what essentially boils down to a book called the “Tao of tmux”, which immediately piqued my interest. My story may be similar to many of yours. I was initially raised on using screen, and screen only for my terminal session and multiplexing needs. Since then I've only had a passing interest in tmux, but its always been one of those utilities I felt was worthy of investing some more time into. (Especially when seeing some of the neat setups some of my peers have with it) Needless to say, this article has been bookmarked, and I've started digesting some of it, but thought it would be good to share with anybody else who finds them-self in a similar situation. The book starts off well, explaining in the simplest terms possible what Tmux really is, by comparing and contrasting it to something we are all familiar with, GUIS! Helpfully they also include a chart which explains some of the terms we will be using frequently when discussing tmux (https://leanpub.com/the-tao-of-tmux/read#leanpub-auto-window-manager-for-the-terminal) One of the things the author does recommend is also making sure you are up to speed on your Terminal knowledge. Before getting into tmux, a few fundamentals of the command line should be reviewed. Often, we're so used to using these out of street smarts and muscle memory a great deal of us never see the relation of where these tools stand next to each other. Seasoned developers are familiar with zsh, Bash, iTerm2, konsole, /dev/tty, shell scripting, and so on. If you use tmux, you'll be around these all the time, regardless of whether you're in a GUI on a local machine or SSH'ing into a remote server. If you want to learn more about how processes and TTY's work at the kernel level (data structures and all) the book The Design and Implementation of the FreeBSD Operating System (2nd Edition) by Marshall Kirk McKusick is nice. In particular, Chapter 4, Process Management and Section 8.6, Terminal Handling. The TTY demystified by Linus Åkesson (available online) dives into the TTY and is a good read as well. We had to get that shout-out of Kirk's book in here ;) From here the boot/article takes us on a whirlwind journey of Sessions, Windows, Panes and more. Every control- command is covered, information on how to customize your statusbar, tips, tricks and the like. There's far more here than we can cover in a single segment, but you are highly encouraged to bookmark this one and start your own adventure into the world of tmux. *** SDF Celebrates 30 years of service in 2017 (https://sdf.org/) HackerNews thread on SDF (https://news.ycombinator.com/item?id=13453774) “Super Dimension Fortress (SDF, also known as freeshell.org) is a non-profit public access UNIX shell provider on the Internet. It has been in continual operation since 1987 as a non-profit social club. The name is derived from the Japanese anime series The Super Dimension Fortress Macross; the original SDF server was a BBS for anime fans[1]. From its BBS roots, which have been well documented as part of the BBS: The Documentary project, SDF has grown into a feature-rich provider serving members around the world.” A public access UNIX system, it was many people's first access to a UNIX shell. In the 90s, Virtual Machines were rare, the software to run them usually cost a lot of money and no one had very much memory to try to run two operating systems at the same time. So for many people, these type of shell accounts were the only way they could access UNIX without having to replace the OS on their only computer This is how I first started with UNIX, eventually moving to paying for access to bigger machines, and then buying my own servers and renting out shell accounts to host IRC servers and channel protection bots. “On June 16th, 1987 Ted Uhlemann (handle: charmin, later iczer) connected his Apple ][e's 300 baud modem to the phone line his mother had just given him for his birthday. He had published the number the night before on as many BBSes around the Dallas Ft. Worth area that he could and he waited for the first caller. He had a copy of Magic Micro BBS which was written in Applesoft BASIC and he named the BBS "SDF-1" after his favorite Japanimation series ROBOTECH (Macross). He hoped to draw users who were interested in anime, industrial music and the Church of the Subgenius.” I too started out in the world of BBSes before I had access to the internet. My parents got my a dedicated phone line for my birthday, so I wouldn't tie up their line all the time. I quickly ended up running my own BBS, the Sudden Death BBS (Renegade (https://en.wikipedia.org/wiki/Renegade_(BBS)) on MS DOS) I credit this early experience for my discovery of a passion for Systems Administration, that lead me to my current career “Slowly, SDF has grown over all these years, never forgetting our past and unlike many sites on the internet, we actually have a past. Some people today may come here and see us as outdated and "retro". But if you get involved, you'll see it is quite alive with new ideas and a platform for opportunity to try many new things. The machines are often refreshed, the quotas are gone, the disk space is expanding as are the features (and user driven features at that) and our cabinets have plenty of space for expansion here in the USA and in Europe (Germany).” “Think about ways you'd like to celebrate SDF's 30th and join us on the 'bboard' to discuss what we could do. I realize many of you have likely moved on yourselves, but I just wanted you to know we're still here and we'll keep doing new and exciting things with a foundation in the UNIX shell.” *** Getting Minecraft to Run on NetBSD (https://www.reddit.com/r/NetBSD/comments/5mtsy1/getting_minecraft_to_run_on_netbsd/) One thing that doesn't come up often on BSDNow is the idea of gaming. I realize most of us are server folks, or perhaps don't play games (The PC is for work, use your fancy-smanzy PS4 and get off my lawn you kids) Today I thought it would be fun to highlight this post over at Reddit talking about running MineCraft on NetBSD Now I realize this may not be news to some of you, but perhaps it is to others. For the record my kids have been playing Minecraft on PC-BSD / TrueOS for years. It's the primary reason they are more often booted into that instead of Windows. (Funny story behind that - Got sick of all the 3rd party mods, which more often than not came helpfully bundled with viruses and malware) On NetBSD the process looks a bit different than on FreeBSD. First up, you'll need to enable Linux Emulation and install Oracle JRE (Not OpenJDK, that path leads to sadness here) The guide will then walk us through the process of fetching the Linux runtime packages, extracting and then enabling bits such as ‘procfs' that is required to run the Linux binaries. Once that's done, minecraft is only a simple “oracle8-jre /path/to/minecraft.jar” command away from starting up, and you'll be “crafting” in no time. (Does anybody even play survival anymore?) *** Beastie Bits UNIX on the Computer Chronicals (https://youtu.be/g7P16mYDIJw) FreeBSD: Atheros AR9380 and later, maximum UDP TX goes from 250mbit to 355mbit. (https://twitter.com/erikarn/status/823298416939659264) Capsicumizing traceroute with casper (https://reviews.freebsd.org/D9303) Feedback/Questions Jason - TarSnap on Windows (http://pastebin.com/Sr1BTzVN) Mike - OpenRC & DO (http://pastebin.com/zpHyhHQG) Anonymous - Old Machines (http://pastebin.com/YnjkrDmk) Matt - Iocage (http://pastebin.com/pBUXtFak) Hjalti - Rclone & FreeNAS (http://pastebin.com/zNkK3epM)

This week on BSDNow, we've got Wifi galore, a new iocage and some RPi3 news and guides to share. Stay tuned for your place to B...SD! This episode was brought to you by Headlines WiFi: 11n hostap mode added to athn(4) driver, testers wanted (http://undeadly.org/cgi?action=article&sid=20170109213803) “OpenBSD as WiFi access points look set to be making a comeback in the near future” “Stefan Sperling added 802.11n hostap mode, with full support initially for the Atheros chips supported by the athn(4) driver.” “Hostap performance is not perfect yet but should be no worse than 11a/b/g modes in the same environment.” “For Linux clients a fix for WME params is needed which I also posted to tech@” “This diff does not modify the known-broken and disabled ar9003 code, apart from making sure it still builds.” “I'm looking for both tests and OKs.” There has also been a flurry of work (http://svnweb.freebsd.org/base/head/sys/net80211/?view=log) in FreeBSD on the ath10k driver, which supports 802.11ac Like this one (https://svnweb.freebsd.org/base?view=revision&revision=310147) and this one (https://svnweb.freebsd.org/base?view=revision&revision=311579) The long-awaited iocage update has landed (https://github.com/iocage/iocage) We've hinted at the new things happening behind the scenes with iocage, and this last week the code has made its first public debut. So what's changed you may ask. The biggest is that iocage has undergone a complete overhaul, moving from its original shell-base to python. The story behind that is that the author (Brandon) works at iXsystems, and the plan is to move away from the legacy warden-based jail management which was also shell-based. This new python re-write will allow it to integrate into FreeNAS (and other projects) better by exposing an API for all jail management tasks. Thats right, no more ugly CLI output parsing just to wrangle jail options either at creation or runtime. But what about users who just run iocage manually from the CLI? No worries, the new iocage is almost identical to the original CLI usage, making the switch over very simple. Just to re-cap, lets look at the new features list: “FEATURES: + Ease of use + Rapid jail creation within seconds + Automatic package installation + Virtual networking stacks (vnet) + Shared IP based jails (non vnet) + Transparent ZFS snapshot management + Export and import “ + The new iocage is available now via ports and packages under sysutils/py-iocage, give it a spin and be sure to report issues back to the developer(s). Reading DHT11 temperature sensors on a Raspberry Pi under FreeBSD (https://smallhacks.wordpress.com/2017/01/14/reading-dht11-temperature-sensor-on-raspberry-pi-under-freebsd/) “DHT-11 is a very cheap temperature/humidity sensor which is commonly used in the IoT devices. It is not very accurate, so for the accurate measurement i would recommend to use DHT21 instead. Anyway, i had DHT-11 in my tool box, so decided to start with it. DHT-11 using very simple 1 wire protocol – host is turning on chip by sending 18ms low signal to the data output and then reading 40 bytes of data.” “To read data from the chip it should be connected to the power (5v) and gpio pin. I used pin 2 as VCC, 6 as GND and 11 as GPIO” “There is no support for this device out of the box on FreeBSD. I found some sample code on the github, see lex/freebsd-gpio-dht11 (https://github.com/lex/freebsd-gpio-dht11) repository. This code was a good starting point, but soon i found 2 issues with it: Results are very unreliable, probably due to gpio decoding algorithm. Checksum is not validated, so sometime values are bogus. “Initially i was thinking to fix this myself, but later found kernel module for this purpose, 1 wire over gpio (http://www.my-tour.ru/FreeBSD/1-wire_over_gpio/). This module contains DHT11 kernel driver (gpio_sw) which implements DHT-11 protocol in the kernel space and exporting /dev/sw0 for the userland. Driver compiles on FreeBSD11/ARM without any changes. Use make install to install the driver.” The articles goes into how to install and configure the driver, including a set of devfs rules to allow non-root users to read from the sensor “Final goal was to add this sensor to the domoticz software. It is using LUA scripting to extend it functionality, e.g. to obtain data from non-supported or non standard devices. So, i decided to read /dev/sw0 from the LUA.” They ran into some trouble with LUA trying to read too much data at once, and had to work around it In the end, they got the results and were able to use them in the monitoring tool *** Tor-ified Home Network using HardenedBSD and a RPi3 (https://github.com/lattera/articles/blob/master/infosec/tor/2017-01-14_torified_home/article.md) Shawn from HardendBSD has posted an article up on GitHub talking about his deployment of a new Tor relay on a RPi3 This particular method was attractive, since it allows running a Relay, but without it being on a machine which may have personal data, such as SSH keys, files, etc While his setup is done on HardendBSD, the same applies to a traditional FreeBSD setup as well. First up, is the list of things needed for this project: Raspberry Pi 3 Model B Rev 1.2 (aka, RPI3) Serial console cable for the RPI3 Belkin F4U047 USB Ethernet Dongle Insignia NS-CR2021 USB 2.0 SD/MMC Memory Card Reader 32GB SanDisk Ultra PLUS MicroSDHC A separate system, running FreeBSD or HardenedBSD HardenedBSD clang 4.0.0 image for the RPI3 An external drive to be formatted A MicroUSB cable to power the RPI3 Two network cables Optional: Edimax N150 EW-7811Un Wireless USB Basic knowledge of vi After getting HBSD running on the RPi3 and serial connection established, he then takes us through the process of installing and enabling the various services needed. (Don't forget to growfs your sdcard first!) Now the tricky part is that some of the packages needed to be compiled from ports, which is somewhat time-consuming on a RPi. He strongly recommends not compiling on the sdcard (it sounds like personal experience has taught him well) and to use iscsi or some external USB drive. With the compiling done, our package / software setup is nearly complete. Next up is firewalling the box, which he helpfully provides a full PF config setup that we can copy-n-paste here. The last bits will be enabling the torrc configuration knobs, which if you follow his example again, will result in a tor public relay, and a local transparent proxy for you. Bonus! Shawn helpfully provides DHCPD configurations, and even Wireless AP configurations, if you want to setup your RPi3 to proxy for devices that connect to it. *** News Roundup Unix Admin. Horror Story Summary, version 1.0 (http://www-uxsup.csx.cam.ac.uk/misc/horror.txt) A great collection of stories, many of which will ring true with our viewers The very first one, is about a user changing root's shell to /usr/local/bin/tcsh but forgetting to make it executable, resulting in not being able to login as root. I too have run into this issue, in a slightly different way. I had tcsh as my user shell (back before tcsh was in base), and after a major OS upgrade, but before I had a chance to recompile all of my ports. Now I couldn't ssh in to the remote machine in order to recompile my shell. Now I always use a shell included in the base system, and test it before rebooting after an upgrade. “Our operations group, a VMS group but trying to learn UNIX, was assigned account administration. They were cleaning up a few non-used accounts like they do on VMS - backup and purge. When they came across the account "sccs", which had never been accessed, away it went. The "deleteuser" utility from DEC asks if you would like to delete all the files in the account. Seems reasonable, huh? Well, the home directory for "sccs" is "/". Enough said :-(“ “I was working on a line printer spooler, which lived in /etc. I wanted to remove it, and so issued the command "rm /etc/lpspl." There was only one problem. Out of habit, I typed "passwd" after "/etc/" and removed the password file. Oops.” I've done things like this as well. Finger memory can be dangerous “I was happily churning along developing something on a Sun workstation, and was getting a number of annoying permission denieds from trying to write into a directory heirarchy that I didn't own. Getting tired of that, I decided to set the permissions on that subtree to 777 while I was working, so I wouldn't have to worry about it. Someone had recently told me that rather than using plain "su", it was good to use "su -", but the implications had not yet sunk in. (You can probably see where this is going already, but I'll go to the bitter end.) Anyway, I cd'd to where I wanted to be, the top of my subtree, and did su -. Then I did chmod -R 777. I then started to wonder why it was taking so damn long when there were only about 45 files in 20 directories under where I (thought) I was. Well, needless to say, su - simulates a real login, and had put me into root's home directory, /, so I was proceeding to set file permissions for the whole system to wide open. I aborted it before it finished, realizing that something was wrong, but this took quite a while to straighten out.” Where is a ZFS snapshot when you need it? *** How individual contributors get stuck (https://medium.com/@skamille/how-do-individual-contributors-get-stuck-63102ba43516) An interesting post looking at the common causes of people getting stuck when trying to create or contribute new code Brainstorming/architecture: “I must have thought through all edge cases of all parts of everything before I can begin this project” Researching possible solutions forever (often accompanied by desire to do a “bakeoff” where they build prototypes in different platforms/languages/etc) Refactoring: “this code could be cleaner and everything would be just so much easier if we cleaned this up… and this up… and…” Helping other people instead of doing their assigned tasks (this one isn't a bad thing in an open source community) Working on side projects instead of the main project (it is your time, it is up to you how to spend it) Excessive testing (rare) Excessive automation (rare) Finish the last 10–20% of a project Start a project completely from scratch Do project planning (You need me to write what now? A roadmap?) (this is why FreeBSD has devsummits, some things you just need to whiteboard) Work with unfamiliar code/libraries/systems Work with other teams (please don't make me go sit with data engineering!!) Talk to other people Ask for help (far beyond the point they realized they were stuck and needed help) Deal with surprises or unexpected setbacks Deal with vendors/external partners Say no, because they can't seem to just say no (instead of saying no they just go into avoidance mode, or worse, always say yes) “Noticing how people get stuck is a super power, and one that many great tech leads (and yes, managers) rely on to get big things done. When you know how people get stuck, you can plan your projects to rely on people for their strengths and provide them help or even completely side-step their weaknesses. You know who is good to ask for which kinds of help, and who hates that particular challenge just as much as you do.” “The secret is that all of us get stuck and sidetracked sometimes. There's actually nothing particularly “bad” about this. Knowing the ways that you get hung up is good because you can choose to either a) get over the fears that are sticking you (lack of knowledge, skills, or confidence), b) avoid such tasks as much as possible, and/or c) be aware of your habits and use extra diligence when faced with tackling these areas.” *** Make Docs! (http://www.mkdocs.org/) “MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.” “MkDocs builds completely static HTML sites that you can host on GitHub pages, Amazon S3, or anywhere else you choose” It is an easy to install python package It includes a server mode that auto-refreshes the page as you write the docs, making it easy to preview your work before you post it online Everything needs docs, and writing docs should be as simple as possible, so that more of them will get written Go write some docs! *** Experimental FreeNAS 11/12 builds (https://forums.freenas.org/index.php?threads/new-freenas-9-10-with-freebsd-11-12-for-testing.49696/#post-341941) We know there's a lot of FreeNAS users who listen to BSDNow, so I felt it was important to share this little tidbit. I've posted something to the forums last night which includes links to brand-new spins of FreeNAS 9.10 based upon FreeBSD 11/stable and 12/current. These builds are updated nightly via our Jenkins infrastructure and hopefully will provide a new playground for technical folks and developers to experiment with FreeBSD features in their FreeNAS environment, long before they make it into a -STABLE release. As usual, the notes of caution do apply, these are nightlies, and as such bugs will abound. Do NOT use these with your production data, unless you are crazy, or just want an excuse to test your backup strategy If you do run these builds, of course feedback is welcome via the usual channels, such as the bug tracker. The hope is that by testing FreeBSD code earlier, we can vet and determine what is safe / ready to go into mainline FreeNAS sooner rather than later. *** Beastie Bits An Explainer on Unix's Most Notorious Code Comment (http://thenewstack.io/not-expected-understand-explainer/) turn your network inside out with one pf.conf trick (http://www.tedunangst.com/flak/post/turn-your-network-inside-out-with-one-pfconf-trick) A story of if_get(9) (http://www.grenadille.net/post/2017/01/13/A-story-of-if_get%289%29) Apple re-affirms its commitment to LLVM/Clang (http://lists.llvm.org/pipermail/llvm-dev/2017-January/108953.html) python 3k17 (http://www.tedunangst.com/flak/post/python-3k17) 2017 presentation proposals (http://blather.michaelwlucas.com/archives/2848) NetBSD 7.1_RC1 available (http://mail-index.netbsd.org/netbsd-announce/2017/01/09/msg000259.html) #define FSUFS2MAGIC 0x19540119 (Happy Birthday to Kirk McKusick tomorrow) *** Feedback/Questions J - LetsEncrypt (http://pastebin.com/nnQ9ZgyN) Mike - OpenRC (http://pastebin.com/EZ4tRiVb) Timothy - ZFS Horror (http://pastebin.com/ZqDFTsnR) Troels (http://pastebin.com/dhZEnREM) Jason - Disk Label (http://pastebin.com/q4F95S6h) ***

Another exciting week on BSDNow, we are queued up with LLVM / Linking news, a look at NetBSD's scheduler, This episode was brought to you by Headlines FreeBSD Kernel and World, and many Ports, can now be linked with lld (https://llvm.org/bugs/show_bug.cgi?id=23214#c40) “With this change applied I can link the entirety of the FreeBSD/amd64 base system (userland world and kernel) with LLD.” “Rafael's done an initial experimental Poudriere FreeBSD package build with lld head, and found almost 20K out of 26K ports built successfully. I'm now looking at getting CI running to test this on an ongoing basis. But, I think we're at the point where an experimental build makes sense.” Such testing will become much easier once llvm 4.0 is imported into -current “I suggest that during development we collect patches in a local git repo -- for example, I've started here for my Poudriere run https://github.com/emaste/freebsd-ports/commits/ports-lld” “It now looks like libtool is responsible for the majority of my failed / skipped ports. Unless we really think we'll add "not GNU" and other hacks to lld we're going to have to address libtool limitations upstream and in the FreeBSD tree. I did look into libtool a few weeks ago, but unfortunately haven't yet managed to produce a patch suitable for sending upstream.” If you are interested in LLVM/Clang/LLD/LLDB etc, check out: A Tourist's Guide to the LLVM Source Code (http://blog.regehr.org/archives/1453) *** Documenting NetBSD's scheduler tweaks (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20170109_2108.html) A followup to our previous coverage of improvements to the scheduler in NetBSD “NetBSD's scheduler was recently changed to better distribute load of long-running processes on multiple CPUs. So far, the associated sysctl tweaks were not documented, and this was changed now, documenting the kern.sched sysctls.” kern.sched.cacheht_time (dynamic): Cache hotness time in which a LWP is kept on one particular CPU and not moved to another CPU. This reduces the overhead of flushing and reloading caches. Defaults to 3ms. Needs to be given in ``hz'' units, see mstohz(9). kern.sched.balance_period (dynamic): Interval at which the CPU queues are checked for re-balancing. Defaults to 300ms. kern.sched.min_catch (dynamic): Minimum count of migratable (runable) threads for catching (stealing) from another CPU. Defaults to 1 but can be increased to decrease chance of thread migration between CPUs. It is important to have good documentation for these tunables, so that users can understand what it is they are adjusting *** FreeBSD Network Gateway on EdgeRouter Lite (http://codeghar.com/blog/freebsd-network-gateway-on-edgerouter-lite.html) “EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running FreeBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.” “Colin Percival has written a great blog post on the subject, titled FreeBSD on EdgeRouter Lite - no serial port required (http://www.daemonology.net/blog/2016-01-10-FreeBSD-EdgeRouter-Lite.html) . In it he provides and describes a shell script to build a bootable image of FreeBSD to be run on ERL, available from GitHub in the freebsd-ERL-build (https://github.com/cperciva/freebsd-ERL-build/) repo. I have built a Vagrant-based workflow to automate the building of the drive image. It's available on GitHub in the freebsd-edgerouterlite-ansible (https://github.com/hamzasheikh/freebsd-edgerouterlite-ansible) repo. It uses the build script Percival wrote.” “Once you've built the disk image it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive. I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I have not tried another third party drive since.” The tutorial covers all of the steps, and the configuration files, including rc.conf, IP configuration, DHCP (and v6), pf, and DNS (unbound) “I'm pretty happy with ERL and FreeBSD. There is great community documentation on how to configure all the pieces of software that make a FreeBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.” “My plan on upgrading the base OS is to get a third party USB drive that works, write a newer FreeBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.” Although something more nanobsd style with 2 partitions on the one drive might be easier. “Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well. ERL is simply a great piece of network hardware. I'm tempted to try Ubiquiti's WiFi products instead of a mixture of DD-WRT and OpenWRT devices I have now. But that is for another day and perhaps another blog post.” *** A highly portable build system targeting modern UNIX systems (https://github.com/michipili/bsdowl) An exciting new/old project is up on GitHub that we wanted to bring your attention to. BSD Owl is a highly portable build-system based around BSD Make that supports a variety of popular (and not so popular) languages, such as: C programs, compiled for several targets C libraries, static and shared, compiled for several targets Shell scripts Python scripts OCaml programs OCaml libraries, with ocamldoc documentation OCaml plugins TeX documents, prepared for several printing devices METAPOST figures, with output as PDF, PS, SVG or PNG, either as part of a TeX document or as standalone documents What about features you may ask? Well BSD Owl has plenty of those to go around: Support of compilation profiles Support of the parallel mode (at the directory level) Support of separate trees for sources and objects Support of architecture-dependant compilation options Support GNU autoconf Production of GPG-signed tarballs Developer subshell, empowered with project-specific scripts Literate programming using noweb Preprocessing with m4 As far as platform support goes, BSD Owl is tested on OSX / Debian Jesse and FreeBSD > 9. Future support for OpenBSD and NetBSD is planned, once they update their respective BSD Make binaries to more modern versions News Roundup find -delete in OpenBSD. Thanks to tedu@ OpenBSD will have this very handy flag to in the future. (https://marc.info/?l=openbsd-tech&m=148342051832692&w=2) OpenBSD's find(1) utility will now support the -delete operation “This option is not posix (not like that's stopped find accumulating a dozen extensions), but it is in gnu and freebsd (for 20 years). it's also somewhat popular among sysadmins and blogs, etc. and perhaps most importantly, it nicely solves one of the more troublesome caveats of find (which the man page actually covers twice because it's so common and easy to screw up). So I think it makes a good addition.” The actual code was borrowed from FreeBSD Using the -delete option is much more performant than forking rm once for each file, and safer because there is no risk of mangling path names If you encounter a system without a -delete option, your best bet is to use the -print0 option of find, which will print each filename terminated by a null byte, and pipe that into xargs -0 rm This avoids any ambiguity caused by files with spaces in the names *** New version of the Lumina desktop released (https://lumina-desktop.org/version-1-2-0-released/) Just in time to kickoff 2017 we have a new release of Lumina Desktop (1.2.0) Some of the notable changes include fixes to make it easier to port to other platforms, and some features: New Panel Plugins: “audioplayer” (panel version of the desktop plugin with the same name): Allows the user to load/play audio files directly through the desktop itself. “jsonmenu” (panel version of the menu plugin with the same name): Allows an external utility/script to be used to generate a menu/contents on demand. New Menu Plugins: “lockdesktop”: Menu option for instantly locking the desktop session. New Utilities: lumina-archiver: This is a pure Qt5 front-end to the “tar” utility for managing/creating archives. This can also use the dd utility to burn a “*.img” file to a USB device for booting.“ Looks like the news already made its rounds to a few different sites, with Phoronix and Softpedia picking it up as well Phoronix (http://www.phoronix.com/scan.php?page=news_item&px=Lumina-1.2-Released) Softpedia (http://news.softpedia.com/news/lumina-1-2-desktop-environments-launches-for-trueos-with-various-enhancements-511495.shtml) TrueOS users running the latest updates are already on the pre-release version of 1.2.1, so nothing has to be done there to get the latest and greatest. dd is not a disk writing tool (http://www.vidarholen.net/contents/blog/?p=479) “If you've ever used dd, you've probably used it to read or write disk images:” > # Write myfile.iso to a USB drive > dd if=myfile.iso of=/dev/sdb bs=1M “Usage of dd in this context is so pervasive that it's being hailed as the magic gatekeeper of raw devices. Want to read from a raw device? Use dd. Want to write to a raw device? Use dd. This belief can make simple tasks complicated. How do you combine dd with gzip? How do you use pv if the source is raw device? How do you dd over ssh?” “The fact of the matter is, dd is not a disk writing tool. Neither “d” is for “disk”, “drive” or “device”. It does not support “low level” reading or writing. It has no special dominion over any kind of device whatsoever.” Then a number of alternatives are discussed “However, this does not mean that dd is useless! The reason why people started using it in the first place is that it does exactly what it's told, no more and no less. If an alias specifies -a, cp might try to create a new block device rather than a copy of the file data. If using gzip without redirection, it may try to be helpful and skip the file for not being regular. Neither of them will write out a reassuring status during or after a copy.” “dd, meanwhile, has one job*: copy data from one place to another. It doesn't care about files, safeguards or user convenience. It will not try to second guess your intent, based on trailing slashes or types of files. When this is no longer a convenience, like when combining it with other tools that already read and write files, one should not feel guilty for leaving dd out entirely.” “dd is the swiss army knife of the open, read, write and seek syscalls. It's unique in its ability to issue seeks and reads of specific lengths, which enables a whole world of shell scripts that have no business being shell scripts. Want to simulate a lseek+execve? Use dd! Want to open a file with O_SYNC? Use dd! Want to read groups of three byte pixels from a PPM file? Use dd!” “It's a flexible, unique and useful tool, and I love it. My only issue is that, far too often, this great tool is being relegated to and inappropriately hailed for its most generic and least interesting capability: simply copying a file from start to finish.” “dd actually has two jobs: Convert and Copy. Legend has it that the intended name, “cc”, was taken by the C compiler, so the letters were shifted by one to give “dd”. This is also why we ended up with a Window system called X.” dd countdown (https://eriknstr.github.io/utils/dd-countdown.htm) *** Bhyve setup for tcp testing (https://www.strugglingcoder.info/index.php/bhyve-setup-for-tcp-testing/) FreeBSD Developer Hiren Panchasara writes about his setup to use bhyve to test changes to the TCP stack in FreeBSD “Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I've already wrote down how I do dummynet (https://www.strugglingcoder.info/index.php/drop-a-packet/) so I'll focus on bhyve part.” “A few months back when I started looking into improving FreeBSD TCP's response to packet loss, I looked around for traffic simulators which can do deterministic packet drop for me.” “I had used dummynet(4) before so I thought of using it but the problem is that it only provided probabilistic drops. You can specify dropping 10% of the total packets” So he wrote a quick hack, hopefully he'll polish it up and get it committed “Setup: I'll create 3 bhyve guests: client, router and server” “Both client and server need their routing tables setup correctly so that they can reach each other. The Dummynet node is the router / traffic shaping node here. We need to enable forwarding between interfaces: sysctl net.inet.ip.forwarding=1” “We need to setup links (called ‘pipes') and their parameters on dummynet node” “For simulations, I run a lighttpd web-server on the server which serves different sized objects and I request them via curl or wget from the client. I have tcpdump running on any/all of four interfaces involved to observe traffic and I can see specified packets getting dropped by dummynet. sysctl net.inet.ip.dummynet.iopktdrop is incremented with each packet that dummynet drops.” “Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.” Create 2 tap interfaces for each end point, and 3 from the router. One each for SSH/control, and the others for the test flows. Then create 3 bridges, the first includes all of the control tap interfaces, and your hosts' real interface. This allows the guests to reach the internet to download packages etc. The other two bridges form the connections between the three VMs The creation and configuration of the VMs is documented in detail I used a setup very similar to this for teaching the basics of how TCP works when I was teaching at a local community college *** Beastie Bits Plan9 on Bhyve (https://twitter.com/pr1ntf/status/817895393824382976) Get your name in the relayd book (http://blather.michaelwlucas.com/archives/2832) Ted Unangst's 2016 Computer Reviews (http://www.tedunangst.com/flak/post/2016-computer-review) Bryan Cantrill on Developer On Fire podcast (http://developeronfire.com/episode-198-bryan-cantrill-persistence-and-action) 2016 in review: pf/ipfw's impact on forwarding performance over time, on 8 core Atom (http://dev.bsdrp.net/benchs/2016.SM5018A-FTN4-Chelsio.png) #Wayland Weston with X and EGL clients, running on #FreeBSD in VBox with new scfb backend. More coming soon! (https://twitter.com/johalun/status/819039940914778112) Feedback/Questions Eddy - TRIM Partitioning (http://pastebin.com/A0LSipCj) Matt - Why FreeBSD? (http://pastebin.com/UE1k4Q99) Shawn - ZFS Horror? (http://pastebin.com/TjTkqHA4) Andrew - Bootloaders (http://pastebin.com/Baxd6Pjy) GELIBoot Paper (http://allanjude.com/talks/AsiaBSDCon2016_geliboot_pdf1a.pdf) FreeBSD Architecture Handbook (https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/boot.html) Bryan - ZFS Error (http://pastebin.com/NygwchFD) ***

This week on BSDNow, we've got all sorts of post-holiday goodies to share. New OpenSSL APIs, Dtrace, OpenBSD This episode was brought to you by Headlines OpenSSL 1.1 API migration path, or the lack thereof (https://www.mail-archive.com/tech@openbsd.org/msg36437.html) As many of you will already be aware, the OpenSSL 1.1.0 release intentionally introduced significant API changes from the previous release. In summary, a large number of data structures that were previously publically visible have been made opaque, with accessor functions being added in order to get and set some of the fields within these now opaque structs. It is worth noting that the use of opaque data structures is generally beneficial for libraries, since changes can be made to these data structures without breaking the ABI. As such, the overall direction of these changes is largely reasonable. However, while API change is generally necessary for progression, in this case it would appear that there is NO transition plan and a complete disregard for the impact that these changes would have on the overall open source ecosystem. So far it seems that the only approach is to place the migration burden onto each and every software project that uses OpenSSL, pushing significant code changes to each project that migrates to OpenSSL 1.1, while maintaining compatibility with the previous API. This is forcing each project to provide their own backwards compatibility shims, which is practically guaranteeing that there will be a proliferation of variable quality implementations; it is almost a certainty that some of these will contain bugs, potentially introducing security issues or memory leaks. I think this will be a bigger issue for other operating systems that do not have the flexibility of the ports tree to deliver a newer version of OpenSSL. If a project switches from the old API to the new API, and the OS only provides the older branch of OpenSSL, how can the application work? Of course, this leaves the issue, if application A wants OpenSSL 1.0, and application B only works with OpenSSL 1.1, how does that work? Due to a number of factors, software projects that make use of OpenSSL cannot simply migrate to the 1.1 API and drop support for the 1.0 API - in most cases they will need to continue to support both. Firstly, I am not aware of any platform that has shipped a production release with OpenSSL 1.1 - any software that supported OpenSSL 1.1 only, would effectively be unusable on every platform for the time being. Secondly, the OpenSSL 1.0.2 release is supported until the 31st of December 2019, while OpenSSL 1.1.0 is only supported until the 31st of August 2018 - any LTS style release is clearly going to consider shipping with 1.0.2 as a result. Platforms that are attempting to ship with OpenSSL 1.1 are already encountering significant challenges - for example, Debian currently has 257 packages (out of 518) that do not build against OpenSSL 1.1. There are also hidden gotchas for situations where different libraries are linked against different OpenSSL versions and then share OpenSSL data structures between them - many of these problems will be difficult to detect since they only fail at runtime. It will be interesting to see what happens with OpenSSL, and LibreSSL Hopefully, most projects will decide to switch to the cleaner APIs provided by s2n or libtls, although they do not provide the entire functionality of the OpenSSL API. Hacker News comments (https://news.ycombinator.com/item?id=13284648) *** exfiltration via receive timing (http://www.tedunangst.com/flak/post/exfiltration-via-receive-timing) Another similar way to create a backchannel but without transmitting anything is to introduce delays in the receiver and measure throughput as observed by the sender. All we need is a protocol with transmission control. Hmmm. Actually, it's easier (and more reliable) to code this up using a plain pipe, but the same principle applies to networked transmissions. For every digit we want to “send” back, we sleep a few seconds, then drain the pipe. We don't care about the data, although if this were a video file or an OS update, we could probably do something useful with it. Continuously fill the pipe with junk data. If (when) we block, calculate the difference between before and after. This is a our secret backchannel data. (The reader and writer use different buffer sizes because on OpenBSD at least, a writer will stay blocked even after a read depending on the space that opens up. Even simple demos have real world considerations.) In this simple example, the secret data (argv) is shared by the processes, but we can see that the writer isn't printing them from its own address space. Nevertheless, it works. Time to add random delays and buffering to firewalls? Probably not. An interesting thought experiment that shows just how many ways there are to covertly convey a message *** OpenBSD Desktop in about 30 Minutes (https://news.ycombinator.com/item?id=13223351) Over at hackernews we have a very non-verbose, but handy guide to getting to a OpenBSD desktop in about 30 minutes! First, the guide will assume you've already installed OpenBSD 6.0, so you'll need to at least be at the shell prompt of your freshly installed system to begin. With that, now its time to do some tuning. Editing some resource limits in login.conf will be our initial task, upping some datasize tunables to 2GB Next up, we will edit some of the default “doas” settings to something a bit more workable for desktop computing Another handy trick, editing your .profile to have your PKG_PATH variables set automatically will make One thing some folks may overlook, but disabling atime can speed disk performance (which you probably don't care about atime on your desktop anyway), so this guide will show you what knobs to tweak in /etc/fstab to do so After some final WPA / Wifi configuration, we then drop to “mere mortal” mode and begin our package installations. In this particular guide, he will be setting up Lumina Desktop (Which yes, it is on OpenBSD) A few small tweaks later for xscreensaver and your xinitrc file, then you are ready to run “startx” and begin your desktop session! All in all, great guide which if you are fast can probably be done in even less than 30 minutes and will result in a rock-solid OpenBSD desktop rocking Lumina none-the-less. *** How DTrace saved Christmas (https://hackernoon.com/dtrace-at-home-145ba773371e) Adam Leventhal, one of the co-creators of DTrace, wrote up this post about how he uses DTrace at home, to save Christmas I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I'm glad I don't have to do 2016 again. It was just before midnight and either I'd make the card that night or leave an empty space on our friends' refrigerators. Adobe Illustrator had other ideas: “Unable to set maximum number of files to be opened” I'm not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions were working for me, and — inspired by Sara Mauskopf's excellent post (https://medium.com/startup-grind/how-to-start-a-company-with-no-free-time-b70fbe7b918a#.uujdblxc6) — I was rapidly running out of the time bounds for the project. Enough; I'd just DTrace it. A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can't imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I'm forever grateful to the Apple team that ported it to Mac OS X) Illustrator is failing on setrlimit(2) and blowing up as result. Let's confirm that it is in fact returning -1:$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }' dtrace: description 'syscall::setrlimit:return' matched 1 probe CPU ID FUNCTION:NAME 0 532 setrlimit:return -1 1 There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let's confirm that it is in fact setting the files limit and check the value to which it's being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process's address space: $ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlimcur, this->r.rlimmax); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe CPU ID FUNCTION:NAME 0 531 setrlimit:entry 1008 2800 7fffffffffffffff Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMITNOFILE | _RLIMITPOSIX_FLAG) The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy: $ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlimt *)alloca(sizeof (rlimt)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlimt), sizeof (rlimt)); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe dtrace: could not enable tracing: Permission denied Oh right. Thank you SIP (System Integrity Protection). This is a new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it's new I haven't yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It's easy enough to disable, but I'm reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb + After using DTrace to get the address of the setrlimit function, Adam used lldb to change the result before it got back to the application: (lldb) break set -n _init Breakpoint 1: 47 locations. (lldb) run … (lldb) di -s 0x1006e5b72 -c 1 0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit (lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90 (lldb) di -s 0x1006e5b72 -c 4 0x1006e5b72: xorl %eax, %eax 0x1006e5b74: nop 0x1006e5b75: nop 0x1006e5b76: nop Next I just did a process detach and got on with making that holiday card… DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn't write a program doesn't mean you can't fix it. News Roundup Say my Blog's name! (https://functionallyparanoid.com/2016/12/22/say-my-blogs-name/) Brian Everly over at functionally paranoid has a treat for us today. Let us give you a moment to get the tin-foil hats on… Ok, done? Let's begin! He starts off with a look at physical security. He begins by listing your options: BIOS passwords – Not something I'm typically impressed with. Most can be avoided by opening up the machine, closing a jumper and powering it up to reset the NVRAM to factory defaults. I don't even bother with them. Full disk encryption – This one really rings my bell in a positive way. If you can kill power to the box (either because the bad actor has to physically steal it and they aren't carrying around a pile of car batteries and an inverter or because you can interrupt power to it some other way), then the disk will be encrypted. The other beauty of this is that if a drive fails (and they all do eventually) you don't have to have any privacy concerns about chucking it into an electronics recycler (or if you are a bad, bad person, into a landfill) because that data is effectively gibberish without the key (or without a long time to brute force it). Two factor auth for logins – I like this one as well. I'm not a fan of biometrics because if your fingerprint is compromised (yes, it can happen – read (https://www.washingtonpost.com/news/federal-eye/wp/2015/07/09/hack-of-security-clearance-system-affected-21-5-million-people-federal-authorities-say/) about the department of defense background checks that were extracted by a bad agent – they included fingerprint images) you can't exactly send off for a new finger. Things like the YubiKey (https://www.yubico.com/) are pretty slick. They require that you have the physical hardware key as well as the password so unless the bad actor lifted your physical key, they would have a much harder time with physical access to your hardware. Out of those options, Brian mentions that he uses disk encryption and yubi-key for all his secure network systems. Next up is network segmentation, in this case the first thing to do is change your admin password for any ISP supplied modem / router. He goes on to scare us of javascript attacks being used not against your local machine, but instead non WAN exposed router admin interface. Scary Stuff! For added security, naturally he firewalls the router by plugging in the LAN port to a OpenBSD box which does the 2nd layer of firewall / router protection. What about privacy and browsing? Here's some more of his tips: I use Unbound as my DNS resolver on my local network (with all UDP port 53 traffic redirected to it by pf so I don't have to configure anything on the clients) and then forward the traffic to DNSCrypt Proxy, caching the results in Unbound. I notice ZERO performance penalty for this and it greatly enhances privacy. This combination of Unbound and DNSCrypt Proxy works very well together. You can even have redundancy by having multiple upstream resolvers running on different ports (basically run the DNSCrypt Proxy daemon multiple times pointing to different public resolvers). I also use Firefox exclusively for my web browsing. By leveraging the tips on this page (https://www.privacytools.io/), you can lock it down to do a great job of privacy protection. The fact that your laptop's battery drain rate can be used to fingerprint your browser completely trips me out but hey – that's the world we live in.' What about the cloud you may ask? Well Brian has a nice solution for that as well: I recently decided I would try to live a cloud-free life and I'll give you a bit of a synopsis on it. I discovered a wonderful Open Source project called FreeNAS (http://www.freenas.org/). What this little gem does is allow you to install a FreeBSD/zfs file server appliance on amd64 hardware and have a slick administrative web interface for managing it. I picked up a nice SuperMicro motherboard and chassis that has 4 hot swap drive bays (and two internal bays that I used to mirror the boot volume on) and am rocking the zfs lifestyle! (Thanks Alan Jude!) One of the nicest features of the FreeNAS is that it provides the ability to leverage the FreeBSD jail functionality in an easy to use way. It also has plugins but the security on those is a bit sketchy (old versions of libraries, etc.) so I decided to roll my own. I created two jails – one to run OwnCloud (yeah, I know about NextCloud and might switch at some point) and the other to run a full SMTP/IMAP email server stack. I used Lets Encrypt (https://letsencrypt.org/) to generate the SSL certificates and made sure I hit an A on SSLLabs (https://www.ssllabs.com/) before I did anything else. His post then goes in to talk about Backups and IoT devices, something else you need to consider in this truely paranoid world we are forced to live in. We even get a nice shout-out near the end! Enter TarSnap (http://www.tarsnap.com/) – a company that advertises itself as “Online Backups for the Truly Paranoid”. It brings a tear to my eye – a kindred spirit! :-) Thanks again to Alan Jude and Kris Moore from the BSD Now podcast (http://www.bsdnow.tv/) for turning me onto this company. It has a very easy command syntax (yes, it isn't a GUI tool – suck it up buttercup, you wanted to learn the shell didn't you?) and even allows you to compile the thing from source if you want to.” We've only covered some of the highlights here, but you really should take a few moments of your time today and read this top to bottom. Lots of good tips here, already thinking how I can secure my home network better. The open source book: “Producing Open Source Software” (http://producingoss.com/en/producingoss.pdf) “How to Run a Successful Free Software Project” by Karl Fogel 9 chapters and over 200 pages of content, plus many appendices Some interesting topics include: Choosing a good name version control bug tracking creating developer guidelines setting up communications channels choosing a license (although this guide leans heavily towards the GPL) setting the tone of the project joining or creating a Non-Profit Organization the economics of open source release engineering, packaging, nightly builds, etc how to deal with forks A lot of good information packaged into this ebook This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License *** DTrace Flamegraphs for node.js on FreeBSD (http://www.venshare.com/dtrace-flamegraphs-for-freebsd-and-node-js-2/) One of the coolest tools built on top of DTrace is flamegraphs They are a very accurate, and visual way to see where a program is spending its time, which can tell you why it is slow, or where it could be improved. Further enhancements include off-cpu flame graphs, which tell you when the program is doing nothing, which can also be very useful > Recently BSD UNIXes are being acknowledged by the application development community as an interesting operating system to deploy to. This is not surprising given that FreeBSD had jails, the original container system, about 17 years ago and a lot of network focused businesses such as netflix see it as the best way to deliver content. This developer interest has led to hosting providers supporting FreeBSD. e.g. Amazon, Azure, Joyent and you can get a 2 months free instance at Digital Ocean. DTrace is another vital feature for anyone who has had to deal with production issues and has been in FreeBSD since version 9. As of FreeBSD 11 the operating system now contains some great work by Fedor Indutny so you can profile node applications and create flamegraphs of node.js processes without any additional runtime flags or restarting of processes. This is one of the most important things about DTrace. Many applications include some debugging functionality, but they require that you stop the application, and start it again in debugging mode. Some even require that you recompile the application in debugging mode. Being able to attach DTrace to an application, while it is under load, while the problem is actively happening, can be critical to figuring out what is going on. In order to configure your FreeBSD instance to utilize this feature make the following changes to the configuration of the server. Load the DTrace module at boot Increase some DTrace limits Install node with the optional DTrace feature compiled in Follow the generic node.js flamegraph tutorial (https://nodejs.org/en/blog/uncategorized/profiling-node-js/) > I hope you find this article useful. The ability to look at a runtime in this manor has saved me twice this year and I hope it will save you in the future too. My next post on freeBSD and node.js will be looking at some scenarios on utilising the ZFS features. Also check out Brendan Gregg's ACM Queue Article (http://queue.acm.org/detail.cfm?id=2927301) “The Flame Graph: This visualization of software execution is a new necessity for performance profiling and debugging” SSHGuard 2.0 Call for Testing (https://sourceforge.net/p/sshguard/mailman/message/35580961/) SSHGuard is a tool for monitoring brute force attempts and blocking them It has been a favourite of mine for a while because it runs as a pipe from syslogd, rather than reading the log files from the disk A lot of work to get SSHGuard working with new log sources (journalctl, macOS log) and backends (firewalld, ipset) has happened in 2.0. The new version also uses a configuration file. Most importantly, SSHGuard has been split into several processes piped into one another (sshg-logmon | sshg-parser | sshg-blocker | sshg-fw). sshg-parser can run with capsicum(4) and pledge(2). sshg-blocker can be sandboxed in its default configuration (without pid file, whitelist, blacklisting) and has not been tested sandboxed in other configurations. Breaking the processes up so that the sensitive bits can be sandboxes is very nice to see *** Beastie Bits pjd's 2007 paper from AsiaBSDCon: “Porting the ZFS file system to the FreeBSD operating system” (https://2007.asiabsdcon.org/papers/P16-paper.pdf) A Message From the FreeBSD Foundation (https://vimeo.com/user60888329) Remembering Roger Faulkner, Unix Champion (http://thenewstack.io/remembering-roger-faulkner/) and A few HN comments (including Bryan Cantrill) (https://news.ycombinator.com/item?id=13293596) Feedback/Questions Peter - TrueOS Network (http://pastebin.com/QtyJeHMk) Chris - Remote Desktop (http://pastebin.com/ru726VTV) Goetz - Geli on Serial (http://pastebin.com/LQZPgF5g) Joe - BGP (http://pastebin.com/jFeL8zKX) Alejandro - BSD Router (http://pastebin.com/Xq9cbmfn) ***

A look back at 2016 This episode was brought to you by LinksZFS in the trenches | BSD Now 123One small step for DRM, one giant leap for BSD | BSD Now 143The Laporte has landed! | BSD Now 152Ham, Radio & Pie, Oh My! | BSD Now 158The Foundation of NetBSD | BSD Now 162Return of the Cantrill | BSD Now 163

This week on the show, we've got some great stories to bring you, a look at the odder side of UNIX history This episode was brought to you by Headlines syspatch in testing state (http://marc.info/?l=openbsd-tech&m=148058309126053&w=2) Antoine Jacoutot ajacoutot@ openbsd has posted a call for testing for OpenBSD's new syspatch tool “syspatch(8), a "binary" patch system for -release is now ready for early testing. This does not use binary diffing to update the system, but regular signed tarballs containing the updated files (ala installer).” “I would appreciate feedback on the tool. But please send it directly to me, there's no need to pollute the list. This is obviously WIP and the tool may or may not change in drastic ways.” “These test binary patches are not endorsed by the OpenBSD project and should not be trusted, I am only providing them to get early feedback on the tool. If all goes as planned, I am hoping that syspatch will make it into the 6.1 release; but for it to happen, I need to know how it breaks your systems :-)” Instructions (http://syspatch.openbsd.org/pub/OpenBSD/6.0/syspatch/amd64/README.txt) If you test it, report back and let us know how it went *** Weston working (https://lists.freebsd.org/pipermail/freebsd-current/2016-December/064198.html) Over the past few years we've had some user-interest in the state of Wayland / Weston on FreeBSD. In the past day or so, Johannes Lundberg has sent in a progress report to the FreeBSD mailing lists. Without further ADO: We had some progress with Wayland that we'd like to share. Wayland (v1.12.0) Working Weston (v1.12.0) Working (Porting WIP) Weston-clients (installed with wayland/weston port) Working XWayland (run X11 apps in Wayland compositor) Works (maximized window only) if started manually but not when launching X11 app from Weston. Most likely problem with Weston IPC. Sway (i3-compatible Wayland compositor) Working SDL20 (Wayland backend) games/stonesoup-sdl briefly tested. https://twitter.com/johalun/status/811334203358867456 GDM (with Wayland) Halted - depends on logind. GTK3 gtk3-demo runs fine on Weston (might have to set GDK_BACKEND=wayland first. GTK3 apps working (gedit, gnumeric, xfce4-terminal tested, xfce desktop (4.12) does not yet support GTK3)“ Johannes goes on to give instructions on how / where you can fetch their WiP and do your own testing. At the moment you'll need Matt Macy's newer Intel video work, as well as their ports tree which includes all the necessary software bits. Before anybody asks, yes we are watching this for TrueOS! *** Where the rubber meets the road (part two) (https://functionallyparanoid.com/2016/12/15/where-the-rubber-meets-the-road-part-two/) Continuing with our story from Brian Everly from a week ago, we have an update today on the process to dual-boot OpenBSD with Arch Linux. As we last left off, Arch was up and running on the laptop, but some quirks in the hardware meant OpenBSD would take a bit longer. With those issues resolved and the HD seen again, the next issue that reared its head was OpenBSD not seeing the partition tables on the disk. After much frustration, it was time to nuke and pave, starting with OpenBSD first this time. After a successful GPT partitioning and install of OpenBSD, he went back to installing Arch, and then the story got more interesting. “I installed Arch as I detailed in my last post; however, when I fired up gdisk I got a weird error message: “Warning! Disk size is smaller than the main header indicates! Loading secondary header from the last sector of the disk! You should use ‘v' to verify disk integrity, and perhaps options on the expert's menu to repair the disk.” Immediately after this, I saw a second warning: “Caution: Invalid backup GPT header, but valid main header; regenerating backup header from main header.” And, not to be outdone, there was a third: “Warning! Main and backup partition tables differ! Use the ‘c' and ‘e' options on the recovery & transformation menu to examine the two tables.” Finally (not kidding), there was a fourth: “Warning! One or more CRCs don't match. You should repair the disk!” Given all of that, I thought to myself, “This is probably why I couldn't see the disk properly when I partitioned it under Linux on the OpenBSD side. I'll let it repair things and I should be good to go.” I then followed the recommendation and repaired things, using the primary GPT table to recreate the backup one. I then installed Arch and figured I was good to go.“ After confirming through several additional re-installs that the behavior was reproducible, he then decided to go full on crazy,and partition with MBR. That in and of itself was a challenge, since as he mentions, not many people dual-boot OpenBSD with Linux on MBR, especially using luks and lvm! If you want to see the details on how that was done, check it out. The story ends in success though! And better yet: “Now that I have everything working, I'll restore my config and data to Arch, configure OpenBSD the way I like it and get moving. I'll take some time and drop a note on the tech@ mailing list for OpenBSD to see if they can figure out what the GPT problem was I was running into. Hopefully it will make that part of the code stronger to get an edge-case bug report like this.” Take note here, if you run into issues like this with any OS, be sure to document in detail what happened so developers can explore solutions to the issue. *** FreeBSD and ZFS as a time capsule for OS X (https://blog.feld.me/posts/2016/12/using-freebsd-as-a-time-capsule-for-osx/) Do you have any Apple users in your life? Perhaps you run FreeBSD for ZFS somewhere else in the house or office. Well today we have a blog post from Mark Felder which shows how you can use FreeBSD as a time-capsule for your OSX systems. The setup is quite simple, to get started you'll need packages for netatalk3 and avahi-app for service discovery. Next up will be your AFP configuration. He helpfully provides a nice example that you should be able to just cut-n-paste. Be sure to check the hosts allow lines and adjust to fit your network. Also of note will be the backup location and valid users to adjust. A little easier should be the avahi setup, which can be a straight copy-n-paste from the site, which will perform the service advertisements. The final piece is just enabling specific services in /etc/rc.conf and either starting them by hand, or rebooting. At this point your OSX systems should be able to discover the new time-capsule provider on the network and DTRT. *** News Roundup netbenches - FreeBSD network forwarding performance benchmark results (https://github.com/ocochard/netbenches) Olivier Cochard-Labbé, original creator of FreeNAS, and leader of the BSD Router Project, has a github repo of network benchmarks There are many interesting results, and all of the scripts, documentation, and configuration files to run the tests yourself IPSec Performance on an Atom C2558, 12-head vs IPSec Performance Branch (https://github.com/ocochard/netbenches/tree/master/Atom_C2558_4Cores-Intel_i350/ipsec/results/fbsd12.projects-ipsec.equilibrium) Compared to: Xeon L5630 2.13GHz (https://github.com/ocochard/netbenches/tree/2f3bb1b3c51e454736f1fcc650c3328071834f8d/Xeon_L5630-4Cores-Intel_82599EB/ipsec/results/fbsd11.0) and IPSec with Authentication (https://github.com/ocochard/netbenches/tree/305235114ba8a3748ad9681c629333f87f82613a/Atom_C2558_4Cores-Intel_i350/ipsec.ah/results/fbsd12.projects-ipsec.equilibrium) I look forward to seeing tests on even more hardware, as people with access to different configurations try out these benchmarks *** A tcpdump Tutorial and Primer with Examples (https://danielmiessler.com/study/tcpdump/) Most users will be familiar with the basics of using tcpdump, but this tutorial/primer is likely to fill in a lot of blanks, and advance many users understanding of tcpdump “tcpdump is the premier network analysis tool for information security professionals. Having a solid grasp of this über-powerful application is mandatory for anyone desiring a thorough understanding of TCP/IP. Many prefer to use higher level analysis tools such as Wireshark, but I believe this to usually be a mistake.” tcpdump is an important tool for any system or network administrator, it is not just for security. It is often the best way to figure out why the network is not behaving as expected. “In a discipline so dependent on a true understanding of concepts vs. rote learning, it's important to stay fluent in the underlying mechanics of the TCP/IP suite. A thorough grasp of these protocols allows one to troubleshoot at a level far beyond the average analyst, but mastery of the protocols is only possible through continued exposure to them.” Not just that, but TCP/IP is a very interesting protocol, considering how little it has changed in its 40+ year history “First off, I like to add a few options to the tcpdump command itself, depending on what I'm looking at. The first of these is -n, which requests that names are not resolved, resulting in the IPs themselves always being displayed. The second is -X, which displays both hex and ascii content within the packet.” “It's also important to note that tcpdump only takes the first 96 bytes of data from a packet by default. If you would like to look at more, add the -s number option to the mix, where number is the number of bytes you want to capture. I recommend using 0 (zero) for a snaplength, which gets everything.” The page has a nice table of the most useful options It also has a great primer on doing basic filtering If you are relatively new to using tcpdump, I highly recommend you spend a few minutes reading through this article *** How Unix made it to the top (http://minnie.tuhs.org/pipermail/tuhs/2016-December/007519.html) Doug McIlroy gives us a nice background post on how “Unix made it to the top” It's fairly short / concise, so I felt it would be good to read in its entirety. “It has often been told how the Bell Labs law department became the first non-research department to use Unix, displacing a newly acquired stand-alone word-processing system that fell short of the department's hopes because it couldn't number the lines on patent applications, as USPTO required. When Joe Ossanna heard of this, he told them about roff and promised to give it line-numbering capability the next day. They tried it and were hooked. Patent secretaries became remote members of the fellowship of the Unix lab. In due time the law department got its own machine. Less well known is how Unix made it into the head office of AT&T. It seems that the CEO, Charlie Brown, did not like to be seen wearing glasses when he read speeches. Somehow his PR assistant learned of the CAT phototypesetter in the Unix lab and asked whether it might be possible to use it to produce scripts in large type. Of course it was. As connections to the top never hurt, the CEO's office was welcomed as another ouside user. The cost--occasionally having to develop film for the final copy of a speech--was not onerous. Having teethed on speeches, the head office realized that Unix could also be useful for things that didn't need phototypesetting. Other documents began to accumulate in their directory. By the time we became aware of it, the hoard came to include minutes of AT&T board meetings. It didn't seem like a very good idea for us to be keeping records from the inner sanctum of the corporation on a computer where most everybody had super-user privileges. A call to the PR guy convinced him of the wisdom of keeping such things on their own premises. And so the CEO's office bought a Unix system. Just as one hears of cars chosen for their cupholders, so were theseusers converted to Unix for trivial reasons: line numbers and vanity.“ Odd Comments and Strange Doings in Unix (http://orkinos.cmpe.boun.edu.tr/~kosar/odd.html) Everybody loves easter-eggs, and today we have some fun odd ones from the history throughout UNIX told by Dennis Ritchie. First up, was a fun one where the “mv” command could sometimes print the following “values of b may give rise to dom!” “Like most of the messages recorded in these compilations, this one was produced in some situation that we considered unlikely or as result of abuse; the details don't matter. I'm recording why the phrase was selected. The very first use of Unix in the "real business" of Bell Labs was to type and produce patent applications, and for a while in the early 1970s we had three typists busily typing away in the grotty lab on the sixth floor. One day someone came in and observed on the paper sticking out of one of the Teletypes, displayed in magnificent isolation, this ominous phrase: values of b may give rise to dom! It was of course obvious that the typist had interrupted a printout (generating the "!" from the ed editor) and moved up the paper, and that the context must have been something like "varying values of beta may give rise to domain wall movement" or some other fragment of a physically plausible patent application.But the phrase itself was just so striking! Utterly meaningless, but it looks like what... a warning? What is "dom?" At the same time, we were experimenting with text-to-voice software by Doug McIlroy and others, and of course the phrase was tried out with it. For whatever reason, its rendition of "give rise to dom!" accented the last word in a way that emphasized the phonetic similarity between "doom" and the first syllable of "dominance." It pronounced "beta" in the British style, "beeta." The entire occurrence became a small, shared treasure.The phrase had to be recorded somewhere, and it was, in the v6 source. Most likely it was Bob Morris who did the deed, but it could just as easily have been Ken. I hope that your browser reproduces the b as a Greek beta.“ Next up is one you might have heard before: /* You are not expected to understand this */> Every now and then on Usenet or elsewhere I run across a reference to a certain comment in the source code of the Sixth Edition Unix operating system. I've even been given two sweatshirts that quote it. Most probably just heard about it, but those who saw it in the flesh either had Sixth Edition Unix (ca. 1975) or read the annotated version of this system by John Lions (which was republished in 1996: ISBN 1-57298-013-7, Peer-to-Peer Communications).It's often quoted as a slur on the quantity or quality of the comments in the Bell Labs research releases of Unix. Not an unfair observation in general, I fear, but in this case unjustified. So we tried to explain what was going on. "You are not expected to understand this" was intended as a remark in the spirit of "This won't be on the exam," rather than as an impudent challenge. There's a few other interesting stories as well, if the odd/fun side of UNIX history at all interests you, I would recommend checking it out. Beastie Bits With patches in review the #FreeBSD base system builds 100% reproducibly (https://twitter.com/ed_maste/status/811289279611682816) BSDCan 2017 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/bsdcan-2017/) ioCell 2.0 released (https://github.com/bartekrutkowski/iocell/releases) who even calls link_ntoa? (http://www.tedunangst.com/flak/post/who-even-calls-link-ntoa) Booting Androidx86 under bhyve (https://twitter.com/pr1ntf/status/809528845673996288) Feedback/Questions Chris - VNET (http://pastebin.com/016BfvU9) Brian - Package Base (http://pastebin.com/8JJeHuRT) Wim - TrueOS Desktop All-n-one (http://pastebin.com/VC0DPQUF) Daniel - Long Boots (http://pastebin.com/q7pFu7pR) Bryan - ZFS / FreeNAS (http://pastebin.com/xgUnbzr7) Bryan - FreeNAS Security (http://pastebin.com/qqCvVTLB) ***

This week on BSDNow, we have a very special guest joining us to tell us a tale of the early days in BSD history. That plus some new OpenSSH goodness, shell scripting utilities and much more. Stay tuned for your place to B...SD! This episode was brought to you by Headlines Call For Testing: OpenSSH 7.4 (http://marc.info/?l=openssh-unix-dev&m=148167688911316&w=2) Getting ready to head into the holidays for for the end of 2016 means some of us will have spare time on our hands. What a perfect time to get some call for testing work done! Damien Miller has issued a public CFT for the upcoming OpenSSH 7.4 release, which considering how much we all rely on SSH I would expect will get some eager volunteers for testing. What are some of the potential breakers? “* This release removes server support for the SSH v.1 protocol. ssh(1): Remove 3des-cbc from the client's default proposal. 64-bit block ciphers are not safe in 2016 and we don't want to wait until attacks like SWEET32 are extended to SSH. As 3des-cbc was the only mandatory cipher in the SSH RFCs, this may cause problems connecting to older devices using the default configuration, but it's highly likely that such devices already need explicit configuration for key exchange and hostkey algorithms already anyway. sshd(8): Remove support for pre-authentication compression. Doing compression early in the protocol probably seemed reasonable in the 1990s, but today it's clearly a bad idea in terms of both cryptography (cf. multiple compression oracle attacks in TLS) and attack surface. Pre-auth compression support has been disabled by default for >10 years. Support remains in the client. ssh-agent will refuse to load PKCS#11 modules outside a whitelist of trusted paths by default. The path whitelist may be specified at run-time. sshd(8): When a forced-command appears in both a certificate and an authorized keys/principals command= restriction, sshd will now refuse to accept the certificate unless they are identical. The previous (documented) behaviour of having the certificate forced-command override the other could be a bit confusing and error-prone. sshd(8): Remove the UseLogin configuration directive and support for having /bin/login manage login sessions.“ What about new features? 7.4 has some of those to wake you up also: “* ssh(1): Add a proxy multiplexing mode to ssh(1) inspired by the version in PuTTY by Simon Tatham. This allows a multiplexing client to communicate with the master process using a subset of the SSH packet and channels protocol over a Unix-domain socket, with the main process acting as a proxy that translates channel IDs, etc. This allows multiplexing mode to run on systems that lack file- descriptor passing (used by current multiplexing code) and potentially, in conjunction with Unix-domain socket forwarding, with the client and multiplexing master process on different machines. Multiplexing proxy mode may be invoked using "ssh -O proxy ..." sshd(8): Add a sshdconfig DisableForwaring option that disables X11, agent, TCP, tunnel and Unix domain socket forwarding, as well as anything else we might implement in the future. Like the 'restrict' authorizedkeys flag, this is intended to be a simple and future-proof way of restricting an account. sshd(8), ssh(1): Support the "curve25519-sha256" key exchange method. This is identical to the currently-support method named "curve25519-sha256@libssh.org". sshd(8): Improve handling of SIGHUP by checking to see if sshd is already daemonised at startup and skipping the call to daemon(3) if it is. This ensures that a SIGHUP restart of sshd(8) will retain the same process-ID as the initial execution. sshd(8) will also now unlink the PidFile prior to SIGHUP restart and re-create it after a successful restart, rather than leaving a stale file in the case of a configuration error. bz#2641 sshd(8): Allow ClientAliveInterval and ClientAliveCountMax directives to appear in sshd_config Match blocks. sshd(8): Add %-escapes to AuthorizedPrincipalsCommand to match those supported by AuthorizedKeysCommand (key, key type, fingerprint, etc.) and a few more to provide access to the contents of the certificate being offered. Added regression tests for string matching, address matching and string sanitisation functions. Improved the key exchange fuzzer harness.“ Get those tests done and be sure to send feedback, both positive and negative. *** How My Printer Caused Excessive Syscalls & UDP Traffic (https://zinascii.com/2014/how-my-printer-caused-excessive-syscalls.html) “3,000 syscalls a second, on an idle machine? That doesn't seem right. I just booted this machine. The only processes running are those required to boot the SmartOS Global Zone, which is minimal.” This is a story from 2014, about debugging a machine that was being slowed down by excessive syscalls and UDP traffic. It is also an excellent walkthrough of the basics of DTrace “Well, at least I have DTrace. I can use this one-liner to figure out what syscalls are being made across the entire system.” dtrace -n 'syscall:::entry { @[probefunc,probename] = count(); }' “Wow! That is a lot of lwpsigmask calls. Now that I know what is being called, it's time to find out who is doing the calling? I'll use another one-liner to show me the most common user stacks invoking lwpsigmask.” dtrace -n 'syscall::lwp_sigmask:entry { @[ustack()] = count(); }' “Okay, so this mdnsd code is causing all the trouble. What is the distribution of syscalls for the mdnsd program?” dtrace -n 'syscall:::entry /execname == "mdnsd"/ { @[probefunc] = count(); } tick-1s { exit(0); }' “Lots of signal masking and polling. What the hell! Why is it doing this? What is mdnsd anyways? Is there a man page? Googling for mdns reveals that it is used for resolving host names in small networks, like my home network. It uses UDP, and requires zero configuration. Nothing obvious to explain why it's flipping out. I feel helpless. I turn to the only thing I can trust, the code.” “Woah boy, this is some messy looking code. This would not pass illumos cstyle checks. Turns out this is code from Darwin—the kernel of OSX.” “Hmmm…an idea pops into my computer animal brain. I wonder…I wonder if my MacBook is also experiencing abnormal syscall rates? Nooo, that can't be it. Why would both my SmartOS server and MacBook both have the same problem? There is no good technical reason to link these two. But, then again, I'm dealing with computers here, and I've seen a lot of strange things over the years—I switch to my laptop.” sudo dtrace -n 'syscall::: { @[execname] = count(); } tick-1s { exit(0); }' Same thing, except mdns is called discoverd on OS X “I ask my friend Steve Vinoski to run the same DTrace one-liner on his OSX machines. He has both Yosemite and the older Mountain Lion. But, to my dismay, neither of his machines are exhibiting high syscall rates. My search continues.” “Not sure what to do next, I open the OSX Activity Monitor. In desperation I click on the Network tab.” “ HOLE—E—SHIT! Two-Hundred-and-Seventy Million packets received by discoveryd. Obviously, I need to stop looking at code and start looking at my network. I hop back onto my SmartOS machine and check network interface statistics.” “Whatever is causing all this, it is sending about 200 packets a second. At this point, the only thing left to do is actually inspect some of these incoming packets. I run snoop(1M) to collect events on the e1000g0 interface, stopping at about 600 events. Then I view the first 15.” “ A constant stream of mDNS packets arriving from IP 10.0.1.8. I know that this IP is not any of my computers. The only devices left are my iPhone, AppleTV, and Canon printer. Wait a minute! The printer! Two days earlier I heard some beeping noises…” “I own a Canon PIXMA MG6120 printer. It has a touch interface with a small LCD at the top, used to set various options. Since it sits next to my desk I sometimes lay things on top of it like a book or maybe a plate after I'm done eating. If I lay things in the wrong place it will activate the touch interface and cause repeated pressing. Each press makes a beeping noise. If the object lays there long enough the printer locks up and I have to reboot it. Just such events occurred two days earlier.” “I fire up dladm again to monitor incoming packets in realtime. Then I turn to the printer. I move all the crap off of it: two books, an empty plate, and the title for my Suzuki SV650 that I've been meaning to sell for the last year. I try to use the touch screen on top of the printer. It's locked up, as expected. I cut power to the printer and whip my head back to my terminal.” No more packet storm “Giddy, I run DTrace again to count syscalls.” “I'm not sure whether to laugh or cry. I laugh, because, LOL computers. There's some new dumb shit you deal with everyday, better to roll with the punches and laugh. You live longer that way. At least I got to flex my DTrace muscles a bit. In fact, I felt a bit like Brendan Gregg when he was debugging why OSX was dropping keystrokes.” “I didn't bother to root cause why my printer turned into a UDP machine gun. I don't intend to either. I have better things to do, and if rebooting solves the problem then I'm happy. Besides, I had to get back to what I was trying to do six hours before I started debugging this damn thing.” There you go. The Internet of Terror has already been on your LAN for years. Making Getaddrinfo Concurrent in Python on Mac OS and BSD (https://emptysqua.re/blog/getaddrinfo-cpython-mac-and-bsd/) We have a very fun blog post today to pass along originally authored by “A. Jesse Jiryu Davis”. Specifically the tale of one man's quest to unify the Getaddrinfo in Python with Mac OS and BSD. To give you a small taste of this tale, let us pass along just the introduction “Tell us about the time you made DNS resolution concurrent in Python on Mac and BSD. No, no, you do not want to hear that story, my friends. It is nothing but old lore and #ifdefs. But you made Python more scalable. The saga of Steve Jobs was sung to you by a mysterious wizard with a fanciful nickname! Tell us! Gather round, then. I will tell you how I unearthed a lost secret, unbound Python from old shackles, and banished an ancient and horrible Mutex Troll. Let us begin at the beginning.“ Is your interest piqued? It should be. I'm not sure we could do this blog post justice trying to read it aloud here, but definetly recommend if you want to see how he managed to get this bit of code working cross platform. (And it's highly entertaining as well) “A long time ago, in the 1980s, a coven of Berkeley sorcerers crafted an operating system. They named it after themselves: the Berkeley Software Distribution, or BSD. For generations they nurtured it, growing it and adding features. One night, they conjured a powerful function that could resolve hostnames to IPv4 or IPv6 addresses. It was called getaddrinfo. The function was mighty, but in years to come it would grow dangerous, for the sorcerers had not made getaddrinfo thread-safe.” “As ages passed, BSD spawned many offspring. There were FreeBSD, OpenBSD, NetBSD, and in time, Mac OS X. Each made its copy of getaddrinfo thread safe, at different times and different ways. Some operating systems retained scribes who recorded these events in the annals. Some did not.” The story continues as our hero battles the Mutex Troll and quests for ancient knowledge “Apple engineers are not like you and me — they are a shy and secretive folk. They publish only what code they must from Darwin. Their comings and goings are recorded in no bug tracker, their works in no changelog. To learn their secrets, one must delve deep.” “There is a tiny coven of NYC BSD users who meet at the tavern called Stone Creek, near my dwelling. They are aged and fierce, but I made the Sign of the Trident and supplicated them humbly for advice, and they were kindly to me.” Spoiler: “Without a word, the mercenary troll shouldered its axe and trudged off in search of other patrons on other platforms. Never again would it hold hostage the worthy smiths forging Python code on BSD.” *** Using release(7) to create FreeBSD images for OpenStack (https://diegocasati.com/2016/12/13/using-release7-to-create-freebsd-images-for-openstack-yes-you-can-do-it/) Following a recent episode where we covered a walk through on how to create FreeBSD guest OpenStack images, we wondered if it would be possible to integrate this process into the FreeBSD release(7) process, so they images could be generated consistently and automatically Being the awesome audience that you are, one of you responded by doing exactly that “During a recent BSDNow podcast, Allan and Kris mentioned that it would be nice to have a tutorial on how to create a FreeBSD image for OpenStack using the official release(7) tools. With that, it came to me that: #1 I do have access to an OpenStack environment and #2 I am interested in having FreeBSD as a guest image in my environment. Looks like I was up for the challenge.” “Previously, I've had success running FreeBSD 11.0-RELEASE on OpenStack but more could/should be done. For instance, as suggested by Allan, wouldn't be nice to deploy the latest code from FreeBSD ? Running -STABLE or even -CURRENT ? Yes, it would. Also, wouldn't it be nice to customize these images for a specific need? I'd say ‘Yes' for that as well.” “After some research I found that the current openstack.conf file, located at /usr/src/release/tools/ could use some extra tweaks to get where I wanted. I've created and attached that to a bugzilla on the same topic. You can read about that here (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213396).” Steps: Fetch the FreeBSD source code and extract it under /usr/src Once the code is in place, follow the regular process of build(7) and perform a make buildworld buildkernel Change into the release directory (/usr/src/release) and perform a make cloudware make cloudware-release WITH_CLOUDWARE=yes CLOUDWARE=OPENSTACK VMIMAGE=2G “That's it! This will generate a qcow2 image with 1.4G in size and a raw image of 2G. The entire process uses the release(7) toolchain to generate the image and should work with newer versions of FreeBSD.” + The patch has already been committed to FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=310047) Interview - Rod Grimes - rgrimes@freebsd.org (mailto:rgrimes@freebsd.org) Want to help fund the development of GPU Passthru? Visit bhyve.org (http://bhyve.org/) *** News Roundup Configuring the FreeBSD automounter (http://blog.khubla.com/freebsd/configuring-the-freebsd-automounter) Ever had to configure the FreeBSD auto-mounting daemon? Today we have a blog post that walks us through a few of the configuration knobs you have at your disposal. First up, Tom shows us his /etc/fstab file, and the various UFS partitions he has setup with the ‘noauto' flag so they are not mounted at system boot. His amd.conf file is pretty basic, with just options enabled to restart mounts, and unmount on exit. Where most users will most likely want to pay attention is in the crafting of an amd.map file Within this file, we have the various command-foo which performs mounts and unmounts of targeted disks / file-systems on demand. Pay special attention to all the special chars, since those all matter and a stray or missing ; could be a source of failure. Lastly a few knobs in rc.conf will enable the various services and a reboot should confirm the functionality. *** l2k16 hackathon report: LibreSSL manuals now in mdoc(7) (http://undeadly.org/cgi?action=article&sid=20161114174451) Hackathon report by Ingo Schwarze “Back in the spring two years ago, Kristaps Dzonsons started the pod2mdoc(1) conversion utility, and less than a month later, the LibreSSL project began. During the general summer hackathon in the same year, g2k14, Anthony Bentley started using pod2mdoc(1) for converting LibreSSL manuals to mdoc(7).” “Back then, doing so still was a pain, because pod2mdoc(1) was still full of bugs and had gaping holes in functionality. For example, Anthony was forced to basically translate the SYNOPSIS sections by hand, and to fix up .Fn and .Xr in the body by hand as well. All the same, he speedily finished all of libssl, and in the autumn of the same year, he mustered the courage to commit his work.” “Near the end of the following winter, i improved the pod2mdoc(1) tool to actually become convenient in practice and started work on libcrypto, converting about 50 out of the about 190 manuals. Max Fillinger also helped a bit, converting a handful of pages, but i fear i tarried too much checking and committing his work, so he quickly gave up on the task. After that, almost nothing happened for a full year.” “Now i was finally fed up with the messy situation and decided to put an end to it. So i went to Toulouse and finished the conversion of the remaining 130 manual pages in libcrypto, such that you can now view the documentation of all functions” Interactive Terminal Utility: smenu (https://github.com/p-gen/smenu) Ok, I've made no secret of my love for shell scripting. Well today we have a new (somewhat new to us) tool to bring your way. Have you ever needed to deal with large lists of data, perhaps as the result of a long specially crafted pipe? What if you need to select a specific value from a range and then continue processing? Enter ‘smenu' which can help make your scripting life easier. “smenu is a selection filter just like sed is an editing filter. This simple tool reads words from the standard input, presents them in a cool interactive window after the current line on the terminal and writes the selected word, if any, on the standard output. After having unsuccessfully searched the NET for what I wanted, I decided to try to write my own. I have tried hard to made its usage as simple as possible. It should work, even when using an old vt100 terminal and is UTF-8 aware.“ What this means, is in your interactive scripts, you can much easier present the user with a cursor driven menu to select from a range of possible choices. (Without needing to craft a bunch of dialog flags) Take a look, and hopefully you'll be able to find creative uses for your shell scripts in the future. *** Ubuntu still isn't free software (http://mjg59.dreamwidth.org/45939.html) “Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries. This does not affect your rights under any open source licence applicable to any of the components of Ubuntu. If you need us to approve, certify or provide modified versions for redistribution you will require a licence agreement from Canonical, for which you may be required to pay. For further information, please contact us” “Mark Shuttleworth just blogged (http://insights.ubuntu.com/2016/12/01/taking-a-stand-against-unstable-risky-unofficial-ubuntu-images/) about their stance against unofficial Ubuntu images. The assertion is that a cloud hoster is providing unofficial and modified Ubuntu images, and that these images are meaningfully different from upstream Ubuntu in terms of their functionality and security. Users are attempting to make use of these images, are finding that they don't work properly and are assuming that Ubuntu is a shoddy product. This is an entirely legitimate concern, and if Canonical are acting to reduce user confusion then they should be commended for that.” “The appropriate means to handle this kind of issue is trademark law. If someone claims that something is Ubuntu when it isn't, that's probably an infringement of the trademark and it's entirely reasonable for the trademark owner to take action to protect the value associated with their trademark. But Canonical's IP policy goes much further than that - it can be interpreted as meaning[1] that you can't distribute works based on Ubuntu without paying Canonical for the privilege, even if you call it something other than Ubuntu. [1]: And by "interpreted as meaning" I mean that's what it says and Canonical refuse to say otherwise” “If you ask a copyright holder if you can give a copy of their work to someone else (assuming it doesn't infringe trademark law), and they say no or insist you need an additional contract, it's not free software. If they insist that you recompile source code before you can give copies to someone else, it's not free software. Asking that you remove trademarks that would otherwise infringe trademark law is fine, but if you can't use their trademarks in non-infringing ways, that's still not free software.” “Canonical's IP policy continues to impose restrictions on all of these things, and therefore Ubuntu is not free software.” Beastie Bits OPNsense 16.7.10 released (https://opnsense.org/opnsense-16-7-10-released/) OpenBSD Foundation Welcomes First Iridium Donor: Smartisan (http://undeadly.org/cgi?action=article&sid=20161123193708&mode=expanded&count=8) Jan Koum donates $500,000 to FreeBSD (https://www.freebsdfoundation.org/blog/foundation-announces-new-uranium-donor/) The Soviet Russia, BSD makes you (https://en.wikipedia.org/wiki/DEMOS) Feedback/Questions Jason - Value (http://pastebin.com/gRN4Lzy8) Hamza - Shell Scripting (http://pastebin.com/GZYjRmSR) Blog link (http://aikchar.me/blog/unix-shell-programming-lessons-learned.html) Dave - Migrating to FreeBSD (http://pastebin.com/hEBu3Drp) Dan - Which BSD? (http://pastebin.com/1HpKqCSt) Zach - AMD Video (http://pastebin.com/4Aj5ebns) ***

Today on the show, we've got a look at running OpenBSD on a APU, some BSD in your Android, managing your own FreeBSD cloud service with ansible and much more. Keep it turned on your place to B...SD! This episode was brought to you by Headlines OpenBSD on PC Engines APU2 (https://github.com/elad/openbsd-apu2) A detailed walkthrough of building an OpenBSD firewall on a PC Engines APU2 It starts with a breakdown of the parts that were purchases, totally around $200 Then the reader is walked through configuring the serial console, flashing the ROM, and updating the BIOS The next step is actually creating a custom OpenBSD install image, and pre-configuring its serial console. Starting with OpenBSD 6.0, this step is done automatically by the installer Installation: Power off the APU2 Insert the bootable OpenBSD installer USB flash drive to one of the USB slots on the APU2 Power on the APU2, press F10 to get to the boot menu, and choose to boot from USB (usually option number 1) At the boot> prompt, remember the serial console settings (see above) Also at the boot> prompt, press Enter to start the installer Follow the installation instructions The driver used for wireless networking is athn(4). It might not work properly out of the box. Once OpenBSD is installed, run fw_update with no arguments. It will figure out which firmware updates are required and will download and install them. When it finishes, reboot. Where the rubber meets the road… (part one) (https://functionallyparanoid.com/2016/11/29/where-the-rubber-meets-the-road-part-one/) A user describes their adventures installing OpenBSD and Arch Linux on a new Lenovo X1 Carbon (4th gen, skylake) They also detail why they moved away from their beloved Macbook, which while long, does describe a journey away from Apple that we've heard elsewhere. The journey begins with getting a new Windows laptop, shrinking the partition and creating space for a triple-boot install, of Windows / Arch / OpenBSD Brian then details how he setup the partitioning and performed the initial Arch installation, getting it tuned to his specifications. Next up was OpenBSD though, and that went sideways initially due to a new NVMe drive that wasn't fully supported (yet) The article is split into two parts (we will bring you the next installment at a future date), but he leaves us with the plan of attack to build a custom OpenBSD kernel with corrected PCI device identifiers. We wish Brian luck, and look forward to the “rest of the story” soon. *** Howto setup a FreeBSD jail server using iocage and ansible. (https://github.com/JoergFiedler/freebsd-ansible-demo) Setting up a FreeBSD jail server can be a daunting task. However when a guide comes along which shows you how to do that, including not exposing a single (non-jailed) port to the outside world, you know we had a take a closer look. This guide comes to us from GitHub, courtesy of Joerg Fielder. The project goals seem notable: Ansible playbook that creates a FreeBSD server which hosts multiple jails. Travis is used to run/test the playbook. No service on the host is exposed externally. All external connections terminate within a jail. Roles can be reused using Ansible Galaxy. Combine any of those roles to create FreeBSD server, which perfectly suits you. To get started, you'll need a machine with Ansible, Vagrant and VirtualBox, and your credentials to AWS if you want it to automatically create / destroy EC2 instances. There's already an impressive list of Anisible roles created for you to start with: freebsd-build-server - Creates a FreeBSD poudriere build server freebsd-jail-host - FreeBSD Jail host freebsd-jailed - Provides a jail freebsd-jailed-nginx - Provides a jailed nginx server freebsd-jailed-php-fpm - Creates a php-fpm pool and a ZFS dataset which is used as web root by php-fpm freebsd-jailed-sftp - Installs a SFTP server freebsd-jailed-sshd - Provides a jailed sshd server. freebsd-jailed-syslogd - Provides a jailed syslogd freebsd-jailed-btsync - Provides a jailed btsync instance server freebsd-jailed-joomla - Installs Joomla freebsd-jailed-mariadb - Provides a jailed MariaDB server freebsd-jailed-wordpress - Provides a jailed Wordpress server. Since the machines have to be customized before starting, he mentions that cloud-init is used to do the following: activate pf firewall add a pass all keep state rule to pf to keep track of connection states, which in turn allows you to reload the pf service without losing the connection install the following packages: sudo bash python27 allow passwordless sudo for user ec2-user “ From there it is pretty straight-forward, just a couple commands to spin up the VM's either locally on your VirtualBox host, or in the cloud with AWS. Internally the VM's are auto-configured with iocage to create jails, where all your actual services run. A neat project, check it out today if you want a shake-n-bake type cloud + jail solution. Colin Percival's bsdiff helps reduce Android apk bandwidth usage by 6 petabytes per day (http://android-developers.blogspot.ca/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html) A post on the official Android-Developers blog, talks about how they used bsdiff (and bspatch) to reduce the size of Android application updates by 65% bsdiff was developed by FreeBSD's Colin Percival Earlier this year, we announced that we started using the bsdiff algorithm (by Colin Percival). Using bsdiff, we were able to reduce the size of app updates on average by 47% compared to the full APK size. This post is actually about the second generation of the code. Today, we're excited to share a new approach that goes further — File-by-File patching. App Updates using File-by-File patching are, on average, 65% smaller than the full app, and in some cases more than 90% smaller. Android apps are packaged as APKs, which are ZIP files with special conventions. Most of the content within the ZIP files (and APKs) is compressed using a technology called Deflate. Deflate is really good at compressing data but it has a drawback: it makes identifying changes in the original (uncompressed) content really hard. Even a tiny change to the original content (like changing one word in a book) can make the compressed output of deflate look completely different. Describing the differences between the original content is easy, but describing the differences between the compressed content is so hard that it leads to inefficient patches. So in the second generation of the code, they use bsdiff on each individual file, then package that, rather than diffing the original and new archives bsdiff is used in a great many other places, including shrinking the updates for the Firefox and Chrome browsers You can find out more about bsdiff here: http://www.daemonology.net/bsdiff/ A far more sophisticated algorithm, which typically provides roughly 20% smaller patches, is described in my doctoral thesis (http://www.daemonology.net/papers/thesis.pdf). Considering the gains, it is interesting that no one has implemented Colin's more sophisticated algorithm Colin had an interesting observation (https://twitter.com/cperciva/status/806426180379230208) last night: “I just realized that bandwidth savings due to bsdiff are now roughly equal to what the total internet traffic was when I wrote it in 2003.” *** News Roundup Distrowatch does an in-depth review of NAS4Free (https://distrowatch.com/weekly.php?issue=20161114#nas4free) Jesse Smith over at DistroWatch has done a pretty in-depth review of Nas4Free. The review starts with mentioning that NAS4Free works on 3 platforms, ARM/i386/AMD64 and for the purposes of this review he would be using AMD64 builds. After going through the initial install (doing typical disk management operations, such as GPT/MBR, etc) he was ready to begin using the product. One concern originally observed was that the initial boot seemed rather slow. Investigation revealed this was due to it loading the entire OS image into memory, and the first (long) disk read did take some time, but once loaded was super responsive. The next steps involved doing the initial configuration, which meant creating a new ZFS storage pool. After this process was done, he did find one puzzling UI option called “VM” which indicated it can be linked to VirtualBox in some way, but the Docs didn't reveal its secrets of usage. Additionally covered were some of the various “Access” methods, including traditional UNIX permissions, AD and LDAP, and then various Sharing services which are typical to a NAS, Such as NFS / Samba and others. One neat feature was the built-in file-browser via the web-interface, which allows you another method of getting at your data when sometimes NFS / Samba or WebDav aren't enough. Jesse gives us a nice round-up conclusion as well Most of the NAS operating systems I have used in the past were built around useful features. Some focused on making storage easy to set up and manage, others focused on services, such as making files available over multiple protocols or managing torrents. Some strive to be very easy to set up. NAS4Free does pretty well in each of the above categories. It may not be the easiest platform to set up, but it's probably a close second. It may not have the prettiest interface for managing settings, but it is quite easy to navigate. NAS4Free may not have the most add-on services and access protocols, but I suspect there are more than enough of both for most people. Where NAS4Free does better than most other solutions I have looked at is security. I don't think the project's website or documentation particularly focuses on security as a feature, but there are plenty of little security features that I liked. NAS4Free makes it very easy to lock the text console, which is good because we do not all keep our NAS boxes behind locked doors. The system is fairly easy to upgrade and appears to publish regular security updates in the form of new firmware. NAS4Free makes it fairly easy to set up user accounts, handle permissions and manage home directories. It's also pretty straight forward to switch from HTTP to HTTPS and to block people not on the local network from accessing the NAS's web interface. All in all, I like NAS4Free. It's a good, general purpose NAS operating system. While I did not feel the project did anything really amazing in any one category, nor did I run into any serious issues. The NAS ran as expected, was fairly straight forward to set up and easy to manage. This strikes me as an especially good platform for home or small business users who want an easy set up, some basic security and a solid collection of features. Browsix: Unix in the browser tab (https://browsix.org/) Browsix is a research project from the PLASMA lab at the University of Massachusetts, Amherst. The goal: Run C, C++, Go and Node.js programs as processes in browsers, including LaTeX, GNU Make, Go HTTP servers, and POSIX shell scripts. “Processes are built on top of Web Workers, letting applications run in parallel and spawn subprocesses. System calls include fork, spawn, exec, and wait.” Pipes are supported with pipe(2) enabling developers to compose processes into pipelines. Sockets include support for TCP socket servers and clients, making it possible to run applications like databases and HTTP servers together with their clients in the browser. Browsix comprises two core parts: A kernel written in TypeScript that makes core Unix features (including pipes, concurrent processes, signals, sockets, and a shared file system) available to web applications. Extended JavaScript runtimes for C, C++, Go, and Node.js that support running programs written in these languages as processes in the browser. This seems like an interesting project, although I am not sure how it would be used as more than a toy *** Book Review: PAM Mastery (https://www.cyberciti.biz/reviews/book-review-pam-mastery/) nixCraft does a book review of Michael W. Lucas' “Pam Mastery” Linux, FreeBSD, and Unix-like systems are multi-user and need some way of authenticating individual users. Back in the old days, this was done in different ways. You need to change each Unix application to use different authentication scheme. Before PAM, if you wanted to use an SQL database to authenticate users, you had to write specific support for that into each of your applications. Same for LDAP, etc. So Open Group lead to the development of PAM for the Unix-like system. Today Linux, FreeBSD, MacOS X and many other Unix-like systems are configured to use a centralized authentication mechanism called Pluggable Authentication Modules (PAM). The book “PAM Mastery” deals with the black magic of PAM. Of course, each OS chose to implement PAM a little bit differently The book starts with the basic concepts about PAM and authentication. You learn about Multi-Factor Authentication and why use PAM instead of changing each program to authenticate the user. The author went into great details about why PAM is useful for developers and sysadmin for several reasons. The examples cover CentOS Linux (RHEL and clones), Debian Linux, and FreeBSD Unix system. I like the way the author described PAM Configuration Files and Common Modules that covers everyday scenarios for the sysadmin. PAM configuration file format and PAM Module Interfaces are discussed in easy to understand language. Control flags in PAM can be very confusing for new sysadmins. Modules can be stacked in a particular order, and the control flags determine how important the success or failure of a particular module. There is also a chapter about using one-time passwords (Google Authenticator) for your application. The final chapter is all about enforcing good password policies for users and apps using PAM. The sysadmin would find this book useful as it covers a common authentication scheme that can be used with a wide variety of applications on Unix. You will master PAM topics and take control over authentication for your organization IT infrastructure. If you are Linux or Unix sysadmin, I would highly recommend this book. Once again Michael W Lucas nailed it. The only book you may need for PAM deployment. get “PAM Mastery” (https://www.michaelwlucas.com/tools/pam) *** Reflections on Trusting Trust - Ken Thompson, co-author of UNIX (http://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html) Ken Thompson's "cc hack" - Presented in the journal, Communication of the ACM, Vol. 27, No. 8, August 1984, in a paper entitled "Reflections on Trusting Trust", Ken Thompson, co-author of UNIX, recounted a story of how he created a version of the C compiler that, when presented with the source code for the "login" program, would automatically compile in a backdoor to allow him entry to the system. This is only half the story, though. In order to hide this trojan horse, Ken also added to this version of "cc" the ability to recognize if it was recompiling itself to make sure that the newly compiled C compiler contained both the "login" backdoor, and the code to insert both trojans into a newly compiled C compiler. In this way, the source code for the C compiler would never show that these trojans existed. The article starts off by talking about a content to write a program that produces its own source code as output. Or rather, a C program, that writes a C program, that produces its own source code as output. The C compiler is written in C. What I am about to describe is one of many "chicken and egg" problems that arise when compilers are written in their own language. In this case, I will use a specific example from the C compiler. Suppose we wish to alter the C compiler to include the sequence "v" to represent the vertical tab character. The extension to Figure 2 is obvious and is presented in Figure 3. We then recompile the C compiler, but we get a diagnostic. Obviously, since the binary version of the compiler does not know about "v," the source is not legal C. We must "train" the compiler. After it "knows" what "v" means, then our new change will become legal C. We look up on an ASCII chart that a vertical tab is decimal 11. We alter our source to look like Figure 4. Now the old compiler accepts the new source. We install the resulting binary as the new official C compiler and now we can write the portable version the way we had it in Figure 3. The actual bug I planted in the compiler would match code in the UNIX "login" command. The replacement code would miscompile the login command so that it would accept either the intended encrypted password or a particular known password. Thus if this code were installed in binary and the binary were used to compile the login command, I could log into that system as any user. Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions. Next “simply add a second Trojan horse to the one that already exists. The second pattern is aimed at the C compiler. The replacement code is a Stage I self-reproducing program that inserts both Trojan horses into the compiler. This requires a learning phase as in the Stage II example. First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere. So now there is a trojan'd version of cc. If you compile a clean version of cc, using the bad cc, you will get a bad cc. If you use the bad cc to compile the login program, it will have a backdoor. The source code for both backdoors no longer exists on the system. You can audit the source code of cc and login all you want, they are trustworthy. The compiler you use to compile your new compiler, is the untrustworthy bit, but you have no way to know it is untrustworthy, and no way to make a new compiler, without using the bad compiler. The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect. Acknowledgment: I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics. I can- not find a more specific reference to this document. I would appreciate it if anyone who can supply this reference would let me know. Beastie Bits Custom made Beastie Stockings (https://www.etsy.com/listing/496638945/freebsd-beastie-christmas-stocking) Migrating ZFS from mirrored pool to raidz1 pool (http://ximalas.info/2016/12/06/migrating-zfs-from-mirrored-pool-to-raidz1-pool/) OpenBSD and you (https://home.nuug.no/~peter/blug2016/) Watson.org FreeBSD and Linux cross reference (http://fxr.watson.org/) OpenGrok (http://bxr.su/) FreeBSD SA-16:37: libc (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:37.libc.asc) -- A 26+ year old bug found in BSD's libc, all BSDs likely affected -- A specially crafted argument can trigger a static buffer overflow in the library, with possibility to rewrite following static buffers that belong to other library functions. HardenedBSD issues correction for libc patch (https://github.com/HardenedBSD/hardenedBSD/commit/fb823297fbced336b6beeeb624e2dc65b67aa0eb) -- original patch improperly calculates how many bytes are remaining in the buffer. From December the 27th until the 30th there the 33rd Chaos Communication Congress[0] is going to take place in Hamburg, Germany. Think of it as the yearly gathering of the european hackerscene and their overseas friends. I am one of the persons organizing the "BSD assembly (https://events.ccc.de/congress/2016/wiki/Assembly:BSD)" as a gathering place for BSD enthusiasts and waving the flag amidst the all the other projects / communities. Feedback/Questions Chris - IPFW + Wifi (http://pastebin.com/WRiuW6nn) Jason - bhyve pci (http://pastebin.com/JgerqZZP) Al - pf errors (http://pastebin.com/3XY5MVca) Zach - Xorg settings (http://pastebin.com/Kty0qYXM) Bart - Wireless Support (http://pastebin.com/m3D81GBW) ***

This week on the show, we've got some new info on the talks from EuroBSDCon, a look at sharing a single ZFS pool between Linux and BSD, Sandboxing and much more! Stay tuned for your place to B...SD! This episode was brought to you by Headlines EuroBSDcon 2016 Presentation Slides (https://2016.eurobsdcon.org/PresentationSlides/) Due to circumstances beyond the control of the organizers of EuroBSDCon, there were not recordings of the talks given at the event. However, they have collected the slide decks from each of the speakers and assembled them on this page for you Also, we have some stuff from MeetBSD already: Youtube Playlist (https://www.youtube.com/playlist?list=PLb87fdKUIo8TAMC2HJLZ7H54edD2BeGWv) Not all of the sessions are posted yet, but the rest should appear shortly MeetBSD 2016 Trip Report: Domagoj Stolfa (https://www.freebsdfoundation.org/blog/meetbsd-2016-trip-report-domagoj-stolfa/) *** Cohabiting FreeBSD and Gentoo Linux on a Common ZFS Volume (https://ericmccorkleblog.wordpress.com/2016/11/15/cohabiting-freebsd-and-gentoo-linux-on-a-common-zfs-volume/) Eric McCorkle, who has contributed ZFS support to the FreeBSD EFI boot-loader code has posted an in-depth look at how he's setup dual-boot with FreeBSD and Gentoo on the same ZFS volume. He starts by giving us some background on how the layout is done. First up, GRUB is used as the boot-loader, allowing boot of both Linux and BSD The next non-typical thing was using /etc/fstab to manage mount-points, instead of the typical ‘zfs mount' usage, (apart from /home datasets) data/home is mounted to /home, with all of its child datasets using the ZFS mountpoint system data/freebsd and its child datasets house the FreeBSD system, and all have their mountpoints set to legacy data/gentoo and its child datasets house the Gentoo system, and have their mountpoints set to legacy as well So, how did he set this up? He helpfully provides an overview of the steps: Use the FreeBSD installer to create the GPT and ZFS pool Install and configure FreeBSD, with the native FreeBSD boot loader Boot into FreeBSD, create the Gentoo Linux datasets, install GRUB Boot into the Gentoo Linux installer, install Gentoo Boot into Gentoo, finish any configuration tasks The rest of the article walks us through the individual commands that make up each of those steps, as well as how to craft a GRUB config file capable of booting both systems. Personally, since we are using EFI, I would have installed rEFInd, and chain-loaded each systems EFI boot code from there, allowing the use of the BSD loader, but to each their own! HardenedBSD introduces Safestack into base (https://hardenedbsd.org/article/shawn-webb/2016-11-27/introducing-safestack) HardenedBSD has integrated SafeStack into its base system and ports tree SafeStack (http://clang.llvm.org/docs/SafeStack.html) is part of the Code Pointer Integrity (CPI) project within clang. “SafeStack is an instrumentation pass that protects programs against attacks based on stack buffer overflows, without introducing any measurable performance overhead. It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack.” “As of 28 November 2016, with clang 3.9.0, SafeStack only supports being applied to applications and not shared libraries. Multiple patches have been submitted to clang by third parties to add support for shared libraries.” SafeStack is only enabled on AMD64 *** pledge(2)… or, how I learned to love web application sandboxing (https://learnbchs.org/pledge.html) We've talked about OpenBSD's sandboxing mechanism pledge() in the past, but today we have a great article by Kristaps Dzonsons, about how he grew to love it for Web Sandboxing. +First up, he gives us his opening argument that should make most of you sit up and listen: I use application-level sandboxing a lot because I make mistakes a lot; and when writing web applications, the price of making mistakes is very dear. In the early 2000s, that meant using systrace(4) on OpenBSD and NetBSD. Then it was seccomp(2) (followed by libseccomp(3)) on Linux. Then there was capsicum(4) on FreeBSD and sandbox_init(3) on Mac OS X. All of these systems are invoked differently; and for the most part, whenever it came time to interface with one of them, I longed for sweet release from the nightmare. Please, try reading seccomp(2). To the end. Aligning web application logic and security policy would require an arduous (and usually trial-and-error or worse, copy-and-paste) process. If there was any process at all — if the burden of writing a policy didn't cause me to abandon sandboxing at the start. And then there was pledge(2). This document is about pledge(2) and why you should use it and love it. “ +Not convinced yet? Maybe you should take his challenge: Let's play a drinking game. The challenge is to stay out of the hospital. 1.Navigate to seccomp(2). 2. Read it to the end. 3. Drink every time you don't understand. For capsicum(4), the challenge is no less difficult. To see these in action, navigate no further than OpenSSH, which interfaces with these sandboxes: sandbox-seccomp-filter.c or sandbox-capsicum.c. (For a history lesson, you can even see sandbox-systrace.c.) Keep in mind that these do little more than restrict resources to open descriptors and the usual necessities of memory, signals, timing, etc. Keep that in mind and be horrified. “ Now Kristaps has his theory on why these are so difficult (NS..), but perhaps there is a better way. He makes the case that pledge() sits right in that sweet-spot, being powerful enough to be useful, but easy enough to implement that developers might actually use it. All in all, a nice read, check it out! Would love to hear other developer success stories using pledge() as well. *** News Roundup Unix history repository, now on GitHub (http://www.osnews.com/story/29513/Unix_history_repository_now_on_GitHub) OS News has an interesting tidbit on their site today, about the entire commit history of Unix now being available online, starting all the way back in 1970 and bringing us forward to today. From the README The history and evolution of the Unix operating system is made available as a revision management repository, covering the period from its inception in 1970 as a 2.5 thousand line kernel and 26 commands, to 2016 as a widely-used 27 million line system. The 1.1GB repository contains about half a million commits and more than two thousand merges. The repository employs Git system for its storage and is hosted on GitHub. It has been created by synthesizing with custom software 24 snapshots of systems developed at Bell Labs, the University of California at Berkeley, and the 386BSD team, two legacy repositories, and the modern repository of the open source FreeBSD system. In total, about one thousand individual contributors are identified, the early ones through primary research. The data set can be used for empirical research in software engineering, information systems, and software archaeology. This is a fascinating find, especially will be of value to students and historians who wish to look back in time to see how UNIX evolved, and in this repo ultimately turned into modern FreeBSD. *** Yandex commits improvements to FreeBSD network stack (https://reviews.freebsd.org/D8526) “Rework ip_tryforward() to use FIB4 KPI.” This commit brings some code from the experimental routing branch into head As you can see from the graphs, it offers some sizable improvements in forwarding and firewalled packets per second commit (https://svnweb.freebsd.org/base?view=revision&revision=309257) *** The brief history of Unix socket multiplexing – select(2) system call (https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/) Ever wondered about the details of socket multiplexing, aka the history of select(2)? Well Marek today gives a treat, with a quick look back at the history that made today's modern multiplexing possible. First, his article starts the way all good ones do, presenting the problem in silent-movie form: In mid-1960's time sharing was still a recent invention. Compared to a previous paradigm - batch-processing - time sharing was truly revolutionary. It greatly reduced the time wasted between writing a program and getting its result. Batch-processing meant hours and hours of waiting often to only see a program error. See this film to better understand the problems of 1960's programmers: "The trials and tribulations of batch processing". Enter the wild world of the 1970's, and we've now reached the birth of UNIX which tried to solve the batch processing problem with time-sharing. These days when a program was executed, it could "stall" (block) only on a couple of things1: + wait for CPU + wait for disk I/O + wait for user input (waiting for a shell command) or console (printing data too fast)“ Jump forward another dozen years or so, and the world changes yet again: This all changed in 1983 with the release of 4.2BSD. This revision introduced an early implementation of a TCP/IP stack and most importantly - the BSD Sockets API.Although today we take the BSD sockets API for granted, it wasn't obvious it was the right API. STREAMS were a competing API design on System V Revision 3. Coming in along with the sockets API was the select(2) call, which our very own Kirk McKusick gives us some background on: Select was introduced to allow applications to multiplex their I/O. Consider a simple application like a remote login. It has descriptors for reading from and writing to the terminal and a descriptor for the (bidirectional) socket. It needs to read from the terminal keyboard and write those characters to the socket. It also needs to read from the socket and write to the terminal. Reading from a descriptor that has nothing queued causes the application to block until data arrives. The application does not know whether to read from the terminal or the socket and if it guesses wrong will incorrectly block. So select was added to let it find out which descriptor had data ready to read. If neither, select blocks until data arrives on one descriptor and then awakens telling which descriptor has data to read. [...] Non-blocking was added at the same time as select. But using non-blocking when reading descriptors does not work well. Do you go into an infinite loop trying to read each of your input descriptors? If not, do you pause after each pass and if so for how long to remain responsive to input? Select is just far more efficient. Select also lets you create a single inetd daemon rather than having to have a separate daemon for every service. The article then wraps up with an interesting conclusion: > CSP = Communicating sequential processes In this discussion I was afraid to phrase the core question. Were Unix processes intended to be CSP-style processes? Are file descriptors a CSP-derived "channels"? Is select equivalent to ALT statement? I think: no. Even if there are design similarities, they are accidental. The file-descriptor abstractions were developed well before the original CSP paper. It seems that an operating socket API's evolved totally disconnected from the userspace CSP-alike programming paradigms. It's a pity though. It would be interesting to see an operating system coherent with the programming paradigms of the user land programs. A long (but good) read, and worth your time if you are interested in the history how modern multiplexing came to be. *** How to start CLion on FreeBSD? (https://intellij-support.jetbrains.com/hc/en-us/articles/206525024-How-to-start-CLion-on-FreeBSD) CLion (pronounced "sea lion") is a cross-platform C and C++ IDE By default, the Linux version comes bundled with some binaries, which obviously won't work with the native FreeBSD build Rather than using Linux emulation, you can replace these components with native versions pkg install openjdk8 cmake gdb Edit clion-2016.3/bin/idea.properties and change run.processes.with.pty=false Start CLion and open Settings | Build, Execution, Deployment | Toolchains Specify CMake path: /usr/local/bin/cmake and GDB path: /usr/local/bin/gdb Without a replacement for fsnotifier, you will get a warning that the IDE may be slow to detect changes to files on disk But, someone has already written a version of fsnotifier that works on FreeBSD and OpenBSD fsnotifier for OpenBSD and FreeBSD (https://github.com/idea4bsd/fsnotifier) -- The fsnotifier is used by IntelliJ for detecting file changes. This version supports FreeBSD and OpenBSD via libinotify and is a replacement for the bundled Linux-only version coming with the IntelliJ IDEA Community Edition. *** Beastie Bits TrueOS Pico – FreeBSD ARM/RPi Thin Clients (https://www.trueos.org/trueos-pico/) A Puppet package provider for FreeBSD's PkgNG package manager. (https://github.com/xaque208/puppet-pkgng) Notes from November London *BSD meetup (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2016-November/014059.html) SemiBug meeting on Dec 20th (http://lists.nycbug.org/pipermail/semibug/2016-November/000131.html) Feedback/Questions Erno - SSH without password (http://pastebin.com/SMvxur9v) Jonathan - Magical ZFS (http://pastebin.com/5ETL7nmj) George - TrueOS (http://pastebin.com/tSVvaV9e) Mohammad - Jails IP (http://pastebin.com/T8nUexd1) Gibheer - BEs (http://pastebin.com/YssXXp70) ***

On today's episode, we are loaded and ready to go. Lots of OpenBSD news, a look at LetsEncrypt usage, the NetBSD scheduler (oh my) and much more. Keep it tuned to your place to B...SD! This episode was brought to you by Headlines Production ready (http://www.tedunangst.com/flak/post/production-ready) Ted Unangst brings us a piece on what it means to be Production Ready He tells the story of a project he worked on that picked a framework that was “production ready” They tested time zones, and it all seemed to work They tested the unicode support in english and various european languages, and it was all good They sent some emails with it, and it just worked The framework said “Production Ready” on the tin, and it passed all the tests. What is the worst that could happen? Now, we built our product on top of this. Some of the bugs were caught internally. Others were discovered by customers, who were of course a little dismayed. Like, how could you possibly ship this? Indeed. We were doing testing, quite a bit really, but when every possible edge case has a bug, it's hard to find them all. A customer from Arizona, which does not observe Daylight Saving Time, crashed the app Some less common unicode characters caused a buffer overflow The email system did not properly escape a period on its own line, truncating the email “Egregious performance because of a naive N^2 algorithm for growing a buffer.” “Egregious performance on some platforms due to using the wrong threading primitives.” “Bizarre database connection bugs for some queries that I can't at all explain.” “In short, everything was “works for me” quality. But is that really production quality?” “There are some obvious contenders for the title of today's most “production ready” software, but it's a more general phenomenon. People who have success don't know what they don't know, what they didn't test, what unused features will crash and burn.” Using Let's Encrypt within FreeBSD.org (https://blog.crashed.org/letsencrypt-in-freebsd-org/) I decided to give Let's Encrypt certificates a shot on my personal web servers earlier this year after a disaster with StartSSL. I'd like to share what I've learned. The biggest gotcha is that people tend to develop bad habits when they only have to deal with certificates once a year or so. The beginning part of the process is manual and the deployment of certificates somehow never quite gets automated, or things get left out. That all changes with Let's Encrypt certificates. Instead of 1-5 year lifetime certificates the Let's Encrypt certificates are only valid for 90 days. Most people will be wanting to renew every 60-80 days. This forces the issue - you really need to automate and make it robust. The Let's Encrypt folks provide tools to do this for you for the common cases. You run it on the actual machine, it manages the certificates and adjusts the server configuration files for you. Their goal is to provide a baseline shake-n-bake solution. I was not willing to give that level of control to a third party tool for my own servers - and it was absolutely out of the question for for the FreeBSD.org cluster. I should probably mention that we do things on the FreeBSD.org cluster that many people would find a bit strange. The biggest problem that we have to deal with is that the traditional model of a firewall/bastion between "us" and "them" does not apply. We design for the assumption that hostile users are already on the "inside" of the network. The cluster is spread over 8 distinct sites with naked internet and no vpn between them. There is actually very little trust between the systems in this network - eg: ssh is for people only - no headless users can ssh. There are no passwords. Sudo can't be used. The command and control systems use signing. We don't trust anything by IPv4/IPv6 address because we have to assume MITM is a thing. And so on. In general, things are constructed to be trigger / polling / pull based. The downside is that this makes automation and integration of Let's Encrypt clients interesting. If server configuration files can't be modified; and replicated web infrastructure is literally read-only (via jails/nullfs); and DNS zone files are static; and headless users can't ssh and therefore cannot do commits, how do you do the verification tokens in an automated fashion? Interesting, indeed. We wanted to be able to use certificates on things like ldap and smtp servers. You can't do http file verification on those so we had to use dns validation of domains. First, a signing request is generated, and the acme-challenge is returned Peter's post then walks through how the script adds the required TXT record to prove control of the domain, regenerates the zone file, DNSSEC signs it, and waits for it to be published, then continues the letsencrypt process. Letsencrypt then issues the actual certificate We export the fullchain files into a publication location. There is another jail that can read the fullchain certificates via nullfs and they are published with our non-secrets update mechanism Since we are using DNSSEC, here is a good opportunity to maintain signed TLSA fingerprints. The catch with TLSA record updates is managing the update event horizon. You are supposed to have both fingerprints listed across the update cycle. We use 'TLSA 3 1 1' records to avoid issues with propagation delays for now. TLSA 3 0 1 changes with every renewal, while 3 1 1 only changes when you generate a new private key. The majority of TLS/SSL servers require a full restart to re-load the certificates if the filename is unchanged. I found out the hard way. There is a great deal more detail in the blog post, I recommend you check it out Learning more about the NetBSD scheduler (... than I wanted to know) Part 1 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161105_1754.html) Part 2 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161109_0059.html) Part 3 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161113_0122.html) Today I had a need to do some number crunching using a home-brewn C program. In order to do some manual load balancing, I was firing up some Amazon AWS instances (which is Xen) with NetBSD 7.0. In this case, the system was assigned two CPUs I started two instances of my program, with the intent to have each one use one CPU. Which is not what happened! Here is what I observed, and how I fixed things for now. ~~ load averages: 2.14, 2.08, 1.83; up 0+00:45:56 18:01:32 27 processes: 4 runnable, 21 sleeping, 2 on CPU CPU0 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Memory: 119M Act, 7940K Exec, 101M File, 3546M Free ~~ ~~ PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 2791 root 25 0 8816K 964K RUN/0 16:10 54.20% 54.20% myprog 2845 root 26 0 8816K 964K RUN/0 17:10 47.90% 47.90% myprog ~~ I expected something like WCPU and CPU being around 100%, assuming that each process was bound to its own CPU. The values I actually saw (and listed above) suggested that both programs were fighting for the same CPU. Huh?! NetBSD allows to create "processor sets", assign CPU(s) to them and then assign processes to the processor sets. Let's have a look! ~~ # psrset -c 1 # psrset -b 0 2791 # psrset -b 1 2845 load averages: 2.02, 2.05, 1.94; up 0+00:59:32 18:15:08 27 processes: 1 runnable, 24 sleeping, 2 on CPU CPU0 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle CPU1 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle Memory: 119M Act, 7940K Exec, 101M File, 3546M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 2845 root 25 0 8816K 964K CPU/1 26:14 100% 100% myprog 2791 root 25 0 8816K 964K RUN/0 25:40 100% 100% myprog ~~ Things are as expected now, with each program being bound to its own CPU. Now why this didn't happen by default is left as an exercise to the reader. I had another look at this today, and was able to reproduce the behaviour using VMWare Fusion with two CPU cores on both NetBSD 7.0_STABLE as well as -current The one hint that I got so far was from Michael van Elst that there may be a rouding error in sched_balance(). Looking at the code, there is not much room for a rounding error. But I am not familiar enough (at all) with the code, so I cannot judge if crucial bits are dropped here, or how that function fits in the whole puzzle. Pondering on the "rounding error", I've setup both VMs with 4 CPUs, and the behaviour shown there is that load is distributed to about 3 and a half CPU - three CPUs under full load, and one not reaching 100%. There's definitely something fishy in there. With multiple CPUs, each CPU has a queue of processes that are either "on the CPU" (running) or waiting to be serviced (run) on that CPU. Those processes count as "migratable" in runqueue_t. Every now and then, the system checks all its run queues to see if a CPU is idle, and can thus "steal" (migrate) processes from a busy CPU. This is done in sched_balance(). Such "stealing" (migration) has the positive effect that the process doesn't have to wait for getting serviced on the CPU it's currently waiting on. On the other side, migrating the process has effects on CPU's data and instruction caches, so switching CPUs shouldn't be taken too easy. All in all, I'd say the patch is a good step forward from the current situation, which does not properly distribute pure CPU hogs, at all. Building Cost-Effective 100-Gbps Firewalls for HPC with FreeBSD (https://www.nas.nasa.gov/SC16/demos/demo9.html) The continuous growth of the NASA Center for Climate Simulation (NCCS) requires providing high-performance security tools and enhancing the network capacity. In order to support the requirements of emerging services, including the Advanced Data Analytics Platform (ADAPT) private cloud, the NCCS security team has proposed an architecture to provide extremely cost-effective 100-gigabit-per-second (Gbps) firewalls. The aim of this project is to create a commodity-based platform that can process enough packets per second (pps) to sustain a 100-Gbps workload within the NCCS computational environment. The test domain consists of several existing systems within the NCCS, including switches (Dell S4084), routers (Dell R530s), servers (Dell R420s, and C6100s), and host card adapters (10-Gbps Mellanox ConnectX2 and Intel 8259 x Ethernet cards). Previous NCCS work testing the FreeBSD operating system for high-performance routing reached a maximum of 4 million pps. Building on this work, we are comparing FreeBSD-11.0 and FreeBSD-Current along with implementing the netmap-fwd Application Programming Interface (API) and tuning the 10-gigabit Ethernet cards. We used the tools iperf3, nuttcp, and netperf to monitor the performance of the maximum bandwidth through the cards. Additional testing has involved enabling the Common Address Redundancy Protocol (CARP) to achieve an active/active architecture. The tests have shown that at the optimally tuned and configured FreeBSD system, it is possible to create a system that can manage the huge amounts of pps needed to create a 100-Gbps firewall with commodity components. Some interesting findings: FreeBSD was able to send more pps as a client than Centos 6. Netmap-fwd increased the pps rate significantly. The choice of network card can have a significant impact on pps, tuning, and netmap support. Further tests will continue verifying the above results with even more capable systems-such as 40-gigabit and 100-gigabit Ethernet cards-to achieve even higher performance. In addition to hardware improvements, updates to the network capabilities in the FreeBSD-Current version will be closely monitored and applied as appropriate. The final result will be a reference architecture with representative hardware and software that will enable the NCCS to build, deploy, and efficiently maintain extremely cost-effective 100-Gbps firewalls. Netflix has already managed to saturate a 100 Gbps interface using only a single CPU Socket (rather than a dual socket server). Forwarding/routing is a bit different, but it is definitely on track to get there. Using a small number of commodity servers to firewall 100 Gbps of traffic just takes some careful planning and load balancing. Soon it will be possible using a single host. News Roundup iocell - A FreeBSD jail manager. (https://github.com/bartekrutkowski/iocell) Another jail manager has arrived on the scene, iocell, which begins life as a fork of the “classic” iocage. Due to its shared heritage, it offers much of the same functionality and flags as iocage users will be familiar with. For those who aren't up to speed with either products, some of those features include: Templates, clones, basejails, fully independent jails Ease of use Zero configuration files Rapid thin provisioning within seconds Automatic package installation Virtual networking stacks (vnet) Shared IP based jails (non vnet) Resource limits (CPU, MEMORY, DISK I/O, etc.) Filesystem quotas and reservations Dedicated ZFS datasets inside jails Transparent ZFS snapshot management Binary updates Differential jail packaging Export and import And many more! The program makes extensive use of ZFS for performing jail operations, so a zpool will be required (But doesn't have to be your boot-pool) It still looks “very” fresh, even using original iocage filenames in the repo, so a safe guess is that you'll be able to switch between iocage and iocell with relative ease. Fail2ban on OpenBSD 6.0 (http://blog.gordonturner.ca/2016/11/20/fail2ban-on-openbsd-6-0/) We've used Fail2Ban in PC-BSD before, due to it's ability to detect and block brute force attempts against a variety of services, including SSH, mail, and others. It even can work to detect jail brute force attempts, blocking IPs on the hosts firewall. However what about OpenBSD users? Well, Gordon Turner comes to the rescue today with a great writeup on deploying Fail2Ban specifically for that platform. Now, Fail2Ban is a python program, so you'll need to pkg install Python first, then he provides instructions on how to manually grab the F2B sources and install on OpenBSD. Helpfully Gordon gives us some handy links to scripts and modifications to get F2B running via RC as well, which is a bit different since F2B has both a server and client that must run together. With the installation bits out of the way, we get to next hit the “fun” stuff, which comes in the way of SSH brute force detection. Naturally we will be configuring F2B to use “pf” to do our actual blocking, but the examples shown give us full control over the knobs used to detect, and then ultimately call ‘pfctl' to do our heavy lifting. The last bits of the article give us a runthrough on how to “prime” pf with the correct block tables and performing basic administrative tasks to control F2B in production. A great article, and if you run an OpenBSD box exposed to the internet, you may want to bookmark this one. openbsd changes of note (http://www.tedunangst.com/flak/post/openbsd-changes-of-note) Continuing with our OpenBSD news for the week, we have a new blog post by TedU, which gives us a bunch of notes on the things which have changed over there as of late: Some of the notables include: mcl2k2 pools and the em conversion. The details are in the commits, but the short story is that due to hardware limitations, a number of tradeoffs need to be made between performance and memory usage. The em chip can (mostly) only be programmed to write to 2k buffers. However, ethernet payloads are not nicely aligned. They're two bytes off. Leading to a costly choice. Provide a 2k buffer, and then copy all the data after the fact, which is slow. Or allocate a larger than 2k buffer, and provide em with a pointer that's 2 bytes offset. Previously, the next size up from 2k was 4k, which is quite wasteful. The new 2k2 buffer size still wastes a bit of memory, but much less. FreeType 2.7 is prettier than ever. vmm for i386. Improve security. vmm is still running with a phenomenal set of privileges, but perhaps some cross-VM attacks may be limited. On the other side of the world, hyperv support is getting better. Remove setlocale. setlocale was sprinkled all throughout the code base many years ago, even though it did nothing, in anticipation of a day when it would do something. We've since decided that day will never come, and so many setlocale calls can go. syspatch is coming. Lots of commits actually. Despite the name, it's more like a system update, since it replaces entire binaries. Then again, replacing a few binaries in a system is like patching small parts of the whole. A syspatch update will be smaller than an entire release. There's a new build system. It kind of works like before, but a lot of the details have changed to support less root. Actually, it'd be accurate to say the whole build privilege system has been flipped. Start as root, which drops down to the build user to do the heavy lifting, instead of starting as a user that can elevate to root at any time. This no longer requires the build user to be pseudo-root; in fact, the goal is that the build user can't elevate. There's several other items on this list, take a look for more details, and he also helpfully provides commit-links if you want to see more about any of these topics. It came from Bell Labs (http://media.bemyapp.com/came-bell-labs/#) A little late for a halloween episode, we have “It came from Bell Labs”, a fascinating article talking about the successor to UNIX, Plan9 There was once an operating system that was intended to be the successor to Unix. Plan 9 From Bell Labs was its name, and playing with it for five minutes is like visiting an alternate dimension where computers are done differently. It was so ahead of its time that it would be considered cutting edge, even today. Find out the weird and woolly history to Plan Nine's inception and eventual consignment as a footnote of operating systems today. So, if you've never heard of Plan 9, how did it exactly differ from the UNIX we know and love today? Here's just a few of the key features under Plan 9's hood + 9P – The distributed file system protocol. Everything runs through this, there is no escaping it. Since everything runs on top of 9P, that makes everything running on a Plan 9 box distributed as well. This means, for example, you can import /dev/audio from another machine on the network to use its sound card when your own machine doesn't have one. + ndb – The namespace server. In conjunction with 9P, it bosses all the programs around and forces them to comply to the Plan 9 way. + Instead of Unix sockets, all the networking just runs through 9P. Thus, everything from ethernet packets to network cards are all just one more kind of file. + While Unicode is implemented ad-hoc in other systems, it's baked into Plan 9 from the first int main(). In fact, even users who don't like Plan 9 have to admit that the character encoding support, together with the beautiful built-in rio font, makes every other operating system look primitive. + The system's own internal programs are built to be a rounded set of user tools from the ground up. So, for instance, it comes with its own editor, acme, built to be its own weird morphing thing that plays nice with the 9P protocol. Sounds neat, but how did it work in the real world? The result was a mixture of both breathtaking efficiency and alienating other-worldliness. Trying out the system is like a visit to an alternate reality where time-traveling gremlins changed how computers are made and used. You can execute anycommand anywhere just by typing its name and middle-clicking on it, even in the middle of reading a file. You can type out your blog post in the middle of a man page and save it right there. Screenshots are made by pointing /dev/screen to a file. When you execute a program in a terminal, the terminal morphs into the program you launched instead of running in the background. The window manager, rio, can be invoked within rio to create an instance of itself running inside itself. You can just keep going like that, until, like Inception, you get lost in which layer you're in. Get used to running Plan 9 long enough, and you will find yourself horribly ill-adapted for dealing with the normal world. While system administrators can't stop praising it, the average home user won't see much benefit unless they happen to run about eight desktop machines scattered all over. But to quote legendary hacker tribal bard Eric S. Raymond: “…Plan 9 failed simply because it fell short of being a compelling enough improvement on Unix to displace its ancestor.” A fascinating article, worth your time to read it through, even though we've pulled some of the best bits here. Nice look at the alternative dimension that could have been. Beastie Bits inks -- Basically Reddit or Hacker News, but without the disagreeable trolls and military industrial complex shills downvoting everything to hide the truth (http://www.tedunangst.com/flak/post/inks) “PAM is Un-American” talk now online (https://youtu.be/Mc2p6sx2s7k) Reddit advertising of “PAM Mastery” (http://blather.michaelwlucas.com/archives/2818) MeetBSD 2016 Report by Michael Dexter (https://www.ixsystems.com/blog/meetbsd-2016-report-michael-dexter/) Various CBSD Tutorials (https://www.bsdstore.ru/en/tutorial.html) Feedback/Questions Dylan - Kaltura Alt (http://pastebin.com/6B96pVcm) Scott - ZFS in Low-Mem (http://pastebin.com/Hrp8qwkP) J - Mixing Ports / Pkgs (http://pastebin.com/85q4Q3Xx) Trenton - Dtract & PC-BSD (http://pastebin.com/RFKY0ERs) Ivan - ZFS Backups (http://pastebin.com/31uqW6vW) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)

This week on BSDNow. Allan and I are back from MeetBSD! A good time was had by all, lots to discuss, so let's jump right into it on your place to B...SD! This episode was brought to you by Headlines Build a FreeBSD 11.0-release Openstack Image with bsd-cloudinit (https://raymii.org/s/tutorials/FreeBSD_11.0-release_Openstack_Image.html) We are going to prepare a FreeBSD image for Openstack deployment. We do this by creating a FreeBSD 11.0-RELEASE instance, installing it and converting it using bsd-cloudinit. We'll use the CloudVPS public Openstack cloud for this. Create an account there and install the Openstack command line tools, like nova, cinder and glance. A FreeBSD image with Cloud Init will automatically resize the disk to the size of the flavor and it will add your SSH key right at boot. You can use Cloud Config to execute a script at first boot, for example, to bootstrap your system into Puppet or Ansible. If you use Ansible to manage OpenStack instances you can integrate it without manually logging in or doing anything manually. Since FreeBSD 10.2-RELEASE there is an rc script which, when the file /firstboot exists, expands the root filesystem to the full disk. While bsd-cloudinit does this as well, if you don't need the whole cloudinit stack, (when you use a static ssh key for example), you can touch that file to make sure the disk is expanded at the first boot A detailed tutorial that shows how to create customized cloud images using the FreeBSD install media There is also the option of using the FreeBSD release tools to build custom cloud images in a more headless fashion Someone should make a tutorial out of that *** iXsystems Announces TrueOS Launch (https://www.ixsystems.com/blog/ixsystems-announces-trueos-launch/) As loyal listeners to this show, you've no doubt heard by now that we are in the middle of undergoing a shift in moving PC-BSD -> TrueOS. Last week during MeetBSD this was made “official” with iX issuing our press release and I was able to give a talk detailing many of the reasons and things going on with this change. The talk should be available online here soon(ish), but for a quick recap: TrueOS is moving to a rolling-release model based on FreeBSD -CURRENT Lumina has become the default desktop for TrueOS LibreSSL is enabled top to bottom We are in the middle of working on conversion to OpenRC for run-control replacement The TrueOS pico was announced, which is our “Thin-Client” solution, right now allowing you to use a TrueOS server pared with a RPI2 device. *** Running FreeBSD 11 on Raspberry Pi (https://vzaigrin.wordpress.com/2016/10/16/running-freebsd-11-on-raspberry-pi/) This article covers some of the changes you will notice if you upgrade your RPI to FreeBSD 11.0 It covers some of the changes to WiFi in 11.0 Pro Tip: you can get a list of WiFi devices by doing: sysctl net.wlan.devices There are official binary packages for ARM with 11.0, so you can just ‘pkg install' your favourite apps Many of the LEDs are exposed via the /dev/led/ interface, which you can just echo 0 or 1 to, or use morse(6) to send a message gpioctl can be used to control the various GPIO pins The post also covers how to setup the real-time clock on the Raspberry Pi There is also limited support for adjusting the CPU frequency of the Pi There are also tips on configuring a one-wire temperature sensor *** void-zones-tools for FreeBSD (https://github.com/cyclaero/void-zones-tools) Adblock has been in the news a bit recently, with some of the more popular browser plugins now accepting brib^...contributions to permit specific ads through. Well today the ad-blockers strike back. We have a great tutorial up on GitHub which demonstrates one of the useful features of using Unbound in FreeBSD to do your own ad-blocking with void-zones. Specifically, void-zones are a way to return NXDOMAIN when DNS requests are made to known malicious or spam sites. Using void-zones-tools software will make managing this easy, by being able to pull in known lists of sites to block from several 3rd party curators. When coupled with our past tutorials on setting up your own FreeBSD router, this may become very useful for a lot of folks who want to do ad-blocking ad at a lower level, allowing it to filter smart-phones or any other devices on a network. *** News Roundup BSD Socket API Revamp (https://raw.githubusercontent.com/sustrik/dsock/master/rfc/sock-api-revamp-01.txt) Martin Sustrik has started a draft RFC to revamp the BSD Sockets API: The progress in the area of network protocols is distinctively lagging behind. While every hobbyist new to the art of programming writes and publishes their small JavaScript libraries, there's no such thing going on with network protocols. Indeed, it looks like the field of network protocols is dominated by big companies and academia, just like programming as a whole used to be before the advent of personal computers. the API proposed in this document doesn't try to virtualize all possible aspects of all possible protocols and provide a single set of functions to deal with all of them. Instead, it acknowledges how varied the protocol landscape is and how much the requirements for individual protocols differ. Therefore, it lets each protocol define its own API and asks only for bare minimum of standardised behaviour needed to implement protocol composability. As a consequence, the new API is much more lightweight and flexible than BSD socket API and allows to decompose today's monolithic protocol monsters into small single-purpose microprotocols that can be easily combined together to achieve desired functionality. The idea behind the new design is to allow the software author to define their own protocols via a generic interface, and easily stack them on top of the existing network protocols, be they the basic protocols like TCP/IP, or a layer 7 protocol like HTTP Example of creating a stack of four protocols: ~~ int s1 = tcpconnect("192.168.0.111:5555"); int s2 = foostart(s1, arg1, arg2, arg3); int s3 = barstart(s2); int s4 = bazstart(s3, arg4, arg5); ~~ It also allows applying generic transformations to the protocols: ~~ int tcps = tcpconnect("192.168.0.111:80"); /* Websockets is a connected protocol. */ int ws = websockconnect(tcps); uint16t compressionalgoritm; mrecv(ws, &compressionalgorithm, 2, -1); /* Compression socket is unconnected. */ int cs = compressstart(ws, compression_algorithm); ~~ *** Updated version of re(4) for DragonflyBSD (http://lists.dragonflybsd.org/pipermail/users/2016-November/313140.html) Sephe over at the Dragonfly project has issued a CFT for a newer version of the “re” driver For those who don't know, that is for Realtek nics, specifically his updates add features: I have made an updated version of re(4), which leverages Realtek driver's chip/PHY reset/initialization code. I hope it can resolve all kinds of weirdness we encountered on this chip so far. Testers, you know what to do! Give this a whirl and let him know if you run into any new issues, or better yet, give feedback if it fixes some long-standing problems you've run into in the past. *** Hackathon reports from OpenBSD's B2K16 b2k16 hackathon report: Jeremy Evans on ports cleaning, progress on postgres, nginx, ruby and more (http://undeadly.org/cgi?action=article&sid=20161112112023) b2k16 hackathon report: Landry Breuil on various ports progress (http://undeadly.org/cgi?action=article&sid=20161112095902) b2k16 hackathon report: Antoine Jacoutot on GNOME's path forward, various ports progress (http://undeadly.org/cgi?action=article&sid=20161109030623) We have a trio of hackathon reports from OpenBSD's B2K16 (Recently held in Budapest) First up - Jeremy Evans give us his rundown which starts with sweeping some of the cruft out of the barn: I started off b2k16 by channeling tedu@, and removing a lot of ports, including lang/ruby/2.0, lang/io, convertors/ruby-json, databases/dbic++, databases/ruby-swift, databases/ruby-jdbc-*, x11/ruby-profiligacy, and mail/ruby-mailfactory. After that, he talks about improvements made to postgres, nginx and ruby ports, fixing things such as pg_upgrade support, breaking nginx down into sub-packages and a major ruby update to about 50% of the packages. Next up - Landry Breuil tells us about his trip, which also started with some major ports pruning, including some stale XFCE bits and drupal6. One of the things he mentions is the Tor browser: Found finally some time again to review properly the pending port for Tor Browser, even if i don't like the way it is developed (600+ patches against upstream firefox-esr !? even if relationship is improving..) nor will endorse its use, i feel that the time that was spent on porting it and updating it and maintaining it shouldn't be lost, and it should get commited - there are only some portswise minor tweaks to fix. Had a bit of discussions about that with other porters... Lastly, Antoine Jacoutot gives us a smaller update on his work: First task of this hackathon was for Jasper and I to upgrade to GNOME 3.22.1 (version 3.22.2 hit the ports tree since). As usual I already updated the core libraries a few days before so that we could start with a nice set of fully updated packages. It ended up being the fastest GNOME update ever, it all went very smoothly. We're still debating the future of GNOME on OpenBSD though. More and more features require systemd interfaces and without a replacement it may not make sense to keep it around. Implementing these interfaces requires time which Jasper and I don't really have these days... Anyway, we'll see. All-n-all, a good trip it sounds like with some much needed hacking taking place. Good to see the cruft getting cleaned up, along with some new exciting ports landing. *** July to September 2016 Status Report (https://www.freebsd.org/news/status/report-2016-07-2016-09.html) The latest FreeBSD quarterly status report is out It includes the induction of the new Core team, and reports from all of the other teams, including Release Engineering, Port Manager, and the FreeBSD Foundation Some other highlights: Capsicum Update The Graphics Stack on FreeBSD Using lld, the LLVM Linker, to Link FreeBSD VirtualBox Shared Folders Filesystem evdev support (better mouse, keyboard, and multi-touch support) ZFS Code Sync with Latest OpenZFS/Illumos The ARC now mostly stores compressed data, the same as is stored on disk, decompressing them on demand. The L2ARC now stores the same (compressed) data as the ARC without recompression, and its RAM usage was further reduced. The largest size of indirect block possible has been increased from 16KB to 128KB, and speculative prefetching of indirect blocks is now performed. Improved ordering of space allocation. The SHA-512t256 and Skein hashing algorithms are now supported. *** Beastie Bits How to Host Your Own Private GitHub with Gogs (http://www.cs.cmu.edu/afs/cs/user/predragp/www/git.html) Nvidia Adds Telemetry To Latest Drivers (https://yro.slashdot.org/story/16/11/07/1427257/nvidia-adds-telemetry-to-latest-drivers) KnoxBUG Upcoming Meeting (http://knoxbug.org/2016-11-29) Feedback/Questions William - Show Music (http://pastebin.com/skvEgkLK) Ray - Mounting a Cell Phone (http://pastebin.com/nMDeSFGM) Ron - TrueOS + Radeon (http://pastebin.com/p5bC1jKU) (Follow-up - He used nvidia card) Kurt - ZFS Migration (http://pastebin.com/ud9vEK2C) Matt Dillon (Yes that Matt Dillon) - vkernels (http://pastebin.com/VPQfsUks) ***

This week on BSDNow, Allan & Kris are out at MeetBSD, but we never forget our loyal listeners. We have a great interview Allan did with Scott Long of Netflix & FreeBSD fame, as well as your questions on the place to B...SD! This episode was brought to you by Interview - Scott Long - scottl@freebsd.org (mailto:scottl@freebsd.org) FreeBSD & Netflix *** Feedback/Questions Zack - USB Config (http://pastebin.com/u77LE0Md) Jens - VMs, Jails and Containers (http://pastebin.com/8KwDK6ay) Ranko - Tarsnap Keys (http://pastebin.com/Kie3EcjN) Alex - OpenBSD in Hyper-V (http://pastebin.com/nRJQ7UPZ) Curt - Discussion Segment (http://pastebin.com/ndx25pQA)

This week on the show, we're loaded up with great stories ranging from System call fuzzing, a history of UNIX Pipes, speeding up MySQL imports and more. Stay tuned, BSDNow is coming your way right now. This episode was brought to you by Headlines System call fuzzing of OpenBSD amd64 using TriforceAFL (i.e. AFL and QEMU) (https://github.com/nccgroup/TriforceOpenBSDFuzzer) The NCCGroup did a series of fuzz testing against the OpenBSD syscall interface, during which they found a number of vulnerabilities, we covered this back in the early summer What we didn't notice, is that they also made the tools they used available. A combination of AFL (American Fuzzy Lop), QEMU, OpenBSD's FlashRD image generation tool, and the “Triforce” driver The other requirement is “a Linux box as host to run the fuzzer (other fuzzer hosts may work as well, we've only run TriforceAFL from a Linux host, specifically Debian/Ubuntu” It would be interesting to see if someone could get this to run from a BSD host It would also be interesting to run the same tests against the other BSDs *** On the Early History and Impact of Unix: the Introduction of Pipes (http://people.fas.harvard.edu/~lib113/reference/unix/unix2.html) Pipes are something we just take for granted today, but there was a time before pipes (How did anything get done?) Ronda Hauben writes up a great look back at the beginning of UNIX, and specifically at how pipes were born: One of the important developments in Unix was the introduction of pipes. Pipes had been suggested by McIlroy during the early days of creating Unix. Ritchie explains how "the idea, explained one afternoon on a blackboard, intrigued us but failed to ignite any immediate action. There were several objections to the idea as put....What a failure of imagination," he admits.(35) McIlroy concurs, describing how the initial effort to add pipes to Unix occurred about the same time in 1969 that Ritchie, Thompson and Canaday were outlining ideas for a file system. "That was when," he writes, "the simple pipeline as a way to combine programs, with data notationally propagating along a chain of (not necessarily concurrent) filters was articulated."(36) However, pipes weren't implemented in Unix until 1972. We also have a great quote from McIlroy on the day pipes were first introduced: Open Systems! Our Systems! How well those who were there remember the pipe-festooned garret where Unix took form. The excitement of creation drew people to work there amidst the whine of the computer's cool- ing fans, even though almost the same computer ac- cess, could be had from one's office or from home. Those raw quarters saw a procession of memorable events. The advent of software pipes precipitated a day-long orgy of one-liners....As people reveled in the power of functional composition in the large, which is even today unavailable to users of other systems. The paper goes on to talk about the invention of other important tools, such as “grep”, “diff” and more. Well worth your time if you want a glimpse into the history of UNIX *** Speeding up MySQL Import on FreeBSD (https://blog.feld.me/posts/2016/09/speeding-up-mysql-import-on-freebsd/) Mark Felder writes a blog post explaining how to speed up MySQL bulk data imports “I was recently tasked with rebuilding a readonly slave database server which only slaves a couple of the available databases. The backup/dump is straightforward and fast, but the restore was being excruciatingly slow. I didn't want to wait a week for this thing to finish, so I had to compile a list of optimizations that would speed up the process. This is the best way to do it on FreeBSD, assuming you're working with InnoDB. Additional optimizations may be required if you're using a different database engine.” “Please note this is assuming no other databases are running on this MySQL instance. Some of these are rather dangerous and you wouldn't want to put other live data at risk.” Most of the changes are meant to be temporary, used on a new server to import a dump of the database, then the settings are to be turned off. Specifically: sync_binlog = 0 innodbflushlogattrx_commit = 0 innodb-doublewrite = 0 He also prepends the following but of SQL before importing the data: set sqllogbin=0; set autocommit=0; set uniquechecks=0; set foreignkey_checks=0; You can also help yourself if your MySQL database lives on ZFS zfs set recordsize=16k pool/var/db/mysql zfs set redundant_metadata=most pool/var/db/mysql Remember, this tuning is ONLY for the initial import, leaving these settings on long term risks losing 5-10 seconds of your data if the server reboots unexpectedly zfs set sync=disabled pool/var/db/mysql zfs set logbias=throughput pool/var/db/mysql *** PostgreSQL and FreeBSD Quick Start (https://cwharton.com/blog/2016/10/postgresql-and-freebsd-quick-start/) There's lots of databases to choose from, but Postgres always has a special place on FreeBSD. Today we have a look at a ‘getting started' guide for those taking the plunge and using it for the first time. Naturally getting started will look familiar to many, a couple simple “pkg” and “sysrc” commands later, and you'll be set. After starting the service (With the “service” command) you'll be ready to start setting up your postgres instance. Next up you'll need to create your initial user/password combo, and a database with access granted to this particular user. If you plan to enable remote access to this DB server, you'll need to make some adjustments to one of the .conf files, allowing other IP's to connect. (If you are hosting something on the same system, this may not be needed) Now yous should be good to go! Enjoy using your brand new Postgres database. If this is your first rodeo, maybe start with something easy, like Apache or Nginx + Wordpress to try it out. *** News Roundup OpenBSD vmm hypervisor test drive (https://www.youtube.com/watch?v=KE_7E1pXy5c) As we asked for a week or two ago, someone has taken OpenBSD's vmm for a test drive, and made a video of it The command line interface for vmm, vmctl, looks quite easy to use. It takes an approach much closer to some of the bhyve management frameworks, rather than bhyve's rather confusing set of switches It also has a config file, the format of which looks very similar to what I designed for bhyveucl, and my first effort to integrate a config file into bhyve itself. The video also looks at accessing the console, configuring the networking, and doing an OpenBSD install in a fresh VM Currently vmm only supports running OpenBSD VMs *** FreeBSD Foundation October 2016 Update (https://www.freebsdfoundation.org/wp-content/uploads/2016/10/FreeBSD-Foundation-October-2016-Update.pdf) Wow, November is already upon us with the Holidays just around the corner. Before things get lost in the noise we wanted to highlight this update from the FreeBSD foundation. Before getting into the stories, they helpfully provide a list of upcoming conferences for this fall/winter, which includes a couple of USENIX gatherings, and the Developer Summit / MeetBSD next week. +The foundation gives us a quick hardware update initially, discussing some of the new ThunderX Cavium servers which are deployed (ARMv8 64Bit) and yes I'm drooling a bit. They also mention that work is ongoing for the RPi3 platform and PINE64. GNN also has an article reprinted from the FreeBSD journal, talking about the achievement of making it to 11.0 over the span of 23 years now. Of course he mentions that the foundation is open to all, and welcomes donations to continue to keep up this tradition of good work being done. Deb Goodkin gives us an update on the “Grace Hopper” convention that took place in Houston TX several weeks back. Roughly 14k women in Tech attended, which is a great turnout, and FreeBSD was well represented there. Next we have a call to potential speakers, don't forget that there are plenty of places you can help present about FreeBSD, not just at *BSD centered conferences, but the SCALES of the world as well. We wrap up with a look at EuroBSDCon 2016, quite a nice writeup, again brought to us by Deb at the foundation, and includes a list of some of those recognized for their contributions to FreeBSD. *** Adhokku – a toy PaaS powered by FreeBSD jails and Ansible (https://github.com/adhokku/adhokku) Described as a toy Platform-as-a-Service, Adhokku is an ansible based automated jail creation framework Based on the concept of Dokku, a single-host open source PaaS for Linux powered by Docker When you deploy an application using Adhokku, Adhokku creates a new jail on the remote host and provisions it from a fixed clean state using the instructions in the Jailfile in your Git repository. All jails sit behind a reverse proxy that directs traffic to one of them based on the domain name or the IP address in the HTTP request. When a new jail has been provisioned for an application, Adhokku seamlessly reconfigures the reverse proxy to send traffic to it instead of the one currently active for that application. The following instructions show how to get Adhokku and an example application running in a VM on your development machine using Vagrant. This process should require no FreeBSD-specific knowledge, through modifying the Jailfile to customize the application may. This seems like an interesting project, and it is good to see people developing workflows so users familiar with docker etc, can easily use BSD instead *** Installing OpenBSD 6.0 on your laptop is really hard (not) (http://sohcahtoa.org.uk/openbsd.html) OpenBSD on a laptop? Difficult? Not hardly. We have a great walkthrough by Keith Burnett, which demonstrates just how easy it can be to get up and running with an XFCE desktop from a fresh OpenBSD installation. For those curious,this was all done with a Thinkpad X60 and 120GB SSD and OpenBSD 6.0. He doesn't really cover the install process itself, that is well covered by the link to the OpenBSD FAQ pages. Once the system is up and running though, we start with the most important portion, getting working internet access (Via wifi) Really just a few ‘ifconfig' commands later and we are in business. Step 2 was getting the package configuration going. (I've never understood why this is still a thing, but no fret, its easy enough to do) With package repos available, now you can grab the binaries for XFCE and friends with just a few simple “pkg_add” commands Steps 4-6 are some specific bits to enable XFCE services, and some handy things such as setting doas permissions to get USB mounting working (For graphical mount/unmount) Lastly, keeping the system updated is important, so we have a nice tutorial on how to do that as well, using a handy “openup” script, which takes some of the guesswork out of it. Bonus! Steps for doing FDE as also included, which isn't for everyone, but you may want it *** Beastie Bits Pi-top with RPi-3 and FreeBSD HEAD (https://twitter.com/gvnn3/status/791475373380804608) NetBSD 7.0.2 released (http://blog.netbsd.org/tnf/entry/netbsd_7_0_1_released1) DragonflyBSD - git: kernel - Fix mmcsd read/write issues (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624851.html) A char device which implements an Enigma machine (FreeBSD & Linux) (https://github.com/rafael-santiago/dev-enigma) *** Feedback/Questions Matt - System Monitoring (http://pastebin.com/ayzvCuaq) Tony - LLVM License (http://pastebin.com/r5axPSE7) Ben - Thanks (http://pastebin.com/MNxCvUtX) David - Write Cache (http://pastebin.com/RswFASqW) Charles - Fonts (http://pastebin.com/e317a32f) ***

This week on BSDNow, we've got voting news for you (No not that election), a closer look at This episode was brought to you by Headlines ARIN 38 involvement, vote! (http://lists.nycbug.org/pipermail/talk/2016-October/016878.html) Isaac (.Ike) Levy, one of our interview guests from earlier this year, is running for a seat on the 15 person ARIN Advisory Council His goal is to represent the entire *BSD community at this important body that makes decisions about how IP addresses are allocated and managed Biographies and statements for all of the candidates are available here (https://www.arin.net/participate/elections/candidate_bios.pdf) The election ends Friday October 28th If elected, Ike will be looking for input from the community *** LibreSSL not just available but default (DragonFlyBSD) (https://www.dragonflydigest.com/2016/10/19/18794.html) DragonFly has become the latest BSD to join the growing LibreSSL family. As mentioned a few weeks back, they were in the process of wiring it up as a replacement for OpenSSL. With this latest commit, you can now build the entire base and OpenSSL isn't built at all. Congrats, and hopefully more BSDs (and Linux) jump on the bandwagon Compat_43 is gone (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624734.html) RiP 4.3 Compat support.. Well for DragonFly anyway. This commit finally puts out to pasture the 4.3 support, which has been disabled by default in DragonFly for almost 5 years now. This is a nice cleanup of their tree, removing more than a thousand lines of code and some of the old cruft still lingering from 4.3. *** Create your first FreeBSD kernel module (http://meltmes.kiloreux.me/create-your-first-freebsd-kernel-module/) This is an interesting tutorial from Abdelhadi Khiati, who is currently a master's student in AI and robotics I have been lucky enough to participate in Google Summer of Code with the FreeBSD foundation. I was amazed by the community surrounding it which was noob friendly and very helpful (Thank you FreeBSD We will run two storage controllers (ctrl-a, ctrl-b) and a host (cln-1). A virtual SAS drive (da0) of 256 MB is configured as “shareable” in Virtual Media Manager and simultaneously connected with both storage controllers The basic settings are applied to both controllers One interesting setting is: kern.cam.ctl.harole – configures default role for the node. So ctrl-a is set as 0 (primary node), ctrl-b – 1 (secondary node). The role also can be specified on per-LUN basis which allows to distribute LUNs over both controllers evenly. Note, kern.cam.ctl.haid and kern.cam.ctl.ha_mode are read-only parameters and must be set only via the /boot/loader.conf file. Once kern.cam.ctl.ha_peer is set, and the peers connect to each other, the log messages should reflect this: CTL: HA link status changed from 0 to 1 CTL: HA link status changed from 1 to 2 The link states can be: 0 – not configured, 1 – configured but not established and 2 – established Then ctld is configured to export /dev/da0 on each of the controllers Then the client is booted, and uses iscsid to connect to each of the exposed targets sysctl kern.iscsi.failondisconnection=1 on the client is needed to drop connection with one of the controllers in case of its failure As we know that da0 and da1 on the client are the same drive, we can put them under multipathing control: gmultipath create -A HA /dev/da0 /dev/da1 The document them shows a file being copied continuously to simulate load. Because the multipath is configured in ‘active/active' mode, the traffic is split between the two controllers Then the secondary controller is turned off, and iscsi disconnects that path, and gmultipath adapts and sends all of the traffic over the primary path. When the secondary node is brought back up, but the primary is taken down, traffic stops The console on the client is filled with errors: “Logical unit not accessible, asymmetric access state transition” The ctl(4) man page explains: > If there is no primary node (both nodes are secondary, or secondary node has no connection to primary one), secondary node(s) report Transitioning state. > Therefore, it looks like a “normal” behavior of CTL HA cluster in a case of disaster and loss of the primary node. It also means that a very lucky administrator can restore the failed primary controller before timeouts are elapsed. If the primary is down, the secondary needs to be promoted by some other process (CARP maybe?): sysctl kern.cam.ctl.ha_role=0 Then traffic follows again This is a very interesting look at this new feature, and I hope to see more about it in the future *** Is SPF Simply Too Hard for Application Developers? (http://bsdly.blogspot.com/2016/10/is-spf-simply-too-hard-for-application.html) Peter Hansteen asks an interesting question: The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers? He tells a story about trying to file his Norwegian taxes, and running into a bug Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site. The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics. The next time I checked my mail at bsdly.net, I found this bounce: support@altinn.no: SMTP error from remote mail server after RCPT TO:: host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed which means that somebody, somewhere tried to send a message to support@altinn.no, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain. What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails. After having a bunch of other problems, he finally gets a message back from the tax authority support staff: It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported. The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem. Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification? Beastie Bits Looking for a very part-time SysAdmin (https://lists.freebsd.org/pipermail/freebsd-jobs/2016-October/000930.html) If anyone wants to build the latest nodejs on OpenBSD... (https://twitter.com/qb1t/status/789610796380598272) IBM considers donating Power8 servers to OpenBSD (https://marc.info/?l=openbsd-misc&m=147680858507662&w=2) Install and configure DNS server in FreeBSD (https://galaxy.ansible.com/vbotka/freebsd-dns/) bhyve vulnerability in FreeBSD 11.0 (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:32.bhyve.asc) Feedback/Questions Larry - Pkg Issue (http://pastebin.com/8hwDVQjL) Larry - Followup (http://pastebin.com/3nswwk90) Jason - TrueOS (http://pastebin.com/pjfYWdXs) Matias - ZFS HALP! (http://pastebin.com/2tAmR5Wz) Robroy - User/Group (http://pastebin.com/7vWvUr8K) ***

This week on the show, we've got all sorts of goodies to discuss. Starting with, vmm, vkernels, raspberry pi and much more! Some iX folks are visiting from out of This episode was brought to you by Headlines vmm enabled (http://undeadly.org/cgi?action=article&sid=20161012092516&mode=flat&count=15) VMM, the OpenBSD hypervisor, has been imported into current It has similar hardware requirements to bhyve, a Intel Nehalem or newer CPU with the hardware virtualization features enabled in the BIOS AMD support has not been started yet OpenBSD is the only supported guest It would be interesting to hear from viewers that have tried it, and hear how it does, and what still needs more work *** vkernels go COW (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624675.html) The DragonflyBSD feature, vkernels, has gained a new Copy-On-Write functionality Disk images can now be mounted RO or RW, but changes will not be written back to the image file This allows multiple vkernels to share the same disk image “Note that when the vkernel operates on an image in this mode, modifications will eat up system memory and swap, so the user should be cognizant of the use-case. Still, the flexibility of being able to mount the image R+W should not be underestimated.” This is another feature we'd love to hear from viewers that have tried it out. *** Basic support for the RPI3 has landed in FreeBSD-CURRENT (https://wiki.freebsd.org/arm64/rpi3) The long awaited bits to allow FreeBSD to boot on the Raspberry Pi 3 have landed There is still a bit of work to be done, some of the as mentioned in Oleksandr's blog post: Raspberry Pi support in HEAD (https://kernelnomicon.org/?p=690) “Raspberry Pi 3 limited support was committed to HEAD. Most of drivers should work with upstream dtb, RNG requires attention because callout mode seems to be broken and there is no IRQ in upstream device tree file. SMP is work in progress. There are some compatibility issue with VCHIQ driver due to some assumptions that are true only for ARM platform. “ This is exciting work. No HDMI support (yet), so if you plan on trying this out make sure you have your USB->Serial adapter cables ready to go. Full Instructions to get started with your RPI 3 can be found on the FreeBSD Wiki (https://wiki.freebsd.org/arm64/rpi3) Relatively soon, I imagine there will be a RaspBSD build for the RPI3 to make it easier to get started Eventually there will be official FreeBSD images as well *** OpenBSD switches softraid crypto from PKCS5 PBKDF2 to bcrypt PBKDF. (https://github.com/openbsd/src/commit/2ba69c71e92471fe05f305bfa35aeac543ebec1f) After the discussion a few weeks ago when a user wrote a tool to brute force their forgotten OpenBSD Full Disk Encryption password (from a password list of possible variations of their password), it was discovered that OpenBSD defaulted to using just 8192 iterations of PKCSv5 for the key derivation function with a SHA1-HMAC The number of iterations can be manually controlled by the user when creating the softraid volume By comparison, FreeBSDs GELI full disk encryption used a benchmark to pick a number of iterations that would take more than 2 seconds to complete, generally resulting in a number of iterations over 1 million on most modern hardware. The algorithm is based on a SHA512-HMAC However, inefficiency in the implementation of PKCSv5 in GELI resulted in the implementation being 50% slower than some other implementations, meaning the effective security was only about 1 second per attempt, rather than the intended 2 seconds. The improved PKCSv5 implementation is out for review currently. This commit to OpenBSD changes the default key derivation function to be based on bcrypt and a SHA512-HMAC instead. OpenBSD also now uses a benchmark to pick a number of of iterations that will take approximately 1 second per attempt “One weakness of PBKDF2 is that while its number of iterations can be adjusted to make it take an arbitrarily large amount of computing time, it can be implemented with a small circuit and very little RAM, which makes brute-force attacks using application-specific integrated circuits or graphics processing units relatively cheap. The bcrypt key derivation function requires a larger amount of RAM (but still not tunable separately, i. e. fixed for a given amount of CPU time) and is slightly stronger against such attacks, while the more modern scrypt key derivation function can use arbitrarily large amounts of memory and is therefore more resistant to ASIC and GPU attacks.” The upgrade to the bcrypt, which has proven to be quite resistant to cracking by GPUs is a significant enhancement to OpenBSDs encrypted softraid feature *** Interview - Josh Paetzel - email@email (mailto:email@email) / @bsdunix4ever (https://twitter.com/bsdunix4ever) MeetBSD ZFS Panel FreeNAS - graceful network reload Pxeboot *** News Roundup EC2's most dangerous feature (http://www.daemonology.net/blog/2016-10-09-EC2s-most-dangerous-feature.html) Colin Percival, FreeBSD's unofficial EC2 maintainer, has published a blog post about “EC2's most dangerous feature” “As a FreeBSD developer — and someone who writes in C — I believe strongly in the idea of "tools, not policy". If you want to shoot yourself in the foot, I'll help you deliver the bullet to your foot as efficiently and reliably as possible. UNIX has always been built around the idea that systems administrators are better equipped to figure out what they want than the developers of the OS, and it's almost impossible to prevent foot-shooting without also limiting useful functionality. The most powerful tools are inevitably dangerous, and often the best solution is to simply ensure that they come with sufficient warning labels attached; but occasionally I see tools which not only lack important warning labels, but are also designed in a way which makes them far more dangerous than necessary. Such a case is IAM Roles for Amazon EC2.” “A review for readers unfamiliar with this feature: Amazon IAM (Identity and Access Management) is a service which allows for the creation of access credentials which are limited in scope; for example, you can have keys which can read objects from Amazon S3 but cannot write any objects. IAM Roles for EC2 are a mechanism for automatically creating such credentials and distributing them to EC2 instances; you specify a policy and launch an EC2 instance with that Role attached, and magic happens making time-limited credentials available via the EC2 instance metadata. This simplifies the task of creating and distributing credentials and is very convenient; I use it in my FreeBSD AMI Builder AMI, for example. Despite being convenient, there are two rather scary problems with this feature which severely limit the situations where I'd recommend using it.” “The first problem is one of configuration: The language used to specify IAM Policies is not sufficient to allow for EC2 instances to be properly limited in their powers. For example, suppose you want to allow EC2 instances to create, attach, detach, and delete Elastic Block Store volumes automatically — useful if you want to have filesystems automatically scaling up and down depending on the amount of data which they contain. The obvious way to do this is would be to "tag" the volumes belonging to an EC2 instance and provide a Role which can only act on volumes tagged to the instance where the Role was provided; while the second part of this (limiting actions to tagged volumes) seems to be possible, there is no way to require specific API call parameters on all permitted CreateVolume calls, as would be necessary to require that a tag is applied to any new volumes being created by the instance.” “As problematic as the configuration is, a far larger problem with IAM Roles for Amazon EC2 is access control — or, to be more precise, the lack thereof. As I mentioned earlier, IAM Role credentials are exposed to EC2 instances via the EC2 instance metadata system: In other words, they're available from http://169.254.169.254/. (I presume that the "EC2ws" HTTP server which responds is running in another Xen domain on the same physical hardware, but that implementation detail is unimportant.) This makes the credentials easy for programs to obtain... unfortunately, too easy for programs to obtain. UNIX is designed as a multi-user operating system, with multiple users and groups and permission flags and often even more sophisticated ACLs — but there are very few systems which control the ability to make outgoing HTTP requests. We write software which relies on privilege separation to reduce the likelihood that a bug will result in a full system compromise; but if a process which is running as user nobody and chrooted into /var/empty is still able to fetch AWS keys which can read every one of the objects you have stored in S3, do you really have any meaningful privilege separation? To borrow a phrase from Ted Unangst, the way that IAM Roles expose credentials to EC2 instances makes them a very effective exploit mitigation mitigation technique.” “To make it worse, exposing credentials — and other metadata, for that matter — via HTTP is completely unnecessary. EC2 runs on Xen, which already has a perfectly good key-value data store for conveying metadata between the host and guest instances. It would be absolutely trivial for Amazon to place EC2 metadata, including IAM credentials, into XenStore; and almost as trivial for EC2 instances to expose XenStore as a filesystem to which standard UNIX permissions could be applied, providing IAM Role credentials with the full range of access control functionality which UNIX affords to files stored on disk. Of course, there is a lot of code out there which relies on fetching EC2 instance metadata over HTTP, and trivial or not it would still take time to write code for pushing EC2 metadata into XenStore and exposing it via a filesystem inside instances; so even if someone at AWS reads this blog post and immediately says "hey, we should fix this", I'm sure we'll be stuck with the problems in IAM Roles for years to come.” “So consider this a warning label: IAM Roles for EC2 may seem like a gun which you can use to efficiently and reliably shoot yourself in the foot; but in fact it's more like a gun which is difficult to aim and might be fired by someone on the other side of the room snapping his fingers. Handle with care!” *** Open-source storage that doesn't suck? Our man tries to break TrueNAS (http://www.theregister.co.uk/2016/10/18/truenas_review/) The storage reviewer over at TheRegister got their hands on a TrueNAS and gave it a try “Data storage is difficult, and ZFS-based storage doubly so. There's a lot of money to be made if you can do storage right, so it's uncommon to see a storage company with an open-source model deliver storage that doesn't suck.” “To become TrueNAS, FreeNAS's code is feature-frozen and tested rigorously. Bleeding-edge development continues with FreeNAS, and FreeNAS comes with far fewer guarantees than does TrueNAS.” “iXsystems provided a Z20 hybrid storage array. The Z20 is a dual-controller, SAS-based, high-availability, hybrid storage array. The testing unit came with a 2x 10GbE NIC per controller and retails around US$24k. The unit shipped with 10x 300GB 10k RPM magnetic hard drives, an 8GB ZIL SSD and a 200GB L2ARC SSD. 50GiB of RAM was dedicated to the ARC by the system's autotune feature.” The review tests the performance of the TrueNAS, which they found acceptable for spinning rust, but they also tested the HA features While the look of the UI didn't impress them, the functionality and built in help did “The UI contains truly excellent mouseover tooltips that provide detailed information and rationale for almost every setting. An experienced sysadmin will be able to navigate the TrueNAS UI with ease. An experienced storage admin who knows what all the terms mean won't have to refer to a wiki or the more traditional help manual, but the same can't be said for the uninitiated.” “After a lot of testing, I'd trust my data to the TrueNAS. I am convinced that it will ensure the availability of my data to within any reasonable test, and do so as a high availability solution. That's more than I can say for a lot of storage out there.” “iXsystems produce a storage array that is decent enough to entice away some existing users of the likes of EMC, NetApp, Dell or HP. Honestly, that's not something I thought possible going into this review. It's a nice surprise.” *** OpenBSD now officially on GitHub (https://github.com/openbsd) Got a couple of new OpenBSD items to bring to your attention today. First up, for those who didn't know, OpenBSD development has (always?) taken place in CVS, similar to NetBSD and previously FreeBSD. However today, Git fans can rejoice, since there is now an “official” read-only github mirror of their sources for public consumption. Since this is read-only, I will assume (unless told otherwise) that pull-requests and whatnot aren't taken. But this will come in handy for the “git-enabled” among us who need an easier way to checkout OpenBSD sources. There is also not yet a guarantee about the stability of the exporter. If you base a fork on the github branch, and something goes wrong with the exporter, the data may be reexported with different hashes, making it difficult to rebase your fork. How to install LibertyBSD or OpenBSD on a libreboot system (https://libreboot.org/docs/bsd/openbsd.html) For the second part of our OpenBSD stories, we have a pretty detailed document posted over at LibreBoot.org with details on how to boot-strap OpenBSD (Or LibertyBSD) using their open-source bios replacement. We've covered blog posts and other tidbits about this process in the past, but this seems to be the definitive version (so far) to reference. Some of the niceties include instructions on getting the USB image formatted not just on OpenBSD, but also FreeBSD, Linux and NetBSD. Instructions on how to boot without full-disk-encryption are provided, with a mention that so far Libreboot + Grub does not support FDE (yet). I would imagine somebody will need to port over the openBSD FDE crypto support to GRUB, as was done with GELI at some point. Lastly some instructions on how to configure grub, and troubleshoot if something goes wrong will help round-out this story. Give it a whirl, let us know if you run into issues. Editorial Aside - Personally I find the libreboot stuff fascinating. It really is one of the last areas that we don't have full control of our systems with open-source. With the growth of EFI, it seems we rely on a closed-source binary / mini-OS of sorts just to boot our Open Source solutions, which needs to be addressed. Hats off to the LibreBoot folks for taking on this important challenge. *** FreeNAS 9.10 – LAGG & VLAN Overview (https://www.youtube.com/watch?v=wqSH_uQSArQ) A video tutorial on FreeNAS's official YouTube Channel Covers the advanced networking features, Link Aggregation and VLANs Covers what the features do, and in the case of LAGG, how each of the modes work and when you might want to use it *** Beastie Bits Remote BSD Developer Position is up for grabs (https://www.cybercoders.com/bsd-developer-remote-job-305206) Isilon is hiring for a FreeBSD Security position (https://twitter.com/jeamland/status/785965716717441024) Google has ported the Networked real-time multi-player BSD game (https://github.com/google/web-bsd-hunt) A bunch of OpenBSD Tips (http://www.vincentdelft.be) The last OpenBSD 6.0 Limited Edition CD has sold (http://www.ebay.com/itm/-/332000602939) Dan spots George Neville-Neil on TV at the Airport (https://twitter.com/DLangille/status/788477000876892162) gnn on CNN (https://www.youtube.com/watch?v=h7zlxgtBA6o) SoloBSD releases v 6.0 built upon OpenBSD (http://solobsd.blogspot.com/2016/10/release-solobsd-60-openbsd-edition.html) Upcoming KnoxBug looks at PacBSD - Oct 25th (http://knoxbug.org/content/2016-10-25) Feedback/Questions Morgan - Ports and Packages (http://pastebin.com/Kr9ykKTu) Mat - ZFS Memory (http://pastebin.com/EwpTpp6D) Thomas - FreeBSD Path Length (http://pastebin.com/HYMPtfjz) Cy - OpenBSD and NetHogs (http://pastebin.com/vGxZHMWE) Lars - Editors (http://pastebin.com/5FMz116T) ***

The wait is over, 11.0 of FreeBSD has (officially) launched. We'll have coverage of this, plus a couple looks back at UNIX history, and a crowd-favorite guest today. This episode was brought to you by Headlines FreeBSD 11.0-RELEASE Now Available (https://lists.freebsd.org/pipermail/freebsd-announce/2016-October/001760.html) FreeBSD 11.0-RELEASE is now officially out. A last minute reroll to pickup OpenSSL updates and a number of other security fixes meant the release was a little behind schedule, and shipped as 11.0-RELEASE-p1, but the release is better for it Improved support for 802.11n and various wifi drivers Support for the AArch64 (arm64) architecture has been added. Native graphics support has been added to the bhyve(8) hypervisor. A new flag, “onifconsole” has been added to /etc/ttys. This allows the system to provide a login prompt via serial console if the device is an active kernel console, otherwise it is equivalent to off. The xz(1) utility has been updated to support multi-threaded compression. A number of kernel panics related to VNET have been fixed The IMAGACT_BINMISC kernel configuration option has been enabled by default, which enables application execution through emulators, such as QEMU via binmiscctl(8). The GENERIC kernel configuration has been updated to include the IPSEC option by default. The kern.osrelease and kern.osreldate are now configurable jail(8) parameters A new sysctl(8), kern.racct.enable, has been added, which when set to a non-zero value allows using rctl(8) with the GENERIC kernel. A new kernel configuration option, RACCT_DISABLED has also been added. The minimum (arcmin) and maximum (arcmax) values for the ZFS adaptive replacement cache can be modified at runtime. Changes to watch out for: OpenSSH DSA key generation has been disabled by default. It is important to update OpenSSH keys prior to upgrading. Additionally, Protocol 1 support has been removed. By default, the ifconfig(8) utility will set the default regulatory domain to FCC on wireless interfaces. As a result, newly created wireless interfaces with default settings will have less chance to violate country-specific regulations. An issue was discovered with Amazon® EC2™ images which would cause the virtual machine to hang during boot when upgrading from previous FreeBSD versions. New EC2™ installations are not affected, but existing installations running earlier releases are advised to wait until the issue is resolved in an Errata Notice before upgrading. An Errata Notice to address this is planned following the release. *** process listing consistency (http://www.tedunangst.com/flak/post/process-listing-consistency) Ted Unangst asks: how consistent is the output of ps(1)? If processes are starting and exiting constantly, and you run ps(1), is the output guaranteed to reflect that exact moment in time, or might it include some processes that have gone away before ps(1) exited, and include some processes that did not exist when ps(1) was started? Ted provides a little example chicken/egg program to try to create such an inconsistency, so you can test out your OS On OpenBSD ps(1) was switched away from the reading kernel memory directly, and instead uses the KERNPROCALL sysctl Thus sysctl can iterate over the entire process list, copying out information to ps(1), without blocking. If we prevent processes from forking or exiting during this time, we get a consistent snapshot. The snapshot may be stale, but it will never show us a viewpoint that never happened. So, OpenBSD will always be consistent, or will it? Is there a way to trick ps on OpenBSD? Not everything is consistent. There's a separate sysctl, KERNPROCARGV, that reads the command line arguments for a process, but it only works on one process at a time. Processes can modify their own argv at any time. A second test program changes the process title of both the chicken and the egg, and if you run ps(1), you can get back a result that never actually happened. The argv of the first program is read by ps(1), and in the meantime, it changes to a different value. The second program also changes its value, so now when ps(1) reads it, it sees the new value, not the original value from when ps(1) was started. So the output is not that consistent, but is it worth the effort to try to make it so? DragonFlyBSD - if_iwm - Add basic powermanagement support via ifconfig wlan0 powersave (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624673.html) WiFi can often be one of the biggest drains on your laptop battery, so anything we can do to improve the situation should be embraced. Imre Vadász over at the DragonFly project has done that, porting over a new set of power management support from Linux to the if_iwm driver. if_iwm - Add basic powermanagement support via ifconfig wlan0 powersave. The DEVICEPOWERFLAGSCAMMSK flag was removed in the upstream iwlwifi in Linux commit ceef91c89480dd18bb3ac51e91280a233d0ca41f. Add scpsdisabled flag to struct iwmsoftc, which corresponds to mvm->psdisabled in struct iwl_mvm in Linux iwlwifi. Adds a hw.iwm.powerscheme tunable which corresponds to the powerscheme module parameter in Linux iwlwifi. Set this to 1 for completely disabling power management, 2 (default) for balanced powermanagement, and 3 for lowerpower mode (which does dtim period skipping). Imports the constants.h file from iwlwifi as ifiwmconstants.h. This doesn't allow changing the powermanagement setting while connected, also one can only choose between enabled and disabled powersaving with ifconfig (so switching between balanced and low-power mode requires rebooting to change the tunable). After any changes to powermanagement (i.e. "ifconfig wlan0 powersave" to enable powermanagement, or "ifconfig wlan0 -powersave" for disabling powermanagement), one has to disconnect and reconnect to the accespoint for the change to take effect.“ Good stuff! These positive changes need to happen more often and sooner, so we can all eek out every drop of power from our respective laptops. *** Helping out an Internet Friend…Dual boot OpenBSD (https://functionallyparanoid.com/2016/10/03/helping-out-an-internet-friend/) Dual-booting OpenBSD and Linux, via UEFI. A year ago we wouldn't be discussing this, but today we have an article where somebody has done exactly that. This Journey was undertaken by Brian Everly (Indiana Bug), partly due to a friend who wanted to dual-boot his laptop which already has an existing UEFI install on it. As a proof of concept, he began by replicating the setup in VMware with UEFI He started by throwing Ubuntu into the VM, with some special attention paid to partitioning to ensure enough room left-over for OpenBSD later. I created a 64MB EFI partition at the front of the disk. Next, I created a 20GB primary partition at the beginning of the space, mounted as the root (/) filesystem. I then added a 4096MB swap partition for Ubuntu. Finally, I used the rest of the free space to create a Reserved BIOS Boot Area FAT32 partition that was not associated with a mount point – this is where I will be installing OpenBSD. With that done, he wrapped up the Ubuntu installation and then turned over to to the OpenBSD side. Some manual partitioning was required to install to the “Reserved FAT32” partition. I mashed through the defaults in the OpenBSD installer until I got to the disk partitioning. Since I told VMWare to make my hard drive an IDE one, I knew I was playing around with wd0 and not sd0 (my USB key). I dumped into fdisk by selecting to (E)dit the partition scheme and saw my setup from Linux. First was the EFI partition (I am guessing I'll have to copy my bootx64.efi file to that at some point), second was the Linux etx4 partition, third was my Linux swap partition and fourth was a weird looking one that is the “Reserved BIOS Boot” partition. That's the one I'll fiddle with. Issuing the command “edit 3” allowed me to fiddle with that partition #3 (remember, we start counting at zero). I set it's type to “A6” (OpenBSD) and then took the defaults with the exception of naming it “OpenBSD”. A quick “write” followed by a “quit” allowed me to update my new partition and get back to the installer. Once the installation was wrapped up (OpenBSD helpfully already created the /boot/EFI partition with the correct EFI loader installed) he was able to reboot and select between the two systems at the UEFI bios screen. For kicks, he lastly went into Ubuntu and grabbed refind. Installing refind provided a fancy graphical selector between the two systems without too much trouble. Next step will be to replicate this process on his friend's laptop. Wishing you luck with that journey! Interview - Bryan Cantrill - email@email (mailto:email@email) / @twitter (https://twitter.com/user) CTO of Joyent *** News Roundup After 22 Years, 386BSD Gets An Update (https://bsd.slashdot.org/story/16/10/09/0230203/after-22-years-386bsd-gets-an-update) Slashdot brings us an interesting mention this week, specifically that after 22 years, we now have an update to 386BSD. 386BSD was last released back in 1994 with a series of articles in Dr. Dobb's Journal -- but then developers for this BSD-based operating system started migrating to both FreeBSD and NetBSD. An anonymous Slashdot reader writes: The last known public release was version 0.1. Until Wednesday, when Lynne Jolitz, one of the co-authors of 386BSD, released the source code to version 1.0 as well as 2.0 on Github. 386BSD takes us back to the days when you could count every file in your Unix distribution and more importantly, read and understand all of your OS source code. 386BSD is also the missing link between BSD and Linux. One can find fragments of Linus Torvalds's math emulation code in the source code of 386BSD. To quote Linus: "If 386BSD had been available when I started on Linux, Linux would probably never had happened.” Though it was designed for Intel 80386 microprocessors, there's already instructions for launching it on the hosted hardware virtualization service Qemu. There you have it! Go grab the new hotness that is 386BSD and run it in 2016! Or perhaps you want FreeBSD 11, but to each their own. *** Progress of the OpenBSD Limited Edition Signed CD set (http://undeadly.org/cgi?action=article&sid=20160929230557&mode=expanded) An update from a story last week! We mentioned the “very” limited edition OpenBSD 6.0 signed CD sets that had gone up for Auction on Ebay. (With proceeds to support for Foundation) As of today, here's where we stand: CD set #1 (Sep 29th + 5 days) sold for $4200 (http://www.ebay.com/itm/-/331985953783) CD set #2 (Oct 4th + 3 days) sold for $3000 (http://www.ebay.com/itm/-/331990536246) CD set #3 (Oct 8th + 3 days) sold for $817 (http://www.ebay.com/itm/-/331994217419) CD set #4 (Oct 11th + 3 days) is currently up for bidding (http://www.ebay.com/itm/-/331997031152) There you have it! The 4th set is almost wrapped up bidding, and the 5th and last set is not far behind. Be sure to grab your piece of BSD history before its gone! PROTOTYPE FreeBSD Jail/ZFS based implementation of the Application Container Specification (https://github.com/3ofcoins/jetpack) “Jetpack is an experimental and incomplete implementation of the App Container Specification for FreeBSD. It uses jails as isolation mechanism, and ZFS for layered storage.” “This document uses some language used in Rocket, the reference implementation of the App Container Specification. While the documentation will be expanded in the future, currently you need to be familiar at least with Rocket's README to understand everything.” + A standard with multiple implementations, that allow substitution of components, such as FreeBSD Jails instead of docker/lxc etc, and ZFS instead of overlayfs etc, is very exciting Microsoft's Forgotten Unix-based Operating System (https://fossbytes.com/xenix-history-microsoft-unix-operating-system/) Do you remember the good old days. You know, when Microsoft was the driving force behind UNIX? Wait, what did you say you may be thinking? It's true, and lets sit back and let FossBytes tell us a tale of what once was reality. The story begins sometime in the late 70's: Turning back the pages to the late 1970's, Microsoft entered into an agreement with AT&T Corporation to license Unix from AT&T. While the company didn't sell the OS to public, it licensed it to other OEM vendors like Intel, SCO, and Tandy. As Microsoft had to face legal trouble due to “Unix” name, the company renamed it and came up with its own Unix distribution. So, AT&T licensed Unix to Redmond that was passed on to other OEMs as Xenix. It's interesting to recall a time when Microsoft enabled people to run Unix — an operating system originally designed for large and multiuser systems — on a microcomputer. Even though it came first, Unix was probably more powerful than MS-DOS. So whatever happened to this microsoft-flavored UNIX you may ask? Sadly it was ditched for DOS due to $REASONS: In early 1980's, IBM was looking for an OS to power its PC. As IBM didn't want to maintain any ties with the recently split AT&T, Xenix was automatically rejected. To fulfill, the tech giant's demand, Microsoft bought 86-DOS from Seattle Computer Products and managed to convince IBM to use it in their systems. Slowly, Microsoft started losing interest in Xenix and traded the full rights of Xenix with SCO, a Xenix partner company. The company filed bankruptcy in 2007 before taking the Xenix legacy to the 21st century in the form of Open Server, previously known as SCO Unix and SCO Open Desktop. An interesting chapter in UNIX history to be sure, and funny enough may come full-circle someday with Microsoft beginning to show interest in UNIX and BSD once again. *** Beastie Bits Ohio LinuxFest 2016 wrap-up (http://blather.michaelwlucas.com/archives/2791) Learn X in Y minutes Where X=zfs (https://learnxinyminutes.com/docs/zfs/) Add touchscreen support for the official 7" RPi touch display (https://svnweb.freebsd.org/base?view=revision&revision=306430) 64-bit U-Boot on Raspberry Pi 3 (https://kernelnomicon.org/?p=682) SNIA SDC 2016 Recap: Michael Dexter (https://www.ixsystems.com/blog/snia-sdc-2016-recap-michael-dexter/) OpenZFS: Stronger than ever (https://www.ixsystems.com/blog/openzfs-devsummit-2016/) Accurate, Traceable, and Verifiable Time Synchronization for World Financial Markets (http://nvlpubs.nist.gov/nistpubs/jres/121/jres.121.023.pdf) ON HOLY WARS AND A PLEA FOR PEACE (https://www.ietf.org/rfc/ien/ien137.txt) Feedback/Questions Morgan - Zero-Filling an VM (http://pastebin.com/CYcqmW7P) Charlie - ZFS Bit-Rot (http://pastebin.com/12mNW57h) Matias - TrueOS / Launchd (http://pastebin.com/NfYWt2cu) Dale - DO Feedback (http://pastebin.com/UvKh2WcF) James - DO / FreeBSD Locks? (http://pastebin.com/0cdMc88U) ***

This week on the show, we'll be talking to Petra about the NetBSD foundation, about how they operate and assist NetBSD behind the scenes. That plus lots of news This episode was brought to you by Headlines What is new on EC2 for FreeBSD 11.0-RELEASE (http://www.daemonology.net/blog/2016-10-03-FreeBSD-EC2-11-0-RELEASE.html) “FreeBSD 11.0-RELEASE is just around the corner, and it will be bringing a long list of new features and improvements — far too many for me to list here. I think there are some improvements in FreeBSD 11.0 which are particularly noteworthy for EC2 users.” “First, the EC2 Console Screenshot functionality now works with FreeBSD. This provides a "VGA" output as opposed to the traditional "serial port" which EC2 has exposed as "console output" for the past decade, and is useful largely because the "VGA" output becomes available immediately whereas the "serial port" output can lag by several minutes. This improvement is a simple configuration change — older releases didn't waste time writing to a non-serial console because it didn't go anywhere until Amazon added support on their side — and can be enabled on older FreeBSD releases by changing the line console="comconsole" to boot_multicons="YES" in /boot/loader.conf.” “The second notable change is support for EC2 "Enhanced Networking" using Intel 82599 hardware; on the C3, C4, R3, I2, D2, and M4 (excluding m4.16xlarge) families, this provides increased network throughput and reduced latency and jitter, since it allows FreeBSD to talk directly to the networking hardware rather than via a Xen paravirtual interface. Getting this working took much longer than I had hoped, but the final problem turned out not to be in FreeBSD at all — we were tickling an interrupt-routing bug in a version of Xen used in EC2. Unfortunately FreeBSD does not yet have support for the new "Elastic Network Adapter" enhanced networking used in P2 and X1 instance families and the m4.16xlarge instance type; I'm hoping that we'll have a driver for that before FreeBSD 11.1 arrives.” “The third notable change is an improvement in EC2 disk throughput. This comes thanks to enabling indirect segment I/Os in FreeBSD's blkfront driver; while the support was present in 10.3, I had it turned off by default due to performance anomalies on some EC2 instances. (Those EC2 performance problems have been resolved, and disk I/O performance in EC2 on FreeBSD 10.3 can now be safely improved by removing the line hw.xbd.xbdenableindirect="0" from /boot/loader.conf.)” “Finally, FreeBSD now supports all 128 CPUs in the x1.32xlarge instance type. This improvement comes thanks to two changes: The FreeBSD default kernel was modified in 2014 to support up to 256 CPUs (up from 64), but that resulted in a (fixed-size) section of preallocated memory being exhausted early in the boot process on systems with 92 or more CPUs; a few months ago I changed that value to tune automatically so that FreeBSD can now boot and not immediately panic with an out-of-the-box setup on such large systems.” “I think FreeBSD/EC2 users will be very happy with FreeBSD 11.0-RELEASE; but I'd like to end with an important reminder: No matter what you might see on FTP servers, in EC2, or available via freebsd-update, the new release has not been released until you see a GPG-signed email from the release engineer. This is not just a theoretical point: In my time as a FreeBSD developer I've seen multiple instances of last-minute release re-rolls happening due to problems being discovered very late, so the fact that you can see bits doesn't necessarily mean that they are ready to be downloaded. I hope you're looking forward to 11.0-RELEASE, but please be patient.” *** Upgrading Amazon EC2 instance from 10.3 to 11.0-PRERELEASE results in hang at boot (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213196) As if to underscore that last point, a last minute bug was found on sunday night A user reported that they used freebsd-update to upgrade an EC2 instance from 10.3 to 11.0 and it started hanging during boot After some quick investigation by Colin, the problem was reproduced Since I had done a lot of work in the loader recently, I helped Colin build a version of the loader with a lot of the debugging enabled, and some more added to try to isolate where in the loader the freeze was happening Colin and I worked late into the night, but eventually found the read from disk that was causing the hang Unlike most of the other reads, that were going into the heap, this read was into a very low memory address, right near the 640kb border. This initially distracted us from the real cause of the problem With more debugging added, it was determined that the problem was in the GELIBoot code, when reading the last sector of each partition to determine if it is encrypted. In cases where the partition is not 4k aligned, and butts up against the end of the disk, the formula used could result in a read past the end of the disk The formula rounds the last sector byte address down to the nearest factor of 4096, then reads 4096 bytes. Then that buffer is examined to determine if the partition is encrypted. If it is a 512b sector drive, the metadata will be in the last 512 bytes of that 4096 byte buffer. However, if the partition is not 4k aligned, the rounding will produce a value that is less than 4096 bytes from the end of the disk, and attempting to read 4096 bytes, will read past the end of the disk Normally this isn't that big of a problem, the BIOS will just return an error. The loader will retry up to three times, then give up and move on, continuing to boot normally. Some BIOSes are buggy, and will initiate their own retries, and the combination might result in a stall of up to 30 seconds for each attempt to read past the end of the disk But it seems that Amazon EC2 instances, (and possibly other virtual instances), will just hang in this case. This bug has existed for 6 months, but was not caught because almost all installations are 4k aligned thanks to changes made to the installer over the last few years, and most hardware continues to boot with no sign of a problem Even the EC2 snapshot images of 11.0 do not have the problem, as they use a newer disk layout that is 4k aligned by default now. The problem only seems to happen when older disk images are upgraded The fix has been committed and will be merged the the branches over the next few days An Errata notice will be issues, and the fix will be available via freebsd-update It is recommended that EC2 users, and anyone who wants to be especially cautious, wait until this errata notice goes out before attempting to upgrade from FreeBSD 10.3 to 11.0 You can determine if your partitions are 4k aligned by running ‘gpart show'. If there is free space after your last partition, you won't have any issues. *** OpenBSD 6.0 Limited Edition CD set (signed by developers) (http://undeadly.org/cgi?action=article&sid=20160929230557&mode=expanded) The first one went for .$4,200.00 (http://www.ebay.com/itm/-/331985953783) + Looking for your piece of OpenBSD history? At the recent g2k16 hackathon in Cambridge UK, 40 OpenBSD developers put pen to paper and signed 5 copies of the new 6.0 release. + Each of these will be auctioned off on ebay, with the proceeds to benefit the OpenBSD foundation. + The first auction has already ended, and CD set went for a whopping $4200! + The next set only has 2 days left, and currently stands at $3000! (http://www.ebay.com/itm/-/331990536246) + Get your bids in soon, these are VERY unique, the odds of getting the same 40 developers in a room together and signing a new .0 release may make this a once-in-a-lifetime opportunity. + Additionally, if you are just starting your OpenBSD collection, here's a nice image to make you envious: A nice collection of OpenBSD CD Sets (http://i.imgur.com/OrE0Gsa.png) [What typing ^D really does on Unix ](https://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffects) + How often have you used a ^D to generate an EOF? Do you really know what that does? + Chris Siebenmann has posted a look at this on his blog, which might not be what you think “Typing ^D causes the tty driver to immediately finish a read().” He continues on: Normally doing a read() from a terminal is line-buffered inside the tty driver; your program only wakes up when the tty driver sees the newline, at which point you get back the full line. (Note that this buffering is distinct from anything that your language's IO system may be doing.) Typing ^D causes the tty driver to stop waiting for a newline and immediately return from the read() with however much of the line has been accumulated to date. If you haven't typed anything on the line yet, there is nothing accumulated and the read() will return 0 bytes, which is conveniently the signal for end of file. If you have typed something the program will get it; because it doesn't have a trailing newline, the program's own line-buffering may take over and keep read()ing to get the rest of the line. (Other programs will immediately process the partial line with no buffering; cat is one example of this.) Once you've typed ^D on a partial line, that portion of the line is immutable because it's already been given to the program. Most Unixes won't let you backspace over such partial lines; effectively they become output, not input. (Note that modern shells are not good examples of this, because they don't do line-buffered input; to support command line editing, they switch terminal input into an uninterpreted mode. So they get the raw ^D and can do whatever they want with it, and they can let you edit as much of the pending line as they want.) Fascinating stuff, and interesting to see behind the curtain at exactly what's going on with your programs buffering and tty driver interaction. Interview - Petra Zeidler - spz@netbsd.org (mailto:spz@netbsd.org) NetBSD Foundation *** News Roundup Running FreeBSD in Travis-CI Thanks to KQEmu (http://erouault.blogspot.com/2016/09/running-freebsd-in-travis-ci.html) Travis-CI is the most popular testing framework on Github, but it doesn't support any of the BSDs This didn't discourage Even Rouault, who managed to run FreeBSD in KQEMU on the Linux instances provided by Travis-CI “Travis-CI has a free offer for software having public repository at GitHub. Travis-CI provides cloud instances running Linux or Mac OS X. To increase portability tests of GDAL, I wondered if it was somehow possible to run another operating system with Travis-CI, for example FreeBSD. A search lead me to this question (https://github.com/travis-ci/travis-ci/issues/1818) in their bug tracker but the outcome seems to be that it is not possible, nor in their medium or long term plans.” “One idea that came quickly to mind was to use the QEMU machine emulator that can simulate full machines, of several hardware architectures.” They found an existing image of FreeBSD 9.2 and configured the Travis job to download it and fire it up in QEMU. “Here we go: ./configure && make ! That works, but 50 minutes later (the maximum length of a Travis-CI job), our job is killed with perhaps only 10% of the GDAL code base being compiled. The reason is that we used the pure software emulation mode of QEMU that involves on-the-fly disassembling of the code to be run and re-assembling.” Travis-CI runs in Google Compute Engine, which does not allow nested virtualization, so hardware virtualization is not an option to speed up QEMU “Here comes the time for good old memories and a bit of software archeology. QEMU was started by Fabrice Bellard. If you didn't know his name yet, F. Bellard created FFMPEG and QEMU, holds a world record for the number of decimals of Pi computed on a COTS PC, has ported QEMU in JavaScript to run the Linux kernel in your browser, devised BPG, a new compression based on HEVC, etc....” “At the time where his interest was focused on QEMU, he created KQemu, a kernel module (for Linux, Windows, FreeBSD hosts), that could significantly enhance QEMU performance when the guest and hosts are x86/x86_64 and does not require (nor use) hardware virtualization instructions.” “Running it on Travis-CI was successful too, with the compilation being done in 20 minutes, so probably half of the speed of bare metal, which is good enough.” “I could also have potentially tried VirtualBox because, as mentioned above, it supports software virtualization with acceleration. But that is only for 32 bit guests (and I didn't find a ready-made FreeBSD 32bit image that you can directly ssh into). For 64 bit guests, VirtualBox require hardware virtualization to be available in the host. To the best of my knowledge, KQemu is (was) the only solution to enable acceleration of 64 bit guests without hardware requirements.” It will be interesting to see if enough people do this hack, maybe Travis-CI will consider properly supporting FreeBSD *** OpenBSD EuroBSDcon 2016 Papers are online (https://www.openbsd.org/events.html) Slides from the OpenBSD talks at EuroBSDCon are online now Landry Breuil, Building packages on exotic architectures (https://rhaalovely.net/~landry/eurobsdcon2016/) Peter Hessler, Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD (https://www.openbsd.org/papers/eurobsdcon2016-bfd.pdf) Ingo Schwarze, Why and how you ought to keep multibyte character support simple (https://www.openbsd.org/papers/eurobsdcon2016-utf8.pdf) (roff/mm/gpresent source code (https://www.openbsd.org/papers/eurobsdcon2016-utf8.roff)) Stefan Sperling, OpenBSD meets 802.11n (https://www.openbsd.org/papers/eurobsdcon2016-openbsd-11n.pdf) Antoine Jacoutot, OpenBSD rc.d(8) (https://www.bsdfrog.org/pub/events/openbsd-rcd-EuroBSDcon2016.pdf) Marc Espie, Retrofitting privsep into dpb and pkg_add (https://www.openbsd.org/papers/eurobsdcon2016-privsep.pdf) Martin Pieuchot, Embracing the BSD routing table (https://www.openbsd.org/papers/eurobsdcon2016-embracingbsdrt.pdf) I am working to build a similar website for the FreeBSD project, but there is still a lot of work to do I also managed to find the slides from the keynotes: Opening Keynote: George Neville-Neil: Looking Backwards: The coming decades of BSD (https://papers.freebsd.org/2016/EuroBSDCon/LookingBackwards.pdf) Closing Keynote: Gert Döring: Internet Attacks, Self-Governance, and the Consequences (http://www.monobsd.com/files/16_ddos_and_consequences.pptx) *** VirtualBox Shared Folders on FreeBSD: progress report (https://kernelnomicon.org/?p=650) In the past month or so, VirtualBox in the FreeBSD ports tree got bumped to version 5, which while bringing new features, did cause a regression in Shared Folders. FreeBSD developer gonzo@ (Oleksandr Tymoshenko) has been tackling this issue in recent days and provides us with a look behind the curtain at the challenges involved. Specifically he started by implementing the various needed VOPs: “lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write.” He then continues with details about how complete this is: ““Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host's md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.” The bigger issue faced is with the rename VOP though: I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task. + We wish him luck in getting those issues solved. I know quite a few of our users rely on shared folders as well. FreeBSD News Issue #1 (http://support.rossw.net/FreeBSD-Issue1.pdf) Issue #1 of FreeBSD News, from summer of 1997 Contains an article by Yahoo! co-founder David Filo about their early use of FreeBSD, on 100mhz Pentium machines with 64MB of ram Java Development Kit 1.0.2 ported to FreeBSD What is FreeBSD? Running the world's busiest FTP site (cdrom.com) on FreeBSD Xi Graphics announces the release of CDE Business Desktop, the first and only integrated desktop for FreeBSD, on AcceleratedX, a fully supported commercial grade X display server Get FreeBSD 2.2.2 Today! *** Beastie Bits Call for testing: newly MPSAFE nvme(4) (http://mail-index.netbsd.org/current-users/2016/09/21/msg030183.html) Thinking about starting a BUG in Indianapolis, IN USA (http://lists.dragonflybsd.org/pipermail/users/2016-September/313061.html) The cost of forsaking C: Why students still need to learn C (https://medium.com/bradfield-cs/the-cost-of-forsaking-c-113986438784#.o2m5gv8y7) OpenBSD (U)EFI bootloader howto (https://blog.jasper.la/openbsd-uefi-bootloader-howto/) Michael Lucas sets his eyes on OpenBSD's web stack for his next book (http://blather.michaelwlucas.com/archives/2780) LibreSSL 2.5.0 released (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.5.0-relnotes.txt) OPNsense 16.7.5 released (https://opnsense.org/opnsense-16-7-5-released/) Feedback/Questions Jonas - ZFS on DO (http://pastebin.com/XeJhK0AJ) Ricardo - OpenBSD Encrypted Disk (http://pastebin.com/Z9JRjcvb) WiskerTickle - Storage Benchmark (http://pastebin.com/XAD0UevP) Phil - Thanks (http://pastebin.com/N52JhYru) Luis - Misc Questions (http://pastebin.com/57qS0wrx) ***