Unix-like, desktop-oriented operating system
POPULARITY
This week Adam talks with Kris Moore, Senior Vice President of Engineering at iXsystems, about all things TrueNAS. They discuss the history of TrueNAS starting from its origins as a FreeBSD project, TrueNAS Core being in maintenance mode, the momentum and innovation happening in TrueNAS Scale, the evolution of the TrueNAS user interface, managing ZFS compatibility in TrueNAS, the business model of iXsystems and their commitment to the open-source community, and of course what's to come in the upcoming Dragonfish release of TrueNAS Scale.
This week Adam talks with Kris Moore, Senior Vice President of Engineering at iXsystems, about all things TrueNAS. They discuss the history of TrueNAS starting from its origins as a FreeBSD project, TrueNAS Core being in maintenance mode, the momentum and innovation happening in TrueNAS Scale, the evolution of the TrueNAS user interface, managing ZFS compatibility in TrueNAS, the business model of iXsystems and their commitment to the open-source community, and of course what's to come in the upcoming Dragonfish release of TrueNAS Scale.
Kris Moore is the Vice President of Engineering for iXsystems. Kris is an experienced executive with a demonstrated history of working in the computer software industry and is closely involved in the TrueNAS Open Storage community. He holds strong information technology skills and expertise in Shell scripting, C++, operating systems, FreeBSD, and system administration. He started learning about open source in the 90's where he was involved in developing web services using a very early version of FreeBSD. From that experience, he learned the basics of shell interface and how to check his e-mail without a GUI by using the “pine” command, which according to him, “was very uber-l33t” at the time. During his college years, Kris played around with other Unix and Linux platforms (caldera Linux, SuSE), however when he needed to do serious work, such as setting up a web or file server, he always fell back to his FreeBSD roots. To this day, he still prefers FreeBSD because it offers a greater degree of stability than other operating systems. You can find Kris Moore on the following sites: Twitter LinkedIn Here are some links provided by Kris Moore: iXsystems TrueNAS PLEASE SUBSCRIBE TO THE PODCAST Spotify: http://isaacl.dev/podcast-spotify Apple Podcasts: http://isaacl.dev/podcast-apple Google Podcasts: http://isaacl.dev/podcast-google RSS: http://isaacl.dev/podcast-rss You can check out more episodes of Coffee and Open Source on https://www.coffeeandopensource.com Coffee and Open Source is hosted by Isaac Levin (https://twitter.com/isaacrlevin) --- Support this podcast: https://podcasters.spotify.com/pod/show/coffeandopensource/support
Here are this week's special guests and energy experts on The Crude Life Week In Review Episode 488. Hunter Berg has a fantastic job these days. Finding students who want a “free education” at Williston State College. According to Berg, executive director, Williston State College Foundation, the university has been [...]
Kris Moore LinkedIn: https://www.linkedin.com/in/kris-moore-24b9612/ Notes: Credits: Music by ikson: https://www.iksonmusic.com Special Guest: Kris Moore.
Your hosts Donovon Moore and Ayush Ajmani invite a special guest on the show to discuss the history of the Ethiopian province and how the current instability in the Tigray region could cause another civil war.
Time Codes: 00:00:27 - Introduction 00:03:03 - Fare-thee-well to Paul! 00:05:51 - The Cardboard Kingdom 00:28:41 - All Summer Long 00:44:11 -Be Prepared 01:06:46 - Wrap up 01:08:09 - Contact us On this episode of the Comics Alternative's Young Readers show, Gwen and Derek discuss summer 2018 new releases, all geared to middle-grade readers. The first text, edited and illustrated by Chad Sell, is The Cardboard Kingdom, released by Random House Graphic. Readers learn about the lives and dreams of a group of neighborhood kids in short stories written by Jay Fuller, David DeMeo, Katie Schenkel, Kris Moore, Molly Muldoon, Vid Alliger, Manuel Betancourt, Michael Cole, Cloud Jacobs, and Barbara Perez Marquez. The second text they discuss is Hope Larson's All Summer Long from Farrar Straus Giroux. And finally, Gwen and Derek review Vera Brosgol's long-awaited memoir, Be Prepared, released by First Second Books. They start by reviewing a comic that demonstrates the experimentation that is currently taking place in the young reader category: Chad Sell's edited collection of connected short stories, The Cardboard Kingdom. Set in a suburban neighborhood and featuring a truly diverse and engaging group of young kids, these stories show how imagination can function as a coping device. Young reader short story collections are not entirely new. Random House has also supported the Comics Squadseries, edited by Jennifer Holm and a rotating cast of co-editors, that have included themed volumes on "Recess," "Lunch," and "Detention." However, this short story collection boast only one artist, Chad Sell, and is presented as a cohesive narrative, with authors identified only at the end of the text. As such, the comic has a more cohesive feel, rather than a collection of fundamentally different stories that are linked only by theme. After that, Gwen and Derek check out Hope Larson's All Summer Long. This is the story of a 13-year-old, Bina, and her attempts to find meaning over the course of a summer. Growing up, she had been used to spending the summer with her best friend Austin, but during this particular summer, Austin goes off to soccer camp, leaving Bina to fend for herself. And part of this fending includes Austin's aggressive older sister, Charlie. This is a book all about growing up and finding your way over the course of significant life changes. Finally, Gwen and Derek discuss Vera Brosgol's BePrepared. This memoir focuses on the summer between 4th and 5th grades, when she attends Russian summer camp in the hope of finding friends with whom she will have something in common. The opening vignette in the novel focuses on young Vera's sense of cultural and economic isolation: as an immigrant from Russia and the daughter of a single mom living in a prosperous east coast suburb, she is often slow to pick up on the latest trends -- such as American Girl dolls -- and unable to approximate the lavish birthday parties that her classmates' parents are able to throw for their children. At the end of the school year, Vera listens to the plans that her friends are making, attending girl scout camp, taking vacations to faraway destinations…and she feels left out again. However, at the Russian Orthodox church that she attends, Vera learns from Ksenya, a Sunday school friend, about Orra, a Russian heritage camp, and she is certain that it will not only be fun, but will give her something to talk about with her school friends in autumn. While the experience is certainly life changing, it nonetheless becomes something quite different than what Vera had expected.
We cover an interview about Unix Architecture Evolution, another vBSDcon trip report, how to teach an old Unix about backspace, new NUMA support coming to FreeBSD, and stack pointer checking in OpenBSD. This episode was brought to you by Headlines Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD (https://fosdem.org/2018/interviews/diomidis-spinellis/) Q: Could you briefly introduce yourself? I'm a professor of software engineering, a programmer at heart, and a technology author. Currently I'm also the editor in chief of the IEEE Software magazine. I recently published the book Effective Debugging, where I detail 66 ways to debug software and systems. Q: What will your talk be about, exactly? I will describe how the architecture of the Unix operating system evolved over the past half century, starting from an unnamed system written in PDP-7 assembly language and ending with a modern FreeBSD system. My talk is based, first, on a GitHub repository where I tried to record the system's history from 1970 until today and, second, on the evolution of documented facilities (user commands, system calls, library functions) across revisions. I will thus present the early system's defining architectural features (layering, system calls, devices as files, an interpreter, and process management) and the important ones that followed in subsequent releases: the tree directory structure, user contributed code, I/O redirection, the shell as a user program, groups, pipes, scripting, and little languages. Q: Why this topic? Unix stands out as a major engineering breakthrough due to its exemplary design, its numerous technical contributions, its impact, its development model, and its widespread use. Furthermore, the design of the Unix programming environment has been characterized as one offering unusual simplicity, power, and elegance. Consequently, there are many lessons that we can learn by studying the evolution of the Unix architecture, which we can apply to the design of new systems. I often see modern systems that suffer from a bloat of architectural features and a lack of clear form on which functionality can be built. I believe that many of the modern Unix architecture defining features are excellent examples of what we should strive toward as system architects. Q: What do you hope to accomplish by giving this talk? What do you expect? I'd like FOSDEM attendees to leave the talk with their mind full with architectural features of timeless quality. I want them to realize that architectural elegance isn't derived by piling design patterns and does not need to be expensive in terms of resources. Rather, beautiful architecture can be achieved on an extremely modest scale. Furthermore, I want attendees to appreciate the importance of adopting flexible conventions rather than rigid enforcement mechanisms. Finally, I want to demonstrate through examples that the open source culture was part of Unix from its earliest days. Q: What are the most significant milestones in the development of Unix? The architectural development of Unix follows a path of continuous evolution, albeit at a slowing pace, so I don't see here the most important milestones. I would however define as significant milestones two key changes in the way Unix was developed. The first occurred in the late 1970s when significant activity shifted from a closely-knit team of researchers at the AT&T Bell Labs to the Computer Science Research Group in the University of California at Berkeley. This opened the system to academic contributions and growth through competitive research funding. The second took place in the late 1980s and the 1990s when Berkeley open-sourced the the code it had developed (by that time a large percentage of the system) and enthusiasts built on it to create complete open source operating system distributions: 386BSD, and then FreeBSD, NetBSD, OpenBSD, and others. Q: In which areas has the development of Unix stalled? The data I will show demonstrate that there were in the past some long periods where the number of C library functions and system calls remained mostly stable. Nowadays there is significant growth in the number of all documented facilities with the exception of file formats. I'm looking forward to a discussion regarding the meaning of these growth patterns in the Q&A session after the talk. Q: What are the core features that still link the 1970 PDP-7 system to the latest FreeBSD 11.1 release, almost half a century apart? Over the past half-century the Unix system has grown by four orders of magnitude from a few thousand lines of code to many millions. Nevertheless, looking at a 1970s architecture diagram and a current one reveals that the initial architectural blocks are still with us today. Furthermore, most system calls, user programs, and C library functions of that era have survived until today with essentially similar functionality. I've even found in modern FreeBSD some lines of code that have survived unchanged for 40 years. Q: Can we still add innovative changes to operating systems like FreeBSD without breaking the ‘Unix philosophy'? Will there be a moment where FreeBSD isn't recognizable anymore as a descendant of the 1970 PDP-7 system? There's a saying that “form liberates”. So having available a time-tested form for developing operating system functionality allows you to innovate in areas that matter rather than reinventing the wheel. Such concepts include having commands act as a filter, providing manual pages with a consistent structure, supplying build information in the form of a Makefile, installing files in a well-defined directory hierarchy, implementing filesystems with an standardized object-oriented interface, and packaging reusable functions as a library. Within this framework there's ample space for both incremental additions (think of jq, the JSON query command) and radical innovations (consider the Solaris-derived ZFS and dtrace functionality). For this reason I think that BSD and Linux systems will always be recognizable as direct or intellectual descendants of the 1970s Research Unix editions. Q: Have you enjoyed previous FOSDEM editions? Immensely! As an academic I need to attend many scientific conferences and meetings in order to present research results and interact with colleagues. This means too much time spent traveling and away from home, and a limited number of conferences I'm in the end able to attend. Nevertheless, attending FOSDEM is an easy decision due to the world-changing nature of its theme, the breadth of the topics presented, the participants' enthusiasm and energy, as well as the exemplary, very efficient conference organization. Another vBSDCon trip report we just found (https://www.weaponizedawesome.com/blog/?cat=53) We just got tipped about another trip report from vBSDCon, this time from one of the first time speakers: W. Dean Freeman Recently I had the honor of co-presenting on the internals of FreeBSD's Kernel RNG with John-Mark Gurney at the 3rd biennial vBSDCon, hosted in Reston, VA hosted by Verisign. I've been in and out of the FreeBSD community for about 20 years. As I've mentioned on here before, my first Unix encounter was FreeBSD 2.2.8 when I was in the 7th or 8th grade. However, for all that time I've never managed to get out to any of the cons. I've been to one or two BUG meetings and I've met some folks from IRC before, but nothing like this. A BSD conference is a very different experience than anything else out there. You have to try it, it is the only way to truly understand it. I'd also not had to do a stand-up presentation really since college before this. So, my first BSD con and my first time presenting rolled into one made for an interesting experience. See, he didn't say terrifying. It went very well. You should totally submit a talk for the next conference, even if it is your first. That said, it was amazing and invigorating experience. I got to meet a few big names in the FreeBSD community, discuss projects, ideas for FreeBSD, etc. I did seem to spend an unusual amount of time talking about FIPS and Common Criteria with folks, but to me that's a good sign and indicative that there is interest in working to close gaps between FreeBSD and the current requirements so that we can start getting FreeBSD and more BSD-based products into the government and start whittling away the domination of Linux (especially since Oracle has cut Solaris, SPARC and the ZFS storage appliance business units). There is nothing that can match the high bandwidth interchange of ideas in person. The internet has made all kinds of communication possible, and we use it all the time, but every once in a while, getting together in person is hugely valuable. Dean then went on to list some of the talks he found most valuable, including DTrace, Capsicum, bhyve, *BSD security tools, and Paul Vixie's talk about gets() I think the talk that really had the biggest impact on me, however, was Kyle Kneisl's talk on BSD community dynamics. One of the key points he asked was whether the things that drew us to the BSD community in the first place would be able to happen today. Obviously, I'm not a 12 or 13 year old kid anymore, but it really got me thinking. That, combined with getting face time with people I'd previously only known as screen names has recently drawn me back into participating in IRC and rejoining mailing lists (wdf on freenode. be on the lookout!) Then Dean covered some thoughts on his own talk: JMG and my talk seems to have been well received, with people paying lots of attention. I don't know what a typical number of questions is for one of these things, but on day one there weren't that many questions. We got about 5 during our question time and spent most of the rest of the day fielding questions from interested attendees. Getting a “great talk!” from GNN after coming down from the stage was probably one of the major highlights for me. I remember my first solo talk, and GNN asking the right question in the middle to get me to explain a part of it I had missed. It was very helpful. I think key to the interest in our presentation was that JMG did a good job framing a very complicated topic's importance in terms everyone could understand. It also helped that we got to drop some serious truth bombs. Final Thoughts: I met a lot of folks in person for the first time, and met some people I'd never known online before. It was a great community and I'm glad I got a chance to expand my network. Verisign were excellent hosts and they took good care of both speakers (covering airfare, rooms, etc.) and also conference attendees at large. The dinners that they hosted were quite good as well. I'm definitely interested in attending vBSDCon again and now that I've had a taste of meeting IRL with the community on scale of more than a handful, I have every intention of finally making it to BSDCan next year (I'd said it in 2017, but then moved to Texas for a new job and it wasn't going to be practical). This year for sure, though! Teaching an Almost 40-year Old UNIX about Backspace (https://virtuallyfun.com/2018/01/17/teaching_an_almost_40-year_old_unix_about_backspace/) Introduction I have been messing with the UNIX® operating system, Seventh Edition (commonly known as UNIX V7 or just V7) for a while now. V7 dates from 1979, so it's about 40 years old at this point. The last post was on V7/x86, but since I've run into various issues with it, I moved on to a proper installation of V7 on SIMH. The Internet has some really good resources on installing V7 in SIMH. Thus, I set out on my own journey on installing and using V7 a while ago, but that was remarkably uneventful. One convenience that I have been dearly missing since the switch from V7/x86 is a functioning backspace key. There seem to be multiple different definitions of backspace: BS, as in ASCII character 8 (010, 0x08, also represented as ^H), and DEL, as in ASCII character 127 (0177, 0x7F, also represented as ^?). V7 does not accept either for input by default. Instead, # is used as the erase character and @ is used as the kill character. These defaults have been there since UNIX V1. In fact, they have been “there” since Multics, where they got chosen seemingly arbitrarily. The erase character erases the character before it. The kill character kills (deletes) the whole line. For example, “ba##gooo#d” would be interpreted as “good” and “bad line@good line” would be interpreted as “good line”. There is some debate on whether BS or DEL is the correct character for terminals to send when the user presses the backspace key. However, most programs have settled on DEL today. tmux forces DEL, even if the terminal emulator sends BS, so simply changing my terminal to send BS was not an option. The change from the defaults outlined here to today's modern-day defaults occurred between 4.1BSD and 4.2BSD. enf on Hacker News has written a nice overview of the various conventions Getting the Diff For future generations as well as myself when I inevitably majorly break this installation of V7, I wanted to make a diff. However, my V7 is installed in SIMH. I am not a very intelligent man, I didn't keep backup copies of the files I'd changed. Getting data out of this emulated machine is an exercise in frustration. In the end, I printed everything on screen using cat(1) and copied that out. Then I performed a manual diff against the original source code tree because tabs got converted to spaces in the process. Then I applied the changes to clean copies that did have the tabs. And finally, I actually invoked diff(1). Closing Thoughts Figuring all this out took me a few days. Penetrating how the system is put together was surprisingly fairly hard at first, but then the difficulty curve eased up. It was an interesting exercise in some kind of “reverse engineering” and I definitely learned something about tty handling. I was, however, not pleased with using ed(1), even if I do know the basics. vi(1) is a blessing that I did not appreciate enough until recently. Had I also been unable to access recursive grep(1) on my host and scroll through the code, I would've probably given up. Writing UNIX under those kinds of editing conditions is an amazing feat. I have nothing but the greatest respect for software developers of those days. News Roundup New NUMA support coming to FreeBSD CURRENT (https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068145.html) Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem, and vmpage*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VMNUMAALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VMNUMAALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff Stack pointer checking – OpenBSD (https://marc.info/?l=openbsd-tech&m=151572838911297&w=2) Stefan (stefan@) and I have been working for a few months on this diff, with help from a few others. At every trap and system call, it checks if the stack-pointer is on a page that is marked MAPSTACK. execve() is changed to create such mappings for the process stack. Also, libpthread is taught the new MAPSTACK flag to use with mmap(). There is no corresponding system call which can set MAP_FLAG on an existing page, you can only set the flag by mapping new memory into place. That is a piece of the security model. The purpose of this change is to twart stack pivots, which apparently have gained some popularity in JIT ROP attacks. It makes it difficult to place the ROP stack in regular data memory, and then perform a system call from it. Workarounds are cumbersome, increasing the need for far more gadgetry. But also the trap case -- if any memory experiences a demand page fault, the same check will occur and potentially also kill the process. We have experimented a little with performing this check during device interrupts, but there are some locking concerns and performance may then become a concern. It'll be best to gain experience from handle of syncronous trap cases first. chrome and other applications I use run fine! I'm asking for some feedback to discover what ports this breaks, we'd like to know. Those would be ports which try to (unconventionally) create their stacks in malloc()'d memory or inside another Data structure. Most of them are probably easily fixed ... Qt 5.9 on FreeBSD (https://euroquis.nl/bobulate/?p=1768) Tobias and Raphael have spent the past month or so hammering on the Qt 5.9 branch, which has (finally!) landed in the official FreeBSD ports tree. This brings FreeBSD back up-to-date with current Qt releases and, more importantly, up-to-date with the Qt release KDE software is increasingly expecting. With Qt 5.9, the Elisa music player works, for instance (where it has run-time errors with Qt 5.7, even if it compiles). The KDE-FreeBSD CI system has had Qt 5.9 for some time already, but that was hand-compiled and jimmied into the system, rather than being a “proper” ports build. The new Qt version uses a new build system, which is one of the things that really slowed us down from a packaging perspective. Some modules have been reshuffled in the process. Some applications depending on Qt internal-private headers have been fixed along the way. The Telegram desktop client continues to be a pain in the butt that way. Following on from Qt 5.9 there has been some work in getting ready for Clang 6 support; in general the KDE and Qt stack is clean and modern C++, so it's more infrastructural tweaks than fixing code. Outside of our silo, I still see lots of wonky C++ code being fixed and plenty of confusion between pointers and integers and strings and chars and .. ugh. Speaking of ugh, I'm still planning to clean up Qt4 on ARM aarch64 for FreeBSD; this boils down to stealing suitable qatomic implementations from Arch Linux. For regular users of Qt applications on FreeBSD, there should be few to no changes required outside the regular upgrade cycle. For KDE Plasma users, note that development of the ports has changed branches; as we get closer to actually landing modern KDE bits, things have been renamed and reshuffled and mulled over so often that the old plasma5 branch wasn't really right anymore. The kde5-import branch is where it's at nowadays, and the instructions are the same: the x11/kde5 metaport will give you all the KDE Frameworks 5, KDE Plasma Desktop and modern KDE Applications you need. Adding IPv6 to an Nginx website on FreeBSD / FreshPorts (https://dan.langille.org/2018/01/13/adding-ipv6-to-an-nginx-website-on-freebsd-freshports/) FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized. There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation. All of this took about 3 hours. We are using: FreeBSD 11.1 Bind 9.9.11 nginx 1.12.2 Fallout I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses. ZFS on TrueOS: Why We Love OpenZFS (https://www.trueos.org/blog/zfs-trueos-love-openzfs/) TrueOS was the first desktop operating system to fully implement the OpenZFS (Zettabyte File System or ZFS for short) enterprise file system in a stable production environment. To fully understand why we love ZFS, we will look back to the early days of TrueOS (formerly PC-BSD). The development team had been using the UFS file system in TrueOS because of its solid track record with FreeBSD-based computer systems and its ability to check file consistency with the built-in check utility fsck. However, as computing demands increased, problems began to surface. Slow fsck file verification on large file systems, slow replication speeds, and inconsistency in data integrity while using UFS logging / journaling began to hinder users. It quickly became apparent that TrueOS users would need a file system that scales with evolving enterprise storage needs, offers the best data protection, and works just as well on a hobbyist system or desktop computer. Kris Moore, the founder of the TrueOS project, first heard about OpenZFS in 2007 from chatter on the FreeBSD mailing lists. In 2008, the TrueOS development team was thrilled to learn that the FreeBSD Project had ported ZFS. At the time, ZFS was still unproven as a graphical desktop solution, but Kris saw a perfect opportunity to offer ZFS as a cutting-edge file system option in the TrueOS installer, allowing the TrueOS project to act as an indicator of how OpenZFS would fair in real-world production use. The team was blown away by the reception and quality of OpenZFS on FreeBSD-based systems. By its nature, ZFS is a copy-on-write (CoW) file system that won't move a block of data until it both writes the data and verifies its integrity. This is very different from most other file systems in use today. ZFS is able to assure that data stays consistent between writes by automatically comparing write checksums, which mitigates bit rot. ZFS also comes with native RaidZ functionality that allows for enterprise data management and redundancy without the need for expensive traditional RAID cards. ZFS snapshots allow for system configuration backups in a split-second. You read that right. TrueOS can backup or restore snapshots in less than a second using the ZFS file system. Given these advantages, the TrueOS team decided to use ZFS as its exclusive file system starting in 2013, and we haven't looked back since. ZFS offers TrueOS users the stable workstation experience they want, while simultaneously scaling to meet the increasing demands of the enterprise storage market. TrueOS users are frequently commenting on how easy it is to use ZFS snapshots with our built-in snapshot utility. This allows users the freedom to experiment with their system knowing they can restore it in seconds if anything goes wrong. If you haven't had a chance to try ZFS with TrueOS, browse to our download page and make sure to grab a copy of TrueOS. You'll be blown away by the ease of use, data protection functionality, and incredible flexibility of RaidZ. Beastie Bits Source Code Podcast Interview with Michael W Lucas (https://blather.michaelwlucas.com/archives/3099) Operating System of the Year 2017: NetBSD Third place (https://w3techs.com/blog/entry/web_technologies_of_the_year_2017) OPNsense 18.1-RC1 released (https://opnsense.org/opnsense-18-1-rc1-released/) Personal OpenBSD Wiki Notes (https://balu-wiki.readthedocs.io/en/latest/security/openbsd.html) BSD section can use some contribution (https://guide.freecodecamp.org/bsd-os/) The Third Research Edition Unix Programmer's Manual (now available in PDF) (https://github.com/dspinellis/unix-v3man) Feedback/Questions Alex - my first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) Brian - Brian - openbsd 6.2 and enlightenment .17 (http://dpaste.com/24DYF1J#wrap) ***
We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)
OpenBSD 6.2 is here, style arguments, a second round of viewer interview questions, how to set CPU affinity for FreeBSD jails, containers on FreeNAS & more! Headlines OpenBSD 6.2 Released (https://www.openbsd.org/62.html) OpenBSD continues their six month release cadence with the release of 6.2, the 44th release On a disappointing note, the song for 6.2 will not be released until December Highlights: Improved hardware support on modern platforms including ARM64/ARMv7 and octeon, while amd64 users will appreciate additional support for the Intel Kaby Lake video cards. Network stack improvements include extensive SMPization improvements and a new FQ-CoDel queueing discipline, as well as enhanced WiFi support in general and improvements to iwn(4), iwm(4) and anthn(4) drivers. Improvements in vmm(4)/vmd include VM migration, as well as various compatibility and performance improvements. Security enhancements including a new freezero(3) function, further pledge(2)ing of base system programs and conversion of several daemons to the fork+exec model. Trapsleds, KARL, and random linking for libcrypto and ld.so, dramatically increase security by making it harder to find helpful ROP gadgets, and by creating a unique order of objects per-boot. A unique kernel is now created by the installer to boot from after install/upgrade. The base system compiler on the amd64 and i386 platforms has switched to clang(1). New versions of OpenSSH, OpenSMTPd, LibreSSL and mandoc are also included. The kernel no longer handles IPv6 Stateless Address Autoconfiguration (RFC 4862), allowing cleanup and simplification of the IPv6 network stack. Improved IPv6 checks for IPsec policies and made them consistent with IPv4. Enabled the use of per-CPU caches in the network packet allocators. Improved UTF-8 line editing support for ksh(1) Emacs and Vi input mode. breaking change for nvme(4) users with GPT: If you are booting from an nvme(4) drive with a GPT disk layout, you are affected by an off-by-one in the driver with the consequence that the sector count in your partition table may be incorrect. The only way to fix this is to re-initialize the partition table. Backup your data to another disk before you upgrade. In the new bsd.rd, drop to a shell and re-initialize the GPT: fdisk -iy -g -b 960 sdN Why we argue: style (https://www.sandimetz.com/blog/2017/6/1/why-we-argue-style) I've been thinking about why we argue about code, and how we might transform vehement differences of opinion into active forces for good. My thoughts spring from a very specific context. Ten or twelve times a year I go to an arbitrary business and spend three or more days teaching a course in object-oriented design. I'm an outsider, but for a few days these business let me in on their secrets. Here's what I've noticed. In some places, folks are generally happy. Programmers get along. They feel as if they are all "in this together." At businesses like this I spend most of my time actually teaching object-oriented design. Other places, folks are surprisingly miserable. There's a lot of discord, and the programmers have devolved into competing "camps." In these situations the course rapidly morphs away from OO Design and into wide-ranging group discussions about how to resolve deeply embedded conflicts. Tolstoy famously said that "Happy families are all alike; every unhappy family is unhappy in its own way." This is known as the Anna Karenina Principle, and describes situations in which success depends on meeting all of a number of criteria. The only way to be happy is to succeed at every one of them. Unhappiness, unfortunately, can be achieved by any combination of failure. Thus, all happy businesses are similar, but unhappy ones appear unique in their misery. Today I'm interested in choices of syntax, i.e whether or not your shop has agreed upon and follows a style guide. If you're surprised that I'm starting with this apparently mundane issue, consider yourself lucky in your choice of workplace. If you're shaking your head in rueful agreement about the importance of this topic, I feel your pain. I firmly believe that all of the code that I personally have to examine should come to me in a consistent format. Code is read many more times than it is written, which means that the ultimate cost of code is in its reading. It therefore follows that code should be optimized for readability, which in turn dictates that an application's code should all follow the same style. This is why FreeBSD, and most other open source projects, have a preferred style. Some projects are less specific and less strict about it. Most programmers agree with the prior paragraph, but here's where things begin to break down. As far as I'm concerned, my personal formatting style is clearly the best. However, I'm quite sure that you feel the same. It's easy for a group of programmers to agree that all code should follow a common style, but surprisingly difficult to get them to agree on just what that common style should be. Avoid appointing a human "style cop", which just forces someone to be an increasingly ill-tempered nag. Instead, supply programmers with the information they need to remedy their own transgressions. By the time a pull request is submitted, mis-stylings should long since have been put right. Pull request conversations ought to be about what code does rather than how code looks. What about old code? Ignore it. You don't have to re-style all existing code, just do better from this day forward. Defer updating old code until you touch it for other reasons. Following this strategy means that the code you most often work on will gradually take on a common style. It also means that some of your existing code might never get updated, but if you never look at it, who cares? If you choose to re-style code that you otherwise have no need to touch, you're declaring that changing the look of this old code has more value to your business than delivering the next item on the backlog. The opportunity cost of making a purely aesthetic change includes losing the benefit of what you could have done instead. The rule-of-thumb is: Don't bother updating the styling of stable, existing code unless not doing so costs you money. Most open source projects also avoid reformatting code just to change the style, because of the merge conflicts this will cause for downstream consumers If you disagree with the style guide upon which your team agrees, you have only two honorable options: First, you can obey the guide despite your aversion. As with me in the Elm story above, this act is likely to change your thinking so that over time you come to prefer the new style. It's possible that if you follow the guide you'll begin to like it. Alternatively, you can decide you will not obey the style guide. Making this decision demands that you leave your current project and find some other project whose guide matches your preferred style. Go there and follow that one. Notice that both of these choices have you following a guide. This part is not optional. The moral of this story? It's more important for all code to be formatted the same than it is for any one of us to get our own way. Commit to agreeing upon and following a style guide. And if you find that your team cannot come to an agreement, step away from this problem and start a discussion about power. There have been many arguments about style, and it can often be one of the first complaints of people new to any open source project This article covers it fairly well from both sides, a) you should follow the style guide of the project you are contributing to, b) the project should review your actual code, then comment on the style after, and provide gentle guidance towards the right style, and avoid being “style cops” *** Interview - The BSDNow Crew, Part II News Roundup Building FreeBSD for the Onion Omega 2 (https://github.com/sysadminmike/freebsd-onion-omega2-build) I got my Onion Omega 2 devices in the mail quite a while ago, but I had never gotten around to trying to install FreeBSD on them. They are a different MIPS SoC than the Onion Omega 1, so it would not work out of the box at the time. Now, the SoC is supported! This guide provides the steps to build an image for the Omega 2 using the freebsd-wifi-build infrastructure First some config files are modified to make the image small enough for the Omega 2's flash chip The DTS (Device Tree Source) files are not yet included in FreeBSD, so they are fetched from github Then the build for the ralink SoC is run, with the provided DTS file and the MT7628_FDT kernel config Once the build is complete, you'll have a tftp image file. Then that image is compressed, and bundled into a uboot image Write the files to a USB stick, and plug it into the Omega's dock Turn it on while holding the reset button with console open Press 1 to get into the command line. You will need to reset the usb: usb reset Then load the kernel boot image: fatload usb 0:1 0x80800000 kernel.MT7628_FDT.lzma.uImage And boot it: bootm 0x80800000 At this point FreeBSD should boot Mount a userland, and you should end up in multi-user mode Hopefully this will get even easier in the next few weeks, and we'll end up with a more streamlined process to tftp boot the device, then write FreeBSD into the onboard flash so it boots automatically. *** Setting the CPU Affinity on FreeBSD Jails with ezjail (https://www.neelc.org/setting-the-cpu-affinity-on-freebsd-jails-with-ezjail/) While there are more advanced resource controls available for FreeBSD jails, one of the most basic ways to control CPU usage is to limit the subset of CPUs that each jail can use. This can make sure that every jail has access to some dedicated resources, while at the same time doesn't have the ability to entirely dominate the machine I just got a new home server: a HP ProLiant ML110 G6. Being a FreeBSD person myself, it was natural that I used it on my server instead of Linux I chose to use ezjail to manage the jails on my ProLiant, with the initial one being a Tor middle node. Despite the fact that where my ML110 is, the upstream is only 35mbps (which is pretty good for cable), I did not want to give my Tor jail access to all four cores. Setting the CPU Affinity would let you choose a specific CPU core (or a range of cores) you want to use. However, it does not just let you pick the number of CPU cores you want and make FreeBSD choose the core running your jail. Going forward, I assumed that you have already created a jail using ezjail-admin. I also do not cover limiting a jail to a certain percentage of CPU usage. ezjail-admin config -c [CORENUMBERFIRST]-[CORENUMBERLAST] [JAIL_NAME] or ezjail-admin config -c [CORENUMBERFIRST],[CORENUMBERSECOND],...,[CORENUMBERN] [JAILNAME] And hopefully, you should have your ezjail-managed FreeBSD jail limited to the CPU cores you want. While I did not cover a CPU percentage or RAM usage, this can be done with rctl I'll admit: it doesn't really matter which CPU a jail runs on, but it might matter if you don't want a jail to have access to all the CPU cores available and only want [JAILNAME] to use one core. Since it's not really possible just specify the number of CPU cores with ezjail (or even iocell), a fallback would be to use CPU affinity, and that requires you to specify an exact CPU core. I know it's not the best solution (it would be better if we could let the scheduler choose provided a jail only runs on one core), but it's what works. We use this at work on high core count machines. When we have multiple databases colocated on the same machine, we make sure each one has a few cores to itself, while it shares other cores with the rest of the machine. We often reserve a core or two for the base system as well. *** A practical guide to containers on FreeNAS for a depraved psychopath. (https://medium.com/@andoriyu/a-practical-guide-to-containers-on-freenas-for-a-depraved-psychopath-c212203c0394) If you are interested in playing with Docker, this guide sets up a Linux VM running on FreeBSD or FreeNAS under bhyve, then runs linux docker containers on top of it You know that jails are dope and I know that jails are dope, yet no one else knows it. So here we are stuck with docker. Two years ago I would be the last person to recommend using docker, but a whole lot of things has changes past years… This tutorial uses iohyve to manage the VMs on the FreeBSD or FreeNAS There are many Linux variants you can choose from — RancherOS, CoreOS are the most popular for docker-only hosts. We going to use RancherOS because it's more lightweight out of the box. Navigate to RancherOS website and grab link to latest version sudo iohyve setup pool=zpool kmod=1 net=em0 sudo iohyve fetch https://releases.rancher.com/os/latest/rancheros.iso sudo iohyve renameiso rancheros.iso rancheros-v1.0.4.iso sudo pkg install grub2-bhyve sudo iohyve create rancher 32G sudo iohyve set rancher loader=grub-bhyve ram=8G cpu=8 con=nmdm0 os=debian sudo iohyve install rancher rancheros-v1.0.4.iso sudo iohyve console rancher Then the tutorial does some basic configuration of RancherOS, and some house keeping in iohyve to make RancherOS come up unattended at boot The whole point of this guide is to reduce pain, and using the docker CLI is still painful. There are a lot of Web UIs to control docker. Most of them include a lot of orchestrating services, so it's just overkill. Portainer is very lightweight and can be run even on Raspberry Pi Create a config file as described After reboot you will be able to access WebUI on 9000 port. Setup is very easy, so I won't go over it The docker tools for FreeBSD are still being worked on. Eventually you will be able to host native FreeBSD docker containers on FreeBSD jails, but we are not quite there yet In the meantime, you can install sysutils/docker and use it to manage the docker instances running on a remote machine, or in this case, the RancherOS VM running in bhyve *** Beastie Bits The Ghost of Invention: A Visit to Bell Labs, excerpt from the forthcoming book: “Kitten Clone: Inside Alcatel-Lucent” (https://www.wired.com/2014/09/coupland-bell-labs/) OpenBSD Cookbook (set of Ansible playbooks) (https://github.com/ligurio/openbsd-cookbooks) 15 useful sockstat commands to find open ports on FreeBSD (https://www.tecmint.com/sockstat-command-examples-to-find-open-ports-in-freebsd/) A prehistory of Slashdot (https://medium.freecodecamp.org/a-pre-history-of-slashdot-6403341dabae) Using ed, the unix line editor (https://medium.com/@claudio.santos.ribeiro/using-ed-the-unix-line-editor-557ed6466660) *** Feedback/Questions Malcolm - ZFS snapshots (http://dpaste.com/16EB3ZA#wrap) Darryn - Zones (http://dpaste.com/1DGHQJP#wrap) Mohammad - SSH Keys (http://dpaste.com/08G3VTB#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
This episode gives you the full dose of BSDCan 2017 recap as well as a blog post on conference speaking advice. Headlines Pre-conference activities: Goat BoF, FreeBSD Foundation Board Meeting, and FreeBSD Journal Editorial Board Meeting The FreeBSD Foundation has a new President as Justin Gibbs is busy this year with building a house, so George Neville-Neil took up the task to serve as President, with Justin Gibbs as Secretary. Take a look at the updated Board of Directors (https://www.freebsdfoundation.org/about/board-of-directors/). We also have a new staff member (https://www.freebsdfoundation.org/about/staff/): Scott Lamons joined the Foundation team as senior program manager. Scott's work for the Foundation will focus on managing and evangelizing programs for advanced technologies in FreeBSD including preparing project plans, coordinating resources, and facilitating interactions between commercial vendors, the Foundation, and the FreeBSD community. The Foundation also planned various future activities, visits of upcoming conferences, and finding new ways to support and engage the community. The Foundation now has interns in the form of co-op students from the University of Waterloo, Canada. This is described further in the May 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/may-2017-development-projects-update/). Both students (Siva and Charlie) were also the conference, helping out at the Foundation table, demonstrating the tinderbox dashboard. Follow the detailed instructions (https://www.freebsdfoundation.org/news-and-events/blog/blog-post/building-a-physical-freebsd-build-status-dashboard/) to build one of your own. The Foundation put out a call for Project Proposal Solicitation for 2017 (https://www.freebsdfoundation.org/blog/freebsd-foundation-2017-project-proposal-solicitation/). If you think you have a good proposal for work relating to any of the major subsystems or infrastructure for FreeBSD, we'd be happy to review it. Don't miss the deadlines for travel grants to some of the upcoming conferences. You can find the necessary forms and deadlines at the Travel Grant page (https://www.freebsdfoundation.org/what-we-do/travel-grants/travel-grants/) on the Foundation website. Pictures from the Goat BoF can be found on Keltia.net (https://assets.keltia.net/photos/BSDCan-2017/Royal%20Oak/index.html) Overlapping with the GoatBoF, members of the FreeBSD Journal editorial board met in a conference room in the Novotel to plan the upcoming issues. Topics were found, authors identified, and new content was discussed to appeal to even more readers. Check out the FreeBSD Journal website (https://www.freebsdfoundation.org/journal/) and subscribe if you like to support the Foundation in that way. FreeBSD Devsummit Day 1 & 2 (https://wiki.freebsd.org/DevSummit/201706) The first day of the Devsummit began with introductory slides by Gordon Tetlow, who organized the devsummit very well. Benno Rice of the FreeBSD core team presented the work done on the new Code of Conduct, which will become effective soon. A round of Q&A followed, with positive feedback from the other devsummit attendees supporting the new CoC. After that, Allan Jude joined to talk about the new FreeBSD Community Proposal (FCP) (https://github.com/freebsd/fcp) process. Modelled after IETF RFCs, Joyent RFDs, and Python PEP, it is a new way for the project to reach consensus on the design or implementation of new features or processes. The FCP repo contains FCP#0 that describes the process, and a template for writing a proposal. Then, the entire core team (except John Baldwin, who could not make it this year) and core secretary held a core Q&A session, Answering questions, gathering feedback and suggestions. After the coffee break, we had a presentation about Intel's QAT integration in FreeBSD. When the lunch was over, people spread out into working groups about BearSSL, Transport (TCP/IP), and OpenZFS. OpenZFS working group (https://pbs.twimg.com/media/DBu_IMsWAAId2sN.jpg:large): Matt Ahrens lead the group, and spent most of the first session providing a status update about what features have been recently committed, are out for review, on the horizon, or in the design phase. Existing Features Compressed ARC Compressed Send/Recv Recently Upstreamed A recent commit improved RAID-Z write speeds by declaring writes to padding blocks to be optional, and to always write them if they can be aggregated with the next write. Mostly impacts large record sizes. ABD (ARC buffer scatter/gather) Upstreaming In Progress Native Encryption Channel Programs Device Removal (Mirrors and Stripes) Redacted Send/recv Native TRIM Support (FreeBSD has its own, but this is better and applies to all ZFS implementations) Faster (mostly sequential) scrub/resilver DRAID (A great deal of time was spent explaining how this works, with diagrams on the chalk board) vdev metadata classes (store metadata on SSDs with data is on HDDs, or similar setups. Could also be modified to do dedup to SSD) Multi-mount protection (“safe import”, for dual-headed storage shelves) zpool checkpoint (rollback an entire pool, including zfs rename and zfs destroy) Further Out Import improvements Import with missing top-level vdevs (some blocks unreadable, but might let you get some data) Improved allocator performance -- vdev spacemap log ZIL performance Persistent L2ARC ZSTD Compression Day 2 Day two started with the Have/Want/Need session for FreeBSD 12.0. A number of features that various people have or are in the process of building, were discussed with an eye towards upstreaming them. Features we want to have in time for 12.0 (early 2019) were also discussed. After the break was the Vendor summit, which continued the discussion of how FreeBSD and its vendors can work together to make a better operating system, and better products based on it After lunch, the group broke up into various working groups: Testing/CI, Containers, Hardening UFS, and GELI Improvements Allan lead the GELI Improvements session. The main thrust of the discussions was fixing an outstanding bug in GELI when using both key slots with passphrases. To solve this, and make GELI more extensible, the metadata format will be extended to allow it to store more than 512 bytes of data (currently 511 bytes are used). The new format will allow arbitrarily large metadata, defined at creation time by selecting the number of user key slots desired. The new extended metadata format will contain mostly the same fields, except the userkey will no longer be a byte array of IV-key, Data-key, HMAC, but a struct that will contain all data about that key This new format will store the number of pkcs5v2 iterations per key, instead of only having a single location to store this number for all keys (the source of the original bug) A new set of flags per key, to control some aspects of the key (does it require a keyfile, etc), as well as possibly the role of the key. An auxdata field related to the flags, this would allow a specific key with a specific flag set, to boot a different partition, rather than decrypt the main partition. A URI to external key material is also stored per key, allowing GELI to uniquely identify the correct data to load to be able to use a specific decryption key And the three original parts of the key are stored in separate fields now. The HMAC also has a type field, allowing for a different HMAC algorithm to be used in the future. The main metadata is also extended to include a field to store the number of user keys, and to provide an overall HMAC of the metadata, so that it can be verified using the master key (provide any of the user keys) Other topics discussed: Ken Merry presented sedutil, a tool for managing Self Encrypting Drives, as may be required by certain governments and other specific use cases. Creating a deniable version of GELI, where the metadata is also encrypted The work to implemented GELI in the UEFI loader was discussed, and a number of developers volunteered to review and test the code Following the end of the Dev Summit, the “Newcomers orientation and mentorship” session was run by Michael W. Lucas, which attempts to pair up first time attendees with oldtimers, to make sure they always know a few people they can ask if they have questions, or if they need help getting introduced to the right people. News Roundup Conference Day 1 (http://www.bsdcan.org/2017/schedule/day_2017-06-09.en.html) The conference opened with some short remarks from Dan Langille, and then the opening keynote by Dr Michael Geist, a law professor at the University of Ottawa where he holds the Canada Research Chair in Internet and E-commerce Law. The keynote focused on what some of the currently issues are, and how the technical community needs to get involved at all levels. In Canada especially, contacting your representatives is quite effective, and when it does not happen, they only hear the other side of the story, and often end up spouting talking points from lobbyists as if they were facts. The question period for the keynote ran well overtime because of the number of good questions the discussion raised, including how do we fight back against large telcos with teams of lawyers and piles of money. Then the four tracks of talks started up for the day The day wrapped up with the Work In Progress (WIP) session. Allan Jude presented work on ZSTD compression in ZFS Drew Gallatin presented about work at Netflix on larger mbufs, to avoid the need for chaining and to allow more data to be pushed at once. Results in an 8% CPU time reduction when pushing 90 gbps of TLS encrypted traffic Dan Langille presented about letsencrypt (the acme.sh tool specifically), and bacula Samy Al Bahra presented about Concurrency Kit *** Conference Day 2 (http://www.bsdcan.org/2017/schedule/day_2017-06-10.en.html) Because Dan is a merciful soul, BSDCan starts an hour later on the second day Another great round of talks and BoF sessions over lunch The hallway track was great as always, and I spent most of the afternoon just talking with people Then the final set of talks started, and I was torn between all four of them Then there was the auction, and the closing party *** BSDCan 2017 Auction Swag (https://blather.michaelwlucas.com/archives/2962) Groff Fundraiser Pins: During the conference, You could get a unique Groff pin, by donating more than the last person to either the FreeBSD or OpenBSD foundation Michael W. Lucas and his wife Liz donated some interesting home made and local items to the infamous Charity Auction I donated the last remaining copy of the “Canadian Edition” of “FreeBSD Mastery: Advanced ZedFS”, and a Pentium G4400 (Skylake) CPU (Supports ECC or non-ECC) Peter Hessler donated his pen (Have you read “Git Commit Murder” yet?) Theo De Raadt donated his autographed conference badge David Maxwell donated a large print of the group photo from last years FreeBSD Developers Summit, which was purchased by Allan There was also a FreeBSD Dev Summit T-Shirt (with the Slogan: What is Core doing about it?) autographed by all of the attending members of core, with a forged jhb@ signature. Lastly, someone wrote “I
A pledge of love to OpenBSD, combating ransomware like WannaCry with OpenZFS, and using PFsense to maximize your non-gigabit Internet connection This episode was brought to you by Headlines ino64 project committed to FreeBSD 12-CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=318736) The ino64 project has been completed and merged into FreeBSD 12-CURRENT Extend the inot, devt, nlinkt types to 64-bit ints. Modify struct dirent layout to add doff, increase the size of dfileno to 64-bits, increase the size of dnamlen to 16-bits, and change the required alignment. Increase struct statfs fmntfromname[] and fmntonname[] array length MNAMELEN to 1024 This means the length of a mount point (MNAMELEN) has been increased from 88 byte to 1024 bytes. This allows longer ZFS dataset names and more nesting, and generally improves the usefulness of nested jails It also allow more than 4 billion files to be stored in a single file system (both UFS and ZFS). It also deals with a number of NFS problems, such as Amazon's EFS (cloud NFS), which uses 64 bit IDs even with small numbers of files. ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and by employing other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat around are broken in backward and forward incompatible ways. A bug in poudriere that may cause some packages to not rebuild is being fixed. Many packages like perl will need to be rebuilt after this change Update note: strictly follow the instructions in UPDATING. Build and install the new kernel with COMPAT_FREEBSD11 option enabled, then reboot, and only then install new world. So you need the new GENERIC kernel with the COMPAT_FREEBSD11 option, so that your old userland will work with the new kernel, and you need to build, install, and reboot onto the new kernel before attempting to install world. The usual process of installing both and then rebooting will NOT WORK Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kmoore) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib). Sponsored by: The FreeBSD Foundation (emaste, kib) Why I love OpenBSD (https://medium.com/@h3artbl33d/why-i-love-openbsd-ca760cf53941) Jeroen Janssen writes: I do love open source software. Oh boy, I really do love open source software. It's extendable, auditable, and customizable. What's not to love? I'm astonished by the idea that tens, hundreds, and sometimes even thousands of enthusiastic, passionate developers collaborate on an idea. Together, they make the world a better place, bit by bit. And this leads me to one of my favorite open source projects: the 22-year-old OpenBSD operating system. The origins of my love affair with OpenBSD From Linux to *BSD The advantages of OpenBSD It's extremely secure It's well documented It's open source > It's neat and clean My take on OpenBSD ** DO ** Combating WannaCry and Other Ransomware with OpenZFS Snapshots (https://www.ixsystems.com/blog/combating-ransomware/) Ransomware attacks that hold your data hostage using unauthorized data encryption are spreading rapidly and are particularly nefarious because they do not require any special access privileges to your data. A ransomware attack may be launched via a sophisticated software exploit as was the case with the recent “WannaCry” ransomware, but there is nothing stopping you from downloading and executing a malicious program that encrypts every file you have access to. If you fail to pay the ransom, the result will be indistinguishable from your simply deleting every file on your system. To make matters worse, ransomware authors are expanding their attacks to include just about any storage you have access to. The list is long, but includes network shares, Cloud services like DropBox, and even “shadow copies” of data that allow you to open previous versions of files. To make matters even worse, there is little that your operating system can do to prevent you or a program you run from encrypting files with ransomware just as it can't prevent you from deleting the files you own. Frequent backups are touted as one of the few effective strategies for recovering from ransomware attacks but it is critical that any backup be isolated from the attack to be immune from the same attack. Simply copying your files to a mounted disk on your computer or in the Cloud makes the backup vulnerable to infection by virtue of the fact that you are backing up using your regular permissions. If you can write to it, the ransomware can encrypt it. Like medical workers wearing hazmat suits for isolation when combating an epidemic, you need to isolate your backups from ransomware. OpenZFS snapshots to the rescue OpenZFS is the powerful file system at the heart of every storage system that iXsystems sells and of its many features, snapshots can provide fast and effective recovery from ransomware attacks at both the individual user and enterprise level as I talked about in 2015. As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks. OpenZFS snapshots in practice While OpenZFS is available on a number of desktop operating systems such as TrueOS and macOS, the most effective way to bring the benefits of OpenZFS snapshots to the largest number of users is with a network of iXsystems TrueNAS, FreeNAS Certified and FreeNAS Mini unified NAS and SAN storage systems. All of these can provide OpenZFS-backed SMB, NFS, AFP, and iSCSI file and block storage to the smallest workgroups up through the largest enterprises and TrueNAS offers available Fibre Channel for enterprise deployments. By sharing your data to your users using these file and block protocols, you can provide them with a storage infrastructure that can quickly recover from any ransomware attack thrown at it. To mitigate ransomware attacks against individual workstations, TrueNAS and FreeNAS can provide snapshotted storage to your VDI or virtualization solution of choice. Best of all, every iXsystems TrueNAS, FreeNAS Certified, and FreeNAS Mini system includes a consistent user interface and the ability to replicate between one another. This means that any topology of individual offices and campuses can exchange backup data to quickly mitigate ransomware attacks on your organization at all levels. Join us for a free webinar (http://www.onlinemeetingnow.com/register/?id=uegudsbc75) with iXsystems Co-Founder Matt Olander and learn more about why businesses everywhere are replacing their proprietary storage platforms with TrueNAS then email us at info@ixsystems.com or call 1-855-GREP-4-IX (1-855-473-7449), or 1-408-493-4100 (outside the US) to discuss your storage needs with one of our solutions architects. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @twitter (https://twitter.com/mwlauthor) Books, conferences, and how these two combine + BR: Welcome back. Tell us what you've been up to since the last time we interviewed you regarding books and such. + AJ: Tell us a little bit about relayd and what it can do. + BR: What other books do you have in the pipeline? + AJ: What are your criteria that qualifies a topic for a mastery book? + BR: Can you tell us a little bit about these writing workshops that you attend and what happens there? + AJ: Without spoiling too much: How did you come up with the idea for git commit murder? + BR: Speaking of BSDCan, can you tell the first timers about what to expect in the http://www.bsdcan.org/2017/schedule/events/890.en.html (Newcomers orientation and mentorship) session on Thursday? + AJ: Tell us about the new WIP session at BSDCan. Who had the idea and how much input did you get thus far? + BR: Have you ever thought about branching off into a new genre like children's books or medieval fantasy novels? + AJ: Is there anything else before we let you go? News Roundup Using LLDP on FreeBSD (https://tetragir.com/freebsd/networking/using-lldp-on-freebsd.html) LLDP, or Link Layer Discovery Protocol allows system administrators to easily map the network, eliminating the need to physically run the cables in a rack. LLDP is a protocol used to send and receive information about a neighboring device connected directly to a networking interface. It is similar to Cisco's CDP, Foundry's FDP, Nortel's SONMP, etc. It is a stateless protocol, meaning that an LLDP-enabled device sends advertisements even if the other side cannot do anything with it. In this guide the installation and configuration of the LLDP daemon on FreeBSD as well as on a Cisco switch will be introduced. If you are already familiar with Cisco's CDP, LLDP won't surprise you. It is built for the same purpose: to exchange device information between peers on a network. While CDP is a proprietary solution and can be used only on Cisco devices, LLDP is a standard: IEEE 802.3AB. Therefore it is implemented on many types of devices, such as switches, routers, various desktop operating systems, etc. LLDP helps a great deal in mapping the network topology, without spending hours in cabling cabinets to figure out which device is connected with which switchport. If LLDP is running on both the networking device and the server, it can show which port is connected where. Besides physical interfaces, LLDP can be used to exchange a lot more information, such as IP Address, hostname, etc. In order to use LLDP on FreeBSD, net-mgmt/lldpd has to be installed. It can be installed from ports using portmaster: #portmaster net-mgmt/lldpd Or from packages: #pkg install net-mgmt/lldpd By default lldpd sends and receives all the information it can gather , so it is advisable to limit what we will communicate with the neighboring device. The configuration file for lldpd is basically a list of commands as it is passed to lldpcli. Create a file named lldpd.conf under /usr/local/etc/ The following configuration gives an example of how lldpd can be configured. For a full list of options, see %man lldpcli To check what is configured locally, run #lldpcli show chassis detail To see the neighbors run #lldpcli show neighbors details Check out the rest of the article about enabling LLDP on a Cisco switch experiments with prepledge (http://www.tedunangst.com/flak/post/experiments-with-prepledge) Ted Unangst takes a crack at a system similar to the one being designed for Capsicum, Oblivious Sandboxing (See the presentation at BSDCan), where the application doesn't even know it is in the sandbox MP3 is officially dead, so I figure I should listen to my collection one last time before it vanishes entirely. The provenance of some of these files is a little suspect however, and since I know one shouldn't open files from strangers, I'd like to take some precautions against malicious malarkey. This would be a good use for pledge, perhaps, if we can get it working. At the same time, an occasional feature request for pledge is the ability to specify restrictions before running a program. Given some untrusted program, wrap its execution in a pledge like environment. There are other system call sandbox mechanisms that can do this (systrace was one), but pledge is quite deliberately designed not to support this. But maybe we can bend it to our will. Our pledge wrapper can't be an external program. This leaves us with the option of injecting the wrapper into the target program via LD_PRELOAD. Before main even runs, we'll initialize what needs initializing, then lock things down with a tight pledge set. Our eventual target will be ffplay, but hopefully the design will permit some flexibility and reuse. So the new code is injected to override the open syscall, and reads a list of files from an environment variable. Those files are opened and the path and file descriptor are put into a linked list, and then pledge is used to restrict further access to the file system. The replacement open call now searches just that linked list, returning the already opened file descriptors. So as long as your application only tries to open files that you have preopened, it can function without modification within the sandbox. Or at least that is the goal... ffplay tries to dlopen() some things, and because of the way dlopen() works, it doesn't go via the libc open() wrapper, so it doesn't get overridden ffplay also tries to call a few ioctl's, not allowed After stubbing both of those out, it still doesn't work and it is just getting worse Ted switches to a new strategy, using ffmpeg to convert the .mp3 to a .wav file and then just cat it to /dev/audio A few more stubs for ffmpeg, including access(), and adding tty access to the list of pledges, and it finally works This point has been made from the early days, but I think this exercise reinforces it, that pledge works best with programs where you understand what the program is doing. A generic pledge wrapper isn't of much use because the program is going to do something unexpected and you're going to have a hard time wrangling it into submission. Software is too complex. What in the world is ffplay doing? Even if I were working with the source, how long would it take to rearrange the program into something that could be pledged? One can try using another program, but I would wager that as far as multiformat media players go, ffplay is actually on the lower end of the complexity spectrum. Most of the trouble comes from using SDL as an abstraction layer, which performs a bunch of console operations. On the flip side, all of this early init code is probably the right design. Once SDL finally gets its screen handle setup, we could apply pledge and sandbox the actual media decoder. That would be the right way to things. Is pledge too limiting? Perhaps, but that's what I want. I could have just kept adding permissions until ffplay had full access to my X socket, but what kind of sandbox is that? I don't want naughty MP3s scraping my screen and spying on my keystrokes. The sandbox I created had all the capabilities one needs to convert an MP3 to audible sound, but the tool I wanted to use wasn't designed to work in that environment. And in its defense, these were new post hoc requirements. Other programs, even sed, suffer from less than ideal pledge sets as well. The best summary might be to say that pledge is designed for tomorrow's programs, not yesterday's (and vice versa). There were a few things I could have done better. In particular, I gave up getting audio to work, even though there's a nice description of how to work with pledge in the sio_open manual. Alas, even going back and with a bit more effort I still haven't succeeded. The requirements to use libsndio are more permissive than I might prefer. How I Maximized the Speed of My Non-Gigabit Internet Connection (https://medium.com/speedtest-by-ookla/engineer-maximizes-internet-speed-story-c3ec0e86f37a) We have a new post from Brennen Smith, who is the Lead Systems Engineer at Ookla, the company that runs Speedtest.net, explaining how he used pfSense to maximize his internet connection I spend my time wrangling servers and internet infrastructure. My daily goals range from designing high performance applications supporting millions of users and testing the fastest internet connections in the world, to squeezing microseconds from our stack —so at home, I strive to make sure that my personal internet performance is running as fast as possible. I live in an area with a DOCSIS ISP that does not provide symmetrical gigabit internet — my download and upload speeds are not equal. Instead, I have an asymmetrical plan with 200 Mbps download and 10 Mbps upload — this nuance considerably impacted my network design because asymmetrical service can more easily lead to bufferbloat. We will cover bufferbloat in a later article, but in a nutshell, it's an issue that arises when an upstream network device's buffers are saturated during an upload. This causes immense network congestion, latency to rise above 2,000 ms., and overall poor quality of internet. The solution is to shape the outbound traffic to a speed just under the sending maximum of the upstream device, so that its buffers don't fill up. My ISP is notorious for having bufferbloat issues due to the low upload performance, and it's an issue prevalent even on their provided routers. They walk through a list of router devices you might consider, and what speeds they are capable of handling, but ultimately ended up using a generic low power x86 machine running pfSense 2.3 In my research and testing, I also evaluated IPCop, VyOS, OPNSense, Sophos UTM, RouterOS, OpenWRT x86, and Alpine Linux to serve as the base operating system, but none were as well supported and full featured as PFSense. The main setting to look at is the traffic shaping of uploads, to keep the pipe from getting saturated and having a large buffer build up in the modem and further upstream. This build up is what increases the latency of the connection As with any experiment, any conclusions need to be backed with data. To validate the network was performing smoothly under heavy load, I performed the following experiment: + Ran a ping6 against speedtest.net to measure latency. + Turned off QoS to simulate a “normal router”. + Started multiple simultaneous outbound TCP and UDP streams to saturate my outbound link. + Turned on QoS to the above settings and repeated steps 2 and 3. As you can see from the plot below, without QoS, my connection latency increased by ~1,235%. However with QoS enabled, the connection stayed stable during the upload and I wasn't able to determine a statistically significant delta. That's how I maximized the speed on my non-gigabit internet connection. What have you done with your network? FreeBSD on 11″ MacBook Air (https://www.geeklan.co.uk/?p=2214) Sevan Janiyan writes in his tech blog about his experiences running FreeBSD on an 11'' MacBook Air This tiny machine has been with me for a few years now, It has mostly run OS X though I have tried OpenBSD on it (https://www.geeklan.co.uk/?p=1283). Besides the screen resolution I'm still really happy with it, hardware wise. Software wise, not so much. I use an external disk containing a zpool with my data on it. Among this data are several source trees. CVS on a ZFS filesystem on OS X is painfully slow. I dislike that builds running inside Terminal.app are slow at the expense of a responsive UI. The system seems fragile, at the slightest push the machine will either hang or become unresponsive. Buggy serial drivers which do not implement the break signal and cause instability are frustrating. Last week whilst working on Rump kernel (http://rumpkernel.org/) builds I introduced some new build issues in the process of fixing others, I needed to pick up new changes from CVS by updating my copy of the source tree and run builds to test if issues were still present. I was let down on both counts, it took ages to update source and in the process of cross compiling a NetBSD/evbmips64-el release, the system locked hard. That was it, time to look what was possible elsewhere. While I have been using OS X for many years, I'm not tied to anything exclusive on it, maybe tweetbot, perhaps, but that's it. On the BSDnow podcast they've been covering changes coming in to TrueOS (formerly PC-BSD – a desktop focused distro based on FreeBSD), their experiments seemed interesting, the project now tracks FreeBSD-CURRENT, they've replaced rcng with OpenRC as the init system and it comes with a pre-configured desktop environment, using their own window manager (Lumina). Booting the USB flash image it made it to X11 without any issue. The dock has a widget which states the detected features, no wifi (Broadcom), sound card detected and screen resolution set to 1366×768. I planned to give it a try on the weekend. Friday, I made backups and wiped the system. TrueOS installed without issue, after a short while I had a working desktop, resuming from sleep worked out of the box. I didn't spend long testing TrueOS, switching out NetBSD-HEAD only to realise that I really need ZFS so while I was testing things out, might as well give stock FreeBSD 11-STABLE a try (TrueOS was based on -CURRENT). Turns out sleep doesn't work yet but sound does work out of the box and with a few invocations of pkg(8) I had xorg, dwm, firefox, CVS and virtuabox-ose installed from binary packages. VirtualBox seems to cause the system to panic (bug 219276) but I should be able to survive without my virtual machines over the next few days as I settle in. I'm considering ditching VirtualBox and converting the vdi files to raw images so that they can be written to a new zvol for use with bhyve. As my default keyboard layout is Dvorak, OS X set the EFI settings to this layout. The first time I installed FreeBSD 11-STABLE, I opted for full disk encryption but ran into this odd issue where on boot the keyboard layout was Dvorak and password was accepted, the system would boot and as it went to mount the various filesystems it would switch back to QWERTY. I tried entering my password with both layout but wasn't able to progress any further, no bug report yet as I haven't ruled myself out as the problem. Thunderbolt gigabit adapter –bge(4) (https://www.freebsd.org/cgi/man.cgi?query=bge) and DVI adapter both worked on FreeBSD though the gigabit adapter needs to be plugged in at boot to be detected. The trackpad bind to wsp(4) (https://www.freebsd.org/cgi/man.cgi?query=wsp), left, right and middle clicks are available through single, double and tripple finger tap. Sound card binds to snd_hda(4) (https://www.freebsd.org/cgi/man.cgi?query=snd_hda) and works out of the box. For wifi I'm using a urtw(4) (https://www.freebsd.org/cgi/man.cgi?query=urtw) Alfa adapter which is a bit on the large side but works very reliably. A copy of the dmesg (https://www.geeklan.co.uk/files/macbookair/freebsd-dmesg.txt) is here. Beastie Bits OPNsense - call-for-testing for SafeStack (https://forum.opnsense.org/index.php?topic=5200.0) BSD 4.4: cat (https://www.rewritinghistorycasts.com/screencasts/bsd-4.4:-cat) Continuous Unix commit history from 1970 until today (https://github.com/dspinellis/unix-history-repo) Update on Unix Architecture Evolution Diagrams (https://www.spinellis.gr/blog/20170510/) “Relayd and Httpd Mastery” is out! (https://blather.michaelwlucas.com/archives/2951) Triangle BSD User Group Meeting -- libxo (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) *** Feedback/Questions Carlos - ASUS Tinkerboard (http://dpaste.com/1GJHPNY#wrap) James - Firewall question (http://dpaste.com/0QCW933#wrap) Adam - ZFS books (http://dpaste.com/0GMG5M2#wrap) David - Managing zvols (http://dpaste.com/2GP8H1E#wrap) ***
This week on BSDNow, reports from AsiaBSDcon, TrueOS and FreeBSD news, Optimizing IllumOS Kernel, your questions and more. This episode was brought to you by Headlines AsiaBSDcon Reports and Reviews () AsiaBSDcon schedule (https://2017.asiabsdcon.org/program.html.en) Schedule and slides from the 4th bhyvecon (http://bhyvecon.org/) Michael Dexter's trip report on the iXsystems blog (https://www.ixsystems.com/blog/ixsystems-attends-asiabsdcon-2017) NetBSD AsiaBSDcon booth report (http://mail-index.netbsd.org/netbsd-advocacy/2017/03/13/msg000729.html) *** TrueOS Community Guidelines are here! (https://www.trueos.org/blog/trueos-community-guidelines/) TrueOS has published its new Community Guidelines The TrueOS Project has existed for over ten years. Until now, there was no formally defined process for interested individuals in the TrueOS community to earn contributor status as an active committer to this long-standing project. The current core TrueOS developers (Kris Moore, Ken Moore, and Joe Maloney) want to provide the community more opportunities to directly impact the TrueOS Project, and wish to formalize the process for interested people to gain full commit access to the TrueOS repositories. These describe what is expected of community members and committers They also describe the process of getting commit access to the TrueOS repo: Previously, Kris directly handed out commit bits. Now, the Core developers have provided a small list of requirements for gaining a TrueOS commit bit: Create five or more pull requests in a TrueOS Project repository within a single six month period. Stay active in the TrueOS community through at least one of the available community channels (Gitter, Discourse, IRC, etc.). Request commit access from the core developers via core@trueos.org OR Core developers contact you concerning commit access. Pull requests can be any contribution to the project, from minor documentation tweaks to creating full utilities. At the end of every month, the core developers review the commit logs, removing elements that break the Project or deviate too far from its intended purpose. Additionally, outstanding pull requests with no active dissension are immediately merged, if possible. For example, a user submits a pull request which adds a little-used OpenRC script. No one from the community comments on the request or otherwise argues against its inclusion, resulting in an automatic merge at the end of the month. In this manner, solid contributions are routinely added to the project and never left in a state of “limbo”. The page also describes the perks of being a TrueOS committer: Contributors to the TrueOS Project enjoy a number of benefits, including: A personal TrueOS email alias: @trueos.org Full access for managing TrueOS issues on GitHub. Regular meetings with the core developers and other contributors. Access to private chat channels with the core developers. Recognition as part of an online Who's Who of TrueOS developers. The eternal gratitude of the core developers of TrueOS. A warm, fuzzy feeling. Intel Donates 250.000 $ to the FreeBSD Foundation (https://www.freebsdfoundation.org/news-and-events/latest-news/new-uranium-level-donation-and-collaborative-partnership-with-intel/) More details about the deal: Systems Thinking: Intel and the FreeBSD Project (https://www.freebsdfoundation.org/blog/systems-thinking-intel-and-the-freebsd-project/) Intel will be more actively engaging with the FreeBSD Foundation and the FreeBSD Project to deliver more timely support for Intel products and technologies in FreeBSD. Intel has contributed code to FreeBSD for individual device drivers (i.e. NICs) in the past, but is now seeking a more holistic “systems thinking” approach. Intel Blog Post (https://01.org/blogs/imad/2017/intel-increases-support-freebsd-project) We will work closely with the FreeBSD Foundation to ensure the drivers, tools, and applications needed on Intel® SSD-based storage appliances are available to the community. This collaboration will also provide timely support for future Intel® 3D XPoint™ products. Thank you very much, Intel! *** Applied FreeBSD: Basic iSCSI (https://globalengineer.wordpress.com/2017/03/05/applied-freebsd-basic-iscsi/) iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered. The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it. Then, enable ctld and start it: sysrc ctld_enable=”YES” service ctld start You can use the ctladm command to see what is going on: root@bsdtarget:/dev # ctladm lunlist (7:0:0/0): Fixed Direct Access SPC-4 SCSI device (7:0:1/1): Fixed Direct Access SPC-4 SCSI device root@bsdtarget:/dev # ctladm devlist LUN Backend Size (Blocks) BS Serial Number Device ID 0 block 10485760 512 MYSERIAL 0 MYDEVID 0 1 block 10485760 512 MYSERIAL 1 MYDEVID 1 Now, let's configure the client side: In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. sysrc iscsid_enable=”YES” service iscsid start Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session: + iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget You should now have a new device (check dmesg), in this case, da1 The guide them walks through partitioning the disk, and laying down a UFS file system, and mounting it This it walks through how to disconnect iscsi, incase you don't want it anymore This all looked nice and easy, and it works very well. Now lets see what happens when you try to mount the iSCSI from Windows Ok, that wasn't so bad. Now, instead of sharing an entire space disk on the host via iSCSI, share a zvol. Now your windows machine can be backed by ZFS. All of your problems are solved. Interview - Philipp Buehler - pbuehler@sysfive.com (mailto:pbuehler@sysfive.com) Technical Lead at SysFive, and Former OpenBSD Committer News Roundup Half a dozen new features in mandoc -T html (http://undeadly.org/cgi?action=article&sid=20170316080827) mandoc (http://man.openbsd.org/mandoc.1)'s HTML output mode got some new features Even though mdoc(7) is a semantic markup language, traditionally none of the semantic annotations were communicated to the reader. [...] Now, at least in -T html output mode, you can see the semantic function of marked-up words by hovering your mouse over them. In terminal output modes, we have the ctags(1)-like internal search facility built around the less(1) tag jump (:t) feature for quite some time now. We now have a similar feature in -T html output mode. To jump to (almost) the same places in the text, go to the address bar of the browser, type a hash mark ('#') after the URI, then the name of the option, command, variable, error code etc. you want to jump to, and hit enter. Check out the full report by Ingo Schwarze (schwarze@) and try out these new features *** Optimizing IllumOS Kernel Crypto (http://zfs-create.blogspot.com/2014/05/optimizing-illumos-kernel-crypto.html) Sašo Kiselkov, of ZFS fame, looked into the performance of the OpenSolaris kernel crypto framework and found it lacking. The article also spends a few minutes on the different modes and how they work. Recently I've had some motivation to look into the KCF on Illumos and discovered that, unbeknownst to me, we already had an AES-NI implementation that was automatically enabled when running on Intel and AMD CPUs with AES-NI support. This work was done back in 2010 by Dan Anderson.This was great news, so I set out to test the performance in Illumos in a VM on my Mac with a Core i5 3210M (2.5GHz normal, 3.1GHz turbo). The initial tests of “what the hardware can do” were done in OpenSSL So now comes the test for the KCF. I wrote a quick'n'dirty crypto test module that just performed a bunch of encryption operations and timed the results. KCF got around 100 MB/s for each algorithm, except half that for AES-GCM OpenSSL had done over 3000 MB/s for CTR mode, 500 MB/s for CBC, and 1000 MB/s for GCM What the hell is that?! This is just plain unacceptable. Obviously we must have hit some nasty performance snag somewhere, because this is comical. And sure enough, we did. When looking around in the AES-NI implementation I came across this bit in aes_intel.s that performed the CLTS instruction. This is a problem: 3.1.2 Instructions That Cause VM Exits ConditionallyCLTS. The CLTS instruction causes a VM exit if the bits in position 3 (corresponding to CR0.TS) are set in both the CR0 guest/host mask and the CR0 read shadow. The CLTS instruction signals to the CPU that we're about to use FPU registers (which is needed for AES-NI), which in VMware causes an exit into the hypervisor. And we've been doing it for every single AES block! Needless to say, performing the equivalent of a very expensive context switch every 16 bytes is going to hurt encryption performance a bit. The reason why the kernel is issuing CLTS is because for performance reasons, the kernel doesn't save and restore FPU register state on kernel thread context switches. So whenever we need to use FPU registers inside the kernel, we must disable kernel thread preemption via a call to kpreemptdisable() and kpreemptenable() and save and restore FPU register state manually. During this time, we cannot be descheduled (because if we were, some other thread might clobber our FPU registers), so if a thread does this for too long, it can lead to unexpected latency bubbles The solution was to restructure the AES and KCF block crypto implementations in such a way that we execute encryption in meaningfully small chunks. I opted for 32k bytes, for reasons which I'll explain below. Unfortunately, doing this restructuring work was a bit more complicated than one would imagine, since in the KCF the implementation of the AES encryption algorithm and the block cipher modes is separated into two separate modules that interact through an internal API, which wasn't really conducive to high performance (we'll get to that later). Anyway, having fixed the issue here and running the code at near native speed, this is what I get: AES-128/CTR: 439 MB/s AES-128/CBC: 483 MB/s AES-128/GCM: 252 MB/s Not disastrous anymore, but still, very, very bad. Of course, you've got keep in mind, the thing we're comparing it to, OpenSSL, is no slouch. It's got hand-written highly optimized inline assembly implementations of most of these encryption functions and their specific modes, for lots of platforms. That's a ton of code to maintain and optimize, but I'll be damned if I let this kind of performance gap persist. Fixing this, however, is not so trivial anymore. It pertains to how the KCF's block cipher mode API interacts with the cipher algorithms. It is beautifully designed and implemented in a fashion that creates minimum code duplication, but this also means that it's inherently inefficient. ECB, CBC and CTR gained the ability to pass an algorithm-specific "fastpath" implementation of the block cipher mode, because these functions benefit greatly from pipelining multiple cipher calls into a single place. ECB, CTR and CBC decryption benefit enormously from being able to exploit the wide XMM register file on Intel to perform encryption/decryption operations on 8 blocks at the same time in a non-interlocking manner. The performance gains here are on the order of 5-8x.CBC encryption benefits from not having to copy the previously encrypted ciphertext blocks into memory and back into registers to XOR them with the subsequent plaintext blocks, though here the gains are more modest, around 1.3-1.5x. After all of this work, this is how the results now look on Illumos, even inside of a VM: Algorithm/Mode 128k ops AES-128/CTR: 3121 MB/s AES-128/CBC: 691 MB/s AES-128/GCM: 1053 MB/s So the CTR and GCM speeds have actually caught up to OpenSSL, and CBC is actually faster than OpenSSL. On the decryption side of things, CBC decryption also jumped from 627 MB/s to 3011 MB/s. Seeing these performance numbers, you can see why I chose 32k for the operation size in between kernel preemption barriers. Even on the slowest hardware with AES-NI, we can expect at least 300-400 MB/s/core of throughput, so even in the worst case, we'll be hogging the CPU for at most ~0.1ms per run. Overall, we're even a little bit faster than OpenSSL in some tests, though that's probably down to us encrypting 128k blocks vs 8k in the "openssl speed" utility. Anyway, having fixed this monstrous atrocity of a performance bug, I can now finally get some sleep. To made these tests repeatable, and to ensure that the changes didn't break the crypto algorithms, Saso created a crypto_test kernel module. I have recently created a FreeBSD version of crypto_test.ko, for much the same purposes Initial performance on FreeBSD is not as bad, if you have the aesni.ko module loaded, but it is not up to speed with OpenSSL. You cannot directly compare to the benchmarks Saso did, because the CPUs are vastly different. Performance results (https://wiki.freebsd.org/OpenCryptoPerformance) I hope to do some more tests on a range of different sized CPUs in order to determine how the algorithms scale across different clock speeds. I also want to look at, or get help and have someone else look at, implementing some of the same optimizations that Saso did. It currently seems like there isn't a way to perform addition crypto operations in the same session without regenerating the key table. Processing additional buffers in an existing session might offer a number of optimizations for bulk operations, although in many cases, each block is encrypted with a different key and/or IV, so it might not be very useful. *** Brendan Gregg's special freeware tools for sysadmins (http://www.brendangregg.com/specials.html) These tools need to be in every (not so) serious sysadmins toolbox. Triple ROT13 encryption algorithm (beware: export restrictions may apply) /usr/bin/maybe, in case true and false don't provide too little choice... The bottom command lists you all the processes using the least CPU cycles. Check out the rest of the tools. You wrote similar tools and want us to cover them in the show? Send us an email to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** A look at 2038 (http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/) I remember the Y2K problem quite vividly. The world was going crazy for years, paying insane amounts of money to experts to fix critical legacy systems, and there was a neverending stream of predictions from the media on how it's all going to fail. Most didn't even understand what the problem was, and I remember one magazine writing something like the following: Most systems store the current year as a two-digit value to save space. When the value rolls over on New Year's Eve 1999, those two digits will be “00”, and “00” means “halt operation” in the machine language of many central processing units. If you're in an elevator at this time, it will stop working and you may fall to your death. I still don't know why they thought a computer would suddenly interpret data as code, but people believed them. We could see a nearby hydropower plant from my parents' house, and we expected it to go up in flames as soon as the clock passed midnight, while at least two airplanes crashed in our garden at the same time. Then nothing happened. I think one of the most “severe” problems was the police not being able to open their car garages the next day because their RFID tokens had both a start and end date for validity, and the system clock had actually rolled over to 1900, so the tokens were “not yet valid”. That was 17 years ago. One of the reasons why Y2K wasn't as bad as it could have been is that many systems had never used the “two-digit-year” representation internally, but use some form of “timestamp” relative to a fixed date (the “epoch”). The actual problem with time and dates rolling over is that systems calculate timestamp differences all day. Since a timestamp derived from the system clock seemingly only increases with each query, it is very common to just calculate diff = now - before and never care about the fact that now could suddenly be lower than before because the system clock has rolled over. In this case diff is suddenly negative, and if other parts of the code make further use of the suddenly negative value, things can go horribly wrong. A good example was a bug in the generator control units (GCUs) aboard Boeing 787 “Dreamliner” aircrafts, discovered in 2015. An internal timestamp counter would overflow roughly 248 days after the system had been powered on, triggering a shut down to “safe mode”. The aircraft has four generator units, but if all were powered up at the same time, they would all fail at the same time. This sounds like an overflow caused by a signed 32-bit counter counting the number of centiseconds since boot, overflowing after 248.55 days, and luckily no airline had been using their Boing 787 models for such a long time between maintenance intervals. The “obvious” solution is to simply switch to 64-Bit values and call it day, which would push overflow dates far into the future (as long as you don't do it like the IBM S/370 mentioned before). But as we've learned from the Y2K problem, you have to assume that computer systems, computer software and stored data (which often contains timestamps in some form) will stay with us for much longer than we might think. The years 2036 and 2038 might be far in the future, but we have to assume that many of the things we make and sell today are going to be used and supported for more than just 19 years. Also many systems have to store dates which are far in the future. A 30 year mortgage taken out in 2008 could have already triggered the bug, and for some banks it supposedly did. sysgettimeofday() is one of the most used system calls on a generic Linux system and returns the current time in form of an UNIX timestamp (timet data type) plus fraction (susecondst data type). Many applications have to know the current time and date to do things, e.g. displaying it, using it in game timing loops, invalidating caches after their lifetime ends, perform an action after a specific moment has passed, etc. In a 32-Bit UNIX system, timet is usually defined as a signed 32-Bit Integer. When kernel, libraries and applications are compiled, the compiler will turn this assumption machine code and all components later have to match each other. So a 32-Bit Linux application or library still expects the kernel to return a 32-Bit value even if the kernel is running on a 64-Bit architecture and has 32-Bit compatibility. The same holds true for applications calling into libraries. This is a major problem, because there will be a lot of legacy software running in 2038. Systems which used an unsigned 32-Bit Integer for timet push the problem back to 2106, but I don't know about many of those. The developers of the GNU C library (glibc), the default standard C library for many GNU/Linux systems, have come up with a design for year 2038 proofness for their library. Besides the timet data type itself, a number of other data structures have fields based on timet or the combined struct timespec and struct timeval types. Many methods beside those intended for setting and querying the current time use timestamps 32-Bit Windows applications, or Windows applications defining _USE32BITTIMET, can be hit by the year 2038 problem too if they use the timet data type. The _time64t data type had been available since Visual C 7.1, but only Visual C 8 (default with Visual Studio 2015) expanded timet to 64 bits by default. The change will only be effective after a recompilation, legacy applications will continue to be affected. If you live in a 64-Bit world and use a 64-Bit kernel with 64-Bit only applications, you might think you can just ignore the problem. In such a constellation all instances of the standard time_t data type for system calls, libraries and applications are signed 64-Bit Integers which will overflow in around 292 billion years. But many data formats, file systems and network protocols still specify 32-Bit time fields, and you might have to read/write this data or talk to legacy systems after 2038. So solving the problem on your side alone is not enough. Then the article goes on to describe how all of this will break your file systems. Not to mention your databases and other file formats. Also see Theo De Raadt's EuroBSDCon 2013 Presentation (https://www.openbsd.org/papers/eurobsdcon_2013_time_t/mgp00001.html) *** Beastie Bits Michael Lucas: Get your name in “Absolute FreeBSD 3rd Edition” (https://blather.michaelwlucas.com/archives/2895) ZFS compressed ARC stats to top (https://svnweb.freebsd.org/base?view=revision&revision=r315435) Matthew Dillon discovered HAMMER was repeating itself when writing to disk. Fixing that issue doubled write speeds (https://www.dragonflydigest.com/2017/03/14/19452.html) TedU on Meaningful Short Names (http://www.tedunangst.com/flak/post/shrt-nms-fr-clrty) vBSDcon and EuroBSDcon Call for Papers are open (https://www.freebsdfoundation.org/blog/submit-your-work-vbsdcon-and-eurobsdcon-cfps-now-open/) Feedback/Questions Craig asks about BSD server management (http://pastebin.com/NMshpZ7n) Michael asks about jails as a router between networks (http://pastebin.com/UqRwMcRk) Todd asks about connecting jails (http://pastebin.com/i1ZD6eXN) Dave writes in with an interesting link (http://pastebin.com/QzW5c9wV) > applications crash more often due to errors than corruptions. In the case of corruption, a few applications (e.g., Log-Cabin, ZooKeeper) can use checksums and redundancy to recover, leading to a correct behavior; however, when the corruption is transformed into an error, these applications crash, resulting in reduced availability. ***
This is a very special BSD Now! New exciting changes are coming to the show and we're gonna cover them, so stick around or you'll miss it! Interview – Kris Moore – kris@trueos.org / @pcbsdKrisTrueOS founder, FreeNAS developer, BSD Now co-hostBenedict Reuschling – bcr@freebsd.org / @bsdbcrFreeBSD commiter & FreeBSD Foundation Vice President, BSD Now co-host Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
This week on BSDNow, Allan and I are in Tokyo for AsiaBSDCon, but not to worry, we have a full episode lined up and ready to go. Hackathon reports This episode was brought to you by Headlines OpenBSD A2k17 hackathon reports a2k17 hackathon report: Patrick Wildt on the arm64 port (http://undeadly.org/cgi?action=article&sid=20170131101827) a2k17 hackathon report: Antoine Jacoutot on syspatch, rc.d improvements and more (http://undeadly.org/cgi?action=article&sid=20170203232049) a2k17 hackathon report: Martin Pieuchot on NET_LOCK and much more (http://undeadly.org/cgi?action=article&sid=20170127154356) a2k17 hackathon report: Kenneth Westerback on the hidden wonders of the build system, the network stack and more (http://undeadly.org/cgi?action=article&sid=20170127031836) a2k17 hackathon report: Bob Beck on LibreSSL progress and more (http://undeadly.org/cgi?action=article&sid=20170125225403) *** NetBSD is now reproducible (https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds) Christos Zoulas posts to the NetBSD blog that he has completed his project to make fully reproducible NetBSD builds for amd64 and sparc64 I have been working on and off for almost a year trying to get reproducible builds (the same source tree always builds an identical cdrom) on NetBSD. I did not think at the time it would take as long or be so difficult, so I did not keep a log of all the changes I needed to make. I was also not the only one working on this. Other NetBSD developers have been making improvements for the past 6 years. I would like to acknowledge the NetBSD build system (aka build.sh) which is a fully portable cross-build system. This build system has given us a head-start in the reproducible builds work. I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file. Finally other developers who have started, motivated and did a lot of work getting us here like Joerg Sonnenberger and Thomas Klausner for their work on reproducible builds, and Todd Vierling and Luke Mewburn for their work on build.sh. Some of the stumbling blocks that were overcome: Timestamps Date/time/author embedded in source files Timezone sensitive code Directory order / build order Non-sanitized data stored in files Symbolic links / paths General tool inconsistencies: including gcc profiling, the fact that GPT partition tables, are by definition, globally unique each time they are created, and the iso9660 standard calls for a timestamp with a timezone. Toolchain Build information / tunables / environment. NetBSD now has a knob ‘MKREPRO', if set to YES it sets a long list of variables to a consistent set of a values. The post walks through how these problems where solves Future Work: Vary more parameters and find more inconsistencies Verify that cross-building is reproducible Verify that unprivileged builds are reproducible Test on other platforms *** Features are faults redux (http://www.tedunangst.com/flak/post/features-are-faults-redux) From Ted Unangst Last week I gave a talk for the security class at Notre Dame based on features are faults but with some various commentary added. It was an exciting trip, with the opportunity to meet and talk with the computer vision group as well. Some other highlights include the Indiana skillet I had for breakfast, which came with pickles and was amazing, and explaining the many wonders of cvs to the Linux users group over lunch. After that came the talk, which went a little something like this. I got started with OpenBSD back about the same time I started college, although I had a slightly different perspective then. I was using OpenBSD because it included so many security features, therefore it must be the most secure system, right? For example, at some point I acquired a second computer. What's the first thing anybody does when they get a second computer? That's right, set up a kerberos domain. The idea that more is better was everywhere. This was also around the time that ipsec was getting its final touches, and everybody knew ipsec was going to be the most secure protocol ever because it had more options than any other secure transport. We'll revisit this in a bit. There's been a partial attitude adjustment since then, with more people recognizing that layering complexity doesn't result in more security. It's not an additive process. There's a whole talk there, about the perfect security that people can't or won't use. OpenBSD has definitely switched directions, including less code, not more. All the kerberos code was deleted a few years ago. Let's assume about one bug per 100 lines of code. That's probably on the low end. Now say your operating system has 100 million lines of code. If I've done the math correctly, that's literally a million bugs. So that's one reason to avoid adding features. But that's a solveable problem. If we pick the right language and the right compiler and the right tooling and with enough eyeballs and effort, we can fix all the bugs. We know how to build mostly correct software, we just don't care. As we add features to software, increasing its complexity, new unexpected behaviors start to emerge. What are the bounds? How many features can you add before craziness is inevitable? We can make some guesses. Less than a thousand for sure. Probably less than a hundred? Ten maybe? I'll argue the answer is quite possibly two. Interesting corollary is that it's impossible to have a program with exactly two features. Any program with two features has at least a third, but you don't know what it is My first example is a bug in the NetBSD ftp client. We had one feature, we added a second feature, and just like that we got a third misfeature (http://marc.info/?l=oss-security&m=141451507810253&w=2) Our story begins long ago. The origins of this bug are probably older than I am. In the dark times before the web, FTP sites used to be a pretty popular way of publishing files. You run an ftp client, connect to a remote site, and then you can browse the remote server somewhat like a local filesystem. List files, change directories, get files. Typically there would be a README file telling you what's what, but you don't need to download a copy to keep. Instead we can pipe the output to a program like more. Right there in the ftp client. No need to disconnect. Fast forward a few decades, and http is the new protocol of choice. http is a much less interactive protocol, but the ftp client has some handy features for batch downloads like progress bars, etc. So let's add http support to ftp. This works pretty well. Lots of code reused. http has one quirk however that ftp doesn't have. Redirects. The server can redirect the client to a different file. So now you're thinking, what happens if I download http://somefile and the server sends back 302 http://|reboot. ftp reconnects to the server, gets the 200, starts downloading and saves it to a file called |reboot. Except it doesn't. The function that saves files looks at the first character of the name and if it's a pipe, runs that command instead. And now you just rebooted your computer. Or worse. It's pretty obvious this is not the desired behavior, but where exactly did things go wrong? Arguably, all the pieces were working according to spec. In order to see this bug coming, you needed to know how the save function worked, you needed to know about redirects, and you needed to put all the implications together. The post then goes into a lot more detail about other issues. We just don't have time to cover it all today, but you should go read it, it is very enlightening What do we do about this? That's a tough question. It's much easier to poke fun at all the people who got things wrong. But we can try. My attitudes are shaped by experiences with the OpenBSD project, and I think we are doing a decent job of containing the complexity. Keep paring away at dependencies and reducing interactions. As a developer, saying “no” to all feature requests is actually very productive. It's so much faster than implementing the feature. Sometimes users complain, but I've often received later feedback from users that they'd come to appreciate the simplicity. There was a question about which of these vulnerabilities were found by researchers, as opposed to troublemakers. The answer was most, if not all of them, but it made me realize one additional point I hadn't mentioned. Unlike the prototypical buffer overflow vulnerability, exploiting features is very reliable. Exploiting something like shellshock or imagetragick requires no customized assembly and is independent of CPU, OS, version, stack alignment, malloc implementation, etc. Within about 24 hours of the initial release of shellshock, I had logs of people trying to exploit it. So unless you're on about a 12 hour patch cycle, you're going to have a bad time. reimplement zfsctl (.zfs) support (https://svnweb.freebsd.org/changeset/base/314048) avg@ (Andriy Gapon) has rewritten the .zfs support in FreeBSD The current code is written on top of GFS, a library with the generic support for writing filesystems, which was ported from Illumos. Because of significant differences between illumos VFS and FreeBSD VFS models, both the GFS and zfsctl code were heavily modified to work on FreeBSD. Nonetheless, they still contain quite a few ugly hacks and bugs. This is a reimplementation of the zfsctl code where the VFS-specific bits are written from scratch and only the code that interacts with the rest of ZFS is reused. Some ideas are picked from an independent work by Will (wca@) This work improves the overall quality of the ZFS port to FreeBSD The code that provides support for ZFS .zfs/ directory functionality has been reimplemented. It is no longer possible to create a snapshot by mkdir under .zfs/snapshot/. That should be the only user visible change. TIL: On IllumOS, you can create, rename, and destroy snapshots, by manipulating the virtual directories in the .zfs/snapshots directory. If enough people would find this feature useful, maybe it could be implemented (rm and rename have never existed on FreeBSD). At the same time, it seems like rather a lot of work, when the ZFS command line tools work so well. Although wca@ pointed out on IRC, it can be useful to be able to create a snapshot over NFS, or SMB. Interview - Konrad Witaszczyk - def@freebsd.org (mailto:def@freebsd.org) Encrypted Kernel Crash Dumps *** News Roundup PBKDF2 Performance improvements on FreeBSD (https://svnweb.freebsd.org/changeset/base/313962) Joe Pixton did some research (https://jbp.io/2015/08/11/pbkdf2-performance-matters/) and found that, because of the way the spec is written, most PBKDF2 implementations are 2x slower than they need to be. Since the PBKDF is used to derive a key, used for encryption, this poses a problem. The attacker can derive a key twice as fast as you can. On FreeBSD the PBKDF2 was configured to derive a SHA512-HMAC key that would take approximately 2 seconds to calculate. That is 2 seconds on one core. So an attacker can calculate the same key in 1 second, and use many cores. Luckily, 1 second is still a long time for each brute force guess. On modern CPUs with the fast algorithm, you can do about 500,000 iterations of PBKDF per second (per core). Until a recent change, OpenBSD used only 8192 iterations. It now uses a similar benchmark of ~2 seconds, and uses bcrypt instead of a SHA1-HMAC. Joe's research showed that the majority of implementations were done the ‘slow' way. Calculating the initial part of the outer round each iteration, instead of reusing the initial calculation over and over for each round. Joe submitted a match to FreeBSD to solve this problem. That patch was improved, and a test of tests were added by jmg@, but then work stalled I picked up the work, and fixed some merge conflicts in the patch that had cropped up based on work I had done that moved the HMAC code to a separate file. This work is now committed. With this change, all newly generated GELI keys will be approximately 2x as strong. Previously generated keys will take half as long to calculate, resulting in faster mounting of encrypted volumes. Users may choose to rekey, to generate a new key with the larger default number of iterations using the geli(8) setkey command. Security of existing data is not compromised, as ~1 second per brute force attempt is still a very high threshold. If you are interested in the topic, I recommend the video of Joe's presentation from the Passwords15 conference in Las Vegas *** Quick How-To: Updating a screenshot in the TrueOS Handbook (https://www.trueos.org/blog/quick-updating-screenshot-trueos-handbook/) Docs writers, might be time to pay attention. This week we have a good walk-through of adding / updating new screenshots to the TrueOS Sphinx Documentation. For those who have not looked in the past, TrueOS and FreeNAS both have fantastic docs by the team over at iXsystems using Sphinx as their doc engine. Often we get questions from users asking what “they can do to help” but don't necessarily have programming skills to apply. The good news is that using Sphinx is relatively easy, and after learning some minio rst syntax you can easily help fix, or even contribute to new sections of the TrueOS (Or FreeNAS) documentation. In this example, Tim takes us through the process of replacing an old out of date screenshot in the handbook with the latest hotness. Starting with a .png file, he then locates the old screenshot name and adds the updated version “lumina-e.png” to “lumina-f.png”. With the file added to the tree, the relevant section of .rst code can be adjusted and the sphinx build run to verify the output HTML looks correct. Using this method you can easily start to get involved with other aspects of documentation and next thing you know you'll be writing boot-loaders like Allan! *** Learn C Programming With 9 Excellent Open Source Books (https://www.ossblog.org/learn-c-programming-with-9-excellent-open-source-books/) Now that you've easily mastered all your documentation skills, you may be ready to take on a new challenge. (Come on, that boot-loader isn't going to write itself!) We wanted to point out some excellent resources to get you started on your journey into writing C. Before you think, “oh, more books to purchase”, wait there's good news. These are the top-9 open-source books that you can download in digital form free of charge. Now I bet we got your attention. We start the rundown with “The C Book”, by Mike Banahan, Declan Brady and Mark Doran, which will lay the groundwork with your introduction into the C language and concepts. Next up, if you are going to do anything, do it with style, so take a read through the “C Elements of Style” which will make you popular at all the parties. (We can't vouch for that statement) From here we have a book on using C to build your own minimal “lisp” interpreter, reference guides on GNU C and some other excellent introduction / mastery books to help round-out your programming skill set. Your C adventure awaits, hopefully these books can not only teach you good C, but also make you feel confident when looking at bits of the FreeBSD world or kernel with a proper foundation to back it up. *** Running a Linux VM on OpenBSD (http://eradman.com/posts/linuxvm-on-openbsd.html) Over the past few years we've talked a lot about Virtualization, Bhyve or OpenBSD's ‘vmm', but qemu hasn't gotten much attention. Today we have a blog post with details on how to deploy qemu to run Linux on top of an OpenBSD host system. The starts by showing us how to first provision the storage for qemu, using the handy ‘qemu-img' command, which in this example only creates a 4GB disk, you'll probably want more for real-world usage though. Next up the qemu command will be run, pay attention to the particular flags for network and memory setup. You'll probably want to bump it up past the recommended 256M of memory. Networking is always the fun part, as the author describes his intended setup I want OpenBSD and Debian to be able to obtain an IP via DHCP on their wired interfaces and I don't want external networking required for an NFS share to the VM. To accomplish this I need two interfaces since dhclient will erase any other IPv4 addresses already assigned. We can't assign an address directly to the bridge, but we can configure a virtual Ethernet device and add it. The setup for this portion involves touching a few more files, but isn't that painless. Some “pf” rules to enable NAT for and dhcpd setup to assign a “fixed” IP to the vm will get us going, along with some additional details on how to configure the networking for inside the debian VM. Once those steps are completed you should be able to mount NFS and share data from the host to the VM painlessly. Beastie Bits MacObserver: Interview with Open Source Developer & Former Apple Manager Jordan Hubbard (https://www.macobserver.com/podcasts/background-mode-jordan-hubbard/) 2016 Google Summer of Code Mentor Summit and MeetBSD Trip Report: Gavin Atkinson (https://www.freebsdfoundation.org/blog/2016-google-summer-of-code-mentor-summit-and-meetbsd-trip-report-gavin-atkinson/) Feedback/Questions Joe - BGP / Vultr Followup (http://pastebin.com/TNyHBYwT) Ryan Moreno asks about Laptops (http://pastebin.com/s4Ypezsz) ***
This week on BSDNow, we have “Weird Unix Things”, “Is it getting Steamy in here?” and an Interview about BSD Sockets API. (Those This episode was brought to you by Headlines playonbsd with TrueOS: It's Getting Steamy in Here and I've Had Too Much Wine (https://www.trueos.org/blog/playonbsd-trueos-getting-steamy-ive-much-wine/) We've done a couple of tutorials in the past on using Steam and Wine with PC-BSD, but now with the addition of playonbsd to the AppCafe library, you have more options than ever before to game on your TrueOS system. We're going to have a look today at playonbsd, how it works with TrueOS, and what you can expect if you want to give it a try on your own system. Let's dive right in! Once playonbsd is installed, go back to your blank desktop, right-click on the wallpaper, and select terminal. Playonbsd does almost all the configuring for you, but there are still a couple of simple options you'll want to configure to give yourself the best experience. In your open terminal, type: playonbsd. You can also find playonbsd by doing a fast search using Lumina's built-in search function in the start menu after it's been installed. Once opened, a graphical interface greets us with easy to navigate menus and even does most of the work for you. A nice graphical UI that hides the complexity of setting up WINE and Steam, and lets you pick select the game you want, and get it setup Start gaming quicker, without the headache If you're a PC gamer, you should definitely give playonbsd a try! You may be surprised at how well it works. If you want to know ahead of time if your games are well supported or not, head on over to WineHQ and do a search. Many people have tested and provided feedback and even solutions for potential problems with a large variety of video games. This is a great resource if you run into a glitch or other problem. Weird Unix thing: 'cd //' (https://jvns.ca/blog/2017/02/08/weird-unix-things-cd/) So why can you do ‘cd //tmp', and it isn't the same as ‘cd /tmp'? The spec says: An implementation may further simplify curpath by removing any trailing characters that are not also leading characters, replacing multiple non-leading consecutive characters with a single , and replacing three or more leading characters with a single . If, as a result of this canonicalization, the curpath variable is null, no further steps shall be taken. “So! We can replace “three or more leading / characters with a single slash”. That does not say anything about what to do when there are 2 / characters though, which presumably is why cd //tmp leaves you at //tmp.” A pathname that begins with two successive slashes may be interpreted in an implementation-defined manner So what is it for? Well, the blog did a bit of digging and came up with this stackoverflow answer (http://unix.stackexchange.com/questions/256497/on-what-systems-is-foo-bar-different-from-foo-bar/256569#256569) In cygwin and some other systems // is treated as a unix-ified version of , to access UNC windows file sharing paths like servershare Perforce, the vcs, uses // to denote a path relative to the depot It seems to have been used in the path for a bunch of different network file systems, but also for myriad other things Testing out snapshots in Apple's next-generation APFS file system (https://arstechnica.com/apple/2017/02/testing-out-snapshots-in-apples-next-generation-apfs-file-system/) Adam Leventhal takes his DTrace hammer to Apple's new file system to see what is going on Back in June, Apple announced its new upcoming file system: APFS, or Apple File System. There was no mention of it in the WWDC keynote, but devotees needed no encouragement. They picked over every scintilla of data from the documentation on Apple's developer site, extrapolating, interpolating, eager for whatever was about to come. In the WWDC session hall, the crowd buzzed with a nervous energy, eager for the grand unveiling of APFS. I myself badge-swapped my way into the conference just to get that first glimpse of Apple's first original filesystem in the 30+ years since HFS Apple's presentation didn't disappoint the hungry crowd. We hoped for a modern filesystem, optimized for next generation hardware, rich with features that have become the norm for data centers and professionals. With APFS, Apple showed a path to meeting those expectations. Dominic Giampaolo and Eric Tamura, leaders of the APFS team, shared performance optimizations, data integrity design, volume management, efficient storage of copied data, and snapshots—arguably the feature of APFS most directly in the user's control. It's 2017, and Apple already appears to be making good on its promise with the revelation that the forthcoming iOS 10.3 will use APFS. The number of APFS tinkerers using it for their personal data has instantly gone from a few hundred to a few million. Beta users of iOS 10.3 have already made the switch apparently without incident. They have even ascribed unscientifically-significant performance improvements to APFS. Previously Adam had used DTrace to find a new syscall introduced in OS X, fs_snapshot, but he had not dug into how to use it. Now it seems, the time has come Learning from XNU and making some educated guesses, I wrote my first C program to create an APFS snapshot. This section has a bit of code, which you can find in this Github repo (https://github.com/ahl/apfs) That just returned “fs_snapshot: Operation not permitted” So, being Adam, he used DTrace to figure out what the problem was Running this DTrace script in one terminal while running the snapshot program in another shows the code flow through the kernel as the program executes In the code flow, the privcheckcred() function jumps out as a good place to continue because of its name, the fact that fs_snapshot calls it directly, and the fact that it returns 1 which corresponds with EPERM, the error we were getting. Turns out, it just requires some sudo With a little more testing I wrote my own version of Apple's unreleased snapUtil command from the WWDC demo We figured out the proper use of the fssnapshot system call and reconstructed the WWDC snapUtil. But all this time an equivalent utility has been lurking on macOS Sierra. If you look in /System/Library/Filesystems/apfs.fs/Contents/Resources/, Apple has included a number of APFS-related utilities, including apfssnapshot (and, tantalizingly, a tool called hfs_convert). Snapshots let you preserve state to later peruse; we can also revert an APFS volume to a previous state to restore its contents. The current APFS semantics around rollback are a little odd. The revert operation succeeds, but it doesn't take effect until the APFS volume is next mounted Another reason Apple may not have wanted people messing around with snapshots is that the feature appears to be incomplete. Winding yourself into a state where only a reboot can clear a mounted snapshot is easy, and using snapshots seems to break some of the diskutil APFS output It is interesting to see what you can do with DTrace, as well as to see what a DTrace and ZFS developer things of APFS *** Interview - Tom Jones - tj@enoti.me (mailto:tj@enoti.me) Replacing the BSD Sockets API *** News Roundup FreeBSD rc.d script to map ethernet device names by MAC address (https://github.com/eborisch/ethname) Self-contained FreeBSD rc.d script for re-naming devices based on their MAC address. I needed it due to USB Ethernet devices coming up in different orders across OS upgrades. Copy ethname into /usr/local/etc/rc.d/ Add the following to rc.conf: ethnameenable="YES" ethnamedevices="em0 ue0 ue1" # Replace with desired devices to rename Create /usr/local/etc/ifmap in the following format: 01:23:45:67:89:ab eth0 01:23:45:67:89:ac eth1 That's it. Use ifconfig_="" settings in rc.conf with the new names. I know MFSBSD has something like this, but a polished up hybrid of the two should likely be part of the base system if something is not already available This would be a great “Junior Job”, if say, a viewer wanted to get started with their first FreeBSD patch *** Mog: A different take on the Unix tool cat (https://github.com/witchard/mog) Do you abuse cat to view files? Did you know cat is meant for concatenating files, meaning: cat part1 part2 part3 > wholething.txt mog is a tool for actually viewing files, and it adds quite a few nice features Syntax highlight scripts Print a hex dump of binary files Show details of image files Perform objdump on executables List a directory mog reads the $HOME/.mogrc config file which describes a series of operations it can do in an ordered manner. Each operation has a match command and an action command. For each file you give to mog it will test each match command in turn, when one matches it will perform the action. A reasonably useful config file is generated when you first run it. How Unix erases things when you type a backspace while entering text (https://utcc.utoronto.ca/~cks/space/blog/unix/HowUnixBackspaces) Yesterday I mentioned in passing that printing a DEL character doesn't actually erase anything. This raises an interesting question, because when you're typing something into a Unix system and hit your backspace key, Unix sure erases the last character that you entered. So how is it doing that? The answer turns out to be basically what you'd expect, although the actual implementation rapidly gets complex. When you hit backspace, the kernel tty line discipline rubs out your previous character by printing (in the simple case) Ctrl-H, a space, and then another Ctrl-H. Of course just backing up one character is not always the correct way of erasing input, and that's when it gets complicated for the kernel. To start with we have tabs, because when you (the user) backspace over a tab you want the cursor to jump all the way back, not just move back one space. The kernel has a certain amount of code to work out what column it thinks you're on and then back up an appropriate number of spaces with Ctrl-Hs. Then we have the case when you quoted a control character while entering it, eg by typing Ctrl-V Ctrl-H; this causes the kernel to print the Ctrl-H instead of acting on it, and it prints it as the two character sequence ^H. When you hit backspace to erase that, of course you want both (printed) characters to be rubbed out, not just the 'H'. So the kernel needs to keep track of that and rub out two characters instead of just one. Chris then provides an example, from IllumOS, of the kernel trying to deal with multibyte characters FreeBSD also handles backspacing a space specially, because you don't need to actually rub that out with a 'b b' sequence; you can just print a plain b. Other kernels don't seem to bother with this optimization. The FreeBSD code for this is in sys/kern/ttyttydisc.c in the ttydiscrubchar function PS: If you want to see the kernel's handling of backspace in action, you usually can't test it at your shell prompt, because you're almost certainly using a shell that supports command line editing and readline and so on. Command line editing requires taking over input processing from the kernel, and so such shells are handling everything themselves. My usual way to see what the kernel is doing is to run 'cat >/dev/null' and then type away. And you thought the backspace key would be simple... *** FreeBSD ports now have Wayland (http://www.freshports.org/graphics/wayland/) We've discussed the pending Wayland work, but we wanted to point you today to the ports which are in mainline FreeBSD ports tree now. First of all, (And I was wondering how they would deal with this) it has landed in the “graphics” category, since Wayland is the Anti-X11, putting it in x11/ didn't make a lot of sense. Couple of notes before you start installing new packages and expecting wayland to “just work” First, this does require that you have working DRM from the kernel side. You'll want to grab TrueOS or build from Matt Macy's FreeBSD branches on GitHub before testing on any kind of modern Intel GPU. Nvidia with modesetting should be supported. Next, not all desktops will “just work”. You may need to grab experimental Weston for compositor. KDE / Gnome (And Lumina) and friends will grow Wayland support in the future, so don't expect to just fire up $whatever and have it all work out of box. Feedback is needed! This is brand new functionality for FreeBSD, and the maintainers will want to hear your results. For us on the TrueOS side we are interested as well, since we want to port Lumina over to Wayland soon(ish) Happy Experimenting! *** Beastie Bits Faces of FreeBSD 2017: Joseph Kong (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2017-joseph-kong/) OPNsense 17.1 “Eclectic Eagle”, based on FreeBSD 11 Released (https://opnsense.org/opnsense-17-1-released/) Why you should start programming on UNIX (http://www.koszek.com/blog/2017/01/28/why-you-should-start-programming-on-unix/) OpenSMTPD Mail Filtering (http://eradman.com/posts/opensmtpd-filtering.html) Feedback/Questions Zane - Databases and Jails (http://pastebin.com/89AyGe5F) Mohammad - USB Install (http://pastebin.com/Te8sz9id) Chuck - Updating Jails (http://pastebin.com/G2SzahWL) David - Lumina / LXQt (http://pastebin.com/71ExJLpL) ***
This week on the show, we've got FreeBSD quarterly Status reports to discuss, OpenBSD changes to the installer, EC2 and IPv6 and more. Stay This episode was brought to you by Headlines OpenBSD changes of note 6 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-6) OpenBSD can now be cross built with clang. Work on this continues Build ld.so with -fno-builtin because otherwise clang would optimize the local versions of functions like dlmemset into a call to memset, which doesn't exist. Add connection timeout for ftp (http). Mostly for the installer so it can error out and try something else. Complete https support for the installer. I wonder how they handle certificate verification. I need to look into this as I'd like to switch the FreeBSD installer to this as well New ocspcheck utility to validate a certificate against its ocsp responder. net lock here, net lock there, net lock not quite everywhere but more than before. More per cpu counters in networking code as well. Disable and lock Silicon Debug feature on modern Intel CPUs. Prevent wireless frame injection attack described at 33C3 in the talk titled “Predicting and Abusing WPA2/802.11 Group Keys” by Mathy Vanhoef. Add support for multiple transmit ifqueues per network interface. Supported drivers include bge, bnx, em, myx, ix, hvn, xnf. pledge now tracks when a file as opened and uses this to permit or deny ioctl. Reimplement httpd's support for byte ranges. Fixes a memory DOS. FreeBSD 2016Q4 Status Report (https://www.freebsd.org/news/status/report-2016-10-2016-12.html) An overview of some of the work that happened in October - December 2016 The ports tree saw many updates and surpassed 27,000 ports The core team was busy as usual, and the foundation attended and/or sponsored a record 24 events in 2016. CEPH on FreeBSD seems to be coming along nicely. For those that do not know, CEPH is a distributed filesystem that can sit on top of another filesystem. That is, you can use it to create a clustered filesystem out of a bunch of ZFS servers. Would love to have some viewers give it a try and report back. OpenBSM, the FreeBSD audit framework, got some updates Ed Schouten committed a front end to export sysctl data in a format usable by Prometheus, the open source monitoring system. This is useful for other monitoring software too. Lots of updates for various ARM boards There is an update on Reproducible Builds in FreeBSD, “ It is now possible to build the FreeBSD base system (kernel and userland) completely reproducibly, although it currently requires a few non-default settings”, and the ports tree is at 80% reproducible Lots of toolchain updates (gcc, lld, gdb) Various updates from major ports teams *** Amazon rolls out IPv6 support on EC2 (http://www.daemonology.net/blog/2017-01-26-IPv6-on-FreeBSD-EC2.html) A few hours ago Amazon announced that they had rolled out IPv6 support in EC2 to 15 regions — everywhere except the Beijing region, apparently. This seems as good a time as any to write about using IPv6 in EC2 on FreeBSD instances. First, the good news: Future FreeBSD releases will support IPv6 "out of the box" on EC2. I committed changes to HEAD last week, and merged them to the stable/11 branch moments ago, to have FreeBSD automatically use whatever IPv6 addresses EC2 makes available to it. Next, the annoying news: To get IPv6 support in EC2 from existing FreeBSD releases (10.3, 11.0) you'll need to run a few simple commands. I consider this unfortunate but inevitable: While Amazon has been unusually helpful recently, there's nothing they could have done to get support for their IPv6 networking configuration into FreeBSD a year before they launched it. You need the dual-dhclient port: pkg install dual-dhclient And the following lines in your /etc/rc.conf: ifconfigDEFAULT="SYNCDHCP acceptrtadv" ipv6activateallinterfaces="YES" dhclientprogram="/usr/local/sbin/dual-dhclient" + It is good to see FreeBSD being ready to use this feature on day 0, not something we would have had in the past Finally, one important caveat: While EC2 is clearly the most important place to have IPv6 support, and one which many of us have been waiting a long time to get, this is not the only service where IPv6 support is important. Of particular concern to me, Application Load Balancer support for IPv6 is still missing in many regions, and Elastic Load Balancers in VPC don't support IPv6 at all — which matters to those of us who run non-HTTP services. Make sure that IPv6 support has been rolled out for all the services you need before you start migrating. Colin's blog also has the details on how to actually activate IPv6 from the Amazon side, if only it was as easy as configuring it on the FreeBSD side *** FreeBSD's George Neville-Neil tries valiantly for over an hour to convince a Linux fan of the error of their ways (https://www.youtube.com/watch?v=cofKxtIO3Is) In today's episode of the Lunduke Hour I talk to George Neville-Neil -- author and FreeBSD advocate. He tries to convince me, a Linux user, that FreeBSD is better. + They cover quite a few topics, including: + licensing, and the motivations behind it + vendor relations + community + development model + drivers and hardware support + George also talks about his work with the FreeBSD Foundation, and the book he co-authored, “The Design and Implementation of the FreeBSD Operating System, 2nd Edition” News Roundup An interactive script that makes it easy to install 50+ desktop environments following a base install of FreeBSD 11 (https://github.com/rosedovell/unixdesktops) And I thought I was doing good when I wrote a patch for the installer that enables your choice of 3 desktop environments... This is a collection of scripts meant to install desktop environments on unix-like operating systems following a base install. I call one of these 'complete' when it meets the following requirements: + A graphical logon manager is presented without user intervention after powering on the machine + Logging into that graphical logon manager takes the user into the specified desktop environment + The user can open a terminal emulator I need to revive my patch, and add Lumina to it *** Firefox 51 on sparc64 - we did not hit the wall yet (https://blog.netbsd.org/tnf/entry/firefox_51_on_sparc64_we) A NetBSD developers tells the story of getting Firefox 51 running on their sparc64 machine It turns out the bug impacted amd64 as well, so it was quickly fixed They are a bit less hopeful about the future, since Firefox will soon require rust to compile, and rust is not working on sparc64 yet Although there has been some activity on the rust on sparc64 front, so maybe there is hope The post also look at a few alternative browsers, but it not hopeful *** Introducing Bloaty McBloatface: a size profiler for binaries (http://blog.reverberate.org/2016/11/07/introducing-bloaty-mcbloatface.html) I'm very excited to announce that today I'm open-sourcing a tool I've been working on for several months at Google. It's called Bloaty McBloatface, and it lets you explore what's taking up space in your .o, .a, .so, and executable binary files. Bloaty is available under the Apache 2 license. All of the code is available on GitHub: github.com/google/bloaty. It is quick and easy to build, though it does require a somewhat recent compiler since it uses C++11 extensively. Bloaty primarily supports ELF files (Linux, BSD, etc) but there is some support for Mach-O files on OS X too. I'm interested in expanding Bloaty's capabilities to more platforms if there is interest! I need to try this one some of the boot code files, to see if there are places we can trim some fat We've been using Bloaty a lot on the Protocol Buffers team at Google to evaluate the binary size impacts of our changes. If a change causes a size increase, where did it come from? What sections/symbols grew, and why? Bloaty has a diff mode for understanding changes in binary size The diff mode looks especially interesting. It might be worth setting up some kind of CI testing that alerts if a change results in a significant size increase in a binary or library *** A BSD licensed mdns responder (https://github.com/kristapsdz/mdnsd) One of the things we just have to deal with in the modern world is service and system discovery. Many of us have fiddled with avahi or mdnsd and related “mdns” services. For various reasons those often haven't been the best-fit on BSD systems. Today we have a github project to point you at, which while a bit older, has recently been updated with pledge() support for OpenBSD. First of all, why do we need an alternative? They list their reasons: This is an attempt to bring native mdns/dns-sd to OpenBSD. Mainly cause all the other options suck and proper network browsing is a nice feature these days. Why not Apple's mdnsd ? 1 - It sucks big time. 2 - No BSD License (Apache-2). 3 - Overcomplex API. 4 - Not OpenBSD-like. Why not Avahi ? 1 - No BSD License (LGPL). 2 - Overcomplex API. 3 - Not OpenBSD-like 4 - DBUS and lots of dependencies. Those already sound like pretty compelling reasons. What makes this “new” information again is the pledge support, and perhaps it's time for more BSD's to start considering importing something like mdnsd into their base system to make system discovery more “automatic” *** Beastie Bits Benno Rice at Linux.Conf.Au: The Trouble with FreeBSD (https://www.youtube.com/watch?v=Ib7tFvw34DM) State of the Port of VMS to x86 (http://vmssoftware.com/pdfs/State_of_Port_20170105.pdf) Microsoft Azure now offers Patent Troll Protection (https://thestack.com/cloud/2017/02/08/microsoft-azure-now-offers-patent-troll-ip-protection/) FreeBSD Storage Summit 2017 (https://www.freebsdfoundation.org/news-and-events/event-calendar/freebsd-storage-summit-2017/) If you are going to be in Tokyo, make sure you come to (http://bhyvecon.org/) Feedback/Questions Farhan - Laptops (http://pastebin.com/bVqsvM3r) Hjalti - rclone (http://pastebin.com/7KWYX2Mg) Ivan - Jails (http://pastebin.com/U5XyzMDR) Jungle - Traffic Control (http://pastebin.com/sK7uEDpn) ***
This week on BSDNow we have a cantrill special to bring you! All three interviews back to back in their original glory, you won't want to miss This episode was brought to you by – Show Notes: –FOSDEM 2017 BSD Dev Room Videos Ubuntu Slaughters Kittens | BSD Now 103The Cantrill Strikes Back | BSD Now 117Return of the Cantrill | BSD Now 163
This week on BSDNow, I'm out of town but we have a great interview with Ken Moore (My brother) about the latest in BSD desktop computing and This episode was brought to you by Interview - Ken Moore - ken@trueos.org (mailto:ken@trueos.org) TrueOS, Lumina, Sys Admin, The BSD Desktop Ecosystem + KM: Thank you for joining us again, can you believe it has been an entire year? + AJ: Let's start by getting an update on Lumina, what has happened in the last year? + KM: What is the change you are most proud of in that time? + AJ: What do you think of the recent introduction of Wayland to the ports tree? Do you think this will impact Lumina? Do you have any plans? + KM: + AJ: What has changed with SysAdm after a year of development? + KM: What plans do you have for the future of SysAdm? + AJ: How has it been working with the drm-next branch? Does it feel like that is progressing? + KM: Can you tell us about some of the other TrueOS work you have been doing? + AJ: What are your thoughts on how the BSD Desktop Ecosystem has changed over the last year? Do you think the future looks better or worse now? + KM: Do you think systemd is going to continue to make things work? Or does it seem like there is enough resistance to it that fewer projects are going to throw out support for anything not-systemd + AJ: Anything else you want to add?
This week on BSDNow, we're going to be leading off with the latest news about Wayland and Xorg support on FreeBSD, then a look at OpenBSD ARM64 This episode was brought to you by Headlines Wayland is now in the FreeBSD Ports tree (https://svnweb.freebsd.org/ports?view=revision&revision=432406) This commit brings Wayland, the new windowing system, into the FreeBSD ports tree “This port was first created by Koop Mast (kwm@) then updated and improved by Johannes Lundberg” “Wayland is intended as a simpler replacement for X, easier to develop and maintain. GNOME and KDE are expected to be ported to it.” Wayland is designed for desktop and laptop use, rather than X, which was designed for use over the network, where clients were not powerful enough to run the applications locally. “Wayland is a protocol for a compositor to talk to its clients as well as a C library implementation of that protocol. The compositor can be a standalone display server running on Linux kernel modesetting and evdev input devices, an X application, or a wayland client itself. The clients can be traditional applications, X servers (rootless or fullscreen) or other display servers.” “Please report bugs to the FreeBSD bugtracker!” It is good to see this project progressing, as it seems in a few generations, high performance graphics drivers may only be actively developed for Wayland. *** Call For Testing: xorg 1.18.4 and newer intel/ati DDX (https://lists.freebsd.org/pipermail/freebsd-x11/2017-January/018738.html) Baptiste Daroussin, and the FreeBSD X11 team, have issued a call for testing for the upgrade to Xorg 1.18.4 Along with it comes newer ATI/AMD and Intel drivers “Note that you will need to rebuild all the xf86-* packages to work with thatnewer xorg (hence the bump of the revision)” “Do not expect newer gpu supported as this is not the kernel part”, it only provides the newer Xorg driver, not the kernel mode setting driver (this is a separate project) “If you experience any issue with intel or radeon driver please try to use the new modesetting driver provided by xorg directly (note that fedora and debian recommend the use of the new driver instead of the ati/intel one)” *** Error handling in C (http://www.tedunangst.com/flak/post/to-errno-or-to-error) “Unlike other languages which have one preferred means of signalling an error, C is a multi error paradigm language. Error handling styles in C can be organized into one of several distinct styles, such as popular or correct. Some examples of each.” “One very popular option is the classic unix style. -1 is returned to indicate an error.” “Another option seen in the standard C library is NULL for errors.” “The latter has the advantage that NULL is a false value, which makes it easier to write logical conditions. File descriptor 0 is valid (stdin) but false, while -1 is invalid but true.” “And of course, there's the worst of both worlds approach requiring a special sentinel that you'll probably forget to use” “Other unix functions, those that don't need to return a file descriptor, stick to just 0 and -1” “Of course, none of these functions reveal anything about the nature of the error. For that, you need to consult the errno on the side” The article goes on to describe different ways of dealing with the issue, and return values. There is also coverage of more complex examples and involve a context that might contain the error message It is really interesting to see the differences, and the pitfalls of each approach *** Fixing POSIX Filenames (http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html) “Traditionally, Unix/Linux/POSIX pathnames and filenames can be almost any sequence of bytes. A pathname lets you select a particular file, and may include zero or more “/” characters. Each pathname component (separated by “/”) is a filename; filenames cannot contain “/”. Neither filenames nor pathnames can contain the ASCII NUL character (), because that is the terminator.” “This lack of limitations is flexible, but it also creates a legion of unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws). It also makes it impossible to consistently and accurately display filenames, causes portability problems, and confuses users.” “This article will try to convince you that adding some tiny limitations on legal Unix/Linux/POSIX filenames would be an improvement. Many programs already presume these limitations, the POSIX standard already permits such limitations, and many Unix/Linux filesystems already embed such limitations — so it'd be better to make these (reasonable) assumptions true in the first place. This article will discuss, in particular, the three biggest problems: control characters in filenames (including newline, tab, and escape), leading dashes in filenames, and the lack of a standard character encoding scheme (instead of using UTF-8). These three problems impact programs written in any language on Unix/Linux/POSIX system. There are other problems, of course. Spaces in filenames can cause problems; it's probably hopeless to ban them outright, but resolving some of the other issues will simplify handling spaces in filenames. For example, when using a Bourne shell, you can use an IFS trick (using IFS=printf 'nt') to eliminate some problems with spaces. Similarly, special metacharacters in filenames cause some problems; I suspect few if any metacharacters could be forbidden on all POSIX systems, but it'd be great if administrators could locally configure systems so that they could prevent or escape such filenames when they want to. I then discuss some other tricks that can help.” “After limiting filenames slightly, creating completely-correct programs is much easier, and some vulnerabilities in existing programs disappear. This article then notes some others' opinions; I knew that some people wouldn't agree with me, but I'm heartened that many do agree that something should be done. Finally, I briefly discuss some methods for solving this long-term; these include forbidding creation of such names (hiding them if they already exist on the underlying filesystem), implementing escaping mechanisms, or changing how tools work so that these are no longer problems (e.g., when globbing/scanning, have the libraries prefix “./” to any filename beginning with “-”). Solving this is not easy, and I suspect that several solutions will be needed. In fact, this paper became long over time because I kept finding new problems that needed explaining (new “worms under the rocks”). If I've convinced you that this needs improving, I'd like your help in figuring out how to best do it!” “Filename problems affect programs written in any programming language. However, they can be especially tricky to deal with when using Bourne shells (including bash and dash). If you just want to write shell programs that can handle filenames correctly, you should see the short companion article Filenames and Pathnames in Shell: How to do it correctly (http://www.dwheeler.com/essays/filenames-in-shell.html).” Imagine that you don't know Unix/Linux/POSIX (I presume you really do), and that you're trying to do some simple tasks. For our purposes we will create simple scripts on the command line (using a Bourne shell) for these tasks, though many of the underlying problems affect any program. For example, let's try to print out the contents of all files in the current directory, putting the contents into a file in the parent directory: cat * > ../collection # WRONG cat ./* > ../collection # CORRECT cat find . -type f > ../collection # WRONG ( set -f ; for file in find . -type f ; do # WRONG cat "$file" done ) > ../collection ( find . -type f | xargs cat ) > ../collection # WRONG, WAY WRONG Just think about trying to remove a file named: -rf / *** News Roundup OpenBSD ARM64 (https://www.openbsd.org/arm64.html) A new page has appeared on the OpenBSD website, offering images for ARM64 “The current target platforms are the Pine64 and the Raspberry Pi 3.” “OpenBSD/arm64 bundles various platforms sharing the 64-bit ARM architecture. Due to the fact that there are many System on a Chips (SoC) around, OpenBSD/arm64 differentiates between various SoCs and may have a different level of support between them” The page contains a list of the devices that are supported, and which components have working drivers At the time of recording, the link to download the snapshots did not work yet, but by time this airs a week from now, it should be working. *** The design of Chacha20 (http://loup-vaillant.fr/tutorials/chacha20-design) Seems like every few episodes we end up discussing Ciphers (With their o-so amusing naming) and today is no exception. We have a great writeup on the D & I of the ‘chacha20' cipher written by “Loup Vaillant” First of all, is this story for you? Maybe the summary will help make that call: “Quick summary: Chacha20 is ARX-based hash function, keyed, running in counter mode. It embodies the idea that one can use a hash function to encrypt data.” If your eyes didn't glaze over, then you are cleared to proceed. Chacha20 is built around stream ciphers: While Chacha20 is mainly used for encryption, its core is a pseudo-random number generator. The cipher text is obtained by XOR'ing the plain text with a pseudo-random stream: ciphertext = plaintext XOR chacha_stream(key, nonce) Provided you never use the same nonce with the same key twice, you can treat that stream as a one time pad. This makes it very simple: unlike block ciphers, you don't have to worry about padding, and decryption is the same operation as encryption: plaintext = ciphertext XOR chacha_stream(key, nonce) Now we just have to get that stream. The idea that the streams can mimic the concept of a one-time pad does make chacha20 very attractive, even to a non-crypto guy such as myself. From here the article goes into depth on how the cipher scrambles 512bit blocks using the quarter-round method (A forth of a block or 4 32bit numbers) Some ascii art is used here to help visualize how this done, in the quarter round-phase, then to the complete block as the 4 quarters are run in parallel over the entire 512 bit block. From here the article goes more into depth, looking at the complete chacha block, and the importance of a seemingly unnecessary 32byte constant (Hint: it's really important) If crypto is something you find fascinating, you'll want to make sure you give this one a full read-through. *** CyberChef - Coming to a FreeBSD Ports tree near you (https://twitter.com/DLangille/status/823915729430913025) Dan Langille tweets that he will be creating a port of GCHQ's CyberChef tool “CyberChef is a simple, intuitive web app for carrying out all manner of "cyber" operations within a web browser. These operations include creating hexdumps, simple encoding like XOR or Base64, more complex encryption like AES, DES and Blowfish, data compression and decompression, calculating hashes and checksums, IPv6 and X.509 parsing, and much more.” “The tool is designed to enable both technical and non-technical analysts to manipulate data in complex ways without having to deal with complex tools or algorithms. It was conceived, designed, built and incrementally improved by an analyst in their 10% innovation time over several years. Every effort has been made to structure the code in a readable and extendable format, however it should be noted that the analyst is not a professional developer and the code has not been peer-reviewed for compliance with a formal specification.” Some handy functions, beyond stuff like base64 encoding: Network Enumeration (CIDR to list of IPS) (https://gchq.github.io/CyberChef/?recipe=%5B%7B%22op%22%3A%22Parse%20IP%20range%22%2C%22args%22%3A%5Btrue%2Ctrue%2Cfalse%5D%7D%5D&input=MTcyLjIxLjAuMzIvMjcK) Browser User Agent Parser (what browser is that, based on your HTTP logs) XOR Brute Force: enter some XOR'd text, and try every possible key to find plaintext. Optionally give it a regex of known plaintext to find the right key. Calculate the “Shannon Entropy” of the input (how random is this data) It also has a number of built in regular expressions for common things, very useful The project is up on github if you want to play with the code *** Building Electron and VSCode in FreeBSD11 (https://gist.github.com/prash-wghats/89be1ee069d2acf23c289e9c606616e1) A patch and set of instructions for building Electron and VSCode on FreeBSD “Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring. It is also customizable, so users can change the editor's theme, keyboard shortcuts, and preferences. It is free and open-source, although the official download is under a proprietary license.” “Visual Studio Code is based on Electron, a framework which is used to deploy Node.js applications for the desktop running on the Blink layout engine. Although it uses the Electron framework, the software is not a fork of Atom, it is actually based on Visual Studio Online's editor (codename "Monaco")” It would be interesting to see official support for VSCode on FreeBSD Has anyone tried VSCode on the FreeBSD Code base? *** Beastie Bits Soft Label Keys (http://roy.marples.name/blog/blog/soft-label-keys) WPA1 (TKIP) disabled by default (OpenBSD) (https://www.mail-archive.com/source-changes@openbsd.org/msg84599.html) Cool but obscure unix tools (https://kkovacs.eu/cool-but-obscure-unix-tools) KDE Frameworks and Plasma on FreeBSD (http://euroquis.nl/bobulate/?p=1521) Initiative to migrate OpenBSD mirrors to HTTPS (https://www.mail-archive.com/source-changes@openbsd.org/msg84904.html) That moment you realize FreeBSD has got some Star Wars fans (http://i.imgur.com/dC7c1y4.png) Pagelink (https://wiki.freebsd.org/PortsSubversionPrimer)
This week on BSD Now, we will be discussing a wide variety of topics including Routers, Run-Controls, the “Rule” of silence and some This episode was brought to you by Headlines Ports no longer build on EOL FreeBSD versions (https://www.reddit.com/r/freebsd/comments/5ouvmp/ports_no_longer_build_on_eol_freebsd_versions/) The FreeBSD ports tree has been updated to automatically fail if you try to compile ports on EOL versions of FreeBSD (any version of 9.x or earlier, 10.0 - 10.2, or 11 from before 11.0) This is to prevent shooting yourself in the food, as the compatibility code for those older OSes has been removed now that they are no longer supported. If you use pkg, you will also run into problems on old releases. Packages are always built on the oldest supported release in a branch. Until recently, this meant packages for 10.1, 10.2, and 10.3 were compiled on 10.1. Now that 10.1 and 10.2 are EOL, packages for 10.x are compiled on 10.3. This matters because 10.3 supports the new openat() and various other *at() functions used by capsicum. Now that pkg and packages are built on a version that supports this new feature, they will not run on systems that do not support it. So pkg will exit with an error as soon as it tries to open a file. You can work around this temporarily by using the pkg-static command, but you should upgrade to a supported release immediately. *** Improving TrueOS: OpenRC (https://www.trueos.org/blog/improving-trueos-openrc/) With TrueOS moving to a rolling-release model, we've decided to be a bit more proactive in sharing news about new features that are landing. This week we've posted an article talking about the transition to OpenRC In past episodes you've heard me mention OpenRC, but hopefully today we can help answer any of those lingering questions you may still have about it The first thing always asked, is “What is OpenRC?” OpenRC is a dependency-based init system working with the system provided init program. It is used with several Linux distributions, including Gentoo and Alpine Linux. However, OpenRC was created by the NetBSD developer Roy Marples in one of those interesting intersections of Linux and BSD development. OpenRC's development history, portability, and 2-clause BSD license make its integration into TrueOS an easy decision. Now that we know a bit about what it is, how does it behave differently than traditional RC? TrueOS now uses OpenRC to manage all system services, as opposed to FreeBSD's RC. Instead of using rc.d for base system rc scripts, OpenRC uses init.d. Also, every service in OpenRC has its own user configuration file, located in /etc/conf.d/ for the base system and /usr/local/etc.conf.d/ for ports. Finally, OpenRC uses runlevels, as opposed to the FreeBSD single- or multi- user modes. You can view the services and their runlevels by typing $ rc-update show -v in a CLI. Also, TrueOS integrates OpenRC service management into SysAdm with the Service Manager tool One of the prime benefits of OpenRC is much faster boot-times, which is important in a portable world of laptops (and desktops as well). But service monitoring and crash detection are also important parts of what make OpenRC a substantial upgrade for TrueOS. Lastly people have asked us about migration, what is done, what isn't? As of now almost all FreeBSD base system services have been migrated over. In addition most desktop-facing services required to run Lumina and the like are also ported. We are still going through the ports tree and converting legacy rc.d scripts to init.d, but the process takes time. Several new folks have begun contributing OpenRC scripts and we hope to have all the roughly 1k ports converted over this year. BSDRP Releases 1.70 (https://sourceforge.net/projects/bsdrp/files/BSD_Router_Project/1.70/) A new release of the BSD Router Project This distro is designed to replace high end routers, like those from Cisco and Juniper, with FreeBSD running on regular off-the-shelf server. Highlights: Upgraded to FreeBSD 11.0-STABLE r312663 (skip 11.0 for massive performance improvement) Re-Added: netmap-fwd (https://github.com/Netgate/netmap-fwd) Add FIBsync patch to netmap-fwd from Zollner Robert netmap pkt-gen supports IPv6, thanks to Andrey V. Elsukov (ae@freebsd.org) bird 1.6.3 (add BGP Large communities support) OpenVPN 2.4.0 (adds the high speed AEAD GCM cipher) All of the other packages have also been upgraded A lot of great work has been done on BSDRP, and it has also generated a lot of great benchmarks and testing that have resulted in performance increases and improved understanding of how FreeBSD networking scales across different CPU types and speeds *** DragonFlyBSD gets UEFI support (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/7b1aa074fcd99442a1345fb8a695b62d01d9c7fd) This commit adds support for UEFI to the Dragonfly Installer, allowing new systems to be installed to boot from UEFI This script (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/9d53bd00e9be53d6b893afd79111370ee0c053b0) provides a way to build a HAMMER filesystem that works with UEFI There is also a UEFI man page (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/d195d5099328849c500d4a1b94d6915d3c72c71e) The install media (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/5fa778d7b36ab0981ff9dcbd96c71ebf653a6a19) has also been updated to support booting from either UEFI or MBR, in the same way that the FreeBSD images work *** News Roundup The Rule of Silence (http://www.linfo.org/rule_of_silence.html) “The rule of silence, also referred to as the silence is golden rule, is an important part of the Unix philosophy that states that when a program has nothing surprising, interesting or useful to say, it should say nothing. It means that well-behaved programs should treat their users' attention and concentration as being valuable and thus perform their tasks as unobtrusively as possible. That is, silence in itself is a virtue.” This doesn't mean a program cannot be verbose, it just means you have to ask it for the additional output, rather than having it by default “There is no single, standardized statement of the Unix philosophy, but perhaps the simplest description would be: "Write programs that are small, simple and transparent. Write them so that they do only one thing, but do it well and can work together with other programs." That is, the philosophy centers around the concepts of smallness, simplicity, modularity, craftsmanship, transparency, economy, diversity, portability, flexibility and extensibility.” “This philosophy has been fundamental to the the fact that Unix-like operating systems have been thriving for more than three decades, far longer than any other family of operating systems, and can be expected to see continued expansion of use in the years to come” “The rule of silence is one of the oldest and most persistent design rules of such operating systems. As intuitive as this rule might seem to experienced users of such systems, it is frequently ignored by the developers of other types of operating systems and application programs for them. The result is often distraction, annoyance and frustration for users.” “There are several very good reasons for the rule of silence: (1) One is to avoid cluttering the user's mind with information that might not be necessary or might not even be desired. That is, unnecessary information can be a distraction. Moreover, unnecessary messages generated by some operating systems and application programs are sometimes poorly worded, and can cause confusion or needless worry on the part of users.” No news is good news. When there is bad news, error messages should be descriptive, and ideally tell the user what they might do about the error. “A third reason is that command line programs (i.e., all-text mode programs) on Unix-like operating systems are designed to work together with pipes, i.e., the output from one program becomes the input of another program. This is a major feature of such systems, and it accounts for much of their power and flexibility. Consequently, it is important to have only the truly important information included in the output of each program, and thus in the input of the next program.” Have you ever had to try to strip out useless output so you could feed that data into another program? “The rule of silence originally applied to command line programs, because all programs were originally command line programs. However, it is just as applicable to GUI (graphical user interfaces) programs. That is, unnecessary and annoying information should be avoided regardless of the type of user interface.” “A example is the useless and annoying dialog boxes (i.e., small windows) that pop up on the display screen with with surprising frequency on some operating systems and programs. These dialog boxes contain some obvious, cryptic or unnecessary message and require the user to click on them in order to close them and proceed with work. This is an interruption of concentration and a waste of time for most users. Such dialog boxes should be employed only in situations in which some unexpected result might occur or to protect important data.” It goes on to make an analogy about Public Address systems. If too many unimportant messages, like advertisements, are sent over the PA system, people will start to ignore them, and miss the important announcements. *** The Tao of tmux (https://leanpub.com/the-tao-of-tmux/read) An interesting article floated across my news feed a few weeks back. It's what essentially boils down to a book called the “Tao of tmux”, which immediately piqued my interest. My story may be similar to many of yours. I was initially raised on using screen, and screen only for my terminal session and multiplexing needs. Since then I've only had a passing interest in tmux, but its always been one of those utilities I felt was worthy of investing some more time into. (Especially when seeing some of the neat setups some of my peers have with it) Needless to say, this article has been bookmarked, and I've started digesting some of it, but thought it would be good to share with anybody else who finds them-self in a similar situation. The book starts off well, explaining in the simplest terms possible what Tmux really is, by comparing and contrasting it to something we are all familiar with, GUIS! Helpfully they also include a chart which explains some of the terms we will be using frequently when discussing tmux (https://leanpub.com/the-tao-of-tmux/read#leanpub-auto-window-manager-for-the-terminal) One of the things the author does recommend is also making sure you are up to speed on your Terminal knowledge. Before getting into tmux, a few fundamentals of the command line should be reviewed. Often, we're so used to using these out of street smarts and muscle memory a great deal of us never see the relation of where these tools stand next to each other. Seasoned developers are familiar with zsh, Bash, iTerm2, konsole, /dev/tty, shell scripting, and so on. If you use tmux, you'll be around these all the time, regardless of whether you're in a GUI on a local machine or SSH'ing into a remote server. If you want to learn more about how processes and TTY's work at the kernel level (data structures and all) the book The Design and Implementation of the FreeBSD Operating System (2nd Edition) by Marshall Kirk McKusick is nice. In particular, Chapter 4, Process Management and Section 8.6, Terminal Handling. The TTY demystified by Linus Åkesson (available online) dives into the TTY and is a good read as well. We had to get that shout-out of Kirk's book in here ;) From here the boot/article takes us on a whirlwind journey of Sessions, Windows, Panes and more. Every control- command is covered, information on how to customize your statusbar, tips, tricks and the like. There's far more here than we can cover in a single segment, but you are highly encouraged to bookmark this one and start your own adventure into the world of tmux. *** SDF Celebrates 30 years of service in 2017 (https://sdf.org/) HackerNews thread on SDF (https://news.ycombinator.com/item?id=13453774) “Super Dimension Fortress (SDF, also known as freeshell.org) is a non-profit public access UNIX shell provider on the Internet. It has been in continual operation since 1987 as a non-profit social club. The name is derived from the Japanese anime series The Super Dimension Fortress Macross; the original SDF server was a BBS for anime fans[1]. From its BBS roots, which have been well documented as part of the BBS: The Documentary project, SDF has grown into a feature-rich provider serving members around the world.” A public access UNIX system, it was many people's first access to a UNIX shell. In the 90s, Virtual Machines were rare, the software to run them usually cost a lot of money and no one had very much memory to try to run two operating systems at the same time. So for many people, these type of shell accounts were the only way they could access UNIX without having to replace the OS on their only computer This is how I first started with UNIX, eventually moving to paying for access to bigger machines, and then buying my own servers and renting out shell accounts to host IRC servers and channel protection bots. “On June 16th, 1987 Ted Uhlemann (handle: charmin, later iczer) connected his Apple ][e's 300 baud modem to the phone line his mother had just given him for his birthday. He had published the number the night before on as many BBSes around the Dallas Ft. Worth area that he could and he waited for the first caller. He had a copy of Magic Micro BBS which was written in Applesoft BASIC and he named the BBS "SDF-1" after his favorite Japanimation series ROBOTECH (Macross). He hoped to draw users who were interested in anime, industrial music and the Church of the Subgenius.” I too started out in the world of BBSes before I had access to the internet. My parents got my a dedicated phone line for my birthday, so I wouldn't tie up their line all the time. I quickly ended up running my own BBS, the Sudden Death BBS (Renegade (https://en.wikipedia.org/wiki/Renegade_(BBS)) on MS DOS) I credit this early experience for my discovery of a passion for Systems Administration, that lead me to my current career “Slowly, SDF has grown over all these years, never forgetting our past and unlike many sites on the internet, we actually have a past. Some people today may come here and see us as outdated and "retro". But if you get involved, you'll see it is quite alive with new ideas and a platform for opportunity to try many new things. The machines are often refreshed, the quotas are gone, the disk space is expanding as are the features (and user driven features at that) and our cabinets have plenty of space for expansion here in the USA and in Europe (Germany).” “Think about ways you'd like to celebrate SDF's 30th and join us on the 'bboard' to discuss what we could do. I realize many of you have likely moved on yourselves, but I just wanted you to know we're still here and we'll keep doing new and exciting things with a foundation in the UNIX shell.” *** Getting Minecraft to Run on NetBSD (https://www.reddit.com/r/NetBSD/comments/5mtsy1/getting_minecraft_to_run_on_netbsd/) One thing that doesn't come up often on BSDNow is the idea of gaming. I realize most of us are server folks, or perhaps don't play games (The PC is for work, use your fancy-smanzy PS4 and get off my lawn you kids) Today I thought it would be fun to highlight this post over at Reddit talking about running MineCraft on NetBSD Now I realize this may not be news to some of you, but perhaps it is to others. For the record my kids have been playing Minecraft on PC-BSD / TrueOS for years. It's the primary reason they are more often booted into that instead of Windows. (Funny story behind that - Got sick of all the 3rd party mods, which more often than not came helpfully bundled with viruses and malware) On NetBSD the process looks a bit different than on FreeBSD. First up, you'll need to enable Linux Emulation and install Oracle JRE (Not OpenJDK, that path leads to sadness here) The guide will then walk us through the process of fetching the Linux runtime packages, extracting and then enabling bits such as ‘procfs' that is required to run the Linux binaries. Once that's done, minecraft is only a simple “oracle8-jre /path/to/minecraft.jar” command away from starting up, and you'll be “crafting” in no time. (Does anybody even play survival anymore?) *** Beastie Bits UNIX on the Computer Chronicals (https://youtu.be/g7P16mYDIJw) FreeBSD: Atheros AR9380 and later, maximum UDP TX goes from 250mbit to 355mbit. (https://twitter.com/erikarn/status/823298416939659264) Capsicumizing traceroute with casper (https://reviews.freebsd.org/D9303) Feedback/Questions Jason - TarSnap on Windows (http://pastebin.com/Sr1BTzVN) Mike - OpenRC & DO (http://pastebin.com/zpHyhHQG) Anonymous - Old Machines (http://pastebin.com/YnjkrDmk) Matt - Iocage (http://pastebin.com/pBUXtFak) Hjalti - Rclone & FreeNAS (http://pastebin.com/zNkK3epM)
This week on BSDNow, we've got Wifi galore, a new iocage and some RPi3 news and guides to share. Stay tuned for your place to B...SD! This episode was brought to you by Headlines WiFi: 11n hostap mode added to athn(4) driver, testers wanted (http://undeadly.org/cgi?action=article&sid=20170109213803) “OpenBSD as WiFi access points look set to be making a comeback in the near future” “Stefan Sperling added 802.11n hostap mode, with full support initially for the Atheros chips supported by the athn(4) driver.” “Hostap performance is not perfect yet but should be no worse than 11a/b/g modes in the same environment.” “For Linux clients a fix for WME params is needed which I also posted to tech@” “This diff does not modify the known-broken and disabled ar9003 code, apart from making sure it still builds.” “I'm looking for both tests and OKs.” There has also been a flurry of work (http://svnweb.freebsd.org/base/head/sys/net80211/?view=log) in FreeBSD on the ath10k driver, which supports 802.11ac Like this one (https://svnweb.freebsd.org/base?view=revision&revision=310147) and this one (https://svnweb.freebsd.org/base?view=revision&revision=311579) The long-awaited iocage update has landed (https://github.com/iocage/iocage) We've hinted at the new things happening behind the scenes with iocage, and this last week the code has made its first public debut. So what's changed you may ask. The biggest is that iocage has undergone a complete overhaul, moving from its original shell-base to python. The story behind that is that the author (Brandon) works at iXsystems, and the plan is to move away from the legacy warden-based jail management which was also shell-based. This new python re-write will allow it to integrate into FreeNAS (and other projects) better by exposing an API for all jail management tasks. Thats right, no more ugly CLI output parsing just to wrangle jail options either at creation or runtime. But what about users who just run iocage manually from the CLI? No worries, the new iocage is almost identical to the original CLI usage, making the switch over very simple. Just to re-cap, lets look at the new features list: “FEATURES: + Ease of use + Rapid jail creation within seconds + Automatic package installation + Virtual networking stacks (vnet) + Shared IP based jails (non vnet) + Transparent ZFS snapshot management + Export and import “ + The new iocage is available now via ports and packages under sysutils/py-iocage, give it a spin and be sure to report issues back to the developer(s). Reading DHT11 temperature sensors on a Raspberry Pi under FreeBSD (https://smallhacks.wordpress.com/2017/01/14/reading-dht11-temperature-sensor-on-raspberry-pi-under-freebsd/) “DHT-11 is a very cheap temperature/humidity sensor which is commonly used in the IoT devices. It is not very accurate, so for the accurate measurement i would recommend to use DHT21 instead. Anyway, i had DHT-11 in my tool box, so decided to start with it. DHT-11 using very simple 1 wire protocol – host is turning on chip by sending 18ms low signal to the data output and then reading 40 bytes of data.” “To read data from the chip it should be connected to the power (5v) and gpio pin. I used pin 2 as VCC, 6 as GND and 11 as GPIO” “There is no support for this device out of the box on FreeBSD. I found some sample code on the github, see lex/freebsd-gpio-dht11 (https://github.com/lex/freebsd-gpio-dht11) repository. This code was a good starting point, but soon i found 2 issues with it: Results are very unreliable, probably due to gpio decoding algorithm. Checksum is not validated, so sometime values are bogus. “Initially i was thinking to fix this myself, but later found kernel module for this purpose, 1 wire over gpio (http://www.my-tour.ru/FreeBSD/1-wire_over_gpio/). This module contains DHT11 kernel driver (gpio_sw) which implements DHT-11 protocol in the kernel space and exporting /dev/sw0 for the userland. Driver compiles on FreeBSD11/ARM without any changes. Use make install to install the driver.” The articles goes into how to install and configure the driver, including a set of devfs rules to allow non-root users to read from the sensor “Final goal was to add this sensor to the domoticz software. It is using LUA scripting to extend it functionality, e.g. to obtain data from non-supported or non standard devices. So, i decided to read /dev/sw0 from the LUA.” They ran into some trouble with LUA trying to read too much data at once, and had to work around it In the end, they got the results and were able to use them in the monitoring tool *** Tor-ified Home Network using HardenedBSD and a RPi3 (https://github.com/lattera/articles/blob/master/infosec/tor/2017-01-14_torified_home/article.md) Shawn from HardendBSD has posted an article up on GitHub talking about his deployment of a new Tor relay on a RPi3 This particular method was attractive, since it allows running a Relay, but without it being on a machine which may have personal data, such as SSH keys, files, etc While his setup is done on HardendBSD, the same applies to a traditional FreeBSD setup as well. First up, is the list of things needed for this project: Raspberry Pi 3 Model B Rev 1.2 (aka, RPI3) Serial console cable for the RPI3 Belkin F4U047 USB Ethernet Dongle Insignia NS-CR2021 USB 2.0 SD/MMC Memory Card Reader 32GB SanDisk Ultra PLUS MicroSDHC A separate system, running FreeBSD or HardenedBSD HardenedBSD clang 4.0.0 image for the RPI3 An external drive to be formatted A MicroUSB cable to power the RPI3 Two network cables Optional: Edimax N150 EW-7811Un Wireless USB Basic knowledge of vi After getting HBSD running on the RPi3 and serial connection established, he then takes us through the process of installing and enabling the various services needed. (Don't forget to growfs your sdcard first!) Now the tricky part is that some of the packages needed to be compiled from ports, which is somewhat time-consuming on a RPi. He strongly recommends not compiling on the sdcard (it sounds like personal experience has taught him well) and to use iscsi or some external USB drive. With the compiling done, our package / software setup is nearly complete. Next up is firewalling the box, which he helpfully provides a full PF config setup that we can copy-n-paste here. The last bits will be enabling the torrc configuration knobs, which if you follow his example again, will result in a tor public relay, and a local transparent proxy for you. Bonus! Shawn helpfully provides DHCPD configurations, and even Wireless AP configurations, if you want to setup your RPi3 to proxy for devices that connect to it. *** News Roundup Unix Admin. Horror Story Summary, version 1.0 (http://www-uxsup.csx.cam.ac.uk/misc/horror.txt) A great collection of stories, many of which will ring true with our viewers The very first one, is about a user changing root's shell to /usr/local/bin/tcsh but forgetting to make it executable, resulting in not being able to login as root. I too have run into this issue, in a slightly different way. I had tcsh as my user shell (back before tcsh was in base), and after a major OS upgrade, but before I had a chance to recompile all of my ports. Now I couldn't ssh in to the remote machine in order to recompile my shell. Now I always use a shell included in the base system, and test it before rebooting after an upgrade. “Our operations group, a VMS group but trying to learn UNIX, was assigned account administration. They were cleaning up a few non-used accounts like they do on VMS - backup and purge. When they came across the account "sccs", which had never been accessed, away it went. The "deleteuser" utility from DEC asks if you would like to delete all the files in the account. Seems reasonable, huh? Well, the home directory for "sccs" is "/". Enough said :-(“ “I was working on a line printer spooler, which lived in /etc. I wanted to remove it, and so issued the command "rm /etc/lpspl." There was only one problem. Out of habit, I typed "passwd" after "/etc/" and removed the password file. Oops.” I've done things like this as well. Finger memory can be dangerous “I was happily churning along developing something on a Sun workstation, and was getting a number of annoying permission denieds from trying to write into a directory heirarchy that I didn't own. Getting tired of that, I decided to set the permissions on that subtree to 777 while I was working, so I wouldn't have to worry about it. Someone had recently told me that rather than using plain "su", it was good to use "su -", but the implications had not yet sunk in. (You can probably see where this is going already, but I'll go to the bitter end.) Anyway, I cd'd to where I wanted to be, the top of my subtree, and did su -. Then I did chmod -R 777. I then started to wonder why it was taking so damn long when there were only about 45 files in 20 directories under where I (thought) I was. Well, needless to say, su - simulates a real login, and had put me into root's home directory, /, so I was proceeding to set file permissions for the whole system to wide open. I aborted it before it finished, realizing that something was wrong, but this took quite a while to straighten out.” Where is a ZFS snapshot when you need it? *** How individual contributors get stuck (https://medium.com/@skamille/how-do-individual-contributors-get-stuck-63102ba43516) An interesting post looking at the common causes of people getting stuck when trying to create or contribute new code Brainstorming/architecture: “I must have thought through all edge cases of all parts of everything before I can begin this project” Researching possible solutions forever (often accompanied by desire to do a “bakeoff” where they build prototypes in different platforms/languages/etc) Refactoring: “this code could be cleaner and everything would be just so much easier if we cleaned this up… and this up… and…” Helping other people instead of doing their assigned tasks (this one isn't a bad thing in an open source community) Working on side projects instead of the main project (it is your time, it is up to you how to spend it) Excessive testing (rare) Excessive automation (rare) Finish the last 10–20% of a project Start a project completely from scratch Do project planning (You need me to write what now? A roadmap?) (this is why FreeBSD has devsummits, some things you just need to whiteboard) Work with unfamiliar code/libraries/systems Work with other teams (please don't make me go sit with data engineering!!) Talk to other people Ask for help (far beyond the point they realized they were stuck and needed help) Deal with surprises or unexpected setbacks Deal with vendors/external partners Say no, because they can't seem to just say no (instead of saying no they just go into avoidance mode, or worse, always say yes) “Noticing how people get stuck is a super power, and one that many great tech leads (and yes, managers) rely on to get big things done. When you know how people get stuck, you can plan your projects to rely on people for their strengths and provide them help or even completely side-step their weaknesses. You know who is good to ask for which kinds of help, and who hates that particular challenge just as much as you do.” “The secret is that all of us get stuck and sidetracked sometimes. There's actually nothing particularly “bad” about this. Knowing the ways that you get hung up is good because you can choose to either a) get over the fears that are sticking you (lack of knowledge, skills, or confidence), b) avoid such tasks as much as possible, and/or c) be aware of your habits and use extra diligence when faced with tackling these areas.” *** Make Docs! (http://www.mkdocs.org/) “MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.” “MkDocs builds completely static HTML sites that you can host on GitHub pages, Amazon S3, or anywhere else you choose” It is an easy to install python package It includes a server mode that auto-refreshes the page as you write the docs, making it easy to preview your work before you post it online Everything needs docs, and writing docs should be as simple as possible, so that more of them will get written Go write some docs! *** Experimental FreeNAS 11/12 builds (https://forums.freenas.org/index.php?threads/new-freenas-9-10-with-freebsd-11-12-for-testing.49696/#post-341941) We know there's a lot of FreeNAS users who listen to BSDNow, so I felt it was important to share this little tidbit. I've posted something to the forums last night which includes links to brand-new spins of FreeNAS 9.10 based upon FreeBSD 11/stable and 12/current. These builds are updated nightly via our Jenkins infrastructure and hopefully will provide a new playground for technical folks and developers to experiment with FreeBSD features in their FreeNAS environment, long before they make it into a -STABLE release. As usual, the notes of caution do apply, these are nightlies, and as such bugs will abound. Do NOT use these with your production data, unless you are crazy, or just want an excuse to test your backup strategy If you do run these builds, of course feedback is welcome via the usual channels, such as the bug tracker. The hope is that by testing FreeBSD code earlier, we can vet and determine what is safe / ready to go into mainline FreeNAS sooner rather than later. *** Beastie Bits An Explainer on Unix's Most Notorious Code Comment (http://thenewstack.io/not-expected-understand-explainer/) turn your network inside out with one pf.conf trick (http://www.tedunangst.com/flak/post/turn-your-network-inside-out-with-one-pfconf-trick) A story of if_get(9) (http://www.grenadille.net/post/2017/01/13/A-story-of-if_get%289%29) Apple re-affirms its commitment to LLVM/Clang (http://lists.llvm.org/pipermail/llvm-dev/2017-January/108953.html) python 3k17 (http://www.tedunangst.com/flak/post/python-3k17) 2017 presentation proposals (http://blather.michaelwlucas.com/archives/2848) NetBSD 7.1_RC1 available (http://mail-index.netbsd.org/netbsd-announce/2017/01/09/msg000259.html) #define FSUFS2MAGIC 0x19540119 (Happy Birthday to Kirk McKusick tomorrow) *** Feedback/Questions J - LetsEncrypt (http://pastebin.com/nnQ9ZgyN) Mike - OpenRC (http://pastebin.com/EZ4tRiVb) Timothy - ZFS Horror (http://pastebin.com/ZqDFTsnR) Troels (http://pastebin.com/dhZEnREM) Jason - Disk Label (http://pastebin.com/q4F95S6h) ***
Another exciting week on BSDNow, we are queued up with LLVM / Linking news, a look at NetBSD's scheduler, This episode was brought to you by Headlines FreeBSD Kernel and World, and many Ports, can now be linked with lld (https://llvm.org/bugs/show_bug.cgi?id=23214#c40) “With this change applied I can link the entirety of the FreeBSD/amd64 base system (userland world and kernel) with LLD.” “Rafael's done an initial experimental Poudriere FreeBSD package build with lld head, and found almost 20K out of 26K ports built successfully. I'm now looking at getting CI running to test this on an ongoing basis. But, I think we're at the point where an experimental build makes sense.” Such testing will become much easier once llvm 4.0 is imported into -current “I suggest that during development we collect patches in a local git repo -- for example, I've started here for my Poudriere run https://github.com/emaste/freebsd-ports/commits/ports-lld” “It now looks like libtool is responsible for the majority of my failed / skipped ports. Unless we really think we'll add "not GNU" and other hacks to lld we're going to have to address libtool limitations upstream and in the FreeBSD tree. I did look into libtool a few weeks ago, but unfortunately haven't yet managed to produce a patch suitable for sending upstream.” If you are interested in LLVM/Clang/LLD/LLDB etc, check out: A Tourist's Guide to the LLVM Source Code (http://blog.regehr.org/archives/1453) *** Documenting NetBSD's scheduler tweaks (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20170109_2108.html) A followup to our previous coverage of improvements to the scheduler in NetBSD “NetBSD's scheduler was recently changed to better distribute load of long-running processes on multiple CPUs. So far, the associated sysctl tweaks were not documented, and this was changed now, documenting the kern.sched sysctls.” kern.sched.cacheht_time (dynamic): Cache hotness time in which a LWP is kept on one particular CPU and not moved to another CPU. This reduces the overhead of flushing and reloading caches. Defaults to 3ms. Needs to be given in ``hz'' units, see mstohz(9). kern.sched.balance_period (dynamic): Interval at which the CPU queues are checked for re-balancing. Defaults to 300ms. kern.sched.min_catch (dynamic): Minimum count of migratable (runable) threads for catching (stealing) from another CPU. Defaults to 1 but can be increased to decrease chance of thread migration between CPUs. It is important to have good documentation for these tunables, so that users can understand what it is they are adjusting *** FreeBSD Network Gateway on EdgeRouter Lite (http://codeghar.com/blog/freebsd-network-gateway-on-edgerouter-lite.html) “EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running FreeBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.” “Colin Percival has written a great blog post on the subject, titled FreeBSD on EdgeRouter Lite - no serial port required (http://www.daemonology.net/blog/2016-01-10-FreeBSD-EdgeRouter-Lite.html) . In it he provides and describes a shell script to build a bootable image of FreeBSD to be run on ERL, available from GitHub in the freebsd-ERL-build (https://github.com/cperciva/freebsd-ERL-build/) repo. I have built a Vagrant-based workflow to automate the building of the drive image. It's available on GitHub in the freebsd-edgerouterlite-ansible (https://github.com/hamzasheikh/freebsd-edgerouterlite-ansible) repo. It uses the build script Percival wrote.” “Once you've built the disk image it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive. I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I have not tried another third party drive since.” The tutorial covers all of the steps, and the configuration files, including rc.conf, IP configuration, DHCP (and v6), pf, and DNS (unbound) “I'm pretty happy with ERL and FreeBSD. There is great community documentation on how to configure all the pieces of software that make a FreeBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.” “My plan on upgrading the base OS is to get a third party USB drive that works, write a newer FreeBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.” Although something more nanobsd style with 2 partitions on the one drive might be easier. “Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well. ERL is simply a great piece of network hardware. I'm tempted to try Ubiquiti's WiFi products instead of a mixture of DD-WRT and OpenWRT devices I have now. But that is for another day and perhaps another blog post.” *** A highly portable build system targeting modern UNIX systems (https://github.com/michipili/bsdowl) An exciting new/old project is up on GitHub that we wanted to bring your attention to. BSD Owl is a highly portable build-system based around BSD Make that supports a variety of popular (and not so popular) languages, such as: C programs, compiled for several targets C libraries, static and shared, compiled for several targets Shell scripts Python scripts OCaml programs OCaml libraries, with ocamldoc documentation OCaml plugins TeX documents, prepared for several printing devices METAPOST figures, with output as PDF, PS, SVG or PNG, either as part of a TeX document or as standalone documents What about features you may ask? Well BSD Owl has plenty of those to go around: Support of compilation profiles Support of the parallel mode (at the directory level) Support of separate trees for sources and objects Support of architecture-dependant compilation options Support GNU autoconf Production of GPG-signed tarballs Developer subshell, empowered with project-specific scripts Literate programming using noweb Preprocessing with m4 As far as platform support goes, BSD Owl is tested on OSX / Debian Jesse and FreeBSD > 9. Future support for OpenBSD and NetBSD is planned, once they update their respective BSD Make binaries to more modern versions News Roundup find -delete in OpenBSD. Thanks to tedu@ OpenBSD will have this very handy flag to in the future. (https://marc.info/?l=openbsd-tech&m=148342051832692&w=2) OpenBSD's find(1) utility will now support the -delete operation “This option is not posix (not like that's stopped find accumulating a dozen extensions), but it is in gnu and freebsd (for 20 years). it's also somewhat popular among sysadmins and blogs, etc. and perhaps most importantly, it nicely solves one of the more troublesome caveats of find (which the man page actually covers twice because it's so common and easy to screw up). So I think it makes a good addition.” The actual code was borrowed from FreeBSD Using the -delete option is much more performant than forking rm once for each file, and safer because there is no risk of mangling path names If you encounter a system without a -delete option, your best bet is to use the -print0 option of find, which will print each filename terminated by a null byte, and pipe that into xargs -0 rm This avoids any ambiguity caused by files with spaces in the names *** New version of the Lumina desktop released (https://lumina-desktop.org/version-1-2-0-released/) Just in time to kickoff 2017 we have a new release of Lumina Desktop (1.2.0) Some of the notable changes include fixes to make it easier to port to other platforms, and some features: New Panel Plugins: “audioplayer” (panel version of the desktop plugin with the same name): Allows the user to load/play audio files directly through the desktop itself. “jsonmenu” (panel version of the menu plugin with the same name): Allows an external utility/script to be used to generate a menu/contents on demand. New Menu Plugins: “lockdesktop”: Menu option for instantly locking the desktop session. New Utilities: lumina-archiver: This is a pure Qt5 front-end to the “tar” utility for managing/creating archives. This can also use the dd utility to burn a “*.img” file to a USB device for booting.“ Looks like the news already made its rounds to a few different sites, with Phoronix and Softpedia picking it up as well Phoronix (http://www.phoronix.com/scan.php?page=news_item&px=Lumina-1.2-Released) Softpedia (http://news.softpedia.com/news/lumina-1-2-desktop-environments-launches-for-trueos-with-various-enhancements-511495.shtml) TrueOS users running the latest updates are already on the pre-release version of 1.2.1, so nothing has to be done there to get the latest and greatest. dd is not a disk writing tool (http://www.vidarholen.net/contents/blog/?p=479) “If you've ever used dd, you've probably used it to read or write disk images:” > # Write myfile.iso to a USB drive > dd if=myfile.iso of=/dev/sdb bs=1M “Usage of dd in this context is so pervasive that it's being hailed as the magic gatekeeper of raw devices. Want to read from a raw device? Use dd. Want to write to a raw device? Use dd. This belief can make simple tasks complicated. How do you combine dd with gzip? How do you use pv if the source is raw device? How do you dd over ssh?” “The fact of the matter is, dd is not a disk writing tool. Neither “d” is for “disk”, “drive” or “device”. It does not support “low level” reading or writing. It has no special dominion over any kind of device whatsoever.” Then a number of alternatives are discussed “However, this does not mean that dd is useless! The reason why people started using it in the first place is that it does exactly what it's told, no more and no less. If an alias specifies -a, cp might try to create a new block device rather than a copy of the file data. If using gzip without redirection, it may try to be helpful and skip the file for not being regular. Neither of them will write out a reassuring status during or after a copy.” “dd, meanwhile, has one job*: copy data from one place to another. It doesn't care about files, safeguards or user convenience. It will not try to second guess your intent, based on trailing slashes or types of files. When this is no longer a convenience, like when combining it with other tools that already read and write files, one should not feel guilty for leaving dd out entirely.” “dd is the swiss army knife of the open, read, write and seek syscalls. It's unique in its ability to issue seeks and reads of specific lengths, which enables a whole world of shell scripts that have no business being shell scripts. Want to simulate a lseek+execve? Use dd! Want to open a file with O_SYNC? Use dd! Want to read groups of three byte pixels from a PPM file? Use dd!” “It's a flexible, unique and useful tool, and I love it. My only issue is that, far too often, this great tool is being relegated to and inappropriately hailed for its most generic and least interesting capability: simply copying a file from start to finish.” “dd actually has two jobs: Convert and Copy. Legend has it that the intended name, “cc”, was taken by the C compiler, so the letters were shifted by one to give “dd”. This is also why we ended up with a Window system called X.” dd countdown (https://eriknstr.github.io/utils/dd-countdown.htm) *** Bhyve setup for tcp testing (https://www.strugglingcoder.info/index.php/bhyve-setup-for-tcp-testing/) FreeBSD Developer Hiren Panchasara writes about his setup to use bhyve to test changes to the TCP stack in FreeBSD “Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I've already wrote down how I do dummynet (https://www.strugglingcoder.info/index.php/drop-a-packet/) so I'll focus on bhyve part.” “A few months back when I started looking into improving FreeBSD TCP's response to packet loss, I looked around for traffic simulators which can do deterministic packet drop for me.” “I had used dummynet(4) before so I thought of using it but the problem is that it only provided probabilistic drops. You can specify dropping 10% of the total packets” So he wrote a quick hack, hopefully he'll polish it up and get it committed “Setup: I'll create 3 bhyve guests: client, router and server” “Both client and server need their routing tables setup correctly so that they can reach each other. The Dummynet node is the router / traffic shaping node here. We need to enable forwarding between interfaces: sysctl net.inet.ip.forwarding=1” “We need to setup links (called ‘pipes') and their parameters on dummynet node” “For simulations, I run a lighttpd web-server on the server which serves different sized objects and I request them via curl or wget from the client. I have tcpdump running on any/all of four interfaces involved to observe traffic and I can see specified packets getting dropped by dummynet. sysctl net.inet.ip.dummynet.iopktdrop is incremented with each packet that dummynet drops.” “Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.” Create 2 tap interfaces for each end point, and 3 from the router. One each for SSH/control, and the others for the test flows. Then create 3 bridges, the first includes all of the control tap interfaces, and your hosts' real interface. This allows the guests to reach the internet to download packages etc. The other two bridges form the connections between the three VMs The creation and configuration of the VMs is documented in detail I used a setup very similar to this for teaching the basics of how TCP works when I was teaching at a local community college *** Beastie Bits Plan9 on Bhyve (https://twitter.com/pr1ntf/status/817895393824382976) Get your name in the relayd book (http://blather.michaelwlucas.com/archives/2832) Ted Unangst's 2016 Computer Reviews (http://www.tedunangst.com/flak/post/2016-computer-review) Bryan Cantrill on Developer On Fire podcast (http://developeronfire.com/episode-198-bryan-cantrill-persistence-and-action) 2016 in review: pf/ipfw's impact on forwarding performance over time, on 8 core Atom (http://dev.bsdrp.net/benchs/2016.SM5018A-FTN4-Chelsio.png) #Wayland Weston with X and EGL clients, running on #FreeBSD in VBox with new scfb backend. More coming soon! (https://twitter.com/johalun/status/819039940914778112) Feedback/Questions Eddy - TRIM Partitioning (http://pastebin.com/A0LSipCj) Matt - Why FreeBSD? (http://pastebin.com/UE1k4Q99) Shawn - ZFS Horror? (http://pastebin.com/TjTkqHA4) Andrew - Bootloaders (http://pastebin.com/Baxd6Pjy) GELIBoot Paper (http://allanjude.com/talks/AsiaBSDCon2016_geliboot_pdf1a.pdf) FreeBSD Architecture Handbook (https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/boot.html) Bryan - ZFS Error (http://pastebin.com/NygwchFD) ***
This week on BSDNow, we've got all sorts of post-holiday goodies to share. New OpenSSL APIs, Dtrace, OpenBSD This episode was brought to you by Headlines OpenSSL 1.1 API migration path, or the lack thereof (https://www.mail-archive.com/tech@openbsd.org/msg36437.html) As many of you will already be aware, the OpenSSL 1.1.0 release intentionally introduced significant API changes from the previous release. In summary, a large number of data structures that were previously publically visible have been made opaque, with accessor functions being added in order to get and set some of the fields within these now opaque structs. It is worth noting that the use of opaque data structures is generally beneficial for libraries, since changes can be made to these data structures without breaking the ABI. As such, the overall direction of these changes is largely reasonable. However, while API change is generally necessary for progression, in this case it would appear that there is NO transition plan and a complete disregard for the impact that these changes would have on the overall open source ecosystem. So far it seems that the only approach is to place the migration burden onto each and every software project that uses OpenSSL, pushing significant code changes to each project that migrates to OpenSSL 1.1, while maintaining compatibility with the previous API. This is forcing each project to provide their own backwards compatibility shims, which is practically guaranteeing that there will be a proliferation of variable quality implementations; it is almost a certainty that some of these will contain bugs, potentially introducing security issues or memory leaks. I think this will be a bigger issue for other operating systems that do not have the flexibility of the ports tree to deliver a newer version of OpenSSL. If a project switches from the old API to the new API, and the OS only provides the older branch of OpenSSL, how can the application work? Of course, this leaves the issue, if application A wants OpenSSL 1.0, and application B only works with OpenSSL 1.1, how does that work? Due to a number of factors, software projects that make use of OpenSSL cannot simply migrate to the 1.1 API and drop support for the 1.0 API - in most cases they will need to continue to support both. Firstly, I am not aware of any platform that has shipped a production release with OpenSSL 1.1 - any software that supported OpenSSL 1.1 only, would effectively be unusable on every platform for the time being. Secondly, the OpenSSL 1.0.2 release is supported until the 31st of December 2019, while OpenSSL 1.1.0 is only supported until the 31st of August 2018 - any LTS style release is clearly going to consider shipping with 1.0.2 as a result. Platforms that are attempting to ship with OpenSSL 1.1 are already encountering significant challenges - for example, Debian currently has 257 packages (out of 518) that do not build against OpenSSL 1.1. There are also hidden gotchas for situations where different libraries are linked against different OpenSSL versions and then share OpenSSL data structures between them - many of these problems will be difficult to detect since they only fail at runtime. It will be interesting to see what happens with OpenSSL, and LibreSSL Hopefully, most projects will decide to switch to the cleaner APIs provided by s2n or libtls, although they do not provide the entire functionality of the OpenSSL API. Hacker News comments (https://news.ycombinator.com/item?id=13284648) *** exfiltration via receive timing (http://www.tedunangst.com/flak/post/exfiltration-via-receive-timing) Another similar way to create a backchannel but without transmitting anything is to introduce delays in the receiver and measure throughput as observed by the sender. All we need is a protocol with transmission control. Hmmm. Actually, it's easier (and more reliable) to code this up using a plain pipe, but the same principle applies to networked transmissions. For every digit we want to “send” back, we sleep a few seconds, then drain the pipe. We don't care about the data, although if this were a video file or an OS update, we could probably do something useful with it. Continuously fill the pipe with junk data. If (when) we block, calculate the difference between before and after. This is a our secret backchannel data. (The reader and writer use different buffer sizes because on OpenBSD at least, a writer will stay blocked even after a read depending on the space that opens up. Even simple demos have real world considerations.) In this simple example, the secret data (argv) is shared by the processes, but we can see that the writer isn't printing them from its own address space. Nevertheless, it works. Time to add random delays and buffering to firewalls? Probably not. An interesting thought experiment that shows just how many ways there are to covertly convey a message *** OpenBSD Desktop in about 30 Minutes (https://news.ycombinator.com/item?id=13223351) Over at hackernews we have a very non-verbose, but handy guide to getting to a OpenBSD desktop in about 30 minutes! First, the guide will assume you've already installed OpenBSD 6.0, so you'll need to at least be at the shell prompt of your freshly installed system to begin. With that, now its time to do some tuning. Editing some resource limits in login.conf will be our initial task, upping some datasize tunables to 2GB Next up, we will edit some of the default “doas” settings to something a bit more workable for desktop computing Another handy trick, editing your .profile to have your PKG_PATH variables set automatically will make One thing some folks may overlook, but disabling atime can speed disk performance (which you probably don't care about atime on your desktop anyway), so this guide will show you what knobs to tweak in /etc/fstab to do so After some final WPA / Wifi configuration, we then drop to “mere mortal” mode and begin our package installations. In this particular guide, he will be setting up Lumina Desktop (Which yes, it is on OpenBSD) A few small tweaks later for xscreensaver and your xinitrc file, then you are ready to run “startx” and begin your desktop session! All in all, great guide which if you are fast can probably be done in even less than 30 minutes and will result in a rock-solid OpenBSD desktop rocking Lumina none-the-less. *** How DTrace saved Christmas (https://hackernoon.com/dtrace-at-home-145ba773371e) Adam Leventhal, one of the co-creators of DTrace, wrote up this post about how he uses DTrace at home, to save Christmas I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I'm glad I don't have to do 2016 again. It was just before midnight and either I'd make the card that night or leave an empty space on our friends' refrigerators. Adobe Illustrator had other ideas: “Unable to set maximum number of files to be opened” I'm not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions were working for me, and — inspired by Sara Mauskopf's excellent post (https://medium.com/startup-grind/how-to-start-a-company-with-no-free-time-b70fbe7b918a#.uujdblxc6) — I was rapidly running out of the time bounds for the project. Enough; I'd just DTrace it. A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can't imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I'm forever grateful to the Apple team that ported it to Mac OS X) Illustrator is failing on setrlimit(2) and blowing up as result. Let's confirm that it is in fact returning -1:$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }' dtrace: description 'syscall::setrlimit:return' matched 1 probe CPU ID FUNCTION:NAME 0 532 setrlimit:return -1 1 There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let's confirm that it is in fact setting the files limit and check the value to which it's being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process's address space: $ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlimcur, this->r.rlimmax); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe CPU ID FUNCTION:NAME 0 531 setrlimit:entry 1008 2800 7fffffffffffffff Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMITNOFILE | _RLIMITPOSIX_FLAG) The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy: $ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlimt *)alloca(sizeof (rlimt)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlimt), sizeof (rlimt)); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe dtrace: could not enable tracing: Permission denied Oh right. Thank you SIP (System Integrity Protection). This is a new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it's new I haven't yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It's easy enough to disable, but I'm reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb + After using DTrace to get the address of the setrlimit function, Adam used lldb to change the result before it got back to the application: (lldb) break set -n _init Breakpoint 1: 47 locations. (lldb) run … (lldb) di -s 0x1006e5b72 -c 1 0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit (lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90 (lldb) di -s 0x1006e5b72 -c 4 0x1006e5b72: xorl %eax, %eax 0x1006e5b74: nop 0x1006e5b75: nop 0x1006e5b76: nop Next I just did a process detach and got on with making that holiday card… DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn't write a program doesn't mean you can't fix it. News Roundup Say my Blog's name! (https://functionallyparanoid.com/2016/12/22/say-my-blogs-name/) Brian Everly over at functionally paranoid has a treat for us today. Let us give you a moment to get the tin-foil hats on… Ok, done? Let's begin! He starts off with a look at physical security. He begins by listing your options: BIOS passwords – Not something I'm typically impressed with. Most can be avoided by opening up the machine, closing a jumper and powering it up to reset the NVRAM to factory defaults. I don't even bother with them. Full disk encryption – This one really rings my bell in a positive way. If you can kill power to the box (either because the bad actor has to physically steal it and they aren't carrying around a pile of car batteries and an inverter or because you can interrupt power to it some other way), then the disk will be encrypted. The other beauty of this is that if a drive fails (and they all do eventually) you don't have to have any privacy concerns about chucking it into an electronics recycler (or if you are a bad, bad person, into a landfill) because that data is effectively gibberish without the key (or without a long time to brute force it). Two factor auth for logins – I like this one as well. I'm not a fan of biometrics because if your fingerprint is compromised (yes, it can happen – read (https://www.washingtonpost.com/news/federal-eye/wp/2015/07/09/hack-of-security-clearance-system-affected-21-5-million-people-federal-authorities-say/) about the department of defense background checks that were extracted by a bad agent – they included fingerprint images) you can't exactly send off for a new finger. Things like the YubiKey (https://www.yubico.com/) are pretty slick. They require that you have the physical hardware key as well as the password so unless the bad actor lifted your physical key, they would have a much harder time with physical access to your hardware. Out of those options, Brian mentions that he uses disk encryption and yubi-key for all his secure network systems. Next up is network segmentation, in this case the first thing to do is change your admin password for any ISP supplied modem / router. He goes on to scare us of javascript attacks being used not against your local machine, but instead non WAN exposed router admin interface. Scary Stuff! For added security, naturally he firewalls the router by plugging in the LAN port to a OpenBSD box which does the 2nd layer of firewall / router protection. What about privacy and browsing? Here's some more of his tips: I use Unbound as my DNS resolver on my local network (with all UDP port 53 traffic redirected to it by pf so I don't have to configure anything on the clients) and then forward the traffic to DNSCrypt Proxy, caching the results in Unbound. I notice ZERO performance penalty for this and it greatly enhances privacy. This combination of Unbound and DNSCrypt Proxy works very well together. You can even have redundancy by having multiple upstream resolvers running on different ports (basically run the DNSCrypt Proxy daemon multiple times pointing to different public resolvers). I also use Firefox exclusively for my web browsing. By leveraging the tips on this page (https://www.privacytools.io/), you can lock it down to do a great job of privacy protection. The fact that your laptop's battery drain rate can be used to fingerprint your browser completely trips me out but hey – that's the world we live in.' What about the cloud you may ask? Well Brian has a nice solution for that as well: I recently decided I would try to live a cloud-free life and I'll give you a bit of a synopsis on it. I discovered a wonderful Open Source project called FreeNAS (http://www.freenas.org/). What this little gem does is allow you to install a FreeBSD/zfs file server appliance on amd64 hardware and have a slick administrative web interface for managing it. I picked up a nice SuperMicro motherboard and chassis that has 4 hot swap drive bays (and two internal bays that I used to mirror the boot volume on) and am rocking the zfs lifestyle! (Thanks Alan Jude!) One of the nicest features of the FreeNAS is that it provides the ability to leverage the FreeBSD jail functionality in an easy to use way. It also has plugins but the security on those is a bit sketchy (old versions of libraries, etc.) so I decided to roll my own. I created two jails – one to run OwnCloud (yeah, I know about NextCloud and might switch at some point) and the other to run a full SMTP/IMAP email server stack. I used Lets Encrypt (https://letsencrypt.org/) to generate the SSL certificates and made sure I hit an A on SSLLabs (https://www.ssllabs.com/) before I did anything else. His post then goes in to talk about Backups and IoT devices, something else you need to consider in this truely paranoid world we are forced to live in. We even get a nice shout-out near the end! Enter TarSnap (http://www.tarsnap.com/) – a company that advertises itself as “Online Backups for the Truly Paranoid”. It brings a tear to my eye – a kindred spirit! :-) Thanks again to Alan Jude and Kris Moore from the BSD Now podcast (http://www.bsdnow.tv/) for turning me onto this company. It has a very easy command syntax (yes, it isn't a GUI tool – suck it up buttercup, you wanted to learn the shell didn't you?) and even allows you to compile the thing from source if you want to.” We've only covered some of the highlights here, but you really should take a few moments of your time today and read this top to bottom. Lots of good tips here, already thinking how I can secure my home network better. The open source book: “Producing Open Source Software” (http://producingoss.com/en/producingoss.pdf) “How to Run a Successful Free Software Project” by Karl Fogel 9 chapters and over 200 pages of content, plus many appendices Some interesting topics include: Choosing a good name version control bug tracking creating developer guidelines setting up communications channels choosing a license (although this guide leans heavily towards the GPL) setting the tone of the project joining or creating a Non-Profit Organization the economics of open source release engineering, packaging, nightly builds, etc how to deal with forks A lot of good information packaged into this ebook This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License *** DTrace Flamegraphs for node.js on FreeBSD (http://www.venshare.com/dtrace-flamegraphs-for-freebsd-and-node-js-2/) One of the coolest tools built on top of DTrace is flamegraphs They are a very accurate, and visual way to see where a program is spending its time, which can tell you why it is slow, or where it could be improved. Further enhancements include off-cpu flame graphs, which tell you when the program is doing nothing, which can also be very useful > Recently BSD UNIXes are being acknowledged by the application development community as an interesting operating system to deploy to. This is not surprising given that FreeBSD had jails, the original container system, about 17 years ago and a lot of network focused businesses such as netflix see it as the best way to deliver content. This developer interest has led to hosting providers supporting FreeBSD. e.g. Amazon, Azure, Joyent and you can get a 2 months free instance at Digital Ocean. DTrace is another vital feature for anyone who has had to deal with production issues and has been in FreeBSD since version 9. As of FreeBSD 11 the operating system now contains some great work by Fedor Indutny so you can profile node applications and create flamegraphs of node.js processes without any additional runtime flags or restarting of processes. This is one of the most important things about DTrace. Many applications include some debugging functionality, but they require that you stop the application, and start it again in debugging mode. Some even require that you recompile the application in debugging mode. Being able to attach DTrace to an application, while it is under load, while the problem is actively happening, can be critical to figuring out what is going on. In order to configure your FreeBSD instance to utilize this feature make the following changes to the configuration of the server. Load the DTrace module at boot Increase some DTrace limits Install node with the optional DTrace feature compiled in Follow the generic node.js flamegraph tutorial (https://nodejs.org/en/blog/uncategorized/profiling-node-js/) > I hope you find this article useful. The ability to look at a runtime in this manor has saved me twice this year and I hope it will save you in the future too. My next post on freeBSD and node.js will be looking at some scenarios on utilising the ZFS features. Also check out Brendan Gregg's ACM Queue Article (http://queue.acm.org/detail.cfm?id=2927301) “The Flame Graph: This visualization of software execution is a new necessity for performance profiling and debugging” SSHGuard 2.0 Call for Testing (https://sourceforge.net/p/sshguard/mailman/message/35580961/) SSHGuard is a tool for monitoring brute force attempts and blocking them It has been a favourite of mine for a while because it runs as a pipe from syslogd, rather than reading the log files from the disk A lot of work to get SSHGuard working with new log sources (journalctl, macOS log) and backends (firewalld, ipset) has happened in 2.0. The new version also uses a configuration file. Most importantly, SSHGuard has been split into several processes piped into one another (sshg-logmon | sshg-parser | sshg-blocker | sshg-fw). sshg-parser can run with capsicum(4) and pledge(2). sshg-blocker can be sandboxed in its default configuration (without pid file, whitelist, blacklisting) and has not been tested sandboxed in other configurations. Breaking the processes up so that the sensitive bits can be sandboxes is very nice to see *** Beastie Bits pjd's 2007 paper from AsiaBSDCon: “Porting the ZFS file system to the FreeBSD operating system” (https://2007.asiabsdcon.org/papers/P16-paper.pdf) A Message From the FreeBSD Foundation (https://vimeo.com/user60888329) Remembering Roger Faulkner, Unix Champion (http://thenewstack.io/remembering-roger-faulkner/) and A few HN comments (including Bryan Cantrill) (https://news.ycombinator.com/item?id=13293596) Feedback/Questions Peter - TrueOS Network (http://pastebin.com/QtyJeHMk) Chris - Remote Desktop (http://pastebin.com/ru726VTV) Goetz - Geli on Serial (http://pastebin.com/LQZPgF5g) Joe - BGP (http://pastebin.com/jFeL8zKX) Alejandro - BSD Router (http://pastebin.com/Xq9cbmfn) ***
A look back at 2016 This episode was brought to you by LinksZFS in the trenches | BSD Now 123One small step for DRM, one giant leap for BSD | BSD Now 143The Laporte has landed! | BSD Now 152Ham, Radio & Pie, Oh My! | BSD Now 158The Foundation of NetBSD | BSD Now 162Return of the Cantrill | BSD Now 163
This week on the show, we've got some great stories to bring you, a look at the odder side of UNIX history This episode was brought to you by Headlines syspatch in testing state (http://marc.info/?l=openbsd-tech&m=148058309126053&w=2) Antoine Jacoutot ajacoutot@ openbsd has posted a call for testing for OpenBSD's new syspatch tool “syspatch(8), a "binary" patch system for -release is now ready for early testing. This does not use binary diffing to update the system, but regular signed tarballs containing the updated files (ala installer).” “I would appreciate feedback on the tool. But please send it directly to me, there's no need to pollute the list. This is obviously WIP and the tool may or may not change in drastic ways.” “These test binary patches are not endorsed by the OpenBSD project and should not be trusted, I am only providing them to get early feedback on the tool. If all goes as planned, I am hoping that syspatch will make it into the 6.1 release; but for it to happen, I need to know how it breaks your systems :-)” Instructions (http://syspatch.openbsd.org/pub/OpenBSD/6.0/syspatch/amd64/README.txt) If you test it, report back and let us know how it went *** Weston working (https://lists.freebsd.org/pipermail/freebsd-current/2016-December/064198.html) Over the past few years we've had some user-interest in the state of Wayland / Weston on FreeBSD. In the past day or so, Johannes Lundberg has sent in a progress report to the FreeBSD mailing lists. Without further ADO: We had some progress with Wayland that we'd like to share. Wayland (v1.12.0) Working Weston (v1.12.0) Working (Porting WIP) Weston-clients (installed with wayland/weston port) Working XWayland (run X11 apps in Wayland compositor) Works (maximized window only) if started manually but not when launching X11 app from Weston. Most likely problem with Weston IPC. Sway (i3-compatible Wayland compositor) Working SDL20 (Wayland backend) games/stonesoup-sdl briefly tested. https://twitter.com/johalun/status/811334203358867456 GDM (with Wayland) Halted - depends on logind. GTK3 gtk3-demo runs fine on Weston (might have to set GDK_BACKEND=wayland first. GTK3 apps working (gedit, gnumeric, xfce4-terminal tested, xfce desktop (4.12) does not yet support GTK3)“ Johannes goes on to give instructions on how / where you can fetch their WiP and do your own testing. At the moment you'll need Matt Macy's newer Intel video work, as well as their ports tree which includes all the necessary software bits. Before anybody asks, yes we are watching this for TrueOS! *** Where the rubber meets the road (part two) (https://functionallyparanoid.com/2016/12/15/where-the-rubber-meets-the-road-part-two/) Continuing with our story from Brian Everly from a week ago, we have an update today on the process to dual-boot OpenBSD with Arch Linux. As we last left off, Arch was up and running on the laptop, but some quirks in the hardware meant OpenBSD would take a bit longer. With those issues resolved and the HD seen again, the next issue that reared its head was OpenBSD not seeing the partition tables on the disk. After much frustration, it was time to nuke and pave, starting with OpenBSD first this time. After a successful GPT partitioning and install of OpenBSD, he went back to installing Arch, and then the story got more interesting. “I installed Arch as I detailed in my last post; however, when I fired up gdisk I got a weird error message: “Warning! Disk size is smaller than the main header indicates! Loading secondary header from the last sector of the disk! You should use ‘v' to verify disk integrity, and perhaps options on the expert's menu to repair the disk.” Immediately after this, I saw a second warning: “Caution: Invalid backup GPT header, but valid main header; regenerating backup header from main header.” And, not to be outdone, there was a third: “Warning! Main and backup partition tables differ! Use the ‘c' and ‘e' options on the recovery & transformation menu to examine the two tables.” Finally (not kidding), there was a fourth: “Warning! One or more CRCs don't match. You should repair the disk!” Given all of that, I thought to myself, “This is probably why I couldn't see the disk properly when I partitioned it under Linux on the OpenBSD side. I'll let it repair things and I should be good to go.” I then followed the recommendation and repaired things, using the primary GPT table to recreate the backup one. I then installed Arch and figured I was good to go.“ After confirming through several additional re-installs that the behavior was reproducible, he then decided to go full on crazy,and partition with MBR. That in and of itself was a challenge, since as he mentions, not many people dual-boot OpenBSD with Linux on MBR, especially using luks and lvm! If you want to see the details on how that was done, check it out. The story ends in success though! And better yet: “Now that I have everything working, I'll restore my config and data to Arch, configure OpenBSD the way I like it and get moving. I'll take some time and drop a note on the tech@ mailing list for OpenBSD to see if they can figure out what the GPT problem was I was running into. Hopefully it will make that part of the code stronger to get an edge-case bug report like this.” Take note here, if you run into issues like this with any OS, be sure to document in detail what happened so developers can explore solutions to the issue. *** FreeBSD and ZFS as a time capsule for OS X (https://blog.feld.me/posts/2016/12/using-freebsd-as-a-time-capsule-for-osx/) Do you have any Apple users in your life? Perhaps you run FreeBSD for ZFS somewhere else in the house or office. Well today we have a blog post from Mark Felder which shows how you can use FreeBSD as a time-capsule for your OSX systems. The setup is quite simple, to get started you'll need packages for netatalk3 and avahi-app for service discovery. Next up will be your AFP configuration. He helpfully provides a nice example that you should be able to just cut-n-paste. Be sure to check the hosts allow lines and adjust to fit your network. Also of note will be the backup location and valid users to adjust. A little easier should be the avahi setup, which can be a straight copy-n-paste from the site, which will perform the service advertisements. The final piece is just enabling specific services in /etc/rc.conf and either starting them by hand, or rebooting. At this point your OSX systems should be able to discover the new time-capsule provider on the network and DTRT. *** News Roundup netbenches - FreeBSD network forwarding performance benchmark results (https://github.com/ocochard/netbenches) Olivier Cochard-Labbé, original creator of FreeNAS, and leader of the BSD Router Project, has a github repo of network benchmarks There are many interesting results, and all of the scripts, documentation, and configuration files to run the tests yourself IPSec Performance on an Atom C2558, 12-head vs IPSec Performance Branch (https://github.com/ocochard/netbenches/tree/master/Atom_C2558_4Cores-Intel_i350/ipsec/results/fbsd12.projects-ipsec.equilibrium) Compared to: Xeon L5630 2.13GHz (https://github.com/ocochard/netbenches/tree/2f3bb1b3c51e454736f1fcc650c3328071834f8d/Xeon_L5630-4Cores-Intel_82599EB/ipsec/results/fbsd11.0) and IPSec with Authentication (https://github.com/ocochard/netbenches/tree/305235114ba8a3748ad9681c629333f87f82613a/Atom_C2558_4Cores-Intel_i350/ipsec.ah/results/fbsd12.projects-ipsec.equilibrium) I look forward to seeing tests on even more hardware, as people with access to different configurations try out these benchmarks *** A tcpdump Tutorial and Primer with Examples (https://danielmiessler.com/study/tcpdump/) Most users will be familiar with the basics of using tcpdump, but this tutorial/primer is likely to fill in a lot of blanks, and advance many users understanding of tcpdump “tcpdump is the premier network analysis tool for information security professionals. Having a solid grasp of this über-powerful application is mandatory for anyone desiring a thorough understanding of TCP/IP. Many prefer to use higher level analysis tools such as Wireshark, but I believe this to usually be a mistake.” tcpdump is an important tool for any system or network administrator, it is not just for security. It is often the best way to figure out why the network is not behaving as expected. “In a discipline so dependent on a true understanding of concepts vs. rote learning, it's important to stay fluent in the underlying mechanics of the TCP/IP suite. A thorough grasp of these protocols allows one to troubleshoot at a level far beyond the average analyst, but mastery of the protocols is only possible through continued exposure to them.” Not just that, but TCP/IP is a very interesting protocol, considering how little it has changed in its 40+ year history “First off, I like to add a few options to the tcpdump command itself, depending on what I'm looking at. The first of these is -n, which requests that names are not resolved, resulting in the IPs themselves always being displayed. The second is -X, which displays both hex and ascii content within the packet.” “It's also important to note that tcpdump only takes the first 96 bytes of data from a packet by default. If you would like to look at more, add the -s number option to the mix, where number is the number of bytes you want to capture. I recommend using 0 (zero) for a snaplength, which gets everything.” The page has a nice table of the most useful options It also has a great primer on doing basic filtering If you are relatively new to using tcpdump, I highly recommend you spend a few minutes reading through this article *** How Unix made it to the top (http://minnie.tuhs.org/pipermail/tuhs/2016-December/007519.html) Doug McIlroy gives us a nice background post on how “Unix made it to the top” It's fairly short / concise, so I felt it would be good to read in its entirety. “It has often been told how the Bell Labs law department became the first non-research department to use Unix, displacing a newly acquired stand-alone word-processing system that fell short of the department's hopes because it couldn't number the lines on patent applications, as USPTO required. When Joe Ossanna heard of this, he told them about roff and promised to give it line-numbering capability the next day. They tried it and were hooked. Patent secretaries became remote members of the fellowship of the Unix lab. In due time the law department got its own machine. Less well known is how Unix made it into the head office of AT&T. It seems that the CEO, Charlie Brown, did not like to be seen wearing glasses when he read speeches. Somehow his PR assistant learned of the CAT phototypesetter in the Unix lab and asked whether it might be possible to use it to produce scripts in large type. Of course it was. As connections to the top never hurt, the CEO's office was welcomed as another ouside user. The cost--occasionally having to develop film for the final copy of a speech--was not onerous. Having teethed on speeches, the head office realized that Unix could also be useful for things that didn't need phototypesetting. Other documents began to accumulate in their directory. By the time we became aware of it, the hoard came to include minutes of AT&T board meetings. It didn't seem like a very good idea for us to be keeping records from the inner sanctum of the corporation on a computer where most everybody had super-user privileges. A call to the PR guy convinced him of the wisdom of keeping such things on their own premises. And so the CEO's office bought a Unix system. Just as one hears of cars chosen for their cupholders, so were theseusers converted to Unix for trivial reasons: line numbers and vanity.“ Odd Comments and Strange Doings in Unix (http://orkinos.cmpe.boun.edu.tr/~kosar/odd.html) Everybody loves easter-eggs, and today we have some fun odd ones from the history throughout UNIX told by Dennis Ritchie. First up, was a fun one where the “mv” command could sometimes print the following “values of b may give rise to dom!” “Like most of the messages recorded in these compilations, this one was produced in some situation that we considered unlikely or as result of abuse; the details don't matter. I'm recording why the phrase was selected. The very first use of Unix in the "real business" of Bell Labs was to type and produce patent applications, and for a while in the early 1970s we had three typists busily typing away in the grotty lab on the sixth floor. One day someone came in and observed on the paper sticking out of one of the Teletypes, displayed in magnificent isolation, this ominous phrase: values of b may give rise to dom! It was of course obvious that the typist had interrupted a printout (generating the "!" from the ed editor) and moved up the paper, and that the context must have been something like "varying values of beta may give rise to domain wall movement" or some other fragment of a physically plausible patent application.But the phrase itself was just so striking! Utterly meaningless, but it looks like what... a warning? What is "dom?" At the same time, we were experimenting with text-to-voice software by Doug McIlroy and others, and of course the phrase was tried out with it. For whatever reason, its rendition of "give rise to dom!" accented the last word in a way that emphasized the phonetic similarity between "doom" and the first syllable of "dominance." It pronounced "beta" in the British style, "beeta." The entire occurrence became a small, shared treasure.The phrase had to be recorded somewhere, and it was, in the v6 source. Most likely it was Bob Morris who did the deed, but it could just as easily have been Ken. I hope that your browser reproduces the b as a Greek beta.“ Next up is one you might have heard before: /* You are not expected to understand this */> Every now and then on Usenet or elsewhere I run across a reference to a certain comment in the source code of the Sixth Edition Unix operating system. I've even been given two sweatshirts that quote it. Most probably just heard about it, but those who saw it in the flesh either had Sixth Edition Unix (ca. 1975) or read the annotated version of this system by John Lions (which was republished in 1996: ISBN 1-57298-013-7, Peer-to-Peer Communications).It's often quoted as a slur on the quantity or quality of the comments in the Bell Labs research releases of Unix. Not an unfair observation in general, I fear, but in this case unjustified. So we tried to explain what was going on. "You are not expected to understand this" was intended as a remark in the spirit of "This won't be on the exam," rather than as an impudent challenge. There's a few other interesting stories as well, if the odd/fun side of UNIX history at all interests you, I would recommend checking it out. Beastie Bits With patches in review the #FreeBSD base system builds 100% reproducibly (https://twitter.com/ed_maste/status/811289279611682816) BSDCan 2017 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/bsdcan-2017/) ioCell 2.0 released (https://github.com/bartekrutkowski/iocell/releases) who even calls link_ntoa? (http://www.tedunangst.com/flak/post/who-even-calls-link-ntoa) Booting Androidx86 under bhyve (https://twitter.com/pr1ntf/status/809528845673996288) Feedback/Questions Chris - VNET (http://pastebin.com/016BfvU9) Brian - Package Base (http://pastebin.com/8JJeHuRT) Wim - TrueOS Desktop All-n-one (http://pastebin.com/VC0DPQUF) Daniel - Long Boots (http://pastebin.com/q7pFu7pR) Bryan - ZFS / FreeNAS (http://pastebin.com/xgUnbzr7) Bryan - FreeNAS Security (http://pastebin.com/qqCvVTLB) ***
This week on BSDNow, we have a very special guest joining us to tell us a tale of the early days in BSD history. That plus some new OpenSSH goodness, shell scripting utilities and much more. Stay tuned for your place to B...SD! This episode was brought to you by Headlines Call For Testing: OpenSSH 7.4 (http://marc.info/?l=openssh-unix-dev&m=148167688911316&w=2) Getting ready to head into the holidays for for the end of 2016 means some of us will have spare time on our hands. What a perfect time to get some call for testing work done! Damien Miller has issued a public CFT for the upcoming OpenSSH 7.4 release, which considering how much we all rely on SSH I would expect will get some eager volunteers for testing. What are some of the potential breakers? “* This release removes server support for the SSH v.1 protocol. ssh(1): Remove 3des-cbc from the client's default proposal. 64-bit block ciphers are not safe in 2016 and we don't want to wait until attacks like SWEET32 are extended to SSH. As 3des-cbc was the only mandatory cipher in the SSH RFCs, this may cause problems connecting to older devices using the default configuration, but it's highly likely that such devices already need explicit configuration for key exchange and hostkey algorithms already anyway. sshd(8): Remove support for pre-authentication compression. Doing compression early in the protocol probably seemed reasonable in the 1990s, but today it's clearly a bad idea in terms of both cryptography (cf. multiple compression oracle attacks in TLS) and attack surface. Pre-auth compression support has been disabled by default for >10 years. Support remains in the client. ssh-agent will refuse to load PKCS#11 modules outside a whitelist of trusted paths by default. The path whitelist may be specified at run-time. sshd(8): When a forced-command appears in both a certificate and an authorized keys/principals command= restriction, sshd will now refuse to accept the certificate unless they are identical. The previous (documented) behaviour of having the certificate forced-command override the other could be a bit confusing and error-prone. sshd(8): Remove the UseLogin configuration directive and support for having /bin/login manage login sessions.“ What about new features? 7.4 has some of those to wake you up also: “* ssh(1): Add a proxy multiplexing mode to ssh(1) inspired by the version in PuTTY by Simon Tatham. This allows a multiplexing client to communicate with the master process using a subset of the SSH packet and channels protocol over a Unix-domain socket, with the main process acting as a proxy that translates channel IDs, etc. This allows multiplexing mode to run on systems that lack file- descriptor passing (used by current multiplexing code) and potentially, in conjunction with Unix-domain socket forwarding, with the client and multiplexing master process on different machines. Multiplexing proxy mode may be invoked using "ssh -O proxy ..." sshd(8): Add a sshdconfig DisableForwaring option that disables X11, agent, TCP, tunnel and Unix domain socket forwarding, as well as anything else we might implement in the future. Like the 'restrict' authorizedkeys flag, this is intended to be a simple and future-proof way of restricting an account. sshd(8), ssh(1): Support the "curve25519-sha256" key exchange method. This is identical to the currently-support method named "curve25519-sha256@libssh.org". sshd(8): Improve handling of SIGHUP by checking to see if sshd is already daemonised at startup and skipping the call to daemon(3) if it is. This ensures that a SIGHUP restart of sshd(8) will retain the same process-ID as the initial execution. sshd(8) will also now unlink the PidFile prior to SIGHUP restart and re-create it after a successful restart, rather than leaving a stale file in the case of a configuration error. bz#2641 sshd(8): Allow ClientAliveInterval and ClientAliveCountMax directives to appear in sshd_config Match blocks. sshd(8): Add %-escapes to AuthorizedPrincipalsCommand to match those supported by AuthorizedKeysCommand (key, key type, fingerprint, etc.) and a few more to provide access to the contents of the certificate being offered. Added regression tests for string matching, address matching and string sanitisation functions. Improved the key exchange fuzzer harness.“ Get those tests done and be sure to send feedback, both positive and negative. *** How My Printer Caused Excessive Syscalls & UDP Traffic (https://zinascii.com/2014/how-my-printer-caused-excessive-syscalls.html) “3,000 syscalls a second, on an idle machine? That doesn't seem right. I just booted this machine. The only processes running are those required to boot the SmartOS Global Zone, which is minimal.” This is a story from 2014, about debugging a machine that was being slowed down by excessive syscalls and UDP traffic. It is also an excellent walkthrough of the basics of DTrace “Well, at least I have DTrace. I can use this one-liner to figure out what syscalls are being made across the entire system.” dtrace -n 'syscall:::entry { @[probefunc,probename] = count(); }' “Wow! That is a lot of lwpsigmask calls. Now that I know what is being called, it's time to find out who is doing the calling? I'll use another one-liner to show me the most common user stacks invoking lwpsigmask.” dtrace -n 'syscall::lwp_sigmask:entry { @[ustack()] = count(); }' “Okay, so this mdnsd code is causing all the trouble. What is the distribution of syscalls for the mdnsd program?” dtrace -n 'syscall:::entry /execname == "mdnsd"/ { @[probefunc] = count(); } tick-1s { exit(0); }' “Lots of signal masking and polling. What the hell! Why is it doing this? What is mdnsd anyways? Is there a man page? Googling for mdns reveals that it is used for resolving host names in small networks, like my home network. It uses UDP, and requires zero configuration. Nothing obvious to explain why it's flipping out. I feel helpless. I turn to the only thing I can trust, the code.” “Woah boy, this is some messy looking code. This would not pass illumos cstyle checks. Turns out this is code from Darwin—the kernel of OSX.” “Hmmm…an idea pops into my computer animal brain. I wonder…I wonder if my MacBook is also experiencing abnormal syscall rates? Nooo, that can't be it. Why would both my SmartOS server and MacBook both have the same problem? There is no good technical reason to link these two. But, then again, I'm dealing with computers here, and I've seen a lot of strange things over the years—I switch to my laptop.” sudo dtrace -n 'syscall::: { @[execname] = count(); } tick-1s { exit(0); }' Same thing, except mdns is called discoverd on OS X “I ask my friend Steve Vinoski to run the same DTrace one-liner on his OSX machines. He has both Yosemite and the older Mountain Lion. But, to my dismay, neither of his machines are exhibiting high syscall rates. My search continues.” “Not sure what to do next, I open the OSX Activity Monitor. In desperation I click on the Network tab.” “ HOLE—E—SHIT! Two-Hundred-and-Seventy Million packets received by discoveryd. Obviously, I need to stop looking at code and start looking at my network. I hop back onto my SmartOS machine and check network interface statistics.” “Whatever is causing all this, it is sending about 200 packets a second. At this point, the only thing left to do is actually inspect some of these incoming packets. I run snoop(1M) to collect events on the e1000g0 interface, stopping at about 600 events. Then I view the first 15.” “ A constant stream of mDNS packets arriving from IP 10.0.1.8. I know that this IP is not any of my computers. The only devices left are my iPhone, AppleTV, and Canon printer. Wait a minute! The printer! Two days earlier I heard some beeping noises…” “I own a Canon PIXMA MG6120 printer. It has a touch interface with a small LCD at the top, used to set various options. Since it sits next to my desk I sometimes lay things on top of it like a book or maybe a plate after I'm done eating. If I lay things in the wrong place it will activate the touch interface and cause repeated pressing. Each press makes a beeping noise. If the object lays there long enough the printer locks up and I have to reboot it. Just such events occurred two days earlier.” “I fire up dladm again to monitor incoming packets in realtime. Then I turn to the printer. I move all the crap off of it: two books, an empty plate, and the title for my Suzuki SV650 that I've been meaning to sell for the last year. I try to use the touch screen on top of the printer. It's locked up, as expected. I cut power to the printer and whip my head back to my terminal.” No more packet storm “Giddy, I run DTrace again to count syscalls.” “I'm not sure whether to laugh or cry. I laugh, because, LOL computers. There's some new dumb shit you deal with everyday, better to roll with the punches and laugh. You live longer that way. At least I got to flex my DTrace muscles a bit. In fact, I felt a bit like Brendan Gregg when he was debugging why OSX was dropping keystrokes.” “I didn't bother to root cause why my printer turned into a UDP machine gun. I don't intend to either. I have better things to do, and if rebooting solves the problem then I'm happy. Besides, I had to get back to what I was trying to do six hours before I started debugging this damn thing.” There you go. The Internet of Terror has already been on your LAN for years. Making Getaddrinfo Concurrent in Python on Mac OS and BSD (https://emptysqua.re/blog/getaddrinfo-cpython-mac-and-bsd/) We have a very fun blog post today to pass along originally authored by “A. Jesse Jiryu Davis”. Specifically the tale of one man's quest to unify the Getaddrinfo in Python with Mac OS and BSD. To give you a small taste of this tale, let us pass along just the introduction “Tell us about the time you made DNS resolution concurrent in Python on Mac and BSD. No, no, you do not want to hear that story, my friends. It is nothing but old lore and #ifdefs. But you made Python more scalable. The saga of Steve Jobs was sung to you by a mysterious wizard with a fanciful nickname! Tell us! Gather round, then. I will tell you how I unearthed a lost secret, unbound Python from old shackles, and banished an ancient and horrible Mutex Troll. Let us begin at the beginning.“ Is your interest piqued? It should be. I'm not sure we could do this blog post justice trying to read it aloud here, but definetly recommend if you want to see how he managed to get this bit of code working cross platform. (And it's highly entertaining as well) “A long time ago, in the 1980s, a coven of Berkeley sorcerers crafted an operating system. They named it after themselves: the Berkeley Software Distribution, or BSD. For generations they nurtured it, growing it and adding features. One night, they conjured a powerful function that could resolve hostnames to IPv4 or IPv6 addresses. It was called getaddrinfo. The function was mighty, but in years to come it would grow dangerous, for the sorcerers had not made getaddrinfo thread-safe.” “As ages passed, BSD spawned many offspring. There were FreeBSD, OpenBSD, NetBSD, and in time, Mac OS X. Each made its copy of getaddrinfo thread safe, at different times and different ways. Some operating systems retained scribes who recorded these events in the annals. Some did not.” The story continues as our hero battles the Mutex Troll and quests for ancient knowledge “Apple engineers are not like you and me — they are a shy and secretive folk. They publish only what code they must from Darwin. Their comings and goings are recorded in no bug tracker, their works in no changelog. To learn their secrets, one must delve deep.” “There is a tiny coven of NYC BSD users who meet at the tavern called Stone Creek, near my dwelling. They are aged and fierce, but I made the Sign of the Trident and supplicated them humbly for advice, and they were kindly to me.” Spoiler: “Without a word, the mercenary troll shouldered its axe and trudged off in search of other patrons on other platforms. Never again would it hold hostage the worthy smiths forging Python code on BSD.” *** Using release(7) to create FreeBSD images for OpenStack (https://diegocasati.com/2016/12/13/using-release7-to-create-freebsd-images-for-openstack-yes-you-can-do-it/) Following a recent episode where we covered a walk through on how to create FreeBSD guest OpenStack images, we wondered if it would be possible to integrate this process into the FreeBSD release(7) process, so they images could be generated consistently and automatically Being the awesome audience that you are, one of you responded by doing exactly that “During a recent BSDNow podcast, Allan and Kris mentioned that it would be nice to have a tutorial on how to create a FreeBSD image for OpenStack using the official release(7) tools. With that, it came to me that: #1 I do have access to an OpenStack environment and #2 I am interested in having FreeBSD as a guest image in my environment. Looks like I was up for the challenge.” “Previously, I've had success running FreeBSD 11.0-RELEASE on OpenStack but more could/should be done. For instance, as suggested by Allan, wouldn't be nice to deploy the latest code from FreeBSD ? Running -STABLE or even -CURRENT ? Yes, it would. Also, wouldn't it be nice to customize these images for a specific need? I'd say ‘Yes' for that as well.” “After some research I found that the current openstack.conf file, located at /usr/src/release/tools/ could use some extra tweaks to get where I wanted. I've created and attached that to a bugzilla on the same topic. You can read about that here (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213396).” Steps: Fetch the FreeBSD source code and extract it under /usr/src Once the code is in place, follow the regular process of build(7) and perform a make buildworld buildkernel Change into the release directory (/usr/src/release) and perform a make cloudware make cloudware-release WITH_CLOUDWARE=yes CLOUDWARE=OPENSTACK VMIMAGE=2G “That's it! This will generate a qcow2 image with 1.4G in size and a raw image of 2G. The entire process uses the release(7) toolchain to generate the image and should work with newer versions of FreeBSD.” + The patch has already been committed to FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=310047) Interview - Rod Grimes - rgrimes@freebsd.org (mailto:rgrimes@freebsd.org) Want to help fund the development of GPU Passthru? Visit bhyve.org (http://bhyve.org/) *** News Roundup Configuring the FreeBSD automounter (http://blog.khubla.com/freebsd/configuring-the-freebsd-automounter) Ever had to configure the FreeBSD auto-mounting daemon? Today we have a blog post that walks us through a few of the configuration knobs you have at your disposal. First up, Tom shows us his /etc/fstab file, and the various UFS partitions he has setup with the ‘noauto' flag so they are not mounted at system boot. His amd.conf file is pretty basic, with just options enabled to restart mounts, and unmount on exit. Where most users will most likely want to pay attention is in the crafting of an amd.map file Within this file, we have the various command-foo which performs mounts and unmounts of targeted disks / file-systems on demand. Pay special attention to all the special chars, since those all matter and a stray or missing ; could be a source of failure. Lastly a few knobs in rc.conf will enable the various services and a reboot should confirm the functionality. *** l2k16 hackathon report: LibreSSL manuals now in mdoc(7) (http://undeadly.org/cgi?action=article&sid=20161114174451) Hackathon report by Ingo Schwarze “Back in the spring two years ago, Kristaps Dzonsons started the pod2mdoc(1) conversion utility, and less than a month later, the LibreSSL project began. During the general summer hackathon in the same year, g2k14, Anthony Bentley started using pod2mdoc(1) for converting LibreSSL manuals to mdoc(7).” “Back then, doing so still was a pain, because pod2mdoc(1) was still full of bugs and had gaping holes in functionality. For example, Anthony was forced to basically translate the SYNOPSIS sections by hand, and to fix up .Fn and .Xr in the body by hand as well. All the same, he speedily finished all of libssl, and in the autumn of the same year, he mustered the courage to commit his work.” “Near the end of the following winter, i improved the pod2mdoc(1) tool to actually become convenient in practice and started work on libcrypto, converting about 50 out of the about 190 manuals. Max Fillinger also helped a bit, converting a handful of pages, but i fear i tarried too much checking and committing his work, so he quickly gave up on the task. After that, almost nothing happened for a full year.” “Now i was finally fed up with the messy situation and decided to put an end to it. So i went to Toulouse and finished the conversion of the remaining 130 manual pages in libcrypto, such that you can now view the documentation of all functions” Interactive Terminal Utility: smenu (https://github.com/p-gen/smenu) Ok, I've made no secret of my love for shell scripting. Well today we have a new (somewhat new to us) tool to bring your way. Have you ever needed to deal with large lists of data, perhaps as the result of a long specially crafted pipe? What if you need to select a specific value from a range and then continue processing? Enter ‘smenu' which can help make your scripting life easier. “smenu is a selection filter just like sed is an editing filter. This simple tool reads words from the standard input, presents them in a cool interactive window after the current line on the terminal and writes the selected word, if any, on the standard output. After having unsuccessfully searched the NET for what I wanted, I decided to try to write my own. I have tried hard to made its usage as simple as possible. It should work, even when using an old vt100 terminal and is UTF-8 aware.“ What this means, is in your interactive scripts, you can much easier present the user with a cursor driven menu to select from a range of possible choices. (Without needing to craft a bunch of dialog flags) Take a look, and hopefully you'll be able to find creative uses for your shell scripts in the future. *** Ubuntu still isn't free software (http://mjg59.dreamwidth.org/45939.html) “Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries. This does not affect your rights under any open source licence applicable to any of the components of Ubuntu. If you need us to approve, certify or provide modified versions for redistribution you will require a licence agreement from Canonical, for which you may be required to pay. For further information, please contact us” “Mark Shuttleworth just blogged (http://insights.ubuntu.com/2016/12/01/taking-a-stand-against-unstable-risky-unofficial-ubuntu-images/) about their stance against unofficial Ubuntu images. The assertion is that a cloud hoster is providing unofficial and modified Ubuntu images, and that these images are meaningfully different from upstream Ubuntu in terms of their functionality and security. Users are attempting to make use of these images, are finding that they don't work properly and are assuming that Ubuntu is a shoddy product. This is an entirely legitimate concern, and if Canonical are acting to reduce user confusion then they should be commended for that.” “The appropriate means to handle this kind of issue is trademark law. If someone claims that something is Ubuntu when it isn't, that's probably an infringement of the trademark and it's entirely reasonable for the trademark owner to take action to protect the value associated with their trademark. But Canonical's IP policy goes much further than that - it can be interpreted as meaning[1] that you can't distribute works based on Ubuntu without paying Canonical for the privilege, even if you call it something other than Ubuntu. [1]: And by "interpreted as meaning" I mean that's what it says and Canonical refuse to say otherwise” “If you ask a copyright holder if you can give a copy of their work to someone else (assuming it doesn't infringe trademark law), and they say no or insist you need an additional contract, it's not free software. If they insist that you recompile source code before you can give copies to someone else, it's not free software. Asking that you remove trademarks that would otherwise infringe trademark law is fine, but if you can't use their trademarks in non-infringing ways, that's still not free software.” “Canonical's IP policy continues to impose restrictions on all of these things, and therefore Ubuntu is not free software.” Beastie Bits OPNsense 16.7.10 released (https://opnsense.org/opnsense-16-7-10-released/) OpenBSD Foundation Welcomes First Iridium Donor: Smartisan (http://undeadly.org/cgi?action=article&sid=20161123193708&mode=expanded&count=8) Jan Koum donates $500,000 to FreeBSD (https://www.freebsdfoundation.org/blog/foundation-announces-new-uranium-donor/) The Soviet Russia, BSD makes you (https://en.wikipedia.org/wiki/DEMOS) Feedback/Questions Jason - Value (http://pastebin.com/gRN4Lzy8) Hamza - Shell Scripting (http://pastebin.com/GZYjRmSR) Blog link (http://aikchar.me/blog/unix-shell-programming-lessons-learned.html) Dave - Migrating to FreeBSD (http://pastebin.com/hEBu3Drp) Dan - Which BSD? (http://pastebin.com/1HpKqCSt) Zach - AMD Video (http://pastebin.com/4Aj5ebns) ***
Today on the show, we've got a look at running OpenBSD on a APU, some BSD in your Android, managing your own FreeBSD cloud service with ansible and much more. Keep it turned on your place to B...SD! This episode was brought to you by Headlines OpenBSD on PC Engines APU2 (https://github.com/elad/openbsd-apu2) A detailed walkthrough of building an OpenBSD firewall on a PC Engines APU2 It starts with a breakdown of the parts that were purchases, totally around $200 Then the reader is walked through configuring the serial console, flashing the ROM, and updating the BIOS The next step is actually creating a custom OpenBSD install image, and pre-configuring its serial console. Starting with OpenBSD 6.0, this step is done automatically by the installer Installation: Power off the APU2 Insert the bootable OpenBSD installer USB flash drive to one of the USB slots on the APU2 Power on the APU2, press F10 to get to the boot menu, and choose to boot from USB (usually option number 1) At the boot> prompt, remember the serial console settings (see above) Also at the boot> prompt, press Enter to start the installer Follow the installation instructions The driver used for wireless networking is athn(4). It might not work properly out of the box. Once OpenBSD is installed, run fw_update with no arguments. It will figure out which firmware updates are required and will download and install them. When it finishes, reboot. Where the rubber meets the road… (part one) (https://functionallyparanoid.com/2016/11/29/where-the-rubber-meets-the-road-part-one/) A user describes their adventures installing OpenBSD and Arch Linux on a new Lenovo X1 Carbon (4th gen, skylake) They also detail why they moved away from their beloved Macbook, which while long, does describe a journey away from Apple that we've heard elsewhere. The journey begins with getting a new Windows laptop, shrinking the partition and creating space for a triple-boot install, of Windows / Arch / OpenBSD Brian then details how he setup the partitioning and performed the initial Arch installation, getting it tuned to his specifications. Next up was OpenBSD though, and that went sideways initially due to a new NVMe drive that wasn't fully supported (yet) The article is split into two parts (we will bring you the next installment at a future date), but he leaves us with the plan of attack to build a custom OpenBSD kernel with corrected PCI device identifiers. We wish Brian luck, and look forward to the “rest of the story” soon. *** Howto setup a FreeBSD jail server using iocage and ansible. (https://github.com/JoergFiedler/freebsd-ansible-demo) Setting up a FreeBSD jail server can be a daunting task. However when a guide comes along which shows you how to do that, including not exposing a single (non-jailed) port to the outside world, you know we had a take a closer look. This guide comes to us from GitHub, courtesy of Joerg Fielder. The project goals seem notable: Ansible playbook that creates a FreeBSD server which hosts multiple jails. Travis is used to run/test the playbook. No service on the host is exposed externally. All external connections terminate within a jail. Roles can be reused using Ansible Galaxy. Combine any of those roles to create FreeBSD server, which perfectly suits you. To get started, you'll need a machine with Ansible, Vagrant and VirtualBox, and your credentials to AWS if you want it to automatically create / destroy EC2 instances. There's already an impressive list of Anisible roles created for you to start with: freebsd-build-server - Creates a FreeBSD poudriere build server freebsd-jail-host - FreeBSD Jail host freebsd-jailed - Provides a jail freebsd-jailed-nginx - Provides a jailed nginx server freebsd-jailed-php-fpm - Creates a php-fpm pool and a ZFS dataset which is used as web root by php-fpm freebsd-jailed-sftp - Installs a SFTP server freebsd-jailed-sshd - Provides a jailed sshd server. freebsd-jailed-syslogd - Provides a jailed syslogd freebsd-jailed-btsync - Provides a jailed btsync instance server freebsd-jailed-joomla - Installs Joomla freebsd-jailed-mariadb - Provides a jailed MariaDB server freebsd-jailed-wordpress - Provides a jailed Wordpress server. Since the machines have to be customized before starting, he mentions that cloud-init is used to do the following: activate pf firewall add a pass all keep state rule to pf to keep track of connection states, which in turn allows you to reload the pf service without losing the connection install the following packages: sudo bash python27 allow passwordless sudo for user ec2-user “ From there it is pretty straight-forward, just a couple commands to spin up the VM's either locally on your VirtualBox host, or in the cloud with AWS. Internally the VM's are auto-configured with iocage to create jails, where all your actual services run. A neat project, check it out today if you want a shake-n-bake type cloud + jail solution. Colin Percival's bsdiff helps reduce Android apk bandwidth usage by 6 petabytes per day (http://android-developers.blogspot.ca/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html) A post on the official Android-Developers blog, talks about how they used bsdiff (and bspatch) to reduce the size of Android application updates by 65% bsdiff was developed by FreeBSD's Colin Percival Earlier this year, we announced that we started using the bsdiff algorithm (by Colin Percival). Using bsdiff, we were able to reduce the size of app updates on average by 47% compared to the full APK size. This post is actually about the second generation of the code. Today, we're excited to share a new approach that goes further — File-by-File patching. App Updates using File-by-File patching are, on average, 65% smaller than the full app, and in some cases more than 90% smaller. Android apps are packaged as APKs, which are ZIP files with special conventions. Most of the content within the ZIP files (and APKs) is compressed using a technology called Deflate. Deflate is really good at compressing data but it has a drawback: it makes identifying changes in the original (uncompressed) content really hard. Even a tiny change to the original content (like changing one word in a book) can make the compressed output of deflate look completely different. Describing the differences between the original content is easy, but describing the differences between the compressed content is so hard that it leads to inefficient patches. So in the second generation of the code, they use bsdiff on each individual file, then package that, rather than diffing the original and new archives bsdiff is used in a great many other places, including shrinking the updates for the Firefox and Chrome browsers You can find out more about bsdiff here: http://www.daemonology.net/bsdiff/ A far more sophisticated algorithm, which typically provides roughly 20% smaller patches, is described in my doctoral thesis (http://www.daemonology.net/papers/thesis.pdf). Considering the gains, it is interesting that no one has implemented Colin's more sophisticated algorithm Colin had an interesting observation (https://twitter.com/cperciva/status/806426180379230208) last night: “I just realized that bandwidth savings due to bsdiff are now roughly equal to what the total internet traffic was when I wrote it in 2003.” *** News Roundup Distrowatch does an in-depth review of NAS4Free (https://distrowatch.com/weekly.php?issue=20161114#nas4free) Jesse Smith over at DistroWatch has done a pretty in-depth review of Nas4Free. The review starts with mentioning that NAS4Free works on 3 platforms, ARM/i386/AMD64 and for the purposes of this review he would be using AMD64 builds. After going through the initial install (doing typical disk management operations, such as GPT/MBR, etc) he was ready to begin using the product. One concern originally observed was that the initial boot seemed rather slow. Investigation revealed this was due to it loading the entire OS image into memory, and the first (long) disk read did take some time, but once loaded was super responsive. The next steps involved doing the initial configuration, which meant creating a new ZFS storage pool. After this process was done, he did find one puzzling UI option called “VM” which indicated it can be linked to VirtualBox in some way, but the Docs didn't reveal its secrets of usage. Additionally covered were some of the various “Access” methods, including traditional UNIX permissions, AD and LDAP, and then various Sharing services which are typical to a NAS, Such as NFS / Samba and others. One neat feature was the built-in file-browser via the web-interface, which allows you another method of getting at your data when sometimes NFS / Samba or WebDav aren't enough. Jesse gives us a nice round-up conclusion as well Most of the NAS operating systems I have used in the past were built around useful features. Some focused on making storage easy to set up and manage, others focused on services, such as making files available over multiple protocols or managing torrents. Some strive to be very easy to set up. NAS4Free does pretty well in each of the above categories. It may not be the easiest platform to set up, but it's probably a close second. It may not have the prettiest interface for managing settings, but it is quite easy to navigate. NAS4Free may not have the most add-on services and access protocols, but I suspect there are more than enough of both for most people. Where NAS4Free does better than most other solutions I have looked at is security. I don't think the project's website or documentation particularly focuses on security as a feature, but there are plenty of little security features that I liked. NAS4Free makes it very easy to lock the text console, which is good because we do not all keep our NAS boxes behind locked doors. The system is fairly easy to upgrade and appears to publish regular security updates in the form of new firmware. NAS4Free makes it fairly easy to set up user accounts, handle permissions and manage home directories. It's also pretty straight forward to switch from HTTP to HTTPS and to block people not on the local network from accessing the NAS's web interface. All in all, I like NAS4Free. It's a good, general purpose NAS operating system. While I did not feel the project did anything really amazing in any one category, nor did I run into any serious issues. The NAS ran as expected, was fairly straight forward to set up and easy to manage. This strikes me as an especially good platform for home or small business users who want an easy set up, some basic security and a solid collection of features. Browsix: Unix in the browser tab (https://browsix.org/) Browsix is a research project from the PLASMA lab at the University of Massachusetts, Amherst. The goal: Run C, C++, Go and Node.js programs as processes in browsers, including LaTeX, GNU Make, Go HTTP servers, and POSIX shell scripts. “Processes are built on top of Web Workers, letting applications run in parallel and spawn subprocesses. System calls include fork, spawn, exec, and wait.” Pipes are supported with pipe(2) enabling developers to compose processes into pipelines. Sockets include support for TCP socket servers and clients, making it possible to run applications like databases and HTTP servers together with their clients in the browser. Browsix comprises two core parts: A kernel written in TypeScript that makes core Unix features (including pipes, concurrent processes, signals, sockets, and a shared file system) available to web applications. Extended JavaScript runtimes for C, C++, Go, and Node.js that support running programs written in these languages as processes in the browser. This seems like an interesting project, although I am not sure how it would be used as more than a toy *** Book Review: PAM Mastery (https://www.cyberciti.biz/reviews/book-review-pam-mastery/) nixCraft does a book review of Michael W. Lucas' “Pam Mastery” Linux, FreeBSD, and Unix-like systems are multi-user and need some way of authenticating individual users. Back in the old days, this was done in different ways. You need to change each Unix application to use different authentication scheme. Before PAM, if you wanted to use an SQL database to authenticate users, you had to write specific support for that into each of your applications. Same for LDAP, etc. So Open Group lead to the development of PAM for the Unix-like system. Today Linux, FreeBSD, MacOS X and many other Unix-like systems are configured to use a centralized authentication mechanism called Pluggable Authentication Modules (PAM). The book “PAM Mastery” deals with the black magic of PAM. Of course, each OS chose to implement PAM a little bit differently The book starts with the basic concepts about PAM and authentication. You learn about Multi-Factor Authentication and why use PAM instead of changing each program to authenticate the user. The author went into great details about why PAM is useful for developers and sysadmin for several reasons. The examples cover CentOS Linux (RHEL and clones), Debian Linux, and FreeBSD Unix system. I like the way the author described PAM Configuration Files and Common Modules that covers everyday scenarios for the sysadmin. PAM configuration file format and PAM Module Interfaces are discussed in easy to understand language. Control flags in PAM can be very confusing for new sysadmins. Modules can be stacked in a particular order, and the control flags determine how important the success or failure of a particular module. There is also a chapter about using one-time passwords (Google Authenticator) for your application. The final chapter is all about enforcing good password policies for users and apps using PAM. The sysadmin would find this book useful as it covers a common authentication scheme that can be used with a wide variety of applications on Unix. You will master PAM topics and take control over authentication for your organization IT infrastructure. If you are Linux or Unix sysadmin, I would highly recommend this book. Once again Michael W Lucas nailed it. The only book you may need for PAM deployment. get “PAM Mastery” (https://www.michaelwlucas.com/tools/pam) *** Reflections on Trusting Trust - Ken Thompson, co-author of UNIX (http://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html) Ken Thompson's "cc hack" - Presented in the journal, Communication of the ACM, Vol. 27, No. 8, August 1984, in a paper entitled "Reflections on Trusting Trust", Ken Thompson, co-author of UNIX, recounted a story of how he created a version of the C compiler that, when presented with the source code for the "login" program, would automatically compile in a backdoor to allow him entry to the system. This is only half the story, though. In order to hide this trojan horse, Ken also added to this version of "cc" the ability to recognize if it was recompiling itself to make sure that the newly compiled C compiler contained both the "login" backdoor, and the code to insert both trojans into a newly compiled C compiler. In this way, the source code for the C compiler would never show that these trojans existed. The article starts off by talking about a content to write a program that produces its own source code as output. Or rather, a C program, that writes a C program, that produces its own source code as output. The C compiler is written in C. What I am about to describe is one of many "chicken and egg" problems that arise when compilers are written in their own language. In this case, I will use a specific example from the C compiler. Suppose we wish to alter the C compiler to include the sequence "v" to represent the vertical tab character. The extension to Figure 2 is obvious and is presented in Figure 3. We then recompile the C compiler, but we get a diagnostic. Obviously, since the binary version of the compiler does not know about "v," the source is not legal C. We must "train" the compiler. After it "knows" what "v" means, then our new change will become legal C. We look up on an ASCII chart that a vertical tab is decimal 11. We alter our source to look like Figure 4. Now the old compiler accepts the new source. We install the resulting binary as the new official C compiler and now we can write the portable version the way we had it in Figure 3. The actual bug I planted in the compiler would match code in the UNIX "login" command. The replacement code would miscompile the login command so that it would accept either the intended encrypted password or a particular known password. Thus if this code were installed in binary and the binary were used to compile the login command, I could log into that system as any user. Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions. Next “simply add a second Trojan horse to the one that already exists. The second pattern is aimed at the C compiler. The replacement code is a Stage I self-reproducing program that inserts both Trojan horses into the compiler. This requires a learning phase as in the Stage II example. First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere. So now there is a trojan'd version of cc. If you compile a clean version of cc, using the bad cc, you will get a bad cc. If you use the bad cc to compile the login program, it will have a backdoor. The source code for both backdoors no longer exists on the system. You can audit the source code of cc and login all you want, they are trustworthy. The compiler you use to compile your new compiler, is the untrustworthy bit, but you have no way to know it is untrustworthy, and no way to make a new compiler, without using the bad compiler. The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect. Acknowledgment: I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics. I can- not find a more specific reference to this document. I would appreciate it if anyone who can supply this reference would let me know. Beastie Bits Custom made Beastie Stockings (https://www.etsy.com/listing/496638945/freebsd-beastie-christmas-stocking) Migrating ZFS from mirrored pool to raidz1 pool (http://ximalas.info/2016/12/06/migrating-zfs-from-mirrored-pool-to-raidz1-pool/) OpenBSD and you (https://home.nuug.no/~peter/blug2016/) Watson.org FreeBSD and Linux cross reference (http://fxr.watson.org/) OpenGrok (http://bxr.su/) FreeBSD SA-16:37: libc (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:37.libc.asc) -- A 26+ year old bug found in BSD's libc, all BSDs likely affected -- A specially crafted argument can trigger a static buffer overflow in the library, with possibility to rewrite following static buffers that belong to other library functions. HardenedBSD issues correction for libc patch (https://github.com/HardenedBSD/hardenedBSD/commit/fb823297fbced336b6beeeb624e2dc65b67aa0eb) -- original patch improperly calculates how many bytes are remaining in the buffer. From December the 27th until the 30th there the 33rd Chaos Communication Congress[0] is going to take place in Hamburg, Germany. Think of it as the yearly gathering of the european hackerscene and their overseas friends. I am one of the persons organizing the "BSD assembly (https://events.ccc.de/congress/2016/wiki/Assembly:BSD)" as a gathering place for BSD enthusiasts and waving the flag amidst the all the other projects / communities. Feedback/Questions Chris - IPFW + Wifi (http://pastebin.com/WRiuW6nn) Jason - bhyve pci (http://pastebin.com/JgerqZZP) Al - pf errors (http://pastebin.com/3XY5MVca) Zach - Xorg settings (http://pastebin.com/Kty0qYXM) Bart - Wireless Support (http://pastebin.com/m3D81GBW) ***
This week on the show, we've got some new info on the talks from EuroBSDCon, a look at sharing a single ZFS pool between Linux and BSD, Sandboxing and much more! Stay tuned for your place to B...SD! This episode was brought to you by Headlines EuroBSDcon 2016 Presentation Slides (https://2016.eurobsdcon.org/PresentationSlides/) Due to circumstances beyond the control of the organizers of EuroBSDCon, there were not recordings of the talks given at the event. However, they have collected the slide decks from each of the speakers and assembled them on this page for you Also, we have some stuff from MeetBSD already: Youtube Playlist (https://www.youtube.com/playlist?list=PLb87fdKUIo8TAMC2HJLZ7H54edD2BeGWv) Not all of the sessions are posted yet, but the rest should appear shortly MeetBSD 2016 Trip Report: Domagoj Stolfa (https://www.freebsdfoundation.org/blog/meetbsd-2016-trip-report-domagoj-stolfa/) *** Cohabiting FreeBSD and Gentoo Linux on a Common ZFS Volume (https://ericmccorkleblog.wordpress.com/2016/11/15/cohabiting-freebsd-and-gentoo-linux-on-a-common-zfs-volume/) Eric McCorkle, who has contributed ZFS support to the FreeBSD EFI boot-loader code has posted an in-depth look at how he's setup dual-boot with FreeBSD and Gentoo on the same ZFS volume. He starts by giving us some background on how the layout is done. First up, GRUB is used as the boot-loader, allowing boot of both Linux and BSD The next non-typical thing was using /etc/fstab to manage mount-points, instead of the typical ‘zfs mount' usage, (apart from /home datasets) data/home is mounted to /home, with all of its child datasets using the ZFS mountpoint system data/freebsd and its child datasets house the FreeBSD system, and all have their mountpoints set to legacy data/gentoo and its child datasets house the Gentoo system, and have their mountpoints set to legacy as well So, how did he set this up? He helpfully provides an overview of the steps: Use the FreeBSD installer to create the GPT and ZFS pool Install and configure FreeBSD, with the native FreeBSD boot loader Boot into FreeBSD, create the Gentoo Linux datasets, install GRUB Boot into the Gentoo Linux installer, install Gentoo Boot into Gentoo, finish any configuration tasks The rest of the article walks us through the individual commands that make up each of those steps, as well as how to craft a GRUB config file capable of booting both systems. Personally, since we are using EFI, I would have installed rEFInd, and chain-loaded each systems EFI boot code from there, allowing the use of the BSD loader, but to each their own! HardenedBSD introduces Safestack into base (https://hardenedbsd.org/article/shawn-webb/2016-11-27/introducing-safestack) HardenedBSD has integrated SafeStack into its base system and ports tree SafeStack (http://clang.llvm.org/docs/SafeStack.html) is part of the Code Pointer Integrity (CPI) project within clang. “SafeStack is an instrumentation pass that protects programs against attacks based on stack buffer overflows, without introducing any measurable performance overhead. It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack.” “As of 28 November 2016, with clang 3.9.0, SafeStack only supports being applied to applications and not shared libraries. Multiple patches have been submitted to clang by third parties to add support for shared libraries.” SafeStack is only enabled on AMD64 *** pledge(2)… or, how I learned to love web application sandboxing (https://learnbchs.org/pledge.html) We've talked about OpenBSD's sandboxing mechanism pledge() in the past, but today we have a great article by Kristaps Dzonsons, about how he grew to love it for Web Sandboxing. +First up, he gives us his opening argument that should make most of you sit up and listen: I use application-level sandboxing a lot because I make mistakes a lot; and when writing web applications, the price of making mistakes is very dear. In the early 2000s, that meant using systrace(4) on OpenBSD and NetBSD. Then it was seccomp(2) (followed by libseccomp(3)) on Linux. Then there was capsicum(4) on FreeBSD and sandbox_init(3) on Mac OS X. All of these systems are invoked differently; and for the most part, whenever it came time to interface with one of them, I longed for sweet release from the nightmare. Please, try reading seccomp(2). To the end. Aligning web application logic and security policy would require an arduous (and usually trial-and-error or worse, copy-and-paste) process. If there was any process at all — if the burden of writing a policy didn't cause me to abandon sandboxing at the start. And then there was pledge(2). This document is about pledge(2) and why you should use it and love it. “ +Not convinced yet? Maybe you should take his challenge: Let's play a drinking game. The challenge is to stay out of the hospital. 1.Navigate to seccomp(2). 2. Read it to the end. 3. Drink every time you don't understand. For capsicum(4), the challenge is no less difficult. To see these in action, navigate no further than OpenSSH, which interfaces with these sandboxes: sandbox-seccomp-filter.c or sandbox-capsicum.c. (For a history lesson, you can even see sandbox-systrace.c.) Keep in mind that these do little more than restrict resources to open descriptors and the usual necessities of memory, signals, timing, etc. Keep that in mind and be horrified. “ Now Kristaps has his theory on why these are so difficult (NS..), but perhaps there is a better way. He makes the case that pledge() sits right in that sweet-spot, being powerful enough to be useful, but easy enough to implement that developers might actually use it. All in all, a nice read, check it out! Would love to hear other developer success stories using pledge() as well. *** News Roundup Unix history repository, now on GitHub (http://www.osnews.com/story/29513/Unix_history_repository_now_on_GitHub) OS News has an interesting tidbit on their site today, about the entire commit history of Unix now being available online, starting all the way back in 1970 and bringing us forward to today. From the README The history and evolution of the Unix operating system is made available as a revision management repository, covering the period from its inception in 1970 as a 2.5 thousand line kernel and 26 commands, to 2016 as a widely-used 27 million line system. The 1.1GB repository contains about half a million commits and more than two thousand merges. The repository employs Git system for its storage and is hosted on GitHub. It has been created by synthesizing with custom software 24 snapshots of systems developed at Bell Labs, the University of California at Berkeley, and the 386BSD team, two legacy repositories, and the modern repository of the open source FreeBSD system. In total, about one thousand individual contributors are identified, the early ones through primary research. The data set can be used for empirical research in software engineering, information systems, and software archaeology. This is a fascinating find, especially will be of value to students and historians who wish to look back in time to see how UNIX evolved, and in this repo ultimately turned into modern FreeBSD. *** Yandex commits improvements to FreeBSD network stack (https://reviews.freebsd.org/D8526) “Rework ip_tryforward() to use FIB4 KPI.” This commit brings some code from the experimental routing branch into head As you can see from the graphs, it offers some sizable improvements in forwarding and firewalled packets per second commit (https://svnweb.freebsd.org/base?view=revision&revision=309257) *** The brief history of Unix socket multiplexing – select(2) system call (https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/) Ever wondered about the details of socket multiplexing, aka the history of select(2)? Well Marek today gives a treat, with a quick look back at the history that made today's modern multiplexing possible. First, his article starts the way all good ones do, presenting the problem in silent-movie form: In mid-1960's time sharing was still a recent invention. Compared to a previous paradigm - batch-processing - time sharing was truly revolutionary. It greatly reduced the time wasted between writing a program and getting its result. Batch-processing meant hours and hours of waiting often to only see a program error. See this film to better understand the problems of 1960's programmers: "The trials and tribulations of batch processing". Enter the wild world of the 1970's, and we've now reached the birth of UNIX which tried to solve the batch processing problem with time-sharing. These days when a program was executed, it could "stall" (block) only on a couple of things1: + wait for CPU + wait for disk I/O + wait for user input (waiting for a shell command) or console (printing data too fast)“ Jump forward another dozen years or so, and the world changes yet again: This all changed in 1983 with the release of 4.2BSD. This revision introduced an early implementation of a TCP/IP stack and most importantly - the BSD Sockets API.Although today we take the BSD sockets API for granted, it wasn't obvious it was the right API. STREAMS were a competing API design on System V Revision 3. Coming in along with the sockets API was the select(2) call, which our very own Kirk McKusick gives us some background on: Select was introduced to allow applications to multiplex their I/O. Consider a simple application like a remote login. It has descriptors for reading from and writing to the terminal and a descriptor for the (bidirectional) socket. It needs to read from the terminal keyboard and write those characters to the socket. It also needs to read from the socket and write to the terminal. Reading from a descriptor that has nothing queued causes the application to block until data arrives. The application does not know whether to read from the terminal or the socket and if it guesses wrong will incorrectly block. So select was added to let it find out which descriptor had data ready to read. If neither, select blocks until data arrives on one descriptor and then awakens telling which descriptor has data to read. [...] Non-blocking was added at the same time as select. But using non-blocking when reading descriptors does not work well. Do you go into an infinite loop trying to read each of your input descriptors? If not, do you pause after each pass and if so for how long to remain responsive to input? Select is just far more efficient. Select also lets you create a single inetd daemon rather than having to have a separate daemon for every service. The article then wraps up with an interesting conclusion: > CSP = Communicating sequential processes In this discussion I was afraid to phrase the core question. Were Unix processes intended to be CSP-style processes? Are file descriptors a CSP-derived "channels"? Is select equivalent to ALT statement? I think: no. Even if there are design similarities, they are accidental. The file-descriptor abstractions were developed well before the original CSP paper. It seems that an operating socket API's evolved totally disconnected from the userspace CSP-alike programming paradigms. It's a pity though. It would be interesting to see an operating system coherent with the programming paradigms of the user land programs. A long (but good) read, and worth your time if you are interested in the history how modern multiplexing came to be. *** How to start CLion on FreeBSD? (https://intellij-support.jetbrains.com/hc/en-us/articles/206525024-How-to-start-CLion-on-FreeBSD) CLion (pronounced "sea lion") is a cross-platform C and C++ IDE By default, the Linux version comes bundled with some binaries, which obviously won't work with the native FreeBSD build Rather than using Linux emulation, you can replace these components with native versions pkg install openjdk8 cmake gdb Edit clion-2016.3/bin/idea.properties and change run.processes.with.pty=false Start CLion and open Settings | Build, Execution, Deployment | Toolchains Specify CMake path: /usr/local/bin/cmake and GDB path: /usr/local/bin/gdb Without a replacement for fsnotifier, you will get a warning that the IDE may be slow to detect changes to files on disk But, someone has already written a version of fsnotifier that works on FreeBSD and OpenBSD fsnotifier for OpenBSD and FreeBSD (https://github.com/idea4bsd/fsnotifier) -- The fsnotifier is used by IntelliJ for detecting file changes. This version supports FreeBSD and OpenBSD via libinotify and is a replacement for the bundled Linux-only version coming with the IntelliJ IDEA Community Edition. *** Beastie Bits TrueOS Pico – FreeBSD ARM/RPi Thin Clients (https://www.trueos.org/trueos-pico/) A Puppet package provider for FreeBSD's PkgNG package manager. (https://github.com/xaque208/puppet-pkgng) Notes from November London *BSD meetup (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2016-November/014059.html) SemiBug meeting on Dec 20th (http://lists.nycbug.org/pipermail/semibug/2016-November/000131.html) Feedback/Questions Erno - SSH without password (http://pastebin.com/SMvxur9v) Jonathan - Magical ZFS (http://pastebin.com/5ETL7nmj) George - TrueOS (http://pastebin.com/tSVvaV9e) Mohammad - Jails IP (http://pastebin.com/T8nUexd1) Gibheer - BEs (http://pastebin.com/YssXXp70) ***
On today's episode, we are loaded and ready to go. Lots of OpenBSD news, a look at LetsEncrypt usage, the NetBSD scheduler (oh my) and much more. Keep it tuned to your place to B...SD! This episode was brought to you by Headlines Production ready (http://www.tedunangst.com/flak/post/production-ready) Ted Unangst brings us a piece on what it means to be Production Ready He tells the story of a project he worked on that picked a framework that was “production ready” They tested time zones, and it all seemed to work They tested the unicode support in english and various european languages, and it was all good They sent some emails with it, and it just worked The framework said “Production Ready” on the tin, and it passed all the tests. What is the worst that could happen? Now, we built our product on top of this. Some of the bugs were caught internally. Others were discovered by customers, who were of course a little dismayed. Like, how could you possibly ship this? Indeed. We were doing testing, quite a bit really, but when every possible edge case has a bug, it's hard to find them all. A customer from Arizona, which does not observe Daylight Saving Time, crashed the app Some less common unicode characters caused a buffer overflow The email system did not properly escape a period on its own line, truncating the email “Egregious performance because of a naive N^2 algorithm for growing a buffer.” “Egregious performance on some platforms due to using the wrong threading primitives.” “Bizarre database connection bugs for some queries that I can't at all explain.” “In short, everything was “works for me” quality. But is that really production quality?” “There are some obvious contenders for the title of today's most “production ready” software, but it's a more general phenomenon. People who have success don't know what they don't know, what they didn't test, what unused features will crash and burn.” Using Let's Encrypt within FreeBSD.org (https://blog.crashed.org/letsencrypt-in-freebsd-org/) I decided to give Let's Encrypt certificates a shot on my personal web servers earlier this year after a disaster with StartSSL. I'd like to share what I've learned. The biggest gotcha is that people tend to develop bad habits when they only have to deal with certificates once a year or so. The beginning part of the process is manual and the deployment of certificates somehow never quite gets automated, or things get left out. That all changes with Let's Encrypt certificates. Instead of 1-5 year lifetime certificates the Let's Encrypt certificates are only valid for 90 days. Most people will be wanting to renew every 60-80 days. This forces the issue - you really need to automate and make it robust. The Let's Encrypt folks provide tools to do this for you for the common cases. You run it on the actual machine, it manages the certificates and adjusts the server configuration files for you. Their goal is to provide a baseline shake-n-bake solution. I was not willing to give that level of control to a third party tool for my own servers - and it was absolutely out of the question for for the FreeBSD.org cluster. I should probably mention that we do things on the FreeBSD.org cluster that many people would find a bit strange. The biggest problem that we have to deal with is that the traditional model of a firewall/bastion between "us" and "them" does not apply. We design for the assumption that hostile users are already on the "inside" of the network. The cluster is spread over 8 distinct sites with naked internet and no vpn between them. There is actually very little trust between the systems in this network - eg: ssh is for people only - no headless users can ssh. There are no passwords. Sudo can't be used. The command and control systems use signing. We don't trust anything by IPv4/IPv6 address because we have to assume MITM is a thing. And so on. In general, things are constructed to be trigger / polling / pull based. The downside is that this makes automation and integration of Let's Encrypt clients interesting. If server configuration files can't be modified; and replicated web infrastructure is literally read-only (via jails/nullfs); and DNS zone files are static; and headless users can't ssh and therefore cannot do commits, how do you do the verification tokens in an automated fashion? Interesting, indeed. We wanted to be able to use certificates on things like ldap and smtp servers. You can't do http file verification on those so we had to use dns validation of domains. First, a signing request is generated, and the acme-challenge is returned Peter's post then walks through how the script adds the required TXT record to prove control of the domain, regenerates the zone file, DNSSEC signs it, and waits for it to be published, then continues the letsencrypt process. Letsencrypt then issues the actual certificate We export the fullchain files into a publication location. There is another jail that can read the fullchain certificates via nullfs and they are published with our non-secrets update mechanism Since we are using DNSSEC, here is a good opportunity to maintain signed TLSA fingerprints. The catch with TLSA record updates is managing the update event horizon. You are supposed to have both fingerprints listed across the update cycle. We use 'TLSA 3 1 1' records to avoid issues with propagation delays for now. TLSA 3 0 1 changes with every renewal, while 3 1 1 only changes when you generate a new private key. The majority of TLS/SSL servers require a full restart to re-load the certificates if the filename is unchanged. I found out the hard way. There is a great deal more detail in the blog post, I recommend you check it out Learning more about the NetBSD scheduler (... than I wanted to know) Part 1 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161105_1754.html) Part 2 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161109_0059.html) Part 3 (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20161113_0122.html) Today I had a need to do some number crunching using a home-brewn C program. In order to do some manual load balancing, I was firing up some Amazon AWS instances (which is Xen) with NetBSD 7.0. In this case, the system was assigned two CPUs I started two instances of my program, with the intent to have each one use one CPU. Which is not what happened! Here is what I observed, and how I fixed things for now. ~~ load averages: 2.14, 2.08, 1.83; up 0+00:45:56 18:01:32 27 processes: 4 runnable, 21 sleeping, 2 on CPU CPU0 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Memory: 119M Act, 7940K Exec, 101M File, 3546M Free ~~ ~~ PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 2791 root 25 0 8816K 964K RUN/0 16:10 54.20% 54.20% myprog 2845 root 26 0 8816K 964K RUN/0 17:10 47.90% 47.90% myprog ~~ I expected something like WCPU and CPU being around 100%, assuming that each process was bound to its own CPU. The values I actually saw (and listed above) suggested that both programs were fighting for the same CPU. Huh?! NetBSD allows to create "processor sets", assign CPU(s) to them and then assign processes to the processor sets. Let's have a look! ~~ # psrset -c 1 # psrset -b 0 2791 # psrset -b 1 2845 load averages: 2.02, 2.05, 1.94; up 0+00:59:32 18:15:08 27 processes: 1 runnable, 24 sleeping, 2 on CPU CPU0 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle CPU1 states: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle Memory: 119M Act, 7940K Exec, 101M File, 3546M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 2845 root 25 0 8816K 964K CPU/1 26:14 100% 100% myprog 2791 root 25 0 8816K 964K RUN/0 25:40 100% 100% myprog ~~ Things are as expected now, with each program being bound to its own CPU. Now why this didn't happen by default is left as an exercise to the reader. I had another look at this today, and was able to reproduce the behaviour using VMWare Fusion with two CPU cores on both NetBSD 7.0_STABLE as well as -current The one hint that I got so far was from Michael van Elst that there may be a rouding error in sched_balance(). Looking at the code, there is not much room for a rounding error. But I am not familiar enough (at all) with the code, so I cannot judge if crucial bits are dropped here, or how that function fits in the whole puzzle. Pondering on the "rounding error", I've setup both VMs with 4 CPUs, and the behaviour shown there is that load is distributed to about 3 and a half CPU - three CPUs under full load, and one not reaching 100%. There's definitely something fishy in there. With multiple CPUs, each CPU has a queue of processes that are either "on the CPU" (running) or waiting to be serviced (run) on that CPU. Those processes count as "migratable" in runqueue_t. Every now and then, the system checks all its run queues to see if a CPU is idle, and can thus "steal" (migrate) processes from a busy CPU. This is done in sched_balance(). Such "stealing" (migration) has the positive effect that the process doesn't have to wait for getting serviced on the CPU it's currently waiting on. On the other side, migrating the process has effects on CPU's data and instruction caches, so switching CPUs shouldn't be taken too easy. All in all, I'd say the patch is a good step forward from the current situation, which does not properly distribute pure CPU hogs, at all. Building Cost-Effective 100-Gbps Firewalls for HPC with FreeBSD (https://www.nas.nasa.gov/SC16/demos/demo9.html) The continuous growth of the NASA Center for Climate Simulation (NCCS) requires providing high-performance security tools and enhancing the network capacity. In order to support the requirements of emerging services, including the Advanced Data Analytics Platform (ADAPT) private cloud, the NCCS security team has proposed an architecture to provide extremely cost-effective 100-gigabit-per-second (Gbps) firewalls. The aim of this project is to create a commodity-based platform that can process enough packets per second (pps) to sustain a 100-Gbps workload within the NCCS computational environment. The test domain consists of several existing systems within the NCCS, including switches (Dell S4084), routers (Dell R530s), servers (Dell R420s, and C6100s), and host card adapters (10-Gbps Mellanox ConnectX2 and Intel 8259 x Ethernet cards). Previous NCCS work testing the FreeBSD operating system for high-performance routing reached a maximum of 4 million pps. Building on this work, we are comparing FreeBSD-11.0 and FreeBSD-Current along with implementing the netmap-fwd Application Programming Interface (API) and tuning the 10-gigabit Ethernet cards. We used the tools iperf3, nuttcp, and netperf to monitor the performance of the maximum bandwidth through the cards. Additional testing has involved enabling the Common Address Redundancy Protocol (CARP) to achieve an active/active architecture. The tests have shown that at the optimally tuned and configured FreeBSD system, it is possible to create a system that can manage the huge amounts of pps needed to create a 100-Gbps firewall with commodity components. Some interesting findings: FreeBSD was able to send more pps as a client than Centos 6. Netmap-fwd increased the pps rate significantly. The choice of network card can have a significant impact on pps, tuning, and netmap support. Further tests will continue verifying the above results with even more capable systems-such as 40-gigabit and 100-gigabit Ethernet cards-to achieve even higher performance. In addition to hardware improvements, updates to the network capabilities in the FreeBSD-Current version will be closely monitored and applied as appropriate. The final result will be a reference architecture with representative hardware and software that will enable the NCCS to build, deploy, and efficiently maintain extremely cost-effective 100-Gbps firewalls. Netflix has already managed to saturate a 100 Gbps interface using only a single CPU Socket (rather than a dual socket server). Forwarding/routing is a bit different, but it is definitely on track to get there. Using a small number of commodity servers to firewall 100 Gbps of traffic just takes some careful planning and load balancing. Soon it will be possible using a single host. News Roundup iocell - A FreeBSD jail manager. (https://github.com/bartekrutkowski/iocell) Another jail manager has arrived on the scene, iocell, which begins life as a fork of the “classic” iocage. Due to its shared heritage, it offers much of the same functionality and flags as iocage users will be familiar with. For those who aren't up to speed with either products, some of those features include: Templates, clones, basejails, fully independent jails Ease of use Zero configuration files Rapid thin provisioning within seconds Automatic package installation Virtual networking stacks (vnet) Shared IP based jails (non vnet) Resource limits (CPU, MEMORY, DISK I/O, etc.) Filesystem quotas and reservations Dedicated ZFS datasets inside jails Transparent ZFS snapshot management Binary updates Differential jail packaging Export and import And many more! The program makes extensive use of ZFS for performing jail operations, so a zpool will be required (But doesn't have to be your boot-pool) It still looks “very” fresh, even using original iocage filenames in the repo, so a safe guess is that you'll be able to switch between iocage and iocell with relative ease. Fail2ban on OpenBSD 6.0 (http://blog.gordonturner.ca/2016/11/20/fail2ban-on-openbsd-6-0/) We've used Fail2Ban in PC-BSD before, due to it's ability to detect and block brute force attempts against a variety of services, including SSH, mail, and others. It even can work to detect jail brute force attempts, blocking IPs on the hosts firewall. However what about OpenBSD users? Well, Gordon Turner comes to the rescue today with a great writeup on deploying Fail2Ban specifically for that platform. Now, Fail2Ban is a python program, so you'll need to pkg install Python first, then he provides instructions on how to manually grab the F2B sources and install on OpenBSD. Helpfully Gordon gives us some handy links to scripts and modifications to get F2B running via RC as well, which is a bit different since F2B has both a server and client that must run together. With the installation bits out of the way, we get to next hit the “fun” stuff, which comes in the way of SSH brute force detection. Naturally we will be configuring F2B to use “pf” to do our actual blocking, but the examples shown give us full control over the knobs used to detect, and then ultimately call ‘pfctl' to do our heavy lifting. The last bits of the article give us a runthrough on how to “prime” pf with the correct block tables and performing basic administrative tasks to control F2B in production. A great article, and if you run an OpenBSD box exposed to the internet, you may want to bookmark this one. openbsd changes of note (http://www.tedunangst.com/flak/post/openbsd-changes-of-note) Continuing with our OpenBSD news for the week, we have a new blog post by TedU, which gives us a bunch of notes on the things which have changed over there as of late: Some of the notables include: mcl2k2 pools and the em conversion. The details are in the commits, but the short story is that due to hardware limitations, a number of tradeoffs need to be made between performance and memory usage. The em chip can (mostly) only be programmed to write to 2k buffers. However, ethernet payloads are not nicely aligned. They're two bytes off. Leading to a costly choice. Provide a 2k buffer, and then copy all the data after the fact, which is slow. Or allocate a larger than 2k buffer, and provide em with a pointer that's 2 bytes offset. Previously, the next size up from 2k was 4k, which is quite wasteful. The new 2k2 buffer size still wastes a bit of memory, but much less. FreeType 2.7 is prettier than ever. vmm for i386. Improve security. vmm is still running with a phenomenal set of privileges, but perhaps some cross-VM attacks may be limited. On the other side of the world, hyperv support is getting better. Remove setlocale. setlocale was sprinkled all throughout the code base many years ago, even though it did nothing, in anticipation of a day when it would do something. We've since decided that day will never come, and so many setlocale calls can go. syspatch is coming. Lots of commits actually. Despite the name, it's more like a system update, since it replaces entire binaries. Then again, replacing a few binaries in a system is like patching small parts of the whole. A syspatch update will be smaller than an entire release. There's a new build system. It kind of works like before, but a lot of the details have changed to support less root. Actually, it'd be accurate to say the whole build privilege system has been flipped. Start as root, which drops down to the build user to do the heavy lifting, instead of starting as a user that can elevate to root at any time. This no longer requires the build user to be pseudo-root; in fact, the goal is that the build user can't elevate. There's several other items on this list, take a look for more details, and he also helpfully provides commit-links if you want to see more about any of these topics. It came from Bell Labs (http://media.bemyapp.com/came-bell-labs/#) A little late for a halloween episode, we have “It came from Bell Labs”, a fascinating article talking about the successor to UNIX, Plan9 There was once an operating system that was intended to be the successor to Unix. Plan 9 From Bell Labs was its name, and playing with it for five minutes is like visiting an alternate dimension where computers are done differently. It was so ahead of its time that it would be considered cutting edge, even today. Find out the weird and woolly history to Plan Nine's inception and eventual consignment as a footnote of operating systems today. So, if you've never heard of Plan 9, how did it exactly differ from the UNIX we know and love today? Here's just a few of the key features under Plan 9's hood + 9P – The distributed file system protocol. Everything runs through this, there is no escaping it. Since everything runs on top of 9P, that makes everything running on a Plan 9 box distributed as well. This means, for example, you can import /dev/audio from another machine on the network to use its sound card when your own machine doesn't have one. + ndb – The namespace server. In conjunction with 9P, it bosses all the programs around and forces them to comply to the Plan 9 way. + Instead of Unix sockets, all the networking just runs through 9P. Thus, everything from ethernet packets to network cards are all just one more kind of file. + While Unicode is implemented ad-hoc in other systems, it's baked into Plan 9 from the first int main(). In fact, even users who don't like Plan 9 have to admit that the character encoding support, together with the beautiful built-in rio font, makes every other operating system look primitive. + The system's own internal programs are built to be a rounded set of user tools from the ground up. So, for instance, it comes with its own editor, acme, built to be its own weird morphing thing that plays nice with the 9P protocol. Sounds neat, but how did it work in the real world? The result was a mixture of both breathtaking efficiency and alienating other-worldliness. Trying out the system is like a visit to an alternate reality where time-traveling gremlins changed how computers are made and used. You can execute anycommand anywhere just by typing its name and middle-clicking on it, even in the middle of reading a file. You can type out your blog post in the middle of a man page and save it right there. Screenshots are made by pointing /dev/screen to a file. When you execute a program in a terminal, the terminal morphs into the program you launched instead of running in the background. The window manager, rio, can be invoked within rio to create an instance of itself running inside itself. You can just keep going like that, until, like Inception, you get lost in which layer you're in. Get used to running Plan 9 long enough, and you will find yourself horribly ill-adapted for dealing with the normal world. While system administrators can't stop praising it, the average home user won't see much benefit unless they happen to run about eight desktop machines scattered all over. But to quote legendary hacker tribal bard Eric S. Raymond: “…Plan 9 failed simply because it fell short of being a compelling enough improvement on Unix to displace its ancestor.” A fascinating article, worth your time to read it through, even though we've pulled some of the best bits here. Nice look at the alternative dimension that could have been. Beastie Bits inks -- Basically Reddit or Hacker News, but without the disagreeable trolls and military industrial complex shills downvoting everything to hide the truth (http://www.tedunangst.com/flak/post/inks) “PAM is Un-American” talk now online (https://youtu.be/Mc2p6sx2s7k) Reddit advertising of “PAM Mastery” (http://blather.michaelwlucas.com/archives/2818) MeetBSD 2016 Report by Michael Dexter (https://www.ixsystems.com/blog/meetbsd-2016-report-michael-dexter/) Various CBSD Tutorials (https://www.bsdstore.ru/en/tutorial.html) Feedback/Questions Dylan - Kaltura Alt (http://pastebin.com/6B96pVcm) Scott - ZFS in Low-Mem (http://pastebin.com/Hrp8qwkP) J - Mixing Ports / Pkgs (http://pastebin.com/85q4Q3Xx) Trenton - Dtract & PC-BSD (http://pastebin.com/RFKY0ERs) Ivan - ZFS Backups (http://pastebin.com/31uqW6vW) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
This week on BSDNow. Allan and I are back from MeetBSD! A good time was had by all, lots to discuss, so let's jump right into it on your place to B...SD! This episode was brought to you by Headlines Build a FreeBSD 11.0-release Openstack Image with bsd-cloudinit (https://raymii.org/s/tutorials/FreeBSD_11.0-release_Openstack_Image.html) We are going to prepare a FreeBSD image for Openstack deployment. We do this by creating a FreeBSD 11.0-RELEASE instance, installing it and converting it using bsd-cloudinit. We'll use the CloudVPS public Openstack cloud for this. Create an account there and install the Openstack command line tools, like nova, cinder and glance. A FreeBSD image with Cloud Init will automatically resize the disk to the size of the flavor and it will add your SSH key right at boot. You can use Cloud Config to execute a script at first boot, for example, to bootstrap your system into Puppet or Ansible. If you use Ansible to manage OpenStack instances you can integrate it without manually logging in or doing anything manually. Since FreeBSD 10.2-RELEASE there is an rc script which, when the file /firstboot exists, expands the root filesystem to the full disk. While bsd-cloudinit does this as well, if you don't need the whole cloudinit stack, (when you use a static ssh key for example), you can touch that file to make sure the disk is expanded at the first boot A detailed tutorial that shows how to create customized cloud images using the FreeBSD install media There is also the option of using the FreeBSD release tools to build custom cloud images in a more headless fashion Someone should make a tutorial out of that *** iXsystems Announces TrueOS Launch (https://www.ixsystems.com/blog/ixsystems-announces-trueos-launch/) As loyal listeners to this show, you've no doubt heard by now that we are in the middle of undergoing a shift in moving PC-BSD -> TrueOS. Last week during MeetBSD this was made “official” with iX issuing our press release and I was able to give a talk detailing many of the reasons and things going on with this change. The talk should be available online here soon(ish), but for a quick recap: TrueOS is moving to a rolling-release model based on FreeBSD -CURRENT Lumina has become the default desktop for TrueOS LibreSSL is enabled top to bottom We are in the middle of working on conversion to OpenRC for run-control replacement The TrueOS pico was announced, which is our “Thin-Client” solution, right now allowing you to use a TrueOS server pared with a RPI2 device. *** Running FreeBSD 11 on Raspberry Pi (https://vzaigrin.wordpress.com/2016/10/16/running-freebsd-11-on-raspberry-pi/) This article covers some of the changes you will notice if you upgrade your RPI to FreeBSD 11.0 It covers some of the changes to WiFi in 11.0 Pro Tip: you can get a list of WiFi devices by doing: sysctl net.wlan.devices There are official binary packages for ARM with 11.0, so you can just ‘pkg install' your favourite apps Many of the LEDs are exposed via the /dev/led/ interface, which you can just echo 0 or 1 to, or use morse(6) to send a message gpioctl can be used to control the various GPIO pins The post also covers how to setup the real-time clock on the Raspberry Pi There is also limited support for adjusting the CPU frequency of the Pi There are also tips on configuring a one-wire temperature sensor *** void-zones-tools for FreeBSD (https://github.com/cyclaero/void-zones-tools) Adblock has been in the news a bit recently, with some of the more popular browser plugins now accepting brib^...contributions to permit specific ads through. Well today the ad-blockers strike back. We have a great tutorial up on GitHub which demonstrates one of the useful features of using Unbound in FreeBSD to do your own ad-blocking with void-zones. Specifically, void-zones are a way to return NXDOMAIN when DNS requests are made to known malicious or spam sites. Using void-zones-tools software will make managing this easy, by being able to pull in known lists of sites to block from several 3rd party curators. When coupled with our past tutorials on setting up your own FreeBSD router, this may become very useful for a lot of folks who want to do ad-blocking ad at a lower level, allowing it to filter smart-phones or any other devices on a network. *** News Roundup BSD Socket API Revamp (https://raw.githubusercontent.com/sustrik/dsock/master/rfc/sock-api-revamp-01.txt) Martin Sustrik has started a draft RFC to revamp the BSD Sockets API: The progress in the area of network protocols is distinctively lagging behind. While every hobbyist new to the art of programming writes and publishes their small JavaScript libraries, there's no such thing going on with network protocols. Indeed, it looks like the field of network protocols is dominated by big companies and academia, just like programming as a whole used to be before the advent of personal computers. the API proposed in this document doesn't try to virtualize all possible aspects of all possible protocols and provide a single set of functions to deal with all of them. Instead, it acknowledges how varied the protocol landscape is and how much the requirements for individual protocols differ. Therefore, it lets each protocol define its own API and asks only for bare minimum of standardised behaviour needed to implement protocol composability. As a consequence, the new API is much more lightweight and flexible than BSD socket API and allows to decompose today's monolithic protocol monsters into small single-purpose microprotocols that can be easily combined together to achieve desired functionality. The idea behind the new design is to allow the software author to define their own protocols via a generic interface, and easily stack them on top of the existing network protocols, be they the basic protocols like TCP/IP, or a layer 7 protocol like HTTP Example of creating a stack of four protocols: ~~ int s1 = tcpconnect("192.168.0.111:5555"); int s2 = foostart(s1, arg1, arg2, arg3); int s3 = barstart(s2); int s4 = bazstart(s3, arg4, arg5); ~~ It also allows applying generic transformations to the protocols: ~~ int tcps = tcpconnect("192.168.0.111:80"); /* Websockets is a connected protocol. */ int ws = websockconnect(tcps); uint16t compressionalgoritm; mrecv(ws, &compressionalgorithm, 2, -1); /* Compression socket is unconnected. */ int cs = compressstart(ws, compression_algorithm); ~~ *** Updated version of re(4) for DragonflyBSD (http://lists.dragonflybsd.org/pipermail/users/2016-November/313140.html) Sephe over at the Dragonfly project has issued a CFT for a newer version of the “re” driver For those who don't know, that is for Realtek nics, specifically his updates add features: I have made an updated version of re(4), which leverages Realtek driver's chip/PHY reset/initialization code. I hope it can resolve all kinds of weirdness we encountered on this chip so far. Testers, you know what to do! Give this a whirl and let him know if you run into any new issues, or better yet, give feedback if it fixes some long-standing problems you've run into in the past. *** Hackathon reports from OpenBSD's B2K16 b2k16 hackathon report: Jeremy Evans on ports cleaning, progress on postgres, nginx, ruby and more (http://undeadly.org/cgi?action=article&sid=20161112112023) b2k16 hackathon report: Landry Breuil on various ports progress (http://undeadly.org/cgi?action=article&sid=20161112095902) b2k16 hackathon report: Antoine Jacoutot on GNOME's path forward, various ports progress (http://undeadly.org/cgi?action=article&sid=20161109030623) We have a trio of hackathon reports from OpenBSD's B2K16 (Recently held in Budapest) First up - Jeremy Evans give us his rundown which starts with sweeping some of the cruft out of the barn: I started off b2k16 by channeling tedu@, and removing a lot of ports, including lang/ruby/2.0, lang/io, convertors/ruby-json, databases/dbic++, databases/ruby-swift, databases/ruby-jdbc-*, x11/ruby-profiligacy, and mail/ruby-mailfactory. After that, he talks about improvements made to postgres, nginx and ruby ports, fixing things such as pg_upgrade support, breaking nginx down into sub-packages and a major ruby update to about 50% of the packages. Next up - Landry Breuil tells us about his trip, which also started with some major ports pruning, including some stale XFCE bits and drupal6. One of the things he mentions is the Tor browser: Found finally some time again to review properly the pending port for Tor Browser, even if i don't like the way it is developed (600+ patches against upstream firefox-esr !? even if relationship is improving..) nor will endorse its use, i feel that the time that was spent on porting it and updating it and maintaining it shouldn't be lost, and it should get commited - there are only some portswise minor tweaks to fix. Had a bit of discussions about that with other porters... Lastly, Antoine Jacoutot gives us a smaller update on his work: First task of this hackathon was for Jasper and I to upgrade to GNOME 3.22.1 (version 3.22.2 hit the ports tree since). As usual I already updated the core libraries a few days before so that we could start with a nice set of fully updated packages. It ended up being the fastest GNOME update ever, it all went very smoothly. We're still debating the future of GNOME on OpenBSD though. More and more features require systemd interfaces and without a replacement it may not make sense to keep it around. Implementing these interfaces requires time which Jasper and I don't really have these days... Anyway, we'll see. All-n-all, a good trip it sounds like with some much needed hacking taking place. Good to see the cruft getting cleaned up, along with some new exciting ports landing. *** July to September 2016 Status Report (https://www.freebsd.org/news/status/report-2016-07-2016-09.html) The latest FreeBSD quarterly status report is out It includes the induction of the new Core team, and reports from all of the other teams, including Release Engineering, Port Manager, and the FreeBSD Foundation Some other highlights: Capsicum Update The Graphics Stack on FreeBSD Using lld, the LLVM Linker, to Link FreeBSD VirtualBox Shared Folders Filesystem evdev support (better mouse, keyboard, and multi-touch support) ZFS Code Sync with Latest OpenZFS/Illumos The ARC now mostly stores compressed data, the same as is stored on disk, decompressing them on demand. The L2ARC now stores the same (compressed) data as the ARC without recompression, and its RAM usage was further reduced. The largest size of indirect block possible has been increased from 16KB to 128KB, and speculative prefetching of indirect blocks is now performed. Improved ordering of space allocation. The SHA-512t256 and Skein hashing algorithms are now supported. *** Beastie Bits How to Host Your Own Private GitHub with Gogs (http://www.cs.cmu.edu/afs/cs/user/predragp/www/git.html) Nvidia Adds Telemetry To Latest Drivers (https://yro.slashdot.org/story/16/11/07/1427257/nvidia-adds-telemetry-to-latest-drivers) KnoxBUG Upcoming Meeting (http://knoxbug.org/2016-11-29) Feedback/Questions William - Show Music (http://pastebin.com/skvEgkLK) Ray - Mounting a Cell Phone (http://pastebin.com/nMDeSFGM) Ron - TrueOS + Radeon (http://pastebin.com/p5bC1jKU) (Follow-up - He used nvidia card) Kurt - ZFS Migration (http://pastebin.com/ud9vEK2C) Matt Dillon (Yes that Matt Dillon) - vkernels (http://pastebin.com/VPQfsUks) ***
This week on BSDNow, Allan & Kris are out at MeetBSD, but we never forget our loyal listeners. We have a great interview Allan did with Scott Long of Netflix & FreeBSD fame, as well as your questions on the place to B...SD! This episode was brought to you by Interview - Scott Long - scottl@freebsd.org (mailto:scottl@freebsd.org) FreeBSD & Netflix *** Feedback/Questions Zack - USB Config (http://pastebin.com/u77LE0Md) Jens - VMs, Jails and Containers (http://pastebin.com/8KwDK6ay) Ranko - Tarsnap Keys (http://pastebin.com/Kie3EcjN) Alex - OpenBSD in Hyper-V (http://pastebin.com/nRJQ7UPZ) Curt - Discussion Segment (http://pastebin.com/ndx25pQA)
This week on the show, we're loaded up with great stories ranging from System call fuzzing, a history of UNIX Pipes, speeding up MySQL imports and more. Stay tuned, BSDNow is coming your way right now. This episode was brought to you by Headlines System call fuzzing of OpenBSD amd64 using TriforceAFL (i.e. AFL and QEMU) (https://github.com/nccgroup/TriforceOpenBSDFuzzer) The NCCGroup did a series of fuzz testing against the OpenBSD syscall interface, during which they found a number of vulnerabilities, we covered this back in the early summer What we didn't notice, is that they also made the tools they used available. A combination of AFL (American Fuzzy Lop), QEMU, OpenBSD's FlashRD image generation tool, and the “Triforce” driver The other requirement is “a Linux box as host to run the fuzzer (other fuzzer hosts may work as well, we've only run TriforceAFL from a Linux host, specifically Debian/Ubuntu” It would be interesting to see if someone could get this to run from a BSD host It would also be interesting to run the same tests against the other BSDs *** On the Early History and Impact of Unix: the Introduction of Pipes (http://people.fas.harvard.edu/~lib113/reference/unix/unix2.html) Pipes are something we just take for granted today, but there was a time before pipes (How did anything get done?) Ronda Hauben writes up a great look back at the beginning of UNIX, and specifically at how pipes were born: One of the important developments in Unix was the introduction of pipes. Pipes had been suggested by McIlroy during the early days of creating Unix. Ritchie explains how "the idea, explained one afternoon on a blackboard, intrigued us but failed to ignite any immediate action. There were several objections to the idea as put....What a failure of imagination," he admits.(35) McIlroy concurs, describing how the initial effort to add pipes to Unix occurred about the same time in 1969 that Ritchie, Thompson and Canaday were outlining ideas for a file system. "That was when," he writes, "the simple pipeline as a way to combine programs, with data notationally propagating along a chain of (not necessarily concurrent) filters was articulated."(36) However, pipes weren't implemented in Unix until 1972. We also have a great quote from McIlroy on the day pipes were first introduced: Open Systems! Our Systems! How well those who were there remember the pipe-festooned garret where Unix took form. The excitement of creation drew people to work there amidst the whine of the computer's cool- ing fans, even though almost the same computer ac- cess, could be had from one's office or from home. Those raw quarters saw a procession of memorable events. The advent of software pipes precipitated a day-long orgy of one-liners....As people reveled in the power of functional composition in the large, which is even today unavailable to users of other systems. The paper goes on to talk about the invention of other important tools, such as “grep”, “diff” and more. Well worth your time if you want a glimpse into the history of UNIX *** Speeding up MySQL Import on FreeBSD (https://blog.feld.me/posts/2016/09/speeding-up-mysql-import-on-freebsd/) Mark Felder writes a blog post explaining how to speed up MySQL bulk data imports “I was recently tasked with rebuilding a readonly slave database server which only slaves a couple of the available databases. The backup/dump is straightforward and fast, but the restore was being excruciatingly slow. I didn't want to wait a week for this thing to finish, so I had to compile a list of optimizations that would speed up the process. This is the best way to do it on FreeBSD, assuming you're working with InnoDB. Additional optimizations may be required if you're using a different database engine.” “Please note this is assuming no other databases are running on this MySQL instance. Some of these are rather dangerous and you wouldn't want to put other live data at risk.” Most of the changes are meant to be temporary, used on a new server to import a dump of the database, then the settings are to be turned off. Specifically: sync_binlog = 0 innodbflushlogattrx_commit = 0 innodb-doublewrite = 0 He also prepends the following but of SQL before importing the data: set sqllogbin=0; set autocommit=0; set uniquechecks=0; set foreignkey_checks=0; You can also help yourself if your MySQL database lives on ZFS zfs set recordsize=16k pool/var/db/mysql zfs set redundant_metadata=most pool/var/db/mysql Remember, this tuning is ONLY for the initial import, leaving these settings on long term risks losing 5-10 seconds of your data if the server reboots unexpectedly zfs set sync=disabled pool/var/db/mysql zfs set logbias=throughput pool/var/db/mysql *** PostgreSQL and FreeBSD Quick Start (https://cwharton.com/blog/2016/10/postgresql-and-freebsd-quick-start/) There's lots of databases to choose from, but Postgres always has a special place on FreeBSD. Today we have a look at a ‘getting started' guide for those taking the plunge and using it for the first time. Naturally getting started will look familiar to many, a couple simple “pkg” and “sysrc” commands later, and you'll be set. After starting the service (With the “service” command) you'll be ready to start setting up your postgres instance. Next up you'll need to create your initial user/password combo, and a database with access granted to this particular user. If you plan to enable remote access to this DB server, you'll need to make some adjustments to one of the .conf files, allowing other IP's to connect. (If you are hosting something on the same system, this may not be needed) Now yous should be good to go! Enjoy using your brand new Postgres database. If this is your first rodeo, maybe start with something easy, like Apache or Nginx + Wordpress to try it out. *** News Roundup OpenBSD vmm hypervisor test drive (https://www.youtube.com/watch?v=KE_7E1pXy5c) As we asked for a week or two ago, someone has taken OpenBSD's vmm for a test drive, and made a video of it The command line interface for vmm, vmctl, looks quite easy to use. It takes an approach much closer to some of the bhyve management frameworks, rather than bhyve's rather confusing set of switches It also has a config file, the format of which looks very similar to what I designed for bhyveucl, and my first effort to integrate a config file into bhyve itself. The video also looks at accessing the console, configuring the networking, and doing an OpenBSD install in a fresh VM Currently vmm only supports running OpenBSD VMs *** FreeBSD Foundation October 2016 Update (https://www.freebsdfoundation.org/wp-content/uploads/2016/10/FreeBSD-Foundation-October-2016-Update.pdf) Wow, November is already upon us with the Holidays just around the corner. Before things get lost in the noise we wanted to highlight this update from the FreeBSD foundation. Before getting into the stories, they helpfully provide a list of upcoming conferences for this fall/winter, which includes a couple of USENIX gatherings, and the Developer Summit / MeetBSD next week. +The foundation gives us a quick hardware update initially, discussing some of the new ThunderX Cavium servers which are deployed (ARMv8 64Bit) and yes I'm drooling a bit. They also mention that work is ongoing for the RPi3 platform and PINE64. GNN also has an article reprinted from the FreeBSD journal, talking about the achievement of making it to 11.0 over the span of 23 years now. Of course he mentions that the foundation is open to all, and welcomes donations to continue to keep up this tradition of good work being done. Deb Goodkin gives us an update on the “Grace Hopper” convention that took place in Houston TX several weeks back. Roughly 14k women in Tech attended, which is a great turnout, and FreeBSD was well represented there. Next we have a call to potential speakers, don't forget that there are plenty of places you can help present about FreeBSD, not just at *BSD centered conferences, but the SCALES of the world as well. We wrap up with a look at EuroBSDCon 2016, quite a nice writeup, again brought to us by Deb at the foundation, and includes a list of some of those recognized for their contributions to FreeBSD. *** Adhokku – a toy PaaS powered by FreeBSD jails and Ansible (https://github.com/adhokku/adhokku) Described as a toy Platform-as-a-Service, Adhokku is an ansible based automated jail creation framework Based on the concept of Dokku, a single-host open source PaaS for Linux powered by Docker When you deploy an application using Adhokku, Adhokku creates a new jail on the remote host and provisions it from a fixed clean state using the instructions in the Jailfile in your Git repository. All jails sit behind a reverse proxy that directs traffic to one of them based on the domain name or the IP address in the HTTP request. When a new jail has been provisioned for an application, Adhokku seamlessly reconfigures the reverse proxy to send traffic to it instead of the one currently active for that application. The following instructions show how to get Adhokku and an example application running in a VM on your development machine using Vagrant. This process should require no FreeBSD-specific knowledge, through modifying the Jailfile to customize the application may. This seems like an interesting project, and it is good to see people developing workflows so users familiar with docker etc, can easily use BSD instead *** Installing OpenBSD 6.0 on your laptop is really hard (not) (http://sohcahtoa.org.uk/openbsd.html) OpenBSD on a laptop? Difficult? Not hardly. We have a great walkthrough by Keith Burnett, which demonstrates just how easy it can be to get up and running with an XFCE desktop from a fresh OpenBSD installation. For those curious,this was all done with a Thinkpad X60 and 120GB SSD and OpenBSD 6.0. He doesn't really cover the install process itself, that is well covered by the link to the OpenBSD FAQ pages. Once the system is up and running though, we start with the most important portion, getting working internet access (Via wifi) Really just a few ‘ifconfig' commands later and we are in business. Step 2 was getting the package configuration going. (I've never understood why this is still a thing, but no fret, its easy enough to do) With package repos available, now you can grab the binaries for XFCE and friends with just a few simple “pkg_add” commands Steps 4-6 are some specific bits to enable XFCE services, and some handy things such as setting doas permissions to get USB mounting working (For graphical mount/unmount) Lastly, keeping the system updated is important, so we have a nice tutorial on how to do that as well, using a handy “openup” script, which takes some of the guesswork out of it. Bonus! Steps for doing FDE as also included, which isn't for everyone, but you may want it *** Beastie Bits Pi-top with RPi-3 and FreeBSD HEAD (https://twitter.com/gvnn3/status/791475373380804608) NetBSD 7.0.2 released (http://blog.netbsd.org/tnf/entry/netbsd_7_0_1_released1) DragonflyBSD - git: kernel - Fix mmcsd read/write issues (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624851.html) A char device which implements an Enigma machine (FreeBSD & Linux) (https://github.com/rafael-santiago/dev-enigma) *** Feedback/Questions Matt - System Monitoring (http://pastebin.com/ayzvCuaq) Tony - LLVM License (http://pastebin.com/r5axPSE7) Ben - Thanks (http://pastebin.com/MNxCvUtX) David - Write Cache (http://pastebin.com/RswFASqW) Charles - Fonts (http://pastebin.com/e317a32f) ***
This week on BSDNow, we've got voting news for you (No not that election), a closer look at This episode was brought to you by Headlines ARIN 38 involvement, vote! (http://lists.nycbug.org/pipermail/talk/2016-October/016878.html) Isaac (.Ike) Levy, one of our interview guests from earlier this year, is running for a seat on the 15 person ARIN Advisory Council His goal is to represent the entire *BSD community at this important body that makes decisions about how IP addresses are allocated and managed Biographies and statements for all of the candidates are available here (https://www.arin.net/participate/elections/candidate_bios.pdf) The election ends Friday October 28th If elected, Ike will be looking for input from the community *** LibreSSL not just available but default (DragonFlyBSD) (https://www.dragonflydigest.com/2016/10/19/18794.html) DragonFly has become the latest BSD to join the growing LibreSSL family. As mentioned a few weeks back, they were in the process of wiring it up as a replacement for OpenSSL. With this latest commit, you can now build the entire base and OpenSSL isn't built at all. Congrats, and hopefully more BSDs (and Linux) jump on the bandwagon Compat_43 is gone (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624734.html) RiP 4.3 Compat support.. Well for DragonFly anyway. This commit finally puts out to pasture the 4.3 support, which has been disabled by default in DragonFly for almost 5 years now. This is a nice cleanup of their tree, removing more than a thousand lines of code and some of the old cruft still lingering from 4.3. *** Create your first FreeBSD kernel module (http://meltmes.kiloreux.me/create-your-first-freebsd-kernel-module/) This is an interesting tutorial from Abdelhadi Khiati, who is currently a master's student in AI and robotics I have been lucky enough to participate in Google Summer of Code with the FreeBSD foundation. I was amazed by the community surrounding it which was noob friendly and very helpful (Thank you FreeBSD We will run two storage controllers (ctrl-a, ctrl-b) and a host (cln-1). A virtual SAS drive (da0) of 256 MB is configured as “shareable” in Virtual Media Manager and simultaneously connected with both storage controllers The basic settings are applied to both controllers One interesting setting is: kern.cam.ctl.harole – configures default role for the node. So ctrl-a is set as 0 (primary node), ctrl-b – 1 (secondary node). The role also can be specified on per-LUN basis which allows to distribute LUNs over both controllers evenly. Note, kern.cam.ctl.haid and kern.cam.ctl.ha_mode are read-only parameters and must be set only via the /boot/loader.conf file. Once kern.cam.ctl.ha_peer is set, and the peers connect to each other, the log messages should reflect this: CTL: HA link status changed from 0 to 1 CTL: HA link status changed from 1 to 2 The link states can be: 0 – not configured, 1 – configured but not established and 2 – established Then ctld is configured to export /dev/da0 on each of the controllers Then the client is booted, and uses iscsid to connect to each of the exposed targets sysctl kern.iscsi.failondisconnection=1 on the client is needed to drop connection with one of the controllers in case of its failure As we know that da0 and da1 on the client are the same drive, we can put them under multipathing control: gmultipath create -A HA /dev/da0 /dev/da1 The document them shows a file being copied continuously to simulate load. Because the multipath is configured in ‘active/active' mode, the traffic is split between the two controllers Then the secondary controller is turned off, and iscsi disconnects that path, and gmultipath adapts and sends all of the traffic over the primary path. When the secondary node is brought back up, but the primary is taken down, traffic stops The console on the client is filled with errors: “Logical unit not accessible, asymmetric access state transition” The ctl(4) man page explains: > If there is no primary node (both nodes are secondary, or secondary node has no connection to primary one), secondary node(s) report Transitioning state. > Therefore, it looks like a “normal” behavior of CTL HA cluster in a case of disaster and loss of the primary node. It also means that a very lucky administrator can restore the failed primary controller before timeouts are elapsed. If the primary is down, the secondary needs to be promoted by some other process (CARP maybe?): sysctl kern.cam.ctl.ha_role=0 Then traffic follows again This is a very interesting look at this new feature, and I hope to see more about it in the future *** Is SPF Simply Too Hard for Application Developers? (http://bsdly.blogspot.com/2016/10/is-spf-simply-too-hard-for-application.html) Peter Hansteen asks an interesting question: The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers? He tells a story about trying to file his Norwegian taxes, and running into a bug Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site. The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics. The next time I checked my mail at bsdly.net, I found this bounce: support@altinn.no: SMTP error from remote mail server after RCPT TO:: host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed which means that somebody, somewhere tried to send a message to support@altinn.no, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain. What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails. After having a bunch of other problems, he finally gets a message back from the tax authority support staff: It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported. The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem. Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification? Beastie Bits Looking for a very part-time SysAdmin (https://lists.freebsd.org/pipermail/freebsd-jobs/2016-October/000930.html) If anyone wants to build the latest nodejs on OpenBSD... (https://twitter.com/qb1t/status/789610796380598272) IBM considers donating Power8 servers to OpenBSD (https://marc.info/?l=openbsd-misc&m=147680858507662&w=2) Install and configure DNS server in FreeBSD (https://galaxy.ansible.com/vbotka/freebsd-dns/) bhyve vulnerability in FreeBSD 11.0 (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:32.bhyve.asc) Feedback/Questions Larry - Pkg Issue (http://pastebin.com/8hwDVQjL) Larry - Followup (http://pastebin.com/3nswwk90) Jason - TrueOS (http://pastebin.com/pjfYWdXs) Matias - ZFS HALP! (http://pastebin.com/2tAmR5Wz) Robroy - User/Group (http://pastebin.com/7vWvUr8K) ***
This week on the show, we've got all sorts of goodies to discuss. Starting with, vmm, vkernels, raspberry pi and much more! Some iX folks are visiting from out of This episode was brought to you by Headlines vmm enabled (http://undeadly.org/cgi?action=article&sid=20161012092516&mode=flat&count=15) VMM, the OpenBSD hypervisor, has been imported into current It has similar hardware requirements to bhyve, a Intel Nehalem or newer CPU with the hardware virtualization features enabled in the BIOS AMD support has not been started yet OpenBSD is the only supported guest It would be interesting to hear from viewers that have tried it, and hear how it does, and what still needs more work *** vkernels go COW (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624675.html) The DragonflyBSD feature, vkernels, has gained a new Copy-On-Write functionality Disk images can now be mounted RO or RW, but changes will not be written back to the image file This allows multiple vkernels to share the same disk image “Note that when the vkernel operates on an image in this mode, modifications will eat up system memory and swap, so the user should be cognizant of the use-case. Still, the flexibility of being able to mount the image R+W should not be underestimated.” This is another feature we'd love to hear from viewers that have tried it out. *** Basic support for the RPI3 has landed in FreeBSD-CURRENT (https://wiki.freebsd.org/arm64/rpi3) The long awaited bits to allow FreeBSD to boot on the Raspberry Pi 3 have landed There is still a bit of work to be done, some of the as mentioned in Oleksandr's blog post: Raspberry Pi support in HEAD (https://kernelnomicon.org/?p=690) “Raspberry Pi 3 limited support was committed to HEAD. Most of drivers should work with upstream dtb, RNG requires attention because callout mode seems to be broken and there is no IRQ in upstream device tree file. SMP is work in progress. There are some compatibility issue with VCHIQ driver due to some assumptions that are true only for ARM platform. “ This is exciting work. No HDMI support (yet), so if you plan on trying this out make sure you have your USB->Serial adapter cables ready to go. Full Instructions to get started with your RPI 3 can be found on the FreeBSD Wiki (https://wiki.freebsd.org/arm64/rpi3) Relatively soon, I imagine there will be a RaspBSD build for the RPI3 to make it easier to get started Eventually there will be official FreeBSD images as well *** OpenBSD switches softraid crypto from PKCS5 PBKDF2 to bcrypt PBKDF. (https://github.com/openbsd/src/commit/2ba69c71e92471fe05f305bfa35aeac543ebec1f) After the discussion a few weeks ago when a user wrote a tool to brute force their forgotten OpenBSD Full Disk Encryption password (from a password list of possible variations of their password), it was discovered that OpenBSD defaulted to using just 8192 iterations of PKCSv5 for the key derivation function with a SHA1-HMAC The number of iterations can be manually controlled by the user when creating the softraid volume By comparison, FreeBSDs GELI full disk encryption used a benchmark to pick a number of iterations that would take more than 2 seconds to complete, generally resulting in a number of iterations over 1 million on most modern hardware. The algorithm is based on a SHA512-HMAC However, inefficiency in the implementation of PKCSv5 in GELI resulted in the implementation being 50% slower than some other implementations, meaning the effective security was only about 1 second per attempt, rather than the intended 2 seconds. The improved PKCSv5 implementation is out for review currently. This commit to OpenBSD changes the default key derivation function to be based on bcrypt and a SHA512-HMAC instead. OpenBSD also now uses a benchmark to pick a number of of iterations that will take approximately 1 second per attempt “One weakness of PBKDF2 is that while its number of iterations can be adjusted to make it take an arbitrarily large amount of computing time, it can be implemented with a small circuit and very little RAM, which makes brute-force attacks using application-specific integrated circuits or graphics processing units relatively cheap. The bcrypt key derivation function requires a larger amount of RAM (but still not tunable separately, i. e. fixed for a given amount of CPU time) and is slightly stronger against such attacks, while the more modern scrypt key derivation function can use arbitrarily large amounts of memory and is therefore more resistant to ASIC and GPU attacks.” The upgrade to the bcrypt, which has proven to be quite resistant to cracking by GPUs is a significant enhancement to OpenBSDs encrypted softraid feature *** Interview - Josh Paetzel - email@email (mailto:email@email) / @bsdunix4ever (https://twitter.com/bsdunix4ever) MeetBSD ZFS Panel FreeNAS - graceful network reload Pxeboot *** News Roundup EC2's most dangerous feature (http://www.daemonology.net/blog/2016-10-09-EC2s-most-dangerous-feature.html) Colin Percival, FreeBSD's unofficial EC2 maintainer, has published a blog post about “EC2's most dangerous feature” “As a FreeBSD developer — and someone who writes in C — I believe strongly in the idea of "tools, not policy". If you want to shoot yourself in the foot, I'll help you deliver the bullet to your foot as efficiently and reliably as possible. UNIX has always been built around the idea that systems administrators are better equipped to figure out what they want than the developers of the OS, and it's almost impossible to prevent foot-shooting without also limiting useful functionality. The most powerful tools are inevitably dangerous, and often the best solution is to simply ensure that they come with sufficient warning labels attached; but occasionally I see tools which not only lack important warning labels, but are also designed in a way which makes them far more dangerous than necessary. Such a case is IAM Roles for Amazon EC2.” “A review for readers unfamiliar with this feature: Amazon IAM (Identity and Access Management) is a service which allows for the creation of access credentials which are limited in scope; for example, you can have keys which can read objects from Amazon S3 but cannot write any objects. IAM Roles for EC2 are a mechanism for automatically creating such credentials and distributing them to EC2 instances; you specify a policy and launch an EC2 instance with that Role attached, and magic happens making time-limited credentials available via the EC2 instance metadata. This simplifies the task of creating and distributing credentials and is very convenient; I use it in my FreeBSD AMI Builder AMI, for example. Despite being convenient, there are two rather scary problems with this feature which severely limit the situations where I'd recommend using it.” “The first problem is one of configuration: The language used to specify IAM Policies is not sufficient to allow for EC2 instances to be properly limited in their powers. For example, suppose you want to allow EC2 instances to create, attach, detach, and delete Elastic Block Store volumes automatically — useful if you want to have filesystems automatically scaling up and down depending on the amount of data which they contain. The obvious way to do this is would be to "tag" the volumes belonging to an EC2 instance and provide a Role which can only act on volumes tagged to the instance where the Role was provided; while the second part of this (limiting actions to tagged volumes) seems to be possible, there is no way to require specific API call parameters on all permitted CreateVolume calls, as would be necessary to require that a tag is applied to any new volumes being created by the instance.” “As problematic as the configuration is, a far larger problem with IAM Roles for Amazon EC2 is access control — or, to be more precise, the lack thereof. As I mentioned earlier, IAM Role credentials are exposed to EC2 instances via the EC2 instance metadata system: In other words, they're available from http://169.254.169.254/. (I presume that the "EC2ws" HTTP server which responds is running in another Xen domain on the same physical hardware, but that implementation detail is unimportant.) This makes the credentials easy for programs to obtain... unfortunately, too easy for programs to obtain. UNIX is designed as a multi-user operating system, with multiple users and groups and permission flags and often even more sophisticated ACLs — but there are very few systems which control the ability to make outgoing HTTP requests. We write software which relies on privilege separation to reduce the likelihood that a bug will result in a full system compromise; but if a process which is running as user nobody and chrooted into /var/empty is still able to fetch AWS keys which can read every one of the objects you have stored in S3, do you really have any meaningful privilege separation? To borrow a phrase from Ted Unangst, the way that IAM Roles expose credentials to EC2 instances makes them a very effective exploit mitigation mitigation technique.” “To make it worse, exposing credentials — and other metadata, for that matter — via HTTP is completely unnecessary. EC2 runs on Xen, which already has a perfectly good key-value data store for conveying metadata between the host and guest instances. It would be absolutely trivial for Amazon to place EC2 metadata, including IAM credentials, into XenStore; and almost as trivial for EC2 instances to expose XenStore as a filesystem to which standard UNIX permissions could be applied, providing IAM Role credentials with the full range of access control functionality which UNIX affords to files stored on disk. Of course, there is a lot of code out there which relies on fetching EC2 instance metadata over HTTP, and trivial or not it would still take time to write code for pushing EC2 metadata into XenStore and exposing it via a filesystem inside instances; so even if someone at AWS reads this blog post and immediately says "hey, we should fix this", I'm sure we'll be stuck with the problems in IAM Roles for years to come.” “So consider this a warning label: IAM Roles for EC2 may seem like a gun which you can use to efficiently and reliably shoot yourself in the foot; but in fact it's more like a gun which is difficult to aim and might be fired by someone on the other side of the room snapping his fingers. Handle with care!” *** Open-source storage that doesn't suck? Our man tries to break TrueNAS (http://www.theregister.co.uk/2016/10/18/truenas_review/) The storage reviewer over at TheRegister got their hands on a TrueNAS and gave it a try “Data storage is difficult, and ZFS-based storage doubly so. There's a lot of money to be made if you can do storage right, so it's uncommon to see a storage company with an open-source model deliver storage that doesn't suck.” “To become TrueNAS, FreeNAS's code is feature-frozen and tested rigorously. Bleeding-edge development continues with FreeNAS, and FreeNAS comes with far fewer guarantees than does TrueNAS.” “iXsystems provided a Z20 hybrid storage array. The Z20 is a dual-controller, SAS-based, high-availability, hybrid storage array. The testing unit came with a 2x 10GbE NIC per controller and retails around US$24k. The unit shipped with 10x 300GB 10k RPM magnetic hard drives, an 8GB ZIL SSD and a 200GB L2ARC SSD. 50GiB of RAM was dedicated to the ARC by the system's autotune feature.” The review tests the performance of the TrueNAS, which they found acceptable for spinning rust, but they also tested the HA features While the look of the UI didn't impress them, the functionality and built in help did “The UI contains truly excellent mouseover tooltips that provide detailed information and rationale for almost every setting. An experienced sysadmin will be able to navigate the TrueNAS UI with ease. An experienced storage admin who knows what all the terms mean won't have to refer to a wiki or the more traditional help manual, but the same can't be said for the uninitiated.” “After a lot of testing, I'd trust my data to the TrueNAS. I am convinced that it will ensure the availability of my data to within any reasonable test, and do so as a high availability solution. That's more than I can say for a lot of storage out there.” “iXsystems produce a storage array that is decent enough to entice away some existing users of the likes of EMC, NetApp, Dell or HP. Honestly, that's not something I thought possible going into this review. It's a nice surprise.” *** OpenBSD now officially on GitHub (https://github.com/openbsd) Got a couple of new OpenBSD items to bring to your attention today. First up, for those who didn't know, OpenBSD development has (always?) taken place in CVS, similar to NetBSD and previously FreeBSD. However today, Git fans can rejoice, since there is now an “official” read-only github mirror of their sources for public consumption. Since this is read-only, I will assume (unless told otherwise) that pull-requests and whatnot aren't taken. But this will come in handy for the “git-enabled” among us who need an easier way to checkout OpenBSD sources. There is also not yet a guarantee about the stability of the exporter. If you base a fork on the github branch, and something goes wrong with the exporter, the data may be reexported with different hashes, making it difficult to rebase your fork. How to install LibertyBSD or OpenBSD on a libreboot system (https://libreboot.org/docs/bsd/openbsd.html) For the second part of our OpenBSD stories, we have a pretty detailed document posted over at LibreBoot.org with details on how to boot-strap OpenBSD (Or LibertyBSD) using their open-source bios replacement. We've covered blog posts and other tidbits about this process in the past, but this seems to be the definitive version (so far) to reference. Some of the niceties include instructions on getting the USB image formatted not just on OpenBSD, but also FreeBSD, Linux and NetBSD. Instructions on how to boot without full-disk-encryption are provided, with a mention that so far Libreboot + Grub does not support FDE (yet). I would imagine somebody will need to port over the openBSD FDE crypto support to GRUB, as was done with GELI at some point. Lastly some instructions on how to configure grub, and troubleshoot if something goes wrong will help round-out this story. Give it a whirl, let us know if you run into issues. Editorial Aside - Personally I find the libreboot stuff fascinating. It really is one of the last areas that we don't have full control of our systems with open-source. With the growth of EFI, it seems we rely on a closed-source binary / mini-OS of sorts just to boot our Open Source solutions, which needs to be addressed. Hats off to the LibreBoot folks for taking on this important challenge. *** FreeNAS 9.10 – LAGG & VLAN Overview (https://www.youtube.com/watch?v=wqSH_uQSArQ) A video tutorial on FreeNAS's official YouTube Channel Covers the advanced networking features, Link Aggregation and VLANs Covers what the features do, and in the case of LAGG, how each of the modes work and when you might want to use it *** Beastie Bits Remote BSD Developer Position is up for grabs (https://www.cybercoders.com/bsd-developer-remote-job-305206) Isilon is hiring for a FreeBSD Security position (https://twitter.com/jeamland/status/785965716717441024) Google has ported the Networked real-time multi-player BSD game (https://github.com/google/web-bsd-hunt) A bunch of OpenBSD Tips (http://www.vincentdelft.be) The last OpenBSD 6.0 Limited Edition CD has sold (http://www.ebay.com/itm/-/332000602939) Dan spots George Neville-Neil on TV at the Airport (https://twitter.com/DLangille/status/788477000876892162) gnn on CNN (https://www.youtube.com/watch?v=h7zlxgtBA6o) SoloBSD releases v 6.0 built upon OpenBSD (http://solobsd.blogspot.com/2016/10/release-solobsd-60-openbsd-edition.html) Upcoming KnoxBug looks at PacBSD - Oct 25th (http://knoxbug.org/content/2016-10-25) Feedback/Questions Morgan - Ports and Packages (http://pastebin.com/Kr9ykKTu) Mat - ZFS Memory (http://pastebin.com/EwpTpp6D) Thomas - FreeBSD Path Length (http://pastebin.com/HYMPtfjz) Cy - OpenBSD and NetHogs (http://pastebin.com/vGxZHMWE) Lars - Editors (http://pastebin.com/5FMz116T) ***
The wait is over, 11.0 of FreeBSD has (officially) launched. We'll have coverage of this, plus a couple looks back at UNIX history, and a crowd-favorite guest today. This episode was brought to you by Headlines FreeBSD 11.0-RELEASE Now Available (https://lists.freebsd.org/pipermail/freebsd-announce/2016-October/001760.html) FreeBSD 11.0-RELEASE is now officially out. A last minute reroll to pickup OpenSSL updates and a number of other security fixes meant the release was a little behind schedule, and shipped as 11.0-RELEASE-p1, but the release is better for it Improved support for 802.11n and various wifi drivers Support for the AArch64 (arm64) architecture has been added. Native graphics support has been added to the bhyve(8) hypervisor. A new flag, “onifconsole” has been added to /etc/ttys. This allows the system to provide a login prompt via serial console if the device is an active kernel console, otherwise it is equivalent to off. The xz(1) utility has been updated to support multi-threaded compression. A number of kernel panics related to VNET have been fixed The IMAGACT_BINMISC kernel configuration option has been enabled by default, which enables application execution through emulators, such as QEMU via binmiscctl(8). The GENERIC kernel configuration has been updated to include the IPSEC option by default. The kern.osrelease and kern.osreldate are now configurable jail(8) parameters A new sysctl(8), kern.racct.enable, has been added, which when set to a non-zero value allows using rctl(8) with the GENERIC kernel. A new kernel configuration option, RACCT_DISABLED has also been added. The minimum (arcmin) and maximum (arcmax) values for the ZFS adaptive replacement cache can be modified at runtime. Changes to watch out for: OpenSSH DSA key generation has been disabled by default. It is important to update OpenSSH keys prior to upgrading. Additionally, Protocol 1 support has been removed. By default, the ifconfig(8) utility will set the default regulatory domain to FCC on wireless interfaces. As a result, newly created wireless interfaces with default settings will have less chance to violate country-specific regulations. An issue was discovered with Amazon® EC2™ images which would cause the virtual machine to hang during boot when upgrading from previous FreeBSD versions. New EC2™ installations are not affected, but existing installations running earlier releases are advised to wait until the issue is resolved in an Errata Notice before upgrading. An Errata Notice to address this is planned following the release. *** process listing consistency (http://www.tedunangst.com/flak/post/process-listing-consistency) Ted Unangst asks: how consistent is the output of ps(1)? If processes are starting and exiting constantly, and you run ps(1), is the output guaranteed to reflect that exact moment in time, or might it include some processes that have gone away before ps(1) exited, and include some processes that did not exist when ps(1) was started? Ted provides a little example chicken/egg program to try to create such an inconsistency, so you can test out your OS On OpenBSD ps(1) was switched away from the reading kernel memory directly, and instead uses the KERNPROCALL sysctl Thus sysctl can iterate over the entire process list, copying out information to ps(1), without blocking. If we prevent processes from forking or exiting during this time, we get a consistent snapshot. The snapshot may be stale, but it will never show us a viewpoint that never happened. So, OpenBSD will always be consistent, or will it? Is there a way to trick ps on OpenBSD? Not everything is consistent. There's a separate sysctl, KERNPROCARGV, that reads the command line arguments for a process, but it only works on one process at a time. Processes can modify their own argv at any time. A second test program changes the process title of both the chicken and the egg, and if you run ps(1), you can get back a result that never actually happened. The argv of the first program is read by ps(1), and in the meantime, it changes to a different value. The second program also changes its value, so now when ps(1) reads it, it sees the new value, not the original value from when ps(1) was started. So the output is not that consistent, but is it worth the effort to try to make it so? DragonFlyBSD - if_iwm - Add basic powermanagement support via ifconfig wlan0 powersave (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624673.html) WiFi can often be one of the biggest drains on your laptop battery, so anything we can do to improve the situation should be embraced. Imre Vadász over at the DragonFly project has done that, porting over a new set of power management support from Linux to the if_iwm driver. if_iwm - Add basic powermanagement support via ifconfig wlan0 powersave. The DEVICEPOWERFLAGSCAMMSK flag was removed in the upstream iwlwifi in Linux commit ceef91c89480dd18bb3ac51e91280a233d0ca41f. Add scpsdisabled flag to struct iwmsoftc, which corresponds to mvm->psdisabled in struct iwl_mvm in Linux iwlwifi. Adds a hw.iwm.powerscheme tunable which corresponds to the powerscheme module parameter in Linux iwlwifi. Set this to 1 for completely disabling power management, 2 (default) for balanced powermanagement, and 3 for lowerpower mode (which does dtim period skipping). Imports the constants.h file from iwlwifi as ifiwmconstants.h. This doesn't allow changing the powermanagement setting while connected, also one can only choose between enabled and disabled powersaving with ifconfig (so switching between balanced and low-power mode requires rebooting to change the tunable). After any changes to powermanagement (i.e. "ifconfig wlan0 powersave" to enable powermanagement, or "ifconfig wlan0 -powersave" for disabling powermanagement), one has to disconnect and reconnect to the accespoint for the change to take effect.“ Good stuff! These positive changes need to happen more often and sooner, so we can all eek out every drop of power from our respective laptops. *** Helping out an Internet Friend…Dual boot OpenBSD (https://functionallyparanoid.com/2016/10/03/helping-out-an-internet-friend/) Dual-booting OpenBSD and Linux, via UEFI. A year ago we wouldn't be discussing this, but today we have an article where somebody has done exactly that. This Journey was undertaken by Brian Everly (Indiana Bug), partly due to a friend who wanted to dual-boot his laptop which already has an existing UEFI install on it. As a proof of concept, he began by replicating the setup in VMware with UEFI He started by throwing Ubuntu into the VM, with some special attention paid to partitioning to ensure enough room left-over for OpenBSD later. I created a 64MB EFI partition at the front of the disk. Next, I created a 20GB primary partition at the beginning of the space, mounted as the root (/) filesystem. I then added a 4096MB swap partition for Ubuntu. Finally, I used the rest of the free space to create a Reserved BIOS Boot Area FAT32 partition that was not associated with a mount point – this is where I will be installing OpenBSD. With that done, he wrapped up the Ubuntu installation and then turned over to to the OpenBSD side. Some manual partitioning was required to install to the “Reserved FAT32” partition. I mashed through the defaults in the OpenBSD installer until I got to the disk partitioning. Since I told VMWare to make my hard drive an IDE one, I knew I was playing around with wd0 and not sd0 (my USB key). I dumped into fdisk by selecting to (E)dit the partition scheme and saw my setup from Linux. First was the EFI partition (I am guessing I'll have to copy my bootx64.efi file to that at some point), second was the Linux etx4 partition, third was my Linux swap partition and fourth was a weird looking one that is the “Reserved BIOS Boot” partition. That's the one I'll fiddle with. Issuing the command “edit 3” allowed me to fiddle with that partition #3 (remember, we start counting at zero). I set it's type to “A6” (OpenBSD) and then took the defaults with the exception of naming it “OpenBSD”. A quick “write” followed by a “quit” allowed me to update my new partition and get back to the installer. Once the installation was wrapped up (OpenBSD helpfully already created the /boot/EFI partition with the correct EFI loader installed) he was able to reboot and select between the two systems at the UEFI bios screen. For kicks, he lastly went into Ubuntu and grabbed refind. Installing refind provided a fancy graphical selector between the two systems without too much trouble. Next step will be to replicate this process on his friend's laptop. Wishing you luck with that journey! Interview - Bryan Cantrill - email@email (mailto:email@email) / @twitter (https://twitter.com/user) CTO of Joyent *** News Roundup After 22 Years, 386BSD Gets An Update (https://bsd.slashdot.org/story/16/10/09/0230203/after-22-years-386bsd-gets-an-update) Slashdot brings us an interesting mention this week, specifically that after 22 years, we now have an update to 386BSD. 386BSD was last released back in 1994 with a series of articles in Dr. Dobb's Journal -- but then developers for this BSD-based operating system started migrating to both FreeBSD and NetBSD. An anonymous Slashdot reader writes: The last known public release was version 0.1. Until Wednesday, when Lynne Jolitz, one of the co-authors of 386BSD, released the source code to version 1.0 as well as 2.0 on Github. 386BSD takes us back to the days when you could count every file in your Unix distribution and more importantly, read and understand all of your OS source code. 386BSD is also the missing link between BSD and Linux. One can find fragments of Linus Torvalds's math emulation code in the source code of 386BSD. To quote Linus: "If 386BSD had been available when I started on Linux, Linux would probably never had happened.” Though it was designed for Intel 80386 microprocessors, there's already instructions for launching it on the hosted hardware virtualization service Qemu. There you have it! Go grab the new hotness that is 386BSD and run it in 2016! Or perhaps you want FreeBSD 11, but to each their own. *** Progress of the OpenBSD Limited Edition Signed CD set (http://undeadly.org/cgi?action=article&sid=20160929230557&mode=expanded) An update from a story last week! We mentioned the “very” limited edition OpenBSD 6.0 signed CD sets that had gone up for Auction on Ebay. (With proceeds to support for Foundation) As of today, here's where we stand: CD set #1 (Sep 29th + 5 days) sold for $4200 (http://www.ebay.com/itm/-/331985953783) CD set #2 (Oct 4th + 3 days) sold for $3000 (http://www.ebay.com/itm/-/331990536246) CD set #3 (Oct 8th + 3 days) sold for $817 (http://www.ebay.com/itm/-/331994217419) CD set #4 (Oct 11th + 3 days) is currently up for bidding (http://www.ebay.com/itm/-/331997031152) There you have it! The 4th set is almost wrapped up bidding, and the 5th and last set is not far behind. Be sure to grab your piece of BSD history before its gone! PROTOTYPE FreeBSD Jail/ZFS based implementation of the Application Container Specification (https://github.com/3ofcoins/jetpack) “Jetpack is an experimental and incomplete implementation of the App Container Specification for FreeBSD. It uses jails as isolation mechanism, and ZFS for layered storage.” “This document uses some language used in Rocket, the reference implementation of the App Container Specification. While the documentation will be expanded in the future, currently you need to be familiar at least with Rocket's README to understand everything.” + A standard with multiple implementations, that allow substitution of components, such as FreeBSD Jails instead of docker/lxc etc, and ZFS instead of overlayfs etc, is very exciting Microsoft's Forgotten Unix-based Operating System (https://fossbytes.com/xenix-history-microsoft-unix-operating-system/) Do you remember the good old days. You know, when Microsoft was the driving force behind UNIX? Wait, what did you say you may be thinking? It's true, and lets sit back and let FossBytes tell us a tale of what once was reality. The story begins sometime in the late 70's: Turning back the pages to the late 1970's, Microsoft entered into an agreement with AT&T Corporation to license Unix from AT&T. While the company didn't sell the OS to public, it licensed it to other OEM vendors like Intel, SCO, and Tandy. As Microsoft had to face legal trouble due to “Unix” name, the company renamed it and came up with its own Unix distribution. So, AT&T licensed Unix to Redmond that was passed on to other OEMs as Xenix. It's interesting to recall a time when Microsoft enabled people to run Unix — an operating system originally designed for large and multiuser systems — on a microcomputer. Even though it came first, Unix was probably more powerful than MS-DOS. So whatever happened to this microsoft-flavored UNIX you may ask? Sadly it was ditched for DOS due to $REASONS: In early 1980's, IBM was looking for an OS to power its PC. As IBM didn't want to maintain any ties with the recently split AT&T, Xenix was automatically rejected. To fulfill, the tech giant's demand, Microsoft bought 86-DOS from Seattle Computer Products and managed to convince IBM to use it in their systems. Slowly, Microsoft started losing interest in Xenix and traded the full rights of Xenix with SCO, a Xenix partner company. The company filed bankruptcy in 2007 before taking the Xenix legacy to the 21st century in the form of Open Server, previously known as SCO Unix and SCO Open Desktop. An interesting chapter in UNIX history to be sure, and funny enough may come full-circle someday with Microsoft beginning to show interest in UNIX and BSD once again. *** Beastie Bits Ohio LinuxFest 2016 wrap-up (http://blather.michaelwlucas.com/archives/2791) Learn X in Y minutes Where X=zfs (https://learnxinyminutes.com/docs/zfs/) Add touchscreen support for the official 7" RPi touch display (https://svnweb.freebsd.org/base?view=revision&revision=306430) 64-bit U-Boot on Raspberry Pi 3 (https://kernelnomicon.org/?p=682) SNIA SDC 2016 Recap: Michael Dexter (https://www.ixsystems.com/blog/snia-sdc-2016-recap-michael-dexter/) OpenZFS: Stronger than ever (https://www.ixsystems.com/blog/openzfs-devsummit-2016/) Accurate, Traceable, and Verifiable Time Synchronization for World Financial Markets (http://nvlpubs.nist.gov/nistpubs/jres/121/jres.121.023.pdf) ON HOLY WARS AND A PLEA FOR PEACE (https://www.ietf.org/rfc/ien/ien137.txt) Feedback/Questions Morgan - Zero-Filling an VM (http://pastebin.com/CYcqmW7P) Charlie - ZFS Bit-Rot (http://pastebin.com/12mNW57h) Matias - TrueOS / Launchd (http://pastebin.com/NfYWt2cu) Dale - DO Feedback (http://pastebin.com/UvKh2WcF) James - DO / FreeBSD Locks? (http://pastebin.com/0cdMc88U) ***
This week on the show, we'll be talking to Petra about the NetBSD foundation, about how they operate and assist NetBSD behind the scenes. That plus lots of news This episode was brought to you by Headlines What is new on EC2 for FreeBSD 11.0-RELEASE (http://www.daemonology.net/blog/2016-10-03-FreeBSD-EC2-11-0-RELEASE.html) “FreeBSD 11.0-RELEASE is just around the corner, and it will be bringing a long list of new features and improvements — far too many for me to list here. I think there are some improvements in FreeBSD 11.0 which are particularly noteworthy for EC2 users.” “First, the EC2 Console Screenshot functionality now works with FreeBSD. This provides a "VGA" output as opposed to the traditional "serial port" which EC2 has exposed as "console output" for the past decade, and is useful largely because the "VGA" output becomes available immediately whereas the "serial port" output can lag by several minutes. This improvement is a simple configuration change — older releases didn't waste time writing to a non-serial console because it didn't go anywhere until Amazon added support on their side — and can be enabled on older FreeBSD releases by changing the line console="comconsole" to boot_multicons="YES" in /boot/loader.conf.” “The second notable change is support for EC2 "Enhanced Networking" using Intel 82599 hardware; on the C3, C4, R3, I2, D2, and M4 (excluding m4.16xlarge) families, this provides increased network throughput and reduced latency and jitter, since it allows FreeBSD to talk directly to the networking hardware rather than via a Xen paravirtual interface. Getting this working took much longer than I had hoped, but the final problem turned out not to be in FreeBSD at all — we were tickling an interrupt-routing bug in a version of Xen used in EC2. Unfortunately FreeBSD does not yet have support for the new "Elastic Network Adapter" enhanced networking used in P2 and X1 instance families and the m4.16xlarge instance type; I'm hoping that we'll have a driver for that before FreeBSD 11.1 arrives.” “The third notable change is an improvement in EC2 disk throughput. This comes thanks to enabling indirect segment I/Os in FreeBSD's blkfront driver; while the support was present in 10.3, I had it turned off by default due to performance anomalies on some EC2 instances. (Those EC2 performance problems have been resolved, and disk I/O performance in EC2 on FreeBSD 10.3 can now be safely improved by removing the line hw.xbd.xbdenableindirect="0" from /boot/loader.conf.)” “Finally, FreeBSD now supports all 128 CPUs in the x1.32xlarge instance type. This improvement comes thanks to two changes: The FreeBSD default kernel was modified in 2014 to support up to 256 CPUs (up from 64), but that resulted in a (fixed-size) section of preallocated memory being exhausted early in the boot process on systems with 92 or more CPUs; a few months ago I changed that value to tune automatically so that FreeBSD can now boot and not immediately panic with an out-of-the-box setup on such large systems.” “I think FreeBSD/EC2 users will be very happy with FreeBSD 11.0-RELEASE; but I'd like to end with an important reminder: No matter what you might see on FTP servers, in EC2, or available via freebsd-update, the new release has not been released until you see a GPG-signed email from the release engineer. This is not just a theoretical point: In my time as a FreeBSD developer I've seen multiple instances of last-minute release re-rolls happening due to problems being discovered very late, so the fact that you can see bits doesn't necessarily mean that they are ready to be downloaded. I hope you're looking forward to 11.0-RELEASE, but please be patient.” *** Upgrading Amazon EC2 instance from 10.3 to 11.0-PRERELEASE results in hang at boot (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213196) As if to underscore that last point, a last minute bug was found on sunday night A user reported that they used freebsd-update to upgrade an EC2 instance from 10.3 to 11.0 and it started hanging during boot After some quick investigation by Colin, the problem was reproduced Since I had done a lot of work in the loader recently, I helped Colin build a version of the loader with a lot of the debugging enabled, and some more added to try to isolate where in the loader the freeze was happening Colin and I worked late into the night, but eventually found the read from disk that was causing the hang Unlike most of the other reads, that were going into the heap, this read was into a very low memory address, right near the 640kb border. This initially distracted us from the real cause of the problem With more debugging added, it was determined that the problem was in the GELIBoot code, when reading the last sector of each partition to determine if it is encrypted. In cases where the partition is not 4k aligned, and butts up against the end of the disk, the formula used could result in a read past the end of the disk The formula rounds the last sector byte address down to the nearest factor of 4096, then reads 4096 bytes. Then that buffer is examined to determine if the partition is encrypted. If it is a 512b sector drive, the metadata will be in the last 512 bytes of that 4096 byte buffer. However, if the partition is not 4k aligned, the rounding will produce a value that is less than 4096 bytes from the end of the disk, and attempting to read 4096 bytes, will read past the end of the disk Normally this isn't that big of a problem, the BIOS will just return an error. The loader will retry up to three times, then give up and move on, continuing to boot normally. Some BIOSes are buggy, and will initiate their own retries, and the combination might result in a stall of up to 30 seconds for each attempt to read past the end of the disk But it seems that Amazon EC2 instances, (and possibly other virtual instances), will just hang in this case. This bug has existed for 6 months, but was not caught because almost all installations are 4k aligned thanks to changes made to the installer over the last few years, and most hardware continues to boot with no sign of a problem Even the EC2 snapshot images of 11.0 do not have the problem, as they use a newer disk layout that is 4k aligned by default now. The problem only seems to happen when older disk images are upgraded The fix has been committed and will be merged the the branches over the next few days An Errata notice will be issues, and the fix will be available via freebsd-update It is recommended that EC2 users, and anyone who wants to be especially cautious, wait until this errata notice goes out before attempting to upgrade from FreeBSD 10.3 to 11.0 You can determine if your partitions are 4k aligned by running ‘gpart show'. If there is free space after your last partition, you won't have any issues. *** OpenBSD 6.0 Limited Edition CD set (signed by developers) (http://undeadly.org/cgi?action=article&sid=20160929230557&mode=expanded) The first one went for .$4,200.00 (http://www.ebay.com/itm/-/331985953783) + Looking for your piece of OpenBSD history? At the recent g2k16 hackathon in Cambridge UK, 40 OpenBSD developers put pen to paper and signed 5 copies of the new 6.0 release. + Each of these will be auctioned off on ebay, with the proceeds to benefit the OpenBSD foundation. + The first auction has already ended, and CD set went for a whopping $4200! + The next set only has 2 days left, and currently stands at $3000! (http://www.ebay.com/itm/-/331990536246) + Get your bids in soon, these are VERY unique, the odds of getting the same 40 developers in a room together and signing a new .0 release may make this a once-in-a-lifetime opportunity. + Additionally, if you are just starting your OpenBSD collection, here's a nice image to make you envious: A nice collection of OpenBSD CD Sets (http://i.imgur.com/OrE0Gsa.png) [What typing ^D really does on Unix ](https://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffects) + How often have you used a ^D to generate an EOF? Do you really know what that does? + Chris Siebenmann has posted a look at this on his blog, which might not be what you think “Typing ^D causes the tty driver to immediately finish a read().” He continues on: Normally doing a read() from a terminal is line-buffered inside the tty driver; your program only wakes up when the tty driver sees the newline, at which point you get back the full line. (Note that this buffering is distinct from anything that your language's IO system may be doing.) Typing ^D causes the tty driver to stop waiting for a newline and immediately return from the read() with however much of the line has been accumulated to date. If you haven't typed anything on the line yet, there is nothing accumulated and the read() will return 0 bytes, which is conveniently the signal for end of file. If you have typed something the program will get it; because it doesn't have a trailing newline, the program's own line-buffering may take over and keep read()ing to get the rest of the line. (Other programs will immediately process the partial line with no buffering; cat is one example of this.) Once you've typed ^D on a partial line, that portion of the line is immutable because it's already been given to the program. Most Unixes won't let you backspace over such partial lines; effectively they become output, not input. (Note that modern shells are not good examples of this, because they don't do line-buffered input; to support command line editing, they switch terminal input into an uninterpreted mode. So they get the raw ^D and can do whatever they want with it, and they can let you edit as much of the pending line as they want.) Fascinating stuff, and interesting to see behind the curtain at exactly what's going on with your programs buffering and tty driver interaction. Interview - Petra Zeidler - spz@netbsd.org (mailto:spz@netbsd.org) NetBSD Foundation *** News Roundup Running FreeBSD in Travis-CI Thanks to KQEmu (http://erouault.blogspot.com/2016/09/running-freebsd-in-travis-ci.html) Travis-CI is the most popular testing framework on Github, but it doesn't support any of the BSDs This didn't discourage Even Rouault, who managed to run FreeBSD in KQEMU on the Linux instances provided by Travis-CI “Travis-CI has a free offer for software having public repository at GitHub. Travis-CI provides cloud instances running Linux or Mac OS X. To increase portability tests of GDAL, I wondered if it was somehow possible to run another operating system with Travis-CI, for example FreeBSD. A search lead me to this question (https://github.com/travis-ci/travis-ci/issues/1818) in their bug tracker but the outcome seems to be that it is not possible, nor in their medium or long term plans.” “One idea that came quickly to mind was to use the QEMU machine emulator that can simulate full machines, of several hardware architectures.” They found an existing image of FreeBSD 9.2 and configured the Travis job to download it and fire it up in QEMU. “Here we go: ./configure && make ! That works, but 50 minutes later (the maximum length of a Travis-CI job), our job is killed with perhaps only 10% of the GDAL code base being compiled. The reason is that we used the pure software emulation mode of QEMU that involves on-the-fly disassembling of the code to be run and re-assembling.” Travis-CI runs in Google Compute Engine, which does not allow nested virtualization, so hardware virtualization is not an option to speed up QEMU “Here comes the time for good old memories and a bit of software archeology. QEMU was started by Fabrice Bellard. If you didn't know his name yet, F. Bellard created FFMPEG and QEMU, holds a world record for the number of decimals of Pi computed on a COTS PC, has ported QEMU in JavaScript to run the Linux kernel in your browser, devised BPG, a new compression based on HEVC, etc....” “At the time where his interest was focused on QEMU, he created KQemu, a kernel module (for Linux, Windows, FreeBSD hosts), that could significantly enhance QEMU performance when the guest and hosts are x86/x86_64 and does not require (nor use) hardware virtualization instructions.” “Running it on Travis-CI was successful too, with the compilation being done in 20 minutes, so probably half of the speed of bare metal, which is good enough.” “I could also have potentially tried VirtualBox because, as mentioned above, it supports software virtualization with acceleration. But that is only for 32 bit guests (and I didn't find a ready-made FreeBSD 32bit image that you can directly ssh into). For 64 bit guests, VirtualBox require hardware virtualization to be available in the host. To the best of my knowledge, KQemu is (was) the only solution to enable acceleration of 64 bit guests without hardware requirements.” It will be interesting to see if enough people do this hack, maybe Travis-CI will consider properly supporting FreeBSD *** OpenBSD EuroBSDcon 2016 Papers are online (https://www.openbsd.org/events.html) Slides from the OpenBSD talks at EuroBSDCon are online now Landry Breuil, Building packages on exotic architectures (https://rhaalovely.net/~landry/eurobsdcon2016/) Peter Hessler, Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD (https://www.openbsd.org/papers/eurobsdcon2016-bfd.pdf) Ingo Schwarze, Why and how you ought to keep multibyte character support simple (https://www.openbsd.org/papers/eurobsdcon2016-utf8.pdf) (roff/mm/gpresent source code (https://www.openbsd.org/papers/eurobsdcon2016-utf8.roff)) Stefan Sperling, OpenBSD meets 802.11n (https://www.openbsd.org/papers/eurobsdcon2016-openbsd-11n.pdf) Antoine Jacoutot, OpenBSD rc.d(8) (https://www.bsdfrog.org/pub/events/openbsd-rcd-EuroBSDcon2016.pdf) Marc Espie, Retrofitting privsep into dpb and pkg_add (https://www.openbsd.org/papers/eurobsdcon2016-privsep.pdf) Martin Pieuchot, Embracing the BSD routing table (https://www.openbsd.org/papers/eurobsdcon2016-embracingbsdrt.pdf) I am working to build a similar website for the FreeBSD project, but there is still a lot of work to do I also managed to find the slides from the keynotes: Opening Keynote: George Neville-Neil: Looking Backwards: The coming decades of BSD (https://papers.freebsd.org/2016/EuroBSDCon/LookingBackwards.pdf) Closing Keynote: Gert Döring: Internet Attacks, Self-Governance, and the Consequences (http://www.monobsd.com/files/16_ddos_and_consequences.pptx) *** VirtualBox Shared Folders on FreeBSD: progress report (https://kernelnomicon.org/?p=650) In the past month or so, VirtualBox in the FreeBSD ports tree got bumped to version 5, which while bringing new features, did cause a regression in Shared Folders. FreeBSD developer gonzo@ (Oleksandr Tymoshenko) has been tackling this issue in recent days and provides us with a look behind the curtain at the challenges involved. Specifically he started by implementing the various needed VOPs: “lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write.” He then continues with details about how complete this is: ““Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host's md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.” The bigger issue faced is with the rename VOP though: I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task. + We wish him luck in getting those issues solved. I know quite a few of our users rely on shared folders as well. FreeBSD News Issue #1 (http://support.rossw.net/FreeBSD-Issue1.pdf) Issue #1 of FreeBSD News, from summer of 1997 Contains an article by Yahoo! co-founder David Filo about their early use of FreeBSD, on 100mhz Pentium machines with 64MB of ram Java Development Kit 1.0.2 ported to FreeBSD What is FreeBSD? Running the world's busiest FTP site (cdrom.com) on FreeBSD Xi Graphics announces the release of CDE Business Desktop, the first and only integrated desktop for FreeBSD, on AcceleratedX, a fully supported commercial grade X display server Get FreeBSD 2.2.2 Today! *** Beastie Bits Call for testing: newly MPSAFE nvme(4) (http://mail-index.netbsd.org/current-users/2016/09/21/msg030183.html) Thinking about starting a BUG in Indianapolis, IN USA (http://lists.dragonflybsd.org/pipermail/users/2016-September/313061.html) The cost of forsaking C: Why students still need to learn C (https://medium.com/bradfield-cs/the-cost-of-forsaking-c-113986438784#.o2m5gv8y7) OpenBSD (U)EFI bootloader howto (https://blog.jasper.la/openbsd-uefi-bootloader-howto/) Michael Lucas sets his eyes on OpenBSD's web stack for his next book (http://blather.michaelwlucas.com/archives/2780) LibreSSL 2.5.0 released (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.5.0-relnotes.txt) OPNsense 16.7.5 released (https://opnsense.org/opnsense-16-7-5-released/) Feedback/Questions Jonas - ZFS on DO (http://pastebin.com/XeJhK0AJ) Ricardo - OpenBSD Encrypted Disk (http://pastebin.com/Z9JRjcvb) WiskerTickle - Storage Benchmark (http://pastebin.com/XAD0UevP) Phil - Thanks (http://pastebin.com/N52JhYru) Luis - Misc Questions (http://pastebin.com/57qS0wrx) ***
This week on BSDNow, we're going to be hearing about Allan's trip to EuroBSDCon, plus an Interview about “Bro on BSD”! Stay tuned, for your place to This episode was brought to you by Headlines EuroBSDCon 2016 Wrapup Ollivier Robert's Photos from EuroBSDCon (https://assets.keltia.net/photos/EuroBSDCon-2016/) Get your BSDNow die-cut stickers (http://www.jupiterbroadcasting.com/stickers/) NetBSD for newbies - Develop your own Power PC (http://discusscomputerx.blogspot.com/2016/09/netbsd-for-noobies-your-power-laptop.html) We don't get to feature too many stories on NetBSD being deployed as a Power PC (Not PowerPC, you know, a Powerful “PC”), so we jumped at this one. Specifically it starts off with some of the pre-req's that you'll need to get started, such as NetBSD 7.0.1 / amd64, along with some information about which wireless nics you may be using. (NetBSD like other BSD's will give a driver based device name for network interfaces) From there, instructions on how to write your WPA_supplicant config are provided, in order for us to fetch the NetBSD sources and convert to their -STABLE branch. After doing a CVS checkout of the sources, he then provides a walkthrough of doing a kernel compile / install, however it mentions changing the config, but doesn't provide an example of what options were changed. Perhaps to remove drivers we don't need? At this point the rest of the “desktop” setup is pretty straight forward. Some packages are added such as openbox, lxappearance, firefox, etc. To get working sound, firefox requires pulseaudio, which in turn needs dbus, so instructions on getting that service up and running are provided as well. When it's all said and done, you'll end up with your shiny new NetBSD -STABLE desktop (or laptop), bragging rights achieved! *** More about OpenSMTPD 6.0.0 (https://www.poolp.org/tech/posts/2016/09/12/opensmtpd-6-0-0-released/) OpenSMTPd 6.0.0 has just been released “and it's quite different from former releases.” “Unlike most of our releases, it comes out with almost no new feature.”, “Turns out most of the changes are not visible.” Changelog: new fork+reexec model so each process has its own randomized memory space logging format has been reworked a "multi-line response" bug in the LMTP delivery backend has been fixed connections concurrency limits have been bumped artificial delaying in remote sessions have been reduced dhparams option has been removed dhe option has been added, supporting auto and legacy modes smtp engine has been simplified various cosmetic changes, code cleanup and documentation improvement “The OpenSMTPD bootstrap process was quite simple: Upon executation, the parent process would read configuration, build a memory representation of it and would then create a bunch of socketpair() before fork()-ing all of its child processes.” The problem is that this does not take advantage of the new address randomization feature. Each child will have the same memory layout, copied from the parent process “So deraadt@ suggested that if OpenSMTPD would not just fork() children but instead fork() them and reexecute the smtpd binary, then each of the children would have its own randomized memory space.” “The idea itself is neat, however not so trivial to implement because when we reexec the whole "inherit configuration and descriptors" part goes away. It's not just fork and exec, it's fork and exec and figure a way for the parent to pass back all the information and descriptors back to the new post-fork instance so it is the new instance that allocates memory and decides where the information goes.” *** Upgrade a FreeBSD 10.3 Installation with ZFS on Root and Full Disk Encryption to 11.0 (http://ftfl.ca/blog/2016-09-17-zfs-fde-one-pool-conversion.html) While FreeBSD 11.0 is not out yet, Joseph Mingrone has helped me work out and test the instructions for upgrading a FreeBSD 10.3 ZFS on full disk encryption setup (bootpool + zpool) to the new GELIBoot feature, which does not require any unencrypted partitions, just the 128kb bootcode Note: Do not upgrade to FreeBSD 11.0 yet. While some images have landed on the FTP server, they do not contain the final openssl fix and are going to be recreated. Currently, GELIBoot does not support key files, so the first step is to reencrypt the master key with only a passphrase. Next, to avoid GELIBoot picking up encrypted partitions that it does not support, or partitions you do not want decrypted at boot, only partitions with the GELIBoot flag are decrypted, so set the flag on your root partition Then, move the loader, kernel, and other files into /boot on the root filesystem, instead of them living on the bootpool. This allows the kernel to be versioned with boot environments, and is the main purpose of this work Then, install the newer gptzfsboot, as this is required to support GELIBoot The old 2gb bootpool partition is then purposely mislabeled as freebsd-vinum, so it is not picked up by the boot blocks. Later, if the upgrade is successful, this partition can be deleted, and used as addition swap or something In order to boot correctly, you want all boot environments to have the ‘canmount' ZFS property set to ‘noauto' Thank you to Joseph for taking the time to prod me for the information required to write this up, and for testing it and finding all of the issues *** Interview - Michael Shirk - mshirk@daemon-security.com (mailto:mshirk@daemon-security.com) / @shirkdog (https://twitter.com/shirkdog) Running Bro on BSD *** News Roundup FreeBSD based distro for virtual hosting platform and appliance (https://clonos.tekroutine.com/) An interesting new FreeBSD-based project as shown up online, called “ClonOS”, which bills itself as a “free open-source FreeBSD-based platform for virtual environments creation and management” It looks to be leveraging an impressive list of technologies, including Bhyve, Xen, Jails and CBSD / Puppet for management tasks. Among its list of features: ZFS features support; VM cloning, export, import Ethernet SoftSwitch for separated networking jails for lightweight container VNC terminal for VM/containers Templates for VM/containers Configuration management/helpers Multi-node operation Multi-Node? Color me intrigued! Right now it appears to be under heavy development, but we'll reach out to the developer to see if we can get an interview lined up at some point! The Raspberry PI Platform and The Challenges of Developing FreeBSD (https://bsdmag.org/oleksandr_rybalko/) BSDMag recently did an interview with FreeBSD developer Olesandr Rybalko! Oleksandr lives in the Ukraine, and while you may not have heard of him, he has worked on some cool projects for FreeBSD including the new “vt” console driver (Which a lot of people are using now), and ARM/MIPS support. The interview covers some of the work he's done to get the PI support working with FreeBSD: I think, my main help here was a USB OTG driver, which I wrote before for another device (Ralink RT3052), then port it to R-Pi. But it was rewritten by Hans Peter Selasky. I do not know so much about USB as Hans knows. Another useful part of my help is Xorg support. I did a simple Xorg video driver which uses framebuffer exported by virtual terminal subsystem. That is help to many guys to start use RPi as a simple desktop system. He was also asked the question “Why would FreeBSD be good fit for ARM?” FreeBSD is very powerful as a network server. All modern network features in one box, with very fast processing. Another good side of FreeBSD is modularity. It is not required to write code to use some driver that was already written for another system, you can just define it in configuration files (kernel config, kernel hints, FDT). So if you want build a nice, R-Pi based, home server – use FreeBSD. If you want to play with devices attached to R-Pi's GPIO – use FreeBSD. He also discusses his work on the ZRouter project, which is a very light-weight platform for tiny routers / embedded devices. But lastly the RPI comes up again, specifically asking him how interested individuals can get involved. Specifically the wiki.freebsd.org is a great reference point for those intested in getting started with FreeBSD on embedded. The warm community is also a plus! Trying out the FreeBSD powered TrueOS (http://www.phoronix.com/scan.php?page=news_item&px=TrueOS-First-Spin) The folks over at Phoronix have done an early look at the new TrueOS desktop images and given some of their thoughts. First up he gives props to the installer, noting that: The TrueOS desktop installer is basically the same as from the PC-BSD days, just re-branded. Still one of the easiest BSD graphical installers I've dealt with and makes it a breeze for setting up a FreeBSD-on-ZFS system by default. After that they took it for a minimal spin, and thing mostly seem to be working. He mentions some of the default apps (Such as qupzilla and trojita) aren't their favorite, but Lumina has come quite a ways for 1.0, despite a few rough edges still. (We are in the process of changing those default e-mail / browser apps) Lastly the article mentions that it's time to do a more full BSD round-up to see the state of installation of them, which we happen to have next! Trying out 8 BSDs on a modern PC (http://www.phoronix.com/scan.php?page=article&item=trying-8-bsds&num=1) First up was TrueOS again, which no major changes there, easy install and done. From there he tries out DragonFlyBSD, which he mentions that while the installer isn't as easy, it is still one of his favorite BSD's, working with all the hardware they've thrown at it. Next up was GhostBSD, which also has an Easy-To-Use graphical installer similar to TrueOS that made it quick to get loaded and up to the Mate desktop. Also tested was FreeBSD 11.0-RC2, which he mentions was easy to installed, and once done then ‘pkg' could be used to easily get the setup he wanted setup. Turning over to page two we get to the naughty list of BSD's he had troubles with. First up was OpenBSD which he tried 6.0. After installation and first boot, the display kept ‘disappearing' which meant he couldn't get IP information to try SSH'ing into the box. Perhaps a display driver error? NetBSD 7 was up next, where the installer couldn't get past a root device prompt. Most likely trouble finding the install media, which was the same story with MightnightBSD as well. Also tested was “PacBSD” (Formerly ArchBSD) which he did manage to get installed, but not after major fighting with the process. After the process he ran into some issues getting packages up and running, but mentions it may have been bad timing due to them moving to a new server at the time. *** IllumOS imports a modified FreeBSD boot loader to replace grub 0.97 (https://www.listbox.com/member/archive/182181/2016/09/sort/time_rev/page/1/entry/0:1/20160923124232:B7978ED4-81AC-11E6-A6DA-02E3F010038B/) Toomas Soome's work to port the FreeBSD boot loader to IllumOS has been merged into illumos-gate, the upstream repository for all IllumOS distributions Toomas' work has also resulted in a number of commits to FreeBSD, and code sharing in both directions Toomas helped me a lot with the building of the ZFS boot environment listing menu, even though on IllumOS they use a configuration file to list the BEs, rather than interrogating the live zpool like we do in FreeBSD Toomas' work to improve msdosfs and the block cache to speed up booting IllumOS also greatly helped FreeBSD This work means IllumOS can now boot from a RAID-Z (the old grub they used could not), and if the work Toomas has done on FreeBSD is any indication, support for almost all other zpool features is also on the way This work also sets IllumOS on a path to eventually having UEFI boot as well It is good to see this work happening, FreeBSD technology being reused elsewhere, but also the improvements being made for IllumOS are coming back to FreeBSD, often landing upstream first, to make merging them into IllumOS easier. The mailing list post describes how to convert existing systems away from grub, as well as how to opt to remain on grub for a while longer. Grub 0.97 is expected to be removed from IllumOS within a year. *** Beastie Bits A demo of booting CentOS and Windows 10 in FreeBSD Bhyve through VNC headless (https://www.youtube.com/watch?v=8YQQfXqtyaA) This year's anemic output (http://blather.michaelwlucas.com/archives/2762) “PAM Mastery” ebook now out (http://blather.michaelwlucas.com/archives/2771) How-to Install OpenBSD 6.0 plus XFCE desktop and basic applications (https://www.youtube.com/watch?v=oC5D9fenQBs) *** Feedback/Questions Piotr - LibreBoot (http://pastebin.com/yniniNpV) Alan - FreeBSD and PC-BSD (http://pastebin.com/dCNX0yF7) Eduardo - Newcomers (http://pastebin.com/LndNeAYb) Greg - ZFS ACL's (http://pastebin.com/F0y6L6NK) Brian - Laptop Recs (http://pastebin.com/sqMPJGMM) ***
This week on BSDNow, Allan is currently at EuroBSDCon! However due to the magic of video (or time travel), you still get a new episode. (You're Welcome!). Stay tuned This episode was brought to you by Headlines Performance Improvements for FreeBSD Kernel Debugging (http://backtrace.io/blog/blog/2016/08/25/improving-freebsd-kernel-debugging/) “We previously explored FreeBSD userspace coredumps (http://backtrace.io/blog/blog/2015/10/03/whats-a-coredump). Backtrace's debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them.” “A kernel core is typically only generated in exceptional circumstances. Unlike userspace processes, kernel routines cannot fault without sacrificing the machine's availability. This means things like page faults and illegal instructions inside the kernel stop the machine, instead of just one process. At that point, in most cases, it is only usable enough to inspect its state in a debugger, or to generate a core file.” No one likes it when this happens. This is why backtrace.io is focused on being able to figure out why it is happening “A FreeBSD kernel core file can be formatted in several different ways. This depends on which type of dump was performed. Full core dumps are ELF files, similar in structure to userspace core files. However, as RAM size grew, this became more difficult to manage. In 2006, FreeBSD introduced minidumps, which are much smaller without making the core file useless. This has been the default dump type since FreeBSD 6.0.” The article goes into detail on the minidump format, and some basic debugging techniques “Libkvm will first determine whether the virtual address lies within the kernel or direct maps. If it lies in the kernel map, libkvm will consult the page table pages to discover the corresponding physical address. If it lies in the direct map, it can simply mask off the direct map base address. If neither of these applies, the address is illegal. This process is encapsulated by vatopa, or “virtual address to physical address”. Once the physical address is determined, libkvm consults the core file's bitmap to figure out where in the core file it is located.” “minidumps include a sparse bitmap indicating the pages that are included. These pages are dumped sequentially in the last section. Because they are sparse in a not entirely predictable way, figuring the offset into the dump for a particular physical address cannot be reduced to a trivial formula.” The article goes into detail about how lookups against this map are slow, and how they were improved “For typical manual debugger use, the impact of this change isn't noticeable, which is probably why the hash table implementation has been in use for 10 years. However, for any automated debugging process, the extra latency adds up quickly.” “On a sample 8GB kernel core file (generated on a 128GB server), crashinfo improves from 44 seconds to 9 seconds, and uses 30% less memory” “Backtrace began shipping a version of this performance improvement in ptrace in February 2016. This enables us to also offer significantly faster tracing of FreeBSD kernel cores to customers running current and older releases of FreeBSD. On July 17, 2016, our work improving libkvm scaling was committed to FreeBSD/head. It will ship with FreeBSD 12.0.” *** OpenBSD gunzip pipeline tightening (https://www.mail-archive.com/tech@openbsd.org/msg34035.html) OpenBSD has rethought the way they handle package signing Changing from: 1/ fetch data -> 2/ uncompress it -> 3/ check signature -> 4/ process data To: 1/ fetch data -> 2/ check signature -> 3/ uncompress -> 4/ process data “The solution is to move the signature outside of the gzip header” “Now, Since step 1/ is privsep, as long as step 2 is airtight, 3/ and 4/are no longer vulnerable” Guidelines: small, self-contained code to parse simple gzip headers signify-style signature in the gzip comment. Contains checksums of 64K blocks of the compressed archive don't even think about passing the original gzip header through use as a pipeline step: does not need to download full archive to use it, and never ever pass any data to the gunzip part before it's been verified. “Note that afaik we haven't had any hole in our gunzipping process. Well… waiting for an accident to happen is not how we do things. Hopefully, this should prevent future mishaps.” *** OpenVPN On FreeBSD 10.3 (http://ramsdenj.com/2016/07/25/openvpn-on-freebsd-10_3.html) “While trying to setup OpenVPN, I noticed there was no up-to-date information with correct instructions. OpenVPN uses EasyRSA to setup keys, it has recently been changed in version 3. As a result of this, the old steps to configure OpenVPN are no longer correct. I went through the process of setting up a VPN using OpenVPN on FreeBSD 10.3.” I know FreeBSD developer Adrian Chadd complained about this exact problem when he was trying to setup a VPN before attending DEFCON The tutorial walks through the basic steps: Install the needed software Configure EasyRSA Create a CA Generate keys and DH params OpenVPN Server Config OpenVPN Client Config Starting the daemon It even finishes off with bonus instructions on Port Forwarding, Firewalls, and Dynamic DNS *** lsop (https://github.com/606u/lsop) LSOP is the tool a bunch of users have been asking for “a FreeBSD utility to list all processes running with outdated binaries or shared libraries” How does it work? “lsop iterates over all running processes and looks through memory-mapped files with read + execute access; then it checks if those files are still available or have been modified/deleted.” How would you use it? After installing an system update (that doesn't require a reboot to update the kernel), or upgrade your packages, you still need to know which daemons need to be restarted to use the patched libraries and binaries This tool gives you that list Thanks to Bogdan Boyadzhiev for writing this much needed tool *** News Roundup OpenBSD 2016 Fundraising Campaign (http://www.openbsdfoundation.org/campaign2016.html) The OpenBSD fund-raising campaign has given us a status update on the state of 2016. They start by giving us a re-cap of previous years: “2015 was a good year for the foundation financially, with one platinum, one gold, four silver and 3 bronze donors providing half of our total donations. 680 individuals making smaller contributions provided the other half. While the total was down significantly after 2014's blockbuster year, we again exceeded our goal.” As of Sept 5th, they were at approx $115k out of a total goal of 250k. If you are an OpenBSD user, remember to contribute before the end of the year. Small amounts help, and the money of course goes to great causes such as hackathons and running the OpenBSD infrastructure. Update firewall Bad Countries (https://github.com/KaiLoi/update-fw-BC) Network and Systems admins know, sometimes when all else fails you need to break out the HUGE ban-hammer. In this case sometimes entire countries get put on the excrement list until the attacks stop. We have a handy GitHub project today, which will assist you in doing exactly that, enter update-fw-BC. (Update firewall by country) This perl script may be your savior when dealing with instances that require major brute force. It specifically works with IPFW, PF and IPTABLES, which will allow it to run across a variety of BSD's or even Linux. It will ingest a list of IP's that you feed it (perhaps from another tool such as sshguard) and determine what block the IP belongs to, and match according to country. Detailed setup instructions for the various firewalls are included, and some instructions for FreeBSD, although using it on OpenBSD or other $BSD should also be easy to adapt. *** More utilities via moreutils (https://distrowatch.com/weekly.php?issue=20160822#tips) In most BSDs, the “core” set of utilities and commands are just part of the base system, but on Linux, they are usually provided by the “coreutils” package. However, on Linux and now FreeBSD, there is a “moreutils” package, that provides a number of interesting additional basic utilities, including: chronic: Run a task via crontab, and only generate output if the task fails combine: binary AND two text files together, only displaying lines that are in both files errno: look up the text description of a specific error number ifdata: parse out specific information from ifconfig ifne: if-not-empty, only run a command if the output of the pipe is not blank isutf8: determine if a file or stdin contains utf8 lckdo: execute a command with a lock held, to prevent a second copy from spawning mispipe: return the exit code of the first command in a pipe chain, rather than the last parallel: run multiple jobs at once pee: tee standard input to multiple pipes sponge: write standard input to a file, allows you to overwrite a file in place: sort file | sponge file ts: add a timestamp to each line of standard input vidir: edit a directory in vi, great for bulk renames vipe: insert vi into a pipe, edit the content before it is passed to the next command zrun: uncompress the arguments before passing them. Like gzless and friends, but for any command Just goes to show the power of the original UNIX philosophy, chaining together a bunch of small useful tools to do really powerful things *** OpenBSD: SNI support added to libtls, httpd in –current (http://undeadly.org/cgi?action=article&sid=20160823100144) libtls, LibreSSL's improved API to replace the OpenSSL standard, now has a set of functions to implement SNI (Server Name Indication) Until a few years ago, each different SSL/TLS enabled website required a unique IP address, because typical HTTP Virtual Hosting (differentiating which content to serve based on the Host header in the HTTP request), didn't work because the request was encrypted. Finally the TLS standard was updated to include the hostname of the site the user is requesting in the TLS handshake, so the server can return the corresponding certificate, and multiple TLS enabled websites can be hosted on a single IP address The new API includes the ability to provide additional keypairs (via tlsconfigaddkeypair{file,mem}()) And allow the server to determine what servername the client requested viatlsconnservername() This is much easier to use, and therefore safer and less error prone, than the OpenSSL API The libtls API is used in a number of OpenBSD tools, including the httpd *** Beastie Bits Shawn Webb of HardenedBSD joins the OPNSense Core Team (https://opnsense.org/new-core-team-member/) How to install 2.11 BSD on a (simulated) PDP11 (http://vak.ru/doku.php/proj/pdp11/211bsd) OpenBSD Puffy needlepoint pixelart (https://nemessica.tintagel.pl/blog/OpenBSD-Puffy/) PulseAudio has been removed from dports (DragonFly BSD) (http://lists.dragonflybsd.org/pipermail/users/2016-August/313010.html) pfSense 2.4 pre-alpha available for testing, based on FreeBSD 11.0 (https://blog.pfsense.org/?p=2118) Call for Testing - Bhyve HDA Sound Emulation (https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-September/004700.html) *** Feedback/Questions Matthew - ZFS Hole Birth (http://pastebin.com/CrZiDAF0) Hunter - systemd-mount (http://pastebin.com/GztjY4wz) Anonymous - Cool'n'quiet (http://pastebin.com/gG4j4RCi) Nathan - Datacenter (http://pastebin.com/9XgPzMM9) Chuck - OpenBSD w/DO (http://pastebin.com/FM2xYcxh) ***
This week on BSDNow! We've got Netflix + FreeBSD news to discuss, always a crowd pleaser, that plus EuroBSDCon is just around the corner. Stick around for your place This episode was brought to you by Headlines Protecting Netflix Viewing Privacy at Scale, with FreeBSD (http://techblog.netflix.com/search/label/FreeBSD) This blog post from Netflix tells the story of how Netflix developed in-kernel TLS to speed up delivery of video via HTTPS Since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs - from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016. We contribute to this effort on the software side by optimizing every aspect of the software for our unique use case - in particular, focusing on the open source FreeBSD operating system and the NGINX web server that run on the OCAs. In the modern internet world, we have to focus not only on efficiency, but also security. There are many state-of-the-art security mechanisms in place at Netflix, including Transport Level Security (TLS) encryption of customer information, search queries, and other confidential data. We have always relied on pre-encoded Digital Rights Management (DRM) to secure our video streams. Over the past year, we've begun to use Secure HTTP (HTTP over TLS or HTTPS) to encrypt the transport of the video content as well. This helps protect member privacy, particularly when the network is insecure - ensuring that our members are safe from eavesdropping by anyone who might want to record their viewing habits. The goal is to ensure that your government, ISP, and wifi sniffing neighbour cannot tell which Netflix videos you are watching Netflix Open Connect serves over 125 million hours of content per day, all around the world. Given our scale, adding the overhead of TLS encryption calculations to our video stream transport had the potential to greatly reduce the efficiency of our global infrastructure. We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-GCM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously - as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method. All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance. We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L). Netflix and NGINX had previously worked together to improve our HTTP client request and response time via the use of sendfile calls to perform a zero-copy data flow from storage (HDD or SSD) to network socket, keeping the data in the kernel memory address space and relieving some of the CPU burden. The Netflix team specifically added the ability to make the sendfile calls asynchronous - further reducing the data path and enabling more simultaneous connections. However, TLS functionality, which requires the data to be passed to the application layer, was incompatible with the sendfile approach. To retain the benefits of the sendfile model while adding TLS functionality, we designed a hybrid TLS scheme whereby session management stays in the application space, but the bulk encryption is inserted into the sendfile data pipeline in the kernel. This extends sendfile to support encrypting data for TLS/SSL connections. We tested the BoringSSL and ISA-L AES-GCM implementations with our sendfile improvements against a baseline of OpenSSL (with no sendfile changes), under typical Netflix traffic conditions on three different OCA hardware types. Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline - increasing performance by up to 30%, depending on the OCA hardware version. We chose the ISA-L cipher implementation, which had slightly better results. With these improvements in place, we can continue the process of adding TLS to our video streams for clients that support it, without suffering prohibitive performance hits. If you would like more detail, check out the papers from AsiaBSDCon 2015 (https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf) and the updated one from 2016 (https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf) *** OpenBSD on HP Stream 7 (http://www.tedunangst.com/flak/post/OpenBSD-on-HP-Stream-7) Recent events have rocked the mobile computing world to its core. OpenBSD retired the zaurus port, leaving users in desperate need of a new device. And not long before that, Microsoft released the Anniversary Update to Windows 10, but with free space requirements such that it's nigh impossible to install on cheap 32GB eMMC equipped devices such as the HP Stream series, leaving users searching for a new lightweight operating system. With necessity as both mother and father, the scene is set for a truly epic pairing. OpenBSD on the HP Stream 7. The HP Stream line is a series of budget computers in a couple form factors. The Stream 11 is a fairly typical netbook. However, the Stream 7 and 8 are tablets. They look like cheap Android devices, but inside the case, they're real boys, er PCs, with Intel Atom CPUs. To install OpenBSD on such a device, we need a few parts. Obviously, the tablet itself. There's a dearth of ports on these things, but there is a micro USB port. Attaching anything useful requires an OTG “on the go” cable that creates a type A port. Attaching more than one useful thing requires a mini hub. And completing the install requires one each USB stick, keyboard, and network adapter. First, we need to prep the machine to boot from USB. Actually, before doing anything, make sure you have a full charge. It's going to be battery only from here on out. Plug everything in. Flash drive, keyboard, and network into the hub, hub into the OTG cable, cable into the port on top of the Stream. Turn on the machine while holding the volume down button. This launches a mini menu from which we can enter the BIOS. There's a little on screen keyboard in the corner, so this can be done even without a keyboard attached, but the USB keyboard should work. We need to change two settings in the boot section. First, turn off secure boot. Second, switch boot order to prefer USB. Save and exit. The first reboot reveals a confirmation screen checking that we really want to disable secure boot. We must enter a PIN and press enter. Enter the PIN shown on the screen and press enter. And we are go. Then boot up OpenBSD from the USB drive Ted then works there a number of kernel panics and device driver issues, but after disabling ACPI and IntelDRM, the device boots OpenBSD. Of course, there's no X at this point. And definitely no touch screen. And no internal networking. However, by keeping our USB hub attached, we can drive the console and access the network. At least until the battery is depleted, even if we have no way of knowing how long that will be since we disabled all the ACPI devices, which also means no suspend or resume. With some xorg.conf hacking, he did get Xorg working *** DragonflyBSD steps towards base LibreSSL (http://lists.dragonflybsd.org/pipermail/commits/2016-September/624493.html) Project: DragonFlyBSD / Switch base to use private LibreSSL libraries (http://freshbsd.org/commit/dfbsd/304ca408000cd34559ef5319b4b5a6766d6eb35b) DragonFly BSD adopts uses of LibreSSL (http://undeadly.org/cgi?action=article&sid=20160911231651) The number of projects beginning to switch over to LibreSSL is growing and it appears we can now throw DragonFly into that camp. Following something that sounds vaguely familiar (Allan!) DFLY is now creating “private” LibreSSL libraries which are only linked against by base system binaries. For the moment OpenSSL is still built, primarily so that various ports and 3rd party apps can continue to function as before. A NO_OPENSSL option has also been added, but doesn't really do much (yet), since it'll still build and install headers / libraries even if set. *** OpenBSD g2k16 Hackathon g2k16 Hackathon Report: Antoine Jacoutot on Binary Patches (http://undeadly.org/cgi?action=article&sid=20160911012316) g2k16 Hackathon Report: Matthieu Herrb on xenodm (http://undeadly.org/cgi?action=article&sid=20160911231712) g2k16 Hackathon Report: Vincent Gross on iked(8), armv7 and sys/netinet[6] (http://undeadly.org/cgi?action=article&sid=20160911000337) g2k16 Hackathon Report: Florian Obser on httpd, networking, acme-client, and more (http://undeadly.org/cgi?action=article&sid=20160911000052) g2k16 Hackathon Report: Jasper Lievisse Adriaanse on ddb(4) and more (http://undeadly.org/cgi?action=article&sid=20160909012520) g2k16 Hackathon Report: Christian Weisgerber on gettext progress, RTC work, removing kernel cruft (http://undeadly.org/cgi?action=article&sid=20160908002430) g2k16 Hackathon Report: Brent Cook on Chromebooks, crypto, and more (http://undeadly.org/cgi?action=article&sid=20160907131655) g2k16 Hackathon Report: Ted Unangst on doas, signify, code removal (http://undeadly.org/cgi?action=article&sid=20160906230610) g2k16 Hackathon Report: Marc Espie on package signing evolution (http://undeadly.org/cgi?action=article&sid=20160905235911) g2k16 Hackathon Report: Adam Wolk on ports, wireless drivers and more (http://undeadly.org/cgi?action=article&sid=20160906004915) g2k16 Hackathon Report: Mike Larkin on vmm + vmd progress (http://undeadly.org/cgi?action=article&sid=20160905134009&mode=expanded) *** News Roundup OpenBSD (with encrypted softraid) on the Chromebook Pixel (https://jcs.org/notaweblog/2016/08/26/openbsd_chromebook/) Looking for a Laptop to make your OpenBSD road-warrior? If so, we have a great blog tutorial on getting OpenBSD setup on the Chromebook Pixel with encrypted softraid! Author Joshua Stein gives us a very verbose look at how to install and dial-in the laptop perfectly. But first for those wondering about the hardware in the pixel: The Chromebook Pixel LS (2015) has an Intel Core i7 processor (Broadwell) at 2.4Ghz, 16Gb of RAM, a 2560x1700 400-nit IPS screen (239ppi), and Intel 802.11ac wireless. It has a Kingston 64Gib flash chip, of which about 54Gib can be used by OpenBSD when dual-booting with a 1Gb Chrome OS partition. Due to this being a chromebook with seaBIOS, some manual key-press trickery will be required to initially get the OpenBSD Installer up and running. From here you'll want to pay special close attention to the disk partitioning. In particular Joshua will show us how to shrink the existing encrypted /home that ChromeOS uses, keeping the dual-boot intact. This will become important if you ever plan on updating the device. From here, we move back to a more traditional setup, but with the added bonus of doing a soft-raid setup. But the fun isn't over yet! If you want to make OpenBSD the default boot, that'll require cracking the lid on the device and removing a special pink write-protect screw. And of course if you want to remove the default splash-screen image, Joshua has you covered as well, although some flashrom magic will be required. At this point you are nearly done. Final details on enabling specific bits of hardware are discussed. Most things work, apart from Audio and Bluetooth as of right now. *** doas mastery (http://www.tedunangst.com/flak/post/doas-mastery) “doas” mastery - Paging MWL! Our buddy Ted Unangst has written up a great ‘mastery' guide of the doas command, which can come in handy if you are among the un-initiated in doas land. UNIX systems have two classes of user, the super user and regular users. The super user is super, and everybody else is not. This concentration of power keeps things simple, but also means that often too much power is granted. Usually we only need super user powers to perform one task. We would rather not have such power all the time. Think of the responsibility that would entail! Like the sudo command, doas allows for subdivision of super user privileges, granting them only for specific tasks. He starts with the basic doas.conf setup, which starts with an empty config file The doas config is much like a pf ruleset, the default is to block everything > We add the root rule second because doas evaluates rules in a last match manner. root is in the wheel group, so the first rule will match, and then we need to override that with a second rule. Remember to always start with general rules, then make them more specific. *** iXsystems iXsystems to host MeetBSD (https://www.ixsystems.com/blog/ixsystems-host-meetbsd-california-2016-uc-berkeley/) FreeBSD Foundation Welcomes New Board Members New Board Members (https://www.freebsdfoundation.org/blog/freebsd-foundation-welcomes-new-board-members/) The FreeBSD Foundation has added two new board members Interview with Kylie Liang (https://www.freebsdfoundation.org/blog/new-board-member-interview-kylie-liang/) Kylie will focus on representing FreeBSD at conferences and businesses in China I live in China. There, I can act as a bridge between Chinese companies and the FreeBSD community to help drive FreeBSD adoption. Through my leadership role in the FreeBSD Foundation, I will help promote FreeBSD in China and also represent the Foundation at conferences and events in my region. Kylie leads the team the ensures FreeBSD runs well on Hyper-V and Azure, including providing commercial support for customers who run FreeBSD or FreeBSD based appliances on the Azure Cloud I joined Microsoft and started to lead the project called FreeBSD Integration Service to get FreeBSD running well on Hyper-V and Azure. To promote our work and to understand the FreeBSD ecosystem, I started to participate in FreeBSD events where I was inspired by this technical community. Interview with Philip Paeps (https://www.freebsdfoundation.org/blog/new-board-member-interview-philip-paeps/) Philip started with FreeBSD in the early 2000s and got his commit bit in 2004 The patches I submitted to make ACPI and input devices work on that laptop led to a src commit bit in 2004. While I haven't worked on ACPI or input devices since, I have been contributing to different areas of the kernel. Taking up maintainership of some ports I cared about also got me a ports commit bit after some time. Philip will continue to help run EuroBSDCon, but is also spreading the word about FreeBSD in India and Africa Primarily, I think I can be useful! I attend (and organize) a number of conferences around the world every year, particularly in regions that have a mostly “stealthy” FreeBSD community. While I clearly don't need to be on the FreeBSD Foundation board to advocate for FreeBSD, joining as a director will provide an additional asset when working in areas of the world where organizational affiliations are meaningful. Philip has also developed network drivers and various other bits and pieces, and has extensive experience working with and for hardware vendors and appliance vendors Despite intending to eventually contribute their code to the FreeBSD Project as open source, many hardware vendors still find it very difficult to engage directly with the FreeBSD development community. The Foundation helps bridge that gap and helps facilitate collaboration between commercial vendors and the FreeBSD community. I hope to make FreeBSD more visible in regions of the world where it is historically under-represented. I expect I will be attending even more conferences and getting myself invited to even more organizations. more, less, and a story of typical Unix fossilization (https://utcc.utoronto.ca/~cks/space/blog/unix/MoreAndUnixFossilization) Chris Siebenmann from the University of Toronto digs into the history of the difference between ‘less' and ‘more' In the beginning, by which we mean V7, Unix didn't have a pager at all. That was okay; Unix wasn't very visual in those days, partly because it was still sort of the era of the hard copy terminal. Then along came Berkeley and BSD. People at Berkeley were into CRT terminals, and so BSD Unix gave us things like vi and the first pager program, more (which showed up quite early, in 3BSD, although this isn't as early as vi, which appears in 2BSD). Calling a pager more is a little bit odd but it's a Unix type of name and from the beginning more prompted you with '--More--' at the bottom of the screen. All of the Unix vendors that based their work on BSD Unix (like Sun and DEC) naturally shipped versions of more along with the rest of the BSD programs, and so more spread around the BSD side of things. However, more was by no means the best pager ever; as you might expect, it was actually a bit primitive and lacking in features. So fairly early on Mark Nudelman wrote a pager with somewhat more features and it wound up being called less as somewhat of a joke. In a sane world, Unix vendors would have either replaced their version of more with the clearly superior less or at least updated their version of more to the 4.3 BSD version. Maybe less wouldn't have replaced more immediately, but certainly over say the next five years, when it kept on being better and most people kept preferring it when they had a choice.” + “This entire history has led to a series of vaguely absurd outcomes on various modern Unixes. On Solaris derivatives more is of course the traditional version with source code that can probably trace itself all the way back to 3BSD, carefully updated to SUS compliance. Solaris would never dream of changing what more is, not even if the replacement is better. Why, it might disturb someone. Oddly, FreeBSD has done the most sensible thing; they've outright replaced more with less. There is a /usr/bin/more but it's the same binary as less and as you can see the more manpage is just the less manpage. OpenBSD has done the same thing but has a specific manpage for more instead of just giving you the less manpage. So, now you can see why I say that less is more, or more, or both, at several levels. less is certainly more than more, and sometimes less literally is more (or rather more is less, to put it the right way around). Beastie Bits PC-BSD listed in the top 8 'best' alternatives to Windows 10 (http://www.computerworlduk.com/galleries/operating-systems/-free-alternatives-windows-10-3639433/) Creating a quick DNS server with a Rapsberry Pi2 and FreeBSD 11.0-RC1 (http://bsdimp.blogspot.co.uk/2016/08/creating-quick-dns-server-with.html) Dual Boot OpenBSD and Linux + UEFI (https://bsdlaptops.wordpress.com/2016/03/07/vaio-pro-11-part-2/) DesktopBSD 2.0 various versions available (Gnome, Lumina, KDE, LXDE) (http://desktopbsd.boards.net/board/10/announcements) FreeBSD gets new ZFS features including: Compressed ARC (https://svnweb.freebsd.org/base?view=revision&revision=305323) and ZFS Allocation Throttle (https://svnweb.freebsd.org/base?view=revision&revision=305331) One Floppy NetBSD Distribution (https://github.com/user340/fdgw2) A Compendium of BUGs (https://github.com/q5sys/BUGtracker) Feedback/Questions Galahad - OpenBSD X setup (http://pastebin.com/b7W6NHqs) Tang - Subtitles (http://pastebin.com/P4MUs3Pa) Ivan - Zpool Options (http://pastebin.com/LQ8yTp0G) Brad - Replication Issue (http://pastebin.com/XTK5gXMU) MJ - HBA (http://pastebin.com/TdYTMSj9) ***
This week on BSDNow, we'll be talking to Diane Bruce about using it for Ham Radio Enthusiasts, the RPi3 and much more! That plus all the latest news from the week, This episode was brought to you by Headlines PC-BSD is now TrueOS (https://www.trueos.org/2016/09/01/pc-bsd-evolves-into-trueos/) If you've been watching this show the past few months, I've been dropping little hints about the upcoming rename of PC-BSD -> TrueOS. We've made that more official finally, and are asking folks to test out the software before a wider announcement this fall. For those wondering about the name change, it's been something discussed over the past few years at different times. With us beginning to move more aggressively with changes for 11.0 (and eventually 12-CURRENT), the time seemed right to have a fresh start, using it as a spring-board to introduce all the changes in both software, and development / release model. I'll be discussing more about this shift in a talk at MeetBSD2016 (Another reason for you to go), but here's some of the highlights. No longer tied to specific FreeBSD point-releases, TrueOS will instead follow a rolling-release model based upon FreeBSD -CURRENT. Special tooling and features (Such as boot-environments) make this a feasible option that we didn't have as easily in the early days of PC-BSD. In addition, TrueOS builds some things different from vanilla FreeBSD. Specifically Matt Macy's DRM and Linux Compat work, LibreSSL directly in base, built from External Toolchain (No clang in base system package) and much more. New tools have have replaced, and are in the process of replacing the legacy PC-BSD control panel as well, which allows remote operation, either via Qt GUI, or WebSockets / REST API's. I'll be talking about more as things unfold, but for now please feel free to test and let us have feedback while we push towards a more stable release. *** The Voicemail Scammers Never Got Past Our OpenBSD Greylisting (http://bsdly.blogspot.com/2016/08/the-voicemail-scammers-never-got-past.html) Peter Hansteen (That grumpy BSD guy) gives us an interesting look at how their OpenBSD grey-listing prevented spam from ever making it to their inbox. Specifically it looks like it occurred during Aug 23rd and 24th, with a particularly nasty ransomware payload destined to play havoc with Windows systems. Peter then walks us through their three-server mail setup, and how spamd is run in greylisting mode on each. The results? Nothing short of perfection: > “From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois). The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes. There were none. The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0. Zero. Not a single one made it through even to content filtering.” Not bad at all! Looks like spam-trap addresses + grey-listing is the way to go for stopping this kind of foolishness. Checkout Peter's blog post for more details, but perhaps this will encourage you to setup a similar-type system for your business. *** FreeBSD on a tiny system; what's missing (http://adrianchadd.blogspot.com/2016/08/freebsd-on-tiny-system-whats-missing.html) Adrian Chadd talks about some of the bits that are missing to make FreeBSD truly useful on small embedded devices Some of this stuff can be done now, but requires more work than it should “The first is a lack of real service management. FreeBSD doesn't have a service management daemon - the framework assumes that daemons implement their own background and monitoring. It would be much nicer if init or something similar to init could manage services and start/restart them where appropriate.” Of course, on a system with 32mb of memory, such a service manager would need to be very light weight “maybe I want to only start telnetd or dropbear/sshd whenever a connection comes in. But I'd also like to be able to add services for monitoring, such as dnsmasq and hostapd.” telnetd and sshd can be run from inetd, but often depend on special support from the daemon “The next is a lack of suitable syslog daemon. Yes, I'd like to be able to log some messages locally - even if it's only a couple hundred kilobytes of messages. I'd also like to be able to push messages to a remote service. Unfortunately the FreeBSD syslog daemon doesn't do log rotation or maximum log file sizes itself - it's done by "newsyslog" which runs out of cron. This isn't any good for real embedded systems with limited storage.” Syslog leaves much to be desired, especially in its configuration syntax, and filtering capabilities. Having it be able to detect with log files have grown beyond a reasonable size and fire off newsyslog would be very interesting “Then yes, there's a lack of a cron service. It'd be nice to have that integrated into the service management framework so things could be easily added/removed. I may just use cron, but that means cron is also always running which adds memory footprint (~1.3 megabytes) for something that is almost never actually active. When you have 32MB of RAM, that's quite a bit of wasted memory.” Systems have come back full circle, to where 32MB and 64MB are amounts of memory people expect to work with, while other people still want the system to perform well with 32 or 64 GB of memory It will be interesting to see how this balancing act plays out, trying to make the same codebase useful for extremely small and extremely large systems at the same time, while also running it on your middle of the road laptop. *** So I lost my OpenBSD FDE password (https://blog.filippo.io/so-i-lost-my-openbsd-fde-password/) “The other day I set up a new OpenBSD instance with a nice RAID array, encrypted with Full Disk Encryption. And promptly proceeded to forget part of the passphrase.” So they started a little project Goal: “We need to extract enough info from the encrypted disk and rebuild enough of the decryption algorithm to be able to rapidly try many passphrases.” The post walks through how they reverse engineered the encryption system from the source code and a hexdump of a small encrypted memory disk “Now that we know how to extract the data and how to try passphrases against it, it will be trivial to write a bruteforce tool to recover the part of passphrase I forgot.” So, rather than having to try every possible passphrase, they only had to try fuzzing around the known keyword that was their passphrase. “UPDATE: I found it! After fixing a bug or two in the brute force tool and almost losing hope, it found the right combination of forgotten word and (Italian) misspelling.” This work lead to the author recommending that OpenBSD consider strengthening the key derivation algorithm (http://marc.info/?l=openbsd-tech&m=147316661717410&w=2) used in its FDE. Rather than using a fixed number of rounds (8000 currently), do a small benchmark and determine how much work can be done in a reasonable amount of time This is what FreeBSD's GELI FDE does, targeting ‘over 2 million microseconds' of work. On my desktop i5-3570 this results in 974842 rounds. The number will likely not be the same twice because of minor variations in how long it will take in microseconds. *** Interview - Diane Bruce - db@freebsd.org (mailto:db@freebsd.org) / @Dianora_1 (https://twitter.com/Dianora_1) Ham Radio, RPi3 and more! News Roundup See Me (Michael W. Lucas) in 2016 (http://blather.michaelwlucas.com/archives/2739) Looking for a chance to interact with author Michael W Lucas in meat-space? (That sounds wrong) If so, he has posted a list of the up-coming conferences he'll be speaking at, starting with Ohio LinuxFest Oct 7-8, where he'll be giving an introduction to ZFS talk. Nov 8th, he'll also be at MUG (Michigan User Group) giving a PAM talk. Sadly, no MeetBSD for Michael this year [moment of silence], but if you are able to make it to one of the aforementioned gatherings, be sure to bring your books for autographs. We promise he doesn't bite. Much. *** It's hard work printing nothing (http://www.tedunangst.com/flak/post/its-hard-work-printing-nothing) “It all starts with a bug report to LibreSSL that the openssl tool crashes when it tries to print NULL. This bug doesn't manifest on OpenBSD because libc will convert NULL strings to ”(null)” when printing. However, this behavior is not required, and as observed, it's not universal. When snprintf silently accepts NULL, that simply leads to propagating the error.” “There's an argument to be made that silly error messages are better than crashing browsers, but stacking layers of sand seems like a poor means of building robust software in the long term.” “As soon as development for the next release of OpenBSD restarted, some developers began testing a patch that would remove this crutch from printf.” If you'd like to help with this work, see our call for volunteers from 2 weeks ago: opportunity to help: %s audit in mandoc (https://marc.info/?l=openbsd-misc&m=147059272201219&w=2) Of course, immediately things started to complain. The configure script for talloc does a number of checks (check out the additional interesting observations by TedU here) “The test checking that our snprintf function conforms to the C99 standard actually contains, at a minimum, 3 deviations from the standard. It should say “Checking for non-conformant vsnprintf”.” “Of course, we're dealing with NULL pointers, so all bets are off, but I wonder what people who expect printf NULL to work expect out of strlen? Does it return 0? Does it crash?” So, talloc decides that the system printf is no good, and it should use its own bundled implementation “After all the configure testing, eventually the build will fail, because somebody forgot to actually add the replacement object file to the Makefile.” “If the replacement function has never been used, that's hardly reassuring that it is actually better tested than the version we have in libc.” *** Revisiting W^X with OpenBSD 6.0 (http://blog.acumensecurity.net/revisiting-wx-with-openbsd-6-0/) OpenBSD 6.0 includes enforcement of W^X in user-land This prevents an application from being able to map a page of memory with both Write and Execute permissions (protecting mmap(2)) Once mapped a page of memory should not be able to have permissions escalated (protecting mprotect(2)) OpenBSD 6.0 enforces the strict W^X definition, and not the PaX/grsec “once write never execute” type of policy *** OpenBSD imports a letsencrypt client into the base system (http://undeadly.org/cgi?action=article&sid=20160901060733) We've mentioned letskencrypt before (A native C version of the letsencrypt client, developed by Kristaps). Looks like it's undergoing a name-change to “acme-client” and has made it's way into OpenBSD's base system! This should ensure first-class support for management of Let's Encrypt certificates, here's hoping the portable version continues to thrive as well. Congrats to Kristaps! *** Beastie Bits OpenBSD: Release Songs 6.0: "Goodbye" -- no more CD releases (https://www.openbsd.org/lyrics.html#60f) FreeBSD 101 Hacks (https://nanxiao.gitbooks.io/freebsd-101-hacks/content/) LibreSSL enabled by default in HardenedBSD (https://hardenedbsd.org/article/shawn-webb/2016-08-20/libressl-enabled-default) DragonflyBSD removes last bits of 32-bit Linux emulation and has no plans to implement 64-bit linux emulation (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624241.html) OpenBSD has sent 32bit sparc to the great bitbucket in the sky (https://twitter.com/phessler/status/771277693090467840) Front Range BSD User Group September Meeting (http://slexy.org/view/s2hm4HBkb2) KnoxBug TrueOS Wrap-up (http://knoxbug.org/content/going-with-the-flow) Feedback/Questions Cody - TrueOS Questions (http://pastebin.com/mVK8G1Vr) John - FreeNAS Backups (http://pastebin.com/xsUNUfCS) Herminio - PowerPC + OpenBSD (http://pastebin.com/nHkWuNkm) Dennis - pmake vs bmake (http://pastebin.com/NAh7r6Ed) Al - Upgrade conflicts (http://pastebin.com/8HaK7yJ6) ***
This week on BSDNow, we have an interview with Richard Yao, who will be telling us about the experience and challenges of porting ZFS to Linux. That plus the latest news and feedback is coming your way, on your place This episode was brought to you by Headlines Registration for MeetBSD 2016 is now Open (https://www.meetbsd.com/) “Beastie's coming home!” This year, MeetBSD will be held at UC Berkeley's Clark Kerr Campus November 11th and 12th, preceded by a two day FreeBSD Vendor/Dev Summit (Nov 9th and 10th) MeetBSD can be traced back to its humble roots as a local workshop for BSD developers and users, hosted annually in Poland since 2004. Since then, MeetBSD's popularity has spread, and it's now widely recognized as its own conference with participants from all over the world. The US version runs every two years in California since 2008, and now trades off with the east coast vBSDCon which runs on the odd years. “MeetBSD 2016 uses a mixed unConference format featuring both scheduled talks and community-driven events such as birds-of-a-feather meetings, lightning talks, hackable presentations, stump the chumps, and speed geeking sessions. Speakers are to be determined – stay tuned for more information!” Register before September 30th, and get $30 off Kris and I will be there, along with lots of other FreeBSD Developers, Vendors, and Users. MeetBSD's unconference style does a very good job of mingling users with developers and is one of my favourite conferences. *** Dual Booting FreeBSD and Windows UEFI (http://kev009.com/wp/2016/07/freebsd-uefi-root-on-zfs-and-windows-dual-boot/) Looking to install FreeBSD alongside Windows 10? What happens if that that system is pre-installed and UEFI? Well you could run TrueOS, but if that isn't your bag and you want vanilla FreeBSD we have you covered this week! Over on Kevin Bowling's blog, we have a detailed article showing exactly how to do that. First up, as prep you'll need to go into the Windows disk manager and shrink your existing NTFS partition. You'll need to next boot FreeBSD 11 or later. From there the walkthrough takes us through disk partitioning using gpart, and setup of ZFS into a boot-environment friendly layout. Once you get through the typical FreeBSD setup / extraction, the tutorial gives us a nice bonus, showing how to setup “rEFInd” for a graphical boot-menu. A great walkthrough, and hopefully it encourages others to try out dual-booting “EFI-style”. *** ZFS High-Availability NAS (https://github.com/ewwhite/zfs-ha/wiki) Interested in a DiY HA ZFS NAS? Edmund White (ewwhite on github) has posted a very detailed look at how he has custom-rolled his own Linux + ZFS + HA setup. Most of the concepts are already ones used in various other HA products, but it is interesting and informative to see a public detailed look at how ZFS and HA works. In particular this setup require some very specific hardware, such as dual-port SAS drives, so you will have to pre-plan according. The only bummer is this is a ZFS on Linux setup. Maybe this can serve as the guide / inspiration for somebody in our community to do their own FreeBSD + HA + ZFS setup and blog about it in similar detail. *** First public release of chyves - version 0.1.0 (http://chyves.org/) As bhyve continues to mature we are seeing tooling evolve around it. Enter ‘chyves' which started life as a fork of iohyve. We are looking to do an interview with the author in the near future, but we still want to bring you some of the new features / changes in this evolution of bhyve management. First up, nearly every function from iohyve has either been re-written in part or full. Among the new features, a full logging system (master and per-vm logs), multiple pool configurations, properties stored outside of ZFS (for speed) and self-upgrading. (Will that work with pkg'd version?) In addition to the above features, the website has a large chart showing the original ‘iohyve' commands, and how that usage has changed moving to chyves. Give it a spin, let the author know of issues! *** Interview - Richard Yao - ryao@gentoo.org (mailto:ryao@gentoo.org) Sr. Kernel Engineer at ClusterHQ - Major Contributor to ZFS on Linux News Roundup ZFS Deadlock: 'Directory of Death' (http://lists.freebsd.org/pipermail/freebsd-hackers/2016-July/049740.html) A user reports that when they try to install npm (the Node.js package manager), their system deadlocks It turns out, this was also hitting the FreeBSD package building machines PR 209158 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209158) The problem was a race condition in the way renames are handled in the FreeBSD VFS vs how ZFS does them internally This bug has existed since the original import of ZFS, but some other change caused it to happen much more frequently “ZFS POSIX Layer is originally written for Solaris VFS which is very different from FreeBSD VFS. Most importantly many things that FreeBSD VFS manages on behalf of all filesystems are implemented in ZPL in a different Way. Thus, ZPL contains code that is redundant on FreeBSD or duplicates VFS functionality or, in the worst cases, badly interacts / interferes with VFS.” “The most prominent problem is a deadlock caused by the lock order reversal of vnode locks that may happen with concurrent zfsrename() and lookup(). The deadlock is a result of zfsrename() not observing the vnode locking contract expected by VFS.” The fixes have been merged to the 10.x and 11.x branches *** New BSD Magazine out (2016-07) (https://bsdmag.org/download/implementing-memory-cache-beast-architecture/) Articles include: Implementing in-memory cache in the BeaST architecture, Docker Cleanup, FreeNAS Getting Started Guide, and starting at the very beginning with open source The August issue is also out (https://bsdmag.org/download/minix-3-free-open-source-operating-system-highly-reliable-flexible-secure/) This issue features two articles about MINIX 3, continues the FreeNAS getting started guide, Optimizes the in-memory cache for the BeaST architecture, and talks about fixing failed ports for Hardened and LibreBSD We hope to have an interview with the creator of the BeaST architecture in the coming weeks *** DragonflyBSD and UEFI (http://lists.dragonflybsd.org/pipermail/users/2016-July/270796.html) We've featured a few stories and walkthroughs about using UEFI to dual-boot BSD, and now its Dragonfly BSD's turn. Dave McFarlane writes into the DF mailing lists, telling us about the specific steps taken to get UEFI installed and boot-strapped on his system. If you've done a FreeBSD manual UEFI install, the process looks very similar, but you will end up manually running ‘gpt' to create partitions, installing dist files, and eventually installing boot1.efi into the FAT EFI partition. Dave also ran into an issue with resulted in no /etc/fstab being present, and helpfully includes what his system needed to fully boot hammer properly. Somebody should document this fully for DFLY, since I would expect to become more commonplace as commodity hardware is shipped with UEFI on by default. *** Netflix and Fill (http://techblog.netflix.com/2016/08/netflix-and-fill.html) The Netflix team has produced a technical blog post describing how their OpenConnect appliances work First the content is received from the content provider, and the Netflix content team makes it ready for deployment, by transcoding the various bitrates, packaging the subtitles, etc. The finished files are then pushed to Amazon S3 storage “We deploy the majority of our updates proactively during configured fill windows. An important difference between our OpenConnect CDN and other commercial CDNs is the concept of proactive caching. Because we can predict with high accuracy what our members will watch and what time of day they will watch it, we can make use of non-peak bandwidth to download most of the content updates to the OCAs in our network during these configurable time windows. By reducing disk reads (content serving) while we are performing disk writes (adding new content to the OCAs), we are able to optimize our disk efficiency by avoiding read/write contention. The predictability of off-peak traffic patterns helps with this optimization, but we still only have a finite amount of time every day to get our content pre-positioned to where it needs to be before our traffic starts to ramp up and we want to make all of the OCA capacity available for content serving.” The OCA may actually contain more than one copy of the same video, because each disk in the OCA is independent, storing the same video on two different disks will provide twice the available read bandwidth Normally the filesystem cache would obviate the need for this, but the Netflix OCA has so much storage, and not a lot of memory, and the requests from users are offset enough that the cache is useless “OCAs communicate at regular intervals with the control plane services, requesting (among other things) a manifest file that contains the list of titles they should be storing and serving to members. If there is a delta between the list of titles in the manifest and what they are currently storing, each OCA will send a request, during its configured fill window, that includes a list of the new or updated titles that it needs. The response from the control plane in AWS is a ranked list of potential download locations, aka fill sources, for each title.” “It would be inefficient, in terms of both time and cost, to distribute a title directly from S3 to all of our OCAs, so we use a tiered approach. The goal is to ensure that the title is passed from one part of our network to another using the most efficient route possible.” The article then goes on to explain how they calculate the least cost filling source “Now that Netflix operates in 190 countries and we have thousands of appliances embedded within many ISP networks around the world, we are even more obsessed with making sure that our OCAs get the latest content as quickly as possible while continuing to minimize bandwidth cost to our ISP partners.” *** Beastie Bits: Cover reveal for “PAM Mastery” (http://blather.michaelwlucas.com/archives/2734) LibertyBSD 5.9 is out - looking for mirrors (http://libertybsd.net/download.html) Unix for Poets (https://web.stanford.edu/class/cs124/lec/124-UnixForPoets.pdf) Feedback/Questions Chuck / Ingo - Get Involved (http://pastebin.com/ksq0rfph) Oskar - Thanks (http://pastebin.com/YqzcHEMg) Alex - SMF (http://pastebin.com/WvdVZbYc) Raymond - RPI3 (http://pastebin.com/JPWgzSGv) ***
This week on BSDNow, Allan is back from his UK trip and we'll get to hear his thoughts on the developer summit. That plus all the This episode was brought to you by Headlines FreeBSD 11.0-RC1 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085277.html) FreeBSD is marching onwards to 11.0, and with it the first RC1 was released. In addition to the usual amd64 architectures, you may want to give it a whirl on your various ARM boards as well, as it includes images for the following systems: 11.0-RC1 amd64 GENERIC 11.0-RC1 i386 GENERIC 11.0-RC1 powerpc GENERIC 11.0-RC1 powerpc64 GENERIC64 11.0-RC1 sparc64 GENERIC 11.0-RC1 armv6 BANANAPI 11.0-RC1 armv6 BEAGLEBONE 11.0-RC1 armv6 CUBIEBOARD 11.0-RC1 armv6 CUBIEBOARD2 11.0-RC1 armv6 CUBOX-HUMMINGBOARD 11.0-RC1 armv6 GUMSTIX 11.0-RC1 armv6 RPI-B 11.0-RC1 armv6 RPI2 11.0-RC1 armv6 PANDABOARD 11.0-RC1 armv6 WANDBOARD 11.0-RC1 aarch64 GENERIC For those wondering the list of changes between this and BETA4, we have that as well: A NULL pointer dereference in IPSEC has been fixed. Support for SSH Protocol 1 has been removed. OpenSSH DSA keys have been disabled by default. Users upgrading from prior FreeBSD versions are urged to update their SSH keys to RSA or ECDSA keys before upgrading to 11.0-RC1. PCI-e hotplug on bridges with power controllers has been disabled. A loader tunable (hw.pci.enablepciehp) to disable PCI-e HotPlug has been added. A VESA panic on suspend has been fixed. Google Compute Engine image publication has been fixed. An AES-ICM heap corruption typo bug has been fixed. A regression in pf.conf while parsing the 'interval' keyword has been fixed. A ZFS/VFS deadlock has been fixed. RC2 is delayed while some issues are sorted out (https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085323.html) RC2 is looming large, but was pushed back a few days while the following bugs are sorted out: Issue with IPv6 UDP traffic being sent from wrong MAC address Layer2 violation with IPv6 *** OpenBSD just added initial support for the RaspberryPi 2 and 3 devices (https://marc.info/?l=openbsd-cvs&m=147059203101111&w=2) It's a good time to be an ARM and BSD enthusiast. In addition to all the ARM images in FreeBSD 11.0, we also have word that initial support for RPi2 and RPi3 has started to land in OpenBSD. Mark Kettenis has posted the following with his Commit: Initial support for Raspberry Pi 2/3. All the hard work done by patrick@, I just cleaned things up a bit. Any bugs introduced in that process are entirely mine. This doesn't work yet. But when it does, you'll need recent firmware from the Raspberry Pi Foundation git repository at: https://github.com/raspberrypi/firmware The device tree for the Raspberry Pi is somewhat in flux as bits and pieces to support the Raspberry Pi 2 and 3 are committed to the mainline Linux kernel.“ + Exciting news! We will of course keep you informed as to when we have images to play with. Running OpenBSD / PF on a RPi does sound intriguing. drm-4.8-rc2 tagged in drm-next (https://lists.freebsd.org/pipermail/freebsd-x11/2016-August/017840.html) Remember when FreeBSD lagged so far behind in Graphics support? Well, those days are rapidly coming to an end. Matt Macy has posted an update to the FreeBSD X11 list with news of his DRM branch being caught up all the way to Linux 4.8-RC2 now. This is a huge accomplishment, with Matt commenting: As of this moment sys/dev/drm in the drm-next tree is sync with https://github.com/torvalds/linux drivers/gpu/drm (albeit only for the subset of drivers that FreeBSD supports - i915, radeon, and amdgpu). I feel this is a bit of a milestone as it means that it is possible that in the future graphics support on FreeBSD could proceed in lockstep with Linux. For those who want to try out the latest support, you can build from his branch at the following GitHub location: (https://github.com/FreeBSDDesktop/freebsd-base-graphics) Or, if compiling isn't your thing, TrueOS (The re-branded PC-BSD) will be releasing the a new ISO based upon his update to Linux 4.7 in the coming days, with 4.8-RC2 to follow in the next week or two. *** Installing FreeBSD for Raspberry Pi (https://www.freebsdfoundation.org/freebsd/how-to-guides/installing-freebsd-for-raspberry-pi/) People have been running FreeBSD on various RPi devices for a while now, however there are still a lot of people who probably need a hand to get boot-strapped on their RPi system. The FreeBSD foundation has put together a nice tutorial which walks even the most novice user through getting FreeBSD up and running. In particular this could become a good way for students or other FreeBSD newcomers to try out the OS on a relatively low-cost platform outside of a VM. The tutorial starts of with a check-list of the specific items you'll need to get started, for RPi 1 (a/b) or RPi 2 hardware. From there, instructions on how to get the downloaded images onto a sdcard are provided, including Mac and Windows image burning details. With this done, it's really only a matter of plugging in your device to be presented with your new RPi + FreeBSD system. The most important details (the default username/password) at also provided, so don't skim too quickly. *** Interview - Drew Gurkowski Foundation Intern: First time FreeBSD User and Writing Tutorials *** News Roundup FreeBSD's ipfw gets a NAT64 implementation (https://svnweb.freebsd.org/base?view=revision&revision=304046) A new feature has been added to FreeBSD's native firewall, ipfw2 The new loadable module implements stateless and stateful NAT64 “Stateless translation is appropriate when a NAT64 translator is used in front of IPv4-only servers to allow them to be reached by remote IPv6-only clients.” With this setup, you map specific IPv6 addresses to the corresponding IPv4 address, allowing IPv4 only servers to be reachable on the v6 network. “Stateful translation is suitable for deployment at the client side or at the service provider, allowing IPv6-only client hosts to reach remote IPv4-only nodes.” This configuration allows many IPv6 only clients to reach the “legacy” internet. The FreeBSD cluster has been waiting for this feature for a while, because they have limited IP addresses, but many service jails that require access to services like GitHub that are not IPv6 enabled. The work was sponsored by Yandex, the Russian search engine and long time FreeBSD user Example configurations for both types are included in the commit message If you would find this feature useful, please take the time to set it up and document the steps and contribute that to the FreeBSD Handbook. *** Update on using LLVM's lld linker in the FreeBSD base system (https://lists.freebsd.org/pipermail/freebsd-toolchain/2016-August/002240.html) Ed Maste has written a lengthy update on the progress being made towards using LLVM's lld linker as a replacement for GNU's ‘ld'. Ed starts off by giving us some of the potential benefits of using lld vs the 2.17.50 ‘ld' version FreeBSD currently uses: AArch64 (arm64) support Link Time Optimization (LTO) New ABI support Other linker optimization Much faster link times Maintained code base Ed also gives us an update on several of the major blockers: Since the last update in March several lld developers have implemented much of the missing functionality. The main blockers were symbol version support and expression evaluation in the linker script expression parser. Both are now nearly complete“ A detailed plan was also articulated in respect to switching over: Update lld along with the Clang/LLVM 3.9 update that dim@ is working on. Add the bmake build infrastructure, installing as /usr/bin/ld.lld on the same architectures that use Clang (amd64, arm, arm64, i386). I don't think there's a need for a WITH_LLD src.conf knob, but will add one if desired. Update lld again (most likely to a snapshot from upstream SVN) once it is able to link an unmodified FreeBSD kernel. Modify the boot loader and kernel builds to avoid using features not implemented by lld. Introduce a WITHLLDAS_LD knob to have /usr/bin/ld be a ld.lld hardlink instead of /usr/bin/ld.bfd. Request ports exp-runs and issue a call for testing with 3rd party software. Fix issues found during this process. Switch /usr/bin/ld to ld.lld by default in head for the Clang-using architectures. Add a WITHOUTLLDAS_LD knob to switch back to GNU ld. *** How to install FreeBSD with ZFS filesystem on DigitalOcean (https://github.com/fxlv/docs/blob/master/freebsd/freebsd-with-zfs-digitalocean.md) I know we've mentioned using FreeBSD + ZFS on digital ocean in the past, but today we have a nice HowTo by Kaspars Mickevics (fxlv) on GitHub. Before getting started, kaspars mentions some pre-reqs. First up he recommends starting with a Minimum of 2GB of RAM. (The $20/mo droplet). This is to ensure you have plenty of cushion to avoid running out of memory during the process. It is possible to use ZFS with less, but depending on your desired workload this does make sense. From there, checking out “mfsBSD” is discussed, along with details on how to make it suitable for a DO installation. (Mostly just disabling DHCP for the network device) For good measure ‘pkg-static' is also included. With that done, using mfsBSD you will create a tar file, which is then extracted on top of the running system. After rebooting, you will be able to run “bsdinstall” and proceed to installing / formatting your disk with ZFS as normal. A good tutorial, something I may need to do here in the near future. User manages to get OpenBSD and FreeBSD working with Libreboot (https://lists.nongnu.org/archive/html/libreboot/2016-08/msg00058.html) In a short drive-by post to the Libreboot mailing list Piotr Kubaj gives a quick notice that he managed to get OpenBSD and FreeBSD both booting. > I know GNU people don't like BSD, so let me make it quick :) > > > I've succeeded in booting FreeBSD 11.0-RC1 using txt mode on my X200 > with the newest Libreboot. > > To get installer to boot, I used: > kfreebsd (usb0,gpt3)/boot/kernel/kernel > set FreeBSD.vfs.mountfrom=ufs:/dev/da1p3 > boot > > I didn't try to install yet. > The trick looks relatively simple (looks like GRUB), manually loading the kernel with ‘kfreebsd' and then setting the vfs.root.mountfrom variable to find the USB stick. In an update he also mentions booting OpenBSD with ‘kopenbsd' instead of ‘kfreebsd' (again GRUB syntax) Now somebody will need to test installation of the system (he didn't) and see what other issues may crop up in running BSD on a free BIOS. *** Beastie Bits: The ACPICA (ACPI Component Architecture) coding language AML now in DragonFly BSD (http://lists.dragonflybsd.org/pipermail/commits/2016-July/624192.html) Release announcement for 4.3BSD Tahoe from 1988 (https://groups.google.com/forum/#!topic/comp.sys.tahoe/50ManvdM1-s) Feedback/Questions Mike - Jail Uptime (http://pastebin.com/FLpybL6D) Greg - Router Hardware (http://pastebin.com/RGuayhB3) Kristof writes in (http://pastebin.com/NT4zmHiG) Ty - Updates and Logs (http://pastebin.com/CtetZdFg) Benjamin - MTA Bug (http://pastebin.com/Qq3VbQG2) ***
This week on BSDNow, Allen is away in the UK (For BSDCam), but we still have a full episode for you! Don't miss our interview with This episode was brought to you by Headlines My two year journey to becoming an OS Developer (http://zinascii.com/2016/going-to-joyent.html) A blog post by Ryan Zezeski about how he ended doing OS Development instead of working on application We have featured his posts before, including The illumos SYSCALL Handler (http://zinascii.com/2016/the-illumos-syscall-handler.html) It started in the summer of 2014: I had just left Basho after 3.5 years of working on Riak, when I decided I wanted to become an OS developer. I purchased Solaris Internals, cloned illumos-gate, fired up cscope, and got to work. I hardly knew any C, x86 might as well have been Brainfuck, and, frankly, I knew shit about operating systems. But I was determined. I've always learned best by beating my head against something until it makes sense. I'm not a fast learner; I'm persistent. What others have in ability I make up for in effort. And when it comes to OS internals it's all about work ethic. The more you look, the more you realize it's just another program. The main difference being: it's the program all the other programs run on. My strategy: to pick something, anything, that looked interesting, and write a post describing how it works. I wrote several of these posts in 2014 and 2015. More important, it put me in touch with Roger Faulkner: the creator of truss(1), the Solaris process model, and the real /proc filesystem. At the time I didn't like my interaction with Roger. He explained, in what I would later find out to be his typical gruff manner, that I was wrong; so I concluded he is a prick. But over the years I realized that I was being a brat—he was trying to teach me something and I let my ego get in the way. I've come to view that interaction as a blessing. I interacted with one of the greats, a mentor of my mentor's mentor (a Great Great Mentor). A couple of weeks later something even more surreal happened, at illumos Day 2014. Bryan Cantrill was the last speaker of the day. One of my mentors and someone I admire greatly. He was there to regale us with the story of Joyent's resurrection of lx-branded zones: Linux system call emulation on top of the illumos kernel. But before he would do that he decided to speak about me! I couldn't believe it. I was so overwhelmed that I don't remember most of what he said. I was too busy flipping shit—Bryan Cantrill is on stage, in front of other kernel developers I look up to, saying my name. I was in a dream. It turns out, unknown to me at the time, that he wrote the POSIX queue code for both Solaris and QNX, which I wrote about. He compared me to the great expository technical writers Elliott Organick and Richard Stevens. And it was at this moment that I knew I could do this: I could become an OS developer. Never underestimate the effect kind words can have on someone that looks up to you. There is a lot more to the story, and it is definitely worth the read The story then goes on to talk about his recent run in with Bryan Cantrill > A week from now my two year journey to become an OS developer comes to an end; and a new chapter begins. I don't know what specific things I'm going to work on, but I'm sure it will push me to the limit. I look forward to the challenge. *** Version 1.0 of the Lumina Desktop released (https://lumina-desktop.org/version-1-0-0-released/) After 4 years of development, Lumina Desktop has now hit version 1.0! This release brings with it a slew of new features and support: Completely customizable interface! Rather than having to learn how to use a new layout, change the desktop to suit you instead! Simple shortcuts for any application! The “favorites” system makes it easy to find and launch applications at any time. Extremely lightweight! Allows applications to utilize more of your system hardware and revitalizes older systems! Multiple-monitor support! Each monitor is treated as an independent entity – making it great for presentation systems which use a temporary monitor or for workstations which utilize an array of monitors for various tasks. While originally developed on PC-BSD, it already has been ported to a variety of different platforms, including OpenBSD, DragonFly, NetBSD, Debian and Gentoo Lumina has become the defacto desktop environment for TrueOS (Formerly PC-BSD), and looks like will provide a solid framework to continue growing desktop features. *** n2k16 hackathon report: Ken Westerback on dhclient, bridges, routing and more (http://undeadly.org/cgi?action=article&sid=20160804200232) Next up, we have a report from Ken Westerback talking about the recent OpenBSD hackathon in Prague He starts by telling us about the work in bpf: First order of business, stsp@'s weird setup involving bridges and multiple dhclient clients. A bit of bpf(4) programming to restrict dhclient to handling ethernet packets unicast to its interface worked. Cool. Unfortunately it turned out some lazy dhcp servers always use ethernet broadcasts just because some lesser, non-OpenBSD clients ignore unicast packets until they have configured IP. Classic chicken and egg. So this was backed out just before 6.0. Sigh. Next up, he talks about an idea he had on the flight over, specifically with regard to how DHCP leases are stored, and how keeping the SSID information with them could speed up re-connection times, by only trying leases for current SSID's connected. After a day or so of hacking, it was working! However for $REASONS it was shelved for post 6.0, bummer! He then discusses an on-going project with Peter Hessler on passing along relevant PIDs in response to routing messages generated by kernel from ioctl events. This is something they've been hacking at, in order to allow dhclient to recognize its own routing messages. Sounds like they are both still works-in-progress. However, Ken did get something in for 6.0: Diving back into dhclient code I discovered that in situations where multiple offers were received the unused offers were not being declined and discarded. Despite a clear comment saying that's what was being done! Thus dhclient might gradually use up more and more memory. And possibly be retrying offers that should have been discarded. The fix for this did make 6.0! Yay! In Memoriam Roger Faulkner (https://www.usenix.org/memoriam-roger-faulkner) USENIX has re-released Roger Faulkner's original paper on /proc as a free download The UNIX community recently lost one of its original pioneers, Roger Faulkner, whom one commenter described as “The godfather of post-AT&T UNIX” In his memory, the USENIX group as re-released his original paper on the /proc file-system from 1991. Roger worked in many area's of UNIX, however the process file system /proc was his special baby. “/proc began as a debugger interface superseding ptrace(2) but has evolved into a general interface to the process model.” The original /proc only had a file for each process, not a directory. "Data may be transferred from or to any valid locations in the process's address space by applying lseek(2) to position the file at the virtual address of interest followed by read(2) or write(2)." Processes could be controlled using IOCTLs on the file As the USENIX article states: Roger believed that terrible things were sometimes required to create beautiful abstractions, and his trailblazing work on /proc embodies this burden: the innards may be delicate and nasty ("vile," as Roger might say in his distinguished Carolinian accent)—but the resulting abstractions are breathtaking in their power, scope and robustness. RIP Roger, and thanks for the wonderful UNIX legacy you've left us all. Interview - Myke Geiger - myke@servernorth.net (mailto:myke@servernorth.net) / @mWare (https://twitter.com/mWare) Using FreeBSD at a DSL/Cable ISP *** News Roundup New options in bsdinstall - some sysctls and date/time settings (https://www.reddit.com/r/freebsd/comments/4vxnw3/new_options_in_bsdinstall_some_sysctls_and/) bsdinstall in FreeBSD 11.0 will feature a number of new menus. The first, well allow you to set the date and time. Often on computers that have been in storage, or some embedded type devices that have no RTC, the date will be wildly wrong, and ntpd will refuse to run until the date is correctly set. This feature makes it easy to enter the date and time using dialog(1) The second menu, inspired by the existing ‘services' menu, offers a number of ‘hardening' options This menu allows users to easily enable a number of security features, including: Hide processes running as other users/groups Disable reading the kernel message buffer and debugging processes for unprivileged users Randomize the PID of newly created processes Enable the stack guard Erase /tmp at boot Disable remote syslog Disable sendmail All of these options are off by default, so that an install done with the installer will be the same as an install from source, or an upgrade. A number of these options are candidates to become on-by-default in the future, so the hope is that this menu will get more users to test these features and find any negative interactions with applications or general use, so they can be fixed. *** Rawrite32: the NetBSD image writing tool (https://www.netbsd.org/~martin/rawrite32/) Martin of the NetBSD project has released a new version of his USB imaging tool, rawrite32 For those who've not used this tool before, it is a Windows Application that allows writing NetBSD images directly to USB media (other other disk media) This update brings with it support for writing .xz file, and binary signing This may come in handy for writing other OS images to memory sticks as well, especially for those locked into a windows environment who need to switch. *** ZFS-Snap-Diff -- A pretty interface for viewing what changed after a ZFS snapshot (https://github.com/j-keck/zfs-snap-diff) There are lots of nice little utilities to help create and maintain your ZFS snapshots. However today we have something unique to look at, ‘zfs-snap-diff'. What makes it unique, is that it ships with a built-in golang / angularjs GUI for snapshot management It looks very powerful, including a built-in diff utility, so you can even see the changes in text-files, in addition to downloading files, restoring old versions and more. Its nice to see so many ZFS utilities starting to take off, and evolve file-management further. *** Dtrace Conf 2016 Event Videos (https://www.joyent.com/about/events/2016/dtrace-conf) The videos from Dtrace.conf 2016 have been posted Some highlights: Useful DTrace Intro CTF Everywhere Distributed DTrace DTrace for Apps DTrace json() subroutine Implementing (or not) fds[] in FreeBSD OpenDTrace DTrace performance improvements with always-on instrumentation D Syntactic Sugar DTrace and Go, DTrace and Postgres dtrace.conf(16) wrap-up by Bryan Cantrill (https://www.joyent.com/blog/dtrace-conf-16-wrap-up) Once again, it was an eclectic mix of technologists — and once again, the day got kicked off with me providing an introduction to dtrace.conf and its history. (Just to save you the time filling out your Cantrill Presentation Bingo Card: you can find me punching myself at 16:19, me offering unsolicited personal medical history at 20:11, and me getting trolled by unikernels at 38:25.) The next DTrace.conf isn't until 2020 *** Beastie Bits The BSD Daemon features in Mexican candy packaging (https://www.reddit.com/r/BSD/comments/4vngmw/the_bsd_daemon_feature_in_mexican_candy_packaging/) Remove PG_ZERO and zeroidle (page-zeroing) entirely (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624202.html) OpenBSD: Release Songs: 6.0: "Black Hat" (https://www.openbsd.org/lyrics.html#60b) OpenBSD Gaming Resource (http://satterly.neocities.org/openbsd_games.html) LibreSSL 2.4.2 and 2.3.7 Released (http://bsdsec.net/articles/libressl-2-4-2-and-2-3-7-released) Feedback/Questions Pedja - Bhyve GUI (http://pastebin.com/LJcJmNsR) Tim - Jail Management (http://pastebin.com/259x94Rh) Don - X260 (http://pastebin.com/A86yHnzz) David - Updates (http://pastebin.com/wjtcuVSA) Ghislain - Jail Management (http://pastebin.com/DgH9G7p5) ***
This week on BSDNow, we are taking a look at a few different tutorials, including running your very own RPi web-server. (Come-on, you This episode was brought to you by Headlines broken features aren't used (http://www.tedunangst.com/flak/post/broken-features-arent-used) This post from TedU talks about the difficulty of removing features from an operating system “One of the difficulties in removing a feature is identifying all the potential users. A feature here could be a program bundled with an operating system, or a command line option, or maybe just a function in a library. If we remove a feature, users that depend on it will be sad. Unfortunately, absence of evidence is not evidence of absence. I've never heard of anybody running ls -p but it's not impossible that somebody does.” “The reasons why we want to remove an existing feature can vary. Sometimes it's old code that interferes with maintenance. Sometimes a nearly complete rewrite can improve performance. In other cases, the feature in question is really more of a misfeature. It may have security implications, where the existence of the feature can be used to facilitate the exploitation of other vulnerabilities, and removing the feature will help mitigate the exploit.” “There's no general test that can be used, but there is one test that works in many cases. Test that the feature works. If the feature doesn't work, that's compelling evidence that nobody is using it, because nobody can be using it. You don't need to fix it. You can just remove it.” He makes some interesting comments about exhaustive unit tests and the push to keep everything working all the time. If you never break anything to see if someone complains, how do you know if it is still being used? *** A Raspberry Pi FreeBSD Web Server (http://devio.us/~spacemonkey/raspberry_pi_freebsd_web_server_howto.html) Looking at a super-low power solution to host some webpages? If so, we have the tutorial for you. Specifically a walkthrough of getting FreeBSD up on a Pi, and setting up nginx, OpenNTPD, LibreSSL and friends. The walkthrough starts with grabbing a FreeBSD 11 snapshot for arm64 and doing the initial setup process to get to a bootable FreeBSD system. If you are an extreme noob, not to fear. The tutorial walks you through setting up usernames, timezones, even a larger /tmp directory on your new MiniBSD setup. The tedious part comes to play during the setup of packages. The author walks us through setting up LibreSSL and various other packages via ports (Since LibreSSL isn't the default in FreeBSD). This will take some time to compile on your humble RPi device. (Go make a sandwich, walk the dog, fix the gutters, etc) When it's all said and done, you'll end up with a secure little web-server that you've configured all by yourself! (Wondering what the word-press performance would be like on that box) *** Uber switches from PostgreSQL back to MySQL (https://eng.uber.com/mysql-migration/) We often hear success stories of people switching to PostgreSQL and getting huge performance gains, but this stories is the reverse Uber's engineering team has switched back to MySQL, because for their specific workload and design, MySQL's innodb has better performance Of course, it is not just vanilla MySQL, but “Schemaless”, a sharding system that sits on top of MySQL The article goes into detail about the on-disk format used by Postgres, and the specific shortcomings that Uber encountered Uber admits that all of its testing was against the older PostgreSQL 9.2, but one of their complaints is about having difficulty upgrading “We started out with Postgres 9.1 and successfully completed the upgrade process to move to Postgres 9.2. However, the process took so many hours that we couldn't afford to do the process again. By the time Postgres 9.3 came out, Uber's growth increased our dataset substantially, so the upgrade would have been even lengthier. For this reason, our legacy Postgres instances run Postgres 9.2 to this day, even though the current Postgres GA release is 9.5.” There is a followup, from the Postgres side “Why we lost Uber as a user” (https://www.postgresql.org/message-id/5797D5A1.5030009%40agliodbs.com) This thread goes into detail about the specific types of problematic queries that Uber was using “The Uber guy is right that InnoDB handles this better as long as you don't touch the primary key (primary key updates in InnoDB are really bad)” “This is a common problem case we don't have an answer for yet.” The thread then goes on to discuss possibly supporting a “pluggable heap storage layer”, to allow different workloads to use different on-disk formats for best performance *** Getting started with GhostBSD and FreeBSD Part 1 (http://unsolicitedbutoffered.blogspot.com/2016/07/getting-started-with-ghostbsd-and.html) Part 2 (http://unsolicitedbutoffered.blogspot.com/2016/07/getting-started-with-ghostbsd-and_31.html) Part 3 (http://unsolicitedbutoffered.blogspot.com/2016/08/getting-started-with-ghostbsd-and.html) Part 4 (http://unsolicitedbutoffered.blogspot.com/2016/08/getting-started-with-ghostbsd-and_2.html) In what may be our first GhostBSD tutorial, we have a nice walkthrough on the initial getting started with it. For those who don't know, GhostBSD provides a nice XFCE or Mate desktop out of box, and still supports 32bit installs for those who want to keep that older hardware running. The walkthough takes us through the process of grabbing GhostBSD images and getting the installer up and running via bootable USB stick. Once booted, the graphical installer is straight-forward and short, allowing you to get the bits on disk as quickly as possible. (The actual installation took around 45 Minutes on an old Toshiba NB520) The author then takes us on a tour of some of GhostBSD's out-of-box bundled applications (Along with XFCE) and how it compares to similar Linux setups. Lastly covered is the setup of Wireless (The manual way with WPA supplicant, since the GUI tool appeared to not work in this particular case) All in all a good walkthrough, especially if you've not seen GhostBSD in action before, the screenshots are very informative! *** News Roundup Steam on FreeBSD 11-CURRENT (https://github.com/SteamOnFreeBSD/SteamOnFreeBSD) Steam on FreeBSD. Yes, we've heard of setups using WINE, but what about running the Linux binaries natively? Well you are in luck. We have a github project that details getting the Linux native client up and running on a FreeBSD 11-CURRENT system. This github project is rather mysterious, with only the instructions to download a pre-packaged steam.txz file, extract and run the provided install.sh script. Curious I inspected some of the scripts, the installer.sh is fairly straight-forward, but does some ‘non-standard' freebsd things, like fetching packages and extracting specific files/libaries into a new /compat/ubuntu directory. After that, it goes through a huge list of debian/ubuntu packages, also throwing them into the aforementioned ubuntu directory. At runtime, the wrapper script ensures that various linux compat file-systems are mounted in the correct location, then proceeds to run steam with some LDLIBRARYFLAGS set from the users .local/share/steam directory. A tad scary if honest, however it is a neat PoC to see Steam working on FreeBSD. Hopefully somebody can turn this into a more traditional package which can be easily removed / cleaned up afterwards. *** How to run Enlightenment on OpenBSD (http://enform.haxlab.org/) Are you an enlightenment fan? Be honest, we know there are quite a few of you out there! If so, we have a tutorial for you today, which talks about how to run E on OpenBSD. The process is pretty easy, but some steps might be overlooked if you are new to OpenBSD or don't know how to tune / compile things on your own. It starts out with adjusting some sysctl's for better tuning that works on an E based desktop. Next is installing from package some pre-reqs that will enable us to build E from source. After that, we need to download and install EFL from github, and the autogen / gmake commands are helpfully provided for you. Lastly the same is done for E itself, and TADA, E is installed and ready to go on your OpenBSD system. If you do this right, should only take 5-10 minutes to be up and running. *** Myths about FreeBSD (https://wiki.freebsd.org/Myths) Over on the FreeBSD wiki, we have a new “Myths” page which we've never highlighted on the show before. First up, and one I've very familiar with, is the usual “FreeBSD is only for Servers and not Desktops”, along with a good rebuttal about what it does offer and mention of projects such as PC-BSD which do it also. Another prevalent one is the “FreeBSD has a closed development model”, which is easily refuted: FreeBSD has over 400 developers around the world who have commit access to the repository. Many of these are willing to commit patches from third parties. If you want to get an idea of the number of patches that have been committed on behalf of other developers, then search for 'Submitted by' in the commit logs. At the time of writing, this is just under twenty thousand, or about ten percent of all commits. After having a few patches accepted, regular contributors are usually encouraged to apply for commit access. Another one that we are still hearing (Although it is less and less now) was the common “FreeBSD makes me compile everything from source”. Listeners of this show will know that pkg has pretty much made this irrelevant in recent years. However the option to compile yourself from source still exists, but most users won't find this ever necessary. A good list, with many more items on it than we've mentioned here. Take a look, you might find something there you've heard in the wild, or maybe even thought yourself at one point! FreeBSD Area51 testing repo, KDE 5 (https://gist.github.com/swills/8a9a9f8a529256f134830c2a1476db0c) This github GIST from Steve Wills provides the instructions to enable the FreeBSD xorg teams unofficial ‘Area51' testing repo on your machine This gives you access to the new KDE 5 Users should obviously be careful testing early-access software, but bug reports are very welcome, and important to getting KDE 5 working well under FreeBSD *** Beastie Bits NetBSD Machines at Open Source Conference 2016 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2016/08/01/msg000712.html) How to install htop on pfSense (http://www.cyberciti.biz/faq/pfsense-install-htop-using-pkg-command/) The first ever patch for #Chromium on #NetBSD has been committed! (https://twitter.com/krytarowski/status/759909139300491265) CharmBug - Adventures in Hardened BSD - August 24th 2016 (http://www.meetup.com/CharmBUG/events/232474857/) Feedback/Questions Emanual - GoTTY (http://pastebin.com/q24sNcNY) Joe - Disable Device (http://pastebin.com/iTkmpVr9) Allen - ZFS Send/Recv (http://pastebin.com/bPHEP4Ya) Mica - Adobe Replacements (http://pastebin.com/L0ttuNx3) Robroy - RAID (http://pastebin.com/Dr92CGCU) ***
This week on BSDNow, we have a variety of news to discuss, covering quite the spectrum of BSD. (Including a new DragonFly release!). This episode was brought to you by Headlines my int is too big (http://www.tedunangst.com/flak/post/my-int-is-too-big) “The NCC Group report (http://marc.info/?l=oss-security&m=146853062403622&w=2) describes the bugs, but not the history of the code.” “Several of them, as reported by NCC, involved similar integer truncation issues. Actually, they involved very similar modern 64 bit code meeting classic 32 bit code” “The thrsleep system call is a part of the kernel code that supports threads. As the name implies, it gives userland a measure of control over scheduling and lets a thread sleep until something happens. As such, it takes a timeout in the form of a timespec. The kernel, however, internally implements time keeping using ticks (there are HZ, 100, ticks per second). The tsleep function (t is for timed) takes an int number of ticks and performs basic validation by checking that it's not negative. A negative timeout would indicate that the caller has miscalculated. The kernel panics so you can fix the bug, instead of stalling forever.” “The trouble therefore is when userland is allowed to specify a timeout that could be negative. The existing code made an attempt to handle various tricks by converting the timespec to a ticks value stored as a 64 bit long long which was checked against INTMAX before passing to sleep. Any value over INTMAX would be truncated, so we can't allow that. Instead, we saturate the value to INT_MAX. Unfortunately, this check didn't account for the possibility that the tick conversion from the timespec could also overflow and result in a negative value.” Then there is the description of the kqueue flaw: “Every kqueue keeps a list of all the attached events it's watching for. A simple array is used to store file events, indexed by fd.” “This array is scaled to accommodate the largest fd that needs to be stored. This would obviously cause trouble, consuming too much memory, if the identifier were not validated first. Which is exactly what kqueue tries to do. The fdgetfile function checks that the identifier is a file that the process has open. One wrinkle. fdgetfile takes an int argument but ident is a uintptr_t, possibly 64 bits. An ident of 2^32 + 2 will look like a valid file descriptor, but then cause the array to be resized to gargantuan proportions.” “Again, the fix is pretty simple. We must check that the ident is bounded by INTMAX before calling fdgetfile. This bug likely would have been exploitable beyond a panic, but the array allocation was changed to use mallocarray instead of multiplying arguments by hand, thus preventing another overflow.” Then there is a description of the anonymous mmap flaw, and the “secret magic” _MAPNOFAULT flag *** FreeBSD Quarterly Status Report Q2 2016 (https://www.freebsd.org/news/status/report-2016-04-2016-06.html) It's time for another round of FreeBSD Quarterly Status Reports! In this edition, we have status updates from the various teams, including IRC/Bugs/RE/Ports/Core and Foundation We also have updates on some specific projects, including from Konstantin on the on-going work for his implementation of ASLR, including the new ‘proccontrol' command which provides the following: > “The proccontrol(1) utility was written to manage and query ASLR enforcement on a per-process basis. It is required for analyzing ASLR failures in specific programs. This utility leverages the procctl(2) interface which was added to the previous version of the patch, with some bug fixes.” Next are updates on porting CEPH to FreeBSD, the ongoing work to improve EFI+GELI (touched on last week) and more robust Mutexes. Additionally we have an update from Matt Macy and the Xorg team discussing the current work to update FreeBSD's graphic stack: > “All Intel GPUs up to and including the unreleased Kaby Lake are supported. The xf86-video-intel driver will be updated soon. Updating this driver requires updating Xorg, which in turn is blocked on Nvidia updates.” The kernel also got some feature status updates, including on the new Allwinner SoC support, an update on FreeBSD in Hyper-V and VIMAGE In addition to a quick update on the arm64 architecture (It's getting there, RPi3 is almost a thing), we also have a slew of port updates, including support for GitLab in ports, updates on GNOME / KDE and some additional Intel-specific networking tools. *** Vulnerabilities discovered in freebsd-update and portsnap (https://lists.freebsd.org/pipermail/freebsd-security/2016-July/009016.html) There are two vulnerabilities discovered in freebsd-update and portsnap, where an attacker could place files in the portsnap directory and they would be used without being subject to having their checksum verified (but this requires root access), and the second where a man-in-the-middle attacker could guess the name of a file you will fetch by exploiting the time-gap between when you download the initial snapshot, and when you fetch the updated files. There are a number of vulnerabilities that were discovered in libarchive/tar as well There is also an issue with bspatch. A security advisory for bspatch has already been released, as this vulnerabilities was also discovered by the Chromium team, which uses this same code. The patch discussed in this mailing list thread is larger, but secteam@ believes at least one of the additional checks introduced is incorrect and may prevent a valid patch from being applied. The smaller patch was pushed out first, to solve the main attack vector, while the larger patch is investigated. Automated fuzz testing is underway. Great care is being taken fixing bspatch, as if it is broken installing future updates becomes much more difficult secteam@ and core@ would like to emphasize that the FreeBSD project takes these issue very seriously and are working on it > “As a general rule, secteam@ does not announce vulnerabilities for which we don't have patches, but we concede that we should have considered making an exception in this case” Work is underway to re-architect freebsd-update and portsnap to do signature verification on all files before they are passed to libarchive/tar, to help protect users from any future vulnerabilities in libarchive. However, this requires changes to the metadata format to provide these additional signatures, and backwards compatibilities must be preserved, so people can update to the newer versions to get these additional security features There is also discussion of using HTTPS for delivery of the files, but certificate verification and trust are always an issue. FreeBSD does not distribute a certificate trust store by default. There will be more on this in the coming days. *** OpenSSH 7.3 Released (http://www.openssh.com/txt/release-7.3) OpenSSH 7.3 has landed! Primarily a bug-fix release, the release notes do mention the pending deprecation of some more legacy Crypto in the future, including denying all RSA keys < 1024bit, and removal of SSHv1 support. (Already disabled via compile option) On the bug side, there was a security issue addressed in sshd: “sshd(8): Mitigate a potential denial-of-service attack against the system's crypt(3) function via sshd(8). An attacker could send very long passwords that would cause excessive CPU use in crypt(3). sshd(8) now refuses to accept password authentication requests of length greater than 1024 characters” Also a timing issue was resolved in regard to password auth, which could possibly allow an attacker to discern between valid/invalid account names. On the feature side, we have the new ProxyJump option (-J flag) which allows you to do simplified indirection through various SSH jump hosts. Various bugs were fixed, and some compile failures resolved in the portable version to auto-disable some ciphers not supported by OpenSSL. News Roundup OpenBSD Ports - Integrating Third Party Applications [pdf] (http://jggimi.homeip.net/semibug.pdf) A talk from Josh Grosse, presented at SEMIBUG (South-East Michigan BSD Users Group), about OpenBSD Ports It opens by explaining the separation of the ‘base system' from ‘packages', as is common in most all BSDs It explains the contents of OpenBSD package tar file, which contain some metadata files (+CONTENTS and +DESC) and then the actual package files The talk goes on to explain the different branches (-release, -stable, and -current), and warn users that there are no official -stable packages from the project Then it goes on into the development model, including what new contributors should expect Then it walks through the entire process of creating a port and getting it contributed *** NetBSD removes last RWX page in amd64 kernel (http://mail-index.netbsd.org/source-changes/2016/07/27/msg076413.html) NetBSD has purged the last holdout RWX page on the amd64 platform > “Use UVMPROTALL only if UVMKMFEXEC is given as argument. Otherwise, if UVMKMFPAGEABLE is also given as argument, only the VA is allocated and UVM waits for the page to fault before kentering it. When kentering it, it will use the UVMPROT flag that was passed to uvm_map; which means that it will kenter it as RWX. With this change, the number of RWX pages in the amd64 kernel reaches strictly zero.” Break out the party favors! Hopefully any last stragglers in any of the other BSD's gets retired soon as well. *** DragonFly BSD 4.6 launches with home-grown support for NVMe Controllers (http://linux.softpedia.com/blog/dragonfly-bsd-4-6-0-launches-with-home-grown-support-for-nvme-controllers-506908.shtml) Softpedia picked up on the release of DragonFlyBSD 4.6, specifically about their new home-grown NVMe driver. > “We now have a NVMe driver (PCIe SSDs). It currently must be kldloaded with nvme_load="YES" in /boot/loader.conf. The driver uses all concurrency features offered by the chip and will distribute queues and interrupts across multiple CPUs to maximize performance. It has been tested up to around 1.05M IOPS @4K, and roughly 6.5 GBytes/sec @32K (random read from urandom-filled partition, physio, many threads), with the 2xE5-2620v4 (xeon) test server 78% idle in the IOPS test and 72% idle on the bandwidth test. In other words, we maxed out the three NVMe devices we had plugged in and the system still had plenty of suds left over. Please note that a machine's ability to boot from an NVMe device depends on the BIOS, and not DragonFly. Most BIOSes cannot boot from NVMe devices and those that can probably only do it through UEFI. Info on device state is available with the new utility nvmectl.“ In addition to this improved support, 4.6 also brings in the improved graphics support, matching what is in Linux 4.4 and support for Broadwell/Skylake. SMP also got some love: > “SMP performance was already very good. As part of the NVMe driver work we revamped the buffer cache subsystem and a number of other I/O related paths, further reducing lock contention and IPI signalling overheads. We also put topology-aware cpu cache localization into the kernel memory allocator (primarily helps multi-socket systems and systems with high core counts). The network subsystem also continues to receive significant improvement, with modest machine configurations now capable of handling upwards of 580K conns/sec.“ +Full Release Notes (https://www.dragonflybsd.org/release46/) *** The powerd++ daemon monitors the system load and adjusts the CPU clock accordingly and is a drop-in replacement for FreeBSD's native powerd(8). (http://www.freshports.org/sysutils/powerdxx/) As mentioned in our EuroBSDCon 2016 rundown, Dominic Fandrey will be giving a presentation about his powerd replacement, powerd++ The source code is already available on github, and is in ports The major difference is the newer design handle many-core systems much better. The original powerd was written at a time when most laptops only had a single core, and maybe a hyperthread. The new design decides which CPU frequency to use by looking at the busiest core, rather than the average across the cores, resulting in a more meaningful result. It also supports averaging over a longer period of time, to avoid jumping to a higher frequency to quickly powerd++ also avoids ‘slewing' the cpu frequency, ratching it up and down one step at a time, and instead jumps directly to the target frequency. Often times, you will use less battery by jumping to maximum frequency, finishing the work, and going back to a low power state, than trying to do that work over a longer period of time in low power mode *** Beastie Bits Hyper-V: Unmapped I/O improves userland direct disk performance by 35% ~ 135% (https://svnweb.freebsd.org/base?view=revision&revision=303474) One does not simply remove FreeBSD (https://imgur.com/a/gjGoq) A new BSD Podcast "BSD Synergy" has started (https://www.youtube.com/channel/UCBua6yMtJ6W5ExYSREnS3UQ) KnoxBug - Next Meeting - Aug 30th (http://knoxbug.org/content/2016-08-30) Feedback/Questions Daniel - Root/Wheel (http://pastebin.com/8sMyKm6c) Joe - IPV6 Frag (http://pastebin.com/r5Y0gbxf) Paul - ChicagoBug (http://pastebin.com/iVYPYcVs) Chris - SSH BruteBlock (http://pastebin.com/597m9gHa) Todd - Jails (http://pastebin.com/xjbKwSaz) ***
This week on BSDNow, we have some big breaking news about another major switcher to FreeBSD, plus early information about the pending This episode was brought to you by Headlines Leo Laporte tries FreeBSD (http://www.leolaporte.com/blog/a-grand-experiment) Leo Laporte, formerly of TechTV, and now of TWiT.tv, is switching to FreeBSD “The latest debacle over the "forced" upgrade to Windows 10 and Apple's increasingly locked-in ecosystem has got me thinking. Do I really need to use a proprietary operating system to get work done? And while I'm at it, do I need to use commercial cloud services to store my data?” A sometimes Linux user since the mid 90s, Leo talks about his motivations: “But as time went by, even Ubuntu began to seem too commercial to me” “So now for the grand experiment. Is it possible, I wonder, to do everything I need to do on an even more venerable, more robust system: a true UNIX OS, FreeBSD? Here are my requirements” Browsing Email with PGP signing and encryption Coding - I'm a hobbyist programmer requiring support for lisp/scheme/racket, rust, and python (and maybe forth and clojure and meteor and whatever else is cool and new) Writing A password vault. I currently use Lastpass because it syncs with mobile but eventually I'll need to find a FOSS replacement for that, too Photo editing - this is the toughest to replace. I love Photoshop and Lightroom. Can I get by with, say, GIMP and Darktable? I do all of those things on my PCBSD machine all the time “I love Linux and will continue to use it on my laptops, but for my main workhorse desktop I think FreeBSD will be a better choice. I also look forward to learning and administering a true UNIX system.” He got a nice SuperMicro based workstation, with an Intel Xeon E3-1275v5 and an NVIDIA GeForce GTX 960 GPU I have a server with one of those Skylake E3s, it is very nice “450Mbps Wireless N Dual Band PCI-e Adapter w/ 3x 2dBi Antennas (Yes, sad to say, unless I rewire my house I'll have to use Wi-Fi with this beast. I'll probably rewire my house.)” He plans to have a 4x 1TB ZFS pool, plus a second pool backed by a 512 GB NVMe m.2 for the OS “And I'll continue to chronicle my journey into the land of FOSS here when The Beast arrives. But in the meantime, please excuse me, I've got some reading to do.” Leo went so far as to slap a “Power By FreeBSD” sticker (https://youtu.be/vNVst_rxxm0?t=270) on the back of his new Tesla *** OpenBSD 6.0 to be released on Sept 1st, 2016 (http://undeadly.org/cgi?action=article&sid=20160725100831) OpenBSD 6.0 Tenative Released Notes (https://www.openbsd.org/60.html) OpenBSD 6.0 is just around the corner, currently slated for Sept 1st and brings with it a whole slew of exciting new features First up, and let's get this right out of the way.. VAX support has been dropped!! Oh no! However to make up for this devastating loss, armv7 has been added to this release. The tentative release notes are very complete and marks 6.0 as quite an exciting release OpenBSD 6.0 Pre-orders up (http://undeadly.org/cgi?action=article&sid=20160726230851) OpenBSD 6.0 tightens security by losing Linux compatibility (http://www.infoworld.com/article/3099038/open-source-tools/openbsd-60-tightens-security-by-losing-linux-compatibility.html) In related news, infoworld picked up on the pending removal of Linux compat from OpenBSD 6.0. Touted as a security feature, you will soon be unable to run legacy linux binaries on OpenBSD. This has both positives and negatives depending upon your use case. Ironically we're excitedly awaiting improved Linux Compat support in FreeBSD, to allow running some various closed-source applications. (Netflix DRM, Steam, Skype to name a few) *** EuroBSDCon 2016 Schedule released (https://2016.eurobsdcon.org/talks-schedule/) EuroBSDCon 2016 Tutorial Schedule released (https://2016.eurobsdcon.org/tutorials/) EuroBSDCon has announced the list of talks and tutorials for September 22nd-25th's conference! George Neville Neil (Who we've interviewed in the past) is giving the keynote about “The Coming Decades of BSD” *** News Roundup Blast from the past No interview again this week, we're working on getting some people lined up. The Leo Laporte story brought these old gem from TechTV into my youtube playlist: Matt Olander and Murrey Stokey explain FreeBSD on TechTV (https://www.youtube.com/watch?v=d0UsXwRvaIg) Matt Olander and Brooks Davis explain building a cluster with FreeBSD on TechTV (https://www.youtube.com/watch?v=bAsYz5pVwyc) FreeBSD vs Linux Part 1 (https://www.youtube.com/watch?v=91igg2UX7o8) FreeBSD vs Linux Part 2 (https://www.youtube.com/watch?v=oU88fQkwfws) *** Running FreeBSD on the LibreM (https://ericmccorkleblog.wordpress.com/2016/07/16/freebsd-librem-update/) Eric McCorkle (Who has worked on the EFI loader for a while now) has written an update on his efforts to get FreeBSD working properly on the LibreM 13 laptop. Since April the work seems to be progressing nicely Matt Macy's i915 graphics patch works well on the Librem 13, and I personally made sure that the suspend/resume support works. The patch is very stable on the Librem, and I've only had one kernel panic the entire time testing it. The HDMI output Just Works™ with the i915 driver. Even better, it works for both X11 and console modes. Full support for the Atheros 9462 card has been merged in. I've had some occasional issues, but it works for the most part. The vesa weirdness is obviated by i915 support, but it was resolved by using the scfb driver. Some of the outstanding issues still being worked on are support for Synaptics on this particular touchpad, as well as hotkey support for the keyboard, and brightness controls. In addition Eric is still working on the EFI + Geli support, with the eventual goal of getting EFI secure-boot working out of box as well. More OpenBSD syscall fuzzing (http://seclists.org/oss-sec/2016/q3/157) NCC Group's Project Triforce continues its work of fuzzing OpenBSD This time they have found a flaw that allows any user to panic the kernel Attempting to read from the tmpfs_vfsops sysctl tree will panic the system: “attempt to execute user address 0x0 in supervisor mode” This is actually a “good” thing… “Impact: Any user can panic the kernel by using the sysctl call. If a user can manage to map a page at address zero, they may be able to gain kernel code execution and escalate privileges” OpenBSD's default configuration prevents mapping a page at address zero, so the code execution is prevented So while a panic is a bad outcome, it is a lot better than it could have been *** Root privilege escalation on NetBSD (http://akat1.pl/?id=2) This post described a root privilege escalation in NetBSD mail.local is a utility included in the base system for delivering mail to other users on the same system, rather than invoking a mail client and going through the mail server. The mail.local utility contains a ‘time of check / time of use' vulnerability. This means that it checks if a file or permission is valid, and then later accesses that file. If an attacker can change that file between the time when it is checked, and the time when it is used, they may be able to exploit the system by evading the check This is exactly what happens in this case mail.local appends a message to the indicated user's mailbox It first checks if the target user already has an existing mailbox file. If the file exists, but is a link, mail.local exits with an error (to prevent exploits) If the file does not exist, it is created The message is then appended to the file If the file needed to be created, it is chown'd to the owner of the mailbox This is where the problem lies, if mail.local checks and does not find the mailbox, but an attacker then creates a link from the target mailbox to some other file mail.local then appends to that file instead, thinking it is creating the new mailbox Then, mail.local chown's the target file to the user the attacker was trying to send mail to The article explains how this could be used to replace /etc/master.passwd etc, but opts for an easier proof of concept, replacing /usr/bin/atrun, which is run as root every 5 minutes from crontab with a script that will copy the shell to /tmp and mark it setuid The attacker can then run that shell out of /tmp, and be root NetBSD fixed the vulnerability by changing the code flow, separating the cases for opening an existing file from creating a new file. In the case where an existing file is opened, the code then verifies that the file that was opened has the same inode number and is on the same device, as the file that was checked earlier, to ensure it was not a link *** FreeBSD Heap vulnerability in bspatch (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:25.bspatch.asc) An important vuln has been found and fixed in FreeBSD this past week, specifically relating to the ‘bspatch' utility. “Upstream's bspatch.c implementation doesn't check for negative values on the number of bytes to read from the "diff" and "extra" streams, allowing an attacker controlling the patch file to write at arbitrary locations in the heap.” This could result in a crash, or running arbitrary code as the user running bspatch. (Often root) “bspatch's main loop reads three numbers from the "control" stream in the patch: X, Y and Z. The first two are the number of bytes to read from "diff" and "extra" (and thus only non-negative), while the third one could be positive or negative and moves the oldpos pointer on the source image. These 3 values are 64bits signed ints (encoded somehow on the file) that are later passed the function that reads from the streams, but those values are not verified to be non-negative.” “Chrome[OS] has four different implementations of this program, all derived from the same original code by Colin Percival.” Chromium Issue Tracker (https://bugs.chromium.org/p/chromium/issues/detail?id=372525) Patch your systems now! *** Beastie Bits: If you're a BUG member or Organizer, please contact BSD Now (https://twitter.com/q5sys/status/758087886927388673) TedU writes about some interesting localizations to gcc in openbsd, and why they are there (http://www.tedunangst.com/flak/post/one-reason-to-hate-openbsd) List of Products based on FreeBSD -- Help complete the list (https://en.wikipedia.org/wiki/List_of_products_based_on_FreeBSD) Virtualbox v5 hits the FreeBSD Ports tree (http://www.freshports.org/emulators/virtualbox-ose/) Skull Canyon NUC booting FreeBSD 11.0-BETA2 (https://gist.github.com/gonzopancho/b71be467f45594822131f4816d6cb718) 2016 BSDCan Trip Report : Trent Thompson (https://www.freebsdfoundation.org/blog/2016-bsdcan-trip-report-trent-thompson/) August London BSD Meetup (http://mail-index.netbsd.org/regional-london/2016/07/25/msg000542.html) Feedback/Questions Michael Open-Source Alts (http://pastebin.com/eiWbDXTd) Herminio - AP Troubles (http://pastebin.com/w9aCDBut) Jake - Plasma (http://pastebin.com/d15QpVFw) Morgan - Clean DO Droplets (http://pastebin.com/Wj1P7jq8) Chris - Auditd (http://pastebin.com/U9PYEH6K) ***
This week on BSDNow, we have all sorts of interesting news, including a Kernel Fuzzing audit done for OpenBSD, a much improved This episode was brought to you by Headlines Multiple Bugs in OpenBSD Kernel (http://marc.info/?l=oss-security&m=146853062403622&w=2) Its patch Wednesday! (OR last Thursday if you were watching the mailing lists) Jesse Hertz and Tim Newsham (part of the NCC Group calling themselves project Triforce) have been working with the OpenBSD team to fix some newly discovered bugs in the kernel using fuzzing. Specifically they were able to track down several potential methods to corrupt memory or panic the kernel: mmap_panic: Malicious calls to mmap() can trigger an allocation panic or trigger memory corruption. kevent_panic: Any user can panic the kernel with the kevent system call. thrsleeppanic: Any user can panic the kernel with the _thrsleep system Call. thrsigdivertpanic: Any user can panic the kernel with the _thrsigdivert system call. ufsgetdentspanic: Any user can panic the kernel with the getdents system call. mount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when mounting a tmpfs filesystem. unmount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when unmounting a filesystem. tmpfsmknodpanic: Root can panic kernel with mknod on a tmpfs filesystem. This was a great find, and we have a link to more of the results, if you would like to explore them in more detail. NCC Group OpenBSD Kernel fuzzing results (http://www.openwall.com/lists/oss-security/2016/07/14/5) Would like to see more work like this done in all of the BSDs *** Running CockroachDB in a FreeBSD Jail (https://www.cockroachlabs.com/blog/critters-in-a-jar-running-cockroachdb-in-a-freebsd-jail/) The developers behind CockroachDB have written up a nice walkthrough of getting their software to run inside FreeBSD jails. “Manually encapsulating CockroachDB using Linux cgroups is no easy task, which is why tools like Docker exist in the first place. By comparison, running server processes natively in FreeBSD jails is straightforward and robust.” The walkthrough begins with compiling CockroachDB straight from source (A port is pending), which is pretty easy relying upon bash / git / gmake and GO. With the compile finished, the next step will be mounting linprocfs, although that may be going away in the future: “(Note: Linux compatibility files / packages / libraries are not needed further. CockroachDB uses Linux's procfs to inspect system properties via gosigar. If/when gosigar evolves to read FreeBSD properties natively, CockroachDB will not need linprocfs any more.)” With the initial setup complete, the walkthrough then takes us through the process of creating the rc.d script (Which should be included with the port) and ultimately setting up ezjail and deploying CockroachDB within. With the word getting out about jails and their functionality, we hope to see more projects also provide walkthroughs and FreeBSD support natively. Kudos to the CockroachDB team! *** Usermount bugs (https://marc.info/?l=openbsd-announce&m=146854517406640&w=2) kern.usermount, (vfs.usermount on FreeBSD) is a sysctl that can be enabled to allow an unprivileged user to mount filesystems. It is very useful for allowing non-root users to mount a USB stick or other external media. It is not without its dangers though: > “kern.usermount=1 is unsafe for everyone, since it allows any non-pledged program to call the mount/umount system calls. There is no way any user can be expected to keep their system safe / reliable with this feature. Ignore setting to =1, and after release we'll delete the sysctl entirely.” In OpenBSD 6.0 and forward, the setting will no longer work, and root privileges will be required to mount a filesystem If there is a bug in the filesystem driver, the user could potentially exploit that and root the system > “In addition to the patched bugs, several panics were discovered by NCC that can be triggered by root or users with the usermount option set. These bugs are not getting patched because we believe they are only the tip of the iceberg. The mount system call exposes too much code to userland to be considered secure” This is a very pragmatic way of dealing with these issues, as it is not really possible to be sure that EVERY bug has been fixed, and that this feature is no longer an exploit vector usermount being removed from OpenBSD (http://undeadly.org/cgi?action=article&sid=20160715125022) I use this facility in FreeBSD extensively, combined with ZFS permission delegation, to allow non-root users to create and mount new ZFS datasets, and to do replication without requiring any root access There are some safety belts, for instance: the user must own the directory that the new filesystem will be mounted to, so they can't mount to /etc and replace the password file with their own *** Let's Encrypt client from BSD in C (https://kristaps.bsd.lv/letskencrypt/) File this one under the category of “It's about time!”, but Kristaps (Who we've interviewed in the past) has released some new software for interacting with letsencrypt. The header for the project site sums it up nicely: “Be up-front about security: OpenSSL is known to have issues, you can't trust what comes down the pipe, and your private key's integrity is a hard requirement. Not a situation where you can be careless. letskencrypt is a client for Let's Encrypt users, but one designed for security. No Python. No Ruby. No Bash.A straightforward, open source implementation in C that isolates each step of the sequence.” What specifically does it isolate you ask? Right now it is broken down into 6 steps: read and parse an account and domain private key authenticate with the Let's Encrypt server authorise each domain listed for the certificate submit the X509 request receive and serialise the signed X509 certificate request, receive, and serialise the certificate chain from the issuer I don't know about all of you, but I'm going to be switching over one of my systems this weekend. *** News Roundup Videos from the FOSDEM BSD Dev room are now online (https://video.fosdem.org/2016/k4601/) The videos from the BSD Dev room at FOSDEM have been stealthily posted online at some point since last I checked The videos are individually linked from the talks on the Schedule (https://archive.fosdem.org/2016/schedule/track/bsd/) The talk pages also include the slides, which can help you to follow along *** FreeBSD on Jetson TK1 (http://kernelnomicon.org/?p=628) The nVidia Jetson TK1 is a medium sized ARM device that is a big more than your standard Raspberry Pi The device has: NVIDIA 4-Plus-1™ Quad-Core ARM® Cortex™-A15 CPU (2.3 GHz) NVIDIA Kepler GPU with 192 CUDA Cores 2 GB DDR3L x16 Memory with 64-bit Width 16 GB 4.51 eMMC Memory 1 Half Mini-PCIE Slot 1 Full-Size SD/MMC Connector 1 Full-Size HDMI Port 1 USB 2.0 Port, Micro AB 1 USB 3.0 Port, A 1 RS232 Serial Port 1 ALC5639 Realtek Audio Codec with Mic In and Line Out 1 RTL8111GS Realtek GigE LAN 1 SATA Data Port SPI 4 MByte Boot Flash The following signals are available through an expansion port: DP/LVDS Touch SPI 1x4 + 1x1 CSI-2 GPIOs UART HSIC i2c The device costs $192 USD from nVidia or Amazon Oleksandr Tymoshenko (gonzo@freebsd.org) has a post describing what it takes to get FreeBSD running on the Jetson TK1 > “First of all – my TK1 didn't have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader.” They tried using the provided tool, compiled on FreeBSD since it uses libusb, but it gave an error. Falling back to trying from Ubuntu, they got the same error. They then flashed the TK1 with newer firmware, and suddenly, uboot is available. The post then walks through pxe booting FreeBSD on the TK1 The guide then walks through replacing the UBoot with a version compatible with UBLDR, for more features We'll have to wait for another post to get FreeBSD burned onto the device, but at this point, you can reliably boot it without any user interaction I have one of these devices, so I am very interested in this work *** Why we use OpenBSD at VidiGaurd (https://blog.vidiguard.com/why-we-use-openbsd-at-vidiguard-4521f217b2b7#.9r86v742v) VidiGuard (Which makes autonomous drone solutions for security monitoring) has posted an interesting write-up on why they use OpenBSD. Specifically they start by mentioning while they are in business to provide physical security, they just as equally value their data security, especially their customer data. They name 4 specific features that matter to them, starting with Uncompromising Quality and Security: “Over the past 20 years, OpenBSD's focus on uncompromising quality and code correctness has yielded an operating system second-to-none. Code auditing and review is core to the project's development process. The team's focus on security includes integrated cryptography, new security mitigation techniques, and an optional-security-is-no-security stance, making it arguably the most secure operating system available today. This approach pays off in the form of only a few security updates for a given release, compared to other operating systems that might release a handful of updates every week.” High praise indeed! They also mention the sane-defaults, documentation and last but not least, the license as also winning factors in making OpenBSD their operating system of choice. Thanks to VidiGuard for publically detailing the use of BSD, and we hope to see other business follow suit! *** "You can (and should) slow down and learn how things work" – Interview with Dru Lavigne (https://bsdmag.org/dru_lavigne/) If you've been around the BSD community for any length of time, you no doubt have heard of Dru Lavigne (Or perhaps own one of her books!) She was recently interviewed by Luca Ferrari for BSD Magazine and you may find it a fascinating read. The 2nd question asked sounded a lot like our opener to an interview (How did you get into BSD) “ In the mid 90s, I went back to school to learn network and system administration. As graduation grew near and I started looking for a work, I noticed that all the interesting jobs wanted Unix skills. Wanting to increase my skills, and not having any money, I did an Internet search for “Free Unix”. The first hit was freebsd.org. I went to the website and started reading the Handbook and thought “I can do this”. Since I only had access to one computer and wanted to ramp up my skills quickly, I printed out the installation and networking chapters of the Handbook. I replaced the current operating system with FreeBSD and forced myself to learn how to do everything I needed to do on that computer in FreeBSD. It was a painful (and scary) few weeks as I figured out how to transition the family's workflow to FreeBSD, but it was also exhilarating to learn that “yes, I can do this!. Since then, I've had the opportunity to try out or administer the other BSDs, several Linux distros, SCO, and Solaris. I found that the layout, logic, and release engineering process of the BSDs makes the most sense to me and I'm happiest when on a BSD system.” When asked, Dru also had a good response to what challenges potential new UNIX or BSD users may face: “Students who haven't been exposed to open source before are used to thinking of technology in terms of a purchasable brand consisting of “black boxes” that are supposed to “just work”, without having to think about how they work. You can (and should) slow down and learn how things work. It can be a mind shift to learn that the freedom to use and change how something works does exist, and isn't considered stealing. And that learning how something works, while hard, can be fun. BSD culture, in particular, is well suited for those who have the time and temperament to dive into how things work. With over 40 years of freely available source and commit messages, you can dive as deep as you want into learning how things came to be, how they evolved over the years, how they work now, and how they can be improved. There is a diverse range of stuff to choose from: from user tools to networking to memory management to hardware drivers to security mechanisms and so on. There is also a culture of sharing and learning and encouragement for users who demonstrate that they have done their homework and have their own ideas to contribute.” The interview is quite long, and Dru provides fantastic insights into more aspects of BSD in general. Well worth your time to read! *** Beastie Bits: Ed Maste is seeking testing 'withoutgpldtc' (https://twitter.com/ed_maste/status/755474764479672321) “PAM Mastery” tech reviewers wanted (http://blather.michaelwlucas.com/archives/2717) OPNsense 16.7 RC2 (https://opnsense.org/opnsense-16-7-rc2-released/) Jupyter Notebook for bootstrapping Arduino on FreeBSD (https://nbviewer.jupyter.org/github/DadAtH-me/Projects/blob/master/arduino-on-nix.ipynb) The Design and Implementation of the Anykernel and Rump Kernels (second edition) (http://www.fixup.fi/misc/rumpkernel-book/) Complete desktop synchronisation with Unison and FreeBSD jails (xjails) (https://github.com/kbs1/freebsd-synced-xjails) Feedback/Questions Eric - List most popular files (http://pastebin.com/S7u0VeVi) Robroy - ZFS Write Cache (http://pastebin.com/81Zmj0cX) Luis - FreeNAS HW Setup (http://pastebin.com/SfeKR7v2) Emett - Python Followup (http://pastebin.com/wy4ar0YH) Peter - Multicast + Jails (http://pastebin.com/zd2QAu25) ***
Today on the show, we are going to be talking to Jim Brown (of BSD Cert Fame) about his home-brew sprinkler system… Wait for it… This episode was brought to you by Headlines Distrowatch reviews OpenBSD and PCBSD's live upgrade method (http://distrowatch.com/weekly.php?issue=20160620#upgrade) Upgrading… The bane of any sysadmin! Distrowatch has recently done a write-up on the in-place upgrading of various distros / BSDs including PC-BSD and OpenBSD. Lets look first at the PC-BSD attempt, which was done going from 9.2 -> 10. “I soon found trying to upgrade either the base system or pkg would fail. The update manager did not provide details as to what had gone wrong and so I decided to attempt a manual upgrade by following the FreeBSD Handbook as I had when performing a live upgrade of FreeBSD back in May. At first the manual process seemed to work, downloading the necessary patches for FreeBSD 10 and getting me to resolve conflicts between my existing configuration files and the new versions. Part way through, we are asked to reboot and then continue the upgrade process using the freebsd-update command utility. PC-BSD failed to reboot and, in fact, the boot loader no longer found any operating systems to run.” Ouch! I'm not sure on the particular commands used, but to lose the boot-loader indicates something went horribly wrong. There is good news in this though. After the pain experienced in the 9.X upgrade process, 11.0 has been vastly improved to help fix this going forward. The updater is also self-updating, which means future changes to tools such as package can be accounted for in previously released versions. Moving on to OpenBSD, Jesse had much better luck: > “The documentation provided explains how to upgrade OpenBSD 5.8 to version 5.9 step-by-step and the instructions worked exactly as laid out. Upgrading requires two reboots, one to initiate the upgrade process and one to boot into the new version of OpenBSD. Upgrading the base operating system took approximately ten minutes, including the two reboots. Upgrading the third-party packages took another minute or two. The only quirk I ran into was that I had to manually update my repository mirror information to gain access to the new packages available for OpenBSD 5.9. If this step is not done, then the pkg_add package manager will continue to pull in packages from the old repository we set up for OpenBSD 5.8. “ A good read, and they covered some Linux distros such as Mint and OpenMandriva as well, if you want to find out how they fared. *** A curated list of awesome DTrace books, articles, videos, tools and resources (http://awesome-dtrace.com/) The website awesome-dtrace.com compiles a list of resources, including books, articles, videos, tools, and other resources, to help you get the most out of DTrace The list of books includes 2 open source books that are available on the web, and of course Brendan Gregg's official DTrace book There are also cheat sheets, one-liner collections, and a set of DTrace war stories A breakdown of different PID providers and the userspace statically defined tracepoints The videos from DTrace.conf 2008, 2012, and soon 2016 And links to the tools to start using DTrace with your favourite programming language, including Erlang, Node.JS, Perl, PHP, Python, or Ruby There are also DTrace setups for MySQL/MariaDB, and PostreSQL Joyent has even written a mod_usdt DTrace module for the Apache web server This seems like a really good resource, and with the efforts of the new OpenDTrace project, to modernize the dtracetoolkit and make it more useful across the different supported operating systems, there has never been a better time to start learning DTrace *** Installing OpenBSD using a serial console with no external monitor (http://unix.stackexchange.com/questions/292891/how-can-i-install-openbsd-using-the-serial-console-without-external-monitor-wi) Have you found yourself needing to install OpenBSD from USB, but with a twist, as in no external monitor? Well somebody has and asked the question on stackexchange. The answer provided is quite well explained, but in a nut-shell the process involves downloading the USB image and making some tweaks before copying it to the physical media. Specifically with a couple of well-placed echo's into boot.conf, the serial-port can be enabled and ready for use: echo "stty com0 115200" > /mnt/etc/boot.conf echo "set tty com0" >> /mnt/etc/boot.conf + After that, simply boot the box and you are ready to access the serial console and drive the installation as normal! #bsdhacks GSoC 2016 Reports: Split debug symbols for pkgsrc builds (https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug) The NetBSD blog provides a status report on one of the GSoC projects that is nearing its midterm evaluation The project to split debugging data into separate pkgsrc packages, so that users can install the debugging symbols if they need them to debug a failing application The report is very detailed, and includes “A quick introduction to ELF and how debug information are stored/stripped off” It walks through the process of writing a simple example application, compiling it, and dealing with the debug data It includes a number of very useful diagrams, and a summary of what changes needed to be make to the pkgsrc makefile infrastructure With this as a recipe, someone should be able to do something quite similar for FreeBSD's ports tree *** iXsystems iXsystems' TrueNAS Firmware Update Delivers Compelling Performance, Replication, and Graphing Improvements (https://www.ixsystems.com/blog/ixsystems-truenas-firmware-update-delivers-compelling-performance-replication-graphing-improvements/) *** Interview - Jim Brown - jpb@jimby.name (mailto:jpb@jimby.name) FreeBSD+BBB Sprinkler System News Roundup From the past : A Research Unix Reader (http://www.cs.dartmouth.edu/~doug/reader.pdf) A paper by by Douglas McIlroy “Selected pages from the nine research editions of the UNIX® Programmer's Manual illustrate the development of the system” “Accompanying commentary recounts some of the needs, events, and individual contributions that shaped this evolution.” Interesting insight into the evolution of the origin UNIX operating system *** Evolution of C programming practices – Unix 1973–2015 (http://kristerw.blogspot.com/2016/06/evolution-of-c-programming-practices.html) From the author of the recent post we covered, “20 years of NetBSD code bloat”, comes a new post “I found a recent paper that also looks at how the BSD code base has evolved, but from a very different perspective compared to my code-size investigation.” The paper "The Evolution of C Programming Practices: A Study of the Unix Operating System 1973–2015" investigates coding style, and tests seven hypotheses by looking at metrics (line length, number of volatile in the source code, etc.) in 66 releases of Unix from 1973 to 2014. The hypotheses are: > + Programming practices reflect technology affordances (e.g. developers may be more liberal with screen space when using high resolution displays) > + Modularity increases with code size > + New language features are increasingly used to saturation point > + Programmers trust the compiler for register allocation > + Code formatting practices converge to a common standard > + Software complexity evolution follows self correction feedback mechanisms > + Code readability increases and the result is that they seem to be true, as interpreted through the metrics. > “The data points for the releases have somewhat random dates. One issue is that the paper use each release's mean file date (the average of the files' last modification time) instead of the release date (that is why the graphs stop at November 2010, even though FreeBSD 10 was released in 2014). The idea is that this better reflects the age of the code base, but this has the effect of compressing some of the data points (especially the clustering around 1993-1994), and it makes the spline fitting even more suspect.” > “One other problem is that the original data used by the researchers seems to have incorrect timestamps. For example, 4.3BSD Net/1 was released in 1989, but is listed as 1993-12-25 in the paper. The same is true for at least the Net/2 release too, which was released in 1991, but the paper list it as 1993-07-02.” *** [old release pictures] openbsd 2.1 - 5.9, straight from theo's bookshelf. (https://twitter.com/blakkheim/status/747540167112671232) Speaking of old releases, our Producer JT picked up this gem at Southeast Linuxfest this year (https://twitter.com/q5sys/status/748003859012984837) Noah Axon shares a scan of his NetBSD 1.4 disc (https://plus.google.com/+NoahAxon/posts/VsiQhUn3tHb) Jan van den broek shares a pic of his FreeBSD 2.2.5 set (https://plus.google.com/101232368324501316985/posts/4QsaJE2KxXh) *** FreeBSD: Just in Time (https://bsdmag.org/just_in_time/) Another BSDMag goodie this week, we have a small article written by Jonathan Garrido which details their experience switching to FreeBSD for a NTP server. The article is short, but a good read: > “A Few years ago we had a time problem. Suddenly our linux NTP server, for a reason that I still do not know, started to fail giving us a lot of issues within all the equipment and services within our network. After a quick and brief meeting with management, I found out that there was not sufficient budget left for a fancy and well-suited appliance. So, with no time (literally) and no money to spend, I decided to give it a try and utilized a homemade open source solution, and the operating system of choice was FreeBSD 10.0.” “Now, let's pause for a second. You may be thinking, why in the world is this guy doing this, when he has never installed a BSD machine in his life? The answer is very simple; here, in the Dominican Republic, in the heart of the Caribbean, FreeBSD has a very good reputation when it comes to reliability and security. In fact, there is some collective thought within the sysadmin community that says something like: “If you want to deal only once with a service, install it over FreeBSD.”” Jonathan then goes through some of the steps taken to initial deploy NTP services, but with that out of the way, he has a great summary: > “Fascinated with the whole experience, we migrate one of our internal dns servers to a second FreeBSD machine and at the moment of this writing we are testing haproxy, an open source load-balancing proxy into a another server with the same OS. > After all this, no time issues have been reported in the past 2 years, so at least for my environment, FreeBSD came just in time.“ *** Beastie Bits MiniBSD laptop computer (https://hackaday.io/project/643-minibsd-laptop-computer) The state of LibreSSL in FreeBSD (https://attilagyorffy.com/2016/07/02/the-state-of-libressl-in-freebsd/) Justin Sherrill is looking for someone willing to run a Go builder with DragonflyBSD (https://www.dragonflydigest.com/2016/07/01/18372.html) Tiny Unix tools for Windows (https://tinyapps.org/blog/windows/201606040700_tiny_unix_tools_windows.html) OpenBSD's doas added to the FreeBSD Ports Tree (http://www.freshports.org/security/doas/) ubuntuBSD 16.04 to feature a combo of BusyBox and OpenRC, no systemd (http://linux.softpedia.com/blog/ubuntubsd-16-04-will-feature-a-combination-of-busybox-and-openrc-but-no-systemd-505463.shtml) Syncast Podcast 4 : Curl, libcurl and the future of the web, with Daniel Stenberg (http://podcast.sysca.st/podcast/4-curl-libcurl-future-web-daniel-stenberg/) Feedback/Questions Harri - Using beadm / zfssnap (http://pastebin.com/qKeCd63F) Jonathan - bhyve vs Proxmox (http://pastebin.com/EhXDwbWQ) Mohammad - Bhyve gfx passthrough (http://pastebin.com/ZCNk4Bga) Jeremy - Shapshots and more Snapshots (http://pastebin.com/xp7nzEYa) Ron - Microphone (http://pastebin.com/H2xr53CR) ***