Podcasts about cddl

Causing ZFS corruption for fun, NetBSD Assembly Programming Tutorial, The IKEA Lack Rack for Servers, a new OmniOS Community Edition LTS has been published, List Block Devices on FreeBSD lsblk(8) Style, Project Trident 19.10 available, and more. Headlines Causing ZFS corruption for fun and profit (https://datto.engineering/post/causing-zfs-corruption) Datto backs up data, a lot of it. At the time of writing Datto has over 500 PB of data stored on ZFS. This count includes both backup appliances that are sent to customer sites, as well as cloud storage servers that are used for secondary and tertiary backup of those appliances. At this scale drive swaps are a daily occurrence, and data corruption is inevitable. How we handle this corruption when it happens determines whether we truly lose data, or successfully restore from secondary backup. In this post we'll be showing you how at Datto we intentionally cause corruption in our testing environments, to ensure we're building software that can properly handle these scenarios. Causing Corruption Since this is a mirror setup, a naive solution to cause corruption would be to randomly dd the same sectors of both /dev/sdb and /dev/sdc. This works, but is equally likely to just overwrite random unused space, or take down the zpool entirely. What we really want is to corrupt a specific snapshot, or even a specific file in that snapshot, to simulate a more realistic minor corruption event. Luckily we have a tool called zdb that lets us view some low level information about datasets. Conclusion At the 500 PB scale, it's not a matter of if data corruption will happen but when. Intentionally causing corruption is one of the strategies we use to ensure we're building software that can handle these rare (but inevitable) events. To others out there using ZFS: I'm curious to hear how you've solved this problem. We did quite a bit of experimentation with zinject before going with this more brute force method. So I'd be especially interested if you've had luck simply simulating corruption with zinject. NetBSD Assembly Programming Tutorial (https://polprog.net/blog/netbsdasmprog/) A sparc64 version is also being prepared and will be added when done This post describes how to write a simple hello world program in pure assembly on NetBSD/amd64. We will not use (nor link against) libc, nor use gcc to compile it. I will be using GNU as (gas), and therefore the AT&T syntax instead of Intel. Why assembly? Why not? Because it's fun to program in assembly directly. Contrary to a popular belief assembly programs aren't always faster than what optimizing compilers produce. Nevertheless it's good to be able to read assembly, especially when debugging C programs Due to the nature of the guide, visit the site for the complete breakdown News Roundup The IKEA Lack Rack for Servers (https://wiki.eth0.nl/index.php/LackRack) The LackRack First occurrence on eth0:2010 Winterlan, the LackRack is the ultimate, low-cost, high shininess solution for your modular datacenter-in-the-living-room. Featuring the LACK (side table) from Ikea, the LackRack is an easy-to-implement, exact-fit datacenter building block. It's a little known fact that we have seen Google engineers tinker with Lack tables since way back in 2009. The LackRack will certainly make its appearance again this summer at eth0:2010 Summer. Summary When temporarily not in use, multiple LackRacks can be stacked in a space-efficient way without disassembly, unlike competing 19" server racks. The LackRack was first seen on eth0:2010 Winterlan in the no-shoe Lounge area. Its low-cost and perfect fit are great for mounting up to 8 U of 19" hardware, such as switches (see below), or perhaps other 19" gear. It's very easy to assemble, and thanks to the design, they are stable enough to hold (for example) 19" switches and you can put your bottle of Club-Mate on top! Multi-shiny LackRack can also be painted to your specific preferences and the airflow is unprecedented! Howto You can find a howto on buying a LackRack on this page. This includes the proof that a 19" switch can indeed be placed in the LackRack in its natural habitat! OmniOS Community Edition r151030 LTS - Published at May 6, 2019 (https://omniosce.org/article/release-030) The OmniOS Community Edition Association is proud to announce the general availability of OmniOS - r151030. OmniOS is published according to a 6-month release cycle, r151030 LTS takes over from r151028, published in November 2018; and since it is a LTS release it also takes over from r151022. The r151030 LTS release will be supported for 3 Years. It is the first LTS release published by the OmniOS CE Association since taking over the reins from OmniTI in 2017. The next LTS release is scheduled for May 2021. The old stable r151026 release is now end-of-life. See the release schedule for further details. This is only a small selection of the new features, and bug fixes in the new release; review the release notes for full details. If you upgrade from r22 and want to see all new features added since then, make sure to also read the release notes for r24, r26 and r28. For full relase notes including upgrade instructions; release notes (https://omniosce.org/releasenotes.html) upgrade instructions (https://omniosce.org/upgrade.html) List Block Devices on FreeBSD lsblk(8) Style (https://vermaden.wordpress.com/2019/09/27/list-block-devices-on-freebsd-lsblk8-style/) When I have to work on Linux systems I usually miss many nice FreeBSD tools such as these for example to name the few: sockstat, gstat, top -b -o res, top -m io -o total, usbconfig, rcorder, beadm/bectl, idprio/rtprio,… but sometimes – which rarely happens – Linux has some very useful tool that is not available on FreeBSD. An example of such tool is lsblk(8) that does one thing and does it quite well – lists block devices and their contents. It has some problems like listing a disk that is entirely used under ZFS pool on which lsblk(8) displays two partitions instead of information about ZFS just being there – but we all know how much in some circles the CDDL licensed ZFS is unloved in that GPL world. Example lsblk(8) output from Linux system: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 931.5G 0 disk |-sda1 8:1 0 500M 0 part /boot `-sda2 8:2 0 931G 0 part |-vg_local-lv_root (dm-0) 253:0 0 50G 0 lvm / |-vg_local-lv_swap (dm-1) 253:1 0 17.7G 0 lvm [SWAP] `-vg_local-lv_home (dm-2) 253:2 0 1.8T 0 lvm /home sdc 8:32 0 232.9G 0 disk `-sdc1 8:33 0 232.9G 0 part `-md1 9:1 0 232.9G 0 raid10 /data sdd 8:48 0 232.9G 0 disk `-sdd1 8:49 0 232.9G 0 part `-md1 9:1 0 232.9G 0 raid10 /data What FreeBSD offers in this department? The camcontrol(8) and geom(8) commands are available. You can also use gpart(8) command to list partitions. Below you will find output of these commands from my single disk laptop. Please note that because of WordPress limitations I need to change all > < characters to ] [ ones in the commands outputs. See the article for the rest of the guide Project Trident 19.10 Now Available (https://project-trident.org/post/2019-10-05_19.10_available/) This is a general package update to the CURRENT release repository based upon TrueOS 19.10 PACKAGE CHANGES FROM 19.08 New Packages: 601 Deleted Packages: 165 Updated Packages: 3341 Beastie Bits NetBSD building tools (https://imgur.com/gallery/0sG4b1K) Sponsorships open for SNMP Mastery (https://mwl.io/archives/4569) pkgsrc-2019Q3 release announcement (2019-10-03) (http://mail-index.netbsd.org/pkgsrc-users/2019/10/03/msg029485.html) pfetch - A simple system information tool written in POSIX sh (https://github.com/dylanaraps/pfetch) Taking NetBSD kernel bug roast to the next level: Kernel Fuzzers (quick A.D. 2019 overview) (https://netbsd.org/~kamil/eurobsdcon2019_fuzzing/presentation.html#slide1) Cracking Ken Thomson’s password (https://leahneukirchen.org/blog/archive/2019/10/ken-thompson-s-unix-password.html) Feedback/Questions Evilham - Couple Questions (http://dpaste.com/2JC85WV) Rob - APU2 alternatives and GPT partition types (http://dpaste.com/0SDX9ZX) Tom - FreeBSD journal article by A. Fengler (http://dpaste.com/2B43MY1#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.

DragonflyBSD 5.6 is out, OpenBSD Vulkan Support, bad utmp implementations in glibc and FreeBSD, OpenSSH protects itself against Side Channel attacks, ZFS vs OpenZFS, and more. Headlines DragonflyBSD 5.6 is out (https://www.dragonflybsd.org/release56) Version 5.6.0 released 17 June 2019 Version 5.6.1 released 19 June 2019 (https://www.dragonflydigest.com/2019/06/19/23091.html) Big-ticket items Improved VM Informal test results showing the changes from 5.4 to 5.6 are available. Reduce stalls in the kernel vmpagealloc() code (vmpagelist_find()). Improve page allocation algorithm to avoid re-iterating the same queues as the search is widened. Add a vmpagehash*() API that allows the kernel to do heuristical lockless lookups of VM pages. Change vmhold() and vmunhold() semantics to not require any spin-locks. Change vmpagewakeup() to not require any spin-locks. Change wiring vm_page's no longer manipulates the queue the page is on, saving a lot of overhead. Instead, the page will be removed from its queue only if the pageout demon encounters it. This allows pages to enter and leave the buffer cache quickly. Refactor the handling of fictitious pages. Remove m->md.pvlist entirely. VM pages in mappings no longer allocate pventry's, saving an enormous amount of memory when multiple processes utilize large shared memory maps (e.g. postgres database cache). Refactor vmobject shadowing, disconnecting the backing linkages from the vmobject itself and instead organizing the linkages in a new structure called vmmapbacking which hangs off the vmmapentry. pmap operations now iterate vmmapbacking structures (rather than spin-locked page lists based on the vmpage and pventry's), and will test/match operations against the PTE found in the pmap at the requisite location. This doubles VM fault performance on shared pages and reduces the locking overhead for fault and pmap operations. Simplify the collapse code, removing most of the original code and replacing it with simpler per-vmmapentry optimizations to limit the shadow depth. DRM Major updates to the radeon and ttm (amd support code) drivers. We have not quite gotten the AMD support up to the more modern cards or Ryzen APUs yet, however. Improve UEFI framebuffer support. A major deadlock has been fixed in the radeon/ttm code. Refactor the startup delay designed to avoid conflicts between the i915 driver initialization and X startup. Add DRMIOCTLGET_PCIINFO to improve mesa/libdrm support. Fix excessive wired memory build-ups. Fix Linux/DragonFly PAGE_MASK confusion in the DRM code. Fix idr_*() API bugs. HAMMER2 The filesystem sync code has been rewritten to significantly improve performance. Sequential write performance also improved. Add simple dependency tracking to prevent directory/file splits during create/rename/remove operations, for better consistency after a crash. Refactor the snapshot code to reduce flush latency and to ensure a consistent snapshot. Attempt to pipeline the flush code against the frontend, improving flush vs frontend write concurrency. Improve umount operation. Fix an allocator race that could lead to corruption. Numerous other bugs fixed. Improve verbosity of CHECK (CRC error) console messages. OpenBSD Vulkan Support (https://www.phoronix.com/scan.php?page=news_item&px=OpenBSD-Vulkan-Support) Somewhat surprisingly, OpenBSD has added the Vulkan library and ICD loader support as their newest port. This new graphics/vulkan-loader port provides the generic Vulkan library and ICD support that is the common code for Vulkan implementations on the system. This doesn't enable any Vulkan hardware drivers or provide something new not available elsewhere, but is rare seeing Vulkan work among the BSDs. There is also in ports the related components like the SPIR-V headers and tools, glsllang, and the Vulkan tools and validation layers. This is of limited usefulness, at least for the time being considering OpenBSD like the other BSDs lag behind in their DRM kernel driver support that is ported over from the mainline Linux kernel tree but generally years behind the kernel upstream. Particularly with Vulkan, newer kernel releases are needed for some Vulkan features as well as achieving decent performance. The Vulkan drivers of relevance are the open-source Intel ANV Vulkan driver and Radeon RADV drivers, both of which are in Mesa though we haven't seen any testing results to know how well they would work if at all currently on OpenBSD, but they're at least in Mesa and obviously open-source. + A note: The BSDs are no longer that far behind. + FreeBSD 12.0 uses DRM from Linux 4.16 (April 2018), and the drm-devel port is based on Linux 5.0 (March 2019) + OpenBSD -current as of April 2019 uses DRM from Linux 4.19.34 News Roundup Bad utmp implementations in glibc and freebsd (https://davmac.wordpress.com/2019/05/04/bad-utmp-implementations-in-glibc-and-freebsd/) I recently released another version – 0.5.0 – of Dinit, the service manager / init system. There were a number of minor improvements, including to the build system (just running “make” or “gmake” should be enough on any of the systems which have a pre-defined configuration, no need to edit mconfig by hand), but the main features of the release were S6-compatible readiness notification, and support for updating the utmp database. In other words, utmp is a record of who is currently logged in to the system (another file, “wtmp”, records all logins and logouts, as well as, potentially, certain system events such as reboots and time updates). This is a hint at the main motivation for having utmp support in Dinit – I wanted the “who” command to correctly report current logins (and I wanted boot time to be correctly recorded in the wtmp file). I wondered: If the files consist of fixed-sized records, and are readable by regular users, how is consistency maintained? That is – how can a process ensure that, when it updates the database, it doesn’t conflict with another process also attempting to update the database at the same time? Similarly, how can a process reading an entry from the database be sure that it receives a consistent, full record and not a record which has been partially updated? (after all, POSIX allows that a write(2) call can return without having written all the requested bytes, and I’m not aware of Linux or any of the *BSDs documenting that this cannot happen for regular files). Clearly, some kind of locking is needed; a process that wants to write to or read from the database locks it first, performs its operation, and then unlocks the database. Once again, this happens under the hood, in the implementation of the getutent/pututline functions or their equivalents. Then I wondered: if a user process is able to lock the utmp file, and this prevents updates, what’s to stop a user process from manually acquiring and then holding such a lock for a long – even practically infinite – duration? This would prevent the database from being updated, and would perhaps even prevent logins/logouts from completing. Unfortunately, the answer is – nothing; and yes, it is possible on different systems to prevent the database from being correctly updated or even to prevent all other users – including root – from logging in to the system. + A good find + On FreeBSD, even though write(2) can be asynchronous, once the write syscall returns, the data is in the buffer cache (or ARC), and any future read(2) will see that new data even if it has not yet been written to disk. OpenSSH gets an update to protect against Side Channel attacks (https://securityboulevard.com/2019/06/openssh-code-gets-an-update-to-protect-against-side-channel-attacks/) Last week, Damien Miller, a Google security researcher, and one of the popular OpenSSH and OpenBSD developers announced an update to the existing OpenSSH code that can help protect against the side-channel attacks that leak sensitive data from computer’s memory. This protection, Miller says, will protect the private keys residing in the RAM against Spectre, Meltdown, Rowhammer, and the latest RAMBleed attack. SSH private keys can be used by malicious threat actors to connect to remote servers without the need of a password. According to CSO, “The approach used by OpenSSH could be copied by other software projects to protect their own keys and secrets in memory”. However, if the attacker is successful in extracting the data from a computer or server’s RAM, they will only obtain an encrypted version of an SSH private key, rather than the cleartext version. In an email to OpenBSD, Miller writes, “this change encrypts private keys when they are not in use with a symmetric key that is derived from a relatively large ‘prekey’ consisting of random data (currently 16KB).” ZFS vs OpenZFS (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You’ve probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset’ and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs’ command, which on FreeBSD relies on the ‘zfs.ko’ kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. + There was further discussion of how the ZFSOnLinux repo will become the OpenZFS repo in the future once it also contains the bits to build on FreeBSD as well during the June 25th ZFS Leadership Meeting. The videos for all of the meetings are available here (https://www.youtube.com/channel/UC0IK6Y4Go2KtRueHDiQcxow) Beastie Bits How to safely and portably close a file descriptor in a multithreaded process without running into problems with EINTR (https://twitter.com/cperciva/status/1141852451756105729?s=03) KnoxBug Meetup June 27th at 6pm (http://knoxbug.org/2019-06-27) BSD Pizza Night, June 27th at 7pm, Flying Pie Pizzeria, 3 Monroe Pkwy, Ste S, Lake Oswego, OR (https://www.flying-pie.com/locations/lake-oswego/) Difference between $x and ${x} (https://moopost.blogspot.com/2019/06/difference-between-x-and-x.html) Beware of Software Engineering Media Sites (https://www.nemil.com/on-software-engineering/beware-engineering-media.html) How Verizon and a BGP optimizer knocked large parts of the internet offline today (https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/) DragonflyBSD - MDS mitigation added a while ago (http://lists.dragonflybsd.org/pipermail/commits/2019-May/718899.html) Reminder: Register for EuroBSDcon 2019 in Lillehammer, Norway (https://eurobsdcon.org) Feedback/Questions Dave - CheriBSD (http://dpaste.com/38233JC) Neb - Hello from Norway (http://dpaste.com/0B8XKXT#wrap) Lars - Ansible tutorial? (http://dpaste.com/3N85SHR) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** Your browser does not support the HTML5 video tag.

Jim and Wes sit down to bust some ZFS myths and share their tips and tricks for getting the most out of the ultimate filesystem. Plus when not to use ZFS, the surprising way your disks are lying to you, and more!

Jim and Wes are joined by OpenZFS developer Richard Yao to explain why the recent drama over Linux kernel 5.0 is no big deal, and how his fix for the underlying issue might actually make things faster. Plus the nitty-gritty details of vectorized optimizations and kernel preemption, and our thoughts on the future of the relationship between ZFS and Linux. Special Guest: Richard Yao.

OpenZFS and DTrace updates in NetBSD, NetBSD network security stack audit, Performance of MySQL on ZFS, OpenSMTP results from p2k18, legacy Windows backup to FreeNAS, ZFS block size importance, and NetBSD as router on a stick. ##Headlines ZFS and DTrace update lands in NetBSD merge a new version of the CDDL dtrace and ZFS code. This changes the upstream vendor from OpenSolaris to FreeBSD, and this version is based on FreeBSD svn r315983. r315983 is from March 2017 (14 months ago), so there is still more work to do in addition to the 10 years of improvements from upstream, this version also has these NetBSD-specific enhancements: dtrace FBT probes can now be placed in kernel modules. ZFS now supports mmap(). This brings NetBSD 10 years forward, and they should be able to catch the rest of the way up fairly quickly ###NetBSD network stack security audit Maxime Villard has been working on an audit of the NetBSD network stack, a project sponsored by The NetBSD Foundation, which has served all users of BSD-derived operating systems. Over the last five months, hundreds of patches were committed to the source tree as a result of this work. Dozens of bugs were fixed, among which a good number of actual, remotely-triggerable vulnerabilities. Changes were made to strengthen the networking subsystems and improve code quality: reinforce the mbuf API, add many KASSERTs to enforce assumptions, simplify packet handling, and verify compliance with RFCs. This was done in several layers of the NetBSD kernel, from device drivers to L4 handlers. In the course of investigating several bugs discovered in NetBSD, I happened to look at the network stacks of other operating systems, to see whether they had already fixed the issues, and if so how. Needless to say, I found bugs there too. A lot of code is shared between the BSDs, so it is especially helpful when one finds a bug, to check the other BSDs and share the fix. The IPv6 Buffer Overflow: The overflow allowed an attacker to write one byte of packet-controlled data into ‘packetstorage+off’, where ‘off’ could be approximately controlled too. This allowed at least a pretty bad remote DoS/Crash The IPsec Infinite Loop: When receiving an IPv6-AH packet, the IPsec entry point was not correctly computing the length of the IPv6 suboptions, and this, before authentication. As a result, a specially-crafted IPv6 packet could trigger an infinite loop in the kernel (making it unresponsive). In addition this flaw allowed a limited buffer overflow - where the data being written was however not controllable by the attacker. The IPPROTO Typo: While looking at the IPv6 Multicast code, I stumbled across a pretty simple yet pretty bad mistake: at one point the Pim6 entry point would return IPPROTONONE instead of IPPROTODONE. Returning IPPROTONONE was entirely wrong: it caused the kernel to keep iterating on the IPv6 packet chain, while the packet storage was already freed. The PF Signedness Bug: A bug was found in NetBSD’s implementation of the PF firewall, that did not affect the other BSDs. In the initial PF code a particular macro was used as an alias to a number. This macro formed a signed integer. NetBSD replaced the macro with a sizeof(), which returns an unsigned result. The NPF Integer Overflow: An integer overflow could be triggered in NPF, when parsing an IPv6 packet with large options. This could cause NPF to look for the L4 payload at the wrong offset within the packet, and it allowed an attacker to bypass any L4 filtering rule on IPv6. The IPsec Fragment Attack: I noticed some time ago that when reassembling fragments (in either IPv4 or IPv6), the kernel was not removing the MPKTHDR flag on the secondary mbufs in mbuf chains. This flag is supposed to indicate that a given mbuf is the head of the chain it forms; having the flag on secondary mbufs was suspicious. What Now: Not all protocols and layers of the network stack were verified, because of time constraints, and also because of unexpected events: the recent x86 CPU bugs, which I was the only one able to fix promptly. A todo list will be left when the project end date is reached, for someone else to pick up. Me perhaps, later this year? We’ll see. This security audit of NetBSD’s network stack is sponsored by The NetBSD Foundation, and serves all users of BSD-derived operating systems. The NetBSD Foundation is a non-profit organization, and welcomes any donations that help continue funding projects of this kind. DigitalOcean ###MySQL on ZFS Performance I used sysbench to create a table of 10M rows and then, using export/import tablespace, I copied it 329 times. I ended up with 330 tables for a total size of about 850GB. The dataset generated by sysbench is not very compressible, so I used lz4 compression in ZFS. For the other ZFS settings, I used what can be found in my earlier ZFS posts but with the ARC size limited to 1GB. I then used that plain configuration for the first benchmarks. Here are the results with the sysbench point-select benchmark, a uniform distribution and eight threads. The InnoDB buffer pool was set to 2.5GB. In both cases, the load is IO bound. The disk is doing exactly the allowed 3000 IOPS. The above graph appears to be a clear demonstration that XFS is much faster than ZFS, right? But is that really the case? The way the dataset has been created is extremely favorable to XFS since there is absolutely no file fragmentation. Once you have all the files opened, a read IOP is just a single fseek call to an offset and ZFS doesn’t need to access any intermediate inode. The above result is about as fair as saying MyISAM is faster than InnoDB based only on table scan performance results of unfragmented tables and default configuration. ZFS is much less affected by the file level fragmentation, especially for point access type. ZFS stores the files in B-trees in a very similar fashion as InnoDB stores data. To access a piece of data in a B-tree, you need to access the top level page (often called root node) and then one block per level down to a leaf-node containing the data. With no cache, to read something from a three levels B-tree thus requires 3 IOPS. The extra IOPS performed by ZFS are needed to access those internal blocks in the B-trees of the files. These internal blocks are labeled as metadata. Essentially, in the above benchmark, the ARC is too small to contain all the internal blocks of the table files’ B-trees. If we continue the comparison with InnoDB, it would be like running with a buffer pool too small to contain the non-leaf pages. The test dataset I used has about 600MB of non-leaf pages, about 0.1% of the total size, which was well cached by the 3GB buffer pool. So only one InnoDB page, a leaf page, needed to be read per point-select statement. To correctly set the ARC size to cache the metadata, you have two choices. First, you can guess values for the ARC size and experiment. Second, you can try to evaluate it by looking at the ZFS internal data. Let’s review these two approaches. You’ll read/hear often the ratio 1GB of ARC for 1TB of data, which is about the same 0.1% ratio as for InnoDB. I wrote about that ratio a few times, having nothing better to propose. Actually, I found it depends a lot on the recordsize used. The 0.1% ratio implies a ZFS recordsize of 128KB. A ZFS filesystem with a recordsize of 128KB will use much less metadata than another one using a recordsize of 16KB because it has 8x fewer leaf pages. Fewer leaf pages require less B-tree internal nodes, hence less metadata. A filesystem with a recordsize of 128KB is excellent for sequential access as it maximizes compression and reduces the IOPS but it is poor for small random access operations like the ones MySQL/InnoDB does. In order to improve ZFS performance, I had 3 options: Increase the ARC size to 7GB Use a larger Innodb page size like 64KB Add a L2ARC I was reluctant to grow the ARC to 7GB, which was nearly half the overall system memory. At best, the ZFS performance would only match XFS. A larger InnoDB page size would increase the CPU load for decompression on an instance with only two vCPUs; not great either. The last option, the L2ARC, was the most promising. ZFS is much more complex than XFS and EXT4 but, that also means it has more tunables/options. I used a simplistic setup and an unfair benchmark which initially led to poor ZFS results. With the same benchmark, very favorable to XFS, I added a ZFS L2ARC and that completely reversed the situation, more than tripling the ZFS results, now 66% above XFS. Conclusion We have seen in this post why the general perception is that ZFS under-performs compared to XFS or EXT4. The presence of B-trees for the files has a big impact on the amount of metadata ZFS needs to handle, especially when the recordsize is small. The metadata consists mostly of the non-leaf pages (or internal nodes) of the B-trees. When properly cached, the performance of ZFS is excellent. ZFS allows you to optimize the use of EBS volumes, both in term of IOPS and size when the instance has fast ephemeral storage devices. Using the ephemeral device of an i3.large instance for the ZFS L2ARC, ZFS outperformed XFS by 66%. ###OpenSMTPD new config TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentioned at EuroBSDCon the one-line per rule config format was a design error. A new configuration grammar is almost ready and the underlying structures are simplified. Refactor removes ~750 lines of code and solves _many issues that were side-effects of the design error. New features are going to be unlocked thanks to this. Anatomy of a design error OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time. The initial configuration format was very different, I was inspired by pyr@’s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets. When I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed. It helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later. That being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong. One-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals. To get to the point: we should move to two-line rules :-) Anatomy of a design error OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time. The initial configuration format was very different, I was inspired by pyr@’s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets. When I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed. It helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later. That being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong. One-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals. To get to the point: we should move to two-line rules :-) The problem with one-line rules OpenSMTPD decides to accept or reject messages based on one-line rules such as: accept from any for domain poolp.org deliver to mbox Which can essentially be split into three units: the decision: accept/reject the matching: from any for domain poolp.org the (default) action: deliver to mbox To ensure that we meet the requirements of the transactions, the matching must be performed during the SMTP transaction before we take a decision for the recipient. Given that the rule is atomic, that it doesn’t have an identifier and that the action is part of it, the two only ways to make sure we can remember the action to take later on at delivery time is to either: save the action in the envelope, which is what we do today evaluate the envelope again at delivery And this this where it gets tricky… both solutions are NOT ok. The first solution, which we’ve been using for a decade, was to save the action within the envelope and kind of carve it in stone. This works fine… however it comes with the downsides that errors fixed in configuration files can’t be caught up by envelopes, that delivery action must be validated way ahead of time during the SMTP transaction which is much trickier, that the parsing of delivery methods takes place as the _smtpd user rather than the recipient user, and that envelope structures that are passed all over OpenSMTPD carry delivery-time informations, and more, and more, and more. The code becomes more complex in general, less safe in some particular places, and some areas are nightmarish to deal with because they have to deal with completely unrelated code that can’t be dealt with later in the code path. The second solution can’t be done. An envelope may be the result of nested rules, for example an external client, hitting an alias, hitting a user with a .forward file resolving to a user. An envelope on disk may no longer match any rule or it may match a completely different rule If we could ensure that it matched the same rule, evaluating the ruleset may spawn new envelopes which would violate the transaction. Trying to imagine how we could work around this leads to more and more and more RFC violations, incoherent states, duplicate mails, etc… There is simply no way to deal with this with atomic rules, the matching and the action must be two separate units that are evaluated at two different times, failure to do so will necessarily imply that you’re either using our first solution and all its downsides, or that you are currently in a world of pain trying to figure out why everything is burning around you. The minute the action is written to an on-disk envelope, you have failed. A proper ruleset must define a set of matching patterns resolving to an action identifier that is carved in stone, AND a set of named action set that is resolved dynamically at delivery time. Follow the link above to see the rest of the article Break ##News Roundup Backing up a legacy Windows machine to a FreeNAS with rsync I have some old Windows servers (10 years and counting) and I have been using rsync to back them up to my FreeNAS box. It has been working great for me. First of all, I do have my Windows servers backup in virtualized format. However, those are only one-time snapshops that I run once in a while. These are classic ASP IIS web servers that I can easily put up on a new VM. However, many of these legacy servers generate gigabytes of data a day in their repositories. Running VM conversion daily is not ideal. My solution was to use some sort of rsync solution just for the data repos. I’ve tried some applications that didn’t work too well with Samba shares and these old servers have slow I/O. Copying files to external sata or usb drive was not ideal. We’ve moved on from Windows to Linux and do not have any Windows file servers of capacity to provide network backups. Hence, I decided to use Delta Copy with FreeNAS. So here is a little write up on how to set it up. I have 4 Windows 2000 servers backing up daily with this method. First, download Delta Copy and install it. It is open-source and pretty much free. It is basically a wrapper for cygwin’s rsync. When you install it, it will ask you to install the Server services which allows you to run it as a Rsync server on Windows. You don’t need to do this. Instead, you will be just using the Delta Copy Client application. But before we do that, we will need to configure our Rsync service for our Windows Clients on FreeNAS. In FreeNAS, go under Services , Select Rsync > Rsync Modules > Add Rsync Module. Then fill out the form; giving the module a name and set the path. In my example, I simply called it WIN and linked it to a user called backupuser. This process is much easier than trying to configure the daemon rsyncd.conf file by hand. Now, on the Windows Client, start the DeltaCopy Client. You will create a new Profile. You will need to enter the IP of the Rsync server (FreeNAS) and specify the module name which will be called “Virtual Directory Name.” When you pull the select menu, the list of Rsync Modules you created earlier in FreeNAS will populate. You can set authentication. On the server, you can restrict by IP and do other things to lock down your rsync. Next, you will add folders (and/or files) you want to synchronize. Once the paths are set up, you can run a sync by right clicking the profile name. Here, I made a test sync to a home folder of a virtualized windows box. As you can see, I mounted the rsync volume on my mac to see the progress. The rsync worked beautifully. DeltaCopy did what it was told. Once you get everything working. The next thing to do is set schedules. If you done tasks schedules in Windows before, it is pretty straightforward. DeltaCopy has a link in the application to directly create a new task for you. I set my backups to run nightly and it has been working great. There you have it. Windows rsync to FreeNAS using DeltaCopy. The nice thing about FreeNAS is you don’t have to modify /etc/rsyncd.conf files. Everything can be done in the web admin. iXsystems ###How to write ATF tests for NetBSD I have recently started contributing to the amazing NetBSD foundation. I was thinking of trying out a new OS for a long time. Switching to the NetBSD OS has been a fun change. My first contribution to the NetBSD foundation was adding regression tests for the Address Sanitizer (ASan) in the Automated Testing Framework(ATF) which NetBSD has. I managed to complete it with the help of my really amazing mentor Kamil. This post is gonna be about the ATF framework that NetBSD has and how to you can add multiple tests with ease. Intro In ATF tests we will basically be talking about test programs which are a suite of test cases for a specific application or program. The ATF suite of Commands There are a variety of commands that the atf suite offers. These include : atf-check: The versatile command that is a vital part of the checking process. man page atf-run: Command used to run a test program. man page atf-fail: Report failure of a test case. atf-report: used to pretty print the atf-run. man page atf-set: To set atf test conditions. We will be taking a better look at the syntax and usage later. Let’s start with the Basics The ATF testing framework comes preinstalled with a default NetBSD installation. It is used to write tests for various applications and commands in NetBSD. One can write the Test programs in either the C language or in shell script. In this post I will be dealing with the Bash part. Follow the link above to see the rest of the article ###The Importance of ZFS Block Size Warning! WARNING! Don’t just do things because some random blog says so One of the important tunables in ZFS is the recordsize (for normal datasets) and volblocksize (for zvols). These default to 128KB and 8KB respectively. As I understand it, this is the unit of work in ZFS. If you modify one byte in a large file with the default 128KB record size, it causes the whole 128KB to be read in, one byte to be changed, and a new 128KB block to be written out. As a result, the official recommendation is to use a block size which aligns with the underlying workload: so for example if you are using a database which reads and writes 16KB chunks then you should use a 16KB block size, and if you are running VMs containing an ext4 filesystem, which uses a 4KB block size, you should set a 4KB block size You can see it has a 16GB total file size, of which 8.5G has been touched and consumes space - that is, it’s a “sparse” file. The used space is also visible by looking at the zfs filesystem which this file resides in Then I tried to copy the image file whilst maintaining its “sparseness”, that is, only touching the blocks of the zvol which needed to be touched. The original used only 8.42G, but the copy uses 14.6GB - almost the entire 16GB has been touched! What’s gone wrong? I finally realised that the difference between the zfs filesystem and the zvol is the block size. I recreated the zvol with a 128K block size That’s better. The disk usage of the zvol is now exactly the same as for the sparse file in the filesystem dataset It does impact the read speed too. 4K blocks took 5:52, and 128K blocks took 3:20 Part of this is the amount of metadata that has to be read, see the MySQL benchmarks from earlier in the show And yes, using a larger block size will increase the compression efficiency, since the compressor has more redundant data to optimize. Some of the savings, and the speedup is because a lot less metadata had to be written Your zpool layout also plays a big role, if you use 4Kn disks, and RAID-Z2, using a volblocksize of 8k will actually result in a large amount of wasted space because of RAID-Z padding. Although, if you enable compression, your 8k records may compress to only 4k, and then all the numbers change again. ###Using a Raspberry Pi 2 as a Router on a Stick Starring NetBSD Sorry we didn’t answer you quickly enough A few weeks ago I set about upgrading my feeble networking skills by playing around with a Cisco 2970 switch. I set up a couple of VLANs and found the urge to set up a router to route between them. The 2970 isn’t a modern layer 3 switch so what am I to do? Why not make use of the Raspberry Pi 2 that I’ve never used and put it to some good use as a ‘router on a stick’. I could install a Linux based OS as I am quite familiar with it but where’s the fun in that? In my home lab I use SmartOS which by the way is a shit hot hypervisor but as far as I know there aren’t any Illumos distributions for the Raspberry Pi. On the desktop I use Solus OS which is by far the slickest Linux based OS that I’ve had the pleasure to use but Solus’ focus is purely desktop. It’s looking like BSD then! I believe FreeBSD is renowned for it’s top notch networking stack and so I wrote to the BSDNow show on Jupiter Broadcasting for some help but it seems that the FreeBSD chaps from the show are off on a jolly to some BSD conference or another(love the show by the way). It looks like me and the luvverly NetBSD are on a date this Saturday. I’ve always had a secret love for NetBSD. She’s a beautiful, charming and promiscuous lover(looking at the supported architectures) and I just can’t stop going back to her despite her misgivings(ahem, zfs). Just my type of grrrl! Let’s crack on… Follow the link above to see the rest of the article ##Beastie Bits BSD Jobs University of Aberdeen’s Internet Transport Research Group is hiring VR demo on OpenBSD via OpenHMD with OSVR HDK2 patch runs ed, and ed can run anything (mentions FreeBSD and OpenBSD) Alacritty (OpenGL-powered terminal emulator) now supports OpenBSD MAP_STACK Stack Register Checking Committed to -current EuroBSDCon CfP till June 17, 2018 Tarsnap ##Feedback/Questions NeutronDaemon - Tutorial request Kurt - Question about transferability/bi-directionality of ZFS snapshots and send/receive Peter - A Question and much love for BSD Now Peter - netgraph state Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

We explain the physics behind ZFS, DTrace switching to the GPL, Emacs debugging, syncookies coming to PF & FreeBSD's history on EC2. This episode was brought to you by Headlines 128 bit storage: Are you high? (https://blogs.oracle.com/bonwick/128-bit-storage:-are-you-high) For people who have heard about ZFS boiling oceans and wonder where that is coming from, we dug out this old piece from 2004 on the blog of ZFS co-creator Jeff Bonwick, originally from the Sun website. 64 bits would have been plenty ... but then you can't talk out of your ass about boiling oceans then, can you? Well, it's a fair question. Why did we make ZFS a 128-bit storage system? What on earth made us think it's necessary? And how do we know it's sufficient? Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 2^64 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. If 64 bits isn't enough, the next logical step is 128 bits. That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. But it does raise the question: what are the theoretical limits to storage capacity? Although we'd all like Moore's Law to continue forever, quantum mechanics imposes some fundamental limits on the computation rate and information capacity of any physical device. In particular, it has been shown that 1 kilogram of matter confined to 1 liter of space can perform at most 10^51 operations per second on at most 10^31 bits of information [see Seth Lloyd, "Ultimate physical limits to computation." Nature 406, 1047-1054 (2000)]. A fully-populated 128-bit storage pool would contain 2^128 blocks = 2^137 bytes = 2^140 bits; therefore the minimum mass required to hold the bits would be (2^140 bits) / (10^31 bits/kg) = 136 billion kg. That's a lot of gear. To operate at the 1031 bits/kg limit, however, the entire mass of the computer must be in the form of pure energy. By E=mc^2, the rest energy of 136 billion kg is 1.2x1028 J. The mass of the oceans is about 1.4x1021 kg. It takes about 4,000 J to raise the temperature of 1 kg of water by 1 degree Celcius, and thus about 400,000 J to heat 1 kg of water from freezing to boiling. The latent heat of vaporization adds another 2 million J/kg. Thus the energy required to boil the oceans is about 2.4x106 J/kg * 1.4x1021 kg = 3.4x1027 J. Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. Best part of all: you don't have to understand any of this to use ZFS. Rest assured that you won't hit any limits with that filesystem for a long time. You still have to buy bigger disks over time, though... *** dtrace for Linux, Oracle relicenses dtrace (https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-linux-oracle-does-the-right-thing/) At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom. Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port: ``` commit e1744f50ee9bc1978d41db7cc93bcf30687853e6 Author: Tomas Jedlicka tomas.jedlicka@oracle.com Date: Tue Aug 1 09:15:44 2017 -0400 dtrace: Integrate DTrace Modules into kernel proper This changeset integrates DTrace module sources into the main kernel source tree under the GPLv2 license. Sources have been moved to appropriate locations in the kernel tree. ``` That is right, dtrace dropped the CDDL and switched to the GPL! The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out). The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory. The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL. Thank you Oracle for making everyone's life easier by waving your magic relicensing wand! Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest. Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don't always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all. Debugging Emacs or: How I Learned to Stop Worrying and Love DTrace (http://nullprogram.com/blog/2018/01/17/) For some time Elfeed was experiencing a strange, spurious failure. Every so often users were seeing an error (spoiler warning) when updating feeds: “error in process sentinel: Search failed.” If you use Elfeed, you might have even seen this yourself. From the surface it appeared that curl, tasked with the responsibility for downloading feed data, was producing incomplete output despite reporting a successful run. Since the run was successful, Elfeed assumed certain data was in curl's output buffer, but, since it wasn't, it failed hard. Unfortunately this issue was not reproducible. Manually running curl outside of Emacs never revealed any issues. Asking Elfeed to retry fetching the feeds would work fine. The issue would only randomly rear its head when Elfeed was fetching many feeds in parallel, under stress. By the time the error was discovered, the curl process had exited and vital debugging information was lost. Considering that this was likely to be a bug in Emacs itself, there really wasn't a reliable way to capture the necessary debugging information from within Emacs Lisp. And, indeed, this later proved to be the case. A quick-and-dirty work around is to use condition-case to catch and swallow the error. When the bizarre issue shows up, rather than fail badly in front of the user, Elfeed could attempt to swallow the error — assuming it can be reliably detected — and treat the fetch as simply a failure. That didn't sit comfortably with me. Elfeed had done its due diligence checking for errors already. Someone was lying to Elfeed, and I intended to catch them with their pants on fire. Someday. I'd just need to witness the bug on one of my own machines. Elfeed is part of my daily routine, so surely I'd have to experience this issue myself someday. My plan was, should that day come, to run a modified Elfeed, instrumented to capture extra data. I would have also routinely run Emacs under GDB so that I could inspect the failure more deeply. For now I just had to wait to hunt that zebra. Bryan Cantrill, DTrace, and FreeBSD Over the holidays I re-discovered Bryan Cantrill, a systems software engineer who worked for Sun between 1996 and 2010, and is most well known for DTrace. My first exposure to him was in a BSD Now interview in 2015. I had re-watched that interview and decided there was a lot more I had to learn from him. He's become a personal hero to me. So I scoured the internet for more of his writing and talks. Some interesting operating system technology came out of Sun during its final 15 or so years — most notably DTrace and ZFS — and Bryan speaks about it passionately. Almost as a matter of luck, most of it survived the Oracle acquisition thanks to Sun releasing it as open source in just the nick of time. Otherwise it would have been lost forever. The scattered ex-Sun employees, still passionate about their prior work at Sun, along with some of their old customers have since picked up the pieces and kept going as a community under the name illumos. It's like an open source flotilla. Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn't have these Sun technologies available. The main reason is license incompatibility. Sun released its code under the CDDL, which is incompatible with the GPL. Ubuntu does infamously include ZFS, but other distributions are unwilling to take that risk. Porting DTrace is a serious undertaking since it's got its fingers throughout the kernel, which also makes the licensing issues even more complicated. Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that. Rather than adopt ZFS and DTrace, they've been reinvented from scratch: btrfs instead of ZFS, and a slew of partial options instead of DTrace. Normally I'm most interested in system call tracing, and my go to is strace, though it certainly has its limitations — including this situation of debugging curl under Emacs. Another famous example of NIH is Linux's epoll(2), which is a broken version of BSD kqueue(2). So, if I want to try these for myself, I'll need to install a different operating system. I've dabbled with OmniOS, an OS built on illumos, in virtual machines, using it as an alien environment to test some of my software (e.g. enchive). OmniOS has a philosophy called Keep Your Software To Yourself (KYSTY), which is really just code for “we don't do packaging.” Honestly, you can't blame them since they're a tiny community. The best solution to this is probably pkgsrc, which is essentially a universal packaging system. Otherwise you're on your own. There's also openindiana, which is a more friendly desktop-oriented illumos distribution. Still, the short of it is that you're very much on your own when things don't work. The situation is like running Linux a couple decades ago, when it was still difficult to do. If you're interested in trying DTrace, the easiest option these days is probably FreeBSD. It's got a big, active community, thorough documentation, and a huge selection of packages. Its license (the BSD license, duh) is compatible with the CDDL, so both ZFS and DTrace have been ported to FreeBSD. What is DTrace? I've done all this talking but haven't yet described what DTrace really is. I won't pretend to write my own tutorial, but I'll provide enough information to follow along. DTrace is a tracing framework for debugging production systems in real time, both for the kernel and for applications. The “production systems” part means it's stable and safe — using DTrace won't put your system at risk of crashing or damaging data. The “real time” part means it has little impact on performance. You can use DTrace on live, active systems with little impact. Both of these core design principles are vital for troubleshooting those really tricky bugs that only show up in production. There are DTrace probes scattered all throughout the system: on system calls, scheduler events, networking events, process events, signals, virtual memory events, etc. Using a specialized language called D (unrelated to the general purpose programming language D), you can dynamically add behavior at these instrumentation points. Generally the behavior is to capture information, but it can also manipulate the event being traced. Each probe is fully identified by a 4-tuple delimited by colons: provider, module, function, and probe name. An empty element denotes a sort of wildcard. For example, syscall::open:entry is a probe at the beginning (i.e. “entry”) of open(2). syscall:::entry matches all system call entry probes. Unlike strace on Linux which monitors a specific process, DTrace applies to the entire system when active. To run curl under strace from Emacs, I'd have to modify Emacs' behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That's a big deal. So, when it comes to this Elfeed issue, FreeBSD is much better poised for debugging the problem. All I have to do is catch it in the act. However, it's been months since that bug report and I'm not really making this connection yet. I'm just hoping I eventually find an interesting problem where I can apply DTrace. Bryan Cantrill: Talks I have given (http://dtrace.org/blogs/bmc/2018/02/03/talks/) *** News Roundup a2k18 Hackathon preview: Syncookies coming to PF (https://undeadly.org/cgi?action=article;sid=20180207090000) As you may have heard, the a2k18 hackathon is in progress. As can be seen from the commit messages, several items of goodness are being worked on. One eagerly anticipated item is the arrival of TCP syncookies (read: another important tool in your anti-DDoS toolset) in PF. Henning Brauer (henning@) added the code in a series of commits on February 6th, 2018, with this one containing the explanation: ``` syncookies for pf. when syncookies are on, pf will blindly answer each and every SYN with a syncookie-SYNACK. Upon reception of the ACK completing the 3WHS, pf will reconstruct the original SYN, shove it through pf_test, where state will be created if the ruleset permits it. Then massage the freshly created state (we won't see the SYNACK), set up the sequence number modulator, and call into the existing synproxy code to start the 3WHS with the backend host. Add an - somewhat basic for now - adaptive mode where syncookies get enabled if a certain percentage of the state table is filled up with half-open tcp connections. This makes pf firewalls resilient against large synflood attacks. syncookies are off by default until we gained more experience, considered experimental for now. see http://bulabula.org/papers/2017/bsdcan/ for more details. joint work with sashan@, widely discussed and with lots of input by many ``` The first release to have this feature available will probably be the upcoming OpenBSD 6.3 if a sufficient number of people test this in their setups (hint, hint). More info is likely to emerge soon in post-hackathon writeups, so watch this space! [Pale Moon] A Perfect example of how not to approach OS developers/packagers Removed from OpenBSD Ports due to Licensing Issues (https://github.com/jasperla/openbsd-wip/issues/86) FreeBSD Palemoon branding violation (https://lists.freebsd.org/pipermail/freebsd-ports/2018-February/112455.html) Mightnight BSD's response (https://twitter.com/midnightbsd/status/961232422091280386) *** FreeBSD EC2 History (http://www.daemonology.net/blog/2018-02-12-FreeBSD-EC2-history.html) A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years. Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments. August 25, 2006: Amazon EC2 launches. It supports a single version of Ubuntu Linux; FreeBSD is not available. December 13, 2010: I manage to get FreeBSD running on EC2 t1.micro instances. March 22, 2011: I manage to get FreeBSD running on EC2 "cluster compute" instances. July 8, 2011: I get FreeBSD 8.2 running on all 64-bit EC2 instance types, by marking it as "Windows" in order to get access to Xen/HVM virtualization. (Unfortunately this meant that users had to pay the higher "Windows" hourly pricing.) January 16, 2012: I get FreeBSD 9.0 running on 32-bit EC2 instances via the same "defenestration" trick. (Again, paying the "Windows" prices.) August 16, 2012: I move the FreeBSD rc.d scripts which handle "EC2" functionality (e.g., logging SSH host keys to the console) into the FreeBSD ports tree. October 7, 2012: I rework the build process for FreeBSD 9.1-RC1 and later to use "world" bits extracted from the release ISOs; only the kernel is custom-built. Also, the default SSH user changes from "root" to "ec2-user". October 31, 2012: Amazon launches the "M3" family of instances, which support Xen/HVM without FreeBSD needing to pay the "Windows" tax. November 21, 2012: I get FreeBSD added to the AWS Marketplace. October 2, 2013: I finish merging kernel patches into the FreeBSD base system, and rework the AMI build (again) so that FreeBSD 10.0-ALPHA4 and later use bits extracted from the release ISOs for the entire system (world + kernel). FreeBSD Update can now be used for updating everything (because now FreeBSD/EC2 uses a GENERIC kernel). October 27, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA2 and later AMIs will run FreeBSD Update when they first boot in order to download and install any critical updates. December 1, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA4 and later AMIs bootstrap the pkg tool and install packages at boot time (by default, the "awscli" package). December 9, 2013: I add configinit to FreeBSD 10.0-RC1 and later to allow systems to be easily configured via EC2 user-data. July 1, 2014: Amazon launches the "T2" family of instances; now the most modern family for every type of EC2 instance (regular, high-memory, high-CPU, high-I/O, burstable) supports HVM and there should no longer be any need for FreeBSD users to pay the "Windows tax". November 24, 2014: I add code to FreeBSD 10.2 and later to automatically resize their root filesystems when they first boot; this means that a larger root disk can be specified at instance launch time and everything will work as expected. April 1, 2015: I integrate the FreeBSD/EC2 build process into the FreeBSD release building process; FreeBSD 10.2-BETA1 and later AMIs are built by the FreeBSD release engineering team. January 12, 2016: I enable Intel 82599-based "first generation EC2 Enhanced Networking" in FreeBSD 11.0 and later. June 9, 2016: I enable the new EC2 VGA console functionality in FreeBSD 11.0 and later. (The old serial console also continues to work.) June 24, 2016: Intel 82599-based Enhanced Networking works reliably in FreeBSD 11.0 and later thanks to discovering and working around a Xen bug. June 29, 2016: I improve throughput on Xen blkfront devices (/dev/xbd*) by enabling indirect segment I/Os in FreeBSD 10.4 and later. (I wrote this functionality in July 2015, but left it disabled by default a first because a bug in EC2 caused it to hurt performance on some instances.) July 7, 2016: I fix a bug in FreeBSD's virtual memory initialization in order to allow it to support boot with 128 CPUs; aka. FreeBSD 11.0 and later support the EC2 x1.32xlarge instance type. January 26, 2017: I change the default configuration in FreeBSD 11.1 and later to support EC2's IPv6 networking setup out of the box (once you flip all of the necessary switches to enable IPv6 in EC2 itself). May 20, 2017: In collaboration with Rick Macklem, I make FreeBSD 11.1 and later compatible with the Amazon "Elastic File System" (aka. NFSv4-as-a-service) via the newly added "oneopenown" mount option (and lots of bug fixes). May 25, 2017: I enable support for the Amazon "Elastic Network Adapter" in FreeBSD 11.1 and later. (The vast majority of the work — porting the driver code — was done by Semihalf with sponsorship from Amazon.) December 5, 2017: I change the default configuration in FreeBSD 11.2 and later to make use of the Amazon Time Sync Service (aka. NTP-as-a-service). The current status The upcoming FreeBSD release (11.2) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial VGA), and every EC2 instance type (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances). Colin's Patreon' page if you'd like to support him (https://www.patreon.com/cperciva) X network transparency X's network transparency has wound up mostly being a failure (https://utcc.utoronto.ca/~cks/space/blog/unix/XNetworkTransparencyFailure) I was recently reading Mark Dominus's entry about some X keyboard problems, in which he said in passing (quoting himself): I have been wondering for years if X's vaunted network transparency was as big a failure as it seemed: an interesting idea, worth trying out, but one that eventually turned out to be more trouble than it was worth. [...] My first reaction was to bristle, because I use X's network transparency all of the time at work. I have several programs to make it work very smoothly, and some core portions of my environment would be basically impossible without it. But there's a big qualification on my use of X's network transparency, namely that it's essentially all for text. When I occasionally go outside of this all-text environment of xterms and emacs and so on, it doesn't go as well. X's network transparency was not designed as 'it will run xterm well'; originally it was to be something that should let you run almost everything remotely, providing a full environment. Even apart from the practical issues covered in Daniel Stone's slide presentation, it's clear that it's been years since X could deliver a real first class environment over the network. You cannot operate with X over the network in the same way that you do locally. Trying to do so is painful and involves many things that either don't work at all or perform so badly that you don't want to use them. In my view, there are two things that did in general X network transparency. The first is that networks turned out to not be fast enough even for ordinary things that people wanted to do, at least not the way that X used them. The obvious case is web browsers; once the web moved to lots of images and worse, video, that was pretty much it, especially with 24-bit colour. (It's obviously not impossible to deliver video across the network with good performance, since YouTube and everyone else does it. But their video is highly encoded in specialized formats, not handled by any sort of general 'send successive images to the display' system.) The second is that the communication facilities that X provided were too narrow and limited. This forced people to go outside of them in order to do all sorts of things, starting with audio and moving on to things like DBus and other ways of coordinating environments, handling sophisticated configuration systems, modern fonts, and so on. When people designed these additional communication protocols, the result generally wasn't something that could be used over the network (especially not without a bunch of setup work that you had to do in addition to remote X). Basic X clients that use X properties for everything may be genuinely network transparent, but there are very few of those left these days. (Not even xterm is any more, at least if you use XFT fonts. XFT fonts are rendered in the client, and so different hosts may have different renderings of the same thing, cf.) < What remains of X's network transparency is still useful to some of us, but it's only a shadow of what the original design aimed for. I don't think it was a mistake for X to specifically design it in (to the extent that they did, which is less than you might think), and it did help X out pragmatically in the days of X terminals, but that's mostly it. (I continue to think that remote display protocols are useful in general, but I'm in an usual situation. Most people only ever interact with remote machines with either text mode SSH or a browser talking to a web server on the remote machine.) PS: The X protocol issues with synchronous requests that Daniel Stone talks about don't help the situation, but I think that even with those edges sanded off X's network transparency wouldn't be a success. Arguably X's protocol model committed a lesser version of part of the NeWS mistake. X's network transparency was basically free at the time (https://utcc.utoronto.ca/~cks/space/blog/unix/XFreeNetworkTransparency) I recently wrote an entry about how X's network transparency has wound up mostly being a failure for various reasons. However, there is an important flipside to the story of X's network transparency, and that is that X's network transparency was almost free at the time and in the context it was created. Unlike the situation today, in the beginning X did not have to give up lots of performance or other things in order to get network transparency. X originated in the mid 1980s and it was explicitly created to be portable across various Unixes, especially BSD-derived ones (because those were what universities were mostly using at that time). In the mid to late 1980s, Unix had very few IPC methods, especially portable ones. In particular, BSD systems did not have shared memory (it was called 'System V IPC' for the obvious reasons). BSD had TCP and Unix sockets, some System V machines had TCP (and you could likely assume that more would get it), and in general your safest bet was to assume some sort of abstract stream protocol and then allow for switchable concrete backends. Unsurprisingly, this is exactly what X did; the core protocol is defined as a bidirectional stream of bytes over an abstracted channel. (And the concrete implementation of $DISPLAY has always let you specify the transport mechanism, as well as allowing your local system to pick the best mechanism it has.) Once you've decided that your protocol has to run over abstracted streams, it's not that much more work to make it network transparent (TCP provides streams, after all). X could have refused to make the byte order of the stream clear or required the server and the client to have access to some shared files (eg for fonts), but I don't think either would have been a particularly big win. I'm sure that it took some extra effort and care to make X work across TCP from a different machine, but I don't think it took very much. (At the same time, my explanation here is probably a bit ahistorical. X's initial development seems relatively strongly tied to sometimes having clients on different machines than the display, which is not unreasonable for the era. But it doesn't hurt to get a feature that you want anyway for a low cost.) I believe it's important here that X was intended to be portable across different Unixes. If you don't care about portability and can get changes made to your Unix, you can do better (for example, you can add some sort of shared memory or process to process virtual memory transfer). I'm not sure how the 1980s versions of SunView worked, but I believe they were very SunOS dependent. Wikipedia says SunView was partly implemented in the kernel, which is certainly one way to both share memory and speed things up. PS: Sharing memory through mmap() and friends was years in the future at this point and required significant changes when it arrived. Beastie Bits Grace Hopper Celebration 2018 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/grace-hopper-celebration-2018-call-for-participation/) Google Summer of Code: Call for Project Ideas (https://www.freebsdfoundation.org/blog/google-summer-of-code-call-for-project-ideas/) The OpenBSD Foundation 2018 Fundraising Campaign (https://undeadly.org/cgi?action=article;sid=20180129190641) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) AsiaBSDcon 2018 Registration is open (https://2018.asiabsdcon.org/) Tarsnap support for Bitcoin ending April 1st; and a Chrome bug (http://mail.tarsnap.com/tarsnap-announce/msg00042.html) Feedback/Questions Todd - Couple Questions (http://dpaste.com/195HGHY#wrap) Seth - Tar Snap (http://dpaste.com/1N7NQVQ#wrap) Alex - sudo question (http://dpaste.com/3D9P1DW#wrap) Thomas - FreeBSD on ARM? (http://dpaste.com/24NMG47#wrap) Albert - Austria BSD User Group (http://dpaste.com/373CRX7#wrap)

We talk about our recent trip to FOSDEM, we discuss the pros and cons of permissive licensing, cover the installation of OpenBSD on a dedibox with full-disk encryption, the new Lumina guide repository, and we explain ZFS vs. OpenZFS. This episode was brought to you by Headlines [FOSDEM Trip report] Your BSDNow hosts were both at FOSDEM in Brussels, Belgium over the weekend. On the friday before FOSDEM, we held a FreeBSD devsummit (3rd consecutive year), sponsored by the FreeBSD Foundation and organized by Benedict (with the help from Kristof Provost, who did it in previous years but could not make it this year). We had 21 people attend, a good mixture of FreeBSD committers (mostly ports) and guests. After introductions, we collected topics and discussed various topics, including a new plan for a future FreeBSD release roadmap (more frequent releases, so that features from HEAD can be tried out earlier in RELEASES). The devsummit concluded with a nice dinner in a nearby restaurant. On Saturday, first day of FOSDEM, we set up the FreeBSD Foundation table with flyers, stickers, FreeBSD Journal print editions, and a small RPI 3 demo system that Deb Goodkin brought. Our table was located next to the Illumos table like last year. This allowed us to continue the good relationship that we have with the Illumos people and Allan helped a little bit getting bhyve to run on Illumos with UEFI. Meanwhile, our table was visited by a lot of people who would ask questions about FreeBSD, take info material, or talk about their use cases. We were busy refilling the table throughout the day and luckily, we had many helpers at the table. Some items we had ran out in the early afternoon, an indicator of how popular they were. Saturday also featured a BSD devroom (https://twitter.com/fosdembsd), organized by Rodrigo Osorio. You can find the list of talks and the recordings on the BSD Devroom schedule (https://fosdem.org/2018/schedule/track/bsd/). The room was very crowded and popular. Deb Goodkin gave the opening talk with an overview of what the Foundation is doing to change the world. Other speakers from various BSD projects presented their talks after that with a range of topics. Among them, Allan gave his talk about ZFS: Advanced Integration (https://fosdem.org/2018/schedule/event/zfs_advanced_integration/), while Benedict presented his Reflections on Teaching a Unix Class With FreeBSD (https://fosdem.org/2018/schedule/event/reflections_on_reaching_unix_class_with_freebsd/). Sunday was just as busy on the FreeBSD table as Saturday and we finally ran out of stickers and some other goodies. We were happy with the results of the two days. Some very interesting conversations at the table about FreeBSD took place, some of which we're going to follow up afterwards. Check out the FOSDEM schedule as many talk recordings are already available, and especially the ones from the BSD devroom if you could not attend the conference. We would like to thank everyone who attended the FreeBSD devsummit, who helped out at the FreeBSD table and organized the BSD devroom. Also, thanks to all the speakers, organizers, and helping hands making FOSDEM another success this year. *** NetBSD kernel wscons IOCTL vulnerable bug class (http://blog.infosectcbr.com.au/2018/01/netbsd-kernel-wscons-ioctl-vulnerable.html) I discovered this bug class during the InfoSect public code review session we ran looking specifically at the NetBSD kernel. I found a couple of these bugs and then after the session was complete, I went back and realised the same bug was scattered in other drivers. In total, 17 instances of this vulnerability and its variants were discovered. In all fairness, I came across this bug class during my kernel audits in 2002 and most instances were patched. It just seems there are more bugs now in NetBSD while OpenBSD and FreeBSD have practically eliminated them. See slide 41 in http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf (http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf) for exactly the same bug (class) 16 years ago. The format of the this blog post is as follows: Introduction Example of the Bug Class How to Fix How to Detect Automatically with Coccinelle More Bugs Conclusion These source files had bugs ./dev/tc/tfb.c ./dev/ic/bt485.c ./dev/pci/radeonfb.c ./dev/ic/sti.c ./dev/sbus/tcx.c ./dev/tc/mfb.c ./dev/tc/sfb.c ./dev/tc/stic.c ./dev/tc/cfb.c ./dev/tc/xcfb.c ./dev/tc/sfbplus.c ./arch/arm/allwinner/awin_debe.c ./arch/arm/iomd/vidcvideo.c ./arch/pmax/ibus/pm.c ./dev/ic/igfsb.c ./dev/ic/bt463.c ./arch/luna68k/dev/lunafb.c Reporting of the bugs was easy. In less than a week from reporting the specific instances of each bug, patches were committed into the mainline kernel. Thanks to Luke Mewburn from NetBSD for coming to the code review session at InfoSect and coordinating with the NetBSD security team. The patches to fix these issues are in NetBSD: https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html (https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html) "Permissive licensing is wrong!” – Is it? (https://eerielinux.wordpress.com/2017/11/25/permissive-licensing-is-wrong-is-it-1-2/) A few weeks ago I've been attacked by some GNU zealots on a German tech site after speaking in favor of permissive licenses. Unfortunately a discussion was not possible there because that would require the will to actually communicate instead of simply accusing the other side of vile motives. Since I actually do care about this topic and a reader asked for a post about it in comments a while ago, here we go. This first part tries to sum up the most important things around the topic. I deliberately aim for an objective overview that tries not to be one-sided. The second part will then contain my points in defence of permissive licensing. Why license software at all? Licenses exist for reasons of protection. If you're the author/inventor of some software, a story or whatever product, you get to decide what to do with it. You can keep it for yourself or you can give it away. If you decide for the latter, you have to decide who may use it and in which way(s). In case you intend to give it to a (potentially) large group of people, you may not want to be asked for permission to xyz by everybody. That's when you decide to write a license which states what you are allowing and explicitly disallowing. Most of the well-known commercial licenses focus on what you're not allowed to do (usually things like copying, disassembling, etc.). Open source licenses on the other hand are meant to grant the user rights (e.g. the right to distribute) while reserving some rights or only giving permission under certain conditions – and they usually make you claim responsibility for using the software. For these reasons licenses can actually be a good thing! If you got an unlicensed piece of code, you're not legally allowed to do anything with it without getting the author's permission first. And even if you got that permission, your project would be risky, since the author can withdraw it later. A proper license protects both parties. The author doesn't get his mail account full of email asking for permission, he's save from legal trouble if his code breaks anything for you and at the same time you have legal certainty when you decide to put the code to long-term use. Permissive vs. Copyleft (in a nutshell) In short terms, permissive licensing usually goes like this: “Here you are, have fun. Oh, and don't sue me if it does something else than what you expect!” Yes, it's that easy and there's little to dispute over. Copyleft on the other side sounds like this (if you ask somebody in favor of Copyleft): “Sure, you can use it, it's free. Just keep it free, ok?”. Also quite simple. And not too bad, eh? Other people however read the same thing like this: “Yes, you're free to use it. Just read these ten pages of legalese and be dead certain that you comply. If you got something wrong, we will absolutely make you regret it.” The GNU Public license (GPL) The most popular copyleft license in use is the GPL (in various versions) (https://www.gnu.org/licenses/gpl.html). It got more and more complex with each version – and to be fair, it had to, because it was necessary to react to new threats and loop holes that were found later. The GNU project states that they are committed to protect what they call the four freedoms of free software: the freedom to use the software for any purpose the freedom to change the software to suit your needs the freedom to share the software with your friends and neighbors the freedom to share the changes you make These are freedoms that every supporter of open source software should be able to agree with. So what's the deal with all the hostility and fighting between the two camps? Let's take a look at a permissive license, too. The BSD license Unlike the GPL, the BSD family of licenses begun with a rather simple license that span four rules (“original BSD license”). It was later revised and reduced to three (“modified BSD license”). And the modern BSD license that e.g. FreeBSD uses is even just two (“simplified BSD license”). Did you read the GPLv3 that I linked to above? If you are using GPL'd code you really should. In case you don't feel like reading all of it, at least take a look and grasp how long that text is. Now compare it to the complete modern BSD license (https://opensource.org/licenses/bsd-license.php). What's the problem? There are essentially two problems that cause all the trouble. The first one is the question of what should be subject to the freedom that we're talking about. And closely related, the second one is where that freedom needs to end. Ironically both camps claim that freedom is the one important thing and it must not be restricted. The GPL is meant to protect the freedom of the software and enforces the availability of the source code, hence limiting the freedom of actual persons. BSD on the other hand is meant to protect the freedom of human beings who should be able to use the software as they see fit – even if that means closing down former open source code! The GNU camp taunts permissive licenses as being “lax” for not providing the protection that they want. The other camp points out that the GPL is a complex monster and that it is virulent in nature: Since it's very strict in a lot of areas, it's incompatible with many other licenses. This makes it complicated to mix GPL and non-GPL code and in the cases where it's legally possible, the GPL's terms will take precedence and necessarily be in effect for the whole combined work. Who's right? That totally depends on what you want to achieve. There are pros and cons to both – and in fact we're only looking at the big picture here. There's also e.g. the Apache license which is often deemed as kind of middle ground. Then you may want to consider the difference between weak (e.g. LGPL) as well as strong copyleft (GPL). Licensing is a potentially huge topic. But let's keep it simple here because the exact details are actually not necessary to understand the essence of our topic. In the next post I'll present my stance on why permissive licensing is a good thing and copyleft is more problematic than many people may think. “Permissive licensing is wrong?” – No it's not! (https://eerielinux.wordpress.com/2018/01/25/permissive-licensing-is-wrong-no-its-not-2-2/) The previous post gave a short introduction into the topic of software licenses, focusing on the GPL vs. BSD discussion. This one is basically my response to some typical arguments I've seen from people who seem to loathe permissive licensing. I'll write this in dialog style, hoping that this makes it a little lighter to read. Roundup Install OpenBSD on dedibox with full-disk encryption (https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/) TL;DR: I run several "dedibox" servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server OpenBSD is not officially supported, I can't guarantee that this will work for you on any kind of server online.net provides, however I've been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up. Currently, I'm running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them. Recently I've been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references. I wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks. The method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice. Step #1: boot to rescue mode Step #2: boot to the installer Step #3: prepare softraid Step #4: reboot to encrypted OpenBSD system Bonus: further tightening your system enable doas disable the root account update system with syspatch add my ssh public key to my ~/.ssh/authorized_keys disable password authentication within ssh reboot so you boot on a brand new up-to-date system with latest stable kernel VOILA ! January 2018 Development Projects Update (https://www.freebsdfoundation.org/blog/january-2018-development-projects-update/) Spectre and Meltdown in FreeBSD Issues affecting most CPUs used in servers, desktops, laptops, and mobile devices are in the news. These hardware vulnerabilities, known by the code-names “Meltdown” and “Spectre”, allow malicious programs to read data to which they should not have access. This potentially includes credentials, cryptographic material, or other secrets. They were originally identified by a researcher from Google's Project Zero, and were also independently discovered by researchers and academics from Cyberus Technology, Graz University of Technology, the University of Pennsylvania, the University of Maryland, Rambus, the University of Adelaide and Data61. These vulnerabilities affect many CPU architectures supported by FreeBSD, but the 64-bit x86 family of processors from Intel and AMD are the most widely used, and are a high priority for software changes to mitigate the effects of Meltdown and Spectre. In particular, the Meltdown issue affects Intel CPUs and may be used to extract secret data from the running kernel, and therefore, is the most important issue to address. The FreeBSD Foundation collaborates with Intel, and under this relationship participated in a briefing to understand the details of these issues and plan the mitigations to be applied to the x86 architectures supported by FreeBSD. We also made arrangements to have FreeBSD's security officer join me in the briefing. It is through the generous support of the Foundation's donors that we are able to dedicate resources to focus on these issues on demand as they arise. Foundation staff member Konstantin (Kostik) Belousov is an expert on FreeBSD's Virtual Memory (VM) system as well as low-level x86 details, and is developing the x86 kernel mitigations for FreeBSD. The mitigation for Meltdown is known as Page Table Isolation (PTI). Kostik created a PTI implementation which was initially committed in mid-January and is available in the FreeBSD-CURRENT development repository. This is the same approach used by the Linux kernel to mitigate Meltdown. One of the drawbacks of the PTI mitigation is that it incurs a performance regression. Kostik recently reworked FreeBSD's use of Process-Context Identifiers (PCID) in order to regain some of the performance loss incurred by PTI. This change is also now available in FreeBSD-CURRENT. The issue known as Spectre comes in two variants, and variant 2 is the more troubling and pressing one. It may be mitigated in one of two ways: by using a technique called “retpoline” in the compiler, or by making use of a CPU feature introduced in a processor microcode update. Both options are under active development. Kostik's change to implement the CPU-based mitigation is currently in review. Unfortunately, it introduces a significant performance penalty and alternatives are preferred, if available. For most cases, the compiler-based retpoline mitigation is likely to be the chosen mitigation. Having switched to the Clang compiler for the base system and most of the ports collection some years ago, FreeBSD is well-positioned to deploy Clang-based mitigations. FreeBSD developer Dimitry Andric is spearheading the update of Clang/LLVM in FreeBSD to version 6.0 in anticipation of its official release; FreeBSD-CURRENT now includes an interim snapshot. I have been assisting with the import, particularly with respect to LLVM's lld linker, and will support the integration of retpoline. This support is expected to be merged into FreeBSD in the coming weeks. The Foundation's co-op students have also participated in the response to these vulnerabilities. Mitchell Horne developed the patch to control the PTI mitigation default setting, while Arshan Khanifar benchmarked the performance impact of the in-progress mitigation patches. In addition, Arshan and Mitchell each developed changes to FreeBSD's tool chain to support the full set of mitigations that will be applied. These mitigations will continue be tested, benchmarked, and refined in FreeBSD-CURRENT before being merged into stable branches and then being made available as updates to FreeBSD releases. Details on the timing of these merges and releases will be shared as they become available. I would like to acknowledge all of those in the FreeBSD community who have participated in FreeBSD's response to Meltdown and Spectre, for testing, reviewing, and coordinating x86 mitigations, for developing mitigations for other processor architectures and for the Bhyve hypervisor, and for working on the toolchain-based mitigations. Guides: Getting Started & Lumina Theme Submissions (https://lumina-desktop.org/guides-getting-started-lumina-themes/) I am pleased to announce the beginning of a new sub-series of blog posts for the Lumina project: Guides! The TrueOS/Lumina projects want to support our users as they use Lumina or experiment with TrueOS. To that end, we've recently set up a central repository for our users to share instructions or other “how-to” guides with each other! Project developers and contributors will also submit guides to the repository on occasion, but the overall goal is to provide a simple hub for instructions written by any Lumina or TrueOS user. This will make it easier for users to not only find a “how-to” for some procedure, but also a very easy way to “give back” to the community by writing simple instructions or more detailed guides. Guides Repository Our first guide to get the whole thing started was created by the TrueOS Linebacker (https://discourse.trueos.org/t/introducing-the-trueos-linebacker/991) (with technical assistance from our own q5sys). In this guide, Terry Tate will walk you through the steps necessary to submit new wallpaper images to the Lumina Themes collection. This procedure is fully documented with screenshots every step of the way, walking you through a simple procedure that only requires a web browser and a Github account! Guide: Lumina Themes Submissions (https://github.com/trueos/guides/blob/master/lumina-themes-submissions/readme.md) The end result of this guide was that Terry Tate was able to submit this cool new “Lunar-4K” wallpaper to the “lumina-nature” collection. TrueOS Community Guides (https://github.com/trueos/guides/tree/master) ZFS vs. OpenZFS (by Michael Dexter) (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You've probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. Our Senior Analyst clears up what ZFS and OpenZFS refer to and how they differ. I admit that we geeks tend to get caught up in the nuts and bolts of enterprise storage and overlook the more obvious questions that users might have. You've probably noticed that this blog and the FreeNAS blog refer to “ZFS” and “OpenZFS” seemingly at random when talking about the amazing file system at the heart of FreeNAS and every storage product that iXsystems sells. I will do my best to clarify what exactly these two terms refer to. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset' and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Yes, ZFS is really as good as people say it is. After enjoying nearly a decade of refinement by a growing group of developers around the world, ZFS became the property of database vendor Oracle, which ceased public development of both ZFS and OpenSolaris in 2010. Disappointed but undeterred, a group of OpenSolaris users and developers forked the last public release of OpenSolaris as the Illumos project. To this day, Illumos represents the official upstream home of the Open Source OpenSolaris technologies, including ZFS. The Illumos project enjoys healthy vendor and user participation but the portable nature and compelling features of ZFS soon produced far more ZFS users than Illumos users around the world. While most if not all users of Illumos and its derivatives are ZFS users, the majority of ZFS users are not Illumos users, thanks significantly in part to FreeNAS which uses the FreeBSD operating system. This imbalance plus several successful ZFS Day events led ZFS co-founder Matt Ahrens and a group of ZFS developers to announce the OpenZFS project, which would remain a part of the Illumos code base but would be free to coordinate development efforts and events around their favorite file system. ZFS Day has grown into the two-day OpenZFS Developer Summit and is stronger than ever, a testament to the passion and dedication of the OpenZFS community. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs' command, which on FreeBSD relies on the ‘zfs.ko' kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. Beastie Bits Explaining Shell (https://explainshell.com/) OPNsense® 18.1 Released (https://opnsense.org/opnsense-18-1-released/) “SSH Mastery 2/e” copyedits back (https://blather.michaelwlucas.com/archives/3104) Sponsoring a Scam (https://blather.michaelwlucas.com/archives/3106) Thursday, February 8, 2018 - Come to Netflix to talk about FreeBSD (https://www.meetup.com/BAFUG-Bay-Area-FreeBSD-User-Group/events/246623825/) BSD User Group meeting in Stockholm: March 22, 17:30 - 21:00 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) FreeBSD Flavoured talks from Linux.conf.au: You can't unit test C, right? (https://www.youtube.com/watch?v=z-uWt5wVVkU) and A Brief History of I/O (https://www.youtube.com/watch?v=qAhZEI_6lbc) EuroBSDcon 2018 website is up (https://2018.eurobsdcon.org/) Full day bhyvecon Tokyo, Japan, March 9, 2018 (http://bhyvecon.org/) *** Feedback/Questions Thomas - freebsd installer improvements (http://dpaste.com/3G2F7RC#wrap) Mohammad - FreeBSD 11 installation from a read only rescue disk (http://dpaste.com/0HGK3FQ#wrap) Stan - Follow up on guide you covered (http://dpaste.com/2S169SH#wrap) Jalal - couple questions (http://dpaste.com/35N8QXP#wrap)

This week on BSD Now, we clear up some ZFS FUD, show you how to write a NetBSD kernel module, and cover DragonflyBSD on the desktop. This episode was brought to you by Headlines ZFS is the best file system (for now) (http://blog.fosketts.net/2017/07/10/zfs-best-filesystem-now/) In my ongoing effort to fight misinformation and FUD about ZFS, I would like to go through this post in detail and share my thoughts on the current state and future of OpenZFS. The post starts with: ZFS should have been great, but I kind of hate it: ZFS seems to be trapped in the past, before it was sidelined it as the cool storage project of choice; it's inflexible; it lacks modern flash integration; and it's not directly supported by most operating systems. But I put all my valuable data on ZFS because it simply offers the best level of data protection in a small office/home office (SOHO) environment. Here's why. When ZFS first appeared in 2005, it was absolutely with the times, but it's remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10 The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. The lawsuit was resolved, and it didn't really impede ZFS. Some say it is the reason that Apple didn't go with ZFS, but there are other theories too. By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. Yes, Oracle taking over Sun and closing the source for ZFS definitely seemed like a setback at the time, but the OpenZFS project was started and active development has continued as an ever increasing pace. As of today, more than half of the code in OpenZFS has been written since the fork from the last open version of Oracle ZFS. the CDDL license Sun applied to the ZFS code was https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ (judged incompatible) with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world's server operating system. That hasn't stopped the ZFS-on-Linux project, or Ubuntu… Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and http://blog.fosketts.net/2008/09/15/greenbytes-embraces-extends-zfs/ (GreenBytes) helped push ZFS forward in the enterprise, but Oracle's lackluster commitment to Sun in the datacenter started having an impact. Lots of companies have adopted OpenZFS for their products. Before OpenZFS, there were very few non-Sun appliances that used ZFS, now there are plenty. OpenZFS Wiki: Companies with products based on OpenZFS (http://open-zfs.org/wiki/Companies) OpenZFS remains little-changed from what we had a decade ago. Other than the fact that half of the current code did not exist a decade ago… Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. This is one of the weaker points in ZFS. As it turns out, the demand for deduplication is actually not that strong. Most of the win can be had with transparent compression. However, there are a number of suggested designs to work around the dedup problems: Dedup Ceiling: Set a limit on the side of the DDT and just stop deduping new unique blocks when this limit is reached. Allocation Classes: A feature being developed by Intel for a supercomputer, will allow different types of data to be classified, and dedicated vdevs (or even metaslabs within a vdev), to be dedicated to that class of data. This could be extended to having the DDT live on a fast device like an PCIe NVMe, combined with the Dedup Ceiling when the device is full. DDT Pruning: Matt Ahrens described a design where items in the DDT with only a single reference, would be expired in an LRU type fashion, to allow newer blocks to live in the DDT in hopes that they would end up with more than a single reference. This doesn't cause bookkeeping problems since when a block is about to be freed, if it is NOT listed in the DDT, ZFS knows it was never deduplicated, so the current block must be the only reference, and it can safely be freed. This provides a best case scenario compared to Dedup Ceiling, since blocks that will deduplicate well, are likely to be written relatively close together, whereas the chance to a dedup match on a very old block is much lower. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today's prices. As we talked about a few weeks ago, ECC is best, but it is not required. If you want your server to stay up for a long time, to be highly available, you'll put ECC in it. Don't let a lack of ECC stop you from using ZFS, you are just putting your data at more risk. The scrub of death is a myth. ZFS does not ‘require' lots of ram. Your NAS will work happily with 8 GB instead of 32 GB of RAM. Its cache hit ratio will be much lower, so performance will be worse. It won't be able to buffer as many writes, so performance will be worse. Copy-on-Write has some drawbacks, data tends to get scattered and fragmented across the drives when it is written gradually. The ARC (RAM Cache) lessens the pain of this, and allows ZFS to batch incoming writes up into nice contiguous writes. ZFS purposely alternates between reading and writing, since both are faster when the other is not happening. So writes are batched up until there is too much dirty data, or the timeout expires. Then reads are held off while the bulk linear write finishes as quickly as possible, and reads are resumed. Obviously all of this works better and more efficiently in larger batches, which you can do if you have more RAM. ZFS can be tuned to use less RAM, and if you do not have a lot of RAM, or you have a lot of other demand on your RAM, you should do that tuning. And ZFS never really adapted to today's world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It's laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it's everywhere in performance PC's. Make up your mind, is 32GB of ram too expensive or not… the L2ARC exists specifically for the case where it is not possible to just install more RAM. Be it because there are no more slots, of limits of the processor, or limits of your budget. The SLOG is optional, but it never needs to be very big. A number of GBs of SLC flash is all you need, it is only holding writes that have not been flushed to the regular storage devices yet. The reason the documentation talks about SLC specifically is because your SLOG needs a very high write endurance, something never the newest NVMe devices cannot yet provide. Of course you can use NVMe devices with ZFS, lots of people do. All flash ZFS arrays are for sale right now. Other than maybe a little tuning of the device queue depths, ZFS just works and there is nothing to think about. However, to say there is nothing happening in this space is woefully inaccurate. The previously mentioned allocation classes code can be used to allocate metadata (4 KB blocks) on SSD or NVMe, while allocating bulk storage data (up to 16 MB blocks) on spinning disks. Extended a bit beyond what Intel is building for their super computer, this will basically create hybrid storage for ZFS. With the metaslab classes feature, it will even be possible to mix classes on the same device, grouping small allocations and large allocations in different areas, decreasing fragmentation. Then there's the question of flexibility, or lack thereof. Once you build a ZFS volume, it's pretty much fixed for life. There are only three ways to expand a storage pool: Replace each and every drive in the pool with a larger one (which is great but limiting and expensive) It depends on your pool layout. If you design with this in mind using ZFS Mirrors, it can be quite useful Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff) The unbalanced LUNs performance issues were sorted out in 2013-2016. 2014: OpenZFS Allocation Performance (http://open-zfs.org/w/images/3/31/Performance-George_Wilson.pdf) 2016: OpenZFS space allocation: doubling performance on large and fragmented pools (http://www.bsdcan.org/2016/schedule/events/710.en.html) These also mostly solved the performance issues when a pool gets full, you can run a lot closer to the edge now Build a new pool and “zfs send” your datasets to it (which is what I do, even though it's kind of tricky) This is one way to do it, yes. There is another way coming, but I can't talk about it just yet. Look for big news later this year. Apart from option 3 above, you can't shrink a ZFS pool. Device removal is arriving now. It will not work for RAIDZ*, but for Mirrors and Stripes you will be able to remove a device. I've probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it's startlingly limiting and out of touch with the present solid-state-dominated storage world. I don't feel like ZFS is out of touch with solid state. Lots of people are running SSD only pools. I will admit the tiered storage options in ZFS are a bit limited still, but there is a lot of work being done to overcome this. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It's going to be a long time before I trust anything other than ZFS! + I agree. + ZFS has a great track record of doing its most important job, keeping your data safe. + Work is ongoing to make ZFS more performance, and more flexible. The import thing is that this work is never allowed to compromise job #1, keeping your data safe. + Hybrid/tiered storage features, re-RAID-ing, are coming + There is a lot going on with OpenZFS, check out the notes from the last two OpenZFS Developer Summits just to get an idea of what some of those things are: 2015 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2015) & 2016 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2016) Some highlights: Compressed ARC Compressed send/recv ABD (arc buf scatter/gather) ZFS Native Encryption (scrub/resilver, send/recv, etc without encryption keys loaded) Channel Programs (do many administrative operations as one atomic transaction) Device Removal Redacted send/recv ZStandard Compression TRIM Support (FreeBSD has its own, but this will be more performant and universal) Faster Scrub/Resilver (https://youtu.be/SZFwv8BdBj4) Declustered RAID (https://youtu.be/MxKohtFSB4M) Allocation Classes (https://youtu.be/28fKiTWb2oM) Multi-mount protection (for Active/Passive failover) Zpool Checkpoint (undo almost anything) Even more Improved Allocator Performance vdev spacemap log ZIL performance improvements (w/ or w/o SLOG) Persistent L2ARC What I don't think the author of this article understands is how far behind every other filesystem is. 100s of Engineer years have gone into OpenZFS, and the pace is accelerating. I don't see how BtrFS can ever catch up, without a huge cash infusion. Writing a NetBSD kernel module (https://saurvs.github.io/post/writing-netbsd-kern-mod/) Kernel modules are object files used to extend an operating system's kernel functionality at run time. In this post, we'll look at implementing a simple character device driver as a kernel module in NetBSD. Once it is loaded, userspace processes will be able to write an arbitrary byte string to the device, and on every successive read expect a cryptographically-secure pseudorandom permutation of the original byte string. You will need the NetBSD Source Code. This doc (https://www.netbsd.org/docs/guide/en/chap-fetch.html) will explain how you can get it. The article gives an easy line by line walkthrough which is easy to follow and understand. The driver implements the bare minimum: open, close, read, and write, plus the module initialization function It explains the differences in how memory is allocated and freed in the kernel It also describes the process of using UIO to copy data back and forth between userspace and the kernel Create a Makefile, and compile the kernel module Then, create a simple userspace program to use the character device that the kernel module creates All the code is available here (https://github.com/saurvs/rperm-netbsd) *** DragonFlyBSD Desktop! (https://functionallyparanoid.com/2017/07/11/dragonflybsd-desktop/) If you read my last post (https://functionallyparanoid.com/2017/06/30/boot-all-the-things/), you know that I set up a machine (Thinkpad x230) with UEFI and four operating systems on it. One, I had no experience with – DragonFlyBSD (other than using Matthew Dillon's C compiler for the Amiga back in the day!) and so it was uncharted territory for me. After getting the install working, I started playing around inside of DragonFlyBSD and discovered to my delight that it was a great operating system with some really unique features – all with that BSD commitment to good documentation and a solid coupling of kernel and userland that doesn't exist (by design) in Linux. So my goal for my DragonFlyBSD desktop experience was to be as BSD as I possibly could. Given that (and since I'm the maintainer of the port on OpenBSD ), I went with Lumina as the desktop environment and XDM as the graphical login manager. I have to confess that I really like the xfce terminal application so I wanted to make sure I had that as well. Toss in Firefox, libreOffice and ownCloud sync client and I'm good to go! OK. So where to start. First, we need to get WiFi and wired networking happening for the console at login. To do that, I added the following to /etc/rc.conf: wlans_iwn0=”wlan0″ ifconfig_wlan0=”WPA DHCP” ifconfig_em0=”DHCP” I then edited /etc/wpa_supplicant.conf to put in the details of my WiFi network: network={ ssid=”MY-NETWORK-NAME” psk=”my-super-secret-password” } A quick reboot showed that both wired and wireless networking were functional and automatically were assigned IP addresses via DHCP. Next up is to try getting into X with whatever DragonFlyBSD uses for its default window manager. A straight up “startx” met with, shall we say, less than stellar results. Therefore, I used the following command to generate a simple /etc/X11/xorg.conf file: # Xorg -configure # cp /root/xorg.conf.new /etc/X11/xorg.conf With that file in place, I could get into the default window manager, but I had no mouse. After some searching and pinging folks on the mailing list, I was able to figure out what I needed to do. I added the following to my /etc/rc.conf file: moused_enable=”YES” moused_type=”auto” moused_port=”/dev/psm0″ I rebooted (I'm sure there is an easier way to get the changes but I don't know it… yet) and was able to get into a basic X session and have a functional mouse. Next up, installing and configuring Lumina! To do that, I went through the incredibly torturous process of installing Lumina: # pkg install lumina Wow! That was really, really hard. I might need to pause here to catch my breath.

Controlling the Transmissions This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** Headlines FreeBSD 2015 Vendor Dev Summit (https://wiki.freebsd.org/201511VendorDevSummit) FreeBSD Quarterly Status Report - Third Quarter 2015 (https://www.freebsd.org/news/status/report-2015-07-2015-09.html) We have a fresh quarterly status report from the FreeBSD project. Once again it almost merits an entire show, but we will try to hit all the highlights. Bhyve - Porting of the Intel edk2 UEFI firmware, allowing Windows in headless mode, and Illumos support. Also porting to ARM has begun! Improved Support for Acer C720 ChromeBooks High Availability Clustering in CTL (Cam Target Layer) Root Remounting (Similar to pivot_root in Linux). This work allows using “reboot -r” to do a fast-reboot, with a partial shutdown, kill all processes, and re-mount rootfs and boot. Especially useful for booting from mfs or similar then transitioning to iscsi or some other backing storage OpenCL Support in Mesa, as well as kernel progress on the i915 driver Improved support for UEFI FrameBuffer on a bunch of recent MacBook Pro and other Macs, in addition to improvements to “vt” framebuffer driver for high resolution displays. ZFS support for UEFI Boot (Needs testing, but used in PC-BSD for a couple months now), and importing new features from IllumOS (resumable send, receive prefetch, replication checksumming, 50% less ram required for L2ARC, better prefetch) DTrace SDT probes added to TCP code, to replace the old TCPDEBUG kernel option. Recompiling the kernel is no longer required to debug TCP, just use DTrace Ongoing work to bring us a native port/package of GitLab *** Meteor, the popular javascript web application framework has been forked to run on FreeBSD, OpenBSD and NetBSD - FreeBSD testers requested (https://forums.meteor.com/t/freebsd-testers-please/12919/10) We have a public call for testing for FreeBSD users of Meteor by Tom Freudenberg The included link includes all the details on how to currently get meteor boot-strapped on your box and bring up the server So far the reports are positive, many users reporting that it is running on their 10.2 systems / jails just fine. Just a day ago the original porter mentioned that OpenBSD is ready to go for testing using the prepared dev bundle. *** Mike Larkin work continues on an native OpenBSD hypervisor, which he has announced is now booting (http://undeadly.org/cgi?action=article&sid=20151101223132) Speaking of OpenBSD, we have an update from Mike Larkin about the status of the OpenBSD native hypervisor vmm(4). His twitter post included the output from a successful VM bootup of OpenBSD 5.8-current, all the way to multi-user While the code hasn't been committed (yet) we will keep you informed when it lands so you too can begin playing with it. *** This is how I like open source (http://blog.etoilebsd.net/post/This_is_how_I_like_opensource) A blog post by FreeBSD Core Team member, and one of the lead developers of pkg, Baptiste Daroussin One project he has been working on is string collation Garrett d'Amore (of IllumOS) implemented unicode string collation while working for Nexenta and made it BSD license John Marino (from Dragonfly) imported the work done on Illumos into Dragonfly, while he was doing that he decided, it was probably a good idea to rework how locales are handled He discovered that Edwin Groothuis (from FreeBSD) had long ago started a project to simplify locales handling on FreeBSD He extended the tools written by Edwin and has been able to update Dragonfly to the latest (v27 so far) unicode definitions John Marino has worked with Bapt many times on various projects (including bringing pkg and ports to Dragonfly) Bapt decided it was time that FreeBSD got proper string collation support as well, and worked with John to import the support to FreeBSD Bapt spotted a couple of bugs and worked with John on fixing them: issues with eucJP encoding, issues with Russian encoding (John did most of the work on tracking down and fixing the bugs), Bapt also converted localedef (the tool to generate the locales) into using BSD license only code (original version used the CDDL libavl library which I modified to use tree(3)), fixed issues. I also took the locale generation from Edwin (extended by John) This work resulted in a nice flow of patches going from Dragonfly to FreeBSD and from FreeBSD to Dragonfly. And now Garrett is interested in grabbing back our patches into Illumos! The result of this collaboration is that now 3 OS share the same implementation for collation support! This is very good because when one discovers a bug the 3 of them benefit the fix! The biggest win here is that this was a lot of work, and not an area that many people are interested in working on, so it was especially important to share the work rather than reimplement it separately. *** Interview - Hiren Panchasara - hiren@freebsd.org (mailto:hiren@freebsd.org) / @hirenpanchasara (https://twitter.com/hirenpanchasara) Improving TCP *** iXsystems MissonComplete winners (https://www.ixsystems.com/whats-new/october-missioncomplete-winners/) *** News Roundup LibreSSL 2.3.1 released (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.3.1-relnotes.txt) LibreSSl keeps on chugging, the latest release has landed, 2.3.1, which is the second snapshot based upon the OpenBSD 5.9 development branch. Currently they are targeting a stable ABI/API sometime around March 2016 for the 2.3.X series. Included in this update are ASN. 1 cleanups and some compliance fixes for RFC5280 Switched internally to timet, with a check that the host OS supports 64bit timet Various TLS fixes, including the ability to check cert validity times with tlspeercert_not{before|after} Fixed a reported memory leak in OBJ_obj2txt *** Guide for Installing Ghost w/ Nginx on FreeBSD (http://linoxide.com/linux-how-to/install-ghost-nginx-freebsd-10-2/) A nice walkthrough for the week, we've found an article about how to install the Ghost blogging platform on FreeBSD 10.2. For those who don't know, Ghost is a MIT licensed blogging tool, started in 2012 by a former WordPress UI developer and is entirely coded in Node.js While a port for FreeBSD does not yet exist (somebody get on that please), this tutorial can walk you through the process of getting it deployed manually Most of the requirements are simple, www/node, www/npm and sqlite3. With those installed, most of the steps are simply creating the username / home for ghost, and some “npm” setup. The walkthrough even includes a handy rc.d script, making the possibility of a port seem much more likely *** Adrian Chadd on 'Why attention to detail matters when you're a kernel developer (http://adrianchadd.blogspot.com/2015/10/fixing-up-qca9558-performance-on.html) Adrian was correctly trolled in the FreeBSD embedded IRC chatroom and started looking at why the bridging performance in MIPS boards was so bad 120-150 mbit/sec is not really enough anymore Using previous MIPS24k support as a starting point, Adrian managed to get HWPMC (Hardware Performance Monitoring Counters) working on MIPS74k Using the data collected from the performance counters Adrian was able to figure out that packets were being copied in order to meet alignment requirements of the NIC and the FreeBSD networking stack. It turns out this is no longer a requirement for most modern Atheros NICs, so the workaround could be removed Now performance was 180 mbit/sec Next, on the receive side, only the TCP stack requires strict alignment, the ethernet stack does not, so offset the start point by 2 bytes so that TCP ends up aligned, and problem solved. Or not, no performance difference... The problem appeared to be busdma, Ian Lepore had recently made improves in this area on armv6 and helpfully ported these over to MIPS Now 420 mbit/sec. Getting better, but not as fast as Linux After some further investigation, a missing ‘sync' operation was added, and the memory caching was changed from writethrough to writeback Things were so fast now, that the descriptor ring was being run through the ring so quickly as to hit the next descriptor that is still being setup. The first was to mark the first descriptor of a multi-descriptor packet as ‘empty' until the entire chain was setup, so it would not be processed before the latter bits were done being added to the ring. So now MIPS can bridge at 720 mbit/sec, and route 320 mbit/sec Adrian wants to improve the routing speed and get it caught up to the bridging speed, but as always, free time is scarce. *** Switching from OS X to FreeBSD (http://mirrorshades.net/post/132753032310) The story of a user who had used OS X since its beta, but 10.9 and 10.10, became more and more dissatisfied They found they were spending too much time fighting with the system, rather than getting work done They cover the new workstation they bought, and the process of getting FreeBSD going on it, including why they chose FreeBSD rather than PCBSD Also covered it setting up a Lenovo X220 laptop They setup the i3wm and mutt The blog is very detailed and goes so far as to share a github repo of dotfiles and configuration files to ease the transition from OS X. *** BeastieBits The Stack behind Netflix's scaling (http://www.scalescale.com/the-stack-behind-netflix-scaling/) The Amiga port of NetBSD now has xorg support (https://mail-index.netbsd.org/source-changes/2015/11/04/msg069873.html) NetBSD has announced EOL for v5.x to be November 9th (http://blog.netbsd.org/tnf/entry/end_of_life_for_netbsd) RetroArch ports allow playing PlayStation, Sega, Atari, etc., games on FreeBSD (https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058266.html) OpenBSD booting on a 75mhz Cyrex system with 32MB RAM (http://gfycat.com/InnocentSneakyEwe) Matthew Green reports Nouveau Nvidia can support GL with his latest commit (http://mail-index.netbsd.org/source-changes/2015/10/29/msg069729.html) Releases! OPNsense releases 15.7.18 (https://opnsense.org/opnsense-15-7-18-released/) pfSense releases 2.2.5 (https://blog.pfsense.org/?p=1925) Feedback/Questions Eric (http://slexy.org/view/s2ogdURldm) Andrew (http://slexy.org/view/s22bK2LZLm) Joseph (http://slexy.org/view/s2to6ZpBTc) Sean (http://slexy.org/view/s2oLU0KM7Y) Dustin (http://slexy.org/view/s21k6oKvle) *** For those of you curious about Kris' new lighting here are the links to what he is using. Softbox Light Diffuser (http://smile.amazon.com/gp/product/B00OTG6474?psc=1&redirect=true&ref_=oh_aui_detailpage_o01_s00&pldnSite=1) Full Spectrum 5500K CFL Bulb (http://smile.amazon.com/gp/product/B00198U6U6?psc=1&redirect=true&ref_=oh_aui_detailpage_o06_s00) ***

On this week's episode, we'll be talking with Ted Unangst of the OpenBSD team about their new signing infrastructure. After that, we've got a tutorial on how to run your own NTP server. News, your feedback and even... the winner of our tutorial contest will be announced! So stay tuned to BSD Now - the place to B.. SD. This episode was brought to you by Headlines FreeBSD foundation's 2013 fundraising results (http://freebsdfoundation.blogspot.com/2014/01/freebsd-foundation-announces-2013.html) The FreeBSD foundation finally counted all the money they made in 2013 $768,562 from 1659 donors Nice little blog post from the team with a giant beastie picture "We have already started our 2014 fundraising efforts. As of the end of January we are just under $40,000. Our goal is to raise $1,000,000. We are currently finalizing our 2014 budget. We plan to publish both our 2013 financial report and our 2014 budget soon." A special thanks to all the BSD Now listeners that contributed, the foundation was really glad that we sent some people their way (and they mentioned us on Facebook) *** OpenSSH 6.5 released (https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-January/032152.html) We mentioned the CFT last week, and it's finally here (https://news.ycombinator.com/item?id=7154925)! New key exchange using elliptic-curve Diffie Hellman in Daniel Bernstein's Curve25519 (now the default when both clients support it) Ed25519 public keys are now available for host keys and user keys, considered more secure than DSA and ECDSA Funny side effect: if you ONLY enable ed25519 host keys, all the compromised Linux boxes can't even attempt to login (http://slexy.org/view/s2rI13v8F4) lol~ New bcrypt private key type, 500,000,000 times harder to brute force Chacha20-poly1305 transport cipher that builds an encrypted and authenticated stream in one Portable version already in (https://svnweb.freebsd.org/base?view=revision&revision=261320) FreeBSD -CURRENT, and ports (https://svnweb.freebsd.org/ports?view=revision&sortby=date&revision=342618) Lots more bugfixes and features, see the full release note or our interview (http://www.bsdnow.tv/episodes/2013_12_18-cryptocrystalline) with Damien Work has already started on 6.6, which can be used without OpenSSL (https://twitter.com/msfriedl/status/427902493176377344)! *** Crazed Ferrets in a Berkeley Shower (http://blather.michaelwlucas.com/archives/1942) In 2000, MWL (http://www.bsdnow.tv/episodes/2013_11_06-year_of_the_bsd_desktop) wrote an essay for linux.com about why he uses the BSD license: "It's actually stood up fairly well to the test of time, but it's fourteen years old now." This is basically an updated version about why he uses the BSD license, in response to recent comments from Richard Stallman (http://gcc.gnu.org/ml/gcc/2014-01/msg00247.html) Very nice post that gives some history about Berkeley, the basics of the BSD-style licenses and their contrast to the GNU GPL Check out the full post if you're one of those people that gets into license arguments The takeaway is "BSD is about making the world a better place. For everyone." *** OpenBSD on BeagleBone Black (http://www.tedunangst.com/flak/post/OpenBSD-on-BeagleBone-Black) Beaglebone Blacks are cheap little ARM devices similar to a Raspberry Pi A blog post about installing OpenBSD on a BBB from.. our guest for today! He describes it as "everything I wish I knew before installing the newly renamed armv7 port on a BeagleBone Black" It goes through the whole process, details different storage options and some workarounds Could be a really fun weekend project if you're interested in small or embedded devices *** Interview - Ted Unangst - tedu@openbsd.org (mailto:tedu@openbsd.org) / @tedunangst (https://twitter.com/tedunangst) OpenBSD's signify (http://www.tedunangst.com/flak/post/signify) infrastructure, ZFS on OpenBSD Tutorial Running an NTP server (http://www.bsdnow.tv/tutorials/ntpd) News Roundup Getting started with FreeBSD (http://smyck.net/2014/02/01/getting-started-with-freebsd/) A new video and blog series about starting out with FreeBSD The author has been a fan since the 90s and has installed it on every server he's worked with He mentioned some of the advantages of BSD over Linux and how to approach explaining them to new users The first video is the installation, then he goes on to packages and other topics - 4 videos so far *** More OpenBSD hackathon reports (http://undeadly.org/cgi?action=article&sid=20140204080515) As a followup to last week, this time Kenneth Westerback writes about his NZ hackathon experience He arrived with two goals: disklabel fixes for drives with 4k sectors and some dhclient work This summary goes into detail about all the stuff he got done there *** X11 in a jail (https://svnweb.freebsd.org/base?view=revision&revision=261266) We've gotten at least one feedback email about running X in a jail Well.. with this commit, looks like now you can! A new tunable option will let jails access /dev/kmem and similar device nodes Along with a change to DRM, this allows full X11 in a jail Be sure to check out our jail tutorial and jailed VNC tutorial (http://www.bsdnow.tv/tutorials) for ideas *** PCBSD weekly digest (http://blog.pcbsd.org/2014/01/whoami-im-pc-bsd-10-0-weekly-feature-digest-15/) 10.0 "Joule Edition" finally released (http://blog.pcbsd.org/2014/01/pc-bsd-10-0-release-is-now-available/)! AMD graphics are now officially supported GNOME3, MATE and Cinnamon desktops are available Grub updates and fixes PCBSD also got a mention in eweek (http://www.eweek.com/enterprise-apps/slideshows/freebsd-open-source-os-comes-to-the-pc-bsd-desktop.html) *** Feedback/Questions Justin writes in (http://slexy.org/view/s21VnbKZsH) Daniel writes in (http://slexy.org/view/s2nD7RF6bo) Martin writes in (http://slexy.org/view/s2jwRrj7UV) Alex writes in (http://slexy.org/view/s201koMD2c) - unofficial FreeBSD RPI Images (http://people.freebsd.org/~gjb/RPI/) James writes in (http://slexy.org/view/s2AntZmtRU) John writes in (http://slexy.org/view/s20bGjMsIQ) ***

Karen and Bradley discuss the LWN article, GnuTLS, copyright assignment, and GNU project governance and other issues related to copyright assignment. Show Notes: Segment 0 (00:46) Bradley didn't want his words compared to the Ayn Rand's quote from an interview with Phil Donahue where she said I'm not going to die, it's just that world will end. (02:54) Bradley discussed the reaction to on 0x36 that occurred in this identi.ca thread. (04:20) Bradley and Karen discussed the LWN article, GnuTLS, copyright assignment, and GNU project governance. (11:15) Bradley pointed out that every other copyleft license allows for relicensing under newer versions automatically (i.e., they have an automatic -or-later ), and Karen asked whether Sun's CDDL does. Bradley checked later, Karen was correct that CDDL's later version clause (Section 4) is similar to the GPL policy. (23:00) However, Fontana wrote to us on IRC to say CDDL's license upgradeability clause is not entirely like GPL's. The GPL states that if no version number is specified, any version can be used. CDDL does not say this; it seems to assume that it will always be clear what version CDDL code will be distributed under, whereas GPL seems to assume otherwise. Bradley mentioned the interview he did with The H Online on GPL enforcement. (41:57) Send feedback and comments on the cast to . You can keep in touch with Free as in Freedom on our IRC channel, #faif on irc.freenode.net, and by following Conservancy on identi.ca and and Twitter. Free as in Freedom is produced by Dan Lynch of danlynch.org. Theme music written and performed by Mike Tarantino with Charlie Paxson on drums. The content of this audcast, and the accompanying show notes and music are licensed under the Creative Commons Attribution-Share-Alike 4.0 license (CC BY-SA 4.0).

Karen and Bradley discuss the LWN article, GnuTLS, copyright assignment, and GNU project governance and other issues related to copyright assignment. Show Notes: Segment 0 (00:46) Bradley didn't want his words compared to the Ayn Rand's quote from an interview with Phil Donahue where she said I'm not going to die, it's just that world will end. (02:54) Bradley discussed the reaction to on 0x36 that occurred in this identi.ca thread. (04:20) Bradley and Karen discussed the LWN article, GnuTLS, copyright assignment, and GNU project governance. (11:15) Bradley pointed out that every other copyleft license allows for relicensing under newer versions automatically (i.e., they have an automatic -or-later ), and Karen asked whether Sun's CDDL does. Bradley checked later, Karen was correct that CDDL's later version clause (Section 4) is similar to the GPL policy. (23:00) However, Fontana wrote to us on IRC to say CDDL's license upgradeability clause is not entirely like GPL's. The GPL states that if no version number is specified, any version can be used. CDDL does not say this; it seems to assume that it will always be clear what version CDDL code will be distributed under, whereas GPL seems to assume otherwise. Bradley mentioned the interview he did with The H Online on GPL enforcement. (41:57) Send feedback and comments on the cast to . You can keep in touch with Free as in Freedom on our IRC channel, #faif on irc.freenode.net, and by following Conservancy on on Twitter and and FaiF on Twitter. Free as in Freedom is produced by Dan Lynch of danlynch.org. Theme music written and performed by Mike Tarantino with Charlie Paxson on drums. The content of this audcast, and the accompanying show notes and music are licensed under the Creative Commons Attribution-Share-Alike 4.0 license (CC BY-SA 4.0).