POPULARITY
65 episodes with allan jude
5 episodes with allan jude
5 episodes with allan jude
2 episodes with allan jude
3 episodes with allan jude
2 episodes with allan jude
2 episodes with allan jude
3 episodes with allan jude
2 episodes with allan jude
We are joined by Allan Jude to talk about what it's like to run a company that develops and maintains open source software with a focus on upstreaming as much code as possible. Klara November 2023 FreeBSD Vendor Summit – The Value of Upstream First How to upstream code to open source projects FiloSottile (Filippo... Read More
We are joined by Allan Jude to talk about what it's like to run a company that develops and maintains open source software with a focus on upstreaming as much code as possible. Klara November 2023 FreeBSD Vendor Summit – The Value of Upstream First How to upstream code to open source projects FiloSottile (Filippo... Read More
This week we're joined by FreeBSD & OpenZFS developer, Allan Jude, to learn all about FreeBSD. Allan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there & answers every curious question we have about this powerful (yet underrepresented) Unix-based operating system.
This week we're joined by FreeBSD & OpenZFS developer, Allan Jude, to learn all about FreeBSD. Allan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there & answers every curious question we have about this powerful (yet underrepresented) Unix-based operating system.
Recorded at BSDCan 2023 NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines BSDNow - The early years BSDNow - Production Process News Roundup FreeBSD Devsummit BSDCan Beastie Bits How you can help the show! Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Corey asks - what is the status of netbsd 10? How have things changed in the bsds over the history of the show? Announcement As a final thing Allan would like to make an announcement: - Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
Leveraging OpenZFS to Build Your Own Storage Appliance, Install OpenBSD as a VM, Set up your own CalDAV and CardDAV servers on OpenBSD, display basic computer information using DMI table decoder, Gpart CheatSheet, Rob Pike on the Origin of Unix Dot File Names, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines OpenZFS – Leveraging OpenZFS to Build Your Own Storage Appliance (https://klarasystems.com/articles/openzfs-leveraging-openzfs-to-build-your-own-storage-appliance/) Install OpenBSD as a VM (https://byte-sized.de/linux-unix/openbsd-als-vm-installieren/#english) News Roundup Set up your own CalDAV and CardDAV servers on OpenBSD (https://dataswamp.org/~solene/2023-04-23-calendar-and-contacts-with-radicale.html) How to display basic computer information using DMI table decoder (https://sleeplessbeastie.eu/2023/03/31/how-to-display-basic-computer-information-using-dmi-table-decoder/) Gpart CheatSheet - wiping drives, partitioning, & formating (https://forums.FreeBSD.org/threads/gpart-cheatsheet-wiping-drives-partitioning-formating.45411) Rob Pike on the Origin of Unix Dot File Names (http://xahlee.info/UnixResource_dir/writ/unix_origin_of_dot_filename.html) Beastie Bits Hackerstations Mike McQuaid's clean, ergonomic setup in Edinburgh, Scotland (https://hackerstations.com/setups/mike_mcquaid/) Daniel Stenberg and the home of curl in Stockholm, Sweden (https://hackerstations.com/setups/daniel_stenberg/) viogpu(4), a VirtIO GPU driver, added to -current (http://undeadly.org/cgi?action=article;sid=20230421124221) OpenBGPD 8.0 released (http://undeadly.org/cgi?action=article;sid=20230505054214) cron(8) now supports random ranges with steps (http://undeadly.org/cgi?action=article;sid=20230507122935) malloc leak detection available in -current (http://undeadly.org/cgi?action=article;sid=20230417074903) vmd(8) moves to a multi-process model (https://www.undeadly.org/cgi?action=article;sid=20230430051250) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
FreeBSD Foundation Welcomes New Team Members, OpenZFS the Ideal Storage Solution for University Environments, SCaLE20X Conference Report, 916 days of Emacs, XTerm: It's Better Than You Thought, NetBSD Annual General Meeting 2023, and more NOTES** This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Foundation Welcomes New Team Members (https://freebsdfoundation.org/blog/freebsd-foundation-welcomes-new-team-members/) What Makes OpenZFS the Ideal Storage Solution for University Environments (https://klarasystems.com/articles/openzfs-what-makes-openzfs-the-ideal-storage-solution-for-university-environments//) News Roundup SCaLE20X Conference Report (https://freebsdfoundation.org/blog/scale20x-conference-report/) 916 days of Emacs (https://sqrtminusone.xyz/posts/2023-04-13-emacs/) XTerm: It's Better Than You Thought (https://aduros.com/blog/xterm-its-better-than-you-thought/) NetBSD AGM2023: Annual General Meeting, May 13, 21:00 UTC (https://mail-index.netbsd.org/netbsd-announce/2023/05/05/msg000348.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Adrian - Tilde (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/508/feedback/Adrian%20-%20Tilde.md) Dan - Root Shell (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/508/feedback/Dan%20-%20Root%20Shell.md) Florian - Salt Extension (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/508/feedback/Florian%20-%20Salt%20Extension.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
OpenBSD 7.3 released, Accelerating Datacenter Energy Efficiency by Leveraging FreeBSD as Your Server OS, install Cinnamon as a Desktop environment, xmonad FreeBSD set up from scratch, Burgr books in your terminal, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines OpenBSD 7.3 released (http://undeadly.org/cgi?action=article;sid=20230410140049) BSDCan 2023 Schedule posted (https://www.bsdcan.org/events/bsdcan_2023/schedule/) Accelerating Datacenter Energy Efficiency by Leveraging FreeBSD as Your Server OS (https://klarasystems.com/articles/accelerating-datacenter-energy-efficiency-by-leveraging-freebsd-as-your-server-os/) News Roundup FreeBSD – How to install Cinnamon as a Desktop environment (https://byte-sized.de/linux-unix/freebsd-cinnamon-als-gui-installieren/#english) xmonad FreeBSD set up from scratch (https://forums.FreeBSD.org/threads/xmonad-freebsd-set-up-from-scratch.75911/) Burgr books in your terminal (https://blubsblog.bearblog.dev/burgr-books-in-your-terminal/) Pros and Cons of FreeBSD for virtual Servers (https://www.hostzealot.com/blog/about-vps/pros-and-cons-of-freebsd-for-virtual-servers) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Reese - Dans Interview (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/505/feedback/Reese%20-%20Dans%20Interview.md) jj - looking for help (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/505/feedback/jj%20-%20looking%20for%20help.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD 13.2 Release, Using DTrace to find block sizes of ZFS, NFS, and iSCSI, Midnight BSD 3.0.1, Closing a stale SSH connection, How to automatically add identity to the SSH authentication agent, Pros and Cons of FreeBSD for virtual Servers, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD 13.2 Release Announcement (https://www.freebsd.org/releases/13.2R/announce/) Using DTrace to find block sizes of ZFS, NFS, and iSCSI (https://axcient.com/blog/using-dtrace-to-find-block-sizes-of-zfs-nfs-and-iscsi/) News Roundup Midnight BSD 3.0.1 (https://www.phoronix.com/news/MidnightBSD-3.0.1) Closing a stale SSH connection (https://davidisaksson.dev/posts/closing-stale-ssh-connections/) How to automatically add identity to the SSH authentication agent (https://sleeplessbeastie.eu/2023/04/10/how-to-automatically-add-identity-to-the-ssh-authentication-agent/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dan - ZFS question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/504/feedback/Dan%20-%20ZFS%20question.md) Matt - Thanks (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/504/feedback/Matt%20-%20Thanks.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Wireguard VPN Server with Unbound on OpenBSD, Auditing for OpenZFS Storage Performance, OpenBSD 7.2 on a Thinkpad X201, Practical Guides to fzf, Replacing postfix with dma, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines How To Set Up a Wireguard VPN Server with Unbound on OpenBSD (https://marcocetica.com/posts/wireguard_openbsd/) Auditing for OpenZFS Storage Performance (https://klarasystems.com/articles/openzfs-auditing-for-storage-performance/) News Roundup Some notes on OpenBSD 7.2 on a Thinkpad X201 (https://box.matto.nl/some-notes-on-openbsd-72-on-a-thinkpad-x201.html) fzf A Practical Guide to fzf: Building a File Explorer (https://thevaluable.dev/practical-guide-fzf-example/) A Practical Guide to fzf: Shell Integration (https://thevaluable.dev/fzf-shell-integration/) *** Replacing postfix with dma (https://dan.langille.org/2023/02/28/replacing-postfix-with-dma/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dennis - Thanks (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/500/feedback/Dennis%20-%20Thanks.md) Luna - Trillian (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/500/feedback/Luna%20-%20trillian.md) Lyubomir - ipfw question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/500/feedback/Lyubomir%20-%20ipfw%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
How to Catch a Bitcoin Miner, A Call For More Collaboration, zstd updates, hating hackathons, How to monitor multiple log files at once, KeePassXC, sshd random relinking at boot, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Sysadmin Series - How to Catch a Bitcoin Miner (https://klarasystems.com/articles/sysadmin-series-how-to-catch-a-bitcoin-miner/) A Call For More Collaboration & Harmony Among BSD Hardware Drivers (https://fosdem.org/2023/schedule/event/bsd_driver_harmony/) • [Slides](https://fosdem.org/2023/schedule/event/bsd_driver_harmony/attachments/slides/5976/export/events/attachments/bsd_driver_harmony/slides/5976/BSD_Driver_Harmony_FOSDEM.pdf) • Video is embedded on the schedule event page Printing on FreeBSD (https://vermaden.wordpress.com/2023/02/07/print-on-freebsd/) News Roundup zstd updates (https://github.com/facebook/zstd/releases/tag/v1.5.4) I hate hackathons (https://pgpt.substack.com/p/i-hate-hackathons) How to monitor multiple log files at once (https://sleeplessbeastie.eu/2023/02/01/how-to-monitor-multiple-log-files-at-once/) Notes to self: KeePassXC (https://jpmens.net/2023/01/22/notes-to-self-keepassxc/) sshd random relinking at boot (http://undeadly.org/cgi?action=article;sid=20230119075627) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Nelson - aix.md (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/497/feedback/Nelson%20-%20aix.md) Adrian - vbsdcon (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/497/feedback/Adrian%20-%20vbsdcon.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Automation and Hacking Your FreeBSD CLI, Run your own instant messaging service on FreeBSD, Watch Netflix on FreeBSD, HardenedBSD January 2023 Status Report, How To Set Up SSH Keys With YubiKey as two-factor authentication, OpenSSH fixes double-free memory bug that's pokable over the network, A late announcement, but better late than never, Next NYC*BUG and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Automation and Hacking Your FreeBSD CLI (https://klarasystems.com/articles/automation-and-hacking-your-freebsd-cli/) Run your own instant messaging service on FreeBSD (https://xn--gckvb8fzb.com/run-your-own-instant-messaging-service-on-freebsd/) News Roundup Watch Netflix on FreeBSD (https://byte--sized-de.translate.goog/linux-unix/netflix-auf-freebsd-schauen/?_x_tr_sl=de&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp) HardenedBSD January 2023 Status Report (https://hardenedbsd.org/article/shawn-webb/2023-01-31/hardenedbsd-january-2023-status-report) How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2) (https://www.cyberciti.biz/security/how-to-set-up-ssh-keys-with-yubikey-as-two-factor-authentication-u2f-fido2/) OpenSSH fixes double-free memory bug that's pokable over the network (https://nakedsecurity.sophos.com/2023/02/03/openssh-fixes-double-free-memory-bug-thats-pokable-over-the-network/) A late announcement, but better late than never (https://github.com/chettrick/discobsd/releases/tag/DISCOBSD_2_0) Next NYC*BUG: March? April? Certainly May! (https://lists.nycbug.org:8443/pipermail/talk/2023-February/018550.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Daniel - Plan 9 lives (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/496/feedback/Daniel%20-%20Plan%209%20lives.md) Jason - nvd driver (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/496/feedback/Jason%20-%20nvd%20driver.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Write Admin tools from Day One, Differentiating between Data Security and Data Integrity, 45 year-old Unix tool is finally getting an upgrade, OpenBSD 7.2 on an ODROID-HC4, Dotfiles Management, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Write Admin tools from Day One (https://milwaukeemaven.blogspot.com/2022/08/write-admin-tools-from-day-one.html) Differentiating between Data Security and Data Integrity (https://klarasystems.com/articles/openzfs-data-security-vs-integrity/) News Roundup This 45 year-old Unix tool is finally getting an upgrade (https://www.techradar.com/news/45-year-old-unix-tool-finally-gets-an-upgrade) Installing OpenBSD 7.2 on an ODROID-HC4 (https://www.tumfatig.net/2022/install-openbsd-odroid-hc4/) Dotfiles Management (https://mitxela.com/projects/dotfiles_management) Beastie Bits FreeBSD Journal - November/December 2022 - Observability and Metrics (https://freebsdfoundation.org/past-issues/observability-and-metrics/) HAMMER2 file system for NetBSD (https://github.com/kusumi/netbsd_hammer2) Running OpenBSD 7.2 on your laptop is really hard (not) (https://sohcahtoa.org.uk/openbsd.html) MinIO on OpenBSD 7.2: Install (https://dev.to/nabbisen/minio-on-openbsd-72-install-3b3h) WireGuard VPN on OpenBSD (https://www.adrianobarbosa.xyz/blog/openbsd-wireguard.html) A tool for glamorous shell scripts (https://github.com/charmbracelet/gum) Visualize your git commits with a heat map in the terminal (https://github.com/james-stoup/heatwave) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
Writing your own operating system, Continuous Integration and Quality Assurance Update, feeling for the NetBSD community, Testing wanted: execute-only on amd64, GCC uses Modula-2 and Rust, do they work on OpenBSD, Unix is dead; long live Unix, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Part 1: Writing your own operating system (https://o-oconnell.github.io/2023/01/12/p1os.html) 2022 in Review: Continuous Integration and Quality Assurance Update (https://freebsdfoundation.org/blog/2022-in-review-continuous-integration-and-quality-assurance-update/) News Roundup I feel for the NetBSD community (https://rubenerd.com/i-feel-for-the-netbsd-community/) Testing wanted: execute-only on amd64 (https://www.undeadly.org/cgi?action=article;sid=20230115095258) GCC now includes Modula-2 and Rust. Do they work on OpenBSD? (https://briancallahan.net/blog/) Unix is dead. Long live Unix! (https://www.theregister.com/2023/01/17/unix_is_dead/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions • [Kevin - Advent of Computing podcast covers BSD](https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/492/feedback/Kevin%20-%20Advent%20of%20Computing%20podcast%20covers%20BSD.md) • [ilo - thanks](https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/492/feedback/ilo%20-%20thanks.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD vs. Linux – Networking, HDMI sound output through TV speakers on FreeBSD 13, Getting started with tmux, Samba Active Directory, OpenIKED 7.2 released, FreeBSD Plasma 5 GUI Install, DHCP server howto in German, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD vs. Linux – Networking (https://klarasystems.com/articles/freebsd-vs-linux-networking/) (Solved), HDMI sound output through TV speakers Freebsd 13 or @4 plus VCHIQ audio patch - Raspberry Pi Forums (https://forums.raspberrypi.com/viewtopic.php?t=343233) News Roundup Getting started with tmux (https://ittavern.com/getting-started-with-tmux/) Samba Active Directory (https://cromwell-intl.com/open-source/samba-active-directory/freebsd-raspberry-pi.html) OpenIKED 7.2 released (http://undeadly.org/cgi?action=article;sid=20221202230711) FreeBSD Plasma 5 GUI Install (https://byte--sized-de.translate.goog/linux-unix/freebsd-kde-plasma-5-als-gui-installieren/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp) Original German Article (https://byte-sized.de/linux-unix/freebsd-kde-plasma-5-als-gui-installieren/) *** Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Finding a 24 year old bug in ping(8), The Role of Operating Systems in IOT, Authentication gateway with SSH on OpenBSD, FreeBSD 12.4 is out, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Fuzzing ping(8) … and finding a 24 year old bug (https://tlakh.xyz/fuzzing-ping.html) The Role of Operating Systems in IOT (https://klarasystems.com/articles/the-role-of-operating-systems-in-iot/) News Roundup Authentication gateway with SSH on OpenBSD (https://dataswamp.org/~solene/2022-12-01-openbsd-authpf.html) FreeBSD 12.4 is out (https://lists.freebsd.org/archives/freebsd-announce/2022-December/000059.html) Beastie Bits Vagrant FreeBSD Boxbuilder (https://github.com/punktDe/vagrant-freebsd-boxbuilder) LibreSSL 3.7.0 Released (https://undeadly.org/cgi?action=article;sid=20221212183516) OPNsense 22.7.9 released (https://opnsense.org/opnsense-22-7-9-released) BIOS Memory Map for vmd(8) Rewrite in Progress (https://undeadly.org/cgi?action=article;sid=20221211164822) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Research Unix Version 6 in the Open SIMH PDP-11 Emulator, The Hot Tub Time Machine is Your ZFS Turn-Back-Time Method, NFS on NetBSD: server and client side, HardenedBSD October 2022 Status Report, Nushell : Introduction, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Installing and Using Research Unix Version 6 in the Open SIMH PDP-11 Emulator (http://decuser.blogspot.com/2022/10/installing-and-using-research-unix.html) httm – The Hot Tub Time Machine is Your ZFS Turn-Back-Time Method (https://klarasystems.com/articles/httm-is-a-zfs-based-time-machine/) News Roundup NFS on NetBSD: server and client side (https://www.unitedbsd.com/d/959-nfs-on-netbsd-server-and-client-side) HardenedBSD October 2022 Status Report (https://hardenedbsd.org/article/shawn-webb/2022-10-31/hardenedbsd-october-2022-status-report) Nushell : Introduction (https://dataswamp.org/~solene/2022-10-31-nushell.html) Beastie Bits Unix Pipe Game (https://hackaday.com/2022/10/18/if-only-the-kids-knew-about-pipes/) Slides - The “other” FreeBSD optimizations used by Netflix to serve video at 800Gb/s from a single server (https://people.freebsd.org/~gallatin/talks/euro2022.pdf) My FreeBSD Friday Lecture: The Writing Scholar's Guide to FreeBSD (https://www.coreystephan.com/freebsd-friday/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dan - Response to Hans (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/483/feedback/Dan%20-%20Response%20to%20Hans.md) Johnny - bhyve question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/483/feedback/Johnny%20-%20bhyve%20question.md) Manuel - EuroBSDcon social event (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/483/feedback/Manuel%20-%20EuroBSDcon%20social%20event.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
5 Key Reasons to Consider Open Source Storage, OpenBSD Minimalist Desktop, BSD XFCE, Alpine Linux VM on bhyve - with root on ZFS, FreeBSD Jail Quick Setup with Networking, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines 5 Key Reasons to Consider Open Source Storage Over Commercial Offerings (https://klarasystems.com/articles/open-source-storage-over-commercial-offerings/) OpenBSD Minimalist Desktop (https://nechtan.io/articles/openbsd_minimalist_desktop.html) News Roundup BSD-XFCE (https://github.com/Wamphyre/BSD-XFCE) Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted) (https://it-notes.dragas.net/2022/11/01/creating-an-alpine-vm-on-bhyve-with-root-on-zfs-optionally-encrypted/) FreeBSD Jail Quick Setup with Networking (2022) (https://www.shaka.today/freebsd-jail-quick-setup-with-networking-2022/) Beastie Bits EuroBSDcon videos are now up (https://www.youtube.com/c/EuroBSDcon/videos) LibreSSL 3.6.1 released (https://undeadly.org/cgi?action=article;sid=20221104064712) Raspberry Pi 4 with FreeBSD 13-RELEASE: A Perfect Miniature Homelab (https://www.coreystephan.com/pi4-freebsd/) AsiaBSDcon 2023 CfP (https://2023.asiabsdcon.org/cfp.html.en) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions John - Allan's meetup (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/John%20-%20Allan's%20meetup.md) Matthew - atime and a question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/Matthew%20-%20atime%20and%20a%20question.md) Valentin - Becoming a FreeBSD Developer (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/Valentin%20-%20Becoming%20a%20FreeBSD%20Developer.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
EuroBSDcon 2022 as first BSD conference, Red Hat's OpenShift vs FreeBSD Jails, Running a Docker Host under OpenBSD using vmd(8), history of sending signals to Unix process groups, Toolchains adventures - Q3 2022, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines EuroBSDCon 2022, my first BSD conference (and how they are different) (https://eerielinux.wordpress.com/2022/09/25/eurobsdcon-2022-my-first-bsd-conference-and-how-they-are-different/) Red Hat's OpenShift vs FreeBSD Jails (https://klarasystems.com/articles/red-hats-openshift-vs-freebsd-jails/) News Roundup The history of sending signals to Unix process groups (https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessGroupsAndSignals) Running a Docker Host under OpenBSD using vmd(8) (https://www.tumfatig.net/2022/running-docker-host-openbsd-vmd/) Toolchains adventures - Q3 2022 (https://www.cambus.net/toolchains-adventures-q3-2022/) Beastie Bits -current has moved to 7.2 (https://undeadly.org/cgi?action=article;sid=20220912055003) Several /sbin daemons are now dynamically-linked (http://undeadly.org/cgi?action=article;sid=20220830052924) Announcing the pkgsrc 2022Q3 branch (https://mail-index.netbsd.org/netbsd-announce/2022/09/29/msg000341.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Hans - datacenters and dust (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/Hans%20-%20datacenters%20and%20dust.md) Tim - Boot issue (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/Tim%20-%20Boot%20issue.md) aaron- dwm tiling (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/aaron-%20dwm%20tiling%20.md) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Open Source in Enterprise Environments, Your Comprehensive Guide to rc(8): FreeBSD Services and Automation, How Rob Pike got hired by Dennis Richie, what FreeBSD machines rubenerd uses, new debugbreak command, 7 sudo myths debunked NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Open Source in Enterprise Environments - Where Are We Now and What Is Our Way Forward? (https://bsdly.blogspot.com/2022/09/open-source-in-enterprise-environments.html) Your Comprehensive Guide to rc(8): FreeBSD Services and Automation (https://klarasystems.com/articles/rc8-freebsd-services-and-automation/) News Roundup How Rob Pike got hired by Dennis Richie (https://minnie.tuhs.org/pipermail/tuhs/2022-September/026506.html) Cartron asks what FreeBSD machines I use (https://rubenerd.com/cartron-asks-what-freebsd-machines-i-use/) My new debugbreak command (https://nullprogram.com/blog/2022/07/31/) 7 sudo myths debunked (https://opensource.com/article/22/8/debunk-sudo-myths) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Andy - sharing and acls (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/478/feedback/Andy%20-%20sharing%20and%20acls.md) Reptilicus Rex - boot environments (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/478/feedback/Reptilicus%20Rex%20-%20boot%20environments.md) i3luefire - byhve issue (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/478/feedback/i3luefire%20-%20byhve%20issue.md) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Writing FreeBSD kernel modules in Rust, Details behind the FreeBSD aio LPE, Linux subsystem for FreeBSD, FreeBSD Journal: Science, Systems, and FreeBSD, NetBSD improves Amiga support, OpenBSD on Scaleway Elastic Metal, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Writing FreeBSD Kernel modules in Rust (https://research.nccgroup.com/2022/08/31/writing-freebsd-kernel-modules-in-rust/) Details behind the FreeBSD aio LPE (https://accessvector.net/2022/freebsd-aio-lpe) News Roundup Linux Subsystem for FreeBSD (https://medium.com/nttlabs/linux-subsystem-for-freebsd-500b9a88fda4) FreeBSD Journal: Science, Systems, and FreeBSD (https://freebsdfoundation.org/wp-content/uploads/2022/08/03ae2705ab4362602a6bb90c5b9628c595d8b4fa.2.pdf) NetBSD improves its support for the Commodore Amiga (https://thenewstrace.com/netbsd-an-operating-system-that-is-serious-about-being-cross-platform-now-improves-its-support-for-the-commodore-amiga-1985/243892/) Installing OpenBSD on Scaleway Elastic Metal (https://www.senzilla.io/blog/2022/08/10/installing-openbsd-scaleway-elastic-metal/) Beastie Bits /usr/games removed from the default $PATH (http://undeadly.org/cgi?action=article;sid=20220810120423) How to install and configure mDNSResponder (https://forums.FreeBSD.org/threads/how-to-install-and-configure-mdnsresponder.70713/) How to use consistent exit codes in shell scripts (https://sleeplessbeastie.eu/2022/08/12/how-to-use-consistent-exit-codes-in-shell-scripts) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions [TheHolm - zfs question)[https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/469/feedback/TheHolm%20-%20zfs%20question.md] *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD on the Framework Laptop, Win32 is the only stable ABI on Linux, why OpenBSD's documentation is so good, configure dma for mail delivery in jails on internet hosts, introducing muxfs, RAID1C boot support, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD on the Framework laptop (https://xyinn.org/md/freebsd/framework_laptop) Win32 is the only stable ABI on Linux (https://blog.hiler.eu/win32-the-only-stable-abi/) News Roundup Why is the OpenBSD documentation so good? (https://dataswamp.org/~solene/2022-08-18-why-openbsd-documentation-is-good.html) How I configure dma for mail delivery in jails on my internet hosts (https://dan.langille.org/2022/08/15/how-i-configure-dma-for-mail-delivery-in-jails-on-my-internet-hosts/) Introducing muxfs (https://sdadams.org/blog/introducing-muxfs/) RAID 1C boot support added (http://undeadly.org/cgi?action=article;sid=20220813110021) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions [Oliver - shell tip)[https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/469/feedback/Oliver%20-%20shell%20tip.md] Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD Q2 2022 Status Report, FreeBSD in Science, fastest yes(1) in the west, Why Programmers Can't "Reset" Programs With Ctrl-C, Run Slack in FreeBSD's Linuxulator, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Q2 2022 Status Report (https://www.freebsd.org/status/report-2022-04-2022-06/) FreeBSD in Science (https://freebsdfoundation.org/blog/guest-post-freebsd-in-science/) News Roundup Fastest yes(1) in the west (https://codegolf.stackexchange.com/questions/199528/fastest-yes-in-the-west/199622#199622) Ctrl-C: Why Programmers Can't "Reset" Programs With Ctrl-C, but Used to Be Able To, and Why They Should Be Able to Again (https://kevinlawler.com/ctrl-c) Run Slack in FreeBSD's Linuxulator (https://meka.rs/blog/2022/07/01/freebsd-linuxulator/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Advocating for FreeBSD in 2022 and Beyond, NetBSD 9.3 released, OPNsense 22.7 available, CHERI-based computer runs KDE for the first time, Run FreeBSD 13.1-RELEASE for ARM64 in QEMU on Apple Silicon Mac, and more Notes This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Advocating for FreeBSD in 2022 and Beyond (https://freebsdfoundation.org/blog/advocating-for-freebsd-in-2022-and-beyond/) NetBSD 9.3 released (http://blog.netbsd.org/tnf/entry/netbsd_9_3_released) News Roundup OPNsense 22.7 released (https://forum.opnsense.org/index.php?topic=29507.0) CHERI-based computer runs KDE for the first time (https://www.theregister.com/2022/07/26/cheri_computer_runs_kde/) Guide: Run FreeBSD 13.1-RELEASE for ARM64 in QEMU on Apple Silicon Mac (https://gist.github.com/ctsrc/a1f57933a2cde9abc0f07be12889f97f) Beastie Bits • [In -current, dhclient(8) now just logs warnings and executes ifconfig(8)](http://undeadly.org/cgi?action=article;sid=20220703114819) • [Freshly installed #NetBSD 4.0.1 booting on a 80386 DX40 with 8MB of RAM in 2022](https://twitter.com/lefinnois/status/1553246084675375104) • [nerdctl](https://twitter.com/woodsb02/status/1554481441060560898?s=28&t=8K7_A1RiWnCDU_Mme4_Yqw) • [Even more Randomness](https://undeadly.org/cgi?action=article;sid=20220731110742) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
Debugging Lisp in Deep Space, 0 Dependency Websites with OpenBSD & AsciiDoc, Deleting old snapshots on FreeBSD, Full multiprocess support in lldb-server, Basic fix between pf tables and macros, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines NASA Programmer Remembers Debugging Lisp in Deep Space (https://thenewstack.io/nasa-programmer-remembers-debugging-lisp-in-deep-space/) 0 Dependency Websites with OpenBSD & AsciiDoc (https://blog.passwordclass.xyz/blogs/2022/06/0-dependency-websites-with-openbsd-asciidoc.html) News Roundup FreeBSD - Deleting old snapshots (https://www.jan0sch.de/post/deleting-old-zfs-snapshots/) Full multiprocess support in lldb-server (https://www.moritz.systems/blog/full-multiprocess-support-in-lldb-server/) Basic fix between pf tables and macros on FreeBSD (https://rubenerd.com/basic-fix-between-pf-tables-and-macros-on-freebsd/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Ben - Jail Question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/464/feedback/Ben%20-%20Jail%20Question.md) Malcolm - encryption (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/464/feedback/Malcolm%20-%20encryption.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
From 0 to bhyve on FreeBSD, Analyze OpenBSD's Kernel with Domain-Specific Knowledge, OpenBSD Webzine: ISSUE #10, HardenedBSD June 2022 Status Report, two new C compilers: chibicc and kefir in OpenBSD, SSD TRIM in NetBSD HEAD, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines From 0 to Bhyve on FreeBSD 13.1 (https://klarasystems.com/articles/from-0-to-bhyve-on-freebsd-13-1/) Analyze OpenBSD's Kernel with Domain-Specific Knowledge (https://medium.com/@chrissicool/analyze-openbsds-kernel-with-domain-specific-knowledge-ca665d92eebb) News Roundup OpenBSD Webzine: ISSUE #10 (https://webzine.puffy.cafe/issue-10.html) HardenedBSD June 2022 Status Report (https://hardenedbsd.org/article/shawn-webb/2022-06-28/hardenedbsd-june-2022-status-report) OpenBSD has two new C compilers: chibicc and kefir (https://briancallahan.net/blog/20220629.html) SSD TRIM in NetBSD HEAD (-current) (https://www.unitedbsd.com/d/859-ssd-trim-in-netbsd-head-current) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Quarterly Status Report First Quarter 2022 (https://www.freebsd.org/status/report-2022-01-2022-03/) Installing Nginx on OpenBSD 7.1 (https://unixcop.com/installing-nginx-on-openbsd-7-1/) News Roundup Live Webinar: Open-source Virtualization: Getting started with bhyve (https://klarasystems.com/webinars/live-sessions-singup/webinar-open-source-virtualization-getting-started-with-bhyve/) Hosted by Jim Salter and Allan Jude Live July 12th at 13:00 ET Available on-demand a few days later Persistent Memory Allocation (https://queue.acm.org/detail.cfm?id=3534855) Colorize your BSD shell (https://forums.FreeBSD.org/threads/colorize-your-bsd-shell.85458/) How to Install cgit With Gitolite and Nginx on FreeBSD 13 (https://herrbischoff.com/2021/10/how-to-install-cgit-with-gitolite-and-nginx-on-freebsd-13) EuroBSDCon 2022 (Austria) Program announced (https://2022.eurobsdcon.org/program/) Come to Austria and learn about the latest happenings in the BSDs 2 days of tutorials, and 2 days of 3 concurrent tracks of talks Registration is open now. See you there! *** Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Brad - Drive question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Brad%20-%20Drive%20question.md) Carl - Wiring question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Carl%20-%20Wiring%20question.md) Jon - Jails question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Jon%20-%20Jails%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Containerd gains support for launching Linux containers on FreeBSD, OpenBSD 7.1 on PINE64 RockPro64, true minimalistic window manager does not exist, OpenBSD folklore, HardenedBSD May 2022 Status Report, DragonFlyBSD 6.2.2 out, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Containerd gains support for launching Linux containers on FreeBSD (https://github.com/containerd/containerd/pull/7000) Uses Linux compat and the Linux Jails concept to deploy a full Linux container userland on FreeBSD OpenBSD 7.1 on PINE64 RockPro64 (https://bsandro.tech/posts/openbsd-7.1-on-pine64-rockpro64/) News Roundup Live Webinar: Open-source Virtualization: Getting started with bhyve (https://klarasystems.com/webinars/webinar---open-source-virtualization---getting-started-with-bhyve/) Hosted by Jim Salter and Allan Jude Live July 12th at 13:00 ET Available on-demand a few days later The True Minimalistic Window Manager Does Not Exist (https://serhanekici.com/ttmwm.html) OpenBSD folklore and share/misc/airport (https://www.cambus.net/openbsd-folklore-and-share-misc-airport/) HardenedBSD May 2022 Status Report (https://hardenedbsd.org/article/shawn-webb/2022-06-01/hardenedbsd-may-2022-status-report) DragonFlyBSD 6.2.2 out (https://www.dragonflydigest.com/2022/06/10/27047.html) Changelog (https://lists.dragonflybsd.org/pipermail/commits/2022-June/820953.html) *** Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Norbert - question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/460/feedback/Norbert%20-%20question.md) Paulo - network question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/460/feedback/Paulo%20-%20network%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Journey to ZFS RAIDZ1 on NetBSD, FreeBSD networking basics: WiFi and Bluetooth, smuggling code into the playstation via NetBSD driver hole, KDE FreeBSD CI, remembering buildtool, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines The journey to ZFS raidz1 with different sized disks (On NetBSD) (Wheelbarrow optional) (http://netbsd0.blogspot.com/2022/05/the-journey-to-zfs-raidz1-with.html) FreeBSD Networking Basics: WiFi and Bluetooth (https://freebsdfoundation.org/freebsd-project/resources/networking-basics-wifi-and-bluetooth/) News Roundup Playstation: Hole in NetBSD driver could allow code smuggling (https://www.kiratas.com/playstation-hole-in-netbsd-driver-could-allow-code-smuggling-2/) Archive link if the page is down (no images) (https://web.archive.org/web/20220519162432/https://www.kiratas.com/playstation-hole-in-netbsd-driver-could-allow-code-smuggling-2/) Original Announcment (https://hackerone.com/reports/1350653) German Article (https://www.heise.de/news/Playstation-Luecke-in-NetBSD-Treiber-koennte-Codeschmuggel-ermoeglichen-7091153.html) KDE-FreeBSD CI (https://euroquis.nl//kde/2022/04/26/freebsd-ci.html) Remembering Buildtool (https://jmmv.dev/2022/05/remembering-buildtool.html) Beastie Bits By the Way... Kubernetes for FreeBSD (https://medium.com/@norlin.t/by-the-way-kubernetes-for-freebsd-d0ba4dab8d8e) FreeBSD Games Directory (https://github.com/tigersharke/FreeBSD-Games-Directory) Candlelit Console patch set to the framebuffer console (http://undeadly.org/cgi?action=article;sid=20220516093712) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dan - A couple things (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/457/feedback/Dan%20-%20A%20couple%20things.md) Paul - BSD Business Justifications (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/457/feedback/Paul%20-%20BSD%20Business%20Justifications.md) Todd - Feedback to prior feedback (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/457/feedback/Todd%20-%20Feedback%20to%20prior%20feedback.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
FreeBSD 13.1 is released, Unix command line conventions over time, Branching for NetBSD 10, Microbhyve, Own your Calendar and Contacts with OpenBSD, the PSARC case for ZFS, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD 13.1 Release is available (https://www.freebsd.org/releases/13.1R/announce/) Unix command line conventions over time (https://blog.liw.fi/posts/2022/05/07/unix-cli/) News Roundup Branching for NetBSD 10 (https://mail-index.netbsd.org/current-users/2022/05/02/msg042278.html) Microbyhve (https://github.com/cbsd/microbhyve) Own Your Calendar & Contacts With OpenBSD, Baïkal, and FOSS Android (https://baak6.com/baikal-openbsd-fossdroid/) Twenty years ago today, Jeff filed the PSARC case for the ZFS filesystem (https://twitter.com/mmusante/status/1518947283626246145?t=tzR6KeMx2mhjJfeoOqrHIw&s=03) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Scott - FreeBSD and supercomputing (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/456/feedback/Scott%20-%20FreeBSD%20and%20supercomputing.md) Nick - Thanks and some shout outs (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/456/feedback/Nick%20-%20Thanks%20and%20some%20shout%20outs.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Building Your Own FreeBSD-based NAS, Writing a device driver for Unix V6, EC2: What Colin Percival's been up to, Beckhoff releases TwinCAT/BSD Hypervisor, Writing a NetBSD kernel module, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Building Your Own FreeBSD-based NAS (https://klarasystems.com/articles/building-your-own-freebsd-based-nas-with-zfs/) Writing a device driver for Unix V6 (https://mveg.es/posts/writing-a-device-driver-for-unix-v6/) News Roundup FreeBSD/EC2: What I've been up to (https://www.daemonology.net/blog/2022-03-29-FreeBSD-EC2-report.html) Beckhoff has released its TwinCAT/BSD Hypervisor (https://www.automationworld.com/control/article/22144694/beckhoff-hypervisor-enables-virtual-machines-for-control-applications) Writing a NetBSD kernel module (https://saurvs.github.io/post/writing-netbsd-kern-mod/) Benedicts Git Finds Projects Run anything (like full blown GTK apps) under Capsicum (https://github.com/unrelentingtech/capsicumizer) Twitter client for UEFI (https://github.com/arata-nvm/mitnal) n³ The unorthodox terminal file manager (https://github.com/jarun/nnn) OpenVi: Portable OpenBSD vi for UNIX systems (https://github.com/johnsonjh/OpenVi) Gists and Articles Step-by-step instructions on installing the latest NVIDIA drivers on FreeBSD 13.0 and above (https://gist.github.com/Mostly-BSD/4d3cacc0ee2f045ed8505005fd664c6e) FreeBSD SSH Hardening (https://gist.github.com/koobs/e01cf8869484a095605404cd0051eb11) GTFOBins is a curated list of Unix binaries that can be used to bypass local security restrictions in misconfigured systems (https://gtfobins.github.io) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Ben - Backing Up (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/453/feedback/Ben%20-%20Backing%20Up.md) Ethan - Thanks (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/453/feedback/Ethan%20-%20Thanks.md) Maxi - question about note taking (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/453/feedback/Maxi%20%20-%20question%20about%20note%20taking.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
The unknown hackers, Papers we love to read, Dual Boot Homelab in The Bedroom by the bed testbed, OpenSSH 9.0 released, OS battle: OpenBSD vs. NixOS, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines The unknown hackers (https://www.salon.com/2000/05/17/386bsd/) Bill Jolitz passed away in March 2022 (https://minnie.tuhs.org/pipermail/tuhs/2022-April/025643.html) *** FreeBSD Documentation: Papers We Love To Read (https://klarasystems.com/articles/freebsd-documentation-papers-we-love-to-read/) News Roundup FreeBSD/Ubuntu Dual Boot Homelab in The Bedroom by the bed testbed (https://adventurist.me/posts/00307) OpenSSH 9.0 has been released (https://www.openssh.com/txt/release-9.0) Operating systems battle: OpenBSD vs NixOS (https://dataswamp.org/~solene/2022-04-18-openbsd-vs-nixos.html) Beastie Bits Celebrating 50 years of the Unix Operating System (https://www.reddit.com/r/BSD/comments/u4t25c/celebrating_50_years_of_the_unix_operating_system/) Kickstarter Campaign Results (https://mwl.io/archives/13627) FreeBSD Virtualization Series (https://productionwithscissors.run/freebsd-virtualization-series/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Jeff - ZFS checksum repair (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/452/feedback/Jeff%20-%20ZFS%20checksum%20repair.md) Nelson - General Thanks (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/452/feedback/Nelson%20-%20General%20Thanks.md) Sam - FOSS Power Support (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/452/feedback/Sam%20-%20FOSS%20Power%20Support.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD Status Report 4th Quarter 2021, Reproducible clean $HOME in OpenBSD using impermanence, Making RockPro64 a NetBSD Server, helloSystem 0.7.0 is out, lazy approach to FreeBSD dual-booting, going to jail, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Quarterly Status Report 4th Quarter 2021 (https://www.freebsd.org/status/report-2021-10-2021-12/) Reproducible clean $HOME in OpenBSD using impermanence (https://dataswamp.org/~solene/2022-03-15-openbsd-impermanence.html) News Roundup Making RockPro64 a NetBSD Server (https://blog.netbsd.org/tnf/entry/making_rockpro64_a_netbsd_server) helloSystem 0.7.0 is out (https://github.com/helloSystem/ISO/releases/tag/r0.7.0) My lazy approach to FreeBSD dual-booting (https://rubenerd.com/my-lazy-approach-to-freebsd-dual-booting/) Going to jail (https://opekkt.tech/docs/vps_migration/going2jail/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions • No Feedback emails this week, so instead we can have “Story Time with Allan” and he can regale us with an entertaining BSD story. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Controlling Resource Limits with rctl in FreeBSD, It's always DNS, Google Summer of Code in BSD Projects, Rsync Technical Notes - Q4 2021, Userland CPU frequency scheduling for OpenBSD, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Controlling Resource Limits with rctl in FreeBSD (https://klarasystems.com/articles/controlling-resource-limits-with-rctl-in-freebsd/) It's DNS. Of course it's DNS, it's always DNS. (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/DNSVariabilityProblems) News Roundup GSOC • [Work with FreeBSD in Google Summer of Code](https://freebsdfoundation.org/blog/work-with-freebsd-in-google-summer-of-code/) • [The NetBSD Foundation is a mentoring organization at Google Summer of Code 2022](https://blog.netbsd.org/tnf/entry/the_netbsd_foundation_is_a) Rsync Technical Notes - Q4 2021 (https://www.rsync.net/resources/notes/2021-q4-rsync.net_technotes.html) Userland CPU frequency scheduling for OpenBSD (https://tildegit.org/solene/obsdfreqd) Beastie Bits Unofficial HardenedBSD liveCD (https://groups.google.com/a/hardenedbsd.org/g/users/c/QUTUJfm30Dg/m/0VNKUeVhHgAJ) The eurobsdcon 2022 CFP is open (https://2022.eurobsdcon.org/the-call-for-talk-and-presentation-proposals-for-eurobsdcon-2022-is-now-open/) Testing parallel forwarding (http://undeadly.org/cgi?action=article;sid=20220319123157) OpenBSD iwx(4) gains 11ac 80MHz channel support (https://www.undeadly.org/cgi?action=article;sid=20220315070043) OpenBSD/arm64 on Apple M1 systems (https://undeadly.org/cgi?action=article;sid=20220320115932) FreeBSD on the CubieBoard2 (https://www.cambus.net/freebsd-on-the-cubieboard2/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Eric - periodic notifications (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/448/feedback/Eric%20-%20periodic%20notifications.md) Kevin - no question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/448/feedback/Kevin%20-%20no%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Idiot's guide to OpenBSD on the Pinebook Pro, FreeBSD Periodic Scripts, history of service management in Unix, journey from macOS to FreeBSD, Unix processes “infecting” each other, navidrom music server on FreeBSD, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines The complete idiot's guide to OpenBSD on the Pinebook Pro (https://tomscii.sig7.se/2022/02/Guide-to-OpenBSD-on-the-PinebookPro) FreeBSD Periodic Scripts (https://klarasystems.com/articles/freebsd-periodic-scripts/) News Roundup The history (sort of) of service management in Unix (https://utcc.utoronto.ca/~cks/space/blog/unix/ServiceManagementHistory) My journey from macOS to FreeBSD (https://www.boucek.me/blog/from-mac-to-freebsd/) A nice story about Unix processes "infecting" each other (https://rachelbythebay.com/w/2022/02/09/nice/) Navidrome music server on FreeBSD (https://web.archive.org/web/20220101220446/https://www.danschmid.me/article/install-navidrome-music-server-on-freebsd) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Tyler - Is this enough for VMs (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/444/feedback/Tyler%20-%20Is%20this%20enough%20for%20VMs.md) Kevin - BSD from RAMdisk (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/445/feedback/Kevin%20-%20BSD%20from%20RAMdisk.md) Malcolm - wired headset in FreeBSD (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/445/feedback/Malcolm%20-%20wired%20headset%20in%20FreeBSD.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol.
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol.
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol.
Looking at Lumina Desktop 2.0, 2 months of KPTI development in SmartOS, OpenBSD email service, an interview with Ryan Zezeski, NomadBSD released, and John Carmack's programming retreat with OpenBSD. This episode was brought to you by Headlines Looking at Lumina Desktop 2.0 (https://www.trueos.org/blog/looking-lumina-desktop-2-0/) A few weeks ago I sat down with Lead Developer Ken Moore of the TrueOS Project to get answers to some of the most frequently asked questions about Lumina Desktop from the open source community. Here is what he said on Lumina Desktop 2.0. Do you have a question for Ken and the rest of the team over at the TrueOS Project? Make sure to read the interview and comment below. We are glad to answer your questions! Ken: Lumina Desktop 2.0 is a significant overhaul compared to Lumina 1.x. Almost every single subsystem of the desktop has been streamlined, resulting in a nearly-total conversion in many important areas. With Lumina Desktop 2.0 we will finally achieve our long-term goal of turning Lumina into a complete, end-to-end management system for the graphical session and removing all the current runtime dependencies from Lumina 1.x (Fluxbox, xscreensaver, compton/xcompmgr). The functionality from those utilities is now provided by Lumina Desktop itself. Going along with the session management changes, we have compressed the entire desktop into a single, multi-threaded binary. This means that if any rogue script or tool starts trying to muck about with the memory used by the desktop (probably even more relevant now than when we started working on this), the entire desktop session will close/crash rather than allowing targeted application crashes to bypass the session security mechanisms. By the same token, this also prevents “man-in-the-middle” type of attacks because the desktop does not use any sort of external messaging system to communicate (looking at you dbus). This also gives a large performance boost to Lumina Desktop The entire system for how a user's settings get saved and loaded has been completely redone, making it a “layered” settings system which allows the default settings (Lumina) to get transparently replaced by system settings (OS/Distributor/SysAdmin) which can get replaced by individual user settings. This results in the actual changes in the user setting files to be kept to a minimum and allows for a smooth transition between updates to the OS or Desktop. This also provides the ability to “restrict” a user's desktop session (based on a system config file) to the default system settings and read-only user sessions for certain business applications. The entire graphical interface has been written in QML in order to fully-utilize hardware-based GPU acceleration with OpenGL while the backend logic and management systems are still written entirely in C++. This results in blazing fast performance on the backend systems (myriad multi-threaded C++ objects) as well as a smooth and responsive graphical interface with all the bells and whistles (drag and drop, compositing, shading, etc). Q: Are there future plans to implement something like Lumina in a MAC Jail? While I have never tried out Lumina in a MAC jail, I do not see anything on that page which should stop it from running in one right now. Lumina is already designed to be run as an unpriviledged user and is very smart about probing the system to find out what is/not available before showing anything to the user. The only thing that comes to mind is that you might need to open up some other system devices so that X11 itself can draw to the display (graphical environment setup is a bit different than CLI environment). Q: I look forward to these changes. I know the last time I used it when I would scroll I would get flashes like the refresh rate was not high enough. It will be nice to have a fast system as well as I know with the more changes Linux is becoming slower. Not once it has loaded but in the loading process. I will do another download when these changes come out and install again and maybe stay this time. If I recall correctly, one of the very first versions of Lumina (pre-1.0) would occasionally flicker. If that is still happening, you might want to verify that you are using the proper video driver for your hardware and/or enable the compositor within the Lumina settings. Q: Why was enlightenment project not considered for TrueOS? It is BSD licensed and is written in C. This was a common question about 4(?) years ago with the first release of the Lumina desktop and it basically boiled down to long-term support and reliability of the underlying toolkit. Some of the things we had to consider were: cross-platform/cross-architecture support, dependency reliability and support framework (Qt5 > EFL), and runtime requirements and dependency tracking (Qt5 is lighter than the EFL). That plus the fact that the EFL specifically states that it is linux-focused and the BSD's are just an afterthought (especially at the time we were doing the evaluation). Q: I have two questions. 1) The default layout of Unity(menu bar with actual menu entries on top and icon dock on the side) is one of the few things I liked about my first voyage into non-Windows systems, and have been missing since moving on to other distros(and now also other non-Linux systems). However in 1.4.0 screenshots on Lumina's site, the OSX-like layout has the menu attached to the window. Will 2.0 be able to have the menus on the bar? 2) Is there any timeline for a public release, or are you taking a “when it's ready” approach? In Lumina you can already put panels on the left/right side of the screen and give you something like the layout of the Unity desktop. The embedded menu system is not available in Lumina because that is not a specification supported by X11 and the window manager standards at the present time. The way that functionality is currently run on Linux is a hacky-bypass of the display system which only really works with the GTK3 and Qt5 toolkits, resulting in very odd overall desktop behavior in mixed environments where some apps use other graphical toolkits. We are targetting the 18.06 STABLE release of TrueOS for Lumina 2, but that is just a guideline and if necessary we will push back the release date to allow for additional testing/fixing as needed. A long two months (https://blog.cooperi.net/a-long-two-months) IllumOS/SmartOS developer Alex Wilson describes the journey of developing KPTI for IllumOS > On Monday (January 1st) I had the day off work for New Year's day, as is usual in most of the western world, so I slept in late. Lou and her friend decided to go to the wax museum and see several tourist attractions around SF, and I decided to pass the day at home reading. That afternoon, work chat started talking about a Tumblr post by pythonsweetness about an Intel hardware security bug. At the time I definitely did not suspect that this was going to occupy most of my working life for the next (almost) two months. Like many people who work on system security, I had read Anders Fogh's post about a "Negative Result" in speculative execution research in July of 2017. At the time I thought it was an interesting writeup and I remember being glad that researchers were looking into this area. I sent the post to Bryan and asked him about his thoughts on it at the time, to which he replied saying that "it would be shocking if they left a way to directly leak out memory in the speculative execution". None of us seriously thought that there would be low-hanging fruit down that research path, but we also felt it was important that there was someone doing work in the area who was committed to public disclosure. At first, after reading the blog post on Monday, we thought (or hoped) that the bug might "just" be a KASLR bypass and wouldn't require a lot of urgency. We tried to reach out to Intel at work to get more information but were met with silence. (We wouldn't hear back from them until after the disclosure was already made public.) The speculation on Tuesday intensified, until finally on Wednesday morning I arrived at the office to find links to late Tuesday night tweets revealing exploits that allowed arbitrary kernel memory reads. Wednesday was not a happy day. Intel finally responded to our emails -- after they had already initiated public disclosure. We all spent a lot of time reading. An arbitrary kernel memory read (an info leak) is not that uncommon as far as bugs go, but for the most part they tend to be fairly easy to fix. The thing that makes the Meltdown and Spectre bugs particularly notable is that in order to mitigate them, a large amount of change is required in very deep low-level parts of the kernel. The kind of deep parts of the kernel where there are 20-year old errata workarounds that were single-line changes that you have to be very careful to not accidentally undo; the kind of parts where, as they say, mortals fear to tread. On Friday we saw the patches Matthew Dillon put together for DragonFlyBSD for the first time. These were the first patches for KPTI that were very straightforward to read and understand, and applied to a BSD-derived kernel that was similar to those I'm accustomed to working on. To mitigate Meltdown (and partially one of the Spectre variants), you have to make sure that speculative execution cannot reach any sensitive data from a user context. This basically means that the pages the kernel uses for anything potentially sensitive have to be unmapped when we are running user code. Traditionally, CPUs that were built to run a multi-user, UNIX-like OS did this by default (SPARC is an example of such a CPU which has completely separate address spaces for the kernel and userland). However, x86 descends from a single-address-space microcontroller that has grown up avoiding backwards-incompatible changes, and has never really introduced a clean notion of multiple address spaces (segmentation is the closest feature really, and it was thrown out for 64-bit AMD64). Instead, operating systems for x86 have generally wound up (at least in the post-AMD64 era) with flat address space models where the kernel text and data is always present in the page table no matter whether you're in user or kernel mode. The kernel mappings simply have the "supervisor" bit set on them so that user code can't directly access them. The mitigation is basically to stop doing this: to stop mapping the kernel text, data and other memory into the page table while we're running in userland. Unfortunately, the x86 design does not make this easy. In order to be able to take interrupts or traps, the CPU has to have a number of structures mapped in the current page table at all times. There is also no ability to tell an x86 CPU that you want it to switch page tables when an interrupt occurs. So, the code that we jump to when we take an interrupt, as well as space for a stack to push context onto have to be available in both page tables. And finally, of course, we need to be able to figure out somehow what the other page table we should switch to is when we enter the kernel. When we looked at the patches for Linux (and also the DragonFlyBSD patches at the time) on Friday and started asking questions, it became pretty evident that the initial work done by both was done under time constraints. Both had left the full kernel text mapped in both page tables, and the Linux trampoline design seemed over-complex. I started talking over some ideas with Robert Mustacchi about ways to fix these and who we should talk to, and reached out to some of my old workmates from the University of Queensland who were involved with OpenBSD. It seemed to me that the OpenBSD developers would care about these issues even more than we did, and would want to work out how to do the mitigation right. I ended up sending an email to Philip Guenther on Friday afternoon, and on Saturday morning I drove an hour or so to meet up with him for coffee to talk page tables and interrupt trampolines. We wound up spending a good 6 hours at the coffee shop, and I came back with several pages of notes and a half-decent idea of the shape of the work to come. One detail we missed that day was the interaction of per-CPU structures with per-process page tables. Much of the interrupt trampoline work is most easily done by using per-CPU structures in memory (and you definitely want a per-CPU stack!). If you combine that with per-process page tables, however, you have a problem: if you leave all the per-CPU areas mapped in all the processes, you will leak information (via Meltdown) about the state of one process to a different one when taking interrupts. In particular, you will leak things like %rip, which ruins all the work being done with PIE and ASLR pretty quickly. So, there are two options: you can either allocate the per-CPU structures per-process (so you end up with $NCPUS * $NPROCS of them); or you can make the page tables per-CPU. OpenBSD, like Linux and the other implementations so far, decided to go down the road of per-CPU per-process pages to solve this issue. For illumos, we took the other route. In illumos, it turned out that we already had per-CPU page tables. Robert and I re-discovered this on the Sunday of that week. We use them for 32-bit processes due to having full P>V PAE support in our kernel (which is, as it turns out, relatively uncommon amongst open-source OS). The logic to deal with creating and managing them and updating them was all already written, and after reading the code we concluded we could basically make a few small changes and re-use all of it. So we did. By the end of that second week, we had a prototype that could get to userland. But, when working on this kind of kernel change we have a rule of thumb we use: after the first 70% of the patch is done and we can boot again, now it's time for the second 70%. In fact it turned out to be more like the second 200% for us -- a tedious long tail of bugs to solve that ended up necessitating some changes in the design as well. At first we borrowed the method that Matt Dillon used for DragonFlyBSD, by putting the temporary "stack" space and state data for the interrupt trampolines into an extra page tacked onto the end of *%gs (in illumos the structure that lives there is the cpu_t). If you read the existing logic in interrupt handlers for dealing with %gs though, you will quickly notice that the corner cases start to build up. There are a bunch of situations where the kernel temporarily alters %gs, and some of the ways to mess it up have security consequences that end up being worse than the bug we're trying to fix. As it turns out, there are no less than 3 different ways that ISRs use to try to get to having the right cpu_t in %gs on illumos, as it turns out, and they are all subtly different. Trying to tell which you should use when requires a bunch of test logic that in turn requires branches and changes to the CPU state, which is difficult to do in a trampoline where you're trying to avoid altering that state as much as possible until you've got the real stack online to push things into. I kept in touch with Philip Guenther and Mike Larkin from the OpenBSD project throughout the weeks that followed. In one of the discussions we had, we talked about the NMI/MCE handlers and the fact that their handling currently on OpenBSD neglected some nasty corner-cases around interrupting an existing trap handler. A big part of the solution to those issues was to use a feature called IST, which allows you to unconditionally change stacks when you take an interrupt. Traditionally, x86 only changes the stack pointer (%rsp on AMD64) while taking an interrupt when there is a privilege level change. If you take an interrupt while already in the kernel, the CPU does not change the stack pointer, and simply pushes the interrupt stack frame onto the stack you're already using. IST makes the change of stack pointer unconditional. If used unwisely, this is a bad idea: if you stay on that stack and turn interrupts back on, you could take another interrupt and clobber the frame you're already in. However, in it I saw a possible way to simplify the KPTI trampoline logic and avoid having to deal with %gs. A few weeks into the project, John Levon joined us at work. He had previously worked on a bunch of Xen-related stuff as well as other parts of the kernel very close to where we were, so he quickly got up to speed with the KPTI work as well. He and I drafted out a "crazy idea" on the whiteboard one afternoon where we would use IST for all interrupts on the system, and put the "stack" they used in the KPTI page on the end of the cpu_t. Then, they could easily use stack-relative addresses to get the page table to change to, then pivot their stack to the real kernel stack memory, and throw away (almost) all the conditional logic. A few days later, we had convinced each other that this was the way to go. Two of the most annoying x86 issues we had to work around were related to the SYSENTER instruction. This instruction is used to make "fast" system calls in 32-bit userland. It has a couple of unfortunate properties: firstly, it doesn't save or restore RFLAGS, so the kernel code has to take care of this (and be very careful not to clobber any of it before saving or after restoring it). Secondly, if you execute SYSENTER with the TF ("trap"/single-step flag) set by a debugger, the resulting debug trap's frame points at kernel code instead of the user code where it actually happened. The first one requires some careful gymnastics on the entry and return trampolines specifically for SYSENTER, while the second is a nasty case that is incidentally made easier by using IST. With IST, we can simply make the debug trap trampoline check for whether we took the trap in another trampoline's code, and reset %cr3 and the destination stack. This works for single-stepping into any of the handlers, not just the one for SYSENTER. To make debugging easier, we decided that traps like the debug/single-step trap (as well as faults like page faults, #GP, etc.) would push their interrupt frame in a different part of the KPTI state page to normal interrupts. We applied this change to all the traps that can interrupt another trampoline (based on the instructions we used). These "paranoid" traps also set a flag in the KPTI struct to mark it busy (and jump to the double-fault handler if it is), to work around some bugs where double-faults are not correctly generated. It's been a long and busy two months, with lots of time spent building, testing, and validating the code. We've run it on as many kinds of machines as we could get our hands on, to try to make sure we catch issues. The time we've spent on this has been validated several times in the process by finding bugs that could have been nasty in production. One great example: our patches on Westmere-EP Xeons were causing busy machines to throw a lot of L0 I-cache parity errors. This seemed very mysterious at first, and it took us a few times seeing it to believe that it was actually our fault. This was actually caused by the accidental activation of a CPU errata for Westmere (B52, "Memory Aliasing of Code Pages May Cause Unpredictable System Behaviour") -- it turned out we had made a typo and put the "cacheable" flag into a variable named flags instead of attrs where it belonged when setting up the page tables. This was causing performance degradation on other machines, but on Westmere it causes cache parity errors as well. This is a great example of the surprising consequences that small mistakes in this kind of code can end up having. In the end, I'm glad that that erratum existed, otherwise it may have been a long time before we caught that bug. As of this week, Mike and Philip have committed the OpenBSD patches for KPTI to their repository, and the patches for illumos are out for review. It's a nice kind of symmetry that the two projects who started on the work together after the public disclosure at the same time are both almost ready to ship at the same time at the other end. I'm feeling hopeful, and looking forward to further future collaborations like this with our cousins, the BSDs. The IllumOS work has since landed, on March 12th (https://github.com/joyent/illumos-joyent/commit/d85fbfe15cf9925f83722b6d62da49d549af615c) *** OpenBSD Email Service (https://github.com/vedetta-com/caesonia) Features Efficient: configured to run on min. 512MB RAM and 20GB SSD, a KVM (cloud) VPS for around $2.50/mo 15GB+ uncompressed Maildir, rivals top free-email providers (grow by upgrading SSD) Email messages are gzip compressed, at least 1/3 more space with level 6 default Server side full text search (headers and body) can be enabled (to use the extra space) Mobile data friendly: IMAPS connections are compressed Subaddress (+tag) support, to filter and monitor email addresses Virtual domains, aliases, and credentials in files, Berkeley DB, or SQLite3 Naive Bayes rspamd filtering with supervised learning: the lowest false positive spam detection rates Carefree automated Spam/ and Trash/ cleaning service (default: older than 30 days) Automated quota management, gently assists when over quota Easy backup MX setup: using the same configuration, install in minutes on a different host Worry-free automated master/master replication with backup MX, prevents accidental loss of email messages Resilient: the backup MX can be used as primary, even when the primary is not down, both perfect replicas Flexible: switching roles is easy, making the process of changing VPS hosts a breeze (no downtime) DMARC (with DKIM and SPF) email-validation system, to detect and prevent email spoofing Daily (spartan) stats, to keep track of things Your sieve scripts and managesieve configuration, let's get started Considerations By design, email message headers need to be public, for exchanges to happen. The body of the message can be encrypted by the user, if desired. Moreover, there is no way to prevent the host from having access to the virtual machine. Therefore, full disk encryption (at rest) may not be necessary. Given our low memory requirements, and the single-purpose concept of email service, Roundcube or other web-based IMAP email clients should be on a different VPS. Antivirus software users (usually) have the service running on their devices. ClamAV can easily be incorporated into this configuration, if affected by the types of malware it protects against, but will require around 1GB additional RAM (or another VPS). Every email message is important, if properly delivered, for Bayes classification. At least 200 ham and 200 spam messages are required to learn what one considers junk. By default (change to use case), a rspamd score above 50% will send the message to Spam/. Moving messages in and out of Spam/ changes this score. After 95%, the message is flagged as "seen" and can be safely ignored. Spamd is effective at greylisting and stopping high volume spam, if it becomes a problem. It will be an option when IPv6 is supported, along with bgp-spamd. System mail is delivered to an alias mapped to a virtual user served by the service. This way, messages are guaranteed to be delivered via encrypted connection. It is not possible for real users to alias, nor mail an external mail address with the default configuration. e.g. puffy@mercury.example.com is wheel, with an alias mapped to (virtual) puffy@example.com, and user (puffy) can be different for each. Interview - Ryan Zezeski - rpz@joyent.com (mailto:rpz@joyent.com) / @rzezeski (https://twitter.com/rzezeski) News Roundup John Carmack's programming retreat to hermit coding with OpenBSD (https://www.facebook.com/permalink.php?story_fbid=2110408722526967&id=100006735798590) After a several year gap, I finally took another week-long programming retreat, where I could work in hermit mode, away from the normal press of work. My wife has been generously offering it to me the last few years, but I'm generally bad at taking vacations from work. As a change of pace from my current Oculus work, I wanted to write some from-scratch-in-C++ neural network implementations, and I wanted to do it with a strictly base OpenBSD system. Someone remarked that is a pretty random pairing, but it worked out ok. Despite not having actually used it, I have always been fond of the idea of OpenBSD — a relatively minimal and opinionated system with a cohesive vision and an emphasis on quality and craftsmanship. Linux is a lot of things, but cohesive isn't one of them. I'm not a Unix geek. I get around ok, but I am most comfortable developing in Visual Studio on Windows. I thought a week of full immersion work in the old school Unix style would be interesting, even if it meant working at a slower pace. It was sort of an adventure in retro computing — this was fvwm and vi. Not vim, actual BSD vi. In the end, I didn't really explore the system all that much, with 95% of my time in just the basic vi / make / gdb operations. I appreciated the good man pages, as I tried to do everything within the self contained system, without resorting to internet searches. Seeing references to 30+ year old things like Tektronix terminals was amusing. I was a little surprised that the C++ support wasn't very good. G++ didn't support C++11, and LLVM C++ didn't play nicely with gdb. Gdb crashed on me a lot as well, I suspect due to C++ issues. I know you can get more recent versions through ports, but I stuck with using the base system. In hindsight, I should have just gone full retro and done everything in ANSI C. I do have plenty of days where, like many older programmers, I think “Maybe C++ isn't as much of a net positive as we assume...”. There is still much that I like, but it isn't a hardship for me to build small projects in plain C. Maybe next time I do this I will try to go full emacs, another major culture that I don't have much exposure to. I have a decent overview understanding of most machine learning algorithms, and I have done some linear classifier and decision tree work, but for some reason I have avoided neural networks. On some level, I suspect that Deep Learning being so trendy tweaked a little bit of contrarian in me, and I still have a little bit of a reflexive bias against “throw everything at the NN and let it sort it out!” In the spirit of my retro theme, I had printed out several of Yann LeCun's old papers and was considering doing everything completely off line, as if I was actually in a mountain cabin somewhere, but I wound up watching a lot of the Stanford CS231N lectures on YouTube, and found them really valuable. Watching lecture videos is something that I very rarely do — it is normally hard for me to feel the time is justified, but on retreat it was great! I don't think I have anything particularly insightful to add about neural networks, but it was a very productive week for me, solidifying “book knowledge” into real experience. I used a common pattern for me: get first results with hacky code, then write a brand new and clean implementation with the lessons learned, so they both exist and can be cross checked. I initially got backprop wrong both times, comparison with numerical differentiation was critical! It is interesting that things still train even when various parts are pretty wrong — as long as the sign is right most of the time, progress is often made. I was pretty happy with my multi-layer neural net code; it wound up in a form that I can just drop it into future efforts. Yes, for anything serious I should use an established library, but there are a lot of times when just having a single .cpp and .h file that you wrote ever line of is convenient. My conv net code just got to the hacky but working phase, I could have used another day or two to make a clean and flexible implementation. One thing I found interesting was that when testing on MNIST with my initial NN before adding any convolutions, I was getting significantly better results than the non-convolutional NN reported for comparison in LeCun ‘98 — right around 2% error on the test set with a single 100 node hidden layer, versus 3% for both wider and deeper nets back then. I attribute this to the modern best practices —ReLU, Softmax, and better initialization. This is one of the most fascinating things about NN work — it is all so simple, and the breakthrough advances are often things that can be expressed with just a few lines of code. It feels like there are some similarities with ray tracing in the graphics world, where you can implement a physically based light transport ray tracer quite quickly, and produce state of the art images if you have the data and enough runtime patience. I got a much better gut-level understanding of overtraining / generalization / regularization by exploring a bunch of training parameters. On the last night before I had to head home, I froze the architecture and just played with hyperparameters. “Training!” Is definitely worse than “Compiling!” for staying focused. Now I get to keep my eyes open for a work opportunity to use the new skills! I am dreading what my email and workspace are going to look like when I get into the office tomorrow. Stack-register Checking (https://undeadly.org/cgi?action=article;sid=20180310000858) Recently, Theo de Raadt (deraadt@) described a new type of mitigation he has been working on together with Stefan Kempf (stefan@): How about we add another new permission! This is not a hardware permission, but a software permission. It is opportunistically enforced by the kernel. The permission is MAP_STACK. If you want to use memory as a stack, you must mmap it with that flag bit. The kernel does so automatically for the stack region of a process's stack. Two other types of stack occur: thread stacks, and alternate signal stacks. Those are handled in clever ways. When a system call happens, we check if the stack-pointer register points to such a page. If it doesn't, the program is killed. We have tightened the ABI. You may no longer point your stack register at non-stack memory. You'll be killed. This checking code is MI, so it works for all platforms. For more detail, see Theo's original message (https://marc.info/?l=openbsd-tech&m=152035796722258&w=2). This is now available in snapshots, and people are finding the first problems in the ports tree already. So far, few issues have been uncovered, but as Theo points out, more testing is necessary: Fairly good results. A total of 4 problems have been found so far. go, SBCL, and two cases in src/regress which failed the new page-alignment requirement. The SBCL and go ones were found at buildtime, since they use themselves to complete build. But more page-alignment violations may be found in ports at runtime. This is something I worry about a bit. So please everyone out there can help: Use snapshots which contain the stack-check diff, update to new packages, and test all possible packages. Really need a lot of testing for this, so please help out. So, everybody, install the latest snapshot and try all your favorite ports. This is the time to report issues you find, so there is a good chance this additional security feature is present in 6.3 (and works with third party software from packages). NomadBSD 1.0 has been released (https://freeshell.de/~mk/projects/nomadbsd.html) NomadBSD is a live system for flash drives, based on FreeBSD® 11.1 (amd64) Change Log The setup process has been improved. Support for optional geli encryption of the home partition has been added Auto-detection of NVIDIA graphics cards and their corresponding driver has been added. (Thanks to holgerw and lme from BSDForen.de) An rc script to start the GEOM disk scheduler on the root device has been added. More software has been added: accessibility/redshift (starts automatically) audio/cantata audio/musicpd audio/ncmpc ftp/filezilla games/bsdtris mail/neomutt math/galculator net-p2p/transmission-qt5 security/fpm2 sysutils/bsdstats x11/metalock x11/xbindkeys Several smaller improvements and bugfixes. Screenshots https://freeshell.de/~mk/projects/nomadbsd-ss1.png https://freeshell.de/~mk/projects/nomadbsd-ss2.png https://freeshell.de/~mk/projects/nomadbsd-ss3.png https://freeshell.de/~mk/projects/nomadbsd-ss4.png https://freeshell.de/~mk/projects/nomadbsd-ss5.png https://freeshell.de/~mk/projects/nomadbsd-ss6.png Beastie Bits KnoxBug - Nagios (http://knoxbug.org/2018-03-27) vBSDcon videos landing (https://www.youtube.com/playlist?list=PLfJr0tWo35bc9FG_reSki2S5S0G8imqB4) AsiaBSDCon 2017 videos (https://www.youtube.com/playlist?list=PLnTFqpZk5ebBTyXedudGm6CwedJGsE2Py) DragonFlyBSD Adds New "Ptr_Restrict" Security Option (https://www.phoronix.com/scan.php?page=news_item&px=DragonFlyBSD-Ptr-Restrict) A Dexter needs your help (https://twitter.com/michaeldexter/status/975603855407788032) Mike Larkin at bhyvecon 2018: OpenBSD vmm(4) update (https://undeadly.org/cgi?action=article;sid=20180309064801) [HEADS UP] - OFED/RDMA stack update (https://lists.freebsd.org/pipermail/freebsd-arch/2018-March/018900.html) *** Feedback/Questions Ron - Interview someone using DragonflyBSD (http://dpaste.com/3BM6GSW#wrap) Brad - Gaming and all (http://dpaste.com/3X4ZZK2#wrap) Mohammad - Sockets vs TCP (http://dpaste.com/0PJMKRD#wrap) Paul - All or at least most of Bryan Cantrill's Talks (http://dpaste.com/2WXVR1X#wrap) ***
AsiaBSDcon review, Meltdown and Spectre Patches in FreeBSD stable, Interview with MidnightBSD founder, 8 months with TrueOS, mysteries of GNU and BSD split This episode was brought to you by Headlines AsiaBSDCon 2018 has concluded (https://2018.asiabsdcon.org/) We have just returned from AsiaBSDCon in Tokyo, Japan last weekend Please excuse our jetlag The conference consisted two days of meeting followed by 2 days of paper presentations We arrived a few days early to see some sights and take a few extra delicious meals in Tokyo The first day of meetings was a FreeBSD developer summit (while Benedict was teaching his two tutorials) where we discussed the FreeBSD release cycle and our thoughts on improving it, the new Casper capsicum helper service, and developments in SDIO which will eventually enable WiFi and SD card readers on more embedded devices The second day of meetings consisted of bhyvecon, a miniconf that covered development in all hypervisors on all BSDs. It also included presentations on the porting of bhyve to IllumOS. Then the conference started There were a number of great presentations, plus an amazing hallway track as usual It was great to see many old friends and to spend time discussing the latest happenings in BSD. A couple of people came by and asked to take a picture with us and we were happy to do that. *** FreeBSD releases Spectre and Meltdown mitigations for 11.1 (https://www.freebsd.org/security/advisories/FreeBSD-SA-18:03.speculative_execution.asc) Speculative execution vulnerability mitigation is a work in progress. This advisory addresses the most significant issues for FreeBSD 11.1 on amd64 CPUs. We expect to update this advisory to include 10.x for amd64 CPUs. Future FreeBSD releases will address this issue on i386 and other CPUs. freebsd-update will include changes on i386 as part of this update due to common code changes shared between amd64 and i386, however it contains no functional changes for i386 (in particular, it does not mitigate the issue on i386). Many modern processors have implementation issues that allow unprivileged attackers to bypass user-kernel or inter-process memory access restrictions by exploiting speculative execution and shared resources (for example, caches). An attacker may be able to read secret data from the kernel or from a process when executing untrusted code (for example, in a web browser). + Meltdown: The mitigation is known as Page Table Isolation (PTI). PTI largely separates kernel and user mode page tables, so that even during speculative execution most of the kernel's data is unmapped and not accessible. A demonstration of the Meltdown vulnerability is available at https://github.com/dag-erling/meltdown. A positive result is definitive (that is, the vulnerability exists with certainty). A negative result indicates either that the CPU is not affected, or that the test is not capable of demonstrating the issue on the CPU (and may need to be modified). A patched kernel will automatically enable PTI on Intel CPUs. The status can be checked via the vm.pmap.pti sysctl PTI introduces a performance regression. The observed performance loss is significant in microbenchmarks of system call overhead, but is much smaller for many real workloads. + Spectre V2: There are two common mitigations for Spectre V2. This patch includes a mitigation using Indirect Branch Restricted Speculation, a feature available via a microcode update from processor manufacturers. The alternate mitigation, Retpoline, is a feature available in newer compilers. The feasibility of applying Retpoline to stable branches and/or releases is under investigation. The patch includes the IBRS mitigation for Spectre V2. To use the mitigation the system must have an updated microcode; with older microcode a patched kernel will function without the mitigation. IBRS can be disabled via the hw.ibrsdisable sysctl (and tunable), and the status can be checked via the hw.ibrsactive sysctl. IBRS may be enabled or disabled at runtime. Additional detail on microcode updates will follow. + Wiki tracking the vulnerabilities and mitigations on different platforms (https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities) Interview with MidnightBSD Founder and Lead Dev Lucas Holt (https://itsfoss.com/midnightbsd-founder-lucas-holt/) Recently, I have taken a little dip into the world of BSD. As part of my attempt to understand the BSD world a little better, I connected with Lucas Holt (MidnightBSD founder and lead developer) to ask him a few questions about his project. Here are his answers. It's FOSS: Please explain MidnightBSD in a nutshell. How is it different than other BSDs? Lucas Holt: MidnightBSD is a desktop focused operating system. When it's considered stable, it will provide a full desktop experience. This differs from other efforts such as TrueOS or GhostBSD in that it's not a distro of FreeBSD, but rather a fork. MidnightBSD has its own package manager, mport as well as unique package cluster software and several features built into user land such as mDNSresponder, libdispatch, and customizations throughout the system. It's FOSS: Who is MidnightBSD aimed at? Lucas Holt: The goal with MidnightBSD has always been to provide a desktop OS that's usable for everyday tasks and that even somewhat non technical people can use. Early versions of Mac OS X were certainly an inspiration. In practice, we're rather far from that goal at this point, but it's been an excellent learning opportunity. It's FOSS: What is your background in computers? Lucas Holt: I started in technical support at a small ISP and moved into web design and system administration. While there, I learned BSDi, Solaris and Linux. I also started tinkering with programming web apps in ASP and a little perl CGI. I then did a mix of programming and system administration jobs through college and graduated with a bachelors in C.S. from Eastern Michigan University. During that time, I learned NetBSD and FreeBSD. I started working on several projects such as porting Apple's HFS+ code to FreeBSD 6 and working on getting the nforce2 chipset SATA controller working with FreeBSD 6, with the latter getting committed. I got a real taste for BSD and after seeing the lack of interest in the community for desktop BSDs, I started MidnightBSD. I began work on it in late 2005. Currently, I'm a Senior Software Engineer focusing on backend rest services by day and a part-time graduate student at the University of Michigan Flint. It's FOSS: I recently installed TrueOS. I was disappointed that a couple of the programs I wanted were not available. The FreeBSD port system looked mildly complicated for beginners. I'm used to using pacman to get the job done quickly. How does MidnightBSD deal with ports? Lucas Holt: MidnightBSD has it's own port system, mports, which shared similarities with FreeBSD ports as well as some ideas from OpenBSD. We decided early on that decent package management was essential for regular users. Power users will still use ports for certain software, but it's just so time consuming to build everything. We started work on our own package manager, mport. Every package is a tar lzma archive with a sqlite3 manifest file as well as a sqlite 3 index that's downloaded from our server. This allows users to query and customize the package system with standard SQL queries. We're also building more user friendly graphical tools. Package availability is another issue that most BSDs have. Software tends to be written for one or two operating systems and many projects are reluctant to support other systems, particularly smaller projects like MidnightBSD. There are certainly gaps. All of the BSD projects need more volunteers to help with porting software and keeping it up to date. It's FOSS: During your June 2015 interview on BSDNow, you mentioned that even though you support both i386 and amd64, that you recommend people choose amd64. Do you have any plans to drop i386 support in the future, like many have done? Lucas Holt: Yes, we do plan to drop i386 support, mostly because of the extra work needed to build and maintain packages. I've held off on this so far because I had a lot of feedback from users in South America that they still needed it. For now, the plan is to keep i386 support through 1.0 release. That's probably a year or two out. It's FOSS: What desktop environments does MidnightBSD support? Lucas Holt: The original plan was to use Etoile as a desktop environment, but that project changed focus. We currently support Xfce, Gnome 3, WindowMaker + GNUstep + Gworkspace as primary choices. We also have several other window managers and desktop environments available such as Enlightenment, rat poison, afterstep, etc. Early versions offered KDE 3.x but we had some issues with KDE 4. We may revisit that with newer versions. It's FOSS: What is MidnightBSD's default filesystem? Do you support DragonflyBSD's HAMMER filesystem? What other filesystems? Lucas Holt: Boot volumes are UFS2. We also support ZFS for additional storage. We have read support for ExFat, NTFS, ext2, CD9660. NFS v3 and v4 are also supported for network file systems. We do not support HAMMER, although it was considered. I would love to see HAMMER2 get added to MidnightBSD eventually. It's FOSS: Is MidnightBSD affected by the recent Spectre and Meltdown issues? Lucas Holt: Yes. Most operating systems were affected by these issues. We were not informed of the issue until the general public became aware. Work is ongoing to come up with appropriate mitigations. Unfortunately, we do not have a patch yet. It's FOSS: The Raspberry Pi and its many clones have made the ARM platform very popular. Are there any plans to make MidnightBSD available on that platform? Lucas Holt: No immediate plans. ARM is an interesting architecture, but by the very nature of SoC designs, takes a lot of work to support a broad number of devices. It might be possible when we stop supporting i386 or if someone volunteers to work on the ARM port. Eventually, I think most hobby systems will need to run ARM chips. Intel's planning on locking down hardware with UEFI 3 and this may make it difficult to run on commodity hardware in the future not only for MidnightBSD but other systems as well. At one point, MidinightBSD ran on sparc64. When workstations were killed off, we dropped support. A desktop OS on a server platform makes little sense. It's FOSS: Does MidnightBSD offer support for Linux applications? Lucas Holt: Yes, we offer Linux emulation. It's emulating a 2.6.16 kernel currently and that needs to be updated so support newer apps. It's possible to run semi-recent versions of Firefox, Thunderbird, Java, and OpenOffice on it though. I've also used it to host game servers in the past and play older games such as Quake 3, enemy territory, etc. It's FOSS: Could you comment on the recent dust-up between the Pale Moon browser developers and the team behind the OpenBSD ports system? [Author's Note: For those who haven't heard about this, let me summarize. Last month, someone from the OpenBSD team added the Pale Moon browser to their ports collection. A Pale Moon developer demanded that they include Pale Moon's libraries instead of using system libraries. As the conversation continued, it got more hostile, especially on the Pale Moon side. The net result is that Pale Moon will not be available on OpenBSD, MidnightBSD, or FreeBSD.] Lucas Holt: I found this discussion frustrating. Many of the BSD projects hear a lot of complaints about browser availability and compatibility. With Firefox moving to Rust, it makes it even more difficult. Then you get into branding issues. Like Firefox, the Pale Moon developers have decided to protect their brand at the cost of users. Unlike the Firefox devs, they've made even stranger requirements for branding. It is not possible to use a system library version of anything with Pale Moon and keep their branding requirements. As such, we cannot offer Pale Moon in MidnightBSD. The reason this is an issue for an open source project is that many third party libraries are used in something as complex as a web browser. For instance, Gecko-based browsers use several multimedia libraries, sqlite3 (for bookmarks), audio and video codecs, etc. Trying to maintain upstream patches for each of these items is difficult. That's why the BSDs have ports collections to begin with. It allows us to track and manage custom patches to make all these libraries work. We go through a lot of effort in keeping these up to date. Sometimes upstream patches don't get included. That means our versions are the only working copies. With pale moon's policy, we'd need to submit separate patches to their customized versions of all these libraries too and any new release of the browser would not be available as changes occur. It might not even be possible to compile pale moon without a patch locally. With regard to Rust, it requires porting the language, as well as an appropriate version of LLVM before you can even start on the browser. It's FOSS: If someone wanted to contribute to your project, both financial and technical, how can they do that? Lucas Holt: Financial assistance for the project can be submitted online. We have a page outlining how to make donations with Patreon, Paypal or via bitcoin. Donations are not tax deductible. You can learn more at http://www.midnightbsd.org/donate/ We also need assistance with translations, porting applications, and working on the actual OS. Interested parties can contact us on the mailing list or through IRC on freenode #midnightbsd We also could use assistance with mirroring ISOs and packages. I would like to thank Lucas for taking the time to reply to my many questions. For more information about MidnightBSD or to download it, please visit their website. The most recent version of MidnightBSD is 0.8.6. News Roundup 8 months with TrueOS (https://inflo.ws/blog/post/2018-03-03-trueos-8th-month-review/) Purpose of this review - what it is and what it is not. I vowed to write down what I felt about TrueOS if I ever got to the six month mark of usage. This is just that. This is neither a tutorial, nor a piece of evangelism dedicated towards it. This is also not a review of specific parts of TrueOS such as Lumina or AppCafe, since I don't use them at all. In the spirit of presenting a screen shot, here is my i3wm displaying 4 windows in one screen - a configuration that I never use. https://inflo.ws/blog/images/trues-screenshot.png The primary tasks I get done with my computer. I need a tiling wm with multi-desktop capability. As regards what I do with a computer, it is fairly straightforward to describe if I just list down my most frequently used applications. xterm (CLI) Emacs (General editing and org mode) Intellij IDEA (Java, Kotlin, SQL) Firefox (Main web browser, with Multi-Account Containers) Thunderbird (Work e-mail) Notmuchmail (Personal e-mail) Chromium/Iridium (Dumb web browser) Telegram Desktop weechat (with wee-slack) cmus (Music player) mpv (Video player) mps-youtube (Youtube client) transmission-gtk Postgresql10 (daemon) Rabbitmq (daemon) Seafile (file sync) Shotwell (manage pictures) GIMP (Edit pictures) Calibre (Manage e-books) VirtualBox All of these are available as binary packages from the repository. Since I use Intellij Ultimate edition, I decided to download the no-jdk linux version from the website rather than install it. This would make sure that it gets updated regularly. Why did I pick TrueOS ? I ran various Linux distributions from 2001 all the way till 2009, till I discovered Arch, and continued with it till 2017. I tried out Void for two months before I switched to TrueOS. Over the last few years, I started feeling like no matter which Linux distribution I touched, they all just stopped making a lot of sense. Generally in the way things were organised, and particularly in terms of software like systemd, which just got pushed down my throat. I couldn't wrap my head around half the things going on in my computer. Mostly I found that Linux distributions stopped becoming a collection of applications that got developed together to something more coupled by software mechanisms like systemd - and that process was more and more opaque. I don't want to talk about the merits and de-merits of systemd, lets just say that I found it of no use and an unnecessary hassle. In February, I found myself in charge of the entire technology stack of a company, and I was free to make choices. A friend who was a long time FreeBSD user convinced me to try it on the servers. My requirement then was to run Postgres, Rabbitmq, Nginx and a couple of JVM processes. The setup was zero hassle and it hasn't changed much in a year. About three months of running FreeBSD-11.x on servers was enough for me to consider it for my laptop. I was very apprehensive of hardware support, but luckily my computer is a Thinkpad, and Thinkpads sort of work out of the box with various BSDs. My general requirements were: Must run Intellij IDEA. Must have proper graphics and sound driver support. Must be able to run VirtualBox. I had to pick from FreeBSD, NetBSD and OpenBSD, since these were the major BSDs that I was familiar with. One of my requirements was that I needed to be able to run VMs just in case I needed to test something on Windows/Linux. This ruled out OpenBSD. Then I was left with NetBSD and FreeBSD. NetBSD's driver support for newer Intel chip-sets were questionable, and FreeBSD was the only choice then. When I was digging through FreeBSD forums, I found out that running the 11.x RELEASE on my laptop was out of the question since it didn't have proper drivers for my chip-set either. A few more hours of digging led me to GhostBSD and TrueOS. I picked TrueOS straightaway because - well because TrueOS came from the old PC-BSD and it was built off FreeBSD-12-CURRENT with the latest drivers integrated. I downloaded the UNSTABLE version available in June 2017, backed up ALL my data and home directory, and then installed it. There were no glitches during installation - I simply followed the installation as described in the handbook and everything was fine. My entire switch from Arch/Void to TrueOS took about an hour, discounting the time it took to backup my data to an external hard disk. It was that easy. Everything I wanted to work just worked, everything was available in the repo. Tweaks from cooltrainer.org : I discovered this excellent tutorial that describes setting up a FreeBSD 11 desktop. It documents several useful tweaks, some of which I applied. A few examples - Fonts, VirtualBox, Firewall, UTF-8 sections. TrueOS (and FreeBSD) specific things I liked Open-rc The open-rc init system is familiar and is well documented. TrueOS specific parts are described here. When I installed postgresql10-server, there was no open-rc script for it, but I could cobble one together in two hours with zero prior experience writing init scripts. Later on I figured out that the init script for postgresql9 would work for 10 as well, and used that. Boot Environments This was an alien concept to me, but the first time I did an update without waiting for a CDN sync to finish, my computer booted into the shell and remained there. The friendly people at TrueOS discourse asked me to roll back to an older BE and wait for sync to finish. I dug through the forums and found "ZFS / Snapshots basics & How-To's for those new to TrueOS". This describes ZFS and BEs, and is well worth reading. ZFS My experience with boot environments was enough to convince me about the utility of ZFS. I am still reading about it and trying things out, and whatever I read just convinces me more about why it is good. File-system layout Coming from the Linux world, how the FreeBSD file-system is laid out seemed odd at first. Then I realised that it was the Linux distros that were doing the odd thing. e.g : The whole OS is split into base system and applications. All the non base system configurations and apps go into /usr/local. That made a lot of sense. The entire OS is developed along with its applications as a single coherent entity, and that shows. Documentation The handbooks for both TrueOS and FreeBSD are really really good. For e.g, I kept some files in an LUKS encrypted drive (when I used Arch Linux). To find an equivalent, all I had to do was read the handbook and look at the GELI section. It is actually nice being able to go to a source like Handbook and things from there just work. Arch Linux and Gentoo has excellent documentation as well, if anyone is wondering about Linux distros. Community The TrueOS community on both Telegram as well as on Discourse are very friendly and patient. They help out a lot and do not get upset when I pose really stupid questions. TrueOS core developers hangout in the Telegram chat-room too, and it is nice being able to talk to them directly about things. What did not work in TrueOS ? The following things that worked during my Linux tenure doesn't work in TrueOS. Netflix Google Hangouts Electron based applications (Slack, Skype) These are not major concerns for the kind of work I do, so it doesn't bother me much. I run a WinXP VM to play some old games, and a Bunsenlabs installation for Linux things like Hangouts/Netflix. I don't have a video calling system setup in TrueOS because I use my phone for both voice and video calls exclusively. Why am I staying on TrueOS ? Great community - whether on Discourse or on the telegram channel, the people make you feel welcome. If things go unanswered, someone will promise to work on it/file a bug/suggest work-arounds. Switching to TrueOS was philosophical as well - I thought a lot more about licenses, and I have arrived at the conclusion that I like BSD more than GPL. I believe it is a more practical license. I believe TrueOS is improving continuously, and is a great desktop UNIX if you put some time into it. AsiaBSDCon 2016 videos now available (https://www.youtube.com/playlist?list=PLnTFqpZk5ebD-FfVScL-x6ZnZSecMA1jI) The videos from AsiaBSDCon 2016 have been posted to youtube, 30 videos in all We'll cover the videos from 2017 next week The videos from 2018 should be posted in 4-6 weeks I are working on a new version of https://papers.freebsd.org/ that will make it easier to find the papers, slides, and videos of all talks related to FreeBSD *** syspatches will be provided for both supported releases (https://undeadly.org/cgi?action=article;sid=20180307234243) Good news for people doing upgrades only once per year: syspatches will be provided for both supported releases. The commit from T.J. Townsend (tj@) speaks for itself: ``` Subject: CVS: cvs.openbsd.org: www From: T.J. Townsend Date: 2018-03-06 22:09:12 CVSROOT: /cvs Module name: www Changes by: tj@cvs.openbsd.org 2018/03/06 15:09:12 Modified files: . : errata61.html stable.html faq : faq10.html Log message: syspatches will now be provided for both supported releases. ``` Thanks to all the developers involved in providing these! Update: An official announcement has been released: ``` I'm happy to announce that we are now able to provide two releases worth of syspatches on the amd64 and i386 platforms. The binary patches for 6.1 will hit the mirrors shortly, so you will be able to catch up with the errata on https://www.openbsd.org/errata61.html using the syspatch utility. People running amd64 will thus get the meltdown workaround. This means in particular that 6.2 will remain supported by syspatch when 6.3 comes out. Thanks to robert and ajacoutot for their amazing work on syspatch and for all their help. Thanks also to tj and the volunteers from #openbsd for their timely tests and of course to Theo for overseeing it all. ``` Exploring permutations and a mystery with BSD and GNU split filenames (https://www.lorainekv.com/permutations_split_and_gsplit/) Recently, I was playing around with the split command-line tool on Mac OS X, and I decided to chop a 4000-line file into 4000 separate single-line files. However, when I attempted to run split -l1, I ran into a funny error: split: too many files Curious to see if any splitting had occurred, I ran ls and sure enough, a huge list of filenames appeared, such as: xaa xab ... xzy xzz Now I could see why you'd run out of unique filenames - there are only 26 letters in the alphabet and these filenames were only three letters long. Also, they all seemed to begin with the letter "x". BSD split's filename defaults I checked the manual for split's defaults and confirmed what I was seeing: each file into which the file is split is named by the prefix followed by a lexically ordered suffix using suffix_length characters in the range 'a-z'. If -a is not specified, two letters are used as the suffix....with the prefix 'x' and with suffixes as above. Got it, so running split with the defaults for prefix name and suffix length will give me filenames that always start with the letter "x" followed by two-letter alphabetical permutations composed of a-z letters, with repeats allowed. I say "repeats allowed" because I noticed filenames such as xaa and xbb in the output. Side node: The reason why I say "permutations" rather than "combinations" is because letter order matters. For example, xab and xba are two distinct and legitimate filenames. Here's a nice explanation about the difference between permutations and combinations. Some permutation math So how many filenames can you get from the BSD split tool using the defaults? There are permutation formulas out there for repeating values and non-repeating values. Based on split's behavior, I wanted to use the repeating values formula: n^r where n equals the number of possible values (26 for a-z) and r equals the number of values (2, since there are only 2 letters after "x" in the filename). 26^2 = 676 So the total number of filename permutations allowed with BSD split's defaults should be 676. To double check, I ran ls | wc -l to get the total number of files in my split_test directory. The output was 677. If you subtract my original input file, input.txt, then you have 676, or the number of permutations split would allow before running out of filenames! Neat. But I still wanted my 4000 files. Moar permutations pls While 26^2 permutations doesn't support 4000 different filenames, I wondered if I could increase r to 3. Then, I'd have 17,576 different filename permutations to play with - more than enough. Earlier, I remembered the manual mentioning suffix length: -a suffixlength Use suffixlength letters to form the suffix of the file name. So I passed 3 in with the -a flag and guess what? I got my 4000 files! split -l1 -a3 input.txt ls | wc -l 4001 But that was a lot of work. It would be great if split would just handle these permutations and suffix lengths by default! In fact, I vaguely remember splitting large files into smaller ones with numerical filenames, which I prefer. I also remember not having to worry about suffixes in the past. But numerical filenames didn't seem to be an option with split installed on Mac OS X - there was no mention of it in the manual. Turns out that I was remembering GNU split from using the Debian OS two years ago, a different flavor of the split tool with different defaults and behaviors. Beastie Bits Michael Lucas is speaking at mug.org 10 April 2018 (https://blather.michaelwlucas.com/archives/3121) PkgsrcCon 2018 July 7+8 Berlin (http://pkgsrc.org/pkgsrcCon/2018/) Tint2 rocks (http://www.vincentdelft.be/post/post_20180310) Open Source Summit Europe 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/open-source-summit-europe-2018-call-for-proposals/) Travel Grants for BSDCan 2018 (https://www.freebsdfoundation.org/blog/bsdcan-2018-travel-grant-application-now-open/) BSDCan 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/bsdcan-2018-freebsd-developers-summit-call-for-proposals/) OpenBSD vmm(4) update, by Mike Larkin (https://www.openbsd.org/papers/asiabsdcon2018-vmm-slides.pdf) Feedback/Questions Morgan ZFS Install Question (http://dpaste.com/3NZN49P#wrap) Andre - Splitting ZFS Array, or not (http://dpaste.com/3V09BZ5#wrap) Jake - Python Projects (http://dpaste.com/2CY5MRE#wrap) Dave - Screen Sharing & Video Conference (http://dpaste.com/257WGCB#wrap) James - ZFS disk id switching (http://dpaste.com/3HAPZ90#wrap)
We'll cover OpenBSD's defensive approach to OS security, help you Understanding Syscall Conventions for Different Platforms, Mishandling SMTP Sender Verification, how the cd command works, and the LUA boot loader coming to FreeBSD. This episode was brought to you by Headlines Pledge: OpenBSD's defensive approach to OS Security (https://medium.com/@_neerajpal/pledge-openbsds-defensive-approach-for-os-security-86629ef779ce) The meaning of Pledge is same as in the real world, that is, “a solemn promise or undertaking”. So, in OpenBSD: Calling pledge in a program means to promise that the program will only use certain resources. How does it make a program more secure? It limits the operation of a program. Example: You wrote a program named ‘abc' that only needed the stdio to just print something to stdout. You added pledge to use only stdio and nothing else. Then, a malicious user found out that there is a vulnerability in your program which one can exploit and get into shell (or root shell). Exploiting your program to open a shell (or root shell) will result in the kernel killing the process with SIGABRT (which cannot be caught/ignored) and will generate a log (which you can find with dmesg). This happens because before executing other codes of your program, the code first pledges not to use anything other than stdio promise/operations. But, opening a shell or root shell will call several other system-calls which are distributed in lots of other promises like “stdio”, “proc”, “exec” etc. They are all forbidden because the program has already promised not to use any promises other than stdio. Pledge is not a system call filter. So, it is not used to restrict system calls. For example, pledge(“read”,NULL) ? wrong syntax of the pledge() pledge(“stdio inet”,NULL) ? correct syntax of the pledge() Pledge works on stdio, dns, inet, etc. promises but not directly on system calls like read, write, etc. And, unique functionality of pledge() is that it works on behavioral approach not just like 1:1 approach with the system calls. On 11 December 2017, Theo de Raadt said: List: openbsd-tech Subject: pledge execpromises From: Theo de Raadt Date: 2017–12–11 21:20:51 Message-ID: 6735.1513027251 () cvs ! openbsd ! org This will probably be committed in the next day or so. The 2nd argument of pledge() becomes execpromises, which is what will gets activated after execve. There is also a small new feature called “error”, which causes violating system calls to return -1 with ENOSYS rather than killing the process. This must be used with EXTREME CAUTION because libraries and programs are full of unchecked system calls. If you carry on past one of these failures, your program is in uncharted territory and risks of exploitation become high. “error” is being introduced for a different reason: The pre-exec process's expectation of what the post-exec process will do might mismatch, so “error” allows things like starting an editor which has no network access or maybe other restrictions in the future… Every Journey Starts with a FAIL...or Understanding Syscall Conventions for Different Platforms (http://k3research.outerhaven.de/posts/every-journey-starts-with-a-fail.html) Introduction Not long ago I started looking into FreeBSD kernel exploitation. There are only a few resources but probably the best starting point is argp's Phrack article from 2009[0]. And while he does only provide one technique, I wanted to understand it and port it to a modern FreeBSD release before describing new, own researched techniques. Well, at least this was my plan. In reality I ended researching how different operating systems resp. the same operating system but for different architectures implement syscalls. Hence, new exploiting methods have to wait for another post. In this one I want to describe my personal FAIL while porting argp's exploit example to a FreeBSD 11.1-RELEASE running on a 64bit processor. Maybe this will give other people interested in kernel stuff some insights they didn't know before. If you already know how syscalls work on 32bit and 64bit *BSD because you are an experienced exploit or kernel developer, you will probably want to search for something else to read. Moreover, some of the debugging stuff can look laborious because I wanted to show the steps I have done while attacking my problem instead of showing a simple walkthrough to the solution. The Problem argp described in his article vulnerable code consisting of a loadable kernel module which exposes a syscall to the userland. Because it was written around the time when FreeBSD 8-RELEASE came out and because he has written himself that the code needs smaller adjustments to work with this version (it was written for FreeBSD 7) I thought I will first port it to FreeBSD 11.1-RELEASE. Moreover it was written for an Intel 32bit processor architecture as we can see from his shellcode examples. Hence, I wanted to go right away the harder way and modify it to work on an 64bit processor. Why the Original Code Worked While It Was Wrong As written above, the syscall convention for the 32bit architecture is different from the one for the 64bit architecture. Indeed, a syscall on a 32bit FreeBSD system passes the arguments via the stack while the syscall offset is stored in the EAX register. The transfer into the kernel address space is done in 'cpufetchsyscall_args' in 'sys/i386/i386/trap.c'. ``` int cpufetchsyscallargs(struct thread *td, struct syscallargs *sa) { ... frame = td->td_frame; params = (caddr_t)frame->tf_esp + sizeof(int); sa->code = frame->tf_eax; ... if (params != NULL && sa->narg != 0) error = copyin(params, (caddr_t)sa->args, (u_int)(sa->narg * sizeof(int))); else ... } ``` That is, 'params' points to ESP+4 bytes offset. Later, the arguments are copied into the kernel space which is referenced by 'sa->args'. 'args' is an array of eight 'registert' which is defined as 'int32t' on the 32bit platform in comparison to the 64bit platform. And as 'struct args' only consisted of integers they got copied into the syscall arguments which are given to the trigger function inside the kernel module. We could verify this by changing 'int op' to 'long long op' in the kernel module and in trigger.c. We get the following output: root@freebsd64:trigger/ # ./trigger 0x28414000 256 3 1 0x28414000 256 4294967295 2 root@freebsd64:trigger/ # To bring this to an end: argp's version only worked for his special choice of arguments and only on 32bit. On 32bit FreeBSD platforms the arguments are transferred into kernel space by 4 byte integers, hence it will only work for integers anyway. On 64bit FreeBSD platforms we have to use syscall(2) in the intended way. iXsystems New Disks! (https://www.ixsystems.com/blog/gdpr-countdown/) A Life Lesson in Mishandling SMTP Sender Verification (https://bsdly.blogspot.co.uk/2018/02/a-life-lesson-in-mishandling-smtp.html) It all started with one of those rare spam mails that got through. This one was hawking address lists, much like the ones I occasionally receive to addresses that I can not turn into spamtraps. The message was addressed to, of all things, root@skapet.bsdly.net. (The message with full headers has been preserved here for reference). Yes, that's right, they sent their spam to root@. And a quick peek at the headers revealed that like most of those attempts at hawking address lists for spamming that actually make it to a mailbox here, this one had been sent by an outlook.com customer. The problem with spam delivered via outlook.com is that you can't usefully blacklist the sending server, since the largish chunk of the world that uses some sort of Microsoft hosted email solution (Office365 and its ilk) have their usually legitimate mail delivered via the very same infrastructure. And since outlook.com is one of the mail providers that doesn't play well with greylisting (it spreads its retries across no less than 81 subnets (the output of 'echo outlook.com | doas smtpctl spf walk' is preserved here), it's fairly common practice to just whitelist all those networks and avoid the hassle of lost or delayed mail to and from Microsoft customers. I was going to just ignore this message too, but we've seen an increasing number of spammy outfits taking advantage of outlook.com's seeming right of way to innocent third parties' mail boxes. So I decided to try both to do my best at demoralizing this particular sender and alert outlook.com to their problem. I wrote a messsage (preserved here) with a Cc: to abuse@outlook.com where the meat is, ``` Ms Farell, The address root@skapet.bsdly.net has never been subscribed to any mailing list, for obvious reasons. Whoever sold you an address list with that address on it are criminals and you should at least demand your money back. Whoever handles abuse@outlook.com will appreciate the attachment, which is a copy of the message as it arrived here with all headers intact. Yours sincerely, Peter N. M. Hansteen ``` What happened next is quite amazing. If my analysis is correct, it may not be possible for senders who are not themselves outlook.com customers to actually reach the outlook.com abuse team. Any student or practitioner of SMTP mail delivery should know that SPF records should only happen on ingress, that is at the point where the mail traffic enters your infrastructure and the sender IP address is the original one. Leave the check for later when the message may have been forwarded, and you do not have sufficient data to perform the check. Whenever I encounter incredibly stupid and functionally destructive configuration errors like this I tend to believe they're down to simple incompetence and not malice. But this one has me wondering. If you essentially require incoming mail to include the contents of spf.outlook.com (currently no less than 81 subnets) as valid senders for the domain, you are essentially saying that only outlook.com customers are allowed to communicate. If that restriction is a result of a deliberate choice rather than a simple configuration error, the problem moves out of the technical sphere and could conceivably become a legal matter, depending on what outlook.com have specified in their contracts that they are selling to their customers. But let us assume that this is indeed a matter of simple bad luck or incompetence and that the solution is indeed technical. I would have liked to report this to whoever does technical things at that domain via email, but unfortunately there are indications that being their customer is a precondition for using that channel of communication to them. I hope they fix that, and soon. And then move on to terminating their spamming customers' contracts. The main lesson to be learned from this is that when you shop around for email service, please do yourself a favor and make an effort to ensure that your prospective providers actually understand how the modern-ish SMTP addons SPF, DKIM and DMARC actually work. Otherwise you may end up receiving more of the mail you don't want than what you do want, and your own mail may end up not being delivered as intended. News Roundup Running Salt Proxy Minions on OpenBSD (https://mirceaulinic.net/2018-02-14-openbsd-salt-proxy/) As I have previously attempted several times in the past, I am (finally) very close to switch to OpenBSD, a more stable and reliable operating system that I like. Before starting to make the actual change on both personal and work computer, I started testing some of the tools I'm currently using, and understand what are the expectations. In general I didn't encounter issues, or when I did, I found the answers in the documentation (which is really great), or various forums. I didn't find however any questions regarding Proxy Minions on OpenBSD which is why I thought it might be helpful to share my experience. Installation and Startup With these said, I started playing with Salt, and it was simple and straightforward. First step - install Salt: pkg_add salt. This will bring several ports for Python futures, ZeroMQ, or Tornado which are needed for Salt. After configuring the pillar_roots in the /etc/salt/master config file for the Master, I started up the master process using rcctl: Starting up the Proxy Minions The Salt package for OpenBSD comes with the rc file for salt-proxy as well, /etc/rc.d/salt_proxy While typically you run a single regular Minion on a given machine, it is very like that there are multiple Proxy processes. Additionally, the default Salt rc file has the following configuration for the salt-proxy daemon: Starting many Proxy Minions I have managed to startup a Proxy Minion, but what about many? Executing the three commands above for each and every device is tedious and cannot scale very well. I thus have figured the following way: Have a separate rc file per Proxy, each having the daemon instruction explicitly specifying its Minion ID Start the service (using the regular Minion that controls the machine where the Proxy processes are running) And the test Proxy Minion is then up (after accepting the key, i.e,, salt-key -a test) Extending the same to a (very) large number of Proxy Minions, you can easily manage the rc files and start the services using a Salt State executed on the regular Minion: Using the file.managed State function to generate the contents of the rc file for each Proxy, with its own Minion ID. Using the service.running State function start the service. These two steps would suffice to start an arbitrary number of Proxy Minions, and the command executed will always be the same regardless how many processes you aim to manage. Conclusions I am still a novice when it comes to OpenBSD, I have plenty to learn, but it looks like the transition will be much smoother than I expected. I am already looking forward to the handover, and - most importantly - I will no longer be using systemd. :-) LUA boot loader coming very soon (https://lists.freebsd.org/pipermail/freebsd-current/2018-February/068464.html) As you may know, the Lua (http://www.lua.org) boot loader has been in the works for some time. It started out life as a GSoC in 2014 by Pedro Souza mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to try to integrate it. I rebased that effort into a github branch which Pedro Arthur fixed up. Over the past year, I've been cleaning up the boot loader for other reasons, and found the time was ripe to start integrating this into the tree. However, those integration efforts have taken a while as my day-job work on the boot loader took priority. In the mean time, Ed Maste and the FreeBSD Foundation funded Zakary Nafziger to enhance the original GSoC Lua scripts to bring it closer to parity with the evolution of the FORTH menu system since the GSoC project started. I'm pleased to announce that all these threads of development have converged and I'll be pushing the FreeBSD Lua Loader later today. This loader uses Lua as its scripting language instead of FORTH. While co-existance is planned, the timeline for it is looking to be a few weeks and I didn't want to delay pushing this into the tree for that. To try the loader, you'll need to build WITHOUTFORTH=yes and WITHLOADERLUA=yes. Fortunately, you needn't do a full world to do this, you can do it in src/stand and install the result (be sure to have the options for both the build and the install). This will replace your current /boot/loader that is scripted with FORTH to one that's scripted with Lua. It will install the lua scripts in /boot/lua. The boot is scripted with /boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised to create a backup copy of /boot/loader before testing (eg cp /boot/loader /boot/loaderforth), since you'll need to boot that from boot2 if something goes wrong. I've tested it extensively, though, with userboot.so and it's test program, so all the initial kinks of finding the lua scripts, etc have been worked out. While it's possible to build all the /boot/loader variants with Lua, I've just tested a BIOS booting /boot/loader both with and without menus enabled. I've not tested any of the other variants and the instructions for testing some of them may be rather tedious (especially UEFI, if you want a simple path to back out). Since there's not been full convergence testing, you'll almost certainly find bumps in this system. Also, all the build-system APIs are likely not yet final. I put MFC after a month on the commit. Due to the heroic (dare I say almost crazy) work of Kyle Evans on merging all the revs from -current to 11, I'm planning a MFC to 11 after the co-existence issues are hammered out. In 11, FORTH will be the default, and Lua will be built by default, but users will have to do something to use it. 12, both FORTH and Lua will be built and installed, with Lua as default (barring unforeseen complications). Once the co-existence stuff goes in, I imagine we'll make the switch to Lua by default shortly after that. In 13, FORTH will be removed unless there's a really really compelling case made to keep it. So please give it a spin and give me any feedback, documentation updates and/or bug fixes. I'm especially interested in reviews from people that have embedded Lua in other projects or experts in Lua that can improve the robustness of the menu code. Bitcoin Full Node on FreeBSD (https://bsdmag.org/5374-2/) What is a Bitcoin ? Bitcoin is a valuable popular open-source cryptocurrency that was invented by Satoshi Nakamoto in 2009. Bitcoins have value because they possess same characteristics like money (durability, portability, fungibility, scarcity, divisibility, and recognizability), but based on the properties of mathematics rather than on physical properties (like gold and silver) or trust in central authorities (like fiat currencies). In short, Bitcoin is backed by mathematics. Bitcoin is the first decentralized peer-to-peer cryptocurrency that is controlled by its users. Transactions take place directly between users, and are later verified by network nodes with digital signature and then placed in a public distributed ledger called a blockchain. Bitcoin is unique in that only 21 million bitcoins will ever be created. The unit of the bitcoin system is bitcoin or mBTC. What is a Bitcoin Wallet ? A wallet is nothing more than a pair of public and private keys that are created by a client to store the digital credentials for your bitcoin. There are several types of wallets: Desktop Wallet Token Wallet Online Wallet Mobile Wallet A token wallet is the safest way to work with bitcoin network, but you can use your mobile or pc as a bitcoin wallet. What is a Blockchain? A blockchain is a ledger that records bitcoin transactions. The blockchain is a distributed database that achieves independent verification of the chain of ownership. Each network node stores its own copy of the blockchain. Transactions will broadcast on the bitcoin network, and about 2400 transactions create a block. These blocks are building blocks of the blockchain. What is Mining? Mining is the process of dedicating computing power to process transactions, secure the network, and keep everyone in the system synchronized together. It has been designed to be fully decentralized. Miners need mining software with specialized hardware. Mining software listens for transactions broadcasted through the peer-to-peer network and performs appropriate tasks to process and confirm these transactions. Bitcoin miners perform this work because they can earn transaction fees paid by users for faster transaction processing. New transactions have to be confirmed then be included in a block along with a mathematical proof of work. Such proofs are very hard to generate because there is no way to create them other than by trying billions of calculations per second. Hence, miners are required to perform these calculations before their blocks are accepted by the network and before they are rewarded. As more people start to mine, the difficulty of finding valid blocks is automatically increased by the network to ensure that the average time to find a block remains equal to 10 minutes. As a result, mining is a very competitive business where no individual miner can control what is included in the blockchain. The proof of work is also designed to depend on the previous block to force a chronological order in the blockchain. This makes it exponentially difficult to reverse previous transactions because it would require the recalculation of the proofs of work of all the subsequent blocks. When two blocks are found at the same time, miners work on the first block they receive and switch to the longest chain of blocks as soon as the next block is found. This allows mining to secure and maintain a global consensus based on processing power. What is Pooled Mining? You have more chances if you participate with others to create a block. In a pool, all participating miners get paid every time a participating server solves a block. The payment depends on the amount of work an individual miner contributed to help find that block. What is a Full Node? A full node is a client that fully validates transactions and blocks. Full nodes also help the network by accepting transactions and blocks from other full nodes, validating those transactions and blocks, and then relaying them to further full nodes. Many people and organizations volunteer to run full nodes using spare computing and bandwidth resources. What is a Bitcoind? bitcoind is a Bitcoin client under the MIT license in 32-bit and 64-bit versions for Windows, GNU/Linux-based OSes, Mac OS X, OpenBSD and FreeBSD as well. Conclusion Cryptocurrencies are replacement for banking we know today, and bitcoin is the game changer. Mining bitcoin with typical hardware is not a good idea. It needs specialized devices like ASIC, but you can create a full node and help the bitcoin network. Useful Links https://en.wikipedia.org/wiki/Cryptocurrency https://bitcoin.org/en/faq *** Latest DRM Graphics work The DRM Graphics stack from Linux is ported to FreeBSD on an ongoing basis to provide support for accelerated graphics for Intel and AMD GPUs. The LinuxKPI bits that the drm-next-kmod driver port depends on have been merged into stable/11 and will be included as part of the upcoming FreeBSD 11.2 (https://svnweb.freebsd.org/ports?view=revision&revision=462202) Additionally, the version of the drives has been updated from Linux 4.9 to Linux 4.11 with a number of additional devices being supported (https://lists.freebsd.org/pipermail/freebsd-current/2018-February/068690.html) *** How does cd work? (https://blog.safia.rocks/post/171311670379/how-does-cd-work) In my last blog post, I dove into some of the code behind the sudo command. I thought this was pretty fun. sudo is one of those commands that I use quite often but haven't had the chance to look into truly. I started thinking about other commands that I use on a daily basis but had little understanding of the internals of. The first command that came to mind is cd. cd stands for change directory. Simply put, it allows you to set your current working directory to a different directory. I read through some of the code that was defined in this file. Some of it was in functions, and other bits were in templates, but after a while, I figured that most of the code was a wrapper around a function called chdir. A lot of the functions defined in the cd.def file linked above actually just invoke chdir and handle errors and parameter cleaning. So all in all, here is what happens when you run cd on the command line. The cd builtin is invoked as part of the Bash shell. The Bash shell invokes the chdir function. The chdir function is part of Unix and invokes the chdir system call. The Unix kernel executes the chdir call and does its own low-level thing. I could dive in a little bit more into how #4 works, but let's be honest, I've already read too much code at this point, and my eyes are starting to hurt. Beastie Bits Stockholm BSD User Group: March 22 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) Open Source Hardware Camp 2018 (30/06 & 01/07) Call for Participation (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2018-February/014182.html) Initial release schedule announcement for FreeBSD 11.2 (https://www.freebsd.org/releases/11.2R/schedule.html) Serious Shell Programming (Devin Teske) (https://www.gitbook.com/book/freebsdfrau/serious-shell-programming/details) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) TCP Fast Open client side lands in FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=330001) Help the Tor BSD Project increase the OS diversity of Tor nodes, for your own safety, and everyone else's (https://torbsd.org/open-letter.html) 5 Differences Between TrueOS & Linux (https://www.kompulsa.com/2018/02/23/5-differences-trueos-linux/) *** Feedback/Questions Ambrose - Bunch of questions (http://dpaste.com/0KRRG18#wrap) Eddy - ZFSoL with single SSD (http://dpaste.com/0MTXYJN#wrap)
How the term open source was created, running FreeBSD on ThinkPad T530, Moving away from Windows, Unknown Giants, as well as OpenBSD and FreeDOS. This episode was brought to you by Headlines How I coined the term 'open source' (https://opensource.com/article/18/2/coining-term-open-source-software) In a few days, on February 3, the 20th anniversary of the introduction of the term "open source software" is upon us. As open source software grows in popularity and powers some of the most robust and important innovations of our time, we reflect on its rise to prominence. I am the originator of the term "open source software" and came up with it while executive director at Foresight Institute. Not a software developer like the rest, I thank Linux programmer Todd Anderson for supporting the term and proposing it to the group. This is my account of how I came up with it, how it was proposed, and the subsequent reactions. Of course, there are a number of accounts of the coining of the term, for example by Eric Raymond and Richard Stallman, yet this is mine, written on January 2, 2006. It has never been published, until today. The introduction of the term "open source software" was a deliberate effort to make this field of endeavor more understandable to newcomers and to business, which was viewed as necessary to its spread to a broader community of users. The problem with the main earlier label, "free software," was not its political connotations, but that—to newcomers—its seeming focus on price is distracting. A term was needed that focuses on the key issue of source code and that does not immediately confuse those new to the concept. The first term that came along at the right time and fulfilled these requirements was rapidly adopted: open source. This term had long been used in an "intelligence" (i.e., spying) context, but to my knowledge, use of the term with respect to software prior to 1998 has not been confirmed. The account below describes how the term open source software caught on and became the name of both an industry and a movement. Meetings on computer security In late 1997, weekly meetings were being held at Foresight Institute to discuss computer security. Foresight is a nonprofit think tank focused on nanotechnology and artificial intelligence, and software security is regarded as central to the reliability and security of both. We had identified free software as a promising approach to improving software security and reliability and were looking for ways to promote it. Interest in free software was starting to grow outside the programming community, and it was increasingly clear that an opportunity was coming to change the world. However, just how to do this was unclear, and we were groping for strategies. At these meetings, we discussed the need for a new term due to the confusion factor. The argument was as follows: those new to the term "free software" assume it is referring to the price. Oldtimers must then launch into an explanation, usually given as follows: "We mean free as in freedom, not free as in beer." At this point, a discussion on software has turned into one about the price of an alcoholic beverage. The problem was not that explaining the meaning is impossible—the problem was that the name for an important idea should not be so confusing to newcomers. A clearer term was needed. No political issues were raised regarding the free software term; the issue was its lack of clarity to those new to the concept. Releasing Netscape On February 2, 1998, Eric Raymond arrived on a visit to work with Netscape on the plan to release the browser code under a free-software-style license. We held a meeting that night at Foresight's office in Los Altos to strategize and refine our message. In addition to Eric and me, active participants included Brian Behlendorf, Michael Tiemann, Todd Anderson, Mark S. Miller, and Ka-Ping Yee. But at that meeting, the field was still described as free software or, by Brian, "source code available" software. While in town, Eric used Foresight as a base of operations. At one point during his visit, he was called to the phone to talk with a couple of Netscape legal and/or marketing staff. When he was finished, I asked to be put on the phone with them—one man and one woman, perhaps Mitchell Baker—so I could bring up the need for a new term. They agreed in principle immediately, but no specific term was agreed upon. Between meetings that week, I was still focused on the need for a better name and came up with the term "open source software." While not ideal, it struck me as good enough. I ran it by at least four others: Eric Drexler, Mark Miller, and Todd Anderson liked it, while a friend in marketing and public relations felt the term "open" had been overused and abused and believed we could do better. He was right in theory; however, I didn't have a better idea, so I thought I would try to go ahead and introduce it. In hindsight, I should have simply proposed it to Eric Raymond, but I didn't know him well at the time, so I took an indirect strategy instead. Todd had agreed strongly about the need for a new term and offered to assist in getting the term introduced. This was helpful because, as a non-programmer, my influence within the free software community was weak. My work in nanotechnology education at Foresight was a plus, but not enough for me to be taken very seriously on free software questions. As a Linux programmer, Todd would be listened to more closely. The key meeting Later that week, on February 5, 1998, a group was assembled at VA Research to brainstorm on strategy. Attending—in addition to Eric Raymond, Todd, and me—were Larry Augustin, Sam Ockman, and attending by phone, Jon "maddog" Hall. The primary topic was promotion strategy, especially which companies to approach. I said little, but was looking for an opportunity to introduce the proposed term. I felt that it wouldn't work for me to just blurt out, "All you technical people should start using my new term." Most of those attending didn't know me, and for all I knew, they might not even agree that a new term was greatly needed, or even somewhat desirable. Fortunately, Todd was on the ball. Instead of making an assertion that the community should use this specific new term, he did something less directive—a smart thing to do with this community of strong-willed individuals. He simply used the term in a sentence on another topic—just dropped it into the conversation to see what happened. I went on alert, hoping for a response, but there was none at first. The discussion continued on the original topic. It seemed only he and I had noticed the usage. Not so—memetic evolution was in action. A few minutes later, one of the others used the term, evidently without noticing, still discussing a topic other than terminology. Todd and I looked at each other out of the corners of our eyes to check: yes, we had both noticed what happened. I was excited—it might work! But I kept quiet: I still had low status in this group. Probably some were wondering why Eric had invited me at all. Toward the end of the meeting, the question of terminology was brought up explicitly, probably by Todd or Eric. Maddog mentioned "freely distributable" as an earlier term, and "cooperatively developed" as a newer term. Eric listed "free software," "open source," and "sourceware" as the main options. Todd advocated the "open source" model, and Eric endorsed this. I didn't say much, letting Todd and Eric pull the (loose, informal) consensus together around the open source name. It was clear that to most of those at the meeting, the name change was not the most important thing discussed there; a relatively minor issue. Only about 10% of my notes from this meeting are on the terminology question. But I was elated. These were some key leaders in the community, and they liked the new name, or at least didn't object. This was a very good sign. There was probably not much more I could do to help; Eric Raymond was far better positioned to spread the new meme, and he did. Bruce Perens signed on to the effort immediately, helping set up Opensource.org and playing a key role in spreading the new term. For the name to succeed, it was necessary, or at least highly desirable, that Tim O'Reilly agree and actively use it in his many projects on behalf of the community. Also helpful would be use of the term in the upcoming official release of the Netscape Navigator code. By late February, both O'Reilly & Associates and Netscape had started to use the term. Getting the name out After this, there was a period during which the term was promoted by Eric Raymond to the media, by Tim O'Reilly to business, and by both to the programming community. It seemed to spread very quickly. On April 7, 1998, Tim O'Reilly held a meeting of key leaders in the field. Announced in advance as the first "Freeware Summit," by April 14 it was referred to as the first "Open Source Summit." These months were extremely exciting for open source. Every week, it seemed, a new company announced plans to participate. Reading Slashdot became a necessity, even for those like me who were only peripherally involved. I strongly believe that the new term was helpful in enabling this rapid spread into business, which then enabled wider use by the public. A quick Google search indicates that "open source" appears more often than "free software," but there still is substantial use of the free software term, which remains useful and should be included when communicating with audiences who prefer it. A happy twinge When an early account of the terminology change written by Eric Raymond was posted on the Open Source Initiative website, I was listed as being at the VA brainstorming meeting, but not as the originator of the term. This was my own fault; I had neglected to tell Eric the details. My impulse was to let it pass and stay in the background, but Todd felt otherwise. He suggested to me that one day I would be glad to be known as the person who coined the name "open source software." He explained the situation to Eric, who promptly updated his site. Coming up with a phrase is a small contribution, but I admit to being grateful to those who remember to credit me with it. Every time I hear it, which is very often now, it gives me a little happy twinge. The big credit for persuading the community goes to Eric Raymond and Tim O'Reilly, who made it happen. Thanks to them for crediting me, and to Todd Anderson for his role throughout. The above is not a complete account of open source history; apologies to the many key players whose names do not appear. Those seeking a more complete account should refer to the links in this article and elsewhere on the net. FreeBSD on a Laptop - A guide to a fully functional installation of FreeBSD on a ThinkPad T530 (https://www.c0ffee.net/blog/freebsd-on-a-laptop) As I stated my previous post, I recently dug up my old ThinkPad T530 after the embarrassing stream of OS X security bugs this month. Although this ThinkPad ran Gentoo faithfully during my time in graduate school at Clemson, these days I'd much rather spend time my wife and baby than fighting with emerge and USE flags. FreeBSD has always been my OS of choice, and laptop support seems to be much better than it was a few years ago. In this guide, I'll show you the tweaks I made to wrestle FreeBSD into a decent experience on a laptop. Unlike my usual posts, this time I'm going to assume you're already pretty familiar with FreeBSD. If you're a layman looking for your first BSD-based desktop, I highly recommend checking out TrueOS (previously PC-BSD): they've basically taken FreeBSD and packaged it with all the latest drivers, along with a user-friendly installer and custom desktop environment out of the box. TrueOS is an awesome project–the only reason I don't use it is because I'm old, grumpy, and persnickety about having my operating system just so. Anyway, if you'd still like to take the plunge, read on. Keep in mind, I'm using a ThinkPad T530, but other ThinkPads of the same generation should be similarly compatible. Here's what you'll get: Decent battery life (8-9 hours with a new 9-cell battery) UEFI boot and full-disk encryption WiFi (Intel Ultimate-N 6300) Ethernet (Intel PRO/1000) Screen brightness adjustment Suspend/Resume on lid close (make sure to disable TPM in BIOS) Audio (Realtek ALC269 HDA, speakers and headphone jack) Keyboard multimedia buttons Touchpad/Trackpoint Graphics Acceleration (with integrated Intel graphics, NVIDIA card disabled in BIOS) What I haven't tested yet: Bluetooth Webcam Fingerprint reader SD Card slot Installation Power Saving Tweaks for Desktop Use X11 Fonts Login Manager: SLiM Desktop Environment: i3 Applications The LLVM Sanitizers stage accomplished (https://blog.netbsd.org/tnf/entry/the_llvm_sanitizers_stage_accomplished) I've managed to get the Memory Sanitizer to work for the elementary base system utilities, like ps(1), awk(1) and ksh(1). This means that the toolchain is ready for tests and improvements. I've iterated over the basesystem utilities and I looked for bugs, both in programs and in sanitizers. The number of detected bugs in the userland programs was low, there merely was one reading of an uninitialized variable in ps(1). A prebuilt LLVM toolchain I've prepared a prebuilt toolchain with Clang, LLVM, LLDB and compiler-rt for NetBSD/amd64. I prepared the toolchain on 8.99.12, however I have received reports that it works on other older releases. Link: llvm-clang-compilerrt-lldb-7.0.0beta_2018-01-24.tar.bz2 The archive has to be untarballed to /usr/local (however it might work to some extent in other paths). This toolchain contains a prebuilt tree of the LLVM projects from a snapshot of 7.0.0(svn). It is a pristine snapshot of HEAD with patches from pkgsrc-wip for llvm, clang, compiler-rt and lldb. Sanitizers Notable changes in sanitizers, all of them are in the context of NetBSD support. Added fstat(2) MSan interceptor. Support for kvm(3) interceptors in the common sanitizer code. Added devname(3) and devname_r(3) interceptors to the common sanitizer code. Added sysctl(3) familty of functions interceptors in the common sanitizer code. Added strlcpy(3)/strlcat(3) interceptors in the common sanitizer code. Added getgrouplist(3)/getgroupmembership(3) interceptors in the common sanitizer code. Correct ctype(3) interceptors in a code using Native Language Support. Correct tzset(3) interceptor in MSan. Correct localtime(3) interceptor in the common sanitizer code. Added paccept(2) interceptor to the common sanitizer code. Added access(2) and faccessat(2) interceptors to the common sanitizer code. Added acct(2) interceptor to the common sanitizer code. Added accept4(2) interceptor to the common sanitizer code. Added fgetln(3) interceptor to the common sanitizer code. Added interceptors for the pwcache(3)-style functions in the common sanitizer code. Added interceptors for the getprotoent(3)-style functions in the common sanitizer code. Added interceptors for the getnetent(3)-style functions in the common sanitizer code. Added interceptors for the fts(3)-style functions in the common sanitizer code. Added lstat(3) interceptor in MSan. Added strftime(3) interceptor in the common sanitizer code. Added strmode(3) interceptor in the common sanitizer code. Added interceptors for the regex(3)-style functions in the common sanitizer code. Disabled unwanted interceptor __sigsetjmp in TSan. Base system changes I've tidied up inclusion of the internal namespace.h header in libc. This has hidden the usage of public global symbol names of: strlcat -> _strlcat sysconf -> __sysconf closedir -> _closedir fparseln -> _fparseln kill -> _kill mkstemp -> _mkstemp reallocarr -> _reallocarr strcasecmp -> _strcasecmp strncasecmp -> _strncasecmp strptime -> _strptime strtok_r -> _strtok_r sysctl -> _sysctl dlopen -> __dlopen dlclose -> __dlclose dlsym -> __dlsym strlcpy -> _strlcpy fdopen -> _fdopen mmap -> _mmap strdup -> _strdup The purpose of these changes was to stop triggering interceptors recursively. Such interceptors lead to sanitization of internals of unprepared (not recompiled with sanitizers) prebuilt code. It's not trivial to sanitize libc's internals and the sanitizers are not designed to do so. This means that they are not a full replacement of Valgrind-like software, but a a supplement in the developer toolbox. Valgrind translates native code to a bytecode virtual machine, while sanitizers are designed to work with interceptors inside the pristine elementary libraries (libc, libm, librt, libpthread) and embed functionality into the executable's code. I've also reverted the vadvise(2) syscall removal, from the previous month. This caused a regression in legacy code recompiled against still supported compat layers. Newly compiled code will use a libc's stub of vadvise(2). I've also prepared a patch installing dedicated headers for sanitizers along with the base system GCC. It's still discussed and should land the sources soon. Future directions and goals Possible paths in random order: In the quartet of UBSan (Undefined Behavior Sanitizer), ASan (Address Sanitizer), TSan (Thread Sanitizer), MSan (Memory Sanitizer) we need to add the fifth basic sanitizer: LSan (Leak Sanitizer). The Leak Sanitizer (detector of memory leaks) demands a stable ptrace(2) interface for processes with multiple threads (unless we want to build a custom kernel interface). Integrate the sanitizers with the userland framework in order to ship with the native toolchain to users. Port sanitizers from LLVM to GCC. Allow to sanitize programs linked against userland libraries other than libc, librt, libm and libpthread; by a global option (like MKSANITIZER) producing a userland that is partially prebuilt with a desired sanitizer. This is required to run e.g. MSanitized programs against editline(3). So far, there is no Operating System distribution in existence with a native integration with sanitizers. There are 3rd party scripts for certain OSes to build a stack of software dependencies in order to validate a piece of software. Execute ATF tests with the userland rebuilt with supported flavors of sanitizers and catch regressions. Finish porting of modern linkers designed for large C++ software, such as GNU GOLD and LLVM LLD. Today the bottleneck with building the LLVM toolchain is a suboptimal linker GNU ld(1). I've decided to not open new battlefields and return now to porting LLDB and fixing ptrace(2). Plan for the next milestone Keep upstreaming a pile of local compiler-rt patches. Restore the LLDB support for traced programs with a single thread. Interview - Goran Mekic - meka@tilda.center (mailto:meka@tilda.center) / @meka_floss (https://twitter.com/meka_floss) CBSD website (https://bsdstore.ru) Jail and VM Manager *** News Roundup Finally Moving Away From Windows (https://www.manios.ca/blog/2018/01/finally-moving-away-from-windows/) Broken Window Thanks to a combination of some really impressive malware, bad clicking, and poor website choices, I had to blow away my Windows 10 installation. Not that it was Window's fault, but a piece of malware had infected my computer when I tried to download a long lost driver for an even longer lost RAID card for a server. A word of advice – the download you're looking for is never on an ad-infested forum in another language. In any case, I had been meaning to switch away from Windows soon. I didn't have my entire plan ready, but now was as good a time as any. My line of work requires me to maintain some form of Windows installation, so I decided to keep it in a VM rather than dual booting as I was developing code and not running any high-end visual stuff like games. My first thought was to install Arch or Gentoo Linux, but the last time I attempted a Gentoo installation it left me bootless. Not that there is anything wrong with Gentoo, it was probably my fault, but I like the idea of some sort of installer so I looked at rock-solid Debian. My dad had installed Debian on his sweet new cutting-edge Lenovo laptop he received recently from work. He often raves about his cool scripts and much more effective customized experience, but often complains about his hybrid GPU support as he has an Intel/Nvidia hybrid display adapter (he has finally resolved it and now boasts his 6 connected displays). I didn't want to install Windows again, but something didn't feel right about installing some flavour of Linux. Back at home I have a small collection of FreeBSD servers running in all sorts of jails and other physical hardware, with the exception of one Debian server which I had the hardest time dealing with (it would be FreeBSD too if 802.11ac support was there as it is acting as my WiFi/gateway/IDS/IPS). I loved my FreeBSD servers, and yes I will write posts about each one soon enough. I wanted that cleanliness and familiarity on my desktop as well (I really love the ports collection!). It's settled – I will run FreeBSD on my laptop. This also created a new rivalry with my father, which is not a bad thing either. Playing Devil's Advocate The first thing I needed to do was backup my Windows data. This was easy enough, just run a Windows Image Backup and it will- wait, what? Why isn't this working? I didn't want to fiddle with this too long because I didn't actually need an image just the data. I ended up just copying over the files to an external hard disk. Once that was done, I downloaded and verified the latest FreeBSD 11.1 RELEASE memstick image and flashed it to my trusty 8GB Verbatim USB stick. I've had this thing since 2007, it works great for being my re-writable “CD”. I booted it up and started the installation. I knew this installer pretty well as I had test-installed FreeBSD and OpenBSD in VMs when I was researching a Unix style replacement OS last year. In any case, I left most of the defaults (I didn't want to play with custom kernels right now) and I selected all packages. This downloaded them from the FreeBSD FTP server as I only had the memstick image. The installer finished and I was off to my first boot. Great! so far so good. FreeBSD loaded up and I did a ‘pkg upgrade' just to make sure that everything was up to date. Alright, time to get down to business. I needed nano. I just can't use vi, or just not yet. I don't care about being a vi-wizard, that's just too much effort for me. Anyway, just a ‘pkg install nano' and I had my editor. Next was obvious, I needed x11. XFCE was common, and there were plenty of tutorials out there. I wont bore you with those details, but it went something like ‘pkg install xfce' and I got all the dependencies. Don't forget to install SLiM to make it seamless. There are some configs in the .login I think. SLiM needs to be called once the boot drops you to the login so that you get SLiM's nice GUI login instead of the CLI login screen. Then SLiM passes you off to XFCE. I think I followed this and this. Awesome. Now that x11 is working, it's time to get all of my apps from Windows. Obviously, I can't get everything (ie. Visual Studio, Office). But in my Windows installation, I had chosen many open-source or cross-compiled apps as they either worked better or so that I was ready to move away from Windows at a moments notice. ‘pkg install firefox thunderbird hexchat pidgin gpa keepass owncloud-client transmission-qt5 veracrypt openvpn' were some immediate picks. There are a lot more that I downloaded later, but these are a few I use everyday. My laptop also has the same hybrid display adapter config that my dad's has, but I chose to only run Intel graphics, so dual screens are no problem for me. I'll add Nvidia support later, but it's not a priority. After I had imported my private keys and loaded my firefox and thunderbird settings, I wanted to get my Windows VM running right away as I was burning productive days at work fiddling with this. I had only two virtualisation options; qemu/kvm and bhyve. qemu/kvm wasn't available in pkg, and looked real dirty to compile, from FreeBSD's point of view. My dad is using qemu/kvm with virt-manager to manage all of his Windows/Unix VMs alike. I wanted that experience, but I also wanted packages that could be updated and I didn't want to mess up a compile. bhyve was a better choice. It was built-in, it was more compatible with Windows (from what I read), and this is a great step-by-step article for Windows 10 on FreeBSD 11 bhyve! I had already tried to get virt-manager to work with bhyve with no luck. I don't think libvirt connects with bhyve completely, or maybe my config is wrong. But I didn't have time to fiddle with it. I managed it all through command lines and that has worked perfectly so far. Well sorta, there was an issue installing SQL Server, and only SQL Server, on my Windows VM. This was due to a missing ‘sectorsize=512' setting on the disk parameter on the bhyve command line. That was only found after A LOT of digging because the SQL Server install didn't log the error properly. I eventually found out that SQL Server only likes one sector size of disks for the install and my virtual disk geometry was incorrect. Apps Apps Apps I installed Windows 10 on my bhyve VM and I got that all setup with the apps I needed for work. Mostly Office, Visual Studio, and vSphere for managing our server farm. Plus all of the annoying 3rd party VPN software (I'm looking at you Dell and Cisco). Alright, with the Windows VM done, I can now work at work and finish FreeBSD mostly during the nights. I still needed my remote files (I setup an ownCloud instance on a FreeNAS jail at home) so I setup the client. Now, normally on Windows I would come to work and connect to my home network using OpenVPN (again, I have a OpenVPN FreeNAS jail at home) and the ownCloud desktop would be able to handle changing DNS destination IPs Not on FreeBSD (and Linux too?). I ended up just configuring the ownCloud client to just connect to the home LAN IP for the ownCloud server and always connecting the OpenVPN to sync things. It kinda sucks, but at least it works. I left that running at home overnight to get a full sync (~130GB cloud sync, another reason I use it over Google or Microsoft). Once that was done I moved onto the fstab as I had another 1TB SSD in my laptop with other files. I messed around with fstab and my NFS shares to my FreeNAS at home, but took them out as they made the boot time so long when I wasn't at home. I would only mount them when my OpenVPN connected or manually. I really wanted to install SpaceFM, but it's only available as a package on Debian and their non-package install script doesn't work on FreeBSD (packages are named differently). I tried doing it manually, but it was too much work. As my dad was the one who introduced me to it, he still uses it as a use-case for his Debian setup. Instead I kept to the original PCManFM and it works just fine. I also loaded up my Bitcoin and Litecoin wallets and pointed them to the blockchain that I has used on Windows after their sync, they loaded perfectly and my balances were there. I kinda wish there was the Bitcoin-ABC full node Bitcoin Cash wallet package on FreeBSD, but I'm sure it will come out later. The rest is essentially just tweaks and making the environment more comfortable for me, and with most programs installed as packages I feel a lot better with upgrades and audit checking (‘pkg audit -F' is really helpful!). I will always hate Python, actually, I will always hate any app that has it's own package manager. I do miss the GUI GitHub tool on Windows. It was a really good-looking way to view all of my repos. The last thing (which is increasing it's priority every time I go to a social media site or YouTube) is fonts. My god I never thought it was such a problem, and UTF support is complicated. If anyone knows how to get all UTF characters to show up, please let me know. I'd really like Wikipedia articles to load perfectly (I followed this post and there are still some missing). There are some extra tweaks I followed here and here. Conclusion I successfully migrated from Windows 10 to FreeBSD 11.1 with minimal consequence. Shout out goes to the entire FreeBSD community. So many helpful people in there, and the forums are a great place to find tons of information. Also thanks to the ones who wrote the how-to articles I've referenced. I never would have gotten bhyve to work and I'd still probably be messing with my X config without them. I guess my take home from this is to not be afraid to make changes that may change how comfortable I am in an environment. I'm always open to comments and questions, please feel free to make them below. I purposefully didn't include too many technical things or commands in this article as I wanted to focus on the larger picture of the migration as a whole not the struggles of xorg.conf, but if you would like to see some of the configs or commands I used, let me know and I'll include some! TrueOS Rules of Conduct (https://www.trueos.org/rulesofconduct/) We believe code is truly agnostic and embrace inclusiveness regardless of a person's individual beliefs. As such we only ask the following when participating in TrueOS public events and digital forums: Treat each other with respect and professionalism. Leave personal and TrueOS unrelated conversations to other channels. In other words, it's all about the code. Users who feel the above rules have been violated in some way can register a complaint with abuse@trueos.org + Shorter than the BSD License (https://twitter.com/trueos/status/965994363070353413) + Positive response from the community (https://twitter.com/freebsdbytes/status/966567686015782912) I really like the @TrueOS Code of Conduct, unlike some other CoCs. It's short, clear and covers everything. Most #OpenSource projects are labour of love. Why do you need a something that reads like a legal contract? FreeBSD: The Unknown Giant (https://neomoevius.tumblr.com/post/171108458234/freebsd-the-unknown-giant) I decided to write this article as a gratitude for the recent fast answer of the FreeBSD/TrueOS community with my questions and doubts. I am impressed how fast and how they tried to help me about this operating system which I used in the past(2000-2007) but recently in 2017 I began to use it again. + A lot has changed in 10 years I was looking around the internet, trying to do some research about recent information about FreeBSD and other versions or an easy to use spins like PCBSD (now TrueOS) I used to be Windows/Mac user for so many years until 2014 when I decided to use Linux as my desktop OS just because I wanted to use something different. I always wanted to use unix or a unix-like operating system, nowadays my main objective is to learn more about these operating systems (Debian Linux, TrueOS or FreeBSD). FreeBSD has similarities with Linux, with two major differences in scope and licensing: FreeBSD maintains a complete operating system, i.e. the project delivers kernel, device drivers, userland utilities and documentation, as opposed to Linux delivering a kernel and drivers only and relying on third-parties for system software; and FreeBSD source code is generally released under a permissive BSD license as opposed to the copyleft GPL used by Linux.“ But why do I call FreeBSD “The Unknown Giant”?, because the code base of this operating system has been used by other companies to develop their own operating system for products like computers or also game consoles. + FreeBSD is used for storage appliances, firewalls, email scanners, network scanners, network security appliances, load balancers, video servers, and more So many people now will learn that not only “linux is everywhere” but also that “FreeBSD is everywhere too” By the way speaking about movies, Do you remember the movie “The Matrix”? FreeBSD was used to make the movie: “The photo-realistic surroundings generated by this method were incorporated into the bullet time scene, and linear interpolation filled in any gaps of the still images to produce a fluent dynamic motion; the computer-generated “lead in” and “lead out” slides were filled in between frames in sequence to get an illusion of orbiting the scene. Manex Visual Effects used a cluster farm running the Unix-like operating system FreeBSD to render many of the film's visual effects” + FreeBSD Press Release re: The Matrix (https://www.freebsd.org/news/press-rel-1.html) I hope that I gave a good reference, information and now so many people can understand why I am going to use just Debian Linux and FreeBSD(TrueOS) to do so many different stuff (music, 3d animation, video editing and text editing) instead use a Mac or Windows. + FreeBSD really is the unknown giant. OpenBSD and FreeDOS vs the hell in earth (https://steemit.com/openbsd/@npna/openbsd-and-freedos-vs-the-hell-in-earth) Yes sir, yes. Our family, composed until now by OpenBSD, Alpine Linux and Docker is rapidly growing. And yes, sir. Yes. All together we're fighting against your best friends, the infamous, the ugliest, the worst...the dudes called the privacy cannibals. Do you know what i mean, sure? We're working hard, no matter what time is it, no matter in what part in the world we are, no matter if we've no money. We perfectly know that you cannot do nothing against the true. And we're doing our best to expand our true, our doors are opened to all the good guys, there's a lot here but their brain was fucked by your shit tv, your fake news, your laws, etc etc etc. We're alive, we're here to fight against you. Tonight, yes it's a Friday night and we're working, we're ready to welcome with open arms an old guy, his experience will give us more power. Welcome to: FreeDOS But why we want to build a bootable usb stick with FreeDOS under our strong OpenBSD? The answer is as usual to fight against the privacy cannibals! More than one decade ago the old BIOS was silently replaced by the more capable and advanced UEFI, this is absolutely normal because of the pass of the years and exponencial grow of the power of our personal computers. UEFI is a complex system, it's like a standalone system operative with direct access to every component of our (yes, it's our not your!) machine. But...wait a moment...do you know how to use it? Do you ever know that it exist? And one more thing, it's secure? The answer to this question is totally insane, no, it's not secure. The idea is good, the company that started in theory is one of the most important in IT, it's Intel. The history is very large and obviously we're going to go very deep in it, but trust me UEFI and the various friend of him, like ME, TPM are insecure and closed source! Like the hell in earth. A FreeDOS bootable usb image under OpenBSD But let's start preparing our OpenBSD to put order in this chaos: $ mkdir -p freedos/stuff $ cd freedos/stuff $ wget https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/fdboot.img $ wget https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/dos/sys/sys-freedos-linux/sys-freedos-linux.zip $ wget https://download.lenovo.com/consumer/desktop/o35jy19usa_y900.exe $ wget http://145.130.102.57/domoticx/software/amiflasher/AFUDOS%20Flasher%205.05.04.7z Explanation in clear language as usual: create two directory, download the minimal boot disc image of FreeDOS, download Syslinux assembler MBR bootloaders, download the last Windows only UEFI update from Lenovo and download the relative unknown utility from AMI to flash our motherboard UEFI chipset. Go ahead: $ doas pkg_add -U nasm unzip dosfstools cabextract p7zip nasm the Netwide Assembler, a portable 80x86 assembler. unzip list, test and extract compressed files in a ZIP archive. dosfstoolsa collections of utilities to manipulate MS-DOSfs. cabextract program to extract files from cabinet. p7zipcollection of utilities to manipulate 7zip archives. $ mkdir sys-freedos-linux && cd sys-freedos-linux $ unzip ../sys-freedos-linux.zip $ cd ~/freedos && mkdir old new $ dd if=/dev/null of=freedos.img bs=1024 seek=20480 $ mkfs.fat freedos.img Create another working directory, cd into it, unzip the archive that we've downloaded, return to the working root and create another twos directories. dd is one of the most important utilities in the unix world to manipulate at byte level input and output: The dd utility copies the standard input to the standard output, applying any specified conversions. Input data is read and written in 512-byte blocks. If input reads are short, input from multiple reads are aggregated to form the output block. When finished, dd displays the number of complete and partial input and output blocks and truncated input records to the standard error output. We're creating here a virtual disk with bs=1024 we're setting both input and output block to 1024bytes; with seek=20480 we require 20480bytes. This is the result: -rw-r--r-- 1 taglio taglio 20971520 Feb 3 00:11 freedos.img. Next we format the virtual disk using the MS-DOS filesystem. Go ahead: $ doas su $ perl stuff/sys-freedos-linux/sys-freedos.pl --disk=freedos.img $ vnconfig vnd0 stuff/fdboot.img $ vnconfig vnd1 freedos.img $ mount -t msdos /dev/vnd0c old/ $ mount -t msdos /dev/vnd1c new/ We use the perl utility from syslinux to write the MBR of our virtual disk freedos.img. Next we create to loop virtual node using the OpenBSD utility vnconfig. Take care here because it is quite different from Linux, but as usual is clear and simple. The virtual nodes are associated to the downloaded fdboot.img and the newly created freedos.img. Next we mount the two virtual nodes cpartitions; in OpenBSD cpartition describes the entire physical disk. Quite different from Linux, take care. $ cp -R old/* new/ $ cd stuff $ mkdir o35jy19usa $ cabextract -d o35jy19usa o35jy19usa_y900.exe $ doas su $ cp o35jy19usa/ ../new/ $ mkdir afudos && cd afudos $ 7z e ../AFUDOS* $ doas su $ cp AFUDOS.exe ../../new/ $ umount ~/freedos/old/ && umount ~/freedos/new/ $ vnconfig -u vnd1 && vnconfig -u vnd0 Copy all files and directories in the new virtual node partition, extract the Lenovo cabinet in a new directory, copy the result in our new image, extract the afudos utility and like the others copy it. Umount the partitions and destroy the loop vnode. Beastie Bits NetBSD - A modern operating system for your retro battlestation (https://www.geeklan.co.uk/files/fosdem2018-retro) FOSDEM OS distribution (https://twitter.com/pvaneynd/status/960181163578019840/photo/1) Update on two pledge-related changes (https://marc.info/?l=openbsd-tech&m=151268831628549) *execpromises (https://marc.info/?l=openbsd-cvs&m=151304116010721&w=2) Slides for (BSD from scratch - from source to OS with ease on NetBSD) (https://www.geeklan.co.uk/files/fosdem2018-bsd/) Goobyte LastPass: You're fired! (https://blog.crashed.org/goodbye-lastpass/) *** Feedback/Questions Scott - ZFS Mirror with SLOG (http://dpaste.com/22Z8C6Z#wrap) Troels - Question about compressed ARC (http://dpaste.com/3X2R1BV#wrap) Jeff - FreeBSD Desktop DNS (http://dpaste.com/2BQ9HFB#wrap) Jonathon - Bhyve and gpu passthrough (http://dpaste.com/0TTT0DB#wrap) ***
GSoC 2018 Projects announced, tutorial FreeBSD jails with iocage, new Code of Conduct for FreeBSD, libhijack, and fancy monitoring for OpenSMTPD This episode was brought to you by Headlines Google Summer of Code 2018 (https://summerofcode.withgoogle.com/organizations/?sp-page=5) FreeBSD (https://www.freebsd.org/projects/summerofcode.html) FreeBSD Google Summer oF Code Ideas (https://wiki.freebsd.org/SummerOfCodeIdeas) You can join #freebsd-soc on the efnet IRC network to chat with FreeBSD developers interested in mentoring student proposals and projects, past FreeBSD/GSoC students, and other students applying to FreeBSD/GSoC this year. NetBSD (https://mail-index.netbsd.org/netbsd-advocacy/2018/02/12/msg000765.html) You can get a stipend (paid for by Google) and spend a few months getting to know and improving the insides of NetBSD or pkgsrc. ``` The schedule is: 12-27 March Applying 23 April Find out if you were accepted 14 May - 22 August Do the project! We have some suggestions for suitable projects: - ARM EFI bootloader - Using libFuzzer on base tools - Refactoring ALTQ (QoS implementation) and integrating with NPF - Testsuite for libcurses - Improve pkgin Other suggestions and details are at: https://wiki.netbsd.org/projects/gsoc/ ``` These projects are suggestions; you can come up with your own. Suggestions for other suitable projects are welcome. Feel free to contact, or chat around on IRC: irc.freenode.org #netbsd #netbsd-code #pkgsrc Haiku (https://summerofcode.withgoogle.com/organizations/4821756754264064/) Students: How to Apply for a Haiku Idea (https://www.haiku-os.org/community/gsoc/2018/students) Project Ideas (https://www.haiku-os.org/community/gsoc/2018/ideas) > If you have questions you can contact the devs on IRC: irc.freenode.org #haiku FreeBSD Jails with iocage (http://norrist.devio.us/iocage_freebsd.html) Introduction FreeBSD jails allow users to run multiple, isolated instances of FreeBSD on a single server. Iocage simplifies the management of FreeBSD Jails. Following this tutorial, the jails will be configured to bind to an IP address on the jail host's internal network, and the host OS will pass traffic from the external network to the jail. The jails will be managed with Iocage. Iocage uses ZFS properties to store configuration data for each jail, so a ZFS file system is required. Network setup These steps will: Set up the internal network. Enable the pf packet filter Configure pf pass internet traffic to and from the jail. PF is full featured firewall, and can do more than just pass traffic to an internal network. Refer to the PF documentation for additional configuration options. Run the following to configure the internal network and enable pf. sysrc cloned_interfaces+="lo1" sysrc ifconfig_lo1="inet 192.0.2.1/24" sysrc pf_enable="YES" Put the following in /etc/pf.conf ``` Variables ext_if should be set to the hosts external NIC extif = "vtnet0" jailif = "lo1" jailnet = $jailif:network NAT allows the jails to access the external network nat on $extif from $jailnet to any -> ($ext_if) Redirect traffic on port 80 to the web server jail Add similar rules for additional jails rdr pass on $ext_if inet proto tcp to port 80 -> 192.0.2.10 ``` Reboot to activate the network changes ZFS The best way to use ZFS on a VPS is to attach block storage as a new disk. If block storage is not available, you can optionally use a file as the ZFS device. Enable and start ZFS. sysrc zfs_enable="YES" service zfs start ZFS using Block storage List the available disks. If you are using a VPS, the block store will probably be the second disk. geom disk list Create a ZFS pool named jailstore. zpool create jailstore /dev/vtbd1 ZFS using a file Create the ZFS file. dd if=/dev/zero of=/zfsfile bs=1M count=4096 Create a ZFS pool named jailstore. zpool create jailstore /zfsfile Install iocage the easy way pkg install py36-iocage Skip to "Using iocage" Install iocage the hard way Swap file Smaller servers may not have enough RAM to build iocage. If needed, create a swap file and reboot. dd if=/dev/zero of=/swapfile bs=1M count=1024 echo 'swapfile="/swapfile"' >> /etc/rc.conf reboot Install some build dependencies pkg install subversion python36 git-lite libgit2 py36-pip Building iocage requires the FreeBSD source. svn checkout https://svn.freebsd.org/base/releng/11.1 /usr/src Get the latest FreeBSD ports tree. ``` portsnap fetch portsnap extract ``` + build iocage. cd /usr/ports/sysutils/iocage/ make install Using iocage ``` iocage activate jailstore iocage fetch iocage create -n www ip4_addr="lo1|192.0.2.10/24" -r 11.1-RELEASE iocage start www iocage console www ``` Once you have a shell inside the jail, install and start Apache. pkg install apache24 sysrc apache24_enable="yes" service apache24 start Port 80 on the jail will now be accessible on the hosts IP address. Multiple jails. Additional jails can be installed using the example above. Install the new jail with the iocage create command , but use a different IP address Expose the new jail to the network by adding additional rules to pf.conf. iXsystems SNIA Persistent Memory Summit 2018 Report (https://www.ixsystems.com/blog/snia-report-2018/) New FreeBSD Code of Conduct (https://www.freebsd.org/internal/code-of-conduct.html) The FreeBSD Project is inclusive. We want the FreeBSD Project to be a venue where people of all backgrounds can work together to make the best operating system, built by a strong community. These values extend beyond just development to all aspects of the Project. All those given recognition as members of the Project in whatever form are seen as ambassadors of the Project. Diversity is a huge strength and is critical to the long term success of the Project. To that end we have a few ground rules that we ask people to adhere to. This code applies equally to everyone representing the FreeBSD Project in any way, from new members, to committers, to the core team itself. These rules are intended to ensure a safe, harassment-free environment for all and to ensure that everyone feels welcome both working within, and interacting with, the Project. This document is not an exhaustive list of things that you should not do. Rather, consider it a guide to make it easier to enrich all of us and the technical communities in which we participate. This code of conduct applies to all spaces used by the FreeBSD Project, including our mailing lists, IRC channels, and social media, both online and off. Anyone who is found to violate this code of conduct may be sanctioned or expelled from FreeBSD Project controlled spaces at the discretion of the FreeBSD Code of Conduct Committee. Some FreeBSD Project spaces may have additional rules in place, which will be made clearly available to participants. Participants are responsible for knowing and abiding by these rules. Harassment includes but is not limited to: + Comments that reinforce systemic oppression related to gender, gender identity and expression, sexual orientation, disability, mental illness, neurodiversity, physical appearance, body size, age, race, or religion. + Unwelcome comments regarding a person's lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment. + Deliberate misgendering. + Deliberate use of "dead" or rejected names. + Gratuitous or off-topic sexual images or behaviour in spaces where they're not appropriate. + Physical contact and simulated physical contact (e.g., textual descriptions like "hug" or "backrub") without consent or after a request to stop. + Threats of violence. + Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm. + Deliberate intimidation. + Stalking or following. + Harassing photography or recording, including logging online activity for harassment purposes. + Sustained disruption of discussion. + Unwelcome sexual attention. + Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others. + Continued one-on-one communication after requests to cease. + Deliberate "outing" of any private aspect of a person's identity without their consent except as necessary to protect vulnerable people from intentional abuse. + Publication of non-harassing private communication without consent. + Publication of non-harassing private communication with consent but in a way that intentionally misrepresents the communication (e.g., removes context that changes the meaning). + Knowingly making harmful false claims about a person. Interview - Benno Rice - benno@freebsd.org (mailto:benno@freebsd.org) / @jeamland (https://twitter.com/jeamland) News Roundup libhijack in PoC||GTFO 0x17! (https://www.soldierx.com/news/libhijack-PoCGTFO-0x17) Hijacking Your Free Beasties In the land of red devils known as Beasties exists a system devoid of meaningful exploit mitigations. As we explore this vast land of opportunity, we will meet our ELFish friends, [p]tracing their very moves in order to hijack them. Since unprivileged process debugging is enabled by default on FreeBSD, we can abuse PTrace to create anonymous memory mappings, inject code into them, and overwrite PLT/GOT entries. We will revive a tool called libhijack to make our nefarious activities of hijacking ELFs via PTrace relatively easy. Nothing presented here is technically new. However, this type of work has not been documented in this much detail, tying it all into one cohesive work. In Phrack 56, Silvio Cesare taught us ELF research enthusiasts how to hook the PLT/GOT. The Phrack 59 article on Runtime Process Infection briefly introduces the concept of injecting shared objects by injecting shellcode via PTrace that calls dlopen(). No other piece of research, however, has discovered the joys of forcing the application to create anonymous memory mappings in which to inject Code. This is only part one of a series of planned articles that will follow libhijack's development. The end goal is to be able to anonymously inject shared objects. The libhijack project is maintained by the SoldierX community. Previous Research All prior work injects code into the stack, the heap, or existing executable code. All three methods create issues on today's systems. On amd64 and arm64, the two architectures libhijack cares about, the stack is non-executable by default. jemalloc, the heap implementation on FreeBSD, creates non-executable mappings. Obviously overwriting existing executable code destroys a part of the executable image. The Role of ELF > FreeBSD provides a nifty API for inspecting the entire virtual memory space of an application. The results returned from the API tells us the protection flags (readable, writable, executable) of each mapping. If FreeBSD provides such a rich API, why would we need to parse the ELF headers? PLT/GOT hijacking requires parsing ELF headers. One would not be able to find the PLT/GOT without iterating through the Process Headers to find the Dynamic Headers, eventually ending up with the DT_PLTGOT entry. With FreeBSD's libprocstat API, we don't have a need for parsing ELF headers until we get to the PLT/GOT stage, but doing so early makes it easier for the attacker using libhijack The Future of libhijack Writing devious code in assembly is cumbersome. Assembly doesn't scale well to multiple architectures. Instead, we would like to write our devious code in C, compiling to a shared object that gets injected anonymously. This requires writing a remote RTLD within libhijack and is in progress. Writing a remote RTLD will take a while as doing so is not an easy task. Additionally, creation of a general-purpose helper library that gets injected would be helpful. It could aid in PLT/GOT redirection attacks, possibly storing the addresses of functions we've previously hijacked. This work is dependent on the remote RTLD. libhijack currently lacks documentation. Once the ABI and API stabilize, formal documentation will be written. Conclusion Using libhijack, we can easily create anonymous memory mappings, inject into them arbitrary code, and hijack the PLT/GOT on FreeBSD. On HardenedBSD, a hardened derivative of FreeBSD, libhijack is fully mitigated through PaX NOEXEC. We've demonstrated that wrapper-style Capsicum is ineffective on FreeBSD. Through the use of libhijack, we emulate a control flow hijack in which the application is forced to call sandbox_open and fdlopen on the resulting file descriptor. Further work to support anonymous injection of full shared objects, along with their dependencies, will be supported in the future. Imagine injecting libpcap into Apache to sniff traffic whenever "GET /pcap" is sent. In order to prevent abuse of PTrace, FreeBSD should set the security.bsd.unprivilegedprocdebug to 0 by default. In order to prevent process manipulation, FreeBSD should implement PaX NOEXEC. libhijack can be found at https://github.com/SoldierX/libhijack Introduction to POSIX shell (https://sircmpwn.github.io/2018/02/05/Introduction-to-POSIX-shell.html) What the heck is the POSIX shell anyway? Well, the POSIX (the Portable Operating System Interface) shell is the standard Unix shell - standard meaning it was formally defined and shipped in a published standard. This makes shell scripts written for it portable, something no other shell can lay claim to. The POSIX shell is basically a formalized version of the venerable Bourne shell, and on your system it lives at /bin/sh, unless you're one of the unlucky masses for whom this is a symlink to bash. Why use POSIX shell? The “Bourne Again shell”, aka bash, is not standardized. Its grammar, features, and behavior aren't formally written up anywhere, and only one implementation of bash exists. Without a standard, bash is defined by its implementation. POSIX shell, on the other hand, has many competing implementations on many different operating systems - all of which are compatible with each other because they conform to the standard. Any shell that utilizes features specific to Bash are not portable, which means you cannot take them with you to any other system. Many Linux-based systems do not use Bash or GNU coreutils. Outside of Linux, pretty much everyone but Hurd does not ship GNU tools, including bash1. On any of these systems, scripts using “bashisms” will not work. This is bad if your users wish to utilize your software anywhere other than GNU/Linux. If your build tooling utilizes bashisms, your software will not build on anything but GNU/Linux. If you ship runtime scripts that use bashisms, your software will not run on anything but GNU/Linux. The case for sticking to POSIX shell in shipping software is compelling, but I argue that you should stick to POSIX shell for your personal scripts, too. You might not care now, but when you feel like flirting with other Unicies you'll thank me when all of your scripts work. One place where POSIX shell does not shine is for interactive use - a place where I think bash sucks, too. Any shell you want to use for your day-to-day command line work is okay in my book. I use fish. Use whatever you like interactively, but stick to POSIX sh for your scripts. How do I use POSIX shell? At the top of your scripts, put #!/bin/sh. You don't have to worry about using env here like you might have been trained to do with bash: /bin/sh is the standardized location for the POSIX shell, and any standards-conforming system will either put it there or make your script work anyway. The next step is to avoid bashisms. There are many, but here are a few that might trip you up: [[ condition ]] does not work; use [ condition ] Arrays do not work; use IFS Local variables do not work; use a subshell The easiest way to learn about POSIX shell is to read the standard - it's not too dry and shorter than you think. Using standard coreutils The last step to writing portable scripts is to use portable tools. Your system may have GNU coreutils installed, which provides tools like grep and cut. Unfortunately, GNU has extended these tools with its own non-portable flags and tools. It's important that you avoid these. One dead giveaway of a non-portable flag is long flags, e.g. grep --file=FILE as opposed to grep -f. The POSIX standard only defines the getopt function - not the proprietary GNU getopt_long function that's used to interpret long options. As a result, no long flags are standardized. You might worry that this will make your scripts difficult to understand, but I think that on the whole it will not. Shell scripts are already pretty alien and require some knowledge to understand. Is knowledge of what the magic word grep means much different from knowledge of what grep -E means? I also like that short flags allow you to make more concise command lines. Which is better: ps --all --format=user --without-tty, or ps -aux? If you are inclined to think the former, do you also prefer function(a, b, c) { return a + b + c; } over (a, b, c) => a + b + c? Conciseness matters, and POSIX shell supports comments if necessary! Some tips for using short flags: They can be collapsed: cmd -a -b -c is equivalent to cmd -abc If they take additional arguments, either a space or no separation is acceptable: cmd -f"hello world" or cmd -f "hello world" A good reference for learning about standardized commands is, once again, the standard. From this page, search for the command you want, or navigate through “Shell & Utilities” -> “Utilities” for a list. If you have man-pages installed, you will also find POSIX man pages installed on your system with the p postfix, such as man 1p grep. Note: at the time of writing, the POSIX man pages do not use dashes if your locale is UTF-8, which makes searching for flags with / difficult. Use env LC_ALL=POSIX man 1p grep if you need to search for flags, and I'll speak to the maintainer of man-pages about this. FreeBSD Broadcom Wi-Fi Improvements (http://landonf.org/code/freebsd/Broadcom_WiFi_Improvements.20180122.html) Introduction Since 2015, I've been working on improving FreeBSD support for Broadcom Wi-Fi devices and SoCs, including authoring the bhnd(4) driver family, which provides a unified bus and driver programming interface for these devices. First committed in early 2016, bhnd(4) allowed us to quickly bring up FreeBSD/MIPS on Broadcom SoCs, but it has taken much longer to implement the full set of features required to support modern Broadcom SoftMAC Wi-Fi hardware. Thanks to the generosity of the FreeBSD Foundation, I've recently finished implementing the necessary improvements to the bhnd(4) driver family. With these changes in place, I was finally able to port the existing bwn(4) Broadcom SoftMAC Wi-Fi driver to the bhnd(4) bus, and implement initial support for the BCM43224 and BCM43225 chipsets, with additional hardware support to be forthcoming. Now that my efforts on FreeBSD/Broadcom Wi-Fi support have progressed far enough to be generally useful, I wanted to take some time to provide a brief overview of Broadcom's Wi-Fi hardware, and explain how my work provides a foundation for further FreeBSD Broadcom Wi-Fi/SoC improvements. A Brief Background on Broadcom Wi-Fi Hardware Broadcom's Wi-Fi devices are members of the Broadcom Home Networking Division (BHND) device family; other BHND devices include MIPS/ARM SoCs (including Wi-Fi SoCs commonly found in consumer access points), as well as a large variety of related networking hardware. BHND devices utilize a common set of Broadcom IP cores (or "functional blocks") connected via one of two on-chip bus architectures: Hardware designed prior to 2009 used Broadcom's “SSB” backplane architecture, based on Sonics Silicon's interconnect IP. Subsequent hardware adopted Broadcom's “BCMA” backplane, based on ARM's AMBA IP. The IP cores used in earlier SSB-based devices were adapted for compatibility with the new backplane. When BHND hardware is used in a PCI Wi-Fi card, or a SDIO Wi-Fi module, the device's dual-mode peripheral controller is configured to operate as an endpoint device on the host's peripheral bus, bridging access to the SoC hardware: Host access to SoC address space is provided via a set of register windows (e.g., a set of configurable windows into SoC address space mapped via PCI BARs) DMA is supported by the bridge core's sparse mapping of host address space into the backplane address space. These address regions may be used as a target for the on-chip DMA engines. Any backplane interrupt vectors routed to the bridge core may be mapped by the bridge to host interrupts (e.g., PCI INTx/MSI/MSI-X). The host is generally expected to provide drivers for the IP cores found on the SoC backplane; since these cores are found in both BHND SoCs and BHND Wi-Fi devices, it is advantageous to share driver and platform code between the two targets. Modernizing FreeBSD's Broadcom SoftMAC Wi-Fi Support FreeBSD support for Broadcom SoftMAC Wi-Fi adapters is provided by two partially overlapping PCI/CardBus drivers: Legacy Wi-Fi adapters are supported by bwi(4). This driver remains in-tree to support devices incompatible with v4 or later firmware (e.g. BCM4301, BCM4302, BCM4306 rev 1-2), all of which were released prior to December 2002. Modern Wi-Fi adapters are supported by bwn(4), with access to on-chip cores mediated by bhnd(4). Prior to my work porting bwn(4) to bhnd(4), access to on-chip cores was mediated by sibabwn, a PCI/WiFi-specific derivative of the legacy siba(4) SSB bus driver. There were two major limitations to sibabwn that have long blocked adding support for newer SoftMAC Wi-Fi chipsets: the newer BCMA interconnect found in post-2009 hardware was not supported by siba(4), and siba_bwn assumed a PCI/PCIe bridge, preventing its use on FreeBSD/MIPS Broadcom SoCs with interconnect-attached D11 cores. The new bhnd(4) driver family, written as a replacement for siba(4) and siba_bwn, provides: A unified bus driver interface for both SSB and BCMA on-chip interconnects A generic BHND bridge driver framework for host-connected BHND devices (e.g. Wi-Fi adapters, etc) A PCI/PCIe bridge core driver, for PCI-attached BHND devices. An abstract BHND NVRAM API, with support for the varied NVRAM formats found in BHND Wi-Fi adapters and SoCs. Drivers for common BHND platform peripherals (UARTs, SPROM/flash, PMUs, etc) By porting bwn(4) to bhnd(4), we are now able to support existing BCMA devices with MAC/PHY/Radio combinations readily supported by bwn(4), as was the case with the BCM43224 and BCM43225 chipsets. This also opens the door to porting additional PHY support from Broadcom's ISC-licensed Linux drivers, and will allow us to bring up bwn(4) on Broadcom WiSoCs supported by FreeBSD/MIPS. Monitor OpenSMTPD using Logstash and Grafana (https://www.tumfatig.net/20180129/monitor-opensmtpd-using-logstash-grafana/) Logs are usefull. Graphs are sexy. Here's a way to get a view on what happens to your OpenSMTPD traffic, using Web v2.0 tools ; namely Logstash & Grafana. For those who would not be aware of those tools, logstash is some kind of log-parser that can eat syslog formatted logs and write them into elasticsearch ; in “document” format. Grafana is a Web frontend that can dig into various databases and render graphics from requests. I won't go into the whole “how to install” process here. Installation is quite straight forward and online documentation is quite clear. What you need OpenSMTPD deals with emails and logs its activity via Syslog. Syslog is configured to send the logs to Logstash. Logstash has a set of rules configured to transform the text-oriented information into searchable document-oriented data. The transformed data is stored into Elasticsearch. Elasticsearch provides Web API to search and find stuff. Grafana connects to ELS to get data and draw the graphs. Beastie Bits CharmBUG Presentation - Writing FreeBSD Malware (https://www.meetup.com/CharmBUG/events/247995596/) March London *BSD meeting 13/03/18 (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2018-February/014180.html) FreBSD Ports Workshop (https://wiki.freebsd.org/MateuszPiotrowski/Ports/Workshop) The history of NetBSD/atari and support for ATARI compatible Milan / OSC2018Osaka (https://speakerdeck.com/tsutsui/osc2018osaka) SSH Mastery, 2nd Edition (https://www.tiltedwindmillpress.com/?product=ssh-mastery-2nd-edition) *** Feedback/Questions Stephen - Viewer Interview Question (http://dpaste.com/06WTRB9#wrap) pb - trust expanding your 280TB pool (http://dpaste.com/0TZV6CM#wrap) Tim - ZFS questions for the ZFS Man (http://dpaste.com/0759X1E#wrap) Daniel - ZFS full backup question (http://dpaste.com/1SJXSBQ#wrap) ***
We explain the physics behind ZFS, DTrace switching to the GPL, Emacs debugging, syncookies coming to PF & FreeBSD's history on EC2. This episode was brought to you by Headlines 128 bit storage: Are you high? (https://blogs.oracle.com/bonwick/128-bit-storage:-are-you-high) For people who have heard about ZFS boiling oceans and wonder where that is coming from, we dug out this old piece from 2004 on the blog of ZFS co-creator Jeff Bonwick, originally from the Sun website. 64 bits would have been plenty ... but then you can't talk out of your ass about boiling oceans then, can you? Well, it's a fair question. Why did we make ZFS a 128-bit storage system? What on earth made us think it's necessary? And how do we know it's sufficient? Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 2^64 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. If 64 bits isn't enough, the next logical step is 128 bits. That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. But it does raise the question: what are the theoretical limits to storage capacity? Although we'd all like Moore's Law to continue forever, quantum mechanics imposes some fundamental limits on the computation rate and information capacity of any physical device. In particular, it has been shown that 1 kilogram of matter confined to 1 liter of space can perform at most 10^51 operations per second on at most 10^31 bits of information [see Seth Lloyd, "Ultimate physical limits to computation." Nature 406, 1047-1054 (2000)]. A fully-populated 128-bit storage pool would contain 2^128 blocks = 2^137 bytes = 2^140 bits; therefore the minimum mass required to hold the bits would be (2^140 bits) / (10^31 bits/kg) = 136 billion kg. That's a lot of gear. To operate at the 1031 bits/kg limit, however, the entire mass of the computer must be in the form of pure energy. By E=mc^2, the rest energy of 136 billion kg is 1.2x1028 J. The mass of the oceans is about 1.4x1021 kg. It takes about 4,000 J to raise the temperature of 1 kg of water by 1 degree Celcius, and thus about 400,000 J to heat 1 kg of water from freezing to boiling. The latent heat of vaporization adds another 2 million J/kg. Thus the energy required to boil the oceans is about 2.4x106 J/kg * 1.4x1021 kg = 3.4x1027 J. Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. Best part of all: you don't have to understand any of this to use ZFS. Rest assured that you won't hit any limits with that filesystem for a long time. You still have to buy bigger disks over time, though... *** dtrace for Linux, Oracle relicenses dtrace (https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-linux-oracle-does-the-right-thing/) At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom. Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port: ``` commit e1744f50ee9bc1978d41db7cc93bcf30687853e6 Author: Tomas Jedlicka tomas.jedlicka@oracle.com Date: Tue Aug 1 09:15:44 2017 -0400 dtrace: Integrate DTrace Modules into kernel proper This changeset integrates DTrace module sources into the main kernel source tree under the GPLv2 license. Sources have been moved to appropriate locations in the kernel tree. ``` That is right, dtrace dropped the CDDL and switched to the GPL! The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out). The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory. The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL. Thank you Oracle for making everyone's life easier by waving your magic relicensing wand! Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest. Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don't always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all. Debugging Emacs or: How I Learned to Stop Worrying and Love DTrace (http://nullprogram.com/blog/2018/01/17/) For some time Elfeed was experiencing a strange, spurious failure. Every so often users were seeing an error (spoiler warning) when updating feeds: “error in process sentinel: Search failed.” If you use Elfeed, you might have even seen this yourself. From the surface it appeared that curl, tasked with the responsibility for downloading feed data, was producing incomplete output despite reporting a successful run. Since the run was successful, Elfeed assumed certain data was in curl's output buffer, but, since it wasn't, it failed hard. Unfortunately this issue was not reproducible. Manually running curl outside of Emacs never revealed any issues. Asking Elfeed to retry fetching the feeds would work fine. The issue would only randomly rear its head when Elfeed was fetching many feeds in parallel, under stress. By the time the error was discovered, the curl process had exited and vital debugging information was lost. Considering that this was likely to be a bug in Emacs itself, there really wasn't a reliable way to capture the necessary debugging information from within Emacs Lisp. And, indeed, this later proved to be the case. A quick-and-dirty work around is to use condition-case to catch and swallow the error. When the bizarre issue shows up, rather than fail badly in front of the user, Elfeed could attempt to swallow the error — assuming it can be reliably detected — and treat the fetch as simply a failure. That didn't sit comfortably with me. Elfeed had done its due diligence checking for errors already. Someone was lying to Elfeed, and I intended to catch them with their pants on fire. Someday. I'd just need to witness the bug on one of my own machines. Elfeed is part of my daily routine, so surely I'd have to experience this issue myself someday. My plan was, should that day come, to run a modified Elfeed, instrumented to capture extra data. I would have also routinely run Emacs under GDB so that I could inspect the failure more deeply. For now I just had to wait to hunt that zebra. Bryan Cantrill, DTrace, and FreeBSD Over the holidays I re-discovered Bryan Cantrill, a systems software engineer who worked for Sun between 1996 and 2010, and is most well known for DTrace. My first exposure to him was in a BSD Now interview in 2015. I had re-watched that interview and decided there was a lot more I had to learn from him. He's become a personal hero to me. So I scoured the internet for more of his writing and talks. Some interesting operating system technology came out of Sun during its final 15 or so years — most notably DTrace and ZFS — and Bryan speaks about it passionately. Almost as a matter of luck, most of it survived the Oracle acquisition thanks to Sun releasing it as open source in just the nick of time. Otherwise it would have been lost forever. The scattered ex-Sun employees, still passionate about their prior work at Sun, along with some of their old customers have since picked up the pieces and kept going as a community under the name illumos. It's like an open source flotilla. Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn't have these Sun technologies available. The main reason is license incompatibility. Sun released its code under the CDDL, which is incompatible with the GPL. Ubuntu does infamously include ZFS, but other distributions are unwilling to take that risk. Porting DTrace is a serious undertaking since it's got its fingers throughout the kernel, which also makes the licensing issues even more complicated. Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that. Rather than adopt ZFS and DTrace, they've been reinvented from scratch: btrfs instead of ZFS, and a slew of partial options instead of DTrace. Normally I'm most interested in system call tracing, and my go to is strace, though it certainly has its limitations — including this situation of debugging curl under Emacs. Another famous example of NIH is Linux's epoll(2), which is a broken version of BSD kqueue(2). So, if I want to try these for myself, I'll need to install a different operating system. I've dabbled with OmniOS, an OS built on illumos, in virtual machines, using it as an alien environment to test some of my software (e.g. enchive). OmniOS has a philosophy called Keep Your Software To Yourself (KYSTY), which is really just code for “we don't do packaging.” Honestly, you can't blame them since they're a tiny community. The best solution to this is probably pkgsrc, which is essentially a universal packaging system. Otherwise you're on your own. There's also openindiana, which is a more friendly desktop-oriented illumos distribution. Still, the short of it is that you're very much on your own when things don't work. The situation is like running Linux a couple decades ago, when it was still difficult to do. If you're interested in trying DTrace, the easiest option these days is probably FreeBSD. It's got a big, active community, thorough documentation, and a huge selection of packages. Its license (the BSD license, duh) is compatible with the CDDL, so both ZFS and DTrace have been ported to FreeBSD. What is DTrace? I've done all this talking but haven't yet described what DTrace really is. I won't pretend to write my own tutorial, but I'll provide enough information to follow along. DTrace is a tracing framework for debugging production systems in real time, both for the kernel and for applications. The “production systems” part means it's stable and safe — using DTrace won't put your system at risk of crashing or damaging data. The “real time” part means it has little impact on performance. You can use DTrace on live, active systems with little impact. Both of these core design principles are vital for troubleshooting those really tricky bugs that only show up in production. There are DTrace probes scattered all throughout the system: on system calls, scheduler events, networking events, process events, signals, virtual memory events, etc. Using a specialized language called D (unrelated to the general purpose programming language D), you can dynamically add behavior at these instrumentation points. Generally the behavior is to capture information, but it can also manipulate the event being traced. Each probe is fully identified by a 4-tuple delimited by colons: provider, module, function, and probe name. An empty element denotes a sort of wildcard. For example, syscall::open:entry is a probe at the beginning (i.e. “entry”) of open(2). syscall:::entry matches all system call entry probes. Unlike strace on Linux which monitors a specific process, DTrace applies to the entire system when active. To run curl under strace from Emacs, I'd have to modify Emacs' behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That's a big deal. So, when it comes to this Elfeed issue, FreeBSD is much better poised for debugging the problem. All I have to do is catch it in the act. However, it's been months since that bug report and I'm not really making this connection yet. I'm just hoping I eventually find an interesting problem where I can apply DTrace. Bryan Cantrill: Talks I have given (http://dtrace.org/blogs/bmc/2018/02/03/talks/) *** News Roundup a2k18 Hackathon preview: Syncookies coming to PF (https://undeadly.org/cgi?action=article;sid=20180207090000) As you may have heard, the a2k18 hackathon is in progress. As can be seen from the commit messages, several items of goodness are being worked on. One eagerly anticipated item is the arrival of TCP syncookies (read: another important tool in your anti-DDoS toolset) in PF. Henning Brauer (henning@) added the code in a series of commits on February 6th, 2018, with this one containing the explanation: ``` syncookies for pf. when syncookies are on, pf will blindly answer each and every SYN with a syncookie-SYNACK. Upon reception of the ACK completing the 3WHS, pf will reconstruct the original SYN, shove it through pf_test, where state will be created if the ruleset permits it. Then massage the freshly created state (we won't see the SYNACK), set up the sequence number modulator, and call into the existing synproxy code to start the 3WHS with the backend host. Add an - somewhat basic for now - adaptive mode where syncookies get enabled if a certain percentage of the state table is filled up with half-open tcp connections. This makes pf firewalls resilient against large synflood attacks. syncookies are off by default until we gained more experience, considered experimental for now. see http://bulabula.org/papers/2017/bsdcan/ for more details. joint work with sashan@, widely discussed and with lots of input by many ``` The first release to have this feature available will probably be the upcoming OpenBSD 6.3 if a sufficient number of people test this in their setups (hint, hint). More info is likely to emerge soon in post-hackathon writeups, so watch this space! [Pale Moon] A Perfect example of how not to approach OS developers/packagers Removed from OpenBSD Ports due to Licensing Issues (https://github.com/jasperla/openbsd-wip/issues/86) FreeBSD Palemoon branding violation (https://lists.freebsd.org/pipermail/freebsd-ports/2018-February/112455.html) Mightnight BSD's response (https://twitter.com/midnightbsd/status/961232422091280386) *** FreeBSD EC2 History (http://www.daemonology.net/blog/2018-02-12-FreeBSD-EC2-history.html) A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years. Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments. August 25, 2006: Amazon EC2 launches. It supports a single version of Ubuntu Linux; FreeBSD is not available. December 13, 2010: I manage to get FreeBSD running on EC2 t1.micro instances. March 22, 2011: I manage to get FreeBSD running on EC2 "cluster compute" instances. July 8, 2011: I get FreeBSD 8.2 running on all 64-bit EC2 instance types, by marking it as "Windows" in order to get access to Xen/HVM virtualization. (Unfortunately this meant that users had to pay the higher "Windows" hourly pricing.) January 16, 2012: I get FreeBSD 9.0 running on 32-bit EC2 instances via the same "defenestration" trick. (Again, paying the "Windows" prices.) August 16, 2012: I move the FreeBSD rc.d scripts which handle "EC2" functionality (e.g., logging SSH host keys to the console) into the FreeBSD ports tree. October 7, 2012: I rework the build process for FreeBSD 9.1-RC1 and later to use "world" bits extracted from the release ISOs; only the kernel is custom-built. Also, the default SSH user changes from "root" to "ec2-user". October 31, 2012: Amazon launches the "M3" family of instances, which support Xen/HVM without FreeBSD needing to pay the "Windows" tax. November 21, 2012: I get FreeBSD added to the AWS Marketplace. October 2, 2013: I finish merging kernel patches into the FreeBSD base system, and rework the AMI build (again) so that FreeBSD 10.0-ALPHA4 and later use bits extracted from the release ISOs for the entire system (world + kernel). FreeBSD Update can now be used for updating everything (because now FreeBSD/EC2 uses a GENERIC kernel). October 27, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA2 and later AMIs will run FreeBSD Update when they first boot in order to download and install any critical updates. December 1, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA4 and later AMIs bootstrap the pkg tool and install packages at boot time (by default, the "awscli" package). December 9, 2013: I add configinit to FreeBSD 10.0-RC1 and later to allow systems to be easily configured via EC2 user-data. July 1, 2014: Amazon launches the "T2" family of instances; now the most modern family for every type of EC2 instance (regular, high-memory, high-CPU, high-I/O, burstable) supports HVM and there should no longer be any need for FreeBSD users to pay the "Windows tax". November 24, 2014: I add code to FreeBSD 10.2 and later to automatically resize their root filesystems when they first boot; this means that a larger root disk can be specified at instance launch time and everything will work as expected. April 1, 2015: I integrate the FreeBSD/EC2 build process into the FreeBSD release building process; FreeBSD 10.2-BETA1 and later AMIs are built by the FreeBSD release engineering team. January 12, 2016: I enable Intel 82599-based "first generation EC2 Enhanced Networking" in FreeBSD 11.0 and later. June 9, 2016: I enable the new EC2 VGA console functionality in FreeBSD 11.0 and later. (The old serial console also continues to work.) June 24, 2016: Intel 82599-based Enhanced Networking works reliably in FreeBSD 11.0 and later thanks to discovering and working around a Xen bug. June 29, 2016: I improve throughput on Xen blkfront devices (/dev/xbd*) by enabling indirect segment I/Os in FreeBSD 10.4 and later. (I wrote this functionality in July 2015, but left it disabled by default a first because a bug in EC2 caused it to hurt performance on some instances.) July 7, 2016: I fix a bug in FreeBSD's virtual memory initialization in order to allow it to support boot with 128 CPUs; aka. FreeBSD 11.0 and later support the EC2 x1.32xlarge instance type. January 26, 2017: I change the default configuration in FreeBSD 11.1 and later to support EC2's IPv6 networking setup out of the box (once you flip all of the necessary switches to enable IPv6 in EC2 itself). May 20, 2017: In collaboration with Rick Macklem, I make FreeBSD 11.1 and later compatible with the Amazon "Elastic File System" (aka. NFSv4-as-a-service) via the newly added "oneopenown" mount option (and lots of bug fixes). May 25, 2017: I enable support for the Amazon "Elastic Network Adapter" in FreeBSD 11.1 and later. (The vast majority of the work — porting the driver code — was done by Semihalf with sponsorship from Amazon.) December 5, 2017: I change the default configuration in FreeBSD 11.2 and later to make use of the Amazon Time Sync Service (aka. NTP-as-a-service). The current status The upcoming FreeBSD release (11.2) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial VGA), and every EC2 instance type (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances). Colin's Patreon' page if you'd like to support him (https://www.patreon.com/cperciva) X network transparency X's network transparency has wound up mostly being a failure (https://utcc.utoronto.ca/~cks/space/blog/unix/XNetworkTransparencyFailure) I was recently reading Mark Dominus's entry about some X keyboard problems, in which he said in passing (quoting himself): I have been wondering for years if X's vaunted network transparency was as big a failure as it seemed: an interesting idea, worth trying out, but one that eventually turned out to be more trouble than it was worth. [...] My first reaction was to bristle, because I use X's network transparency all of the time at work. I have several programs to make it work very smoothly, and some core portions of my environment would be basically impossible without it. But there's a big qualification on my use of X's network transparency, namely that it's essentially all for text. When I occasionally go outside of this all-text environment of xterms and emacs and so on, it doesn't go as well. X's network transparency was not designed as 'it will run xterm well'; originally it was to be something that should let you run almost everything remotely, providing a full environment. Even apart from the practical issues covered in Daniel Stone's slide presentation, it's clear that it's been years since X could deliver a real first class environment over the network. You cannot operate with X over the network in the same way that you do locally. Trying to do so is painful and involves many things that either don't work at all or perform so badly that you don't want to use them. In my view, there are two things that did in general X network transparency. The first is that networks turned out to not be fast enough even for ordinary things that people wanted to do, at least not the way that X used them. The obvious case is web browsers; once the web moved to lots of images and worse, video, that was pretty much it, especially with 24-bit colour. (It's obviously not impossible to deliver video across the network with good performance, since YouTube and everyone else does it. But their video is highly encoded in specialized formats, not handled by any sort of general 'send successive images to the display' system.) The second is that the communication facilities that X provided were too narrow and limited. This forced people to go outside of them in order to do all sorts of things, starting with audio and moving on to things like DBus and other ways of coordinating environments, handling sophisticated configuration systems, modern fonts, and so on. When people designed these additional communication protocols, the result generally wasn't something that could be used over the network (especially not without a bunch of setup work that you had to do in addition to remote X). Basic X clients that use X properties for everything may be genuinely network transparent, but there are very few of those left these days. (Not even xterm is any more, at least if you use XFT fonts. XFT fonts are rendered in the client, and so different hosts may have different renderings of the same thing, cf.) < What remains of X's network transparency is still useful to some of us, but it's only a shadow of what the original design aimed for. I don't think it was a mistake for X to specifically design it in (to the extent that they did, which is less than you might think), and it did help X out pragmatically in the days of X terminals, but that's mostly it. (I continue to think that remote display protocols are useful in general, but I'm in an usual situation. Most people only ever interact with remote machines with either text mode SSH or a browser talking to a web server on the remote machine.) PS: The X protocol issues with synchronous requests that Daniel Stone talks about don't help the situation, but I think that even with those edges sanded off X's network transparency wouldn't be a success. Arguably X's protocol model committed a lesser version of part of the NeWS mistake. X's network transparency was basically free at the time (https://utcc.utoronto.ca/~cks/space/blog/unix/XFreeNetworkTransparency) I recently wrote an entry about how X's network transparency has wound up mostly being a failure for various reasons. However, there is an important flipside to the story of X's network transparency, and that is that X's network transparency was almost free at the time and in the context it was created. Unlike the situation today, in the beginning X did not have to give up lots of performance or other things in order to get network transparency. X originated in the mid 1980s and it was explicitly created to be portable across various Unixes, especially BSD-derived ones (because those were what universities were mostly using at that time). In the mid to late 1980s, Unix had very few IPC methods, especially portable ones. In particular, BSD systems did not have shared memory (it was called 'System V IPC' for the obvious reasons). BSD had TCP and Unix sockets, some System V machines had TCP (and you could likely assume that more would get it), and in general your safest bet was to assume some sort of abstract stream protocol and then allow for switchable concrete backends. Unsurprisingly, this is exactly what X did; the core protocol is defined as a bidirectional stream of bytes over an abstracted channel. (And the concrete implementation of $DISPLAY has always let you specify the transport mechanism, as well as allowing your local system to pick the best mechanism it has.) Once you've decided that your protocol has to run over abstracted streams, it's not that much more work to make it network transparent (TCP provides streams, after all). X could have refused to make the byte order of the stream clear or required the server and the client to have access to some shared files (eg for fonts), but I don't think either would have been a particularly big win. I'm sure that it took some extra effort and care to make X work across TCP from a different machine, but I don't think it took very much. (At the same time, my explanation here is probably a bit ahistorical. X's initial development seems relatively strongly tied to sometimes having clients on different machines than the display, which is not unreasonable for the era. But it doesn't hurt to get a feature that you want anyway for a low cost.) I believe it's important here that X was intended to be portable across different Unixes. If you don't care about portability and can get changes made to your Unix, you can do better (for example, you can add some sort of shared memory or process to process virtual memory transfer). I'm not sure how the 1980s versions of SunView worked, but I believe they were very SunOS dependent. Wikipedia says SunView was partly implemented in the kernel, which is certainly one way to both share memory and speed things up. PS: Sharing memory through mmap() and friends was years in the future at this point and required significant changes when it arrived. Beastie Bits Grace Hopper Celebration 2018 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/grace-hopper-celebration-2018-call-for-participation/) Google Summer of Code: Call for Project Ideas (https://www.freebsdfoundation.org/blog/google-summer-of-code-call-for-project-ideas/) The OpenBSD Foundation 2018 Fundraising Campaign (https://undeadly.org/cgi?action=article;sid=20180129190641) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) AsiaBSDcon 2018 Registration is open (https://2018.asiabsdcon.org/) Tarsnap support for Bitcoin ending April 1st; and a Chrome bug (http://mail.tarsnap.com/tarsnap-announce/msg00042.html) Feedback/Questions Todd - Couple Questions (http://dpaste.com/195HGHY#wrap) Seth - Tar Snap (http://dpaste.com/1N7NQVQ#wrap) Alex - sudo question (http://dpaste.com/3D9P1DW#wrap) Thomas - FreeBSD on ARM? (http://dpaste.com/24NMG47#wrap) Albert - Austria BSD User Group (http://dpaste.com/373CRX7#wrap)
We talk about our recent trip to FOSDEM, we discuss the pros and cons of permissive licensing, cover the installation of OpenBSD on a dedibox with full-disk encryption, the new Lumina guide repository, and we explain ZFS vs. OpenZFS. This episode was brought to you by Headlines [FOSDEM Trip report] Your BSDNow hosts were both at FOSDEM in Brussels, Belgium over the weekend. On the friday before FOSDEM, we held a FreeBSD devsummit (3rd consecutive year), sponsored by the FreeBSD Foundation and organized by Benedict (with the help from Kristof Provost, who did it in previous years but could not make it this year). We had 21 people attend, a good mixture of FreeBSD committers (mostly ports) and guests. After introductions, we collected topics and discussed various topics, including a new plan for a future FreeBSD release roadmap (more frequent releases, so that features from HEAD can be tried out earlier in RELEASES). The devsummit concluded with a nice dinner in a nearby restaurant. On Saturday, first day of FOSDEM, we set up the FreeBSD Foundation table with flyers, stickers, FreeBSD Journal print editions, and a small RPI 3 demo system that Deb Goodkin brought. Our table was located next to the Illumos table like last year. This allowed us to continue the good relationship that we have with the Illumos people and Allan helped a little bit getting bhyve to run on Illumos with UEFI. Meanwhile, our table was visited by a lot of people who would ask questions about FreeBSD, take info material, or talk about their use cases. We were busy refilling the table throughout the day and luckily, we had many helpers at the table. Some items we had ran out in the early afternoon, an indicator of how popular they were. Saturday also featured a BSD devroom (https://twitter.com/fosdembsd), organized by Rodrigo Osorio. You can find the list of talks and the recordings on the BSD Devroom schedule (https://fosdem.org/2018/schedule/track/bsd/). The room was very crowded and popular. Deb Goodkin gave the opening talk with an overview of what the Foundation is doing to change the world. Other speakers from various BSD projects presented their talks after that with a range of topics. Among them, Allan gave his talk about ZFS: Advanced Integration (https://fosdem.org/2018/schedule/event/zfs_advanced_integration/), while Benedict presented his Reflections on Teaching a Unix Class With FreeBSD (https://fosdem.org/2018/schedule/event/reflections_on_reaching_unix_class_with_freebsd/). Sunday was just as busy on the FreeBSD table as Saturday and we finally ran out of stickers and some other goodies. We were happy with the results of the two days. Some very interesting conversations at the table about FreeBSD took place, some of which we're going to follow up afterwards. Check out the FOSDEM schedule as many talk recordings are already available, and especially the ones from the BSD devroom if you could not attend the conference. We would like to thank everyone who attended the FreeBSD devsummit, who helped out at the FreeBSD table and organized the BSD devroom. Also, thanks to all the speakers, organizers, and helping hands making FOSDEM another success this year. *** NetBSD kernel wscons IOCTL vulnerable bug class (http://blog.infosectcbr.com.au/2018/01/netbsd-kernel-wscons-ioctl-vulnerable.html) I discovered this bug class during the InfoSect public code review session we ran looking specifically at the NetBSD kernel. I found a couple of these bugs and then after the session was complete, I went back and realised the same bug was scattered in other drivers. In total, 17 instances of this vulnerability and its variants were discovered. In all fairness, I came across this bug class during my kernel audits in 2002 and most instances were patched. It just seems there are more bugs now in NetBSD while OpenBSD and FreeBSD have practically eliminated them. See slide 41 in http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf (http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf) for exactly the same bug (class) 16 years ago. The format of the this blog post is as follows: Introduction Example of the Bug Class How to Fix How to Detect Automatically with Coccinelle More Bugs Conclusion These source files had bugs ./dev/tc/tfb.c ./dev/ic/bt485.c ./dev/pci/radeonfb.c ./dev/ic/sti.c ./dev/sbus/tcx.c ./dev/tc/mfb.c ./dev/tc/sfb.c ./dev/tc/stic.c ./dev/tc/cfb.c ./dev/tc/xcfb.c ./dev/tc/sfbplus.c ./arch/arm/allwinner/awin_debe.c ./arch/arm/iomd/vidcvideo.c ./arch/pmax/ibus/pm.c ./dev/ic/igfsb.c ./dev/ic/bt463.c ./arch/luna68k/dev/lunafb.c Reporting of the bugs was easy. In less than a week from reporting the specific instances of each bug, patches were committed into the mainline kernel. Thanks to Luke Mewburn from NetBSD for coming to the code review session at InfoSect and coordinating with the NetBSD security team. The patches to fix these issues are in NetBSD: https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html (https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html) "Permissive licensing is wrong!” – Is it? (https://eerielinux.wordpress.com/2017/11/25/permissive-licensing-is-wrong-is-it-1-2/) A few weeks ago I've been attacked by some GNU zealots on a German tech site after speaking in favor of permissive licenses. Unfortunately a discussion was not possible there because that would require the will to actually communicate instead of simply accusing the other side of vile motives. Since I actually do care about this topic and a reader asked for a post about it in comments a while ago, here we go. This first part tries to sum up the most important things around the topic. I deliberately aim for an objective overview that tries not to be one-sided. The second part will then contain my points in defence of permissive licensing. Why license software at all? Licenses exist for reasons of protection. If you're the author/inventor of some software, a story or whatever product, you get to decide what to do with it. You can keep it for yourself or you can give it away. If you decide for the latter, you have to decide who may use it and in which way(s). In case you intend to give it to a (potentially) large group of people, you may not want to be asked for permission to xyz by everybody. That's when you decide to write a license which states what you are allowing and explicitly disallowing. Most of the well-known commercial licenses focus on what you're not allowed to do (usually things like copying, disassembling, etc.). Open source licenses on the other hand are meant to grant the user rights (e.g. the right to distribute) while reserving some rights or only giving permission under certain conditions – and they usually make you claim responsibility for using the software. For these reasons licenses can actually be a good thing! If you got an unlicensed piece of code, you're not legally allowed to do anything with it without getting the author's permission first. And even if you got that permission, your project would be risky, since the author can withdraw it later. A proper license protects both parties. The author doesn't get his mail account full of email asking for permission, he's save from legal trouble if his code breaks anything for you and at the same time you have legal certainty when you decide to put the code to long-term use. Permissive vs. Copyleft (in a nutshell) In short terms, permissive licensing usually goes like this: “Here you are, have fun. Oh, and don't sue me if it does something else than what you expect!” Yes, it's that easy and there's little to dispute over. Copyleft on the other side sounds like this (if you ask somebody in favor of Copyleft): “Sure, you can use it, it's free. Just keep it free, ok?”. Also quite simple. And not too bad, eh? Other people however read the same thing like this: “Yes, you're free to use it. Just read these ten pages of legalese and be dead certain that you comply. If you got something wrong, we will absolutely make you regret it.” The GNU Public license (GPL) The most popular copyleft license in use is the GPL (in various versions) (https://www.gnu.org/licenses/gpl.html). It got more and more complex with each version – and to be fair, it had to, because it was necessary to react to new threats and loop holes that were found later. The GNU project states that they are committed to protect what they call the four freedoms of free software: the freedom to use the software for any purpose the freedom to change the software to suit your needs the freedom to share the software with your friends and neighbors the freedom to share the changes you make These are freedoms that every supporter of open source software should be able to agree with. So what's the deal with all the hostility and fighting between the two camps? Let's take a look at a permissive license, too. The BSD license Unlike the GPL, the BSD family of licenses begun with a rather simple license that span four rules (“original BSD license”). It was later revised and reduced to three (“modified BSD license”). And the modern BSD license that e.g. FreeBSD uses is even just two (“simplified BSD license”). Did you read the GPLv3 that I linked to above? If you are using GPL'd code you really should. In case you don't feel like reading all of it, at least take a look and grasp how long that text is. Now compare it to the complete modern BSD license (https://opensource.org/licenses/bsd-license.php). What's the problem? There are essentially two problems that cause all the trouble. The first one is the question of what should be subject to the freedom that we're talking about. And closely related, the second one is where that freedom needs to end. Ironically both camps claim that freedom is the one important thing and it must not be restricted. The GPL is meant to protect the freedom of the software and enforces the availability of the source code, hence limiting the freedom of actual persons. BSD on the other hand is meant to protect the freedom of human beings who should be able to use the software as they see fit – even if that means closing down former open source code! The GNU camp taunts permissive licenses as being “lax” for not providing the protection that they want. The other camp points out that the GPL is a complex monster and that it is virulent in nature: Since it's very strict in a lot of areas, it's incompatible with many other licenses. This makes it complicated to mix GPL and non-GPL code and in the cases where it's legally possible, the GPL's terms will take precedence and necessarily be in effect for the whole combined work. Who's right? That totally depends on what you want to achieve. There are pros and cons to both – and in fact we're only looking at the big picture here. There's also e.g. the Apache license which is often deemed as kind of middle ground. Then you may want to consider the difference between weak (e.g. LGPL) as well as strong copyleft (GPL). Licensing is a potentially huge topic. But let's keep it simple here because the exact details are actually not necessary to understand the essence of our topic. In the next post I'll present my stance on why permissive licensing is a good thing and copyleft is more problematic than many people may think. “Permissive licensing is wrong?” – No it's not! (https://eerielinux.wordpress.com/2018/01/25/permissive-licensing-is-wrong-no-its-not-2-2/) The previous post gave a short introduction into the topic of software licenses, focusing on the GPL vs. BSD discussion. This one is basically my response to some typical arguments I've seen from people who seem to loathe permissive licensing. I'll write this in dialog style, hoping that this makes it a little lighter to read. Roundup Install OpenBSD on dedibox with full-disk encryption (https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/) TL;DR: I run several "dedibox" servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server OpenBSD is not officially supported, I can't guarantee that this will work for you on any kind of server online.net provides, however I've been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up. Currently, I'm running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them. Recently I've been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references. I wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks. The method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice. Step #1: boot to rescue mode Step #2: boot to the installer Step #3: prepare softraid Step #4: reboot to encrypted OpenBSD system Bonus: further tightening your system enable doas disable the root account update system with syspatch add my ssh public key to my ~/.ssh/authorized_keys disable password authentication within ssh reboot so you boot on a brand new up-to-date system with latest stable kernel VOILA ! January 2018 Development Projects Update (https://www.freebsdfoundation.org/blog/january-2018-development-projects-update/) Spectre and Meltdown in FreeBSD Issues affecting most CPUs used in servers, desktops, laptops, and mobile devices are in the news. These hardware vulnerabilities, known by the code-names “Meltdown” and “Spectre”, allow malicious programs to read data to which they should not have access. This potentially includes credentials, cryptographic material, or other secrets. They were originally identified by a researcher from Google's Project Zero, and were also independently discovered by researchers and academics from Cyberus Technology, Graz University of Technology, the University of Pennsylvania, the University of Maryland, Rambus, the University of Adelaide and Data61. These vulnerabilities affect many CPU architectures supported by FreeBSD, but the 64-bit x86 family of processors from Intel and AMD are the most widely used, and are a high priority for software changes to mitigate the effects of Meltdown and Spectre. In particular, the Meltdown issue affects Intel CPUs and may be used to extract secret data from the running kernel, and therefore, is the most important issue to address. The FreeBSD Foundation collaborates with Intel, and under this relationship participated in a briefing to understand the details of these issues and plan the mitigations to be applied to the x86 architectures supported by FreeBSD. We also made arrangements to have FreeBSD's security officer join me in the briefing. It is through the generous support of the Foundation's donors that we are able to dedicate resources to focus on these issues on demand as they arise. Foundation staff member Konstantin (Kostik) Belousov is an expert on FreeBSD's Virtual Memory (VM) system as well as low-level x86 details, and is developing the x86 kernel mitigations for FreeBSD. The mitigation for Meltdown is known as Page Table Isolation (PTI). Kostik created a PTI implementation which was initially committed in mid-January and is available in the FreeBSD-CURRENT development repository. This is the same approach used by the Linux kernel to mitigate Meltdown. One of the drawbacks of the PTI mitigation is that it incurs a performance regression. Kostik recently reworked FreeBSD's use of Process-Context Identifiers (PCID) in order to regain some of the performance loss incurred by PTI. This change is also now available in FreeBSD-CURRENT. The issue known as Spectre comes in two variants, and variant 2 is the more troubling and pressing one. It may be mitigated in one of two ways: by using a technique called “retpoline” in the compiler, or by making use of a CPU feature introduced in a processor microcode update. Both options are under active development. Kostik's change to implement the CPU-based mitigation is currently in review. Unfortunately, it introduces a significant performance penalty and alternatives are preferred, if available. For most cases, the compiler-based retpoline mitigation is likely to be the chosen mitigation. Having switched to the Clang compiler for the base system and most of the ports collection some years ago, FreeBSD is well-positioned to deploy Clang-based mitigations. FreeBSD developer Dimitry Andric is spearheading the update of Clang/LLVM in FreeBSD to version 6.0 in anticipation of its official release; FreeBSD-CURRENT now includes an interim snapshot. I have been assisting with the import, particularly with respect to LLVM's lld linker, and will support the integration of retpoline. This support is expected to be merged into FreeBSD in the coming weeks. The Foundation's co-op students have also participated in the response to these vulnerabilities. Mitchell Horne developed the patch to control the PTI mitigation default setting, while Arshan Khanifar benchmarked the performance impact of the in-progress mitigation patches. In addition, Arshan and Mitchell each developed changes to FreeBSD's tool chain to support the full set of mitigations that will be applied. These mitigations will continue be tested, benchmarked, and refined in FreeBSD-CURRENT before being merged into stable branches and then being made available as updates to FreeBSD releases. Details on the timing of these merges and releases will be shared as they become available. I would like to acknowledge all of those in the FreeBSD community who have participated in FreeBSD's response to Meltdown and Spectre, for testing, reviewing, and coordinating x86 mitigations, for developing mitigations for other processor architectures and for the Bhyve hypervisor, and for working on the toolchain-based mitigations. Guides: Getting Started & Lumina Theme Submissions (https://lumina-desktop.org/guides-getting-started-lumina-themes/) I am pleased to announce the beginning of a new sub-series of blog posts for the Lumina project: Guides! The TrueOS/Lumina projects want to support our users as they use Lumina or experiment with TrueOS. To that end, we've recently set up a central repository for our users to share instructions or other “how-to” guides with each other! Project developers and contributors will also submit guides to the repository on occasion, but the overall goal is to provide a simple hub for instructions written by any Lumina or TrueOS user. This will make it easier for users to not only find a “how-to” for some procedure, but also a very easy way to “give back” to the community by writing simple instructions or more detailed guides. Guides Repository Our first guide to get the whole thing started was created by the TrueOS Linebacker (https://discourse.trueos.org/t/introducing-the-trueos-linebacker/991) (with technical assistance from our own q5sys). In this guide, Terry Tate will walk you through the steps necessary to submit new wallpaper images to the Lumina Themes collection. This procedure is fully documented with screenshots every step of the way, walking you through a simple procedure that only requires a web browser and a Github account! Guide: Lumina Themes Submissions (https://github.com/trueos/guides/blob/master/lumina-themes-submissions/readme.md) The end result of this guide was that Terry Tate was able to submit this cool new “Lunar-4K” wallpaper to the “lumina-nature” collection. TrueOS Community Guides (https://github.com/trueos/guides/tree/master) ZFS vs. OpenZFS (by Michael Dexter) (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You've probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. Our Senior Analyst clears up what ZFS and OpenZFS refer to and how they differ. I admit that we geeks tend to get caught up in the nuts and bolts of enterprise storage and overlook the more obvious questions that users might have. You've probably noticed that this blog and the FreeNAS blog refer to “ZFS” and “OpenZFS” seemingly at random when talking about the amazing file system at the heart of FreeNAS and every storage product that iXsystems sells. I will do my best to clarify what exactly these two terms refer to. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset' and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Yes, ZFS is really as good as people say it is. After enjoying nearly a decade of refinement by a growing group of developers around the world, ZFS became the property of database vendor Oracle, which ceased public development of both ZFS and OpenSolaris in 2010. Disappointed but undeterred, a group of OpenSolaris users and developers forked the last public release of OpenSolaris as the Illumos project. To this day, Illumos represents the official upstream home of the Open Source OpenSolaris technologies, including ZFS. The Illumos project enjoys healthy vendor and user participation but the portable nature and compelling features of ZFS soon produced far more ZFS users than Illumos users around the world. While most if not all users of Illumos and its derivatives are ZFS users, the majority of ZFS users are not Illumos users, thanks significantly in part to FreeNAS which uses the FreeBSD operating system. This imbalance plus several successful ZFS Day events led ZFS co-founder Matt Ahrens and a group of ZFS developers to announce the OpenZFS project, which would remain a part of the Illumos code base but would be free to coordinate development efforts and events around their favorite file system. ZFS Day has grown into the two-day OpenZFS Developer Summit and is stronger than ever, a testament to the passion and dedication of the OpenZFS community. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs' command, which on FreeBSD relies on the ‘zfs.ko' kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. Beastie Bits Explaining Shell (https://explainshell.com/) OPNsense® 18.1 Released (https://opnsense.org/opnsense-18-1-released/) “SSH Mastery 2/e” copyedits back (https://blather.michaelwlucas.com/archives/3104) Sponsoring a Scam (https://blather.michaelwlucas.com/archives/3106) Thursday, February 8, 2018 - Come to Netflix to talk about FreeBSD (https://www.meetup.com/BAFUG-Bay-Area-FreeBSD-User-Group/events/246623825/) BSD User Group meeting in Stockholm: March 22, 17:30 - 21:00 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) FreeBSD Flavoured talks from Linux.conf.au: You can't unit test C, right? (https://www.youtube.com/watch?v=z-uWt5wVVkU) and A Brief History of I/O (https://www.youtube.com/watch?v=qAhZEI_6lbc) EuroBSDcon 2018 website is up (https://2018.eurobsdcon.org/) Full day bhyvecon Tokyo, Japan, March 9, 2018 (http://bhyvecon.org/) *** Feedback/Questions Thomas - freebsd installer improvements (http://dpaste.com/3G2F7RC#wrap) Mohammad - FreeBSD 11 installation from a read only rescue disk (http://dpaste.com/0HGK3FQ#wrap) Stan - Follow up on guide you covered (http://dpaste.com/2S169SH#wrap) Jalal - couple questions (http://dpaste.com/35N8QXP#wrap)
We cover an interview about Unix Architecture Evolution, another vBSDcon trip report, how to teach an old Unix about backspace, new NUMA support coming to FreeBSD, and stack pointer checking in OpenBSD. This episode was brought to you by Headlines Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD (https://fosdem.org/2018/interviews/diomidis-spinellis/) Q: Could you briefly introduce yourself? I'm a professor of software engineering, a programmer at heart, and a technology author. Currently I'm also the editor in chief of the IEEE Software magazine. I recently published the book Effective Debugging, where I detail 66 ways to debug software and systems. Q: What will your talk be about, exactly? I will describe how the architecture of the Unix operating system evolved over the past half century, starting from an unnamed system written in PDP-7 assembly language and ending with a modern FreeBSD system. My talk is based, first, on a GitHub repository where I tried to record the system's history from 1970 until today and, second, on the evolution of documented facilities (user commands, system calls, library functions) across revisions. I will thus present the early system's defining architectural features (layering, system calls, devices as files, an interpreter, and process management) and the important ones that followed in subsequent releases: the tree directory structure, user contributed code, I/O redirection, the shell as a user program, groups, pipes, scripting, and little languages. Q: Why this topic? Unix stands out as a major engineering breakthrough due to its exemplary design, its numerous technical contributions, its impact, its development model, and its widespread use. Furthermore, the design of the Unix programming environment has been characterized as one offering unusual simplicity, power, and elegance. Consequently, there are many lessons that we can learn by studying the evolution of the Unix architecture, which we can apply to the design of new systems. I often see modern systems that suffer from a bloat of architectural features and a lack of clear form on which functionality can be built. I believe that many of the modern Unix architecture defining features are excellent examples of what we should strive toward as system architects. Q: What do you hope to accomplish by giving this talk? What do you expect? I'd like FOSDEM attendees to leave the talk with their mind full with architectural features of timeless quality. I want them to realize that architectural elegance isn't derived by piling design patterns and does not need to be expensive in terms of resources. Rather, beautiful architecture can be achieved on an extremely modest scale. Furthermore, I want attendees to appreciate the importance of adopting flexible conventions rather than rigid enforcement mechanisms. Finally, I want to demonstrate through examples that the open source culture was part of Unix from its earliest days. Q: What are the most significant milestones in the development of Unix? The architectural development of Unix follows a path of continuous evolution, albeit at a slowing pace, so I don't see here the most important milestones. I would however define as significant milestones two key changes in the way Unix was developed. The first occurred in the late 1970s when significant activity shifted from a closely-knit team of researchers at the AT&T Bell Labs to the Computer Science Research Group in the University of California at Berkeley. This opened the system to academic contributions and growth through competitive research funding. The second took place in the late 1980s and the 1990s when Berkeley open-sourced the the code it had developed (by that time a large percentage of the system) and enthusiasts built on it to create complete open source operating system distributions: 386BSD, and then FreeBSD, NetBSD, OpenBSD, and others. Q: In which areas has the development of Unix stalled? The data I will show demonstrate that there were in the past some long periods where the number of C library functions and system calls remained mostly stable. Nowadays there is significant growth in the number of all documented facilities with the exception of file formats. I'm looking forward to a discussion regarding the meaning of these growth patterns in the Q&A session after the talk. Q: What are the core features that still link the 1970 PDP-7 system to the latest FreeBSD 11.1 release, almost half a century apart? Over the past half-century the Unix system has grown by four orders of magnitude from a few thousand lines of code to many millions. Nevertheless, looking at a 1970s architecture diagram and a current one reveals that the initial architectural blocks are still with us today. Furthermore, most system calls, user programs, and C library functions of that era have survived until today with essentially similar functionality. I've even found in modern FreeBSD some lines of code that have survived unchanged for 40 years. Q: Can we still add innovative changes to operating systems like FreeBSD without breaking the ‘Unix philosophy'? Will there be a moment where FreeBSD isn't recognizable anymore as a descendant of the 1970 PDP-7 system? There's a saying that “form liberates”. So having available a time-tested form for developing operating system functionality allows you to innovate in areas that matter rather than reinventing the wheel. Such concepts include having commands act as a filter, providing manual pages with a consistent structure, supplying build information in the form of a Makefile, installing files in a well-defined directory hierarchy, implementing filesystems with an standardized object-oriented interface, and packaging reusable functions as a library. Within this framework there's ample space for both incremental additions (think of jq, the JSON query command) and radical innovations (consider the Solaris-derived ZFS and dtrace functionality). For this reason I think that BSD and Linux systems will always be recognizable as direct or intellectual descendants of the 1970s Research Unix editions. Q: Have you enjoyed previous FOSDEM editions? Immensely! As an academic I need to attend many scientific conferences and meetings in order to present research results and interact with colleagues. This means too much time spent traveling and away from home, and a limited number of conferences I'm in the end able to attend. Nevertheless, attending FOSDEM is an easy decision due to the world-changing nature of its theme, the breadth of the topics presented, the participants' enthusiasm and energy, as well as the exemplary, very efficient conference organization. Another vBSDCon trip report we just found (https://www.weaponizedawesome.com/blog/?cat=53) We just got tipped about another trip report from vBSDCon, this time from one of the first time speakers: W. Dean Freeman Recently I had the honor of co-presenting on the internals of FreeBSD's Kernel RNG with John-Mark Gurney at the 3rd biennial vBSDCon, hosted in Reston, VA hosted by Verisign. I've been in and out of the FreeBSD community for about 20 years. As I've mentioned on here before, my first Unix encounter was FreeBSD 2.2.8 when I was in the 7th or 8th grade. However, for all that time I've never managed to get out to any of the cons. I've been to one or two BUG meetings and I've met some folks from IRC before, but nothing like this. A BSD conference is a very different experience than anything else out there. You have to try it, it is the only way to truly understand it. I'd also not had to do a stand-up presentation really since college before this. So, my first BSD con and my first time presenting rolled into one made for an interesting experience. See, he didn't say terrifying. It went very well. You should totally submit a talk for the next conference, even if it is your first. That said, it was amazing and invigorating experience. I got to meet a few big names in the FreeBSD community, discuss projects, ideas for FreeBSD, etc. I did seem to spend an unusual amount of time talking about FIPS and Common Criteria with folks, but to me that's a good sign and indicative that there is interest in working to close gaps between FreeBSD and the current requirements so that we can start getting FreeBSD and more BSD-based products into the government and start whittling away the domination of Linux (especially since Oracle has cut Solaris, SPARC and the ZFS storage appliance business units). There is nothing that can match the high bandwidth interchange of ideas in person. The internet has made all kinds of communication possible, and we use it all the time, but every once in a while, getting together in person is hugely valuable. Dean then went on to list some of the talks he found most valuable, including DTrace, Capsicum, bhyve, *BSD security tools, and Paul Vixie's talk about gets() I think the talk that really had the biggest impact on me, however, was Kyle Kneisl's talk on BSD community dynamics. One of the key points he asked was whether the things that drew us to the BSD community in the first place would be able to happen today. Obviously, I'm not a 12 or 13 year old kid anymore, but it really got me thinking. That, combined with getting face time with people I'd previously only known as screen names has recently drawn me back into participating in IRC and rejoining mailing lists (wdf on freenode. be on the lookout!) Then Dean covered some thoughts on his own talk: JMG and my talk seems to have been well received, with people paying lots of attention. I don't know what a typical number of questions is for one of these things, but on day one there weren't that many questions. We got about 5 during our question time and spent most of the rest of the day fielding questions from interested attendees. Getting a “great talk!” from GNN after coming down from the stage was probably one of the major highlights for me. I remember my first solo talk, and GNN asking the right question in the middle to get me to explain a part of it I had missed. It was very helpful. I think key to the interest in our presentation was that JMG did a good job framing a very complicated topic's importance in terms everyone could understand. It also helped that we got to drop some serious truth bombs. Final Thoughts: I met a lot of folks in person for the first time, and met some people I'd never known online before. It was a great community and I'm glad I got a chance to expand my network. Verisign were excellent hosts and they took good care of both speakers (covering airfare, rooms, etc.) and also conference attendees at large. The dinners that they hosted were quite good as well. I'm definitely interested in attending vBSDCon again and now that I've had a taste of meeting IRL with the community on scale of more than a handful, I have every intention of finally making it to BSDCan next year (I'd said it in 2017, but then moved to Texas for a new job and it wasn't going to be practical). This year for sure, though! Teaching an Almost 40-year Old UNIX about Backspace (https://virtuallyfun.com/2018/01/17/teaching_an_almost_40-year_old_unix_about_backspace/) Introduction I have been messing with the UNIX® operating system, Seventh Edition (commonly known as UNIX V7 or just V7) for a while now. V7 dates from 1979, so it's about 40 years old at this point. The last post was on V7/x86, but since I've run into various issues with it, I moved on to a proper installation of V7 on SIMH. The Internet has some really good resources on installing V7 in SIMH. Thus, I set out on my own journey on installing and using V7 a while ago, but that was remarkably uneventful. One convenience that I have been dearly missing since the switch from V7/x86 is a functioning backspace key. There seem to be multiple different definitions of backspace: BS, as in ASCII character 8 (010, 0x08, also represented as ^H), and DEL, as in ASCII character 127 (0177, 0x7F, also represented as ^?). V7 does not accept either for input by default. Instead, # is used as the erase character and @ is used as the kill character. These defaults have been there since UNIX V1. In fact, they have been “there” since Multics, where they got chosen seemingly arbitrarily. The erase character erases the character before it. The kill character kills (deletes) the whole line. For example, “ba##gooo#d” would be interpreted as “good” and “bad line@good line” would be interpreted as “good line”. There is some debate on whether BS or DEL is the correct character for terminals to send when the user presses the backspace key. However, most programs have settled on DEL today. tmux forces DEL, even if the terminal emulator sends BS, so simply changing my terminal to send BS was not an option. The change from the defaults outlined here to today's modern-day defaults occurred between 4.1BSD and 4.2BSD. enf on Hacker News has written a nice overview of the various conventions Getting the Diff For future generations as well as myself when I inevitably majorly break this installation of V7, I wanted to make a diff. However, my V7 is installed in SIMH. I am not a very intelligent man, I didn't keep backup copies of the files I'd changed. Getting data out of this emulated machine is an exercise in frustration. In the end, I printed everything on screen using cat(1) and copied that out. Then I performed a manual diff against the original source code tree because tabs got converted to spaces in the process. Then I applied the changes to clean copies that did have the tabs. And finally, I actually invoked diff(1). Closing Thoughts Figuring all this out took me a few days. Penetrating how the system is put together was surprisingly fairly hard at first, but then the difficulty curve eased up. It was an interesting exercise in some kind of “reverse engineering” and I definitely learned something about tty handling. I was, however, not pleased with using ed(1), even if I do know the basics. vi(1) is a blessing that I did not appreciate enough until recently. Had I also been unable to access recursive grep(1) on my host and scroll through the code, I would've probably given up. Writing UNIX under those kinds of editing conditions is an amazing feat. I have nothing but the greatest respect for software developers of those days. News Roundup New NUMA support coming to FreeBSD CURRENT (https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068145.html) Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem, and vmpage*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VMNUMAALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VMNUMAALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff Stack pointer checking – OpenBSD (https://marc.info/?l=openbsd-tech&m=151572838911297&w=2) Stefan (stefan@) and I have been working for a few months on this diff, with help from a few others. At every trap and system call, it checks if the stack-pointer is on a page that is marked MAPSTACK. execve() is changed to create such mappings for the process stack. Also, libpthread is taught the new MAPSTACK flag to use with mmap(). There is no corresponding system call which can set MAP_FLAG on an existing page, you can only set the flag by mapping new memory into place. That is a piece of the security model. The purpose of this change is to twart stack pivots, which apparently have gained some popularity in JIT ROP attacks. It makes it difficult to place the ROP stack in regular data memory, and then perform a system call from it. Workarounds are cumbersome, increasing the need for far more gadgetry. But also the trap case -- if any memory experiences a demand page fault, the same check will occur and potentially also kill the process. We have experimented a little with performing this check during device interrupts, but there are some locking concerns and performance may then become a concern. It'll be best to gain experience from handle of syncronous trap cases first. chrome and other applications I use run fine! I'm asking for some feedback to discover what ports this breaks, we'd like to know. Those would be ports which try to (unconventionally) create their stacks in malloc()'d memory or inside another Data structure. Most of them are probably easily fixed ... Qt 5.9 on FreeBSD (https://euroquis.nl/bobulate/?p=1768) Tobias and Raphael have spent the past month or so hammering on the Qt 5.9 branch, which has (finally!) landed in the official FreeBSD ports tree. This brings FreeBSD back up-to-date with current Qt releases and, more importantly, up-to-date with the Qt release KDE software is increasingly expecting. With Qt 5.9, the Elisa music player works, for instance (where it has run-time errors with Qt 5.7, even if it compiles). The KDE-FreeBSD CI system has had Qt 5.9 for some time already, but that was hand-compiled and jimmied into the system, rather than being a “proper” ports build. The new Qt version uses a new build system, which is one of the things that really slowed us down from a packaging perspective. Some modules have been reshuffled in the process. Some applications depending on Qt internal-private headers have been fixed along the way. The Telegram desktop client continues to be a pain in the butt that way. Following on from Qt 5.9 there has been some work in getting ready for Clang 6 support; in general the KDE and Qt stack is clean and modern C++, so it's more infrastructural tweaks than fixing code. Outside of our silo, I still see lots of wonky C++ code being fixed and plenty of confusion between pointers and integers and strings and chars and .. ugh. Speaking of ugh, I'm still planning to clean up Qt4 on ARM aarch64 for FreeBSD; this boils down to stealing suitable qatomic implementations from Arch Linux. For regular users of Qt applications on FreeBSD, there should be few to no changes required outside the regular upgrade cycle. For KDE Plasma users, note that development of the ports has changed branches; as we get closer to actually landing modern KDE bits, things have been renamed and reshuffled and mulled over so often that the old plasma5 branch wasn't really right anymore. The kde5-import branch is where it's at nowadays, and the instructions are the same: the x11/kde5 metaport will give you all the KDE Frameworks 5, KDE Plasma Desktop and modern KDE Applications you need. Adding IPv6 to an Nginx website on FreeBSD / FreshPorts (https://dan.langille.org/2018/01/13/adding-ipv6-to-an-nginx-website-on-freebsd-freshports/) FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized. There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation. All of this took about 3 hours. We are using: FreeBSD 11.1 Bind 9.9.11 nginx 1.12.2 Fallout I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses. ZFS on TrueOS: Why We Love OpenZFS (https://www.trueos.org/blog/zfs-trueos-love-openzfs/) TrueOS was the first desktop operating system to fully implement the OpenZFS (Zettabyte File System or ZFS for short) enterprise file system in a stable production environment. To fully understand why we love ZFS, we will look back to the early days of TrueOS (formerly PC-BSD). The development team had been using the UFS file system in TrueOS because of its solid track record with FreeBSD-based computer systems and its ability to check file consistency with the built-in check utility fsck. However, as computing demands increased, problems began to surface. Slow fsck file verification on large file systems, slow replication speeds, and inconsistency in data integrity while using UFS logging / journaling began to hinder users. It quickly became apparent that TrueOS users would need a file system that scales with evolving enterprise storage needs, offers the best data protection, and works just as well on a hobbyist system or desktop computer. Kris Moore, the founder of the TrueOS project, first heard about OpenZFS in 2007 from chatter on the FreeBSD mailing lists. In 2008, the TrueOS development team was thrilled to learn that the FreeBSD Project had ported ZFS. At the time, ZFS was still unproven as a graphical desktop solution, but Kris saw a perfect opportunity to offer ZFS as a cutting-edge file system option in the TrueOS installer, allowing the TrueOS project to act as an indicator of how OpenZFS would fair in real-world production use. The team was blown away by the reception and quality of OpenZFS on FreeBSD-based systems. By its nature, ZFS is a copy-on-write (CoW) file system that won't move a block of data until it both writes the data and verifies its integrity. This is very different from most other file systems in use today. ZFS is able to assure that data stays consistent between writes by automatically comparing write checksums, which mitigates bit rot. ZFS also comes with native RaidZ functionality that allows for enterprise data management and redundancy without the need for expensive traditional RAID cards. ZFS snapshots allow for system configuration backups in a split-second. You read that right. TrueOS can backup or restore snapshots in less than a second using the ZFS file system. Given these advantages, the TrueOS team decided to use ZFS as its exclusive file system starting in 2013, and we haven't looked back since. ZFS offers TrueOS users the stable workstation experience they want, while simultaneously scaling to meet the increasing demands of the enterprise storage market. TrueOS users are frequently commenting on how easy it is to use ZFS snapshots with our built-in snapshot utility. This allows users the freedom to experiment with their system knowing they can restore it in seconds if anything goes wrong. If you haven't had a chance to try ZFS with TrueOS, browse to our download page and make sure to grab a copy of TrueOS. You'll be blown away by the ease of use, data protection functionality, and incredible flexibility of RaidZ. Beastie Bits Source Code Podcast Interview with Michael W Lucas (https://blather.michaelwlucas.com/archives/3099) Operating System of the Year 2017: NetBSD Third place (https://w3techs.com/blog/entry/web_technologies_of_the_year_2017) OPNsense 18.1-RC1 released (https://opnsense.org/opnsense-18-1-rc1-released/) Personal OpenBSD Wiki Notes (https://balu-wiki.readthedocs.io/en/latest/security/openbsd.html) BSD section can use some contribution (https://guide.freecodecamp.org/bsd-os/) The Third Research Edition Unix Programmer's Manual (now available in PDF) (https://github.com/dspinellis/unix-v3man) Feedback/Questions Alex - my first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) Brian - Brian - openbsd 6.2 and enlightenment .17 (http://dpaste.com/24DYF1J#wrap) ***
We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)
We review Meltdown and Spectre responses from various BSD projects, show you how to run CentOS with bhyve, GhostBSD 11.1 is out, and we look at the case against the fork syscall. This episode was brought to you by Headlines More Meltdown Much has been happened this week, but before we get into a status update of the various mitigations on the other BSDs, some important updates: Intel has recalled the microcode update they issued on January 8th. It turns out this update can cause Haswell and Broadwell based systems to randomly reboot, with some frequency. (https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/) AMD has confirmed that its processors are vulnerable to both variants of Spectre, and the the fix for variant #2 will require a forthcoming microcode update, in addition to OS level mitigations (https://www.amd.com/en/corporate/speculative-execution) Fujitsu has provided a status report for most of its products, including SPARC hardware (https://sp.ts.fujitsu.com/dmsp/Publications/public/Intel-Side-Channel-Analysis-Method-Security-Review-CVE2017-5715-vulnerability-Fujitsu-products.pdf) The Register of course has some commentary (https://www.theregister.co.uk/2018/01/12/intel_warns_meltdown_spectre_fixes_make_broadwells_haswells_unstable/) If new code is needed, Intel will need to get it right: the company already faces numerous class action lawsuits. Data centre operators already scrambling to conduct unplanned maintenance will not be happy about the fix reducing stability. AMD has said that operating system patches alone will address the Spectre bounds check bypass bug. Fixing Spectre's branch target injection flaw will require firmware fixes that AMD has said will start to arrive for Ryzen and EPYC CPUs this week. The Register has also asked other server vendors how they're addressing the bugs. Oracle has patched its Linux, but has told us it has “No comment/statement on this as of now” in response to our query about its x86 systems, x86 cloud, Linux and Solaris on x86. The no comment regarding Linux is odd as fixes for Oracle Linux landed here (https://linux.oracle.com/errata/ELSA-2018-4006.html) on January 9th. SPARC-using Fujitsu, meanwhile, has published advice (PDF) revealing how it will address the twin bugs in its servers and PCs, and also saying its SPARC systems are “under investigation”. Response from OpenBSD: (https://undeadly.org/cgi?action=article;sid=20180106082238) 'Meltdown, aka "Dear Intel, you suck"' (https://marc.info/?t=151521438600001&r=1&w=2) Theo de Raadt's response to Meltdown (https://www.itwire.com/security/81338-handling-of-cpu-bug-disclosure-incredibly-bad-openbsd-s-de-raadt.html) That time in 2007 when Theo talked about how Intel x86 had major design problems in their chips (https://marc.info/?l=openbsd-misc&m=118296441702631&w=2) OpenBSD gets a Microcode updater (https://marc.info/?l=openbsd-cvs&m=151570987406841&w=2) Response from Dragonfly BSD: (http://lists.dragonflybsd.org/pipermail/users/2018-January/313758.html) The longer response in four commits One (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627151.html) Two (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627152.html) Three (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627153.html) Four (http://lists.dragonflybsd.org/pipermail/commits/2018-January/627154.html) Even more Meltdown (https://www.dragonflydigest.com/2018/01/10/20718.html) DragonflyBSD master now has full IBRS and IBPB support (http://lists.dragonflybsd.org/pipermail/users/2018-January/335643.html) IBRS (Indirect Branch Restricted Speculation): The x86 IBRS feature requires corresponding microcode support. It mitigates the variant 2 vulnerability. If IBRS is set, near returns and near indirect jumps/calls will not allow their predicted target address to be controlled by code that executed in a less privileged prediction mode before the IBRS mode was last written with a value of 1 or on another logical processor so long as all RSB entries from the previous less privileged prediction mode are overwritten. Speculation on Skylake and later requires these patches ("dynamic IBRS") be used instead of retpoline. If you are very paranoid or you run on a CPU where IBRS=1 is cheaper, you may also want to run in "IBRS always" mode. IBPB (Indirect Branch Prediction Barrier): Setting of IBPB ensures that earlier code's behavior does not control later indirect branch predictions. It is used when context switching to new untrusted address space. Unlike IBRS, IBPB is a command MSR and does not retain its state. DragonFlyBSD's Meltdown Fix Causing More Slowdowns Than Linux (https://www.phoronix.com/scan.php?page=article&item=dragonfly-bsd-meltdown&num=1) NetBSD HOTPATCH() (http://mail-index.netbsd.org/source-changes/2018/01/07/msg090945.html) NetBSD SVS (Separate Virtual Space) (http://mail-index.netbsd.org/source-changes/2018/01/07/msg090952.html) Running CentOS with Bhyve (https://www.daemon-security.com/2018/01/bhyve-centos-0110.html) With the addition of UEFI in FreeBSD (since version 11), users of bhyve can use the UEFI boot loader instead of the grub2-bhyve port for booting operating systems such as Microsoft Windows, Linux and OpenBSD. The following page provides information necessary for setting up bhyve with UEFI boot loader support: https://wiki.freebsd.org/bhyve/UEFI Features have been added to vmrun.sh to make it easier to setup the UEFI boot loader, but the following is required to install the UEFI firmware pkg: # pkg install -y uefi-edk2-bhyve With graphical support, you can use a vnc client like tigervnc, which can be installed with the following command: # pkg install -y tigervnc In the case of most corporate or government environments, the Linux of choice is RHEL, or CentOS. Utilizing bhyve, you can test and install CentOS in a bhyve VM the same way you would deploy a Linux VM in production. The first step is to download the CentOS iso (for this tutorial I used the CentOS minimal ISO): http://isoredirect.centos.org/centos/7/isos/x8664/CentOS-7-x8664-Minimal-1708.iso I normally use a ZFS Volume (zvol) when running bhyve VMs. Run the following commands to create a zvol (ensure you have enough disk space to perform these operations): # zfs create -V20G -o volmode=dev zroot/centos0 (zroot in this case is the zpool I am using) Similar to my previous post about vmrun.sh, you need certain items to be configured on FreeBSD in order to use bhyve. The following commands are necessary to get things running: ``` echo "vfs.zfs.vol.mode=2" >> /boot/loader.conf kldload vmm ifconfig tap0 create sysctl net.link.tap.uponopen=1 net.link.tap.uponopen: 0 -> 1 ifconfig bridge0 create ifconfig bridge0 addm em0 addm tap0 ifconfig bridge0 up ``` (replace em0 with whatever your physical interface is). There are a number of utilities that can be used to manage bhyve VMs, and I am sure there is a way to use vmrun.sh to run Linux VMs, but since all of the HowTos for running Linux use the bhyve command line, the following script is what I use for running CentOS with bhyve. ``` !/bin/sh General bhyve install/run script for CentOS Based on scripts from pr1ntf and lattera HOST="127.0.0.1" PORT="5901" ISO="/tmp/centos.iso" VMNAME="centos" ZVOL="centos0" SERIAL="nmda0A" TAP="tap1" CPU="1" RAM="1024M" HEIGHT="800" WIDTH="600" if [ "$1" == "install" ]; then Kill it before starting it bhyvectl --destroy --vm=$VMNAME bhyve -c $CPU -m $RAM -H -P -A -s 0,hostbridge -s 2,virtio-net,$TAP -s 3,ahci-cd,$ISO -s 4,virtio-blk,/dev/zvol/zroot/$ZVOL -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,/dev/$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd $VMNAME kill it after bhyvectl --destroy --vm=$VMNAME elif [ "$1" == "run" ]; then Kill it before starting it bhyvectl --destroy --vm=centos bhyve -c $CPU -m $RAM -w -H -s 0,hostbridge -s 2,virtio-net,$TAP -s 4,virtio-blk,/dev/zvol/zroot/$ZVOL -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,/dev/$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd $VMNAME & else echo "Please type install or run"; fi ``` The variables at the top of the script can be adjusted to fit your own needs. With the addition of the graphics output protocol in UEFI (or UEFI-GOP), a VNC console is launched and hosted with the HOST and PORT setting. There is a password option available for the VNC service, but the connection should be treated as insecure. It is advised to only listen on localhost with the VNC console and tunnel into the host of the bhyve VM. Now with the ISO copied to /tmp/centos.iso, and the script saved as centos.sh you can run the following command to start the install: # ./centos.sh install At this point, using vncviewer (on the local machine, or over an SSH tunnel), you should be able to bring up the console and run the CentOS installer as normal. The absolutely most critical item is to resolve an issue with the booting of UEFI after the installation has completed. Because of the path used in bhyve, you need to run the following to be able to boot CentOS after the installation: # cp -f /mnt/sysimage/boot/efi/EFI/centos/grubx64.efi /mnt/sysimage/boot/efi/EFI/BOOT With this setting changed, the same script can be used to launch your CentOS VM as needed: # ./centos.sh run If you are interested in a better solution for managing your Linux VM, take a look at the various bhyve management ports in the FreeBSD ports tree. Interview - newnix architect - @newnix (https://bsd.network/@newnix) News Roundup GhostBSD 11.1 - FreeBSD for the desktop (https://distrowatch.com/weekly.php?issue=20180108#ghostbsd) GhostBSD is a desktop oriented operating system which is based on FreeBSD. The project takes the FreeBSD operating system and adds a desktop environment, some popular applications, a graphical package manager and Linux binary compatibility. GhostBSD is available in two flavours, MATE and Xfce, and is currently available for 64-bit x86 computers exclusively. I downloaded the MATE edition which is available as a 2.3GB ISO file. Installing GhostBSD's system installer is a graphical application which begins by asking us for our preferred language, which we can select from a list. We can then select our keyboard's layout and our time zone. When it comes to partitioning we have three main options: let GhostBSD take over the entire disk using UFS as the file system, create a custom UFS layout or take over the entire disk using ZFS as the file system. UFS is a classic file system and quite popular, it is more or less FreeBSD's equivalent to Linux's ext4. ZFS is a more advanced file system with snapshots, multi-disk volumes and optional deduplication of data. I decided to try the ZFS option. Once I selected ZFS I didn't have many more options to go through. I was given the chance to set the size of my swap space and choose whether to set up ZFS as a plain volume, with a mirrored disk for backup or in a RAID arrangement with multiple disks. I stayed with the plain, single disk arrangement. We are then asked to create a password for the root account and create a username and password for a regular user account. The installer lets us pick our account's shell with the default being fish, which seemed unusual. Other shells, including bash, csh, tcsh, ksh and zsh are available. The installer goes to work copying files and offers to reboot our computer when it is done. Early impressions The newly installed copy of GhostBSD boots to a graphical login screen where we can sign into the account we created during the install process. Signing into our account loads the MATE 1.18 desktop environment. I found MATE to be responsive and applications were quick to open. Early on I noticed odd window behaviour where windows would continue to slide around after I moved them with the mouse, as if the windows were skidding on ice. Turning off compositing in the MATE settings panel corrected this behaviour. I also found the desktop's default font (Montserrat Alternates) to be hard on my eyes as the font is thin and, for lack of a better term, bubbly. Fonts can be easily adjusted in the settings panel. A few minutes after I signed into my account, a notification appeared in the system tray letting me know software updates were available. Clicking the update icon brings up a small window showing us a list of package updates and, if any are available, updates to the base operating system. FreeBSD, and therefore GhostBSD, both separate the core operating system from the applications (packages) which run on the operating system. This means we can update the core of the system separately from the applications. GhostBSD's core remains relatively static and minimal while applications are updated using a semi-rolling schedule. When we are updating the core operating system, the update manager will give us the option of rebooting the system to finish the process. We can dismiss this prompt to continue working, but the wording of the prompt may be confusing. When asked if we want to reboot to continue the update process, the options presented to us are "Continue" or "Restart". The Continue option closes the update manager and returns us to the MATE desktop. The update manager worked well for me and the only issue I ran into was when I dismissed the update manager and then wanted to install updates later. There are two launchers for the update manager, one in MATE's System menu and one in the settings panel. Clicking either of these launchers didn't accomplish anything. Running the update manager from the command line simply caused the process to lock up until killed. I found if I had dismissed the update manager once, I'd have to wait until I logged in again to use it. Alternatively, I could use a command line tool or use the OctoPkg package manager to install package updates. Conclusions Most of my time with GhostBSD, I was impressed and happy with the operating system. GhostBSD builds on a solid, stable FreeBSD core. We benefit from FreeBSD's performance and its large collection of open source software packages. The MATE desktop was very responsive in my trial and the system is relatively light on memory, even when run on ZFS which has a reputation for taking up more memory than other file systems. FreeBSD Looks At Making Wayland Support Available By Default (https://www.phoronix.com/scan.php?page=news_item&px=FreeBSD-Wayland-Availability) There's an active discussion this week about making Wayland support available by default on FreeBSD. FreeBSD has working Wayland support -- well, assuming you have working Intel / Radeon graphics -- and do have Weston and some other Wayland components available via FreeBSD Ports. FreeBSD has offered working Wayland support that is "quite usable" for more than one year. But, it's not too easy to get going with Wayland on FreeBSD. Right now those FreeBSD desktop users wanting to use/develop with Wayland currently need to rebuild the GTK3 tool-kit, Mesa, and other packages with Wayland support enabled. This call for action now is about allowing the wayland=on to be made the default. This move would then allow these dependencies to be built with Wayland support by default, but for the foreseeable future FreeBSD will continue defaulting to X.Org-based sessions. The FreeBSD developers mostly acknowledge that Wayland is the future and the cost of enabling Wayland support by default is just slightly larger packages, but that weight is still leaner than the size of the X.Org code-base and its dependencies. FreeBSD vote thread (https://lists.freebsd.org/pipermail/freebsd-ports/2017-December/111906.html) TrueOS Fliped the switch already (https://github.com/trueos/trueos-core/commit/f48dba9d4e8cefc45d6f72336e7a0b5f42a2f6f1) fork is not my favorite syscall (https://sircmpwn.github.io/2018/01/02/The-case-against-fork.html) This article has been on my to-write list for a while now. In my opinion, fork is one of the most questionable design choices of Unix. I don't understand the circumstances that led to its creation, and I grieve over the legacy rationale that keeps it alive to this day. Let's set the scene. It's 1971 and you're a fly on the wall in Bell Labs, watching the first edition of Unix being designed for the PDP-11/20. This machine has a 16-bit address space with no more than 248 kilobytes of memory. They're discussing how they're going to support programs that spawn new programs, and someone has a brilliant idea. “What if we copied the entire address space of the program into a new process running from the same spot, then let them overwrite themselves with the new program?” This got a rousing laugh out of everyone present, then they moved on to a better design which would become immortalized in the most popular and influential operating system of all time. At least, that's the story I'd like to have been told. In actual fact, the laughter becomes consensus. There's an obvious problem with this approach: every time you want to execute a new program, the entire process space is copied and promptly discarded when the new program begins. Usually when I complain about fork, this the point when its supporters play the virtual memory card, pointing out that modern operating systems don't actually have to copy the whole address space. We'll get to that, but first — First Edition Unix does copy the whole process space, so this excuse wouldn't have held up at the time. By Fourth Edition Unix (the next one for which kernel sources survived), they had wisened up a bit, and started only copying segments when they faulted. This model leads to a number of problems. One is that the new process inherits all of the parent's process descriptors, so you have to close them all before you exec another process. However, unless you're manually keeping tabs on your open file descriptors, there is no way to know what file handles you must close! The hack that solves this is CLOEXEC, the first of many hacks that deal with fork's poor design choices. This file descriptors problem balloons a bit - consider for example if you want to set up a pipe. You have to establish a piped pair of file descriptors in the parent, then close every fd but the pipe in the child, then dup2 the pipe file descriptor over the (now recently closed) file descriptor 1. By this point you've probably had to do several non-trivial operations and utilize a handful of variables from the parent process space, which hopefully were on the stack so that we don't end up copying segments into the new process space anyway. These problems, however, pale in comparison to my number one complaint with the fork model. Fork is the direct cause of the stupidest component I've ever heard of in an operating system: the out-of-memory (aka OOM) killer. Say you have a process which is using half of the physical memory on your system, and wants to spawn a tiny program. Since fork “copies” the entire process, you might be inclined to think that this would make fork fail. But, on Linux and many other operating systems since, it does not fail! They agree that it's stupid to copy the entire process just to exec something else, but because fork is Important for Backwards Compatibility, they just fake it and reuse the same memory map (except read-only), then trap the faults and actually copy later. The hope is that the child will get on with it and exec before this happens. However, nothing prevents the child from doing something other than exec - it's free to use the memory space however it desires! This approach now leads to memory overcommittment - Linux has promised memory it does not have. As a result, when it really does run out of physical memory, Linux will just kill off processes until it has some memory back. Linux makes an awfully big fuss about “never breaking userspace” for a kernel that will lie about memory it doesn't have, then kill programs that try to use the back-alley memory they were given. That this nearly 50 year old crappy design choice has come to this astonishes me. Alas, I cannot rant forever without discussing the alternatives. There are better process models that have been developed since Unix! The first attempt I know of is BSD's vfork syscall, which is, in a nutshell, the same as fork but with severe limitations on what you do in the child process (i.e. nothing other than calling exec straight away). There are loads of problems with vfork. It only handles the most basic of use cases: you cannot set up a pipe, cannot set up a pty, and can't even close open file descriptors you inherited from the parent. Also, you couldn't really be sure of what variables you were and weren't editing or allowed to edit, considering the limitations of the C specification. Overall this syscall ended up being pretty useless. Another model is posixspawn, which is a hell of an interface. It's far too complicated for me to detail here, and in my opinion far too complicated to ever consider using in practice. Even if it could be understood by mortals, it's a really bad implementation of the spawn paradigm — it basically operates like fork backwards, and inherits many of the same flaws. You still have to deal with children inheriting your file descriptors, for example, only now you do it in the parent process. It's also straight-up impossible to make a genuine pipe with posixspawn. (Note: a reader corrected me - this is indeed possible via posixspawnfileactionsadddup2.) Let's talk about the good models - rfork and spawn (at least, if spawn is done right). rfork originated from plan9 and is a beautiful little coconut of a syscall, much like the rest of plan9. They also implement fork, but it's a special case of rfork. plan9 does not distinguish between processes and threads - all threads are processes and vice versa. However, new processes in plan9 are not the everything-must-go fuckfest of your typical fork call. Instead, you specify exactly what the child should get from you. You can choose to include (or not include) your memory space, file descriptors, environment, or a number of other things specific to plan9. There's a cool flag that makes it so you don't have to reap the process, too, which is nice because reaping children is another really stupid idea. It still has some problems, mainly around creating pipes without tremendous file descriptor fuckery, but it's basically as good as the fork model gets. Note: Linux offers this via the clone syscall now, but everyone just fork+execs anyway. The other model is the spawn model, which I prefer. This is the approach I took in my own kernel for KnightOS, and I think it's also used in NT (Microsoft's kernel). I don't really know much about NT, but I can tell you how it works in KnightOS. Basically, when you create a new process, it is kept in limbo until the parent consents to begin. You are given a handle with which you can configure the process - you can change its environment, load it up with file descriptors to your liking, and so on. When you're ready for it to begin, you give the go-ahead and it's off to the races. The spawn model has none of the flaws of fork. Both fork and exec can be useful at times, but spawning is much better for 90% of their use-cases. If I were to write a new kernel today, I'd probably take a leaf from plan9's book and find a happy medium between rfork and spawn, so you could use spawn to start new threads in your process space as well. To the brave OS designers of the future, ready to shrug off the weight of legacy: please reconsider fork. Enable ld.lld as bootstrap linker by default on amd64 (https://svnweb.freebsd.org/changeset/base/327783) Enable ld.lld as bootstrap linker by default on amd64 For some time we have been planning to migrate to LLVM's lld linker. Having a man page was the last blocking issue for using ld.lld to link the base system kernel + userland, now addressed by r327770. Link the kernel and userland libraries and binaries with ld.lld by default, for additional test coverage. This has been a long time in the making. On 2013-04-13 I submitted an upstream tracking issue in LLVM PR 23214: [META] Using LLD as FreeBSD's system linker. Since then 85 individual issues were identified, and submitted as dependencies. These have been addressed along with two and a half years of other lld development and improvement. I'd like to express deep gratitude to upstream lld developers Rui Ueyama, Rafael Espindola, George Rimar and Davide Italiano. They put in substantial effort in addressing the issues we found affecting FreeBSD/amd64. To revert to using ld.bfd as the bootstrap linker, in /etc/src.conf set WITHOUTLLDBOOTSTRAP=yes If you need to set this, please follow up with a PR or post to the freebsd-toolchain mailing list explaining how default WITHLLDBOOTSTRAP failed for your use case. Note that GNU ld.bfd is still installed as /usr/bin/ld, and will still be used for linking ports. ld.lld can be installed as /usr/bin/ld by setting in /etc/src.conf WITH_LLD_IS_LLD=yes A followup commit will set WITHLLDIS_LD by default, possibly after Clang/LLVM/lld 6.0 is merged to FreeBSD. Release notes: Yes Sponsored by: The FreeBSD Foundation Followup: https://www.mail-archive.com/svn-src-all@freebsd.org/msg155493.html *** Beastie Bits BSDCAN2017 Interview with Peter Hessler, Reyk Floeter, and Henning Brauer (https://undeadly.org/cgi?action=article;sid=20171229080944) video (https://www.youtube.com/watch?v=e-Xim3_rJns) DSBMD (https://freeshell.de/~mk/projects/dsbmd.html) ccc34 talk - May contain DTraces of FreeBSD (https://media.ccc.de/v/34c3-9196-may_contain_dtraces_of_freebsd) [scripts to run an OpenBSD mirror, rsync and verify])(https://github.com/bluhm/mirror-openbsd) Old School PC Fonts (https://int10h.org/oldschool-pc-fonts/readme/) Feedback/Questions David - Approach and Tools for Snapshots and Remote Replication (http://dpaste.com/33HKKEM#wrap) Brian - Help getting my FreeBSD systems talking across the city (http://dpaste.com/3QWFEYR#wrap) Malcolm - First BSD Meetup in Stockholm happened and it was great (http://dpaste.com/1Z9Y8H1) Brad - Update on TrueOS system (http://dpaste.com/3EC9RGG#wrap) ***
We review the information about Spectre & Meltdown thus far, we look at NetBSD memory sanitizer progress, Postgres on ZFS & show you a bit about NomadBSD. This episode was brought to you by Headlines Meltdown Spectre Official Site (https://meltdownattack.com/) Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign (https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/) Intel's official response (https://newsroom.intel.com/news/intel-responds-to-security-research-findings/) The Register mocks intels response with pithy annotations (https://www.theregister.co.uk/2018/01/04/intel_meltdown_spectre_bugs_the_registers_annotations/) Intel's Analysis PDF (https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-of-Speculative-Execution-Side-Channels.pdf) XKCD (https://xkcd.com/1938/) Response from FreeBSD (https://lists.freebsd.org/pipermail/freebsd-security/2018-January/009719.html) FreeBSD's patch WIP (https://reviews.freebsd.org/D13797) Why Raspberry Pi isn't vulnerable to Spectre or Meltdown (https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/) Xen mitigation patches (https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00110.html) Overview of affected FreeBSD Platforms/Architectures (https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities) Groff's response (https://twitter.com/GroffTheBSDGoat/status/949372300368867328) ##### We'll cover OpenBSD, NetBSD, and DragonflyBSD's responses in next weeks episode. *** ###The LLVM Memory Sanitizer support work in progress (https://blog.netbsd.org/tnf/entry/the_llvm_memory_sanitizer_support) > In the past 31 days, I've managed to get the core functionality of MSan to work. This is an uninitialized memory usage detector. MSan is a special sanitizer because it requires knowledge of every entry to the basesystem library and every entry to the kernel through public interfaces. This is mandatory in order to mark memory regions as initialized. Most of the work has been done directly for MSan. However, part of the work helped generic features in compiler-rt. Sanitizers > Changes in the sanitizer are listed below in chronological order. Almost all of the changes mentioned here landed upstream. A few small patches were reverted due to breaking non-NetBSD hosts and are rescheduled for further investigation. I maintain these patches locally and have moved on for now to work on the remaining features. NetBSD syscall hooks > I wrote a large patch (815kb!) adding support for NetBSD syscall hooks for use with sanitizers. NetBSD ioctl(2) hooks > Similar to the syscall hooks, there is need to handle every ioctl(2) call. I've created the needed patch, this time shorter - for less than 300kb. New patches still pending for upstream review > There are two corrections that I've created, and they are still pending upstream for review: Add MSan interceptor for fstat(2)](https://reviews.llvm.org/D41637) Correct the setitimer interceptor on NetBSD)](https://reviews.llvm.org/D41502) > I've got a few more local patches that require cleanup before submitting to review. NetBSD basesystem corrections Sanitizers in Go The MSan state as of today Solaris support in sanitizers > I've helped the Solaris team add basic support for Sanitizers (ASan, UBsan). This does not help NetBSD directly, however indirectly it improves the overall support for non-Linux hosts and helps to catch more Linuxisms in the code. Plan for the next milestone > I plan to continue the work on MSan and correct sanitizing of the NetBSD basesystem utilities. This mandates me to iterate over the basesystem libraries implementing the missing interceptors and correcting the current support of the existing ones. My milestone is to build all src/bin programs against Memory Sanitizer and when possible execute them cleanly. This work was sponsored by The NetBSD Foundation. The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: http://netbsd.org/donations/#how-to-donate (http://netbsd.org/donations/#how-to-donate) *** ##News Roundup ###MWL's 2017 Wrap-Up (https://blather.michaelwlucas.com/archives/3078) > The obvious place to start is my 2016 wrap-up post](https://blather.michaelwlucas.com/archives/2822), where I listed goals for 2017. As usual, these goals were wildly delusional. > The short answer is, my iron was back up to normal. My writing speed wasn't, though. I'd lost too much general health, and needed hard exercise to recover it. Yes, writing requires physical endurance. Maintaining that level of concentration for several hours a day demands a certain level of blood flow to the brain. I could have faked it in a day job, but when self-employed as an artist? Not so much. > Then there's travel. I did my usual BSDCan trip, plus two educational trips to Lincoln City, Oregon. The current political mayhem convinced me that if I wanted to hit EuroBSDCon any time in the next few years, I should do it in the very near future. So I went to Paris, where I promptly got pickpocketed. (Thankfully, they didn't get my passport.) I was actively writing the third edition of Absolute FreeBSD, so I visited BSDCam in Cambridge to get the latest information and a sense of where FreeBSD was going. I also did weekends at Kansas LinuxFest (because they asked and paid for my trip) and Penguicon. > (Because people will ask: why EuroBSDCon and not AsiaBSDCon? A six-hour transatlantic flight requires that I take a substantial dose of heavy-grade tranquilizers. I'm incapable of making intelligent decisions while on those drugs, or for several hours afterward. They don't last long enough for twelve-hour flight to Japan, so I need to be accompanied by someone qualified to tell me when I need to take the next dose partway through the flight. This isn't a predetermined time that I can set an alarm for; it depends on how the clonazepam affects me at those altitudes. A drug overdose while flying over the North Pole would be bad. When I can arrange that qualified companion, I'll make the trip.) > I need most of the preceding week to prepare for long trips. I need the following week to recover from time shifts and general exhaustion. Additionally, I have to hoard people juice for a few weeks beforehand so I can deal with folks during these expeditions. Travel disrupts my dojo time as well, which impacts my health. > Taken as a whole: I didn't get nearly as much done as I hoped. I wrote more stories, but Kris Rusch bludgeoned me into submitting them to trad markets. (The woman is a brute, I tell you. Cross her at your peril.) Among my 2017 titles, my fiction outsold the tech books. No, not Prohibition Orcs–all four of the people who buy those love them, but the sales tell me I've done something wrong with those tales. My cozy mystery git commit murder outsold Relayd and Httpd Mastery. But what outdid them both, as well as most of my older books? What title utterly dominated my sales for the last quarter of the year? It was of course, my open source software political satire disguised as porn Savaged by Systemd: an Erotic Unix Encounter. (https://www.michaelwarrenlucas.com/index.php/romance#sbs) > I can't believe I just wrote that paragraph. The good news is, once I recovered from EuroBSDCon, my writing got better. I finished Absolute FreeBSD, 3rd edition and submitted it to the publisher. I wrote the second edition of SSH Mastery (no link, because you can't order it yet.) I'm plowing through git sync murder, the sequel to git commit murder. I don't get to see the new Star Wars movie until I finish GSM, so hopefully that'll be this month. All in all, I wrote 480,200 words in 2017. Most of that was after September. It's annoyingly close to breaking half a million, but after 2016's scandalous 195,700, I'll take it. *** ###PG Phriday: Postgres on ZFS (https://blog.2ndquadrant.com/pg-phriday-postgres-zfs/) > ZFS is a filesystem originally created by Sun Microsystems, and has been available for BSD over a decade. While Postgres will run just fine on BSD, most Postgres installations are historically Linux-based systems. ZFS on Linux has had much more of a rocky road to integration due to perceived license incompatibilities. > As a consequence, administrators were reluctant or outright refused to run ZFS on their Linux clusters. It wasn't until OpenZFS was introduced in 2013 that this slowly began to change. These days, ZFS and Linux are starting to become more integrated, and Canonical of Ubuntu fame even announced direct support for ZFS in their 16.04 LTS release. > So how can a relatively obscure filesystem designed by a now-defunct hardware and software company help Postgres? Let's find out! Eddie waited til he finished high school > Old server hardware is dirt cheap these days, and make for a perfect lab for testing suspicious configurations. This is the server we'll be using for these tests for those following along at home, or want some point of reference: Dell R710 x2 Intel X5660 CPUs, for up to 24 threads 64GB RAM x4 1TB 7200RPM SATA HDDs H200 RAID card configured for Host Bus Adapter (HBA) mode 250GB Samsung 850 EVO SSD > The H200 is particularly important, as ZFS acts as its own RAID system. It also has its own checksumming and other algorithms that don't like RAID cards getting in the way. As such, we put the card itself in a mode that facilitates this use case. > Due to that, we lose out on any battery-backed write cache the RAID card might offer. To make up for it, it's fairly common to use an SSD or other persistent fast storage to act both as a write cache, and a read cache. This also transforms our HDDs into hybrid storage automatically, which is a huge performance boost on a budget. She had a guitar and she taught him some chords > First things first: we need a filesystem. This hardware has four 1TB HDDs, and a 250GB SSD. To keep this article from being too long, we've already placed GPT partition tables on all the HDDs, and split the SSD into 50GB for the OS, 32GB for the write cache, and 150GB for the read cache. A more robust setup would probably use separate SSDs or a mirrored pair for these, but labs are fair game. They moved into a place they both could afford > Let's start by getting a performance baseline for the hardware. We might expect peak performance at 12 or 24 threads because the server has 12 real CPUs and 24 threads, but query throughput actually topped out at concurrent 32 processes. We can scratch our heads over this later, for now, we can consider it the maximum capabilities of this hardware. Here's a small sample: ``` $> pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench ... tps = 264661.135288 (including connections establishing) tps = 264849.345595 (excluding connections establishing) ``` So far, this is pretty standard behavior. 260k prepared queries per second is great read performance, but this is supposed to be a filesystem demonstration. Let's get ZFS involved. + The papers said Ed always played from the heart Let's repeat that same test with writes enabled. Once that happens, filesystem syncs, dirty pages, WAL overhead, and other things should drastically reduce overall throughput. That's an expected result, but how much are we looking at, here? ``` $> pgbench -j 32 -c 32 -M prepared -T 10 pgbench ... tps = 6153.877658 (including connections establishing) tps = 6162.392166 (excluding connections establishing) ``` SSD cache or not, storage overhead is a painful reality. Still, 6000 TPS with writes enabled is a great result for this hardware. Or is it? Can we actually do better? Consider the Postgres fullpagewrites parameter. Tomas Vondra has written about it in the past as a necessity to prevent WAL corruption due to partial writes. The WAL is both streaming replication and crash recovery, so its integrity is of utmost importance. As a result, this is one parameter almost everyone should leave alone. ZFS is Copy on Write (CoW). As a result, it's not possible to have a torn page because a page can't be partially written without reverting to the previous copy. This means we can actually turn off fullpagewrites in the Postgres config. The results are some fairly startling performance gains: $> pgbench -j 32 -c 32 -M prepared -T 10 pgbench tps = 10325.200812 (including connections establishing) tps = 10336.807218 (excluding connections establishing) That's nearly a 70% improvement. Due to write amplification caused by full page writes, Postgres produced 1.2GB of WAL files during a 1-minute pgbench test, but only 160MB with full page writes disabled. To be fair, a 32-thread pgbench write test is extremely abusive and certainly not a typical usage scenario. However, ZFS just ensured our storage a much lower write load by altering one single parameter. That means the capabilities of the hardware have also been extended to higher write workloads as IO bandwidth is not being consumed by WAL traffic. + They both met movie stars, partied and mingled Astute readers may have noticed we didn't change the default ZFS block size from 128k to align with the Postgres default of 8kb. As it turns out, the 128kb blocks allow ZFS to better combine some of those 8kb Postgres pages to save space. That will allow our measly 2TB to go a lot further than is otherwise possible. Please note that this is not de-duplication, but simple lz4 compression, which is nearly real-time in terms of CPU overhead. De-duplication on ZFS is currently an uncertain bizzaro universe populated with misshapen horrors crawling along a broken landscape. It's a world of extreme memory overhead for de-duplication tables, and potential lost data due to inherent conflicts with the CoW underpinnings. Please don't use it, let anyone else use it, or even think about using it, ever. + They made a record and it went in the chart We're still not done. One important aspect of ZFS as a CoW filesystem, is that it has integrated snapshots. Consider the scenario where a dev is connected to the wrong system and drops what they think is a table in a QA environment. It turns out they were in the wrong terminal and just erased a critical production table, and now everyone is frantic. + The future was wide open It's difficult to discount an immediately observable reduction in write overhead. Snapshots have a multitude of accepted and potential use cases, as well. In addition to online low-overhead compression, and the hybrid cache layer, ZFS boasts a plethora of features we didn't explore. Built-in checksums with integrated self-healing suggest it isn't entirely necessary to re-initialize an existing Postgres instance to enable checksums. The filesystem itself ensures checksums are validated and correct, especially if we have more than one drive resource in our pool. It even goes the extra mile and actively corrects inconsistencies when encountered. I immediately discounted ZFS back in 2012 because the company I worked for at the time was a pure Linux shop. ZFS was only available using the FUSE driver back then, meaning ZFS only worked through userspace with no real kernel integration. It was fun to tinker with, but nobody sane would use that on a production server of any description. Things have changed quite drastically since then. I've stopped waiting for btrfs to become viable, and ZFS has probably taken the throne away from XFS as my filesystem of choice. Future editions of the Postgres High Availability Cookbook will reflect this as well. Postgres MVCC and ZFS CoW seem made for each other. I'm curious to see what will transpire over the next few years now that ZFS has reached mainstream acceptance in at least one major Linux distribution. NomadBSD (https://github.com/mrclksr/NomadBSD) About NomadBSD is a live system for flash drives, based on FreeBSD. Screenshots http://freeshell.de/~mk/download/nomadbsd-ss1.png http://freeshell.de/~mk/download/nomadbsd-ss2.png Requirements for building the image A recent FreeBSD system Requirements for running NomadBSD A 4GB (or more) flash drive A System capable running FreeBSD 11.1 (amd64) Building the image ~~ csh # make image ~~ Writing the image to an USB memory stick ~~ csh # dd if=nomadbsd.img of=/dev/da0 bs=10240 conv=sync ~~ Resize filesystem to use the entire USB memory Boot NomadBSD into single user mode, and execute: ~~ # gpart delete -i 2 da0s1 # gpart resize -i 1 da0 # gpart commit da0s1 ~~ Determine the partition size in megabytes using fdisk da0 and calculate the remaining size of da0s1a: = - . ~~ # gpart resize -i 1 -s M da0s1 # gpart add -t freebsd-swap -i 2 da0s1 # glabel label NomadBSDsw da0s1b # service growfs onestart # reboot ~~ FreeBSD forum thread (https://forums.freebsd.org/threads/63888/) A short screen capture video of the NomadBSD system running in VirtualBox (https://freeshell.de/~mk/download/nomad_capture.mp4) *** ##Beastie Bits Coolpkg, a package manager inspired by Nix for OpenBSD (https://github.com/andrewchambers/coolpkg) zrepl - ZFS replication (https://zrepl.github.io/) OpenBSD hotplugd automount script (https://bijanebrahimi.github.io/blog/openbsd-hotplugd-scripting.html) Ancient troff sources vs. modern-day groff (https://virtuallyfun.com/2017/12/22/learn-ancient-troff-sources-vs-modern-day-groff/) Paypal donation balance and status.. thanks everyone! (http://lists.dragonflybsd.org/pipermail/users/2017-December/313752.html) Supervised FreeBSD rc.d script for a Go daemon (updated in last few days) (https://redbyte.eu/en/blog/supervised-freebsd-init-script-for-go-deamon/) A Brief History of sed (https://blog.sourcerer.io/a-brief-history-of-sed-6eaf00302ed) Flamegraph: Why does my AWS instance boot so slow? (http://www.daemonology.net/timestamping/tslog-c5.4xlarge.svg) *** ##Feedback/Questions Jeremy - Replacing Drive in a Zpool (http://dpaste.com/319593M#wrap) Dan's Blog (https://dan.langille.org/2017/08/16/swapping-5tb-in-3tb-out/) Tim - Keeping GELI key through reboot (http://dpaste.com/11QTA06) Brian - Mixing 2.5 and 3.5 drives (http://dpaste.com/2JQVD10#wrap) Troels - zfs swap on FreeBSD (http://dpaste.com/147WAFR#wrap) ***
© 2020-2025 Ivy Podcast Discovery LLC
