POPULARITY
41 episodes with eurobsdcon
4 episodes with eurobsdcon
11 episodes with eurobsdcon
4 episodes with eurobsdcon
2 episodes with eurobsdcon
ZFS Storage Fault Management, FreeBSD 14.2-RELEASE Announcement, I feel that NAT is inevitable even with IPv6, Spell checking in Vim, OpenBSD Memory Conflict Messages, The Biggest Shell Programs in the World, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines ZFS Storage Fault Management (https://klarasystems.com/articles/zfs-storage-fault-management-linux/?utm_source=BSD%20Now&utm_medium=Podcast) FreeBSD 14.2-RELEASE Announcement (https://www.freebsd.org/releases/14.2R/announce/) News Roundup I feel that NAT is inevitable even with IPv6 (https://utcc.utoronto.ca/~cks/space/blog/tech/IPv6AndStillHavingNAT) Spell checking in Vim (https://www.tumfatig.net/2024/spell-checking-in-vim/) OpenBSD Memory Conflict Messages (https://utcc.utoronto.ca/~cks/space/blog/unix/OpenBSDMemoryConflictMessages) The Biggest Shell Programs in the World (https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell-Programs-in-the-World) Beastie Bits The Connectivity of Things: Network Cultures since 1832 (https://direct.mit.edu/books/oa-monograph/5866/The-Connectivity-of-ThingsNetwork-Cultures-since) Initial list of 21 EuroBSDcon 2024 videos released (https://www.undeadly.org/cgi?action=article;sid=20241130184249) -current now has more flexible performance policy (https://www.undeadly.org/cgi?action=article;sid=20241129093132) OpenBSD 5.1 on Sun Ultra 5 (https://eggflix.foolbazar.eu/w/fa211a4f-6984-4c03-a6d2-b8c329d9459d) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/592/feedback/Phillip%20-%20regressions.md Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
Debunking Common Myths About FreeBSD - Part 2, FreeBSD 13.4-RELEASE Announcement, OpenBSD -current has moved to version 7.6, acpidumping,Install snac2 on FreeBSD – An ActivityPub Instance for the Fediverse, Managing dotfiles with chezmoi, Podman testing on FreeBSD, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Debunking Common Myths About FreeBSD - Part 2 (https://klarasystems.com/articles/debunking-common-myths-about-freebsd-2/) FreeBSD 13.4-RELEASE Announcement (https://www.freebsd.org/releases/13.4R/announce/) FreeBSD 14.0 end-of-life (https://bsdsec.net/articles/freebsd-14-0-end-of-life) - You should have upgraded to 14.1 by now OpenBSD -current has moved to version 7.6 (https://www.undeadly.org/cgi?action=article;sid=20240918052239) News Roundup acpidumping (https://adventurist.me/posts/00325) Install snac2 on FreeBSD – An ActivityPub Instance for the Fediverse (https://gyptazy.com/install-snac2-on-freebsd-an-activitypub-instance-for-the-fediverse/) Installing Uptime-Kuma on a FreeBSD Jail (https://it-notes.dragas.net/2024/07/22/install-uptime-kuma-freebsd-jail/) Managing dotfiles with chezmoi (https://stoddart.github.io/2024/09/08/managing-dotfiles-with-chezmoi.html) Podman testing on FreeBSD (https://github.com/oci-playground/freebsd-podman-testing) Undeadly Bits OpenSSH 9.9 released! (https://undeadly.org/cgi?action=article;sid=20240921181110) OpenBSD now enforcing no invalid NUL characters in shell scripts (http://undeadly.org/cgi?action=article;sid=20240924105732) EuroBSDCon 2024 presentations are now up (http://undeadly.org/cgi?action=article;sid=20240924092154) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions rel4x - Secure by default (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/581/feedback/rel4x%20-%20Secure%20by%20default.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
Jason is still on location at EuroBSDcon getting interviews with those in the BSD Community. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Interviews Colin Percival Andrew Hewus Wolfgang Liam Proven Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
Jason is on location at EuroBSDcon getting interviews with those in the BSD Community. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Interviews Vanja Cvelbar Stefano Marinelli Dave Cottlehuber Christos Margiolis Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow) Jason is on location at EuroBSDcon getting interviews with those in the BSD Community.
Navigating FreeBSD's New Quarterly and Biennial Release Schedule, EuroBSDCon 2024 Schedule, From Cloud Chaos to FreeBSD Efficiency, Local-to-anchors tables in PF rules, CloudBSD, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Navigating FreeBSD's New Quarterly and Biennial Release Schedule (https://freebsdfoundation.org/blog/navigating-freebsds-new-quarterly-and-biennial-release-schedule/) https://mccd.space/posts/netbsd-review/ (https://mccd.space/posts/netbsd-review/) EuroBSDCon 2024 Schedule (https://events.eurobsdcon.org/2024/schedule/) News Roundup From Cloud Chaos to FreeBSD Efficiency (https://it-notes.dragas.net/2024/07/04/from-cloud-chaos-to-freebsd-efficiency/) Enable local-to-anchors tables in PF rules (https://undeadly.org/cgi?action=article;sid=20240714154257) CloudBSD (https://cloudbsd.xyz/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Rick - Feedback about Docs Bugs (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/571/feedback/Rick%20-%20Feedback%20about%20Docs%20Bugs.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
GPL 3: The Controversial Licensing Model and Potential Solutions, The Geeks way of checking what the outside weather is like, Alpine on a FreeBSD Jail, DragonFly BSD on a Thinkpad T480s, Dealing with USB Storage devices on OmniOS, Creating a Time Capsule instance using Samba, FreeBSD, and ZFS NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines GPL 3: The Controversial Licensing Model and Potential Solutions (https://klarasystems.com/articles/gpl-3-the-controversial-licensing-model-and-potential-solutions/) The Geeks way of checking what the outside wheather is like (https://blog.netbsd.org/tnf/entry/the_geeks_way_of_checking) News Roundup Alpine on a FreeBSD Jail (https://it-notes.dragas.net/2024/01/18/installing-alpine-linux-on-a-freebsd-jail/) DragonFly BSD on a Thinkpad T480s (https://git.sr.ht/~tomh/dragonflybsd-on-a-laptop/tree/master/item/README.md) Dealing with USB Storage devices on OmniOS (https://www.tumfatig.net/2024/dealing-with-usb-storage-devices-on-omnios/) Creating a Time Capsule instance using Samba, FreeBSD, and ZFS (https://dan.langille.org/2024/01/06/creating-a-time-capsule-instance-using-samba-freebsd-and-zfs-2/) Conferences FOSDEM (https://fosdem.org/2024/) AsiaBSDCon (https://2024.asiabsdcon.org/program.html) BSDCan (https://www.bsdcan.org/2024/papers.php) EuroBSDcon (https://2024.eurobsdcon.org/) Southeast Linuxfest (https://southeastlinuxfest.org/2024/01/self-2024-call-for-participation/) Dont let the name fool you, SELF is BSD friendly and they'd love to have BSD/Unix Talks if you're in the area. JT is staff at SELF, so he can put in a good word for you. ;) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
OpenZFS Storage Best Practices and Use Cases, EuroBSDcon trip report, Disks from the Perspective of a File System, Creating Jails using flavours in pot, OpenIKED 7.3 released, OpenSMTPD 7.4.0p1 Released, FreeBSD can now boot in 25 milliseconds, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines OpenZFS Storage Best Practices and Use Cases - Part 1: Snapshots and Backups (https://klarasystems.com/articles/openzfs-storage-best-practices-and-use-cases-part-1-snapshots-and-backups/) EuroBSDCon 2023 report (1/2) – arrival & tutorial days (https://eerielinux.wordpress.com/2023/09/20/eurobsdcon-2023-report-1-2-arrival-tutorial-days/) EuroBSDCon 2023 report (2/2) – Main conference, social event & conclusion (https://eerielinux.wordpress.com/2023/10/15/eurobsdcon-2023-report-2-2-main-conference-social-event-conclusion/) News Roundup Disks from the Perspective of a File System (https://dl.acm.org/doi/10.1145/2367376.2367378) OpenIKED 7.3 released (https://marc.info/?l=openbsd-announce&m=170042964022226&w=2) OpenSMTPD 7.4.0p1 Released (https://marc.info/?l=openbsd-announce&m=170012963318854&w=2) FreeBSD – Creating Jails using flavours in pot (https://byte-sized.de/linux-unix/freebsd-jails-mithilfe-von-flavours-in-pot-erstellen/#english) FreeBSD can now boot in 25 milliseconds (https://www.theregister.com/2023/08/29/freebsd_boots_in_25ms/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
FreeBSD on the RISC-V Architecture, A bit of XENIX history, pkgbase: Official packages, recover lost text by coredumping firefox, FuguIta 7.4 has been released, LibreSSL 3.8.2 Released, OpenSMTPD 7.4.0p0 Released NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Looking Towards the Future: FreeBSD on the RISC-V Architecture (https://klarasystems.com/articles/looking-towards-the-future-freebsd-on-the-risc-v-architecture/) A bit of XENIX history (http://seefigure1.com/2014/04/15/xenixtime.html) News Roundup Official packages (https://lists.freebsd.org/archives/freebsd-pkgbase/2023-October/000221.html) recover lost text by coredumping firefox (https://j3s.sh/thought/recover-lost-text-by-coredumping-firefox.html) FuguIta 7.4 has been released (https://fuguita.org/?FuguIta/7.4&utm_source=bsdweekly) LibreSSL 3.8.2 Released (https://undeadly.org/cgi?action=article;sid=20231103065952) OpenSMTPD 7.4.0p0 Released (https://undeadly.org/cgi?action=article;sid=20231026121132) Conference News AsiaBSDCon 2024 (https://2024.asiabsdcon.org) BSDCan 2024 (https://www.bsdcan.org) EuroBSDCon 2024 (https://2024.eurobsdcon.org) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
2^18 dollars to open source, EuroBSDCon 2023 Trip Report, FreeBSD vs Linux (Debian), Introduction to sysclean8, Run your own Syncthing discovery server on OpenBSD, FreeBSD years: 2000-2005, Using OpenBSD relayd(8) as an Application Layer Gateway, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines 2^18 dollars to open source (https://www.daemonology.net/blog/2023-10-25-2%5E18-dollars-to-open-source.html) Special Thanks to Colin for supporting BSD Now for over 10 years! *** ### EuroBSDCon 2023 Trip Report – Bojan Novković (https://freebsdfoundation.org/blog/eurobsdcon-2023-trip-report-bojan-novkovic/) *** ### FreeBSD vs Linux (Debian) (https://markmcb.com/freebsd/vs_linux/) *** News Roundup Introduction to sysclean8 (https://www.osnews.com/story/137266/introduction-to-sysclean8-on-openbsd/) Run your own Syncthing discovery server on OpenBSD (https://dataswamp.org/~solene/2023-10-18-syncthing-discovery-server.html) My FreeBSD years: 2000-2005 (https://www.cambus.net/my-freebsd-years-2000-2005/) Using OpenBSD relayd(8) as an Application Layer Gateway (https://www.tumfatig.net/2023/using-openbsd-relayd8-as-an-application-layer-gateway/) Beastie Bits How to send syslog messages using command-line utilities (https://sleeplessbeastie.eu/2023/09/11/how-to-send-syslog-messages-using-command-line-utilities/) A Practical Guide of GNU grep With Examples (https://thevaluable.dev/grep-cli-guide-examples/) FreeBSD Container VM for Podman (https://github.com/davidchisnall/container-vm-scripts) User Certbot to create SSL certificates on FreeBSD (https://www.sharpwriting.net/project/use-certbot-to-create-ssl-certificates-on-freebsd/) OpenBSD's built-in memory leak detection (https://www.undeadly.org/cgi?action=article;sid=20231024064619) OpenBSD Webzine Issue #15 (https://webzine.puffy.cafe/issue-15.html) FreeBSD OpenSSL 3.0 ported (https://github.com/FreeBSD/freebsd-ports/commit/d5ec2e12f399b7813994564b77a0915821a0ac42) Harden FreeBSD Script (https://github.com/wravoc/harden-freebsd) Something odd happened... (https://mastodon.bsd.cafe/@stefano/111257154132788711) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)
This year end episode of BSDNow features a trip report to EuroBSDcon by Mr. BSD.tv, as well as an interview with FreeBSD committer John Baldwin. Happy New Year, 2023! NOTES*** This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) EuroBSDCon 2022 Trip Report (https://freebsdfoundation.org/blog/eurobsdcon-2022-trip-report-patrick-mcevoy/) Interview 3 - John Baldwin - email@email (mailto:email@email) / @twitter (https://twitter.com/user) Interview topic Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
This special episode features two interviews we did at EuroBSDcon in Vienna this year. We talk with FreeBSD developers about how they got started, their current projects and more. Also, consider donating to your favorite BSD Foundation to keep the projects going. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Help the OpenBSD Foundation Reach Its 2022 Funding Goal (http://undeadly.org/cgi?action=article;sid=20221202062601) • [FreeBSD Foundation Donation Link](https://freebsdfoundation.org/donate/) • [NetBSD Foundation Donation Link](http://www.netbsd.org/donations/#how-to-donate) Interview 1 - Brooks Davis - email@email (mailto:email@email) / @twitter (https://twitter.com/user) Interview topic Interview 2 - Olivier Cochard-Labbe - email@email (mailto:email@email) / @twitter (https://twitter.com/user) Interview topic Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. *** Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
5 Key Reasons to Consider Open Source Storage, OpenBSD Minimalist Desktop, BSD XFCE, Alpine Linux VM on bhyve - with root on ZFS, FreeBSD Jail Quick Setup with Networking, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines 5 Key Reasons to Consider Open Source Storage Over Commercial Offerings (https://klarasystems.com/articles/open-source-storage-over-commercial-offerings/) OpenBSD Minimalist Desktop (https://nechtan.io/articles/openbsd_minimalist_desktop.html) News Roundup BSD-XFCE (https://github.com/Wamphyre/BSD-XFCE) Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted) (https://it-notes.dragas.net/2022/11/01/creating-an-alpine-vm-on-bhyve-with-root-on-zfs-optionally-encrypted/) FreeBSD Jail Quick Setup with Networking (2022) (https://www.shaka.today/freebsd-jail-quick-setup-with-networking-2022/) Beastie Bits EuroBSDcon videos are now up (https://www.youtube.com/c/EuroBSDcon/videos) LibreSSL 3.6.1 released (https://undeadly.org/cgi?action=article;sid=20221104064712) Raspberry Pi 4 with FreeBSD 13-RELEASE: A Perfect Miniature Homelab (https://www.coreystephan.com/pi4-freebsd/) AsiaBSDcon 2023 CfP (https://2023.asiabsdcon.org/cfp.html.en) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions John - Allan's meetup (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/John%20-%20Allan's%20meetup.md) Matthew - atime and a question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/Matthew%20-%20atime%20and%20a%20question.md) Valentin - Becoming a FreeBSD Developer (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/482/feedback/Valentin%20-%20Becoming%20a%20FreeBSD%20Developer.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
OpenBSD 7.2 and FuguIta have been released, Learn the Whys and Hows with the FreeBSD Sec Team, how to get notified about FreeBSD updates, using unbound for ad blocking on OpenBSD, further memory protections on OpenBSD current, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines OpenBSD 7.2 has been released (https://www.openbsd.org/72.html) FuguIta 7.2 is out as well (https://fuguita.org/index.php?FuguIta%2F7.2=) *** ### Keeping FreeBSD Secure: Learn the Whys and Hows with the FreeBSD Sec Team (https://freebsdfoundation.org/blog/keeping-freebsd-secure-learn-the-whys-and-hows-with-the-freebsd-sec-team/) News Roundup Howto: be notified of FreeBSD upgrades, security updates and package updates at login (https://forums.freebsd.org/threads/howto-be-notified-of-freebsd-upgrades-security-updates-and-package-updates-at-login.86660/) Ads blocking with OpenBSD unbound(8) (https://www.tumfatig.net/2022/ads-blocking-with-openbsd-unbound8/) Further memory protections committed to -current (http://undeadly.org/cgi?action=article;sid=20221008100649) Beastie Bits • [“OpenBSD Mastery: Filesystems” Print/Ebook Bundle Preorder](https://mwl.io/archives/22352) • [Klara is hiring a FreeBSD Kernel Developer](https://klarasystems.com/careers/freebsd-kernel-developer/) • [FreeBSD 12.4-BETA1 Now Available](https://lists.freebsd.org/archives/freebsd-stable/2022-October/000920.html) • [Hunting kernel lock and interrupt latency](https://mail-index.netbsd.org/tech-kern/2022/10/30/msg028499.html) • [EuroBSDcon 2022 videos available](https://undeadly.org/cgi?action=article;sid=20221027232308) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Charles - BSD Now Bingo (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/480/feedback/Charles%20-%20BSD%20Now%20Bingo.md) Jake - FreeBSD Security defaults (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/480/feedback/Jake%20-%20FreeBSD%20Security%20defaults.md) Sam - FreeBSD and SSDs (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/480/feedback/Sam%20-%20FreeBSD%20and%20SSDs.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
EuroBSDcon 2022 as first BSD conference, Red Hat's OpenShift vs FreeBSD Jails, Running a Docker Host under OpenBSD using vmd(8), history of sending signals to Unix process groups, Toolchains adventures - Q3 2022, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines EuroBSDCon 2022, my first BSD conference (and how they are different) (https://eerielinux.wordpress.com/2022/09/25/eurobsdcon-2022-my-first-bsd-conference-and-how-they-are-different/) Red Hat's OpenShift vs FreeBSD Jails (https://klarasystems.com/articles/red-hats-openshift-vs-freebsd-jails/) News Roundup The history of sending signals to Unix process groups (https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessGroupsAndSignals) Running a Docker Host under OpenBSD using vmd(8) (https://www.tumfatig.net/2022/running-docker-host-openbsd-vmd/) Toolchains adventures - Q3 2022 (https://www.cambus.net/toolchains-adventures-q3-2022/) Beastie Bits -current has moved to 7.2 (https://undeadly.org/cgi?action=article;sid=20220912055003) Several /sbin daemons are now dynamically-linked (http://undeadly.org/cgi?action=article;sid=20220830052924) Announcing the pkgsrc 2022Q3 branch (https://mail-index.netbsd.org/netbsd-announce/2022/09/29/msg000341.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Hans - datacenters and dust (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/Hans%20-%20datacenters%20and%20dust.md) Tim - Boot issue (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/Tim%20-%20Boot%20issue.md) aaron- dwm tiling (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/476/feedback/aaron-%20dwm%20tiling%20.md) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Deploying FreeBSD on Oracle Cloud, A Tale of 300,000 Imaginary Friends, EuroBSDcon 2022 recap, OpenBSD Mastery: Filesystems” Status Report, OpenBGPD 7.6 Released, immutable userland mappings, Portable OpenSSH commits now SSH-signed, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines Deploying FreeBSD on Oracle Cloud (https://klarasystems.com/articles/deploying-freebsd-on-oracle-cloud/) The Things Spammers Believe - A Tale of 300,000 Imaginary Friends (https://bsdly.blogspot.com/2022/09/the-things-spammers-believe-tale-of.html) EuroBSDcon 2022 (https://peter.czanik.hu/posts/eurobsdcon2022/) News Roundup “OpenBSD Mastery: Filesystems” Status Report (https://mwl.io/archives/22031) OpenBGPD 7.6 Released (https://undeadly.org/cgi?action=article;sid=20220916051806) OpenBSD may soon gain further memory protections: immutable userland mappings (http://undeadly.org/cgi?action=article;sid=20220902100648) Portable OpenSSH commits now SSH-signed (https://undeadly.org/cgi?action=article;sid=20220902045137) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Quarterly Status Report First Quarter 2022 (https://www.freebsd.org/status/report-2022-01-2022-03/) Installing Nginx on OpenBSD 7.1 (https://unixcop.com/installing-nginx-on-openbsd-7-1/) News Roundup Live Webinar: Open-source Virtualization: Getting started with bhyve (https://klarasystems.com/webinars/live-sessions-singup/webinar-open-source-virtualization-getting-started-with-bhyve/) Hosted by Jim Salter and Allan Jude Live July 12th at 13:00 ET Available on-demand a few days later Persistent Memory Allocation (https://queue.acm.org/detail.cfm?id=3534855) Colorize your BSD shell (https://forums.FreeBSD.org/threads/colorize-your-bsd-shell.85458/) How to Install cgit With Gitolite and Nginx on FreeBSD 13 (https://herrbischoff.com/2021/10/how-to-install-cgit-with-gitolite-and-nginx-on-freebsd-13) EuroBSDCon 2022 (Austria) Program announced (https://2022.eurobsdcon.org/program/) Come to Austria and learn about the latest happenings in the BSDs 2 days of tutorials, and 2 days of 3 concurrent tracks of talks Registration is open now. See you there! *** Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Brad - Drive question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Brad%20-%20Drive%20question.md) Carl - Wiring question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Carl%20-%20Wiring%20question.md) Jon - Jails question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/461/feedback/Jon%20-%20Jails%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
We talk about our recent trip to FOSDEM, we discuss the pros and cons of permissive licensing, cover the installation of OpenBSD on a dedibox with full-disk encryption, the new Lumina guide repository, and we explain ZFS vs. OpenZFS. This episode was brought to you by Headlines [FOSDEM Trip report] Your BSDNow hosts were both at FOSDEM in Brussels, Belgium over the weekend. On the friday before FOSDEM, we held a FreeBSD devsummit (3rd consecutive year), sponsored by the FreeBSD Foundation and organized by Benedict (with the help from Kristof Provost, who did it in previous years but could not make it this year). We had 21 people attend, a good mixture of FreeBSD committers (mostly ports) and guests. After introductions, we collected topics and discussed various topics, including a new plan for a future FreeBSD release roadmap (more frequent releases, so that features from HEAD can be tried out earlier in RELEASES). The devsummit concluded with a nice dinner in a nearby restaurant. On Saturday, first day of FOSDEM, we set up the FreeBSD Foundation table with flyers, stickers, FreeBSD Journal print editions, and a small RPI 3 demo system that Deb Goodkin brought. Our table was located next to the Illumos table like last year. This allowed us to continue the good relationship that we have with the Illumos people and Allan helped a little bit getting bhyve to run on Illumos with UEFI. Meanwhile, our table was visited by a lot of people who would ask questions about FreeBSD, take info material, or talk about their use cases. We were busy refilling the table throughout the day and luckily, we had many helpers at the table. Some items we had ran out in the early afternoon, an indicator of how popular they were. Saturday also featured a BSD devroom (https://twitter.com/fosdembsd), organized by Rodrigo Osorio. You can find the list of talks and the recordings on the BSD Devroom schedule (https://fosdem.org/2018/schedule/track/bsd/). The room was very crowded and popular. Deb Goodkin gave the opening talk with an overview of what the Foundation is doing to change the world. Other speakers from various BSD projects presented their talks after that with a range of topics. Among them, Allan gave his talk about ZFS: Advanced Integration (https://fosdem.org/2018/schedule/event/zfs_advanced_integration/), while Benedict presented his Reflections on Teaching a Unix Class With FreeBSD (https://fosdem.org/2018/schedule/event/reflections_on_reaching_unix_class_with_freebsd/). Sunday was just as busy on the FreeBSD table as Saturday and we finally ran out of stickers and some other goodies. We were happy with the results of the two days. Some very interesting conversations at the table about FreeBSD took place, some of which we're going to follow up afterwards. Check out the FOSDEM schedule as many talk recordings are already available, and especially the ones from the BSD devroom if you could not attend the conference. We would like to thank everyone who attended the FreeBSD devsummit, who helped out at the FreeBSD table and organized the BSD devroom. Also, thanks to all the speakers, organizers, and helping hands making FOSDEM another success this year. *** NetBSD kernel wscons IOCTL vulnerable bug class (http://blog.infosectcbr.com.au/2018/01/netbsd-kernel-wscons-ioctl-vulnerable.html) I discovered this bug class during the InfoSect public code review session we ran looking specifically at the NetBSD kernel. I found a couple of these bugs and then after the session was complete, I went back and realised the same bug was scattered in other drivers. In total, 17 instances of this vulnerability and its variants were discovered. In all fairness, I came across this bug class during my kernel audits in 2002 and most instances were patched. It just seems there are more bugs now in NetBSD while OpenBSD and FreeBSD have practically eliminated them. See slide 41 in http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf (http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf) for exactly the same bug (class) 16 years ago. The format of the this blog post is as follows: Introduction Example of the Bug Class How to Fix How to Detect Automatically with Coccinelle More Bugs Conclusion These source files had bugs ./dev/tc/tfb.c ./dev/ic/bt485.c ./dev/pci/radeonfb.c ./dev/ic/sti.c ./dev/sbus/tcx.c ./dev/tc/mfb.c ./dev/tc/sfb.c ./dev/tc/stic.c ./dev/tc/cfb.c ./dev/tc/xcfb.c ./dev/tc/sfbplus.c ./arch/arm/allwinner/awin_debe.c ./arch/arm/iomd/vidcvideo.c ./arch/pmax/ibus/pm.c ./dev/ic/igfsb.c ./dev/ic/bt463.c ./arch/luna68k/dev/lunafb.c Reporting of the bugs was easy. In less than a week from reporting the specific instances of each bug, patches were committed into the mainline kernel. Thanks to Luke Mewburn from NetBSD for coming to the code review session at InfoSect and coordinating with the NetBSD security team. The patches to fix these issues are in NetBSD: https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html (https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html) "Permissive licensing is wrong!” – Is it? (https://eerielinux.wordpress.com/2017/11/25/permissive-licensing-is-wrong-is-it-1-2/) A few weeks ago I've been attacked by some GNU zealots on a German tech site after speaking in favor of permissive licenses. Unfortunately a discussion was not possible there because that would require the will to actually communicate instead of simply accusing the other side of vile motives. Since I actually do care about this topic and a reader asked for a post about it in comments a while ago, here we go. This first part tries to sum up the most important things around the topic. I deliberately aim for an objective overview that tries not to be one-sided. The second part will then contain my points in defence of permissive licensing. Why license software at all? Licenses exist for reasons of protection. If you're the author/inventor of some software, a story or whatever product, you get to decide what to do with it. You can keep it for yourself or you can give it away. If you decide for the latter, you have to decide who may use it and in which way(s). In case you intend to give it to a (potentially) large group of people, you may not want to be asked for permission to xyz by everybody. That's when you decide to write a license which states what you are allowing and explicitly disallowing. Most of the well-known commercial licenses focus on what you're not allowed to do (usually things like copying, disassembling, etc.). Open source licenses on the other hand are meant to grant the user rights (e.g. the right to distribute) while reserving some rights or only giving permission under certain conditions – and they usually make you claim responsibility for using the software. For these reasons licenses can actually be a good thing! If you got an unlicensed piece of code, you're not legally allowed to do anything with it without getting the author's permission first. And even if you got that permission, your project would be risky, since the author can withdraw it later. A proper license protects both parties. The author doesn't get his mail account full of email asking for permission, he's save from legal trouble if his code breaks anything for you and at the same time you have legal certainty when you decide to put the code to long-term use. Permissive vs. Copyleft (in a nutshell) In short terms, permissive licensing usually goes like this: “Here you are, have fun. Oh, and don't sue me if it does something else than what you expect!” Yes, it's that easy and there's little to dispute over. Copyleft on the other side sounds like this (if you ask somebody in favor of Copyleft): “Sure, you can use it, it's free. Just keep it free, ok?”. Also quite simple. And not too bad, eh? Other people however read the same thing like this: “Yes, you're free to use it. Just read these ten pages of legalese and be dead certain that you comply. If you got something wrong, we will absolutely make you regret it.” The GNU Public license (GPL) The most popular copyleft license in use is the GPL (in various versions) (https://www.gnu.org/licenses/gpl.html). It got more and more complex with each version – and to be fair, it had to, because it was necessary to react to new threats and loop holes that were found later. The GNU project states that they are committed to protect what they call the four freedoms of free software: the freedom to use the software for any purpose the freedom to change the software to suit your needs the freedom to share the software with your friends and neighbors the freedom to share the changes you make These are freedoms that every supporter of open source software should be able to agree with. So what's the deal with all the hostility and fighting between the two camps? Let's take a look at a permissive license, too. The BSD license Unlike the GPL, the BSD family of licenses begun with a rather simple license that span four rules (“original BSD license”). It was later revised and reduced to three (“modified BSD license”). And the modern BSD license that e.g. FreeBSD uses is even just two (“simplified BSD license”). Did you read the GPLv3 that I linked to above? If you are using GPL'd code you really should. In case you don't feel like reading all of it, at least take a look and grasp how long that text is. Now compare it to the complete modern BSD license (https://opensource.org/licenses/bsd-license.php). What's the problem? There are essentially two problems that cause all the trouble. The first one is the question of what should be subject to the freedom that we're talking about. And closely related, the second one is where that freedom needs to end. Ironically both camps claim that freedom is the one important thing and it must not be restricted. The GPL is meant to protect the freedom of the software and enforces the availability of the source code, hence limiting the freedom of actual persons. BSD on the other hand is meant to protect the freedom of human beings who should be able to use the software as they see fit – even if that means closing down former open source code! The GNU camp taunts permissive licenses as being “lax” for not providing the protection that they want. The other camp points out that the GPL is a complex monster and that it is virulent in nature: Since it's very strict in a lot of areas, it's incompatible with many other licenses. This makes it complicated to mix GPL and non-GPL code and in the cases where it's legally possible, the GPL's terms will take precedence and necessarily be in effect for the whole combined work. Who's right? That totally depends on what you want to achieve. There are pros and cons to both – and in fact we're only looking at the big picture here. There's also e.g. the Apache license which is often deemed as kind of middle ground. Then you may want to consider the difference between weak (e.g. LGPL) as well as strong copyleft (GPL). Licensing is a potentially huge topic. But let's keep it simple here because the exact details are actually not necessary to understand the essence of our topic. In the next post I'll present my stance on why permissive licensing is a good thing and copyleft is more problematic than many people may think. “Permissive licensing is wrong?” – No it's not! (https://eerielinux.wordpress.com/2018/01/25/permissive-licensing-is-wrong-no-its-not-2-2/) The previous post gave a short introduction into the topic of software licenses, focusing on the GPL vs. BSD discussion. This one is basically my response to some typical arguments I've seen from people who seem to loathe permissive licensing. I'll write this in dialog style, hoping that this makes it a little lighter to read. Roundup Install OpenBSD on dedibox with full-disk encryption (https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/) TL;DR: I run several "dedibox" servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server OpenBSD is not officially supported, I can't guarantee that this will work for you on any kind of server online.net provides, however I've been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up. Currently, I'm running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them. Recently I've been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references. I wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks. The method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice. Step #1: boot to rescue mode Step #2: boot to the installer Step #3: prepare softraid Step #4: reboot to encrypted OpenBSD system Bonus: further tightening your system enable doas disable the root account update system with syspatch add my ssh public key to my ~/.ssh/authorized_keys disable password authentication within ssh reboot so you boot on a brand new up-to-date system with latest stable kernel VOILA ! January 2018 Development Projects Update (https://www.freebsdfoundation.org/blog/january-2018-development-projects-update/) Spectre and Meltdown in FreeBSD Issues affecting most CPUs used in servers, desktops, laptops, and mobile devices are in the news. These hardware vulnerabilities, known by the code-names “Meltdown” and “Spectre”, allow malicious programs to read data to which they should not have access. This potentially includes credentials, cryptographic material, or other secrets. They were originally identified by a researcher from Google's Project Zero, and were also independently discovered by researchers and academics from Cyberus Technology, Graz University of Technology, the University of Pennsylvania, the University of Maryland, Rambus, the University of Adelaide and Data61. These vulnerabilities affect many CPU architectures supported by FreeBSD, but the 64-bit x86 family of processors from Intel and AMD are the most widely used, and are a high priority for software changes to mitigate the effects of Meltdown and Spectre. In particular, the Meltdown issue affects Intel CPUs and may be used to extract secret data from the running kernel, and therefore, is the most important issue to address. The FreeBSD Foundation collaborates with Intel, and under this relationship participated in a briefing to understand the details of these issues and plan the mitigations to be applied to the x86 architectures supported by FreeBSD. We also made arrangements to have FreeBSD's security officer join me in the briefing. It is through the generous support of the Foundation's donors that we are able to dedicate resources to focus on these issues on demand as they arise. Foundation staff member Konstantin (Kostik) Belousov is an expert on FreeBSD's Virtual Memory (VM) system as well as low-level x86 details, and is developing the x86 kernel mitigations for FreeBSD. The mitigation for Meltdown is known as Page Table Isolation (PTI). Kostik created a PTI implementation which was initially committed in mid-January and is available in the FreeBSD-CURRENT development repository. This is the same approach used by the Linux kernel to mitigate Meltdown. One of the drawbacks of the PTI mitigation is that it incurs a performance regression. Kostik recently reworked FreeBSD's use of Process-Context Identifiers (PCID) in order to regain some of the performance loss incurred by PTI. This change is also now available in FreeBSD-CURRENT. The issue known as Spectre comes in two variants, and variant 2 is the more troubling and pressing one. It may be mitigated in one of two ways: by using a technique called “retpoline” in the compiler, or by making use of a CPU feature introduced in a processor microcode update. Both options are under active development. Kostik's change to implement the CPU-based mitigation is currently in review. Unfortunately, it introduces a significant performance penalty and alternatives are preferred, if available. For most cases, the compiler-based retpoline mitigation is likely to be the chosen mitigation. Having switched to the Clang compiler for the base system and most of the ports collection some years ago, FreeBSD is well-positioned to deploy Clang-based mitigations. FreeBSD developer Dimitry Andric is spearheading the update of Clang/LLVM in FreeBSD to version 6.0 in anticipation of its official release; FreeBSD-CURRENT now includes an interim snapshot. I have been assisting with the import, particularly with respect to LLVM's lld linker, and will support the integration of retpoline. This support is expected to be merged into FreeBSD in the coming weeks. The Foundation's co-op students have also participated in the response to these vulnerabilities. Mitchell Horne developed the patch to control the PTI mitigation default setting, while Arshan Khanifar benchmarked the performance impact of the in-progress mitigation patches. In addition, Arshan and Mitchell each developed changes to FreeBSD's tool chain to support the full set of mitigations that will be applied. These mitigations will continue be tested, benchmarked, and refined in FreeBSD-CURRENT before being merged into stable branches and then being made available as updates to FreeBSD releases. Details on the timing of these merges and releases will be shared as they become available. I would like to acknowledge all of those in the FreeBSD community who have participated in FreeBSD's response to Meltdown and Spectre, for testing, reviewing, and coordinating x86 mitigations, for developing mitigations for other processor architectures and for the Bhyve hypervisor, and for working on the toolchain-based mitigations. Guides: Getting Started & Lumina Theme Submissions (https://lumina-desktop.org/guides-getting-started-lumina-themes/) I am pleased to announce the beginning of a new sub-series of blog posts for the Lumina project: Guides! The TrueOS/Lumina projects want to support our users as they use Lumina or experiment with TrueOS. To that end, we've recently set up a central repository for our users to share instructions or other “how-to” guides with each other! Project developers and contributors will also submit guides to the repository on occasion, but the overall goal is to provide a simple hub for instructions written by any Lumina or TrueOS user. This will make it easier for users to not only find a “how-to” for some procedure, but also a very easy way to “give back” to the community by writing simple instructions or more detailed guides. Guides Repository Our first guide to get the whole thing started was created by the TrueOS Linebacker (https://discourse.trueos.org/t/introducing-the-trueos-linebacker/991) (with technical assistance from our own q5sys). In this guide, Terry Tate will walk you through the steps necessary to submit new wallpaper images to the Lumina Themes collection. This procedure is fully documented with screenshots every step of the way, walking you through a simple procedure that only requires a web browser and a Github account! Guide: Lumina Themes Submissions (https://github.com/trueos/guides/blob/master/lumina-themes-submissions/readme.md) The end result of this guide was that Terry Tate was able to submit this cool new “Lunar-4K” wallpaper to the “lumina-nature” collection. TrueOS Community Guides (https://github.com/trueos/guides/tree/master) ZFS vs. OpenZFS (by Michael Dexter) (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You've probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. Our Senior Analyst clears up what ZFS and OpenZFS refer to and how they differ. I admit that we geeks tend to get caught up in the nuts and bolts of enterprise storage and overlook the more obvious questions that users might have. You've probably noticed that this blog and the FreeNAS blog refer to “ZFS” and “OpenZFS” seemingly at random when talking about the amazing file system at the heart of FreeNAS and every storage product that iXsystems sells. I will do my best to clarify what exactly these two terms refer to. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset' and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Yes, ZFS is really as good as people say it is. After enjoying nearly a decade of refinement by a growing group of developers around the world, ZFS became the property of database vendor Oracle, which ceased public development of both ZFS and OpenSolaris in 2010. Disappointed but undeterred, a group of OpenSolaris users and developers forked the last public release of OpenSolaris as the Illumos project. To this day, Illumos represents the official upstream home of the Open Source OpenSolaris technologies, including ZFS. The Illumos project enjoys healthy vendor and user participation but the portable nature and compelling features of ZFS soon produced far more ZFS users than Illumos users around the world. While most if not all users of Illumos and its derivatives are ZFS users, the majority of ZFS users are not Illumos users, thanks significantly in part to FreeNAS which uses the FreeBSD operating system. This imbalance plus several successful ZFS Day events led ZFS co-founder Matt Ahrens and a group of ZFS developers to announce the OpenZFS project, which would remain a part of the Illumos code base but would be free to coordinate development efforts and events around their favorite file system. ZFS Day has grown into the two-day OpenZFS Developer Summit and is stronger than ever, a testament to the passion and dedication of the OpenZFS community. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs' command, which on FreeBSD relies on the ‘zfs.ko' kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. Beastie Bits Explaining Shell (https://explainshell.com/) OPNsense® 18.1 Released (https://opnsense.org/opnsense-18-1-released/) “SSH Mastery 2/e” copyedits back (https://blather.michaelwlucas.com/archives/3104) Sponsoring a Scam (https://blather.michaelwlucas.com/archives/3106) Thursday, February 8, 2018 - Come to Netflix to talk about FreeBSD (https://www.meetup.com/BAFUG-Bay-Area-FreeBSD-User-Group/events/246623825/) BSD User Group meeting in Stockholm: March 22, 17:30 - 21:00 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) FreeBSD Flavoured talks from Linux.conf.au: You can't unit test C, right? (https://www.youtube.com/watch?v=z-uWt5wVVkU) and A Brief History of I/O (https://www.youtube.com/watch?v=qAhZEI_6lbc) EuroBSDcon 2018 website is up (https://2018.eurobsdcon.org/) Full day bhyvecon Tokyo, Japan, March 9, 2018 (http://bhyvecon.org/) *** Feedback/Questions Thomas - freebsd installer improvements (http://dpaste.com/3G2F7RC#wrap) Mohammad - FreeBSD 11 installation from a read only rescue disk (http://dpaste.com/0HGK3FQ#wrap) Stan - Follow up on guide you covered (http://dpaste.com/2S169SH#wrap) Jalal - couple questions (http://dpaste.com/35N8QXP#wrap)
We review the information about Spectre & Meltdown thus far, we look at NetBSD memory sanitizer progress, Postgres on ZFS & show you a bit about NomadBSD. This episode was brought to you by Headlines Meltdown Spectre Official Site (https://meltdownattack.com/) Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign (https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/) Intel's official response (https://newsroom.intel.com/news/intel-responds-to-security-research-findings/) The Register mocks intels response with pithy annotations (https://www.theregister.co.uk/2018/01/04/intel_meltdown_spectre_bugs_the_registers_annotations/) Intel's Analysis PDF (https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-of-Speculative-Execution-Side-Channels.pdf) XKCD (https://xkcd.com/1938/) Response from FreeBSD (https://lists.freebsd.org/pipermail/freebsd-security/2018-January/009719.html) FreeBSD's patch WIP (https://reviews.freebsd.org/D13797) Why Raspberry Pi isn't vulnerable to Spectre or Meltdown (https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/) Xen mitigation patches (https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00110.html) Overview of affected FreeBSD Platforms/Architectures (https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities) Groff's response (https://twitter.com/GroffTheBSDGoat/status/949372300368867328) ##### We'll cover OpenBSD, NetBSD, and DragonflyBSD's responses in next weeks episode. *** ###The LLVM Memory Sanitizer support work in progress (https://blog.netbsd.org/tnf/entry/the_llvm_memory_sanitizer_support) > In the past 31 days, I've managed to get the core functionality of MSan to work. This is an uninitialized memory usage detector. MSan is a special sanitizer because it requires knowledge of every entry to the basesystem library and every entry to the kernel through public interfaces. This is mandatory in order to mark memory regions as initialized. Most of the work has been done directly for MSan. However, part of the work helped generic features in compiler-rt. Sanitizers > Changes in the sanitizer are listed below in chronological order. Almost all of the changes mentioned here landed upstream. A few small patches were reverted due to breaking non-NetBSD hosts and are rescheduled for further investigation. I maintain these patches locally and have moved on for now to work on the remaining features. NetBSD syscall hooks > I wrote a large patch (815kb!) adding support for NetBSD syscall hooks for use with sanitizers. NetBSD ioctl(2) hooks > Similar to the syscall hooks, there is need to handle every ioctl(2) call. I've created the needed patch, this time shorter - for less than 300kb. New patches still pending for upstream review > There are two corrections that I've created, and they are still pending upstream for review: Add MSan interceptor for fstat(2)](https://reviews.llvm.org/D41637) Correct the setitimer interceptor on NetBSD)](https://reviews.llvm.org/D41502) > I've got a few more local patches that require cleanup before submitting to review. NetBSD basesystem corrections Sanitizers in Go The MSan state as of today Solaris support in sanitizers > I've helped the Solaris team add basic support for Sanitizers (ASan, UBsan). This does not help NetBSD directly, however indirectly it improves the overall support for non-Linux hosts and helps to catch more Linuxisms in the code. Plan for the next milestone > I plan to continue the work on MSan and correct sanitizing of the NetBSD basesystem utilities. This mandates me to iterate over the basesystem libraries implementing the missing interceptors and correcting the current support of the existing ones. My milestone is to build all src/bin programs against Memory Sanitizer and when possible execute them cleanly. This work was sponsored by The NetBSD Foundation. The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: http://netbsd.org/donations/#how-to-donate (http://netbsd.org/donations/#how-to-donate) *** ##News Roundup ###MWL's 2017 Wrap-Up (https://blather.michaelwlucas.com/archives/3078) > The obvious place to start is my 2016 wrap-up post](https://blather.michaelwlucas.com/archives/2822), where I listed goals for 2017. As usual, these goals were wildly delusional. > The short answer is, my iron was back up to normal. My writing speed wasn't, though. I'd lost too much general health, and needed hard exercise to recover it. Yes, writing requires physical endurance. Maintaining that level of concentration for several hours a day demands a certain level of blood flow to the brain. I could have faked it in a day job, but when self-employed as an artist? Not so much. > Then there's travel. I did my usual BSDCan trip, plus two educational trips to Lincoln City, Oregon. The current political mayhem convinced me that if I wanted to hit EuroBSDCon any time in the next few years, I should do it in the very near future. So I went to Paris, where I promptly got pickpocketed. (Thankfully, they didn't get my passport.) I was actively writing the third edition of Absolute FreeBSD, so I visited BSDCam in Cambridge to get the latest information and a sense of where FreeBSD was going. I also did weekends at Kansas LinuxFest (because they asked and paid for my trip) and Penguicon. > (Because people will ask: why EuroBSDCon and not AsiaBSDCon? A six-hour transatlantic flight requires that I take a substantial dose of heavy-grade tranquilizers. I'm incapable of making intelligent decisions while on those drugs, or for several hours afterward. They don't last long enough for twelve-hour flight to Japan, so I need to be accompanied by someone qualified to tell me when I need to take the next dose partway through the flight. This isn't a predetermined time that I can set an alarm for; it depends on how the clonazepam affects me at those altitudes. A drug overdose while flying over the North Pole would be bad. When I can arrange that qualified companion, I'll make the trip.) > I need most of the preceding week to prepare for long trips. I need the following week to recover from time shifts and general exhaustion. Additionally, I have to hoard people juice for a few weeks beforehand so I can deal with folks during these expeditions. Travel disrupts my dojo time as well, which impacts my health. > Taken as a whole: I didn't get nearly as much done as I hoped. I wrote more stories, but Kris Rusch bludgeoned me into submitting them to trad markets. (The woman is a brute, I tell you. Cross her at your peril.) Among my 2017 titles, my fiction outsold the tech books. No, not Prohibition Orcs–all four of the people who buy those love them, but the sales tell me I've done something wrong with those tales. My cozy mystery git commit murder outsold Relayd and Httpd Mastery. But what outdid them both, as well as most of my older books? What title utterly dominated my sales for the last quarter of the year? It was of course, my open source software political satire disguised as porn Savaged by Systemd: an Erotic Unix Encounter. (https://www.michaelwarrenlucas.com/index.php/romance#sbs) > I can't believe I just wrote that paragraph. The good news is, once I recovered from EuroBSDCon, my writing got better. I finished Absolute FreeBSD, 3rd edition and submitted it to the publisher. I wrote the second edition of SSH Mastery (no link, because you can't order it yet.) I'm plowing through git sync murder, the sequel to git commit murder. I don't get to see the new Star Wars movie until I finish GSM, so hopefully that'll be this month. All in all, I wrote 480,200 words in 2017. Most of that was after September. It's annoyingly close to breaking half a million, but after 2016's scandalous 195,700, I'll take it. *** ###PG Phriday: Postgres on ZFS (https://blog.2ndquadrant.com/pg-phriday-postgres-zfs/) > ZFS is a filesystem originally created by Sun Microsystems, and has been available for BSD over a decade. While Postgres will run just fine on BSD, most Postgres installations are historically Linux-based systems. ZFS on Linux has had much more of a rocky road to integration due to perceived license incompatibilities. > As a consequence, administrators were reluctant or outright refused to run ZFS on their Linux clusters. It wasn't until OpenZFS was introduced in 2013 that this slowly began to change. These days, ZFS and Linux are starting to become more integrated, and Canonical of Ubuntu fame even announced direct support for ZFS in their 16.04 LTS release. > So how can a relatively obscure filesystem designed by a now-defunct hardware and software company help Postgres? Let's find out! Eddie waited til he finished high school > Old server hardware is dirt cheap these days, and make for a perfect lab for testing suspicious configurations. This is the server we'll be using for these tests for those following along at home, or want some point of reference: Dell R710 x2 Intel X5660 CPUs, for up to 24 threads 64GB RAM x4 1TB 7200RPM SATA HDDs H200 RAID card configured for Host Bus Adapter (HBA) mode 250GB Samsung 850 EVO SSD > The H200 is particularly important, as ZFS acts as its own RAID system. It also has its own checksumming and other algorithms that don't like RAID cards getting in the way. As such, we put the card itself in a mode that facilitates this use case. > Due to that, we lose out on any battery-backed write cache the RAID card might offer. To make up for it, it's fairly common to use an SSD or other persistent fast storage to act both as a write cache, and a read cache. This also transforms our HDDs into hybrid storage automatically, which is a huge performance boost on a budget. She had a guitar and she taught him some chords > First things first: we need a filesystem. This hardware has four 1TB HDDs, and a 250GB SSD. To keep this article from being too long, we've already placed GPT partition tables on all the HDDs, and split the SSD into 50GB for the OS, 32GB for the write cache, and 150GB for the read cache. A more robust setup would probably use separate SSDs or a mirrored pair for these, but labs are fair game. They moved into a place they both could afford > Let's start by getting a performance baseline for the hardware. We might expect peak performance at 12 or 24 threads because the server has 12 real CPUs and 24 threads, but query throughput actually topped out at concurrent 32 processes. We can scratch our heads over this later, for now, we can consider it the maximum capabilities of this hardware. Here's a small sample: ``` $> pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench ... tps = 264661.135288 (including connections establishing) tps = 264849.345595 (excluding connections establishing) ``` So far, this is pretty standard behavior. 260k prepared queries per second is great read performance, but this is supposed to be a filesystem demonstration. Let's get ZFS involved. + The papers said Ed always played from the heart Let's repeat that same test with writes enabled. Once that happens, filesystem syncs, dirty pages, WAL overhead, and other things should drastically reduce overall throughput. That's an expected result, but how much are we looking at, here? ``` $> pgbench -j 32 -c 32 -M prepared -T 10 pgbench ... tps = 6153.877658 (including connections establishing) tps = 6162.392166 (excluding connections establishing) ``` SSD cache or not, storage overhead is a painful reality. Still, 6000 TPS with writes enabled is a great result for this hardware. Or is it? Can we actually do better? Consider the Postgres fullpagewrites parameter. Tomas Vondra has written about it in the past as a necessity to prevent WAL corruption due to partial writes. The WAL is both streaming replication and crash recovery, so its integrity is of utmost importance. As a result, this is one parameter almost everyone should leave alone. ZFS is Copy on Write (CoW). As a result, it's not possible to have a torn page because a page can't be partially written without reverting to the previous copy. This means we can actually turn off fullpagewrites in the Postgres config. The results are some fairly startling performance gains: $> pgbench -j 32 -c 32 -M prepared -T 10 pgbench tps = 10325.200812 (including connections establishing) tps = 10336.807218 (excluding connections establishing) That's nearly a 70% improvement. Due to write amplification caused by full page writes, Postgres produced 1.2GB of WAL files during a 1-minute pgbench test, but only 160MB with full page writes disabled. To be fair, a 32-thread pgbench write test is extremely abusive and certainly not a typical usage scenario. However, ZFS just ensured our storage a much lower write load by altering one single parameter. That means the capabilities of the hardware have also been extended to higher write workloads as IO bandwidth is not being consumed by WAL traffic. + They both met movie stars, partied and mingled Astute readers may have noticed we didn't change the default ZFS block size from 128k to align with the Postgres default of 8kb. As it turns out, the 128kb blocks allow ZFS to better combine some of those 8kb Postgres pages to save space. That will allow our measly 2TB to go a lot further than is otherwise possible. Please note that this is not de-duplication, but simple lz4 compression, which is nearly real-time in terms of CPU overhead. De-duplication on ZFS is currently an uncertain bizzaro universe populated with misshapen horrors crawling along a broken landscape. It's a world of extreme memory overhead for de-duplication tables, and potential lost data due to inherent conflicts with the CoW underpinnings. Please don't use it, let anyone else use it, or even think about using it, ever. + They made a record and it went in the chart We're still not done. One important aspect of ZFS as a CoW filesystem, is that it has integrated snapshots. Consider the scenario where a dev is connected to the wrong system and drops what they think is a table in a QA environment. It turns out they were in the wrong terminal and just erased a critical production table, and now everyone is frantic. + The future was wide open It's difficult to discount an immediately observable reduction in write overhead. Snapshots have a multitude of accepted and potential use cases, as well. In addition to online low-overhead compression, and the hybrid cache layer, ZFS boasts a plethora of features we didn't explore. Built-in checksums with integrated self-healing suggest it isn't entirely necessary to re-initialize an existing Postgres instance to enable checksums. The filesystem itself ensures checksums are validated and correct, especially if we have more than one drive resource in our pool. It even goes the extra mile and actively corrects inconsistencies when encountered. I immediately discounted ZFS back in 2012 because the company I worked for at the time was a pure Linux shop. ZFS was only available using the FUSE driver back then, meaning ZFS only worked through userspace with no real kernel integration. It was fun to tinker with, but nobody sane would use that on a production server of any description. Things have changed quite drastically since then. I've stopped waiting for btrfs to become viable, and ZFS has probably taken the throne away from XFS as my filesystem of choice. Future editions of the Postgres High Availability Cookbook will reflect this as well. Postgres MVCC and ZFS CoW seem made for each other. I'm curious to see what will transpire over the next few years now that ZFS has reached mainstream acceptance in at least one major Linux distribution. NomadBSD (https://github.com/mrclksr/NomadBSD) About NomadBSD is a live system for flash drives, based on FreeBSD. Screenshots http://freeshell.de/~mk/download/nomadbsd-ss1.png http://freeshell.de/~mk/download/nomadbsd-ss2.png Requirements for building the image A recent FreeBSD system Requirements for running NomadBSD A 4GB (or more) flash drive A System capable running FreeBSD 11.1 (amd64) Building the image ~~ csh # make image ~~ Writing the image to an USB memory stick ~~ csh # dd if=nomadbsd.img of=/dev/da0 bs=10240 conv=sync ~~ Resize filesystem to use the entire USB memory Boot NomadBSD into single user mode, and execute: ~~ # gpart delete -i 2 da0s1 # gpart resize -i 1 da0 # gpart commit da0s1 ~~ Determine the partition size in megabytes using fdisk da0 and calculate the remaining size of da0s1a: = - . ~~ # gpart resize -i 1 -s M da0s1 # gpart add -t freebsd-swap -i 2 da0s1 # glabel label NomadBSDsw da0s1b # service growfs onestart # reboot ~~ FreeBSD forum thread (https://forums.freebsd.org/threads/63888/) A short screen capture video of the NomadBSD system running in VirtualBox (https://freeshell.de/~mk/download/nomad_capture.mp4) *** ##Beastie Bits Coolpkg, a package manager inspired by Nix for OpenBSD (https://github.com/andrewchambers/coolpkg) zrepl - ZFS replication (https://zrepl.github.io/) OpenBSD hotplugd automount script (https://bijanebrahimi.github.io/blog/openbsd-hotplugd-scripting.html) Ancient troff sources vs. modern-day groff (https://virtuallyfun.com/2017/12/22/learn-ancient-troff-sources-vs-modern-day-groff/) Paypal donation balance and status.. thanks everyone! (http://lists.dragonflybsd.org/pipermail/users/2017-December/313752.html) Supervised FreeBSD rc.d script for a Go daemon (updated in last few days) (https://redbyte.eu/en/blog/supervised-freebsd-init-script-for-go-deamon/) A Brief History of sed (https://blog.sourcerer.io/a-brief-history-of-sed-6eaf00302ed) Flamegraph: Why does my AWS instance boot so slow? (http://www.daemonology.net/timestamping/tslog-c5.4xlarge.svg) *** ##Feedback/Questions Jeremy - Replacing Drive in a Zpool (http://dpaste.com/319593M#wrap) Dan's Blog (https://dan.langille.org/2017/08/16/swapping-5tb-in-3tb-out/) Tim - Keeping GELI key through reboot (http://dpaste.com/11QTA06) Brian - Mixing 2.5 and 3.5 drives (http://dpaste.com/2JQVD10#wrap) Troels - zfs swap on FreeBSD (http://dpaste.com/147WAFR#wrap) ***
Picking a compiler for debuggability, how to port Rust apps to FreeBSD, what the point of Docker is on FreeBSD/Solaris, another EuroBSDcon recap, and network manager control in OpenBSD This episode was brought to you by Headlines Compile once, Debug twice: Picking a compiler for debuggability, part 1 of 3 (https://backtrace.io/blog/compile-once-debug-twice-picking-a-compiler-for-debuggability-1of3/) An interesting look into why when you try to debug a crash, you can often find all of the useful information has been ‘optimized out' Have you ever had an assert get triggered only to result in a useless core dump with missing variable information or an invalid callstack? Common factors that go into selecting a C or C++ compiler are: availability, correctness, compilation speed and application performance. A factor that is often neglected is debug information quality, which symbolic debuggers use to reconcile application executable state to the source-code form that is familiar to most software engineers. When production builds of an application fail, the level of access to program state directly impacts the ability for a software engineer to investigate and fix a bug. If a compiler has optimized out a variable or is unable to express to a symbolic debugger how to reconstruct the value of a variable, the engineer's investigation process is significantly impacted. Either the engineer has to attempt to recreate the problem, iterate through speculative fixes or attempt to perform prohibitively expensive debugging, such as reconstructing program state through executable code analysis. Debug information quality is in fact not proportionally related to the quality of the generated executable code and wildly varies from compiler to compiler. Different compilers emit debug information at varying levels of quality and accuracy. However, certain optimizations will certainly impact any debugger's ability to generate accurate stack traces or extract variable values. In the above program, the value of argv is extracted and then the program is paused. The ckprloadptr function performs a read from the region of memory pointed to by argv, in a manner that prevents the compiler from performing optimization on it. This ensures that the memory access occurs and for this reason, the value of argv must be accessible by the time ckprloadptr is executed. When compiled with gcc, the debugger fails to find the value of the variable. The compiler determines that the value of argv is no longer needed after the ckprload_ptr operation and so doesn't bother paying the cost of saving the value. Some optimizations generate executable code whose call stack cannot be sufficiently disambiguated to reconcile a call stack that mirrors that of the source program. Two common culprits for this are tail call optimization and basic block commoning. In another example If the program receives a first argument of 1, then function is called with the argument of "a". If the program receives a first argument of 2, then function is called with the argument of "b". However, if we compile this program with clang, the stack traces in both cases are identical! clang informs the debugger that the function f invoked the function("b") branch where x = 2 even if x = 1. Though some optimizations will certainly impact the accuracy of a symbolic debugger, some compilers simply lack the ability to generate debug information in the presence of certain optimizations. One common optimization is induction variable elimination. A variable that's incremented or decremented by a constant on every iteration of a loop or derived from another variable that follows this pattern, is an induction variable. Coupled with other optimizations, the compiler is then able to generate code that doesn't actually rely on a dedicated counter variable “i” for maintaining the current offset into “buffer”. As you can see, i is completely optimized out. The compiler determines it doesn't have to pay the cost of maintaining the induction variable i. It maintains the pointer in the register %rdi. The code is effectively rewritten to something closer to this: So the for loop, changes into a while loop, with a condition of the end of the input We have shown some common optimizations that may get in the way of the debuggability of your application and demonstrated a disparity in debug information quality across two popular compilers. In the next blog post of this series, we will examine how gcc and clang stack up with regards to debug information quality across a myriad of synthetic applications and real world applications. Looking forward to part 2 *** This is how you can port your rust application to FreeBSD (https://medium.com/@andoriyu/this-is-how-you-can-port-your-rust-application-to-freebsd-7d3e9f1bc3df) This is how you can port your rust application to FreeBSD The FreeBSD Ports Collection is the way almost everyone installs applications (“ports”) on FreeBSD. Like everything else about FreeBSD, it is primarily a volunteer effort. It is important to keep this in mind when reading this document. In FreeBSD, anyone may submit a new port, or volunteer to maintain an existing unmaintained port. No special commit privilege is needed. For this guide I will use fd tool written by David Peter as example project. Prerequisites FreeBSD installation (VM is fine) Local ports tree (done via svn) portlint (located at devel/portlint) poudriere (located at ports-mgmt/poudriere)[optional] Getting ports tree When you install FreeBSD opt-out of the ports tree. Install svn: pkg install svn svn checkout https://svn.freebsd.org/ports/head /usr/ports Poudriere Sometimes you might get asked to show poudriere build log, sometimes you won't. It's good to have anyway. If you choose to use poudriere, use ZFS. There are plenty of guides on the subject. FreeBSD Porter's Handbook is the most complete source of information on porting to FreeBSD. Makefile Whole porting process in most cases is writing one Makefile. I recommend doing something like this. Here is the one I wrote for fd: Port metadata Each port must have one primary category in case of fd it will be sysutils, therefore it's located in /usr/ports/systuils/fd. PORTNAME= fd CATEGORIES= sysutils Since this port conflicts with other util named fd I specified package suffix as: PKGNAMESUFFIX= -find and indicate conflict: CONFLICTS_INSTALL= fd-[0-9]*. That means to install it from packages user will have to type: pkg install fd-find Licenses This section is different for every port, but in case of fd it's pretty straightforward: LICENSE= MIT APACHE20 LICENSE_COMB= dual Since fd includes the text of licenses you should do this as well: LICENSE_FILE_MIT= ${WRKSRC}/LICENSE-MIT LICENSE_FILE_APACHE20= ${WRKSRC}/LICENSE-APACHE Distfiles FreeBSD has a requirement that all ports must allow offline building. That means you have specified which files are needed to be downloaded. Luckily we now have helpers to download GitHub sources directly from GitHub: USE_GITHUB= yes GH_ACCOUNT= sharkdp Since PORTNANE is fd it will try to download sources for sharkdp/fd. By default it's going to download tag: ${DISTVERSIONPREFIX}${DISTVERSION}${DISTVERSIONSUFFIX} fd uses v as the prefix, therefore we need to specify: DISTVERSIONPREFIX= v. It's also possible to specify GH_TAGNAME in case tag name doesn't match that pattern. Extra packages There are very few rust projects that are standalone and use no crates dependencies. It's used to be PITA to make it work offline, but now cargo is a first class citizen in ports: USES= cargo CARGO_CRATES= aho-corasick-0.6.3 atty-0.2.3 # and so goes on Yes, you have to specify each dependency. Luckily, there is a magic awk script that turns Cargo.lock into what you need. Execute make cargo-crates in the port root. This will fail because you're missing checksum for the original source files: make makesum make cargo-crates This will give you what you need. Double check that result is correct. There is a way to ignore checksum error, but I can't remember… Execute make makesum again. CARGO_OUT If. build.rs relies on that you have to change it. fd allows you to use SHELLCOMPLETIONSDIR to specify where completions go, while ripgrep doesn't. In our case we just specify SHELLCOMPLETIONSDIR: SHELL_COMPLETIONS_DIR= ${WRKDIR}/shell-completions-dir CARGO_ENV= SHELL_COMPLETIONS_DIR=${SHELL_COMPLETIONS_DIR} PLIST FreeBSD is very strict about files it's installing and it won't allow you to install random files that get lost. You have to specify which files you're installing. In this case, it's just two: PLIST_FILES= bin/fd man/man1/fd.1.gz Note that sources for fd have uncompressed man file, while here it's listed as compressed. If port installs a lot of files, specify them in pkg-plist like here. To actually install them: post-install: @${STRIP_CMD} ${STAGEDIR}${PREFIX}/bin/fd ${INSTALL_MAN}${WRKSRC}/doc/fd.1 ${STAGEDIR}${MAN1PREFIX}/man/man1 Shell completions clap-rs can generate shell completions for you, it's usually handled by build.rs script. First, we need to define options: OPTIONS_DEFINE= BASH FISH ZSH # list options OPTIONS_DEFAULT= BASH FISH ZSH # select them by default BASH_PLIST_FILES= etc/bash_completion.d/fd.bash-completion FISH_PLIST_FILES= share/fish/completions/fd.fish ZSH_PLIST_FILES= share/zsh/site-functions/_fd To actually install them: post-install-BASH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/etc/bash_completion.d ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/fd.bash-completion ${STAGEDIR}${PREFIX}/etc/bash_completion.d post-install-FISH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/share/fish/completions ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/fd.fish ${STAGEDIR}${PREFIX}/share/fish/completions post-install-ZSH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/share/zsh/site-functions ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/_fd ${STAGEDIR}${PREFIX}/share/zsh/site-functions Bonus round - Patching source code Sometimes you have to patch it and send the patch upstream. Merging it upstream can take awhile, so you can patch it as part of the install process. An easy way to do it: Go to work/ dir Copy file you want to patch and add .orig suffix to it Edit file you want to patch Execute make makepatch in port's root Submitting port First, make sure portlint -AC doesn't give you any errors or warnings. Second, make sure poudriere can build it on both amd64 and i386. If it can't?—?you have to either fix it or mark port broken for that arch. Follow this steps like I did steps. If you have any issues you can always ask your question in freebsd-ports on freenode try to find your answer in porter's handbook before asking. Conference Recap: EuroBSDCon 2017 Recap (https://www.freebsdfoundation.org/blog/conference-recap-eurobsdcon-2017-recap/) The location was wonderful and I loved sneaking out and exploring the city when I could. From what I heard, it was the largest BSD conference in history, with over 320 attendees! Each venue is unique and draws many local BSD enthusiasts, who normally wouldn't be able to travel to a conference. I love having the chance to talk to these people about how they are involved in the projects and what they would like to do. Most of the time, they are asking me questions about how they can get more involved and how we can help. Magical is how I would describe the conference social event. To stand in front of the dinner cruise on the Seine, with the Eiffel Tower standing tall, lit up in the night, while working – talking to our community members, was incredible. But, let me start at the beginning. We attend these conferences to talk to our community members, to find out what they are working on, determine technologies that should be supported in FreeBSD, and what we can do to help and improve FreeBSD. We started the week with a half-day board meeting on Wednesday. BSD conferences give us a chance to not only meet with community members around the world, but to have face-to-face meetings with our team members, who are also located around the world. We worked on refining our strategic direction and goals, determining what upcoming conferences we want FreeBSD presence at and who can give FreeBSD talks and workshops there, discussed current and potential software development projects, and discussed how we can help raise awareness about and increase the use of FreeBSD in Europe. Thursday was the first day of the FreeBSD developer summit, led by our very own Benedict Reuschling. He surprised us all by having us participate in a very clever quiz on France. 45 of us signed into the software, where he'd show the question on the screen and we had a limited amount of time to select our answers, with the results listed on the screen. It was actually a lot of fun, especially since they didn't publicize the names of the people who got the questions wrong. The lucky or most knowledgeable person on France, was des@freebsd.org. Some of our board members ran tutorials in parallel to the summit. Kirk McKusick gave his legendary tutorial, An Introduction to the FreeBSD Open-Source Operating System , George Neville-Neil gave his tutorial, DTrace for Developers, and Benedict Reuschling gave a tutorial on, Managing BSD systems with Ansible. I was pleased to have two chairs from ACM-W Europe run an “Increasing Diversity in the BSDs” BoF for the second year in a row. We broke up into three groups to discuss different gender bias situations, and what we can do to address these types of situations, to make the BSD projects more diverse, welcoming, and inclusive. At the end, people asked that we continue these discussions at future BSD conferences and suggested having an expert in the field give a talk on how to increase the diversity in our projects. As I mentioned earlier, the social dinner was on a boat cruising along the Seine. I had a chance to talk to community members in a more social environment. With the conference being in France, we had a lot of first time attendees from France. I enjoyed talking to many of them, as well as other people I only get to see at the European conferences. Sunday was full of more presentations and conversations. During the closing session, I gave a short talk on the Foundation and the work we are doing. Then, Benedict Reuschling, Board Vice President, came up and gave out recognition awards to four FreeBSD contributors who have made an impact on the Project. News Roundup Playing with the pine64 (https://chown.me/blog/playing-with-the-pine64.html) Daniel Jakots writes in his blog about his experiences with his two pine64 boards: Finding something to install on it 6 weeks ago, I ordered two pine64 units. I didn't (and still don't) have much plan for them, but I wanted to play with some cheap boards. I finally received them this week. Initially I wanted to install some Linux stuff on it, I didn't have much requirement so I thought I would just look what seems to be easy and/or the best supported systemd flavour. I headed over their wiki. Everything seems either not really maintained, done by some random people or both. I am not saying random people do bad things, just that installing some random things from the Internet is not really my cup of tea. I heard about Armbian (https://www.armbian.com/pine64/) but the server flavour seems to be experimental so I got scared of it. And sadly, the whole things looks like to be alot undermanned. So I went for OpenBSD because I know the stuff and who to har^Wkindly ask for help. Spoiler alert, it's boring because it just works. Getting OpenBSD on it I downloaded miniroot62.fs, dd'ed it on the micro SD card. I was afraid I'd need to fiddle with some things like sysutils/dtb because I don't know what I would have needed to do. That's because I don't know what it does and for this precise reason I was wrong and I didn't need to do anything. So just dd the miniroot62.fs and you can go to next checkpoint. I plugged an HDMI cable, ethernet cable and the power, it booted, I could read for 10 seconds but then it got dark. Of course it's because you need a serial console. Of course I didn't have one. I thought about trying to install OpenBSD blindly, I could have probably succeeded with autoinstall buuuuuut… Following some good pieces of advice from OpenBSD people I bought some cp2102 (I didn't try to understand what it was or what were the other possibilities, I just wanted something that would work :D). I looked how to plug the thing. It appears you can plug it on two different places but if you plug it on the Euler bus it could power a bit the board so if you try to reboot it, it would then mess with the power disruption and could lead a unclean reboot. You just need to plug three cables: GND, TXD and RXD. Of course, the TXD goes on the RXD pin from the picture and the RXD goes on the TXD pin. Guess why I'm telling you that! That's it Then you can connect with the usual $ cu -dl /dev/cuaU0 -s 115200 What's the point of Docker on FreeBSD or Solaris? (http://blog.frankleonhardt.com/2017/whats-the-point-of-docker-on-freebsd-or-solaris/) Penguinisters are very keen on their docker, but for the rest of us it may be difficult to see what the fuss is all about – it's only been around a few years and everyone's talking about it. And someone asked again today. What are we missing? Well docker is a solution to a Linux (and Windows) problem that FreeBSD/Solaris doesn't have. Until recently, the Linux kernel only implemented the original user isolation model involving chroot. More recent kernels have had Control Groups added, which are intended to provide isolation for a group of processes (namespaces). This came out of Google, and they've extended to concept to include processor resource allocation as one of the knobs, which could be a good idea for FreeBSD. The scheduler is aware of the JID of the process it's about to schedule, and I might take a look in the forthcoming winter evenings. But I digress. So if isolation (containerisation in Linux terms) is in the Linux kernel, what is Docker bringing to the party? The only thing I can think of is standardisation and an easy user interface (at the expense of having Python installed). You might think of it in similar terms to ezjail – a complex system intended to do something that is otherwise very simple. To make a jail in FreeBSD all you need do is copy the files for your system to a directory. This can even be a whole server's system disk if you like, and jails can run inside jails. You then create a very simple config file, giving the jail a name, the path to your files and an what IP addresses to pass through (if any) and you're done. Just type “service jail nameofjal start”, and off it goes. Is there any advantage in running Docker? Well, in a way, there is. Docker has a repository of system images that you can just install and run, and this is what a lot of people want. They're a bit like virtual appliances, but not mind-numbingly inefficient. You can actually run docker on FreeBSD. A port was done a couple of years ago, but it relies on the 64-bit Linux emulation that started to appear in 10.x. The newer the version of FreeBSD the better. Docker is in ports/sysutils/docker-freebsd. It makes uses of jails instead of Linux cgroups, and requires ZFS rather than UFS for file system isolation. I believe the Linux version uses Union FS but I could be completely wrong on that. The FreeBSD port works with the Docker hub repository, giving you access to thousands of pre-packaged system images to play with. And that's about as far as I've ever tested it. If you want to run the really tricky stuff (like Windows) you probably want full hardware emulation and something like Xen. If you want to deploy or migrate FreeBSD or Solaris systems, just copy a new tarball in to the directory and go. It's a non-problem, so why make it more complicated? Given the increasing frequency Docker turns up in conversations, it's probably worth taking seriously as Linux applications get packaged up in to images for easy access. Jails/Zones may be more efficient, and Docker images are limited to binary, but convenience tends to win in many environments. Network Manager Control for OpenBSD (http://www.vincentdelft.be/post/post_20171023) I propose you a small script allowing you to easily manage your networks connections. This script is integrated within the openbox dynamic menus. Moreover, it allow you to automatically have the connections you have pre-defined based. I was frustrated to not be able to swap quickly from one network interface to an another, to connect simply and quickly to my wifi, to my cable connection, to the wifi of a friend, ... Every time you have to type the ifconfig commands, .... This is nice, but boring. Surely, when you are in a middle of a presentation and you just want a quick connection to your mobile in tethering mode. Thanks to OpenBSD those commands are not so hard, but this frustrate me to not be able to do it with one click. Directly from my windows environment. Since I'm using Openbox, from a menu of openbox. So, I've looked around to see what is currently existing. One tool I've found was netctl (https://github.com/akpoff/netctl). The idea is to have a repository of hostname.if files ready to use for different cases. The idea sounds great, but I had some difficulties to use it. But what annoys me the most, is that it modify the current hostname.if files in /etc. To my eyes, I would avoid to modify those files because they are my working basis. I want to rely on them and make sure that my network will be back to a normal mode after a reboot. Nevertheless, if I've well understood netctl, you have a feature where it will look for the predefined network config matching the environment where you are. Very cool. So, after having played with netctl, look for alternative on internet, I've decided to create nmctl. A small python script which just perform the mandatory network commands. 1. nmctl: a Network Manager Control tool for OpenBSD Nmctl a small tool that allow you to manage your network connections. Why python ? Just because it's the easiest programming language for me. But I should maybe rewrite it in shell, more standard in the OpenBSD world than python. 1.1. download and install I've put nmctl on my sourceforge account here (https://sourceforge.net/p/nmctl/code/ci/master/tree/) You can dowload the last version here (https://sourceforge.net/p/nmctl/code/ci/master/tarball) To install you just have to run: make install (as root) The per-requists are: - having python2.7 installed - Since nmctl must be run as root, I strongly recommend you to run it via doas (http://man.openbsd.org/doas.conf.5). 1.2. The config file First you have to create a config and store it in /etc/nmctl.conf. This file must respect few rules: Each block must starts with a line having the following format: ''':''' Each following lines must start by at least one space. Those lines have more or less the same format as for hostname.if. You have to create a block with the name "open". This will be used to establish a connection to the Open Wifi around you (in restaurant for example) The order of those elements is important. In case you use the -restart option, nmctl will try each of those network configs one after one until it can ping www.google.com. (if you wan to ping something else, you can change it in the python script if you want). You can use external commands. Just preced them with the "!". You have macors. Macros allow you to perform some actions. The 2 currently implemented are '''''' and ''''''. You can use keywords. Currently the only one implemented is "dhcp" Basically you can put all commands that nmctl will apply to the interface to which those commands are referring to. So, you will always have "ifconfig ". Check the manpage of ifconfig to see how flexible command is. You have currently 2 macros: - which refers to the "nwid " when you select an Open Wifi with the -open option of nmctl. - is a macro generating a random mac address. This is useful test a dhcp server for example. The keyword "dhcp" will trigger a command like "dhclient ". 1.3. Config file sample. Let me show you one nmctl.conf example. It speaks by itself. ``` # the name open is required for Open wifi. # this is the interface that nmctl will take to establish a connection # We must put the macro . This is where nmctl will put the nwid command # and the selected openwifi selected by the parameter --open open:iwn0 !route flush -wpa dhcp cable:em0 !route flush dhcp lgg4:iwn0 !route flush nwid LGG4s_8114 wpakey aanotherpassword dhcp home:iwn0 !route flush nwid Linksys19594 wpakey apassword dhcp college:iwn0 !route flush nwid john wpakey haahaaaguessme dhcp cable_fixip:em0 !route flush inet 192.168.3.3 netmask 255.255.255.0 !route add -host default 192.168.3.1 # with this network interface I'm using the macro # which will do what you guess it will do :-) cable_random:em0 !route flush lladdr dhcp ``` In this config we have several cable's networks associated with my interface "em0" and several wifi networks associated with my wireless interface "iwn0". You see that you can switch from dhcp, to fixed IP and even you can play with the random mac address macro. Thanks to the network called "open", you can connect to any open wifi system. To do that, just type ''' nmctl --open ''' So, now, with just one command you can switch from one network configuration to an another one. That's become cool :-). 2. Integration with openbox Thanks to the dynamic menu feature of oenbox[sic], you can have your different pre-defined networks under one click of your mouse. For that, you just have to add, at the most appropriate place for you, the following code in your ./config/openbox/menu.xml In this case, you see the different networks as defined in the config file just above. 3. Automatically identify your available connection and connect to it in one go But the most interesting part, is coming from a loop through all of your defined networks. This loop is reachable via the -restart option. Basically the idea is to loop from the first network config to the last and test a ping for each of them. Once the ping works, we break the loop and keep this setting. Thus where ever you are, you just have to initiate a nmctl -restart and you will be connected to the network you have defined for this place. There is one small exception, the open-wifis. We do not include them in this loop exercise. Thus the way you define your config file is important. Since the network called "open" is dedicated to "open wifi", it will not be part of this scan exercise. I propose you keep it at the first place. Then, in my case, if my mobile, called lgg4, is open and visible by my laptop, I will connect it immediately. Second, I check if my "home wifi" is visible. Third, if I have a cable connected on my laptop, I'm using this connection and do a dhcp command. Then, I check to see if my laptop is not viewing the "college" wifi. ? and so on until a ping command works. If you do not have a cable in your laptop and if none of your pre-defined wifi connections are visible, the scan will stop. 3.1 examples No cable connected, no pre-defined wifi around me: t420:~$ time doas nmctl -r nwids around you: bbox2-d954 0m02.97s real 0m00.08s user 0m00.11s system t420:~$ t420:~$ I'm at home and my wifi router is running: ``` t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 ifconfig em0 down: 0 default fw done fw 00:22:4d:ac:30:fd done nas link#2 done route flush: 0 ifconfig iwn0 nwid Linksys19594 ...: 0 iwn0: no link ........... sleeping dhclient iwn0: 0 Done. PING www.google.com (216.58.212.164): 56 data bytes 64 bytes from 216.58.212.164: icmp_seq=0 ttl=52 time=12.758 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 12.758/12.758/12.758/0.000 ms ping -c1 -w2 www.google.com: 0 0m22.49s real 0m00.08s user 0m00.11s system t420:~$ ``` I'm at home but tethering is active on my mobile: ``` t420:~$ t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 LGG4s8114 ifconfig em0 down: 0 default fw done fw 00:22:4d:ac:30:fd done nas link#2 done route flush: 0 ifconfig iwn0 nwid LGG4s8114 ...: 0 iwn0: DHCPDISCOVER - interval 1 iwn0: DHCPDISCOVER - interval 2 iwn0: DHCPOFFER from 192.168.43.1 (a0:91:69:be:10:49) iwn0: DHCPREQUEST to 255.255.255.255 iwn0: DHCPACK from 192.168.43.1 (a0:91:69:be:10:49) iwn0: bound to 192.168.43.214 -- renewal in 1800 seconds dhclient iwn0: 0 Done. ping: Warning: www.google.com has multiple addresses; using 173.194.69.99 PING www.google.com (173.194.69.99): 56 data bytes 64 bytes from 173.194.69.99: icmp_seq=0 ttl=43 time=42.863 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 42.863/42.863/42.863/0.000 ms ping -c1 -w2 www.google.com: 0 0m13.78s real 0m00.08s user 0m00.13s system t420:~$ ``` Same situation, but I cut the tethering just after the scan. Thus the dhcp command will not succeed. We see that, after timeouts, nmctl see that the ping is failing (return code 1), thus he pass to the next possible pre-defined network. ``` t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 LGG4s8114 ifconfig em0 down: 0 default 192.168.43.1 done 192.168.43.1 a0:91:69:be:10:49 done route flush: 0 ifconfig iwn0 nwid LGG4s8114 ...: 0 iwn0: no link ........... sleeping dhclient iwn0: 0 Done. ping: no address associated with name ping -c1 -w2 www.google.com: 1 ifconfig em0 down: 0 192.168.43.1 link#2 done route flush: 0 ifconfig iwn0 nwid Linksys19594 ...: 0 iwn0: DHCPREQUEST to 255.255.255.255 iwn0: DHCPACK from 192.168.3.1 (00:22:4d:ac:30:fd) iwn0: bound to 192.168.3.16 -- renewal in 302400 seconds dhclient iwn0: 0 Done. PING www.google.com (216.58.212.164): 56 data bytes 64 bytes from 216.58.212.164: icmp_seq=0 ttl=52 time=12.654 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 12.654/12.654/12.654/0.000 ms ping -c1 -w2 www.google.com: 0 3m34.85s real 0m00.17s user 0m00.20s system t420:~$ ``` OpenVPN Setup Guide for FreeBSD (https://www.c0ffee.net/blog/openvpn-guide) OpenVPN Setup Guide Browse securely from anywhere using a personal VPN with OpenVPN, LDAP, FreeBSD, and PF. A VPN allows you to securely extend a private network over the internet via tunneling protocols and traffic encryption. For most people, a VPN offers two primary features: (1) the ability to access services on your local network over the internet, and (2) secure internet connectivity over an untrusted network. In this guide, I'll describe how to set up a personal VPN using OpenVPN on FreeBSD. The configuration can use both SSL certificates and LDAP credentials for authentication. We'll also be using the PF firewall to NAT traffic from our VPN out to the internet. One important note about running your own VPN: since you are most likely hosting your server using a VPS or hosting provider, with a public IP address allocated specifically to you, your VPN will not give you any extra anonymity on the internet. If anything, you'll be making yourself more of a target, since all your activity can be trivially traced back to your server's IP address. So while your VPN will protect you from a snooping hacker on the free WiFi at Starbucks, it won't protect you from a federal investigation. This guide assumes you are running FreeBSD with the PF firewall. If you're using a different Unix flavor, I'll probably get you most of the way there—but you'll be on your own when configuring your firewall and networking. Finally, I've used example.com and a non-routable public IP address for all the examples in this guide. You'll need to replace them with your own domain name and public IP address. Beastie Bits BSDCan 2017 videos (https://www.youtube.com/channel/UCuQhwHMJ0yK2zlfyRr1XZ_Q/feed) Getting started with OpenBSD device driver development PDF (https://www.openbsd.org/papers/eurobsdcon2017-device-drivers.pdf) AWS CloudWatch Logs agent for FreeBSD (https://macfoo.wordpress.com/2017/10/27/aws-cloudwatch-logs-agent-for-freebsd/) FreeBSD Foundation November 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/november-2017-development-projects-update/) Schedule for the BSD Devroom at FOSDEM 2018 (https://fosdem.org/2018/schedule/track/bsd/) *** Feedback/Questions Matt - The show and Cantrill (http://dpaste.com/35VNXR5#wrap) Paulo - FreeBSD Question (http://dpaste.com/17E9Z2W#wrap) Steven - Virtualization under FreeBSD (http://dpaste.com/1N6F0TC#wrap) ***
We take a look at two-faced Oracle, cover a FAMP installation, how Netflix works the complex stuff, and show you who the patron of yak shaving is. This episode was brought to you by Headlines Why is Oracle so two-faced over open source? (https://www.theregister.co.uk/2017/10/12/oracle_must_grow_up_on_open_source/) Oracle loves open source. Except when the database giant hates open source. Which, according to its recent lobbying of the US federal government, seems to be "most of the time". Yes, Oracle has recently joined the Cloud Native Computing Foundation (CNCF) to up its support for open-source Kubernetes and, yes, it has long supported (and contributed to) Linux. And, yes, Oracle has even gone so far as to (finally) open up Java development by putting it under a foundation's stewardship. Yet this same, seemingly open Oracle has actively hammered the US government to consider that "there is no math that can justify open source from a cost perspective as the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." That punch to the face was delivered in a letter to Christopher Liddell, a former Microsoft CFO and now director of Trump's American Technology Council, by Kenneth Glueck, Oracle senior vice president. The US government had courted input on its IT modernisation programme. Others writing back to Liddell included AT&T, Cisco, Microsoft and VMware. In other words, based on its letter, what Oracle wants us to believe is that open source leads to greater costs and poorly secured, limply featured software. Nor is Oracle content to leave it there, also arguing that open source is exactly how the private sector does not function, seemingly forgetting that most of the leading infrastructure, big data, and mobile software today is open source. Details! Rather than take this counterproductive detour into self-serving silliness, Oracle would do better to follow Microsoft's path. Microsoft, too, used to Janus-face its way through open source, simultaneously supporting and bashing it. Only under chief executive Satya Nadella's reign did Microsoft realise it's OK to fully embrace open source, and its financial results have loved the commitment. Oracle has much to learn, and emulate, in Microsoft's approach. I love you, you're perfect. Now change Oracle has never been particularly warm and fuzzy about open source. As founder Larry Ellison might put it, Oracle is a profit-seeking corporation, not a peace-loving charity. To the extent that Oracle embraces open source, therefore it does so for financial reward, just like every other corporation. Few, however, are as blunt as Oracle about this fact of corporate open-source life. As Ellison told the Financial Times back in 2006: "If an open-source product gets good enough, we'll simply take it. So the great thing about open source is nobody owns it – a company like Oracle is free to take it for nothing, include it in our products and charge for support, and that's what we'll do. "So it is not disruptive at all – you have to find places to add value. Once open source gets good enough, competing with it would be insane... We don't have to fight open source, we have to exploit open source." "Exploit" sounds about right. While Oracle doesn't crack the top-10 corporate contributors to the Linux kernel, it does register a respectable number 12, which helps it influence the platform enough to feel comfortable building its IaaS offering on Linux (and Xen for virtualisation). Oracle has also managed to continue growing MySQL's clout in the industry while improving it as a product and business. As for Kubernetes, Oracle's decision to join the CNCF also came with P&L strings attached. "CNCF technologies such as Kubernetes, Prometheus, gRPC and OpenTracing are critical parts of both our own and our customers' development toolchains," said Mark Cavage, vice president of software development at Oracle. One can argue that Oracle has figured out the exploitation angle reasonably well. This, however, refers to the right kind of exploitation, the kind that even free software activist Richard Stallman can love (or, at least, tolerate). But when it comes to government lobbying, Oracle looks a lot more like Mr Hyde than Dr Jekyll. Lies, damned lies, and Oracle lobbying The current US president has many problems (OK, many, many problems), but his decision to follow the Obama administration's support for IT modernisation is commendable. Most recently, the Trump White House asked for feedback on how best to continue improving government IT. Oracle's response is high comedy in many respects. As TechDirt's Mike Masnick summarises, Oracle's "latest crusade is against open-source technology being used by the federal government – and against the government hiring people out of Silicon Valley to help create more modern systems. Instead, Oracle would apparently prefer the government just give it lots of money." Oracle is very good at making lots of money. As such, its request for even more isn't too surprising. What is surprising is the brazenness of its position. As Masnick opines: "The sheer contempt found in Oracle's submission on IT modernization is pretty stunning." Why? Because Oracle contradicts much that it publicly states in other forums about open source and innovation. More than this, Oracle contradicts much of what we now know is essential to competitive differentiation in an increasingly software and data-driven world. Take, for example, Oracle's contention that "significant IT development expertise is not... central to successful modernization efforts". What? In our "software is eating the world" existence Oracle clearly believes that CIOs are buyers, not doers: "The most important skill set of CIOs today is to critically compete and evaluate commercial alternatives to capture the benefits of innovation conducted at scale, and then to manage the implementation of those technologies efficiently." While there is some truth to Oracle's claim – every project shouldn't be a custom one-off that must be supported forever – it's crazy to think that a CIO – government or otherwise – is doing their job effectively by simply shovelling cash into vendors' bank accounts. Indeed, as Masnick points out: "If it weren't for Oracle's failures, there might not even be a USDS [the US Digital Service created in 2014 to modernise federal IT]. USDS really grew out of the emergency hiring of some top-notch internet engineers in response to the Healthcare.gov rollout debacle. And if you don't recall, a big part of that debacle was blamed on Oracle's technology." In short, blindly giving money to Oracle and other big vendors is the opposite of IT modernisation. In its letter to Liddell, Oracle proceeded to make the fantastic (by which I mean "silly and false") claim that "the fact is that the use of open-source software has been declining rapidly in the private sector". What?!? This is so incredibly untrue that Oracle should score points for being willing to say it out loud. Take a stroll through the most prominent software in big data (Hadoop, Spark, Kafka, etc.), mobile (Android), application development (Kubernetes, Docker), machine learning/AI (TensorFlow, MxNet), and compare it to Oracle's statement. One conclusion must be that Oracle believes its CIO audience is incredibly stupid. Oracle then tells a half-truth by declaring: "There is no math that can justify open source from a cost perspective." How so? Because "the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." Which I guess is why Oracle doesn't use any open source like Linux, Kubernetes, etc. in its services. Oops. The Vendor Formerly Known As Satan The thing is, Oracle doesn't need to do this and, for its own good, shouldn't do this. After all, we already know how this plays out. We need only look at what happened with Microsoft. Remember when Microsoft wanted us to "get the facts" about Linux? Now it's a big-time contributor to Linux. Remember when it told us open source was anti-American and a cancer? Now it aggressively contributes to a huge variety of open-source projects, some of them homegrown in Redmond, and tells the world that "Microsoft loves open source." Of course, Microsoft loves open source for the same reason any corporation does: it drives revenue as developers look to build applications filled with open-source components on Azure. There's nothing wrong with that. Would Microsoft prefer government IT to purchase SQL Server instead of open-source-licensed PostgreSQL? Sure. But look for a single line in its response to the Trump executive order that signals "open source is bad". You won't find it. Why? Because Microsoft understands that open source is a friend, not foe, and has learned how to monetise it. Microsoft, in short, is no longer conflicted about open source. It can compete at the product level while embracing open source at the project level, which helps fuel its overall product and business strategy. Oracle isn't there yet, and is still stuck where Microsoft was a decade ago. It's time to grow up, Oracle. For a company that builds great software and understands that it increasingly needs to depend on open source to build that software, it's disingenuous at best to lobby the US government to put the freeze on open source. Oracle needs to learn from Microsoft, stop worrying and love the open-source bomb. It was a key ingredient in Microsoft's resurgence. Maybe it could help Oracle get a cloud clue, too. Install FAMP on FreeBSD (https://www.linuxsecrets.com/home/3164-install-famp-on-freebsd) The acronym FAMP refers to a set of free open source applications which are commonly used in Web server environments called Apache, MySQL and PHP on the FreeBSD operating system, which provides a server stack that provides web services, database and PHP. Prerequisites sudo Installed and working - Please read Apache PHP5 or PHP7 MySQL or MariaDB Install your favorite editor, ours is vi Note: You don't need to upgrade FreeBSD but make sure all patches have been installed and your port tree is up-2-date if you plan to update by ports. Install Ports portsnap fetch You must use sudo for each indivdual command during installations. Please see link above for installing sudo. Searching Available Apache Versions to Install pkg search apache Install Apache To install Apache 2.4 using pkg. The apache 2.4 user account managing Apache is www in FreeBSD. pkg install apache24 Confirmation yes prompt and hit y for yes to install Apache 2.4 This installs Apache and its dependencies. Enable Apache use sysrc to update services to be started at boot time, Command below adds "apache24enable="YES" to the /etc/rc.conf file. For sysrc commands please read ```sysrc apache24enable=yes Start Apache service apache24 start``` Visit web address by accessing your server's public IP address in your web browser How To find Your Server's Public IP Address If you do not know what your server's public IP address is, there are a number of ways that you can find it. Usually, this is the address you use to connect to your server through SSH. ifconfig vtnet0 | grep "inet " | awk '{ print $2 }' Now that you have the public IP address, you may use it in your web browser's address bar to access your web server. Install MySQL Now that we have our web server up and running, it is time to install MySQL, the relational database management system. The MySQL server will organize and provide access to databases where our server can store information. Install MySQL 5.7 using pkg by typing pkg install mysql57-server Enter y at the confirmation prompt. This installs the MySQL server and client packages. To enable MySQL server as a service, add mysqlenable="YES" to the /etc/rc.conf file. This sysrc command will do just that ```sysrc mysqlenable=yes Now start the MySQL server service mysql-server start Now run the security script that will remove some dangerous defaults and slightly restrict access to your database system. mysqlsecureinstallation``` Answer all questions to secure your newly installed MySQL database. Enter current password for root (enter for none): [RETURN] Your database system is now set up and we can move on. Install PHP5 or PHP70 pkg search php70 Install PHP70 you would do the following by typing pkg install php70-mysqli mod_php70 Note: In these instructions we are using php5.7 not php7.0. We will be coming out with php7.0 instructions with FPM. PHP is the component of our setup that will process code to display dynamic content. It can run scripts, connect to MySQL databases to get information, and hand the processed content over to the web server to display. We're going to install the modphp, php-mysql, and php-mysqli packages. To install PHP 5.7 with pkg, run this command ```pkg install modphp56 php56-mysql php56-mysqli Copy sample PHP configuration file into place. cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini Regenerate the system's cached information about your installed executable files rehash``` Before using PHP, you must configure it to work with Apache. Install PHP Modules (Optional) To enhance the functionality of PHP, we can optionally install some additional modules. To see the available options for PHP 5.6 modules and libraries, you can type this into your system pkg search php56 Get more information about each module you can look at the long description of the package by typing pkg search -f apache24 Optional Install Example pkg install php56-calendar Configure Apache to Use PHP Module Open the Apache configuration file vim /usr/local/etc/apache24/Includes/php.conf DirectoryIndex index.php index.html Next, we will configure Apache to process requested PHP files with the PHP processor. Add these lines to the end of the file: SetHandler application/x-httpd-php SetHandler application/x-httpd-php-source Now restart Apache to put the changes into effect service apache24 restart Test PHP Processing By default, the DocumentRoot is set to /usr/local/www/apache24/data. We can create the info.php file under that location by typing vim /usr/local/www/apache24/data/info.php Add following line to info.php and save it. Details on info.php info.php file gives you information about your server from the perspective of PHP. It' useful for debugging and to ensure that your settings are being applied correctly. If this was successful, then your PHP is working as expected. You probably want to remove info.php after testing because it could actually give information about your server to unauthorized users. Remove file by typing rm /usr/local/www/apache24/data/info.php Note: Make sure Apache / meaning the root of Apache is owned by user which should have been created during the Apache install is the owner of the /usr/local/www structure. That explains FAMP on FreeBSD. IXsystems IXsystems TrueNAS X10 Torture Test & Fail Over Systems In Action with the ZFS File System (https://www.youtube.com/watch?v=GG_NvKuh530) How Netflix works: what happens every time you hit Play (https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b) Not long ago, House of Cards came back for the fifth season, finally ending a long wait for binge watchers across the world who are interested in an American politician's ruthless ascendance to presidency. For them, kicking off a marathon is as simple as reaching out for your device or remote, opening the Netflix app and hitting Play. Simple, fast and instantly gratifying. What isn't as simple is what goes into running Netflix, a service that streams around 250 million hours of video per day to around 98 million paying subscribers in 190 countries. At this scale, providing quality entertainment in a matter of a few seconds to every user is no joke. And as much as it means building top-notch infrastructure at a scale no other Internet service has done before, it also means that a lot of participants in the experience have to be negotiated with and kept satiated?—?from production companies supplying the content, to internet providers dealing with the network traffic Netflix brings upon them. This is, in short and in the most layman terms, how Netflix works. Let us just try to understand how Netflix is structured on the technological side with a simple example. Netflix literally ushered in a revolution around ten years ago by rewriting the applications that run the entire service to fit into a microservices architecture?—?which means that each application, or microservice's code and resources are its very own. It will not share any of it with any other app by nature. And when two applications do need to talk to each other, they use an application programming interface (API)?—?a tightly-controlled set of rules that both programs can handle. Developers can now make many changes, small or huge, to each application as long as they ensure that it plays well with the API. And since the one program knows the other's API properly, no change will break the exchange of information. Netflix estimates that it uses around 700 microservices to control each of the many parts of what makes up the entire Netflix service: one microservice stores what all shows you watched, one deducts the monthly fee from your credit card, one provides your device with the correct video files that it can play, one takes a look at your watching history and uses algorithms to guess a list of movies that you will like, and one will provide the names and images of these movies to be shown in a list on the main menu. And that's the tip of the iceberg. Netflix engineers can make changes to any part of the application and can introduce new changes rapidly while ensuring that nothing else in the entire service breaks down. They made a courageous decision to get rid of maintaining their own servers and move all of their stuff to the cloud?—?i.e. run everything on the servers of someone else who dealt with maintaining the hardware while Netflix engineers wrote hundreds of programs and deployed it on the servers rapidly. The someone else they chose for their cloud-based infrastructure is Amazon Web Services (AWS). Netflix works on thousands of devices, and each of them play a different format of video and sound files. Another set of AWS servers take this original film file, and convert it into hundreds of files, each meant to play the entire show or film on a particular type of device and a particular screen size or video quality. One file will work exclusively on the iPad, one on a full HD Android phone, one on a Sony TV that can play 4K video and Dolby sound, one on a Windows computer, and so on. Even more of these files can be made with varying video qualities so that they are easier to load on a poor network connection. This is a process known as transcoding. A special piece of code is also added to these files to lock them with what is called digital rights management or DRM?—?a technological measure which prevents piracy of films. The Netflix app or website determines what particular device you are using to watch, and fetches the exact file for that show meant to specially play on your particular device, with a particular video quality based on how fast your internet is at that moment. Here, instead of relying on AWS servers, they install their very own around the world. But it has only one purpose?—?to store content smartly and deliver it to users. Netflix strikes deals with internet service providers and provides them the red box you saw above at no cost. ISPs install these along with their servers. These Open Connect boxes download the Netflix library for their region from the main servers in the US?—?if there are multiple of them, each will rather store content that is more popular with Netflix users in a region to prioritise speed. So a rarely watched film might take time to load more than a Stranger Things episode. Now, when you will connect to Netflix, the closest Open Connect box to you will deliver the content you need, thus videos load faster than if your Netflix app tried to load it from the main servers in the US. In a nutshell… This is what happens when you hit that Play button: Hundreds of microservices, or tiny independent programs, work together to make one large Netflix service. Content legally acquired or licensed is converted into a size that fits your screen, and protected from being copied. Servers across the world make a copy of it and store it so that the closest one to you delivers it at max quality and speed. When you select a show, your Netflix app cherry picks which of these servers will it load the video from> You are now gripped by Frank Underwood's chilling tactics, given depression by BoJack Horseman's rollercoaster life, tickled by Dev in Master of None and made phobic to the future of technology by the stories in Black Mirror. And your lifespan decreases as your binge watching turns you into a couch potato. It looked so simple before, right? News Roundup Moving FreshPorts (http://dan.langille.org/2017/11/15/moving-freshports/) Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice. In preparation for this move, I have: DNS TTL reduced to 60s Posted to Twitter Updated the status page Put the website put in offline mode: What was missed I turned off commit processing on the new server, but I did not do this on the old server. I should have: sudo svc -d /var/service/freshports That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless. The offline code Here is the basic stuff I used to put the website into offline mode. The main points are: header(“HTTP/1.1 503 Service Unavailable”); ErrorDocument 404 /index.php I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code. The dump The database dump just started (Sun Nov 5 17:07:22 UTC 2017). root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump That should take about 30 minutes. I have set a timer to remind me. Total time was: 1464.82 real 1324.96 user 37.22 sys The MD5 is: MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5 It is now: Sun Nov 5 17:34:07 UTC 2017 The rsync The rsync should take about 10-20 minutes. I have already done an rsync of yesterday's dump file. The rsync today should copy over only the deltas (i.e. differences). The rsync started at about Sun Nov 5 17:36:05 UTC 2017 That took 2m9.091s The MD5 matches. The restore The restore should take about 30 minutes. I ran this test yesterday. It is now Sun Nov 5 17:40:03 UTC 2017. $ createdb -T template0 -E SQL_ASCII freshports.testing $ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump Done. real 25m21.108s user 1m57.508s sys 0m15.172s It is now Sun Nov 5 18:06:22 UTC 2017. Insert break here About here, I took a 30 minute break to run an errand. It was worth it. Changing DNS I'm ready to change DNS now. It is Sun Nov 5 19:49:20 EST 2017 Done. And nearly immediately, traffic started. How many misses? During this process, XXXXX requests were declined: $ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log XXXXX That's it, we're done Total elapsed time: 1 hour 48 minutes. There are still a number of things to follow up on, but that was the transfers. The new FreshPorts Server (http://dan.langille.org/2017/11/17/x8dtu-3/) *** Using bhyve on top of CEPH (https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005876.html) Hi, Just an info point. I'm preparing for a lecture tomorrow, and thought why not do an actual demo.... Like to be friends with Murphy :) So after I started the cluster: 5 jails with 7 OSDs This what I manually needed to do to boot a memory stick Start een Bhyve instance rbd --dest-pool rbddata --no-progress import memstick.img memstick rbd-ggate map rbddata/memstick ggate-devvice is available on /dev/ggate1 kldload vmm kldload nmdm kldload iftap kldload ifbridge kldload cpuctl sysctl net.link.tap.uponopen=1 ifconfig bridge0 create ifconfig bridge0 addm em0 up ifconfig ifconfig tap11 create ifconfig bridge0 addm tap11 ifconfig tap11 up load the GGate disk in bhyve bhyveload -c /dev/nmdm11A -m 2G -d /dev/ggate1 FB11 and boot a single from it. bhyve -H -P -A -c 1 -m 2G -l com1,/dev/nmdm11A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap11 -s 4,ahci-hd,/dev/ggate1 FB11 & bhyvectl --vm=FB11 --get-stats Connect to the VM cu -l /dev/nmdm11B And that'll give you a bhyve VM running on an RBD image over ggate. In the installer I tested reading from the bootdisk: root@:/ # dd if=/dev/ada0 of=/dev/null bs=32M 21+1 records in 21+1 records out 734077952 bytes transferred in 5.306260 secs (138341865 bytes/sec) which is a nice 138Mb/sec. Hope the demonstration does work out tomorrow. --WjW *** Donald Knuth - The Patron Saint of Yak Shaves (http://yakshav.es/the-patron-saint-of-yakshaves/) Excerpts: In 2015, I gave a talk in which I called Donald Knuth the Patron Saint of Yak Shaves. The reason is that Donald Knuth achieved the most perfect and long-running yak shave: TeX. I figured this is worth repeating. How to achieve the ultimate Yak Shave The ultimate yak shave is the combination of improbable circumstance, the privilege to be able to shave at your hearts will and the will to follow things through to the end. Here's the way it was achieved with TeX. The recount is purely mine, inaccurate and obviously there for fun. I'll avoid the most boring facts that everyone always tells, such as why Knuth's checks have their own Wikipedia page. Community Shaving is Best Shaving Since the release of TeX, the community has been busy working on using it as a platform. If you ever downloaded the full TeX distribution, please bear in mind that you are downloading the amassed work of over 40 years, to make sure that each and every TeX document ever written builds. We're talking about documents here. But mostly, two big projects sprung out of that. The first is LaTeX by Leslie Lamport. Lamport is a very productive researcher, famous for research in formal methods through TLA+ and also known laying groundwork for many distributed algorithms. LaTeX is based on the idea of separating presentation and content. It is based around the idea of document classes, which then describe the way a certain document is laid out. Think Markdown, just much more complex. The second is ConTeXt, which is far more focused on fine grained layout control. The Moral of the Story Whenever you feel like “can't we just replace this whole thing, it can't be so hard” when handling TeX, don't forget how many years of work and especially knowledge were poured into that system. Typesetting isn't the most popular knowledge around programmers. Especially see it in the context of the space it is in: they can't remove legacy. Ever. That would break documents. TeX is also not a programming language. It might resemble one, but mostly, it should be approached as a typesetting system first. A lot of it's confusing lingo gets much better then. It's not programming lingo. By approaching TeX with an understanding for its history, a lot of things can be learned from it. And yes, a replacement would be great, but it would take ages. In any case, I hope I thoroughly convinced you why Donald Knuth is the Patron Saint of Yak Shaves. Extra Credits This comes out of a enjoyable discussion with [Arne from Lambda Island](https://lambdaisland.com/https://lambdaisland.com/, who listened and said “you should totally turn this into a talk”. Vincent's trip to EuroBSDCon 2017 (http://www.vincentdelft.be/post/post_20171016) My euroBSDCon 2017 Posted on 2017-10-16 09:43:00 from Vincent in Open Bsd Let me just share my feedback on those 2 days spent in Paris for the EuroBSDCon. My 1st BSDCon. I'm not a developer, contributor, ... Do not expect to improve your skills with OpenBSD with this text :-) I know, we are on October 16th, and the EuroBSDCon of Paris was 3 weeks ago :( I'm not quick !!! Sorry for that Arrival at 10h, I'm too late for the start of the key note. The few persons behind a desk welcome me by talking in Dutch, mainly because of my name. Indeed, Delft is a city in Netherlands, but also a well known university. I inform them that I'm from Belgium, and the discussion moves to the fact the Fosdem is located in Brussels. I receive my nice T-shirt white and blue, a bit like the marine T-shirts, but with the nice EuroBSDCon logo. I'm asking where are the different rooms reserved for the BSD event. We have 1 big on the 1st floor, 1 medium 1 level below, and 2 smalls 1 level above. All are really easy to access. In this entrance we have 4 or 5 tables with some persons representing their company. Those are mainly the big sponsors of the event providing details about their activity and business. I discuss a little bit with StormShield and Gandi. On other tables people are selling BSD t-shirts, and they will quickly be sold. "Is it done yet ?" The never ending story of pkg tools In the last Fosdem, I've already hear Antoine and Baptiste presenting the OpenBSD and FreeBSD battle, I decide to listen Marc Espie in the medium room called Karnak. Marc explains that he has rewritten completely the pkg_add command. He explains that, at contrario with other elements of OpenBSD, the packages tools must be backward compatible and stable on a longer period than 12 months (the support period for OpenBSD). On the funny side, he explains that he has his best idea inside his bath. Hackathons are also used to validate some ideas with other OpenBSD developers. All in all, he explains that the most time consuming part is to imagine a good solution. Coding it is quite straightforward. He adds that better an idea is, shorter the implementation will be. A Tale of six motherboards, three BSDs and coreboot After the lunch I decide to listen the talk about Coreboot. Indeed, 1 or 2 years ago I had listened the Libreboot project at Fosdem. Since they did several references to Coreboot, it's a perfect occasion to listen more carefully to this project. Piotr and Katazyba Kubaj explains us how to boot a machine without the native Bios. Indeed Coreboot can replace the bios, and de facto avoid several binaries imposed by the vendor. They explain that some motherboards are supporting their code. But they also show how difficult it is to flash a Bios and replace it by Coreboot. They even have destroyed a motherboard during the installation. Apparently because the power supply they were using was not stable enough with the 3v. It's really amazing to see that open source developers can go, by themselves, to such deep technical level. State of the DragonFly's graphics stack After this Coreboot talk, I decide to stay in the room to follow the presentation of Fran?ois Tigeot. Fran?ois is now one of the core developer of DrangonflyBSD, an amazing BSD system having his own filesystem called Hammer. Hammer offers several amazing features like snapshots, checksum data integrity, deduplication, ... Francois has spent his last years to integrate the video drivers developed for Linux inside DrangonflyBSD. He explains that instead of adapting this code for the video card to the kernel API of DrangonflyBSD, he has "simply" build an intermediate layer between the kernel of DragonflyBSD and the video drivers. This is not said in the talk, but this effort is very impressive. Indeed, this is more or less a linux emulator inside DragonflyBSD. Francois explains that he has started with Intel video driver (drm/i915), but now he is able to run drm/radeon quite well, but also drm/amdgpu and drm/nouveau. Discovering OpenBSD on AWS Then I move to the small room at the upper level to follow a presentation made by Laurent Bernaille on OpenBSD and AWS. First Laurent explains that he is re-using the work done by Antoine Jacoutot concerning the integration of OpenBSD inside AWS. But on top of that he has integrated several other Open Source solutions allowing him to build OpenBSD machines very quickly with one command. Moreover those machines will have the network config, the required packages, ... On top of the slides presented, he shows us, in a real demo, how this system works. Amazing presentation which shows that, by putting the correct tools together, a machine builds and configure other machines in one go. OpenBSD Testing Infrastructure Behind bluhm.genua.de Here Jan Klemkow explains us that he has setup a lab where he is able to run different OpenBSD architectures. The system has been designed to be able to install, on demand, a certain version of OpenBSD on the different available machines. On top of that a regression test script can be triggered. This provides reports showing what is working and what is not more working on the different machines. If I've well understood, Jan is willing to provide such lab to the core developers of OpenBSD in order to allow them to validate easily and quickly their code. Some more effort is needed to reach this goal, but with what exists today, Jan and his colleague are quite close. Since his company is using OpenBSD business, to his eyes this system is a "tit for tat" to the OpenBSD community. French story on cybercrime Then comes the second keynote of the day in the big auditorium. This talk is performed by the colonel of french gendarmerie. Mr Freyssinet, who is head of the Cyber crimes unit inside the Gendarmerie. Mr Freyssinet explains that the "bad guys" are more and more volatile across countries, and more and more organized. The small hacker in his room, alone, is no more the reality. As a consequence the different national police investigators are collaborating more inside an organization called Interpol. What is amazing in his talk is that Mr Freyssinet talks about "Crime as a service". Indeed, more and more hackers are selling their services to some "bad and temporary organizations". Social event It's now time for the famous social event on the river: la Seine. The organizers ask us to go, by small groups, to a station. There is a walk of 15 minutes inside Paris. Hopefully the weather is perfect. To identify them clearly several organizers takes a "beastie fork" in their hands and walk on the sidewalk generating some amazing reactions from some citizens and toursits. Some of them recognize the Freebsd logo and ask us some details. Amazing :-) We walk on small and big sidewalks until a small stair going under the street. There, we have a train station a bit like a metro station. 3 stations later they ask us to go out. We walk few minutes and come in front of a boat having a double deck: one inside, with nice tables and chairs and one on the roof. But the crew ask us to go up, on the second deck. There, we are welcome with a glass of wine. The tour Eiffel is just at few 100 meters from us. Every hour the Eiffel tower is blinking for 5 minutes with thousands of small lights. Brilliant :-) We see also the "statue de la libertee" (the small one) which is on a small island in the middle of the river. During the whole night the bar will be open with drinks and some appetizers, snacks, ... Such walking diner is perfect to talk with many different persons. I've discussed with several persons just using BSD, they are not, like me, deep and specialized developers. One was from Switzerland, another one from Austria, and another one from Netherlands. But I've also followed a discussion with Theo de Raadt, several persons of the FreeBSD foundation. Some are very technical guys, other just users, like me. But all with the same passion for one of the BSD system. Amazing evening. OpenBSD's small steps towards DTrace (a tale about DDB and CTF) On the second day, I decide to sleep enough in order to have enough resources to drive back to my home (3 hours by car). So I miss the 1st presentations, and arrive at the event around 10h30. Lot of persons are already present. Some faces are less "fresh" than others. I decide to listen to Dtrace in OpenBSD. After 10 minutes I am so lost into those too technical explainations, that I decide to open and look at my PC. My OpenBSD laptop is rarely leaving my home, so I've never had the need to have a screen locking system. In a crowded environment, this is better. So I was looking for a simple solution. I've looked at how to use xlock. I've combined it with the /ets/apm/suspend script, ... Always very easy to use OpenBSD :-) The OpenBSD web stack Then I decide to follow the presentation of Michael W Lucas. Well know person for his different books about "Absolute OpenBSD", Relayd", ... Michael talks about the httpd daemon inside OpenBSD. But he also present his integration with Carp, Relayd, PF, FastCGI, the rules based on LUA regexp (opposed to perl regexp), ... For sure he emphasis on the security aspect of those tools: privilege separation, chroot, ... OpenSMTPD, current state of affairs Then I follow the presentation of Gilles Chehade about the OpenSMTPD project. Amazing presentation that, on top of the technical challenges, shows how to manage such project across the years. Gilles is working on OpenSMTPD since 2007, thus 10 years !!!. He explains the different decisions they took to make the software as simple as possible to use, but as secure as possible, too: privilege separation, chroot, pledge, random malloc, ? . The development starts on BSD systems, but once quite well known they received lot of contributions from Linux developers. Hoisting: lessons learned integrating pledge into 500 programs After a small break, I decide to listen to Theo de Raadt, the founder of OpenBSD. In his own style, with trekking boots, shorts, backpack. Theo starts by saying that Pledge is the outcome of nightmares. Theo explains that the book called "Hacking blind" presenting the BROP has worried him since few years. That's why he developed Pledge as a tool killing a process as soon as possible when there is an unforeseen behavior of this program. For example, with Pledge a program which can only write to disk will be immediately killed if he tries to reach network. By implementing Pledge in the +-500 programs present in the "base", OpenBSD is becoming more secured and more robust. Conclusion My first EuroBSDCon was a great, interesting and cool event. I've discussed with several BSD enthusiasts. I'm using OpenBSD since 2010, but I'm not a developer, so I was worried to be "lost" in the middle of experts. In fact it was not the case. At EuroBSDCon you have many different type of enthusiasts BSD's users. What is nice with the EuroBSDCon is that the organizers foresee everything for you. You just have to sit and listen. They foresee even how to spend, in a funny and very cool attitude, the evening of Saturday. > The small draw back is that all of this has a cost. In my case the whole weekend cost me a bit more than 500euro. Based on what I've learned, what I've saw this is very acceptable price. Nearly all presentations I saw give me a valuable input for my daily job. For sure, the total price is also linked to my personal choice: hotel, parking. And I'm surely biased because I'm used to go to the Fosdem in Brussels which cost nothing (entrance) and is approximately 45 minutes of my home. But Fosdem is not the same atmosphere and presentations are less linked to my daily job. I do not regret my trip to EuroBSDCon and will surely plan other ones. Beastie Bits Important munitions lawyering (https://www.jwz.org/blog/2017/10/important-munitions-lawyering/) AsiaBSDCon 2018 CFP is now open, until December 15th (https://2018.asiabsdcon.org/) ZSTD Compression for ZFS by Allan Jude (https://www.youtube.com/watch?v=hWnWEitDPlM&feature=share) NetBSD on Allwinner SoCs Update (https://blog.netbsd.org/tnf/entry/netbsd_on_allwinner_socs_update) *** Feedback/Questions Tim - Creating Multi Boot USB sticks (http://dpaste.com/0FKTJK3#wrap) Nomen - ZFS Questions (http://dpaste.com/1HY5MFB) JJ - Questions (http://dpaste.com/3ZGNSK9#wrap) Lars - Hardening Diffie-Hellman (http://dpaste.com/3TRXXN4) ***
FreeBSD 10.4-RELEASE is here, more EuroBSDcon travel notes, the KRACK attack, ZFS and DTrace on NetBSD, and pfsense 2.4. This episode was brought to you by Headlines FreeBSD 10.4-RELEASE Available (https://www.freebsd.org/releases/10.4R/announce.html) FreeBSD 10.4-RELEASE is out. The FreeBSD Project dedicates the FreeBSD 10.4-RELEASE to the memory of Andrey A. Chernov. Some of the highlights: 10.4-RELEASE is the first FreeBSD release to feature full support for eMMC storage, including eMMC partitions, TRIM and bus speed modes up to HS400. Please note, though, that availability of especially the DDR52, HS200 and HS400 modes requires support in the actual sdhci(4) front-end as well as by the hardware used. Also note, that the SDHCI controller part of Intel® Apollo Lake chipsets is affected by several severe silicon bugs. Apparently, it depends on the particular Apollo Lake platform whether the workarounds in place so far are sufficient to avoid timeouts on attaching sdhci(4) there. Also in case a GPT disk label is used, the fsckffs(8) utility now is able to find alternate superblocks. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for accelerating ipsec(4). Support for the Kaby Lake generation of Intel® i219(4)/ i219(5) devices has been added to the em(4) driver. The em(4) driver is now capable of enabling Wake On LAN (WOL) also for Intel® i217, i218 and i219 chips. Note that stale interface configurations from previous unsuccessful attempts to enable WOL for these devices now will actually take effect. For example, an ifconfig em0 wol activates all WOL variants including wolmcast, which might be undesirable. Support for WOL has been added to the igb(4) driver, which was not able to activate this feature on any device before. The same remark regarding stale WOL configurations as for the em(4) driver applies. Userland coredumps can now trigger events such as generating a human readable crash report via devd(8). This feature is off by default. The firmware shipping with the qlxgbe(4) driver has been updated to version 5.4.66. Additionally, this driver has received some TSO and locking fixes, performance optimizations as well as SYSCTLs providing MAC, RX and TX statistics. Mellanox® ConnectX-4 series adapters are now supported by the newly added mlx5ib(4) driver. OpenSSH received an update to version 7.3p1. GNOME has been updated to version 3.18. Xorg-Server has been updated to version 1.18.4. Check out the full release notes and upgrade your systems to 10.4-RELEASE. Thanks to the FreeBSD Release Engineering Team for their efforts. *** EuroBSDcon 2017: "travel notes" after the conference (https://blog.netbsd.org/tnf/entry/eurobsdcon_2017_travel_notes_after) Leonardo Taccari posted in the NetBSD blog about his experiences at EuroBSDcon 2017: Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings! Thursday (21/09): NetBSD developers dinner Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers! Friday (22/09): NetBSD developers summit On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla. NetBSD on Google Compute Engine -- Benny Siegert (bsiegert) Scripting DDB with Forth -- Valery Ushakov (uwe) News from the version control front -- Jörg Sonnenberger (joerg) Afternoon discussions and dinner After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidentally dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together. Saturday (23/09): First day of conference session and Social Event A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry) Hardening pkgsrc -- Pierre Pronchery (khorben) Reproducible builds on NetBSD -- Christos Zoulas (christos) Social event The social event on Saturday evening took place on a boat that cruised on the Seine river. It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community. + Sunday (24/09): Second day of conference session The school of hard knocks - PT1 -- Sevan Janiyan (sevan) The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil) What's in store for NetBSD 8.0? -- Alistair Crooks (agc) Sunday dinner After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers! Conclusion It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!). I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme committee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference! I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit! A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :) Thank you! *** WiFi Vulnerability in WPA2: KRACK (https://www.krackattacks.com/) “We discovered serious weaknesses in WPA2, a protocol that secures all modern protected Wi-Fi networks. An attacker within range of a victim can exploit these weaknesses using key reinstallation attacks (KRACKs). Concretely, attackers can use this novel attack technique to read information that was previously assumed to be safely encrypted. This can be abused to steal sensitive information such as credit card numbers, passwords, chat messages, emails, photos, and so on. The attack works against all modern protected Wi-Fi networks. Depending on the network configuration, it is also possible to inject and manipulate data. For example, an attacker might be able to inject ransomware or other malware into websites.” “Note that if your device supports Wi-Fi, it is most likely affected. During our initial research, we discovered ourselves that Android, Linux, Apple, Windows, OpenBSD, MediaTek, Linksys, and others, are all affected by some variant of the attacks. For more information about specific products, consult the database of CERT/CC, or contact your vendor.” FreeBSD Advisory (https://www.freebsd.org/security/advisories/FreeBSD-SA-17:07.wpa.asc) As of the date of this recording, a few weeks ahead of when this episode will air, the issue is fixed in FreeBSD 11.0 and 11.1, and a workaround has been provided for 10.3 and 10.4 (install newer wpa_supplicant from ports). A fix for 10.3 and 10.4 is expected soon. They will more than likely be out by time you are watching this. The fix for 10.3 and 10.4 is more complicated because the version of wpasupplicant included in the base system is 2.0, from January 2013, so is nearly 5 years old, so the patches do not apply cleanly. The security team is still considering if it will try to patch 2.0, or just replace the version of wpasupplicant with 2.5 from FreeBSD 11.x. OpenBSD was unwilling to wait when the embargo was extended on this vulnerability and stealth fixed the issue on Aug 30th (https://marc.info/?l=openbsd-cvs&m=150410571407760&w=2) stsp@openbsd.org ‘s Mastodon post (https://mastodon.social/@stsp/98837563531323569) Lobste.rs conversation about flaw and OpenBSD's reaction (https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz) “What happened is that he told me on July 15, and gave a 6 weeks embargo until end of August. We already complained back then that this was way too long and leaving people exposed. Then he got CERT (and, thus, US gov agencies) involved and had to extend the embargo even further until today. At that point we already had the ball rolling and decided to stick to the original agreement with him, and he gave us an agreeing nod towards that as well.” “In this situation, a request for keeping the problem and fix secret is a request to leave our users at risk and exposed to insiders who will potentially use the bug to exploit our users. And we have no idea who the other insiders are. We have to assume that information of this kind leaks and dissipates pretty fast in the security “community”.” “We chose to serve the needs of our users who are the vulnerable people in this drama. I stand by that choice.” As a result of this: “To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.” NetBSD: “patches for the WPA issues in KRACK Attacks were committed Oct 16th to HEAD & are pending pullup to 6/7/8 branches” (http://mail-index.netbsd.org/source-changes/2017/10/16/msg088877.html) As of this recording, Dragonfly appears to use wpa_supplicant 2.1 which they imported in 2014 and has not been touched in over a year (https://github.com/DragonFlyBSD/DragonFlyBSD/commits/master/contrib/wpa_supplicant) *** News Roundup NetBSD - dtrace and ZFS update (https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html) Chuck Silvers writes to the tech-kern mailing list of NetBSD: I've been working on updating netbsd's copy of the dtrace and zfs code to rebase from the existing ancient opensolaris version to a recent freebsd version. most of the freebsd changes are pretty close to what netbsd needs, so that seems like a more useful upstream for us. I have things working well enough now that I want to share the code in preparation for committing. this update improves upon our existing dtrace/zfs code in several ways: picks up all the upstream zfs fixes and enhancements from the last decade zfs now supports mmap on netbsd, so you can run executables stored in zfs dtrace fbt probes can now be used in kernel modules (such as zfs) A patch is provided here: http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012 which needs to be applied using “patch -E” as it adds and removes files. He provides the following summary for the diff: freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for netbsd. a few updates to our copy of freebsd's libproc. build system support for using -fno-omit-frame-pointer everywhere and disabling other compiler optimizations that confuse dtrace. sample kernel config changes for a couple evbarm configs (the ones I tested). module/ksyms enhancements for dtrace integration. genfs API enhancements to support zfs. an option to have mutexes not become no-ops during a panic. uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs). Known issues with the patch include: unloading the zfs module fails even with no zpools imported if you've done much with zfs since it was loaded. there's some refcounting problem that I haven't tracked down yet. the module refcounting for active fbt probes is bogus. currently module refcounting is protected by kernconfig_lock(), but taking that lock down in the bowels of dtrace seems likely to create deadlocks. I plan to do something fancier but haven't gotten to it yet. the dtrace uregs[] stuff is probably still wrong. the CTF typeid overflow problem is still there (more on this below). Unsupported features include: the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar" zvols ZFS ACLs (aka. NFSv4 ACLs) NFS exporting a ZFS file system setting dtrace probes in application code using ZFS as the root fs new crypto hashes SHA512_256, skein, and edonr (the last one is not in freebsd yet either) zio delay injection (used for testing zfs) dtrace support for platforms other than x86 and arm A more detailed description of the CTF typeid overflow is also provided. Check out the full thread with followups and try out the patch if you're on NetBSD. *** pfSense 2.4.0-RELEASE Now Available! (https://www.netgate.com/blog/pfsense-2-4-0-release-now-available.html) Jim Pingle writes about the new release: We are excited to announce the release of pfSense® software version 2.4, now available for new installations and upgrades! pfSense software version 2.4.0 was a herculean effort! It is the culmination of 18 months of hard work by Netgate and community contributors, with over 290 items resolved. According to git, 671 files were changed with a total 1651680 lines added, and 185727 lines deleted. Most of those added lines are from translated strings for multiple language support! + Highlights FreeBSD 11.1-RELEASE as the base Operating System New pfSense installer based on bsdinstall, with support for ZFS, UEFI, and multiple types of partition layouts (e.g. GPT, BIOS) Support for Netgate ARM devices such as the SG-1000 OpenVPN 2.4.x support, which brings features like AES-GCM ciphers, speed improvements, Negotiable Crypto Parameters (NCP), TLS encryption, and dual stack/multihome Translation of the GUI into 13 different languages! For more information on contributing to the translation effort, read our previous blog post and visit the project on Zanata WebGUI improvements, such as a new login page, improved GET/POST CSRF handling, significant improvements to the Dashboard and its AJAX handling Certificate Management improvements including CSR signing and international character support Captive Portal has been rewritten to work without multiple instances of ipfw Important Information: 32-bit x86 and NanoBSD have been deprecated and are not supported on pfSense 2.4. Read the full release notes and let them know how you like the new release. *** OpenBSD changes of note 629 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-629) Use getrusage to measure CPU time in md5 benchmarking. Add guard pages at the end of kernel stacks so overflows don't run into important stuff. This would be useful in FreeBSD, even just to detect the condition. I had all kinds of strange crashes when I was accidently overflowing the stack when working on the initial version of the ZSTD patches before ZSTD gained a working heap mode. Add dwxe driver for ethernet found on Allwinner A64, H3 and H5 SoCs. Fix a regression caused by removal of SIGIO from some devices. In malloc, always delay freeing chunks and change ‘F' option to perform a more extensive check for double free. Change sendsyslog prototype to take a string, since there's little point logging not strings. The config program tries to modify zero initialized variables. Previous versions of gcc were patched to place these in the data segment, instead of the bss, but clang has no such patches. Long long ago, this was the default behavior for compilers, which is why gcc was patched to maintain that existing behavior, but now we want a slightly less unusual toolchain. Fix the underlying issue for now by annotating such variables with a data section attribute. *** t2k17 Hackathon Report: Philip Guenther: locking and libc (https://undeadly.org/cgi?action=article;sid=20170824080132) Next up in our series of t2k17 hackathon reports is this one from Philip Guenther: I showed up at t2k17 with a couple hold-over diffs from e2k17 that weren't stable then and hadn't gotten much better since, so after a red-eye through Chicago I arrived in the hackroom, fired up my laptop and synced trees. Meanwhile, people trickled in and the best part of hackathons, the conversations and "what do you think about this?" chats started. Theo introduced me to Todd Mortimer (mortimer@), who's been hacking on clang to implement RETGUARD for C programs. Over the hackathon we discussed a few loose ends that cropped up and what the correct behavior should be for them as well as the mechanics of avoiding 0xc3 bytes (the RET opcode) embedded in the middle of other multi-byte x86 machine code. Fun stuff. Martin (mpi@) and I had a conversation about the desirability of being able to sleep while holding netlock and pretty much came down on "oof, the scheduler does need work before the underlying issue driving this question can be resolved enough to answer it". :-( After some final hammering I got in an enhancement to pool(9) to let a pool use (sleeping) rwlocks instead of (spinning) mutexes and then immediately used that for the per-CPU pool cache pool as well as the futex pool. Further pools are likely to be converted as well kernel upper-level locking changes are made. Speaking of, a larger diff I had been working on for said upper-level locking was still suffering deadlock issues so I took a stab at narrowing it down to just a lock for the process tree, mostly mirroring the FreeBSD proctreelock. That appears to be holding up much better and I just have some code arrangement issues around sysptrace() before that'll go out for final review. Then most of the way through the week, Bob (beck@) vocally complained that life would be easier for libressl if we had some version of pthread_once() and the pthread mutex routines in libc. This would make some other stuff easier too (c.f. /usr/X11R6/lib/libpthread-stubs.*) and the TIB work over the last couple years has basically eliminated the runtime costs of doing so, so I spent most the rest of the hackathon finding the right place to draw a line through libpthread and move everything on the one side of the line into libc. That code seems pretty stable and the xenocara and ports people seem to like—or at least accept—the effects, so it will almost certainly go in with the next libc bump. Lots of other random conversations, hacking, meals, and beer. Many thanks to Ken (krw@) and local conspirators for another excellent Toronto hackathon! Beastie Bits 2017 NetBSD Foundation Officers (https://blog.netbsd.org/tnf/entry/2017_netbsd_foundation_officers) New BSDMag is out - Military Grade Data Wiping in FreeBSD with BCWipe (https://bsdmag.org/download/military-grade-data-wiping-freebsd-bcwipe/) LibertyBSD 6.1 released (http://libertybsd.net/) *** Feedback/Questions Eddy - EuroBSDCon 2017 video and some help (http://dpaste.com/3WDNV05#wrap) Eric - ZFS monitoring (http://dpaste.com/2RP0S60#wrap) Tom - BSD Hosting (http://dpaste.com/31DGH3J#wrap) ***
EuroBSDcon trip report, how to secure OpenBSD's LDAP server, ZFS channel programs in FreeBSD HEAD and why software is storytelling. This episode was brought to you by Headlines EuroBSDcon Trip Report This is from Frank Moore, who has been supplying us with collections of links for the show and who we met at EuroBSDcon in Paris for the first time. Here is his trip report. My attendance at the EuroBSDCon 2017 conference in Paris was sprinkled with several 'firsts'. My first visit to Paris, my first time travelling on a EuroTunnel Shuttle train and my first time at any BSD conference. Hopefully, none of these will turn out to be 'lasts'. I arrived on the Wednesday afternoon before the conference started on Thursday morning. My hotel was conveniently located close to the conference centre in Paris' 3rd arrondissement. This area is well-known as a buzzy enclave of hip cafes, eateries, independent shops, markets, modern galleries and museums. It certainly lived up to its reputation. Even better, the weather held over the course of the conference, only raining once, with the rest of the time being both warm and sunny. The first two days were taken up with attending Dr Kirk McKusick's excellent tutorial 'An Introduction to the FreeBSD Open-Source Operating System'. This is training "straight from the horse's mouth". Kirk has worked extensively on The FreeBSD operating system since the 1980's, helping to design the original BSD filesystem (FFS) and later working on UFS as well. Not only is Kirk an engaging speaker, making what could be a dry topic very interesting, he also sprinkles liberal doses of history and war stories throughout his lectures. Want to know why a protocol was designed the way that it was? Or why a system flag has a particular value or position in a record? Kirk was there and has the first-hand answer. He reminisces about his meetings and work with other Unix and BSD luminaries and debunks and confirms common myths in equal measure. Kirk's teaching style and knowledge are impressive. Every section starts with an overview and a big picture diagram before drilling down into the nitty-gritty detail. Nothing feels superfluous, and everything fits together logically. It's easy to tell that the material and its delivery have been honed over many years, but without feeling stale. Topics covered included the kernel, processes, virtual memory, threads, I/O, devices, FFS, ZFS, and networking. The slides were just as impressive, with additional notes written by a previous student and every slide containing a reference back to the relevant page(s) in the 2nd edition of Kirk's operating system book. As well as a hard copy for those that requested it, Kirk also helpfully supplied soft copies of all the training materials. The breaks in between lectures were useful for meeting the students from the other tutorials and for recovering from the inevitable information overload. It's not often that you can get to hear someone as renowned as Dr McKusick give a lecture on something as important as the FreeBSD operating system. If you have any interest in FreeBSD, Unix history, or operating systems in general, I would urge you to grab the opportunity to attend one of his lectures. You won't be disappointed. The last two days of the conference consisted of various hour-long talks by members of each of the main BSD systems. All of them were fairly evenly represented except Dragonfly BSD which unfortunately only had one talk. With three talks going on at any one time, it was often difficult to pick which one to go to. At other times there might be nothing to pique the interest. Attendance at a talk is not mandatory, so for those times when no talks looked inviting, just hanging out in one of the lobby areas with other attendees was often just as interesting and informative. The conference centre itself was certainly memorable with the interior design of an Egyptian temple or pyramid. All the classrooms were more than adequate while the main auditorium was first-class and easily held the 300+ attendees comfortably. All in all, the facilities, catering and organisation were excellent. Kudos to the EuroBSDCon team, especially Bapt and Antoine for all their hard work and hospitality. As a long-time watcher and occasional contributor to the BSD Now podcast it was good to meet both Allan and Benedict in the flesh. And having done some proofreading for Michael Lucas previously, it was nice to finally meet him as well. My one suggestion to the organisers of the next conference would be to provide more hand-holding for newbies. As a first-time attendee at a BSD conference it would have been nice to have been formally introduced to various people within the projects as the goto people for their areas. I could do this myself, but it's not always easy finding the right person and wrangling an introduction. I also think it was a missed opportunity for each project to recruit new developers to their cause. Apparently, this is already in place at BSDCan, but should probably be rolled out across all BSD conferences. Having said all that, my aims for the conference were to take Dr McKusick's course, meet a few BSD people and make contacts within one of the BSD projects to start contributing. I was successful on all these fronts, so for me this was mission accomplished. Another first! autoconf/clang (No) Fun and Games (https://undeadly.org/cgi?action=article;sid=20170930133438) Robert Nagy (robert@) wrote in with a fascinating story of hunting down a recent problem with ports: You might have been noticing the amount of commits to ports regarding autoconf and nested functions and asking yourself… what the hell is this all about? I was hanging out at my friend Antoine (ajacoutot@)'s place just before EuroBSDCon 2017 started and we were having drinks and he told me that there is this weird bug where Gnome hangs completely after just a couple of seconds of usage and the gnome-shell process just sits in the fsleep state. This started to happen at the time when inteldrm(4) was updated, the default compiler was switched to clang(1) and futexes were turned on by default. The next day we started to have a look at the issue and since the process was hanging in fsleep, it seemed clear that the cause must be futexes, so we had to start bisecting the base system, which resulted in random success and failure. In the end we figured out that it is neither futex nor inteldrm(4) related, so the only thing that was left is the switch to clang. Now the problem is that we have to figure out what part of the system needs to be build with clang to trigger this issue, so we kept on going and systematically recompiled the base system with gcc until everything was ruled out … and it kept on hanging. We were drunk and angry that now we have to go and check hundreds of ports because gnome is not a small standalone port, so between two bottles of wine a build VM was fired up to do a package build with gcc, because manually building all the dependencies would just take too long and we had spent almost two days on this already. Next day ~200 packages were available to bisect and figure out what's going on. After a couple of tries it turned out that the hang is being caused by the gtk+3 package, which is bad since almost everything is using gtk+3. Now it was time to figure out what file the gtk+3 source being built by clang is causing the issue. (Compiler optimizations were ruled out already at this point.) So another set of bisecting happened, building each subdirectory of gtk+3 with clang and waiting for the hang to manifest … and it did not. What the $f? Okay so something else is going on and maybe the configure script of gtk+3 is doing something weird with different compilers, so I quickly did two configure runs with gcc and clang and simply diff'd the two directories. Snippets from the diff: -GDKHIDDENVISIBILITYCFLAGS = -fvisibility=hidden GDKHIDDENVISIBILITYCFLAGS = -ltcvprogcompilerrttiexceptions=no ltcvprogcompilerrttiexceptions=yes -#define GDKEXTERN attribute((visibility("default"))) extern -ltprogcompilernobuiltinflag=' -fno-builtin' +ltprogcompilernobuiltinflag=' -fno-builtin -fno-rtti -fno-exceptions' Okay, okay that's something, but wait … clang has symbol visibility support so what is going on again? Let's take a peek at config.log: configure:29137: checking for -fvisibility=hidden compiler flag configure:29150: cc -c -fvisibility=hidden -I/usr/local/include -I/usr/X11R6/include conftest.c >&5 conftest.c:82:17: error: function definition is not allowed here int main (void) { return 0; } ^ 1 error generated. Okay that's clearly an error but why exactly? autoconf basically generates a huge shell script that will check for whatever you throw at it by creating a file called conftest.c and putting chunks of code into it and then trying to compile it. In this case the relevant part of the code was: | int | main () | { | int main (void) { return 0; } | ; | return 0; | } That is a nested function declaration which is a GNU extension and it is not supported by clang, but that's okay, the question is why the hell would you use nested functions to check for simple compiler flags. The next step was to go and check what is going on in configure.ac to see how the configure script is generated. In the gtk+3 case the following snippet is used: AC_MSG_CHECKING([for -fvisibility=hidden compiler flag]) ACTRYCOMPILE([], [int main (void) { return 0; }], ACMSGRESULT(yes) enablefvisibilityhidden=yes, ACMSGRESULT(no) enablefvisibilityhidden=no) According to the autoconf manual the ACTRYCOMPILE macro accepts the following parameters: That clearly states that a function body has to be specified because the function definition is already provided automatically, so doing ACTRYCOMPILE([], [int main (void) { return 0;}], instead of ACTRYCOMPILE([],[] will result in a nested function declaration, which will work just fine with gcc, even though the autoconf usage is wrong. After fixing the autoconf macro in gtk+3 and rebuilding the complete port from scratch with clang, the hang completely went away as the proper CFLAGS and LDFLAGS were picked up by autoconf for the build. At this point we realized that most of the ports tree uses autoconf so this issue might be a lot bigger than we thought, so I asked sthen@ to do a grep on the ports object directory and just search for "function definition is not allowed here", which resulted in about ~60 additional ports affected. Out of the list of ports there were only two false positive matches. These were actually trying to test whether the compiler supports nested functions. The rest were a combination of several autoconf macros used in a wrong way, e.g: ACTRYCOMPILE, ACTRYLINK. Most of them were fixable by just removing the extra function declaration or by switching to other autoconf macros like ACLANGSOURCE where you can actually declare your own functions if need be. The conclusion is that this issue was a combination of people not reading documentation and just copy/pasting autoconf snippets, instead of reading their documentation and using the macros in the way they were intended, and the fact that switching to a new compiler is never easy and bugs or undefined behaviour are always lurking in the dark. Thanks to everyone who helped fixing all the ports up this quickly! Hopefully all of the changes can be merged upstream, so that others can benefit as well. Interview - David Carlier - @devnexen (https://twitter.com/devnexen) Software Engineer at Afilias *** News Roundup Setting up OpenBSD's LDAP Server (ldapd) with StartTLS and SASL (http://blog.databasepatterns.com/2017/08/setting-up-openbsds-ldap-server-ldapd.html) A tutorial on setting up OpenBSD's native LDAP server with TLS encryption and SASL authentication OpenBSD has its own LDAP server, ldapd. Here's how to configure it for use with StartTLS and SASL authentication Create a certificate (acme-client anyone?) Create a basic config file listen on em0 tls certificate ldapserver This will listen on the em0 interface with tls using the certificate called ldapserver.crt / ldapserver.key Validate the configuration: /usr/sbin/ldapd -n Enable and start the service: rcctl enable ldapd rcctl start ldapd On the client machine: pkg_add openldap-client Copy the certificate to /etc/ssl/trusted.crt Add this line to /etc/openldap/ldap.conf TLS_CACERT /etc/ssl/trusted.crt Enable and start the service rcctl enable saslauthd rcctl start saslauthd Connect to ldapd (-ZZ means force TLS, use -H to specify URI): ldapsearch -H ldap://ldapserver -ZZ FreeBSD Picks Up Support for ZFS Channel Programs in -current (https://svnweb.freebsd.org/base?view=revision&revision=324163) ZFS channel programs (ZCP) adds support for performing compound ZFS administrative actions via Lua scripts in a sandboxed environment (with time and memory limits). This initial commit includes both base support for running ZCP scripts, and a small initial library of API calls which support getting properties and listing, destroying, and promoting datasets. Testing: in addition to the included unit tests, channel programs have been in use at Delphix for several months for batch destroying filesystems. Take a simple task as an example: Create a snapshot, then set a property on that snapshot. In the traditional system for this, when you issue the snapshot command, that closes the currently open transaction group (say #100), and opens a new one, #101. While #100 is being written to disk, other writes are accumulated in #101. Once #100 is flushed to disk, the ‘zfs snapshot' command returns. You can then issue the ‘zfs set' command. This actually ends up going into transaction group #102. Each administrative action needs to wait for the transaction group to flush, which under heavy loads could take multiple seconds. Now if you want to create AND set, you need to wait for two or three transaction groups. Meanwhile, during transaction group #101, the snapshot existed without the property set, which could cause all kinds of side effects. ZFS Channel programs solves this by allowing you to perform a small scripted set of actions as a single atomic operation. In Delphix's appliance, they often needed to do as many as 15 operations together, which might take multiple minutes. Now with channel programs it is much faster, far safer, and has fewer chances of side effects BSDCan 2017 - Matt Ahrens: Building products based on OpenZFS, using channel programs -- Video Soon (http://www.bsdcan.org/2017/schedule/events/854.en.html) Software Is About Storytelling (http://bravenewgeek.com/software-is-about-storytelling/) Tyler Treat writes on the brave new geek blog: Software engineering is more a practice in archeology than it is in building. As an industry, we undervalue storytelling and focus too much on artifacts and tools and deliverables. How many times have you been left scratching your head while looking at a piece of code, system, or process? It's the story, the legacy left behind by that artifact, that is just as important—if not more—than the artifact itself. And I don't mean what's in the version control history—that's often useless. I mean the real, human story behind something. Artifacts, whether that's code or tools or something else entirely, are not just snapshots in time. They're the result of a series of decisions, discussions, mistakes, corrections, problems, constraints, and so on. They're the product of the engineering process, but the problem is they usually don't capture that process in its entirety. They rarely capture it at all. They commonly end up being nothing but a snapshot in time. It's often the sign of an inexperienced engineer when someone looks at something and says, “this is stupid” or “why are they using X instead of Y?” They're ignoring the context, the fact that circumstances may have been different. There is a story that led up to that point, a reason for why things are the way they are. If you're lucky, the people involved are still around. Unfortunately, this is not typically the case. And so it's not necessarily the poor engineer's fault for wondering these things. Their predecessors haven't done enough to make that story discoverable and share that context. I worked at a company that built a homegrown container PaaS on ECS. Doing that today would be insane with the plethora of container solutions available now. “Why aren't you using Kubernetes?” Well, four years ago when we started, Kubernetes didn't exist. Even Docker was just in its infancy. And it's not exactly a flick of a switch to move multiple production environments to a new container runtime, not to mention the politicking with leadership to convince them it's worth it to not ship any new code for the next quarter as we rearchitect our entire platform. Oh, and now the people behind the original solution are no longer with the company. Good luck! And this is on the timescale of about five years. That's maybe like one generation of engineers at the company at most—nothing compared to the decades or more software usually lives (an interesting observation is that timescale, I think, is proportional to the size of an organization). Don't underestimate momentum, but also don't underestimate changing circumstances, even on a small time horizon. The point is, stop looking at technology in a vacuum. There are many facets to consider. Likewise, decisions are not made in a vacuum. Part of this is just being an empathetic engineer. The corollary to this is you don't need to adopt every bleeding-edge tech that comes out to be successful, but the bigger point is software is about storytelling. The question you should be asking is how does your organization tell those stories? Are you deliberate or is it left to tribal knowledge and hearsay? Is it something you truly value and prioritize or simply a byproduct? Documentation is good, but the trouble with documentation is it's usually haphazard and stagnant. It's also usually documentation of how and not why. Documenting intent can go a long way, and understanding the why is a good way to develop empathy. Code survives us. There's a fantastic talk by Bryan Cantrill on oral tradition in software engineering (https://youtu.be/4PaWFYm0kEw) where he talks about this. People care about intent. Specifically, when you write software, people care what you think. As Bryan puts it, future generations of programmers want to understand your intent so they can abide by it, so we need to tell them what our intent was. We need to broadcast it. Good code comments are an example of this. They give you a narrative of not only what's going on, but why. When we write software, we write it for future generations, and that's the most underestimated thing in all of software. Documenting intent also allows you to document your values, and that allows the people who come after you to continue to uphold them. Storytelling in software is important. Without it, software archeology is simply the study of puzzles created by time and neglect. When an organization doesn't record its history, it's bound to repeat the same mistakes. A company's memory is comprised of its people, but the fact is people churn. Knowing how you got here often helps you with getting to where you want to be. Storytelling is how we transcend generational gaps and the inevitable changing of the old guard to the new guard in a maturing engineering organization. The same is true when we expand that to the entire industry. We're too memoryless—shipping code and not looking back, discovering everything old that is new again, and simply not appreciating our lineage. Beastie Bits 1st BSD Users Stockholm Meetup (https://www.meetup.com/en-US/BSD-Users-Stockholm/) Absolute FreeBSD, 3rd Edition draft completed (https://blather.michaelwlucas.com/archives/3020) Absolute FreeBSD, 3rd Edition Table of Contents (https://blather.michaelwlucas.com/archives/2995) t2k17 Hackathon Report: My first time (Aaron Bieber) (https://undeadly.org/cgi?action=article;sid=20170824193521) The release of pfSense 2.4.0 will be slightly delayed to apply patches for vulnerabilities in 3rd party packages that are part of pfSense (https://www.netgate.com/blog/no-plan-survives-contact-with-the-internet.html) Feedback/Questions Ben writes in that zrepl is in ports now (http://dpaste.com/1XMJYMH#wrap) Peter asks us about Netflix on BSD (http://dpaste.com/334WY4T#wrap) meka writes in about dhclient exiting (http://dpaste.com/3GSGKD3#wrap) ***
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
We explore whether a BSD can replicate Cisco router performance; RETGUARD, OpenBSDs new exploit mitigation technology, Dragonfly's HAMMER2 filesystem implementation & more! This episode was brought to you by Headlines Can a BSD system replicate the performance of a Cisco router? (https://www.reddit.com/r/networking/comments/6upchy/can_a_bsd_system_replicate_the_performance_of/) Short Answer: No, but it might be good enough for what you need Traditionally routers were built with a tightly coupled data plane and control plane. Back in the 80s and 90s the data plane was running in software on commodity CPUs with proprietary software. As the needs and desires for more speeds and feeds grew, the data plane had to be implemented in ASICs and FPGAs with custom memories and TCAMs. While these were still programmable in a sense, they certainly weren't programmable by anyone but a small handful of people who developed the hardware platform. The data plane was often layered, where features not handled by the hardware data plane were punted to a software only data path running on a more general CPU. The performance difference between the two were typically an order or two of magnitude. source (https://fd.io/wp-content/uploads/sites/34/2017/07/FDioVPPwhitepaperJuly2017.pdf) Except for encryption (e.g. IPsec) or IDS/IPS, the true measure of router performance is packets forwarded per unit time. This is normally expressed as Packets-per-second, or PPS. To 'line-rate' forward on a 1gbps interface, you must be able to forward packets at 1.488 million pps (Mpps). To forward at "line-rate" between 10Gbps interfaces, you must be able to forward at 14.88Mpps. Even on large hardware, kernel-forwarding is limited to speeds that top out below 2Mpps. George Neville-Neil and I did a couple papers on this back in 2014/2015. You can read the papers (https://github.com/freebsd-net/netperf/blob/master/Documentation/Papers/ABSDCon2015Paper.pdf) for the results. However, once you export the code from the kernel, things start to improve. There are a few open source code bases that show the potential of kernel-bypass networking for building a software-based router. The first of these is netmap-fwd which is the FreeBSD ip_forward() code hosted on top of netmap, a kernel-bypass technology present in FreeBSD (and available for linux). Full-disclosure, netmap-fwd was done at my company, Netgate. netmap-fwd will l3 forward around 5 Mpps per core. slides (https://github.com/Netgate/netmap-fwd/blob/master/netmap-fwd.pdf) The first of these is netmap-fwd (https://github.com/Netgate/netmap-fwd) which is the FreeBSD ip_forward() code hosted on top of netmap (https://github.com/luigirizzo/netmap), a kernel-bypass technology present in FreeBSD (and available for linux). Full-disclosure, netmap-fwd was done at my company, Netgate. (And by "my company" I mean that I co-own it with my spouse.). netmap-fwd will l3 forward around 5 Mpps per core. slides (https://github.com/Netgate/netmap-fwd/blob/master/netmap-fwd.pdf) Nanako Momiyama of the Keio Univ Tokuda Lab presented on IP Forwarding Fastpath (https://www.bsdcan.org/2017/schedule/events/823.en.html) at BSDCan this past May. She got about 5.6Mpps (roughly 10% faster than netmap-fwd) using a similar approach where the ip_foward() function was rewritten as a module for VALE (the netmap-based in-kernel switch). Slides (https://2016.eurobsdcon.org/PresentationSlides/NanakoMomiyama_TowardsFastIPForwarding.pdf) from her previous talk at EuroBSDCon 2016 are available. (Speed at the time was 2.8Mpps.). Also a paper (https://www.ht.sfc.keio.ac.jp/~nanako/conext17-sw.pdf) from that effort, if you want to read it. Of note: They were showing around 1.6Mpps even after replacing the in-kernel routing lookup algorithm with DXR. (DXR was written by Luigi Rizzo, who is also the primary author of netmap.) Not too long after netmap-fwd was open sourced, Ghandi announced packet-journey, an application based on drivers and libraries and from DPDK. Packet-journey is also an L3 router. The GitHub page for packet-journey lists performance as 21,773.47 mbps (so 21.77Gbps) for 64-byte UDP frames with 50 ACLs and 500,000 routes. Since they're using 64-byte frames, this translates to roughly 32.4Mpps. Finally, there is recent work in FreeBSD (which is part of 11.1-RELEASE) that gets performance up to 2x the level of netmap-fwd or the work by Nanako Momiyama. 10 million PPS: Here (http://blog.cochard.me/2015/09/receipt-for-building-10mpps-freebsd.html) is a decent introduction. But of course, even as FreeBSD gets up to being able to do 10gbps at line-rate, 40 and 100 gigabits are not uncommon now Even with the fastest modern CPUs, this is very little time to do any kind of meaningful packet processing. At 10Gbps, your total budget per packet, to receive (Rx) the packet, process the packet, and transmit (Tx) the packet is 67.2 ns. Complicating the task is the simple fact that main memory (RAM) is 70 ns away. The simple conclusion here is that, even at 10Gbps, if you have to hit RAM, you can't generate the PPS required for line-rate forwarding. There is some detail about design tradeoffs in the Ryzen architecture and how that might impact using those machines as routers Anyway... those are all interesting, but the natural winner here is FD.io's Vector Packet Processing (VPP). Read this (http://blogs.cisco.com/sp/a-bigger-helping-of-internet-please) VPP is an efficient, flexible open source data plane. It consists of a set of forwarding nodes arranged in a directed graph and a supporting framework. The framework has all the basic data structures, timers, drivers (and interfaces to both DPDK and netmap), a scheduler which allocates the CPU time between the graph nodes, performance and debugging tools, like counters and built-in packet trace. The latter allows you to capture the paths taken by the packets within the graph with high timestamp granularity, giving full insight into the processing on a per-packet level. The net result here is that Cisco (again, Cisco) has shown the ability to route packets at 1 Tb/s using VPP on a four socket Purley system There is also much discussion of the future of pfSense, as they transition to using VPP This is a very lengthy write up which deserves a full read, plus there are some comments from other people *** RETGUARD, the OpenBSD next level in exploit mitigation, is about to debut (https://marc.info/?l=openbsd-tech&m=150317547021396&w=2) This year I went to BSDCAN in Ottawa. I spent much of it in the 'hallway track', and had an extended conversation with various people regarding our existing security mitigations and hopes for new ones in the future. I spoke a lot with Todd Mortimer. Apparently I told him that I felt return-address protection was impossible, so a few weeks later he sent a clang diff to address that issue... The first diff is for amd64 and i386 only -- in theory RISC architectures can follow this approach soon. The mechanism is like a userland 'stackghost' in the function prologue and epilogue. The preamble XOR's the return address at top of stack with the stack pointer value itself. This perturbs by introducing bits from ASLR. The function epilogue undoes the transform immediately before the RET instruction. ROP attack methods are impacted because existing gadgets are transformed to consist of " RET". That pivots the return sequence off the ROP chain in a highly unpredictable and inconvenient fashion. The compiler diff handles this for all the C code, but the assembly functions have to be done by hand. I did this work first for amd64, and more recently for i386. I've fixed most of the functions and only a handful of complex ones remain. For those who know about polymorphism and pop/jmp or JOP, we believe once standard-RET is solved those concerns become easier to address seperately in the future. In any case a substantial reduction of gadgets is powerful. For those worried about introducing worse polymorphism with these "xor; ret" epilogues themselves, the nested gadgets for 64bit and 32bit variations are +1 "xor %esp,(%rsp); ret", +2 "and $0x24,%al; ret" and +3 "and $0xc3,%al; int3". Not bad. Over the last two weeks, we have received help and advice to ensure debuggers (gdb, egdb, ddb, lldb) can still handle these transformed callframes. Also in the kernel, we discovered we must use a smaller XOR, because otherwise userland addresses are generated, and cannot rely on SMEP as it is really new feature of the architecture. There were also issues with pthreads and dlsym, which leads to a series of uplifts around _builtinreturn_address and DWARF CFI. Application of this diff doesn't require anything special, a system can simply be built twice. Or shortcut by building & installing gnu/usr.bin/clang first, then a full build. We are at the point where userland and base are fully working without regressions, and the remaining impacts are in a few larger ports which directly access the return address (for a variety of reasons). So work needs to continue with handling the RET-addr swizzle in those ports, and then we can move forward. You can find the full message with the diff here (https://marc.info/?l=openbsd-tech&m=150317547021396&w=2) *** Interview - Ed Maste, Charlie & Siva - @ed_maste (https://twitter.com/ed_maste), @yzgyyang (https://twitter.com/yzgyyang) & @svmhdvn (https://twitter.com/svmhdvn) Co-op Students for the FreeBSD Foundation *** News Roundup Next DFly release will have an initial HAMMER2 implementation (http://lists.dragonflybsd.org/pipermail/users/2017-August/313558.html) The next DragonFly release (probably in September some time) will have an initial HAMMER2 implementation. It WILL be considered experimental and won't be an installer option yet. This initial release will only have single-image support operational plus basic features. It will have live dedup (for cp's), compression, fast recovery, snapshot, and boot support out of the gate. This first H2 release will not have clustering or multi-volume support, so don't expect those features to work. I may be able to get bulk dedup and basic mirroring operational by release time, but it won't be very efficient. Also, right now, sync operations are fairly expensive and will stall modifying operations to some degree during the flush, and there is no reblocking (yet). The allocator has a 16KB granularity (on HAMMER1 it was 2MB), so for testing purposes it will still work fairly well even without reblocking. The design is in a good place. I'm quite happy with how the physical layout turned out. Allocations down to 1KB are supported. The freemap has a 16KB granularity with a linear counter (one counter per 512KB) for packing smaller allocations. INodes are 1KB and can directly embed 512 bytes of file data for files 512 bytes. The freemap is also zoned by type for I/O locality. The blockrefs are 'fat' at 128 bytes but enormously powerful. That will allow us to ultimately support up to a 512-bit crypto hash and blind dedup using said hash. Not on release, but that's the plan. I came up with an excellent solution for directory entries. The 1KB allocation granularity was a bit high but I didn't want to reduce it. However, because blockrefs are now 128 byte entities, and directory entries are hashed just like in H1, I was able to code them such that a directory entry is embedded in the blockref itself and does not require a separate data reference or allocation beyond that. Filenames up to 64 bytes long can be accomodated in the blockref using the check-code area of the blockref. Longer filenames will use an additional data reference hanging off the blockref to accomodate up to 255 char filenames. Of course, a minimum of 1KB will have to be allocated in that case, but filenames are
In this episode, we take a look at the reimplementation of NetBSD using a Microkernel, check out what makes DHCP faster, and see what high-process count support for DragonflyBSD has to offer, and we answer the questions you've always wanted to ask us. This episode was brought to you by Headlines A Reimplementation Of Netbsd Using a Microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Minix author Andy Tanenbaum writes in Part 1 of a-reimplementation-of-netbsd-using-a-microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project. Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running. The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this. The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project. Part 2 (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-2-of-2/) is also available. *** Rapid DHCP: Or, how do Macs get on the network so fast? (https://cafbit.com/post/rapid_dhcp_or_how_do/) One of life's minor annoyances is having to wait on my devices to connect to the network after I wake them from sleep. All too often, I'll open the lid on my EeePC netbook, enter a web address, and get the dreaded "This webpage is not available" message because the machine is still working on connecting to my Wi-Fi network. On some occasions, I have to twiddle my thumbs for as long as 10-15 seconds before the network is ready to be used. The frustrating thing is that I know it doesn't have to be this way. I know this because I have a Mac. When I open the lid of my MacBook Pro, it connects to the network nearly instantaneously. In fact, no matter how fast I am, the network comes up before I can even try to load a web page. My curiosity got the better of me, and I set out to investigate how Macs are able to connect to the network so quickly, and how the network connect time in other operating systems could be improved. I figure there are three main categories of time-consuming activities that occur during network initialization: Link establishment. This is the activity of establishing communication with the network's link layer. In the case of Wi-Fi, the radio must be powered on, the access point detected, and the optional encryption layer (e.g. WPA) established. After link establishment, the device is able to send and receive Ethernet frames on the network. Dynamic Host Configuration Protocol (DHCP). Through DHCP handshaking, the device negotiates an IP address for its use on the local IP network. A DHCP server is responsible for managing the IP addresses available for use on the network. Miscellaneous overhead. The operating system may perform any number of mundane tasks during the process of network initialization, including running scripts, looking up preconfigured network settings in a local database, launching programs, etc. My investigation thus far is primarily concerned with the DHCP phase, although the other two categories would be interesting to study in the future. I set up a packet capture environment with a spare wireless access point, and observed the network activity of a number of devices as they initialized their network connection. For a worst-case scenario, let's look at the network activity captured while an Android tablet is connecting: This tablet, presumably in the interest of "optimization", is initially skipping the DHCP discovery phase and immediately requesting its previous IP address. The only problem is this is a different network, so the DHCP server ignores these requests. After about 4.5 seconds, the tablet stubbornly tries again to request its old IP address. After another 4.5 seconds, it resigns itself to starting from scratch, and performs the DHCP discovery needed to obtain an IP address on the new network. In all fairness, this delay wouldn't be so bad if the device was connecting to the same network as it was previously using. However, notice that the tablet waits a full 1.13 seconds after link establishment to even think about starting the DHCP process. Engineering snappiness usually means finding lots of small opportunities to save a few milliseconds here and there, and someone definitely dropped the ball here. In contrast, let's look at the packet dump from the machine with the lightning-fast network initialization, and see if we can uncover the magic that is happening under the hood: The key to understanding the magic is the first three unicast ARP requests. It looks like Mac OS remembers certain information about not only the last connected network, but the last several networks. In particular, it must at least persist the following tuple for each of these networks: > 1. The Ethernet address of the DHCP server > 2. The IP address of the DHCP server > 3. Its own IP address, as assigned by the DHCP server During network initialization, the Mac transmits carefully crafted unicast ARP requests with this stored information. For each network in its memory, it attempts to send a request to the specific Ethernet address of the DHCP server for that network, in which it asks about the server's IP address, and requests that the server reply to the IP address which the Mac was formerly using on that network. Unless network hosts have been radically shuffled around, at most only one of these ARP requests will result in a response—the request corresponding to the current network, if the current network happens to be one of the remembered networks. This network recognition technique allows the Mac to very rapidly discover if it is connected to a known network. If the network is recognized (and presumably if the Mac knows that the DHCP lease is still active), it immediately and presumptuously configures its IP interface with the address it knows is good for this network. (Well, it does perform a self-ARP for good measure, but doesn't seem to wait more than 13ms for a response.) The DHCP handshaking process begins in the background by sending a DHCP request for its assumed IP address, but the network interface is available for use during the handshaking process. If the network was not recognized, I assume the Mac would know to begin the DHCP discovery phase, instead of sending blind requests for a former IP address as the Galaxy Tab does. The Mac's rapid network initialization can be credited to more than just the network recognition scheme. Judging by the use of ARP (which can be problematic to deal with in user-space) and the unusually regular transmission intervals (a reliable 1.0ms delay between each packet sent), I'm guessing that the Mac's DHCP client system is entirely implemented as tight kernel-mode code. The Mac began the IP interface initialization process a mere 10ms after link establishment, which is far faster than any other device I tested. Android devices such as the Galaxy Tab rely on the user-mode dhclient system (part of the dhcpcd package) dhcpcd program, which no doubt brings a lot of additional overhead such as loading the program, context switching, and perhaps even running scripts. The next step for some daring kernel hacker is to implement a similarly aggressive DHCP client system in the Linux kernel, so that I can enjoy fast sign-on speeds on my Android tablet, Android phone, and Ubuntu netbook. There already exists a minimal DHCP client implementation in the Linux kernel, but it lacks certain features such as configuring the DNS nameservers. Perhaps it wouldn't be too much work to extend this code to support network recognition and interface with a user-mode daemon to handle such auxillary configuration information received via DHCP. If I ever get a few spare cycles, maybe I'll even take a stab at it. You can also find other ways of optimizing the dhclient program and how it works in the dhclient tutorial on Calomel.org (https://calomel.org/dhclient.html). *** BSDCam Trip Report (https://www.freebsdfoundation.org/blog/bsdcam-2017-trip-report-michael-lucas/) Over the decades, FreeBSD development and coordination has shifted from being purely on-line to involving more and more in-person coordination and cooperation. The FreeBSD Foundation sponsors a devsummit right before BSDCan, EuroBSDCon, and AsiaBSDCon, so that developers traveling to the con can leverage their airfare and hammer out some problems. Yes, the Internet is great for coordination, but nothing beats a group of developers spending ten minutes together to sketch on a whiteboard and figuring out exactly how to make something bulletproof. In addition to the coordination efforts, though, conference devsummits are hierarchical. There's a rigid schedule, with topics decided in advance. Someone leads the session. Sessions can be highly informative, passionate arguments, or anything in between. BSDCam is… a little different. It's an invaluable part of the FreeBSD ecosystem. However, it's something that I wouldn't normally attend. But right now, is not normal. I'm writing a new edition of Absolute FreeBSD. To my astonishment, people have come to rely on this book when planning their deployments and operations. While I find this satisfying, it also increases the pressure on me to get things correct. When I wrote my first FreeBSD book back in 2000, a dozen mailing lists provided authoritative information on FreeBSD development. One person could read every one of those lists. Today, that's not possible—and the mailing lists are only one narrow aspect of the FreeBSD social system. Don't get me wrong—it's pretty easy to find out what people are doing and how the system works. But it's not that easy to find out what people will be doing and how the system will work. If this book is going to be future-proof, I needed to leave my cozy nest and venture into the wilds of Cambridge, England. Sadly, the BSDCam chair agreed with my logic, so I boarded an aluminum deathtrap—sorry, a “commercial airliner”—and found myself hurtled from Detroit to Heathrow. And one Wednesday morning, I made it to the William Gates building of Cambridge University, consciousness nailed to my body by a thankfully infinite stream of proper British tea. BSDCam attendance is invitation only, and the facilities can only handle fifty folks or so. You need to be actively working on FreeBSD to wrangle an invite. Developers attend from all over the world. Yet, there's no agenda. Robert Watson is the chair, but he doesn't decide on the conference topics. He goes around the room and asks everyone to introduce themselves, say what they're working on, and declare what they want to discuss during the conference. The topics of interest are tallied. The most popular topics get assigned time slots and one of the two big rooms. Folks interested in less popular topics are invited to claim one of the small breakout rooms. Then the real fun begins. I started by eavesdropping in the virtualization workshop. For two hours, people discussed FreeBSD's virtualization needs, strengths, and weaknesses. What needs help? What should this interface look like? What compatibility is important, and what isn't? By the end of the session, the couple dozen people had developed a reasonable consensus and, most importantly, some folks had added items to their to-do lists. Repeat for a dozen more topics. I got a good grip on what's really happening with security mitigation techniques, FreeBSD's cloud support, TCP/IP improvements, advances in teaching FreeBSD, and more. A BSDCan devsummit presentation on packaging the base system is informative, but eavesdropping on two dozen highly educated engineers arguing about how to nail down the final tidbits needed to make that a real thing is far more educational. To my surprise, I was able to provide useful feedback for some sessions. I speak at a lot of events outside of the FreeBSD world, and was able to share much of what I hear at Linux conferences. A tool that works well for an experienced developer doesn't necessarily work well for everyone. Every year, I leave BSDCan tired. I left BSDCam entirely exhausted. These intense, focused discussions stretched my brain. But, I have a really good idea where key parts of FreeBSD development are actually headed. This should help future-proof the new Absolute FreeBSD, as much as any computer book can be future-proof. Plus, BSDCam throws the most glorious conference dinner I've ever seen. I want to thank Robert Watson for his kind invitation, and the FreeBSD Foundation for helping defray the cost of this trip Interview - The BSDNow Crew As a kid, what did you dream of to become as an adult? JT: An Astronaut BR: I wanted to be a private detective, because of all the crime novels that I read back then. I didn't get far with it. However, I think the structured analysis skills (who did what, when, and such) help me in debugging and sysadmin work. AJ: Didn't think about it much How do you manage to stay organized day to day with so much things you're actively doing each day? (Day job, wife/girlfriend, conferences, hobbies, friends, etc.) JT: Who said I was organized? BR: A lot of stuff in my calendar as reminders, open browser tabs as “to read later” list. A few things like task switching when getting stuck helps. Also, focus on a single goal for the day, even though there will be distractions. Slowly, but steadily chip away at the things you're working on. Rather than to procrastinate and put things back to review later, get started early with easy things for a big task and then tackle the hard part. Often, things look totally chaotic and unmanageable, until you start working on them. AJ: I barely manage. Lots of Google Calendar reminders, and the entire wall of my office is covered in whiteboard sheet todo lists. I use pinboard.in to deal with finding and organizing bookmarks. Write things down, don't trust your memory. What hobbies outside of IT do you have? JT: I love photography, but I do that Professional part time, so I'm not sure if that counts as a hobby anymore. I guess it'd have to be working in the garage on my cars. BR: I do Tai Chi to relax once a week in a group, but can also do it alone, pretty much everywhere. Way too much Youtube watching and browsing the web. I did play some games before studying at the university and I'm still proud that I could control it to the bare minimum not to impact my studies. A few “lapses” from time to time, revisiting the old classics since the newer stuff won't run on my machines anyway. Holiday time is pretty much spent for BSD conferences and events, this is where I can relax and talk with like-minded people from around the world, which is fascinating. Plus, it gets me to various places and countries I never would have dared to visit on my own. AJ: I play a few video games, and I like to ski, although I don't go very often as most of my vacation time is spent hanging out with my BSD friends at various conferences How do you relax? JT: What is this word ‘relax' and what does it mean? BR: My Tai Chi plays a big part in it I guess. I really calms you and the constant stream of thoughts for a while. It also gives you better clarity of what's important in life. Watching movies, sleeping long. AJ: Usually watching TV or Movies. Although I have taken to doing most of my TV watching on my exercise bike now, but it is still mentally relaxing If FreeBSD didn't exist, which BSD flavour would you use? Why? JT: I use TrueOS, but if FreeBSD didn't exist, that project might not either… so… My other choice would be HardenedBSD, but since it's also based on FreeBSD I'm in the same dillema. BR: I once installed NetBSD to see what It can do. If FreeBSD wouldn't exist, I would probably try my luck with it. OpenBSD is also appealing, but I've never installed it. AJ: When I started using FreeBSD in 2000, the only other BSD I had heard of at the time was OpenBSD. If FreeBSD wasn't around, I don't think the world would look like it does, so it is hard to speculate. If any of the BSD's weren't around and you had to use Linux, which camp would belong to? (Redhat, SUSE, Debian, Ubuntu, Gentoo?) JT: I learned Linux in the mid 90s using Slackware, which I used consistently up until the mid 2000s, when I joined the PuppyLinux community and eventually became a developer (FYI, Puppy was/is/can be based on Slackware -- its complicated). So I'd go back to using either Slackware or PuppyLinux. BR: I tried various Linux distributions until I landed at Debian. I used is pretty extensively as my desktop OS at home, building custom kernels and packages to install them until I discovered FreeBSD. I ran both side by side for a few months for learning until one day I figured out that I had not booted Debian in a while, so I switched completely. AJ: The first Linux I played with was Slackware, and it is the most BSD like, but the bits of Linux I learned in school were Redhat and so I can somewhat wrap my head around it, although now that they are changing everything to systemd, all of that old knowledge is more harmful than useful. Are you still finding yourself in need to use Windows/Mac OS? Why? JT: I work part time as a professional Photographer, so I do use Windows for my photography work. While I can do everything I need to do in Linux, it comes down to being pragmatic about my time. What takes me several hours to accomplish in Linux I can accomplish in 20 minutes on Windows. BR: I was a long time Windows-only user before my Unix days. But back when Vista was about to come out and I needed a new laptop, my choice was basically learning to cope with Vistas awful features or learn MacOS X. I did the latter, it increased my productivity since it's really a good Unix desktop experience (at least, back then). I only have to use Windows at work from time to time as I manage our Windows Terminal server, which keeps the exposure low enough and I only connect to it to use a certain app not available for the Mac or the BSDs. AJ: I still use Windows to play games, for a lot of video conferencing, and to produce BSD Now. Some of it could be done on BSD but not as easily. I have promised myself that I will switch to 100% BSD rather than upgrade to Windows 10, so we'll see how that goes. Please describe your home networking setup. Router type, router OS, router hardware, network segmentation, wifi apparatus(es), other devices connected, and anything else that might be interesting about your home network. BR: Very simple and boring: Apple Airport Express base station and an AVM FritzBox for DNS, DHCP, and the link to my provider. A long network cable to my desktop machine. That I use less and less often. I just bought an RPI 3 for some home use in the future to replace it. Mostly my brother's and my Macbook Pro's are connected, our phones and the iPad of my mother. AJ: I have a E3-1220 v3 (dual 3.1ghz + HT) with 8 GB of ram, and 4x Intel gigabit server NICs as my router, and it runs vanilla FreeBSD (usually some snapshot of -current). I have 4 different VLANs, Home, Office, DMZ, and Guest WiFi. WiFi is served via a tiny USB powered device I bought in Tokyo years ago, it serves 3 different SSIDs, one for each VLAN except the DMZ. There are ethernet jacks in every room wired for 10 gigabit, although the only machines with 10 gigabit are my main workstation, file server, and some machines in the server rack. There are 3 switches, one for the house (in the laundry room), one for the rack, and one for 10gig stuff. There is a rack in the basement spare bedroom, it has 7 servers in it, mostly storage for live replicas of customer data for my company. How do guys manage to get your work done on FreeBSD desktops? What do you do when you need to a Linux or Windows app that isn't ported, or working? I've made several attempts to switch to FreeBSD, but each attempt failed because of tools not being available (e.g. Zoom, Dropbox, TeamViewer, Crashplan) or broken (e.g. VirtualBox). BR: I use VIrtualBox for everything that is not natively available or Windows-only. Unfortunately, that means no modern games. I mostly do work in the shell when I'm on FreeBSD and when it has to be a graphical application, then I use Fluxbox as the DE. I want to get work done, not look at fancy eye-candy that get's boring after a while. Deactivated the same stuff on my mac due to the same reason. I look for alternative software online, but my needs are relatively easy to satisfy as I'm not doing video editing/rendering and such. AJ: I generally find that I don't need these apps. I use Firefox, Thunderbird, OpenSSH, Quassel, KomodoEdit, and a few other apps, so my needs are not very demanding. It is annoying when packages are broken, but I usually work around this with boot environments, and being able to just roll back to a version that worked for a few days until the problem is solved. I do still have access to a windows machine for the odd time I need specific VPN software or access to Dell/HP etc out-of-band management tools. Which desktop environments are your favorite, and why? For example, I like i3, Xfce, and I'm drawn to Lumina's ethos, but so far always seem to end up back on Xfc because of its ease of use, flexibility, and dashing good looks. JT: As a Lumina Desktop developer, I think my preference is obvious. ;) I am also a long timeOpenBox user, so I have a soft place in my heart for that as well. BR: I use Fluxbox when I need to work with a lot of windows or an application demands X11. KDE and others are too memory heavy for me and I rarely use even 20% of the features they provide. AJ: I was a long time KDE user, but I have adopted Lumina. I find it fast, and that it gets out of my way and lets me do what I want. It had some annoyances early on, but I've nagged the developers into making it work for me. Which command-line shells do you prefer, why, and how (if at all) have you customised the environment or prompt? BR: I use zsh, but without all the fancy stuff you can find online. It might make you more productive, yes. But again, I try to keep things simple. I'm slowly learning tmux and want to work more in it in the future. I sometimes look at other BSD people's laptops and am amazed at what they do with window-management in tmux. My prompt looks like this: bcr@Voyager:~> 20:20 17-08-17 Put this in your .zshrc to get the same result: PROMPT='%n@%m:%~>' RPROMPT='%T %D' AJ: I started using tcsh early on, because it was the shell on the first box I had access to, and because one of the first things I read in “BSD Hacks” was how to enable ‘typo correction”, which made my life a lot better especially on dial up in the early days. My shell prompt looks like this: allan@CA-TOR1-02:/usr/home/allan% What is one thing (or more) missing in FreeBSD you would import from another project or community? Could be tech, process, etc. JT: AUFS from Linux BR: Nohup from Illumos where you can detach an already running process and put it in the background. I often forget that and I'm not in tmux when that happens, so I can see myself use that feature a lot. AJ: Zones (more complete Jails) from IllumOS how do you manage your time to learn about and work on FreeBSD? Does your work/employment enable what you do, or are your contributions mainly done in private time? JT: These days I'm mostly learning things I need for work, so it just falls into something I'm doing while working on work projects. BR: We have a lot of time during the semester holidays to learn on our own, it's part of the idea of being in a university to keep yourself updated, at least for me. Especially in the fast moving world of IT. I also read a lot in my free time. My interests can shift sometimes, but then I devour everything I can find on the topic. Can be a bit excessive, but has gotten me where I am now and I still need a lot to learn (and want to). Since I work with FreeBSD at work (my owndoing), I can try out many things there. AJ: My work means a spend a lot of time working with FreeBSD, but not that much time working ON it. My contributions are mostly done outside of work, but as I own the company I do get more flexibility to take time off for conferences and other FreeBSD related stuff. we know we can bribe Michael W Lucas with gelato (good gelato that is), but what can we use to bribe you guys? Like when I want to have Allan to work on fixing a bug which prevents me from running ZFS on this fancy rock64 board? BR: Desserts of various kinds. AJ: I am probably not the right person to look at your rock64 board. Most people in the project have taken to bribing me with chocolate. In general, my todo list is so long, the best way is a trade, you take this task and I'll take that task. Is your daily mobile device iOS, Android, Windows Mobile, or other? Why? JT: These days I'm using Android on my Blackberry Priv, but until recently I was still a heavy user of Sailfish OS. I would use SailfishOS everyday, if I could find a phone with a keyboard that I could run it on. BR: iOS on the iPhone 7 currently. Never used an Android phone, saw it on other people's devices and what they can do with it (much more). But the infrequent security updates (if any at all) keep me away from it. AJ: I have a Google Nexus 6 (Android 7.1). I wanted the ‘pure' Android experience, and I had been happy with my previous Nexus S. I don't run a custom OS/ROM or anything because I use the phone to verify that video streams work on an ‘average users device'. I am displeased that support for my device will end soon. I am not sure what device I will get next, but it definitely won't be an iPhone. News Roundup Beta Update - Request for (more) Testing (http://undeadly.org/cgi?action=article&sid=20170808065718&mode=flat&count=30) https://beta.undeadly.org/ has received an update. The most significant changes include: The site has been given a less antiquated "look". (As the topic icons have been eliminated, we are no longer seeking help with those graphics.) The site now uses a moderate amount of semantic HTML5. Several bugs in the HTML fragment validator (used for submissions and comments) have been fixed. To avoid generating invalid HTML, submission content which fails validation is no longer displayed in submission/comment previews. Plain text submissions are converted to HTML in a more useful fashion. (Instead of just converting each EOL to , the converter now generates proper paragraphs and interprets two or more consecutive EOLs as indicating a paragraph break.) The redevelopment remains a work-in-progress. Many thanks to those who have contributed! As before, constructive feedback would be appreciated. Of particular interest are reports of bugs in behaviour (for example, in the HTML validator or in authentication) that would preclude the adoption of the current code for the main site. High-process-count support added to master (http://lists.dragonflybsd.org/pipermail/users/2017-August/313552.html) We've fixed a number of bottlenecks that can develop when the number of user processes runs into the tens of thousands or higher. One thing led to another and I said to myself, "gee, we have a 6-digit PID, might as well make it work to a million!". With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. And, in fact, as today's machines start to ratchet up there in both memory capacity and core count, with fast storage (NVMe) and fast networking (10GigE and higher), even in consumer boxes, this is actually something that one might want to do. With AMD's threadripper and EPYC chips now out, the IntelAMD cpu wars are back on! Boasting up to 32 cores (64 threads) per socket and two sockets on EPYC, terabytes of ram, and motherboards with dual 10GigE built-in, the reality is that these numbers are already achievable in a useful manner. In anycase, I've tested these changes on a dual-socket xeon. I can in-fact start 900,000 processes. They don't get a whole lot of cpu and running 'ps' would be painful, but it works and the system is still responsive from the shell with all of that going on. xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 In fact, judging from the memory use, these minimal test processes only eat around 60KB each. 900,000 of them ate only 55GB on a 128GB machine. So even a million processes is not out of the question, depending on the cpu requirements for those processes. Today's modern machines can be stuffed with enormous amounts of memory. Of course, our PIDs are currently limited to 6 digits, so a million is kinda the upper limit in terms of discrete user processes (verses pthreads which are less restricted). I'd rather not go to 7 digits (yet). CFT: Driver for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens (https://lists.freebsd.org/pipermail/freebsd-current/2017-August/066783.html) Following patch [1] adds support for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens via evdev protocol. It is intended to be a native replacement of hid-multitouch.c driver found in Linux distributions and multimedia/webcamd port. Patch is made for 12-CURRENT and most probably can be applied to recent 11-STABLE and 11.1-RELEASE (not tested) How to test" 1. Apply patch [1] 2. To compile this driver into the kernel, place the following lines into your kernel configuration file: device wmt device usb device evdev Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5): wmt_load="YES" 3. Install x11-drivers/xf86-input-evdev or x11-drivers/xf86-input-libinput port 4. Tell XOrg to use evdev or libinput driver for the device: ``` Section "ServerLayout" InputDevice "TouchScreen0" "SendCoreEvents" EndSection Section "InputDevice" Identifier "TouchScreen0" Driver "evdev" # Driver "libinput" Option "Device" "/dev/input/eventXXX" EndSection ``` Exact value of "/dev/input/eventXXX" can be obtained with evemu-record utility from devel/evemu. Note1: Currently, driver does not support pens or touchpads. Note2: wmt.ko should be kld-loaded before uhid driver to take precedence over it! Otherwise uhid can be kld-unloaded after loading of wmt. wmt review: https://reviews.freebsd.org/D12017 Raw diff: https://reviews.freebsd.org/D12017.diff *** Beastie Bits BSDMag Programing Languages Infographic (https://bsdmag.org/programm_history/) t2k17 Hackathon Report: Bob Beck on buffer cache tweaks, libressl and pledge progress (http://undeadly.org/cgi?action=article&sid=20170815171854) New FreeBSD Journal (https://www.freebsdfoundation.org/past-issues/resource-control/) NetBSD machines at Open Source Conference 2017 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2017/08/10/msg000744.html) *** Feedback/Questions Dan - HDD question (http://dpaste.com/3H6TDJV) Benjamin - scrub of death (http://dpaste.com/10F086V) Jason - Router Opinion (http://dpaste.com/2D9102K) Sohrab - Thanks (http://dpaste.com/1XYYTWF) ***
In which we interview a unicorn, FreeNAS 11.0 is out, show you how to run Nextcloud in a FreeBSD jail, and talk about the connection between oil changes and software patches. This episode was brought to you by Headlines FreeNAS 11.0 is Now Here (http://www.freenas.org/blog/freenas-11-0/) The FreeNAS blog informs us: After several FreeNAS Release Candidates, FreeNAS 11.0 was released today. This version brings new virtualization and object storage features to the World's Most Popular Open Source Storage Operating System. FreeNAS 11.0 adds bhyve virtual machines to its popular SAN/NAS, jails, and plugins, letting you use host web-scale VMs on your FreeNAS box. It also gives users S3-compatible object storage services, which turns your FreeNAS box into an S3-compatible server, letting you avoid reliance on the cloud. FreeNAS 11.0 also introduces the beta version of a new administration GUI. The new GUI is based on the popular Angular framework and the FreeNAS team expects the GUI to be themeable and feature complete by 11.1. The new GUI follows the same flow as the existing GUI, but looks better. For now, the FreeNAS team has released it in beta form to get input from the FreeNAS community. The new GUI, as well as the classic GUI, are selectable from the login screen. Also new in FreeNAS 11 is an Alert Service page which configures the system to send critical alerts from FreeNAS to other applications and services such as Slack, PagerDuty, AWS, Hipchat, InfluxDB, Mattermost, OpsGenie, and VictorOps. FreeNAS 11.0 has an improved Services menu that adds the ability to manage which services and applications are started at boot. The FreeNAS community is large and vibrant. We invite you to join us on the FreeNAS forum (https://forums.freenas.org/index.php) and the #freenas IRC channel on Freenode. To download FreeNAS and sign-up for the FreeNAS Newsletter, visit freenas.org/download (http://www.freenas.org/download/). Building an IPsec Gateway With OpenBSD (https://www.exoscale.ch/syslog/2017/06/26/building-an-ipsec-gateway-with-openbsd/) Pierre-Yves Ritschard wrote the following blog article: With private networks just released on Exoscale, there are now more options to implement secure access to Exoscale cloud infrastructure. While we still recommend the bastion approach, as detailed in this article (https://www.exoscale.ch/syslog/2016/01/15/secure-your-cloud-computing-architecture-with-a-bastion/), there are applications or systems which do not lend themselves well to working this way. In these cases, the next best thing is building IPsec gateways. IPsec is a protocol which works directly at layer 3. It uses its configuration to determine which network flows should be sent encrypted on the wire. Once IPsec is correctly configured, selected network flows are transparently encrypted and applications do not need to modify anything to benefit from secured traffic. In addition to encryption, IPSec also authenticates the end points, so you can be sure you are exchanging packets with a trusted host For the purposes of this article we will work under the following assumptions: We want a host to network setup, providing access to cloud-hosted infrastructure from a desktop environment. Only stock tooling should be used on desktop environment, no additional VPN client should be needed. In this case, to ensure no additional software is needed on the client, we will configure an L2TP/IPsec gateway. This article will use OpenBSD as the operating system to implement the gateway. While this choice may sound surprising, OpenBSD excels at building gateways of all sorts thanks to its simple configuration formats and inclusion of all necessary software and documentation to do so in the base system. The tutorial assumes you have setup a local network between the hosts in the cloud, and walks through the configuration of an OpenBSD host as a IPsec gateway On the OpenBSD host, all necessary software is already installed. We will configure the system, as well as pf, npppd, and ipsec + Configure L2TP + Configure IPsec + Configure NAT + Enabled services: ipsec isakmpd npppd The tutorial then walks through configuring a OS X client, but other desktops will be very similar *** Running Nextcloud in a jail on FreeBSD (https://ramsdenj.com/2017/06/05/nextcloud-in-a-jail-on-freebsd.html) I recently setup Nextcloud 12 inside a FreeBSD jail in order to allow me access to files i might need while at University. I figured this would be a optimal solution for files that I might need access to unexpectedly, on computers where I am not in complete control. My Nextcloud instance is externally accessible, and yet if someone were to get inside my Jail, I could rest easy knowing they still didn't have access to the rest of my host server. I chronicled the setup process including jail setup using iocage, https with Lets Encrypt, and full setup of the web stack. Nextcloud has a variety of features such as calendar synchronization, email, collaborative editing, and even video conferencing. I haven't had time to play with all these different offerings and have only utilized the file synchronization, but even if file sync is not needed, Nextcloud has many offerings that make it worth setting up. MariaDB, PHP 7.0, and Apache 2.4 To manage my jails I'm using iocage. In terms of jail managers it's a fairly new player in the game of jail management and is being very actively developed. It just had a full rewrite in Python, and while the code in the background might be different, the actual user interface has stayed the same. Iocage makes use of ZFS clones in order to create “base jails”, which allow for sharing of one set of system packages between multiple jails, reducing the amount of resources necessary. Alternatively, jails can be completely independent from each other; however, using a base jail makes it easier to update multiple jails as well. + pkg install iocage + sysrc iocageenable=YES + iocage fetch -r 11.0-RELEASE + iocage create tag="stratus" jailzfs=on vnet=off boot=on ip4_addr="sge0|172.20.0.100/32" -r 11.0-RELEASE + iocage start stratus + iocage console stratus I have chosen to provide storage to the Nextcloud Jail by mounting a dataset over NFS on my host box. This means my server can focus on serving Nextcloud and my storage box can focus on housing the data. The Nextcloud Jail is not even aware of this since the NFS Mount is simply mounted by the host server into the jail. The other benefit of this is the Nextcloud jail doesn't need to be able to see my storage server, nor the ability to mount the NFS share itself. Using a separate server for storage isn't necessary and if the storage for my Nextcloud server was being stored on the same server I would have created a ZFS dataset on the host and mounted it into the jail. Next I set up a dataset for the database and delegated it into the jail. Using a separate dataset allows me to specify certain properties that are better for a database, it also makes migration easier in case I ever need to move or backup the database. With most of the requirements in place it was time to start setting up Nextcloud. The requirements for Nextcloud include your basic web stack of a web server, database, and PHP. Also covers the setup of acme.sh for LetsEncrypt. This is now available as a package, and doesn't need to be manually fetched Install a few more packages, and do a bit of configuration, and you have a NextCloud server *** Historical: My first OpenBSD Hackathon (http://bad.network/historical-my-first-openbsd-hackathon.html) This is a blog post by our friend, and OpenBSD developer: Peter Hessler This is a story about encouragement. Every time I use the word "I", you should think "I as in me, not I as in the author". In 2003, I was invited to my first OpenBSD Hackathon. Way before I was into networking, I was porting software to my favourite OS. Specifically, I was porting games. On the first night most of the hackathon attendees end up at the bar for food and beer, and I'm sitting next to Theo de Raadt, the founder of OpenBSD. At some point during the evening, he's telling me about all of these "crazy" ideas he has about randomizing libraries, and protections that can be done in ld.so. (ld.so is the part of the OS that loads the libraries your program needs. It's, uh, kinda important.) Theo is encouraging me to help implement some of these ideas! At some point I tell Theo "I'm just a porter, I don't know C." Theo responds with "It isn't hard, I'll have Dale (Rahn) show you how ld.so works, and you can do it." I was hoping that all of this would be forgotten by the next day, but sure enough Dale comes by. "Hey, are you Peter? Theo wanted me to show you how ld.so works" Dale spends an hour or two showing me how it works, the code structure, and how to recover in case of failure. At first I had lots of failures. Then more failures. And even more failures. Once, I broke my machine so badly I had to reinstall it. I learned a lot about how an OS works during this. But, I eventually started doing changes without it breaking. And some even did what I wanted! By the end of the hackathon I had came up with a useful patch, that was committed as part of a larger change. I was a nobody. With some encouragement, enough liquid courage to override my imposter syndrome, and a few hours of mentoring, I'm now doing big projects. The next time you're sitting at a table with someone new to your field, ask yourself: how can you encourage them? You just might make the world better. Thank you Dale. And thank you Theo. Everyone has to start somewhere. One of the things that sets the BSDs apart from certain other open source operating systems, is the welcoming community, and the tradition of mentorship. Sure, someone else in the OpenBSD project could have done the bits that Peter did, likely a lot more quickly, but then OpenBSD wouldn't have gained a new committer. So, if you are interested in working on one of the BSDs, reach out, and we'll try to help you find a mentor. What part of the system do you want to work on? *** Interview - Dan McDonald - allcoms@gmail.com (mailto:allcoms@gmail.com) (danboid) News Roundup FreeBSD 11.1-RC1 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087340.html) 11.1-RC1 Installation images are available for: amd64, i386 powerpc, powerpc64 sparc64 armv6 BANANAPI, BEAGLEBONE, CUBIEBOARD, CUBIEBOARD2, CUBOX-HUMMINGBOARD, GUMSTIX, RPI-B, RPI2, PANDABOARD, WANDBOARD aarch64 (aka arm64), including the RPI3, Pine64, OverDrive 1000, and Cavium Server A summary of changes since BETA3 includes: Several build toolchain related fixes. A use-after-free in RPC client code has been corrected. The ntpd(8) leap-seconds file has been updated. Various VM subsystem fixes. The '_' character is now allowed in newfs(8) labels. A potential sleep while holding a mutex has been corrected in the sa(4) driver. A memory leak in an ioctl handler has been fixed in the ses(4) driver. Virtual Machine Disk Images are available for the amd64 and i386 architectures. Amazon EC2 AMI Images of FreeBSD/amd64 EC2 AMIs are available The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases. Systems running earlier FreeBSD releases can upgrade as follows: freebsd-update upgrade -r 11.1-RC1 During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly. freebsd-update install The system must be rebooted with the newly installed kernel before continuing. shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: freebsd-update install It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 10.x. Alternatively, the user can install misc/compat10x and other compatibility libraries, afterwards the system must be rebooted into the new userland: shutdown -r now Finally, after rebooting, freebsd-update needs to be run again to remove stale files: freebsd-update install Oil changes, safety recalls, and software patches (http://www.daemonology.net/blog/2017-06-14-oil-changes-safety-recalls-software-patches.html) Every few months I get an email from my local mechanic reminding me that it's time to get my car's oil changed. I generally ignore these emails; it costs time and money to get this done (I'm sure I could do it myself, but the time it would cost is worth more than the money it would save) and I drive little enough — about 2000 km/year — that I'm not too worried about the consequences of going for a bit longer than nominally advised between oil changes. I do get oil changes done... but typically once every 8-12 months, rather than the recommended 4-6 months. From what I've seen, I don't think I'm alone in taking a somewhat lackadaisical approach to routine oil changes. On the other hand, there's another type of notification which elicits more prompt attention: Safety recalls. There are two good reasons for this: First, whether for vehicles, food, or other products, the risk of ignoring a safety recall is not merely that the product will break, but rather that the product will be actively unsafe; and second, when there's a safety recall you don't have to pay for the replacement or fix — the cost is covered by the manufacturer. I started thinking about this distinction — and more specifically the difference in user behaviour — in the aftermath of the "WannaCry" malware. While WannaCry attracted widespread attention for its "ransomware" nature, the more concerning aspect of this incident is how it propagated: By exploiting a vulnerability in SMB for which Microsoft issued patches two months earlier. As someone who works in computer security, I find this horrifying — and I was particularly concerned when I heard that the NHS was postponing surgeries because they couldn't access patient records. Think about it: If the NHS couldn't access patient records due to WannaCry, it suggests WannaCry infiltrated systems used to access patient records — meaning that someone else exploiting the same vulnerabilities could have accessed those records. The SMB subsystem in Windows was not merely broken; until patches were applied, it was actively unsafe. I imagine that most people in my industry would agree that security patches should be treated in the same vein as safety recalls — unless you're certain that you're not affected, take care of them as a matter of urgency — but it seems that far more users instead treat security patches more like oil changes: something to be taken care of when convenient... or not at all, if not convenient. It's easy to say that such users are wrong; but as an industry it's time that we think about why they are wrong rather than merely blaming them for their problems. There are a few factors which I think are major contributors to this problem. First, the number of updates: When critical patches occur frequently enough to become routine, alarm fatigue sets in and people cease to give the attention updates deserve, even if on a conscious level they still recognize the importance of applying updates. Colin also talks about his time as the FreeBSD Security Officer, and the problems in ensuring the patches are correct and do not break the system when installed He also points out the problem of systems like Windows Update, the combines optional updates, and things like its license checking tool, in the same interface that delivers important updates. Or my recent machines, that gets constant popups about how some security updates will not be delivered because my processor is too new. My bank sends me special offers in the mail but phones if my credit card usage trips fraud alarms; this is the sort of distinction in intrusiveness we should see for different types of software updates Finally, I think there is a problem with the mental model most people have of computer security. Movies portray attackers as geniuses who can break into any system in minutes; journalists routinely warn people that "nobody is safe"; and insurance companies offer insurance against "cyberattacks" in much the same way as they offer insurance against tornados. Faced with this wall of misinformation, it's not surprising that people get confused between 400 pound hackers sitting on beds and actual advanced persistent threats. Yes, if the NSA wants to break into your computer, they can probably do it — but most attackers are not the NSA, just like most burglars are not Ethan Hunt. You lock your front door, not because you think it will protect you from the most determined thieves, but because it's an easy step which dramatically reduces your risk from opportunistic attack; but users don't see applying security updates as the equivalent of locking their front door when they leave home. SKIP grep, use AWK (http://blog.jpalardy.com/posts/skip-grep-use-awk/) This is a tip from Jonathan Palardy in a series of blog posts about awk. It is especially helpful for people who write a lot of shell scripts or are using a lot of pipes with awk and grep. Over the years, I've seen many people use this pattern (filter-map): $ [data is generated] | grep something | awk '{print $2}' but it can be shortened to: $ [data is generated] | awk '/something/ {print $2}' AWK can take a regular expression (the part between the slashes) and matches that to the input. Anything that matches is being passed to the print $2 action (to print the second column). Why would I do this? I can think of 4 reasons: *it's shorter to type *it spawns one less process *awk uses modern (read “Perl”) regular expressions, by default – like grep -E *it's ready to “augment” with more awk How about matching the inverse (search for patterns that do NOT match)? But “grep -v” is OK… Many people have pointed out that “grep -v” can be done more concisely with: $ [data is generated] | awk '! /something/' See if you have such combinations of grep piped to awk and fix those in your shell scripts. It saves you one process and makes your scripts much more readable. Also, check out the other intro links on the blog if you are new to awk. *** vim Adventures (https://vim-adventures.com) This website, created by Doron Linder, will playfully teach you how to use vim. Hit any key to get started and follow the instructions on the playing field by moving the cursor around. There is also a menu in the bottom left corner to save your game. Try it out, increase your vim-fu, and learn how to use a powerful text editor more efficiently. *** Beastie Bits Slides from PkgSrcCon (http://pkgsrc.org/pkgsrcCon/2017/talks.html) OpenBSD's doas adds systemd compat shim (http://marc.info/?l=openbsd-tech&m=149902196520920&w=2) Deadlock Empire -- “Each challenge below is a computer program of two or more threads. You take the role of the Scheduler - and a cunning one! Your objective is to exploit flaws in the programs to make them crash or otherwise malfunction.” (https://deadlockempire.github.io/) EuroBSDcon 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/eurobsdcon-2017-travel-grant-application-now-open/) Registration for vBSDCon is open (http://www.vbsdcon.com/) - Registration is only $100 if you register before July 31. Discount hotel rooms arranged at the Hyatt for only $100/night while supplies last. BSD Taiwan call for papers opens, closes July 31st (https://bsdtw.org/)Windows Application Versand *** Feedback/Questions Joseph - Server Monitoring (http://dpaste.com/2AM6C2H#wrap) Paulo - Updating Jails (http://dpaste.com/1Z4FBE2#wrap) Kevin - openvpn server (http://dpaste.com/2MNM9GJ#wrap) Todd - several questions (http://dpaste.com/17BVBJ3#wrap) ***
This week on BSD Now, we review the EuroBSDcon schedule, we explore the mysteries of Docker on OpenBSD, and show you how to run PostgreSQL on ZFS. This episode was brought to you by Headlines EuroBSDcon 2017 - Talks & Schedule published (https://2017.eurobsdcon.org/2017/05/26/talks-schedule-published/) The EuroBSDcon website was updated with the tutorial and talk schedule for the upcoming September conference in Paris, France. Tutorials on the 1st day: Kirk McKusick - An Introduction to the FreeBSD Open-Source Operating System, George Neville-Neil - DTrace for Developers, Taylor R Campbell - How to untangle your threads from a giant lock in a multiprocessor system Tutorials on the 2nd day: Kirk continues his Introduction lecture, Michael Lucas - Core concepts of ZFS (half day), Benedict Reuschling - Managing BSD systems with Ansible (half day), Peter Hessler - BGP for developers and sysadmins Talks include 3 keynotes (2 on the first day, beginning and end), another one at the end of the second day by Brendan Gregg Good mixture of talks of the various BSD projects Also, a good amount of new names and faces Check out the full talk schedule (https://2017.eurobsdcon.org/talks-schedule/). Registration is not open yet, but will be soon. *** OpenBSD on the Xiaomi Mi Air 12.5" (https://jcs.org/2017/05/22/xiaomiair) The Xiaomi Mi Air 12.5" (https://xiaomi-mi.com/notebooks/xiaomi-mi-notebook-air-125-silver/) is a basic fanless 12.5" Ultrabook with good build quality and decent hardware specs, especially for the money: while it can usually be had for about $600, I got mine for $489 shipped to the US during a sale about a month ago. Xiaomi offers this laptop in silver and gold. They also make a 13" version but it comes with an NVidia graphics chip. Since these laptops are only sold in China, they come with a Chinese language version of Windows 10 and only one or two distributors that carry them ship to the US. Unfortunately that also means they come with practically no warranty or support. Hardware > The Mi Air 12.5" has a fanless, 6th generation (Skylake) Intel Core m3 processor, 4Gb of soldered-on RAM, and a 128Gb SATA SSD (more on that later). It has a small footprint of 11.5" wide, 8" deep, and 0.5" thick, and weighs 2.3 pounds. > A single USB-C port on the right-hand side is used to charge the laptop and provide USB connectivity. A USB-C ethernet adapter I tried worked fine in OpenBSD. Whether intentional or not, a particular design touch I appreciated was that the USB-C port is placed directly to the right of the power button on the keyboard, so you don't have to look or feel around for the port when plugging in the power cable. > A single USB 3 type-A port is also available on the right side next to the USB-C port. A full-size HDMI port and a headphone jack are on the left-hand side. It has a soldered-on Intel 8260 wireless adapter and Bluetooth. The webcam in the screen bezel attaches internally over USB. > The chassis is all aluminum and has sufficient rigidity in the keyboard area. The 12.5" 1920x1080 glossy IPS screen has a fairly small bezel and while its hinge is properly weighted to allow opening the lid with one hand (if you care about that kind of thing), the screen does have a bit of top-end wobble when open, especially when typing on another laptop on the same desk. > The keyboard has a roomy layout and a nice clicky tactile with good travel. It is backlit, but with only one backlight level. When enabled via Fn+F10 (which is handled by the EC, so no OpenBSD support required), it will automatically shut off after not typing for a short while, automatically turning back once a key is pressed. Upgrades > An interesting feature of the Mi Air is that it comes with a 128Gb SATA SSD but also includes an open PCI-e slot ready to accept an NVMe SSD. > I upgraded mine with a Samsung PM961 256Gb NVMe SSD (left), and while it is possible to run with both drives in at the same time, I removed the Samsung CM871a 128Gb SATA (right) drive to save power. > The bottom case can be removed by removing the seven visible screws, in addition to the one under the foot in the middle back of the case, which just pries off. A spudger tool is needed to release all of the plastic attachment clips along the entire edge of the bottom cover. > Unfortunately this upgrade proved to be quite time consuming due to the combination of the limited UEFI firmware on the Mi Air and a bug in OpenBSD. A Detour into UEFI Firmware Variables > Unlike a traditional BIOS where one can boot into a menu and configure the boot order as well as enabling and disabling options such as "USB Hard Drive", the InsydeH2O UEFI firmware on the Xiaomi Air only provides the ability to adjust the boot order of existing devices. Any change or addition of boot devices must be done from the operating system, which is not possible under OpenBSD. > I booted to a USB key with OpenBSD on it and manually partitioned the new NVME SSD, then rsynced all of the data over from the old drive, but the laptop would not boot to the new NVME drive, instead showing an error message that there was no bootable OS. > Eventually I figured out that the GPT table that OpenBSD created on the NVMe disk was wrong due to a [one-off bug in the nvme driver](https://github.com/openbsd/src/commit/dc8298f669ea2d7e18c8a8efea509eed200cb989) which was causing the GPT table to be one sector too large, causing the backup GPT table to be written in the wrong location (and other utilities under Linux to write it over the OpenBSD area). I'm guessing the UEFI firmware would fail to read the bad GPT table on the disk that the boot variable pointed to, then declare that disk as missing, and then remove any variables that pointed to that disk. OpenBSD Support > The Mi Air's soldered-on Intel 8260 wireless adapter is supported by OpenBSD's iwm driver, including 802.11n support. The Intel sound chip is recognized by the azalia driver. > The Synaptics touchpad is connected via I2C, but is not yet supported. I am actively hacking on my dwiic driver to make this work and the touchpad will hopefully operate as a Windows Precision Touchpad via imt so I don't have to write an entirely new Synaptics driver. > Unfortunately since OpenBSD's inteldrm support that is ported from Linux is lagging quite a bit behind, there is no kernel support for Skylake and Kaby Lake video chips. Xorg works at 1920x1080 through efifb so the machine is at least usable, but X is not very fast and there is a noticeable delay when doing certain redrawing operations in xterm. Screen backlight can be adjusted through my OpenBSD port of intel_backlight. Since there is no hardware graphics support, this also means that suspend and resume do not work because nothing is available to re-POST the video after resume. Having to use efifb also makes it impossible to adjust the screen gamma, so for me, I can't use redshift for comfortable night-time hacking. Flaws > Especially taking into account the cheap price of the laptop, it's hard to find faults with the design. One minor gripe is that the edges of the case along the bottom are quite sharp, so when carrying the closed laptop, it can feel uncomfortable in one's hands. > While all of those things could be overlooked, unfortunately there is also a critical flaw in the rollover support in the keyboard/EC on the laptop. When typing certain combinations of keys quickly, such as holding Shift and typing "NULL", one's fingers may actually hold down the Shift, N, and U keys at the same time for a very brief moment before releasing N. Normally the keyboard/EC would recognize U being pressed after N is already down and send an interrupt for the U key. Unfortunately on this laptop, particular combinations of three keys do not interrupt for the third key at all until the second key is lifted, usually causing the third key not to register at all if typed quickly. I've been able to reproduce this problem in OpenBSD, Linux, and Windows, with the combinations of at least Shift+N+U and Shift+D+F. Holding Shift and typing the two characters in sequence quickly enough will usually fail to register the final character. Trying the combinations without Shift, using Control or Alt instead of Shift, or other character pairs does not trigger the problem. This might be a problem in the firmware on the Embedded Controller, or a defect in the keyboard circuitry itself. As I mentioned at the beginning, getting technical support for this machine is difficult because it's only sold in China. Docker on OpenBSD 6.1-current (https://medium.com/@dave_voutila/docker-on-openbsd-6-1-current-c620513b8110) Dave Voutila writes: So here's the thing. I'm normally a macOS user…all my hardware was designed in Cupertino, built in China. But I'm restless and have been toying with trying to switch my daily machine over to a non-macOS system sort of just for fun. I find Linux messy, FreeBSD not as Apple-laptop-friendly as it should be, and Windows a non-starter. Luckily, I found a friend in Puffy. Switching some of my Apple machines over to dual-boot OpenBSD left a gaping hole in my workflow. Luckily, all the hard work the OpenBSD team has done over the last year seems to have plugged it nicely! OpenBSD's hypervisor support officially made it into the 6.1 release, but after some experimentation it was rather time consuming and too fragile to get a Linux guest up and running (i.e. basically the per-requisite for Docker). Others had reported some success starting with QEMU and doing lots of tinkering, but after a wasted evening I figured I'd grab the latest OpenBSD snapshot and try what the openbsd-misc list suggested was improved Linux support in active development. 10 (11) Steps to docker are provided Step 0 — Install the latest OpenBSD 6.1 snapshot (-current) Step 1 — Configure VMM/VMD Step 2 — Grab an Alpine Linux ISO Step 3 — Make a new virtual disk image Step 4 — Boot Alpine's ISO Step 5 — Inhale that fresh Alpine air Step 6 — Boot Alpine for Reals Step 7 — Install Docker Step 8 — Make a User Step 9 — Ditch the Serial Console Step 10 — Test out your Docker instance I haven't done it yet, but I plan on installing docker-compose via Python's pip package manager. I prefer defining containers in the compose files. PostgreSQL + ZFS Best Practices and Standard Procedures (https://people.freebsd.org/~seanc/postgresql/scale15x-2017-postgresql_zfs_best_practices.pdf) Slides from Sean Chittenden's talk about PostgreSQL and ZFS at Scale 15x this spring Slides start with a good overview of Postgres and ZFS, and how to use them together To start, it walks through the basics of how PostgreSQL interacts with the filesystem (any filesystem) Then it shows the steps to take a good backup of PostgreSQL, then how to do it even better with ZFS Then an intro to ZFS, and how Copy-on-Write changes host PostgreSQL interacts with the filesystem Overview of how ZFS works ZFS Tuning tips: Compression, Recordsize, atime, when to use mostly ARC vs sharedbuffer, plus pgrepack Followed by a discussion of the reliability of SSDs, and their Bit Error Rate (BER) A good SSD has a 4%/year chance of returning the wrong data. A cheap SSD 34% If you put 20 SSDs in a database server, that means 58% (Good SSDs) to 99.975% (Lowest quality commercially viable SSD) chance of an error per year Luckily, ZFS can detect and correct these errors This applies to all storage, not just SSDs, every device fails More Advice: Use quotas and reservations to avoid running out of space Schedule Periodic Scrubs One dataset per database Backups: Live demo of rm -rf'ing the database and getting it back Using clones to test upgrades on real data Naming Conventions: Use a short prefix not on the root filesystem (e.g. /db) Encode the PostgreSQL major version into the dataset name Give each PostgreSQL cluster its own dataset (e.g. pgdb01) Optional but recommended: one database per cluster Optional but recommended: one app per database Optional but recommended: encode environment into DB name Optional but recommended: encode environment into DB username using ZFS Replication Check out the full detailed PDF and implement a similar setup for your database needs *** News Roundup TrueOS Evolving Its "Stable" Release Cycle (https://www.trueos.org/blog/housekeeping-update-infrastructure-trueos-changes/) TrueOS is reformulating its Stable branch based on feedback from users. The goal is to have a “release” of the stable branch every 6 months, for those who do not want to live on the edge with the rapid updates of the full rolling release Most of the TrueOS developers work for iX Systems in their Tennessee office. Last month, the Tennessee office was moved to a different location across town. As part of the move, we need to move all our servers. We're still getting some of the infrastructure sorted before moving the servers, so please bear with us as we continue this process. As we've continued working on TrueOS, we've heard a significant portion of the community asking for a more stable “STABLE” release of TrueOS, maybe something akin to an old PC-BSD version release. In order to meet that need, we're redefining the TrueOS STABLE branch a bit. STABLE releases are now expected to follow a six month schedule, with more testing and lots of polish between releases. This gives users the option to step back a little from the “cutting edge” of development, but still enjoy many of the benefits of the “rolling release” style and the useful elements of FreeBSD Current. Critical updates like emergency patches and utility bug fixes are still expected to be pushed to STABLE on a case-by-case basis, but again with more testing and polish. This also applies to version updates of the Lumina and SysAdm projects. New, released work from those projects will be tested and added to STABLE outside the 6 month window as well. The UNSTABLE branch continues to be our experimental “cutting edge” track, and users who want to follow along with our development and help us or FreeBSD test new features are still encouraged to follow the UNSTABLE track by checking that setting in their TrueOS Update Manager. With boot environments, it will be easy to switch back and forth, so you can have the best of both worlds. Use the latest bleeding edge features, but knowing you can fall back to the stable branch with just a reboot As TrueOS evolves, it is becoming clearer that one role of the system is to function as a “test platform” for FreeBSD. In order to better serve this role, TrueOS will support both OpenRC and the FreeBSD RC init systems, giving users the choice to use either system. While the full functionality isn't quite ready for the next STABLE update, it is planned for addition after the last bit of work and testing is complete. Stay tuned for an upcoming blog post with all the details of this change, along with instructions how to switch between RC and OpenRC. This is the most important change for me. I used TrueOS as an easy way to run the latest version of -CURRENT on my laptop, to use it as a user, but also to do development. When TrueOS deviates from FreeBSD too much, it lessens the power of my expertise, and complicates development and debugging. Being able to switch back to RC, even if it takes another minute to boot, will bring TrueOS back to being FreeBSD + GUI and more by default, instead of a science project. We need both of those things, so having the option, while more work for the TrueOS team, I think will be better for the entire community *** Logical Domains on SunFire T2000 with OpenBSD/sparc64 (http://www.h-i-r.net/2017/05/logical-domains-on-sunfire-t2000-with.html) A couple of years ago, I picked up a Sun Fire T2000. This is a 2U rack mount server. Mine came with four 146GB SAS drives, a 32-core UltraSPARC T1 CPU and 32GB of RAM. Sun Microsystems incorporated Logical Domains (LDOMs) on this class of hardware. You don't often need 32 threads and 32GB of RAM in a single server. LDOMs are a kind of virtualization technology that's a bit closer to bare metal than vmm, Hyper-V, VirtualBox or even Xen. It works a bit like Xen, though. You can allocate processor, memory, storage and other resources to virtual servers on-board, with a blend of firmware that supports the hardware allocation, and some software in userland (on the so-called primary or control domain, similar to Xen DomU) to control it. LDOMs are similar to what IBM calls Logical Partitions (LPARs) on its Mainframe and POWER series computers. My day job from 2006-2010 involved working with both of these virtualization technologies, and I've kind of missed it. While upgrading OpenBSD to 6.1 on my T2000, I decided to delve into LDOM support under OpenBSD. This was pretty easy to do, but let's walk through it Resources: The ldomctl(8) man page (http://man.openbsd.org/OpenBSD-current/man8/sparc64/ldomctl.8) tedu@'s write-up on Flak (for a different class of server) (http://www.tedunangst.com/flak/post/OpenBSD-on-a-Sun-T5120) A Google+ post by bmercer@ (https://plus.google.com/101694200911870273983/posts/jWh4rMKVq97) Once you get comfortable with the fact that there's a little-tiny computer (the ALOM) powered by VXWorks inside that's acting as the management system and console (there's no screen or keyboard/mouse input), Installing OpenBSD on the base server is pretty straightforward. The serial console is an RJ-45 jack, and, yes, the ubiquitous blue-colored serial console cables you find for certain kinds of popular routers will work fine. OpenBSD installs quite easily, with the same installer you find on amd64 and i386. I chose to install to /dev/sd0, the first SAS drive only, leaving the others unused. It's possible to set them up in a hardware RAID configuration using tools available only under Solaris, or use softraid(4) on OpenBSD, but I didn't do this. I set up the primary LDOM to use the first ethernet port, em0. I decided I wanted to bridge the logical domains to the second ethernet port. You could also use a bridge and vether interface, with pf and dhcpd to create a NAT environment, similar to how I networked the vmm(4) systems. Create an LDOM configuration file. You can put this anywhere that's convenient. All of this stuff was in a "vm" subdirectory of my home. I called it ldom.conf: domain primary { vcpu 8 memory 8G } domain puffy { vcpu 8 memory 4G vdisk "/home/axon/vm/ldom1" vnet } Make as many disk images as you want, and make as many additional domain clauses as you wish. Be mindful of system resources. I couldn't actually allocate a full 32GB of RAM across all the LDOMs I eventually provisioned seven LDOMs (in addition to the primary) on the T2000, each with 3GB of RAM and 4 vcpu cores. If you get creative with use of network interfaces, virtual ethernet, bridges and pf rules, you can run a pretty complex environment on a single chassis, with services that are only exposed to other VMs, a DMZ segment, and the internal LAN. A nice tutorial, and an interesting look at an alternative platform that was ahead of its time *** documentation is thoroughly hard (http://www.tedunangst.com/flak/post/documentation-is-thoroughly-hard) Ted Unangst has a new post this week about documentation: Documentation is good, so therefore more documentation must be better, right? A few examples where things may have gotten out of control A fine example is the old OpenBSD install instructions. Once you've installed OpenBSD once or twice, the process is quite simple, but you'd never know this based on reading the instructions. Compare the files for 4.8 INSTALL and 5.8 INSTALL. Both begin with a brief intro to the project. Then 4.8 has an enormous list of mirrors, which seems fairly redundant if you've already found the install file. Followed by an enormous list of every supported variant of every supported device. Including a table of IO port configurations for ISA devices. Finally, after 1600 lines of introduction we get to the actual installation instructions. (Compared to line 231 for 5.8.) This includes a full page of text about how to install from tape, which nobody ever does. It took some time to recognize that all this documentation was actually an impediment to new users. Attempting to answer every possible question floods the reader with information for questions they were never planning to ask. Part of the problem is how the information is organized. Theoretically it makes sense to list supported hardware before instructions. After all, you can't install anything if it's not supported, right? I'm sure that was considered when the device list was originally inserted above the install instructions. But as a practical matter, consulting a device list is neither the easiest nor fastest way to determine what actually works. In the FreeBSD docs tree, we have been doing a facelift project, trying to add ‘quick start' sections to each chapter to let you get to the more important information first. It is also helpful to move data in the forms of lists and tables to appendices or similar, where they can easily be references, but are not blocking your way to the information you are actually hunting for An example of nerdview signage (http://languagelog.ldc.upenn.edu/nll/?p=29866). “They have in effect provided a sign that will tell you exactly what the question is provided you can already supply the answer.” That is, the logical minds of technical people often decide to order information in an order that makes sense to them, rather than in the order that will be most useful to the reader In the end, I think “copy diskimage to USB and follow prompts” is all the instructions one should need, but it's hard to overcome the unease of actually making the jump. What if somebody is confused or uncertain? Why is this paragraph more redundant than that paragraph? (And if we delete both, are we cutting too much?) Sometimes we don't need to delete the information. Just hide it. The instructions to upgrade to 4.8 and upgrade to 5.8 are very similar, with a few differences because every release is a little bit different. The pages look very different, however, because the not at all recommended kernel free procedure, which takes up half the page, has been hidden from view behind some javascript and only expanded on demand. A casual browser will find the page and figure the upgrade process will be easy, as opposed to some long ordeal. This is important as well, it was my original motivation for working on the FreeBSD Handbook's ZFS chapter. The very first section of the chapter was the custom kernel configuration required to run ZFS on i386. That scared many users away. I moved that to the very end, and started with why you might want to use ZFS. Much more approachable. Sometimes it's just a tiny detail that's overspecified. The apmd manual used to explain exactly which CPU idle time thresholds were used to adjust frequency. Those parameters, and the algorithm itself, were adjusted occasionally in response to user feedback, but sometimes the man page lagged behind. The numbers are of no use to a user. They're not adjustable without recompiling. Knowing that the frequency would be reduced at 85% idle vs 90% idle doesn't really offer much guidance as to whether to enable auto scaling or not. Deleting this detail ensured the man page was always correct and spares the user the cognitive load of trying to solve an unnecessary math problem. For fun: For another humorous example, it was recently observed that the deja-dup package provides man page translations for Australia, Canada, and Great Britain. I checked, the pages are in fact not quite identical. Some contain typo fixes that didn't propagate to other translations. Project idea: attempt to identify which country has the most users, or most fastidious users, by bug fixes to localized man pages. lldb on BeagleBone Black (https://lists.freebsd.org/pipermail/freebsd-arm/2017-May/016260.html) I reliably managed to build (lldb + clang/lld) from the svn trunk of LLVM 5.0.0 on my Beaglebone Black running the latest snapshot (May 20th) of FreeBSD 12.0-CURRENT, and the lldb is working very well, and this includes single stepping and ncurses-GUI mode, while single stepping with the latest lldb 4.0.1 from the ports does not work. In order to reliably build LLVM 5.0.0 (svn), I set up a 1 GB swap partition for the BBB on a NFSv4 share on a FreeBSD fileserver in my network - I put a howto of the procedure on my BLog: https://obsigna.net/?p=659 The prerequesites on the Beaglebone are: ``` pkg install tmux pkg install cmake pkg install python pkg install libxml2 pkg install swig30 pkg install ninja pkg install subversion ``` On the FreeBSD fileserver: ``` /pathtothe/bbb_share svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm cd llvm/tools svn co http://llvm.org/svn/llvm-project/cfe/trunk clang svn co http://llvm.org/svn/llvm-project/lld/trunk lld svn co http://llvm.org/svn/llvm-project/lldb/trunk lldb ``` + On the Beaglebone Black: # mount_nfs -o noatime,readahead=4,intr,soft,nfsv4 server:/path_to_the/bbb_share /mnt # cd /mnt # mkdir build # cmake -DLLVM_TARGETS_TO_BUILD="ARM" -DCMAKE_BUILD_TYPE="MinSizeRel" -DLLVM_PARALLEL_COMPILE_JOBS="1" -DLLVM_PARALLEL_LINK_JOBS="1" -G Ninja .. I execute the actual build command from within a tmux session, so I may disconnect during the quite long (40 h) build: ``` tmux new "ninja lldb install" ``` When debugging in GUI mode using the newly build lldb 5.0.0-svn, I see only a minor issue, namely UTF8 strings are not displayed correctly. This happens in the ncurses-GUI only, and this is an ARM issue, since it does not occur on x86 machines. Perhaps this might be related to the signed/unsigned char mismatch between ARM and x86. Beastie Bits Triangle BSD Meetup on June 27th (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) Support for Controller Area Networks (CAN) in NetBSD (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20170521_0113.html) Notes from Monday's meeting (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2017-May/014104.html) RunBSD - A site about the BSD family of operating systems (http://runbsd.info/) BSDCam(bridge) 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/bsdcam-2017-travel-grant-application-now-open/) New BSDMag has been released (https://bsdmag.org/download/nearly-online-zpool-switching-two-freebsd-machines/) *** Feedback/Questions Philipp - A show about byhve (http://dpaste.com/390F9JN#wrap) Jake - byhve Support on AMD (http://dpaste.com/0DYG5BD#wrap) CY - Pledge and Capsicum (http://dpaste.com/1YVBT12#wrap) CY - OpenSSL relicense Issue (http://dpaste.com/3RSYV23#wrap) Andy - Laptops (http://dpaste.com/0MM09EX#wrap) ***
This week on the show we interview Wendell from Level1Techs, cover Google Summer of Code on the different BSD projects, cover YubiKey usage, dive into how NICs work & This episode was brought to you by Headlines Google summer of code for BSDs FreeBSD (https://www.freebsd.org/projects/summerofcode.html) FreeBSD's existing list of GSoC Ideas for potential students (https://wiki.freebsd.org/SummerOfCodeIdeas) FreeBSD/Xen: import the grant-table bus_dma(9) handlers from OpenBSD Add support for usbdump file-format to wireshark and vusb-analyzer Write a new boot environment manager Basic smoke test of all base utilities Port OpenBSD's pf testing framework and tests Userspace Address Space Annotation zstandard integration in libstand Replace mergesort implementation Test Kload (kexec for FreeBSD) Kernel fuzzing suite Integrate MFSBSD into the release building tools NVMe controller emulation for bhyve Verification of bhyve's instruction emulation VGA emulation improvements for bhyve audit framework test suite Add more FreeBSD testing to Xen osstest Lua in bootloader POSIX compliance testing framework coreclr: add Microsoft's coreclr and corefx to the Ports tree. NetBSD (https://wiki.netbsd.org/projects/gsoc/) Kernel-level projects Medium ISDN NT support and Asterisk integration LED/LCD Generic API NetBSD/azure -- Bringing NetBSD to Microsoft Azure OpenCrypto swcrypto(4) enhancements Scalable entropy gathering Userland PCI drivers Hard Real asynchronous I/O Parallelize page queues Tickless NetBSD with high-resolution timers Userland projects Easy Inetd enhancements -- Add new features to inetd Curses library automated testing Medium Make Anita support additional virtual machine systems Create an SQL backend and statistics/query page for ATF test results Light weight precision user level time reading Query optimizer for find(1) Port launchd Secure-PLT - supporting RELRO binaries Sysinst alternative interface Hard Verification tool for NetBSD32 pkgsrc projects Easy Version control config files Spawn support in pkgsrc tools Authentication server meta-package Medium pkgin improvements Unify standard installation tasks Hard Add dependency information to binary packages Tool to find dependencies precisely LLVM (http://llvm.org/OpenProjects.html#gsoc17) Fuzzing the Bitcode reader Description of the project: The optimizer is 25-30% slower when debug info are enabled, it'd be nice to track all the places where we don't do a good job about ignoring them! Extend clang AST to provide information for the type as written in template instantiations. Description of the project: When instantiating a template, the template arguments are canonicalized before being substituted into the template pattern. Clang does not preserve type sugar when subsequently accessing members of the instantiation. Clang should "re-sugar" the type when performing member access on a class template specialization, based on the type sugar of the accessed specialization. Shell auto-completion support for clang. Bash and other shells support typing a partial command and then automatically completing it for the user (or at least providing suggestions how to complete) when pressing the tab key. This is usually only supported for popular programs such as package managers (e.g. pressing tab after typing "apt-get install late" queries the APT package database and lists all packages that start with "late"). As of now clang's frontend isn't supported by any common shell. Clang-based C/C++ diff tool. Description of the project: Every developer has to interact with diff tools daily. The algorithms are usually based on detecting "longest common subsequences", which is agnostic to the file type content. A tool that would understand the structure of the code may provide a better diff experience by being robust against, for example, clang-format changes. Find dereference of pointers. Description of the project: Find dereference of pointer before checking for nullptr. Warn if virtual calls are made from constructors or destructors. Description of the project: Implement a path-sensitive checker that warns if virtual calls are made from constructors and destructors, which is not valid in case of pure virtual calls and could be a sign of user error in non-pure calls. Improve Code Layout Description of the project: The goal for the project is trying to improve the layout/performances of the generated executable. The primary object format considered for the project is ELF but this can be extended to other object formats. The project will touch both LLVM and lld. Why Isn't OpenBSD in Google Summer of Code 2017? (http://marc.info/?l=openbsd-misc&m=149119308705465&w=2) Hacker News Discussion Thread (https://news.ycombinator.com/item?id=14020814) Turtles on the Wire: Understanding How the OS Uses the Modern NIC (http://dtrace.org/blogs/rm/2016/09/15/turtles-on-the-wire-understanding-how-the-os-uses-the-modern-nic/) The Simple NIC MAC Address Filters and Promiscuous Mode Problem: The Single Busy CPU A Swing and a Miss Nine Rings for Packets Doomed to be Hashed Problem: Density, Density, Density A Brief Aside: The Virtual NIC Always Promiscuous? The Classification Challenge Problem: CPUs are too ‘slow' Problem: The Interrupts are Coming in too Hot Solution One: Do Less Work Solution Two: Turn Off Interrupts Recapping Future Directions and More Reading Make Dragonfly BSD great again! (http://akat1.pl/?id=3) Recently I spent some time reading Dragonfly BSD code. While doing so I spotted a vulnerability in the sysvsem subsystem that let user to point to any piece of memory and write data through it (including the kernel space). This can be turned into execution of arbitrary code in the kernel context and by exploiting this, we're gonna make Dragonfly BSD great again! Dragonfly BSD is a BSD system which originally comes from the FreeBSD project. In 2003 Matthew Dillon forked code from the 4.x branch of the FreeBSD and started a new flavour. I thought of Dragonfly BSD as just another fork, but during EuroBSDCon 2015 I accidentally saw the talk about graphical stack in the Dragonfly BSD. I confused rooms, but it was too late to escape as I was sitting in the middle of a row, and the exit seemed light years away from me. :-) Anyway, this talk was a sign to me that it's not just a niche of a niche of a niche of a niche operating system. I recommend spending a few minutes of your precious time to check out the HAMMER file system, Dragonfly's approach to MP, process snapshots and other cool features that it offers. Wikipedia article is a good starter With the exploit, they are able to change the name of the operating system back to FreeBSD, and escalate from an unprivileged user to root. The Bug itself is located in the semctl(2) system call implementation. bcopy(3) in line 385 copies semid_ds structure to memory pointed by arg->buf, this pointer is fully controlled by the user, as it's one of the syscall's arguments. So the bad thing here is that we can copy things to arbitrary address, but we have not idea what we copy yet. This code was introduced by wrongly merging code from the FreeBSD project, bah, bug happens. Using this access, the example code shows how to overwrite the function pointers in the kernel used for the open() syscall, and how to overwrite the ostype global, changing the name of the operating system. In the second example, the reference to the credentials of the user trying to open a file are used to overwrite that data, making the user root. The bug was fixed in uber fast manner (within few hours!) by Matthew Dillon, version 4.6.1 released shortly after that seems to be safe. In case you care, you know what to do! Thanks to Mateusz Kocielski for the detailed post, and finding the bug *** Interview - Wendell - wendell@level1techs.com (mailto:wendell@level1techs.com) / @tekwendell (https://twitter.com/tekwendell) Host of Level1Techs website, podcast and YouTube channel News Roundup Using yubikeys everywhere (http://www.tedunangst.com/flak/post/using-yubikeys-everywhere) Ted Unangst is back, with an interesting post about YUBI Keys Everybody is getting real excited about yubikeys recently, so I figured I should get excited, too. I have so far resisted two factor authorizing everything, but this seemed like another fun experiment. There's a lot written about yubikeys and how you should use one, but nothing I've read answered a few of the specific questions I had To begin with, I ordered two yubikeys. One regular sized 4 and one nano. I wanted to play with different form factors to see which is better for various uses, and I wanted to test having a key and a backup key. Everybody always talks about having one yubikey. And then if you lose it, terrible things happen. Can this problem be alleviated with two keys? I'm also very curious what happens when I try to login to a service with my phone after enabling U2F. We've got three computers (and operating systems) in the mix, along with a number of (mostly web) services. Wherever possible, I want to use a yubikey both to login to the computer and to authorize myself to remote services. I started my adventure on my chromebook. Ultimate goal would be to use the yubikey for local logins. Either as a second factor, or as an alternative factor. First things first and we need to get the yubikey into the account I use to sign into the chromebook. Alas, there is apparently no way to enroll only a security key for a Google account. Every time I tried, it would ask me for my phone number. That is not what I want. Zero stars. Giving up on protecting the chromebook itself, at least maybe I can use it to enable U2F with some other sites. U2F is currently limited to Chrome, but it sounds like everything I want. Facebook signup using U2F was pretty easy. Go to account settings, security subheading, add the device. Tap the button when it glows. Key added. Note that it's possible to add a key without actually enabling two factor auth, in which case you can still login with only a password, but no way to login with no password and only a USB key. Logged out to confirm it would check the key, and everything looked good, so I killed all my other active sessions. Now for the phone test. Not quite as smooth. Tried to login, the Facebook app then tells me it has sent me an SMS and to enter the code in the box. But I don't have a phone number attached. I'm not getting an SMS code. Meanwhile, on my laptop, I have a new notification about a login attempt. Follow the prompts to confirm it's me and permit the login. This doesn't have any effect on the phone, however. I have to tap back, return to the login screen, and enter my password again. This time the login succeeds. So everything works, but there are still some rough patches in the flow. Ideally, the phone would more accurately tell me to visit the desktop site, and then automatically proceed after I approve. (The messenger app crashed after telling me my session had expired, but upon restarting it was able to borrow the Facebook app credentials and I was immediately logged back in.) Let's configure Dropbox next. Dropbox won't let you add a security key to an account until after you've already set up some other mobile authenticator. I already had the Duo app on my phone, so I picked that, and after a short QR scan, I'm ready to add the yubikey. So the key works to access Dropbox via Chrome. Accessing Dropbox via my phone or Firefox requires entering a six digit code. No way to use a yubikey in a three legged configuration I don't use Github, but I know they support two factors, so let's try them next. Very similar to Dropbox. In order to set up a key, I must first set up an authenticator app. This time I went with Yubico's own desktop authenticator. Instead of scanning the QR code, type in some giant number (on my Windows laptop), and it spits out an endless series of six digit numbers, but only while the yubikey is inserted. I guess this is kind of what I want, although a three pound yubikey is kind of unwieldy. As part of my experiment, I noticed that Dropbox verifies passwords before even looking at the second auth. I have a feeling that they should be checked at the same time. No sense allowing my password guessing attack to proceed while I plot how to steal someone's yubikey. In a sense, the yubikey should serve as a salt, preventing me from mounting such an attack until I have it, thus creating a race where the victim notices the key is gone and revokes access before I learn the password. If I know the password, the instant I grab the key I get access. Along similar lines, I was able to complete a password reset without entering any kind of secondary code. Having my phone turn into a second factor is a big part of what I'm looking to avoid with the yubikey. I'd like to be able to take my phone with me, logged into some sites but not all, and unable to login to the rest. All these sites that require using my phone as mobile authenticator are making that difficult. I bought the yubikey because it was cheaper than buying another phone! Using the Yubico desktop authenticator seems the best way around that. The article also provides instructions for configuring the Yubikey on OpenBSD A few notes about OTP. As mentioned, the secret key is the real password. It's stored on whatever laptop or server you login to. Meaning any of those machines can take the key and use it to login to any other machine. If you use the same yubikey to login to both your laptop and a remote server, your stolen laptop can trivially be used to login to the server without the key. Be mindful of that when setting up multiple machines. Also, the OTP counter isn't synced between machines in this setup, which allows limited replay attacks. Ted didn't switch his SSH keys to the Yubikey, because it doesn't support ED25519, and he just finished rotating all of his keys and doesn't want to do it again. I did most of my experimenting with the larger yubikey, since it was easier to move between machines. For operations involving logging into a web site, however, I'd prefer the nano. It's very small, even smaller than the tiniest wireless mouse transcievers I've seen. So small, in fact, I had trouble removing it because I couldn't find anything small enough to fit through the tiny loop. But probably a good thing. Most other micro USB gadgets stick out just enough to snag when pushing a laptop into a bag. Not the nano. You lose a port, but there's really no reason to ever take it out. Just leave it in, and then tap it whenever you login to the tubes. It would not be a good choice for authenticating to the local machine, however. The larger device, sized to fit on a keychain, is much better for that. It is possible to use two keys as backups. Facebook and Dropbox allow adding two U2F keys. This is perhaps a little tiresome if there's lots of sites, as I see no way to clone a key. You have to login to every service. For challenge response and OTP, however, the personalization tool makes it easy to generate lots of yubikeys with the same secrets. On the other hand, a single device supports an infinite number of U2F sites. The programmable interfaces like OTP are limited to only two slots, and the first is already used by the factory OTP setup. What happened to my vlan (http://www.grenadille.net/post/2017/02/13/What-happened-to-my-vlan) A long term goal of the effort I'm driving to unlock OpenBSD's Network Stack is obviously to increase performances. So I'd understand that you find confusing when some of our changes introduce performance regressions. It is just really hard to do incremental changes without introducing temporary regressions. But as much as security is a process, improving performance is also a process. Recently markus@ told me that vlan(4) performances dropped in last releases. He had some ideas why but he couldn't provide evidences. So what really happened? Hrvoje Popovski was kind enough to help me with some tests. He first confirmed that on his Xeon box (E5-2643 v2 @ 3.50GHz), forwarding performances without pf(4) dropped from 1.42Mpps to 880Kpps when using vlan(4) on both interfaces. Together vlaninput() and vlanstart() represent 25% of the time CPU1 spends processing packets. This is not exactly between 33% and 50% but it is close enough. The assumption we made earlier is certainly too simple. If we compare the amount of work done in process context, represented by ifinputprocess() we clearly see that half of the CPU time is not spent in etherinput(). I'm not sure how this is related to the measured performance drop. It is actually hard to tell since packets are currently being processed in 3 different contexts. One of the arguments mikeb@ raised when we discussed moving everything in a single context, is that it is simpler to analyse and hopefully make it scale. With some measurements, a couple of nice pictures, a bit of analysis and some educated guesses we are now in measure of saying that the performances impact observed with vlan(4) is certainly due to the pseudo-driver itself. A decrease of 30% to 50% is not what I would expect from such pseudo-driver. I originally heard that the reason for this regression was the use of SRP but by looking at the profiling data it seems to me that the queuing API is the problem. In the graph above the CPU time spent in ifinput() and ifenqueue() from vlan(4) is impressive. Remember, in the case of vlan(4) these operations are done per packet! When ifinput() has been introduced the queuing API did not exist and putting/taking a single packet on/from an interface queue was cheap. Now it requires a mutex per operation, which in the case of packets received and sent on vlan(4) means grabbing three mutexes per packets. I still can't say if my analysis is correct or not, but at least it could explain the decrease observed by Hrvoje when testing multiple vlan(4) configurations. vlaninput() takes one mutex per packet, so it decreases the number of forwarded packets by ~100Kpps on this machine, while vlanstart() taking two mutexes decreases it by ~200Kpps. An interesting analysis of the routing performance regression on OpenBSD I have asked Olivier Cochard-Labbe about doing a similar comparison of routing performance on FreeBSD when a vlan pseudo interface is added to the forwarding path *** NetBSD: the first BSD introducing a modern process plugin framework in LLDB (https://blog.netbsd.org/tnf/entry/netbsd_the_first_bsd_introducing) Clean up in ptrace(2) ATF tests We have created some maintanance burden for the current ptrace(2) regression tests. The main issues with them is code duplication and the splitting between generic (Machine Independent) and port-specific (Machine Dependent) test files. I've eliminated some of the ballast and merged tests into the appropriate directory tests/lib/libc/sys/. The old location (tests/kernel) was a violation of the tests/README recommendation PTRACE_FORK on !x86 ports Along with the motivation from Martin Husemann we have investigated the issue with PTRACE_FORK ATF regression tests. It was discovered that these tests aren't functional on evbarm, alpha, shark, sparc and sparc64 and likely on other non-x86 ports. We have discovered that there is a missing SIGTRAP emitted from the child, during the fork(2) handshake. The proper order of operations is as follows: parent emits SIGTRAP with sicode=TRAPCHLD and pesetevent=pid of forkee child emits SIGTRAP with sicode=TRAPCHLD and pesetevent=pid of forker Only the x86 ports were emitting the second SIGTRAP signal. PTSYSCALL and PTSYSCALLEMU With the addition of PTSYSCALLEMU we can implement a virtual kernel syscall monitor. It means that we can fake syscalls within a debugger. In order to achieve this feature, we need to use the PTSYSCALL operation, catch SIGTRAP with sicode=TRAPSCE (syscall entry), call PTSYSCALLEMU and perform an emulated userspace syscall that would have been done by the kernel, followed by calling another PTSYSCALL with sicode=TRAPSCX. What has been done in LLDB A lot of work has been done with the goal to get breakpoints functional. This target penetrated bugs in the existing local patches and unveiled missing features required to be added. My initial test was tracing a dummy hello-world application in C. I have sniffed the GDB Remote Protocol packets and compared them between Linux and NetBSD. This helped to streamline both versions and bring the NetBSD support to the required Linux level. Plan for the next milestone I've listed the following goals for the next milestone. watchpoints support floating point registers support enhance core(5) and make it work for multiple threads introduce PTSETSTEP and PTCLEARSTEP in ptrace(2) support threads in the NetBSD Process Plugin research F_GETPATH in fcntl(2) Beyond the next milestone is x86 32-bit support. LibreSSL 2.5.2 released (https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.5.2-relnotes.txt) Added the recallocarray(3) memory allocation function, and converted various places in the library to use it, such as CBB and BUFMEMgrow. recallocarray(3) is similar to reallocarray. Newly allocated memory is cleared similar to calloc(3). Memory that becomes unallocated while shrinking or moving existing allocations is explicitly discarded by unmapping or clearing to 0. Added new root CAs from SECOM Trust Systems / Security Communication of Japan. Added EVP interface for MD5+SHA1 hashes. Fixed DTLS client failures when the server sends a certificate request. Correct handling of padding when upgrading an SSLv2 challenge into an SSLv3/TLS connection. Allow protocols and ciphers to be set on a TLS config object in libtls. Improved nc(1) TLS handshake CPU usage and server-side error reporting. Beastie Bits HardenedBSD Stable v46.16 released (http://hardenedbsd.org/article/op/2017-03-30/stable-release-hardenedbsd-stable-11-stable-v4616) KnoxBUG looking for OpenBSD people in Knoxville TN area (https://www.reddit.com/r/openbsd/comments/5vggn7/knoxbug_looking_for_openbsd_people_in_knoxville/) KnoxBUG Tuesday, April 18, 2017 - 6:00pm : Caleb Cooper: Advanced BASH Scripting](http://knoxbug.org/2017-04-18) e2k17 Nano hackathon report from Bob Beck (http://undeadly.org/cgi?action=article&sid=20170405110059) Noah Chelliah, Host of the Linux Action Show calls Linux a ‘Bad Science Project' and ditches Linux for TrueOS](https://youtu.be/yXB85_olYhQ?t=3238) *** Feedback/Questions James - ZFS Mounting (http://dpaste.com/1H43JGV#wrap) Kevin - Virtualization (http://dpaste.com/18VNAJK#wrap) Ben - Jails (http://dpaste.com/0R7CRZ7#wrap) Florian - ZFS and Migrating Linux userlands (http://dpaste.com/2Z1P23T#wrap) q5sys - question for the community (http://dpaste.com/26M453F#wrap)
This week on BSDNow, reports from AsiaBSDcon, TrueOS and FreeBSD news, Optimizing IllumOS Kernel, your questions and more. This episode was brought to you by Headlines AsiaBSDcon Reports and Reviews () AsiaBSDcon schedule (https://2017.asiabsdcon.org/program.html.en) Schedule and slides from the 4th bhyvecon (http://bhyvecon.org/) Michael Dexter's trip report on the iXsystems blog (https://www.ixsystems.com/blog/ixsystems-attends-asiabsdcon-2017) NetBSD AsiaBSDcon booth report (http://mail-index.netbsd.org/netbsd-advocacy/2017/03/13/msg000729.html) *** TrueOS Community Guidelines are here! (https://www.trueos.org/blog/trueos-community-guidelines/) TrueOS has published its new Community Guidelines The TrueOS Project has existed for over ten years. Until now, there was no formally defined process for interested individuals in the TrueOS community to earn contributor status as an active committer to this long-standing project. The current core TrueOS developers (Kris Moore, Ken Moore, and Joe Maloney) want to provide the community more opportunities to directly impact the TrueOS Project, and wish to formalize the process for interested people to gain full commit access to the TrueOS repositories. These describe what is expected of community members and committers They also describe the process of getting commit access to the TrueOS repo: Previously, Kris directly handed out commit bits. Now, the Core developers have provided a small list of requirements for gaining a TrueOS commit bit: Create five or more pull requests in a TrueOS Project repository within a single six month period. Stay active in the TrueOS community through at least one of the available community channels (Gitter, Discourse, IRC, etc.). Request commit access from the core developers via core@trueos.org OR Core developers contact you concerning commit access. Pull requests can be any contribution to the project, from minor documentation tweaks to creating full utilities. At the end of every month, the core developers review the commit logs, removing elements that break the Project or deviate too far from its intended purpose. Additionally, outstanding pull requests with no active dissension are immediately merged, if possible. For example, a user submits a pull request which adds a little-used OpenRC script. No one from the community comments on the request or otherwise argues against its inclusion, resulting in an automatic merge at the end of the month. In this manner, solid contributions are routinely added to the project and never left in a state of “limbo”. The page also describes the perks of being a TrueOS committer: Contributors to the TrueOS Project enjoy a number of benefits, including: A personal TrueOS email alias: @trueos.org Full access for managing TrueOS issues on GitHub. Regular meetings with the core developers and other contributors. Access to private chat channels with the core developers. Recognition as part of an online Who's Who of TrueOS developers. The eternal gratitude of the core developers of TrueOS. A warm, fuzzy feeling. Intel Donates 250.000 $ to the FreeBSD Foundation (https://www.freebsdfoundation.org/news-and-events/latest-news/new-uranium-level-donation-and-collaborative-partnership-with-intel/) More details about the deal: Systems Thinking: Intel and the FreeBSD Project (https://www.freebsdfoundation.org/blog/systems-thinking-intel-and-the-freebsd-project/) Intel will be more actively engaging with the FreeBSD Foundation and the FreeBSD Project to deliver more timely support for Intel products and technologies in FreeBSD. Intel has contributed code to FreeBSD for individual device drivers (i.e. NICs) in the past, but is now seeking a more holistic “systems thinking” approach. Intel Blog Post (https://01.org/blogs/imad/2017/intel-increases-support-freebsd-project) We will work closely with the FreeBSD Foundation to ensure the drivers, tools, and applications needed on Intel® SSD-based storage appliances are available to the community. This collaboration will also provide timely support for future Intel® 3D XPoint™ products. Thank you very much, Intel! *** Applied FreeBSD: Basic iSCSI (https://globalengineer.wordpress.com/2017/03/05/applied-freebsd-basic-iscsi/) iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered. The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it. Then, enable ctld and start it: sysrc ctld_enable=”YES” service ctld start You can use the ctladm command to see what is going on: root@bsdtarget:/dev # ctladm lunlist (7:0:0/0): Fixed Direct Access SPC-4 SCSI device (7:0:1/1): Fixed Direct Access SPC-4 SCSI device root@bsdtarget:/dev # ctladm devlist LUN Backend Size (Blocks) BS Serial Number Device ID 0 block 10485760 512 MYSERIAL 0 MYDEVID 0 1 block 10485760 512 MYSERIAL 1 MYDEVID 1 Now, let's configure the client side: In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. sysrc iscsid_enable=”YES” service iscsid start Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session: + iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget You should now have a new device (check dmesg), in this case, da1 The guide them walks through partitioning the disk, and laying down a UFS file system, and mounting it This it walks through how to disconnect iscsi, incase you don't want it anymore This all looked nice and easy, and it works very well. Now lets see what happens when you try to mount the iSCSI from Windows Ok, that wasn't so bad. Now, instead of sharing an entire space disk on the host via iSCSI, share a zvol. Now your windows machine can be backed by ZFS. All of your problems are solved. Interview - Philipp Buehler - pbuehler@sysfive.com (mailto:pbuehler@sysfive.com) Technical Lead at SysFive, and Former OpenBSD Committer News Roundup Half a dozen new features in mandoc -T html (http://undeadly.org/cgi?action=article&sid=20170316080827) mandoc (http://man.openbsd.org/mandoc.1)'s HTML output mode got some new features Even though mdoc(7) is a semantic markup language, traditionally none of the semantic annotations were communicated to the reader. [...] Now, at least in -T html output mode, you can see the semantic function of marked-up words by hovering your mouse over them. In terminal output modes, we have the ctags(1)-like internal search facility built around the less(1) tag jump (:t) feature for quite some time now. We now have a similar feature in -T html output mode. To jump to (almost) the same places in the text, go to the address bar of the browser, type a hash mark ('#') after the URI, then the name of the option, command, variable, error code etc. you want to jump to, and hit enter. Check out the full report by Ingo Schwarze (schwarze@) and try out these new features *** Optimizing IllumOS Kernel Crypto (http://zfs-create.blogspot.com/2014/05/optimizing-illumos-kernel-crypto.html) Sašo Kiselkov, of ZFS fame, looked into the performance of the OpenSolaris kernel crypto framework and found it lacking. The article also spends a few minutes on the different modes and how they work. Recently I've had some motivation to look into the KCF on Illumos and discovered that, unbeknownst to me, we already had an AES-NI implementation that was automatically enabled when running on Intel and AMD CPUs with AES-NI support. This work was done back in 2010 by Dan Anderson.This was great news, so I set out to test the performance in Illumos in a VM on my Mac with a Core i5 3210M (2.5GHz normal, 3.1GHz turbo). The initial tests of “what the hardware can do” were done in OpenSSL So now comes the test for the KCF. I wrote a quick'n'dirty crypto test module that just performed a bunch of encryption operations and timed the results. KCF got around 100 MB/s for each algorithm, except half that for AES-GCM OpenSSL had done over 3000 MB/s for CTR mode, 500 MB/s for CBC, and 1000 MB/s for GCM What the hell is that?! This is just plain unacceptable. Obviously we must have hit some nasty performance snag somewhere, because this is comical. And sure enough, we did. When looking around in the AES-NI implementation I came across this bit in aes_intel.s that performed the CLTS instruction. This is a problem: 3.1.2 Instructions That Cause VM Exits ConditionallyCLTS. The CLTS instruction causes a VM exit if the bits in position 3 (corresponding to CR0.TS) are set in both the CR0 guest/host mask and the CR0 read shadow. The CLTS instruction signals to the CPU that we're about to use FPU registers (which is needed for AES-NI), which in VMware causes an exit into the hypervisor. And we've been doing it for every single AES block! Needless to say, performing the equivalent of a very expensive context switch every 16 bytes is going to hurt encryption performance a bit. The reason why the kernel is issuing CLTS is because for performance reasons, the kernel doesn't save and restore FPU register state on kernel thread context switches. So whenever we need to use FPU registers inside the kernel, we must disable kernel thread preemption via a call to kpreemptdisable() and kpreemptenable() and save and restore FPU register state manually. During this time, we cannot be descheduled (because if we were, some other thread might clobber our FPU registers), so if a thread does this for too long, it can lead to unexpected latency bubbles The solution was to restructure the AES and KCF block crypto implementations in such a way that we execute encryption in meaningfully small chunks. I opted for 32k bytes, for reasons which I'll explain below. Unfortunately, doing this restructuring work was a bit more complicated than one would imagine, since in the KCF the implementation of the AES encryption algorithm and the block cipher modes is separated into two separate modules that interact through an internal API, which wasn't really conducive to high performance (we'll get to that later). Anyway, having fixed the issue here and running the code at near native speed, this is what I get: AES-128/CTR: 439 MB/s AES-128/CBC: 483 MB/s AES-128/GCM: 252 MB/s Not disastrous anymore, but still, very, very bad. Of course, you've got keep in mind, the thing we're comparing it to, OpenSSL, is no slouch. It's got hand-written highly optimized inline assembly implementations of most of these encryption functions and their specific modes, for lots of platforms. That's a ton of code to maintain and optimize, but I'll be damned if I let this kind of performance gap persist. Fixing this, however, is not so trivial anymore. It pertains to how the KCF's block cipher mode API interacts with the cipher algorithms. It is beautifully designed and implemented in a fashion that creates minimum code duplication, but this also means that it's inherently inefficient. ECB, CBC and CTR gained the ability to pass an algorithm-specific "fastpath" implementation of the block cipher mode, because these functions benefit greatly from pipelining multiple cipher calls into a single place. ECB, CTR and CBC decryption benefit enormously from being able to exploit the wide XMM register file on Intel to perform encryption/decryption operations on 8 blocks at the same time in a non-interlocking manner. The performance gains here are on the order of 5-8x.CBC encryption benefits from not having to copy the previously encrypted ciphertext blocks into memory and back into registers to XOR them with the subsequent plaintext blocks, though here the gains are more modest, around 1.3-1.5x. After all of this work, this is how the results now look on Illumos, even inside of a VM: Algorithm/Mode 128k ops AES-128/CTR: 3121 MB/s AES-128/CBC: 691 MB/s AES-128/GCM: 1053 MB/s So the CTR and GCM speeds have actually caught up to OpenSSL, and CBC is actually faster than OpenSSL. On the decryption side of things, CBC decryption also jumped from 627 MB/s to 3011 MB/s. Seeing these performance numbers, you can see why I chose 32k for the operation size in between kernel preemption barriers. Even on the slowest hardware with AES-NI, we can expect at least 300-400 MB/s/core of throughput, so even in the worst case, we'll be hogging the CPU for at most ~0.1ms per run. Overall, we're even a little bit faster than OpenSSL in some tests, though that's probably down to us encrypting 128k blocks vs 8k in the "openssl speed" utility. Anyway, having fixed this monstrous atrocity of a performance bug, I can now finally get some sleep. To made these tests repeatable, and to ensure that the changes didn't break the crypto algorithms, Saso created a crypto_test kernel module. I have recently created a FreeBSD version of crypto_test.ko, for much the same purposes Initial performance on FreeBSD is not as bad, if you have the aesni.ko module loaded, but it is not up to speed with OpenSSL. You cannot directly compare to the benchmarks Saso did, because the CPUs are vastly different. Performance results (https://wiki.freebsd.org/OpenCryptoPerformance) I hope to do some more tests on a range of different sized CPUs in order to determine how the algorithms scale across different clock speeds. I also want to look at, or get help and have someone else look at, implementing some of the same optimizations that Saso did. It currently seems like there isn't a way to perform addition crypto operations in the same session without regenerating the key table. Processing additional buffers in an existing session might offer a number of optimizations for bulk operations, although in many cases, each block is encrypted with a different key and/or IV, so it might not be very useful. *** Brendan Gregg's special freeware tools for sysadmins (http://www.brendangregg.com/specials.html) These tools need to be in every (not so) serious sysadmins toolbox. Triple ROT13 encryption algorithm (beware: export restrictions may apply) /usr/bin/maybe, in case true and false don't provide too little choice... The bottom command lists you all the processes using the least CPU cycles. Check out the rest of the tools. You wrote similar tools and want us to cover them in the show? Send us an email to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** A look at 2038 (http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/) I remember the Y2K problem quite vividly. The world was going crazy for years, paying insane amounts of money to experts to fix critical legacy systems, and there was a neverending stream of predictions from the media on how it's all going to fail. Most didn't even understand what the problem was, and I remember one magazine writing something like the following: Most systems store the current year as a two-digit value to save space. When the value rolls over on New Year's Eve 1999, those two digits will be “00”, and “00” means “halt operation” in the machine language of many central processing units. If you're in an elevator at this time, it will stop working and you may fall to your death. I still don't know why they thought a computer would suddenly interpret data as code, but people believed them. We could see a nearby hydropower plant from my parents' house, and we expected it to go up in flames as soon as the clock passed midnight, while at least two airplanes crashed in our garden at the same time. Then nothing happened. I think one of the most “severe” problems was the police not being able to open their car garages the next day because their RFID tokens had both a start and end date for validity, and the system clock had actually rolled over to 1900, so the tokens were “not yet valid”. That was 17 years ago. One of the reasons why Y2K wasn't as bad as it could have been is that many systems had never used the “two-digit-year” representation internally, but use some form of “timestamp” relative to a fixed date (the “epoch”). The actual problem with time and dates rolling over is that systems calculate timestamp differences all day. Since a timestamp derived from the system clock seemingly only increases with each query, it is very common to just calculate diff = now - before and never care about the fact that now could suddenly be lower than before because the system clock has rolled over. In this case diff is suddenly negative, and if other parts of the code make further use of the suddenly negative value, things can go horribly wrong. A good example was a bug in the generator control units (GCUs) aboard Boeing 787 “Dreamliner” aircrafts, discovered in 2015. An internal timestamp counter would overflow roughly 248 days after the system had been powered on, triggering a shut down to “safe mode”. The aircraft has four generator units, but if all were powered up at the same time, they would all fail at the same time. This sounds like an overflow caused by a signed 32-bit counter counting the number of centiseconds since boot, overflowing after 248.55 days, and luckily no airline had been using their Boing 787 models for such a long time between maintenance intervals. The “obvious” solution is to simply switch to 64-Bit values and call it day, which would push overflow dates far into the future (as long as you don't do it like the IBM S/370 mentioned before). But as we've learned from the Y2K problem, you have to assume that computer systems, computer software and stored data (which often contains timestamps in some form) will stay with us for much longer than we might think. The years 2036 and 2038 might be far in the future, but we have to assume that many of the things we make and sell today are going to be used and supported for more than just 19 years. Also many systems have to store dates which are far in the future. A 30 year mortgage taken out in 2008 could have already triggered the bug, and for some banks it supposedly did. sysgettimeofday() is one of the most used system calls on a generic Linux system and returns the current time in form of an UNIX timestamp (timet data type) plus fraction (susecondst data type). Many applications have to know the current time and date to do things, e.g. displaying it, using it in game timing loops, invalidating caches after their lifetime ends, perform an action after a specific moment has passed, etc. In a 32-Bit UNIX system, timet is usually defined as a signed 32-Bit Integer. When kernel, libraries and applications are compiled, the compiler will turn this assumption machine code and all components later have to match each other. So a 32-Bit Linux application or library still expects the kernel to return a 32-Bit value even if the kernel is running on a 64-Bit architecture and has 32-Bit compatibility. The same holds true for applications calling into libraries. This is a major problem, because there will be a lot of legacy software running in 2038. Systems which used an unsigned 32-Bit Integer for timet push the problem back to 2106, but I don't know about many of those. The developers of the GNU C library (glibc), the default standard C library for many GNU/Linux systems, have come up with a design for year 2038 proofness for their library. Besides the timet data type itself, a number of other data structures have fields based on timet or the combined struct timespec and struct timeval types. Many methods beside those intended for setting and querying the current time use timestamps 32-Bit Windows applications, or Windows applications defining _USE32BITTIMET, can be hit by the year 2038 problem too if they use the timet data type. The _time64t data type had been available since Visual C 7.1, but only Visual C 8 (default with Visual Studio 2015) expanded timet to 64 bits by default. The change will only be effective after a recompilation, legacy applications will continue to be affected. If you live in a 64-Bit world and use a 64-Bit kernel with 64-Bit only applications, you might think you can just ignore the problem. In such a constellation all instances of the standard time_t data type for system calls, libraries and applications are signed 64-Bit Integers which will overflow in around 292 billion years. But many data formats, file systems and network protocols still specify 32-Bit time fields, and you might have to read/write this data or talk to legacy systems after 2038. So solving the problem on your side alone is not enough. Then the article goes on to describe how all of this will break your file systems. Not to mention your databases and other file formats. Also see Theo De Raadt's EuroBSDCon 2013 Presentation (https://www.openbsd.org/papers/eurobsdcon_2013_time_t/mgp00001.html) *** Beastie Bits Michael Lucas: Get your name in “Absolute FreeBSD 3rd Edition” (https://blather.michaelwlucas.com/archives/2895) ZFS compressed ARC stats to top (https://svnweb.freebsd.org/base?view=revision&revision=r315435) Matthew Dillon discovered HAMMER was repeating itself when writing to disk. Fixing that issue doubled write speeds (https://www.dragonflydigest.com/2017/03/14/19452.html) TedU on Meaningful Short Names (http://www.tedunangst.com/flak/post/shrt-nms-fr-clrty) vBSDcon and EuroBSDcon Call for Papers are open (https://www.freebsdfoundation.org/blog/submit-your-work-vbsdcon-and-eurobsdcon-cfps-now-open/) Feedback/Questions Craig asks about BSD server management (http://pastebin.com/NMshpZ7n) Michael asks about jails as a router between networks (http://pastebin.com/UqRwMcRk) Todd asks about connecting jails (http://pastebin.com/i1ZD6eXN) Dave writes in with an interesting link (http://pastebin.com/QzW5c9wV) > applications crash more often due to errors than corruptions. In the case of corruption, a few applications (e.g., Log-Cabin, ZooKeeper) can use checksums and redundancy to recover, leading to a correct behavior; however, when the corruption is transformed into an error, these applications crash, resulting in reduced availability. ***
This week on the show, we've got some new info on the talks from EuroBSDCon, a look at sharing a single ZFS pool between Linux and BSD, Sandboxing and much more! Stay tuned for your place to B...SD! This episode was brought to you by Headlines EuroBSDcon 2016 Presentation Slides (https://2016.eurobsdcon.org/PresentationSlides/) Due to circumstances beyond the control of the organizers of EuroBSDCon, there were not recordings of the talks given at the event. However, they have collected the slide decks from each of the speakers and assembled them on this page for you Also, we have some stuff from MeetBSD already: Youtube Playlist (https://www.youtube.com/playlist?list=PLb87fdKUIo8TAMC2HJLZ7H54edD2BeGWv) Not all of the sessions are posted yet, but the rest should appear shortly MeetBSD 2016 Trip Report: Domagoj Stolfa (https://www.freebsdfoundation.org/blog/meetbsd-2016-trip-report-domagoj-stolfa/) *** Cohabiting FreeBSD and Gentoo Linux on a Common ZFS Volume (https://ericmccorkleblog.wordpress.com/2016/11/15/cohabiting-freebsd-and-gentoo-linux-on-a-common-zfs-volume/) Eric McCorkle, who has contributed ZFS support to the FreeBSD EFI boot-loader code has posted an in-depth look at how he's setup dual-boot with FreeBSD and Gentoo on the same ZFS volume. He starts by giving us some background on how the layout is done. First up, GRUB is used as the boot-loader, allowing boot of both Linux and BSD The next non-typical thing was using /etc/fstab to manage mount-points, instead of the typical ‘zfs mount' usage, (apart from /home datasets) data/home is mounted to /home, with all of its child datasets using the ZFS mountpoint system data/freebsd and its child datasets house the FreeBSD system, and all have their mountpoints set to legacy data/gentoo and its child datasets house the Gentoo system, and have their mountpoints set to legacy as well So, how did he set this up? He helpfully provides an overview of the steps: Use the FreeBSD installer to create the GPT and ZFS pool Install and configure FreeBSD, with the native FreeBSD boot loader Boot into FreeBSD, create the Gentoo Linux datasets, install GRUB Boot into the Gentoo Linux installer, install Gentoo Boot into Gentoo, finish any configuration tasks The rest of the article walks us through the individual commands that make up each of those steps, as well as how to craft a GRUB config file capable of booting both systems. Personally, since we are using EFI, I would have installed rEFInd, and chain-loaded each systems EFI boot code from there, allowing the use of the BSD loader, but to each their own! HardenedBSD introduces Safestack into base (https://hardenedbsd.org/article/shawn-webb/2016-11-27/introducing-safestack) HardenedBSD has integrated SafeStack into its base system and ports tree SafeStack (http://clang.llvm.org/docs/SafeStack.html) is part of the Code Pointer Integrity (CPI) project within clang. “SafeStack is an instrumentation pass that protects programs against attacks based on stack buffer overflows, without introducing any measurable performance overhead. It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack.” “As of 28 November 2016, with clang 3.9.0, SafeStack only supports being applied to applications and not shared libraries. Multiple patches have been submitted to clang by third parties to add support for shared libraries.” SafeStack is only enabled on AMD64 *** pledge(2)… or, how I learned to love web application sandboxing (https://learnbchs.org/pledge.html) We've talked about OpenBSD's sandboxing mechanism pledge() in the past, but today we have a great article by Kristaps Dzonsons, about how he grew to love it for Web Sandboxing. +First up, he gives us his opening argument that should make most of you sit up and listen: I use application-level sandboxing a lot because I make mistakes a lot; and when writing web applications, the price of making mistakes is very dear. In the early 2000s, that meant using systrace(4) on OpenBSD and NetBSD. Then it was seccomp(2) (followed by libseccomp(3)) on Linux. Then there was capsicum(4) on FreeBSD and sandbox_init(3) on Mac OS X. All of these systems are invoked differently; and for the most part, whenever it came time to interface with one of them, I longed for sweet release from the nightmare. Please, try reading seccomp(2). To the end. Aligning web application logic and security policy would require an arduous (and usually trial-and-error or worse, copy-and-paste) process. If there was any process at all — if the burden of writing a policy didn't cause me to abandon sandboxing at the start. And then there was pledge(2). This document is about pledge(2) and why you should use it and love it. “ +Not convinced yet? Maybe you should take his challenge: Let's play a drinking game. The challenge is to stay out of the hospital. 1.Navigate to seccomp(2). 2. Read it to the end. 3. Drink every time you don't understand. For capsicum(4), the challenge is no less difficult. To see these in action, navigate no further than OpenSSH, which interfaces with these sandboxes: sandbox-seccomp-filter.c or sandbox-capsicum.c. (For a history lesson, you can even see sandbox-systrace.c.) Keep in mind that these do little more than restrict resources to open descriptors and the usual necessities of memory, signals, timing, etc. Keep that in mind and be horrified. “ Now Kristaps has his theory on why these are so difficult (NS..), but perhaps there is a better way. He makes the case that pledge() sits right in that sweet-spot, being powerful enough to be useful, but easy enough to implement that developers might actually use it. All in all, a nice read, check it out! Would love to hear other developer success stories using pledge() as well. *** News Roundup Unix history repository, now on GitHub (http://www.osnews.com/story/29513/Unix_history_repository_now_on_GitHub) OS News has an interesting tidbit on their site today, about the entire commit history of Unix now being available online, starting all the way back in 1970 and bringing us forward to today. From the README The history and evolution of the Unix operating system is made available as a revision management repository, covering the period from its inception in 1970 as a 2.5 thousand line kernel and 26 commands, to 2016 as a widely-used 27 million line system. The 1.1GB repository contains about half a million commits and more than two thousand merges. The repository employs Git system for its storage and is hosted on GitHub. It has been created by synthesizing with custom software 24 snapshots of systems developed at Bell Labs, the University of California at Berkeley, and the 386BSD team, two legacy repositories, and the modern repository of the open source FreeBSD system. In total, about one thousand individual contributors are identified, the early ones through primary research. The data set can be used for empirical research in software engineering, information systems, and software archaeology. This is a fascinating find, especially will be of value to students and historians who wish to look back in time to see how UNIX evolved, and in this repo ultimately turned into modern FreeBSD. *** Yandex commits improvements to FreeBSD network stack (https://reviews.freebsd.org/D8526) “Rework ip_tryforward() to use FIB4 KPI.” This commit brings some code from the experimental routing branch into head As you can see from the graphs, it offers some sizable improvements in forwarding and firewalled packets per second commit (https://svnweb.freebsd.org/base?view=revision&revision=309257) *** The brief history of Unix socket multiplexing – select(2) system call (https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/) Ever wondered about the details of socket multiplexing, aka the history of select(2)? Well Marek today gives a treat, with a quick look back at the history that made today's modern multiplexing possible. First, his article starts the way all good ones do, presenting the problem in silent-movie form: In mid-1960's time sharing was still a recent invention. Compared to a previous paradigm - batch-processing - time sharing was truly revolutionary. It greatly reduced the time wasted between writing a program and getting its result. Batch-processing meant hours and hours of waiting often to only see a program error. See this film to better understand the problems of 1960's programmers: "The trials and tribulations of batch processing". Enter the wild world of the 1970's, and we've now reached the birth of UNIX which tried to solve the batch processing problem with time-sharing. These days when a program was executed, it could "stall" (block) only on a couple of things1: + wait for CPU + wait for disk I/O + wait for user input (waiting for a shell command) or console (printing data too fast)“ Jump forward another dozen years or so, and the world changes yet again: This all changed in 1983 with the release of 4.2BSD. This revision introduced an early implementation of a TCP/IP stack and most importantly - the BSD Sockets API.Although today we take the BSD sockets API for granted, it wasn't obvious it was the right API. STREAMS were a competing API design on System V Revision 3. Coming in along with the sockets API was the select(2) call, which our very own Kirk McKusick gives us some background on: Select was introduced to allow applications to multiplex their I/O. Consider a simple application like a remote login. It has descriptors for reading from and writing to the terminal and a descriptor for the (bidirectional) socket. It needs to read from the terminal keyboard and write those characters to the socket. It also needs to read from the socket and write to the terminal. Reading from a descriptor that has nothing queued causes the application to block until data arrives. The application does not know whether to read from the terminal or the socket and if it guesses wrong will incorrectly block. So select was added to let it find out which descriptor had data ready to read. If neither, select blocks until data arrives on one descriptor and then awakens telling which descriptor has data to read. [...] Non-blocking was added at the same time as select. But using non-blocking when reading descriptors does not work well. Do you go into an infinite loop trying to read each of your input descriptors? If not, do you pause after each pass and if so for how long to remain responsive to input? Select is just far more efficient. Select also lets you create a single inetd daemon rather than having to have a separate daemon for every service. The article then wraps up with an interesting conclusion: > CSP = Communicating sequential processes In this discussion I was afraid to phrase the core question. Were Unix processes intended to be CSP-style processes? Are file descriptors a CSP-derived "channels"? Is select equivalent to ALT statement? I think: no. Even if there are design similarities, they are accidental. The file-descriptor abstractions were developed well before the original CSP paper. It seems that an operating socket API's evolved totally disconnected from the userspace CSP-alike programming paradigms. It's a pity though. It would be interesting to see an operating system coherent with the programming paradigms of the user land programs. A long (but good) read, and worth your time if you are interested in the history how modern multiplexing came to be. *** How to start CLion on FreeBSD? (https://intellij-support.jetbrains.com/hc/en-us/articles/206525024-How-to-start-CLion-on-FreeBSD) CLion (pronounced "sea lion") is a cross-platform C and C++ IDE By default, the Linux version comes bundled with some binaries, which obviously won't work with the native FreeBSD build Rather than using Linux emulation, you can replace these components with native versions pkg install openjdk8 cmake gdb Edit clion-2016.3/bin/idea.properties and change run.processes.with.pty=false Start CLion and open Settings | Build, Execution, Deployment | Toolchains Specify CMake path: /usr/local/bin/cmake and GDB path: /usr/local/bin/gdb Without a replacement for fsnotifier, you will get a warning that the IDE may be slow to detect changes to files on disk But, someone has already written a version of fsnotifier that works on FreeBSD and OpenBSD fsnotifier for OpenBSD and FreeBSD (https://github.com/idea4bsd/fsnotifier) -- The fsnotifier is used by IntelliJ for detecting file changes. This version supports FreeBSD and OpenBSD via libinotify and is a replacement for the bundled Linux-only version coming with the IntelliJ IDEA Community Edition. *** Beastie Bits TrueOS Pico – FreeBSD ARM/RPi Thin Clients (https://www.trueos.org/trueos-pico/) A Puppet package provider for FreeBSD's PkgNG package manager. (https://github.com/xaque208/puppet-pkgng) Notes from November London *BSD meetup (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2016-November/014059.html) SemiBug meeting on Dec 20th (http://lists.nycbug.org/pipermail/semibug/2016-November/000131.html) Feedback/Questions Erno - SSH without password (http://pastebin.com/SMvxur9v) Jonathan - Magical ZFS (http://pastebin.com/5ETL7nmj) George - TrueOS (http://pastebin.com/tSVvaV9e) Mohammad - Jails IP (http://pastebin.com/T8nUexd1) Gibheer - BEs (http://pastebin.com/YssXXp70) ***
This week on the show, we're loaded up with great stories ranging from System call fuzzing, a history of UNIX Pipes, speeding up MySQL imports and more. Stay tuned, BSDNow is coming your way right now. This episode was brought to you by Headlines System call fuzzing of OpenBSD amd64 using TriforceAFL (i.e. AFL and QEMU) (https://github.com/nccgroup/TriforceOpenBSDFuzzer) The NCCGroup did a series of fuzz testing against the OpenBSD syscall interface, during which they found a number of vulnerabilities, we covered this back in the early summer What we didn't notice, is that they also made the tools they used available. A combination of AFL (American Fuzzy Lop), QEMU, OpenBSD's FlashRD image generation tool, and the “Triforce” driver The other requirement is “a Linux box as host to run the fuzzer (other fuzzer hosts may work as well, we've only run TriforceAFL from a Linux host, specifically Debian/Ubuntu” It would be interesting to see if someone could get this to run from a BSD host It would also be interesting to run the same tests against the other BSDs *** On the Early History and Impact of Unix: the Introduction of Pipes (http://people.fas.harvard.edu/~lib113/reference/unix/unix2.html) Pipes are something we just take for granted today, but there was a time before pipes (How did anything get done?) Ronda Hauben writes up a great look back at the beginning of UNIX, and specifically at how pipes were born: One of the important developments in Unix was the introduction of pipes. Pipes had been suggested by McIlroy during the early days of creating Unix. Ritchie explains how "the idea, explained one afternoon on a blackboard, intrigued us but failed to ignite any immediate action. There were several objections to the idea as put....What a failure of imagination," he admits.(35) McIlroy concurs, describing how the initial effort to add pipes to Unix occurred about the same time in 1969 that Ritchie, Thompson and Canaday were outlining ideas for a file system. "That was when," he writes, "the simple pipeline as a way to combine programs, with data notationally propagating along a chain of (not necessarily concurrent) filters was articulated."(36) However, pipes weren't implemented in Unix until 1972. We also have a great quote from McIlroy on the day pipes were first introduced: Open Systems! Our Systems! How well those who were there remember the pipe-festooned garret where Unix took form. The excitement of creation drew people to work there amidst the whine of the computer's cool- ing fans, even though almost the same computer ac- cess, could be had from one's office or from home. Those raw quarters saw a procession of memorable events. The advent of software pipes precipitated a day-long orgy of one-liners....As people reveled in the power of functional composition in the large, which is even today unavailable to users of other systems. The paper goes on to talk about the invention of other important tools, such as “grep”, “diff” and more. Well worth your time if you want a glimpse into the history of UNIX *** Speeding up MySQL Import on FreeBSD (https://blog.feld.me/posts/2016/09/speeding-up-mysql-import-on-freebsd/) Mark Felder writes a blog post explaining how to speed up MySQL bulk data imports “I was recently tasked with rebuilding a readonly slave database server which only slaves a couple of the available databases. The backup/dump is straightforward and fast, but the restore was being excruciatingly slow. I didn't want to wait a week for this thing to finish, so I had to compile a list of optimizations that would speed up the process. This is the best way to do it on FreeBSD, assuming you're working with InnoDB. Additional optimizations may be required if you're using a different database engine.” “Please note this is assuming no other databases are running on this MySQL instance. Some of these are rather dangerous and you wouldn't want to put other live data at risk.” Most of the changes are meant to be temporary, used on a new server to import a dump of the database, then the settings are to be turned off. Specifically: sync_binlog = 0 innodbflushlogattrx_commit = 0 innodb-doublewrite = 0 He also prepends the following but of SQL before importing the data: set sqllogbin=0; set autocommit=0; set uniquechecks=0; set foreignkey_checks=0; You can also help yourself if your MySQL database lives on ZFS zfs set recordsize=16k pool/var/db/mysql zfs set redundant_metadata=most pool/var/db/mysql Remember, this tuning is ONLY for the initial import, leaving these settings on long term risks losing 5-10 seconds of your data if the server reboots unexpectedly zfs set sync=disabled pool/var/db/mysql zfs set logbias=throughput pool/var/db/mysql *** PostgreSQL and FreeBSD Quick Start (https://cwharton.com/blog/2016/10/postgresql-and-freebsd-quick-start/) There's lots of databases to choose from, but Postgres always has a special place on FreeBSD. Today we have a look at a ‘getting started' guide for those taking the plunge and using it for the first time. Naturally getting started will look familiar to many, a couple simple “pkg” and “sysrc” commands later, and you'll be set. After starting the service (With the “service” command) you'll be ready to start setting up your postgres instance. Next up you'll need to create your initial user/password combo, and a database with access granted to this particular user. If you plan to enable remote access to this DB server, you'll need to make some adjustments to one of the .conf files, allowing other IP's to connect. (If you are hosting something on the same system, this may not be needed) Now yous should be good to go! Enjoy using your brand new Postgres database. If this is your first rodeo, maybe start with something easy, like Apache or Nginx + Wordpress to try it out. *** News Roundup OpenBSD vmm hypervisor test drive (https://www.youtube.com/watch?v=KE_7E1pXy5c) As we asked for a week or two ago, someone has taken OpenBSD's vmm for a test drive, and made a video of it The command line interface for vmm, vmctl, looks quite easy to use. It takes an approach much closer to some of the bhyve management frameworks, rather than bhyve's rather confusing set of switches It also has a config file, the format of which looks very similar to what I designed for bhyveucl, and my first effort to integrate a config file into bhyve itself. The video also looks at accessing the console, configuring the networking, and doing an OpenBSD install in a fresh VM Currently vmm only supports running OpenBSD VMs *** FreeBSD Foundation October 2016 Update (https://www.freebsdfoundation.org/wp-content/uploads/2016/10/FreeBSD-Foundation-October-2016-Update.pdf) Wow, November is already upon us with the Holidays just around the corner. Before things get lost in the noise we wanted to highlight this update from the FreeBSD foundation. Before getting into the stories, they helpfully provide a list of upcoming conferences for this fall/winter, which includes a couple of USENIX gatherings, and the Developer Summit / MeetBSD next week. +The foundation gives us a quick hardware update initially, discussing some of the new ThunderX Cavium servers which are deployed (ARMv8 64Bit) and yes I'm drooling a bit. They also mention that work is ongoing for the RPi3 platform and PINE64. GNN also has an article reprinted from the FreeBSD journal, talking about the achievement of making it to 11.0 over the span of 23 years now. Of course he mentions that the foundation is open to all, and welcomes donations to continue to keep up this tradition of good work being done. Deb Goodkin gives us an update on the “Grace Hopper” convention that took place in Houston TX several weeks back. Roughly 14k women in Tech attended, which is a great turnout, and FreeBSD was well represented there. Next we have a call to potential speakers, don't forget that there are plenty of places you can help present about FreeBSD, not just at *BSD centered conferences, but the SCALES of the world as well. We wrap up with a look at EuroBSDCon 2016, quite a nice writeup, again brought to us by Deb at the foundation, and includes a list of some of those recognized for their contributions to FreeBSD. *** Adhokku – a toy PaaS powered by FreeBSD jails and Ansible (https://github.com/adhokku/adhokku) Described as a toy Platform-as-a-Service, Adhokku is an ansible based automated jail creation framework Based on the concept of Dokku, a single-host open source PaaS for Linux powered by Docker When you deploy an application using Adhokku, Adhokku creates a new jail on the remote host and provisions it from a fixed clean state using the instructions in the Jailfile in your Git repository. All jails sit behind a reverse proxy that directs traffic to one of them based on the domain name or the IP address in the HTTP request. When a new jail has been provisioned for an application, Adhokku seamlessly reconfigures the reverse proxy to send traffic to it instead of the one currently active for that application. The following instructions show how to get Adhokku and an example application running in a VM on your development machine using Vagrant. This process should require no FreeBSD-specific knowledge, through modifying the Jailfile to customize the application may. This seems like an interesting project, and it is good to see people developing workflows so users familiar with docker etc, can easily use BSD instead *** Installing OpenBSD 6.0 on your laptop is really hard (not) (http://sohcahtoa.org.uk/openbsd.html) OpenBSD on a laptop? Difficult? Not hardly. We have a great walkthrough by Keith Burnett, which demonstrates just how easy it can be to get up and running with an XFCE desktop from a fresh OpenBSD installation. For those curious,this was all done with a Thinkpad X60 and 120GB SSD and OpenBSD 6.0. He doesn't really cover the install process itself, that is well covered by the link to the OpenBSD FAQ pages. Once the system is up and running though, we start with the most important portion, getting working internet access (Via wifi) Really just a few ‘ifconfig' commands later and we are in business. Step 2 was getting the package configuration going. (I've never understood why this is still a thing, but no fret, its easy enough to do) With package repos available, now you can grab the binaries for XFCE and friends with just a few simple “pkg_add” commands Steps 4-6 are some specific bits to enable XFCE services, and some handy things such as setting doas permissions to get USB mounting working (For graphical mount/unmount) Lastly, keeping the system updated is important, so we have a nice tutorial on how to do that as well, using a handy “openup” script, which takes some of the guesswork out of it. Bonus! Steps for doing FDE as also included, which isn't for everyone, but you may want it *** Beastie Bits Pi-top with RPi-3 and FreeBSD HEAD (https://twitter.com/gvnn3/status/791475373380804608) NetBSD 7.0.2 released (http://blog.netbsd.org/tnf/entry/netbsd_7_0_1_released1) DragonflyBSD - git: kernel - Fix mmcsd read/write issues (http://lists.dragonflybsd.org/pipermail/commits/2016-October/624851.html) A char device which implements an Enigma machine (FreeBSD & Linux) (https://github.com/rafael-santiago/dev-enigma) *** Feedback/Questions Matt - System Monitoring (http://pastebin.com/ayzvCuaq) Tony - LLVM License (http://pastebin.com/r5axPSE7) Ben - Thanks (http://pastebin.com/MNxCvUtX) David - Write Cache (http://pastebin.com/RswFASqW) Charles - Fonts (http://pastebin.com/e317a32f) ***
This week on the show, we'll be talking to Petra about the NetBSD foundation, about how they operate and assist NetBSD behind the scenes. That plus lots of news This episode was brought to you by Headlines What is new on EC2 for FreeBSD 11.0-RELEASE (http://www.daemonology.net/blog/2016-10-03-FreeBSD-EC2-11-0-RELEASE.html) “FreeBSD 11.0-RELEASE is just around the corner, and it will be bringing a long list of new features and improvements — far too many for me to list here. I think there are some improvements in FreeBSD 11.0 which are particularly noteworthy for EC2 users.” “First, the EC2 Console Screenshot functionality now works with FreeBSD. This provides a "VGA" output as opposed to the traditional "serial port" which EC2 has exposed as "console output" for the past decade, and is useful largely because the "VGA" output becomes available immediately whereas the "serial port" output can lag by several minutes. This improvement is a simple configuration change — older releases didn't waste time writing to a non-serial console because it didn't go anywhere until Amazon added support on their side — and can be enabled on older FreeBSD releases by changing the line console="comconsole" to boot_multicons="YES" in /boot/loader.conf.” “The second notable change is support for EC2 "Enhanced Networking" using Intel 82599 hardware; on the C3, C4, R3, I2, D2, and M4 (excluding m4.16xlarge) families, this provides increased network throughput and reduced latency and jitter, since it allows FreeBSD to talk directly to the networking hardware rather than via a Xen paravirtual interface. Getting this working took much longer than I had hoped, but the final problem turned out not to be in FreeBSD at all — we were tickling an interrupt-routing bug in a version of Xen used in EC2. Unfortunately FreeBSD does not yet have support for the new "Elastic Network Adapter" enhanced networking used in P2 and X1 instance families and the m4.16xlarge instance type; I'm hoping that we'll have a driver for that before FreeBSD 11.1 arrives.” “The third notable change is an improvement in EC2 disk throughput. This comes thanks to enabling indirect segment I/Os in FreeBSD's blkfront driver; while the support was present in 10.3, I had it turned off by default due to performance anomalies on some EC2 instances. (Those EC2 performance problems have been resolved, and disk I/O performance in EC2 on FreeBSD 10.3 can now be safely improved by removing the line hw.xbd.xbdenableindirect="0" from /boot/loader.conf.)” “Finally, FreeBSD now supports all 128 CPUs in the x1.32xlarge instance type. This improvement comes thanks to two changes: The FreeBSD default kernel was modified in 2014 to support up to 256 CPUs (up from 64), but that resulted in a (fixed-size) section of preallocated memory being exhausted early in the boot process on systems with 92 or more CPUs; a few months ago I changed that value to tune automatically so that FreeBSD can now boot and not immediately panic with an out-of-the-box setup on such large systems.” “I think FreeBSD/EC2 users will be very happy with FreeBSD 11.0-RELEASE; but I'd like to end with an important reminder: No matter what you might see on FTP servers, in EC2, or available via freebsd-update, the new release has not been released until you see a GPG-signed email from the release engineer. This is not just a theoretical point: In my time as a FreeBSD developer I've seen multiple instances of last-minute release re-rolls happening due to problems being discovered very late, so the fact that you can see bits doesn't necessarily mean that they are ready to be downloaded. I hope you're looking forward to 11.0-RELEASE, but please be patient.” *** Upgrading Amazon EC2 instance from 10.3 to 11.0-PRERELEASE results in hang at boot (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213196) As if to underscore that last point, a last minute bug was found on sunday night A user reported that they used freebsd-update to upgrade an EC2 instance from 10.3 to 11.0 and it started hanging during boot After some quick investigation by Colin, the problem was reproduced Since I had done a lot of work in the loader recently, I helped Colin build a version of the loader with a lot of the debugging enabled, and some more added to try to isolate where in the loader the freeze was happening Colin and I worked late into the night, but eventually found the read from disk that was causing the hang Unlike most of the other reads, that were going into the heap, this read was into a very low memory address, right near the 640kb border. This initially distracted us from the real cause of the problem With more debugging added, it was determined that the problem was in the GELIBoot code, when reading the last sector of each partition to determine if it is encrypted. In cases where the partition is not 4k aligned, and butts up against the end of the disk, the formula used could result in a read past the end of the disk The formula rounds the last sector byte address down to the nearest factor of 4096, then reads 4096 bytes. Then that buffer is examined to determine if the partition is encrypted. If it is a 512b sector drive, the metadata will be in the last 512 bytes of that 4096 byte buffer. However, if the partition is not 4k aligned, the rounding will produce a value that is less than 4096 bytes from the end of the disk, and attempting to read 4096 bytes, will read past the end of the disk Normally this isn't that big of a problem, the BIOS will just return an error. The loader will retry up to three times, then give up and move on, continuing to boot normally. Some BIOSes are buggy, and will initiate their own retries, and the combination might result in a stall of up to 30 seconds for each attempt to read past the end of the disk But it seems that Amazon EC2 instances, (and possibly other virtual instances), will just hang in this case. This bug has existed for 6 months, but was not caught because almost all installations are 4k aligned thanks to changes made to the installer over the last few years, and most hardware continues to boot with no sign of a problem Even the EC2 snapshot images of 11.0 do not have the problem, as they use a newer disk layout that is 4k aligned by default now. The problem only seems to happen when older disk images are upgraded The fix has been committed and will be merged the the branches over the next few days An Errata notice will be issues, and the fix will be available via freebsd-update It is recommended that EC2 users, and anyone who wants to be especially cautious, wait until this errata notice goes out before attempting to upgrade from FreeBSD 10.3 to 11.0 You can determine if your partitions are 4k aligned by running ‘gpart show'. If there is free space after your last partition, you won't have any issues. *** OpenBSD 6.0 Limited Edition CD set (signed by developers) (http://undeadly.org/cgi?action=article&sid=20160929230557&mode=expanded) The first one went for .$4,200.00 (http://www.ebay.com/itm/-/331985953783) + Looking for your piece of OpenBSD history? At the recent g2k16 hackathon in Cambridge UK, 40 OpenBSD developers put pen to paper and signed 5 copies of the new 6.0 release. + Each of these will be auctioned off on ebay, with the proceeds to benefit the OpenBSD foundation. + The first auction has already ended, and CD set went for a whopping $4200! + The next set only has 2 days left, and currently stands at $3000! (http://www.ebay.com/itm/-/331990536246) + Get your bids in soon, these are VERY unique, the odds of getting the same 40 developers in a room together and signing a new .0 release may make this a once-in-a-lifetime opportunity. + Additionally, if you are just starting your OpenBSD collection, here's a nice image to make you envious: A nice collection of OpenBSD CD Sets (http://i.imgur.com/OrE0Gsa.png) [What typing ^D really does on Unix ](https://utcc.utoronto.ca/~cks/space/blog/unix/TypingEOFEffects) + How often have you used a ^D to generate an EOF? Do you really know what that does? + Chris Siebenmann has posted a look at this on his blog, which might not be what you think “Typing ^D causes the tty driver to immediately finish a read().” He continues on: Normally doing a read() from a terminal is line-buffered inside the tty driver; your program only wakes up when the tty driver sees the newline, at which point you get back the full line. (Note that this buffering is distinct from anything that your language's IO system may be doing.) Typing ^D causes the tty driver to stop waiting for a newline and immediately return from the read() with however much of the line has been accumulated to date. If you haven't typed anything on the line yet, there is nothing accumulated and the read() will return 0 bytes, which is conveniently the signal for end of file. If you have typed something the program will get it; because it doesn't have a trailing newline, the program's own line-buffering may take over and keep read()ing to get the rest of the line. (Other programs will immediately process the partial line with no buffering; cat is one example of this.) Once you've typed ^D on a partial line, that portion of the line is immutable because it's already been given to the program. Most Unixes won't let you backspace over such partial lines; effectively they become output, not input. (Note that modern shells are not good examples of this, because they don't do line-buffered input; to support command line editing, they switch terminal input into an uninterpreted mode. So they get the raw ^D and can do whatever they want with it, and they can let you edit as much of the pending line as they want.) Fascinating stuff, and interesting to see behind the curtain at exactly what's going on with your programs buffering and tty driver interaction. Interview - Petra Zeidler - spz@netbsd.org (mailto:spz@netbsd.org) NetBSD Foundation *** News Roundup Running FreeBSD in Travis-CI Thanks to KQEmu (http://erouault.blogspot.com/2016/09/running-freebsd-in-travis-ci.html) Travis-CI is the most popular testing framework on Github, but it doesn't support any of the BSDs This didn't discourage Even Rouault, who managed to run FreeBSD in KQEMU on the Linux instances provided by Travis-CI “Travis-CI has a free offer for software having public repository at GitHub. Travis-CI provides cloud instances running Linux or Mac OS X. To increase portability tests of GDAL, I wondered if it was somehow possible to run another operating system with Travis-CI, for example FreeBSD. A search lead me to this question (https://github.com/travis-ci/travis-ci/issues/1818) in their bug tracker but the outcome seems to be that it is not possible, nor in their medium or long term plans.” “One idea that came quickly to mind was to use the QEMU machine emulator that can simulate full machines, of several hardware architectures.” They found an existing image of FreeBSD 9.2 and configured the Travis job to download it and fire it up in QEMU. “Here we go: ./configure && make ! That works, but 50 minutes later (the maximum length of a Travis-CI job), our job is killed with perhaps only 10% of the GDAL code base being compiled. The reason is that we used the pure software emulation mode of QEMU that involves on-the-fly disassembling of the code to be run and re-assembling.” Travis-CI runs in Google Compute Engine, which does not allow nested virtualization, so hardware virtualization is not an option to speed up QEMU “Here comes the time for good old memories and a bit of software archeology. QEMU was started by Fabrice Bellard. If you didn't know his name yet, F. Bellard created FFMPEG and QEMU, holds a world record for the number of decimals of Pi computed on a COTS PC, has ported QEMU in JavaScript to run the Linux kernel in your browser, devised BPG, a new compression based on HEVC, etc....” “At the time where his interest was focused on QEMU, he created KQemu, a kernel module (for Linux, Windows, FreeBSD hosts), that could significantly enhance QEMU performance when the guest and hosts are x86/x86_64 and does not require (nor use) hardware virtualization instructions.” “Running it on Travis-CI was successful too, with the compilation being done in 20 minutes, so probably half of the speed of bare metal, which is good enough.” “I could also have potentially tried VirtualBox because, as mentioned above, it supports software virtualization with acceleration. But that is only for 32 bit guests (and I didn't find a ready-made FreeBSD 32bit image that you can directly ssh into). For 64 bit guests, VirtualBox require hardware virtualization to be available in the host. To the best of my knowledge, KQemu is (was) the only solution to enable acceleration of 64 bit guests without hardware requirements.” It will be interesting to see if enough people do this hack, maybe Travis-CI will consider properly supporting FreeBSD *** OpenBSD EuroBSDcon 2016 Papers are online (https://www.openbsd.org/events.html) Slides from the OpenBSD talks at EuroBSDCon are online now Landry Breuil, Building packages on exotic architectures (https://rhaalovely.net/~landry/eurobsdcon2016/) Peter Hessler, Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD (https://www.openbsd.org/papers/eurobsdcon2016-bfd.pdf) Ingo Schwarze, Why and how you ought to keep multibyte character support simple (https://www.openbsd.org/papers/eurobsdcon2016-utf8.pdf) (roff/mm/gpresent source code (https://www.openbsd.org/papers/eurobsdcon2016-utf8.roff)) Stefan Sperling, OpenBSD meets 802.11n (https://www.openbsd.org/papers/eurobsdcon2016-openbsd-11n.pdf) Antoine Jacoutot, OpenBSD rc.d(8) (https://www.bsdfrog.org/pub/events/openbsd-rcd-EuroBSDcon2016.pdf) Marc Espie, Retrofitting privsep into dpb and pkg_add (https://www.openbsd.org/papers/eurobsdcon2016-privsep.pdf) Martin Pieuchot, Embracing the BSD routing table (https://www.openbsd.org/papers/eurobsdcon2016-embracingbsdrt.pdf) I am working to build a similar website for the FreeBSD project, but there is still a lot of work to do I also managed to find the slides from the keynotes: Opening Keynote: George Neville-Neil: Looking Backwards: The coming decades of BSD (https://papers.freebsd.org/2016/EuroBSDCon/LookingBackwards.pdf) Closing Keynote: Gert Döring: Internet Attacks, Self-Governance, and the Consequences (http://www.monobsd.com/files/16_ddos_and_consequences.pptx) *** VirtualBox Shared Folders on FreeBSD: progress report (https://kernelnomicon.org/?p=650) In the past month or so, VirtualBox in the FreeBSD ports tree got bumped to version 5, which while bringing new features, did cause a regression in Shared Folders. FreeBSD developer gonzo@ (Oleksandr Tymoshenko) has been tackling this issue in recent days and provides us with a look behind the curtain at the challenges involved. Specifically he started by implementing the various needed VOPs: “lookup, access, readdir, read, getattr, readlink, remove, rmdir, symlink, close, create, open, write.” He then continues with details about how complete this is: ““Kind of implemented” means that I was able to mount directory, traverse it, read file, calculate md5 sums and compare with host's md5sum, create/remove directories, unzip zip file, etc but I doubt it would survive stress-test. Locking is all wrong at the moment and read/write VOPs allocate buffers for every operation.” The bigger issue faced is with the rename VOP though: I hit a roadblock with rename VOP: it involves some non-trivial locking logic and also there is a problem with cached paths. VBox hypervisor operates on full paths so we cache them in vboxfs nodes, but if one of parent directories is renamed, all cached names should be modified accordingly. I am going to tackle these two problems once I have long enough stretch of time time sit and concentrate on task. + We wish him luck in getting those issues solved. I know quite a few of our users rely on shared folders as well. FreeBSD News Issue #1 (http://support.rossw.net/FreeBSD-Issue1.pdf) Issue #1 of FreeBSD News, from summer of 1997 Contains an article by Yahoo! co-founder David Filo about their early use of FreeBSD, on 100mhz Pentium machines with 64MB of ram Java Development Kit 1.0.2 ported to FreeBSD What is FreeBSD? Running the world's busiest FTP site (cdrom.com) on FreeBSD Xi Graphics announces the release of CDE Business Desktop, the first and only integrated desktop for FreeBSD, on AcceleratedX, a fully supported commercial grade X display server Get FreeBSD 2.2.2 Today! *** Beastie Bits Call for testing: newly MPSAFE nvme(4) (http://mail-index.netbsd.org/current-users/2016/09/21/msg030183.html) Thinking about starting a BUG in Indianapolis, IN USA (http://lists.dragonflybsd.org/pipermail/users/2016-September/313061.html) The cost of forsaking C: Why students still need to learn C (https://medium.com/bradfield-cs/the-cost-of-forsaking-c-113986438784#.o2m5gv8y7) OpenBSD (U)EFI bootloader howto (https://blog.jasper.la/openbsd-uefi-bootloader-howto/) Michael Lucas sets his eyes on OpenBSD's web stack for his next book (http://blather.michaelwlucas.com/archives/2780) LibreSSL 2.5.0 released (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.5.0-relnotes.txt) OPNsense 16.7.5 released (https://opnsense.org/opnsense-16-7-5-released/) Feedback/Questions Jonas - ZFS on DO (http://pastebin.com/XeJhK0AJ) Ricardo - OpenBSD Encrypted Disk (http://pastebin.com/Z9JRjcvb) WiskerTickle - Storage Benchmark (http://pastebin.com/XAD0UevP) Phil - Thanks (http://pastebin.com/N52JhYru) Luis - Misc Questions (http://pastebin.com/57qS0wrx) ***
This week on BSDNow, we're going to be hearing about Allan's trip to EuroBSDCon, plus an Interview about “Bro on BSD”! Stay tuned, for your place to This episode was brought to you by Headlines EuroBSDCon 2016 Wrapup Ollivier Robert's Photos from EuroBSDCon (https://assets.keltia.net/photos/EuroBSDCon-2016/) Get your BSDNow die-cut stickers (http://www.jupiterbroadcasting.com/stickers/) NetBSD for newbies - Develop your own Power PC (http://discusscomputerx.blogspot.com/2016/09/netbsd-for-noobies-your-power-laptop.html) We don't get to feature too many stories on NetBSD being deployed as a Power PC (Not PowerPC, you know, a Powerful “PC”), so we jumped at this one. Specifically it starts off with some of the pre-req's that you'll need to get started, such as NetBSD 7.0.1 / amd64, along with some information about which wireless nics you may be using. (NetBSD like other BSD's will give a driver based device name for network interfaces) From there, instructions on how to write your WPA_supplicant config are provided, in order for us to fetch the NetBSD sources and convert to their -STABLE branch. After doing a CVS checkout of the sources, he then provides a walkthrough of doing a kernel compile / install, however it mentions changing the config, but doesn't provide an example of what options were changed. Perhaps to remove drivers we don't need? At this point the rest of the “desktop” setup is pretty straight forward. Some packages are added such as openbox, lxappearance, firefox, etc. To get working sound, firefox requires pulseaudio, which in turn needs dbus, so instructions on getting that service up and running are provided as well. When it's all said and done, you'll end up with your shiny new NetBSD -STABLE desktop (or laptop), bragging rights achieved! *** More about OpenSMTPD 6.0.0 (https://www.poolp.org/tech/posts/2016/09/12/opensmtpd-6-0-0-released/) OpenSMTPd 6.0.0 has just been released “and it's quite different from former releases.” “Unlike most of our releases, it comes out with almost no new feature.”, “Turns out most of the changes are not visible.” Changelog: new fork+reexec model so each process has its own randomized memory space logging format has been reworked a "multi-line response" bug in the LMTP delivery backend has been fixed connections concurrency limits have been bumped artificial delaying in remote sessions have been reduced dhparams option has been removed dhe option has been added, supporting auto and legacy modes smtp engine has been simplified various cosmetic changes, code cleanup and documentation improvement “The OpenSMTPD bootstrap process was quite simple: Upon executation, the parent process would read configuration, build a memory representation of it and would then create a bunch of socketpair() before fork()-ing all of its child processes.” The problem is that this does not take advantage of the new address randomization feature. Each child will have the same memory layout, copied from the parent process “So deraadt@ suggested that if OpenSMTPD would not just fork() children but instead fork() them and reexecute the smtpd binary, then each of the children would have its own randomized memory space.” “The idea itself is neat, however not so trivial to implement because when we reexec the whole "inherit configuration and descriptors" part goes away. It's not just fork and exec, it's fork and exec and figure a way for the parent to pass back all the information and descriptors back to the new post-fork instance so it is the new instance that allocates memory and decides where the information goes.” *** Upgrade a FreeBSD 10.3 Installation with ZFS on Root and Full Disk Encryption to 11.0 (http://ftfl.ca/blog/2016-09-17-zfs-fde-one-pool-conversion.html) While FreeBSD 11.0 is not out yet, Joseph Mingrone has helped me work out and test the instructions for upgrading a FreeBSD 10.3 ZFS on full disk encryption setup (bootpool + zpool) to the new GELIBoot feature, which does not require any unencrypted partitions, just the 128kb bootcode Note: Do not upgrade to FreeBSD 11.0 yet. While some images have landed on the FTP server, they do not contain the final openssl fix and are going to be recreated. Currently, GELIBoot does not support key files, so the first step is to reencrypt the master key with only a passphrase. Next, to avoid GELIBoot picking up encrypted partitions that it does not support, or partitions you do not want decrypted at boot, only partitions with the GELIBoot flag are decrypted, so set the flag on your root partition Then, move the loader, kernel, and other files into /boot on the root filesystem, instead of them living on the bootpool. This allows the kernel to be versioned with boot environments, and is the main purpose of this work Then, install the newer gptzfsboot, as this is required to support GELIBoot The old 2gb bootpool partition is then purposely mislabeled as freebsd-vinum, so it is not picked up by the boot blocks. Later, if the upgrade is successful, this partition can be deleted, and used as addition swap or something In order to boot correctly, you want all boot environments to have the ‘canmount' ZFS property set to ‘noauto' Thank you to Joseph for taking the time to prod me for the information required to write this up, and for testing it and finding all of the issues *** Interview - Michael Shirk - mshirk@daemon-security.com (mailto:mshirk@daemon-security.com) / @shirkdog (https://twitter.com/shirkdog) Running Bro on BSD *** News Roundup FreeBSD based distro for virtual hosting platform and appliance (https://clonos.tekroutine.com/) An interesting new FreeBSD-based project as shown up online, called “ClonOS”, which bills itself as a “free open-source FreeBSD-based platform for virtual environments creation and management” It looks to be leveraging an impressive list of technologies, including Bhyve, Xen, Jails and CBSD / Puppet for management tasks. Among its list of features: ZFS features support; VM cloning, export, import Ethernet SoftSwitch for separated networking jails for lightweight container VNC terminal for VM/containers Templates for VM/containers Configuration management/helpers Multi-node operation Multi-Node? Color me intrigued! Right now it appears to be under heavy development, but we'll reach out to the developer to see if we can get an interview lined up at some point! The Raspberry PI Platform and The Challenges of Developing FreeBSD (https://bsdmag.org/oleksandr_rybalko/) BSDMag recently did an interview with FreeBSD developer Olesandr Rybalko! Oleksandr lives in the Ukraine, and while you may not have heard of him, he has worked on some cool projects for FreeBSD including the new “vt” console driver (Which a lot of people are using now), and ARM/MIPS support. The interview covers some of the work he's done to get the PI support working with FreeBSD: I think, my main help here was a USB OTG driver, which I wrote before for another device (Ralink RT3052), then port it to R-Pi. But it was rewritten by Hans Peter Selasky. I do not know so much about USB as Hans knows. Another useful part of my help is Xorg support. I did a simple Xorg video driver which uses framebuffer exported by virtual terminal subsystem. That is help to many guys to start use RPi as a simple desktop system. He was also asked the question “Why would FreeBSD be good fit for ARM?” FreeBSD is very powerful as a network server. All modern network features in one box, with very fast processing. Another good side of FreeBSD is modularity. It is not required to write code to use some driver that was already written for another system, you can just define it in configuration files (kernel config, kernel hints, FDT). So if you want build a nice, R-Pi based, home server – use FreeBSD. If you want to play with devices attached to R-Pi's GPIO – use FreeBSD. He also discusses his work on the ZRouter project, which is a very light-weight platform for tiny routers / embedded devices. But lastly the RPI comes up again, specifically asking him how interested individuals can get involved. Specifically the wiki.freebsd.org is a great reference point for those intested in getting started with FreeBSD on embedded. The warm community is also a plus! Trying out the FreeBSD powered TrueOS (http://www.phoronix.com/scan.php?page=news_item&px=TrueOS-First-Spin) The folks over at Phoronix have done an early look at the new TrueOS desktop images and given some of their thoughts. First up he gives props to the installer, noting that: The TrueOS desktop installer is basically the same as from the PC-BSD days, just re-branded. Still one of the easiest BSD graphical installers I've dealt with and makes it a breeze for setting up a FreeBSD-on-ZFS system by default. After that they took it for a minimal spin, and thing mostly seem to be working. He mentions some of the default apps (Such as qupzilla and trojita) aren't their favorite, but Lumina has come quite a ways for 1.0, despite a few rough edges still. (We are in the process of changing those default e-mail / browser apps) Lastly the article mentions that it's time to do a more full BSD round-up to see the state of installation of them, which we happen to have next! Trying out 8 BSDs on a modern PC (http://www.phoronix.com/scan.php?page=article&item=trying-8-bsds&num=1) First up was TrueOS again, which no major changes there, easy install and done. From there he tries out DragonFlyBSD, which he mentions that while the installer isn't as easy, it is still one of his favorite BSD's, working with all the hardware they've thrown at it. Next up was GhostBSD, which also has an Easy-To-Use graphical installer similar to TrueOS that made it quick to get loaded and up to the Mate desktop. Also tested was FreeBSD 11.0-RC2, which he mentions was easy to installed, and once done then ‘pkg' could be used to easily get the setup he wanted setup. Turning over to page two we get to the naughty list of BSD's he had troubles with. First up was OpenBSD which he tried 6.0. After installation and first boot, the display kept ‘disappearing' which meant he couldn't get IP information to try SSH'ing into the box. Perhaps a display driver error? NetBSD 7 was up next, where the installer couldn't get past a root device prompt. Most likely trouble finding the install media, which was the same story with MightnightBSD as well. Also tested was “PacBSD” (Formerly ArchBSD) which he did manage to get installed, but not after major fighting with the process. After the process he ran into some issues getting packages up and running, but mentions it may have been bad timing due to them moving to a new server at the time. *** IllumOS imports a modified FreeBSD boot loader to replace grub 0.97 (https://www.listbox.com/member/archive/182181/2016/09/sort/time_rev/page/1/entry/0:1/20160923124232:B7978ED4-81AC-11E6-A6DA-02E3F010038B/) Toomas Soome's work to port the FreeBSD boot loader to IllumOS has been merged into illumos-gate, the upstream repository for all IllumOS distributions Toomas' work has also resulted in a number of commits to FreeBSD, and code sharing in both directions Toomas helped me a lot with the building of the ZFS boot environment listing menu, even though on IllumOS they use a configuration file to list the BEs, rather than interrogating the live zpool like we do in FreeBSD Toomas' work to improve msdosfs and the block cache to speed up booting IllumOS also greatly helped FreeBSD This work means IllumOS can now boot from a RAID-Z (the old grub they used could not), and if the work Toomas has done on FreeBSD is any indication, support for almost all other zpool features is also on the way This work also sets IllumOS on a path to eventually having UEFI boot as well It is good to see this work happening, FreeBSD technology being reused elsewhere, but also the improvements being made for IllumOS are coming back to FreeBSD, often landing upstream first, to make merging them into IllumOS easier. The mailing list post describes how to convert existing systems away from grub, as well as how to opt to remain on grub for a while longer. Grub 0.97 is expected to be removed from IllumOS within a year. *** Beastie Bits A demo of booting CentOS and Windows 10 in FreeBSD Bhyve through VNC headless (https://www.youtube.com/watch?v=8YQQfXqtyaA) This year's anemic output (http://blather.michaelwlucas.com/archives/2762) “PAM Mastery” ebook now out (http://blather.michaelwlucas.com/archives/2771) How-to Install OpenBSD 6.0 plus XFCE desktop and basic applications (https://www.youtube.com/watch?v=oC5D9fenQBs) *** Feedback/Questions Piotr - LibreBoot (http://pastebin.com/yniniNpV) Alan - FreeBSD and PC-BSD (http://pastebin.com/dCNX0yF7) Eduardo - Newcomers (http://pastebin.com/LndNeAYb) Greg - ZFS ACL's (http://pastebin.com/F0y6L6NK) Brian - Laptop Recs (http://pastebin.com/sqMPJGMM) ***
This week on BSDNow, Allan is currently at EuroBSDCon! However due to the magic of video (or time travel), you still get a new episode. (You're Welcome!). Stay tuned This episode was brought to you by Headlines Performance Improvements for FreeBSD Kernel Debugging (http://backtrace.io/blog/blog/2016/08/25/improving-freebsd-kernel-debugging/) “We previously explored FreeBSD userspace coredumps (http://backtrace.io/blog/blog/2015/10/03/whats-a-coredump). Backtrace's debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them.” “A kernel core is typically only generated in exceptional circumstances. Unlike userspace processes, kernel routines cannot fault without sacrificing the machine's availability. This means things like page faults and illegal instructions inside the kernel stop the machine, instead of just one process. At that point, in most cases, it is only usable enough to inspect its state in a debugger, or to generate a core file.” No one likes it when this happens. This is why backtrace.io is focused on being able to figure out why it is happening “A FreeBSD kernel core file can be formatted in several different ways. This depends on which type of dump was performed. Full core dumps are ELF files, similar in structure to userspace core files. However, as RAM size grew, this became more difficult to manage. In 2006, FreeBSD introduced minidumps, which are much smaller without making the core file useless. This has been the default dump type since FreeBSD 6.0.” The article goes into detail on the minidump format, and some basic debugging techniques “Libkvm will first determine whether the virtual address lies within the kernel or direct maps. If it lies in the kernel map, libkvm will consult the page table pages to discover the corresponding physical address. If it lies in the direct map, it can simply mask off the direct map base address. If neither of these applies, the address is illegal. This process is encapsulated by vatopa, or “virtual address to physical address”. Once the physical address is determined, libkvm consults the core file's bitmap to figure out where in the core file it is located.” “minidumps include a sparse bitmap indicating the pages that are included. These pages are dumped sequentially in the last section. Because they are sparse in a not entirely predictable way, figuring the offset into the dump for a particular physical address cannot be reduced to a trivial formula.” The article goes into detail about how lookups against this map are slow, and how they were improved “For typical manual debugger use, the impact of this change isn't noticeable, which is probably why the hash table implementation has been in use for 10 years. However, for any automated debugging process, the extra latency adds up quickly.” “On a sample 8GB kernel core file (generated on a 128GB server), crashinfo improves from 44 seconds to 9 seconds, and uses 30% less memory” “Backtrace began shipping a version of this performance improvement in ptrace in February 2016. This enables us to also offer significantly faster tracing of FreeBSD kernel cores to customers running current and older releases of FreeBSD. On July 17, 2016, our work improving libkvm scaling was committed to FreeBSD/head. It will ship with FreeBSD 12.0.” *** OpenBSD gunzip pipeline tightening (https://www.mail-archive.com/tech@openbsd.org/msg34035.html) OpenBSD has rethought the way they handle package signing Changing from: 1/ fetch data -> 2/ uncompress it -> 3/ check signature -> 4/ process data To: 1/ fetch data -> 2/ check signature -> 3/ uncompress -> 4/ process data “The solution is to move the signature outside of the gzip header” “Now, Since step 1/ is privsep, as long as step 2 is airtight, 3/ and 4/are no longer vulnerable” Guidelines: small, self-contained code to parse simple gzip headers signify-style signature in the gzip comment. Contains checksums of 64K blocks of the compressed archive don't even think about passing the original gzip header through use as a pipeline step: does not need to download full archive to use it, and never ever pass any data to the gunzip part before it's been verified. “Note that afaik we haven't had any hole in our gunzipping process. Well… waiting for an accident to happen is not how we do things. Hopefully, this should prevent future mishaps.” *** OpenVPN On FreeBSD 10.3 (http://ramsdenj.com/2016/07/25/openvpn-on-freebsd-10_3.html) “While trying to setup OpenVPN, I noticed there was no up-to-date information with correct instructions. OpenVPN uses EasyRSA to setup keys, it has recently been changed in version 3. As a result of this, the old steps to configure OpenVPN are no longer correct. I went through the process of setting up a VPN using OpenVPN on FreeBSD 10.3.” I know FreeBSD developer Adrian Chadd complained about this exact problem when he was trying to setup a VPN before attending DEFCON The tutorial walks through the basic steps: Install the needed software Configure EasyRSA Create a CA Generate keys and DH params OpenVPN Server Config OpenVPN Client Config Starting the daemon It even finishes off with bonus instructions on Port Forwarding, Firewalls, and Dynamic DNS *** lsop (https://github.com/606u/lsop) LSOP is the tool a bunch of users have been asking for “a FreeBSD utility to list all processes running with outdated binaries or shared libraries” How does it work? “lsop iterates over all running processes and looks through memory-mapped files with read + execute access; then it checks if those files are still available or have been modified/deleted.” How would you use it? After installing an system update (that doesn't require a reboot to update the kernel), or upgrade your packages, you still need to know which daemons need to be restarted to use the patched libraries and binaries This tool gives you that list Thanks to Bogdan Boyadzhiev for writing this much needed tool *** News Roundup OpenBSD 2016 Fundraising Campaign (http://www.openbsdfoundation.org/campaign2016.html) The OpenBSD fund-raising campaign has given us a status update on the state of 2016. They start by giving us a re-cap of previous years: “2015 was a good year for the foundation financially, with one platinum, one gold, four silver and 3 bronze donors providing half of our total donations. 680 individuals making smaller contributions provided the other half. While the total was down significantly after 2014's blockbuster year, we again exceeded our goal.” As of Sept 5th, they were at approx $115k out of a total goal of 250k. If you are an OpenBSD user, remember to contribute before the end of the year. Small amounts help, and the money of course goes to great causes such as hackathons and running the OpenBSD infrastructure. Update firewall Bad Countries (https://github.com/KaiLoi/update-fw-BC) Network and Systems admins know, sometimes when all else fails you need to break out the HUGE ban-hammer. In this case sometimes entire countries get put on the excrement list until the attacks stop. We have a handy GitHub project today, which will assist you in doing exactly that, enter update-fw-BC. (Update firewall by country) This perl script may be your savior when dealing with instances that require major brute force. It specifically works with IPFW, PF and IPTABLES, which will allow it to run across a variety of BSD's or even Linux. It will ingest a list of IP's that you feed it (perhaps from another tool such as sshguard) and determine what block the IP belongs to, and match according to country. Detailed setup instructions for the various firewalls are included, and some instructions for FreeBSD, although using it on OpenBSD or other $BSD should also be easy to adapt. *** More utilities via moreutils (https://distrowatch.com/weekly.php?issue=20160822#tips) In most BSDs, the “core” set of utilities and commands are just part of the base system, but on Linux, they are usually provided by the “coreutils” package. However, on Linux and now FreeBSD, there is a “moreutils” package, that provides a number of interesting additional basic utilities, including: chronic: Run a task via crontab, and only generate output if the task fails combine: binary AND two text files together, only displaying lines that are in both files errno: look up the text description of a specific error number ifdata: parse out specific information from ifconfig ifne: if-not-empty, only run a command if the output of the pipe is not blank isutf8: determine if a file or stdin contains utf8 lckdo: execute a command with a lock held, to prevent a second copy from spawning mispipe: return the exit code of the first command in a pipe chain, rather than the last parallel: run multiple jobs at once pee: tee standard input to multiple pipes sponge: write standard input to a file, allows you to overwrite a file in place: sort file | sponge file ts: add a timestamp to each line of standard input vidir: edit a directory in vi, great for bulk renames vipe: insert vi into a pipe, edit the content before it is passed to the next command zrun: uncompress the arguments before passing them. Like gzless and friends, but for any command Just goes to show the power of the original UNIX philosophy, chaining together a bunch of small useful tools to do really powerful things *** OpenBSD: SNI support added to libtls, httpd in –current (http://undeadly.org/cgi?action=article&sid=20160823100144) libtls, LibreSSL's improved API to replace the OpenSSL standard, now has a set of functions to implement SNI (Server Name Indication) Until a few years ago, each different SSL/TLS enabled website required a unique IP address, because typical HTTP Virtual Hosting (differentiating which content to serve based on the Host header in the HTTP request), didn't work because the request was encrypted. Finally the TLS standard was updated to include the hostname of the site the user is requesting in the TLS handshake, so the server can return the corresponding certificate, and multiple TLS enabled websites can be hosted on a single IP address The new API includes the ability to provide additional keypairs (via tlsconfigaddkeypair{file,mem}()) And allow the server to determine what servername the client requested viatlsconnservername() This is much easier to use, and therefore safer and less error prone, than the OpenSSL API The libtls API is used in a number of OpenBSD tools, including the httpd *** Beastie Bits Shawn Webb of HardenedBSD joins the OPNSense Core Team (https://opnsense.org/new-core-team-member/) How to install 2.11 BSD on a (simulated) PDP11 (http://vak.ru/doku.php/proj/pdp11/211bsd) OpenBSD Puffy needlepoint pixelart (https://nemessica.tintagel.pl/blog/OpenBSD-Puffy/) PulseAudio has been removed from dports (DragonFly BSD) (http://lists.dragonflybsd.org/pipermail/users/2016-August/313010.html) pfSense 2.4 pre-alpha available for testing, based on FreeBSD 11.0 (https://blog.pfsense.org/?p=2118) Call for Testing - Bhyve HDA Sound Emulation (https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-September/004700.html) *** Feedback/Questions Matthew - ZFS Hole Birth (http://pastebin.com/CrZiDAF0) Hunter - systemd-mount (http://pastebin.com/GztjY4wz) Anonymous - Cool'n'quiet (http://pastebin.com/gG4j4RCi) Nathan - Datacenter (http://pastebin.com/9XgPzMM9) Chuck - OpenBSD w/DO (http://pastebin.com/FM2xYcxh) ***
This week on BSDNow! We've got Netflix + FreeBSD news to discuss, always a crowd pleaser, that plus EuroBSDCon is just around the corner. Stick around for your place This episode was brought to you by Headlines Protecting Netflix Viewing Privacy at Scale, with FreeBSD (http://techblog.netflix.com/search/label/FreeBSD) This blog post from Netflix tells the story of how Netflix developed in-kernel TLS to speed up delivery of video via HTTPS Since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs - from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016. We contribute to this effort on the software side by optimizing every aspect of the software for our unique use case - in particular, focusing on the open source FreeBSD operating system and the NGINX web server that run on the OCAs. In the modern internet world, we have to focus not only on efficiency, but also security. There are many state-of-the-art security mechanisms in place at Netflix, including Transport Level Security (TLS) encryption of customer information, search queries, and other confidential data. We have always relied on pre-encoded Digital Rights Management (DRM) to secure our video streams. Over the past year, we've begun to use Secure HTTP (HTTP over TLS or HTTPS) to encrypt the transport of the video content as well. This helps protect member privacy, particularly when the network is insecure - ensuring that our members are safe from eavesdropping by anyone who might want to record their viewing habits. The goal is to ensure that your government, ISP, and wifi sniffing neighbour cannot tell which Netflix videos you are watching Netflix Open Connect serves over 125 million hours of content per day, all around the world. Given our scale, adding the overhead of TLS encryption calculations to our video stream transport had the potential to greatly reduce the efficiency of our global infrastructure. We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-GCM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously - as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method. All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance. We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L). Netflix and NGINX had previously worked together to improve our HTTP client request and response time via the use of sendfile calls to perform a zero-copy data flow from storage (HDD or SSD) to network socket, keeping the data in the kernel memory address space and relieving some of the CPU burden. The Netflix team specifically added the ability to make the sendfile calls asynchronous - further reducing the data path and enabling more simultaneous connections. However, TLS functionality, which requires the data to be passed to the application layer, was incompatible with the sendfile approach. To retain the benefits of the sendfile model while adding TLS functionality, we designed a hybrid TLS scheme whereby session management stays in the application space, but the bulk encryption is inserted into the sendfile data pipeline in the kernel. This extends sendfile to support encrypting data for TLS/SSL connections. We tested the BoringSSL and ISA-L AES-GCM implementations with our sendfile improvements against a baseline of OpenSSL (with no sendfile changes), under typical Netflix traffic conditions on three different OCA hardware types. Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline - increasing performance by up to 30%, depending on the OCA hardware version. We chose the ISA-L cipher implementation, which had slightly better results. With these improvements in place, we can continue the process of adding TLS to our video streams for clients that support it, without suffering prohibitive performance hits. If you would like more detail, check out the papers from AsiaBSDCon 2015 (https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf) and the updated one from 2016 (https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf) *** OpenBSD on HP Stream 7 (http://www.tedunangst.com/flak/post/OpenBSD-on-HP-Stream-7) Recent events have rocked the mobile computing world to its core. OpenBSD retired the zaurus port, leaving users in desperate need of a new device. And not long before that, Microsoft released the Anniversary Update to Windows 10, but with free space requirements such that it's nigh impossible to install on cheap 32GB eMMC equipped devices such as the HP Stream series, leaving users searching for a new lightweight operating system. With necessity as both mother and father, the scene is set for a truly epic pairing. OpenBSD on the HP Stream 7. The HP Stream line is a series of budget computers in a couple form factors. The Stream 11 is a fairly typical netbook. However, the Stream 7 and 8 are tablets. They look like cheap Android devices, but inside the case, they're real boys, er PCs, with Intel Atom CPUs. To install OpenBSD on such a device, we need a few parts. Obviously, the tablet itself. There's a dearth of ports on these things, but there is a micro USB port. Attaching anything useful requires an OTG “on the go” cable that creates a type A port. Attaching more than one useful thing requires a mini hub. And completing the install requires one each USB stick, keyboard, and network adapter. First, we need to prep the machine to boot from USB. Actually, before doing anything, make sure you have a full charge. It's going to be battery only from here on out. Plug everything in. Flash drive, keyboard, and network into the hub, hub into the OTG cable, cable into the port on top of the Stream. Turn on the machine while holding the volume down button. This launches a mini menu from which we can enter the BIOS. There's a little on screen keyboard in the corner, so this can be done even without a keyboard attached, but the USB keyboard should work. We need to change two settings in the boot section. First, turn off secure boot. Second, switch boot order to prefer USB. Save and exit. The first reboot reveals a confirmation screen checking that we really want to disable secure boot. We must enter a PIN and press enter. Enter the PIN shown on the screen and press enter. And we are go. Then boot up OpenBSD from the USB drive Ted then works there a number of kernel panics and device driver issues, but after disabling ACPI and IntelDRM, the device boots OpenBSD. Of course, there's no X at this point. And definitely no touch screen. And no internal networking. However, by keeping our USB hub attached, we can drive the console and access the network. At least until the battery is depleted, even if we have no way of knowing how long that will be since we disabled all the ACPI devices, which also means no suspend or resume. With some xorg.conf hacking, he did get Xorg working *** DragonflyBSD steps towards base LibreSSL (http://lists.dragonflybsd.org/pipermail/commits/2016-September/624493.html) Project: DragonFlyBSD / Switch base to use private LibreSSL libraries (http://freshbsd.org/commit/dfbsd/304ca408000cd34559ef5319b4b5a6766d6eb35b) DragonFly BSD adopts uses of LibreSSL (http://undeadly.org/cgi?action=article&sid=20160911231651) The number of projects beginning to switch over to LibreSSL is growing and it appears we can now throw DragonFly into that camp. Following something that sounds vaguely familiar (Allan!) DFLY is now creating “private” LibreSSL libraries which are only linked against by base system binaries. For the moment OpenSSL is still built, primarily so that various ports and 3rd party apps can continue to function as before. A NO_OPENSSL option has also been added, but doesn't really do much (yet), since it'll still build and install headers / libraries even if set. *** OpenBSD g2k16 Hackathon g2k16 Hackathon Report: Antoine Jacoutot on Binary Patches (http://undeadly.org/cgi?action=article&sid=20160911012316) g2k16 Hackathon Report: Matthieu Herrb on xenodm (http://undeadly.org/cgi?action=article&sid=20160911231712) g2k16 Hackathon Report: Vincent Gross on iked(8), armv7 and sys/netinet[6] (http://undeadly.org/cgi?action=article&sid=20160911000337) g2k16 Hackathon Report: Florian Obser on httpd, networking, acme-client, and more (http://undeadly.org/cgi?action=article&sid=20160911000052) g2k16 Hackathon Report: Jasper Lievisse Adriaanse on ddb(4) and more (http://undeadly.org/cgi?action=article&sid=20160909012520) g2k16 Hackathon Report: Christian Weisgerber on gettext progress, RTC work, removing kernel cruft (http://undeadly.org/cgi?action=article&sid=20160908002430) g2k16 Hackathon Report: Brent Cook on Chromebooks, crypto, and more (http://undeadly.org/cgi?action=article&sid=20160907131655) g2k16 Hackathon Report: Ted Unangst on doas, signify, code removal (http://undeadly.org/cgi?action=article&sid=20160906230610) g2k16 Hackathon Report: Marc Espie on package signing evolution (http://undeadly.org/cgi?action=article&sid=20160905235911) g2k16 Hackathon Report: Adam Wolk on ports, wireless drivers and more (http://undeadly.org/cgi?action=article&sid=20160906004915) g2k16 Hackathon Report: Mike Larkin on vmm + vmd progress (http://undeadly.org/cgi?action=article&sid=20160905134009&mode=expanded) *** News Roundup OpenBSD (with encrypted softraid) on the Chromebook Pixel (https://jcs.org/notaweblog/2016/08/26/openbsd_chromebook/) Looking for a Laptop to make your OpenBSD road-warrior? If so, we have a great blog tutorial on getting OpenBSD setup on the Chromebook Pixel with encrypted softraid! Author Joshua Stein gives us a very verbose look at how to install and dial-in the laptop perfectly. But first for those wondering about the hardware in the pixel: The Chromebook Pixel LS (2015) has an Intel Core i7 processor (Broadwell) at 2.4Ghz, 16Gb of RAM, a 2560x1700 400-nit IPS screen (239ppi), and Intel 802.11ac wireless. It has a Kingston 64Gib flash chip, of which about 54Gib can be used by OpenBSD when dual-booting with a 1Gb Chrome OS partition. Due to this being a chromebook with seaBIOS, some manual key-press trickery will be required to initially get the OpenBSD Installer up and running. From here you'll want to pay special close attention to the disk partitioning. In particular Joshua will show us how to shrink the existing encrypted /home that ChromeOS uses, keeping the dual-boot intact. This will become important if you ever plan on updating the device. From here, we move back to a more traditional setup, but with the added bonus of doing a soft-raid setup. But the fun isn't over yet! If you want to make OpenBSD the default boot, that'll require cracking the lid on the device and removing a special pink write-protect screw. And of course if you want to remove the default splash-screen image, Joshua has you covered as well, although some flashrom magic will be required. At this point you are nearly done. Final details on enabling specific bits of hardware are discussed. Most things work, apart from Audio and Bluetooth as of right now. *** doas mastery (http://www.tedunangst.com/flak/post/doas-mastery) “doas” mastery - Paging MWL! Our buddy Ted Unangst has written up a great ‘mastery' guide of the doas command, which can come in handy if you are among the un-initiated in doas land. UNIX systems have two classes of user, the super user and regular users. The super user is super, and everybody else is not. This concentration of power keeps things simple, but also means that often too much power is granted. Usually we only need super user powers to perform one task. We would rather not have such power all the time. Think of the responsibility that would entail! Like the sudo command, doas allows for subdivision of super user privileges, granting them only for specific tasks. He starts with the basic doas.conf setup, which starts with an empty config file The doas config is much like a pf ruleset, the default is to block everything > We add the root rule second because doas evaluates rules in a last match manner. root is in the wheel group, so the first rule will match, and then we need to override that with a second rule. Remember to always start with general rules, then make them more specific. *** iXsystems iXsystems to host MeetBSD (https://www.ixsystems.com/blog/ixsystems-host-meetbsd-california-2016-uc-berkeley/) FreeBSD Foundation Welcomes New Board Members New Board Members (https://www.freebsdfoundation.org/blog/freebsd-foundation-welcomes-new-board-members/) The FreeBSD Foundation has added two new board members Interview with Kylie Liang (https://www.freebsdfoundation.org/blog/new-board-member-interview-kylie-liang/) Kylie will focus on representing FreeBSD at conferences and businesses in China I live in China. There, I can act as a bridge between Chinese companies and the FreeBSD community to help drive FreeBSD adoption. Through my leadership role in the FreeBSD Foundation, I will help promote FreeBSD in China and also represent the Foundation at conferences and events in my region. Kylie leads the team the ensures FreeBSD runs well on Hyper-V and Azure, including providing commercial support for customers who run FreeBSD or FreeBSD based appliances on the Azure Cloud I joined Microsoft and started to lead the project called FreeBSD Integration Service to get FreeBSD running well on Hyper-V and Azure. To promote our work and to understand the FreeBSD ecosystem, I started to participate in FreeBSD events where I was inspired by this technical community. Interview with Philip Paeps (https://www.freebsdfoundation.org/blog/new-board-member-interview-philip-paeps/) Philip started with FreeBSD in the early 2000s and got his commit bit in 2004 The patches I submitted to make ACPI and input devices work on that laptop led to a src commit bit in 2004. While I haven't worked on ACPI or input devices since, I have been contributing to different areas of the kernel. Taking up maintainership of some ports I cared about also got me a ports commit bit after some time. Philip will continue to help run EuroBSDCon, but is also spreading the word about FreeBSD in India and Africa Primarily, I think I can be useful! I attend (and organize) a number of conferences around the world every year, particularly in regions that have a mostly “stealthy” FreeBSD community. While I clearly don't need to be on the FreeBSD Foundation board to advocate for FreeBSD, joining as a director will provide an additional asset when working in areas of the world where organizational affiliations are meaningful. Philip has also developed network drivers and various other bits and pieces, and has extensive experience working with and for hardware vendors and appliance vendors Despite intending to eventually contribute their code to the FreeBSD Project as open source, many hardware vendors still find it very difficult to engage directly with the FreeBSD development community. The Foundation helps bridge that gap and helps facilitate collaboration between commercial vendors and the FreeBSD community. I hope to make FreeBSD more visible in regions of the world where it is historically under-represented. I expect I will be attending even more conferences and getting myself invited to even more organizations. more, less, and a story of typical Unix fossilization (https://utcc.utoronto.ca/~cks/space/blog/unix/MoreAndUnixFossilization) Chris Siebenmann from the University of Toronto digs into the history of the difference between ‘less' and ‘more' In the beginning, by which we mean V7, Unix didn't have a pager at all. That was okay; Unix wasn't very visual in those days, partly because it was still sort of the era of the hard copy terminal. Then along came Berkeley and BSD. People at Berkeley were into CRT terminals, and so BSD Unix gave us things like vi and the first pager program, more (which showed up quite early, in 3BSD, although this isn't as early as vi, which appears in 2BSD). Calling a pager more is a little bit odd but it's a Unix type of name and from the beginning more prompted you with '--More--' at the bottom of the screen. All of the Unix vendors that based their work on BSD Unix (like Sun and DEC) naturally shipped versions of more along with the rest of the BSD programs, and so more spread around the BSD side of things. However, more was by no means the best pager ever; as you might expect, it was actually a bit primitive and lacking in features. So fairly early on Mark Nudelman wrote a pager with somewhat more features and it wound up being called less as somewhat of a joke. In a sane world, Unix vendors would have either replaced their version of more with the clearly superior less or at least updated their version of more to the 4.3 BSD version. Maybe less wouldn't have replaced more immediately, but certainly over say the next five years, when it kept on being better and most people kept preferring it when they had a choice.” + “This entire history has led to a series of vaguely absurd outcomes on various modern Unixes. On Solaris derivatives more is of course the traditional version with source code that can probably trace itself all the way back to 3BSD, carefully updated to SUS compliance. Solaris would never dream of changing what more is, not even if the replacement is better. Why, it might disturb someone. Oddly, FreeBSD has done the most sensible thing; they've outright replaced more with less. There is a /usr/bin/more but it's the same binary as less and as you can see the more manpage is just the less manpage. OpenBSD has done the same thing but has a specific manpage for more instead of just giving you the less manpage. So, now you can see why I say that less is more, or more, or both, at several levels. less is certainly more than more, and sometimes less literally is more (or rather more is less, to put it the right way around). Beastie Bits PC-BSD listed in the top 8 'best' alternatives to Windows 10 (http://www.computerworlduk.com/galleries/operating-systems/-free-alternatives-windows-10-3639433/) Creating a quick DNS server with a Rapsberry Pi2 and FreeBSD 11.0-RC1 (http://bsdimp.blogspot.co.uk/2016/08/creating-quick-dns-server-with.html) Dual Boot OpenBSD and Linux + UEFI (https://bsdlaptops.wordpress.com/2016/03/07/vaio-pro-11-part-2/) DesktopBSD 2.0 various versions available (Gnome, Lumina, KDE, LXDE) (http://desktopbsd.boards.net/board/10/announcements) FreeBSD gets new ZFS features including: Compressed ARC (https://svnweb.freebsd.org/base?view=revision&revision=305323) and ZFS Allocation Throttle (https://svnweb.freebsd.org/base?view=revision&revision=305331) One Floppy NetBSD Distribution (https://github.com/user340/fdgw2) A Compendium of BUGs (https://github.com/q5sys/BUGtracker) Feedback/Questions Galahad - OpenBSD X setup (http://pastebin.com/b7W6NHqs) Tang - Subtitles (http://pastebin.com/P4MUs3Pa) Ivan - Zpool Options (http://pastebin.com/LQ8yTp0G) Brad - Replication Issue (http://pastebin.com/XTK5gXMU) MJ - HBA (http://pastebin.com/TdYTMSj9) ***
This week on BSDNow, we have a variety of news to discuss, covering quite the spectrum of BSD. (Including a new DragonFly release!). This episode was brought to you by Headlines my int is too big (http://www.tedunangst.com/flak/post/my-int-is-too-big) “The NCC Group report (http://marc.info/?l=oss-security&m=146853062403622&w=2) describes the bugs, but not the history of the code.” “Several of them, as reported by NCC, involved similar integer truncation issues. Actually, they involved very similar modern 64 bit code meeting classic 32 bit code” “The thrsleep system call is a part of the kernel code that supports threads. As the name implies, it gives userland a measure of control over scheduling and lets a thread sleep until something happens. As such, it takes a timeout in the form of a timespec. The kernel, however, internally implements time keeping using ticks (there are HZ, 100, ticks per second). The tsleep function (t is for timed) takes an int number of ticks and performs basic validation by checking that it's not negative. A negative timeout would indicate that the caller has miscalculated. The kernel panics so you can fix the bug, instead of stalling forever.” “The trouble therefore is when userland is allowed to specify a timeout that could be negative. The existing code made an attempt to handle various tricks by converting the timespec to a ticks value stored as a 64 bit long long which was checked against INTMAX before passing to sleep. Any value over INTMAX would be truncated, so we can't allow that. Instead, we saturate the value to INT_MAX. Unfortunately, this check didn't account for the possibility that the tick conversion from the timespec could also overflow and result in a negative value.” Then there is the description of the kqueue flaw: “Every kqueue keeps a list of all the attached events it's watching for. A simple array is used to store file events, indexed by fd.” “This array is scaled to accommodate the largest fd that needs to be stored. This would obviously cause trouble, consuming too much memory, if the identifier were not validated first. Which is exactly what kqueue tries to do. The fdgetfile function checks that the identifier is a file that the process has open. One wrinkle. fdgetfile takes an int argument but ident is a uintptr_t, possibly 64 bits. An ident of 2^32 + 2 will look like a valid file descriptor, but then cause the array to be resized to gargantuan proportions.” “Again, the fix is pretty simple. We must check that the ident is bounded by INTMAX before calling fdgetfile. This bug likely would have been exploitable beyond a panic, but the array allocation was changed to use mallocarray instead of multiplying arguments by hand, thus preventing another overflow.” Then there is a description of the anonymous mmap flaw, and the “secret magic” _MAPNOFAULT flag *** FreeBSD Quarterly Status Report Q2 2016 (https://www.freebsd.org/news/status/report-2016-04-2016-06.html) It's time for another round of FreeBSD Quarterly Status Reports! In this edition, we have status updates from the various teams, including IRC/Bugs/RE/Ports/Core and Foundation We also have updates on some specific projects, including from Konstantin on the on-going work for his implementation of ASLR, including the new ‘proccontrol' command which provides the following: > “The proccontrol(1) utility was written to manage and query ASLR enforcement on a per-process basis. It is required for analyzing ASLR failures in specific programs. This utility leverages the procctl(2) interface which was added to the previous version of the patch, with some bug fixes.” Next are updates on porting CEPH to FreeBSD, the ongoing work to improve EFI+GELI (touched on last week) and more robust Mutexes. Additionally we have an update from Matt Macy and the Xorg team discussing the current work to update FreeBSD's graphic stack: > “All Intel GPUs up to and including the unreleased Kaby Lake are supported. The xf86-video-intel driver will be updated soon. Updating this driver requires updating Xorg, which in turn is blocked on Nvidia updates.” The kernel also got some feature status updates, including on the new Allwinner SoC support, an update on FreeBSD in Hyper-V and VIMAGE In addition to a quick update on the arm64 architecture (It's getting there, RPi3 is almost a thing), we also have a slew of port updates, including support for GitLab in ports, updates on GNOME / KDE and some additional Intel-specific networking tools. *** Vulnerabilities discovered in freebsd-update and portsnap (https://lists.freebsd.org/pipermail/freebsd-security/2016-July/009016.html) There are two vulnerabilities discovered in freebsd-update and portsnap, where an attacker could place files in the portsnap directory and they would be used without being subject to having their checksum verified (but this requires root access), and the second where a man-in-the-middle attacker could guess the name of a file you will fetch by exploiting the time-gap between when you download the initial snapshot, and when you fetch the updated files. There are a number of vulnerabilities that were discovered in libarchive/tar as well There is also an issue with bspatch. A security advisory for bspatch has already been released, as this vulnerabilities was also discovered by the Chromium team, which uses this same code. The patch discussed in this mailing list thread is larger, but secteam@ believes at least one of the additional checks introduced is incorrect and may prevent a valid patch from being applied. The smaller patch was pushed out first, to solve the main attack vector, while the larger patch is investigated. Automated fuzz testing is underway. Great care is being taken fixing bspatch, as if it is broken installing future updates becomes much more difficult secteam@ and core@ would like to emphasize that the FreeBSD project takes these issue very seriously and are working on it > “As a general rule, secteam@ does not announce vulnerabilities for which we don't have patches, but we concede that we should have considered making an exception in this case” Work is underway to re-architect freebsd-update and portsnap to do signature verification on all files before they are passed to libarchive/tar, to help protect users from any future vulnerabilities in libarchive. However, this requires changes to the metadata format to provide these additional signatures, and backwards compatibilities must be preserved, so people can update to the newer versions to get these additional security features There is also discussion of using HTTPS for delivery of the files, but certificate verification and trust are always an issue. FreeBSD does not distribute a certificate trust store by default. There will be more on this in the coming days. *** OpenSSH 7.3 Released (http://www.openssh.com/txt/release-7.3) OpenSSH 7.3 has landed! Primarily a bug-fix release, the release notes do mention the pending deprecation of some more legacy Crypto in the future, including denying all RSA keys < 1024bit, and removal of SSHv1 support. (Already disabled via compile option) On the bug side, there was a security issue addressed in sshd: “sshd(8): Mitigate a potential denial-of-service attack against the system's crypt(3) function via sshd(8). An attacker could send very long passwords that would cause excessive CPU use in crypt(3). sshd(8) now refuses to accept password authentication requests of length greater than 1024 characters” Also a timing issue was resolved in regard to password auth, which could possibly allow an attacker to discern between valid/invalid account names. On the feature side, we have the new ProxyJump option (-J flag) which allows you to do simplified indirection through various SSH jump hosts. Various bugs were fixed, and some compile failures resolved in the portable version to auto-disable some ciphers not supported by OpenSSL. News Roundup OpenBSD Ports - Integrating Third Party Applications [pdf] (http://jggimi.homeip.net/semibug.pdf) A talk from Josh Grosse, presented at SEMIBUG (South-East Michigan BSD Users Group), about OpenBSD Ports It opens by explaining the separation of the ‘base system' from ‘packages', as is common in most all BSDs It explains the contents of OpenBSD package tar file, which contain some metadata files (+CONTENTS and +DESC) and then the actual package files The talk goes on to explain the different branches (-release, -stable, and -current), and warn users that there are no official -stable packages from the project Then it goes on into the development model, including what new contributors should expect Then it walks through the entire process of creating a port and getting it contributed *** NetBSD removes last RWX page in amd64 kernel (http://mail-index.netbsd.org/source-changes/2016/07/27/msg076413.html) NetBSD has purged the last holdout RWX page on the amd64 platform > “Use UVMPROTALL only if UVMKMFEXEC is given as argument. Otherwise, if UVMKMFPAGEABLE is also given as argument, only the VA is allocated and UVM waits for the page to fault before kentering it. When kentering it, it will use the UVMPROT flag that was passed to uvm_map; which means that it will kenter it as RWX. With this change, the number of RWX pages in the amd64 kernel reaches strictly zero.” Break out the party favors! Hopefully any last stragglers in any of the other BSD's gets retired soon as well. *** DragonFly BSD 4.6 launches with home-grown support for NVMe Controllers (http://linux.softpedia.com/blog/dragonfly-bsd-4-6-0-launches-with-home-grown-support-for-nvme-controllers-506908.shtml) Softpedia picked up on the release of DragonFlyBSD 4.6, specifically about their new home-grown NVMe driver. > “We now have a NVMe driver (PCIe SSDs). It currently must be kldloaded with nvme_load="YES" in /boot/loader.conf. The driver uses all concurrency features offered by the chip and will distribute queues and interrupts across multiple CPUs to maximize performance. It has been tested up to around 1.05M IOPS @4K, and roughly 6.5 GBytes/sec @32K (random read from urandom-filled partition, physio, many threads), with the 2xE5-2620v4 (xeon) test server 78% idle in the IOPS test and 72% idle on the bandwidth test. In other words, we maxed out the three NVMe devices we had plugged in and the system still had plenty of suds left over. Please note that a machine's ability to boot from an NVMe device depends on the BIOS, and not DragonFly. Most BIOSes cannot boot from NVMe devices and those that can probably only do it through UEFI. Info on device state is available with the new utility nvmectl.“ In addition to this improved support, 4.6 also brings in the improved graphics support, matching what is in Linux 4.4 and support for Broadwell/Skylake. SMP also got some love: > “SMP performance was already very good. As part of the NVMe driver work we revamped the buffer cache subsystem and a number of other I/O related paths, further reducing lock contention and IPI signalling overheads. We also put topology-aware cpu cache localization into the kernel memory allocator (primarily helps multi-socket systems and systems with high core counts). The network subsystem also continues to receive significant improvement, with modest machine configurations now capable of handling upwards of 580K conns/sec.“ +Full Release Notes (https://www.dragonflybsd.org/release46/) *** The powerd++ daemon monitors the system load and adjusts the CPU clock accordingly and is a drop-in replacement for FreeBSD's native powerd(8). (http://www.freshports.org/sysutils/powerdxx/) As mentioned in our EuroBSDCon 2016 rundown, Dominic Fandrey will be giving a presentation about his powerd replacement, powerd++ The source code is already available on github, and is in ports The major difference is the newer design handle many-core systems much better. The original powerd was written at a time when most laptops only had a single core, and maybe a hyperthread. The new design decides which CPU frequency to use by looking at the busiest core, rather than the average across the cores, resulting in a more meaningful result. It also supports averaging over a longer period of time, to avoid jumping to a higher frequency to quickly powerd++ also avoids ‘slewing' the cpu frequency, ratching it up and down one step at a time, and instead jumps directly to the target frequency. Often times, you will use less battery by jumping to maximum frequency, finishing the work, and going back to a low power state, than trying to do that work over a longer period of time in low power mode *** Beastie Bits Hyper-V: Unmapped I/O improves userland direct disk performance by 35% ~ 135% (https://svnweb.freebsd.org/base?view=revision&revision=303474) One does not simply remove FreeBSD (https://imgur.com/a/gjGoq) A new BSD Podcast "BSD Synergy" has started (https://www.youtube.com/channel/UCBua6yMtJ6W5ExYSREnS3UQ) KnoxBug - Next Meeting - Aug 30th (http://knoxbug.org/content/2016-08-30) Feedback/Questions Daniel - Root/Wheel (http://pastebin.com/8sMyKm6c) Joe - IPV6 Frag (http://pastebin.com/r5Y0gbxf) Paul - ChicagoBug (http://pastebin.com/iVYPYcVs) Chris - SSH BruteBlock (http://pastebin.com/597m9gHa) Todd - Jails (http://pastebin.com/xjbKwSaz) ***
This week on BSDNow, we have some big breaking news about another major switcher to FreeBSD, plus early information about the pending This episode was brought to you by Headlines Leo Laporte tries FreeBSD (http://www.leolaporte.com/blog/a-grand-experiment) Leo Laporte, formerly of TechTV, and now of TWiT.tv, is switching to FreeBSD “The latest debacle over the "forced" upgrade to Windows 10 and Apple's increasingly locked-in ecosystem has got me thinking. Do I really need to use a proprietary operating system to get work done? And while I'm at it, do I need to use commercial cloud services to store my data?” A sometimes Linux user since the mid 90s, Leo talks about his motivations: “But as time went by, even Ubuntu began to seem too commercial to me” “So now for the grand experiment. Is it possible, I wonder, to do everything I need to do on an even more venerable, more robust system: a true UNIX OS, FreeBSD? Here are my requirements” Browsing Email with PGP signing and encryption Coding - I'm a hobbyist programmer requiring support for lisp/scheme/racket, rust, and python (and maybe forth and clojure and meteor and whatever else is cool and new) Writing A password vault. I currently use Lastpass because it syncs with mobile but eventually I'll need to find a FOSS replacement for that, too Photo editing - this is the toughest to replace. I love Photoshop and Lightroom. Can I get by with, say, GIMP and Darktable? I do all of those things on my PCBSD machine all the time “I love Linux and will continue to use it on my laptops, but for my main workhorse desktop I think FreeBSD will be a better choice. I also look forward to learning and administering a true UNIX system.” He got a nice SuperMicro based workstation, with an Intel Xeon E3-1275v5 and an NVIDIA GeForce GTX 960 GPU I have a server with one of those Skylake E3s, it is very nice “450Mbps Wireless N Dual Band PCI-e Adapter w/ 3x 2dBi Antennas (Yes, sad to say, unless I rewire my house I'll have to use Wi-Fi with this beast. I'll probably rewire my house.)” He plans to have a 4x 1TB ZFS pool, plus a second pool backed by a 512 GB NVMe m.2 for the OS “And I'll continue to chronicle my journey into the land of FOSS here when The Beast arrives. But in the meantime, please excuse me, I've got some reading to do.” Leo went so far as to slap a “Power By FreeBSD” sticker (https://youtu.be/vNVst_rxxm0?t=270) on the back of his new Tesla *** OpenBSD 6.0 to be released on Sept 1st, 2016 (http://undeadly.org/cgi?action=article&sid=20160725100831) OpenBSD 6.0 Tenative Released Notes (https://www.openbsd.org/60.html) OpenBSD 6.0 is just around the corner, currently slated for Sept 1st and brings with it a whole slew of exciting new features First up, and let's get this right out of the way.. VAX support has been dropped!! Oh no! However to make up for this devastating loss, armv7 has been added to this release. The tentative release notes are very complete and marks 6.0 as quite an exciting release OpenBSD 6.0 Pre-orders up (http://undeadly.org/cgi?action=article&sid=20160726230851) OpenBSD 6.0 tightens security by losing Linux compatibility (http://www.infoworld.com/article/3099038/open-source-tools/openbsd-60-tightens-security-by-losing-linux-compatibility.html) In related news, infoworld picked up on the pending removal of Linux compat from OpenBSD 6.0. Touted as a security feature, you will soon be unable to run legacy linux binaries on OpenBSD. This has both positives and negatives depending upon your use case. Ironically we're excitedly awaiting improved Linux Compat support in FreeBSD, to allow running some various closed-source applications. (Netflix DRM, Steam, Skype to name a few) *** EuroBSDCon 2016 Schedule released (https://2016.eurobsdcon.org/talks-schedule/) EuroBSDCon 2016 Tutorial Schedule released (https://2016.eurobsdcon.org/tutorials/) EuroBSDCon has announced the list of talks and tutorials for September 22nd-25th's conference! George Neville Neil (Who we've interviewed in the past) is giving the keynote about “The Coming Decades of BSD” *** News Roundup Blast from the past No interview again this week, we're working on getting some people lined up. The Leo Laporte story brought these old gem from TechTV into my youtube playlist: Matt Olander and Murrey Stokey explain FreeBSD on TechTV (https://www.youtube.com/watch?v=d0UsXwRvaIg) Matt Olander and Brooks Davis explain building a cluster with FreeBSD on TechTV (https://www.youtube.com/watch?v=bAsYz5pVwyc) FreeBSD vs Linux Part 1 (https://www.youtube.com/watch?v=91igg2UX7o8) FreeBSD vs Linux Part 2 (https://www.youtube.com/watch?v=oU88fQkwfws) *** Running FreeBSD on the LibreM (https://ericmccorkleblog.wordpress.com/2016/07/16/freebsd-librem-update/) Eric McCorkle (Who has worked on the EFI loader for a while now) has written an update on his efforts to get FreeBSD working properly on the LibreM 13 laptop. Since April the work seems to be progressing nicely Matt Macy's i915 graphics patch works well on the Librem 13, and I personally made sure that the suspend/resume support works. The patch is very stable on the Librem, and I've only had one kernel panic the entire time testing it. The HDMI output Just Works™ with the i915 driver. Even better, it works for both X11 and console modes. Full support for the Atheros 9462 card has been merged in. I've had some occasional issues, but it works for the most part. The vesa weirdness is obviated by i915 support, but it was resolved by using the scfb driver. Some of the outstanding issues still being worked on are support for Synaptics on this particular touchpad, as well as hotkey support for the keyboard, and brightness controls. In addition Eric is still working on the EFI + Geli support, with the eventual goal of getting EFI secure-boot working out of box as well. More OpenBSD syscall fuzzing (http://seclists.org/oss-sec/2016/q3/157) NCC Group's Project Triforce continues its work of fuzzing OpenBSD This time they have found a flaw that allows any user to panic the kernel Attempting to read from the tmpfs_vfsops sysctl tree will panic the system: “attempt to execute user address 0x0 in supervisor mode” This is actually a “good” thing… “Impact: Any user can panic the kernel by using the sysctl call. If a user can manage to map a page at address zero, they may be able to gain kernel code execution and escalate privileges” OpenBSD's default configuration prevents mapping a page at address zero, so the code execution is prevented So while a panic is a bad outcome, it is a lot better than it could have been *** Root privilege escalation on NetBSD (http://akat1.pl/?id=2) This post described a root privilege escalation in NetBSD mail.local is a utility included in the base system for delivering mail to other users on the same system, rather than invoking a mail client and going through the mail server. The mail.local utility contains a ‘time of check / time of use' vulnerability. This means that it checks if a file or permission is valid, and then later accesses that file. If an attacker can change that file between the time when it is checked, and the time when it is used, they may be able to exploit the system by evading the check This is exactly what happens in this case mail.local appends a message to the indicated user's mailbox It first checks if the target user already has an existing mailbox file. If the file exists, but is a link, mail.local exits with an error (to prevent exploits) If the file does not exist, it is created The message is then appended to the file If the file needed to be created, it is chown'd to the owner of the mailbox This is where the problem lies, if mail.local checks and does not find the mailbox, but an attacker then creates a link from the target mailbox to some other file mail.local then appends to that file instead, thinking it is creating the new mailbox Then, mail.local chown's the target file to the user the attacker was trying to send mail to The article explains how this could be used to replace /etc/master.passwd etc, but opts for an easier proof of concept, replacing /usr/bin/atrun, which is run as root every 5 minutes from crontab with a script that will copy the shell to /tmp and mark it setuid The attacker can then run that shell out of /tmp, and be root NetBSD fixed the vulnerability by changing the code flow, separating the cases for opening an existing file from creating a new file. In the case where an existing file is opened, the code then verifies that the file that was opened has the same inode number and is on the same device, as the file that was checked earlier, to ensure it was not a link *** FreeBSD Heap vulnerability in bspatch (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:25.bspatch.asc) An important vuln has been found and fixed in FreeBSD this past week, specifically relating to the ‘bspatch' utility. “Upstream's bspatch.c implementation doesn't check for negative values on the number of bytes to read from the "diff" and "extra" streams, allowing an attacker controlling the patch file to write at arbitrary locations in the heap.” This could result in a crash, or running arbitrary code as the user running bspatch. (Often root) “bspatch's main loop reads three numbers from the "control" stream in the patch: X, Y and Z. The first two are the number of bytes to read from "diff" and "extra" (and thus only non-negative), while the third one could be positive or negative and moves the oldpos pointer on the source image. These 3 values are 64bits signed ints (encoded somehow on the file) that are later passed the function that reads from the streams, but those values are not verified to be non-negative.” “Chrome[OS] has four different implementations of this program, all derived from the same original code by Colin Percival.” Chromium Issue Tracker (https://bugs.chromium.org/p/chromium/issues/detail?id=372525) Patch your systems now! *** Beastie Bits: If you're a BUG member or Organizer, please contact BSD Now (https://twitter.com/q5sys/status/758087886927388673) TedU writes about some interesting localizations to gcc in openbsd, and why they are there (http://www.tedunangst.com/flak/post/one-reason-to-hate-openbsd) List of Products based on FreeBSD -- Help complete the list (https://en.wikipedia.org/wiki/List_of_products_based_on_FreeBSD) Virtualbox v5 hits the FreeBSD Ports tree (http://www.freshports.org/emulators/virtualbox-ose/) Skull Canyon NUC booting FreeBSD 11.0-BETA2 (https://gist.github.com/gonzopancho/b71be467f45594822131f4816d6cb718) 2016 BSDCan Trip Report : Trent Thompson (https://www.freebsdfoundation.org/blog/2016-bsdcan-trip-report-trent-thompson/) August London BSD Meetup (http://mail-index.netbsd.org/regional-london/2016/07/25/msg000542.html) Feedback/Questions Michael Open-Source Alts (http://pastebin.com/eiWbDXTd) Herminio - AP Troubles (http://pastebin.com/w9aCDBut) Jake - Plasma (http://pastebin.com/d15QpVFw) Morgan - Clean DO Droplets (http://pastebin.com/Wj1P7jq8) Chris - Auditd (http://pastebin.com/U9PYEH6K) ***
This week on BSDNow, we have an interview with Matthew Macy, who has some exciting news to share with us regarding the state of graphics This episode was brought to you by Headlines How the number of states affects pf's performance of FreeBSD (http://blog.cochard.me/2016/05/playing-with-freebsd-packet-filter.html) Our friend Olivier of FreeNAS and BSDRP fame has an interesting blog post this week detailing his unique issue with finding a firewall that can handle upwards of 4 million state table entries. He begins in the article with benchmarking the defaults, since without that we don't have a framework to compare the later results. All done on his Netgate RCC-VE 4860 (4 cores ATOM C2558, 8GB RAM) under FreeBSD 10.3. “We notice a little performance impact when we reach the default 10K state table limit: From 413Kpps with 128 states in-used, it lower to 372Kpps.” With the initial benchmarks done and graphed, he then starts the tuning process by adjusting the “net.pf.states_hashsize”sysctl, and then playing with the number of states for the firewall to keep. “For the next bench, the number of flow will be fixed for generating 9800 pf state entries, but I will try different value of pf.states_hashsize until the maximum allowed on my 8GB RAM server (still with the default max states of 10k):” Then he cranks it up to 4 million states “There is only 12% performance penalty between pf 128 pf states and 4 million pf states.” “With 10M state, pf performance lower to 362Kpps: Still only 12% lower performance than with only 128 states” He then looks at what this does of pfsync, the protocol to sync the state table between two redundant pf firewalls Conclusions: There need to be a linear relationship between the pf hard-limit of states and the pf.stateshashsize; RAM needed for pf.stateshashsize = pf.stateshashsize * 80 Byte and pf.stateshashsize should be a power of 2 (from the manual page); Even small hardware can manage large number of sessions (it's a matter of RAM), but under too lot's of pressure pfsync will suffer. Introducing the BCHS Stack = BSD, C, httpd, SQLite (http://www.learnbchs.org/) Pronounced Beaches “It's a hipster-free, open source software stack for web applications” “Don't just write C. Write portable and secure C.” “Get to know your security tools. OpenBSD has systrace(4) and pledge(2). FreeBSD has capsicum(4).” “Statically scan your binary with LLVM” and “Run your application under valgrind” “Don't forget: BSD is a community of professionals. Go to conferences (EuroBSDCon, AsiaBSDCon, BSDCan, etc.)” This seems like a really interesting project, we'll have to get Kristaps Dzonsons back on the show to talk about it *** Installing OpenBSD's httpd server, MariaDB, PHP 5.6 on OpenBSD 5.9 (https://www.rootbsd.net/kb/339/Installing-OpenBSDandsharp039s-httpd-server-MariaDB-PHP-56-on-OpenBSD-59.html) Looking to deploy your next web-stack on OpenBSD 5.9? If so this next article from rootbsd.net is for you. Specifically it will walk you through the process of getting OpenBSD's own httpd server up and running, followed by MariaDB and PHP 5.6. Most of the setup is pretty straight-forward, the httpd syntax may be different to you, if this is your first time trying it out. Once the various packages are installed / configured, the rest of the tutorial will be easy, walking you through the standard hello world PHP script, and enabling the services to run at reboot. A good article for those wanting to start hosting PHP/DB content (wordpress anyone?) on your OpenBSD system. *** The infrastructure behind Varnish (https://www.varnish-cache.org/news/20160425_website.html) Dogfooding. It's a term you hear often in the software community, which essentially means to “Run your own stuff”. Today we have an article by PKH over at varnish-cache, talking about what that means to them. Specifically, they recently went through a website upgrade, which will enable them to run more of their own stuff. He has a great quote on what OS they use:“So, dogfood: Obviously FreeBSD. Apart from the obvious reason that I wrote a lot of FreeBSD and can get world-class support by bugging my buddies about it, there are two equally serious reasons for the Varnish Project to run on FreeBSD: Dogfood and jails.Varnish Cache is not “software for Linux”, it is software for any competent UNIX-like operating system, and FreeBSD is our primary “keep us honest about this” platform.“ He then goes through the process of explaining how they would setup a new Varnish-cache website, or upgrade it. All together a great read, and if you are one of the admin-types, you really should pay attention to how they build from the ground up. Some valuable knowledge here which every admin should try to replicate. I can not reiterate the value of having your config files in a private source control repo strongly enough The biggest take-away is: “And by doing it this way, I know it will work next time also.” *** Interview - Matt Macy - mmacy@nextbsd.org (mailto:mmacy@nextbsd.org)Graphics Stack Update (https://lists.freebsd.org/pipermail/freebsd-x11/2016-May/017560.html) News Roundup Followup on packaging base with pkg(8) (https://lists.freebsd.org/pipermail/freebsd-pkgbase/2016-May/000238.html) In spite of the heroic last minute effort by a team of contributors, pkg'd base will not be ready in time for FreeBSD 11.0 There are just too many issues that were discovered during testing The plan is to continue using freebsd-update in the meantime, and introduce a pkg based upgrade mechanism in FreeBSD 11.1 With the new support model for the FreeBSD 11 branch, 11.1 may come sooner than with previous major releases *** FreeBSD Core Election (https://www.freebsd.org/internal/bylaws.html) It is time once again for the FreeBSD Core Election Application period begins: Wednesday, 18 May 2016 at 18:00:00 UTC Application period ends: Wednesday, 25 May 2016 at 18:00:00 UTC Voting begins: Wednesday, 25 May 2016 at 18:00:00 UTC Voting ends: Wednesday, 22 June 2016 at 18:00:00 UTC Results announced Wednesday, 29 June 2016 New core team takes office: Wednesday, 6 July 2016 As of the time I was writing these notes, 3 hours before the application deadline, the candidates are: Allan Jude: Filling in the potholes Marcelo Araujo: We are not vampires, but we need new blood. Baptiste Daroussin (incumbent): Keep on improving Benedict Reuschling: Learn and Teach Benno Rice: Revitalising The Community Devin Teske: Here to help Ed Maste (incumbent): FreeBSD is people George V. Neville-Neil (incumbent): There is much to do… Hiroki Sato (incumbent): Keep up with our good community and technical strength John Baldwin: Ready to work Juli Mallett: Caring for community. Kris Moore: User-Focused Mathieu Arnold: Someone ask for fresh blood ? Ollivier Robert: Caring for the project and you, its developers The deadline for applications is around the time we finish recording the live show We welcome any of the candidates to schedule an interview in the next few weeks. We will make an attempt to hunt many of them down at BSDCan as well. *** Wayland/Weston with XWayland works on DragonFly (http://lists.dragonflybsd.org/pipermail/users/2016-May/249620.html) We haven't talked a lot about Wayland on BSD recently (or much at all), but today we have a post from Peter to the dragonfly mailing list, detailing his experience with it. Specifically he talks about getting XWayland working, which provides the compat bits for native X applications to run on WayLand displays. So far on the working list of apps: “gtk3: gedit nautilus evince xfce4: - xfce4-terminal - atril firefox spyder scilab” A pretty impressive list, although he said “chrome” failed with a seg-fault This is something I'm personally interested in. Now with the newer DRM bits landing in FreeBSD, perhaps it's time for some further looking into Wayland. Broadcom WiFi driver update (http://adrianchadd.blogspot.ca/2016/05/updating-broadcom-softmac-driver-bwn-or.html) In this blog post Adrian Chadd talks about his recent work on the bwn(4) driver for Broadcom WiFi chips This work has added support for a number of older 802.11g chips, including the one from 2009-era Macbooks Work is ongoing, and the hope is to add 802.11n and 5ghz support as well Adrian is mentoring a number of developers working on embedded or wifi related things, to try to increase the projects bandwidth in those areas If you are interested in driver development, or wifi internals, the blog post has lots of interesting details and covers the story of Adrian's recent adventures in bringing the drivers up *** Beastie Bits The Design of the NetBSD I/O Subsystems (2002) (http://arxiv.org/abs/1605.05810) ZFS, BTRFS, XFS, EXT4 and LVM with KVM – a storage performance comparison (http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html?print=true) Swift added to FreeBSD Ports (http://www.freshports.org/lang/swift/) misc@openbsd: 'NSA addition to ifconfig' (http://marc.info/?l=openbsd-misc&m=146391388912602&w=2) Papers We Love: Memory by the Slab: The Tale of Bonwick's Slab Allocator (http://paperswelove.org/2015/video/ryan-zezeski-memory-by-the-slab/) Feedback/Questions Lars - Poudriere (http://pastebin.com/HRRyfxev) Warren - .NET (http://pastebin.com/fESV1egk) Eddy - Sys Init (http://pastebin.com/kQecpA1X) Tim - ZFS Resources (http://pastebin.com/5096cGXr) Morgan - Ports and Kernel (http://pastebin.com/rYr1CDcV) ***
This week on BSDNow, Allan is back in down from Europe! We'll get to hear some of his wrap-up and get caught up on the latest BSD This episode was brought to you by Headlines FreeBSD Quarterly Report (http://www.freebsd.org/news/status/report-2016-01-2016-03.html) This quarterly status report starts with a rather interesting introduction by Warren Block ASLR Porting CEPH to FreeBSD RCTL I/O Rate Limiting The Graphics Stack on FreeBSD (Haswell is in, work is progressing on the next update) CAM I/O Scheduler NFS Server updates, working around the 16 group limit, and implementing pNFS, allowing NFS to scale beyond a single server Static Analysis of the FreeBSD Kernel with PVS Studio PCI-express HotPlug GitLab Port committed! WITHFASTDEPEND and other improvements to the FreeBSD build system Lots of other interesting stuff *** A Prog By Any Other Name (http://www.tedunangst.com/flak/post/a-prog-by-any-other-name) Ted Unangst looks at what goes into the name of a program “Sometimes two similar programs are really the same program with two names. For example, grep and egrep are two commands that perform very similar functions and are therefore implemented as a single program. Running ls -i and observing the inode number of each file will reveal that there is only one file. Calling the program egrep is a shorthand for -E and does the same thing.” So BSD provides __progname in libc, so a program can tell what its name is But, what if it has more than one name? “In fact, every program has three names: its name in the filesystem, the name it has been invoked with, and whatever it believes its own name to be.” Of course it is not that easy. “there's another set of choices for each name, the full path and the basename” “It's even possible on some systems for argv[0] to be NULL.” He then goes on to rename doas (the OpenBSD light replacement for sudo) to banana and discuss what happens “On that note, another possible bug is to realize that syslog by default uses progname. A user may be able to evade log monitoring by invoking doas with a different name. (Just fixed.)” Another interesting article from our friend Ted *** FreeBSD (https://summerofcode.withgoogle.com/organizations/4892834293350400/) and NetBSD (https://summerofcode.withgoogle.com/organizations/6246531984261120/) Google Summer of Code projects have been announced Some FreeBSD highlights: Add SCSI passthrough to CTL (share an optical drive via iSCSI) Add USB target mode driver based on CTL (share a USB device via iSCSI) API to link created /dev entries to sysctl nodes Implement Ethernet Ring Protection Switching (ERPS) HD Audio device model in userspace for bhyve Some NetBSD highlights: Implement Ext4fs support in ReadOnly mode NPF and blacklistd web interface Port U-Boot so it can be compiled on NetBSD Split debug symbols for pkgsrc builds *** libressl - more vague priomises (http://www.tedunangst.com/flak/post/libressl-more-vague-promises) We haven't had a Ted U article on the show as of late, however this week we get several! In his next entry “LibreSSL, more vague promises” He then goes into some detail on what has happened with LibreSSL in the past while, as well as future plans going forward. “With an eye to the future, what new promises can we make? Some time ago I joked that we only promised to make a better TLS implementation, not a better TLS. Remains true, but fortunately there are people working on that, too. TLS 1.3 support is on the short term watchlist. The good news is we may be ahead of the game, having already removed compression. How much more work can there be?” “LibreSSL integrated the draft chacha20-poly1305 construction from BoringSSL. The IETF has since standardized a slightly different version because if it were the same it wouldn't be different. Support for standard variant, and the beginning of deprecation for the existing code, should be landing very shortly. Incidentally, some people got bent out of shape because shipping chacha20 meant exposing non IANA approved numbers to Internet. No promises that won't happen again.” *** Interview - Samy Al Bahra - @0xF390 (https://twitter.com/0xF390) Backtrace *** News Roundup systrace(1) is removed for OpenBSD 6.0 (http://marc.info/?l=openbsd-cvs&m=146161167911029&w=2) OpenBSD has removed systrace, an older mechanism for limiting what syscalls an application can make It is mostly replaced by the pledge() system OpenBSD was the first implementation, most others have been unmaintained for some time The last reported Linux version was for kernel 2.6.1 NetBSD removed systrace in 2007 *** pfSense Video Series: Comprehensive Guide To pfSense 2.3 (https://www.youtube.com/playlist?list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk) A series of videos (11 so far), about pfSense Covers Why you would use it, how to pick your hardware, and installation Then the series covers some networking basics, to make sure you are up to speed before configuring your pfSense Then a comprehensive tour of the WebUI Then goes on to cover graphing, backing up and restoring configuration There are also videos on running DHCP, NTP, and DNS servers *** DuckDuckGo announces its 2016 FOSS Donations (https://duck.co/blog/post/303/2016-foss-donations-announcement) The theme is “raising the standard of trust online” Supported projects include: OpenBSD Foundation announces DuckDuckGo as a Gold Sponsor (http://undeadly.org/cgi?action=article&sid=20160503085227&mode=expanded) the Freedom of the Press Foundation for SecureDrop the Freenet Project the CrypTech Project the Tor Project Fight for the Future for Save Security Open Source Technology Improvement Fund for VeraCrypt (based on TrueCrypt) Riseup Labs for LEAP (LEAP Encryption Access Project) GPGTools for GPGMail *** Larry the BSD Guy hangs up his hat at FOSS Force (http://fossforce.com/2016/04/bsd-linuxfest-northwest/) After 15 years, Larry the BSD Guy has decided to hang it up, and walk into the sunset! (Figuratively of course) After wrapping up coverage of recent LinuxFest NorthWest (Which he didn't attend), Larry has decided it's time for a change and is giving up his column over at FOSS Force, as well as stepping away from all things technical. His last write-up is a good one, and he has some nice plugs for both Dru Lavigne and Michael Dexter of the BSD community. He will be missed, but we wish him all the luck with the future! He also puts out the plug that FOSS Force will be needing a new columnist in the near future, so if you are interested please let them know! *** Beastie Bits If you sponsored “FreeBSD Mastery: Advanced ZFS”, check your mail box (http://blather.michaelwlucas.com/archives/2648) pkg-1.7.0 is an order of magnitude slower than pkg-1.6.4 (https://marc.info/?l=freebsd-ports&m=146001143408868&w=2) -- Caused by a problem not in pkg LinuxFest Northwest 2016 Recap (https://www.ixsystems.com/blog/linuxfest-northwest-2016/) Dru Lavigne's 'Doc like an Egyption' talk from LFNW (https://www.linuxfestnorthwest.org/2016/sessions/doc-egyptian) Michael Dexters' 'Switching to BSD from Linux' talk from LFNW (https://www.linuxfestnorthwest.org/2016/sessions/devil-details-switching-bsd-linux) Michael Dexters' 'Secrets to enduring user groups' talk from LFNW (https://www.linuxfestnorthwest.org/2016/sessions/20-year-and-counting-secrets-enduring-user-groups) January issue of Freebsd Journal online for free (https://www.freebsdfoundation.org/journal/) Ghost BSD releases 10.3 Alpha1 for testing (http://ghostbsd.org/10.3_alpha1) EuroBSDcon 2016 - Call for Papers - Dealine: May 8th (https://www.freebsdnews.com/2016/04/15/eurobsdcon-2016-call-for-papers/) KnoxBUG Initial Meeting (http://www.knoxbug.org/content/knoxbug-maiden-voyage) Photos, slides, and videos from the Open Source Data Center Conference (https://www.netways.de/en/events_trainings/osdc/archive/osdc2016/) *** Feedback/Questions Mohammad - Replication (http://pastebin.com/KDnyWf6Y) John - Rolling new packages (http://pastebin.com/mAbRwbEF) Clint - Unicast (http://pastebin.com/BNa6pyir) Bill - GhostBSD (http://pastebin.com/KDjS2Hxa) Charles - BSD Videos (http://pastebin.com/ABUUtzWM) ***
This week, Allan is out of town, but since when has that ever stopped us from bringing you a new episode of BSDNow? We have news, This episode was brought to you by Headlines Unix's file durability problem (https://utcc.utoronto.ca/~cks/space/blog/unix/FileSyncProblem) Another article by Chris Siebenmann from the University of Toronto This time, the issue was a lost comment on his Python based blog which uses files on disk rather than a database After an unexpected restart of the system, a recently posted comment no longer existed The post goes on to investigate what the ‘right way' to ensure file durability is The answer, as you might expect, is “it depends…” Normally, fsync() should work, but it seems with ext4 and some other file systems, you must also fsync() the directory where the file was created, or it might not be possible to find the file after a crash Do you need to fsync() the parent of that directory too? Then what is fdatasync() for? What about just calling sync()? “One issue is that unlike many other Unix API issues, it's impossible to test to see if you got it all correct and complete. If your steps are incomplete, you don't get any errors; your data is just silently sometimes at risk. Even with a test setup to create system crashes or abrupt power loss (which VMs make much easier), you need uncommon instrumentation to know things like if your OS actually issued disk flushes or just did normal buffered writes. And straightforward testing can't tell you if what you're doing will work all the time, because what is required varies by Unix, kernel version, and the specific filesystem involved.” Second post by author: How I'm trying to do durable disk writes (https://utcc.utoronto.ca/~cks/space/blog/python/HowISyncDataDWiki) Additional Discussion on Hacker News (https://news.ycombinator.com/item?id=11511269) The discussion on HN also gets into AIO and other more complicated facilities, but even those seem to be vague about when your data is actually safe At least ZFS ensures you never get half of your new data, and half of your old data. *** Build a FreeBSD 10.3-release Openstack Image with bsd-cloudinit (https://raymii.org/s/tutorials/FreeBSD_10.3-release_Openstack_Image.html) Are you using FreeBSD and OpenStack or would you like to be? We next have a great tutorial which explains the ins-and-outs of doing exactly that. Remy van Elst brings us a great walkthrough on his site on how to get started, and hint it involves just a few ‘pip' commands. After getting the initial Python tools bootstrapped, next he shows us how to save our OpenStack settings in a sourceable shell command, which comes in handy before doing admin on a instance. Next the ‘glance' and ‘cinder' tools are used to upload the target OS ISO file and then create a volume for it to install onto. Next the VM is started and some specific steps are outlined on getting FreeBSD 10.3 installed into the instance. It includes some helpful hints as how to fix a mountroot error, if you installed to ada0, but need to mount via vtdb0 instead now. After the installation is finished, the prep for ‘cloudinit' is done, and the resulting image is compressed and made ready for deployment. We've kinda stepped through some of the more gory steps here, but if OpenStack is something you work with, this tutorial should be at the top of your “must read” list. *** Undeadly and HTTPS (http://undeadly.org/cgi?action=article&sid=20160411201504) Undeadly, the OpenBSD journal, is thinking of moving to HTTPS only In order to do this, they would like some help rewriting part of the site Currently, when you login to post comments, this is done over HTTPS, but to an stunnel instance running a custom script that gives you a cookie, and sends you back to the non-HTTPS site They would like to better integrate the authentication system, and otherwise improve the code for the site There is some pushback as well, questioning whether it makes sense to block users who are unable to use HTTPS for one reason or another I think it makes sense to have the site default to HTTPS, but, maybe HTTPS only doesn't make sense. There is nothing private on the site, other than the authentication system which is optional, not required to post a comment. There is also some discussion about the code for the site, including the fact that when the code was released, the salt for the password database was included This is not actually a security problem, but the discussion may be interesting to some viewers *** FreeBSD Journal March/April Edition (https://www.freebsdfoundation.org/journal/browser-based-edition/) The next issue of the FreeBSD Journal is here, and this time it is about Teaching with Operating Systems In addition to the usual columns, including: svn update, the ports report, a conference report from FOSDEM, a meetup report from PortsCamp Taipei, A book review of "The Algorithm Design Manual", and the Events Calendar; there are a set of feature articles about teaching Teaching with FreeBSD through Tracing, Analysis, and Experimentation CHERI: Building a foundation for secure, trusted computing bases A brief history of Fast Filesystems There is also an interview with Gleb Smirnoff, a member of the Core team, release engineering, and the deputy security officer, as well as a senior software developer at Netflix Get the latest issue from your favourite mobile store, or the “Desktop Edition” directly in your browser from the FreeBSD Foundation's website *** Interview - Brooks Davis - brooks@FreeBSD.org (mailto:brooks@FreeBSD.org) / @brooksdavis (https://twitter.com/brooksdavis) CHERI and Capabilities *** TrueNAS Three-Peats!!! (https://www.ixsystems.com/blog/truenas-three-peats/) News Roundup UbuntuBSD Is Looking To Become An Official Ubuntu Flavor (http://linux.softpedia.com/blog/ubuntubsd-is-looking-to-become-an-official-ubuntu-flavor-502746.shtml) You may recall a few weeks back that we were a bit surprised by the UbuntuBSD project and its longevity / goals. However the project seems to be pushing forward, with news on softpedia.com that they are now seeking to become an ‘official' Ubuntu Flavor. They've already released a forth beta, so it seems the project currently has some developers pushing it forward: "I would like to contribute all my work to Ubuntu Community and, if you think it is worthy, make ubuntuBSD an official Ubuntu project like Xubuntu or Edubuntu," said Jon Boden. "If you're interested, please let me know how would you like me to proceed." It's Just Bits (http://blog.appliedcompscilab.com/its_just_bits/index.html) We have next an interesting blog post talking about the idea that “It's just all bits!” The author then takes us down the idea of no matter how old or mysterious the code may be, in the end it is ending up as bits arranged a certain way. Then the article transitions and takes us through the idea that old bits, and bits that have grown too large should often be good candidates for replacement by “simpler” bits, using OpenBSD as an example. “The OpenBSD community exemplifies this in many ways by taking existing solutions and simplifying them. Processing man pages is as old as Unix, and even in the 21st century OpenBSD has taken the time to rewrite the existing solution to be simpler and safer. It's just bits that need to be turned into other bits. Similarly, OpenBSD has introduced doas as an alternative to sudo. While not replacing sudo entirely, doas makes the 99.99% case of what people use sudo for easier and safer. They are just bits that need to be authenticated. “ All in all, a good read, and it reinforces the point that nothing is really truly “finished”. As computing advances and new technologies / practices are made available, sometimes it makes a lot of sense to go back and re-write things in order to simplify the complexity that has snuck in over time. *** Disk IO limiting is coming to FreeBSD (https://lists.freebsd.org/pipermail/svn-src-head/2016-April/084288.html) A much requested feature for both Jails and VM's on FreeBSD has just landed with experimental support in -HEAD, Disk IO limiting! The Commit message states as follows: “Add four new RCTL resources - readbps, readiops, writebps and writeiops, for limiting disk (actually filesystem) IO. Note that in some cases these limits are not quite precise. It's ok, as long as it's within some reasonable bounds. Testing - and review of the code, in particular the VFS and VM parts - is very welcome.” Well, what are you waiting for? This is a fantastic new feature which I'm sure will get incorporated into other tools for controlling jails and VM's down the road. If you give it a spin, be sure to report back bugs so they can get quashed in time for 11. *** BeastieBits PC-BSD 10.3 Is the Last in the Series, PC-BSD 11.0 Arrives Later This Year (http://news.softpedia.com/news/pc-bsd-10-3-is-the-last-in-the-series-pc-bsd-11-0-arrives-later-this-year-502570.shtml) ASLR now on by default in NetBSD amd64 (http://mail-index.netbsd.org/source-changes/2016/04/10/msg073939.html) Daniel Bilik's fix for hangs on Baytrail (http://lists.dragonflybsd.org/pipermail/users/2016-April/228682.html) Don't forget about PGCon 2016 (http://www.pgcon.org/2016/) Get your paper in for EuroBSDCon 2016, deadline is May 8th (https://2016.eurobsdcon.org/call-for-papers/) Feedback/Questions John - Destroy all Dataset (http://pastebin.com/QdGWn0TW) Thomas - Misc Questions (http://pastebin.com/43YkwBjP) Ben - ZFS Copy (http://pastebin.com/gdi3pswe) Bryson - SysV IPC (http://pastebin.com/E9n938D1) Drin - IPSEC (http://pastebin.com/bgGTmbDG) ***
This week on the show, we will be talking to Benedict Reushling about his role with the FreeBSD foundation and the journey that took him This episode was brought to you by Headlines HardenedBSD introduces full PIE support (https://hardenedbsd.org/article/shawn-webb/2016-04-15/introducing-full-pie-support) PIE base for amd64 and i386 Only nine applications are not compiled as PIEs Tested PIE base on several amd64 systems, both virtualized and bare metal Hoped to be to enabled it for ARM64 before or during BSDCan. Shawn will be bringing ten Raspberry Pi 3 devices (which are ARM64) with to BSDCan, eight of which will be given out to lucky individuals. “We want the BSD community to hack on them and get ARM64/Aarch64 fully functional on them.” *** Lessons learned from 30 years of MINIX (http://m.cacm.acm.org/magazines/2016/3/198874-lessons-learned-from-30-years-of-minix/fulltext) Eat your own dog food. By not relying on idiosyncratic features of the hardware, one makes porting to new platforms much easier. The Internet is like an elephant; it never forgets. When standards exist (such as ANSI Standard C) stick to them. Even after you have adopted a strategy, you should nevertheless reexamine it from time to time. Keep focused on your real goal, Einstein was right: Things should be as simple as possible but not simpler. *** pfSense 2.3 released (https://blog.pfsense.org/?p=2008) Rewrite of the webGUI utilizing Bootstrap TLS v1.0 disabled for the GUI Moved to a FreeBSD 10.3-RELEASE base PHP Upgraded to 5.6 The "Full Backup" feature has been deprecated Closed 760 total tickets of which 137 are fixed bugs Known Regressions OpenVPN topology change IP aliases with CARP IP parent lose their parent interface association post-upgrade IPsec IPComp does not work. IGMP Proxy does not work with VLAN interfaces. Many other updates and changes *** OPNsense 16.1.10 released (https://opnsense.org/opnsense-16-1-10-released/) openvpn: revive windows installer binaries system: improved config history and backup pages layout system: increased backup count default from 30 to 60 system: /var /tmp MFS awareness for crash dumps added trust: add “IP security IKE intermediate” to server key usage firmware: moved reboot, halt and defaults pages to new home languages: updates to Russian, French, German and Japanese Many other updates and changes *** Interview - Benedict Reuschling - bcr@freebsd.org (mailto:bcr@freebsd.org) FreeBSD Foundation in Europe *** News Roundup Write opinionated workarounds (http://www.daemonology.net/blog/2016-04-11-write-opinionated-workarounds.html) Colin Percival has written a great blog post this past week, specifically talking about his policy of writing “opinionated workarounds”. The idea came about due to his working on multi-platform software, and the frustrations of dealing with POSIX violations The crux of the post is how he deals with these workarounds. Specifically by only applying them to the particular system in which it was required. And doing so loudly. This has some important benefits. First, it doesn't potentially expose other systems to bugs / security flaws when a workaround doesn't “work” on a system for which it wasn't designed. Secondly it's important to complain. Loudly. This lets the user know that they are running on a system that doesn't adhere to POSIX compliance, and maybe even get the attention of a developer who could remedy the situation. *** Privilege escalation in calendar(1) (http://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2016-003.txt.asc) File this one under “Ouch that hurts” a new security vuln has been posted, this time against NetBSD's ‘calendar' command. Specifically it looks like some of the daily scripts uses the ‘-a' flag, which requires super-user privs in order to process all users calendar files and mail the results. However the bug occurred because the calendar command didn't drop priv properly before executing external commands (whoops!) To workaround you can set run_calendar=NO in the daily.conf file, or apply the fixed binary from upstream. *** PGCon 2016 (http://www.pgcon.org/2016/) PGCon 2016 is now only 4 weeks away The conference will be held at the University of Ottawa (same venue as BSDCan) from May 17th to 20th Tutorials: 17-18 May 2016 (Tue & Wed) Talks: 19-20 May 2016 (Thu-Fri) Wednesday is a developer unconference. Saturday is a user unconference. “PGCon is an annual conference for users and developers of PostgreSQL, a leading relational database, which just happens to be open source. PGCon is the place to meet, discuss, build relationships, learn valuable insights, and generally chat about the work you are doing with PostgreSQL. If you want to learn why so many people are moving to PostgreSQL, PGCon will be the place to find out why. Whether you are a casual user or you've been working with PostgreSQL for years, PGCon will have something for you.” New to PGSQL? Just a user? Long time developers? This conference has something for you. A great lineup of talks (https://www.pgcon.org/2016/schedule/events.en.html), plus unconference days focused on both users and developers *** CfP EuroBSDCon 2016 (https://2016.eurobsdcon.org/call-for-papers/) The call for papers has been issued for EuroBSDCon 2016 in Belgrade, Serbia The conference will be held from the 22nd to 25th of September, 2016 The deadline for talk submissions is: Sunday the 8th of May, 2016 Submit your talk or tutorial proposal before it is too late *** Beastie Bits “FreeBSD Mastery: Advanced ZFS” has officially been released (https://www.michaelwlucas.com/nonfiction/fmaz) Support of OpenBSD pledge(2) in programming Languages (https://gist.github.com/ligurio/f6114bd1df371047dd80ea9b8a55c104) pkgsrcCon 2016 -Call for Presentations (http://daemonforums.org/showthread.php?t=9781) Christos Zoulas talks about blacklistd (http://blog.netbsd.org/tnf/entry/talks_about_blacklistd) Penguicon 2016 Lucas Track Schedule (http://blather.michaelwlucas.com/archives/2617) Feedback/Questions Peter - NVME (http://pastebin.com/HiiDpGcT) Jeremy - Wireless Gear (http://pastebin.com/L5XeVS1H) Ted - Rpi2 Packages (http://pastebin.com/yrCEnkWt) - Cross Building Wiki (https://wiki.freebsd.org/FreeBSD/arm/crossbuild) Geoff - Jail Failover (http://pastebin.com/pYFC1vdQ) Zach - Graphical Bhyve? (http://pastebin.com/WEgN0ZVw) ***
This week on BSDNow, we are going to be talking to Pawel about how his This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** Headlines Note the recent passing of 2 members of the BSD community Juergen Lock / Nox (https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributors/contrib-develinmemoriam.html) Benjamin Perrault / creepingfur (https://twitter.com/michaeldexter/status/676290499389485057) Memories from Michael Dexter (http://pastebin.com/4BQ5uVsT) Additional Memories (http://www.filis.org/rip_ben.txt) Benjamin and Allan at Ben's local bar (http://www.allanjude.com/bsd/bp/IMG_20151101_161727-auto.jpg) Benjamin treated Allan and Michael Dexter to their first ever Bermese food (http://www.allanjude.com/bsd/bp/IMG_20151101_191344-auto.jpg) Benjamin enjoying the hallway track at EuroBSDCon 2015 (http://www.allanjude.com/bsd/bp/IMG_20151003_105457-auto.jpg) *** NGINX as Reverse Proxy for Apache on FreeBSD 10.2 (http://linoxide.com/linux-how-to/install-nginx-reverse-proxy-apache-freebsd-10-2/) A tutorial on setting up NGINX as a reverse proxy for Apache Sometimes your users or application require some feature of Apache, that cannot be easily replicated in NGINX, like .htaccess files or a custom apache module In addition, because the default worker model in Apache does not accept new work until it is finished sending the request, a user with a slow connection can tie down that worker for a long time With NGINX as a reverse proxy, it will receive the data from the Apache worker over localhost, freeing that worker to answer the next request, while NGINX takes care of sending the data to the user The tutorial walks through the setup, which is very easy on modern FreeBSD One could also add mod_rpaf2 to the Apache, to securely pass through the users' real IP address for use by Apache's logging and the PHP scripts *** FreeBSD and FreeNAS in Business by Randy Westlund (http://bsdmag.org/freebsd_freenas/) The story of how a Tent & Awning company switched from managing orders with paper, to a computerized system backed by a FreeNAS “At first, I looked at off-the-shelf solutions. I found a number of cloud services that were like Dropbox, but with some generic management stuff layered on top. Not only did these all feel like a poor solution, they were very expensive. If the provider were to go out of business, what would happen to my dad's company?” “Fortunately, sourcing the hardware and setting up the OS was the easiest part; I talked to iXsystems. I ordered a FreeNAS Mini and a nice workstation tower” “I have r2d2 (the tower, which hosts the database) replicating ZFS snapshots to c3po (the FreeNAS mini), and the data is backed up off-site regularly. This data is absolutely mission-critical, so I can't take any risks. I'm glad I have ZFS on my side.” “I replaced Dropbox with Samba on c3po, and the Windows machines in the office now store important data on the NAS, rather than their local drives.” “I also replaced their router with an APU board running pfSense and replaced their PPTP VPN with OpenVPN and certificate authorization.” “FreeBSD (in three different incarnations) helped me focus on improving the company's workflow without spending much time on the OS. And now there's an awning company that is, in a very real sense, powered by FreeBSD.” *** Tutorial, Windows running under bhyve (http://pr1ntf.xyz/windowsunderbhyve.html) With the recent passing of the world's foremost expert on running Windows under bhyve on FreeBSD, this tutorial will help you get up to speed “The secret sauce to getting Windows running under bhyve is the new UEFI support. This is pretty great news, because when you utilize UEFI in bhyve, you don't have to load the operating system in bhyveload or grub-bhyve first.” The author works on iohyve, and wanted to migrate away from VirtualBox, the only thing stopping that was support for Windows Guests iohyve now has support for managing Windows VMs The tutorial uses a script to extract the Windows Server 2008 ISO and set up AutoUnattend.xml to handle the installation of Windows, including setting the default administrator password, this is required because there is no graphical console yet The AutoUnattended setup also includes setting the IP address, laying out the partitions, and configuring the serial console A second script is then used to make a new ISO with the modifications The user is directed to fetch the UEFI firmware and some other bits Then iohyve is used to create the Windows VM The first boot uses the newly created ISO to install Windows Server 2008 Subsequent boots start Windows directly from the virtual disk Remote Desktop is enabled, so the user can manage the Windows Server graphically, using FreeRDP or a Windows client iohyve can then be used to take snapshots of the machine, and clone it *** BSD Router Project has released 1.58 (http://sourceforge.net/projects/bsdrp/files/BSD_Router_Project/1.58/) The BSD Router project has announced the release of version 1.58 with some notable new features Update to FreeBSD 10.2-RELEASE-p8 Disabled some Chelsio Nic features not used by a router Added new easy installation helper option, use with “system install ” Added the debugging symbols for userland Includes the iperf package, and flashrom package, which allows updating system BIOS on supported boxes IMPORTANT: Corrects an important UFS label bug introduced on 1.57. If you are running 1.57, you will need to fetch their fixlabel.sh script before upgrading to 1.58 *** OPNsense 15.7.22 Released (https://opnsense.org/opnsense-15-7-22-released/) An update to OPNsense has landed this week which includes the important updates to OpenSSL 1.0.2e and LibreSSL 2.2.5 A long-standing annoying bug with filter reload timeouts has finally been identified and sorted out as well, allowing the functionality to run quickly and “glitch free” again. Some newer ports for curl (7.46), squid (3.5.12) and lighttpd (1.4.38) have also been thrown in for good measure Some other minor UI fixes have also been included as well With the holidays coming up, if you are still running a consumer router, this may be a good time to convert over to a OPNsense or PFsense box and get yourself ready for the new year. *** iXsystems iXSystems releases vCenter Web Client Plug-in for TrueNAS (https://www.ixsystems.com/whats-new/2015/12/vcenter-web-client-plug-in-for-truenas-now-available/) Interview - Pawel Jakub Dawidek - pjd@FreeBSD.org (mailto:pjd@FreeBSD.org) News Roundup Developer claims the PS4 has been jail-broken (http://www.networkworld.com/article/3014714/security/developer-claims-ps4-officially-jailbroken.html) While not exactly a well-kept secret, the PS4's proprietary “OrbOS” is FreeBSD based. Using this knowledge and a Kernel exploit, developer CTurt (https://twitter.com/CTurtE/) claims he was able jailbreak a WebKit process and gain access to the system. He has posted a small tease to GitHub, detailing some of the information gleaned from the exploit, such as PID list and root FS dump As such with these kinds of jailbreaks, he already requested that users stop sending him requests about game piracy, but the ability to hack on / run homebrew apps on the PS4 seems intriguing *** Sepherosa Ziehau is looking for testers if you have a em(4), emx(4), or igb(4) Intel device (http://lists.dragonflybsd.org/pipermail/users/2015-December/228461.html) DragonFly Testers wanted! Sephe has posted a request for users of the em(4), emx(4) and igb(4) intel drivers to test his latest branch and report back results He mentions that he has tested the models 82571, 82574 and 82573 (em/emx); 82575, 82576, 82580 and i350 specifically, so if you have something different, I'm sure he would be much appreciative of the help. It looks like the em(4) driver has been updated to 7.5.2, and igb(4) 2.4.3, and adds support for the I219-LM and I219-V NICS. *** OpenBSD Xen Support (https://marc.info/?l=openbsd-tech&m=144933933119525&w=2) Filed under the “Ohh, look what's coming soon” section, it appears that patches are starting to surface for OpenBSD Xen DOMU support. For those who aren't up on their Xen terminology, DomU is the unprivileged domain (I.E. Guest mode) Right now the patch exists at the link above, and adds a new (commented out) device to the GENERIC kernel, but this gives Xen users something new to watch for updates to. *** Thinkpad Backlit Keyboard support being worked on (http://freshbsd.org/commit/openbsd/b355449caa22e7bb6c460f7a647874836ef604f0) Another reason why Lenovo / ThinkPads are some of the best laptops currently to use with BSD, the kettenis over at the OpenBSD project has committed a patch to enable support for the “ThinkLight” For those who don't know, this is the little light that helps illuminate the laptop's keyboard under low-light situations. While the initial patch only supports the “real-deal” ThinkLight, he does mention that support will be added soon for the others on ThinkPads No sysctl's to fiddle with, this works directly with the ACPI / keyboard function keys directly, nice! *** Deadline is approaching for Submissions of Tutorial Proposals for AsiaBSDCon 2016 (https://2016.asiabsdcon.org/cfp.html) Call for Papers for BSDCAN 2016 now open (http://www.bsdcan.org/2016/papers.php) + The next two major BSD conferences both have their CFP up right now. First up is AsiaBSDCon in Tokyo from March 10th-13th, followed by BSDCan in Ottawa, June 8th-11th. + If you are working on anything interesting in the BSD community, this is a good way to get the word out about your project, plus the conference pays for Hotel / Travel. + If you can make it to both, DO SO, you won't regret it. Both Allan and Kris will be attending and we would look forward to meeting you. iohyve lands in ports (https://github.com/pr1ntf/iohyve) (http://www.freshports.org/sysutils/iohyve/) + Something we've mentioned in passing has taken its first steps in becoming reality for users! “iohyve” has now landed in the FreeBSD ports tree + While it shares a similar name to “iocage” its not directly related, different developers and such. However it does share a very similar syntax and some principles of ZFS usage + The current version is 0.7, but it already has a rather large feature set + Among the current features are ISO Management, resource management, snapshot support (via ZFS), and support for OpenBSD, NetBSD and Linux (Using grub-bhyve port) BeastieBits hammer mount is forced noatime by default (http://lists.dragonflybsd.org/pipermail/users/2015-November/228445.html) Show your support for FreeBSD (http://freebsdfoundation.blogspot.com/2015/12/show-your-support-for-freebsd.html) OpenBSD running in an Amazon EC2 t2.micro (https://gist.github.com/reyk/e23fde95354d4bc35a40) NetBSD's 2015Q4 Package freeze is coming (http://mail-index.netbsd.org/tech-pkg/2015/12/05/msg016059.html) ‘Screenshots from Developers' that we covered previously from 2002, updated for 2015 (https://anders.unix.se/2015/12/10/screenshots-from-developers--2002-vs.-2015/) Feedback/Questions (slexy was down when I made these, I only did 3, since the last is really long, save rest for next week) Mark - BSD laptops (http://pastebin.com/g0DnFG95) Jamie - zxfer (http://pastebin.com/BNCmDgTe) Anonymous - Long Story (http://pastebin.com/iw0dXZ9P) ***
This week on BSDNow, we get to hear all of Allans post EuroBSDCon wrap-up and a great interview with Benno Rice from Isilon. We got to discuss some of the pain of doing major forklift upgrades, and why your business should track -CURRENT. This episode was brought to you by Headlines EuroBSDCon Videos EuroBSDCon has started posting videos of the talks online already. The videos posted online are archives of the live stream, so some of the videos contain multiple talks Due to a technical complication, some videos only have 1 channel of audio EuroBSDCon Talk Schedule (https://2015.eurobsdcon.org/talks-and-schedule/talk-schedule/) Red Room Videos (https://www.youtube.com/channel/UCBPvcqZrNuKZuP1LQhlCp-A) Yellow Room Videos (https://www.youtube.com/channel/UCJk8Kls9LT-Txu-Jhv7csfw) Blue Room Videos (https://www.youtube.com/channel/UC-3DOxIOI5oHXE1H57g3FzQ) Photos of the conference courtersy of Ollivier Robert (https://assets.keltia.net/photos/EuroBSDCon-2015/) *** A series of OpenSMTPd patches fix multiple vulnerabilities (http://undeadly.org/cgi?action=article&sid=20151005200020) Qualys recently published an audit of the OpenSNMPd source code (https://www.qualys.com/2015/10/02/opensmtpd-audit-report.txt) The fixes for these vulnerabilities were released as 5.7.2 After its release, two additional vulnerabilities (http://www.openwall.com/lists/oss-security/2015/10/04/2) were found. One, in the portable version, newer code that was added after the audit started All users are strongly encouraged to upgrade to 5.7.3 OpenBSD users should apply the latest errata or upgrade to the newest snapshot *** FreeBSD updates in -CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=288917) Looks like Xen header support has been bumped in FreeBSD from 4.2 -> 4.6 It also enables support for ARM Update to Clang / LLVM to 3.7.0 (https://lists.freebsd.org/pipermail/freebsd-current/2015-October/057691.html) http://llvm.org/releases/3.7.0/docs/ReleaseNotes.html ZFS gets FRU (field replaceable unit) tracking (https://svnweb.freebsd.org/base?view=revision&revision=287745) OpenCL makes it way into the ports tree (https://svnweb.freebsd.org/ports?view=revision&revision=397198) bhyve has grown UEFI support, plus a CSM module bhyve can now boot Windows (https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-October/003832.html) Currently there is still only a serial console, so the post includes an unattended install .xml file and instructions on how to repack the ISO. Once Windows is installed, you can RDP into the machine bhyve can also now run IllumOS (https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-October/003833.html) *** OpenBSD Initial Support for Broadwell Graphics (http://marc.info/?l=openbsd-cvs&m=144304997800589&w=2) OpenBSD joins DragonFly now with initial support for broadwell GPUs landing in their development branch This brings Open up to Linux 3.14.52 DRM, and Mark Kettenis mentions that it isn.t perfect yet, and may cause some issues with older hardware, although no major regressions yet *** OpenBSD Slides for TAME (http://www.openbsd.org/papers/tame-fsec2015/) and libTLS APIs (http://www.openbsd.org/papers/libtls-fsec-2015/) The first set of slides are from a talk Theo de Raadt gave in Croatia, they describe the history and impetus for tame Theo specifically avoids comparisons to other sandboxing techniques like capsicum and seccomp, because he is not impartial tame() itself is only about 1200 lines of code Sandboxing the file(1) command with systrace: 300 lines of code, with tame: 4 lines Theo makes the point that .optional security. is irrelevant. If a mitigation feature has a knob to turn it off, some program will break and advise users to turn the feature off. Eventually, no one uses the feature, and it dies This has lead to OpenBSD.s policy: .Once working, these features cannot be disabled. Application bugs must be fixed. The second talk is by Bob Beck, about LibreSSL when LibreSSL was forked from OpenSSL 1.0.1g, it contained 388,000 lines of C code 30 days in LibreSSL, they had deleted 90,000 lines of C OpenSSL 1.0.2d has 432,000 lines of C (728k total), and OpenSSL Current has 411,000 lines of C (over 1 million total) LibreSSL today, contains 297,000 lines of C (511k total) None of the high risk CVEs against OpenSSL (there have been 5) have affected LibreSSL. It turns out removing old code and unneeded features is good for security. The talk focuses on libtls, an alternative to the OpenSSL API, designed to be easier to use and less error prone In the libtls api, if -1 is returned, it is always an error. In OpenSSL, it might not be an error, needs additional code to check errno In OpenBSD: ftp, nc, ntpd, httpd, spamd, syslog have been converted to the new API The OpenBSD Foundation is looking for donations in order to sponsor 2-3 developers to spend 6 months dedicated to LibreSSL *** Interview - Benno Rice - benno@FreeBSD.org (mailto:benno@FreeBSD.org) / @jeamland (https://twitter.com/jeamland) Isilon and building products on top of FreeBSD News Roundup ReLaunchd (https://github.com/mheily/relaunchd/blob/master/doc/rationale.txt) This past week we got a heads up about another init/launchd replacement, this time .Relaunchd. The goals of this project appear to be keeping launchd functionality, while being portable enough to run on FreeBSD / Linux, etc. It also has aspirations of being .container-aware. with support for jailed services, ala-docker, as well as cluster awareness. Written in ruby :(, it also maintains that it wishes to NOT take over PID1 or replace the initial system boot scripts, but extend / leverage them in new ways. *** Static Intrusion Detection in NetBSD (https://mail-index.netbsd.org/source-changes/2015/09/24/msg069028.html) Alistar Crooks has committed a new .sid. utility to NetBSD, which allows intrusion detection by comparing the file-system contents to a database of known good values The utility can compare the entire root file system of a modest NetBSD machine in about 15 seconds The following parameters of each file can be checked: atime, block count, ctime, file type, flags, group, inode, link target, mtime, number of links, permissions, size, user, crc32c checksum, sha256 checksum, sha512 checksum A JSON report is issued at the end, for any detected variances *** LibreSSL 2.3.0 in PC-BSD If you.re running PC-BSD 10.2-EDGE or October's -CURRENT image, LibreSSL 2.3.0 is now a thing Thanks to the hard work of Bernard Spil and others, we have merged in the latest LibreSSL which actually removes SSL support in favor of TLS Quite a number of bugs have been fixed, as well as patches brought over from OpenBSD to fix numerous ports. Allan has started a patchset that sets the OpenSSL in base to "private" (http://allanjude.com/bsd/privatessl_2015-10-07.patch) This hides the library so that applications and ports cannot find it, so only tools in the base system, like fetch, will be able to use it. This makes OpenSSL no longer part of the base system ABI, meaning the version can be upgraded without breaking the stable ABI promise. This feature may be important in the future as OpenSSL versions now have EoL dates, that may be sooner than the EoL on the FreeBSD stable branches. *** PC-BSD and boot-environments without GRUB (http://lists.pcbsd.org/pipermail/testing/2015-October/010173.html) In this month.s -CURRENT image of PC-BSD, we began the process of moving back from the GRUB boot-loader, in favor of FreeBSD.s A couple of patches have been included, which enables boot-environment support via the 4th menus (Thanks Allan) and support for booting ZFS on root via UEFI "beadm" has also been updated to seamlessly support both boot-loaders No full-disk encryption support yet (hopefully soon), but GRUB is still available on installer for those who need it *** Import of IWM wireless to DragonFly (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/24a8d46a22f9106b0c1466c41ba73460d7d22262) Matthew Dillon has recently imported the newer if_iwm driver from FreeBSD -> DragonFly Across the internet, users with newer Intel chipsets rejoiced! Coupled with the latest Broadwell DRM improvements, DragonFly sounds very ready for the latest laptop chipsets Also, looks like progress is being made on i386 removal (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/cf37dc2040cea9f384bd7d3dcaf24014f441b8a6) *** Feedback/Questions Dan writes in about PCBSD (http://slexy.org/view/s27ZeOiM4t) Matt writes in about ZFS (http://slexy.org/view/s219J3ebx5) Anonymous writes in about problems booting (http://slexy.org/view/s21uuMAmZb) ***
This week on the show, Allan is heading to Sweden, but we have a great interview with Andrew Pantyukhin to bring you. We will be discussing everything from contributions to FreeBSD, which technologies worked best in the datacenter, config management and more. This episode was brought to you by Headlines Allan is away this week, traveling to Sweden for the ACM womENcourage conference followed by EuroBSDCon, but we have an excellent interview for you, so sit back and enjoy the show. Allan will be back on October 5th, so we look forward to bringing you a live show, with all the details about EuroBSD and more! Interview - Andrew Pantyukhin - infofarmer@gmail.com (mailto:infofarmer@gmail.com) / @infofarmer (https://twitter.com/infofarmer) Building products with FreeBSD
It's already our two-year anniversary! This time on the show, we'll be chatting with Scott Courtney, vice president of infrastructure engineering at Verisign, about this year's vBSDCon. What's it have to offer in an already-crowded BSD conference space? We'll find out. This episode was brought to you by Headlines OpenBSD hypervisor coming soon (https://www.marc.info/?l=openbsd-tech&m=144104398132541&w=2) Our buddy Mike Larkin never rests, and he posted some very tight-lipped console output (http://pastebin.com/raw.php?i=F2Qbgdde) on Twitter recently From what little he revealed at the time (https://twitter.com/mlarkin2012/status/638265767864070144), it appeared to be a new hypervisor (https://en.wikipedia.org/wiki/Hypervisor) (that is, X86 hardware virtualization) running on OpenBSD -current, tentatively titled "vmm" Later on, he provided a much longer explanation on the mailing list, detailing a bit about what the overall plan for the code is Originally started around the time of the Australia hackathon, the work has since picked up more steam, and has gotten a funding boost from the OpenBSD foundation One thing to note: this isn't just a port of something like Xen or Bhyve; it's all-new code, and Mike explains why he chose to go that route He also answered some basic questions about the requirements, when it'll be available, what OSes it can run, what's left to do, how to get involved and so on *** Why FreeBSD should not adopt launchd (http://blog.darknedgy.net/technology/2015/08/26/0/) Last week (http://www.bsdnow.tv/episodes/2015_08_26-beverly_hills_25519) we mentioned a talk Jordan Hubbard gave about integrating various parts of Mac OS X into FreeBSD One of the changes, perhaps the most controversial item on the list, was the adoption of launchd to replace the init system (replacing init systems seems to cause backlash, we've learned) In this article, the author talks about why he thinks this is a bad idea He doesn't oppose the integration into FreeBSD-derived projects, like FreeNAS and PC-BSD, only vanilla FreeBSD itself - this is also explained in more detail The post includes both high-level descriptions and low-level technical details, and provides an interesting outlook on the situation and possibilities Reddit had quite a bit (https://www.reddit.com/r/BSD/comments/3ilhpk) to say (https://www.reddit.com/r/freebsd/comments/3ilj4i) about this one, some in agreement and some not *** DragonFly graphics improvements (http://lists.dragonflybsd.org/pipermail/commits/2015-August/458108.html) The DragonFlyBSD guys are at it again, merging newer support and fixes into their i915 (Intel) graphics stack This latest update brings them in sync with Linux 3.17, and includes Haswell fixes, DisplayPort fixes, improvements for Broadwell and even Cherryview GPUs You should also see some power management improvements, longer battery life and various other bug fixes If you're running DragonFly, especially on a laptop, you'll want to get this stuff on your machine quick - big improvements all around *** OpenBSD tames the userland (https://www.marc.info/?l=openbsd-tech&m=144070638327053&w=2) Last week we mentioned OpenBSD's tame framework getting support for file whitelists, and said that the userland integration was next - well, now here we are Theo posted a mega diff of nearly 100 smaller diffs, adding tame support to many areas of the userland tools It's still a work-in-progress version; there's still more to be added (including the file path whitelist stuff) Some classic utilities are even being reworked to make taming them easier - the "w" command (https://www.marc.info/?l=openbsd-cvs&m=144103945031253&w=2), for example The diff provides some good insight on exactly how to restrict different types of utilities, as well as how easy it is to actually do so (and en masse) More discussion can be found on HN (https://news.ycombinator.com/item?id=10135901), as one might expect If you're a software developer, and especially if your software is in ports already, consider adding some more fine-grained tame support in your next release *** Interview - Scott Courtney - vbsdcon@verisign.com (mailto:vbsdcon@verisign.com) / @verisign (https://twitter.com/verisign) vBSDCon (http://vbsdcon.com/) 2015 News Roundup OPNsense, beyond the fork (https://opnsense.org/opnsense-beyond-the-fork) We first heard about (http://www.bsdnow.tv/episodes/2015_01_14-common_sense_approach) OPNsense back in January, and they've since released nearly 40 versions, spanning over 5,000 commits This is their first big status update, covering some of the things that've happened since the project was born There's been a lot of community growth and participation, mass bug fixing, new features added, experimental builds with ASLR and much more - the report touches on a little of everything *** LibreSSL nukes SSLv3 (http://undeadly.org/cgi?action=article&sid=20150827112006) With their latest release, LibreSSL began to turn off SSLv3 (http://disablessl3.com) support, starting with the "openssl" command At the time, SSLv3 wasn't disabled entirely because of some things in the OpenBSD ports tree requiring it (apache being one odd example) They've now flipped the switch, and the process of complete removal has started From the Undeadly summary, "This is an important step for the security of the LibreSSL library and, by extension, the ports tree. It does, however, require lots of testing of the resulting packages, as some of the fallout may be at runtime (so not detected during the build). That is part of why this is committed at this point during the release cycle: it gives the community more time to test packages and report issues so that these can be fixed. When these fixes are then pushed upstream, the entire software ecosystem will benefit. In short: you know what to do!" With this change and a few more to follow shortly, LibreSSL won't actually support SSL anymore - time to rename it "LibreTLS" *** FreeBSD MPTCP updated (http://caia.swin.edu.au/urp/newtcp/mptcp/tools/v05/mptcp-readme-v0.5.txt) For anyone unaware, Multipath TCP (https://en.wikipedia.org/wiki/Multipath_TCP) is "an ongoing effort of the Internet Engineering Task Force's (IETF) Multipath TCP working group, that aims at allowing a Transmission Control Protocol (TCP) connection to use multiple paths to maximize resource usage and increase redundancy." There's been work out of an Australian university to add support for it to the FreeBSD kernel, and the patchset was recently updated Including in this latest version is an overview of the protocol, how to get it compiled in, current features and limitations and some info about the routing requirements Some big performance gains can be had with MPTCP, but only if both the client and server systems support it - getting it into the FreeBSD kernel would be a good start *** UEFI and GPT in OpenBSD (https://www.marc.info/?l=openbsd-cvs&m=144092912907778&w=2) There hasn't been much fanfare about it yet, but some initial UEFI and GPT-related commits have been creeping into OpenBSD recently Some support (https://github.com/yasuoka/openbsd-uefi) for UEFI booting has landed in the kernel, and more bits are being slowly enabled after review This comes along with a number (https://www.marc.info/?l=openbsd-cvs&m=143732984925140&w=2) of (https://www.marc.info/?l=openbsd-cvs&m=144088136200753&w=2) other (https://www.marc.info/?l=openbsd-cvs&m=144046793225230&w=2) commits (https://www.marc.info/?l=openbsd-cvs&m=144045760723039&w=2) related to GPT, much of which is being refactored and slowly reintroduced Currently, you have to do some disklabel wizardry to bypass the MBR limit and access more than 2TB of space on a single drive, but it should "just work" with GPT (once everything's in) The UEFI bootloader support has been committed (https://www.marc.info/?l=openbsd-cvs&m=144115942223734&w=2), so stay tuned for more updates (http://undeadly.org/cgi?action=article&sid=20150902074526&mode=flat) as further (https://twitter.com/kotatsu_mi/status/638909417761562624) progress (https://twitter.com/yojiro/status/638189353601097728) is made *** Feedback/Questions John writes in (http://slexy.org/view/s2sIWfb3Qh) Mason writes in (http://slexy.org/view/s2Ybrx00KI) Earl writes in (http://slexy.org/view/s20FpmR7ZW) ***
Coming up this time on the show, we'll be chatting with Lee Sharp. He's recently revived the m0n0wall codebase, now known as SmallWall, and we'll find out what the future holds for this new addition to the BSD family. Answers to your emails and all this week's news, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines BSDCan and pkgsrcCon videos (https://www.youtube.com/channel/UCAEx6zhR2sD2pAGKezasAjA/videos) Even more BSDCan 2015 videos are slowly but surely making their way to the internet Nigel Williams, Multipath TCP for FreeBSD (https://www.youtube.com/watch?v=P3vB_FWtyIs) Stephen Bourne, Early days of Unix and design of sh (https://www.youtube.com/watch?v=2kEJoWfobpA) John Criswell, Protecting FreeBSD with Secure Virtual Architecture (https://www.youtube.com/watch?v=hRIC_aF_u24) Shany Michaely, Expanding RDMA capability over Ethernet in FreeBSD (https://www.youtube.com/watch?v=stsaeKvF3no) John-Mark Gurney, Adding AES-ICM and AES-GCM to OpenCrypto (https://www.youtube.com/watch?v=JaufZ7yCrLU) Sevan Janiyan, Adventures in building (https://www.youtube.com/watch?v=-HMXyzybgdM) open source software (https://www.youtube.com/watch?v=Xof-uKnQ6cY) And finally, the BSDCan 2015 closing (https://www.youtube.com/watch?v=Ynm0bGnYdfY) Some videos (https://vimeo.com/channels/pkgsrccon/videos) from this year's pkgsrcCon (http://pkgsrc.org/pkgsrcCon/2015/) are also starting to appear online Sevan Janiyan, A year of pkgsrc 2014 - 2015 (https://vimeo.com/channels/pkgsrccon/132767946) Pierre Pronchery, pkgsrc meets pkg-ng (https://vimeo.com/channels/pkgsrccon/132766052) Jonathan Perkin, pkgsrc at Joyent (https://vimeo.com/channels/pkgsrccon/132760863) Jörg Sonnenberger, pkg_install script framework (https://vimeo.com/channels/pkgsrccon/132757658) Benny Siegert, New Features in BulkTracker (https://vimeo.com/channels/pkgsrccon/132751897) This is the first time we've ever seen recordings from the conference - hopefully they continue this trend *** OPNsense 15.7 released (https://forum.opnsense.org/index.php?topic=839.0) The OPNsense team has released version 15.7, almost exactly six months after their initial debut (http://www.bsdnow.tv/episodes/2015_01_14-common_sense_approach) In addition to pulling in the latest security fixes from upstream FreeBSD, 15.7 also includes new integration of an intrusion detection system (and new GUI for it) as well as new blacklisting options for the proxy server Taking a note from upstream PF's playbook, ALTQ traffic shaping support has finally been retired as of this release (it was deprecated from OpenBSD a few years ago, and the code was completely removed (http://undeadly.org/cgi?action=article&sid=20140419151959) just over a year ago) The LibreSSL flavor has been promoted to production-ready, and users can easily migrate over from OpenSSL via the GUI - switching between the two is simple; no commitment needed Various third party ports have also been bumped up to their latest versions to keep things fresh, and there's the usual round of bug fixes included Shortly afterwards, 15.7.1 (https://forum.opnsense.org/index.php?topic=915.0) was released with a few more small fixes *** NetBSD at Open Source Conference 2015 Okinawa (https://mail-index.netbsd.org/netbsd-advocacy/2015/07/04/msg000688.html) If you liked last week's episode (http://www.bsdnow.tv/episodes/2015_07_01-lost_technology) then you'll probably know what to expect with this one The NetBSD users group of Japan hit another open source conference, this time in Okinawa This time, they had a few interesting NetBSD machines on display that we didn't get to see in the interview last week We'd love to see something like this in North America or Europe too - anyone up for installing BSD on some interesting devices and showing them off at a Linux con? *** OpenBSD BGP and VRFs (http://firstyear.id.au/entry/21) "VRFs (https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding), or in OpenBSD rdomains, are a simple, yet powerful (and sometimes confusing) topic" This article aims to explain both BGP and rdomains, using network diagrams, for some network isolation goodness With multiple rdomains, it's also possible to have two upstream internet connections, but lock different groups of your internal network to just one of them The idea of a "guest network" can greatly benefit from this separation as well, even allowing for the same IP ranges to be used without issues Combining rdomains with the BGP protocol allows for some very selective and precise blocking/passing of traffic between networks, which is also covered in detail here The BSDCan talk on rdomains (https://www.youtube.com/watch?v=BizrC8Zr-YY) expands on the subject a bit more if you haven't seen it, as well as a few related (https://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/) posts (http://cybermashup.com/2013/05/21/complex-routing-with-openbsd/) *** Interview - Lee Sharp - lee@smallwall.org (mailto:lee@smallwall.org) SmallWall (http://smallwall.org), a continuation of m0n0wall News Roundup Solaris adopts more BSD goodies (https://blogs.oracle.com/solarisfw/entry/pf_for_solaris) We mentioned a while back that Oracle developers have begun porting a current version of OpenBSD's PF firewall to their next version, even contributing back patches for SMP and other bug fixes They recently published an article about PF, talking about what's different about it on their platform compared to others - not especially useful for BSD users, but interesting to read if you like firewalls Darren Moffat, who was part of originally getting an SSH implementation into Solaris, has a second blog post (https://blogs.oracle.com/darren/entry/openssh_in_solaris_11_3) up about their "SunSSH" fork Going forward, their next version is going to offer a completely vanilla OpenSSH option as well, with the plan being to phase out SunSSH after that The article talks a bit about the history of getting SSH into the OS, forking the code and also lists some of the differences between the two In a third blog post (https://blogs.oracle.com/darren/entry/solaris_new_system_calls_getentropy), they talk about a new system call they're borrowing from OpenBSD, getentropy(2) (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/getentropy.2), as well as the addition of arc4random (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man3/arc4random.3) to their libc With an up-to-date and SMP-capable PF, ZFS with native encryption, jail-like Zones, unaltered OpenSSH and secure entropy calls… is Solaris becoming better than us? Look forward to the upcoming "Solaris Now" podcast (not really) *** EuroBSDCon 2015 talks and tutorials (https://2015.eurobsdcon.org/talks/) This year's EuroBSDCon is set to be held in Sweden at the beginning of October, and the preliminary list of accepted presentations has been published The list looks pretty well-balanced between the different BSDs, something Paul would be happy to see if he was still with us It even includes an interesting DragonFly talk and a couple talks from NetBSD developers, in addition to plenty of FreeBSD and OpenBSD of course There are also a few tutorials (https://2015.eurobsdcon.org/tutorials/) planned for the event, some you've probably seen already and some you haven't Registration for the event will be opening very soon (likely this week or next) *** Using ZFS replication to improve offsite backups (https://www.iceflatline.com/2015/07/using-zfs-replication-features-in-freebsd-to-improve-my-offsite-backups/) If you take backups seriously, you're probably using ZFS and probably keeping an offsite copy of the data This article covers doing just that, but with a focus on making use of the replication capability It'll walk you through taking a snapshot of your pool and then replicating it to another remote system, using "zfs send" and SSH - this has the benefit of only transferring the files that have changed since the last time you did it Steps are also taken to allow a regular user to take and manage snapshots, so you don't need to be root for the SSH transfer Data integrity is a long process - filesystem-level checksums, resistance to hardware failure, ECC memory, multiple copies in different locations... they all play a role in keeping your files secure; don't skip out on any of them One thing the author didn't mention in his post: having an offline copy of the data, ideally sealed in a safe place, is also important *** Block encryption in OpenBSD (http://anadoxin.org/blog/blog/20150705/block-encryption-in-openbsd/) We've covered (http://www.bsdnow.tv/tutorials/fde) ways to do fully-encrypted installations of OpenBSD (and FreeBSD) before, but that requires dedicating a whole drive or partition to the sensitive data This blog post takes you through the process of creating encrypted containers in OpenBSD, à la TrueCrypt - that is, a file-backed virtual device with an encrypted filesystem It goes through creating a file that looks like random data, pointing vnconfig at it, setting up the crypto and finally using it as a fake storage device The encrypted container method offers the advantage of being a bit more portable across installations than other ways *** Docker hits FreeBSD ports (https://svnweb.freebsd.org/ports?view=revision&revision=391421) The inevitable has happened, and an early FreeBSD port of docker is finally here Some details and directions (https://github.com/kvasdopil/docker/blob/freebsd-compat/FREEBSD-PORTING.md) are available to read if you'd like to give it a try, as well as a list of which features work and which don't There was also some Hacker News discussion (https://news.ycombinator.com/item?id=9840025) on the topic *** Microsoft donates to OpenSSH (http://undeadly.org/cgi?action=article&sid=20150708134520&mode=flat) We've talked about big businesses using BSD and contributing back before, even mentioning a few other large public donations - now it's Microsoft's turn With their recent decision to integrate OpenSSH into an upcoming Windows release, Microsoft has donated a large sum of money to the OpenBSD foundation, making them a gold-level sponsor They've also posted some contract work offers on the OpenSSH mailing list, and say that their changes will be upstreamed if appropriate - we're always glad to see this *** Feedback/Questions Joe writes in (http://slexy.org/view/s2NqbhwOoH) Mike writes in (http://slexy.org/view/s2T3NEia98) Randy writes in (http://slexy.org/view/s20RlTK6Ha) Tony writes in (http://slexy.org/view/s2rjCd0bGX) Kevin writes in (http://slexy.org/view/s21PfSIyG5) ***
This time on the show, we'll be talking with Pascal Stumpf about static PIE in the upcoming OpenBSD release. He'll tell us what types of attacks it prevents, and why it's such a big deal. We've also got answers to questions from you in the audience and all this week's news, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Solaris' networking future is with OpenBSD (http://bsdly.blogspot.com/2015/04/solaris-admins-for-glimpse-of-your.html) A curious patch from someone with an Oracle email address was recently sent in (https://www.marc.info/?l=openbsd-tech&m=142822852613581&w=2) to one of the OpenBSD mailing lists It was revealed that future releases of Solaris are going to drop their IPFilter firewall entirely, in favor of a port of the current version of PF For anyone unfamiliar with the history of PF, it was actually made as a replacement for IPFilter in OpenBSD, due to some licensing issues What's more, Solaris was the original development platform for IPFilter, so the fact that it would be replaced in its own home is pretty interesting This blog post goes through some of the backstory of the two firewalls PF is in a lot of places - other BSDs, Mac OS X and iOS - but there are plenty of other OpenBSD-developed technologies end up ported to other projects too "Many of the world's largest corporations and government agencies are heavy Solaris users, meaning that even if you're neither an OpenBSD user or a Solaris user, your kit is likely interacting intensely with both kinds, and with Solaris moving to OpenBSD's PF for their filtering needs, we will all be benefiting even more from the OpenBSD project's emphasis on correctness, quality and security" You're welcome, Oracle *** BAFUG discussion videos (https://www.youtube.com/watch?v=Cb--h-iOQEM#t=15) The Bay Area FreeBSD users group has been uploading some videos from their recent meetings Sean Bruno gave a recap of his experiences at EuroBSDCon last year, including the devsummit and some proposed ideas from it (as well as their current status) Craig Rodrigues also gave a talk (https://www.youtube.com/watch?v=kPs8Dni_g3M#t=15) about Kyua and the FreeBSD testing framework Lastly, Kip Macy gave a talk (https://www.youtube.com/watch?v=Q13WtuqbZ7E#t=15) titled "network stack changes, user-level FreeBSD" The main two subjects there are some network stack changes, and how to get more people contributing, but there's also open discussion about a variety of FreeBSD topics If you're close to the Bay Area in California, be sure to check out their group and attend a meeting sometime *** More than just a makefile (http://homing-on-code.blogspot.com/2015/04/ports-are-more-than-just-makefile.html) If you're not a BSD user just yet, you might be wondering how the various ports and pkgsrc systems compare to the binary way of doing things on Linux This blog entry talks about the ports system in OpenBSD, but a lot of the concepts apply to all the ports systems across the BSDs As it turns out, the ports system really isn't that different from a binary package manager - they are what's used to create binary packages, after all The author goes through what makefiles do, customizing which options software is compiled with, patching source code to build and getting those patches back upstream After that, he shows you how to get your new port tested, if you're interesting in doing some porting yourself, and getting involved with the rest of the community This post is very long and there's a lot more to it, so check it out (and more discussion on Hacker News (https://news.ycombinator.com/item?id=9360827)) *** Securing your home fences (http://www.scip.ch/en/?labs.20150409) Hopefully all our listeners have realized that trusting your network(s) to a consumer router is a bad (http://www.devttys0.com/2015/04/hacking-the-d-link-dir-890l/) idea (https://threatpost.com/12-million-home-routers-vulnerable-to-takeover/109970) by now We hear from a lot of users who want to set up some kind of BSD-based firewall, but don't hear back from them after they've done it.. until now In this post, someone goes through the process of setting up a home firewall using OPNsense on a PCEngines APU board (http://www.pcengines.ch/apu1d4.htm) He notes that you have a lot of options software-wise, including vanilla FreeBSD (http://blog.pcbsd.org/2015/01/using-trueos-as-a-ipfw-based-home-router/), OpenBSD (http://www.bsdnow.tv/tutorials/openbsd-router) or even Linux, but decided to go with OPNsense because of the easy interface and configuration The post covers all the hardware you'll need, getting the OS installed to a flash drive or SD card and going through the whole process Finally, he goes through setting up the firewall with the graphical interface, applying updates and finishing everything up If you don't have any experience using a serial console, this guide also has some good info for beginners about those (which also applies to regular FreeBSD) We love super-detailed guides like this, so everyone should write more and send them to us immediately *** Interview - Pascal Stumpf - pascal@openbsd.org (mailto:pascal@openbsd.org) Static PIE in OpenBSD News Roundup LLVM's new libFuzzer (http://blog.llvm.org/2015/04/fuzz-all-clangs.html) We've discussed fuzzing on the show a number of times, albeit mostly with the American Fuzzy Lop utility It looks like LLVM is going to have their own fuzzing tool too now The Clang and LLVM guys are no strangers to this type of code testing, but decided to "close the loop" and start fuzzing parts of LLVM (including Clang) using LLVM itself With Clang being the default in both FreeBSD and Bitrig, and with the other BSDs considering the switch, this could make for some good bug hunting across all the projects in the future *** HardenedBSD upgrades secadm (http://hardenedbsd.org/article/shawn-webb/2015-04-14/introducing-secadm-02) The HardenedBSD guys have released a new version of their secadm tool, with the showcase feature being integriforce support We covered both the secadm tool and integriforce in previous episodes, but the short version is that it's a way to prevent files from being altered (even as root) Their integriforce feature itself has also gotten a couple improvements: shared objects are now checked too, instead of just binaries, and it uses more caching to speed up the whole process now *** RAID5 returns to OpenBSD (https://www.marc.info/?l=openbsd-tech&m=142877132517229&w=2) OpenBSD's softraid (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/softraid.4) subsystem, somewhat similar to FreeBSD's GEOM, has had experimental RAID5 support for a while However, it was exactly that - experimental - and required a recompile to enable With some work from recent hackathons, the final piece (https://www.marc.info/?l=openbsd-cvs&m=142876943116907&w=2) was added to enable resuming partial array rebuilds Now it's on by default (https://www.marc.info/?l=openbsd-cvs&m=142877026917030&w=2), and there's a call for testing being put out, so grab a snapshot and put the code through its paces The bioctl softraid command also now supports (https://www.marc.info/?l=openbsd-cvs&m=142877223817406&w=2) DUIDs during pseudo-device detachment, possibly paving the way for the installer to drop (https://www.marc.info/?l=openbsd-tech&m=142643313416298&w=2) the "do you want to enable DUIDs?" question entirely *** pkgng 1.5.0 released (https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055463.html) Going back to what we talked about last week (http://www.bsdnow.tv/episodes/2015_04_08-pkg_remove_freebsd-update), the final version of pkgng 1.5.0 is out The "provides" and "requires" support is finally in a regular release A new "-r" switch will allow for direct installation to a chroot or alternate root directory Memory usage should be much better now, and some general code speed-ups were added This version also introduces support for Mac OS X, NetBSD and EdgeBSD - it'll be interesting to see if anything comes of that Many more bugs were fixed, so check the mailing list announcement for the rest (and plenty new bugs were added, according to bapt) *** p2k15 hackathon reports (http://undeadly.org/cgi?action=article&sid=20150411160247) There was another OpenBSD hackathon that just finished up in the UK - this time it was mainly for ports work As usual, the developers sent in reports of some of the things they got done at the event Landry Breuil, both an upstream Mozilla developer and an OpenBSD developer, wrote in about the work he did on the Firefox port (specifically WebRTC) and some others, as well as reviewing lots of patches that were ready to commit Stefan Sperling wrote in (http://undeadly.org/cgi?action=article&sid=20150414064710), detailing his work with wireless chipsets, specifically when the vendor doesn't provide any hardware documentation, as well as updating some of the games in ports Ken Westerback also sent in a report (http://undeadly.org/cgi?action=article&sid=20150413163333), but decided to be a rebel and not work on ports at all - he got a lot of GPT-related work done, and also reviewed the RAID5 support we talked about earlier *** Feedback/Questions Shaun writes in (http://slexy.org/view/s2iNBo2swq) Hrishi writes in (http://slexy.org/view/s202BRLwrd) Randy writes in (http://slexy.org/view/s2KT7M35uY) Zach writes in (http://slexy.org/view/s2Q5lOoxzl) Ben writes in (http://slexy.org/view/s2ynDjuzVi) *** Mailing List Gold Gstreamer hates us (https://www.marc.info/?l=openbsd-ports&m=142884995931428&w=2) At least he's honest (https://lists.torproject.org/pipermail/tor-relays/2015-April/006765.html) I find myself in a situation (https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055390.html) ***
Coming up this week on the show, we'll be talking to Kamila Součková, a Google intern. She's been working on the FreeBSD pager daemon, and also tells us about her initial experiences trying out BSD and going to a conference. As always, all the week's news and answers to your emails, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Major changes coming in PCBSD 11 (http://blog.pcbsd.org/2015/04/huge-announcement-for-pc-bsd/) The PCBSD team has announced that version 11.0 will have some more pretty big changes (as they've been known to do lately with NTP daemons and firewalls) Switching from PF to IPFW provided some benefits for VIMAGE, but the syntax was just too complicated for regular everyday users To solve this, they've ported over Linux's iptables, giving users a much more straightforward configuration (http://dpaste.com/2F1KM6T.txt) While ZFS has served them well as the default filesystem for a while, Kris decided that Btrfs would be a better choice going forward Since the FreeBSD kernel doesn't support it natively, all filesystem calls will be through FUSE from now on - performance is Good Enough People often complain about PCBSD's huge ISO download, so, to save space, the default email client will be switched to mutt, and KDE will be replaced with DWM as the default window manager To reconfigure it, or make any appearance changes, users just need to edit a simple C header file and recompile - easy peasy As we've mentioned on the show, PCBSD has been promoting safe backup solutions for a long time with its "life preserver" utility, making it simple to manage multiple snapshots too To test if people have been listening to this advice, Kris recently activated the backdoor he put in life preserver that deletes all the users' files - hope you had that stuff backed up *** NetBSD and FreeBSD join forces (http://www.freebsddiary.org/fretbsd.php) The BSD community has been running into one of the same problems Linux has lately: we just have too many different BSDs to choose from What's more, none of them have any specific areas they focus on or anything like that (they're all basically the same) That situation is about to improve somewhat, as FreeBSD and NetBSD have just merged codebases... say hello to FretBSD Within a week, all mailing lists and webservers for the legacy NetBSD and FreeBSD projects will be terminated - the mailing list for the new combined project will be hosted from the United Nations datacenter on a Microsoft Exchange server As UN monitors will be moderating the mailing lists to prevent disagreements and divisive arguments before they begin, this system is expected to be adequate for the load With FretBSD, your toaster can now run ZFS, so you'll never need to worry about the bread becoming silently corrupted again *** Puffy in the cloud (http://homing-on-code.blogspot.com/2015/03/puffy-in-cloud.html) If you've ever wanted to set up a backup server, especially for family members or someone who's not as technology-savvy, you've probably realized there are a lot of options This post explores the option of setting up your own Dropbox-like service with Owncloud and PostgreSQL, running atop the new OpenBSD http daemon Doing it this way with your own setup, you can control all the security aspects - disk encryption, firewall rules, who can access what and from where, etc He also mentions our pf tutorial (http://www.bsdnow.tv/tutorials/pf) being helpful in blocking script kiddies from hammering the box Be sure to encourage your less-technical friends to always back up their important data *** NetBSD at AsiaBSDCon (https://blog.netbsd.org/tnf/entry/asiabsdcon_2015) Some NetBSD developers have put together a report of what they did at the most recent event in Tokyo It includes a wrap-up of the event, as well as a list of presentations (https://www.netbsd.org/gallery/presentations/#asiabsdcon2015) that NetBSD developers gave Have you ever wanted even more pictures of NetBSD running on lots of devices? There's a never-ending supply, apparently At the BSD research booth of AsiaBSDCon, there were a large number of machines on display, and someone has finally uploaded pictures of all of them (http://www.ki.nu/~makoto/p15/20150315/) There's also a video (https://www.youtube.com/watch?v=K1y9cdmLFjw) of an OMRON LUNA-II running the luna68k port *** Interview - Kamila Součková - kamila@ksp.sk (mailto:kamila@ksp.sk) / @anotherkamila (https://twitter.com/anotherkamila) BSD conferences, Google Summer of Code, various topics News Roundup FreeBSD foundation March update (https://www.freebsdfoundation.org/press/2015marchupdate.pdf) The FreeBSD foundation has published their March update for fundraising and sponsored projects In the document, you'll find information about upcoming ARMv8 enhancements, some event recaps and a Google Summer of Code status update They also mention our interview with the foundation president (http://www.bsdnow.tv/episodes/2015_03_11-the_pcbsd_tour_ii) - be sure to check it out if you haven't *** Inside OpenBSD's new httpd (http://sdtimes.com/inside-openbsds-new-httpd-web-server/) BSD news continues to dominate mainstream tech news sites… well not really, but they talk about it once in a while The SD Times is featuring an article about OpenBSD's in-house HTTP server, after seeing Reyk's AsiaBSDCon presentation (http://www.openbsd.org/papers/httpd-slides-asiabsdcon2015.pdf) about it (which he's giving at BSDCan this year, too) In this article, they talk about the rapid transition of webservers in the base system - apache being replaced with nginx, only to be replaced with httpd shortly thereafter Since the new daemon has had almost a full release cycle to grow, new features and fixes have been pouring in The post also highlights some of the security features: everything runs in a chroot with privsep by default, and it also leverages strong TLS 1.2 defaults (including Perfect Forward Secrecy) *** Using poudriere without OpenSSL (http://bsdxbsdx.blogspot.com/2015/04/build-packages-in-poudriere-without.html) Last week we talked about (http://www.bsdnow.tv/episodes/2015_03_25-ssl_in_the_wild) using LibreSSL in FreeBSD for all your ports One of the problems that was mentioned is that some ports are configured improperly, and end up linking against the OpenSSL in the base system even when you tell them not to This blog post shows how to completely strip OpenSSL out of the poudriere (http://www.bsdnow.tv/tutorials/poudriere) build jails, something that's a lot more difficult than you'd think If you're a port maintainer, pay close attention to this post, and get your ports fixed to adhere to the make.conf options properly *** HAMMER and GPT in OpenBSD (https://www.marc.info/?l=openbsd-tech&m=142755452428573&w=2) Someone, presumably a Google Summer of Code student, wrote in to the lists about his HAMMER FS (http://www.bsdnow.tv/tutorials/hammer) porting proposal He outlined the entire process and estimated timetable, including what would be supported and which aspects were beyond the scope of his work (like the clustering stuff) There's no word yet on if it will be accepted, but it's an interesting idea to explore, especially when you consider that HAMMER really only has one developer In more disk-related news, Ken Westerback (http://www.bsdnow.tv/episodes/2015_02_25-from_the_foundation_2) has been committing quite a lot of GPT-related fixes (https://www.marc.info/?l=openbsd-cvs&w=2&r=1&s=gpt&q=b) recently Full GPT support will most likely be finished before 5.8, but anything involving HAMMER FS is still anyone's guess *** Feedback/Questions Morgan writes in (http://slexy.org/view/s20e30p4qf) Dustin writes in (http://slexy.org/view/s20clKByMP) Stan writes in (http://slexy.org/view/s20aBlmaT5) Mica writes in (http://slexy.org/view/s2ufFrZY9y) *** Mailing List Gold Developers in freefall (https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055281.html) Xorg thieves pt. 1 (https://www.marc.info/?l=openbsd-cvs&m=142786808725483&w=4) Xorg thieves pt. 2 (https://www.marc.info/?l=openbsd-cvs&m=142790740405547&w=4) ***
Coming up this week, we'll be chatting with Bernard Spil about wider adoption of LibreSSL in other communities. He's been doing a lot of work with FreeBSD ports specifically, but also working with upstream projects. As usual, all this weeks news and answers to your questions, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines EuroBSDCon 2015 call for papers (https://2015.eurobsdcon.org/call-for-papers/) The call for papers has been announced for the next EuroBSDCon (http://www.bsdnow.tv/episodes/2014_12_03-conference-connoisseur), which is set to be held in Sweden this year According to their site, the call for presentation proposals period will start on Monday the 23rd of March until Friday the 17th of April If giving a full talk isn't your thing, there's also a call for tutorials - if you're comfortable teaching other people about something BSD-related, this could be a great thing too You're not limited to one proposal - several speakers gave multiple in 2014 - so don't hesitate if you've got more than one thing you'd like to talk about We'd like to see a more balanced conference schedule than BSDCan's having this year, but that requires effort on both sides - if you're doing anything cool with any BSD, we'd encourage you submit a proposal (or two) Check the announcement for all the specific details and requirements If your talk gets accepted, the conference even pays for your travel expenses *** Making security sausage (http://www.tedunangst.com/flak/post/making-security-sausage) Ted Unangst (http://www.bsdnow.tv/episodes/2014_02_05-time_signatures) has a new blog post up, detailing his experiences with some recent security patches both in and out of OpenBSD "Unfortunately, I wrote the tool used for signing patches which somehow turned into a responsibility for also creating the inputs to be signed. That was not the plan!" The post first takes us through a few OpenBSD errata patches, explaining how some can get fixed very quickly, but others are more complicated and need a bit more review It also covers security in upstream codebases, and how upstream projects sometimes treat security issues as any other bug Following that, it leads to the topic of FreeType - and a much more complicated problem with backporting patches between versions The recent OpenSSL vulnerabilities were also mentioned, with an interesting story to go along with them Just 45 minutes before the agreed-upon announcement, OpenBSD devs found a problem with the patch OpenSSL planned to release - it had to be redone at the last minute It was because of this that FreeBSD actually had to release a security update to their security update (https://lists.freebsd.org/pipermail/freebsd-security-notifications/2015-March/000237.html) He concludes with "My number one wish would be that every project provide small patches for security issues. Dropping enormous feature releases along with a note 'oh, and some security too' creates downstream mayhem." *** Running FreeBSD on the server, a sysadmin speaks (http://www.itwire.com/business-it-news/open-source/67420-running-freebsd-on-the-server-a-sysadmin-speaks) More BSD content is appearing on mainstream technology sites, and, more importantly, BSD Now is being mentioned ITWire recently did an interview with Allan about running FreeBSD on servers (possibly to go with their earlier interview with Kris about desktop usage) They discuss some of the advantages BSD brings to the table for sysadmins that might be used to Linux or some other UNIX flavor It also covers specific features like jails, ZFS, long-term support, automating tasks and even… what to name your computers If you've been considering switching your servers over from Linux to FreeBSD, but maybe wanted to hear some first-hand experience, this is the article for you *** NetBSD ported to Hardkernel ODROID-C1 (https://blog.netbsd.org/tnf/entry/netbsd_ported_to_hardkernel_odroid) In their never-ending quest to run on every new board that comes out, NetBSD has been ported to the Hardkernel ODROID-C1 (http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141578608433) This one features a quad-core ARMv7 CPU at 1.5GHz, has a gig of ram and gigabit ethernet... all for just $35 There's a special kernel config file for this board's hardware, available in both -current and the upcoming 7.0 More info can be found on their wiki page (https://wiki.netbsd.org/ports/evbarm/odroid-c1/) After this was written, basic framebuffer console support was also committed (http://mail-index.netbsd.org/source-changes/2015/03/21/msg064156.html), allowing a developer to run XFCE (https://pbs.twimg.com/media/CAqU5CnWEAAEhH2.png:large) on the device *** Interview - Bernard Spil - brnrd@freebsd.org (mailto:brnrd@freebsd.org) / @sp1l (https://twitter.com/sp1l) LibreSSL adoption in FreeBSD ports (https://wiki.freebsd.org/LibreSSL) and the wider software ecosystem News Roundup Monitoring pf logs with Gource (http://www.echothrust.com/blogs/monitoring-pf-logs-gource) If you're using pf (http://www.bsdnow.tv/tutorials/pf) on any of the BSDs, maybe you've gotten bored of grepping logs and want to do something more fancy This article will show you how to get set up with Gource for a cinematic-like experience If you've never heard of Gource, it's "an OpenGL-based 3D visualization tool intended for visualizing activity on source control repositories" When you put all the tools together, you can end up with some pretty eye-catching animations of your firewall traffic One of our listeners wrote in to say that he set this up and, almost immediately, noticed his girlfriend's phone had been compromised - graphical representations of traffic could be useful for detecting suspicious network activity *** pkgng 1.5.0 alpha1 released (https://svnweb.freebsd.org/ports?view=revision&revision=381573) The development version of pkgng was updated to 1.4.99.14, or 1.5.0 alpha1 This update introduces support for provides/requires, something that we've been wanting for a long time It will also now print which package is the reason for direct dependency change Another interesting addition is the "pkg -r" switch, allowing cross installation of packages Remember this isn't the stable version, so maybe don't upgrade to it just yet on any production systems DragonFly will also likely pick up this update once it's marked stable *** Welcome to OpenBSD (http://devio.us/~bcallah/rcos2015.pdf) We mentioned last week that our listener Brian was giving a talk in the Troy, New York area The slides from that talk are now online, and they've been generating quite a bit of discussion (https://news.ycombinator.com/item?id=9240533) online (https://www.reddit.com/r/openbsd/comments/2ztokc/welcome_to_openbsd/) It's simply titled "Welcome to OpenBSD" and gives the reader an introduction to the OS (and how easy it is to get involved with contributing) Topics include a quick history of the project, who the developers are and what they do, some proactive security techniques and finally how to get involved As you may know, NetBSD has almost 60 supported platforms (https://www.netbsd.org/ports/) and their slogan is "of course it runs NetBSD" - Brian says, with 17 platforms (http://www.openbsd.org/plat.html) over 13 CPU architectures, "it probably runs OpenBSD" No matter which BSD you might be interested in, these slides are a great read, especially for any beginners looking to get their feet wet Try to guess which font he used... *** BSDTalk episode 252 (http://bsdtalk.blogspot.com/2015/03/bsdtalk252-devious-with-brian-callahan.html) And somehow Brian has snuck himself into another news item this week He makes an appearance in the latest episode of BSD Talk (http://www.bsdnow.tv/episodes/2014_03_05-bsd_now_vs_bsdtalk), where he chats with Will about running a BSD-based shell provider If that sounds familiar, it's probably because we did the same thing (http://www.bsdnow.tv/episodes/2014_06_18-devious_methods), albeit with a different member of their team In this interview, they discuss what a shell provider does, hardware requirements and how to weed out the spammers in favor of real people They also talk a bit about the community aspect of a shared server, as opposed to just running a virtual machine by yourself *** Feedback/Questions Christian writes in (http://slexy.org/view/s2O81pixhq) Stefan writes in (http://slexy.org/view/s2dhr2WfVc) Possnfiffer writes in (http://slexy.org/view/s2Kisq2EqT) Ruudsch writes in (http://slexy.org/view/s2Xr0e5YAJ) Shane writes in (http://slexy.org/view/s2Xz7BNoJE) *** Mailing List Gold Accidental support (https://lists.freebsd.org/pipermail/svn-src-head/2015-March/069679.html) Larry's tears (https://www.marc.info/?l=openbsd-cvs&m=142686812913221&w=2) The boy who sailed with BSD (https://lists.freebsd.org/pipermail/freebsd-hardware/2015-March/007625.html) ***
This week on the show, we'll be chatting with Alex Reece and Matt Ahrens about what's new in the world of OpenZFS. After that, we're starting a new tutorial series on submitting your first patch. All the latest BSD news and answers to your emails, coming up on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Revisiting FreeBSD after 20 years (http://changelog.complete.org/archives/9317-has-linux-lost-its-way-comments-prompt-a-debian-developer-to-revisit-freebsd-after-20-years) With comments like "has Linux lost its way?" floating around, a Debian developer was prompted to revisit FreeBSD after nearly two decades This blog post goes through his experiences trying out a modern BSD variant, and includes the good, the bad and the ugly - not just praise this time He loves ZFS and the beadm tool, and finds the FreeBSD implementation to be much more stable than ZoL On the topic of jails, he summarizes: "Linux has tried so hard to get this right, and fallen on its face so many times, a person just wants to take pity sometimes. We've had linux-vserver, openvz, lxc, and still none of them match what FreeBSD jails have done for a long time." The post also goes through the "just plain different" aspects of a complete OS vs. a distribution of various things pieced together Finally, he includes some things he wasn't so happy about: subpar laptop support, virtualization being a bit behind, a myriad of complaints about pkgng and a few other things There was some decent discussion (https://news.ycombinator.com/item?id=9063216) on Hacker News about this article too, with counterpoints from both sides *** s2k15 hackathon report: network stack SMP (http://undeadly.org/cgi?action=article&sid=20150218085759) The first trip report from the recent OpenBSD hackathon in Australia has finally been submitted One of the themes of this hackathon was SMP (symmetric multiprocessing) improvement, and Martin Pieuchot did some hacking on the network stack If you're not familiar with him, he gave a presentation (http://www.openbsd.org/papers/tamingdragons.pdf) at EuroBSDCon last year, titled Taming OpenBSD Network Stack Dragons (https://va.ludost.net/files/eurobsdcon/2014/Rodopi/03.Saturday/03.Taming%20OpenBSD%20Network%20Stack%20Dragons%20-%20Martin%20Pieuchot.mp4) Teaming up with David Gwynne, they worked on getting some bits of the networking code out of the big lock (https://en.wikipedia.org/wiki/Giant_lock) Hopefully more trip reports will be sent in during the coming weeks Most of the big code changes should probably appear after the 5.7-release testing period *** From BIND to NSD and Unbound (https://www.tumfatig.net/20150215/bind-nsd-unbound-openbsd-5-6/) If you've been running a DNS server on any of the BSDs, you've probably noticed a semi-recent trend: BIND being replaced with Unbound BIND was ripped out in FreeBSD 10.0 and will be gone in OpenBSD 5.7, but both systems include Unbound now as an alternative OpenBSD goes a step further, also including NSD in the base system, whereas you'll need to install that from ports on FreeBSD Instead of one daemon doing everything like BIND tried to do, this new setup splits the authoritative nameserver and the caching resolver into two separate daemons This post takes you through the transitional phase of going from a single BIND setup to a combination of NSD and Unbound All in all, everyone wins here, as there will be a lot less security advisories in both BSDs because of it... *** m0n0wall calls it quits (http://m0n0.ch/wall/end_announcement.php) The original, classic BSD firewall distribution m0n0wall (https://en.wikipedia.org/wiki/M0n0wall) has finally decided to close up shop For those unfamiliar, m0n0wall was a FreeBSD-based firewall project that put a lot of focus on embedded devices: running from a CF card, CD, USB drive or even a floppy disk It started over twelve years ago, which is pretty amazing when you consider that's around half of FreeBSD itself's lifespan The project was probably a lot of people's first encounter with BSD in any form If you were a m0n0wall user, fear not, you've got plenty of choices for a potential replacement: doing it yourself with something like FreeBSD (http://blog.pcbsd.org/2015/01/using-trueos-as-a-ipfw-based-home-router/) or OpenBSD (http://www.bsdnow.tv/tutorials/openbsd-router), or going the premade route with something like pfSense (http://www.bsdnow.tv/episodes/2014_02_19-a_sixth_pfsense), OPNsense (http://www.bsdnow.tv/episodes/2015_01_14-common_sense_approach) or the BSD Router Project (http://www.bsdnow.tv/episodes/2014_10_22-dont_buy_a_router) The founder's announcement includes these closing words: "m0n0wall has served as the seed for several other well known open source projects, like pfSense, FreeNAS and AskoziaPBX. The newest offspring, OPNsense, aims to continue the open source spirit of m0n0wall while updating the technology to be ready for the future. In my view, it is the perfect way to bring the m0n0wall idea into 2015, and I encourage all current m0n0wall users to check out OPNsense and contribute if they can." While m0n0wall didn't get a lot of on-air mention, surely a lot of our listeners will remember it fondly *** Interview - Alex Reece & Matt Ahrens - alex@delphix.com (mailto:alex@delphix.com) & matt@delphix.com (mailto:matt@delphix.com) / @openzfs (https://twitter.com/openzfs) What's new in OpenZFS Tutorial Making your first patch (OpenBSD) (http://www.bsdnow.tv/tutorials/patching-obsd) News Roundup Overlaying remote LANs with OpenBSD's VXLAN (http://www.echothrust.com/blogs/using-openbsd-and-vxlan-overlay-remote-lans) Have you ever wanted to "merge" multiple remote LANs? OpenBSD's vxlan(4) (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/vxlan.4) is exactly what you need This article talks about using it to connect two virtualized infrastructures on different ESXi servers It gives a bit of networking background first, in case you're not quite up to speed on all this stuff This tool opens up a lot of very cool possibilities, even possibly doing a "remote" LAN party Be sure to check the AsiaBSDCon talk (https://www.youtube.com/watch?v=ufeEP_hzFN0) about VXLANs if you haven't already *** 2020, year of the PCBSD desktop (http://lukewolf.blogspot.com/2015/02/a-prediction-2020-year-of-pc-bsd-on.html) Here we have a blog post about BSD on the desktop, straight from a KDE developer He predicts that PCBSD is going to take off before the year 2020, possibly even overtaking Linux's desktop market share (small as it may be) With PCBSD making a preconfigured FreeBSD desktop a reality, and the new KMS work, the author is impressed with how far BSD has come as a viable desktop option ZFS and easy-to-use boot environments top the list of things he says differentiate the BSD desktop experience from the Linux one There was also some discussion on Slashdot (http://bsd.slashdot.org/story/15/02/16/2355236/pc-bsd-set-for-serious-growth) that might be worth reading *** OpenSSH host key rotation, redux (http://blog.djm.net.au/2015/02/hostkey-rotation-redux.html) We mentioned the new OpenSSH host key rotation and other goodies in a previous episode (http://www.bsdnow.tv/episodes/2015_02_04-from_the_foundation_1), but things have changed a little bit since then djm (http://www.bsdnow.tv/episodes/2013_12_18-cryptocrystalline) says "almost immediately after smugly declaring 'mission accomplished', the bug reports started rolling in." There were some initial complaints from developers about the new options, and a serious bug shortly thereafter After going back to the drawing board, he refactored some of the new code (and API) and added some more regression tests Most importantly, the bigger big fix was described as: "a malicious server (say, "host-a") could advertise the public key of another server (say, "host-b"). Then, when the client subsequently connects back to host-a, instead of answering the connection as usual itself, host-a could proxy the connection to host-b. This would cause the user to connect to host-b when they think they are connecting to host-a, which is a violation of the authentication the host key is supposed to provide." None of this code has been in a formal OpenSSH release just yet, but hopefully it will soon *** PCBSD tries out LibreSSL (https://github.com/pcbsd/pcbsd/commit/6ede13117dcee1272d7a7060b16818506874286e) PCBSD users may soon be seeing a lot less security problems because of two recent changes After switching over to OpenNTPD last week (http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change), PCBSD decides to give the portable LibreSSL (http://www.bsdnow.tv/episodes/2014_07_30-liberating_ssl) a try too Note that this is only for the packages built from ports, not the base system unfortunately They're not the first ones to do this - OPNsense has been experimenting with replacing OpenSSL in their ports tree for a little while now, and of course all of OpenBSD's ports are built against it A good number of patches (https://github.com/pcbsd/freebsd-ports/commit/2eee669f4d6ab9a641162ecda29b62ab921438eb) are still not committed in vanilla FreeBSD ports, so they had to borrow some from Bugzilla Look forward to Kris wearing a "keep calm and abandon OpenSSL (https://www.openbsdstore.com/cgi-bin/live/ecommerce.pl?site=shop_openbsdeurope_com&state=item&dept_id=01&sub_dept_id=01&product_id=TSHIRTOSSL)" shirt in the near future *** Feedback/Questions Benjamin writes in (http://slexy.org/view/s28nyJ5omV) Mike writes in (http://slexy.org/view/s2wYUmUmh0) Brad writes in (http://slexy.org/view/s2BAKAQvMt) *** Mailing List Gold Debian (https://lists.freebsd.org/pipermail/svn-src-head/2015-February/068405.html) Dejavu (https://lists.freebsd.org/pipermail/freebsd-current/2015-February/054580.html) Package gone missing (http://lists.dragonflybsd.org/pipermail/users/2015-February/207475.html) ***
© 2020-2025 Ivy Podcast Discovery LLC
