POPULARITY
24 episodes with freebsd foundation
3 episodes with freebsd foundation
2 episodes with freebsd foundation
Thank you to the folks at Sustain (https://sustainoss.org/) for providing the hosting account for CHAOSSCast! CHAOSScast – Episode 103 In this episode of CHAOSScast, host Alice Sowerby welcomes guests Miguel Ángel Fernández, Ed Maste, and Moin Rahman to talk about FreeBSD's Project adoption of GrimoireLab as part of work commissioned by the Sovereign Tech Agency (https://www.sovereign.tech/). They discuss the FreeBSD Foundation's need to understand their extensive bug backlog and how they leveraged GrimoireLab to gain insights and optimize bug management. The conversation also covers the challenges of deploying GrimoireLab on FreeBSD, resulting enhancements in the GrimoireLab metrics, and the overall benefits seen in managing the FreeBSD project. Press download to hear more! [00:00:45] The guests introduce themselves and share their backgrounds. [00:03:00] Alice explains the adoption of GrimoireLab by the FreeBSD project, facilitated by Bitergia who Miguel works for, to enhance project metrics and dashboarding capabilities. [00:04:02] Ed details FreeBSD's history, it integrated approach to software distribution, and its licensing benefits. [00:05:50] Miguel describes Bitergia's role in providing analytics for open source projects through GrimoireLab and how it integrates data from various development platforms. [00:07:38] Ed discusses the FreeBSD project's need for better tools to manage its large bug backlog, prompting the adoption of GrimoireLab. [00:10:57] Alice and Miguel discuss the translation of specific project needs into functional dashboards within GrimoireLab. [00:15:38] Ed highlights the utility of the Backlog Management Index (BMI) dashboard, which helps assess the effectiveness of efforts to manage and reduce the bug backlog. [00:18:20] Moin speaks on the practical use of GrimoireLab during bug busting sessions, which helped identify and close outdates or irrelevant bugs. [00:20:14] Moin explains the challenges and solutions of deploying GrimoireLab on FreeBSD, emphasizing the importance of utilizing FreeBSD's native capabilities over Linux-based solutions. [00:24:26] Miguel mentions new features added to GrimoireLab, inspired by their collaboration with the FreeBSD project, improving metrics for bug management. [00:28:05] Ed provides insights into the patterns of bug response times and highlights the need for proactive management of bugs that are unlikely to be addressed, and Miguel shares his positive thoughts on the community responding to the bugs. Value Adds (Picks) of the week: [00:32:10] Alice's pick is taking a walk in the Black Forest in Germany. [00:32:38] Ed's pick is going downhill skiing and taking a trip to a larger ski resort in Ontario. [00:33:21] Miguel's pick is starting to learn to play the piano a few months ago with an online course and how rewarding it's been. [00:34:17] Moin's pick is joining the FreeBSD Foundation and the strong sense of purpose it has given him. Panelist: Alice Sowerby Guests: Miguel Ángel Fernández Ed Maste Moin Rahman Links: CHAOSS (https://chaoss.community/) CHAOSS Project X (https://twitter.com/chaossproj) CHAOSScast Podcast (https://podcast.chaoss.community/) podcast@chaoss.community (mailto:podcast@chaoss.community) Alice Sowerby LinkedIn (https://www.linkedin.com/in/alice-sowerby-ba692a13/?originalSubdomain=uk) Miguel Ángel Fernández LinkedIn (https://www.linkedin.com/in/mafesan/) Ed Maste LinkedIn (https://www.linkedin.com/in/edmaste/?originalSubdomain=ca) Moin Rahman LinkedIn (https://www.linkedin.com/in/mmoinurrahman/) Sovereign Tech Agency (https://www.sovereign.tech/) Sovereign Tech Fund to Invest €686,400 in FreeBSD Infrastructure Modernization (https://freebsdfoundation.org/blog/sovereign-tech-fund-to-invest-e686400-in-freebsd-infrastructure-modernization/) FreeBSD's GrimoireLab instance (https://grimoire.freebsd.org/) Documentation for FreeBSD's GrimoireLab instance (https://github.com/freebsd/grimoire) FreeBSD Project (https://www.freebsd.org/) FreeBSD Foundation (https://freebsdfoundation.org/) Bitergia (https://bitergia.com/) Bitergia-GrimoireLab (https://bitergia.com/blog/opensource/grimoirelab-to-measure-organizations-private-development/) Black Forest-Germany (https://en.wikipedia.org/wiki/Black_Forest) Special Guests: Ed Maste, Miguel Ángel Fernández Sánchez , and Moin Rahman.
Meet Deb Goodkin! Deb is the Executive Director of the FreeBSD Foundation. The Foundation supports the FreeBSD project and community. FreeBSD is a powerful open-source operating system known for its reliability and security and used by companies like Netflix to power their servers and networks.With a strong background in engineering, Deb is passionate about open-source technology, and after this episode, you might become passionate about it, too! In this episode of the Scrimba podcast, you'll learn how a complex project like FreeBSD works from an organizational standpoint, why open-source is a great place for even newer developers, what are the key differences between community-driven projects and working for a corporation, and where should you start if you're looking to get into open-source.
Ever wondered how the backbone of streaming giants like Netflix or gaming consoles like Sony PlayStation ticks? Dive deep with us in this episode as we unravel the legacy and impact of FreeBSD, one of the unsung heroes of the tech world. Join us in an insightful conversation with Deb Goodkin, the Executive Director of the FreeBSD Foundation, as we explore the history, evolution, and significance of FreeBSD. From its roots in Berkeley UNIX to its integral role in modern technology, Deb sheds light on why this powerful, cohesive operating system stands out from Linux and how its permissive licensing has attracted major players. Discover the fascinating story behind FreeBSD's stability, performance, and community-driven development. Tune in for more intriguing details and expert insights you won't want to miss!
Guest Deb Goodkin Panelist Richard Littauer Show Notes This episode of Sustain features host Richard Littauer having a conversation with guest Deb Goodkin, the Executive Director of the FreeBSD Foundation. They discuss the history, development, and distinct characteristics of FreeBSD as an operating system independent from Linux, and the foundational support provided by the FreeBSD Foundation. The conversation delves into the organization's efforts in software development, security, community engagement, and plans to increase diversity. They also highlight the celebration of FreeBSD Day and Deb's passion for maintaining and promoting FreeBSD. Press download now to hear more! [00:00:55] Deb explains her role and clarifies that FreeBSD is an independent operating system, originally derived from Unix developed at Bell Labs. [00:02:27] Richard mentions “BSD” stands for “Berkeley Software Distribution.” Deb confirms the existence of the BSD license and its relevance. [00:03:27] Deb admits it's challenging to determine the number of FreeBSD users due to the nature of the BSD license but estimates hundreds of thousands based on device usage. [00:04:08] Deb distinguishes the FreeBSD Foundation from the FreeBSD Project, explaining the foundation's role in supporting the project and the community through funding, development, and advocacy. [00:06:47] Deb compares the FreeBSD Foundation to other foundations like the GNOME Foundation and the Linux Foundation, emphasizing FreeBSD's focus on software development and community support. [00:08:15] Deb reveals most of the foundation's funding comes from corporate sponsors. She highlights the foundation's increasing focus on security in response to global standards and government guidelines, mentioning FreeBSD's involvement in security research and development. [00:12:45] There's a discussion on the foundation's approach to policy and standards, noting the limited resources but emphasizing the importance of corporate and community input in deciding priorities and projects. [00:15:52] Richard questions Deb about feedback mechanisms used to gauge the FreeBSD community's concerns about security. He wonders about the participation dynamics. Deb details the communication channels within the FreeBSD community, mentions a recent community survey and what the responses revealed, and explains the 1% negative feedback. [00:19:56] The conversation shifts towards diversity and inclusion within FreeBSD, questioning the foundation's strategies to enhance geographical and demographic diversity. Deb discusses the foundation's efforts to increase gender diversity, mentioning initiatives targeting women, especially through involvement in computing events and universities. [00:24:05] Richard inquires about the prevalence of women in executive roles within open source foundations, sparking a conversation on gender dynamics in leadership and the intense dedication required in these roles. Deb shares her admiration for the passion and intelligence within the FreeBSD community and emphasizes the foundation's commitment to supporting the project and its people. [00:26:51] Why is Deb so passionate about FreeBSD? She reflects on her deep connection with the community, her continuous learning in the field, and her desire to see the project thrive long-term. [00:29:06] Richard acknowledges the upcoming FreeBSD Day, and Deb shares how it showcases the uses of FreeBSD with a week of activities. [00:31:26] Find out here you can learn more about FreeBSD and Deb's work online. {00:33:34] Richard asks about the FreeBSD logo, confused by the demon mascot. Deb clarifies that the mascot, named Beastie, is a playful representation related to the UNIX's background processes (daemons) and not an actual logo. Spotlight [00:35:34] Richard's spotlight is Philip Pullman's Trilogy book series. [00:36:12] Deb's spotlight is Michael Lucas's book, Absolute FreeBSD. Links SustainOSS (https://sustainoss.org/) SustainOSS Twitter (https://twitter.com/SustainOSS?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) SustainOSS Discourse (https://discourse.sustainoss.org/) podcast@sustainoss.org (mailto:podcast@sustainoss.org) SustainOSS Mastodon (https://mastodon.social/tags/sustainoss) Open Collective-SustainOSS (Contribute) (https://opencollective.com/sustainoss) Richard Littauer Socials (https://www.burntfen.com/2023-05-30/socials) Deb Goodkin X/Twitter (https://x.com/dgoodkin?lang=en) Deb Goodkin LinkedIn (https://www.linkedin.com/in/deb-goodkin-b282924a/) FreeBSD Foundation (https://freebsdfoundation.org/) 2024 FreeBSD Community Survey is Here-FreeBSD Foundation (https://freebsdfoundation.org/blog/2024-freebsd-community-survey-is-here/) FreeBSD Foundation YouTube (https://www.youtube.com/@freebsdfoundation2324) The FreeBSD Project (https://www.freebsd.org/) His Dark Materials by Philip Pullman-Wikipedia (https://en.wikipedia.org/wiki/His_Dark_Materials) Absolute FreeBSD by Michael W. Lucas (no starch press) (https://nostarch.com/absfreebsd3) Credits Produced by Richard Littauer (https://www.burntfen.com/) Edited by Paul M. Bahr at Peachtree Sound (https://www.peachtreesound.com/) Show notes by DeAnn Bahr Peachtree Sound (https://www.peachtreesound.com/) Special Guest: Deb Goodkin.
In this episode, SD Times Editor-in-Chief David Rubinstein is joined by FreeBSD Foundation executive director Deb Goodkin to discuss the 30th anniversary of the project, how it has evolved from its beginnings, and looks at how open-source software has exploded over that time.
FreeBSD Foundation's Software Development review of 2022, what can we learn from Vintage Computing, OpenBSD KDE Status Report 2022, a Decade of HardenedBSD, In Praise of Plan9, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines 2022 in Review: Software Development (https://freebsdfoundation.org/blog/2022-in-review-software-development/) What can we learn from Vintage Computing (https://github.com/readme/featured/vintage-computing) News Roundup OpenBSD KDE Status Report 2022 (https://www.sizeofvoid.org/posts/2022-26-12-openbsd-kde-status-report-2022/) A Decade of HardenedBSD (https://git.hardenedbsd.org/shawn.webb/articles/-/blob/master/hardenedbsd/2023-01_decade/article.md) In Praise of Plan9 (https://drewdevault.com/2022/11/12/In-praise-of-Plan-9.html) Beastie Bits LibreSSL 3.7.0 Released (https://undeadly.org/cgi?action=article;sid=20221212183516) OPNsense 22.7.10 released (https://opnsense.org/opnsense-22-7-10-released/) BSDCan 2023 call for papers (https://lists.bsdcan.org/pipermail/bsdcan-announce/2022-December/000194.html) How to lock OpenSSH authentication agent (https://sleeplessbeastie.eu/2022/12/28/how-to-lock-openssh-authentication-agent/) Once upon a time long ago, I was sitting alone in the UCLA ARPANET site... (https://mastodon.laurenweinstein.org/@lauren/109588605178700335) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Got a sale going right now on Lunduke Journal Subscriptions!* Founding Member Subscriptions: $75 / Year (normally $125)* Lifetime Subscriptions: $200 (normally $350)For the details on how to take advantage of it, see this post on Lunduke.Locals.com.It's time for some weird computer news! None of that mainstream, Big Tech mumbo jumbo. Just the weird, fun, alternative OS and retro stuff!Haiku Week is coming!From November 9th through November 16th, The Lunduke Journal Community will be celebrating “Haiku Week”.Much like with our previous “DOS Week” and “Command Line Week”, this will be an opportunity for all of us to embrace the wonderful world of the Haiku operating system (originally started as an open source replacement for BeOS).There is a very real possibility that our Haiku Week will line up with the Haiku Beta 4 release (which is “imminent”). So make sure you've got your Lunduke.Locals.com account setup and come join in the fun!FreeBSD 12.4 hits BetaThe first beta release of FreeBSD 12.4 have been released:“The first BETA build for the FreeBSD 12.4 release cycle is now available. ISO images for the amd64, armv6, armv7, arm64, i386, powerpc, powerpc64, powerpcspe, and sparc64 architectures are available.”The final release of 12.4 is expected during the first week of December (a little over a month from now). After which the release process for FreeBSD 13.2 is set to begin in January.The 12.x line is continuing support for the previous major release (as 13.x is current) for those still using it.In related news, the FreeBSD Foundation is falling dramatically behind in their fundraising goals — having raised less than 10% of their annual budget.“Our goal this year is to raise at a minimum $1,400,000 towards a spending budget of around $2,000,000. As we enter the last quarter of 2022, our donation total sits at $167,348”What that means for the viability of continuing to fund the FreeBSD Foundation remains to be seen. But it doesn't seem great.Ladybird adds support for more websites Last week we briefly touched on the rapid progress that Ladybird (the web browser engine and project that began within Serenity OS) has been making.In the few days since then, Andreas Kling (the head honcho of Serenity OS and Ladybird) has continued to showcase current improvements to how major websites are rendered in their new web browser. Including YouTube, Google, Instagram, and Google Maps.The results are nothing short of outstanding. Take a look:Why I bring this up:This is, quite possibly, one of the biggest shake-ups in the modern computing world… and it is happening rather quietly. Under the radar of most of the tech press.A brand new web browser (and engine) — developed entirely from scratch — is rapidly gaining significant compatibility with modern websites. Not only is this a massive win for Serenity (as modern web browsing is one of the major obstacles for new OS adoption nowadays)… but this could become a competitor to Mozilla and Google. Quite possibly very quickly (at least if the current development speed is maintained). All without the baggage of legacy code that those companies are saddled with.MACE - Mac Application Compatibility Environment I recently discovered MACE… and was intrigued.“The goal of M.A.C.E. project is the create an runtime library and executing environment for old Mac applications, similar to Apple's deprecated Classic environment, but without needing any ROM images or System Software binaries by Apple.”Running classic Mac applications — on a modern OS — without ROMs or the Mac System Software? Cool!According to their (rather extensive) July 2022 status update, they are working towards a full release:“We will continue working on the missing features as always, trying to reach the goal of a generic runtime release as soon as possible. We also will try to find solution for the binary file distribution in near future, so anybody interested trying the new features out could do that as soon as possible (especially considering the last released binaries are already getting quite old). We will keep you updated on this blog on our progress as always.”Want to really blow your mind? Here's a screenshot of MACE (which, again, no system software used) running SoftWindows (a PC emulator for 68k Macs)… running Windows 3.1.That's my kind of weird. Now, here's hoping the development team can get a release out so we can all play with it.This reminds me somewhat of “Executor” — a similar “run Mac software without MacOS” application that was released for DOS (and Linux).Seriously. That sale. It's rad.Got a sale going right now on Lunduke Journal Subscriptions!* Founding Member Subscriptions: $75 / Year (normally $125)* Lifetime Subscriptions: $200 (normally $350)For the details on how to take advantage of it, see this post on Lunduke.Locals.com. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit lunduke.substack.com/subscribe
Deb Goodkin Deb Goodkin is the Executive Director of the FreeBSD Foundation. LinkedIn: https://www.linkedin.com/in/deb-goodkin-b282924a/ Twitter: https://twitter.com/dgoodkin Notes: https://freebsdfoundation.org/ Credits: Music by ikson: https://www.iksonmusic.com Special Guest: Deb Goodkin.
ACM: It takes a community, Don't use discord for OSS projects, Unix in a browser tab, OpenIndiana Hipster 2021.10 available, Omni OS CE v11 is out, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines It takes a community - ACM (https://queue.acm.org/detail.cfm?id=3501361) PSA: Dont use Discord for Open Source Projects Jeffrey Paul - Discord Is Not An Acceptable Choice For Free Software Projects (https://sneak.berlin/20200220/discord-is-not-an-acceptable-choice-for-free-software-projects/) Drew deVault - Dont use Discord for FOSS (https://drewdevault.com/2021/12/28/Dont-use-Discord-for-FOSS.html) News Roundup Unix in your Browser Tab (https://browsix.org/) OpenIndiana Hipster 2021.10 is here (https://www.openindiana.org/2021/12/05/openindiana-hipster-2021-10-is-here/) Omni OS CE v11 r151040 is out (https://github.com/omniosorg/omnios-build/blob/r151040/doc/ReleaseNotes.md) Beastie Bits Deb from the FreeBSD Foundation on FLOSS Weekly (https://twit.tv/shows/floss-weekly/episodes/662?autostart=false) Jailfox - BastilleBSD template to bootstrap Firefox. (https://github.com/ddowse/jailfox) FreeBSD Journal Nov/Dec 2021 (https://freebsdfoundation.org/past-issues/storage-2/) First call through the 3ESS (https://www.youtube.com/watch?v=WUUsAK21f20) OpenBSD for minimalists (https://github.com/krzysztofengineer/openbsd) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dale - two zfs questions (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/439/feedback/Dale%20-%20two%20zfs%20questions.md) Johnny - home question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/439/feedback/Johnny%20-%20home%20question.md) Mike - GhostBSD in a VM (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/439/feedback/Mike%20-%20GhostBSD%20in%20a%20VM.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD Foundation reviews 2021 activities, DragonflyBSD 6.2.1 is here, Lumina Desktop 1.6.2 available, toolchain adventures, The OpenBSD BASED Challenge Day 7, Bastille Template: AdGuard Home, setting up ZSH on FreeBSD and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD Foundation 2021 in Review Software Development (https://freebsdfoundation.org/blog/2021-in-review-software-development/) Year End Fundraising Report (https://freebsdfoundation.org/blog/2021-year-end-fundraising-report/) Infrastructure Support (https://freebsdfoundation.org/blog/2021-in-review-infrastructure-support/) Advocacy (https://freebsdfoundation.org/blog/2021-in-review-advocacy/) FreeBSD 2022 CfP (https://freebsdfoundation.org/blog/freebsd-foundation-2022-call-for-proposals/) DragonFlyBSD 6.2.1 is out (https://www.dragonflybsd.org/release62/) News Roundup Lumina Desktop 1.6.2 is out (https://lumina-desktop.org/post/2021-12-25/) Toolchain Adventures (https://www.cambus.net/toolchains-adventures-q4-2021/) The OpenBSD BASED Challenge Day 7 (https://write.as/adventures-in-bsd/) Bastille Template: AdGuard Home (https://bastillebsd.org/blog/2022/01/03/bastille-template-examples-adguardhome/) Setting up ZSH on FreeBSD (https://www.danschmid.me/article/setting-up-zsh-on-freebsd) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions • Producers Note: We did get some Christmas AMA questions in after we recorded that episode (since we recorded it early) but don't worry, I've made a note of them and we'll save them for our next AMA episode. Patrick - Volume (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/438/feedback/Patrick%20-%20Volume.md) Reptilicus Rex - FreeBSD Docs Team (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/438/feedback/Reptilicus%20Rex%20-%20FreeBSD%20Docs%20Team.md) michael - question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/438/feedback/michael%20-%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Deb Goodkin, Executive Director of the FreeBSD Foundation joins Doc Searls and Shawn Powers on this episode of FLOSS Weekly. There are lots of ways FreeBSD is not Linux or vice versa. Or ways they're actually better for some purposes. Just ask Netflix. In a lively hour of conversation, Goodkin visits many operating systems and related topics regarding the open source world as well as FreeBSD. Hosts: Doc Searls and Shawn Powers Guest: Deb Goodkin Download or subscribe to this show at https://twit.tv/shows/floss-weekly Think your open source project should be on FLOSS Weekly? Email floss@twit.tv. Thanks to Lullabot's Jeff Robbins, web designer and musician, for our theme music. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsor: kolide.com/twit
Deb Goodkin, Executive Director of the FreeBSD Foundation joins Doc Searls and Shawn Powers on this episode of FLOSS Weekly. There are lots of ways FreeBSD is not Linux or vice versa. Or ways they're actually better for some purposes. Just ask Netflix. In a lively hour of conversation, Goodkin visits many operating systems and related topics regarding the open source world as well as FreeBSD. Hosts: Doc Searls and Shawn Powers Guest: Deb Goodkin Download or subscribe to this show at https://twit.tv/shows/floss-weekly Think your open source project should be on FLOSS Weekly? Email floss@twit.tv. Thanks to Lullabot's Jeff Robbins, web designer and musician, for our theme music. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsor: kolide.com/twit
Deb Goodkin, Executive Director of the FreeBSD Foundation joins Doc Searls and Shawn Powers on this episode of FLOSS Weekly. There are lots of ways FreeBSD is not Linux or vice versa. Or ways they're actually better for some purposes. Just ask Netflix. In a lively hour of conversation, Goodkin visits many operating systems and related topics regarding the open source world as well as FreeBSD. Hosts: Doc Searls and Shawn Powers Guest: Deb Goodkin Download or subscribe to this show at https://twit.tv/shows/floss-weekly Think your open source project should be on FLOSS Weekly? Email floss@twit.tv. Thanks to Lullabot's Jeff Robbins, web designer and musician, for our theme music. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsor: kolide.com/twit
Deb Goodkin, Executive Director of the FreeBSD Foundation joins Doc Searls and Shawn Powers on this episode of FLOSS Weekly. There are lots of ways FreeBSD is not Linux or vice versa. Or ways they're actually better for some purposes. Just ask Netflix. In a lively hour of conversation, Goodkin visits many operating systems and related topics regarding the open source world as well as FreeBSD. Hosts: Doc Searls and Shawn Powers Guest: Deb Goodkin Download or subscribe to this show at https://twit.tv/shows/floss-weekly Think your open source project should be on FLOSS Weekly? Email floss@twit.tv. Thanks to Lullabot's Jeff Robbins, web designer and musician, for our theme music. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsor: kolide.com/twit
Why You Should Use BSD Licensing for Your Next Open Source Project or Product, Update on FreeBSD Foundation Investment in Linuxulator, OPNsense 21.1.5 released, FreeBSD meetings on the Desktop, Running FreeBSD jails with containerd 1.5, Markdown, DocBook, and the quest for semantic documentation on NetBSD.org, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines Why You Should Use BSD Licensing for Your Next Open Source Project or Product (https://klarasystems.com/articles/why-you-should-use-bsd-licensing-for-your-next-open-source-project-or-product/) The term “open source” has its origins in the context of software development, designating a specific approach to developing computer programs. Nowadays, however, it stands for a broad set of values – open source means open exchange, transparency, collaborative participation and development for the benefit of the entire community. Update on FreeBSD Foundation Investment in Linuxulator (https://freebsdfoundation.org/blog/update-on-freebsd-foundation-investment-in-linuxulator/) Dr. Emmett Brown’s similar-sounding Flux Capacitor from the movie Back to the Future bridged the dimension of time, uniting past, present, and future for the McFlys. Similarly, the FreeBSDⓇ Linuxulator project also bridges dimensions – in our case, these are LinuxⓇ and FreeBSD. News Roundup OPNsense 21.1.5 released (https://opnsense.org/opnsense-21-1-5-released/) This is mainly a security and reliablility update. There are several FreeBSD security advisories and updates for third party tools such as curl. + OPNsense to rebase on FreeBSD 13 (https://forum.opnsense.org/index.php?topic=22761.msg108313#msg108313) FreeBSD meetings on the Desktop (https://euroquis.nl//freebsd/2021/04/20/fbsd-bbb.html) FreeBSD on the desktop is a whole stack - X11, Qt, KDE Frameworks, KDE Plasma and KDE Gear, and Wayland, and Poppler and GTK - o my! Running FreeBSD jails with containerd 1.5 (https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/) containerd 1.5.0 was released today and now works on a new operating system: FreeBSD! This new release includes a series of patches (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) which allow containerd to build, enable the native and zfs snapshotters, and use a compatible runtime like runj. Markdown, DocBook, and the quest for semantic documentation on NetBSD.org (https://washbear.neocities.org/markdown.html) Recently, I’ve been doing a lot of maintenance of the NetBSD website. It contains a boatload of documentation, much of which was originally written in the 2000s. It has some special requirements: it has to work in text-based web browsers like lynx, or maybe even without any working browser installed at all, or just ftp(1) for downloading plain text over HTTP. Naturally, the most important parts are static, suitable for serving from the standard NetBSD http server, which runs from inetd by default. Beastie Bits Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Alrekur - An Interesting FreeBSD Find (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/403/feedback/Alrekur%20-%20An%20Interesting%20FreeBSD%20Find) They presented at the FreeBSD Vendor summit last year too: https://www.youtube.com/watch?v=8LUdZseNrpE Sven - feedback (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/403/feedback/Sven%20-%20feedback) Robert - firewalling (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/403/feedback/Robert%20-%20firewalling) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD 13 BETA Benchmarks, FreeBSD Jails Deep Dive by Klara Systems, FreeBSD Foundation looking for a Senior Arm Kernel Engineer & OSS Project Coordinator, macOS-Inspired BSD Desktop OS by helloSystem, A Trip into FreeBSD and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD 13 BETA Benchmarks - Performance Is Much Better (https://www.phoronix.com/scan.php?page=article&item=freebsd-13-beta1&num=6) FreeBSD Jails – Deep Dive into the Beginning of FreeBSD Containers (https://klarasystems.com/articles/freebsd-jails-the-beginning-of-freebsd-containers/) In recent years, containers and virtualization have become a buzzword in the Linux community, especially with the rise of Docker and Kubernetes. What many people probably don’t realize is that these ideas have been around for a very long time. Today, we will be looking at Jails and how they became part of FreeBSD. News Roundup FreeBSD Jobs The FreeBSD Foundation is looking for a Senior Arm Kernel Engineer (https://www.fossjobs.net/job/10369/senior-arm-kernel-engineer-at-the-freebsd-foundation/) The FreeBSD Foundation is also looking for an Open Source Project Coordinator. (https://www.fossjobs.net/job/10367/freebsd-open-source-project-coordinator-at-freebsd/) *** ### helloSystem Releases New ISOs For This macOS-Inspired BSD Desktop OS (https://www.phoronix.com/scan.php?page=news_item&px=helloSystem-New-12.1-Exp-ISOs) > The helloSystem motto is being a "desktop system for creators with focus on simplicity, elegance, and usability. Based on FreeBSD. Less, but better!" The desktop utilities are written with PyQt5. *** ### A Trip into FreeBSD (https://christine.website/blog/a-trip-into-freebsd-2021-02-13) > I normally deal with Linux machines. Linux is what I know and it's what I've been using since I was in college. A friend of mine has been coaxing me into trying out FreeBSD, and I decided to try it out and see what it's like. Here's some details about my experience and what I've learned. *** ###Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Beastie Bits Testing Linux Steam Proton on GhostBSD with BSD linuxulator - NO Audio (https://www.youtube.com/watch?v=H8ihW0m3bRQ) New Build of DragonFlyBSD 5.8 (http://lists.dragonflybsd.org/pipermail/users/2021-February/381550.html) Install OpenBSD 6.8 on PINE64 ROCK64 Media Board (https://github.com/krjdev/rock64_openbsd) FOSDEM BSD Track Videos are up (https://fosdem.org/2021/schedule/track/bsd/) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** Special Guest: Dan Langille.
FreeBSD Remote Process Plugin Final Milestone achieved, Tailscale for OpenBSD, macOS to FreeBSD migration, monitoring of our OpenBSD machines, OPNsense 20.7.6 released, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD Remote Process Plugin: Final Milestone Achieved (https://www.moritz.systems/blog/freebsd-remote-plugin-final-milestone-achieved/) Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are working on a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one. Tailscale on OpenBSD (https://rakhesh.com/linux-bsd/tailscale-on-openbsd/) I spent some time setting this up today evening and thought I’d post the steps here. Nothing fancy, just putting together various pieces actually. I assume you know what Tailscale is; if not check out their website. Basically it is a mesh network built on top of Wireguard. Using it you can have all your devices both within your LAN(s) and outside be on one overlay network as if they are all on the same LAN and can talk to each other. It’s my new favourite thing! News Roundup macOS to FreeBSD migration a.k.a why I left macOS (https://antranigv.am/weblog_en/posts/macos_to_freebsd/) This is not a technical documentation for how I migrated from macOS to FreeBSD. This is a high-level for why I migrated from macOS to FreeBSD. Not so long ago, I was using macOS as my daily driver. The main reason why I got a macbook was the underlying BSD Unix and the nice graphics it provides. Also, I have an iPhone. But they were also the same reasons for why I left macOS. Our monitoring of our OpenBSD machines, such as it is (as of November 2020 (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/OurOpenBSDMonitoring) We have a number of OpenBSD firewalls in service (along with some other OpenBSD servers for things like VPN endpoints), and I was recently asked how we monitor PF and overall network traffic on them. I had to disappoint the person who asked with my answer, because right now we mostly don't (although this is starting to change). OPNsense 20.7.6 released (https://opnsense.org/opnsense-20-7-6-released/) This update brings the usual mix of reliability fixes, plugin and third party software updates: FreeBSD, HardenedBSD, PHP, OpenSSH, StrongSwan, Suricata and Syslog-ng amongst others. Please note that Let's Encrypt users need to reissue their certificates manually after upgrading to this version to fix the embedded certificate chain issue with the current signing CA switch going on. NYC Bug Jan 2021 with Michael W. Lucas (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/nycbug) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions cy - .so files (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/cy%20-%20.so%20files) ben - mixer volume (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/ben%20-%20mixer%20volume) probono - live cds (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/probono%20-%20live%20cds) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
bhyve - The FreeBSD Hypervisor, udf information leak, being a vim user instead of classic vi, FreeBSD on ESXi ARM Fling: Fixing Virtual Hardware, new FreeBSD Remote Process Plugin in LLDB, OpenBSD Laptop, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines bhyve - The FreeBSD Hypervisor (https://klarasystems.com/articles/bhyve-the-freebsd-hypervisor/) FreeBSD has had varying degrees of support as a hypervisor host throughout its history. For a time during the mid-2000s, VMWare Workstation 3.x could be made to run under FreeBSD’s Linux Emulation, and Qemu was ported in 2004, and later the kQemu accelerator in 2005. Then in 2009 a port for VirtualBox was introduced. All of these solutions suffered from being a solution designed for a different operating system and then ported to FreeBSD, requiring constant maintenance. ZFS and FreeBSD Support Klara offers flexible Support Subscriptions for your ZFS and FreeBSD infrastructure. Get a world class team of experts to back you up. Check it out on our website! (https://klarasystems.com/support/) udf info leak (https://gist.github.com/CTurt/a00fb4164e13342567830b052aaed94b) FreeBSD UDF driver info leak Analysis done on FreeBSD release 11.0 because that's what I had around. + Fix committed to FreeBSD (https://svnweb.freebsd.org/changeset/base/366005) News Roundup I'm now a user of Vim, not classical Vi (partly because of windows) (https://utcc.utoronto.ca/~cks/space/blog/unix/VimNowAUser) In the past I've written entries (such as this one) where I said that I was pretty much a Vi user, not really a Vim user, because I almost entirely stuck to Vi features. In a comment on my entry on not using and exploring Vim features, rjc reinforced this, saying that I seemed to be using vi instead of vim (and that there was nothing wrong with this). For a long time I thought this way myself, but these days this is not true any more. These days I really want Vim, not classical Vi. FreeBSD on ESXi ARM Fling: Fixing Virtual Hardware (https://vincerants.com/freebsd-on-esxi-arm-fling-fixing-virtual-hardware/) With the current state of FreeBSD on ARM in general, a number of hardware drivers are either set to not auto-load on boot, or are entirely missing altogether. This page is to document my findings with various bits of hardware, and if possible, list fixes. Introduction of a new FreeBSD Remote Process Plugin in LLDB (https://www.moritz.systems/blog/introduction-of-a-new-freebsd-remote-process-plugin-in-lldb/) Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are writing a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one. OpenBSD Laptop (https://functionallyparanoid.com/2020/10/14/openbsd-laptop/) Hi, I know it’s been a while. I recently had to nuke and re-pave my personal laptop and I thought it would be a nice thing to share with the community how I set up OpenBSD on it so that I have a useful, modern, secure environment for getting work done. I’m not going to say I’m the expert on this or that this is the BEST way to set up OpenBSD, but I thought it would be worthwhile for folks doing Google searches to at least get my opinion on this. So, given that, let’s go… Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Ethan - Linux user wanting to try out OpenBSD (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/375/feedback/Ethan%20-%20Linux%20user%20wanting%20to%20try%20out%20OpenBSD.md) iian - Learning IT (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/375/feedback/iian%20-%20Learning%20IT.md) johnny - bsd swag (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/375/feedback/johnny%20-%20bsd%20swag.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
This is a very special episode of a 45 minute full interview with someone who worked on BSD and remains actively involved in FreeBSD as well. Dr. Marshall Kirk McKusick is a computer scientist, known for his extensive work on BSD UNIX, from the 1980s to FreeBSD in the present day. Questions *What do you feel when you look back on the history of BSD and what it became today including FreeBSD? * Did You get involved eventually with FreeBSD after BSD ceased to exist because you felt FreeBSD could be the proper continuation of the BSD flag so to say? * Text editor of your choice, Which one are you using? *What do you think of the ZFS file system? *If BSD was not frozen up for 3 long years during the AT&T Lawsuit period which allowed Linux to get a head start.. Would BSD be what Linux is today? * Linus Torvalds wrote an article back in 2004 about the prediction of the death of BSD, How much of Linus’ prediction has come true? * Kirk’s comments on my statement regarding the Linux Foundation receiving a lot of money through major corporations as Gold and Platinum sponsors (Microsoft, etc.) which is kind of a takeover is already happening in place steering its focus towards what these company’s have in their interests while the FreeBSD Foundation receiving very little money compared to the Linux Foundation while still the FreeBSD Foundation is doing an impressive job with the much smaller amount they are getting. * Did You ever miss the chance you were given to go and work for Sun Microsystems? More info: https://www.mckusick.com https://tsr-podcast.com
The FreeBSD Foundation celebrated its 20th anniversary this year. We talk to Deb Goodkin, executive director of the foundation, about how the foundation got started and where it is going as well as some new resources for developers to take advantage of.
FreeBSD on Power, DragonflyBSD 5.8 is here, Unifying FreeNAS/TrueNAS, OpenBSD vs. Prometheus and Go, gcc 4.2.1 removed from FreeBSD base, and more. Headlines FreeBSD on Power (https://www.freebsdfoundation.org/blog/power-to-the-people-making-freebsd-a-first-class-citizen-on-power/) The power and promise of all open source software is freedom. Another way to express freedom is choice — choice of platforms, deployment models, stacks, configurations, etc. The FreeBSD Foundation is dedicated to supporting and promoting the FreeBSD Project and community worldwide. But, what does this mean, exactly, you may wonder. The truth is it means many different things, but in all cases the Foundation acts to expand freedom and choice so that FreeBSD users have the power to serve their varied compute needs. This blog tells the story of one specific way the Foundation helps a member of the community provide greater hardware choice for all FreeBSD users. Dragonfly 5.8 (https://www.dragonflybsd.org/release58/) DragonFly version 5.8 brings a new dsynth utility for building your own binary dports packages, plus significant support work to speed up that build - up to and including the entire collection. Additional progress has been made on GPU and signal support. The details of all commits between the 5.6 and 5.8 branches are available in the associated commit messages for 5.8.0rc1 and 5.8.0. Also see /usr/src/UPDATING for specific file changes in PAM. See article for rest of information 2nd HamBUG meeting recap (https://www.hambug.ca/) The second meeting of the Hamilton BSD Users Group took place last night The next meeting is scheduled for the 2nd Tuesday of the month, April 14th 2020 News Roundup FreeNAS/TrueNAS Brand Unification (https://www.ixsystems.com/blog/freenas-truenas-unification/) FreeNAS and TrueNAS have been separate-but-related members of the #1 Open Source storage software family since 2012. FreeNAS is the free Open Source version with an expert community and has led the pursuit of innovations like Plugins and VMs. TrueNAS is the enterprise version for organizations of all sizes that need additional uptime and performance, as well as the enterprise-grade support necessary for critical data and applications. From the beginning at iXsystems, we’ve developed, tested, documented, and released both as separate products, even though the vast majority of code is shared. This was a deliberate technical decision in the beginning but over time became less of a necessity and more of “just how we’ve always done it”. Furthermore, to change it was going to require a serious overhaul to how we build and package both products, among other things, so we continued to kick the can down the road. As we made systematic improvements to development and QA efficiency over the past few years, the redundant release process became almost impossible to ignore as our next major efficiency roadblock to overcome. So, we’ve finally rolled up our sleeves. With the recent 11.3 release, TrueNAS gained parity with FreeNAS on features like VMs and Plugins, further homogenizing the code. Today, we announce the next phase of evolution for FreeNAS and TrueNAS. OpenBSD versus Prometheus (and Go). (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/OpenBSDVsPrometheusAndGo) We have a decent number of OpenBSD machines that do important things (and that have sometimes experienced problems like running out of disk space), and we have a Prometheus based metrics and monitoring system. The Prometheus host agent has enough support for OpenBSD to be able to report on critical metrics, including things like local disk space. Despite all of this, after some investigation I've determined that it's not really sensible to even try to deploy the host agent on our OpenBSD machines. This is due to a combination of factors that have at their root OpenBSD's lack of ABI stability FreeBSD removed gcc from base (https://svnweb.freebsd.org/base?view=revision&revision=358454) As described in Warner's email message[1] to the FreeBSD-arch mailing list we have reached GCC 4.2.1's retirement date. At this time all supported architectures either use in-tree Clang, or rely on external toolchain (i.e., a contemporary GCC version from ports). GCC 4.2.1 was released July 18, 2007 and was imported into FreeBSD later that year, in r171825. GCC has served us well, but version 4.2.1 is obsolete and not used by default on any architecture in FreeBSD. It does not support modern C and does not support arm64 or RISC-V. Beastie Bits New Archive location for Dragonfly 4.x (https://www.dragonflydigest.com/2020/03/10/24276.html) A dead simple git cheat sheet (https://hub.iwebthings.com/a-dead-simple-git-cheatsheet/) Xorg 1.20.7 on HardenedBSD Comes with IE/RELRO+BIND_NOW/CFI/SafeStack Protections (https://twitter.com/lattera/status/1233412881569415168) Feedback/Questions Niclas writes in Regarding the Lenovo E595 user (episode 340) (http://dpaste.com/2YJ6PFW#wrap) Lyubomir writes about GELI and ZFS (http://dpaste.com/1S0DGT3#wrap) Peter writes in about scaling FreeBSD jails (http://dpaste.com/2FSZQ8V#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
Hyperbola Developer interview, why you should migrate from Linux to BSD, FreeBSD is an amazing OS, improving the ptrace(2) API in LLVM 10, First FreeBSD conference in Australia, and a guide to containers on FreeNAS. Headlines FreeBSD is an amazing operating System (https://www.unixsheikh.com/articles/freebsd-is-an-amazing-operating-system.html) Update 2020-01-21: Since I wrote this article it got posted on Hacker News, Reddit and Lobster, and a few people have emailed me with comments. I have updated the article with comments where I have found it needed. As an important side note I would like to point out that I am not a FreeBSD developer, there may be things going on in the FreeBSD world that I know absolutely nothing about. I am also not glued to the FreeBSD developer mailing lists. I am not a FreeBSD "fanboy". I have been using GNU/Linux a ton more for the past two decades than FreeBSD, mainly due to hardware incompatibility (lacking or buggy drivers), and I love both Debian GNU/Linux and Arch Linux just as much as FreeBSD. However, I am concerned about the development of GNU/Linux as of late. Also this article is not about me trying to make anyone switch from something else to FreeBSD. It's about why I like FreeBSD and that I recommend you try it out if you're into messing with operating systems. I think the year was late 1999 or mid 2000 when I one day was browsing computer books at my favorite bookshop and I discovered the book The Complete FreeBSD third edition from 1999 by Greg Lehey. With the book came 4 CD Roms with FreeBSD 3.3. I had already familiarized myself with GNU/Linux in 1998, and I was in the process of migrating every server and desktop operating system away from Microsoft Windows, both at home and at my company, to GNU/Linux, initially Red Hat Linux and then later Debian GNU/Linux, which eventually became my favorite GNU/Linux distribution for many years. When I first saw The Complete FreeBSD book by Greg Lehey I remember noticing the text on the front page that said, "The Free Version of Berkeley UNIX" and "Rock Solid Stability", and I was immediately intrigued! What was that all about? A free UNIX operating system! And rock solid stability? That sounded amazing. Hyperbola Dev Interview (https://itsfoss.com/hyperbola-linux-bsd/) In late December 2019, Hyperbola announced that they would be making major changes to their project. They have decided to drop the Linux kernel in favor of forking the OpenBSD kernel. This announcement only came months after Project Trident announced that they were going in the opposite direction (from BSD to Linux). Hyperbola also plans to replace all software that is not GPL v3 compliant with new versions that are. To get more insight into the future of their new project, I interviewed Andre, co-founder of Hyperbola. News Roundup Improving the ptrace(2) API and preparing for LLVM-10.0 (https://blog.netbsd.org/tnf/entry/improving_the_ptrace_2_api) This month I have improved the NetBSD ptrace(2) API, removing one legacy interface with a few flaws and replacing it with two new calls with new features, and removing technical debt. As LLVM 10.0 is branching now soon (Jan 15th 2020), I worked on proper support of the LLVM features for NetBSD 9.0 (today RC1) and NetBSD HEAD (future 10.0). The first FreeBSD conference in Australia (https://rubenerd.com/the-first-freebsd-conference-in-australia/) FreeBSD has existed as an operating system, project, and foundation for more than twenty years, and its earlier incantations have exited for far longer. The old guard have been developing code, porting software, and writing documentation for longer than I’ve existed. I’ve been using it for more than a decade for personal projects, and professionally for half that time. While there are many prominent Australian FreeBSD contributors, sysadmins, and users, we’ve always had to venture overseas for conferences. We’re always told Australians are among the most ardent travellers, but I always wondered if we could do a domestic event as well. And on Tuesday, we did! Deb Goodkin and the FreeBSD Foundation graciously organised and chaired a dedicated FreeBSD miniconf at the long-running linux.conf.au event held each year in a different city in Australia and New Zealand. A practical guide to containers on FreeNAS for a depraved psychopath (https://medium.com/@andoriyu/a-practical-guide-to-containers-on-freenas-for-a-depraved-psychopath-c212203c0394) This is a simple write-up to setup Docker on FreeNAS 11 or FreeBSD 11. But muh jails? You know that jails are dope and you know that jails are dope, yet no one else knows it. So here we are stuck with docker. Two years ago I would be the last person to recommend using docker, but a whole lot of things has changes past years… So jails are dead then? No, jails are still dope, but jails lack tools to manage them. Yes, there are a few tools, but they meant for hard-core FreeBSD users who used to suffering. Docker allows you to run applications without deep knowledge of application you’re running. It will also allow you to run applications that are not ported to FreeBSD. Why you should migrate everything from Linux to BSD (https://www.unixsheikh.com/articles/why-you-should-migrate-everything-from-linux-to-bsd.html) As an operating system GNU/Linux has become a real mess because of the fragmented nature of the project, the bloatware in the kernel, and because of the jerking around by commercial interests. Response Should you migrate from Linux to BSD? It depends. (https://fediverse.blog/~/AllGoodThings/should-you-migrate-from-linux-to-bsd-it-depends) Beastie Bits Using the OpenBSD ports tree with dedicated users (https://dataswamp.org/~solene/2020-01-11-privsep.html) broot on FreeBSD (https://vermaden.wordpress.com/2020/01/10/run-broot-on-freebsd/) A Trip down Memory Lane (https://svnweb.freebsd.org/base/head/share/misc/bsd-family-tree?view=co) Running syslog-ng in BastilleBSD (https://www.syslog-ng.com/community/b/blog/posts/running-syslog-ng-in-bastillebsd) NASA : Using Software Packages in pkgsrc (https://www.nas.nasa.gov/hecc/support/kb/using-software-packages-in-pkgsrc_493.html) Feedback/Questions All of our questions this week were pretty technical in nature so I'm going to save those for the next episode so Allan can weigh in on them, since if we cover them now we're basically going to be deferring to Allan anyway. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
Your Impact on FreeBSD in 2019, Wireguard on OpenBSD Router, Amazon now has FreeBSD/ARM 12, pkgsrc-2019Q4, The Joys of UNIX Keyboards, OpenBSD on Digital Ocean, and more. Headlines Your Impact on FreeBSD in 2019 (https://www.freebsdfoundation.org/blog/your-impact-on-freebsd-in-2019/) It’s hard to believe that 2019 is nearly over. It has been an amazing year for supporting the FreeBSD Project and community! Why do I say that? Because as I reflect over the past 12 months, I realize how many events we’ve attended all over the world, and how many lives we’ve touched in so many ways. From advocating for FreeBSD to implementing FreeBSD features, my team has been there to help make FreeBSD the best open source project and operating system out there. In 2019, we focused on supporting a few key areas where the Project needed the most help. The first area was software development. Whether it was contracting FreeBSD developers to work on projects like wifi support, to providing internal staff to quickly implement hardware workarounds, we’ve stepped in to help keep FreeBSD innovative, secure, and reliable. Software development includes supporting the tools and infrastructure that make the development process go smoothly, and we’re on it with team members heading up the Continuous Integration efforts, and actively involved in the clusteradmin and security teams. Our advocacy efforts focused on recruiting new users and contributors to the Project. We attended and participated in 38 conferences and events in 21 countries. From giving FreeBSD presentations and workshops to staffing tables, we were able to have 1:1 conversations with thousands of attendees. Our travels also provided opportunities to talk directly with FreeBSD commercial and individual users, contributors, and future FreeBSD user/contributors. We’ve seen an increase in use and interest in FreeBSD from all of these organizations and individuals. These meetings give us a chance to learn more about what organizations need and what they and other individuals are working on. The information helps inform the work we should fund. Wireguard on OpenBSD Router (https://obscurity.xyz/bsd/open/wireguard.html) wireguard (wg) is a modern vpn protocol, using the latest class of encryption algorithms while at the same time promising speed and a small code base. modern crypto and lean code are also tenants of openbsd, thus it was a no brainer to migrate my router from openvpn over to wireguard. my setup : a collection of devices, both wired and wireless, that are nat’d through my router (openbsd 6.6) out via my vpn provider azire* and out to the internet using wg-quick to start wg. running : doubtless this could be improved on, but currently i start wg manually when my router boots. this, and the nat'ing on the vpn interface mean its impossible for clients to connect to the internet without the vpn being up. as my router is on a ups and only reboots when a kernel patch requires it, it’s a compromise i can live with. run wg-quick (please replace vpn with whatever you named your wg .conf file.) and reload pf rules. News Roundup Amazon now has FreeBSD/ARM 12 (https://aws.amazon.com/marketplace/pp/B081NF7BY7) AWS, the cloud division of Amazon, announced in December the next generation of its ARM processors, the Graviton2. This is a custom chip design with a 7nm architecture. It is based on 64-bit ARM Neoverse cores. Compared to first-generation Graviton processors (A1), today’s new chips should deliver up to 7x the performance of A1 instances in some cases. Floating point performance is now twice as fast. There are additional memory channels and cache speed memory access should be much faster. The company is working on three types of Graviton2 EC2 instances that should be available soon. Instances with a “g” suffix are powered by Graviton2 chips. If they have a “d” suffix, it also means that they have NVMe local storage. General-purpose instances (M6g and M6gd) Compute-optimized instances (C6g and C6gd) Memory-optimized instances (R6g and R6gd) You can choose instances with up to 64 vCPUs, 512 GiB of memory and 25 Gbps networking. And you can see that ARM-powered servers are not just a fad. AWS already promises a 40% better price/performance ratio with ARM-based instances when you compare them with x86-based instances. AWS has been working with operating system vendors and independent software vendors to help them release software that runs on ARM. ARM-based EC2 instances support Amazon Linux 2, Ubuntu, Red Hat, SUSE, Fedora, Debian and FreeBSD. It also works with multiple container services (Docker, Amazon ECS, and Amazon Elastic Kubernetes Service). Coverage of AWS Announcement (https://techcrunch.com/2019/12/03/aws-announces-new-arm-based-instances-with-graviton2-processors/) Announcing the pkgsrc-2019Q4 release (https://mail-index.netbsd.org/pkgsrc-users/2020/01/06/msg030130.html) The pkgsrc developers are proud to announce the 65th quarterly release of pkgsrc, the cross-platform packaging system. pkgsrc is available with more than 20,000 packages, running on 23 separate platforms; more information on pkgsrc itself is available at https://www.pkgsrc.org/ In total, 190 packages were added, 96 packages were removed, and 1,868 package updates (to 1388 unique packages) were processed since the pkgsrc-2019Q3 release. As usual, a large number of updates and additions were processed for packages for go (14), guile (11), perl (170), php (10), python (426), and ruby (110). This continues pkgsrc's tradition of adding useful packages, updating many packages to more current versions, and pruning unmaintained packages that are believed to have essentially no users. The Joys of UNIX Keyboards (https://donatstudios.com/UNIX-Keyboards) I fell in love with a dead keyboard layout. A decade or so ago while helping a friends father clean out an old building, we came across an ancient Sun Microsystems server. We found it curious. Everything about it was different from what we were used to. The command line was black on white, the connectors strange and foreign, and the keyboard layout was bizarre. We never did much with it; turning it on made all the lights in his home dim, and our joint knowledge of UNIX was nonexistent. It sat in his bedroom for years supporting his television at the foot of his bed. I never forgot that keyboard though. The thought that there was this alternative layout out there seemed intriguing to me. OpenBSD on Digital Ocean (https://www.going-flying.com/blog/openbsd-on-digitalocean.html) Last night I had a need to put together a new OpenBSD machine. Since I already use DigitalOcean for one of my public DNS servers I wanted to use them for this need but sadly like all too many of the cloud providers they don't support OpenBSD. Now they do support FreeBSD and I found a couple writeups that show how to use FreeBSD as a shim to install OpenBSD. They are both sort of old at this point and with OpenBSD 6.6 out I ran into a bit of a snag. The default these days is to use a GPT partition table to enable EFI booting. This is generally pretty sane but it looks to me like the FreeBSD droplet doesn't support this. After the installer rebooted the VM failed to boot, being unable to find the bootloader. Thankfully DigitalOcean has a recovery ISO that you can boot by simply switching to it and powering off and then on your Droplet. Beastie Bits FreeBSD defaults to LLVM on PPC (https://svnweb.freebsd.org/base?view=revision&revision=356111) Theo De Raadt Interview between Ottawa 2019 Hackathon and BSDCAN 2019 (https://undeadly.org/cgi?action=article;sid=20191231214356) Bastille Poll about what people would like to see in 2020 (https://twitter.com/BastilleBSD/status/1211475103143251968) Notes on the classic book : The Design of the UNIX Operating System (https://github.com/suvratapte/Maurice-Bach-Notes) Multics History (https://www.multicians.org/) First meeting of the Hamilton BSD user group, February 11, 2020 18:30 - 21:00, Boston Pizza on Upper James St (http://studybsd.com/) Feedback/Questions Bill - 1.1 CDROM (http://dpaste.com/2H9CW6R) Greg - More 50 Year anniversary information (http://dpaste.com/2SGA3KY) Dave - Question time for Allan (http://dpaste.com/3ZAEKHD#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
vBSDcon 2019 recap, Unix at 50, OpenBSD on fan-less Tuxedo InfinityBook, humungus - an hg server, how to configure a network dump in FreeBSD, and more. Headlines vBSDcon Recap Allan and Benedict attended vBSDcon 2019, which ended last week. It was held again at the Hyatt Regency Reston and the main conference was organized by Dan Langille of BSDCan fame.The two day conference was preceded by a one day FreeBSD hackathon, where FreeBSD developers had the chance to work on patches and PRs. In the evening, a reception was held to welcome attendees and give them a chance to chat and get to know each other over food and drinks. The first day of the conference was opened with a Keynote by Paul Vixie about DNS over HTTPS (DoH). He explained how we got to the current state and what challenges (technical and social) this entails. If you missed this talk and are dying to see it, it will also be presented at EuroBSDCon next week John Baldwin followed up by giving an overview of the work on “In-Kernel TLS Framing and Encryption for FreeBSD” abstract (https://www.vbsdcon.com/schedule/2019-09-06.html#talk:132615) and the recent commit we covered in episode 313. Meanwhile, Brian Callahan was giving a separate session in another room about “Learning to (Open)BSD through its porting system: an attendee-driven educational session” where people had the chance to learn about how to create ports for the BSDs. David Fullard’s talk about “Transitioning from FreeNAS to FreeBSD” was his first talk at a BSD conference and described how he built his own home NAS setup trying to replicate FreeNAS’ functionality on FreeBSD, and why he transitioned from using an appliance to using vanilla FreeBSD. Shawn Webb followed with his overview talk about the “State of the Hardened Union”. Benedict’s talk about “Replacing an Oracle Server with FreeBSD, OpenZFS, and PostgreSQL” was well received as people are interested in how we liberated ourselves from the clutches of Oracle without compromising functionality. Entertaining and educational at the same time, Michael W. Lucas talk about “Twenty Years in Jail: FreeBSD Jails, Then and Now” closed the first day. Lucas also had a table in the hallway with his various tech and non-tech books for sale. People formed small groups and went into town for dinner. Some returned later that night to some work in the hacker lounge or talk amongst fellow BSD enthusiasts. Colin Percival was the keynote speaker for the second day and had an in-depth look at “23 years of software side channel attacks”. Allan reprised his “ELI5: ZFS Caching” talk explaining how the ZFS adaptive replacement cache (ARC) work and how it can be tuned for various workloads. “By the numbers: ZFS Performance Results from Six Operating Systems and Their Derivatives” by Michael Dexter followed with his approach to benchmarking OpenZFS on various platforms. Conor Beh was also a new speaker to vBSDcon. His talk was about “FreeBSD at Work: Building Network and Storage Infrastructure with pfSense and FreeNAS”. Two OpenBSD talks closed the talk session: Kurt Mosiejczuk with “Care and Feeding of OpenBSD Porters” and Aaron Poffenberger with “Road Warrior Disaster Recovery: Secure, Synchronized, and Backed-up”. A dinner and reception was enjoyed by the attendees and gave more time to discuss the talks given and other things until late at night. We want to thank the vBSDcon organizers and especially Dan Langille for running such a great conference. We are grateful to Verisign as the main sponsor and The FreeBSD Foundation for sponsoring the tote bags. Thanks to all the speakers and attendees! humungus - an hg server (https://humungus.tedunangst.com/r/humungus) Features View changes, files, changesets, etc. Some syntax highlighting. Read only. Serves multiple repositories. Allows cloning via the obvious URL. Supports go get. Serves files for downloads. Online documentation via mandoc. Terminal based admin interface. News Roundup OpenBSD on fan-less Tuxedo InfinityBook 14″ v2. (https://hazardous.org/archive/blog/openbsd/2019/09/02/OpenBSD-on-Infinitybook14) The InfinityBook 14” v2 is a fanless 14” notebook. It is an excellent choice for running OpenBSD - but order it with the supported wireless card (see below.). I’ve set it up in a dual-boot configuration so that I can switch between Linux and OpenBSD - mainly to spot differences in the drivers. TUXEDO allows a variety of configurations through their webshop. The dual boot setup with grub2 and EFI boot will be covered in a separate blogpost. My tests were done with OpenBSD-current - which is as of writing flagged as 6.6-beta. See Article for breakdown of CPU, Wireless, Video, Webcam, Audio, ACPI, Battery, Touchpad, and MicroSD Card Reader Unix at 50: How the OS that powered smartphones started from failure (https://arstechnica.com/gadgets/2019/08/unix-at-50-it-starts-with-a-mainframe-a-gator-and-three-dedicated-researchers/) Maybe its pervasiveness has long obscured its origins. But Unix, the operating system that in one derivative or another powers nearly all smartphones sold worldwide, was born 50 years ago from the failure of an ambitious project that involved titans like Bell Labs, GE, and MIT. Largely the brainchild of a few programmers at Bell Labs, the unlikely story of Unix begins with a meeting on the top floor of an otherwise unremarkable annex at the sprawling Bell Labs complex in Murray Hill, New Jersey. It was a bright, cold Monday, the last day of March 1969, and the computer sciences department was hosting distinguished guests: Bill Baker, a Bell Labs vice president, and Ed David, the director of research. Baker was about to pull the plug on Multics (a condensed form of MULTiplexed Information and Computing Service), a software project that the computer sciences department had been working on for four years. Multics was two years overdue, way over budget, and functional only in the loosest possible understanding of the term. Trying to put the best spin possible on what was clearly an abject failure, Baker gave a speech in which he claimed that Bell Labs had accomplished everything it was trying to accomplish in Multics and that they no longer needed to work on the project. As Berk Tague, a staffer present at the meeting, later told Princeton University, “Like Vietnam, he declared victory and got out of Multics.” Within the department, this announcement was hardly unexpected. The programmers were acutely aware of the various issues with both the scope of the project and the computer they had been asked to build it for. Still, it was something to work on, and as long as Bell Labs was working on Multics, they would also have a $7 million mainframe computer to play around with in their spare time. Dennis Ritchie, one of the programmers working on Multics, later said they all felt some stake in the success of the project, even though they knew the odds of that success were exceedingly remote. Cancellation of Multics meant the end of the only project that the programmers in the Computer science department had to work on—and it also meant the loss of the only computer in the Computer science department. After the GE 645 mainframe was taken apart and hauled off, the computer science department’s resources were reduced to little more than office supplies and a few terminals. Some of Allan’s favourite excerpts: In the early '60s, Bill Ninke, a researcher in acoustics, had demonstrated a rudimentary graphical user interface with a DEC PDP-7 minicomputer. Acoustics still had that computer, but they weren’t using it and had stuck it somewhere out of the way up on the sixth floor. And so Thompson, an indefatigable explorer of the labs’ nooks and crannies, finally found that PDP-7 shortly after Davis and Baker cancelled Multics. With the rest of the team’s help, Thompson bundled up the various pieces of the PDP-7—a machine about the size of a refrigerator, not counting the terminal—moved it into a closet assigned to the acoustics department, and got it up and running. One way or another, they convinced acoustics to provide space for the computer and also to pay for the not infrequent repairs to it out of that department’s budget. McIlroy’s programmers suddenly had a computer, kind of. So during the summer of 1969, Thompson, Ritchie, and Canaday hashed out the basics of a file manager that would run on the PDP-7. This was no simple task. Batch computing—running programs one after the other—rarely required that a computer be able to permanently store information, and many mainframes did not have any permanent storage device (whether a tape or a hard disk) attached to them. But the time-sharing environment that these programmers had fallen in love with required attached storage. And with multiple users connected to the same computer at the same time, the file manager had to be written well enough to keep one user’s files from being written over another user’s. When a file was read, the output from that file had to be sent to the user that was opening it. It was a challenge that McIlroy’s team was willing to accept. They had seen the future of computing and wanted to explore it. They knew that Multics was a dead-end, but they had discovered the possibilities opened up by shared development, shared access, and real-time computing. Twenty years later, Ritchie characterized it for Princeton as such: “What we wanted to preserve was not just a good environment in which to do programming, but a system around which a fellowship could form.” Eventually when they had the file management system more or less fleshed out conceptually, it came time to actually write the code. The trio—all of whom had terrible handwriting—decided to use the Labs’ dictating service. One of them called up a lab extension and dictated the entire code base into a tape recorder. And thus, some unidentified clerical worker or workers soon had the unenviable task of trying to convert that into a typewritten document. Of course, it was done imperfectly. Among various errors, “inode” came back as “eye node,” but the output was still viewed as a decided improvement over their assorted scribbles. In August 1969, Thompson’s wife and son went on a three-week vacation to see her family out in Berkeley, and Thompson decided to spend that time writing an assembler, a file editor, and a kernel to manage the PDP-7 processor. This would turn the group’s file manager into a full-fledged operating system. He generously allocated himself one week for each task. Thompson finished his tasks more or less on schedule. And by September, the computer science department at Bell Labs had an operating system running on a PDP-7—and it wasn’t Multics. By the summer of 1970, the team had attached a tape drive to the PDP-7, and their blossoming OS also had a growing selection of tools for programmers (several of which persist down to this day). But despite the successes, Thompson, Canaday, and Ritchie were still being rebuffed by labs management in their efforts to get a brand-new computer. It wasn’t until late 1971 that the computer science department got a truly modern computer. The Unix team had developed several tools designed to automatically format text files for printing over the past year or so. They had done so to simplify the production of documentation for their pet project, but their tools had escaped and were being used by several researchers elsewhere on the top floor. At the same time, the legal department was prepared to spend a fortune on a mainframe program called “AstroText.” Catching wind of this, the Unix crew realized that they could, with only a little effort, upgrade the tools they had written for their own use into something that the legal department could use to prepare patent applications. The computer science department pitched lab management on the purchase of a DEC PDP-11 for document production purposes, and Max Mathews offered to pay for the machine out of the acoustics department budget. Finally, management gave in and purchased a computer for the Unix team to play with. Eventually, word leaked out about this operating system, and businesses and institutions with PDP-11s began contacting Bell Labs about their new operating system. The Labs made it available for free—requesting only the cost of postage and media from anyone who wanted a copy. The rest has quite literally made tech history. See the link for the rest of the article How to configure a network dump in FreeBSD? (https://www.oshogbo.vexillium.org/blog/68/) A network dump might be very useful for collecting kernel crash dumps from embedded machines and machines with a larger amount of RAM then available swap partition size. Besides net dumps we can also try to compress the core dump. However, often this may still not be enough swap to keep whole core dump. In such situation using network dump is a convenient and reliable way for collecting kernel dump. So, first, let’s talk a little bit about history. The first implementation of the network dumps was implemented around 2000 for the FreeBSD 4.x as a kernel module. The code was implemented in 2010 with the intention of being part of FreeBSD 9.0. However, the code never landed in FreeBSD. Finally, in 2018 with the commit r333283 by Mark Johnston the netdump client code landed in the FreeBSD. Subsequently, many other commitments were then implemented to add support for the different drivers (for example r333289). The first official release of FreeBSD, which support netdump is FreeBSD 12.0. Now, let’s get back to the main topic. How to configure the network dump? Two machines are needed. One machine is to collect core dump, let’s call it server. We will use the second one to send us the core dump - the client. See the link for the rest of the article Beastie Bits Sudo Mastery 2nd edition is not out (https://mwl.io/archives/4530) Empirical Notes on the Interaction Between Continuous Kernel Fuzzing and Development (http://users.utu.fi/kakrind/publications/19/vulnfuzz_camera.pdf) soso (https://github.com/ozkl/soso) GregKH - OpenBSD was right (https://youtu.be/gUqcMs0svNU?t=254) Game of Trees (https://gameoftrees.org/faq.html) Feedback/Questions BostJan - Another Question (http://dpaste.com/1ZPCCQY#wrap) Tom - PF (http://dpaste.com/3ZSCB8N#wrap) JohnnyK - Changing VT without keys (http://dpaste.com/3QZQ7Q5#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
The Linux Foundation hosted the executive director of the FreeBSD Foundation, Deb Goodkin, at the Open Source Summit in San Diego. In this episode of Let’s Talk, we sat down with Goodkin to talk about the FreeBSD project and the foundation.
FreeBSD 11.3-beta 1 is out, BSDCan 2019 recap, OpenIndiana 2019.04 is out, Overview of ZFS Pools in FreeNAS, why open source firmware is important for security, a new Opnsense release, wireguard on OpenBSD, and more. Headlines FreeBSD 11.3-b1 is out BSDCan 2019 Recap We’re back from BSDCan and it was a packed week as always. It started with bhyvecon on Tuesday. Meanwhile, Benedict spent the whole day in productive meetings: annual FreeBSD Foundation board meeting and FreeBSD Journal editorial board meeting. On Wednesday, tutorials for BSDCan started as well as the FreeBSD Developer Summit. In the mornings, there were presentations in the big auditorium, while working groups about networking, failsafe bootcode, development web services, swap space management, and testing/CI were held. Friday had a similar format with an update from the FreeBSD core team and the “have, need, want” session for FreeBSD 13. In the afternoon, there were working groups about translation tools, package base, GSoC/Outreachy, or general hacking. Benedict held his Icinga tutorial in the afternoon with about 15 people attending. Devsummit presentation slides can be found on the wiki page and video recordings done by ScaleEngine are available on FreeBSD’s youtube channel. The conference program was a good mixture of sysadmin and tech talks across the major BSDs. Benedict saw the following talks: How ZFS snapshots really work by Matt Ahrens, 20 years in Jail by Michael W. Lucas, OpenZFS BOF session, the future of OpenZFS and FreeBSD, MQTT for system administrators by Jan-Piet Mens, and spent the rest of the time in between in the hallway track. Photos from the event are available on Ollivier Robert’s talegraph and Diane Bruce’s website for day 1, day 2, conference day 1, and conference day 2. Thanks to all the sponsors, supporters, organizers, speakers, and attendees for making this yet another great BSDCan. Next year’s BSDCan will be from June 2 - 6, 2020. OpenIndiana 2019.04 is out We have released a new OpenIndiana Hipster snapshot 2019.04. The noticeable changes: Firefox was updated to 60.6.3 ESR Virtualbox packages were added (including guest additions) Mate was updated to 1.22 IPS has received updates from OmniOS CE and Oracle IPS repos, including automatic boot environment naming Some OI-specific applications have been ported from Python 2.7/GTK 2 to Python 3.5/GTK 3 Quick Demo Video: https://www.youtube.com/watch?v=tQ0-fo3XNrg News Roundup Overview of ZFS Pools in FreeNAS FreeNAS uses the OpenZFS (ZFS) file system, which handles both disk and volume management. ZFS offers RAID options mirror, stripe, and its own parity distribution called RAIDZ that functions like RAID5 on hardware RAID. The file system is extremely flexible and secure, with various drive combinations, checksums, snapshots, and replication all possible. For a deeper dive on ZFS technology, read the ZFS Primer section of the FreeNAS documentation. SUGGEST LAYOUT attempts to balance usable capacity and redundancy by automatically choosing an ideal vdev layout for the number of available disks. The following vdev layout options are available when creating a pool: Stripe data is shared on two drives, similar to RAID0) Mirror copies data on two drives, similar to RAID1 but not limited to 2 disks) RAIDZ1 single parity similar to RAID5 RAIDZ2 double parity similar to RAID6 RAIDZ3 which uses triple parity and has no RAID equivalent Why OpenSource Firmware is Important for Security Roots of Trust The goal of the root of trust should be to verify that the software installed in every component of the hardware is the software that was intended. This way you can know without a doubt and verify if hardware has been hacked. Since we have very little to no visibility into the code running in a lot of places in our hardware it is hard to do this. How do we really know that the firmware in a component is not vulnerable or that is doesn’t have any backdoors? Well we can’t. Not unless it was all open source. Every cloud and vendor seems to have their own way of doing a root of trust. Microsoft has Cerberus, Google has Titan, and Amazon has Nitro. These seem to assume an explicit amount of trust in the proprietary code (the code we cannot see). This leaves me with not a great feeling. Wouldn’t it be better to be able to use all open source code? Then we could verify without a doubt that the code you can read and build yourself is the same code running on hardware for all the various places we have firmware. We could then verify that a machine was in a correct state without a doubt of it being vulnerable or with a backdoor. It makes me wonder what the smaller cloud providers like DigitalOcean or Packet have for a root of trust. Often times we only hear of these projects from the big three or five. OPNsense This update addresses several privilege escalation issues in the access control implementation and new memory disclosure issues in Intel CPUs. We would like to thank Arnaud Cordier and Bill Marquette for the top-notch reports and coordination. Here are the full patch notes: system: address CVE-2019-11816 privilege escalation bugs[1] (reported by Arnaud Cordier) system: /etc/hosts generation without interfacehasgateway() system: show correct timestamp in config restore save message (contributed by nhirokinet) system: list the commands for the pluginctl utility when n+ argument is given system: introduce and use userIsAdmin() helper function instead of checking for 'page-all' privilege directly system: use absolute path in widget ACLs (reported by Netgate) system: RRD-related cleanups for less code exposure interfaces: add EN DUID Generation using OPNsense PEN (contributed by Team Rebellion) interfaces: replace legacygetallinterface_addresses() usage firewall: fix port validation in aliases with leading / trailing spaces firewall: fix outbound NAT translation display in overview page firewall: prevent CARP outgoing packets from using the configured gateway firewall: use CARP net.inet.carp.demotion to control current demotion in status page firewall: stop live log poller on error result dhcpd: change rule priority to 1 to avoid bogon clash dnsmasq: only admins may edit custom options field firmware: use insecure mode for base and kernel sets when package fingerprints are disabled firmware: add optional device support for base and kernel sets firmware: add Hostcentral mirror (HTTP, Melbourne, Australia) ipsec: always reset rightallowany to default when writing configuration lang: say "hola" to Spanish as the newest available GUI language lang: updates for Chinese, Czech, Japanese, German, French, Russian and Portuguese network time: only admins may edit custom options field openvpn: call openvpnrefreshcrls() indirectly via plugin_configure() for less code exposure openvpn: only admins may edit custom options field to prevent privilege escalation (reported by Bill Marquette) openvpn: remove custom options field from wizard unbound: only admins may edit custom options field wizard: translate typehint as well plugins: os-freeradius 1.9.3 fixes string interpolation in LDAP filters (contributed by theq86) plugins: os-nginx 1.12[2] plugins: os-theme-cicada 1.17 (contributed by Team Rebellion) plugins: os-theme-tukan 1.17 (contributed by Team Rebellion) src: timezone database information update[3] src: install(1) broken with partially matching relative paths[4] src: microarchitectural Data Sampling (MDS) mitigation[5] ports: carootnss 3.44 ports: php 7.2.18[6] ports: sqlite 3.28.0[7] ports: strongswan custom XAuth generic patch removed wiregaurd on OpenBSD Earlier this week I imported a port for WireGuard into the OpenBSD ports tree. At the moment we have the userland daemon and the tools available. The in-kernel implementation is only available for Linux. At the time of writing there are packages available for -current. Jason A. Donenfeld (WireGuard author) has worked to support OpenBSD in WireGuard and as such his post on ports@ last year got me interested in WireGuard, since then others have toyed with WireGuard on OpenBSD before and as such I've used Ted's article as a reference. Note however that some of the options mentioned there are no longer valid. Also, I'll be using two OpenBSD peers here. The setup will be as follows: two OpenBSD peers, of which we'll dub wg1 the server and wg2 the client. The WireGuard service on wg1 is listening on 100.64.4.3:51820. Conclusion WireGuard (cl)aims to be easier to setup and faster than OpenVPN and while I haven't been able to verify the latter, the first is certainly true...once you've figured it out. Most documentation out there is for Linux so I had to figure out the wireguardgo service and the tun parameters. But all in all, sure, it's easier. Especially the client configuration on iOS which I didn't cover here because it's essentially pkgadd libqrencode ; cat client.conf | qrencode -t ansiutf8, scan the code with the WireGuard app and you're good to go. What is particularly neat is that WireGuard on iOS supports Always-on. Beastie Bits Serenity OS vkernels vs pmap Brian Kernighan interviews Ken Thompson Improvements in forking, threading, and signal code DragonFly 5.4.3 NetBSD on the Odroid C2 Feedback/Questions Paulo - Laptops A Listener - Thanks Bostjan - Extend a pool and lower RAM footprint Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.
OpenBSD 6.4 released, GhostBSD RC2 released, MeetBSD - the ultimate hallway track, DragonflyBSD desktop on a Thinkpad, Porting keybase to NetBSD, OpenSSH 7.9, and draft-ietf-6man-ipv6only-flag in FreeBSD. ##Headlines OpenBSD 6.4 released See a detailed log of changes between the 6.3 and 6.4 releases. See the information on the FTP page for a list of mirror machines. Have a look at the 6.4 errata page for a list of bugs and workarounds. signify(1) pubkeys for this release: base: RWQq6XmS4eDAcQW4KsT5Ka0KwTQp2JMOP9V/DR4HTVOL5Bc0D7LeuPwA fw: RWRoBbjnosJ/39llpve1XaNIrrQND4knG+jSBeIUYU8x4WNkxz6a2K97 pkg: RWRF5TTY+LoN/51QD5kM2hKDtMTzycQBBPmPYhyQEb1+4pff/H6fh/kA ###GhostBSD 18.10 RC2 Announced This second release candidate of GhostBSD 18.10 is the second official release of GhostBSD with TrueOS under the hood. The official desktop of GhostBSD is MATE. However, in the future, there might be an XFCE community release, but for now, there is no community release yet. What has changed since RC1 Removed drm-stable-kmod and we will let users installed the propper drm-*-kmod Douglas Joachin added libva-intel-driver libva-vdpau-driver to supports accelerated some video driver for Intel Issues that got fixed Bug #70 Cannot run Octopi, missing libgksu error. Bug #71 LibreOffice doesn’t start because of missing libcurl.so.4 Bug #72 libarchive is a missing dependency Again thanks to iXsystems, TrueOS, Joe Maloney, Kris Moore, Ken Moore, Martin Wilke, Neville Goddard, Vester “Vic” Thacker, Douglas Joachim, Alex Lyakhov, Yetkin Degirmenci and many more who helped to make the transition from FreeBSD to TrueOS smoother. Updating from RC1 to RC2: sudo pkg update -f sudo pkg install -f libarchive curl libgksu sudo pkg upgrade Where to download: All images checksum, hybrid ISO(DVD, USB) and torrent are available here: https://www.ghostbsd.org/download [ScreenShots] https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-2013-22-41.png https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-20_13-27-26.png ###OpenSSH 7.9 has been released and it has support for OpenSSL 1.1 Changes since OpenSSH 7.8 This is primarily a bugfix release. New Features ssh(1), sshd(8): allow most port numbers to be specified using service names from getservbyname(3) (typically /etc/services). ssh(1): allow the IdentityAgent configuration directive to accept environment variable names. This supports the use of multiple agent sockets without needing to use fixed paths. sshd(8): support signalling sessions via the SSH protocol. A limited subset of signals is supported and only for login or command sessions (i.e. not subsystems) that were not subject to a forced command via authorizedkeys or sshdconfig. bz#1424 ssh(1): support "ssh -Q sig" to list supported signature options. Also "ssh -Q help" to show the full set of supported queries. ssh(1), sshd(8): add a CASignatureAlgorithms option for the client and server configs to allow control over which signature formats are allowed for CAs to sign certificates. For example, this allows banning CAs that sign certificates using the RSA-SHA1 signature algorithm. sshd(8), ssh-keygen(1): allow key revocation lists (KRLs) to revoke keys specified by SHA256 hash. ssh-keygen(1): allow creation of key revocation lists directly from base64-encoded SHA256 fingerprints. This supports revoking keys using only the information contained in sshd(8) authentication log messages. Bugfixes ssh(1), ssh-keygen(1): avoid spurious "invalid format" errors when attempting to load PEM private keys while using an incorrect passphrase. bz#2901 sshd(8): when a channel closed message is received from a client, close the stderr file descriptor at the same time stdout is closed. This avoids stuck processes if they were waiting for stderr to close and were insensitive to stdin/out closing. bz#2863 ssh(1): allow ForwardX11Timeout=0 to disable the untrusted X11 forwarding timeout and support X11 forwarding indefinitely. Previously the behaviour of ForwardX11Timeout=0 was undefined. sshd(8): when compiled with GSSAPI support, cache supported method OIDs regardless of whether GSSAPI authentication is enabled in the main section of sshd_config. This avoids sandbox violations if GSSAPI authentication was later enabled in a Match block. bz#2107 sshd(8): do not fail closed when configured with a text key revocation list that contains a too-short key. bz#2897 ssh(1): treat connections with ProxyJump specified the same as ones with a ProxyCommand set with regards to hostname canonicalisation (i.e. don't try to canonicalise the hostname unless CanonicalizeHostname is set to 'always'). bz#2896 ssh(1): fix regression in OpenSSH 7.8 that could prevent public- key authentication using certificates hosted in a ssh-agent(1) or against sshd(8) from OpenSSH
FreeBSD Foundation September Update, tiny C lib for programming Unix daemons, EuroBSDcon trip reports, GhostBSD tested on real hardware, and a BSD auth module for duress. ##Headlines FreeBSD Foundation Update, September 2018 MESSAGE FROM THE EXECUTIVE DIRECTOR Dear FreeBSD Community Member, It is hard to believe that September is over. The Foundation team had a busy month promoting FreeBSD all over the globe, bug fixing in preparation for 12.0, and setting plans in motion to kick off our 4th quarter fundraising and advocacy efforts. Take a minute to see what we’ve been up to and please consider making a donation to help us continue our efforts supporting FreeBSD! September 2018 Development Projects Update In preparation for the release of FreeBSD 12.0, I have been working on investigating and fixing a backlog of kernel bug reports. Of course, this kind of work is never finished, and we will continue to make progress after the release. In the past couple of months I have fixed a combination of long-standing issues and recent regressions. Of note are a pair of UNIX domain socket bugs which had been affecting various applications for years. In particular, Chromium tabs would frequently hang unless a workaround was manually applied to the system, and the bug had started affecting recent versions of Firefox as well. Fixing these issues gave me an opportunity to revisit and extend our regression testing for UNIX sockets, which, in turn, resulted in some related bugs being identified and fixed. Of late I have also been investigating reports of issues with ZFS, particularly, those reported on FreeBSD 11.2. A number of regressions, including a kernel memory leak and issues with ARC reclamation, have already been fixed for 12.0; investigation of other reports is ongoing. Those who closely follow FreeBSD-CURRENT know that some exciting work to improve memory usage on NUMA systems is now enabled by default. As is usually the case when new code is deployed in a diverse array of systems and workloads, a number of problems since have been identified. We are working on resolving them as soon as possible to ensure the quality of the release. I’m passionate about maintaining FreeBSD’s stability and dependability as it continues to expand and grow new features, and I’m grateful to the FreeBSD Foundation for sponsoring this work. We depend on users to report problems to the mailing lists and via the bug tracker, so please try running the 12.0 candidate builds and help us make 12.0 a great release. Fundraising Update: Supporting the Project It’s officially Fall here at Foundation headquarters and we’re heading full-steam into our final fundraising campaign of the year. We couldn’t even have begun to reach our funding goal of $1.25 million dollars without the support from the companies who have partnered with us this year. Thank you to Verisign for becoming a Silver Partner. They now join a growing list of companies like Xiplink, NetApp, Microsoft, Tarsnap, VMware, and NeoSmart Technologies that are stepping up and showing their commitment to FreeBSD! Funding from commercial users like these and individual users like yourself, help us continue our efforts of supporting critical areas of FreeBSD such as: Operating System Improvements: Providing staff to immediately respond to urgent problems and implement new features and functionality allowing for the innovation and stability you’ve come to rely on. Security: Providing engineering resources to bolster the capacity and responsiveness of the Security team providing your users with piece of mind when security issues arise. Release Engineering: Continue providing a full-time release engineer, resulting in timely and reliable releases you can plan around. Quality Assurance: Improving and increasing test coverage, continuous integration, and automated testing with a full-time software engineer to ensure you receive the highest quality, secure, and reliable operating system. New User Experience: Improving the process and documentation for getting new people involved with FreeBSD, and supporting those people as they become integrated into the FreeBSD Community providing the resources you may need to get new folks up to speed. Training: Supporting more FreeBSD training for undergraduates, graduates, and postgraduates. Growing the community means reaching people and catching their interest in systems software as early as possible and providing you with a bigger pool of candidates with the FreeBSD skills you’re looking for. Face-to-Face Opportunities: Facilitating collaboration among members of the community, and building connections throughout the industry to support a healthy and growing ecosystem and make it easier for you to find resources when questions emerge . We can continue the above work, if we meet our goal this year! If your company uses FreeBSD, please consider joining our growing list of 2018 partners. If you haven’t made your donation yet, please consider donating today. We are indebted to the individual donors, and companies listed above who have already shown their commitment to open source. Thank you for supporting FreeBSD and the Foundation! September 2018 Release Engineering Update The FreeBSD Release Engineering team continued working on the upcoming 12.0 RELEASE. At present, the 12.0 schedule had been adjusted by one week to allow for necessary works-in-progress to be completed. Of note, one of the works-in-progress includes updating OpenSSL from 1.0.2 to 1.1.1, in order to avoid breaking the application binary interface (ABI) on an established stable branch. Due to the level of non-trivial intrusiveness that had already been discovered and addressed in a project branch of the repository, it is possible (but not yet definite) that the schedule will need to be adjusted by another week to allow more time for larger and related updates for this particular update. Should the 12.0-RELEASE schedule need to be adjusted at any time during the release cycle, the schedule on the FreeBSD project website will be updated accordingly. The current schedule is available at: https://www.freebsd.org/releases/12.0R/schedule.html BSDCam 2018 Trip Report: Marie Helene Kvello-Aune I’d like to start by thanking the FreeBSD Foundation for sponsoring my trip to BSDCam(bridge) 2018. I wouldn’t have managed to attend otherwise. I’ve used FreeBSD in both personal and professional deployments since the year 2000, and over the last few years I have become more involved with development and documentation. I arrived in Gatwick, London at midnight. On Monday, August 13, I took the train to Cambridge, and decided to do some touristy activities as I walked from the train station to Churchill College. I ran into Allan outside the hotel right before the sky decided it was time for a heavy rainfall. Monday was mostly spent settling in, recouping after travel, and hanging out with Allan, Brad, Will and Andy later in the afternoon/evening. Read more… Continuous Integration Update The FreeBSD Foundation has sponsored the development of the Project’s continuous integration system, available at https://ci.FreeBSD.org, since June. Over the summer, we improved both the software and hardware infrastructure, and also added some new jobs for extending test coverage of the -CURRENT and -STABLE branches. Following are some highlights. New Hardware The Foundation purchased 4 new build machines for scaling up the computation power for the various test jobs. These newer, faster machines substantially speed up the time it takes to test amd64 builds, so that failing changes can be identified more quickly. Also, in August, we received a donation of 2 PINE A64-LTS boards from PINE64.org, which will be put in the hardware test lab as one part of the continuous tests. CI Staging Environment We used hardware from a previous generation CI system to build a staging environment for the CI infrastructure, which is available at https://ci-dev.freebsd.org. It executes the configurations and scripts from the “staging” branch of the FreeBSD-CI repository, and the development feature branches. We also use it to experiment with the new version of the jenkins server and plugins. Having a staging environment avoids affecting the production CI environment, reducing downtime. Mail Notification In July, we turned on failure notification for all the kernel and world build jobs. Committers will receive email containing the build information and failure log to inform them of possible problems with their modification on certain architectures. For amd64 of the -CURRENT branch, we also enabled the notification on failing regression test cases. Currently mail is sent only to the individual committers, but with help from postmaster team, we have created a dev-ci mailing list and will soon be also sending notifications there. New Test Job In August, we updated the embedded script of the virtual machine image. Originally it only executed pre-defined tests, but now this behavior can be modified by the data on the attached disk. This mechanism is used for adding new ZFS tests jobs. We are also working on analyzing and fixing the failing and skipped test cases. Work in Progress In August and September, we had two developer summits, one in Cambridge, UK and one in Bucharest, Romania. In these meetings, we discussed running special tests, such as ztest, which need a longer run time. We also planned the network testing for TCP/IP stack ###Daemonize - a Tiny C Library for Programming the UNIX Daemons Whatever they say, writing System-V style UNIX daemons is hard. One has to follow many rules to make a daemon process behave correctly on diverse UNIX flavours. Moreover, debugging such a code might be somewhat tricky. On the other hand, the process of daemon initialisation is rigid and well defined so the corresponding code has to be written and debugged once and later can be reused countless number of times. Developers of BSD UNIX were very aware of this, as there a C library function daemon() was available starting from version 4.4. The function, although non-standard, is present on many UNIXes. Unfortunately, it does not follow all the required steps to reliably run a process in the background on systems which follow System-V semantics (e.g. Linux). The details are available at the corresponding Linux man page. The main problem here, as I understand it, is that daemon() does not use the double-forking technique to avoid the situation when zombie processes appear. Whenever I encounter a problem like this one, I know it is time to write a tiny C library which solves it. This is exactly how ‘daemonize’ was born (GitHub mirror). The library consists of only two files which are meant to be integrated into the source tree of your project. Recently I have updated the library and realised that it would be good to describe how to use it on this site. If for some reason you want to make a Windows service, I have a battle tested template code for you as well. System-V Daemon Initialisation Procedure To make discussion clear we shall quote the steps which have to be performed during a daemon initialisation (according to daemon(7) manual page on Linux). I do it to demonstrate that this task is more tricky than one might expect. So, here we go: Close all open file descriptors except standard input, output, and error (i.e. the first three file descriptors 0, 1, 2). This ensures that no accidentally passed file descriptor stays around in the daemon process. On Linux, this is best implemented by iterating through /proc/self/fd, with a fallback of iterating from file descriptor 3 to the value returned by getrlimit() for RLIMITNOFILE. Reset all signal handlers to their default. This is best done by iterating through the available signals up to the limit of _NSIG and resetting them to SIGDFL. Reset the signal mask using sigprocmask(). Sanitize the environment block, removing or resetting environment variables that might negatively impact daemon runtime. Call fork(), to create a background process. In the child, call setsid() to detach from any terminal and create an independent session. In the child, call fork() again, to ensure that the daemon can never re-acquire a terminal again. Call exit() in the first child, so that only the second child (the actual daemon process) stays around. This ensures that the daemon process is re-parented to init/PID 1, as all daemons should be. In the daemon process, connect /dev/null to standard input, output, and error. In the daemon process, reset the umask to 0, so that the file modes passed to open(), mkdir() and suchlike directly control the access mode of the created files and directories. In the daemon process, change the current directory to the root directory (/), in order to avoid that the daemon involuntarily blocks mount points from being unmounted. In the daemon process, write the daemon PID (as returned by getpid()) to a PID file, for example /run/foobar.pid (for a hypothetical daemon “foobar”) to ensure that the daemon cannot be started more than once. This must be implemented in race-free fashion so that the PID file is only updated when it is verified at the same time that the PID previously stored in the PID file no longer exists or belongs to a foreign process. In the daemon process, drop privileges, if possible and applicable. From the daemon process, notify the original process started that initialization is complete. This can be implemented via an unnamed pipe or similar communication channel that is created before the first fork() and hence available in both the original and the daemon process. Call exit() in the original process. The process that invoked the daemon must be able to rely on that this exit() happens after initialization is complete and all external communication channels are established and accessible. The discussed library does most of the above-mentioned initialisation steps as it becomes immediately evident that implementation details for some of them heavily dependent on the internal logic of an application itself, so it is not possible to implement them in a universal library. I believe it is not a flaw, though, as the missed parts are safe to implement in an application code. The Library’s Application Programming Interface The generic programming interface was loosely modelled after above-mentioned BSD’s daemon() function. The library provides two user available functions (one is, in fact, implemented on top of the other) as well as a set of flags to control a daemon creation behaviour. Conclusion The objective of the library is to hide all the trickery of programming a daemon so you could concentrate on the more creative parts of your application. I hope it does this well. If you are not only interested in writing a daemon, but also want to make yourself familiar with the techniques which are used to accomplish that, the source code is available. Moreover, I would advise anyone, who starts developing for a UNIX environment to do that, as it shows many intricacies of programming for these platforms. ##News Roundup EuroBSDCon 2018 travel report and obligatory pics This was my first big BSD conference. We also planned - planned might be a big word - thought about doing a devsummit on Friday. Since the people who were in charge of that had a change of plans, I was sure it’d go horribly wrong. The day before the devsummit and still in the wrong country, I mentioned the hours and venue on the wiki, and booked a reservation for a restaurant. It turns out that everything was totally fine, and since the devsummit was at the conference venue (that was having tutorials that day), they even had signs pointing at the room we were given. Thanks EuroBSDCon conference organizers! At the devsummit, we spent some time hacking. A few people came with “travel laptops” without access to anything, like Riastradh, so I gave him access to my own laptop. This didn’t hold very long and I kinda forgot about it, but for a few moments he had access to a NetBSD source tree and an 8 thread, 16GB RAM machine with which to build things. We had a short introduction and I suggested we take some pictures, so here’s the ones we got. A few people were concerned about privacy, so they’re not pictured. We had small team to hold the camera :-) At the actual conference days, I stayed at the speaker hotel with the other speakers. I’ve attempted to make conversation with some visibly FreeBSD/OpenBSD people, but didn’t have plans to talk about anything, so there was a lot of just following people silently. Perhaps for the next conference I’ll prepare a list of questions to random BSD people and then very obviously grab a piece of paper and ask, “what was…”, read a bit from it, and say, “your latest kernel panic?”, I’m sure it’ll be a great conversation starter. At the conference itself, was pretty cool to have folks like Kirk McKusick give first person accounts of some past events (Kirk gave a talk about governance at FreeBSD), or the second keynote by Ron Broersma. My own talk was hastily prepared, it was difficult to bring the topic together into a coherent talk. Nevertheless, I managed to talk about stuff for a while 40 minutes, though usually I skip over so many details that I have trouble putting together a sufficiently long talk. I mentioned some of my coolest bugs to solve (I should probably make a separate article about some!). A few people asked for the slides after the talk, so I guess it wasn’t totally incoherent. It was really fun to meet some of my favourite NetBSD people. I got to show off my now fairly well working laptop (it took a lot of work by all of us!). After the conference I came back with a conference cold, and it took a few days to recover from it. Hopefully I didn’t infect too many people on the way back. ###GhostBSD tested on real hardware T410 – better than TrueOS? You might have heard about FreeBSD which is ultimately derived from UNIX back in the days. It is not Linux even though it is similar in many ways because Linux was designed to follow UNIX principles. Seeing is believing, so check out the video of the install and some apps as well! Nowadays if you want some of that BSD on your personal desktop how to go about? Well there is a full package or distro called GhostBSD which is based on FreeBSD current with a Mate or XFCE desktop preconfigured. I did try another package called TrueOS before and you can check out my blog post as well. Let’s give it a try on my Lenovo ThinkPad T410. You can download the latest version from ghostbsd.org. Creating a bootable USB drive was surprisingly difficult as rufus did not work and created a corrupted drive. You have to follow this procedure under Windows: download the 2.5GB .iso file and rename the extension to .img. Download Win32 Disk imager and burn the img file to an USB drive and boot from it. You will be able to start a live session and use the onboard setup to install GhostBSD unto a disk. I did encounter some bugs or quirks along the way. The installer failed the first time for some unknown reason but worked on the second attempt. The first boot stopped upon initialization of the USB3 ports (the T410 does not have USB3) but I could use some ‘exit’ command line magic to continue. The second boot worked fine. Audio was only available through headphones, not speakers but that could partially be fixed using the command line again. Lot’s of installed apps did not show up in the start menu and on goes the quirks list. Overall it is still better than TrueOS for me because drivers did work very well and I could address most of the existing bugs. On the upside: Free and open source FreeBSD package ready to go Mate or XFCE desktop (Mate is the only option for daily builds) Drivers work fine including LAN, WiFi, video 2D & 3D, audio, etc UFS or ZFS advanced file systems available Some downsides: Less driver and direct app support than Linux Installer and desktop have some quirks and bugs App-store is cumbersome, inferior to TrueOS ##Beastie Bits EuroBSDCon 2018 and NetBSD sanitizers New mandoc feature: -T html -O toc EuroBSDcon 2018 Polish BSD User Group garbage[43]: What year is it? The Demo @ 50 Microsoft ports DTrace from FreeBSD to Windows 10 OpenBSD joins Twitter NetBSD curses ripoffline improvements FCP-0101: Deprecating most 10/100 Ethernet drivers Announcing the pkgsrc-2018Q3 release Debian on OpenBSD vmd (without qemu or another debian system) A BSD authentication module for duress passwords (Joshua Stein) Disk Price/Performance Analysis ##Feedback/Questions DJ - Zombie ZFS Josua - arm tier 1? how to approach it -Gamah - 5ghz Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
We report from our experiences at EuroBSDcon, disenchant software, LLVM 7.0.0 has been released, Thinkpad BIOS update options, HardenedBSD Foundation announced, and ZFS send vs. rsync. ##Headlines ###[FreeBSD DevSummit & EuroBSDcon 2018 in Romania] Your hosts are back from EuroBSDcon 2018 held in Bucharest, Romania this year. The first two days of the conference are used for tutorials and devsummits (FreeBSD and NetBSD), while the last two are for talks. Although Benedict organized the devsummit in large parts, he did not attend it this year. He held his Ansible tutorial in the morning of the first day, followed by Niclas Zeising’s new ports and poudriere tutorial (which had a record attendance). It was intended for beginners that had never used poudriere before and those who wanted to create their first port. The tutorial was well received and Niclas already has ideas for extending it for future conferences. On the second day, Benedict took Kirk McKusick’s “An Introduction to the FreeBSD Open-Source Operating System” tutorial, held as a one full day class this year. Although it was reduced in content, it went into enough depth of many areas of the kernel and operating system to spark many questions from attendees. Clearly, this is a good start into kernel programming as Kirk provides enough material and backstories to understand why certain things are implemented as they are. Olivier Robert took https://www.talegraph.com/tales/l2o9ltrvsE (pictures from the devsummit) and created a nice gallery out of it. Devsummit evenings saw dinners at two restaurants that allowed developers to spend some time talking over food and drinks. The conference opened on the next day with the opening session held by Mihai Carabas. He introduced the first keynote speaker, a colleague of his who presented “Lightweight virtualization with LightVM and Unikraft”. Benedict helped out at the FreeBSD Foundation sponsor table and talked to people. He saw the following talks in between: Selfhosting as an alternative to the public cloud (by Albert Dengg) Using Boot Environments at Scale (by Allan Jude) Livepatching FreeBSD kernel (by Maciej Grochowski) FreeBSD: What to (Not) Monitor (by Andrew Fengler) FreeBSD Graphics (by Niclas Zeising) Allan spent a lot of time talking to people and helping track down issues they were having, in addition to attending many talks: Hacking together a FreeBSD presentation streaming box – For as little as possible (by Tom Jones) Introduction of FreeBSD in new environments (by Baptiste Daroussin) Keynote: Some computing and networking historical perspectives (by Ron Broersma) Livepatching FreeBSD kernel (by Maciej Grochowski) FreeBSD: What to (Not) Monitor (by Andrew Fengler) Being a BSD user (by Roller Angel) From “Hello World” to the VFS Layer: building a beadm for DragonFly BSD (by Michael Voight) We also met the winner of our Power Bagel raffle from Episode 2^8. He received the item in the meantime and had it with him at the conference, providing a power outlet to charge other people’s devices. During the closing session, GroffTheBSDGoat was handed over to Deb Goodkin, who will bring the little guy to the Grace Hopper Celebration of Women in Computing conference and then to MeetBSD later this year. It was also revealed that next year’s EuroBSDcon will be held in Lillehammer, Norway. Thanks to all the speakers, helpers, sponsors, organizers, and attendees for making it a successful conferences. There were no talks recorded this year, but the slides will be uploaded to the EuroBSDcon website in a couple of weeks. The OpenBSD talks are already available, so check them out. ###Software disenchantment I’ve been programming for 15 years now. Recently our industry’s lack of care for efficiency, simplicity, and excellence started really getting to me, to the point of me getting depressed by my own career and the IT in general. Modern cars work, let’s say for the sake of argument, at 98% of what’s physically possible with the current engine design. Modern buildings use just enough material to fulfill their function and stay safe under the given conditions. All planes converged to the optimal size/form/load and basically look the same. Only in software, it’s fine if a program runs at 1% or even 0.01% of the possible performance. Everybody just seems to be ok with it. People are often even proud about how much inefficient it is, as in “why should we worry, computers are fast enough”: @tveastman: I have a Python program I run every day, it takes 1.5 seconds. I spent six hours re-writing it in rust, now it takes 0.06 seconds. That efficiency improvement means I’ll make my time back in 41 years, 24 days :-) You’ve probably heard this mantra: “programmer time is more expensive than computer time”. What it means basically is that we’re wasting computers at an unprecedented scale. Would you buy a car if it eats 100 liters per 100 kilometers? How about 1000 liters? With computers, we do that all the time. Everything is unbearably slow Look around: our portable computers are thousands of times more powerful than the ones that brought man to the moon. Yet every other webpage struggles to maintain a smooth 60fps scroll on the latest top-of-the-line MacBook Pro. I can comfortably play games, watch 4K videos but not scroll web pages? How is it ok? Google Inbox, a web app written by Google, running in Chrome browser also by Google, takes 13 seconds to open moderately-sized emails: It also animates empty white boxes instead of showing their content because it’s the only way anything can be animated on a webpage with decent performance. No, decent doesn’t mean 60fps, it’s rather “as fast as this web page could possibly go”. I’m dying to see web community answer when 120Hz displays become mainstream. Shit barely hits 60Hz already. Windows 10 takes 30 minutes to update. What could it possibly be doing for that long? That much time is enough to fully format my SSD drive, download a fresh build and install it like 5 times in a row. Pavel Fatin: Typing in editor is a relatively simple process, so even 286 PCs were able to provide a rather fluid typing experience. Modern text editors have higher latency than 42-year-old Emacs. Text editors! What can be simpler? On each keystroke, all you have to do is update tiny rectangular region and modern text editors can’t do that in 16ms. It’s a lot of time. A LOT. A 3D game can fill the whole screen with hundreds of thousands (!!!) of polygons in the same 16ms and also process input, recalculate the world and dynamically load/unload resources. How come? As a general trend, we’re not getting faster software with more features. We’re getting faster hardware that runs slower software with the same features. Everything works way below the possible speed. Ever wonder why your phone needs 30 to 60 seconds to boot? Why can’t it boot, say, in one second? There are no physical limitations to that. I would love to see that. I would love to see limits reached and explored, utilizing every last bit of performance we can get for something meaningful in a meaningful way. Everything is HUUUUGE And then there’s bloat. Web apps could open up to 10× faster if you just simply block all ads. Google begs everyone to stop shooting themselves in their feet with AMP initiative—a technology solution to a problem that doesn’t need any technology, just a little bit of common sense. If you remove bloat, the web becomes crazy fast. How smart do you have to be to understand that? Android system with no apps takes almost 6 Gb. Just think for a second how obscenely HUGE that number is. What’s in there, HD movies? I guess it’s basically code: kernel, drivers. Some string and resources too, sure, but those can’t be big. So, how many drivers do you need for a phone? Windows 95 was 30Mb. Today we have web pages heavier than that! Windows 10 is 4Gb, which is 133 times as big. But is it 133 times as superior? I mean, functionally they are basically the same. Yes, we have Cortana, but I doubt it takes 3970 Mb. But whatever Windows 10 is, is Android really 150% of that? Google keyboard app routinely eats 150 Mb. Is an app that draws 30 keys on a screen really five times more complex than the whole Windows 95? Google app, which is basically just a package for Google Web Search, is 350 Mb! Google Play Services, which I do not use (I don’t buy books, music or videos there)—300 Mb that just sit there and which I’m unable to delete. All that leaves me around 1 Gb for my photos after I install all the essential (social, chats, maps, taxi, banks etc) apps. And that’s with no games and no music at all! Remember times when an OS, apps and all your data fit on a floppy? Your desktop todo app is probably written in Electron and thus has userland driver for Xbox 360 controller in it, can render 3d graphics and play audio and take photos with your web camera. A simple text chat is notorious for its load speed and memory consumption. Yes, you really have to count Slack in as a resource-heavy application. I mean, chatroom and barebones text editor, those are supposed to be two of the less demanding apps in the whole world. Welcome to 2018. At least it works, you might say. Well, bigger doesn’t imply better. Bigger means someone has lost control. Bigger means we don’t know what’s going on. Bigger means complexity tax, performance tax, reliability tax. This is not the norm and should not become the norm. Overweight apps should mean a red flag. They should mean run away scared. Better world manifesto I want to see progress. I want change. I want state-of-the-art in software engineering to improve, not just stand still. I don’t want to reinvent the same stuff over and over, less performant and more bloated each time. I want something to believe in, a worthy end goal, a future better than what we have today, and I want a community of engineers who share that vision. What we have today is not progress. We barely meet business goals with poor tools applied over the top. We’re stuck in local optima and nobody wants to move out. It’s not even a good place, it’s bloated and inefficient. We just somehow got used to it. So I want to call it out: where we are today is bullshit. As engineers, we can, and should, and will do better. We can have better tools, we can build better apps, faster, more predictable, more reliable, using fewer resources (orders of magnitude fewer!). We need to understand deeply what are we doing and why. We need to deliver: reliably, predictably, with topmost quality. We can—and should–take pride in our work. Not just “given what we had…”—no buts! I hope I’m not alone at this. I hope there are people out there who want to do the same. I’d appreciate if we at least start talking about how absurdly bad our current situation in the software industry is. And then we maybe figure out how to get out. ##News Roundup [llvm-announce] LLVM 7.0.0 Release I am pleased to announce that LLVM 7 is now available. Get it here: https://llvm.org/releases/download.html#7.0.0 The release contains the work on trunk up to SVN revision 338536 plus work on the release branch. It is the result of the community's work over the past six months, including: function multiversioning in Clang with the 'target' attribute for ELF-based x86/x86_64 targets, improved PCH support in clang-cl, preliminary DWARF v5 support, basic support for OpenMP 4.5 offloading to NVPTX, OpenCL C++ support, MSan, X-Ray and libFuzzer support for FreeBSD, early UBSan, X-Ray and libFuzzer support for OpenBSD, UBSan checks for implicit conversions, many long-tail compatibility issues fixed in lld which is now production ready for ELF, COFF and MinGW, new tools llvm-exegesis, llvm-mca and diagtool. And as usual, many optimizations, improved diagnostics, and bug fixes. For more details, see the release notes: https://llvm.org/releases/7.0.0/docs/ReleaseNotes.html https://llvm.org/releases/7.0.0/tools/clang/docs/ReleaseNotes.html https://llvm.org/releases/7.0.0/tools/clang/tools/extra/docs/ReleaseNotes.html https://llvm.org/releases/7.0.0/tools/lld/docs/ReleaseNotes.html Thanks to everyone who helped with filing, fixing, and code reviewing for the release-blocking bugs! Special thanks to the release testers and packagers: Bero Rosenkränzer, Brian Cain, Dimitry Andric, Jonas Hahnfeld, Lei Huang Michał Górny, Sylvestre Ledru, Takumi Nakamura, and Vedant Kumar. For questions or comments about the release, please contact the community on the mailing lists. Onwards to LLVM 8! Cheers, Hans ###Update your Thinkpad’s bios with Linux or OpenBSD Get your new bios At first, go to the Lenovo website and download your new bios: Go to lenovo support Use the search bar to find your product (example for me, x270) Choose the right product (if necessary) and click search On the right side, click on Update Your System Click on BIOS/UEFI Choose *BIOS Update (Bootable CD) for Windows * Download For me the file is called like this : r0iuj25wd.iso Extract bios update Now you will need to install geteltorito. With OpenBSD: $ doas pkgadd geteltorito quirks-3.7 signed on 2018-09-09T13:15:19Z geteltorito-0.6: ok With Debian: $ sudo apt-get install genisoimage Now we will extract the bios update : $ geteltorito -o biosupdate.img r0iuj25wd.iso Booting catalog starts at sector: 20 Manufacturer of CD: NERO BURNING ROM VER 12 Image architecture: x86 Boot media type is: harddisk El Torito image starts at sector 27 and has 43008 sector(s) of 512 Bytes Image has been written to file "biosupdate.img". This will create a file called biosupdate.img. Put the image on an USB key CAREFULL : on my computer, my USB key is sda1 on Linux and sd1 on OpenBSD. Please check twice on your computer the name of your USB key. With OpenBSD : $ doas dd if=biosupdate.img of=/dev/rsd1c With Linux : $ sudo dd if=biosupdate.img of=/dev/sda Now all you need is to reboot, to boot on your USB key and follow the instructions. Enjoy 😉 ###Announcing The HardenedBSD Foundation In June of 2018, we announced our intent to become a not-for-profit, tax-exempt 501©(3) organization in the United States. It took a dedicated team months of work behind-the-scenes to make that happen. On 06 September 2018, HardenedBSD Foundation Corp was granted 501©(3) status, from which point all US-based persons making donations can deduct the donation from their taxes. We are grateful for those who contribute to HardenedBSD in whatever way they can. Thank you for making HardenedBSD possible. We look forward to a bright future, driven by a helpful and positive community. ###How you migrate ZFS filesystems matters If you want to move a ZFS filesystem around from one host to another, you have two general approaches; you can use ‘zfs send’ and ‘zfs receive’, or you can use a user level copying tool such as rsync (or ‘tar -cf | tar -xf’, or any number of similar options). Until recently, I had considered these two approaches to be more or less equivalent apart from their convenience and speed (which generally tilted in favour of ‘zfs send’). It turns out that this is not necessarily the case and there are situations where you will want one instead of the other. We have had two generations of ZFS fileservers so far, the Solaris ones and the OmniOS ones. When we moved from the first generation to the second generation, we migrated filesystems across using ‘zfs send’, including the filesystem with my home directory in it (we did this for various reasons). Recently I discovered that some old things in my filesystem didn’t have file type information in their directory entries. ZFS has been adding file type information to directories for a long time, but not quite as long as my home directory has been on ZFS. This illustrates an important difference between the ‘zfs send’ approach and the rsync approach, which is that zfs send doesn’t update or change at least some ZFS on-disk data structures, in the way that re-writing them from scratch from user level does. There are both positives and negatives to this, and a certain amount of rewriting does happen even in the ‘zfs send’ case (for example, all of the block pointers get changed, and ZFS will re-compress your data as applicable). I knew that in theory you had to copy things at the user level if you wanted to make sure that your ZFS filesystem and everything in it was fully up to date with the latest ZFS features. But I didn’t expect to hit a situation where it mattered in practice until, well, I did. Now I suspect that old files on our old filesystems may be partially missing a number of things, and I’m wondering how much of the various changes in ‘zfs upgrade -v’ apply even to old data. (I’d run into this sort of general thing before when I looked into ext3 to ext4 conversion on Linux.) With all that said, I doubt this will change our plans for migrating our ZFS filesystems in the future (to our third generation fileservers). ZFS sending and receiving is just too convenient, too fast and too reliable to give up. Rsync isn’t bad, but it’s not the same, and so we only use it when we have to (when we’re moving only some of the people in a filesystem instead of all of them, for example). PS: I was going to try to say something about what ‘zfs send’ did and didn’t update, but having looked briefly at the code I’ve concluded that I need to do more research before running my keyboard off. In the mean time, you can read the OpenZFS wiki page on ZFS send and receive, which has plenty of juicy technical details. PPS: Since eliminating all-zero blocks is a form of compression, you can turn zero-filled files into sparse files through a ZFS send/receive if the destination has compression enabled. As far as I know, genuine sparse files on the source will stay sparse through a ZFS send/receive even if they’re sent to a destination with compression off. ##Beastie Bits BSD Users Stockholm Meetup #4: Tuesday, November 13, 2018 at 18:00 BSD Poland User Group: Next Meeting: October 11, 2018, 18:15 - 21:15 at Warsaw University of Technology n2k18 Hackathon report: Ken Westerback (krw@) on disklabel(8) work, dhclient(8) progress Running MirageOS Unikernels on OpenBSD in vmm (Now Works) vmm(4) gets support for qcow2 MeetBSD and SecurityBsides Colin Percival reduced FreeBSD startup time from 10627ms (11.2) to 4738ms (12.0) FreeBSD 11.1 end-of-life KnoxBug: Monday, October 1, 2018 at 18:00: Real-world Performance Advantages of NVDIMM and NVMe: Case Study with OpenZFS ##Feedback/Questions Todd - 2 Nics, 1 bhyve and a jail cell Thomas - Deep Dive Morgan - Send/Receive to Manage Fragmentation? Dominik - hierarchical jails -> networking Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
FreeBSD ULE vs. Linux CFS, OpenBSD on Tuxedo InfinityBook, how zfs diff reports filenames efficiently, why choose FreeBSD over Linux, PS4 double free exploit, OpenBSD’s wifi autojoin, and FreeBSD jails the hard way. Win Celebrate our 256th episode with us. You can win a Mogics Power Bagel (not sponsored). To enter, go find the 4 episodes we did in December of 2017. In the opening, find the 4 letters in the bookshelf behind me. They spell different words in each of the 4 episodes. Send us these words in order to feedback@bsdnow.tv with the subject “bsdnow256” until August 8th, 2018 18:00 UTC and we’ll randomly draw the winner on the live show. We’ll then contact you to ship the item. Only one item to win. All decisions are final. Better luck next time. Headlines Battle of the Schedulers: FreeBSD ULE vs. Linux CFS Introduction This paper analyzes the impact on application performance of the design and implementation choices made in two widely used open-source schedulers: ULE, the default FreeBSD scheduler, and CFS, the default Linux scheduler. We compare ULE and CFS in otherwise identical circumstances. We have ported ULE to Linux, and use it to schedule all threads that are normally scheduled by CFS. We compare the performance of a large suite of applications on the modified kernel running ULE and on the standard Linux kernel running CFS. The observed performance differences are solely the result of scheduling decisions, and do not reflect differences in other subsystems between FreeBSD and Linux. There is no overall winner. On many workloads the two schedulers perform similarly, but for some workloads there are significant and even surprising differences. ULE may cause starvation, even when executing a single application with identical threads, but this starvation may actually lead to better application performance for some workloads. The more complex load balancing mechanism of CFS reacts more quickly to workload changes, but ULE achieves better load balance in the long run. Operating system kernel schedulers are responsible for maintaining high utilization of hardware resources (CPU cores, memory, I/O devices) while providing fast response time to latency-sensitive applications. They have to react to workload changes, and handle large numbers of cores and threads with minimal overhead [12]. This paper provides a comparison between the default schedulers of two of the most widely deployed open-source operating systems: the Completely Fair Scheduler (CFS) used in Linux, and the ULE scheduler used in FreeBSD. Our goal is not to declare an overall winner. In fact, we find that for some workloads ULE is better and for others CFS is better. Instead, our goal is to illustrate how differences in the design and the implementation of the two schedulers are reflected in application performance under different workloads. ULE and CFS are both designed to schedule large numbers of threads on large multicore machines. Scalability considerations have led both schedulers to adopt per-core run-queues. On a context switch, a core accesses only its local run-queue to find the next thread to run. Periodically and at select times, e.g., when a thread wakes up, both ULE and CFS perform load balancing, i.e., they try to balance the amount of work waiting in the run-queues of different cores. ULE and CFS, however, differ greatly in their design and implementation choices. FreeBSD ULE is a simple scheduler (2,950 lines of code in FreeBSD 11.1), while Linux CFS is much more complex (17,900 lines of code in the latest LTS Linux kernel, Linux 4.9). FreeBSD run-queues are FIFO. For load balancing, FreeBSD strives to even out the number of threads per core. In Linux, a core decides which thread to run next based on prior execution time, priority, and perceived cache behavior of the threads in its runqueue. Instead of evening out the number of threads between cores, Linux strives to even out the average amount of pending work. Performance analysis We now analyze the impact of the per-core scheduling on the performance of 37 applications. We define “performance” as follows: for database workloads and NAS applications, we compare the number of operations per second, and for the other applications we compare “execution time”. The higher the “performance”, the better a scheduler performs. Figure 5 presents the performance difference between CFS and ULE on a single core, with percentages above 0 meaning that the application executes faster with ULE than CFS. Overall, the scheduler has little influence on most workloads. Indeed, most applications use threads that all perform the same work, thus both CFS and ULE endup scheduling all of the threads in a round-robin fashion. The average performance difference is 1.5%, in favor of ULE. Still, scimark is 36% slower on ULE than CFS, and apache is 40% faster on ULE than CFS. Scimark is a single-threaded Java application. It launches one compute thread, and the Java runtime executes other Java system threads in the background (for the garbage collector, I/O, etc.). When the application is executed with ULE, the compute thread can be delayed, because Java system threads are considered interactive and get priority over the computation thread. The apache workload consists of two applications: the main server (httpd) running 100 threads, and ab, a single-threaded load injector. The performance difference between ULE and CFS is explained by different choices regarding thread preemption. In ULE, full preemption is disabled, while CFS preempts the running thread when the thread that has just been woken up has a vruntime that is much smaller than the vruntime of the currently executing thread (1ms difference in practice). In CFS, ab is preempted 2 million times during the benchmark, while it never preempted with ULE. This behavior is explained as follows: ab starts by sending 100 requests to the httpd server, and then waits for the server to answer. When ab is woken up, it checks which requests have been processed and sends new requests to the server. Since ab is single-threaded, all requests sent to the server are sent sequentially. In ULE, ab is able to send as many new requests as it has received responses. In CFS, every request sent by ab wakes up a httpd thread, which preempts ab. Conclusion Scheduling threads on a multicore machine is hard. In this paper, we perform a fair comparison of the design choices of two widely used schedulers: the ULE scheduler from FreeBSD and CFS from Linux. We show that they behave differently even on simple workloads, and that no scheduler performs better than the other on all workloads. OpenBSD 6.3 on Tuxedo InfinityBook Disclaimer: I came across the Tuxedo Computers InfinityBook last year at the Open! Conference where Tuxedo had a small booth. Previously they came to my attention since they’re a member of the OSB Alliance on whose board I’m a member. Furthermore Tuxedo Computers are a sponsor of the OSBAR which I’m part of the organizational team. OpenBSD on the Tuxedo InfinityBook I’ve asked the guys over at Tuxedo Computers whether they would be interested to have some tests with *BSD done and that I could test drive one of their machines and give feedback on what works and what does not - and possibly look into it.+ Within a few weeks they shipped me a machine and last week the InfinityBook Pro 14” arrived. Awesome. Thanks already to the folks at Tuxedo Computers. The machine arrived accompanied by lot’s of swag :) The InfinityBook is a very nice machine and allows a wide range of configuration. The configuration that was shipped to me: Intel Core i7-8550U 1x 16GB RAM 2400Mhz Crucial Ballistix Sport LT 250 GB Samsung 860 EVO (M.2 SATAIII) I used a USB-stick to boot install63.fs and re-installed the machine with OpenBSD. Full dmesg. The installation went flawlessly, the needed intel firmware is being installed after installation automatically via fw_update(1). Out of the box the graphics works and once installed the machine presents the login. Video When X starts the display is turned off for some reason. You will need to hit fn+f12 (the key with the moon on it) then the display will go on. Aside from that little nit, X works just fine and presents one the expected resolution. External video is working just fine as well. Either via hdmi output or via the mini displayport connector. The buttons for adjusting brightness (fn+f8 and fn+f9) are not working. Instead one has to use wsconsctl(8) to adjust the brightness. Networking The infinityBook has built-in ethernet, driven by re(4) And for the wireless interface the iwm(4) driver is being used. Both work as expected. ACPI Neither suspend nor hibernate work. Reporting of battery status is bogus as well. Some of the keyboard function keys work: LCD on/off works (fn+f2) Keyboard backlight dimming works (fn+f4) Volume (fn+f5 / fn+f6) works Sound The azalia chipset is being used for audio processing. Works as expected, volume can be controlled via buttons (fn+f5, fn+f6) or via mixerctl. Touchpad Can be controlled via wsconsctl(8). So far I must say, that the InfinityBook makes a nice machine - and I’m enjoying working with it. iXsystems iXsystems - Its all NAS How ZFS makes things like ‘zfs diff’ report filenames efficiently As a copy on write (file)system, ZFS can use the transaction group (txg) numbers that are embedded in ZFS block pointers to efficiently find the differences between two txgs; this is used in, for example, ZFS bookmarks. However, as I noted at the end of my entry on block pointers, this doesn’t give us a filesystem level difference; instead, it essentially gives us a list of inodes (okay, dnodes) that changed. In theory, turning an inode or dnode number into the path to a file is an expensive operation; you basically have to search the entire filesystem until you find it. In practice, if you’ve ever run ‘zfs diff’, you’ve likely noticed that it runs pretty fast. Nor is this the only place that ZFS quickly turns dnode numbers into full paths, as it comes up in ‘zpool status’ reports about permanent errors. At one level, zfs diff and zpool status do this so rapidly because they ask the ZFS code in the kernel to do it for them. At another level, the question is how the kernel’s ZFS code can be so fast. The interesting and surprising answer is that ZFS cheats, in a way that makes things very fast when it works and almost always works in normal filesystems and with normal usage patterns. The cheat is that ZFS dnodes record their parent’s object number. If you’re familiar with the twists and turns of Unix filesystems, you’re now wondering how ZFS deals with hardlinks, which can cause a file to be in several directories at once and so have several parents (and then it can be removed from some of the directories). The answer is that ZFS doesn’t; a dnode only ever tracks a single parent, and ZFS accepts that this parent information can be inaccurate. I’ll quote the comment in zfsobjto_pobj: When a link is removed [the file’s] parent pointer is not changed and will be invalid. There are two cases where a link is removed but the file stays around, when it goes to the delete queue and when there are additional links. Before I get into the details, I want to say that I appreciate the brute force elegance of this cheat. The practical reality is that most Unix files today don’t have extra hardlinks, and when they do most hardlinks are done in ways that won’t break ZFS’s parent stuff. The result is that ZFS has picked an efficient implementation that works almost all of the time; in my opinion, the great benefit we get from having it around are more than worth the infrequent cases where it fails or malfunctions. Both zfs diff and having filenames show up in zpool status permanent error reports are very useful (and there may be other cases where this gets used). The current details are that any time you hardlink a file to somewhere or rename it, ZFS updates the file’s parent to point to the new directory. Often this will wind up with a correct parent even after all of the dust settles; for example, a common pattern is to write a file to an initial location, hardlink it to its final destination, and then remove the initial location version. In this case, the parent will be correct and you’ll get the right name. News Roundup What is FreeBSD? Why Should You Choose It Over Linux? Not too long ago I wondered if and in what situations FreeBSD could be faster than Linux and we received a good amount of informative feedback. So far, Linux rules the desktop space and FreeBSD rules the server space. In the meantime, though, what exactly is FreeBSD? And at what times should you choose it over a GNU/Linux installation? Let’s tackle these questions. FreeBSD is a free and open source derivative of BSD (Berkeley Software Distribution) with a focus on speed, stability, security, and consistency, among other features. It has been developed and maintained by a large community ever since its initial release many years ago on November 1, 1993. BSD is the version of UNIX® that was developed at the University of California in Berkeley. And being a free and open source version, “Free” being a prefix to BSD is a no-brainer. What’s FreeBSD Good For? FreeBSD offers a plethora of advanced features and even boasts some not available in some commercial Operating Systems. It makes an excellent Internet and Intranet server thanks to its robust network services that allow it to maximize memory and work with heavy loads to deliver and maintain good response times for thousands of simultaneous user processes. FreeBSD runs a huge number of applications with ease. At the moment, it has over 32,000 ported applications and libraries with support for desktop, server, and embedded environments. with that being said, let me also add that FreeBSD is excellent for working with advanced embedded platforms. Mail and web appliances, timer servers, routers, MIPS hardware platforms, etc. You name it! FreeBSD is available to install in several ways and there are directions to follow for any method you want to use; be it via CD-ROM, over a network using NFS or FTP, or DVD. FreeBSD is easy to contribute to and all you have to do is to locate the section of the FreeBSD code base to modify and carefully do a neat job. Potential contributors are also free to improve on its artwork and documentation, among other project aspects. FreeBSD is backed by the FreeBSD Foundation, a non-profit organization that you can contribute to financially and all direct contributions are tax deductible. FreeBSD’s license allows users to incorporate the use of proprietary software which is ideal for companies interested in generating revenues. Netflix, for example, could cite this as one of the reasons for using FreeBSD servers. Why Should You Choose It over Linux? From what I’ve gathered about both FreeBSD and Linux, FreeBSD has a better performance on servers than Linux does. Yes, its packaged applications are configured to offer better a performance than Linux and it is usually running fewer services by default, there really isn’t a way to certify which is faster because the answer is dependent on the running hardware and applications and how the system is tuned. FreeBSD is reportedly more secure than Linux because of the way the whole project is developed and maintained. Unlike with Linux, the FreeBSD project is controlled by a large community of developers around the world who fall into any of these categories; core team, contributors, and committers. FreeBSD is much easier to learn and use because there aren’t a thousand and one distros to choose from with different package managers, DEs, etc. FreeBSD is more convenient to contribute to because it is the entire OS that is preserved and not just the kernel and a repo as is the case with Linux. You can easily access all of its versions since they are sorted by release numbers. Apart from the many documentations and guides that you can find online, FreeBSD has a single official documentation wherein you can find the solution to virtually any issue you will come across. So, you’re sure to find it resourceful. FreeBSD has close to no software issues compared to Linux because it has Java, is capable of running Windows programs using Wine, and can run .NET programs using Mono. FreeBSD’s ports/packages system allows you to compile software with specific configurations, thereby avoiding conflicting dependency and version issues. Both the FreeBSD and GNU/Linux project are always receiving updates. The platform you decide to go with is largely dependent on what you want to use it for, your technical know-how, willingness to learn new stuff, and ultimately your preference. What is your take on the topic? For what reasons would you choose FreeBSD over Linux if you would? Let us know what you think about both platforms in the comments section below. PS4 5.05 BPF Double Free Kernel Exploit Writeup Introduction Welcome to the 5.0x kernel exploit write-up. A few months ago, a kernel vulnerability was discovered by qwertyoruiopz and an exploit was released for BPF which involved crafting an out-of-bounds (OOB) write via use-after-free (UAF) due to the lack of proper locking. It was a fun bug, and a very trivial exploit. Sony then removed the write functionality from BPF, so that exploit was patched. However, the core issue still remained (being the lack of locking). A very similar race condition still exists in BPF past 4.55, which we will go into detail below on. The full source of the exploit can be found here. This bug is no longer accessible however past 5.05 firmware, because the BPF driver has finally been blocked from unprivileged processes - WebKit can no longer open it. Sony also introduced a new security mitigation in 5.0x firmwares to prevent the stack pointer from pointing into user space, however we’ll go more in detail on this a bit further down. Assumptions Some assumptions are made of the reader’s knowledge for the writeup. The avid reader should have a basic understanding of how memory allocators work - more specifically, how malloc() and free() allocate and deallocate memory respectively. They should also be aware that devices can be issued commands concurrently, as in, one command could be received while another one is being processed via threading. An understanding of C, x86, and exploitation basics is also very helpful, though not necessarily required. Background This section contains some helpful information to those newer to exploitation, or are unfamiliar with device drivers, or various exploit techniques such as heap spraying and race conditions. Feel free to skip to the “A Tale of Two Free()'s” section if you’re already familiar with this material. What Are Drivers? There are a few ways that applications can directly communicate with the operating system. One of which is system calls, which there are over 600 of in the PS4 kernel, ~500 of which are FreeBSD - the rest are Sony-implemented. Another method is through something called “Device Drivers”. Drivers are typically used to bridge the gap between software and hardware devices (usb drives, keyboard/mouse, webcams, etc) - though they can also be used just for software purposes. There are a few operations that a userland application can perform on a driver (if it has sufficient permissions) to interface with it after opening it. In some instances, one can read from it, write to it, or in some cases, issue more complex commands to it via the ioctl() system call. The handlers for these commands are implemented in kernel space - this is important, because any bugs that could be exploited in an ioctl handler can be used as a privilege escalation straight to ring0 - typically the most privileged state. Drivers are often the more weaker points of an operating system for attackers, because sometimes these drivers are written by developers who don’t understand how the kernel works, or the drivers are older and thus not wise to newer attack methods. The BPF Device Driver If we take a look around inside of WebKit’s sandbox, we’ll find a /dev directory. While this may seem like the root device driver path, it’s a lie. Many of the drivers that the PS4 has are not exposed to this directory, but rather only ones that are needed for WebKit’s operation (for the most part). For some reason though, BPF (aka. the “Berkely Packet Filter”) device is not only exposed to WebKit’s sandbox - it also has the privileges to open the device as R/W. This is very odd, because on most systems this driver is root-only (and for good reason). If you want to read more into this, refer to my previous write-up with 4.55FW. What Are Packet Filters? Below is an excerpt from the 4.55 bpfwrite writeup. Since the bug is directly in the filter system, it is important to know the basics of what packet filters are. Filters are essentially sets of pseudo-instructions that are parsed by bpf_filter() (which are ran when packets are received). While the pseudo-instruction set is fairly minimal, it allows you to do things like perform basic arithmetic operations and copy values around inside it’s buffer. Breaking down the BPF VM in it’s entirety is far beyond the scope of this write-up, just know that the code produced by it is ran in kernel mode - this is why read/write access to /dev/bpf should be privileged. Race Conditions Race conditions occur when two processes/threads try to access a shared resource at the same time without mutual exclusion. The problem was ultimately solved by introducing concepts such as the “mutex” or “lock”. The idea is when one thread/process tries to access a resource, it will first acquire a lock, access it, then unlock it once it’s finished. If another thread/process tries to access it while the other has the lock, it will wait until the other thread is finished. This works fairly well - when it’s used properly. Locking is hard to get right, especially when you try to implement fine-grained locking for performance. One single instruction or line of code outside the locking window could introduce a race condition. Not all race conditions are exploitable, but some are (such as this one) - and they can give an attacker very powerful bugs to work with. Heap Spraying The process of heap spraying is fairly simple - allocate a bunch of memory and fill it with controlled data in a loop and pray your allocation doesn’t get stolen from underneath you. It’s a very useful technique when exploiting something such as a use-after-free(), as you can use it to get controlled data into your target object’s backing memory. By extension, it’s useful to do this for a double free() as well, because once we have a stale reference, we can use a heap spray to control the data. Since the object will be marked “free” - the allocator will eventually provide us with control over this memory, even though something else is still using it. That is, unless, something else has already stolen the pointer from you and corrupts it - then you’ll likely get a system crash, and that’s no fun. This is one factor that adds to the variance of exploits, and typically, the smaller the object, the more likely this is to happen. Follow the link to read more of the article DigitalOcean http://do.co/bsdnow OpenBSD gains Wi-Fi “auto-join” In a change which is bound to be welcomed widely, -current has gained “auto-join” for Wi-Fi networks. Peter Hessler (phessler@) has been working on this for quite some time and he wrote about it in his p2k18 hackathon report. He has committed the work from the g2k18 hackathon in Ljubljana: CVSROOT: /cvs Module name: src Changes by: phessler@cvs.openbsd.org 2018/07/11 14:18:09 Modified files: sbin/ifconfig : ifconfig.8 ifconfig.c sys/net80211 : ieee80211ioctl.c ieee80211ioctl.h ieee80211node.c ieee80211node.h ieee80211_var.h Log message: Introduce 'auto-join' to the wifi 802.11 stack. This allows a system to remember which ESSIDs it wants to connect to, any relevant security configuration, and switch to it when the network we are currently connected to is no longer available. Works when connecting and switching between WPA2/WPA1/WEP/clear encryptions. example hostname.if: join home wpakey password join work wpakey mekmitasdigoat join open-lounge join cafe wpakey cafe2018 join "wepnetwork" nwkey "12345" dhcp inet6 autoconf up OK stsp@ reyk@ and enthusiasm from every hackroom I've been in for the last 3 years The usage should be clear from the commit message, but basically you ‘join’ all the networks you want to auto-join as you would previously use ‘nwid’ to connect to one specific network. Then the kernel will join the network that’s actually in range and do the rest automagically for you. When you move out of range of that network you lose connectivity until you come in range of the original (where things will continue to work as you’ve been used to) or one of the other networks (where you will associate and then get a new lease). Thanks to Peter for working on this feature - something many a Wi-Fi using OpenBSD user will be able to benefit from. FreeBSD Jails the hard way There are many great options for managing FreeBSD Jails. iocage, warden and ez-jail aim to streamline the process and make it quick an easy to get going. But sometimes the tools built right into the OS are overlooked. This post goes over what is involved in creating and managing jails using only the tools built into FreeBSD. For this guide, I’m going to be putting my jails in /usr/local/jails. I’ll start with a very simple, isolated jail. Then I’ll go over how to use ZFS snapshots, and lastly nullfs mounts to share the FreeBSD base files with multiple jails. I’ll also show some examples of how to use the templating power of jail.conf to apply similar settings to all your jails. Full Jail Make a directory for the jail, or a zfs dataset if you prefer. Download the FreeBSD base files, and any other parts of FreeBSD you want. In this example I’ll include the 32 bit libraries as well. Update your FreeBSD base install. Verify your download. We’re downloading these archives over FTP after all, we should confirm that this download is valid and not tampered with. The freebsd-update IDS command verifies the installation using a PGP key which is in your base system, which was presumably installed with an ISO that you verified using the FreeBSD signed checksums. Admittedly this step is a bit of paranoia, but I think it’s prudent. Make sure you jail has the right timezone and dns servers and a hostname in rc.conf. Edit jail.conf with the details about your jail. Start and login to your jail. 11 commands and a config file, but this is the most tedious way to make a jail. With a little bit of templating it can be even easier. So I’ll start by making a template. Making a template is basically the same as steps 1, 2 and 3 above, but with a different destination folder, I’ll condense them here. Creating a template Create a template or a ZFS dataset. If you’d like to use the zfs clone method of deploying templates, you’ll need to create a zfs dataset instead of a folder. Update your template with freebsd-update. Verify your install And that’s it, now you have a fully up to date jail template. If you’ve made this template with zfs, you can easily deploy it using zfs snapshots. Deploying a template with ZFS snapshots Create a snapshot. My last freebsd-update to my template brought it to patch level 17, so I’ll call my snapshot p10. Clone the snapshot to a new jail. Configure the jail hostname. Add the jail definition to jail.conf, make sure you have the global jail settings from jail.conf listed in the fulljail example. Start the jail. The downside with the zfs approach is that each jail is now a fully independent, and if you need to update your jails, you have to update them all individually. By sharing a template using nullfs mounts you can have only one copy of the base system that only needs to be updated once. Follow the link to see the rest of the article about Thin jails using NullFS mounts Simplifying jail.conf Hopefully this has helped you understand the process of how to create and manage FreeBSD jails without tools that abstract away all the details. Those tools are often quite useful, but there is always benefit in learning to do things the hard way. And in this case, the hard way doesn’t seem to be that hard after all. Beastie Bits Meetup in Zurich #4, July edition (July 19) – Which you likely missed, but now you know to look for the August edition! The next two BSD-PL User group meetings in Warsaw have been scheduled for July 30th and Aug 9th @ 1830 CEST – Submit your topic proposals now Linux Geek Books - Humble Bundle Extend loader(8) geli support to all architectures and all disk-like devices Upgrading from a bootpool to a single encrypted pool – skip the gptzfsboot part, and manually update your EFI partition with loader.efi The pkgsrc 2018Q2 for Illumos is available with 18500+ binary packages NetBSD ARM64 Images Available with SMP for RPi3 / NanoPi / Pine64 Boards Recently released CDE 2.3.0 running on Tribblix (Illumos) An Interview With Tech & Science Fiction Author Michael W Lucas A reminder : MeetBSD CFP EuroBSDCon talk acceptances have gone out, and once the tutorials are confirmed, registration will open. That will likely have happened by time you see this episode, so go register! See you in Romania Tarsnap Feedback/Questions Wilyarti - Adblocked on FreeBSD Continued… Andrew - A Question and a Story Matthew - Thanks Brian - PCI-E Controller Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
Fanless server setup with FreeBSD, NetBSD on pinebooks, another BSDCan trip report, transparent network audio, MirBSD's Korn Shell on Plan9, static site generators on OpenBSD, and more. ##Headlines Silent Fanless FreeBSD Desktop/Server Today I will write about silent fanless FreeBSD desktop or server computer … or NAS … or you name it, it can have multiple purposes. It also very low power solution, which also means that it will not overheat. Silent means no fans at all, even for the PSU. The format of the system should also be brought to minimum, so Mini-ITX seems best solution here. I have chosen Intel based solutions as they are very low power (6-10W), if you prefer AMD (as I often do) the closest solution in comparable price and power is Biostar A68N-2100 motherboard with AMD E1-2100 CPU and 9W power. Of course AMD has even more low power SoC solutions but finding the Mini-ITX motherboard with decent price is not an easy task. For comparison Intel has lots of such solutions below 6W whose can be nicely filtered on the ark.intel.com page. Pity that AMD does not provide such filtration for their products. I also chosen AES instructions as storage encryption (GELI on FreeBSD) today seems as obvious as HTTPS for the web pages. Here is how the system look powered up and working This motherboard uses Intel J3355 SoC which uses 10W and has AES instructions. It has two cores at your disposal but it also supports VT-x and EPT extensions so you can even run Bhyve on it. Components Now, an example system would look like that one below, here are the components with their prices. $49 CPU/Motherboard ASRock J3355B-ITX Mini-ITX $14 RAM Crucial 4 GB DDR3L 1.35V (low power) $17 PSU 12V 160W Pico (internal) $11 PSU 12V 96W FSP (external) $5 USB 2.0 Drive 16 GB ADATA $4 USB Wireless 802.11n $100 TOTAL The PSU 12V 160W Pico (internal) and PSU 12V 96W FSP can be purchased on aliexpress.com or ebay.com for example, at least I got them there. Here is the 12V 160W Pico (internal) PSU and its optional additional cables to power the optional HDDs. If course its one SATA power and one MOLEX power so additional MOLEX-SATA power adapter for about 1$ would be needed. Here is the 12V 96W FSP (external) PSU without the power cord. This gives as total silent fanless system price of about $120. Its about ONE TENTH OF THE COST of the cheapest FreeNAS hardware solution available – the FreeNAS Mini (Diskless) costs $1156 also without disks. You can put plain FreeBSD on top of it or Solaris/Illumos distribution OmniOSce which is server oriented. You can use prebuilt NAS solution based on FreeBSD like FreeNAS, NAS4Free, ZFSguru or even Solaris/Illumos based storage with napp-it appliance. ###An annotated look at a NetBSD Pinebook’s startup Pinebook is an affordable 64-bit ARM notebook. Today we’re going to take a look at the kernel output at startup and talk about what hardware support is available on NetBSD. Photo Pinebook comes with 2GB RAM standard. A small amount of this is reserved by the kernel and framebuffer. NetBSD uses flattened device-tree (FDT) to enumerate devices on all Allwinner based SoCs. On a running system, you can inspect the device tree using the ofctl(8) utility: Pinebook’s Allwinner A64 processor is based on the ARM Cortex-A53. It is designed to run at frequencies up to 1.2GHz. The A64 is a quad core design. NetBSD’s aarch64 pmap does not yet support SMP, so three cores are disabled for now. The interrupt controller is a standard ARM GIC-400 design. Clock drivers for managing PLLs, module clock dividers, clock gating, software resets, etc. Information about the clock tree is exported in the hw.clk sysctl namespace (root access required to read these values). # sysctl hw.clk.sun50ia64ccu0.mmc2 hw.clk.sun50ia64ccu0.mmc2.rate = 200000000 hw.clk.sun50ia64ccu0.mmc2.parent = pllperiph02x hw.clk.sun50ia64ccu0.mmc2.parent_domain = sun50ia64ccu0 Digital Ocean http://do.co/bsdnow ###BSDCan 2018 Trip Report: Mark Johnston BSDCan is a highlight of my summers: the ability to have face-to-face conversations with fellow developers and contributors is invaluable and always helps refresh my enthusiasm for FreeBSD. While in a perfect world we would all be able to communicate effectively over the Internet, it’s often noted that locking a group of developers together in a room can be a very efficient way to make progress on projects that otherwise get strung out over time, and to me this is one of the principal functions of BSD conferences. In my case I was able to fix some kgdb bugs that had been hindering me for months; get some opinions on the design of a feature I’ve been working on for FreeBSD 12.0; hear about some ongoing usage of code that I’ve worked on; and do some pair-debugging of an issue that has been affecting another developer. As is tradition, on Tuesday night I dropped off my things at the university residence where I was staying, and headed straight to the Royal Oak. This year it didn’t seem quite as packed with BSD developers, but I did meet several long-time colleagues and get a chance to catch up. In particular, I chatted with Justin Hibbits and got to hear about the bring-up of FreeBSD on POWER9, a new CPU family released by IBM. Justin was able to acquire a workstation based upon this CPU, which is a great motivator for getting FreeBSD into shape on that platform. POWER9 also has some promise in the server market, so it’s important for FreeBSD to be a viable OS choice there. Wednesday morning saw the beginning of the two-day FreeBSD developer summit, which precedes the conference proper. Gordon Tetlow led the summit and did an excellent job organizing things and keeping to the schedule. The first presentation was by Deb Goodkin of the FreeBSD Foundation, who gave an overview of the Foundation’s role and activities. After Deb’s presentation, present members of the FreeBSD core team discussed the work they had done over the past two years, as well as open tasks that would be handed over to the new core team upon completion of the ongoing election. Finally, Marius Strobl rounded off the day’s presentations by discussing the state and responsibilities of FreeBSD’s release engineering team. One side discussion of interest to me was around the notion of tightening integration with our Bugzilla instance; at moment we do not have any good means to mark a given bug as blocking a release, making it easy for bugs to slip into releases and thus lowering our overall quality. With FreeBSD 12.0 upon us, I plan to help with the triage and fixes for known regressions before the release process begins. After a break, the rest of the morning was devoted to plans for features in upcoming FreeBSD releases. This is one of my favorite discussion topics and typically takes the form of have/need/want, where developers collectively list features that they’ve developed and intend to upstream (have), features that they are missing (need), and nice-to-have features (want). This year, instead of the usual format, we listed features that are intended to ship in FreeBSD 12.0. The compiled list ended up being quite ambitious given how close we are to the beginning of the release cycle, but many individual developers (including myself) have signed up to deliver work. I’m hopeful that most, if not all of it, will make it into the release. After lunch, I attended a discussion led by Matt Ahrens and Alexander Motin on OpenZFS. Of particular interest to me were some observations made regarding the relative quantity and quality of contributions made by different “camps” of OpenZFS users (illumos, FreeBSD and ZoL), and their respective track records of upstreaming enhancements to the OpenZFS project. In part due to the high pace of changes in ZoL, the definition of “upstream” for ZFS has become murky, and of late ZFS changes have been ported directly from ZoL. Alexander discussed some known problems with ZFS on FreeBSD that have been discovered through performance testing. While I’m not familiar with ZFS internals, Alexander noted that ZFS’ write path has poor SMP scalability on FreeBSD owing to some limitations in a certain kernel API called taskqueue(9). I would like to explore this problem further and perhaps integrate a relatively new alternative interface which should perform better. Friday and Saturday were, of course, taken up by BSDCan talks. Friday’s keynote was by Benno Rice, who provided some history of UNIX boot systems as a precursor to some discussion of systemd and the difficulties presented by a user and developer community that actively resist change. The rest of the morning was consumed by talks and passed by quickly. First was Colin Percival’s detailed examination of where the FreeBSD kernel spends time during boot, together with an overview of some infrastructure he added to track boot times. He also provided a list of improvements that have been made since he started taking measurements, and some areas we can further improve. Colin’s existing work in this area has already brought about substantial reductions in boot time; amusingly, one of the remaining large delays comes from the keyboard driver, which contains a workaround for old PS/2 keyboards. While there seems to be general agreement that the workaround is probably no longer needed on most systems, the lingering uncertainty around this prevents us from removing the workaround. This is, sadly, a fairly typical example of an OS maintenance burden, and underscores the need to carefully document hardware bug workarounds. After this talk, I got to see some rather novel demonstrations of system tracing using dwatch, a new utility by Devin Teske, which aims to provide a user-friendly interface to DTrace. After lunch, I attended talks on netdump, a protocol for transmitting kernel dumps over a network after the system has panicked, and on a VPC implementation for FreeBSD. After the talks ended, I headed yet again to the hacker lounge and had some fruitful discussions on early microcode loading (one of my features for FreeBSD 12.0). These led me to reconsider some aspects of my approach and saved me a lot of time. Finally, I continued my debugging session from Wednesday with help from a couple of other developers. Saturday’s talks included a very thorough account by Li-Wen Hsu of his work in organizing a BSD conference in Taipei last year. As one of the attendees, I had felt that the conference had gone quite smoothly and was taken aback by the number of details and pitfalls that Li-Wen enumerated during his talk. This was followed by an excellent talk by Baptiste Daroussin on the difficulties one encounters when deploying FreeBSD in new environments. Baptiste offered criticisms of a number of aspects of FreeBSD, some of which hit close to home as they involved portions of the system that I’ve worked on. At the conclusion of the talks, we all gathered in the main lecture hall, where Dan led a traditional and quite lively auction for charity. I managed to snag a Pine64 board and will be getting FreeBSD installed on it the first chance I get. At the end of the auction, we all headed to ByWard for dinner, concluding yet another BSDCan. Thanks to Mark for sharing his experiences at this years BSDCan ##News Roundup Transparent network audio with mpd & sndiod Landry Breuil (landry@ when wearing his developer hat) wrote in… I've been a huge fan of MPD over the years to centralize my audio collection, and i've been using it with the http output to stream the music as a radio on the computer i'm currently using… audio_output { type "sndio" name "Local speakers" mixer_type "software" } audio_output { type "httpd" name "HTTP stream" mixer_type "software" encoder "vorbis" port "8000" format "44100:16:2" } this setup worked for years, allows me to stream my home radio to $work by tunnelling the port 8000 over ssh via LocalForward, but that still has some issues: a distinct timing gap between the 'local output' (ie the speakers connected to the machine where MPD is running) and the 'http output' caused by the time it takes to reencode the stream, which is ugly when you walk through the house and have a 15s delay sometimes mplayer as a client doesn't detect the pauses in the stream and needs to be restarted i need to configure/start a client on each computer and point it at the sound server url (can do via gmpc shoutcast client plugin…) it's not that elegant to reencode the stream, and it wastes cpu cycles So the current scheme is: mpd -> http output -> network -> mplayer -> sndiod on remote machine | -> sndio output -> sndiod on soundserver Fiddling a little bit with mpd outputs and reading the sndio output driver, i remembered sndiod has native network support… and the mpd sndio output allows you to specify a device (it uses SIO_DEVANY by default). So in the end, it's super easy to: enable network support in sndio on the remote machine i want the audio to play by adding -L to sndiod_flags (i have two audio devices, with an input coming from the webcam): sndiod_flags="-L10.246.200.10 -f rsnd/0 -f rsnd/1" open pf on port 11025 from the sound server ip: pass in proto tcp from 10.246.200.1 to any port 11025 configure a new output in mpd: audio_output { type "sndio" name "sndio on renton" device "snd@10.246.200.10/0" mixer_type "software" } and enable the new output in mpd: $mpc enable 2 Output 1 (Local speakers) is disabled Output 2 (sndio on renton) is enabled Output 3 (HTTP stream) is disabled Results in a big win: no gap anymore with the local speakers, no reencoding, no need to configure a client to play the stream, and i can still probably reproduce the same scheme over ssh from $work using a RemoteForward. mpd -> sndio output 2 -> network -> sndiod on remote machine | -> sndio output 1 -> sndiod on soundserver Thanks ratchov@ for sndiod :) ###MirBSD’s Korn Shell on Plan9 Jehanne Let start by saying that I’m not really a C programmer. My last public contribution to a POSIX C program was a little improvement to the Snort’s react module back in 2008. So while I know the C language well enough, I do not know anything about the subtleness of the standard library and I have little experience with POSIX semantics. This is not a big issue with Plan 9, since the C library and compiler are not standard anyway, but with Jehanne (a Plan 9 derivative of my own) I want to build a simple, loosely coupled, system that can actually run useful free software ported from UNIX. So I ported RedHat’s newlib to Jehanne on top of a new system library I wrote, LibPOSIX, that provides the necessary emulations. I wrote several test, checking they run the same on Linux and Jehanne, and then I begun looking for a real-world, battle tested, application to port first. I approached MirBSD’s Korn Shell for several reason: it is simple, powerful and well written it has been ported to several different operating systems it has few dependencies it’s the default shell in Android, so it’s really battle tested I was very confident. I had read the POSIX standard after all! And I had a test suite! I remember, I thought “Given newlib, how hard can it be?” The porting begun on September 1, 2017. It was completed by tg on January 5, 2018. 125 nights later. Turn out, my POSIX emulation was badly broken. Not just because of the usual bugs that any piece of C can have: I didn’t understood most POSIX semantics at all! iXsystems ###Static site generator with rsync and lowdown on OpenBSD ssg is a tiny POSIX-compliant shell script with few dependencies: lowdown(1) to parse markdown, rsync(1) to copy temporary files, and entr(1) to watch file changes. It generates Markdown articles to a static website. It copies the current directory to a temporary on in /tmp skipping .* and _*, renders all Markdown articles to HTML, generates RSS feed based on links from index.html, extracts the first tag from every article to generate a sitemap and use it as a page title, then wraps articles with a single HTML template, copies everything from the temporary directory to $DOCS/ Why not Jekyll or “$X”? ssg is one hundred times smaller than Jekyll. ssg and its dependencies are about 800KB combined. Compare that to 78MB of ruby with Jekyll and all the gems. So ssg can be installed in just few seconds on almost any Unix-like operating system. Obviously, ssg is tailored for my needs, it has all features I need and only those I use. Keeping ssg helps you to master your Unix-shell skills: awk, grep, sed, sh, cut, tr. As a web developer you work with lots of text: code and data. So you better master these wonderful tools. Performance 100 pps. On modern computers ssg generates a hundred pages per second. Half of a time for markdown rendering and another half for wrapping articles into the template. I heard good static site generators work—twice as fast—at 200 pps, so there’s lots of performance that can be gained. ;) ###Why does FreeBSD have virtually no (0%) desktop market share? Because someone made a horrible design decision back in 1984. In absolute fairness to those involved, it was an understandable decision, both from a research perspective, and from an economic perspective, although likely not, from a technology perspective. Why and what. The decision was taken because the X Window System was intended to run on cheap hardware, and, at the time, that meant reduced functionality in the end-point device with the physical display attached to it. At the same time, another force was acting to also limit X displays to display services only, rather than rolling in both window management and specific widget instances for common operational paradigms. Mostly, common operational paradigms didn’t really exist for windowing systems because they also simply didn’t exist at the time, and no one really knew how people were going to use the things, and so researchers didn’t want to commit future research to a set of hard constraints. So a decision was made: separate the display services from the application at the lowest level of graphics primitives currently in use at the time. The ramifications of this were pretty staggering. First, it guaranteed that all higher level graphics would live on the host side of the X protocol, instead of on the display device side of the protocol. Despite a good understanding of Moore’s law, and the fact that, since no X Terminals existed at the time as hardware, but were instead running as emulations on workstations that had sufficient capability, this put the higher level GUI object libraries — referred to as “widgets” — in host libraries linked into the applications. Second, it guaranteed that display organization and management paradigms would also live on the host side of the protocol — assumed, in contradiction to the previous decision, to be running on the workstation. But, presumably, at some point, as lightweight X Terminals became available, to migrate to a particular host computer managing compute resource login/access services. Between these early decisions reigned chaos. Specifically, the consequences of these decisions have been with us ever since: Look-and-feel are a consequence of the toolkit chosen by the application programmer, rather than a user decision which applies universally to all applications. You could call this “lack of a theme”, and — although I personally despise the idea of customizing or “theming” desktops — this meant that one paradigm chosen by the user would not apply universally across all applications, no matter who had written them. Window management style is a preference. You could call this a more radical version of “theming” — which you will remember, I despise — but a consequence to this is that training is not universal across personnel using such systems, nor is it transferrable. In other words, I can’t send someone to a class, and have them come back and use the computers in the office as a tool, with the computer itself — and the elements not specific to the application itself — disappearing into the background. Both of these ultimately render an X-based system unsuitable for desktops. I can’t pay once for training. Training that I do pay for does not easily and naturally translate between applications. Each new version may radically alter the desktop management paradigm into unrecognizability. Is there hope for the future? Well, the Linux community has been working on something called Wayland, and it is very promising… …In the same way X was “very promising” in 1984, because, unfortunately, they are making exactly the same mistakes X made in 1984, rather than correcting them, now that we have 20/20 hindsight, and know what a mature widget library should look like. So Wayland is screwing up again. But hey, it only took us, what, 25 years to get from X in 1987 to Wayland in in 2012. Maybe if we try again in 2037, we can get to where Windows was in 1995. ##Beastie Bits New washing machine comes with 7 pages of open source licenses! BSD Jobs Site FreeBSD Foundation Update, May 2018 FreeBSD Journal looking for book reviewers zedenv ZFS Boot Environment Manager Tarsnap ##Feedback/Questions Wouter - Feedback Efraim - OS Suggestion kevr - Raspberry Pi2/FreeBSD/Router on a Stick Vanja - Interview Suggestion Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
FreeBSD 11.2 has been released, setting up an MTA behind Tor, running pfsense on DigitalOcean, one year of C, using OpenBGPD to announce VM networks, the power to serve, and a BSDCan trip report. ##Headlines FreeBSD 11.2-RELEASE Available FreeBSD 11.2 was released today (June 27th) and is ready for download Highlights: OpenSSH has been updated to version 7.5p1. OpenSSL has been updated to version 1.0.2o. The clang, llvm, lldb and compiler-rt utilities have been updated to version 6.0.0. The libarchive(3) library has been updated to version 3.3.2. The libxo(3) library has been updated to version 0.9.0. Major Device driver updates to: cxgbe(4) – Chelsio 10/25/40/50/100 gigabit NICs – version 1.16.63.0 supports T4, T5 and T6 ixl(4) – Intel 10 and 40 gigabit NICs, updated to version 1.9.9-k ng_pppoe(4) – driver has been updated to add support for user-supplied Host-Uniq tags New drivers: + drm-next-kmod driver supporting integrated Intel graphics with the i915 driver. mlx5io(4) – a new IOCTL interface for Mellanox ConnectX-4 and ConnectX-5 10/20/25/40/50/56/100 gigabit NICs ocs_fc(4) – Emulex Fibre Channel 8/16/32 gigabit Host Adapters smartpqi(4) – HP Gen10 Smart Array Controller Family The newsyslog(8) utility has been updated to support RFC5424-compliant messages when rotating system logs The diskinfo(8) utility has been updated to include two new flags, -s which displays the disk identity (usually the serial number), and -p which displays the physical path to the disk in a storage controller. The top(1) utility has been updated to allow filtering on multiple user names when the -U flag is used The umount(8) utility has been updated to include a new flag, -N, which is used to forcefully unmount an NFS mounted filesystem. The ps(1) utility has been updated to display if a process is running with capsicum(4) capability mode, indicated by the flag ‘C’ The service(8) utility has been updated to include a new flag, -j, which is used to interact with services running within a jail(8). The argument to -j can be either the name or numeric jail ID The mlx5tool(8) utility has been added, which is used to manage Connect-X 4 and Connect-X 5 devices supported by mlx5io(4). The ifconfig(8) utility has been updated to include a random option, which when used with the ether option, generates a random MAC address for an interface. The dwatch(1) utility has been introduced The efibootmgr(8) utility has been added, which is used to manipulate the EFI boot manager. The etdump(1) utility has been added, which is used to view El Torito boot catalog information. The linux(4) ABI compatibility layer has been updated to include support for musl consumers. The fdescfs(5) filesystem has been updated to support Linux®-specific fd(4) /dev/fd and /proc/self/fd behavior Support for virtio_console(4) has been added to bhyve(4). The length of GELI passphrases entered when booting a system with encrypted disks is now hidden by default. See the configuration options in geli(8) to restore the previous behavior. In addition to the usual CD/DVD ISO, Memstick, and prebuilt VM images (raw, qcow2, vhd, and vmdk), FreeBSD 11.2 is also available on: Amazon EC2 Google Compute Engine Hashicorp/Atlas Vagrant Microsoft Azure In addition to a generic ARM64 image for devices like the Pine64 and Raspberry Pi 3, specific images are provided for: GUMSTIX BANANAPI BEAGLEBONE CUBIEBOARD CUBIEBOARD2 CUBOX-HUMMINGBOARD RASPBERRY PI 2 PANDABOARD WANDBOARD Full Release Notes ###Setting up an MTA Behind Tor This article will document how to set up OpenSMTPD behind a fully Tor-ified network. Given that Tor’s DNS resolver code does not support MX record lookups, care must be taken for setting up an MTA behind a fully Tor-ified network. OpenSMTPD was chosen because it was easy to modify to force it to fall back to A/AAAA lookups when MX lookups failed with a DNS result code of NOTIMP (4). Note that as of 08 May 2018, the OpenSMTPD project is planning a configuration file language change. The proposed change has not landed. Once it does, this article will be updated to reflect both the old language and new. The reason to use an MTA behing a fully Tor-ified network is to be able to support email behind the .onion TLD. This setup will only allow us to send and receive email to and from the .onion TLD. Requirements: A fully Tor-ified network HardenedBSD as the operating system A server (or VM) running HardenedBSD behind the fully Tor-ified network. /usr/ports is empty Or is already pre-populated with the HardenedBSD Ports tree Why use HardenedBSD? We get all the features of FreeBSD (ZFS, DTrace, bhyve, and jails) with enhanced security through exploit mitigations and system hardening. Tor has a very unique threat landscape and using a hardened ecosystem is crucial to mitigating risks and threats. Also note that this article reflects how I’ve set up my MTA. I’ve included configuration files verbatim. You will need to replace the text that refers to my .onion domain with yours. On 08 May 2018, HardenedBSD’s version of OpenSMTPD just gained support for running an MTA behind Tor. The package repositories do not yet contain the patch, so we will compile OpenSMTPD from ports. Steps Installation Generating Cryptographic Key Material Tor Configuration OpenSMTPD Configuration Dovecot Configuration Testing your configuration Optional: Webmail Access iXsystems https://www.forbes.com/sites/forbestechcouncil/2018/06/21/strings-attached-knowing-when-and-when-not-to-accept-vc-funding/#30f9f18f46ec https://www.ixsystems.com/blog/self-2018-recap/ ###Running pfSense on a Digital Ocean Droplet I love pfSense (and opnSense, no discrimination here). I use it for just about anything, from homelab to large scale deployments and I’ll give out on any fancy for a pfSense setup on a decent hardware. I also love DigitalOcean, if you ever used them, you know why, if you never did, head over and try, you’ll understand why. . Unfortunately, while DO offers tremendous amount of useful distros and applications, pfSense isn’t one of them. But, where there’s a will, there’s a way, and here’s how to get pfSense up and running on DO so you can have it as the gatekeeper to your kingdom. Start by creating a FreeBSD droplet, choose your droplet size (for modest setups, I find the 5$ to be quite awesome): There are many useful things you can do with pfSense on your droplet, from OpenVPN, squid, firewalling, fancy routing, url filtering, dns black listing and much much more. One note though, before we wrap up: You have two ways to initiate the initial setup wizard of the web-configurator: Spin up another droplet, log into it and browse your way to the INTERNAL ip address of the internal NIC you’ve set up. This is the long and tedious way, but it’s also somewhat safer as it eliminates the small window of risk the second method poses. or Once your WAN address is all setup, your pfSense is ready to accept https connection to start the initial web-configurator setup. Thing is, there’s a default, well known set of credential to this initial wizard (admin:pfsense), so, there is a slight window of opportunity that someone can swoop in (assuming they know you’ve installed pfsense + your wan IP address + the exact time window between setting up the WAN interface and completing the wizard) and do . I leave it up to you which of the path you’d like to go, either way, once you’re done with the web-configurator wizard, you’ll have a shiny new pfSense installation at your disposal running on your favorite VPS. Hopefully this was helpful for someone, I hope to get a similar post soon detailing how to get FreeNAS up and running on DO. Many thanks to Tubsta and his blogpost as well as to Allan Jude, Kris Moore and Benedict Reuschling for their AWESOME and inspiring podcast, BSD Now. ##News Roundup One year of C It’s now nearly a year that I started writing non-trivial amounts of C code again (the first sokol_gfx.h commit was on the 14-Jul-2017), so I guess it’s time for a little retrospective. In the beginning it was more of an experiment: I wanted to see how much I would miss some of the more useful C++ features (for instance namespaces, function overloading, ‘simple’ template code for containers, …), and whether it is possible to write non-trivial codebases in C without going mad. Here are all the github projects I wrote in C: sokol: a slowly growing set of platform-abstraction headers sokol-samples - examples for Sokol chips - 8-bit chip emulators chips-test - tests and examples for the chip- emulators, including some complete home computer emulators (minus sound) All in all these are around 32k lines of code (not including 3rd party code like flextGL and HandmadeMath). I think I wrote more C code in the recent 10 months than any other language. So one thing seems to be clear: yes, it’s possible to write a non-trivial amount of C code that does something useful without going mad (and it’s even quite enjoyable I might add). Here’s a few things I learned: Pick the right language for a problem C is a perfect match for WebAssembly C99 is a huge improvement over C89 The dangers of pointers and explicit memory management are overrated Less Boilerplate Code Less Language Feature ‘Anxiety’ Conclusion All in all my “C experiment” is a success. For a lot of problems, picking C over C++ may be the better choice since C is a much simpler language (btw, did you notice how there are hardly any books, conferences or discussions about C despite being a fairly popular language? Apart from the neverending bickering about undefined behaviour from the compiler people of course ;) There simply isn’t much to discuss about a language that can be learned in an afternoon. I don’t like some of the old POSIX or Linux APIs as much as the next guy (e.g. ioctl(), the socket API or some of the CRT library functions), but that’s an API design problem, not a language problem. It’s possible to build friendly C APIs with a bit of care and thinking, especially when C99’s designated initialization can be used (C++ should really make sure that the full C99 language can be used from inside C++ instead of continuing to wander off into an entirely different direction). ###Configuring OpenBGPD to announce VM’s virtual networks We use BGP quite heavily at work, and even though I’m not interacting with that directly, it feels like it’s something very useful to learn at least on some basic level. The most effective and fun way of learning technology is finding some practical application, so I decided to see if it could help to improve networking management for my Virtual Machines. My setup is fairly simple: I have a host that runs bhyve VMs and I have a desktop system from where I ssh to VMs, both hosts run FreeBSD. All VMs are connected to each other through a bridge and have a common network 10.0.1/24. The point of this exercise is to be able to ssh to these VMs from desktop without adding static routes and without adding vmhost’s external interfaces to the VMs bridge. I’ve installed openbgpd on both hosts and configured it like this: vmhost: /usr/local/etc/bgpd.conf AS 65002 router-id 192.168.87.48 fib-update no network 10.0.1.1/24 neighbor 192.168.87.41 { descr "desktop" remote-as 65001 } Here, router-id is set vmhost’s IP address in my home network (192.168.87/24), fib-update no is set to forbid routing table update, which I initially set for testing, but keeping it as vmhost is not supposed to learn new routes from desktop anyway. network announces my VMs network and neighbor describes my desktop box. Now the desktop box: desktop: /usr/local/etc/bgpd.conf AS 65001 router-id 192.168.87.41 fib-update yes neighbor 192.168.87.48 { descr "vmhost" remote-as 65002 } It’s pretty similar to vmhost’s bgpd.conf, but no networks are announced here, and fib-update is set to yes because the whole point is to get VM routes added. Both hosts have to have the openbgpd service enabled: /etc/rc.conf.local openbgpdenable="YES" Conclusion As mentioned already, similar result could be achieved without using BGP by using either static routes or bridging interfaces differently, but the purpose of this exercise is to get some basic hands-on experience with BGP. Right now I’m looking into extending my setup in order to try more complex BGP schema. I’m thinking about adding some software switches in front of my VMs or maybe adding a second VM host (if budget allows). You’re welcome to comment if you have some ideas how to extend this setup for educational purposes in the context of BGP and networking. As a side note, I really like openbgpd so far. Its configuration file format is clean and simple, documentation is good, error and information messages are clear, and CLI has intuitive syntax. Digital Ocean ###The Power to Serve All people within the IT Industry should known where the slogan “The Power To Serve” is exposed every day to millions of people. But maybe too much wishful thinking from me. But without “The Power To Serve” the IT industry today will look totally different. Companies like Apple, Juniper, Cisco and even WatsApp would not exist in their current form. I provide IT architecture services to make your complex IT landscape manageable and I love to solve complex security and privacy challenges. Complex challenges where people, processes and systems are heavily interrelated. For this knowledge intensive work I often run some IT experiments. When you run experiments nowadays you have a choice: Rent some cloud based services or DIY (Do IT Yourself) on premise Running your own developments experiments on your own infrastructure can be time consuming. However smart automation saves time and money. And by creating your own CICD pipeline (Continuous Integration, Continuous Deployment) you stay on top of core infrastructure developments. Even hands-on. Knowing how things work from a technical ‘hands-on’ perspective gives great advantages when it comes to solving complex business IT problems. Making a clear distinguish between a business problem or IT problem is useless. Business and IT problems are related. Sometimes causal related, but more often indirect by one or more non linear feedback loops. Almost every business depends of IT systems. Bad IT means often that your customers will leave your business. One of the things of FeeBSD for me is still FreeBSD Jails. In 2015 I had luck to attend to a presentation of the legendary hacker Poul-Henning Kamp . Check his BSD bio to see what he has done for the FreeBSD community! FreeBSD jails are a light way to visualize your system without enormous overhead. Now that the development on Linux for LXD/LXD is more mature (lxd is the next generation system container manager on linux) there is finally again an alternative for a nice chroot Linux based system again. At least when you do not need the overhead and management complexity that comes with Kubernetes or Docker. FreeBSD means control and quality for me. When there is an open source package I need, I want to install it from source. It gives me more control and always some extra knowledge on how things work. So no precompiled binaries for me on my BSD systems! If a build on FreeBSD fails most of the time this is an alert regarding the quality for me. If a complex OSS package is not available at all in the FreeBSD ports collection there should be a reason for it. Is it really that nobody on the world wants to do this dirty maintenance work? Or is there another cause that running this software on FreeBSD is not possible…There are currently 32644 ports available on FreeBSD. So all the major programming language, databases and middleware libraries are present. The FreeBSD organization is a mature organization and since this is one of the largest OSS projects worldwide learning how this community manages to keep innovation and creates and maintains software is a good entrance for learning how complex IT systems function. FreeBSD is of course BSD licensed. It worked well! There is still a strong community with lots of strong commercial sponsors around the community. Of course: sometimes a GPL license makes more sense. So beside FreeBSD I also love GPL software and the rationale and principles behind it. So my hope is that maybe within the next 25 years the hard battle between BSD vs GPL churches will be more rationalized and normalized. Principles are good, but as all good IT architects know: With good principles alone you never make a good system. So use requirements and not only principles to figure out what OSS license fits your project. There is never one size fits all. June 19, 1993 was the day the official name for FreeBSD was agreed upon. So this blog is written to celebrate 25th anniversary of FreeBSD. ###Dave’s BSDCan trip report So far, only one person has bothered to send in a BSDCan trip report. Our warmest thanks to Dave for doing his part. Hello guys! During the last show, you asked for a trip report regarding BSDCan 2018. This was my first time attending BSDCan. However, BSDCan was my second BSD conference overall, my first being vBSDCon 2017 in Reston, VA. Arriving early Thursday evening and after checking into the hotel, I headed straight to the Red Lion for the registration, picked up my badge and swag and then headed towards the ‘DMS’ building for the newbies talk. The only thing is, I couldn’t find the DMS building! Fortunately I found a BSDCan veteran who was heading there themselves. My only suggestion is to include the full building name and address on the BSDCan web site, or even a link to Google maps to help out with the navigation. The on-campus street maps didn’t have ‘DMS’ written on them anywhere. But I digress. Once I made it to the newbies talk hosted by Dan Langille and Michael W Lucas, it highlighted places to meet, an overview of what is happening, details about the ‘BSDCan widow/widower tours’ and most importantly, the 6-2-1 rule! The following morning, we were present with tea/coffee, muffins and other goodies to help prepare us for the day ahead. The first talk, “The Tragedy of systemd” covered what systemd did wrong and how the BSD community could improve on the ideas behind it. With the exception of Michael W Lucas, SSH Key Management and Kirk McKusick, The Evolution of FreeBSD Governance talk, I pretty much attended all of the ZFS talks including the lunchtime BoF session, hosted by Allan Jude. Coming from FreeNAS and being involved in the community, this is where my main interest and motivation lies. Since then I have been able to share some of that information with the FreeNAS community forums and chatroom. I also attended the “Speculating about Intel” lunchtime BoF session hosted by Theo de Raddt, which proved to be “interesting”. The talks ended with the wrap up session with a few words from Dan, covering the record attendance and made very clear there “was no cabal”. Followed by the the handing over of Groff the BSD goat to a new owner, thank you’s from the FreeBSD Foundation to various community committers and maintainers, finally ending with the charity auction, where a things like a Canadian $20 bill sold for $40, a signed FreeBSD Foundation shirt originally worn by George Neville-Neil, a lost laptop charger, Michael’s used gelato spoon, various books, the last cookie and more importantly, the second to last cookie! After the auction, we all headed to the Red Lion for food and drinks, sponsored by iXsystems. I would like to thank the BSDCan organizers, speakers and sponsors for a great conference. I will certainly hope to attend next year! Regards, Dave (aka m0nkey) Thanks to Dave for sharing his experiences with us and our viewers ##Beastie Bits Robert Watson (from 2008) on how much FreeBSD is in Mac OS X Why Intel Skylake CPUs are sometimes 50% slower than older CPUs Kristaps Dzonsons is looking for somebody to maintain this as mentioned at this link camcontrol(8) saves the day again! Formatting floppy disks in a USB floppy disk drive 32+ great indie games now playable on OpenBSD -current; 7 currently on sale! Warsaw BSD User Group. June 27 2018 18:30-21:00, Wheel Systems Office, Aleje Jerozolimskie 178, Warsaw Tarsnap ##Feedback/Questions Ron - Adding a disk to ZFS Marshall - zfs question Thomas - Allan, the myth perpetuator Ross - ZFS IO stats per dataset Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
DragonflyBSD’s hammer1 encrypted master/slave setup, second part of our BSDCan recap, NomadBSD 1.1-RC1 available, OpenBSD adds an LDAP client to base, FreeBSD gets pNFS support, Intel FPU Speculation Vulnerability confirmed, and what some Unix command names mean. ##Headlines DragonflyBSD: Towards a HAMMER1 master/slave encrypted setup with LUKS I just wanted to share my experience with setting up DragonFly master/slave HAMMER1 PFS’s on top of LUKS So after a long time using an Synology for my NFS needs, I decided it was time to rethink my setup a little since I had several issues with it : You cannot run NFS on top of encrypted partitions easily I suspect I am having some some data corruption (bitrot) on the ext4 filesystem the NIC was stcuk to 100 Mbps instead of 1 Gbps even after swapping cables, switches, you name it It’s proprietary I have been playing with DragonFly in the past and knew about HAMMER, now I just had the perfect excuse to actually use it in production :) After setting up the OS, creating the LUKS partition and HAMMER FS was easy : kdload dm cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox newfs_hammer -L hammer1_secure_master /dev/mapper/fort_knox cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox_slave newfs_hammer -L hammer1_secure_slave /dev/mapper/fort_knox_slave Mount the 2 drives : mount /dev/mapper/fort_knox /fort_knox mount /dev/mapper_fort_know_slave /fort_knox_slave You can now put your data under /fort_knox Now, off to setting up the replication, first get the shared-uuid of /fort_knox hammer pfs-status /fort_knox Create a PFS slave “linked” to the master hammer pfs-slave /fort_knox_slave/pfs/slave shared-uuid=f9e7cc0d-eb59-10e3-a5b5-01e6e7cefc12 And then stream your data to the slave PFS ! hammer mirror-stream /fort_knox /fort_knox_slave/pfs/slave After that, setting NFS is fairly trivial even though I had problem with the /etc/exports syntax which is different than Linux There’s a few things I wish would be better though but nothing too problematic or without workarounds : Cannot unlock LUKS partitions at boot time afaik (Acceptable tradeoff for the added security LUKS gives me vs my old Synology setup) but this force me to run a script to unlock LUKS, mount hammer and start mirror-stream at each boot No S1/S3 sleep so I made a script to shutdown the system when there’s no network neighborgs to serve the NFS As my system isn’t online 24/7 for energy reasons, I guess will have to run hammer cleanup myself from time to time Some uncertainty because hey, it’s kind of exotic but exciting too :) Overall, I am happy, HAMMER1 and PFS are looking really good, DragonFly is a neat Unix and the community is super friendly (Matthew Dillon actually provided me with a kernel patch to fix the broken ACPI on the PC holding this setup, many thanks!), the system is still a “work in progress” but it is already serving my files as I write this post. Let’s see in 6 months how it goes in the longer run ! Helpful resources : https://www.dragonflybsd.org/docs/how_to_implement_hammer_pseudo_file_system__40___pfs___41___slave_mirroring_from_pfs_master/ ###BSDCan 2018 Recap As promised, here is our second part of our BSDCan report, covering the conference proper. The last tutorials/devsummit of that day lead directly into the conference, as people could pick up their registration packs at the Red Lion and have a drink with fellow BSD folks. Allan and I were there only briefly, as we wanted to get back to the “Newcomers orientation and mentorship” session lead by Michael W. Lucas. This session is intended for people that are new to BSDCan (maybe their first BSD conference ever?) and may have questions. Michael explained everything from the 6-2-1 rule (hours of sleep, meals per day, and number of showers that attendees should have at a minimum), to the partner and widowers program (lead by his wife Liz), to the sessions that people should not miss (opening, closing, and hallway track). Old-time BSDCan folks were asked to stand up so that people can recognize them and ask them any questions they might have during the conferences. The session was well attended. Afterwards, people went for dinner in groups, a big one lead by Michael Lucas to his favorite Shawarma place, followed by gelato (of course). This allowed newbies to mingle over dinner and ice cream, creating a welcoming atmosphere. The next day, after Dan Langille opened the conference, Benno Rice gave the keynote presentation about “The Tragedy of Systemd”. Benedict went to the following talks: “Automating Network Infrastructures with Ansible on FreeBSD” in the DevSummit track. A good talk that connected well with his Ansible tutorial and even allowed some discussions among participants. “All along the dwatch tower”: Devin delivered a well prepared talk. I first thought that the number of slides would not fit into the time slot, but she even managed to give a demo of her work, which was well received. The dwatch tool she wrote should make it easy for people to get started with DTrace without learning too much about the syntax at first. The visualizations were certainly nice to see, combining different tools together in a new way. ZFS BoF, lead by Allan and Matthew Ahrens SSH Key Management by Michael W. Lucas. Yet another great talk where I learned a lot. I did not get to the SSH CA chapter in the new SSH Mastery book, so this was a good way to wet my appetite for it and motivated me to look into creating one for the cluster that I’m managing. The rest of the day was spent at the FreeBSD Foundation table, talking to various folks. Then, Allan and I had an interview with Kirk McKusick for National FreeBSD Day, then we had a core meeting, followed by a core dinner. Day 2: “Flexible Disk Use in OpenZFS”: Matthew Ahrens talking about the feature he is implementing to expand a RAID-Z with a single disk, as well as device removal. Allan’s talk about his efforts to implement ZSTD in OpenZFS as another compression algorithm. I liked his overview slides with the numbers comparing the algorithms for their effectiveness and his personal story about the sometimes rocky road to get the feature implemented. “zrepl - ZFS replication” by Christian Schwarz, was well prepared and even had a demo to show what his snapshot replication tool can do. We covered it on the show before and people can find it under sysutils/zrepl. Feedback and help is welcome. “The Evolution of FreeBSD Governance” by Kirk McKusick was yet another great talk by him covering the early days of FreeBSD until today, detailing some of the progress and challenges the project faced over the years in terms of leadership and governance. This is an ongoing process that everyone in the community should participate in to keep the project healthy and infused with fresh blood. Closing session and auction were funny and great as always. All in all, yet another amazing BSDCan. Thank you Dan Langille and your organizing team for making it happen! Well done. Digital Ocean ###NomadBSD 1.1-RC1 Released The first – and hopefully final – release candidate of NomadBSD 1.1 is available! Changes The base system has been upgraded to FreeBSD 11.2-RC3 EFI booting has been fixed. Support for modern Intel GPUs has been added. Support for installing packages has been added. Improved setup menu. More software packages: benchmarks/bonnie++ DSBDisplaySettings DSBExec DSBSu mail/thunderbird net/mosh ports-mgmt/octopkg print/qpdfview security/nmap sysutils/ddrescue sysutils/fusefs-hfsfuse sysutils/fusefs-sshfs sysutils/sleuthkit www/lynx x11-wm/compton x11/xev x11/xterm Many improvements and bugfixes The image and instructions can be found here. ##News Roundup LDAP client added to -current CVSROOT: /cvs Module name: src Changes by: reyk@cvs.openbsd.org 2018/06/13 09:45:58 Log message: Import ldap(1), a simple ldap search client. We have an ldapd(8) server and ypldap in base, so it makes sense to have a simple LDAP client without depending on the OpenLDAP package. This tool can be used in an ssh(1) AuthorizedKeysCommand script. With feedback from many including millert@ schwarze@ gilles@ dlg@ jsing@ OK deraadt@ Status: Vendor Tag: reyk Release Tags: ldap_20180613 N src/usr.bin/ldap/Makefile N src/usr.bin/ldap/aldap.c N src/usr.bin/ldap/aldap.h N src/usr.bin/ldap/ber.c N src/usr.bin/ldap/ber.h N src/usr.bin/ldap/ldap.1 N src/usr.bin/ldap/ldapclient.c N src/usr.bin/ldap/log.c N src/usr.bin/ldap/log.h No conflicts created by this import ###Intel® FPU Speculation Vulnerability Confirmed Earlier this month, Philip Guenther (guenther@) committed (to amd64 -current) a change from lazy to semi-eager FPU switching to mitigate against rumored FPU state leakage in Intel® CPUs. Theo de Raadt (deraadt@) discussed this in his BSDCan 2018 session. Using information disclosed in Theo’s talk, Colin Percival developed a proof-of-concept exploit in around 5 hours. This seems to have prompted an early end to an embargo (in which OpenBSD was not involved), and the official announcement of the vulnerability. FPU change in FreeBSD Summary: System software may utilize the Lazy FP state restore technique to delay the restoring of state until an instruction operating on that state is actually executed by the new process. Systems using Intel® Core-based microprocessors may potentially allow a local process to infer data utilizing Lazy FP state restore from another process through a speculative execution side channel. Description: System software may opt to utilize Lazy FP state restore instead of eager save and restore of the state upon a context switch. Lazy restored states are potentially vulnerable to exploits where one process may infer register values of other processes through a speculative execution side channel that infers their value. · CVSS - 4.3 Medium CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:L/I:N/A:N Affected Products: Intel® Core-based microprocessors. Recommendations: If an XSAVE-enabled feature is disabled, then we recommend either its state component bitmap in the extended control register (XCR0) is set to 0 (e.g. XCR0[bit 2]=0 for AVX, XCR0[bits 7:5]=0 for AVX512) or the corresponding register states of the feature should be cleared prior to being disabled. Also for relevant states (e.g. x87, SSE, AVX, etc.), Intel recommends system software developers utilize Eager FP state restore in lieu of Lazy FP state restore. Acknowledgements: Intel would like to thank Julian Stecklina from Amazon Germany, Thomas Prescher from Cyberus Technology GmbH (https://www.cyberus-technology.de/), Zdenek Sojka from SYSGO AG (http://sysgo.com), and Colin Percival for reporting this issue and working with us on coordinated disclosure. iXsystems iX Ad Spot iX Systems - BSDCan 2018 Recap ###FreeBSD gets pNFS support Merge the pNFS server code from projects/pnfs-planb-server into head. This code merge adds a pNFS service to the NFSv4.1 server. Although it is a large commit it should not affect behaviour for a non-pNFS NFS server. Some documentation on how this works can be found at: Merge the pN http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt and will hopefully be turned into a proper document soon. This is a merge of the kernel code. Userland and man page changes will come soon, once the dust settles on this merge. It has passed a "make universe", so I hope it will not cause build problems. It also adds NFSv4.1 server support for the "current stateid". Here is a brief overview of the pNFS service: A pNFS service separates the Read/Write operations from all the other NFSv4.1 Metadata operations. It is hoped that this separation allows a pNFS service to be configured that exceeds the limits of a single NFS server for either storage capacity and/or I/O bandwidth. It is possible to configure mirroring within the data servers (DSs) so that the data storage file for an MDS file will be mirrored on two or more of the DSs. When this is used, failure of a DS will not stop the pNFS service and a failed DS can be recovered once repaired while the pNFS service continues to operate. Although two way mirroring would be the norm, it is possible to set a mirroring level of up to four or the number of DSs, whichever is less. The Metadata server will always be a single point of failure, just as a single NFS server is. A Plan B pNFS service consists of a single MetaData Server (MDS) and K Data Servers (DS), all of which are recent FreeBSD systems. Clients will mount the MDS as they would a single NFS server. When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size 0. Each of these files will also have two extended attributes in the system attribute name space: pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file(s) on DS(s) for this file. pnfsd.dsattr - This extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file, so that the MDS doesn't need to acquire the attributes from the DS for every Getattr operation. For each regular (VREG) file, the MDS creates a data storage file on one (or more if mirroring is enabled) of the DSs in one of the "dsNN" subdirectories. The name of this file is the file handle of the file on the MDS in hexadecimal so that the name is unique. The DSs use subdirectories named "ds0" to "dsN" so that no one directory gets too large. The value of "N" is set via the sysctl vfs.nfsd.dsdirsize on the MDS, with the default being 20. For production servers that will store a lot of files, this value should probably be much larger. It can be increased when the "nfsd" daemon is not running on the MDS, once the "dsK" directories are created. For pNFS aware NFSv4.1 clients, the FreeBSD server will return two pieces of information to the client that allows it to do I/O directly to the DS. DeviceInfo - This is relatively static information that defines what a DS is. The critical bits of information returned by the FreeBSD server is the IP address of the DS and, for the Flexible File layout, that NFSv4.1 is to be used and that it is "tightly coupled". There is a "deviceid" which identifies the DeviceInfo. Layout - This is per file and can be recalled by the server when it is no longer valid. For the FreeBSD server, there is support for two types of layout, call File and Flexible File layout. Both allow the client to do I/O on the DS via NFSv4.1 I/O operations. The Flexible File layout is a more recent variant that allows specification of mirrors, where the client is expected to do writes to all mirrors to maintain them in a consistent state. The Flexible File layout also allows the client to report I/O errors for a DS back to the MDS. The Flexible File layout supports two variants referred to as "tightly coupled" vs "loosely coupled". The FreeBSD server always uses the "tightly coupled" variant where the client uses the same credentials to do I/O on the DS as it would on the MDS. For the "loosely coupled" variant, the layout specifies a synthetic user/group that the client uses to do I/O on the DS. The FreeBSD server does not do striping and always returns layouts for the entire file. The critical information in a layout is Read vs Read/Writea and DeviceID(s) that identify which DS(s) the data is stored on. At this time, the MDS generates File Layout layouts to NFSv4.1 clients that know how to do pNFS for the non-mirrored DS case unless the sysctl vfs.nfsd.default_flexfile is set non-zero, in which case Flexible File layouts are generated. The mirrored DS configuration always generates Flexible File layouts. For NFS clients that do not support NFSv4.1 pNFS, all I/O operations are done against the MDS which acts as a proxy for the appropriate DS(s). When the MDS receives an I/O RPC, it will do the RPC on the DS as a proxy. If the DS is on the same machine, the MDS/DS will do the RPC on the DS as a proxy and so on, until the machine runs out of some resource, such as session slots or mbufs. As such, DSs must be separate systems from the MDS. *** ###[What does {some strange unix command name} stand for?](http://www.unixguide.net/unix/faq/1.3.shtml) + awk = "Aho Weinberger and Kernighan" + grep = "Global Regular Expression Print" + fgrep = "Fixed GREP". + egrep = "Extended GREP" + cat = "CATenate" + gecos = "General Electric Comprehensive Operating Supervisor" + nroff = "New ROFF" + troff = "Typesetter new ROFF" + tee = T + bss = "Block Started by Symbol + biff = "BIFF" + rc (as in ".cshrc" or "/etc/rc") = "RunCom" + Don Libes' book "Life with Unix" contains lots more of these tidbits. *** ##Beastie Bits + [RetroBSD: Unix for microcontrollers](http://retrobsd.org/wiki/doku.php) + [On the matter of OpenBSD breaking embargos (KRACK)](https://marc.info/?l=openbsd-tech&m=152910536208954&w=2) + [Theo's Basement Computer Paradise (1998)](https://zeus.theos.com/deraadt/hosts.html) + [Airport Extreme runs NetBSD](https://jcs.org/2018/06/12/airport_ssh) + [What UNIX shell could have been](https://rain-1.github.io/shell-2.html) *** Tarsnap ad *** ##Feedback/Questions + We need more feedback and questions. Please email feedback@bsdnow.tv + Also, many of you owe us BSDCan trip reports! We have shared what our experience at BSDCan was like, but we want to hear about yours. What can we do better next year? What was it like being there for the first time? + [Jason writes in](https://slexy.org/view/s205jU58X2) + https://www.wheelsystems.com/en/products/wheel-fudo-psm/ + [June 19th was National FreeBSD Day](https://twitter.com/search?src=typd&q=%23FreeBSDDay) *** - Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [feedback@bsdnow.tv](mailto:feedback@bsdnow.tv) ***
TrueOS becoming a downstream fork with Trident, our BSDCan 2018 recap, HardenedBSD Foundation founding efforts, VPN with OpenIKED on OpenBSD, FreeBSD on a System76 Galago Pro, and hardware accelerated crypto on Octeons. ##Headlines## TrueOS to Focus on Core Operating System The TrueOS Project has some big plans in the works, and we want to take a minute and share them with you. Many have come to know TrueOS as the “graphical FreeBSD” that makes things easy for newcomers to the BSDs. Today we’re announcing that TrueOS is shifting our focus a bit to become a cutting-edge operating system that keeps all of the stability that you know and love from ZFS (OpenZFS) and FreeBSD, and adds additional features to create a fresh, innovative operating system. Our goal is to create a core-centric operating system that is modular, functional, and perfect for do-it-yourselfers and advanced users alike. TrueOS will become a downstream fork that will build on FreeBSD by integrating new software technologies like OpenRC and LibreSSL. Work has already begun which allows TrueOS to be used as a base platform for other projects, including JSON-based manifests, integrated Poudriere / pkg tools and much more. We’re planning on a six month release cycle to keep development moving and fresh, allowing us to bring you hot new features to ZFS, bhyve and related tools in a timely manner. This makes TrueOS the perfect fit to serve as the basis for building other distributions. Some of you are probably asking yourselves “But what if I want to have a graphical desktop?” Don’t worry! We’re making sure that everyone who knows and loves the legacy desktop version of TrueOS will be able to continue using a FreeBSD-based, graphical operating system in the future. For instance, if you want to add KDE, just use sudo pkg install kde and voila! You have your new shiny desktop. Easy right? This allows us to get back to our roots of being a desktop agnostic operating system. If you want to add a new desktop environment, you get to pick the one that best suits your use. We know that some of you will still be looking for an out-of-the-box solution similar to legacy PC-BSD and TrueOS. We’re happy to announce that Project Trident will take over graphical FreeBSD development going forward. Not much is going to change in that regard other than a new name! You’ll still have Lumina Desktop as a lightweight and feature-rich desktop environment and tons of utilities from the legacy TrueOS toolchain like sysadm and AppCafe. There will be migration paths available for those that would like to move to other FreeBSD-based distributions like Project Trident or GhostBSD. We look forward to this new chapter for TrueOS and hope you will give the new edition a spin! Tell us what you think about the new changes by leaving us a comment. Don’t forget you can ask us questions on our Twitter and be a part of our community by joining the new TrueOS Forums when they go live in about a week. Thanks for being a loyal fan of TrueOS. ###Project Trident FAQ Q: Why did you pick the name “Project Trident”? A: We were looking for a name that was unique, yet would still relate to the BSD community. Since Beastie (the FreeBSD mascot) is always pictured with a trident, it felt like that would be a great name. Q: Where can users go for technical support? A: At the moment, Project Trident will continue sharing the TrueOS community forums and Telegram channels. We are currently evaluating dedicated options for support channels in the future. Q: Can I help contribute to the project? A: We are always looking for developers who want to join the project. If you’re not a developer you can still help, as a community project we will be more reliant on contributions from the community in the form of how-to guides and other user-centric documentation and support systems. Q: How is the project supported financially? A: Project Trident is sponsored by the community, from both individuals and corporations. iXsystems has stepped up as the first enterprise-level sponsor of the project, and has been instrumental in getting Project Trident up and running. Please visit the Sponsors page to see all the current sponsors. Q: How can I help support the project financially? A: Several methods exist, from one time or recurring donations via Paypal to limited time swag t-shirt campaigns during the year. We are also looking into more alternative methods of support, so please visit the Sponsors page to see all the current methods of sponsorship. Q: Will there be any transparency of the financial donations and expenditures? A: Yes, we will be totally open with how much money comes into the project and what it is spent on. Due to concerns of privacy, we will not identify individuals and their donation amounts unless they specifically request to be identified. We will release a monthly overview in/out ledger, so that community members can see where their money is going. Relationship with TrueOS Project Trident does have very close ties to the TrueOS project, since most of the original Project Trident developers were once part of the TrueOS project before it became a distribution platform. For users of the TrueOS desktop, we have some additional questions and answers below. Q: Do we need to be at a certain TrueOS install level/release to upgrade? A: As long as you have a TrueOS system which has been updated to at least the 18.03 release you should be able to just perform a system update to be automatically upgraded to Project Trident. Q: Which members moved from TrueOS to Project Trident? A: Project Trident is being led by prior members of the TrueOS desktop team. Ken and JT (development), Tim (documentation) and Rod (Community/Support). Since Project Trident is a community-first project, we look forward to working with new members of the team. iXsystems ###BSDCan BSDCan finished Saturday last week It started with the GoatBoF on Tuesday at the Royal Oak Pub, where people had a chance to meet and greet. Benedict could not attend due to an all-day FreeBSD Foundation meeting and and even FreeBSD Journal Editorial Board meeting. The FreeBSD devsummit was held the next two days in parallel to the tutorials. Gordon Tetlow, who organized the devsummit, opened the devsummit. Deb Goodkin from the FreeBSD Foundation gave the first talk with a Foundation update, highlighting current and future efforts. Li-Wen Hsu is now employed by the Foundation to assist in QA work (Jenkins, CI/CD) and Gordon Tetlow has a part-time contract to help secteam as their secretary. Next, the FreeBSD core team (among them Allan and Benedict) gave a talk about what has happened this last term. With a core election currently running, some of these items will carry over to the next core team, but there were also some finished ones like the FCP process and FreeBSD members initiative. People in the audience asked questions on various topics of interest. After the coffee break, the release engineering team gave a talk about their efforts in terms of making releases happen in time and good quality. Benedict had to give his Ansible tutorial in the afternoon, which had roughly 15 people attending. Most of them beginners, we could get some good discussions going and I also learned a few new tricks. The overall feedback was positive and one even asked what I’m going to teach next year. The second day of the FreeBSD devsummit began with Gordon Tetlow giving an insight into the FreeBSD Security team (aka secteam). He gave a overview of secteam members and responsibilities, explaining the process based on a long past advisory. Developers were encouraged to help out secteam. NDAs and proper disclosure of vulnerabilities were also discussed, and the audience had some feedback and questions. When the coffee break was over, the FreeBSD 12.0 planning session happened. A Google doc served as a collaborative way of gathering features and things left to do. People signed up for it or were volunteered. Some features won’t make it into 12.0 as they are not 100% ready for prime time and need a few more rounds of testing and bugfixing. Still, 12.0 will have some compelling features. A 360° group picture was taken after lunch, and then people split up into the working groups for the afternoon or started hacking in the UofO Henderson residence. Benedict and Allan both attended the OpenZFS working group, lead by Matt Ahrens. He presented the completed and outstanding work in FreeBSD, without spoiling too much of the ZFS presentations of various people that happened later at the conference. Benedict joined the boot code session a bit late (hallway track is the reason) when most things seem to have already been discussed. BSDCan 2018 — Ottawa (In Pictures) iXsystems Photos from BSDCan 2018 ##News Roundup June HardenedBSD Foundation Update We at HardenedBSD are working towards starting up a 501©(3) not-for-profit organization in the USA. Setting up this organization will allow future donations to be tax deductible. We’ve made progress and would like to share with you the current state of affairs. We have identified, sent invitations out, and received acceptance letters from six people who will serve on the HardenedBSD Foundation Board of Directors. You can find their bios below. In the latter half of June 2018 or the beginning half of July 2018, we will meet for the first time as a board and formally begin the process of creating the documentation needed to submit to the local, state, and federal tax services. Here’s a brief introduction to those who will serve on the board: W. Dean Freeman (Advisor): Dean has ten years of professional experience with deploying and security Unix and networking systems, including assessing systems security for government certification and assessing the efficacy of security products. He was introduced to Unix via FreeBSD 2.2.8 on an ISP shell account as a teenager. Formerly, he was the Snort port maintainer for FreeBSD while working in the Sourcefire VRT, and has contributed entropy-related patches to the FreeBSD and HardenedBSD projects – a topic on which he presented at vBSDCon 2017. Ben La Monica (Advisor): Ben is a Senior Technology Manager of Software Engineering at Morningstar, Inc and has been developing software for over 15 years in a variety of languages. He advocates open source software and enjoys tinkering with electronics and home automation. George Saylor (Advisor): George is a Technical Directory at G2, Inc. Mr. Saylor has over 28 years of information systems and security experience in a broad range of disciplines. His core focus areas are automation and standards in the event correlation space as well as penetration and exploitation of computer systems. Mr Saylor was also a co-founder of the OpenSCAP project. Virginia Suydan (Accountant and general administrator): Accountant and general administrator for the HardenedBSD Foundation. She has worked with Shawn Webb for tax and accounting purposes for over six years. Shawn Webb (Director): Co-founder of HardenedBSD and all-around infosec wonk. He has worked and played in the infosec industry, doing both offensive and defensive research, for around fifteen years. He loves open source technologies and likes to frustrate the bad guys. Ben Welch (Advisor): Ben is currently a Security Engineer at G2, Inc. He graduated from Pennsylvania College of Technology with a Bachelors in Information Assurance and Security. Ben likes long walks, beaches, candlelight dinners, and attending various conferences like BSides and ShmooCon. ###Your own VPN with OpenIKED & OpenBSD Remote connectivity to your home network is something I think a lot of people find desirable. Over the years, I’ve just established an SSH tunnel and use it as a SOCKS proxy, sending my traffic through that. It’s a nice solution for a “poor man’s VPN”, but it can be a bit clunky, and it’s not great having to expose SSH to the world, even if you make sure to lock everything down I set out the other day to finally do it properly. I’d come across this great post by Gordon Turner: OpenBSD 6.2 VPN Endpoint for iOS and macOS Whilst it was exactly what I was looking for, it outlined how to set up an L2TP VPN. Really, I wanted IKEv2 for performance and security reasons (I won’t elaborate on this here, if you’re curious about the differences, there’s a lot of content out on the web explaining this). The client systems I’d be using have native support for IKEv2 (iOS, macOS, other BSD systems). But, I couldn’t find any tutorials in the same vein. So, let’s get stuck in! A quick note ✍️ This guide will walk through the set up of an IKEv2 VPN using OpenIKED on OpenBSD. It will detail a “road warrior” configuration, and use a PSK (pre-shared-key) for authentication. I’m sure it can be easily adapted to work on any other platforms that OpenIKED is available on, but keep in mind my steps are specifically for OpenBSD. Server Configuration As with all my home infrastructure, I crafted this set-up declaratively. So, I had the deployment of the VM setup in Terraform (deployed on my private Triton cluster), and wrote the configuration in Ansible, then tied them together using radekg/terraform-provisioner-ansible. One of the reasons I love Ansible is that its syntax is very simplistic, yet expressive. As such, I feel it fits very well into explaining these steps with snippets of the playbook I wrote. I’ll link the full playbook a bit further down for those interested. See the full article for the information on: sysctl parameters The naughty list (optional) Configure the VPN network interface Configure the firewall Configure the iked service Gateway configuration Client configuration Troubleshooting DigitalOcean ###FreeBSD on a System76 Galago Pro Hey all, It’s been a while since I last posted but I thought I would hammer something out here. My most recent purchase was a System76 Galago Pro. I thought, afer playing with POP! OS a bit, is there any reason I couldn’t get BSD on this thing. Turns out the answer is no, no there isnt and it works pretty decently. To get some accounting stuff out of the way I tested this all on FreeBSD Head and 11.1, and all of it is valid as of May 10, 2018. Head is a fast moving target so some of this is only bound to improve. The hardware Intel Core i5 Gen 8 UHD Graphics 620 16 GB DDR4 Ram RTL8411B PCI Express Card Reader RTL8111 Gigabit ethernet controller Intel HD Audio Samsung SSD 960 PRO 512GB NVMe The caveats There are a few things that I cant seem to make work straight out of the box, and that is the SD Card reader, the backlight, and the audio is a bit finicky. Also the trackpad doesn’t respond to two finger scrolling. The wiki is mostly up to date, there are a few edits that need to be made still but there is a bug where I cant register an account yet so I haven’t made all the changes. Processor It works like any other Intel processor. Pstates and throttling work. Graphics The boot menu sets itself to what looks like 1024x768, but works as you expect in a tiny window. The text console does the full 3200x1800 resolution, but the text is ultra tiny. There isnt a font for the console that covers hidpi screens yet. As for X Windows it requres the drm-kmod-next package. Once installed follow the directions from the package and it works with almost no fuss. I have it running on X with full intel acceleration, but it is running at it’s full 3200x1800 resolution, to scale that down just do xrandr --output eDP-1 --scale 0.5x0.5 it will blow it up to roughly 200%. Due to limitations with X windows and hidpi it is harder to get more granular. Intel Wireless 8265 The wireless uses the iwm module, as of right now it does not seem to automagically load right now. Adding iwm_load=“YES” will cause the module to load on boot and kldload iwm Battery I seem to be getting about 5 hours out of the battery, but everything reports out of the box as expected. I could get more by throttling the CPU down speed wise. Overall impression It is a pretty decent experience. While not as polished as a Thinkpad there is a lot of potential with a bit of work and polishing. The laptop itself is not bad, the keyboard is responsive. The build quality is pretty solid. My only real complaint is the trackpad is stiff to click and sort of tiny. They seem to be a bit indifferent to non linux OSes running on the gear but that isnt anything new. I wont have any problems using it and is enough that when I work through this laptop, but I’m not sure at this stage if my next machine will be a System76 laptop, but they have impressed me enough to put them in the running when I go to look for my next portable machine but it hasn’t yet replaced the hole left in my heart by lenovo messing with the thinkpad. ###Hardware accelerated AES/HMAC-SHA on octeons In this commit, visa@ submitted code (disabled for now) to use built-in acceleration on octeon CPUs, much like AESNI for x86s. I decided to test tcpbench(1) and IPsec, before and after updating and enabling the octcrypto(4) driver. I didn't capture detailed perf stats from before the update, I had heard someone say that Edgerouter Lite boxes would only do some 6MBit/s over ipsec, so I set up a really simple ipsec.conf with ike esp from A to B leading to a policy of esp tunnel from A to B spi 0xdeadbeef auth hmac-sha2-256 enc aes going from one ERL to another (I collect octeons, so I have a bunch to test with) and let tcpbench run for a while on it. My numbers hovered around 7Mbit/s, which coincided with what I've heard, and also that most of the CPU gets used while doing it. Then I edited /sys/arch/octeon/conf/GENERIC, removed the # from octcrypto0 at mainbus0 and recompiled. Booted into the new kernel and got a octcrypto0 line in dmesg, and it was time to rock the ipsec tunnel again. The crypto algorithm and HMAC used by default on ipsec coincides nicely with the list of accelerated functions provided by the driver. Before we get to tunnel traffic numbers, just one quick look at what systat pigs says while the ipsec is running at full steam: PID USER NAME CPU 20 40 60 80 100 58917 root crypto 52.25 ################# 42636 root softnet 42.48 ############## (idle) 29.74 ######### 1059 root tcpbench 24.22 ####### 67777 root crynlk 19.58 ###### So this indicates that the load from doing ipsec and generating the traffic is somewhat nicely evened out over the two cores in the Edgerouter, and there's even some CPU left unused, which means I can actually ssh into it and have it usable. I have had it running for almost 2 days now, moving some 2.1TB over the tunnel. Now for the new and improved performance numbers: 204452123 4740752 37.402 100.00% Conn: 1 Mbps: 37.402 Peak Mbps: 58.870 Avg Mbps: 37.402 204453149 4692968 36.628 100.00% Conn: 1 Mbps: 36.628 Peak Mbps: 58.870 Avg Mbps: 36.628 204454167 5405552 42.480 100.00% Conn: 1 Mbps: 42.480 Peak Mbps: 58.870 Avg Mbps: 42.480 204455188 5202496 40.804 100.00% Conn: 1 Mbps: 40.804 Peak Mbps: 58.870 Avg Mbps: 40.804 204456194 5062208 40.256 100.00% Conn: 1 Mbps: 40.256 Peak Mbps: 58.870 Avg Mbps: 40.256 The tcpbench numbers fluctuate up and down a bit, but the output is nice enough to actually keep tabs on the peak values. Peaking to 58.8MBit/s! Of course, as you can see, the average is lower but nice anyhow. A manyfold increase in performance, which is good enough in itself, but also moves the throughput from a speed that would make a poor but cheap gateway to something actually useful and decent for many home network speeds. Biggest problem after this gets enabled will be that my options to buy cheap used ERLs diminish. ##Beastie Bits Using FreeBSD Text Dumps llvm’s lld now the default linker for amd64 on FreeBSD Author Discoverability Pledge and Unveil in OpenBSD {pdf} EuroBSDCon 2018 CFP Closes June 17, hurry up and get your submissions in Just want to attend, but need help getting to the conference? Applications for the Paul Schenkeveld travel grant accepted until June 15th Tarsnap ##Feedback/Questions Casey - ZFS on Digital Ocean Jürgen - A Question Kevin - Failover best practice Dennis - SQL Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
FreeBSD internship learnings, exciting developments coming to FreeBSD, running FreeNAS on DigitalOcean, Network Manager control for OpenBSD, OpenZFS User Conference Videos are here and batch editing files with ed. Headlines What I learned during my FreeBSD intership Hi, my name is Mitchell Horne. I am a computer engineering student at the University of Waterloo, currently in my third year of studies, and fortunate to have been one of the FreeBSD Foundation’s co-op students this past term (January to April). During this time I worked under Ed Maste, in the Foundation’s small Kitchener office, along with another co-op student Arshan Khanifar. My term has now come to an end, and so I’d like to share a little bit about my experience as a newcomer to FreeBSD and open-source development. I’ll begin with some quick background — and a small admission of guilt. I have been an open-source user for a large part of my life. When I was a teenager I started playing around with Linux, which opened my eyes to the wider world of free software. Other than some small contributions to GNOME, my experience has been mostly as an end user; however, the value of these projects and the open-source philosophy was not lost on me, and is most of what motivated my interest in this position. Before beginning this term I had no personal experience with any of the BSDs, although I knew of their existence and was extremely excited to receive the position. I knew it would be a great opportunity for growth, but I must confess that my naivety about FreeBSD caused me to make the silent assumption that this would be a form of compromise — a stepping stone that would eventually allow me to work on open-source projects that are somehow “greater” or more “legitimate”. After four months spent immersed in this project I have learned how it operates, witnessed its community, and learned about its history. I am happy to admit that I was completely mistaken. Saying it now seems obvious, but FreeBSD is a project with its own distinct uses, goals, and identity. For many there may exist no greater opportunity than to work on FreeBSD full time, and with what I know now I would have a hard time coming up with a project that is more “legitimate”. What I Liked In all cases, the work I submitted this term was reviewed by no less than two people before being committed. The feedback and criticism I received was always both constructive and to the point, and it commented on everything from high-level ideas to small style issues. I appreciate having these thorough reviews in place, since I believe it ultimately encourages people to accept only their best work. It is indicative of the high quality that already exists within every aspect of this project, and this commitment to quality is something that should continue to be honored as a core value. As I’ve discovered in some of my previous work terms, it is all too easy cut corners in the name of a deadline or changing priorities, but the fact that FreeBSD doesn’t need to make these types of compromises is a testament to the power of free software. It’s a small thing, but the quality and completeness of the FreeBSD documentation was hugely helpful throughout my term. Everything you might need to know about utilities, library functions, the kernel, and more can be found in a man page; and the handbook is a great resource as both an introduction to the operating system and a reference. I only wish I had taken some time earlier in the term to explore the different documents more thoroughly, as they cover a wide range of interesting and useful topics. The effort people put into writing and maintaining FreeBSD’s documentation is easy to overlook, but its value cannot be overstated. What I Learned Although there was a lot I enjoyed, there were certainly many struggles I faced throughout the term, and lessons to be learned from them. I expect that some of issues I faced may be specific to FreeBSD, while others may be common to open-source projects in general. I don’t have enough experience to speculate on which is which, so I will leave this to the reader. The first lesson can be summed up simply: you have to advocate for your own work. FreeBSD is made up in large part by volunteer efforts, and in many cases there is more work to go around than people available to do it. A consequence of this is that there will not be anybody there to check up on you. Even in my position where I actually had a direct supervisor, Ed often had his plate full with so many other things that the responsibility to find someone to look at my work fell to me. Admittedly, a couple of smaller changes I worked on got left behind or stuck in review simply because there wasn’t a clear person/place to reach out to. I think this is both a barrier of entry to FreeBSD and a mental hurdle that I needed to get over. If there’s a change you want to see included or reviewed, then you may have to be the one to push for it, and there’s nothing wrong with that. Perhaps this process should be easier for newcomers or infrequent contributors (the disconnect between Bugzilla and Phabricator definitely leaves a lot to be desired), but we also have to be aware that this simply isn’t the reality right now. Getting your work looked at may require a little bit more self-motivation, but I’d argue that there are much worse problems a project like FreeBSD could have than this. I understand this a lot better now, but it is still something I struggle with. I’m not naturally the type of person who easily connects with others or asks for help, so I see this as an area for future growth rather than simply a struggle I encountered and overcame over the course of this work term. Certainly it is an important skill to understand the value of your own work, and equally important is the ability to communicate that value to others. I also learned the importance of starting small. My first week or two on the job mainly involved getting set up and comfortable with the workflow. After this initial stage, I began exploring the project and found myself overwhelmed by its scale. With so many possible areas to investigate, and so much work happening at once, I felt quite lost on where to begin. Many of the potential projects I found were too far beyond my experience level, and most small bugs were picked up and fixed quickly by more experienced contributors before I could even get to them. It’s easy to make the mistake that FreeBSD is made up solely of a few rock-star committers that do everything. This is how it appears at face-value, as reading through commits, bug reports, and mailing lists yields a few of the same names over and over. The reality is that just as important are the hundreds of users and infrequent contributors who take the time to submit bug reports, patches, or feedback. Even though there are some people who would fall under the umbrella of a rock-star committer, they didn’t get there overnight. Rather, they have built their skills and knowledge through many years of involvement in FreeBSD and similar projects. As a student coming into this project and having high expectations of myself, it was easy to set the bar too high by comparing myself against those big committers, and feel that my work was insignificant, inadequate, and simply too infrequent. In reality, there is no reason I should have felt this way. In a way, this comparison is disrespectful to those who have reached this level, as it took them a long time to get there, and it’s a humbling reminder that any skill worth learning requires time, patience, and dedication. It is easy to focus on an end product and simply wish to be there, but in order to be truly successful one must start small, and find satisfaction in the struggle of learning something new. I take pride in the many small successes I’ve had throughout my term here, and appreciate the fact that my journey into FreeBSD and open-source software is only just beginning. Closing Thoughts I would like to close with some brief thank-you’s. First, to everyone at the Foundation for being so helpful, and allowing this position to exist in the first place. I am extremely grateful to have been given this unique opportunity to learn about and give back to the open-source world. I’d also like to thank my office mates; Ed: for being an excellent mentor, who offered an endless wealth of knowledge and willingness to share it. My classmate and fellow intern Arshan: for giving me a sense of camaraderie and the comforting reminder that at many moments he was as lost as I was. Finally, a quick thanks to everyone else I crossed paths with who offered reviews and advice. I appreciate your help and look forward to working with you all further. I am walking away from this co-op with a much greater appreciation for this project, and have made it a goal to remain involved in some capacity. I feel that I’ve gained a little bit of a wider perspective on my place in the software world, something I never really got from my previous co-ops. Whether it ends up being just a stepping stone, or the beginning of much larger involvement, I thoroughly enjoyed my time here. Recent Developments in FreeBSD Support for encrypted, compressed (gzip and zstd), and network crash dumps enabled by default on most platforms Intel Microcode Splitter Intel Spec Store Bypass Disable control Raspberry Pi 3B+ Ethernet Driver IBRS for i386 Upcoming: Microcode updater for AMD CPUs the RACK TCP/IP stack, from Netflix Voting in the FreeBSD Core Election begins today: DigitalOcean Digital Ocean Promo Link for BSD Now Listeners Running FreeNAS on a DigitalOcean Droplet Need to backup your FreeNAS offsite? Run a locked down instance in the cloud, and replicate to it The tutorial walks though the steps of converting a fresh FreeBSD based droplet into a FreeNAS Create a droplet, and add a small secondary block-storage device Boot the droplet, login, and download FreeNAS Disable swap, enable ‘foot shooting’ mode in GEOM use dd to write the FreeNAS installer to the boot disk Reboot the droplet, and use the FreeNAS installer to install FreeNAS to the secondary block storage device Now, reimage the droplet with FreeBSD again, to replace the FreeNAS installer Boot, and dd FreeNAS from the secondary block storage device back to the boot disk You can now destroy the secondary block device Now you have a FreeNAS, and can take it from there. Use the FreeNAS replication wizard to configure sending snapshots from your home NAS to your cloud NAS Note: You might consider creating a new block storage device to create a larger pool, that you can more easily grow over time, rather than using the boot device in the droplet as your main pool. News Roundup Network Manager Control for OpenBSD (Updated) Generalities I just remind the scope of this small tool: allow you to pre-define several cable or wifi connections let nmctl to connect automatically to the first available one allow you to easily switch from one network connection to an other one create openbox dynamic menus Enhancements in this version This is my second development version: 0.2. I've added performed several changes in the code: code style cleanup, to better match the python recommendations adapt the tool to allow to connect to an Open-wifi having blancs in the name. This happens in some hotels implement a loop as work-around concerning the arp table issue. The source code is still on the git of Sourceforge.net. You can see the files here And you can download the last version here Feedbacks after few months I'm using this script on my OpenBSD laptop since about 5 months. In my case, I'm mainly using the openbox menus and the --restart option. The Openbox menus The openbox menus are working fine. As explain in my previous blog, I just have to create 2 entries in my openbox's menu.xml file, and all the rest comes automatically from nmctl itself thanks to the --list and --scan options. I've not changed this part of nmctl since it works as expected (for me :-) ). The --restart option Because I'm very lazy, and because OpenBSD is very simple to use, I've added the command "nmctl --restart" in the /etc/apm/resume script. Thanks to apmd, this script will be used each time I'm opening the lid of my laptop. In other words, each time I'll opening my laptop, nmctl will search the optimum network connection for me. But I had several issues in this scenario. Most of the problems were linked to the arp table issues. Indeed, in some circumstances, my proxy IP address was associated to the cable interface instead of the wifi interface or vice-versa. As consequence I'm not able to connect to the proxy, thus not able to connect to internet. So the ping to google (final test nmctl perform) is failing. Knowing that anyhow, I'm doing a full arp cleanup, it's not clear for me from where this problem come from. To solve this situation I've implemented a "retry" concept. In other words, before testing an another possible network connection (as listed in my /etc/nmctl.conf file), the script try 3x the current connection's parameters. If you want to reduce or increase this figures, you can do it via the --retry parameter. Results of my expertise with this small tool Where ever I'm located, my laptop is now connecting automatically to the wifi / cable connection previously identified for this location. Currently I have 3 places where I have Wifi credentials and 2 offices places where I just have to plug the network cable. Since the /etc/apm/resume scripts is triggered when I open the lid of the laptop, I just have to make sure that I plug the RJ45 before opening the laptop. For the rest, I do not have to type any commands, OpenBSD do all what is needed ;-). I hotels or restaurants, I can just connect to the Open Wifi thanks to the openbox menu created by "nmctl --scan". Next steps Documentation The tool is missing lot of documentation. I appreciate OpenBSD for his great documentation, so I have to do the same. I plan to write a README and a man page at first instances. But since my laziness, I will do it as soon as I see some interest for this tool from other persons. Tests I now have to travel and see how to see the script react on the different situations. Interested persons are welcome to share with me the outcome of their tests. I'm curious how it work. OpenBSD 6.3 on EdgeRouter Lite simple upgrade method TL;DR OpenBSD 6.3 oceton upgrade instructions may not factor that your ERL is running from the USB key they want wiped with the miniroot63.fs image loaded on. Place the bsd.rd for OpenBSD 6.3 on the sd0i slice used by U-Boot for the kernel, and then edit the boot command to run it. a tiny upgrade The OpenBSD documentation is comprehensive, but there might be rough corners around what are probably edge cases in their user base. People running EdgeRouter Lite hardware for example, who are looking to upgrade from 6.2 to 6.3. The documentation, which gave us everything we needed last time, left me with some questions about how to upgrade. In INSTALL.octeon, the Upgrading section does mention: The best solution, whenever possible, is to backup your data and reinstall from scratch I had to check if that directive existed in the documentation for other architectures. I wondered if oceton users were getting singled out. We were not. Just simplicity and pragmatism. Reading on: To upgrade OpenBSD 6.3 from a previous version, start with the general instructions in the section "Installing OpenBSD". But that section requires us to boot off of TFTP or NFS. Which I don’t want to do right now. Could also use a USB stick with the miniroot63.fs installed on it. But as the ERL only has a single USB port, we would have to remove the USB stick with the current install on it. Once we get to the Install or Upgrade prompt, there would be nothing to upgrade. Well, I guess I could use a USB hub. But the ERL’s USB port is inside the case. With all the screws in. And the tools are neatly put away. And I’d have to pull the USB hub from behind a workstation. And it’s two am. And I cleaned up the cabling in the lab this past weekend. Looks nice for once. So I don’t want to futz around with all that. There must be an almost imperceptibly easier way of doing this than setting up a TFTP server or NFS share in five minutes… Right? iXsystems Boise Technology Show 2018 Recap OpenZFS User Conference Slides & Videos Thank you ZFS ZSTD Compression Pool Layout Considerations ZFS Releases Helping Developers Help You ZFS and MySQL on Linux Micron OSNEXUS ZFS at Six Feet Up Flexible Disk Use with OpenZFS Batch editing files with ed what’s ‘ed’? ed is this sort of terrifying text editor. A typical interaction with ed for me in the past has gone something like this: $ ed help ? h ? asdfasdfasdfsadf ? Basically if you do something wrong, ed will just print out a single, unhelpful, ?. So I’d basically dismissed ed as an old arcane Unix tool that had no practical use today. vi is a successor to ed, except with a visual interface instead of this ? surprise: Ed is actually sort of cool and fun So if Ed is a terrifying thing that only prints ? at you, why am I writing a blog post about it? WELL!!!! On April 1 this year, Michael W Lucas published a new short book called Ed Mastery. I like his writing, and even though it was sort of an april fool’s joke, it was ALSO a legitimate actual real book, and so I bought it and read it to see if his claims that Ed is actually interesting were true. And it was so cool!!!! I found out: how to get Ed to give you better error messages than just ? that the name of the grep command comes from ed syntax (g/re/p) the basics of how to navigate and edit files using ed All of that was a cool Unix history lesson, but did not make me want to actually use Ed in real life. But!!! The other neat thing about Ed (that did make me want to use it!) is that any Ed session corresponds to a script that you can replay! So if I know Ed, then I can use Ed basically as a way to easily apply vim-macro-like programs to my files. Beastie Bits FreeBSD Mastery: Jails -- Help make it happen Video: OpenZFS Basics presented by George Wilson and Matt Ahrens at Scale 16x back in March 2018 DragonFlyBSD’s IPFW gets highspeed lockless in-kernel NAT A Love Letter to OpenBSD New talks, and the F-bomb Practical UNIX Manuals: mdoc BSD Meetup in Zurich: May 24th BSD Meetup in Warsaw: May 24th MeetBSD 2018 Tarsnap Feedback/Questions Seth - First time poudriere Builder Farhan - Why we didn't go FreeBSD architech - Encryption Feedback Dave - Handy Tip on setting up automated coredump handling for FreeBSD Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
How Intel docs were misinterpreted by almost any OS, a look at the mininet SDN emulator, do’s and don’ts for FreeBSD, OpenBSD community going gold, ed mastery is a must read, and the distributed object store minio on FreeBSD. Headlines Intel documentation flaw sees instruction misimplemented in almost every OS A statement in the System Programming Guide of the Intel 64 and IA-32 Architectures Software Developer's Manual (SDM) was mishandled in the development of some or all operating-system kernels, resulting in unexpected behavior for #DB exceptions that are deferred by MOV SS or POP SS, as demonstrated by (for example) privilege escalation in Windows, macOS, some Xen configurations, or FreeBSD, or a Linux kernel crash. OS kernels may not expect this order of events and may therefore experience unexpected behavior when it occurs. + A detailed white paper describes this behavior here + FreeBSD Commit Thank you to the MSRC Incident Response Team, and in particular Greg Lenti and Nate Warfield, for coordinating the response to this issue across multiple vendors. Thanks to Computer Recycling at The Working Center of Kitchener for making hardware available to allow us to test the patch on additional CPU families. + FreeBSD Security Advisory + DragonFlyBSD Post + NetBSD does not support debug register and so is not affected. + OpenBSD also appears to not be affected, “We are not aware of further vendor information regarding this vulnerability.” + IllumOS Not Impacted Guest Post – A Look at SDN Emulator Mininet A guest post on the FreeBSD Foundation’s blog by developer Ayaka Koshibe At this year’s AsiaBSDCon, I presented a talk about a SDN network emulator called Mininet, and my ongoing work to make it more portable. That presentation was focused on the OpenBSD version of the port, and I breezed past the detail that I also had a version or Mininet working on FreeBSD. Because I was given the opportunity, I’d like to share a bit about the FreeBSD version of Mininet. It will not only be about what Mininet is and why it might be interesting, but also a recounting of my experience as a user making a first-time attempt at porting an application to FreeBSD. Mininet started off as a tool used by academic researchers to emulate OpenFlow networks when they didn’t have convenient access to actual networks. Because of its history, Mininet became associated strongly with networks that use OpenFlow for their control channels. But, it has also become fairly popular among developers working in, and among several universities for research and teaching about, SDN (Software Defined Networking) I began using Mininet as an intern at my university’s network research lab. I was using FreeBSD by that time, and wasn’t too happy to learn that Mininet wouldn’t work on anything but Linux. I gradually got tired of having to run a Linux VM just to use Mininet, and one day it clicked in my mind that I can actually try porting it to FreeBSD. Mininet creates a topology using the resource virtualization features that Linux has. Specifically, nodes are bash processes running in network namespaces, and the nodes are interconnected using veth virtual Ethernet links. Switches and controllers are just nodes whose shells have run the right commands to configure a software switch or start a controller application. Mininet can therefore be viewed as a series of Python libraries that run the system commands necessary to create network namespaces and veth interfaces, assemble a specified topology, and coordinate how user commands aimed at nodes (since they are just shells) are run. Coming back to the port, I chose to use vnet jails to replace the network namespaces, and epair(4) links to replace the veth links. For the SDN functionality, I needed at least one switch and controller that can be run on FreeBSD. I chose OpenvSwitch(OVS) for the switch, since it was available in ports and is well-known by the SDN world, and Ryu for the controller since it’s being actively developed and used and supports more recent versions of OpenFlow. I have discussed the possibility of upstreaming my work. Although they were excited about it, I was asked about a script for creating VMs with Mininet preinstalled, and continuous integration support for my fork of the repository. I started taking a look at the release scripts for creating a VM, and after seeing that it would be much easier to use the scripts if I can get Mininet and Ryu added to the ports tree, I also tried a hand at submitting some ports. For CI support, Mininet uses Travis, which unfortunately doesn’t support FreeBSD. For this, I plan to look at a minimalistic CI tool called contbuild, which looks simple enough to get running and is written portably. This is very much a work-in-progress, and one going at a glacial pace. Even though the company that I work for does use Mininet, but doesn’t use FreeBSD, so this is something that I’ve been working on in my free time. Earlier on, it was the learning curve that made progress slow. When I started, I hadn’t done anything more than run FreeBSD on a laptop, and uneventfully build a few applications from the ports tree. Right off the bat, using vnet jails meant learning how to build and run a custom kernel. This was the easy part, as the handbook was clear about how to do this. When I moved from using FreeBSD 10.3 to 11, I found that I can panic my machine by quickly creating and destroying OVS switches and jails. I submitted a bug report, but decided to go one step further and actually try to debug the panic for myself. With the help of a few people well-versed in systems programming and the developer’s handbook, I was able to come up with a fix, and get it accepted. This pretty much brings my porting experiment to the present day, where I’m slowly working out the pieces that I mentioned earlier. In the beginning, I thought that this Mininet port would be a weekend project where I come out knowing thing or two about using vnet jails and with one less VM to run. Instead, it became a crash course in building and debugging kernels and submitting bug reports, patches, and ports. It’d like to mention that I wouldn’t have gotten far at all if it weren’t for the helpful folks, the documentation, and how debuggable FreeBSD is. I enjoy good challenges and learning experiences, and this has definitely been both. Thank you to Ayaka for working to port Mininet to the BSDs, and for sharing her experiences with us. If you want to see the OpenBSD version of the talk, the video from AsiaBSDCon is here, and it will be presented again at BSDCan. **iXsystems** [iXsystems LFNW Recap](https://www.ixsystems.com/blog/lfnw-2018-recap/) 10 Beginner Do's and Don't for FreeBSD 1) Don't mix ports and binary packages 2) Don't edit 'default' files 3) Don't mess with /etc/crontab 4) Don't mess with /etc/passwd and /etc/groups either! 5) Reconsider the removal of any options from your customized kernel configuration 6) Don't change the root shell to something else 7) Don't use the root user all the time 8) /var/backups is a thing 9) Check system integrity using /etc/mtree 10) What works for me doesn't have to work for you! News Roundup OpenBSD Community Goes Gold for 2018! Ken Westerback (krw@ when wearing his developer hat) writes: ``` Monthly paypal donations from the OpenBSD community have made the community the OpenBSD Foundation's first Gold level contributor for 2018! 2018 is the third consecutive year that the community has reached Gold status or better. These monthly paypal commitments by the community are our most reliable source of funds and thus the most useful for financial planning purposes. We are extremely thankful for the continuing support and hope the community matches their 2017 achievement of Platinum status. Or even their 2016 achievement of Iridium status. Sign up now for a monthly donation! Note that Bitcoin contributions have been re-enabled now that our Bitcoin intermediary has re-certified our Canadian paperwork. https://www.openbsdfoundation.org/donations.html ``` ed(1) mastery is a must read for real unix people In some circles on the Internet, your choice of text editor is a serious matter. We've all seen the threads on mailing lits, USENET news groups and web forums about the relative merits of Emacs vs vi, including endless iterations of flame wars, and sometimes even involving lesser known or non-portable editing environments. And then of course, from the Linux newbies we have seen an endless stream of tweeted graphical 'memes' about the editor vim (aka 'vi Improved') versus the various apparently friendlier-to-some options such as GNU nano. Apparently even the 'improved' version of the classical and ubiquitous vi(1) editor is a challenge even to exit for a significant subset of the younger generation. Yes, your choice of text editor or editing environment is a serious matter. Mainly because text processing is so fundamental to our interactions with computers. But for those of us who keep our systems on a real Unix (such as OpenBSD or FreeBSD), there is no real contest. The OpenBSD base system contains several text editors including vi(1) and the almost-emacs mg(1), but ed(1) remains the standard editor. Now Michael Lucas has written a book to guide the as yet uninitiated to the fundamentals of the original Unix text editor. It is worth keeping in mind that much of Unix and its original standard text editor written back when the standard output and default user interface was more likely than not a printing terminal. To some of us, reading and following the narrative of Ed Mastery is a trip down memory lane. To others, following along the text will illustrate the horror of the world of pre-graphic computer interfaces. For others again, the fact that ed(1) doesn't use your terminal settings much at all offers hope of fixing things when something or somebody screwed up your system so you don't have a working terminal for that visual editor. DigitalOcean Digital Ocean Promo Link for BSD Now Listeners Distributed Object Storage with Minio on FreeBSD Free and open source distributed object storage server compatible with Amazon S3 v2/v4 API. Offers data protection against hardware failures using erasure code and bitrot detection. Supports highly available distributed setup. Provides confidentiality, integrity and authenticity assurances for encrypted data with negligible performance overhead. Both server side and client side encryption are supported. Below is the image of example Minio setup. Architecture Diagram The Minio identifies itself as the ZFS of Cloud Object Storage. This guide will show You how to setup highly available distributed Minio storage on the FreeBSD operating system with ZFS as backend for Minio data. For convenience we will use FreeBSD Jails operating system level virtualization. Setup The setup will assume that You have 3 datacenters and assumption that you have two datacenters in whose the most of the data must reside and that the third datacenter is used as a ‘quorum/witness’ role. Distributed Minio supports up to 16 nodes/drives total, so we may juggle with that number to balance data between desired datacenters. As we have 16 drives to allocate resources on 3 sites we will use 7 + 7 + 2 approach here. The datacenters where most of the data must reside have 7/16 ratio while the ‘quorum/witness’ datacenter have only 2/16 ratio. Thanks to built in Minio redundancy we may loose (turn off for example) any one of those machines and our object storage will still be available and ready to use for any purpose. Jails First we will create 3 jails for our proof of concept Minio setup, storage1 will have the ‘quorum/witness’ role while storage2 and storage3 will have the ‘data’ role. To distinguish commands I type on the host system and storageX Jail I use two different prompts, this way it should be obvious what command to execute and where. WeI know the FreeNAS people have been working on integrating this Best practises for pledge(2) security Let's set the record straight for securing kcgi CGI and FastCGI applications with pledge(2). This is focussed on secure OpenBSD deployments. Theory Internally, kcgi makes considerable use of available security tools. But it's also designed to be invoked in a secure environment. We'll start with pledge(2), which has been around on OpenBSD since version 5.9. If you're reading this tutorial, you're probably on OpenBSD, and you probably have knowledge of pledge(2). How to begin? Read kcgi(3). It includes canonical information on which pledge(2) promises you'll need for each function in the library. This is just a tutorial—the manpage is canonical and overrides what you may read here. Next, assess the promises that your application needs. From kcgi(3), it's easy to see which promises we'll need to start. You'll need to augment this list with whichever tools you're also using. The general push is to start with the broadest set of required promises, then restrict as quickly as possible. Sometimes this can be done in a single pledge(2), but other times it takes a few. Beastie Bits April's London *BSD meetup - notes May’s London *BSD Meetup: May 22nd Call for Papers for EuroBSDcon 2018 FreeBSD Journal March/April Desktop/Laptop issue LWN followup on the PostgreSQL fsync() issue The Association for Computing Machinery recognizes Steve Bourne for outstanding contributions Feedback/Questions Ray - Speaking at Conferences Casey - Questions Jeremy - zfs in the enterprise HAST + ZFS Lars - Civil Infrastructure Platform use of *BSD Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
TrueOS Stable 18.03 released, a look at F-stack, the secret to an open source business model, intro to jails and jail networking, FreeBSD Foundation March update, and the ipsec Errata. Headlines TrueOS STABLE 18.03 Release The TrueOS team is pleased to announce the availability of a new STABLE release of the TrueOS project (version 18.03). This is a special release due to the security issues impacting the computing world since the beginning of 2018. In particular, mitigating the “Meltdown” and “Spectre” system exploits make it necessary to update the entire package ecosystem for TrueOS. This release does not replace the scheduled June STABLE update, but provides the necessary and expected security updates for the STABLE release branch of TrueOS, even though this is part-way through our normal release cycle. Important changes between version 17.12 and 18.03 “Meltdown” security fixes: This release contains all the fixes to FreeBSD which mitigate the security issues for systems that utilize Intel-based processors when running virtual machines such as FreeBSD jails. Please note that virtual machines or jails must also be updated to a version of FreeBSD or TrueOS which contains these security fixes. “Spectre” security mitigations: This release contains all current mitigations from FreeBSD HEAD for the Spectre memory-isolation attacks (Variant 2). All 3rd-party packages for this release are also compiled with LLVM/Clang 6 (the “retpoline” mitigation strategy). This fixes many memory allocation issues and enforces stricter requirements for code completeness and memory usage within applications. Unfortunately, some 3rd-party applications became unavailable as pre-compiled packages due to non-compliance with these updated standards. These applications are currently being fixed either by the upstream authors or the FreeBSD port maintainers. If there are any concerns about the availability of a critical application for a specific workflow, please search through the changelog of packages between TrueOS 17.12 and 18.03 to verify the status of the application. Most systems will need microcode updates for additional Spectre mitigations. The microcode updates are not enabled by default. This work is considered experimental because it is in active development by the upstream vendors. If desired, the microcode updates are available with the new devcpu-data package, which is available in the Appcafe. Install this package and enable the new microcode_update service to apply the latest runtime code when booting the system. Important security-based package updates LibreSSL is updated from version 2.6.3 -> 2.6.4 Reminder: LibreSSL is used on TrueOS to build any package which does not explicitly require OpenSSL. All applications that utilize the SSL transport layer are now running with the latest security updates. Browser updates: (Keep in mind that many browsers have also implemented their own security mitigations in the aftermath of the Spectre exploit.) Firefox: 57.0.1 -> 58.0.2 Chromium: 61.0.3163.100 -> 63.0.3239.132 Qt5 Webengine (QupZilla, Falkon, many others): 5.7.1 -> 5.9.4 All pre-compiled packages for this release are built with the latest versions of LLVM/Clang, unless the package explicitly requires GCC. These packages also utilize the latest compile-time mitigations for memory-access security concerns. F-Stack F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API. http://www.f-stack.org Introduction With the rapid development of NIC, the poor performance of data packets processing with Linux kernel has become the bottleneck. However, the rapid development of the Internet needs high performance of network processing, kernel bypass has caught more and more attentions. There are various similar technologies appear, such as DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow, all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copying, thread scheduling, system calls and interrupts. Furthermore, kernel bypass can achieve higher performance with multi optimizing methods. Within various techniques, DPDK has been widely used because of its more thorough isolation from kernel scheduling and active community support. F-Stack is an open source network framework with high performance based on DPDK. With following characteristics Ultra high network performance which can achieve network card under full load, 10 million concurrent connections, 5 million RPS, 1 million CPS. Transplant FreeBSD 11.01 user space stack, provides a complete stack function, cut a great amount of irrelevant features. Therefore greatly enhance the performance. Support Nginx, Redis and other mature applications, service can easily use F-Stack With Multi-process architecture, easy to extend Provide micro thread interface. Various applications with stateful app can easily use F-Stack to get high performance without processing complex asynchronous logic. Provide Epoll/Kqueue interface that allow many kinds of applications easily use F-Stack History In order to deal with the increasingly severe DDoS attacks, authorized DNS server of Tencent Cloud DNSPod switched from Gigabit Ethernet to 10-Gigabit at the end of 2012. We faced several options, one is to continue to use the original model another is to use kernel bypass technology. After several rounds of investigation, we finally chose to develop our next generation of DNS server based on DPDK. The reason is DPDK provides ultra-high performance and can be seamlessly extended to 40G, or even 100G NIC in the future. After several months of development and testing, DKDNS, high-performance DNS server based on DPDK officially released in October 2013. It's capable of achieving up to 11 million QPS with a single 10GE port and 18.2 million QPS with two 10GE ports. And then we developed a user-space TCP/IP stack called F-Stack that can process 0.6 million RPS with a single 10GE port. With the fast growth of Tencent Cloud, more and more services need higher network access performance. Meanwhile, F-Stack was continuous improving driven by the business growth, and ultimately developed into a general network access framework. But this TCP/IP stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high, we've tried several plans and finally determined to port FreeBSD(11.0 stable) TCP/IP stack into F-Stack. Thus, we can reduce the cost of maintenance and follow up the improvement from community quickly.Thanks to libplebnet and libuinet, this work becomes a lot easier. With the rapid development of all kinds of application, in order to help different APPs quick and easily use F-Stack, F-Stack has integrated Nginx, Redis and other commonly used APPs, and a micro thread framework, and provides a standard Epoll/Kqueue interface. Currently, besides authorized DNS server of DNSPod, there are various products in Tencent Cloud has used the F-Stack, such as HttpDNS (D+), COS access module, CDN access module, etc.. iXsystems Leadership Is The Secret To An Open Source Business Model A Forbes article by Mike Lauth, CEO of iXsystems There is a good chance you’ve never heard of open source software and an even greater one that you’re using it every day without even realizing it. Open source software is computer software that is available under a variety of licenses that all encourage the sharing of the software and its underlying source code. Open source has powered the internet from day one and today powers the cloud and just about everything connected to it from your mobile phone to virtually every internet of things device. FreeNAS is one of two open source operating systems that my company, iXsystems, develops and distributes free of charge and is at the heart of our line of TrueNAS enterprise storage products. While some of our competitors sell storage software similar to FreeNAS, we not only give it away but also do so with truly no strings attached -- competitors can and do take FreeNAS and build products based on it with zero obligation to share their changes. The freedom to do so is the fundamental tenet of permissively licensed open source software, and while it sounds self-defeating to be this generous, we’ve proven that leadership, not licensing, is the true secret to a successful open source business model. We each have our own personal definition of what is fair when it comes to open source. At iXsystems, we made a conscious decision to base FreeNAS and TrueOS on the FreeBSD operating system developed by the FreeBSD project. We stand on the shoulders of giants by using FreeBSD and we consider it quite reasonable to give back on the same generous terms that the FreeBSD project offers us. We could be selective in what we provide free of charge, but we believe that doing so would be short-sighted. In the long game we’re playing, the leadership we provide over the open source projects we produce is infinitely more important than any restrictions provided by the licenses of those and other open source projects. Twenty years in, we have no reason to change our free-software-on-great-hardware business model and giving away the software has brought an unexpected side-benefit: the largest Q/A department in the world, staffed by our passionate users who volunteer to let us know every thought they have about our software. We wouldn’t change a thing, and I encourage you to find exactly what win-win goodwill you and your company can provide to your constituents to make them not just a customer base but a community. Drive The Conversation It took a leap of faith for us to give away the heart of our products in exchange for a passionate community, but doing so changes your customer's relationship with your brand from priced to priceless. This kind of relationship leverages a social contract instead of a legal one. Taking this approach empowers your users in ways they will not experience with other companies and it is your responsibility to lead, rather than control them with a project like FreeNAS Relieve Customer Pain Points With Every New Release Responsiveness to the needs of your constituents is what distinguishes project leadership from project dictatorship. Be sure to balance your vision for your products and projects with the “real world” needs of your users. While our competition can use the software we develop, they will at best wow users with specific features rather than project-wide ones. Never underestimate how grateful a user will be when you make their job easier. Accept That A Patent Is Not A Business Model Patents are considered the ultimate control mechanism in the technology industry, but they only provide a business model if you have a monopoly and monopolies are illegal. Resist getting hung up on the control you can establish over your customers and spend your time acquiring and empowering them. The moment you both realize that your success is mutual, you have a relationship that will last longer than any single sale. You’ll be pleasantly surprised how the relationships you build will transcend the specific companies that friends you make work for. Distinguish Leadership From Management Every company has various levels of management, but leadership is the magic that creates markets where they did not exist and aligns paying customers with value that you can deliver in a profitable manner. Leadership and vision are ultimately the most proprietary aspects of a technology business, over every patentable piece of hardware or licensable piece of software. Whether you create a new market or bring efficiency to an existing one, your leadership is your secret weapon -- not your level of control. News Roundup Introduction to Jails and Jail Networking on FreeBSD Jails basically partition a FreeBSD system into various isolated sub-systems called jails. The syscall and userspace tools first appeared in FreeBSD 4.0 (~ March 2000) with subsequent releases expanding functionality and improving existing features as well as usability. + For Linux users, jails are similar to LXC, used for resource/process isolation. Unlike LXC however, jails are a first-class concept and are well integrated into the base system. Essentially however, both offer a chroot-with-extra-separation feeling. Setting up a jail is a fairly simple process, which can essentially be split into three steps: + Place the stuff you want to run and the stuff it needs to run somewhere on your filesystem. + Add some basic configuration for the jail in jail.conf. + Fire up the jail. To confirm that the jail started successfully we can use the jls utility: We can now enter the jailed environment by using jexec, which will by default execute a root shell inside the named jail A jail can only see and use addresses that have been passed down to it by the parent system. This creates a slight problem with the loopback address: The host would probably like to keep that address to itself and not share it with any jail. Because of this, the loopback-address inside a jail is emulated by the system: + 127.0.0.1 is an alias for the first IPv4-address assigned to the jail. + ::1 is an alias for the first IPv6-address assigned to the jail. While this looks simple enough and usually works just fine[tm], it is also a source of many problems. Just imagine if your jail has only one single global IPv4 assigned to it. A daemon binding its (possibly unsecured) control port to the loopback-address would then unwillingly be exposed to the rest of the internet, which is hardly ever a good idea. + So, create an extra loopback adapter, and make the first IP in each jail a private loopback address + The tutorial goes on to cover making multiple jails share a single public IP address using NAT + It also covers more advanced concepts like ‘thin’ jails, to save some disk space if you are going to create a large number of jails, and how to upgrade them after the fact + Finally, it covers the integration with a lot of common tools, like identifying and filter jailed processes using top and ps, or using the package managers support for jails to install packages in a jail from the outside. **DigitalOcean** SmartOS release-20180315 ``` Hello All, The latest bi-weekly "release" branch build of SmartOS is up: curl -C - -O https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos-latest.iso curl -C - -O https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos-latest-USB.img.bz2 curl -C - -O https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos-latest.vmwarevm.tar.bz2 A generated changelog is here: https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos.html#20180329T002644Z The full build bits directory, for those interested, is here in Manta: /Joyent_Dev/public/SmartOS/20180329T002644Z Highlights Firewall rules created with fwadm(1M) can now use the PRIORITY keyword to specify a higher precedence for a rule. This release has includes mitigation of the Intel Meltdown vulnerability in the form of kpti (kernel page table isolation) with PCID (process context identifier) support This release also includes experimental support for bhyve branded zones. General Info Every second Thursday we roll a "release-YYYYMMDD" release branch and builds for SmartOS (and Triton DataCenter and Manta, as well). Cheers, Josh Wilsdon, on behalf of the SmartOS developers https://smartos.org ``` Here's a screencap from q5sys' machine showing the output of sysinfo: https://i.imgur.com/MFkNi76.jpg FreeBSD Foundation March 2018 Update > Syzkaller update: Syzkaller is a coverage-guided system call fuzzer. It invokes syscalls with arbitrary and changing inputs, and is intended to use code coverage data to guide changes to system call inputs in order to access larger and larger portions of the kernel in the search for bugs. > Last term’s student focused largely on scripts to deploy and configure Syzkaller on Packet.net’s hosting infrastructure, but did not get to the code coverage integration required for Syzkaller to be effective. This term co-op student Mitchell Horne has been adding code coverage support in FreeBSD for Syzkaller. > The Linux code coverage support for Syzkaller is known as kcov and was submitted by Dmitry Vyukov, Syzkaller’s author. Kcov is purposebuilt for Syzkaller: > kcov provides code coverage collection for coverage-guided fuzzing (randomized testing). Coverage-guided fuzzing is a testing technique that uses coverage feedback to determine new interesting inputs to a system. > kcov does not aim to collect as much coverage as possible. It aims to collect more or less stable coverage that is function of syscall inputs. To achieve this goal it does not collect coverage in soft/hard interrupts and instrumentation of some inherently non-deterministic or non-interesting parts of kernel is disabled (e.g. scheduler, locking). > Mitchell implemented equivalent functionality for FreeBSD - a distinct implementation, but modelled on the one in Linux. These patches are currently in review, as are minor changes to Syzkaller to use the new interface on FreeBSD. > We still have some additional work to fully integrate Syzkaller and run it on a consistent basis, but the brief testing that has been completed suggests this work will provide a very valuable improvement in test coverage and opportunities for system hardening: we tested Syzkaller with Mitchell's code coverage patch over a weekend. It provoked kernel crashes hundreds of times faster than without his work. > I want to say thank you to NetApp for becoming an Iridium Partner again this year! (Donations between $100,000 - $249,999) It’s companies like NetApp, who recognize the importance of supporting our efforts, that allow us to continue to provide software improvements, advocate for FreeBSD, and help lead the release engineering and security efforts. > Conference Recap: FOSSASIA 2018 Foundation Director Philip Paeps went to FOSSASIA, which is possibly the largest open source event in Asia. The FreeBSD Foundation sponsored the conference. Our booth had a constant stream of traffic over the weekend and we handed out hundreds of FreeBSD stickers, pens and flyers. Many attendees of FOSSASIA had never heard of FreeBSD before and are now keen to start exploring and perhaps even contributing. By the end of the conference, there were FreeBSD stickers everywhere! > One particular hallway-track conversation led to an invitation to present FreeBSD at a "Women Who Code" evening in Kuala Lumpur later this week (Thursday 29th March). I spent the days after the conference meeting companies who use (or want to use) FreeBSD in Singapore. > SCaLE 16x: The Foundation sponsored a FreeBSD table in the expo hall that was staffed by Dru Lavigne, Warren Block, and Deb Goodkin. Our purpose was to promote FreeBSD, and attract more users and contributors to the Project. We had a steady flow of people stopping by our table, asking inquisitive questions, and picking up some cool swag and FreeBSD handouts. Deb Goodkin took some tutorials/trainings there and talked to a lot of other open source projects. Next year, we have the opportunity to have a BSD track, similar to the BSD Devroom at FOSDEM. We are looking for some volunteers in Southern California who can help organize this one or two-day event and help us educate more people about the BSDs. Let us know if you would like to help with this effort. Roll Call: #WhoUsesFreeBSD Many of you probably saw our post on social media asking Who Uses FreeBSD. Please help us answer this question to assist us in determining FreeBSD market share data, promote how companies are successfully using FreeBSD to encourage more companies to embrace FreeBSD, and to update the list of users on our website. Knowing who uses FreeBSD helps our contributors know where to look for jobs; knowing what universities teach with FreeBSD, helps companies know where to recruit, and knowing what products use FreeBSD helps us determine what features and technologies to support. New Hosting Partner: Oregon State University Open Source Lab > We are pleased to announce that the Oregon State University (OSU) Open Source Lab (OSL), which hosts infrastructure for over 160 different open source projects, has agreed to host some of our servers for FreeBSD development. The first server, which should be arriving shortly, is an HP Enterprise Proliant DL360 Gen10 configured with NVDIMM memory which will be initially used for further development and testing of permanent memory support in the kernel. Stay tuned for more news from the FreeBSD Foundation in May (next newsletter). Beastie Bits cURL is 20 today A Note on SYSVIPC and Jails on FreeBSD OpenBSD Errata: March 20th, 2018 (ipsec) FreeBSD Security Advisories for IPSEC and vt 23 Useful PKG Command Examples to Manage Packages in FreeBSD Tarsnap Feedback/Questions Casey - Cool Editor Nelson - New article on FreeBSD vs MacOS Damian - Mysterious Reverse Proxy 504 Nelson - FreeBSD, rsync, nasty bug, now fixed Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
We'll cover OpenBSD's defensive approach to OS security, help you Understanding Syscall Conventions for Different Platforms, Mishandling SMTP Sender Verification, how the cd command works, and the LUA boot loader coming to FreeBSD. This episode was brought to you by Headlines Pledge: OpenBSD's defensive approach to OS Security (https://medium.com/@_neerajpal/pledge-openbsds-defensive-approach-for-os-security-86629ef779ce) The meaning of Pledge is same as in the real world, that is, “a solemn promise or undertaking”. So, in OpenBSD: Calling pledge in a program means to promise that the program will only use certain resources. How does it make a program more secure? It limits the operation of a program. Example: You wrote a program named ‘abc' that only needed the stdio to just print something to stdout. You added pledge to use only stdio and nothing else. Then, a malicious user found out that there is a vulnerability in your program which one can exploit and get into shell (or root shell). Exploiting your program to open a shell (or root shell) will result in the kernel killing the process with SIGABRT (which cannot be caught/ignored) and will generate a log (which you can find with dmesg). This happens because before executing other codes of your program, the code first pledges not to use anything other than stdio promise/operations. But, opening a shell or root shell will call several other system-calls which are distributed in lots of other promises like “stdio”, “proc”, “exec” etc. They are all forbidden because the program has already promised not to use any promises other than stdio. Pledge is not a system call filter. So, it is not used to restrict system calls. For example, pledge(“read”,NULL) ? wrong syntax of the pledge() pledge(“stdio inet”,NULL) ? correct syntax of the pledge() Pledge works on stdio, dns, inet, etc. promises but not directly on system calls like read, write, etc. And, unique functionality of pledge() is that it works on behavioral approach not just like 1:1 approach with the system calls. On 11 December 2017, Theo de Raadt said: List: openbsd-tech Subject: pledge execpromises From: Theo de Raadt Date: 2017–12–11 21:20:51 Message-ID: 6735.1513027251 () cvs ! openbsd ! org This will probably be committed in the next day or so. The 2nd argument of pledge() becomes execpromises, which is what will gets activated after execve. There is also a small new feature called “error”, which causes violating system calls to return -1 with ENOSYS rather than killing the process. This must be used with EXTREME CAUTION because libraries and programs are full of unchecked system calls. If you carry on past one of these failures, your program is in uncharted territory and risks of exploitation become high. “error” is being introduced for a different reason: The pre-exec process's expectation of what the post-exec process will do might mismatch, so “error” allows things like starting an editor which has no network access or maybe other restrictions in the future… Every Journey Starts with a FAIL...or Understanding Syscall Conventions for Different Platforms (http://k3research.outerhaven.de/posts/every-journey-starts-with-a-fail.html) Introduction Not long ago I started looking into FreeBSD kernel exploitation. There are only a few resources but probably the best starting point is argp's Phrack article from 2009[0]. And while he does only provide one technique, I wanted to understand it and port it to a modern FreeBSD release before describing new, own researched techniques. Well, at least this was my plan. In reality I ended researching how different operating systems resp. the same operating system but for different architectures implement syscalls. Hence, new exploiting methods have to wait for another post. In this one I want to describe my personal FAIL while porting argp's exploit example to a FreeBSD 11.1-RELEASE running on a 64bit processor. Maybe this will give other people interested in kernel stuff some insights they didn't know before. If you already know how syscalls work on 32bit and 64bit *BSD because you are an experienced exploit or kernel developer, you will probably want to search for something else to read. Moreover, some of the debugging stuff can look laborious because I wanted to show the steps I have done while attacking my problem instead of showing a simple walkthrough to the solution. The Problem argp described in his article vulnerable code consisting of a loadable kernel module which exposes a syscall to the userland. Because it was written around the time when FreeBSD 8-RELEASE came out and because he has written himself that the code needs smaller adjustments to work with this version (it was written for FreeBSD 7) I thought I will first port it to FreeBSD 11.1-RELEASE. Moreover it was written for an Intel 32bit processor architecture as we can see from his shellcode examples. Hence, I wanted to go right away the harder way and modify it to work on an 64bit processor. Why the Original Code Worked While It Was Wrong As written above, the syscall convention for the 32bit architecture is different from the one for the 64bit architecture. Indeed, a syscall on a 32bit FreeBSD system passes the arguments via the stack while the syscall offset is stored in the EAX register. The transfer into the kernel address space is done in 'cpufetchsyscall_args' in 'sys/i386/i386/trap.c'. ``` int cpufetchsyscallargs(struct thread *td, struct syscallargs *sa) { ... frame = td->td_frame; params = (caddr_t)frame->tf_esp + sizeof(int); sa->code = frame->tf_eax; ... if (params != NULL && sa->narg != 0) error = copyin(params, (caddr_t)sa->args, (u_int)(sa->narg * sizeof(int))); else ... } ``` That is, 'params' points to ESP+4 bytes offset. Later, the arguments are copied into the kernel space which is referenced by 'sa->args'. 'args' is an array of eight 'registert' which is defined as 'int32t' on the 32bit platform in comparison to the 64bit platform. And as 'struct args' only consisted of integers they got copied into the syscall arguments which are given to the trigger function inside the kernel module. We could verify this by changing 'int op' to 'long long op' in the kernel module and in trigger.c. We get the following output: root@freebsd64:trigger/ # ./trigger 0x28414000 256 3 1 0x28414000 256 4294967295 2 root@freebsd64:trigger/ # To bring this to an end: argp's version only worked for his special choice of arguments and only on 32bit. On 32bit FreeBSD platforms the arguments are transferred into kernel space by 4 byte integers, hence it will only work for integers anyway. On 64bit FreeBSD platforms we have to use syscall(2) in the intended way. iXsystems New Disks! (https://www.ixsystems.com/blog/gdpr-countdown/) A Life Lesson in Mishandling SMTP Sender Verification (https://bsdly.blogspot.co.uk/2018/02/a-life-lesson-in-mishandling-smtp.html) It all started with one of those rare spam mails that got through. This one was hawking address lists, much like the ones I occasionally receive to addresses that I can not turn into spamtraps. The message was addressed to, of all things, root@skapet.bsdly.net. (The message with full headers has been preserved here for reference). Yes, that's right, they sent their spam to root@. And a quick peek at the headers revealed that like most of those attempts at hawking address lists for spamming that actually make it to a mailbox here, this one had been sent by an outlook.com customer. The problem with spam delivered via outlook.com is that you can't usefully blacklist the sending server, since the largish chunk of the world that uses some sort of Microsoft hosted email solution (Office365 and its ilk) have their usually legitimate mail delivered via the very same infrastructure. And since outlook.com is one of the mail providers that doesn't play well with greylisting (it spreads its retries across no less than 81 subnets (the output of 'echo outlook.com | doas smtpctl spf walk' is preserved here), it's fairly common practice to just whitelist all those networks and avoid the hassle of lost or delayed mail to and from Microsoft customers. I was going to just ignore this message too, but we've seen an increasing number of spammy outfits taking advantage of outlook.com's seeming right of way to innocent third parties' mail boxes. So I decided to try both to do my best at demoralizing this particular sender and alert outlook.com to their problem. I wrote a messsage (preserved here) with a Cc: to abuse@outlook.com where the meat is, ``` Ms Farell, The address root@skapet.bsdly.net has never been subscribed to any mailing list, for obvious reasons. Whoever sold you an address list with that address on it are criminals and you should at least demand your money back. Whoever handles abuse@outlook.com will appreciate the attachment, which is a copy of the message as it arrived here with all headers intact. Yours sincerely, Peter N. M. Hansteen ``` What happened next is quite amazing. If my analysis is correct, it may not be possible for senders who are not themselves outlook.com customers to actually reach the outlook.com abuse team. Any student or practitioner of SMTP mail delivery should know that SPF records should only happen on ingress, that is at the point where the mail traffic enters your infrastructure and the sender IP address is the original one. Leave the check for later when the message may have been forwarded, and you do not have sufficient data to perform the check. Whenever I encounter incredibly stupid and functionally destructive configuration errors like this I tend to believe they're down to simple incompetence and not malice. But this one has me wondering. If you essentially require incoming mail to include the contents of spf.outlook.com (currently no less than 81 subnets) as valid senders for the domain, you are essentially saying that only outlook.com customers are allowed to communicate. If that restriction is a result of a deliberate choice rather than a simple configuration error, the problem moves out of the technical sphere and could conceivably become a legal matter, depending on what outlook.com have specified in their contracts that they are selling to their customers. But let us assume that this is indeed a matter of simple bad luck or incompetence and that the solution is indeed technical. I would have liked to report this to whoever does technical things at that domain via email, but unfortunately there are indications that being their customer is a precondition for using that channel of communication to them. I hope they fix that, and soon. And then move on to terminating their spamming customers' contracts. The main lesson to be learned from this is that when you shop around for email service, please do yourself a favor and make an effort to ensure that your prospective providers actually understand how the modern-ish SMTP addons SPF, DKIM and DMARC actually work. Otherwise you may end up receiving more of the mail you don't want than what you do want, and your own mail may end up not being delivered as intended. News Roundup Running Salt Proxy Minions on OpenBSD (https://mirceaulinic.net/2018-02-14-openbsd-salt-proxy/) As I have previously attempted several times in the past, I am (finally) very close to switch to OpenBSD, a more stable and reliable operating system that I like. Before starting to make the actual change on both personal and work computer, I started testing some of the tools I'm currently using, and understand what are the expectations. In general I didn't encounter issues, or when I did, I found the answers in the documentation (which is really great), or various forums. I didn't find however any questions regarding Proxy Minions on OpenBSD which is why I thought it might be helpful to share my experience. Installation and Startup With these said, I started playing with Salt, and it was simple and straightforward. First step - install Salt: pkg_add salt. This will bring several ports for Python futures, ZeroMQ, or Tornado which are needed for Salt. After configuring the pillar_roots in the /etc/salt/master config file for the Master, I started up the master process using rcctl: Starting up the Proxy Minions The Salt package for OpenBSD comes with the rc file for salt-proxy as well, /etc/rc.d/salt_proxy While typically you run a single regular Minion on a given machine, it is very like that there are multiple Proxy processes. Additionally, the default Salt rc file has the following configuration for the salt-proxy daemon: Starting many Proxy Minions I have managed to startup a Proxy Minion, but what about many? Executing the three commands above for each and every device is tedious and cannot scale very well. I thus have figured the following way: Have a separate rc file per Proxy, each having the daemon instruction explicitly specifying its Minion ID Start the service (using the regular Minion that controls the machine where the Proxy processes are running) And the test Proxy Minion is then up (after accepting the key, i.e,, salt-key -a test) Extending the same to a (very) large number of Proxy Minions, you can easily manage the rc files and start the services using a Salt State executed on the regular Minion: Using the file.managed State function to generate the contents of the rc file for each Proxy, with its own Minion ID. Using the service.running State function start the service. These two steps would suffice to start an arbitrary number of Proxy Minions, and the command executed will always be the same regardless how many processes you aim to manage. Conclusions I am still a novice when it comes to OpenBSD, I have plenty to learn, but it looks like the transition will be much smoother than I expected. I am already looking forward to the handover, and - most importantly - I will no longer be using systemd. :-) LUA boot loader coming very soon (https://lists.freebsd.org/pipermail/freebsd-current/2018-February/068464.html) As you may know, the Lua (http://www.lua.org) boot loader has been in the works for some time. It started out life as a GSoC in 2014 by Pedro Souza mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to try to integrate it. I rebased that effort into a github branch which Pedro Arthur fixed up. Over the past year, I've been cleaning up the boot loader for other reasons, and found the time was ripe to start integrating this into the tree. However, those integration efforts have taken a while as my day-job work on the boot loader took priority. In the mean time, Ed Maste and the FreeBSD Foundation funded Zakary Nafziger to enhance the original GSoC Lua scripts to bring it closer to parity with the evolution of the FORTH menu system since the GSoC project started. I'm pleased to announce that all these threads of development have converged and I'll be pushing the FreeBSD Lua Loader later today. This loader uses Lua as its scripting language instead of FORTH. While co-existance is planned, the timeline for it is looking to be a few weeks and I didn't want to delay pushing this into the tree for that. To try the loader, you'll need to build WITHOUTFORTH=yes and WITHLOADERLUA=yes. Fortunately, you needn't do a full world to do this, you can do it in src/stand and install the result (be sure to have the options for both the build and the install). This will replace your current /boot/loader that is scripted with FORTH to one that's scripted with Lua. It will install the lua scripts in /boot/lua. The boot is scripted with /boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised to create a backup copy of /boot/loader before testing (eg cp /boot/loader /boot/loaderforth), since you'll need to boot that from boot2 if something goes wrong. I've tested it extensively, though, with userboot.so and it's test program, so all the initial kinks of finding the lua scripts, etc have been worked out. While it's possible to build all the /boot/loader variants with Lua, I've just tested a BIOS booting /boot/loader both with and without menus enabled. I've not tested any of the other variants and the instructions for testing some of them may be rather tedious (especially UEFI, if you want a simple path to back out). Since there's not been full convergence testing, you'll almost certainly find bumps in this system. Also, all the build-system APIs are likely not yet final. I put MFC after a month on the commit. Due to the heroic (dare I say almost crazy) work of Kyle Evans on merging all the revs from -current to 11, I'm planning a MFC to 11 after the co-existence issues are hammered out. In 11, FORTH will be the default, and Lua will be built by default, but users will have to do something to use it. 12, both FORTH and Lua will be built and installed, with Lua as default (barring unforeseen complications). Once the co-existence stuff goes in, I imagine we'll make the switch to Lua by default shortly after that. In 13, FORTH will be removed unless there's a really really compelling case made to keep it. So please give it a spin and give me any feedback, documentation updates and/or bug fixes. I'm especially interested in reviews from people that have embedded Lua in other projects or experts in Lua that can improve the robustness of the menu code. Bitcoin Full Node on FreeBSD (https://bsdmag.org/5374-2/) What is a Bitcoin ? Bitcoin is a valuable popular open-source cryptocurrency that was invented by Satoshi Nakamoto in 2009. Bitcoins have value because they possess same characteristics like money (durability, portability, fungibility, scarcity, divisibility, and recognizability), but based on the properties of mathematics rather than on physical properties (like gold and silver) or trust in central authorities (like fiat currencies). In short, Bitcoin is backed by mathematics. Bitcoin is the first decentralized peer-to-peer cryptocurrency that is controlled by its users. Transactions take place directly between users, and are later verified by network nodes with digital signature and then placed in a public distributed ledger called a blockchain. Bitcoin is unique in that only 21 million bitcoins will ever be created. The unit of the bitcoin system is bitcoin or mBTC. What is a Bitcoin Wallet ? A wallet is nothing more than a pair of public and private keys that are created by a client to store the digital credentials for your bitcoin. There are several types of wallets: Desktop Wallet Token Wallet Online Wallet Mobile Wallet A token wallet is the safest way to work with bitcoin network, but you can use your mobile or pc as a bitcoin wallet. What is a Blockchain? A blockchain is a ledger that records bitcoin transactions. The blockchain is a distributed database that achieves independent verification of the chain of ownership. Each network node stores its own copy of the blockchain. Transactions will broadcast on the bitcoin network, and about 2400 transactions create a block. These blocks are building blocks of the blockchain. What is Mining? Mining is the process of dedicating computing power to process transactions, secure the network, and keep everyone in the system synchronized together. It has been designed to be fully decentralized. Miners need mining software with specialized hardware. Mining software listens for transactions broadcasted through the peer-to-peer network and performs appropriate tasks to process and confirm these transactions. Bitcoin miners perform this work because they can earn transaction fees paid by users for faster transaction processing. New transactions have to be confirmed then be included in a block along with a mathematical proof of work. Such proofs are very hard to generate because there is no way to create them other than by trying billions of calculations per second. Hence, miners are required to perform these calculations before their blocks are accepted by the network and before they are rewarded. As more people start to mine, the difficulty of finding valid blocks is automatically increased by the network to ensure that the average time to find a block remains equal to 10 minutes. As a result, mining is a very competitive business where no individual miner can control what is included in the blockchain. The proof of work is also designed to depend on the previous block to force a chronological order in the blockchain. This makes it exponentially difficult to reverse previous transactions because it would require the recalculation of the proofs of work of all the subsequent blocks. When two blocks are found at the same time, miners work on the first block they receive and switch to the longest chain of blocks as soon as the next block is found. This allows mining to secure and maintain a global consensus based on processing power. What is Pooled Mining? You have more chances if you participate with others to create a block. In a pool, all participating miners get paid every time a participating server solves a block. The payment depends on the amount of work an individual miner contributed to help find that block. What is a Full Node? A full node is a client that fully validates transactions and blocks. Full nodes also help the network by accepting transactions and blocks from other full nodes, validating those transactions and blocks, and then relaying them to further full nodes. Many people and organizations volunteer to run full nodes using spare computing and bandwidth resources. What is a Bitcoind? bitcoind is a Bitcoin client under the MIT license in 32-bit and 64-bit versions for Windows, GNU/Linux-based OSes, Mac OS X, OpenBSD and FreeBSD as well. Conclusion Cryptocurrencies are replacement for banking we know today, and bitcoin is the game changer. Mining bitcoin with typical hardware is not a good idea. It needs specialized devices like ASIC, but you can create a full node and help the bitcoin network. Useful Links https://en.wikipedia.org/wiki/Cryptocurrency https://bitcoin.org/en/faq *** Latest DRM Graphics work The DRM Graphics stack from Linux is ported to FreeBSD on an ongoing basis to provide support for accelerated graphics for Intel and AMD GPUs. The LinuxKPI bits that the drm-next-kmod driver port depends on have been merged into stable/11 and will be included as part of the upcoming FreeBSD 11.2 (https://svnweb.freebsd.org/ports?view=revision&revision=462202) Additionally, the version of the drives has been updated from Linux 4.9 to Linux 4.11 with a number of additional devices being supported (https://lists.freebsd.org/pipermail/freebsd-current/2018-February/068690.html) *** How does cd work? (https://blog.safia.rocks/post/171311670379/how-does-cd-work) In my last blog post, I dove into some of the code behind the sudo command. I thought this was pretty fun. sudo is one of those commands that I use quite often but haven't had the chance to look into truly. I started thinking about other commands that I use on a daily basis but had little understanding of the internals of. The first command that came to mind is cd. cd stands for change directory. Simply put, it allows you to set your current working directory to a different directory. I read through some of the code that was defined in this file. Some of it was in functions, and other bits were in templates, but after a while, I figured that most of the code was a wrapper around a function called chdir. A lot of the functions defined in the cd.def file linked above actually just invoke chdir and handle errors and parameter cleaning. So all in all, here is what happens when you run cd on the command line. The cd builtin is invoked as part of the Bash shell. The Bash shell invokes the chdir function. The chdir function is part of Unix and invokes the chdir system call. The Unix kernel executes the chdir call and does its own low-level thing. I could dive in a little bit more into how #4 works, but let's be honest, I've already read too much code at this point, and my eyes are starting to hurt. Beastie Bits Stockholm BSD User Group: March 22 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) Open Source Hardware Camp 2018 (30/06 & 01/07) Call for Participation (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2018-February/014182.html) Initial release schedule announcement for FreeBSD 11.2 (https://www.freebsd.org/releases/11.2R/schedule.html) Serious Shell Programming (Devin Teske) (https://www.gitbook.com/book/freebsdfrau/serious-shell-programming/details) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) TCP Fast Open client side lands in FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=330001) Help the Tor BSD Project increase the OS diversity of Tor nodes, for your own safety, and everyone else's (https://torbsd.org/open-letter.html) 5 Differences Between TrueOS & Linux (https://www.kompulsa.com/2018/02/23/5-differences-trueos-linux/) *** Feedback/Questions Ambrose - Bunch of questions (http://dpaste.com/0KRRG18#wrap) Eddy - ZFSoL with single SSD (http://dpaste.com/0MTXYJN#wrap)
GSoC 2018 Projects announced, tutorial FreeBSD jails with iocage, new Code of Conduct for FreeBSD, libhijack, and fancy monitoring for OpenSMTPD This episode was brought to you by Headlines Google Summer of Code 2018 (https://summerofcode.withgoogle.com/organizations/?sp-page=5) FreeBSD (https://www.freebsd.org/projects/summerofcode.html) FreeBSD Google Summer oF Code Ideas (https://wiki.freebsd.org/SummerOfCodeIdeas) You can join #freebsd-soc on the efnet IRC network to chat with FreeBSD developers interested in mentoring student proposals and projects, past FreeBSD/GSoC students, and other students applying to FreeBSD/GSoC this year. NetBSD (https://mail-index.netbsd.org/netbsd-advocacy/2018/02/12/msg000765.html) You can get a stipend (paid for by Google) and spend a few months getting to know and improving the insides of NetBSD or pkgsrc. ``` The schedule is: 12-27 March Applying 23 April Find out if you were accepted 14 May - 22 August Do the project! We have some suggestions for suitable projects: - ARM EFI bootloader - Using libFuzzer on base tools - Refactoring ALTQ (QoS implementation) and integrating with NPF - Testsuite for libcurses - Improve pkgin Other suggestions and details are at: https://wiki.netbsd.org/projects/gsoc/ ``` These projects are suggestions; you can come up with your own. Suggestions for other suitable projects are welcome. Feel free to contact, or chat around on IRC: irc.freenode.org #netbsd #netbsd-code #pkgsrc Haiku (https://summerofcode.withgoogle.com/organizations/4821756754264064/) Students: How to Apply for a Haiku Idea (https://www.haiku-os.org/community/gsoc/2018/students) Project Ideas (https://www.haiku-os.org/community/gsoc/2018/ideas) > If you have questions you can contact the devs on IRC: irc.freenode.org #haiku FreeBSD Jails with iocage (http://norrist.devio.us/iocage_freebsd.html) Introduction FreeBSD jails allow users to run multiple, isolated instances of FreeBSD on a single server. Iocage simplifies the management of FreeBSD Jails. Following this tutorial, the jails will be configured to bind to an IP address on the jail host's internal network, and the host OS will pass traffic from the external network to the jail. The jails will be managed with Iocage. Iocage uses ZFS properties to store configuration data for each jail, so a ZFS file system is required. Network setup These steps will: Set up the internal network. Enable the pf packet filter Configure pf pass internet traffic to and from the jail. PF is full featured firewall, and can do more than just pass traffic to an internal network. Refer to the PF documentation for additional configuration options. Run the following to configure the internal network and enable pf. sysrc cloned_interfaces+="lo1" sysrc ifconfig_lo1="inet 192.0.2.1/24" sysrc pf_enable="YES" Put the following in /etc/pf.conf ``` Variables ext_if should be set to the hosts external NIC extif = "vtnet0" jailif = "lo1" jailnet = $jailif:network NAT allows the jails to access the external network nat on $extif from $jailnet to any -> ($ext_if) Redirect traffic on port 80 to the web server jail Add similar rules for additional jails rdr pass on $ext_if inet proto tcp to port 80 -> 192.0.2.10 ``` Reboot to activate the network changes ZFS The best way to use ZFS on a VPS is to attach block storage as a new disk. If block storage is not available, you can optionally use a file as the ZFS device. Enable and start ZFS. sysrc zfs_enable="YES" service zfs start ZFS using Block storage List the available disks. If you are using a VPS, the block store will probably be the second disk. geom disk list Create a ZFS pool named jailstore. zpool create jailstore /dev/vtbd1 ZFS using a file Create the ZFS file. dd if=/dev/zero of=/zfsfile bs=1M count=4096 Create a ZFS pool named jailstore. zpool create jailstore /zfsfile Install iocage the easy way pkg install py36-iocage Skip to "Using iocage" Install iocage the hard way Swap file Smaller servers may not have enough RAM to build iocage. If needed, create a swap file and reboot. dd if=/dev/zero of=/swapfile bs=1M count=1024 echo 'swapfile="/swapfile"' >> /etc/rc.conf reboot Install some build dependencies pkg install subversion python36 git-lite libgit2 py36-pip Building iocage requires the FreeBSD source. svn checkout https://svn.freebsd.org/base/releng/11.1 /usr/src Get the latest FreeBSD ports tree. ``` portsnap fetch portsnap extract ``` + build iocage. cd /usr/ports/sysutils/iocage/ make install Using iocage ``` iocage activate jailstore iocage fetch iocage create -n www ip4_addr="lo1|192.0.2.10/24" -r 11.1-RELEASE iocage start www iocage console www ``` Once you have a shell inside the jail, install and start Apache. pkg install apache24 sysrc apache24_enable="yes" service apache24 start Port 80 on the jail will now be accessible on the hosts IP address. Multiple jails. Additional jails can be installed using the example above. Install the new jail with the iocage create command , but use a different IP address Expose the new jail to the network by adding additional rules to pf.conf. iXsystems SNIA Persistent Memory Summit 2018 Report (https://www.ixsystems.com/blog/snia-report-2018/) New FreeBSD Code of Conduct (https://www.freebsd.org/internal/code-of-conduct.html) The FreeBSD Project is inclusive. We want the FreeBSD Project to be a venue where people of all backgrounds can work together to make the best operating system, built by a strong community. These values extend beyond just development to all aspects of the Project. All those given recognition as members of the Project in whatever form are seen as ambassadors of the Project. Diversity is a huge strength and is critical to the long term success of the Project. To that end we have a few ground rules that we ask people to adhere to. This code applies equally to everyone representing the FreeBSD Project in any way, from new members, to committers, to the core team itself. These rules are intended to ensure a safe, harassment-free environment for all and to ensure that everyone feels welcome both working within, and interacting with, the Project. This document is not an exhaustive list of things that you should not do. Rather, consider it a guide to make it easier to enrich all of us and the technical communities in which we participate. This code of conduct applies to all spaces used by the FreeBSD Project, including our mailing lists, IRC channels, and social media, both online and off. Anyone who is found to violate this code of conduct may be sanctioned or expelled from FreeBSD Project controlled spaces at the discretion of the FreeBSD Code of Conduct Committee. Some FreeBSD Project spaces may have additional rules in place, which will be made clearly available to participants. Participants are responsible for knowing and abiding by these rules. Harassment includes but is not limited to: + Comments that reinforce systemic oppression related to gender, gender identity and expression, sexual orientation, disability, mental illness, neurodiversity, physical appearance, body size, age, race, or religion. + Unwelcome comments regarding a person's lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment. + Deliberate misgendering. + Deliberate use of "dead" or rejected names. + Gratuitous or off-topic sexual images or behaviour in spaces where they're not appropriate. + Physical contact and simulated physical contact (e.g., textual descriptions like "hug" or "backrub") without consent or after a request to stop. + Threats of violence. + Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm. + Deliberate intimidation. + Stalking or following. + Harassing photography or recording, including logging online activity for harassment purposes. + Sustained disruption of discussion. + Unwelcome sexual attention. + Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others. + Continued one-on-one communication after requests to cease. + Deliberate "outing" of any private aspect of a person's identity without their consent except as necessary to protect vulnerable people from intentional abuse. + Publication of non-harassing private communication without consent. + Publication of non-harassing private communication with consent but in a way that intentionally misrepresents the communication (e.g., removes context that changes the meaning). + Knowingly making harmful false claims about a person. Interview - Benno Rice - benno@freebsd.org (mailto:benno@freebsd.org) / @jeamland (https://twitter.com/jeamland) News Roundup libhijack in PoC||GTFO 0x17! (https://www.soldierx.com/news/libhijack-PoCGTFO-0x17) Hijacking Your Free Beasties In the land of red devils known as Beasties exists a system devoid of meaningful exploit mitigations. As we explore this vast land of opportunity, we will meet our ELFish friends, [p]tracing their very moves in order to hijack them. Since unprivileged process debugging is enabled by default on FreeBSD, we can abuse PTrace to create anonymous memory mappings, inject code into them, and overwrite PLT/GOT entries. We will revive a tool called libhijack to make our nefarious activities of hijacking ELFs via PTrace relatively easy. Nothing presented here is technically new. However, this type of work has not been documented in this much detail, tying it all into one cohesive work. In Phrack 56, Silvio Cesare taught us ELF research enthusiasts how to hook the PLT/GOT. The Phrack 59 article on Runtime Process Infection briefly introduces the concept of injecting shared objects by injecting shellcode via PTrace that calls dlopen(). No other piece of research, however, has discovered the joys of forcing the application to create anonymous memory mappings in which to inject Code. This is only part one of a series of planned articles that will follow libhijack's development. The end goal is to be able to anonymously inject shared objects. The libhijack project is maintained by the SoldierX community. Previous Research All prior work injects code into the stack, the heap, or existing executable code. All three methods create issues on today's systems. On amd64 and arm64, the two architectures libhijack cares about, the stack is non-executable by default. jemalloc, the heap implementation on FreeBSD, creates non-executable mappings. Obviously overwriting existing executable code destroys a part of the executable image. The Role of ELF > FreeBSD provides a nifty API for inspecting the entire virtual memory space of an application. The results returned from the API tells us the protection flags (readable, writable, executable) of each mapping. If FreeBSD provides such a rich API, why would we need to parse the ELF headers? PLT/GOT hijacking requires parsing ELF headers. One would not be able to find the PLT/GOT without iterating through the Process Headers to find the Dynamic Headers, eventually ending up with the DT_PLTGOT entry. With FreeBSD's libprocstat API, we don't have a need for parsing ELF headers until we get to the PLT/GOT stage, but doing so early makes it easier for the attacker using libhijack The Future of libhijack Writing devious code in assembly is cumbersome. Assembly doesn't scale well to multiple architectures. Instead, we would like to write our devious code in C, compiling to a shared object that gets injected anonymously. This requires writing a remote RTLD within libhijack and is in progress. Writing a remote RTLD will take a while as doing so is not an easy task. Additionally, creation of a general-purpose helper library that gets injected would be helpful. It could aid in PLT/GOT redirection attacks, possibly storing the addresses of functions we've previously hijacked. This work is dependent on the remote RTLD. libhijack currently lacks documentation. Once the ABI and API stabilize, formal documentation will be written. Conclusion Using libhijack, we can easily create anonymous memory mappings, inject into them arbitrary code, and hijack the PLT/GOT on FreeBSD. On HardenedBSD, a hardened derivative of FreeBSD, libhijack is fully mitigated through PaX NOEXEC. We've demonstrated that wrapper-style Capsicum is ineffective on FreeBSD. Through the use of libhijack, we emulate a control flow hijack in which the application is forced to call sandbox_open and fdlopen on the resulting file descriptor. Further work to support anonymous injection of full shared objects, along with their dependencies, will be supported in the future. Imagine injecting libpcap into Apache to sniff traffic whenever "GET /pcap" is sent. In order to prevent abuse of PTrace, FreeBSD should set the security.bsd.unprivilegedprocdebug to 0 by default. In order to prevent process manipulation, FreeBSD should implement PaX NOEXEC. libhijack can be found at https://github.com/SoldierX/libhijack Introduction to POSIX shell (https://sircmpwn.github.io/2018/02/05/Introduction-to-POSIX-shell.html) What the heck is the POSIX shell anyway? Well, the POSIX (the Portable Operating System Interface) shell is the standard Unix shell - standard meaning it was formally defined and shipped in a published standard. This makes shell scripts written for it portable, something no other shell can lay claim to. The POSIX shell is basically a formalized version of the venerable Bourne shell, and on your system it lives at /bin/sh, unless you're one of the unlucky masses for whom this is a symlink to bash. Why use POSIX shell? The “Bourne Again shell”, aka bash, is not standardized. Its grammar, features, and behavior aren't formally written up anywhere, and only one implementation of bash exists. Without a standard, bash is defined by its implementation. POSIX shell, on the other hand, has many competing implementations on many different operating systems - all of which are compatible with each other because they conform to the standard. Any shell that utilizes features specific to Bash are not portable, which means you cannot take them with you to any other system. Many Linux-based systems do not use Bash or GNU coreutils. Outside of Linux, pretty much everyone but Hurd does not ship GNU tools, including bash1. On any of these systems, scripts using “bashisms” will not work. This is bad if your users wish to utilize your software anywhere other than GNU/Linux. If your build tooling utilizes bashisms, your software will not build on anything but GNU/Linux. If you ship runtime scripts that use bashisms, your software will not run on anything but GNU/Linux. The case for sticking to POSIX shell in shipping software is compelling, but I argue that you should stick to POSIX shell for your personal scripts, too. You might not care now, but when you feel like flirting with other Unicies you'll thank me when all of your scripts work. One place where POSIX shell does not shine is for interactive use - a place where I think bash sucks, too. Any shell you want to use for your day-to-day command line work is okay in my book. I use fish. Use whatever you like interactively, but stick to POSIX sh for your scripts. How do I use POSIX shell? At the top of your scripts, put #!/bin/sh. You don't have to worry about using env here like you might have been trained to do with bash: /bin/sh is the standardized location for the POSIX shell, and any standards-conforming system will either put it there or make your script work anyway. The next step is to avoid bashisms. There are many, but here are a few that might trip you up: [[ condition ]] does not work; use [ condition ] Arrays do not work; use IFS Local variables do not work; use a subshell The easiest way to learn about POSIX shell is to read the standard - it's not too dry and shorter than you think. Using standard coreutils The last step to writing portable scripts is to use portable tools. Your system may have GNU coreutils installed, which provides tools like grep and cut. Unfortunately, GNU has extended these tools with its own non-portable flags and tools. It's important that you avoid these. One dead giveaway of a non-portable flag is long flags, e.g. grep --file=FILE as opposed to grep -f. The POSIX standard only defines the getopt function - not the proprietary GNU getopt_long function that's used to interpret long options. As a result, no long flags are standardized. You might worry that this will make your scripts difficult to understand, but I think that on the whole it will not. Shell scripts are already pretty alien and require some knowledge to understand. Is knowledge of what the magic word grep means much different from knowledge of what grep -E means? I also like that short flags allow you to make more concise command lines. Which is better: ps --all --format=user --without-tty, or ps -aux? If you are inclined to think the former, do you also prefer function(a, b, c) { return a + b + c; } over (a, b, c) => a + b + c? Conciseness matters, and POSIX shell supports comments if necessary! Some tips for using short flags: They can be collapsed: cmd -a -b -c is equivalent to cmd -abc If they take additional arguments, either a space or no separation is acceptable: cmd -f"hello world" or cmd -f "hello world" A good reference for learning about standardized commands is, once again, the standard. From this page, search for the command you want, or navigate through “Shell & Utilities” -> “Utilities” for a list. If you have man-pages installed, you will also find POSIX man pages installed on your system with the p postfix, such as man 1p grep. Note: at the time of writing, the POSIX man pages do not use dashes if your locale is UTF-8, which makes searching for flags with / difficult. Use env LC_ALL=POSIX man 1p grep if you need to search for flags, and I'll speak to the maintainer of man-pages about this. FreeBSD Broadcom Wi-Fi Improvements (http://landonf.org/code/freebsd/Broadcom_WiFi_Improvements.20180122.html) Introduction Since 2015, I've been working on improving FreeBSD support for Broadcom Wi-Fi devices and SoCs, including authoring the bhnd(4) driver family, which provides a unified bus and driver programming interface for these devices. First committed in early 2016, bhnd(4) allowed us to quickly bring up FreeBSD/MIPS on Broadcom SoCs, but it has taken much longer to implement the full set of features required to support modern Broadcom SoftMAC Wi-Fi hardware. Thanks to the generosity of the FreeBSD Foundation, I've recently finished implementing the necessary improvements to the bhnd(4) driver family. With these changes in place, I was finally able to port the existing bwn(4) Broadcom SoftMAC Wi-Fi driver to the bhnd(4) bus, and implement initial support for the BCM43224 and BCM43225 chipsets, with additional hardware support to be forthcoming. Now that my efforts on FreeBSD/Broadcom Wi-Fi support have progressed far enough to be generally useful, I wanted to take some time to provide a brief overview of Broadcom's Wi-Fi hardware, and explain how my work provides a foundation for further FreeBSD Broadcom Wi-Fi/SoC improvements. A Brief Background on Broadcom Wi-Fi Hardware Broadcom's Wi-Fi devices are members of the Broadcom Home Networking Division (BHND) device family; other BHND devices include MIPS/ARM SoCs (including Wi-Fi SoCs commonly found in consumer access points), as well as a large variety of related networking hardware. BHND devices utilize a common set of Broadcom IP cores (or "functional blocks") connected via one of two on-chip bus architectures: Hardware designed prior to 2009 used Broadcom's “SSB” backplane architecture, based on Sonics Silicon's interconnect IP. Subsequent hardware adopted Broadcom's “BCMA” backplane, based on ARM's AMBA IP. The IP cores used in earlier SSB-based devices were adapted for compatibility with the new backplane. When BHND hardware is used in a PCI Wi-Fi card, or a SDIO Wi-Fi module, the device's dual-mode peripheral controller is configured to operate as an endpoint device on the host's peripheral bus, bridging access to the SoC hardware: Host access to SoC address space is provided via a set of register windows (e.g., a set of configurable windows into SoC address space mapped via PCI BARs) DMA is supported by the bridge core's sparse mapping of host address space into the backplane address space. These address regions may be used as a target for the on-chip DMA engines. Any backplane interrupt vectors routed to the bridge core may be mapped by the bridge to host interrupts (e.g., PCI INTx/MSI/MSI-X). The host is generally expected to provide drivers for the IP cores found on the SoC backplane; since these cores are found in both BHND SoCs and BHND Wi-Fi devices, it is advantageous to share driver and platform code between the two targets. Modernizing FreeBSD's Broadcom SoftMAC Wi-Fi Support FreeBSD support for Broadcom SoftMAC Wi-Fi adapters is provided by two partially overlapping PCI/CardBus drivers: Legacy Wi-Fi adapters are supported by bwi(4). This driver remains in-tree to support devices incompatible with v4 or later firmware (e.g. BCM4301, BCM4302, BCM4306 rev 1-2), all of which were released prior to December 2002. Modern Wi-Fi adapters are supported by bwn(4), with access to on-chip cores mediated by bhnd(4). Prior to my work porting bwn(4) to bhnd(4), access to on-chip cores was mediated by sibabwn, a PCI/WiFi-specific derivative of the legacy siba(4) SSB bus driver. There were two major limitations to sibabwn that have long blocked adding support for newer SoftMAC Wi-Fi chipsets: the newer BCMA interconnect found in post-2009 hardware was not supported by siba(4), and siba_bwn assumed a PCI/PCIe bridge, preventing its use on FreeBSD/MIPS Broadcom SoCs with interconnect-attached D11 cores. The new bhnd(4) driver family, written as a replacement for siba(4) and siba_bwn, provides: A unified bus driver interface for both SSB and BCMA on-chip interconnects A generic BHND bridge driver framework for host-connected BHND devices (e.g. Wi-Fi adapters, etc) A PCI/PCIe bridge core driver, for PCI-attached BHND devices. An abstract BHND NVRAM API, with support for the varied NVRAM formats found in BHND Wi-Fi adapters and SoCs. Drivers for common BHND platform peripherals (UARTs, SPROM/flash, PMUs, etc) By porting bwn(4) to bhnd(4), we are now able to support existing BCMA devices with MAC/PHY/Radio combinations readily supported by bwn(4), as was the case with the BCM43224 and BCM43225 chipsets. This also opens the door to porting additional PHY support from Broadcom's ISC-licensed Linux drivers, and will allow us to bring up bwn(4) on Broadcom WiSoCs supported by FreeBSD/MIPS. Monitor OpenSMTPD using Logstash and Grafana (https://www.tumfatig.net/20180129/monitor-opensmtpd-using-logstash-grafana/) Logs are usefull. Graphs are sexy. Here's a way to get a view on what happens to your OpenSMTPD traffic, using Web v2.0 tools ; namely Logstash & Grafana. For those who would not be aware of those tools, logstash is some kind of log-parser that can eat syslog formatted logs and write them into elasticsearch ; in “document” format. Grafana is a Web frontend that can dig into various databases and render graphics from requests. I won't go into the whole “how to install” process here. Installation is quite straight forward and online documentation is quite clear. What you need OpenSMTPD deals with emails and logs its activity via Syslog. Syslog is configured to send the logs to Logstash. Logstash has a set of rules configured to transform the text-oriented information into searchable document-oriented data. The transformed data is stored into Elasticsearch. Elasticsearch provides Web API to search and find stuff. Grafana connects to ELS to get data and draw the graphs. Beastie Bits CharmBUG Presentation - Writing FreeBSD Malware (https://www.meetup.com/CharmBUG/events/247995596/) March London *BSD meeting 13/03/18 (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2018-February/014180.html) FreBSD Ports Workshop (https://wiki.freebsd.org/MateuszPiotrowski/Ports/Workshop) The history of NetBSD/atari and support for ATARI compatible Milan / OSC2018Osaka (https://speakerdeck.com/tsutsui/osc2018osaka) SSH Mastery, 2nd Edition (https://www.tiltedwindmillpress.com/?product=ssh-mastery-2nd-edition) *** Feedback/Questions Stephen - Viewer Interview Question (http://dpaste.com/06WTRB9#wrap) pb - trust expanding your 280TB pool (http://dpaste.com/0TZV6CM#wrap) Tim - ZFS questions for the ZFS Man (http://dpaste.com/0759X1E#wrap) Daniel - ZFS full backup question (http://dpaste.com/1SJXSBQ#wrap) ***
We talk about our recent trip to FOSDEM, we discuss the pros and cons of permissive licensing, cover the installation of OpenBSD on a dedibox with full-disk encryption, the new Lumina guide repository, and we explain ZFS vs. OpenZFS. This episode was brought to you by Headlines [FOSDEM Trip report] Your BSDNow hosts were both at FOSDEM in Brussels, Belgium over the weekend. On the friday before FOSDEM, we held a FreeBSD devsummit (3rd consecutive year), sponsored by the FreeBSD Foundation and organized by Benedict (with the help from Kristof Provost, who did it in previous years but could not make it this year). We had 21 people attend, a good mixture of FreeBSD committers (mostly ports) and guests. After introductions, we collected topics and discussed various topics, including a new plan for a future FreeBSD release roadmap (more frequent releases, so that features from HEAD can be tried out earlier in RELEASES). The devsummit concluded with a nice dinner in a nearby restaurant. On Saturday, first day of FOSDEM, we set up the FreeBSD Foundation table with flyers, stickers, FreeBSD Journal print editions, and a small RPI 3 demo system that Deb Goodkin brought. Our table was located next to the Illumos table like last year. This allowed us to continue the good relationship that we have with the Illumos people and Allan helped a little bit getting bhyve to run on Illumos with UEFI. Meanwhile, our table was visited by a lot of people who would ask questions about FreeBSD, take info material, or talk about their use cases. We were busy refilling the table throughout the day and luckily, we had many helpers at the table. Some items we had ran out in the early afternoon, an indicator of how popular they were. Saturday also featured a BSD devroom (https://twitter.com/fosdembsd), organized by Rodrigo Osorio. You can find the list of talks and the recordings on the BSD Devroom schedule (https://fosdem.org/2018/schedule/track/bsd/). The room was very crowded and popular. Deb Goodkin gave the opening talk with an overview of what the Foundation is doing to change the world. Other speakers from various BSD projects presented their talks after that with a range of topics. Among them, Allan gave his talk about ZFS: Advanced Integration (https://fosdem.org/2018/schedule/event/zfs_advanced_integration/), while Benedict presented his Reflections on Teaching a Unix Class With FreeBSD (https://fosdem.org/2018/schedule/event/reflections_on_reaching_unix_class_with_freebsd/). Sunday was just as busy on the FreeBSD table as Saturday and we finally ran out of stickers and some other goodies. We were happy with the results of the two days. Some very interesting conversations at the table about FreeBSD took place, some of which we're going to follow up afterwards. Check out the FOSDEM schedule as many talk recordings are already available, and especially the ones from the BSD devroom if you could not attend the conference. We would like to thank everyone who attended the FreeBSD devsummit, who helped out at the FreeBSD table and organized the BSD devroom. Also, thanks to all the speakers, organizers, and helping hands making FOSDEM another success this year. *** NetBSD kernel wscons IOCTL vulnerable bug class (http://blog.infosectcbr.com.au/2018/01/netbsd-kernel-wscons-ioctl-vulnerable.html) I discovered this bug class during the InfoSect public code review session we ran looking specifically at the NetBSD kernel. I found a couple of these bugs and then after the session was complete, I went back and realised the same bug was scattered in other drivers. In total, 17 instances of this vulnerability and its variants were discovered. In all fairness, I came across this bug class during my kernel audits in 2002 and most instances were patched. It just seems there are more bugs now in NetBSD while OpenBSD and FreeBSD have practically eliminated them. See slide 41 in http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf (http://www.blackhat.com/presentations/bh-usa-03/bh-us-03-cesare.pdf) for exactly the same bug (class) 16 years ago. The format of the this blog post is as follows: Introduction Example of the Bug Class How to Fix How to Detect Automatically with Coccinelle More Bugs Conclusion These source files had bugs ./dev/tc/tfb.c ./dev/ic/bt485.c ./dev/pci/radeonfb.c ./dev/ic/sti.c ./dev/sbus/tcx.c ./dev/tc/mfb.c ./dev/tc/sfb.c ./dev/tc/stic.c ./dev/tc/cfb.c ./dev/tc/xcfb.c ./dev/tc/sfbplus.c ./arch/arm/allwinner/awin_debe.c ./arch/arm/iomd/vidcvideo.c ./arch/pmax/ibus/pm.c ./dev/ic/igfsb.c ./dev/ic/bt463.c ./arch/luna68k/dev/lunafb.c Reporting of the bugs was easy. In less than a week from reporting the specific instances of each bug, patches were committed into the mainline kernel. Thanks to Luke Mewburn from NetBSD for coming to the code review session at InfoSect and coordinating with the NetBSD security team. The patches to fix these issues are in NetBSD: https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html (https://mail-index.netbsd.org/source-changes/2018/01/24/msg091428.html) "Permissive licensing is wrong!” – Is it? (https://eerielinux.wordpress.com/2017/11/25/permissive-licensing-is-wrong-is-it-1-2/) A few weeks ago I've been attacked by some GNU zealots on a German tech site after speaking in favor of permissive licenses. Unfortunately a discussion was not possible there because that would require the will to actually communicate instead of simply accusing the other side of vile motives. Since I actually do care about this topic and a reader asked for a post about it in comments a while ago, here we go. This first part tries to sum up the most important things around the topic. I deliberately aim for an objective overview that tries not to be one-sided. The second part will then contain my points in defence of permissive licensing. Why license software at all? Licenses exist for reasons of protection. If you're the author/inventor of some software, a story or whatever product, you get to decide what to do with it. You can keep it for yourself or you can give it away. If you decide for the latter, you have to decide who may use it and in which way(s). In case you intend to give it to a (potentially) large group of people, you may not want to be asked for permission to xyz by everybody. That's when you decide to write a license which states what you are allowing and explicitly disallowing. Most of the well-known commercial licenses focus on what you're not allowed to do (usually things like copying, disassembling, etc.). Open source licenses on the other hand are meant to grant the user rights (e.g. the right to distribute) while reserving some rights or only giving permission under certain conditions – and they usually make you claim responsibility for using the software. For these reasons licenses can actually be a good thing! If you got an unlicensed piece of code, you're not legally allowed to do anything with it without getting the author's permission first. And even if you got that permission, your project would be risky, since the author can withdraw it later. A proper license protects both parties. The author doesn't get his mail account full of email asking for permission, he's save from legal trouble if his code breaks anything for you and at the same time you have legal certainty when you decide to put the code to long-term use. Permissive vs. Copyleft (in a nutshell) In short terms, permissive licensing usually goes like this: “Here you are, have fun. Oh, and don't sue me if it does something else than what you expect!” Yes, it's that easy and there's little to dispute over. Copyleft on the other side sounds like this (if you ask somebody in favor of Copyleft): “Sure, you can use it, it's free. Just keep it free, ok?”. Also quite simple. And not too bad, eh? Other people however read the same thing like this: “Yes, you're free to use it. Just read these ten pages of legalese and be dead certain that you comply. If you got something wrong, we will absolutely make you regret it.” The GNU Public license (GPL) The most popular copyleft license in use is the GPL (in various versions) (https://www.gnu.org/licenses/gpl.html). It got more and more complex with each version – and to be fair, it had to, because it was necessary to react to new threats and loop holes that were found later. The GNU project states that they are committed to protect what they call the four freedoms of free software: the freedom to use the software for any purpose the freedom to change the software to suit your needs the freedom to share the software with your friends and neighbors the freedom to share the changes you make These are freedoms that every supporter of open source software should be able to agree with. So what's the deal with all the hostility and fighting between the two camps? Let's take a look at a permissive license, too. The BSD license Unlike the GPL, the BSD family of licenses begun with a rather simple license that span four rules (“original BSD license”). It was later revised and reduced to three (“modified BSD license”). And the modern BSD license that e.g. FreeBSD uses is even just two (“simplified BSD license”). Did you read the GPLv3 that I linked to above? If you are using GPL'd code you really should. In case you don't feel like reading all of it, at least take a look and grasp how long that text is. Now compare it to the complete modern BSD license (https://opensource.org/licenses/bsd-license.php). What's the problem? There are essentially two problems that cause all the trouble. The first one is the question of what should be subject to the freedom that we're talking about. And closely related, the second one is where that freedom needs to end. Ironically both camps claim that freedom is the one important thing and it must not be restricted. The GPL is meant to protect the freedom of the software and enforces the availability of the source code, hence limiting the freedom of actual persons. BSD on the other hand is meant to protect the freedom of human beings who should be able to use the software as they see fit – even if that means closing down former open source code! The GNU camp taunts permissive licenses as being “lax” for not providing the protection that they want. The other camp points out that the GPL is a complex monster and that it is virulent in nature: Since it's very strict in a lot of areas, it's incompatible with many other licenses. This makes it complicated to mix GPL and non-GPL code and in the cases where it's legally possible, the GPL's terms will take precedence and necessarily be in effect for the whole combined work. Who's right? That totally depends on what you want to achieve. There are pros and cons to both – and in fact we're only looking at the big picture here. There's also e.g. the Apache license which is often deemed as kind of middle ground. Then you may want to consider the difference between weak (e.g. LGPL) as well as strong copyleft (GPL). Licensing is a potentially huge topic. But let's keep it simple here because the exact details are actually not necessary to understand the essence of our topic. In the next post I'll present my stance on why permissive licensing is a good thing and copyleft is more problematic than many people may think. “Permissive licensing is wrong?” – No it's not! (https://eerielinux.wordpress.com/2018/01/25/permissive-licensing-is-wrong-no-its-not-2-2/) The previous post gave a short introduction into the topic of software licenses, focusing on the GPL vs. BSD discussion. This one is basically my response to some typical arguments I've seen from people who seem to loathe permissive licensing. I'll write this in dialog style, hoping that this makes it a little lighter to read. Roundup Install OpenBSD on dedibox with full-disk encryption (https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/) TL;DR: I run several "dedibox" servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server OpenBSD is not officially supported, I can't guarantee that this will work for you on any kind of server online.net provides, however I've been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up. Currently, I'm running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them. Recently I've been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references. I wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks. The method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice. Step #1: boot to rescue mode Step #2: boot to the installer Step #3: prepare softraid Step #4: reboot to encrypted OpenBSD system Bonus: further tightening your system enable doas disable the root account update system with syspatch add my ssh public key to my ~/.ssh/authorized_keys disable password authentication within ssh reboot so you boot on a brand new up-to-date system with latest stable kernel VOILA ! January 2018 Development Projects Update (https://www.freebsdfoundation.org/blog/january-2018-development-projects-update/) Spectre and Meltdown in FreeBSD Issues affecting most CPUs used in servers, desktops, laptops, and mobile devices are in the news. These hardware vulnerabilities, known by the code-names “Meltdown” and “Spectre”, allow malicious programs to read data to which they should not have access. This potentially includes credentials, cryptographic material, or other secrets. They were originally identified by a researcher from Google's Project Zero, and were also independently discovered by researchers and academics from Cyberus Technology, Graz University of Technology, the University of Pennsylvania, the University of Maryland, Rambus, the University of Adelaide and Data61. These vulnerabilities affect many CPU architectures supported by FreeBSD, but the 64-bit x86 family of processors from Intel and AMD are the most widely used, and are a high priority for software changes to mitigate the effects of Meltdown and Spectre. In particular, the Meltdown issue affects Intel CPUs and may be used to extract secret data from the running kernel, and therefore, is the most important issue to address. The FreeBSD Foundation collaborates with Intel, and under this relationship participated in a briefing to understand the details of these issues and plan the mitigations to be applied to the x86 architectures supported by FreeBSD. We also made arrangements to have FreeBSD's security officer join me in the briefing. It is through the generous support of the Foundation's donors that we are able to dedicate resources to focus on these issues on demand as they arise. Foundation staff member Konstantin (Kostik) Belousov is an expert on FreeBSD's Virtual Memory (VM) system as well as low-level x86 details, and is developing the x86 kernel mitigations for FreeBSD. The mitigation for Meltdown is known as Page Table Isolation (PTI). Kostik created a PTI implementation which was initially committed in mid-January and is available in the FreeBSD-CURRENT development repository. This is the same approach used by the Linux kernel to mitigate Meltdown. One of the drawbacks of the PTI mitigation is that it incurs a performance regression. Kostik recently reworked FreeBSD's use of Process-Context Identifiers (PCID) in order to regain some of the performance loss incurred by PTI. This change is also now available in FreeBSD-CURRENT. The issue known as Spectre comes in two variants, and variant 2 is the more troubling and pressing one. It may be mitigated in one of two ways: by using a technique called “retpoline” in the compiler, or by making use of a CPU feature introduced in a processor microcode update. Both options are under active development. Kostik's change to implement the CPU-based mitigation is currently in review. Unfortunately, it introduces a significant performance penalty and alternatives are preferred, if available. For most cases, the compiler-based retpoline mitigation is likely to be the chosen mitigation. Having switched to the Clang compiler for the base system and most of the ports collection some years ago, FreeBSD is well-positioned to deploy Clang-based mitigations. FreeBSD developer Dimitry Andric is spearheading the update of Clang/LLVM in FreeBSD to version 6.0 in anticipation of its official release; FreeBSD-CURRENT now includes an interim snapshot. I have been assisting with the import, particularly with respect to LLVM's lld linker, and will support the integration of retpoline. This support is expected to be merged into FreeBSD in the coming weeks. The Foundation's co-op students have also participated in the response to these vulnerabilities. Mitchell Horne developed the patch to control the PTI mitigation default setting, while Arshan Khanifar benchmarked the performance impact of the in-progress mitigation patches. In addition, Arshan and Mitchell each developed changes to FreeBSD's tool chain to support the full set of mitigations that will be applied. These mitigations will continue be tested, benchmarked, and refined in FreeBSD-CURRENT before being merged into stable branches and then being made available as updates to FreeBSD releases. Details on the timing of these merges and releases will be shared as they become available. I would like to acknowledge all of those in the FreeBSD community who have participated in FreeBSD's response to Meltdown and Spectre, for testing, reviewing, and coordinating x86 mitigations, for developing mitigations for other processor architectures and for the Bhyve hypervisor, and for working on the toolchain-based mitigations. Guides: Getting Started & Lumina Theme Submissions (https://lumina-desktop.org/guides-getting-started-lumina-themes/) I am pleased to announce the beginning of a new sub-series of blog posts for the Lumina project: Guides! The TrueOS/Lumina projects want to support our users as they use Lumina or experiment with TrueOS. To that end, we've recently set up a central repository for our users to share instructions or other “how-to” guides with each other! Project developers and contributors will also submit guides to the repository on occasion, but the overall goal is to provide a simple hub for instructions written by any Lumina or TrueOS user. This will make it easier for users to not only find a “how-to” for some procedure, but also a very easy way to “give back” to the community by writing simple instructions or more detailed guides. Guides Repository Our first guide to get the whole thing started was created by the TrueOS Linebacker (https://discourse.trueos.org/t/introducing-the-trueos-linebacker/991) (with technical assistance from our own q5sys). In this guide, Terry Tate will walk you through the steps necessary to submit new wallpaper images to the Lumina Themes collection. This procedure is fully documented with screenshots every step of the way, walking you through a simple procedure that only requires a web browser and a Github account! Guide: Lumina Themes Submissions (https://github.com/trueos/guides/blob/master/lumina-themes-submissions/readme.md) The end result of this guide was that Terry Tate was able to submit this cool new “Lunar-4K” wallpaper to the “lumina-nature” collection. TrueOS Community Guides (https://github.com/trueos/guides/tree/master) ZFS vs. OpenZFS (by Michael Dexter) (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You've probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. Our Senior Analyst clears up what ZFS and OpenZFS refer to and how they differ. I admit that we geeks tend to get caught up in the nuts and bolts of enterprise storage and overlook the more obvious questions that users might have. You've probably noticed that this blog and the FreeNAS blog refer to “ZFS” and “OpenZFS” seemingly at random when talking about the amazing file system at the heart of FreeNAS and every storage product that iXsystems sells. I will do my best to clarify what exactly these two terms refer to. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset' and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Yes, ZFS is really as good as people say it is. After enjoying nearly a decade of refinement by a growing group of developers around the world, ZFS became the property of database vendor Oracle, which ceased public development of both ZFS and OpenSolaris in 2010. Disappointed but undeterred, a group of OpenSolaris users and developers forked the last public release of OpenSolaris as the Illumos project. To this day, Illumos represents the official upstream home of the Open Source OpenSolaris technologies, including ZFS. The Illumos project enjoys healthy vendor and user participation but the portable nature and compelling features of ZFS soon produced far more ZFS users than Illumos users around the world. While most if not all users of Illumos and its derivatives are ZFS users, the majority of ZFS users are not Illumos users, thanks significantly in part to FreeNAS which uses the FreeBSD operating system. This imbalance plus several successful ZFS Day events led ZFS co-founder Matt Ahrens and a group of ZFS developers to announce the OpenZFS project, which would remain a part of the Illumos code base but would be free to coordinate development efforts and events around their favorite file system. ZFS Day has grown into the two-day OpenZFS Developer Summit and is stronger than ever, a testament to the passion and dedication of the OpenZFS community. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs' command, which on FreeBSD relies on the ‘zfs.ko' kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. Beastie Bits Explaining Shell (https://explainshell.com/) OPNsense® 18.1 Released (https://opnsense.org/opnsense-18-1-released/) “SSH Mastery 2/e” copyedits back (https://blather.michaelwlucas.com/archives/3104) Sponsoring a Scam (https://blather.michaelwlucas.com/archives/3106) Thursday, February 8, 2018 - Come to Netflix to talk about FreeBSD (https://www.meetup.com/BAFUG-Bay-Area-FreeBSD-User-Group/events/246623825/) BSD User Group meeting in Stockholm: March 22, 17:30 - 21:00 (https://www.meetup.com/BSD-Users-Stockholm/events/247552279/) FreeBSD Flavoured talks from Linux.conf.au: You can't unit test C, right? (https://www.youtube.com/watch?v=z-uWt5wVVkU) and A Brief History of I/O (https://www.youtube.com/watch?v=qAhZEI_6lbc) EuroBSDcon 2018 website is up (https://2018.eurobsdcon.org/) Full day bhyvecon Tokyo, Japan, March 9, 2018 (http://bhyvecon.org/) *** Feedback/Questions Thomas - freebsd installer improvements (http://dpaste.com/3G2F7RC#wrap) Mohammad - FreeBSD 11 installation from a read only rescue disk (http://dpaste.com/0HGK3FQ#wrap) Stan - Follow up on guide you covered (http://dpaste.com/2S169SH#wrap) Jalal - couple questions (http://dpaste.com/35N8QXP#wrap)
We provide you with updates to Spectre and Meltdown from various BSD projects, a review of TrueOS from Linux, how to set up FreeBSD on ThinkPad x240, and a whole bunch of beastie bits. This episode was brought to you by Headlines KPTI patch lands in FreeBSD -current (https://svnweb.freebsd.org/base?view=revision&revision=328083) After a heroic effort by Konstantin Belousov kib@FreeBSD.org, the first meltdown patch has landed in FreeBSD This creates separate page tables for the Kernel and userland, and switches between them when executions enters the kernel, and when it returns to userland It is currently off by default, but you are encouraged to test it, so it can be merged back to the release branches. Set vm.pmap.pti=1 in /boot/loader.conf The existing implementation of PCID (process-context identifiers), is not compatible with the new PTI code, and is disabled when PTI is enabled, decreasing performance. A future patch will use PCID in a way that is compatible with PTI. PCID allows the OS to annotate memory mappings to specific processes, so that they can be flushed selectively, and so that they are only used when in the context of that application. Once the developers are relatively confident in the correctness of the code that has landed in -current, it will be ported back to FreeBSD 10 and 11, and released as a security advisory. Apparently porting back to FreeBSD 11 only has some relatively simple merge conflicts, but 10 will be more work. Former FreeBSD Security Officer Dag-Erling Smørgrav has created a meltdown testing and PoC tool (https://github.com/dag-erling/meltdown) that you can use to check your system. It is not finished yet, and doesn't seem to work with newer processors (haswell and newer). The first partial mitigation for Spectre variant 2 (https://svnweb.freebsd.org/changeset/base/328011) for bhyve on AMD64 has also been committed The latest information is always available on the FreeBSD Wiki (https://wiki.freebsd.org/action/edit/SpeculativeExecutionVulnerabilities) *** Some thoughts on Spectre and Meltdown (http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html) Colin Percival breaks down how these vulnerabilities work, with same nice analogies What is a side channel: I want to know when my girlfriend's passport expires, but she won't show me her passport (she complains that it has a horrible photo) and refuses to tell me the expiry date. I tell her that I'm going to take her to Europe on vacation in August and watch what happens: If she runs out to renew her passport, I know that it will expire before August; while if she doesn't get her passport renewed, I know that it will remain valid beyond that date. Her desire to ensure that her passport would be valid inadvertently revealed to me some information: Whether its expiry date was before or after August. Spectre Variant 1: I tell my girlfriend that I'm going to take her on vacation in June, but I don't tell her where yet; however, she knows that it will either be somewhere within Canada (for which she doesn't need a passport, since we live in Vancouver) or somewhere in Europe. She knows that it takes time to get a passport renewed, so she checks her passport and (if it was about to expire) gets it renewed just in case I later reveal that I'm going to take her to Europe. If I tell her later that I'm only taking her to Ottawa — well, she didn't need to renew her passport after all, but in the meantime her behaviour has already revealed to me whether her passport was about to expire. This is what Google refers to "variant 1" of the Spectre vulnerability: Even though she didn't need her passport, she made sure it was still valid just in case she was going to need it. Spectre Variant 2: I spend a week talking about how Oxford is a wonderful place to visit and I really enjoyed the years I spent there, and then I tell her that I want to take her on vacation. She very reasonably assumes that — since I've been talking about Oxford so much — I must be planning on taking her to England, and runs off to check her passport and potentially renew it... but in fact I tricked her and I'm only planning on taking her to Ottawa. Meltdown: I tell my girlfriend that I want to take her to the Korean peninsula. She knows that her passport is valid for long enough; but she immediately runs off to check that her North Korean visa hasn't expired. Why does she have a North Korean visa, you ask? Good question. She doesn't — but she runs off to check its expiry date anyway! Because she doesn't have a North Korean visa, she (somehow) checks the expiry date on someone else's North Korean visa, and then (if it is about to expire) runs out to renew it — and so by telling her that I want to take her to Korea for a vacation I find out something she couldn't have told me even if she wanted to. Final thoughts on vulnerability disclosure The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. Now, FreeBSD can occasionally respond very quickly to security vulnerabilities, even when they arise at inconvenient times — on November 30th 2009 a vulnerability was reported at 22:12 UTC, and on December 1st I provided a patch at 01:20 UTC, barely over 3 hours later — but that was an extremely simple bug which needed only a few lines of code to fix; the Spectre and Meltdown issues are orders of magnitude more complex. To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. As a result, despite an embargo date being set for January 9th, by January 4th anyone who cared knew about the issues and there was code being passed around on Twitter for exploiting them. This is not the first time I've seen people get sloppy with embargoes recently, but it's by far the worst case. As an industry we pride ourselves on the concept of responsible disclosure — ensuring that people are notified in time to prepare fixes before an issue is disclosed publicly — but in this case there was far too much disclosure and nowhere near enough responsibility. We can do better, and I sincerely hope that next time we do. CPU microcode update code for amd64 (https://undeadly.org/cgi?action=article;sid=20180115073406) (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick Wildt (patrick@) recently committed some code that will update the Intel microcode on many Intel CPUs, a diff initially written by Stefan Fritsch (sf@). The microcode of your CPU is basically the firmware that runs on your (Intel) processor, defining its instruction set in terms of so called "microinstructions". The new code depends, of course, on the corresponding firmware package, ported by Patrick which can be installed using a very recent fw_update(1). Of course, this all plays into the recently revealed problems in Intel (and other) CPUs, Meltdown and Spectre. Now Theo has explained the workings of the code on openbsd-tech, detailing some of the challenges in updating microcode on CPUs where your OS is already starting to run. Theo hints at future updates to the intel-firmware package in his mail: (https://marc.info/?l=openbsd-tech&m=151588857304763&w=2) Patrick and others committed amd64 Intel cpu microcode update code over the last few days. The approach isn't perfect, but it is good enough for a start. I want to explain the situation. When you fw_update, you'll get the firmware files. Upon a reboot, it will attempt to update the microcode on your cpus. Maybe there isn't a new microcode. Maybe your BIOS has a copy of the microcode and installs it before booting OpenBSD. This firmware installation is done a little late. Doing it better will require some work in the bootblocks to find the firmware files, but time is a bit short to do that right now. The branch-target-cache flushing features added in new microcode are not being used yet. There is more code which has to be written, but again other work is happening first. Also, Intel is saying their new microcodes sucks and people should wait a little. "Hi, my name is Intel and I'm an cheating speculator". Several developers are working on mitigations for these issues, attacking the problem from several angles. Expect to see more updates to a CVS tree near you soon. Intel: as a *BSD user, I am fucking pissed! (https://malcont.net/2018/01/dont-like-meltdown-spectre-releated-bugs-handled/) I wasn't going to write anything on the recently found x64 architecture – related bugs. I'm not a kernel developer nor even a programmer and I can't say that I have a solid understanding of what Meltdown and Spectre attacks are. Also there already is a ton of articles and posts written by people who have no grasp of the subject. I'm however a malcontent and I find this a good way to express my feelings: Intel: as a *BSD user, I am fucking pissed! Meltdown, Spectre and BSD – the “pissed” part Part of my work is UNIX-like systems administration – including BSDs and Linuces. As much as I am happy with Linux changes already made, I am beyond pissed about how the BSDs were handled by Intel – because they were not. FreeBSD Security Team received some heads-up just before Xmas, while OpenBSD, NetBSD and DragonflyBSD teams received no prior warnings. Meltdown and Spectre attacks are hard to perform. It is a hard work to mitigate them in the software, as the bugs lay in the CPUs and are not fixable by microcode updates. Developers are trying to mitigate these bugs in a way that will deliver smallest performance losses. A lot of time consuming work is needed to fix CPU vendors' mistakes. Linux developers had this time. BSD developers did not. BSD user base too small? BSD user base is small in comparison to Linux. Seems that it's too small for Intel. PlayStation4 consoles are FreeBSD-based (and use AMD CPUs) but I think it's safe to say that gaming devices are not the most important systems to be fixed. Netflix serves their content off FreeBSD but the bugs are not remotely exploitable (possibly not including JavaScript, but it's running someone's code locally) so there's probably not much harm to be done here either. However gamers and Netflix aren't the only ones who use *BSD systems. I'd say that there is more than a few FreeBSD, NetBSD, OpenBSD and DragonFlyBSD servers on the internet. In March 2017, Intel promised “more timely support to FreeBSD”. They knew about flaws in their CPUs in June and decided that a timely manner is the end of December – short before the embargo was to be lifted. Intel and Google (probably Intel more): it was your job to pick the correct people to whom the bugs can be disclosed. In my humble opinion you chose poorly by disclosing these issues with ONLY Apple, Microsoft, and the Linux Foundation, of OS vendors. You did much harm to the BSD community. Intel: It's your bugs. And you offered “more support” to the FreeBSD Foundation less than 3 months prior to being informed (my guess is that you knew much earlier) on the flaws in YOUR products. I don't want to write more here as the wording would be too strong. Interview - Viewer Questions These days, do you consider yourself more of an programmer or a sysadmin? Which one do you enjoy more? Does FreeBSD/BSD enable your business or would another OS suit your needs just as well? You've hinted that you use FreeBSD as part of your business. Can you elaborate on that and give some technical detail on how it's used in that environment? If you were allowed three wishes for anything at all to be implemented or changed in ZFS, what would they be, and why? Per Dataset throughput and IOPS limiting Per-File Cloning and/or zfsmv (move a file from one dataset to another, without copying) Cluster support Allan, you have previously mentioned that you have worked on FreeBSD on MIPS, what made you choose the Onion Omega over something like the Raspberry Pi? What is BSD Now's association with Jupiter broadcasting, and how did the relationship come to be? Jupiter seems to be associated with several Linux-themed podcasts, and I'm wondering how and why BSD Now joined Jupiter. The two communities (the Linuxes and BSDs) don't always seem to mix freely -- or do they? What kind of keyboard is that? Have you ever tried an ErgoDox? The ErgoDox EZ is made by a Canadian. You mentioned when doing one of your talks on UCL for FreeBSD that you had only recently learned C. I am also aware of your history also on contributing to the FreeBSD handbook and to documentation in general. Given you started with C relatively recently, what made you want to learn it, how quickly did you pick it up, and is it your favourite language? It is most inspiring to me, as you are clearly so talented, and of all the languages I have learned (including C++), I still prefer C in my heart of hearts. I'd be really interested to hear your answer, many thanks. *** News Roundup LinuxAndUbuntu Review Of TrueOS A Unix Based OS (http://www.linuxandubuntu.com/home/linuxandubuntu-review-of-trueos-a-unix-based-os) Trust me, the name TrueOS takes me back to 1990s when Tru64 UNIX operating system made its presence. TrueOS is PC-BSD's new unified brand built upon FreeBSD-CURRENT code base. Note that TrueOS is not a Linux distro but is BSD Unix. FreeBSD is known for its cutting-edge features, security, scalability, and ability to work both as a server and desktop operating system. TrueOS aims at having user-friendliness with the power of FreeBSD OS. Let us start with going into details of different aspects of the TrueOS. TrueOS History ? TrueOS was founded by Kris Moore in 2005 with name PC-BSD. Initial version focused to make FreeBSD easy to use starting with providing GUI based installer (to relatively complicated FreeBSD installer). In the year 2006, PC-BSD was acquired by iXsystems. Before rebranding as TrueOS in Sept 2016, PC-BSD reached a stage starting considering better than vanilla FreeBSD. Older PC-BSD version used to support both x86 and x86-64 architecture. Kris Moore, the developer founder, says about rebranding: “We've already been using TrueOS for the server side of PC-BSD, and it made sense to unify the names. PC-BSD doesn't reflect server or embedded well. TrueOS Desktop/Server/Embedded can be real products, avoids some of the alphabet soup, and gives us a more catchy name.” TrueOS First Impression ? The startup is little longer; may be due to starting up of many services. The heavy KDE well suited to PC-BSD. The C++/Qt5 based Lumina desktop environment is light and fast. The Lumina offers an easy way to configure menu and panels. I did not face any problems for continuous use of two weeks on a virtual machine having the minimal configuration: 1 GB RAM, 20 GB hard disk and Intel 3.06 GHz i3 processor. The Lumina desktop is light and fast. The developers of Lumina know what they are doing and have a good idea of what makes a good IDE. As it happens with any new desktop environment, it needs some time to settle. Let us hope that they keep to the path they are on with it. Conclusion ? The TrueOS is impressive when consider it as relatively young. It is a daring step that TrueOS developers took FreeBSD Current rather than FreeBSD Stable code base. Overall it has created its own place from the legacy shadow of PC-BSD. Starting with easy installation TrueOS is a good combination of software and utilities that make the system ready to use. Go and get a TrueOS ISO to unleash the “bleeding edge” tag of FreeBSD Thinkpad x240 - FreeBSD Setup (http://stygix.org/nix/x240-freebsd.php) What follows is a record of how I set up FreeBSD to be my daily driver OS on the Lenovo Thinkpad X240. Everything seems to work great. Although, the touchpad needs some tweaking. I've tried several configurations, even recompiling Xorg with EVDEV support and all that, to no avail. Eventually I will figure it out. Do not sleep the laptop from the command line. Do it from within Xorg, or it will not wake up. I don't know why. You can do it from a terminal within Xorg, just not from the naked command line without Xorg started. It also will not sleep by closing the lid. I included a sudo config that allows you to run /usr/sbin/zzz without a password, so what I do is I have a key combo assigned within i3wm to run "sudo /usr/sbin/zzz". It works fine this way. I go into detail when it comes to setting up Xorg with i3wm. You can skip this if you want, but if you've never used a tiling window manager, it will handle screen real estate very efficiently on a laptop with a 12.5-inch screen and a touchpad. First, download the amd64 image for 11.1-RELEASE and flash it to a USB pen drive. For the Unices, use this: # dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync Obviously, you'll change /dev/da0 to whatever the USB pen drive is assigned. Plug it in, check dmesg. Leave it plugged in, restart the laptop. When prompted, tap Enter to halt the boot process, then F12 to select a bootable device. Choose the USB drive. I won't go through the actual install process, but it is pretty damn easy so just look at a guide or two and you'll be fine. If you can install Debian, you can install FreeBSD. I will, however, recommend ZFS if you have over 4GB of RAM (my particular variant of the X240 has 8GB of RAM, so yours should have at least 4GB), along with an encrypted disk, and an encrypted SWAP partition. When prompted to add an additional user, and you get to the question where it asks for additional groups, please make sure you add the user to "wheel". The rest should be self-explanatory during the install. Now for the good shit. You just booted into a fresh FreeBSD install. Now what? Well, time to fire up vi and open some config files... CNN Article about CDROM.com and FreeBSD, from 1999 (https://www.cnn.com/TECH/computing/9904/08/cdrom.idg/index.html) Walnut Creek CDROM sells a lot of CD-ROMs, but it gives away even more data. Specifically, anyone who has Internet access is free to log into wcarchive (ftp.cdrom.com) and start downloading bits. Even with a good Internet connection, however, you should expect to be at it for a while. At the present time, wcarchive resides on half a terabyte (500 GB) of RAID 5-disk storage. Even if your 56-Kbps modem can deliver seven kilobytes per second, downloading the complete archive would take you 70 million seconds. Even then, some of the files would be more than two years out of date, so a bit of "back and fill" would be needed. Of course, nobody uses wcarchive that way. Instead, they just drop in when they need the odd file or two. The FTP server is very accommodating; 3,600 simultaneous download sessions is the current limit and an upgrade to 10,000 sessions is in the works. This translates to about 800 GB per day of downloads. Bob Bruce (Walnut Creek's founder) says he's thinking about issuing a press release when they reach a terabyte a day. But 800 GB isn't all that shabby.... The hardware Because FTP archives don't do a lot of thinking, wcarchive doesn't need a massive cluster of CPUs. In fact, it gets by with a single 200-MHz P6 Pentium Pro and a measly(!) 1 GB of RAM. The I/O support, however, is fairly impressive. A six-channel Mylex RAID controller (DAC960SXI; Ultra-Wide SCSI-SCSI) is the centerpiece of the I/O subsystem. Two channels link it to the PC ("Personal Computer"!?!), via a dual-channel Adaptec card (AHA-3940AUW; PCI to Ultra-Wide SCSI). An 256-MB internal cache helps it to eliminate recurring disk accesses. Four nine-drive disk arrays provide the actual storage. The two larger arrays use 18-GB IBM drives; the two smaller arrays use 9-GB Micropolis and Quantum drives. A separate 4-GB Quantum drive is used as the "system disk." The output side is handled by a single Intel 100Base-T controller (Pro/100B PCI), which feeds into the Internet through a number of shared DS3 (45 Mbps) and OC3 (155 Mbps) circuits. A detailed description of the system is available as ftp.cdrom.com/archive-info/configuration; The software The system software is rather prosaic: a copy of FreeBSD, supplemented by home-grown FTP mirroring and server code. Because of the massive hardware support, the software "only" needs to keep the I/O going in an efficient and reliable manner. FreeBSD, the "prosaic" operating system mentioned above, merits a bit more discussion. Like Linux, FreeBSD is open source. Anyone can examine, modify, and/or redistribute the source code. And, like Linux, an active user community helps the authors to find bugs, improve documentation, and generally support the OS. Unlike Linux, FreeBSD is derived from the Berkeley Unix code that forms the foundation for most commercial Unix variants. When you use the "fast file system" (cylinder groups, long file names, symbolic links, etc.), TCP/IP networking, termcap, or even vi, you are using Berkeley Unix additions. The version of BSD underlying FreeBSD, however, is "pure" BSD; don't look for the System V modifications you see in Solaris. Instead, think of it as SunOS, brought up to date with Kerberos, modern sendmail, an updated filesystem, and more. Solid, fast, and free! One of FreeBSD's finest innovations, the Ports Collection, makes FreeBSD a delight for open source application users. The Ports Collection automates the downloading, building, and installation (including de-installation) of 2,300+ open source packages. The company Walnut Creek CDROM has been around for several years now, so you are likely to be familiar with its offerings. You may not realize, however, that it provides the major financial support for FreeBSD. The FreeBSD support has two purposes. First, it provides the company with a solid base to run wcarchive and other massive projects. Second, it ties in with the company's mission of making software (and data) economically accessible. Bob Bruce, the firm's founder, is an interesting guy: laid back and somewhat conservative in manner, but productive and innovative in practice. Here is a possibly illustrative story. When Bob started selling CD-ROMs, disc caddies were selling for $15 each. Bob thought that was rather high, so he started investigating the marketplace. A long-distance call to Japan got him Sony's fax number; a series of faxes got him in touch with the salespeople. It turned out that caddies were available, in bulk, for only a few dollars each. Bulk, in this case, meant pallet-loads of 10,000 caddies. In an act of great faith, Bob purchased a pallet of caddies, then proceeded to sell them for five dollars each. The results were everything he might have wished. Folks who bought his CD-ROMs added caddies to their orders; folks who bought piles of caddies added in a disc or two. Either way, Walnut Creek CDROM was making a name for itself. Many pallet-loads later, the company is still selling caddies, making and distributing CD-ROMs, and giving away bits. Walnut Creek CDROM is a real open-source success story; its breadth and depth of offerings is well worth a look. Beastie Bits OpenBSD adds kqueue event support to DRM, to detect device changes like HDMI cables being plugged in, and trigger randr events (https://github.com/openbsd/src/commit/b8584f4233dc11a328cd245a5843ec3d67462200) Thesis describing QUAD3, a unix-like, multi-tasking operating system for the 6502 processor (https://archive.org/details/AMultiTaskingOperatingSystemForMicrocomputers) Windows is getting chmod and chown... (https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/) Timeline: How they kept Meltdown and Spectre secret for so long (https://www.theverge.com/platform/amp/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux) bsd.network is a *BSD-themed Mastodon Instance (https://bsd.network/): Peter Hessler is administering a new Mastodon instance, running in an OpenBSD VM on top of an OpenBSD vmm hypervisor Computer-Aided Instruction on UNIX (https://virtuallyfun.com/wordpress/wp-content/uploads/2017/12/whfUb.pdf) AsiaBSDCon 2018 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/asiabsdcon-2018-travel-grant-application-now-open/) AsiaBSDCon 2018 FreeBSD Developers Summit Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/asiabsdcon-2018-freebsd-developers-summit-call-for-proposals/) LinuxFest Northwest 2018 Call for Proposals (https://www.freebsdfoundation.org/news-and-events/call-for-papers/linuxfest-northwest-2018-call-for-proposals/) Feedback/Questions Jason - Dont break my ports (http://dpaste.com/05PRNG2) Wilyarti - show content (http://dpaste.com/1BG8GZW) https://clinetworking.wordpress.com/2017/12/08/data-de-duplication-file-diff-ing-and-s3-style-object-storage-using-digital-ocean-spaces Scott - Your show is Perfect! (http://dpaste.com/0KER8YE#wrap) Ken - Community Culture (http://dpaste.com/0WT8285#wrap)
We walk through dumping a PS4 kernel in only 6 days, tell you the news that NetBSD 7.1.1 has been released, details on how to run FreeBSD on a Thinkpad T470, and there's progress in OpenBSD's pledge. This episode was brought to you by Headlines NetBSD 7.1.1 released (http://www.netbsd.org/releases/formal-7/NetBSD-7.1.1.html) The NetBSD Project is pleased to announce NetBSD 7.1.1, the first security/critical update of the NetBSD 7.1 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. Complete source and binaries for NetBSD 7.1.1 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at https://www.NetBSD.org/mirrors/. We encourage users who wish to install via ISO or USB disk images to download via BitTorrent by using the torrent files supplied in the images area. A list of hashes for the NetBSD 7.1.1 distribution has been signed with the well-connected PGP key for the NetBSD Security Officer: https://ftp.NetBSD.org/pub/NetBSD/security/hashes/NetBSD-7.1.1_hashes.asc NetBSD is free. All of the code is under non-restrictive licenses, and may be used without paying royalties to anyone. Free support services are available via our mailing lists and website. Commercial support is available from a variety of sources. More extensive information on NetBSD is available from our website: NetBSD website (www.NetBSD.org) +Changes Between 7.1 and 7.1.1 Below is an abbreviated list of changes in this release. The complete list can be found in the CHANGES-7.1.1 file in the top level directory of the NetBSD 7.1.1 release tree. Security Advisory Fixes The following security advisories were fixed: NetBSD-SA2017-004 buffer overflow via cmap for 4 graphics drivers. NetBSD-SA2017-005 x86: vulnerabilities in context handling. NetBSD-SA2017-006 Vnode reference leak in the openat system call. NetBSD-SA2018-001 Several vulnerabilities in context handling NetBSD-SA2018-002 Local DoS in virecover Note: Advisories prior to NetBSD-SA2017-004 do not affect NetBSD 7.1.1. Userland changes dhcrelay(8): Fix bug that prevented proper operation when run in the background. Heimdal: Update to 7.1. Fix CVE-2017-11103. mtree(8): Don't modify strings stored in hash, otherwise filling up of directory hierarchy stops if the same hash value occurs in directory and leaf. ping(8): Fix cksum calculation for clearing the cached route. resize_ffs(8): Fix numerous overflow errors which can lead to superblock corruption on large filesystems. rtadvd(8): Fix the default value of rltime. PR bin/51994. Update BIND to 9.10.5-P2. Update expat to 2.2.1. Update ntp to 4.2.8p10. Update root.cache to 2017102400. Update tzdata to 2017c. vi(1): Don't garble display when when resizing nvi in xterm. wpa_supplicant/hostapd: Update to 2.6. Apply fixes for CVEs 2017-13077 through 2017-13082 and CVEs 2017-13086 through 2017-13088. X: Apply fixes for CVEs 2017-12176 through 2017-12187, 2017-10971, 2017-10972, 2017-13722, 2017-13720, 2017-16611, and 2017-16612. *** ###Dumping a PS4 Kernel in "Only" 6 Days (https://fail0verflow.com/blog/2017/ps4-crashdump-dump/) > What if a secure device had an attacker-viewable crashdump format? What if that same device allowed putting arbitrary memory into the crashdump? Amazingly, the ps4 tempted fate by supporting both of these features! Let's see how that turned out… Crashdumps on PS4 The crash handling infrastructure of the ps4 kernel is interesting for 2 main reasons: It is ps4-specific code (likely to be buggy) If the crashdump can be decoded, we will gain very useful info for finding bugs and creating reliable exploits On a normal FreeBSD system, a kernel panic will create a dump by calling kernreboot with the RBDUMP flag. This then leads to doadump being called, which will dump a rather tiny amount of information about the kernel image itself to some storage device. On ps4, the replacement for doadump is mdbgrundump, which can be called from panic or directly from trapfatal. The amount of information stored into the dump is gigantic by comparison - kernel state for all process, thread, and vm objects are included, along with some metadata about loaded libraries. Other obvious changes from the vanilla FreeBSD method are that the mdbgrun_dump encodes data recorded into the dump on a field-by-field basis and additionally encrypts the resulting buffer before finally storing it to disk. Dumping Anything Let's zoom in to a special part of mdbgrundump - where it iterates over all process' threads and tries to dump some pthread state: dumpstate is a temporary buffer which will eventually make it into the crashdump. To summarize, sysdump_internalcallreaduser can be made to function as a read-anywhere oracle. This is because fsbase will point into our (owned) webkit process' usermode address space. Thus, even without changing the actual fsbase value, we may freely change the value of tcbthread, which is stored at fsbase + 0x10. Further, sysdump_internalcall_readuser will happily read from a kernel address and put the result into the dump. We can now put any kernel location into the dump, but we still need to decrypt and decode it… Aside from that, there's also the issue that we may only add 0x10 bytes per thread in this manner… Further reading: Crashdump Crypto Crashdump Decoding Crashdump Automation Triggering the Vulnerability The Fix (Kind of…) Fin Appendix Crashdump Decryptor NXDP Decoder *** ###BSDTW 2017 Conference Recap: Li-Wen Hsu (https://www.freebsdfoundation.org/blog/bsdtw-2017-conference-recap-li-wen-hsu/) BSDTW 2017 Conference Recap: Li-Wen Hsu 12/28/2017 > Last month, we held BSDTW 2017 on November 11-12th, 2017 in Taipei, Taiwan. It was the second largest BSD conference in Taiwan and the first one in this decade. In 2004, the first AsiaBSDCon was also held in Taipei. Then all of the following AsiaBSDCon conferences were held in Tokyo, Japan. (AsiaBSDCon 2018 will be in Tokyo again next year, please submit your talk proposal by December 31th 2017, and attend the conference on March 8th-11th) > We wanted to start small with the first BSDTW because we were not sure how much sponsorship or how many volunteers we might have. BSDTW 2017 was a single track, two-day conference with 11 selected 50 minute presentations and 1 WIP/lightning talk session consisting of 8 short talks. I do regret that we did not have any local presenters this year. It is also a similar problem at AsiaBSDCon. Unsurprisingly, as with AsiaBSDcon, the travel reimbursement took up a large part of the whole conference budget. We do have many good people that work in Asia, but we still need to encourage people to present their work more. > We had over 130 registered attendees, with 30% of them coming from outside of Taiwan. To our knowledge, in recent years, this is the only open source conference in Taiwan to be held entirely in English, and to have such a large portion of international attendees. This is also the first open source conference in Taiwan to focus entirely on operating systems. The attendees included students, professors, engineers or CTOs, and CEOs from technology companies. This is also the first time that GroffTheBSDGoat visited Taiwan! We were surprised that after the silence for so many years, there are still so many people that use and love BSD near us. We saw many old friends, who had “disappeared” for a long time, came back, and were glad to meet many new friends at the conference. I am really happy that this conference was able to bring together these people, from local and abroad. After attending BSD conferences around the world for many years, I feel that the friendship between BSD users is the most important thing in the BSD community, and one of the main reasons people stay. It has been my pleasure to bring this community back to my friends in my homeland. > After the two-day event, I truly understand that bootstrapping a new conference is a very hard job. One with many aspects that you don't even imagine until you're really in the process of planning an event. I now have an even greater respect for all of the conference organizers and realize that we need to have more people help them, to keep these conferences continue to get better and better. Plus, there will always be room for a new conference! > Thanks to the FreeBSD Foundation for being the biggest sponsor of BSDTW 2017 and always being the strongest backend of our community. We are excited about the many local companies and organizations that helped us whether with people, materials or financially. We even had 21 personal sponsors, more than two times the number of other big open source conferences in Taiwan. > As I said in the closing session, I'm not sure if there will be 2nd BSDTW next year. It still depends on the amount of sponsorship and number of volunteers. However, we will definitely hold more smaller meetups in the next year to keep building up the local BSD community. > Finally, in the beginning of this month, we had a “post-conference media workshop” for organizing the media files we collected in the BSDTW 2017. Here are the review article in Traditional Chinese and the photos: https://medium.com/@bsdtw/bsdtw-2017-總回顧-a402788daede (https://medium.com/@bsdtw/bsdtw-2017-總回顧-a402788daede) && https://www.flickr.com/photos/bsdtw/albums/72157689410035911 (https://www.flickr.com/photos/bsdtw/albums/72157689410035911) *** ##News Roundup ###Running FreeBSD on a Lenovo T470s (https://blog.grem.de/pages/t470s.html) Running FreeBSD on the Lenovo T470s ThinkPad > Installing FreeBSD on this machine was super easy. As I couldn't find a comprehensive/encouraging how-to about installing FreeBSD on a recent ThinkPad, I just wrote up the one below. It includes details about my personal setup, which are not required to run FreeBSD on this model, but which are more to my own taste. I still think this can be a quite useful inspiration for others who want to run their own customized configurations. Specs > The system I use has these specifications: Type: 20JS-001EGE CPU: Intel Core i7-6600U, 2x 2.60GHz RAM: 20GB DDR4 SSD: 512GB NVMe Graphics: Intel HD Graphics 520 (IGP), 1x HDMI 1.4 Display: 14", 1920x1080, non-glare, IPS Ports: 3x USB-A 3.0, 1x Thunderbolt 3, 1x Gb LAN Wireless: WLAN 802.11a/b/g/n/ac, Bluetooth 4.1, LTE (Micro-SIM) Cardreader: SD/SDHC/SDXC/MMC Webcam: 0.9 Megapixel Extras: MIL-STD-810G, Pointing Stick, Fingerprint-Reader, Docking port Things that work > Basically everything I care about: Accelerated video Keyboard Touchpad/ClickPad (like expected in a modern laptop) SSD WiFi Sound HDMI out Suspend to RAM Webcam Things that don't work Fingerprint reader Potentially anything I didn't test Battery life is okay, but could be better. Installation of the base system > I used a snapshot release of 12-CURRENT as the basis of my installation, particularly the one of 13th of December 2017. > I dd'ed it onto a memory stick and boot the laptop. I started a standard installation and created an encrypted ZFS pool on nvme0, using encryption, swap encryption and partition scheme "GPT (UEFI)". > After installation, it boots straight up. Ports tree used > All work is based on a head ports tree from about Dec 18, 22:15 CET, which should be more or less r456672. Preferred ClickPad configuration > As I'm not a fan of the the pointing stick, I disabled it in the bios. My final ClickPad configuration will be: Click to click (not tap), no middle button, right button in the lower right corner. As the old synaptics driver doesn't provide good thumb detection, libinput will be used. Check out the laptop list on the FreeBSD wiki for compatibility: (https://wiki.freebsd.org/Laptops/) *** ###FreeBSD desktop LiveCD creator (https://github.com/pkgdemon/comet) Introduction > The purpose of this tool is quickly generate bloat free images containing stock FreeBSD, and supported desktop environments. Features FreeBSD 11.1-RELEASE AMD64 Gnome & KDE desktop environments Hybrid DVD/USB image Screenshots [Gnome LiveCD])https://github.com/pkgdemon/comet/raw/master/screenshots/gnome-livecd.png?raw=true) KDE LiveCD (https://github.com/pkgdemon/comet/raw/master/screenshots/kde-livecd.png?raw=true) System Requirements FreeBSD 11.1, or higher for AMD64 20GB of free disk space 1GB of free memory UFS, or ZFS Initial Setup Install the required packages: pkg install git grub2-pcbsd grub2-efi xorriso Clone the repo: git clone https://www.github.com/pkgdemon/comet Enter the directory for running the LiveCD creator: cd comet/src Credentials for live media > User: liveuser > Password: freebsd *** ###iXsystems StorageCrypter Ransomware: Security Threat or Clickbait? (https://www.ixsystems.com/blog/storagecrypter/) ###pledge() work in progress (https://undeadly.org/cgi?action=article;sid=20171208082246) > I wanted to give an update that a two pledge-related changes are being worked on. The semantics and integration are complicated so it is taking some time. > One is execpromises. This will become the 2nd argument of pledge(). This allows one to set the pledge for the new image after pledge "exec"-allowed execve(). A warning though: utilizing this in software isn't as easy as you might think! The fork+exec + startup sequences needed to be studied quite carefully to ensure the newly-executed child doesn't ask for more than the parent's execpromises. In my experiments such a circumstance is exceedingly common, so the problem is eased by introducing a new pledge feature which allows pledge violations to return ENOSYS or such rather than killing the process. > This feature also needs to be used with great caution (especially in privileged programs) because programs which fail to observe errors may continue operating forward very incorrectly; you've lost the ability to catch it failing, and provide care by fixing the problem. > The other is pledgepaths. The semantics are still being tuned a bit. Before the first call to pledge() in a process, one can pledgepath() directories. Then later after pledge(), file access operations only work if the traversal of the path crosses one of those pre-declared directories (but better make sure you don't move a directory, because the kernel remembers and reasons about the vnode of the directory rather than the path). Something similar is being worked on for files, but we are still adjusting that, as well as a flag parameter for the pledgepath() call which may constrain the operations done on such files. > As such, pledgepath() will become a filesystem containment mechanism unlike chroot() because paths will still be based upon true /. > Patience. *** ###The anatomy of tee program on OpenBSD (http://nanxiao.me/en/the-anatomy-of-tee-program-on-openbsd/) > The tee command is used to read content from standard input and displays it not only in standard output but also saves to other files simultaneously. The source code of tee in OpenBSD is very simple, and I want to give it an analysis: > (1) tee leverages Singlely-linked List defined in sys/queue.h to manage outputted files (including standard output): struct list { SLIST_ENTRY(list) next; int fd; char *name; }; SLIST_HEAD(, list) head; ...... static void add(int fd, char *name) { struct list *p; ...... SLIST_INSERT_HEAD(&head, p, next); } int main(int argc, char *argv[]) { struct list *p; ...... SLIST_INIT(&head); ...... SLIST_FOREACH(p, &head, next) { ...... } } > To understand it easily, I extract the macros from sys/queue.h and created a file which utilizes the marcos: #define SLIST_HEAD(name, type) struct name { struct type *slh_first; /* first element */ } #define SLIST_ENTRY(type) struct { struct type *sle_next; /* next element */ } #define SLIST_FIRST(head) ((head)->slh_first) #define SLIST_END(head) NULL #define SLIST_EMPTY(head) (SLIST_FIRST(head) == SLIST_END(head)) #define SLIST_NEXT(elm, field) ((elm)->field.sle_next) #define SLIST_FOREACH(var, head, field) for((var) = SLIST_FIRST(head); (var) != SLIST_END(head); (var) = SLIST_NEXT(var, field)) #define SLIST_INIT(head) { SLIST_FIRST(head) = SLIST_END(head); } #define SLIST_INSERT_HEAD(head, elm, field) do { (elm)->field.sle_next = (head)->slh_first; (head)->slh_first = (elm); } while (0) struct list { SLIST_ENTRY(list) next; int fd; char *name; }; SLIST_HEAD(, list) head; int main(int argc, char *argv[]) { struct list *p; SLIST_INIT(&head); SLIST_INSERT_HEAD(&head, p, next); SLIST_FOREACH(p, &head, next) { } } > Then employed gcc‘s pre-processing function: # gcc -E slist.c # 1 "slist.c" # 1 "" # 1 "" # 1 "slist.c" # 30 "slist.c" struct list { struct { struct list *sle_next; } next; int fd; char *name; }; struct { struct list *slh_first; } head; int main(int argc, char *argv[]) { struct list *p; { ((&head)->slh_first) = NULL; }; do { (p)->next.sle_next = (&head)->slh_first; (&head)->slh_first = (p); } while (0); for((p) = ((&head)->slh_first); (p) != NULL; (p) = ((p)->next.sle_next)) { } } > It becomes clear now! The head node in list contains only 1 member: slhfirst, which points to the first valid node. For the elements in the list, it is embedded with next struct which uses slenext to refer to next buddy. > (2) By default, tee will overwrite the output files. If you want to append it, use -a option, and the code is as following: while (*argv) { if ((fd = open(*argv, O_WRONLY | O_CREAT | (append ? O_APPEND : O_TRUNC), DEFFILEMODE)) == -1) { ...... } ...... } > (3) The next part is the skeleton of saving content to files: while ((rval = read(STDIN_FILENO, buf, sizeof(buf))) > 0) { SLIST_FOREACH(p, &head, next) { n = rval; bp = buf; do { if ((wval = write(p->fd, bp, n)) == -1) { ...... } bp += wval; } while (n -= wval); } } > We need to iterates every opened file descriptor and write contents into it. > (4) Normally, theinterrupt signal will cause tee exit: # tee fdkfkdfjk fdkfkdfjk ^C # > To disable this feature, use -i option: # tee -i fdhfhd fdhfhd ^C^C > The corresponding code is like this: ...... case 'i': (void)signal(SIGINT, SIG_IGN); break; *** ##Beastie Bits What I learned from reading the OpenBSD's network stack source code (https://bijanebrahimi.github.io/blog/openbsds-network-stack-part-1.html) Broadcom BCM43224 and BCM43225 Wi-Fi cards now supported by bwn(4) (https://github.com/freebsd/freebsd/commit/888843e26a4e393f405c1c6cbdfc5b701670d363) Ingo details searching man pages (https://marc.info/?l=openbsd-misc&m=151320195122669&w=2) DTrace & ZFS Being Updated On NetBSD, Moving Away From Old OpenSolaris Code (https://www.phoronix.com/scan.php?page=news_item&px=NetBSD-ZFS-DTrace-Updating) Linux Professional Institute and BSD Certification Group Join Efforts (http://www.lpi.org/articles/linux-professional-institute-and-bsd-certification-group-join-efforts) The FreeBSD Foundation thanks Donors (https://www.freebsdfoundation.org/blog/thank-you-2/) ##Feedback/Questions Alex - My first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) ***
We look at how Netflix serves 100 Gbps from an Open Connect Appliance, read through the 2nd quarter FreeBSD status report, show you a freebsd-update speedup via nginx reverse proxy, and customize your OpenBSD default shell. This episode was brought to you by Headlines Serving 100 Gbps from an Open Connect Appliance (https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99) In the summer of 2015, the Netflix Open Connect CDN team decided to take on an ambitious project. The goal was to leverage the new 100GbE network interface technology just coming to market in order to be able to serve at 100 Gbps from a single FreeBSD-based Open Connect Appliance (OCA) using NVM Express (NVMe)-based storage. At the time, the bulk of our flash storage-based appliances were close to being CPU limited serving at 40 Gbps using single-socket Xeon E5–2697v2. The first step was to find the CPU bottlenecks in the existing platform while we waited for newer CPUs from Intel, newer motherboards with PCIe Gen3 x16 slots that could run the new Mellanox 100GbE NICs at full speed, and for systems with NVMe drives. Fake NUMA Normally, most of an OCA's content is served from disk, with only 10–20% of the most popular titles being served from memory (see our previous blog, Content Popularity for Open Connect (https://medium.com/@NetflixTechBlog/content-popularity-for-open-connect-b86d56f613b) for details). However, our early pre-NVMe prototypes were limited by disk bandwidth. So we set up a contrived experiment where we served only the very most popular content on a test server. This allowed all content to fit in RAM and therefore avoid the temporary disk bottleneck. Surprisingly, the performance actually dropped from being CPU limited at 40 Gbps to being CPU limited at only 22 Gbps! The ultimate solution we came up with is what we call “Fake NUMA”. This approach takes advantage of the fact that there is one set of page queues per NUMA domain. All we had to do was to lie to the system and tell it that we have one Fake NUMA domain for every 2 CPUs. After we did this, our lock contention nearly disappeared and we were able to serve at 52 Gbps (limited by the PCIe Gen3 x8 slot) with substantial CPU idle time. After we had newer prototype machines, with an Intel Xeon E5 2697v3 CPU, PCIe Gen3 x16 slots for 100GbE NIC, and more disk storage (4 NVMe or 44 SATA SSD drives), we hit another bottleneck, also related to a lock on a global list. We were stuck at around 60 Gbps on this new hardware, and we were constrained by pbufs. Our first problem was that the list was too small. We were spending a lot of time waiting for pbufs. This was easily fixed by increasing the number of pbufs allocated at boot time by increasing the kern.nswbuf tunable. However, this update revealed the next problem, which was lock contention on the global pbuf mutex. To solve this, we changed the vnode pager (which handles paging to files, rather than the swap partition, and hence handles all sendfile() I/O) to use the normal kernel zone allocator. This change removed the lock contention, and boosted our performance into the 70 Gbps range. As noted above, we make heavy use of the VM page queues, especially the inactive queue. Eventually, the system runs short of memory and these queues need to be scanned by the page daemon to free up memory. At full load, this was happening roughly twice per minute. When this happened, all NGINX processes would go to sleep in vm_wait() and the system would stop serving traffic while the pageout daemon worked to scan pages, often for several seconds. This problem is actually made progressively worse as one adds NUMA domains, because there is one pageout daemon per NUMA domain, but the page deficit that it is trying to clear is calculated globally. So if the vm pageout daemon decides to clean, say 1GB of memory and there are 16 domains, each of the 16 pageout daemons will individually attempt to clean 1GB of memory. To solve this problem, we decided to proactively scan the VM page queues. In the sendfile path, when allocating a page for I/O, we run the pageout code several times per second on each VM domain. The pageout code is run in its lightest-weight mode in the context of one unlucky NGINX process. Other NGINX processes continue to run and serve traffic while this is happening, so we can avoid bursts of pager activity that blocks traffic serving. Proactive scanning allowed us to serve at roughly 80 Gbps on the prototype hardware. Hans Petter Selasky, Mellanox's 100GbE driver developer, came up with an innovative solution to our problem. Most modern NICs will supply an Receive Side Scaling (RSS) hash result to the host. RSS is a standard developed by Microsoft wherein TCP/IP traffic is hashed by source and destination IP address and/or TCP source and destination ports. The RSS hash result will almost always uniquely identify a TCP connection. Hans' idea was that rather than just passing the packets to the LRO engine as they arrive from the network, we should hold the packets in a large batch, and then sort the batch of packets by RSS hash result (and original time of arrival, to keep them in order). After the packets are sorted, packets from the same connection are adjacent even when they arrive widely separated in time. Therefore, when the packets are passed to the FreeBSD LRO routine, it can aggregate them. With this new LRO code, we were able to achieve an LRO aggregation rate of over 2 packets per aggregation, and were able to serve at well over 90 Gbps for the first time on our prototype hardware for mostly unencrypted traffic. So the job was done. Or was it? The next goal was to achieve 100 Gbps while serving only TLS-encrypted streams. By this point, we were using hardware which closely resembles today's 100GbE flash storage-based OCAs: four NVMe PCIe Gen3 x4 drives, 100GbE ethernet, Xeon E5v4 2697A CPU. With the improvements described in the Protecting Netflix Viewing Privacy at Scale blog entry, we were able to serve TLS-only traffic at roughly 58 Gbps. In the lock contention problems we'd observed above, the cause of any increased CPU use was relatively apparent from normal system level tools like flame graphs, DTrace, or lockstat. The 58 Gbps limit was comparatively strange. As before, the CPU use would increase linearly as we approached the 58 Gbps limit, but then as we neared the limit, the CPU use would increase almost exponentially. Flame graphs just showed everything taking longer, with no apparent hotspots. We finally had a hunch that we were limited by our system's memory bandwidth. We used the Intel® Performance Counter Monitor Tools to measure the memory bandwidth we were consuming at peak load. We then wrote a simple memory thrashing benchmark that used one thread per core to copy between large memory chunks that did not fit into cache. According to the PCM tools, this benchmark consumed the same amount of memory bandwidth as our OCA's TLS-serving workload. So it was clear that we were memory limited. At this point, we became focused on reducing memory bandwidth usage. To assist with this, we began using the Intel VTune profiling tools to identify memory loads and stores, and to identify cache misses. Because we are using sendfile() to serve data, encryption is done from the virtual memory page cache into connection-specific encryption buffers. This preserves the normal FreeBSD page cache in order to allow serving of hot data from memory to many connections. One of the first things that stood out to us was that the ISA-L encryption library was using half again as much memory bandwidth for memory reads as it was for memory writes. From looking at VTune profiling information, we saw that ISA-L was somehow reading both the source and destination buffers, rather than just writing to the destination buffer. We realized that this was because the AVX instructions used by ISA-L for encryption on our CPUs worked on 256-bit (32-byte) quantities, whereas the cache line size was 512-bits (64 bytes)?—?thus triggering the system to do read-modify-writes when data was written. The problem is that the the CPU will normally access the memory system in 64 byte cache line-sized chunks, reading an entire 64 bytes to access even just a single byte. After a quick email exchange with the ISA-L team, they provided us with a new version of the library that used non-temporal instructions when storing encryption results. Non-temporals bypass the cache, and allow the CPU direct access to memory. This meant that the CPU was no longer reading from the destination buffers, and so this increased our bandwidth from 58 Gbps to 65 Gbps. At 100 Gbps, we're moving about 12.5 GB/s of 4K pages through our system unencrypted. Adding encryption doubles that to 25 GB/s worth of 4K pages. That's about 6.25 Million mbufs per second. When you add in the extra 2 mbufs used by the crypto code for TLS metadata at the beginning and end of each TLS record, that works out to another 1.6M mbufs/sec, for a total of about 8M mbufs/second. With roughly 2 cache line accesses per mbuf, that's 128 bytes * 8M, which is 1 GB/s (8 Gbps) of data that is accessed at multiple layers of the stack (alloc, free, crypto, TCP, socket buffers, drivers, etc). At this point, we're able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack. However, the goalposts keep moving. We've found that when we use more advanced TCP algorithms, such as RACK and BBR, we are still a bit short of our goal. We have several ideas that we are currently pursuing, which range from optimizing the new TCP code to increasing the efficiency of LRO to trying to do encryption closer to the transfer of the data (either from the disk, or to the NIC) so as to take better advantage of Intel's DDIO and save memory bandwidth. FreeBSD April to June 2017 Status Report (https://www.freebsd.org/news/status/report-2017-04-2017-06.html) FreeBSD Team Reports FreeBSD Release Engineering Team Ports Collection The FreeBSD Core Team The FreeBSD Foundation The Postmaster Team Projects 64-bit Inode Numbers Capability-Based Network Communication for Capsicum/CloudABI Ceph on FreeBSD DTS Updates Kernel Coda revival FreeBSD Driver for the Annapurna Labs ENA Intel 10G Driver Update pNFS Server Plan B Architectures FreeBSD on Marvell Armada38x FreeBSD/arm64 Userland Programs DTC Using LLVM's LLD Linker as FreeBSD's System Linker Ports A New USES Macro for Porting Cargo-Based Rust Applications GCC (GNU Compiler Collection) GNOME on FreeBSD KDE on FreeBSD New Port: FRRouting PHP Ports: Help Improving QA Rust sndio Support in the FreeBSD Ports Collection TensorFlow Updating Port Metadata for non-x86 Architectures Xfce on FreeBSD Documentation Absolute FreeBSD, 3rd Edition Doc Version Strings Improved by Their Absence New Xen Handbook Section Miscellaneous BSD Meetups at Rennes (France) Third-Party Projects HardenedBSD DPDK, VPP, and the future of pfSense @ the DPDK Summit (https://www.pscp.tv/DPDKProject/1dRKZnleWbmKB?t=5h1m0s) The DPDK (Data Plane Development Kit) conference included a short update from the pfSense project The video starts with a quick introduction to pfSense and the company behind it It covers the issues they ran into trying to scale to 10gbps and beyond, and some of the solutions they tried: libuinet, netmap, packet-journey Then they discovered VPP (Vector Packet Processing) The video then covers the architecture of the new pfSense pfSense has launched of EC2, on Azure soon, and will launch support for the new Atom C3000 and Xeon hardware with built-in QAT (Quick-Assist crypto offload) in November The future: 100gbps, MPLS, VXLANs, and ARM64 hardware support *** News Roundup Local nginx reverse proxy cache for freebsd-update (https://wiki.freebsd.org/VladimirKrstulja/Guides/FreeBSDUpdateReverseProxy) Vladimir Krstulja has created this interesting tutorial on the FreeBSD wiki about a freebsd-update reverse proxy cache Either because you're a good netizen and don't want to repeatedly hammer the FreeBSD mirrors to upgrade all your systems, or you want to benefit from the speed of having a local "mirror" (cache, more precisely), running a freebsd update reverse proxy cache with, say, nginx is dead simple. 1. Install nginx somewhere 2. Configure nginx for a subdomain, say, freebsd-update.example.com 3. On all your hosts, in all your jails, configure /etc/freebsd-update.conf for new ServerName And... that's it. Running freebsd-update will use the ServerName domain which is your reverse nginx proxy. Note the comment about using a "nearby" server is not quite true. FreeBSD update mirrors are frequently slow and running such a reverse proxy cache significantly speeds things up. Caveats: This is a simple cache. That means it doesn't consider the files as a whole repository, which in turn means updates to your cache are not atomic. It'd be advised to nuke your cache before your update run, as its point is only to retain the files in a local cache for some short period of time required for all your machines to be updated. ClonOS is a free, open-source FreeBSD-based platform for virtual environment creation and management (https://clonos.tekroutine.com/) The operating system uses FreeBSD's development branch (12.0-CURRENT) as its base. ClonOS uses ZFS as the default file system and includes web-based administration tools for managing virtual machines and jails. The project's website also mentions the availability of templates for quickly setting up new containers and web-based VNC access to jails. Puppet, we are told, can be used for configuration management. ClonOS can be downloaded as a disk image file (IMG) or as an optical media image (ISO). I downloaded the ISO file which is 1.6GB in size. Booting from ClonOS's media displays a text console asking us to select the type of text terminal we are using. There are four options and most people can probably safely take the default, xterm, option. The operating system, on the surface, appears to be a full installation of FreeBSD 12. The usual collection of FreeBSD packages are available, including manual pages, a compiler and the typical selection of UNIX command line utilities. The operating system uses ZFS as its file system and uses approximately 3.3GB of disk space. ClonOS requires about 50MB of active memory and 143MB of wired memory before any services or jails are created. Most of the key features of ClonOS, the parts which set it apart from vanilla FreeBSD, can be accessed through a web-based control panel. When we connect to this control panel, over a plain HTTP connection, using our web browser, we are not prompted for an account name or password. The web-based interface has a straight forward layout. Down the left side of the browser window we find categories of options and controls. Over on the right side of the window are the specific options or controls available in the selected category. At the top of the page there is a drop-down menu where we can toggle the displayed language between English and Russian, with English being the default. There are twelve option screens we can access in the ClonOS interface and I want to quickly give a summary of each one: Overview - this page shows a top-level status summary. The page lists the number of jails and nodes in the system. We are also shown the number of available CPU cores and available RAM on the system. Jail containers - this page allows us to create and delete jails. We can also change some basic jail settings on this page, adjusting the network configuration and hostname. Plus we can click a button to open a VNC window that allows us to access the jail's command line interface. Template for jails - provides a list of available jail templates. Each template is listed with its name and a brief description. For example, we have a Wordpress template and a bittorrent template. We can click a listed template to create a new jail with a vanilla installation of the selected software included. We cannot download or create new templates from this page. Bhyve VMs - this page is very much like the Jails containers page, but concerns the creation of new virtual machines and managing them. Virtual Private Network - allows for the management of subnets Authkeys - upload security keys for something, but it is not clear for what these keys will be used. Storage media - upload ISO files that will be used when creating virtual machines and installing an operating system in the new virtual environment. FreeBSD Bases - I think this page downloads and builds source code for alternative versions of FreeBSD, but I am unsure and could not find any associated documentation for this page. FreeBSD Sources - download source code for various versions of FreeBSD. TaskLog - browse logs of events, particularly actions concerning jails. SQLite admin - this page says it will open an interface for managing a SQLite database. Clicking link on the page gives a file not found error. Settings - this page simply displays a message saying the settings page has not been implemented yet. While playing with ClonOS, I wanted to perform a couple of simple tasks. I wanted to use the Wordpress template to set up a blog inside a jail. I wanted a generic, empty jail in which I could play and run commands without harming the rest of the operating system. I also wanted to try installing an operating system other than FreeBSD inside a Bhyve virtual environment. I thought this would give me a pretty good idea of how quick and easy ClonOS would make common tasks. Conclusions ClonOS appears to be in its early stages of development, more of a feature preview or proof-of-concept than a polished product. A few of the settings pages have not been finished yet, the web-based controls for jails are unable to create jails that connect to the network and I was unable to upload even small ISO files to create virtual machines. The project's website mentions working with Puppet to handle system configuration, but I did not encounter any Puppet options. There also does not appear to be any documentation on using Puppet on the ClonOS platform. One of the biggest concerns I had was the lack of security on ClonOS. The web-based control panel and terminal both automatically login as the root user. Passwords we create for our accounts are ignored and we cannot logout of the local terminal. This means anyone with physical access to the server automatically gains root access and, in addition, anyone on our local network gets access to the web-based admin panel. As it stands, it would not be safe to install ClonOS on a shared network. Some of the ideas present are good ones. I like the idea of jail templates and have used them on other systems. The graphical Bhyve tools could be useful too, if the limitations of the ISO manager are sorted out. But right now, ClonOS still has a way to go before it is likely to be safe or practical to use. Customize ksh display for OpenBSD (http://nanxiao.me/en/customize-ksh-display-for-openbsd/) The default shell for OpenBSD is ksh, and it looks a little monotonous. To make its user-experience more friendly, I need to do some customizations: (1) Modify the “Prompt String” to display the user name and current directory: PS1='$USER:$PWD# ' (2) Install colorls package: pkg_add colorls Use it to replace the shipped ls command: alias ls='colorls -G' (3) Change LSCOLORS environmental variable to make your favorite color. For example, I don't want the directory is displayed in default blue, change it to magenta: LSCOLORS=fxexcxdxbxegedabagacad For detailed explanation of LSCOLORS, please refer manual of colorls. This is my final modification of .profile: PS1='$USER:$PWD# ' export PS1 LSCOLORS=fxexcxdxbxegedabagacad export LSCOLORS alias ls='colorls -G' DragonFly 5 release candidate (https://www.dragonflydigest.com/2017/10/02/20295.html) Commit (http://lists.dragonflybsd.org/pipermail/commits/2017-September/626463.html) I tagged DragonFly 5.0 (commit message list in that link) over the weekend, and there's a 5.0 release candidate for download (http://mirror-master.dragonflybsd.org/iso-images/). It's RC2 because the recent Radeon changes had to be taken out. (http://lists.dragonflybsd.org/pipermail/commits/2017-September/626476.html) Beastie Bits Faster forwarding (http://www.grenadille.net/post/2017/08/21/Faster-forwarding) DRM-Next-Kmod hits the ports tree (http://www.freshports.org/graphics/drm-next-kmod/) OpenBSD Community Goes Platinum (https://undeadly.org/cgi?action=article;sid=20170829025446) Setting up iSCSI on TrueOS and FreeBSD12 (https://www.youtube.com/watch?v=4myESLZPXBU) *** Feedback/Questions Christopher - Virtualizing FreeNAS (http://dpaste.com/38G99CK#wrap) Van - Tar Question (http://dpaste.com/3MEPD3S#wrap) Joe - Book Reviews (http://dpaste.com/0T623Z6#wrap) ***
The costs of open sourcing a project are explored, we discover why PS4 downloads are so slow, delve into the history of UNIX man pages, and more. This episode was brought to you by Headlines The Cost Of Open Sourcing Your Project (https://meshedinsights.com/2016/09/20/open-source-unlikely-to-be-abandonware/) Accusing a company of “dumping” their project as open source is probably misplaced – it's an expensive business no-one would do frivolously. If you see an active move to change software licensing or governance, it's likely someone is paying for it and thus could justify the expense to an executive. A Little History Some case study cameos may help. From 2004 onwards, Sun Microsystems had a policy of all its software moving to open source. The company migrated almost all products to open source licenses, and had varying degrees of success engaging communities around the various projects, largely related to the outlooks of the product management and Sun developers for the project. Sun occasionally received requests to make older, retired products open source. For example, Sun acquired a company called Lighthouse Design which created a respected suite of office productivity software for Steve Jobs' NeXT platform. Strategy changes meant that software headed for the vault (while Jonathan Schwartz, a founder of Lighthouse, headed for the executive suite). Members of the public asked if Sun would open source some of this software, but these requests were declined because there was no business unit willing to fund the move. When Sun was later bought by Oracle, a number of those projects that had been made open source were abandoned. “Abandoning” software doesn't mean leaving it for others; it means simply walking away from wherever you left it. In the case of Sun's popular identity middleware products, that meant Oracle let the staff go and tried to migrate customers to other products, while remaining silent in public on the future of the project. But the code was already open source, so the user community was able to pick up the pieces and carry on, with help from Forgerock. It costs a lot of money to open source a mature piece of commercial software, even if all you are doing is “throwing a tarball over the wall”. That's why companies abandoning software they no longer care about so rarely make it open source, and those abandoning open source projects rarely move them to new homes that benefit others. If all you have thought about is the eventual outcome, you may be surprised how expensive it is to get there. Costs include: For throwing a tarball over the wall: Legal clearance. Having the right to use the software is not the same as giving everyone in the world an unrestricted right to use it and create derivatives. Checking every line of code to make sure you have the rights necessary to release under an OSI-approved license is a big task requiring high-value employees on the “liberation team”. That includes both developers and lawyers; neither come cheap. Repackaging. To pass it to others, a self-contained package containing all necessary source code, build scripts and non-public source and tool dependencies has to be created since it is quite unlikely to exist internally. Again, the liberation team will need your best developers. Preserving provenance. Just because you have confidence that you have the rights to the code, that doesn't mean anyone else will. The version control system probably contains much of the information that gives confidence about who wrote which code, so the repackaging needs to also include a way to migrate the commit information. Code cleaning. The file headers will hopefully include origin information but the liberation team had better check. They also need to check the comments for libel and profanities, not to mention trade secrets (especially those from third parties) and other IP issues. For a sustainable project, all the above plus: Compliance with host governance. It is a fantastic idea to move your project to a host like Apache, Conservancy, Public Software and so on. But doing so requires preparatory work. As a minimum you will need to negotiate with the new host organisation, and they may well need you to satisfy their process requirements. Paperwork obviously, but also the code may need conforming copyright statements and more. That's more work for your liberation team. Migration of rights. Your code has an existing community who will need to migrate to your new host. That includes your staff – they are community too! They will need commit rights, governance rights, social media rights and more. Your liberation team will need your community manager, obviously, but may also need HR input. Endowment. Keeping your project alive will take money. It's all been coming from you up to this point, but if you simply walk away before the financial burden has been accepted by the new community and hosts there may be a problem. You should consider making an endowment to your new host to pay for their migration costs plus the cost of hosting the community for at least a year. Marketing. Explaining the move you are making, the reasons why you are making it and the benefits for you and the community is important. If you don't do it, there are plenty of trolls around who will do it for you. Creating a news blog post and an FAQ — the minimum effort necessary — really does take someone experienced and you'll want to add such a person to your liberation team. Motivations There has to be some commercial reason that makes the time, effort and thus expense worth incurring. Some examples of motivations include: Market Strategy. An increasing number of companies are choosing to create substantial, openly-governed open source communities around software that contributes to their business. An open multi-stakeholder co-developer community is an excellent vehicle for innovation at the lowest cost to all involved. As long as your market strategy doesn't require creating artificial scarcity. Contract with a third party. While the owner of the code may no longer be interested, there may be one or more parties to which they owe a contractual responsibility. Rather than breaching that contract, or buying it out, a move to open source may be better. Some sources suggest a contractual obligation to IBM was the reason Oracle abandoned OpenOffice.org by moving it over to the Apache Software Foundation for example. Larger dependent ecosystem. You may have no further use for the code itself, but you may well have other parts of your business which depend on it. If they are willing to collectively fund development you might consider an “inner source” strategy which will save you many of the costs above. But the best way to proceed may well be to open the code so your teams and those in other companies can fund the code. Internal politics. From the outside, corporations look monolithic, but from the inside it becomes clear they are a microcosm of the market in which they exist. As a result, they have political machinations that may be addressed by open source. One of Oracle's motivations for moving NetBeans to Apache seems to have been political. Despite multiple internal groups needing it to exist, the code was not generating enough direct revenue to satisfy successive executive owners, who allegedly tried to abandon it on more than one occasion. Donating it to Apache meant that couldn't happen again. None of this is to say a move to open source guarantees the success of a project. A “Field of Dreams” strategy only works in the movies, after all. But while it may be tempting to look at a failed corporate liberation and describe it as “abandonware”, chances are it was intended as nothing of the kind. Why PS4 downloads are so slow (https://www.snellman.net/blog/archive/2017-08-19-slow-ps4-downloads/) From the blog that brought us “The origins of XXX as FIXME (https://www.snellman.net/blog/archive/2017-04-17-xxx-fixme/)” and “The mystery of the hanging S3 downloads (https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/)”, this week it is: “Why are PS4 downloads so slow?” Game downloads on PS4 have a reputation of being very slow, with many people reporting downloads being an order of magnitude faster on Steam or Xbox. This had long been on my list of things to look into, but at a pretty low priority. After all, the PS4 operating system is based on a reasonably modern FreeBSD (9.0), so there should not be any crippling issues in the TCP stack. The implication is that the problem is something boring, like an inadequately dimensioned CDN. But then I heard that people were successfully using local HTTP proxies as a workaround. It should be pretty rare for that to actually help with download speeds, which made this sound like a much more interesting problem. Before running any experiments, it's good to have a mental model of how the thing we're testing works, and where the problems might be. If nothing else, it will guide the initial experiment design. The speed of a steady-state TCP connection is basically defined by three numbers. The amount of data the client is will to receive on a single round-trip (TCP receive window), the amount of data the server is willing to send on a single round-trip (TCP congestion window), and the round trip latency between the client and the server (RTT). To a first approximation, the connection speed will be: speed = min(rwin, cwin) / RTT With this model, how could a proxy speed up the connection? The speed through the proxy should be the minimum of the speed between the client and proxy, and the proxy and server. It should only possibly be slower With a local proxy the client-proxy RTT will be very low; that connection is almost guaranteed to be the faster one. The improvement will have to be from the server-proxy connection being somehow better than the direct client-server one. The RTT will not change, so there are just two options: either the client has a much smaller receive window than the proxy, or the client is somehow causing the server's congestion window to decrease. (E.g. the client is randomly dropping received packets, while the proxy isn't). After setting up a test rig, where the PS4's connection was bridged through a linux box so packets could be captured, and artificial latency could be added, some interested results came up: The differences in receive windows at different times are striking. And more important, the changes in the receive windows correspond very well to specific things I did on the PS4 When the download was started, the game Styx: Shards of Darkness was running in the background (just idling in the title screen). The download was limited by a receive window of under 7kB. This is an incredibly low value; it's basically going to cause the downloads to take 100 times longer than they should. And this was not a coincidence, whenever that game was running, the receive window would be that low. Having an app running (e.g. Netflix, Spotify) limited the receive window to 128kB, for about a 5x reduction in potential download speed. Moving apps, games, or the download window to the foreground or background didn't have any effect on the receive window. Playing an online match in a networked game (Dreadnought) caused the receive window to be artificially limited to 7kB. I ran a speedtest at a time when downloads were limited to 7kB receive window. It got a decent receive window of over 400kB; the conclusion is that the artificial receive window limit appears to only apply to PSN downloads. When a game was started (causing the previously running game to be stopped automatically), the receive window could increase to 650kB for a very brief period of time. Basically it appears that the receive window gets unclamped when the old game stops, and then clamped again a few seconds later when the new game actually starts up. I did a few more test runs, and all of them seemed to support the above findings. The only additional information from that testing is that the rest mode behavior was dependent on the PS4 settings. Originally I had it set up to suspend apps when in rest mode. If that setting was disabled, the apps would be closed when entering in rest mode, and the downloads would proceed at full speed. The PS4 doesn't make it very obvious exactly what programs are running. For games, the interaction model is that opening a new game closes the previously running one. This is not how other apps work; they remain in the background indefinitely until you explicitly close them. So, FreeBSD and its network stack are not to blame Sony used a poor method to try to keep downloads from interfering with your gameplay The impact of changing the receive window is highly dependant upon RTT, so it doesn't work as evenly as actual traffic shaping or queueing would. An interesting deep dive, it is well worth reading the full article and checking out the graphs *** OpenSSH 7.6 Released (http://www.openssh.com/releasenotes.html#7.6) From the release notes: This release includes a number of changes that may affect existing configurations: ssh(1): delete SSH protocol version 1 support, associated configuration options and documentation. ssh(1)/sshd(8): remove support for the hmac-ripemd160 MAC. ssh(1)/sshd(8): remove support for the arcfour, blowfish and CAST Refuse RSA keys
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
In this episode, we take a look at the reimplementation of NetBSD using a Microkernel, check out what makes DHCP faster, and see what high-process count support for DragonflyBSD has to offer, and we answer the questions you've always wanted to ask us. This episode was brought to you by Headlines A Reimplementation Of Netbsd Using a Microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Minix author Andy Tanenbaum writes in Part 1 of a-reimplementation-of-netbsd-using-a-microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project. Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running. The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this. The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project. Part 2 (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-2-of-2/) is also available. *** Rapid DHCP: Or, how do Macs get on the network so fast? (https://cafbit.com/post/rapid_dhcp_or_how_do/) One of life's minor annoyances is having to wait on my devices to connect to the network after I wake them from sleep. All too often, I'll open the lid on my EeePC netbook, enter a web address, and get the dreaded "This webpage is not available" message because the machine is still working on connecting to my Wi-Fi network. On some occasions, I have to twiddle my thumbs for as long as 10-15 seconds before the network is ready to be used. The frustrating thing is that I know it doesn't have to be this way. I know this because I have a Mac. When I open the lid of my MacBook Pro, it connects to the network nearly instantaneously. In fact, no matter how fast I am, the network comes up before I can even try to load a web page. My curiosity got the better of me, and I set out to investigate how Macs are able to connect to the network so quickly, and how the network connect time in other operating systems could be improved. I figure there are three main categories of time-consuming activities that occur during network initialization: Link establishment. This is the activity of establishing communication with the network's link layer. In the case of Wi-Fi, the radio must be powered on, the access point detected, and the optional encryption layer (e.g. WPA) established. After link establishment, the device is able to send and receive Ethernet frames on the network. Dynamic Host Configuration Protocol (DHCP). Through DHCP handshaking, the device negotiates an IP address for its use on the local IP network. A DHCP server is responsible for managing the IP addresses available for use on the network. Miscellaneous overhead. The operating system may perform any number of mundane tasks during the process of network initialization, including running scripts, looking up preconfigured network settings in a local database, launching programs, etc. My investigation thus far is primarily concerned with the DHCP phase, although the other two categories would be interesting to study in the future. I set up a packet capture environment with a spare wireless access point, and observed the network activity of a number of devices as they initialized their network connection. For a worst-case scenario, let's look at the network activity captured while an Android tablet is connecting: This tablet, presumably in the interest of "optimization", is initially skipping the DHCP discovery phase and immediately requesting its previous IP address. The only problem is this is a different network, so the DHCP server ignores these requests. After about 4.5 seconds, the tablet stubbornly tries again to request its old IP address. After another 4.5 seconds, it resigns itself to starting from scratch, and performs the DHCP discovery needed to obtain an IP address on the new network. In all fairness, this delay wouldn't be so bad if the device was connecting to the same network as it was previously using. However, notice that the tablet waits a full 1.13 seconds after link establishment to even think about starting the DHCP process. Engineering snappiness usually means finding lots of small opportunities to save a few milliseconds here and there, and someone definitely dropped the ball here. In contrast, let's look at the packet dump from the machine with the lightning-fast network initialization, and see if we can uncover the magic that is happening under the hood: The key to understanding the magic is the first three unicast ARP requests. It looks like Mac OS remembers certain information about not only the last connected network, but the last several networks. In particular, it must at least persist the following tuple for each of these networks: > 1. The Ethernet address of the DHCP server > 2. The IP address of the DHCP server > 3. Its own IP address, as assigned by the DHCP server During network initialization, the Mac transmits carefully crafted unicast ARP requests with this stored information. For each network in its memory, it attempts to send a request to the specific Ethernet address of the DHCP server for that network, in which it asks about the server's IP address, and requests that the server reply to the IP address which the Mac was formerly using on that network. Unless network hosts have been radically shuffled around, at most only one of these ARP requests will result in a response—the request corresponding to the current network, if the current network happens to be one of the remembered networks. This network recognition technique allows the Mac to very rapidly discover if it is connected to a known network. If the network is recognized (and presumably if the Mac knows that the DHCP lease is still active), it immediately and presumptuously configures its IP interface with the address it knows is good for this network. (Well, it does perform a self-ARP for good measure, but doesn't seem to wait more than 13ms for a response.) The DHCP handshaking process begins in the background by sending a DHCP request for its assumed IP address, but the network interface is available for use during the handshaking process. If the network was not recognized, I assume the Mac would know to begin the DHCP discovery phase, instead of sending blind requests for a former IP address as the Galaxy Tab does. The Mac's rapid network initialization can be credited to more than just the network recognition scheme. Judging by the use of ARP (which can be problematic to deal with in user-space) and the unusually regular transmission intervals (a reliable 1.0ms delay between each packet sent), I'm guessing that the Mac's DHCP client system is entirely implemented as tight kernel-mode code. The Mac began the IP interface initialization process a mere 10ms after link establishment, which is far faster than any other device I tested. Android devices such as the Galaxy Tab rely on the user-mode dhclient system (part of the dhcpcd package) dhcpcd program, which no doubt brings a lot of additional overhead such as loading the program, context switching, and perhaps even running scripts. The next step for some daring kernel hacker is to implement a similarly aggressive DHCP client system in the Linux kernel, so that I can enjoy fast sign-on speeds on my Android tablet, Android phone, and Ubuntu netbook. There already exists a minimal DHCP client implementation in the Linux kernel, but it lacks certain features such as configuring the DNS nameservers. Perhaps it wouldn't be too much work to extend this code to support network recognition and interface with a user-mode daemon to handle such auxillary configuration information received via DHCP. If I ever get a few spare cycles, maybe I'll even take a stab at it. You can also find other ways of optimizing the dhclient program and how it works in the dhclient tutorial on Calomel.org (https://calomel.org/dhclient.html). *** BSDCam Trip Report (https://www.freebsdfoundation.org/blog/bsdcam-2017-trip-report-michael-lucas/) Over the decades, FreeBSD development and coordination has shifted from being purely on-line to involving more and more in-person coordination and cooperation. The FreeBSD Foundation sponsors a devsummit right before BSDCan, EuroBSDCon, and AsiaBSDCon, so that developers traveling to the con can leverage their airfare and hammer out some problems. Yes, the Internet is great for coordination, but nothing beats a group of developers spending ten minutes together to sketch on a whiteboard and figuring out exactly how to make something bulletproof. In addition to the coordination efforts, though, conference devsummits are hierarchical. There's a rigid schedule, with topics decided in advance. Someone leads the session. Sessions can be highly informative, passionate arguments, or anything in between. BSDCam is… a little different. It's an invaluable part of the FreeBSD ecosystem. However, it's something that I wouldn't normally attend. But right now, is not normal. I'm writing a new edition of Absolute FreeBSD. To my astonishment, people have come to rely on this book when planning their deployments and operations. While I find this satisfying, it also increases the pressure on me to get things correct. When I wrote my first FreeBSD book back in 2000, a dozen mailing lists provided authoritative information on FreeBSD development. One person could read every one of those lists. Today, that's not possible—and the mailing lists are only one narrow aspect of the FreeBSD social system. Don't get me wrong—it's pretty easy to find out what people are doing and how the system works. But it's not that easy to find out what people will be doing and how the system will work. If this book is going to be future-proof, I needed to leave my cozy nest and venture into the wilds of Cambridge, England. Sadly, the BSDCam chair agreed with my logic, so I boarded an aluminum deathtrap—sorry, a “commercial airliner”—and found myself hurtled from Detroit to Heathrow. And one Wednesday morning, I made it to the William Gates building of Cambridge University, consciousness nailed to my body by a thankfully infinite stream of proper British tea. BSDCam attendance is invitation only, and the facilities can only handle fifty folks or so. You need to be actively working on FreeBSD to wrangle an invite. Developers attend from all over the world. Yet, there's no agenda. Robert Watson is the chair, but he doesn't decide on the conference topics. He goes around the room and asks everyone to introduce themselves, say what they're working on, and declare what they want to discuss during the conference. The topics of interest are tallied. The most popular topics get assigned time slots and one of the two big rooms. Folks interested in less popular topics are invited to claim one of the small breakout rooms. Then the real fun begins. I started by eavesdropping in the virtualization workshop. For two hours, people discussed FreeBSD's virtualization needs, strengths, and weaknesses. What needs help? What should this interface look like? What compatibility is important, and what isn't? By the end of the session, the couple dozen people had developed a reasonable consensus and, most importantly, some folks had added items to their to-do lists. Repeat for a dozen more topics. I got a good grip on what's really happening with security mitigation techniques, FreeBSD's cloud support, TCP/IP improvements, advances in teaching FreeBSD, and more. A BSDCan devsummit presentation on packaging the base system is informative, but eavesdropping on two dozen highly educated engineers arguing about how to nail down the final tidbits needed to make that a real thing is far more educational. To my surprise, I was able to provide useful feedback for some sessions. I speak at a lot of events outside of the FreeBSD world, and was able to share much of what I hear at Linux conferences. A tool that works well for an experienced developer doesn't necessarily work well for everyone. Every year, I leave BSDCan tired. I left BSDCam entirely exhausted. These intense, focused discussions stretched my brain. But, I have a really good idea where key parts of FreeBSD development are actually headed. This should help future-proof the new Absolute FreeBSD, as much as any computer book can be future-proof. Plus, BSDCam throws the most glorious conference dinner I've ever seen. I want to thank Robert Watson for his kind invitation, and the FreeBSD Foundation for helping defray the cost of this trip Interview - The BSDNow Crew As a kid, what did you dream of to become as an adult? JT: An Astronaut BR: I wanted to be a private detective, because of all the crime novels that I read back then. I didn't get far with it. However, I think the structured analysis skills (who did what, when, and such) help me in debugging and sysadmin work. AJ: Didn't think about it much How do you manage to stay organized day to day with so much things you're actively doing each day? (Day job, wife/girlfriend, conferences, hobbies, friends, etc.) JT: Who said I was organized? BR: A lot of stuff in my calendar as reminders, open browser tabs as “to read later” list. A few things like task switching when getting stuck helps. Also, focus on a single goal for the day, even though there will be distractions. Slowly, but steadily chip away at the things you're working on. Rather than to procrastinate and put things back to review later, get started early with easy things for a big task and then tackle the hard part. Often, things look totally chaotic and unmanageable, until you start working on them. AJ: I barely manage. Lots of Google Calendar reminders, and the entire wall of my office is covered in whiteboard sheet todo lists. I use pinboard.in to deal with finding and organizing bookmarks. Write things down, don't trust your memory. What hobbies outside of IT do you have? JT: I love photography, but I do that Professional part time, so I'm not sure if that counts as a hobby anymore. I guess it'd have to be working in the garage on my cars. BR: I do Tai Chi to relax once a week in a group, but can also do it alone, pretty much everywhere. Way too much Youtube watching and browsing the web. I did play some games before studying at the university and I'm still proud that I could control it to the bare minimum not to impact my studies. A few “lapses” from time to time, revisiting the old classics since the newer stuff won't run on my machines anyway. Holiday time is pretty much spent for BSD conferences and events, this is where I can relax and talk with like-minded people from around the world, which is fascinating. Plus, it gets me to various places and countries I never would have dared to visit on my own. AJ: I play a few video games, and I like to ski, although I don't go very often as most of my vacation time is spent hanging out with my BSD friends at various conferences How do you relax? JT: What is this word ‘relax' and what does it mean? BR: My Tai Chi plays a big part in it I guess. I really calms you and the constant stream of thoughts for a while. It also gives you better clarity of what's important in life. Watching movies, sleeping long. AJ: Usually watching TV or Movies. Although I have taken to doing most of my TV watching on my exercise bike now, but it is still mentally relaxing If FreeBSD didn't exist, which BSD flavour would you use? Why? JT: I use TrueOS, but if FreeBSD didn't exist, that project might not either… so… My other choice would be HardenedBSD, but since it's also based on FreeBSD I'm in the same dillema. BR: I once installed NetBSD to see what It can do. If FreeBSD wouldn't exist, I would probably try my luck with it. OpenBSD is also appealing, but I've never installed it. AJ: When I started using FreeBSD in 2000, the only other BSD I had heard of at the time was OpenBSD. If FreeBSD wasn't around, I don't think the world would look like it does, so it is hard to speculate. If any of the BSD's weren't around and you had to use Linux, which camp would belong to? (Redhat, SUSE, Debian, Ubuntu, Gentoo?) JT: I learned Linux in the mid 90s using Slackware, which I used consistently up until the mid 2000s, when I joined the PuppyLinux community and eventually became a developer (FYI, Puppy was/is/can be based on Slackware -- its complicated). So I'd go back to using either Slackware or PuppyLinux. BR: I tried various Linux distributions until I landed at Debian. I used is pretty extensively as my desktop OS at home, building custom kernels and packages to install them until I discovered FreeBSD. I ran both side by side for a few months for learning until one day I figured out that I had not booted Debian in a while, so I switched completely. AJ: The first Linux I played with was Slackware, and it is the most BSD like, but the bits of Linux I learned in school were Redhat and so I can somewhat wrap my head around it, although now that they are changing everything to systemd, all of that old knowledge is more harmful than useful. Are you still finding yourself in need to use Windows/Mac OS? Why? JT: I work part time as a professional Photographer, so I do use Windows for my photography work. While I can do everything I need to do in Linux, it comes down to being pragmatic about my time. What takes me several hours to accomplish in Linux I can accomplish in 20 minutes on Windows. BR: I was a long time Windows-only user before my Unix days. But back when Vista was about to come out and I needed a new laptop, my choice was basically learning to cope with Vistas awful features or learn MacOS X. I did the latter, it increased my productivity since it's really a good Unix desktop experience (at least, back then). I only have to use Windows at work from time to time as I manage our Windows Terminal server, which keeps the exposure low enough and I only connect to it to use a certain app not available for the Mac or the BSDs. AJ: I still use Windows to play games, for a lot of video conferencing, and to produce BSD Now. Some of it could be done on BSD but not as easily. I have promised myself that I will switch to 100% BSD rather than upgrade to Windows 10, so we'll see how that goes. Please describe your home networking setup. Router type, router OS, router hardware, network segmentation, wifi apparatus(es), other devices connected, and anything else that might be interesting about your home network. BR: Very simple and boring: Apple Airport Express base station and an AVM FritzBox for DNS, DHCP, and the link to my provider. A long network cable to my desktop machine. That I use less and less often. I just bought an RPI 3 for some home use in the future to replace it. Mostly my brother's and my Macbook Pro's are connected, our phones and the iPad of my mother. AJ: I have a E3-1220 v3 (dual 3.1ghz + HT) with 8 GB of ram, and 4x Intel gigabit server NICs as my router, and it runs vanilla FreeBSD (usually some snapshot of -current). I have 4 different VLANs, Home, Office, DMZ, and Guest WiFi. WiFi is served via a tiny USB powered device I bought in Tokyo years ago, it serves 3 different SSIDs, one for each VLAN except the DMZ. There are ethernet jacks in every room wired for 10 gigabit, although the only machines with 10 gigabit are my main workstation, file server, and some machines in the server rack. There are 3 switches, one for the house (in the laundry room), one for the rack, and one for 10gig stuff. There is a rack in the basement spare bedroom, it has 7 servers in it, mostly storage for live replicas of customer data for my company. How do guys manage to get your work done on FreeBSD desktops? What do you do when you need to a Linux or Windows app that isn't ported, or working? I've made several attempts to switch to FreeBSD, but each attempt failed because of tools not being available (e.g. Zoom, Dropbox, TeamViewer, Crashplan) or broken (e.g. VirtualBox). BR: I use VIrtualBox for everything that is not natively available or Windows-only. Unfortunately, that means no modern games. I mostly do work in the shell when I'm on FreeBSD and when it has to be a graphical application, then I use Fluxbox as the DE. I want to get work done, not look at fancy eye-candy that get's boring after a while. Deactivated the same stuff on my mac due to the same reason. I look for alternative software online, but my needs are relatively easy to satisfy as I'm not doing video editing/rendering and such. AJ: I generally find that I don't need these apps. I use Firefox, Thunderbird, OpenSSH, Quassel, KomodoEdit, and a few other apps, so my needs are not very demanding. It is annoying when packages are broken, but I usually work around this with boot environments, and being able to just roll back to a version that worked for a few days until the problem is solved. I do still have access to a windows machine for the odd time I need specific VPN software or access to Dell/HP etc out-of-band management tools. Which desktop environments are your favorite, and why? For example, I like i3, Xfce, and I'm drawn to Lumina's ethos, but so far always seem to end up back on Xfc because of its ease of use, flexibility, and dashing good looks. JT: As a Lumina Desktop developer, I think my preference is obvious. ;) I am also a long timeOpenBox user, so I have a soft place in my heart for that as well. BR: I use Fluxbox when I need to work with a lot of windows or an application demands X11. KDE and others are too memory heavy for me and I rarely use even 20% of the features they provide. AJ: I was a long time KDE user, but I have adopted Lumina. I find it fast, and that it gets out of my way and lets me do what I want. It had some annoyances early on, but I've nagged the developers into making it work for me. Which command-line shells do you prefer, why, and how (if at all) have you customised the environment or prompt? BR: I use zsh, but without all the fancy stuff you can find online. It might make you more productive, yes. But again, I try to keep things simple. I'm slowly learning tmux and want to work more in it in the future. I sometimes look at other BSD people's laptops and am amazed at what they do with window-management in tmux. My prompt looks like this: bcr@Voyager:~> 20:20 17-08-17 Put this in your .zshrc to get the same result: PROMPT='%n@%m:%~>' RPROMPT='%T %D' AJ: I started using tcsh early on, because it was the shell on the first box I had access to, and because one of the first things I read in “BSD Hacks” was how to enable ‘typo correction”, which made my life a lot better especially on dial up in the early days. My shell prompt looks like this: allan@CA-TOR1-02:/usr/home/allan% What is one thing (or more) missing in FreeBSD you would import from another project or community? Could be tech, process, etc. JT: AUFS from Linux BR: Nohup from Illumos where you can detach an already running process and put it in the background. I often forget that and I'm not in tmux when that happens, so I can see myself use that feature a lot. AJ: Zones (more complete Jails) from IllumOS how do you manage your time to learn about and work on FreeBSD? Does your work/employment enable what you do, or are your contributions mainly done in private time? JT: These days I'm mostly learning things I need for work, so it just falls into something I'm doing while working on work projects. BR: We have a lot of time during the semester holidays to learn on our own, it's part of the idea of being in a university to keep yourself updated, at least for me. Especially in the fast moving world of IT. I also read a lot in my free time. My interests can shift sometimes, but then I devour everything I can find on the topic. Can be a bit excessive, but has gotten me where I am now and I still need a lot to learn (and want to). Since I work with FreeBSD at work (my owndoing), I can try out many things there. AJ: My work means a spend a lot of time working with FreeBSD, but not that much time working ON it. My contributions are mostly done outside of work, but as I own the company I do get more flexibility to take time off for conferences and other FreeBSD related stuff. we know we can bribe Michael W Lucas with gelato (good gelato that is), but what can we use to bribe you guys? Like when I want to have Allan to work on fixing a bug which prevents me from running ZFS on this fancy rock64 board? BR: Desserts of various kinds. AJ: I am probably not the right person to look at your rock64 board. Most people in the project have taken to bribing me with chocolate. In general, my todo list is so long, the best way is a trade, you take this task and I'll take that task. Is your daily mobile device iOS, Android, Windows Mobile, or other? Why? JT: These days I'm using Android on my Blackberry Priv, but until recently I was still a heavy user of Sailfish OS. I would use SailfishOS everyday, if I could find a phone with a keyboard that I could run it on. BR: iOS on the iPhone 7 currently. Never used an Android phone, saw it on other people's devices and what they can do with it (much more). But the infrequent security updates (if any at all) keep me away from it. AJ: I have a Google Nexus 6 (Android 7.1). I wanted the ‘pure' Android experience, and I had been happy with my previous Nexus S. I don't run a custom OS/ROM or anything because I use the phone to verify that video streams work on an ‘average users device'. I am displeased that support for my device will end soon. I am not sure what device I will get next, but it definitely won't be an iPhone. News Roundup Beta Update - Request for (more) Testing (http://undeadly.org/cgi?action=article&sid=20170808065718&mode=flat&count=30) https://beta.undeadly.org/ has received an update. The most significant changes include: The site has been given a less antiquated "look". (As the topic icons have been eliminated, we are no longer seeking help with those graphics.) The site now uses a moderate amount of semantic HTML5. Several bugs in the HTML fragment validator (used for submissions and comments) have been fixed. To avoid generating invalid HTML, submission content which fails validation is no longer displayed in submission/comment previews. Plain text submissions are converted to HTML in a more useful fashion. (Instead of just converting each EOL to , the converter now generates proper paragraphs and interprets two or more consecutive EOLs as indicating a paragraph break.) The redevelopment remains a work-in-progress. Many thanks to those who have contributed! As before, constructive feedback would be appreciated. Of particular interest are reports of bugs in behaviour (for example, in the HTML validator or in authentication) that would preclude the adoption of the current code for the main site. High-process-count support added to master (http://lists.dragonflybsd.org/pipermail/users/2017-August/313552.html) We've fixed a number of bottlenecks that can develop when the number of user processes runs into the tens of thousands or higher. One thing led to another and I said to myself, "gee, we have a 6-digit PID, might as well make it work to a million!". With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. And, in fact, as today's machines start to ratchet up there in both memory capacity and core count, with fast storage (NVMe) and fast networking (10GigE and higher), even in consumer boxes, this is actually something that one might want to do. With AMD's threadripper and EPYC chips now out, the IntelAMD cpu wars are back on! Boasting up to 32 cores (64 threads) per socket and two sockets on EPYC, terabytes of ram, and motherboards with dual 10GigE built-in, the reality is that these numbers are already achievable in a useful manner. In anycase, I've tested these changes on a dual-socket xeon. I can in-fact start 900,000 processes. They don't get a whole lot of cpu and running 'ps' would be painful, but it works and the system is still responsive from the shell with all of that going on. xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 In fact, judging from the memory use, these minimal test processes only eat around 60KB each. 900,000 of them ate only 55GB on a 128GB machine. So even a million processes is not out of the question, depending on the cpu requirements for those processes. Today's modern machines can be stuffed with enormous amounts of memory. Of course, our PIDs are currently limited to 6 digits, so a million is kinda the upper limit in terms of discrete user processes (verses pthreads which are less restricted). I'd rather not go to 7 digits (yet). CFT: Driver for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens (https://lists.freebsd.org/pipermail/freebsd-current/2017-August/066783.html) Following patch [1] adds support for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens via evdev protocol. It is intended to be a native replacement of hid-multitouch.c driver found in Linux distributions and multimedia/webcamd port. Patch is made for 12-CURRENT and most probably can be applied to recent 11-STABLE and 11.1-RELEASE (not tested) How to test" 1. Apply patch [1] 2. To compile this driver into the kernel, place the following lines into your kernel configuration file: device wmt device usb device evdev Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5): wmt_load="YES" 3. Install x11-drivers/xf86-input-evdev or x11-drivers/xf86-input-libinput port 4. Tell XOrg to use evdev or libinput driver for the device: ``` Section "ServerLayout" InputDevice "TouchScreen0" "SendCoreEvents" EndSection Section "InputDevice" Identifier "TouchScreen0" Driver "evdev" # Driver "libinput" Option "Device" "/dev/input/eventXXX" EndSection ``` Exact value of "/dev/input/eventXXX" can be obtained with evemu-record utility from devel/evemu. Note1: Currently, driver does not support pens or touchpads. Note2: wmt.ko should be kld-loaded before uhid driver to take precedence over it! Otherwise uhid can be kld-unloaded after loading of wmt. wmt review: https://reviews.freebsd.org/D12017 Raw diff: https://reviews.freebsd.org/D12017.diff *** Beastie Bits BSDMag Programing Languages Infographic (https://bsdmag.org/programm_history/) t2k17 Hackathon Report: Bob Beck on buffer cache tweaks, libressl and pledge progress (http://undeadly.org/cgi?action=article&sid=20170815171854) New FreeBSD Journal (https://www.freebsdfoundation.org/past-issues/resource-control/) NetBSD machines at Open Source Conference 2017 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2017/08/10/msg000744.html) *** Feedback/Questions Dan - HDD question (http://dpaste.com/3H6TDJV) Benjamin - scrub of death (http://dpaste.com/10F086V) Jason - Router Opinion (http://dpaste.com/2D9102K) Sohrab - Thanks (http://dpaste.com/1XYYTWF) ***
We read a trip report about FreeBSD in China, look at how Unix deals with Signals, a stats collector in DragonFlyBSD & much more! This episode was brought to you by Headlines Trip Report: FreeBSD in China at COPU and LinuxCon (https://www.freebsdfoundation.org/blog/trip-report-freebsd-in-china-at-copu-and-linuxcon/) This trip report is from Deb Goodkin, the Executive Director of the FreeBSD Foundation. She travelled to China in May 2017 to promote FreeBSD, meet with companies, and participate in discussions around Open Source. > In May of 2017, we were invited to give a talk about FreeBSD at COPU's (China Open Source Promotional Unit) Open Source China, Open Source World Summit, which took place June 21-22, in Beijing. This was a tremendous opportunity to talk about the advantages of FreeBSD to the open source leaders and organizations interested in open source. I was honored to represent the Project and Foundation and give the presentation “FreeBSD Advantages and Applications”. > Since I was already going to be in Beijing, and LinuxCon China was being held right before the COPU event, Microsoft invited me to be part of a women-in-tech panel they were sponsoring. There were six of us on the panel including two from Microsoft, one from the Linux Foundation, one from Accenture of China, and one from Women Who Code. Two of us spoke in English, with everyone else speaking Chinese. It was disappointing that we didn't have translators, because I would have loved hearing everyone's answers. We had excellent questions from the audience at the end. I also had a chance to talk with a journalist from Beijing, where I emphasized how contributing to an open source project, like FreeBSD, is a wonderful way to get experience to boost your resume for a job. > The first day of LinuxCon also happened to be FreeBSD Day. I had my posters with me and was thrilled to have the Honorary Chairman of COPU (also known as the “Father of Open Source in China”) hold one up for a photo op. Unfortunately, I haven't been able to get a copy of that photo for proof (I'm still working on it!). We spent a long time discussing the strengths of FreeBSD. He believes there are many applications in China that could benefit from FreeBSD, especially for embedded devices, university research, and open source education. We had more time throughout the week to discuss FreeBSD in more detail. > Since I was at LinuxCon, I had a chance to meet with people from the Linux Foundation, other open source projects, and some of our donors. With LinuxCon changing its name to Open Source Summit, I discussed how important it is to include minority voices like ours to contribute to improving the open source ecosystem. The people I talked to within the Linux Foundation agreed and suggested that we get someone from the Project to give a talk at the Open Source Summit in Prague this October. Jim Zemlin, the Linux Foundation Executive Director, suggested having a BSD track at the summits. We did miss the call for proposals for that conference, but we need to get people to consider submitting proposals for the Open Source Summits in 2018. > I talked to a CTO from a company that donates to us and he brought up his belief that FreeBSD is much easier to get started on as a contributor. He talked about the steep path in Linux to getting contributions accepted due to having over 10,000 developers and the hierarchy of decision makers, from Linus to his main lieutenants to the layers beneath him. It can take 6 months to get your changes in! > On Tuesday, Kylie and I met with a representative from Huawei, who we've been meeting over the phone with over the past few months. Huawei has a FreeBSD contributor and is looking to add more. We were thrilled to hear they decided to donate this year. We look forward to helping them get up to speed with FreeBSD and collaborate with the Project. > Wednesday marked the beginning of COPU and the reason I flew all the way to Beijing! We started the summit with having a group photo of all the speakers:The honorary chairman, Professor Lu in the front middle. > My presentation was called “FreeBSD Advantages and Applications”. A lot of the material came from Foundation Board President, George-Neville-Neil's presentation, “FreeBSD is not a Linux Distribution”, which is a wonderful introduction to FreeBSD and includes the history of FreeBSD, who uses it and why, and which features stand out. My presentation went well, with Professor Lu and others engaged through the translators. Afterwards, I was invited to a VIP dinner, which I was thrilled about. > The only hitch was that Kylie and I were running a FreeBSD meetup that evening, and both were important! Beijing during rush hour is crazy, even trying to go only a couple of miles is challenging. We made plans that I would go to the meetup and give the same presentation, and then head back to the dinner. Amazingly, it worked out. Check out the rest of her trip report and stay tuned for more news from the region as this is one of the focus areas of the Foundation. *** Unix: Dealing with signals (http://www.networkworld.com/article/3211296/linux/unix-dealing-with-signals.html) Signals on Unix systems are critical to the way processes live and die. This article looks at how they're generated, how they work, and how processes receive or block them On Unix systems, there are several ways to send signals to processes—with a kill command, with a keyboard sequence (like control-C), or through a program Signals are also generated by hardware exceptions such as segmentation faults and illegal instructions, timers and child process termination. But how do you know what signals a process will react to? After all, what a process is programmed to do and able to ignore is another issue. Fortunately, the /proc file system makes information about how processes handle signals (and which they block or ignore) accessible with commands like the one shown below. In this command, we're looking at information related to the login shell for the current user, the "$$" representing the current process. On FreeBSD, you can use procstat -i PID to get that and even more information, and easier to digest form P if signal is pending in the global process queue I if signal delivery disposition is SIGIGN C if signal delivery is to catch it Catching a signal requires that a signal handling function exists in the process to handle a given signal. The SIGKILL (9) and SIGSTOP (#) signals cannot be ignored or caught. For example, if you wanted to tell the kernel that ctrl-C's are to be ignored, you would include something like this in your source code: signal(SIGINT, SIGIGN); To ensure that the default action for a signal is taken, you would do something like this instead: signal(SIGSEGV, SIGDFL); + The article then shows some ways to send signals from the command line, for example to send SIGHUP to a process with pid 1234: kill -HUP 1234 + You can get a list of the different signals by running kill -l On Unix systems, signals are used to send all kinds of information to running processes, and they come from user commands, other processes, and the kernel itself. Through /proc, information about how processes are handling signals is now easily accessible and, with just a little manipulation of the data, easy to understand. links owned by NGZ erroneously marked as on loan (https://smartos.org/bugview/OS-6274) NGZ (Non-Global Zone), is IllumOS speak for their equivalent to a jail > As reported by user brianewell in smartos-live#737, NGZ ip tunnels stopped persisting across zone reboot. This behavior appeared in the 20170202 PI and was not present in previous releases. After much spelunking I determined that this was caused by a regression introduced in commit 33df115 (part of the OS-5363 work). The regression was a one-line change to link_activate() which marks NGZ links as on loan when they are in fact not loaned because the NGZ created and owns the link. “On loan” means the interface belongs to the host (GZ, Global Zone), and has been loaned to the NGZ (Jail) This regression was easy to introduce because of the subtle nature of this code and lack of comments. I'm going to remove the regressive line, add clarifying comments, and also add some asserts. The following is a detailed analysis of the issue, how I debugged it, and why my one-line change caused the regression: To start I verified that PI 20170119 work as expected: booted 20170119 created iptun (named v4sys76) inside of a native NGZ (names sos-zone) performed a reboot of sos-zone zlogin to sos-zone and verify iptun still exists after reboot Then I booted the GZ into PI 20170202 and verified the iptun did not show up booted 20170202 started sos-zone zlogin and verified the iptun was missing At this point I thought I would recreate the iptun and see if I could monitor the zone halt/boot process for the culprit, but instead I received an error from dladm: "object already exists". I didn't expect this. So I used mdb to inspect the dlmgmtd state. Sure enough the iptun exists in dlmgmtd. Okay, so if the link already exists, why doesn't it show up (in either the GZ or the NGZ)? If a link is not marked as active then it won't show up when you query dladm. When booting the zone on 20170119 the llflags for the iptun contained the value 0x3. So the problem is the link is not marked as active on the 20170202 PI. The linkactivate() function is responsible for marking a link as active. I used dtrace to verify this function was called on the 20170202 PI and that the dlmgmtlinkt had the correct llflags value. So the iptun link structure has the correct llflags when linkactivate() returns but when I inspect the same structure with mdb afterwards the value has changed. Sometime after linkactivate() completes some other process changed the llflags value. My next question was: where is linkactivate() called and what comes after it that might affect the llflags? I did another trace and got this stack. The dlmgmtupid() function calls dlmgmtwritedbentry() after linkactivate() and that can change the flags. But dtrace proved the llflags value was still 0x3 after returning from this function. With no obvious questions left I then asked cscope to show me all places where llflags is modified. As I walked through the list I used dtrace to eliminate candidates one at a time -- until I reached dlmgmtdestroycommon(). I would not have expected this function to show up during zone boot but sure enough it was being called somehow, and by someone. Who? Since there is no easy way to track door calls it was at this point I decided to go nuclear and use the dtrace stop action to stop dlmgmtd when it hits dlmgmtdestroycommon(). Then I used mdb -k to inspect the door info for the dlmgmtd threads and look for my culprit. The culprit is doupiptun() caused by the dladm up-iptun call. Using ptree I then realized this was happening as part of the zone boot under the network/iptun svc startup. At this point it was a matter of doing a zlogin to sos-zone and running truss on dladm up-iptun to find the real reason why dladmdestroydatalinkid() is called. So the link is marked as inactive because dladmgetsnapconf() fails with DLADMSTATUSDENIED which is mapped to EACCESS. Looking at the dladmgetsnapconf() code I see the following “The caller is in a non-global zone and the persistent configuration belongs to the global zone.” What this is saying is that if a link is marked "on loan" (meaning it's technically owned/created by the GZ but assigned/loaned to the NGZ) and the zone calling dladmgetsnapconf() is an NGZ then return EACCESS because the configuration of the link is up to the GZ, not the NGZ. This code is correct and should be enforced, but why is it tripping in PI 20170202 and not 20170119? It comes back to my earlier observation that in the 20170202 PI we marked the iptun as "on loan" but not in the older one. Why? Well as it turns out while fixing OS-5363 I fixed what I thought was a bug in linkactivate() When I first read this code it was my understanding that anytime we added a link to a zone's datalink list, by calling zoneadddatalink(), that link was then considered "on loan". My understanding was incorrect. The linkactivate() code has a subtleness that eluded me. There are two cases in linkactivate(): 1. The link is under an NGZ's datalink list but it's lllinkid doesn't reflect that (e.g., the link is found under zoneid 3 but lllinkid is 0). In this case the link is owned by the GZ but is being loaned to an NGZ and the link state should be updated accordingly. We get in this situation when dlmgmtd is restated for some reason (it must resync it's in-memory state with the state of the system). 2. The link is NOT under any NGZ's (zonecheckdatalink() is only concerned with NGZs) datalink list but its llzoneid holds the value of an NGZ. This indicates that the link is owned by an NGZ but for whatever reason is not currently under the NGZ's datalink list (e.g., because we are booting the zone and we now need to assign the link to its list). So the fix is to revert that one line change as well as add some clarifying comments and also some asserts to prevent further confusion in the future. + A nice breakdown by Ryan Zezeski of how he accidently introduced a regression, and how he tracked it down using dtrace and mdb New experimental statistics collector in master (http://dpaste.com/2YP0X9C) Master now has an in-kernel statistics collector which is enabled by default, and a (still primitive) user land program to access it. This recorder samples the state of the machine once every 10 seconds and records it in a large FIFO, all in-kernel. The FIFO typically contains 8192 entries, or around the last 23 hours worth of data. Statistics recorded include current load, user/sys/idle cpu use, swap use, VM fault rate, VM memory statistics, and counters for syscalls, path lookups, and various interrupt types. A few more useful counters will probably be added... I'd like to tie cpu temperature, fork rate, and exec rate in at some point, as well as network and disk traffic. The statistics gathering takes essentially no real overhead and is always on, so any user at the spur of the moment with no prior intent can query the last 23 hours worth of data. There is a user frontend to the data called 'kcollect' (its tied into the buildworld now). Currently still primitive. Ultimately my intention is to integrate it with a dbm database for long-term statistical data retention (if desired) using an occasional (like once-an-hour) cron-job to soak up anything new, with plenty of wiggle room due to the amount of time the kernel keeps itself. This is better and less invasive than having a userland statistics gathering script running every few minutes from cron and has the advantage of giving you a lot of data on the spur of the moment without having to ask for it before-hand. If you have gnuplot installed (pkg install gnuplot), kcollect can generate some useful graphs based on the in-kernel data. Well, it will be boring if the machine isn't doing anything :-). There are options to use gnuplot to generate a plot window in X or a .jpg or .png file, and other options to set the width and height and such. At the moment the gnuplot output uses a subset of statically defined fields to plot but ultimately the field list it uses will be specifiable. Sample image generated during a synth run (http://apollo.backplane.com/DFlyMisc/kcollect03.jpg) News Roundup openbsd changes of note 626 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-626) Hackerthon is imminent. There are two signals one can receive after accessing invalid memory, SIGBUS and SIGSEGV. Nobody seems to know what the difference is or should be, although some theories have been unearthed. Make some attempt to be slightly more consistent and predictable in OpenBSD. Introduces jiffies in an effort to appease our penguin oppressors. Clarify that IP.OF.UPSTREAM.RESOLVER is not actually the hostname of a server you can use. Switch acpibat to use _BIX before _BIF, which means you might see discharge cycle counts, too. Assorted clang compatibility. clang uses -Oz to mean optimize for size and -Os for something else, so make gcc accept -Oz so all makefiles can be the same. Adjust some hardlinks. Make sure we build gcc with gcc. The SSLcheckprivate_key function is a lie. Switch the amd64 and i386 compiler to clang and see what happens. We are moving towards using wscons (wstpad) as the driver for touchpads. Dancing with the stars, er, NET_LOCK(). clang emits lots of warnings. Fix some of them. Turn off a bunch of clang builtins because we have a strong preference that code use our libc versions. Some other changes because clang is not gcc. Among other curiosities, static variables in the special .openbsd.randomdata are sometimes assumed to be all zero, leading the clang optimizer to eliminate reads of such variables. Some more pledge rules for sed. If the script doesn't require opening new files, don't let it. Backport a bajillion fixes to stable. Release errata. RFC 1885 was obsoleted nearly 20 years ago by RFC 2463 which was obsoleted over 10 years ago by RFC 4443. We are probably not going back. Update libexpat to 2.2.3. vmm: support more than 3855MB guest memory. Merge libdrm 2.4.82. Disable SSE optimizations on i386/amd64 for SlowBcopy. It is supposed to be slow. Prevents crashes when talking to memory mapped video memory in a hypervisor. The $25 “FREEDOM Laptop!” (https://functionallyparanoid.com/2017/08/08/the-25-freedom-laptop/) Time to get back to the original intent of this blog – talking about my paranoid obsession with information security! So break out your tinfoil hats my friends because this will be a fun ride. I'm looking for the most open source / freedom respecting portable computing experience I can possibly find and I'm going to document my work in real-time so you will get to experience the ups (and possibly the downs) of that path through the universe. With that said, let's get rolling. When I built my OpenBSD router using the APU2 board, I discovered that there are some amd64 systems that use open source BIOS. This one used Coreboot and after some investigation I discovered that there was an even more paranoid open source BIOS called Libreboot out there. That started to feel like it might scratch my itch. Well, after playing around with some lower-powered systems like my APU2 board, my Thinkpad x230 and my SPARC64 boxes, I thought, if it runs amd64 code and I can run an open source operating system on it, the thing should be powerful enough for me to do most (if not all) of what I need it to do. At this point, I started looking for a viable machine. From a performance perspective, it looked like the Thinkpad x200, T400, T500 and W500 were all viable candidates. After paying attention on eBay for a while, I saw something that was either going to be a sweet deal, or a throwaway piece of garbage! I found a listing for a Thinkpad T500 that said it didn't come with a power adapter and was 100% untested. From looking at the photos, it seemed like there was nothing that had been molested about it. Obviously, nobody was jumping on something this risky so I thought, “what the heck” and dropped a bit at the opening price of $24.99. Well, guess what. I won the auction. Now to see what I got. When the laptop showed up, I discovered it was minus its hard drive (but the outside plastic cover was still in place). I plugged in my x230's power adapter and hit the button. I got lights and was dropped to the BIOS screen. To my eternal joy, I discovered that the machine I had purchased for $25 was 100% functional and included the T9400 2.54 GHz Core 2 Duo CPU and the 1680×1050 display panel. W00t! First things first, I need to get this machine a hard drive and get the RAM upgraded from the 2GB that it showed up with to 8GB. Good news is that these two purchases only totaled $50 for the pair. An aftermarket 9-cell replacement battery was another $20. Throw in a supported WiFi card that doesn't require a non-free blob from Libreboot at $5.99 off of eBay and $5 for a hard drive caddy and I'm looking at about $65 in additional parts bringing the total cost of the laptop, fully loaded up at just over $100. Not bad at all… Once all of the parts arrived and were installed, now for the fun part. Disassembling the entire thing down to the motherboard so we can re-flash the BIOS with Libreboot. The guide looks particularly challenging for this but hey, I have a nice set of screwdrivers from iFixit and a remarkable lack of fear when it comes to disassembling things. Should be fun! Well, fun didn't even come close. I wish I had shot some pictures along the way because at one point I had a heap of parts in one corner of my “workbench” (the dining room table) and just the bare motherboard, minus the CPU sitting in front of me. With the help of a clip and a bunch of whoops wires (patch cables), I connected my Beaglebone Black to the BIOS chip on the bare motherboard and attempted to read the chip. #fail I figured out after doing some more digging that you need to use the connector on the left side of the BBB if you hold it with the power connector facing away from you. In addition, you should probably read the entire process through instead of stopping at the exciting pinout connector diagram because I missed the bit about the 3.3v power supply need to have ground connected to pin 2 of the BIOS chip. Speaking of that infamous 3.3v power supply, I managed to bend a paperclip into a U shape and jam it into the connector of an old ATX power supply I had in a closet and source power from that. I felt like MacGyver for that one! I was able to successfully read the original Thinkpad BIOS and then flash the Libreboot + Grub2 VESA framebuffer image onto the laptop! I gulped loudly and started the reassembly process. Other than having some cable routing difficulties because the replacement WiFi card didn't have a 5Ghz antenna, it all went back together. Now for the moment of truth! I hit the power button and everything worked!!! At this point I happily scurried to download the latest snapshot of OpenBSD – current and install it. Well, things got a little weird here. Looks like I have to use GRUB to boot this machine now and GRUB won't boot an OpenBSD machine with Full Disk Encryption. That was a bit of a bummer for me. I tilted against that windmill for several days and then finally admitted defeat. So now what to do? Install Arch? Well, here's where I think the crazy caught up to me. I decided to be an utter sell out and install Ubuntu Gnome Edition 17.04 (since that will be the default DE going forward) with full disk encryption. I figured I could have fun playing around in a foreign land and try to harden the heck out of that operating system. I called Ubuntu “grandma's Linux” because a friend of mine installed it on his mom's laptop for her but I figured what the heck – let's see how the other half live! At this point, while I didn't have what I originally set out to do – build a laptop with Libreboot and OpenBSD, I did have a nice compromise that is as well hardened as I can possibly make it and very functional in terms of being able to do what I need to do on a day to day basis. Do I wish it was more portable? Of course. This thing is like a six or seven pounder. However, I feel much more secure in knowing that the vast majority of the code running on this machine is open source and has all the eyes of the community on it, versus something that comes from a vendor that we cannot inspect. My hope is that someone with the talent (unfortunately I lack those skills) takes an interest in getting FDE working with Libreboot on OpenBSD and I will most happily nuke and repave this “ancient of days” machine to run that! FreeBSD Programmers Report Ryzen SMT Bug That Hangs Or Resets Machines (https://hothardware.com/news/freebsd-programmers-report-ryzen-smt-bug-that-hangs-or-resets-machines) It's starting to look like there's an inherent bug with AMD's Zen-based chips that is causing issues on Unix-based operating systems, with both Linux and FreeBSD confirmed. The bug doesn't just affect Ryzen desktop chips, but also AMD's enterprise EPYC chips. It seems safe to assume that Threadripper will bundle it in, as well. It's not entirely clear what is causing the issue, but it's related to the CPU being maxed out in operations, thus causing data to get shifted around in memory, ultimately resulting in unstable software. If the bug is exercised a certain way, it can even cause machines to reset. The revelation about the issue on FreeBSD was posted to the official repository, where the issue is said to happen when threads can lock up, and then cause the system to become unstable. Getting rid of the issue seems as simple as disabling SMT, but that would then negate the benefits provided by having so many threads at-the-ready. On the Linux side of the Unix fence, Phoronix reports on similar issues, where stressing Zen chips with intensive benchmarks can cause one segmentation fault after another. The issue is so profound, that Phoronix Test Suite developer Michael Larabel introduced a special test that can be run to act as a bit of a proof-of-concept. To test another way, PTS can be run with this command: PTS_CONCURRENT_TEST_RUNS=4 TOTAL_LOOP_TIME=60 phoronix-test-suite stress-run build-linux-kernel build-php build-apache build-imagemagick Running this command will compile four different software projects at once, over and over, for an hour. Before long, segfaults should begin to appear (as seen in the shot above). It's not entirely clear if both sets of issues here are related, but seeing as both involve stressing the CPU to its limit, it seems likely. Whether or not this could be patched on a kernel or EFI level is something yet to be seen. TrueOS - UNSTABLE update: 8/7/17 (https://www.trueos.org/blog/unstable-update-8717/) A new UNSTABLE update for TrueOS is available! Released regularly, UNSTABLE updates are the full “rolling release” of TrueOS. UNSTABLE includes experimental features, bugfixes, and other CURRENT FreeBSD work. It is meant to be used by those users interested in using the latest TrueOS and FreeBSD developments to help test and improve these projects. WARNING: UNSTABLE updates are released primarily for TrueOS and FreeBSD testing/experimentation purposes. Update and run UNSTABLE “at your own risk”. Note: There was a CDN issue over the weekend that caused issues for early updaters. Everything appears to be resolved and the update is fully available again. If you encountered instability or package issues from updating on 8/6 or 8/5, roll back to a previous boot environment and run the update again. Changes: UNSTABLE .iso and .img files beginning with TrueOS-2017-08-3-x64 will be available to download from http://download.trueos.org/unstable/amd64/. Due to CDN issues, these are not quite available, look for them later today or tomorrow (8/8/17). This update resyncs all ports with FreeBSD as of 8.1.2017. This includes: New/updated FreeBSD Kernel and World & New DRM (Direct Rendering Manager) next. Experimental patch for libhyve-remote: (From htps://github.com/trueos/freebsd/commit/a67a73e49538448629ea27, thanks araujobsd) The libhyve-remote aims to abstract functionalities from other third party libraries like libvncserver, freerdp, and spice to be used in hypervisor implementation. With a basic data structure it is easy to implement any remote desktop protocol without digging into the protocol specification or third part libraries – check some of our examples.We don't statically link any third party library, instead we use a dynamic linker and load only the functionality necessary to launch the service.Our target is to abstract functionalities from libvncserver, freerdp and spice. Right now, libhyve-remote only supports libvncserver. It is possible to launch a VNC server with different screen resolution as well as with authentication.With this patch we implement support for bhyve to use libhyve-remote that basically abstract some functionalities from libvncserver. We can: Enable wait state, Enable authentication, Enable different resolutions< Have a better compression. Also, we add a new -s flag for vncserver, if the libhyve-remote library is not present in the system, we fallback to bhyve RFB implementation. For example: -s 2,fbuf,tcp=0.0.0.0:5937,w=800,h=600,password=1234567,vncserver,wait New SysAdm Client pages under the System Management category: System Control: This is an interface to browse all the sysctl's on the system. Devices: This lists all known information about devices on the designated system. Lumina Theming: Lumina is testing new theming functionality! By default (in UNSTABLE), a heavily customized version of the Qt5ct engine is included and enabled. This is intended to allow users to quickly adjust themes/icon packs without needing to log out and back in. This also fixes a bug in Insight with different icons loading for the side and primary windows. Look for more information about this new functionality to be discussed on the Lumina Website. Update to Iridium Web Browser: Iridium is a Chromium based browser built with user privacy and security as the primary concern, but still maintaining the speed and usability of Chromium. It is now up to date – give it a try and let us know what you think (search for iridium-browser in AppCafe). Beastie Bits GhostBSD 11.1 Alpha1 is ready (http://www.ghostbsd.org/11.1-ALPHA1) A Special CharmBUG announcement (https://www.meetup.com/CharmBUG/events/242563414/) Byhve Obfuscation Part 1 of Many (https://github.com/HardenedBSD/hardenedBSD/commit/59eabffdca53275086493836f732f24195f3a91d) New BSDMag is out (https://bsdmag.org/download/bsd-magazine-overriding-libc-functions/) git: kernel - Lower VMMAXUSER_ADDRESS to finalize work-around for Ryzen bug (http://lists.dragonflybsd.org/pipermail/commits/2017-August/626190.html) Ken Thompson corrects one of his biggest regrets (https://twitter.com/_rsc/status/897555509141794817) *** Feedback/Questions Hans - zxfer (http://dpaste.com/2SQYQV2) Harza - Google Summer of Code (http://dpaste.com/2175GEB) tadslot - Microphones, Proprietary software, and feedback (http://dpaste.com/154MY1H) Florian - ZFS/Jail (http://dpaste.com/2V9VFAC) Modifying a ZFS root system to a beadm layout (http://dan.langille.org/2015/03/11/modifying-a-zfs-root-system-to-a-beadm-layout/) ***
This week on BSD Now, we clear up some ZFS FUD, show you how to write a NetBSD kernel module, and cover DragonflyBSD on the desktop. This episode was brought to you by Headlines ZFS is the best file system (for now) (http://blog.fosketts.net/2017/07/10/zfs-best-filesystem-now/) In my ongoing effort to fight misinformation and FUD about ZFS, I would like to go through this post in detail and share my thoughts on the current state and future of OpenZFS. The post starts with: ZFS should have been great, but I kind of hate it: ZFS seems to be trapped in the past, before it was sidelined it as the cool storage project of choice; it's inflexible; it lacks modern flash integration; and it's not directly supported by most operating systems. But I put all my valuable data on ZFS because it simply offers the best level of data protection in a small office/home office (SOHO) environment. Here's why. When ZFS first appeared in 2005, it was absolutely with the times, but it's remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10 The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. The lawsuit was resolved, and it didn't really impede ZFS. Some say it is the reason that Apple didn't go with ZFS, but there are other theories too. By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. Yes, Oracle taking over Sun and closing the source for ZFS definitely seemed like a setback at the time, but the OpenZFS project was started and active development has continued as an ever increasing pace. As of today, more than half of the code in OpenZFS has been written since the fork from the last open version of Oracle ZFS. the CDDL license Sun applied to the ZFS code was https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ (judged incompatible) with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world's server operating system. That hasn't stopped the ZFS-on-Linux project, or Ubuntu… Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and http://blog.fosketts.net/2008/09/15/greenbytes-embraces-extends-zfs/ (GreenBytes) helped push ZFS forward in the enterprise, but Oracle's lackluster commitment to Sun in the datacenter started having an impact. Lots of companies have adopted OpenZFS for their products. Before OpenZFS, there were very few non-Sun appliances that used ZFS, now there are plenty. OpenZFS Wiki: Companies with products based on OpenZFS (http://open-zfs.org/wiki/Companies) OpenZFS remains little-changed from what we had a decade ago. Other than the fact that half of the current code did not exist a decade ago… Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. This is one of the weaker points in ZFS. As it turns out, the demand for deduplication is actually not that strong. Most of the win can be had with transparent compression. However, there are a number of suggested designs to work around the dedup problems: Dedup Ceiling: Set a limit on the side of the DDT and just stop deduping new unique blocks when this limit is reached. Allocation Classes: A feature being developed by Intel for a supercomputer, will allow different types of data to be classified, and dedicated vdevs (or even metaslabs within a vdev), to be dedicated to that class of data. This could be extended to having the DDT live on a fast device like an PCIe NVMe, combined with the Dedup Ceiling when the device is full. DDT Pruning: Matt Ahrens described a design where items in the DDT with only a single reference, would be expired in an LRU type fashion, to allow newer blocks to live in the DDT in hopes that they would end up with more than a single reference. This doesn't cause bookkeeping problems since when a block is about to be freed, if it is NOT listed in the DDT, ZFS knows it was never deduplicated, so the current block must be the only reference, and it can safely be freed. This provides a best case scenario compared to Dedup Ceiling, since blocks that will deduplicate well, are likely to be written relatively close together, whereas the chance to a dedup match on a very old block is much lower. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today's prices. As we talked about a few weeks ago, ECC is best, but it is not required. If you want your server to stay up for a long time, to be highly available, you'll put ECC in it. Don't let a lack of ECC stop you from using ZFS, you are just putting your data at more risk. The scrub of death is a myth. ZFS does not ‘require' lots of ram. Your NAS will work happily with 8 GB instead of 32 GB of RAM. Its cache hit ratio will be much lower, so performance will be worse. It won't be able to buffer as many writes, so performance will be worse. Copy-on-Write has some drawbacks, data tends to get scattered and fragmented across the drives when it is written gradually. The ARC (RAM Cache) lessens the pain of this, and allows ZFS to batch incoming writes up into nice contiguous writes. ZFS purposely alternates between reading and writing, since both are faster when the other is not happening. So writes are batched up until there is too much dirty data, or the timeout expires. Then reads are held off while the bulk linear write finishes as quickly as possible, and reads are resumed. Obviously all of this works better and more efficiently in larger batches, which you can do if you have more RAM. ZFS can be tuned to use less RAM, and if you do not have a lot of RAM, or you have a lot of other demand on your RAM, you should do that tuning. And ZFS never really adapted to today's world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It's laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it's everywhere in performance PC's. Make up your mind, is 32GB of ram too expensive or not… the L2ARC exists specifically for the case where it is not possible to just install more RAM. Be it because there are no more slots, of limits of the processor, or limits of your budget. The SLOG is optional, but it never needs to be very big. A number of GBs of SLC flash is all you need, it is only holding writes that have not been flushed to the regular storage devices yet. The reason the documentation talks about SLC specifically is because your SLOG needs a very high write endurance, something never the newest NVMe devices cannot yet provide. Of course you can use NVMe devices with ZFS, lots of people do. All flash ZFS arrays are for sale right now. Other than maybe a little tuning of the device queue depths, ZFS just works and there is nothing to think about. However, to say there is nothing happening in this space is woefully inaccurate. The previously mentioned allocation classes code can be used to allocate metadata (4 KB blocks) on SSD or NVMe, while allocating bulk storage data (up to 16 MB blocks) on spinning disks. Extended a bit beyond what Intel is building for their super computer, this will basically create hybrid storage for ZFS. With the metaslab classes feature, it will even be possible to mix classes on the same device, grouping small allocations and large allocations in different areas, decreasing fragmentation. Then there's the question of flexibility, or lack thereof. Once you build a ZFS volume, it's pretty much fixed for life. There are only three ways to expand a storage pool: Replace each and every drive in the pool with a larger one (which is great but limiting and expensive) It depends on your pool layout. If you design with this in mind using ZFS Mirrors, it can be quite useful Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff) The unbalanced LUNs performance issues were sorted out in 2013-2016. 2014: OpenZFS Allocation Performance (http://open-zfs.org/w/images/3/31/Performance-George_Wilson.pdf) 2016: OpenZFS space allocation: doubling performance on large and fragmented pools (http://www.bsdcan.org/2016/schedule/events/710.en.html) These also mostly solved the performance issues when a pool gets full, you can run a lot closer to the edge now Build a new pool and “zfs send” your datasets to it (which is what I do, even though it's kind of tricky) This is one way to do it, yes. There is another way coming, but I can't talk about it just yet. Look for big news later this year. Apart from option 3 above, you can't shrink a ZFS pool. Device removal is arriving now. It will not work for RAIDZ*, but for Mirrors and Stripes you will be able to remove a device. I've probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it's startlingly limiting and out of touch with the present solid-state-dominated storage world. I don't feel like ZFS is out of touch with solid state. Lots of people are running SSD only pools. I will admit the tiered storage options in ZFS are a bit limited still, but there is a lot of work being done to overcome this. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It's going to be a long time before I trust anything other than ZFS! + I agree. + ZFS has a great track record of doing its most important job, keeping your data safe. + Work is ongoing to make ZFS more performance, and more flexible. The import thing is that this work is never allowed to compromise job #1, keeping your data safe. + Hybrid/tiered storage features, re-RAID-ing, are coming + There is a lot going on with OpenZFS, check out the notes from the last two OpenZFS Developer Summits just to get an idea of what some of those things are: 2015 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2015) & 2016 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2016) Some highlights: Compressed ARC Compressed send/recv ABD (arc buf scatter/gather) ZFS Native Encryption (scrub/resilver, send/recv, etc without encryption keys loaded) Channel Programs (do many administrative operations as one atomic transaction) Device Removal Redacted send/recv ZStandard Compression TRIM Support (FreeBSD has its own, but this will be more performant and universal) Faster Scrub/Resilver (https://youtu.be/SZFwv8BdBj4) Declustered RAID (https://youtu.be/MxKohtFSB4M) Allocation Classes (https://youtu.be/28fKiTWb2oM) Multi-mount protection (for Active/Passive failover) Zpool Checkpoint (undo almost anything) Even more Improved Allocator Performance vdev spacemap log ZIL performance improvements (w/ or w/o SLOG) Persistent L2ARC What I don't think the author of this article understands is how far behind every other filesystem is. 100s of Engineer years have gone into OpenZFS, and the pace is accelerating. I don't see how BtrFS can ever catch up, without a huge cash infusion. Writing a NetBSD kernel module (https://saurvs.github.io/post/writing-netbsd-kern-mod/) Kernel modules are object files used to extend an operating system's kernel functionality at run time. In this post, we'll look at implementing a simple character device driver as a kernel module in NetBSD. Once it is loaded, userspace processes will be able to write an arbitrary byte string to the device, and on every successive read expect a cryptographically-secure pseudorandom permutation of the original byte string. You will need the NetBSD Source Code. This doc (https://www.netbsd.org/docs/guide/en/chap-fetch.html) will explain how you can get it. The article gives an easy line by line walkthrough which is easy to follow and understand. The driver implements the bare minimum: open, close, read, and write, plus the module initialization function It explains the differences in how memory is allocated and freed in the kernel It also describes the process of using UIO to copy data back and forth between userspace and the kernel Create a Makefile, and compile the kernel module Then, create a simple userspace program to use the character device that the kernel module creates All the code is available here (https://github.com/saurvs/rperm-netbsd) *** DragonFlyBSD Desktop! (https://functionallyparanoid.com/2017/07/11/dragonflybsd-desktop/) If you read my last post (https://functionallyparanoid.com/2017/06/30/boot-all-the-things/), you know that I set up a machine (Thinkpad x230) with UEFI and four operating systems on it. One, I had no experience with – DragonFlyBSD (other than using Matthew Dillon's C compiler for the Amiga back in the day!) and so it was uncharted territory for me. After getting the install working, I started playing around inside of DragonFlyBSD and discovered to my delight that it was a great operating system with some really unique features – all with that BSD commitment to good documentation and a solid coupling of kernel and userland that doesn't exist (by design) in Linux. So my goal for my DragonFlyBSD desktop experience was to be as BSD as I possibly could. Given that (and since I'm the maintainer of the port on OpenBSD ), I went with Lumina as the desktop environment and XDM as the graphical login manager. I have to confess that I really like the xfce terminal application so I wanted to make sure I had that as well. Toss in Firefox, libreOffice and ownCloud sync client and I'm good to go! OK. So where to start. First, we need to get WiFi and wired networking happening for the console at login. To do that, I added the following to /etc/rc.conf: wlans_iwn0=”wlan0″ ifconfig_wlan0=”WPA DHCP” ifconfig_em0=”DHCP” I then edited /etc/wpa_supplicant.conf to put in the details of my WiFi network: network={ ssid=”MY-NETWORK-NAME” psk=”my-super-secret-password” } A quick reboot showed that both wired and wireless networking were functional and automatically were assigned IP addresses via DHCP. Next up is to try getting into X with whatever DragonFlyBSD uses for its default window manager. A straight up “startx” met with, shall we say, less than stellar results. Therefore, I used the following command to generate a simple /etc/X11/xorg.conf file: # Xorg -configure # cp /root/xorg.conf.new /etc/X11/xorg.conf With that file in place, I could get into the default window manager, but I had no mouse. After some searching and pinging folks on the mailing list, I was able to figure out what I needed to do. I added the following to my /etc/rc.conf file: moused_enable=”YES” moused_type=”auto” moused_port=”/dev/psm0″ I rebooted (I'm sure there is an easier way to get the changes but I don't know it… yet) and was able to get into a basic X session and have a functional mouse. Next up, installing and configuring Lumina! To do that, I went through the incredibly torturous process of installing Lumina: # pkg install lumina Wow! That was really, really hard. I might need to pause here to catch my breath.
We look at an OpenBSD setup on a new laptop, revel in BSDCan trip reports, and visit daemons and friendly ninjas. This episode was brought to you by Headlines OpenBSD and the modern laptop (http://bsdly.blogspot.de/2017/07/openbsd-and-modern-laptop.html) Peter Hansteen has a new blog post about OpenBSD (http://www.openbsd.org/) on laptops: Did you think that OpenBSD is suitable only for firewalls and high-security servers? Think again. Here are my steps to transform a modern mid to high range laptop into a useful Unix workstation with OpenBSD. One thing that never ceases to amaze me is that whenever I'm out and about with my primary laptop at conferences and elsewhere geeks gather, a significant subset of the people I meet have a hard time believing that my laptop runs OpenBSD, and that it's the only system installed. and then it takes a bit of demonstrating that yes, the graphics runs with the best available resolution the hardware can offer, the wireless network is functional, suspend and resume does work, and so forth. And of course, yes, I do use that system when writing books and articles too. Apparently heavy users of other free operating systems do not always run them on their primary workstations. Peter goes on to describe the laptops he's had over the years (all running OpenBSD) and after BSDCan 2017, he needed a new one due to cracks in the display. So the time came to shop around for a replacement. After a bit of shopping around I came back to Multicom, a small computers and parts supplier outfit in rural Åmli in southern Norway, the same place I had sourced the previous one. One of the things that attracted me to that particular shop and their own-branded offerings is that they will let you buy those computers with no operating system installed. That is of course what you want to do when you source your operating system separately, as we OpenBSD users tend to do. The last time around I had gone for a "Thin and lightweight" 14 inch model (Thickness 20mm, weight 2.0kg) with 16GB RAM, 240GB SSD for system disk and 1TB HD for /home (since swapped out for a same-size SSD, as the dmesg will show). Three years later, the rough equivalent with some added oomph for me to stay comfortable for some years to come ended me with a 13.3 inch model, 18mm and advertised as 1.3kg (but actually weighing in at 1.5kg, possibly due to extra components), 32GB RAM, 512GB SSD and 2TB harddisk. For now the specification can be viewed online here (https://www.multicom.no/systemconfigurator.aspx?q=st:10637291;c:100559;fl:0#4091-10500502-1;4086-10637290-1;4087-8562157-2;4088-9101982-1;4089-9101991-1) (the site language is Norwegian, but product names and units of measure are not in fact different). The OpenBSD installer is a wonder of straightforward, no-nonsense simplicity that simply gets the job done. Even so, if you are not yet familiar with OpenBSD, it is worth spending some time reading the OpenBSD FAQ's installation guidelines and the INSTALL.platform file (in our case, INSTALL.amd64) to familiarize yourself with the procedure. If you're following this article to the letter and will be installing a snapshot, it is worth reading the notes on following -current too. The main hurdle back when I was installing the 2014-vintage 14" model was getting the system to consider the SSD which showed up as sd1 the automatic choice for booting (I solved that by removing the MBR, setting the size of the MBR on the hard drive that showed up as sd0 to 0 and enlarging the OpenBSD part to fill the entire drive). + He goes on to explain the choices he made in the installer and settings made after the reboot to set up his work environment. Peter closes with: If you have any questions on running OpenBSD as a primary working environment, I'm generally happy to answer but in almost all cases I would prefer that you use the mailing lists such as misc@openbsd.org or the OpenBSD Facebook (https://www.facebook.com/groups/2210554563/) group so the question and hopefully useful answers become available to the general public. Browsing the slides for my recent OpenBSD and you (https://home.nuug.no/~peter/openbsd_and_you/) user group talk might be beneficial if you're not yet familiar with the system. And of course, comments on this article are welcome. BSDCan 2017 Trip Report: Roller Angel (https://www.freebsdfoundation.org/blog/2017-bsdcan-trip-report-roller-angel/) We could put this into next week's show, because we have another trip report already that's quite long. After dropping off my luggage, I headed straight over to the Goat BoF which took place at The Royal Oak. There were already a number of people there engaged in conversation with food and drink. I sat down at a table and was delighted that the people sitting with me were also into the BSD's and were happy to talk about it the whole time. I felt right at home from the start as people were very nice to me, and were interested in what I was working on. I honestly didn't know that I would fit in so well. I had a preconceived notion that people may be a bit hard to approach as they are famous and so technically advanced. At first, people seemed to only be working in smaller circles. Once you get more familiar with the faces, you realize that these circles don't always contain the same people and that they are just people talking about specific topics. I found that it was easy to participate in the conversation and also found out that people are happy to get your feedback on the subject as well. I was actually surprised how easily I got along with everyone and how included I felt in the activities. I volunteered to help wherever possible and got to work on the video crew that recorded the audio and slides of the talks. The people at BSDCan are incredibly easy to talk to, are actually interested in what you're doing with BSD, and what they can do to help. It's nice to feel welcome in the community. It's like going home. Dan mentioned in his welcome on the first day of BSDCan that the conference is like home for many in the community. The trip report is very detailed and chronicles the two days of the developer summit, and the two days of the conference There was some discussion about a new code of conduct by Benno Rice who mentioned that people are welcome to join a body of people that is forming that helps work out issues related to code of conduct and forwards their recommendations on to core. Next, Allan introduced the idea of creating a process for formally discussing big project changes or similar discussions that is going to be known as FCP or FreeBSD Community Proposal. In Python we have the Python Enhancement Proposal or PEP which is very similar to the idea of FCP. I thought this idea is a great step for FreeBSD to be implementing as it has been a great thing for Python to have. There was some discussion about taking non-code contributions from people and how to recognize those people in the project. There was a suggestion to have a FreeBSD Member status created that can be given to people whose non-code contributions are valuable to the project. This idea seemed to be on a lot of people's minds as something that should be in place soon. The junior jobs on the FreeBSD Wiki were also brought up as a great place to look for ideas on how to get involved in contributing to FreeBSD. Roller wasted no time, and started contributing to EdgeBSD at the conference. On the first day of BSDCan I arrived at the conference early to coordinate with the team that records the talks. We selected the rooms that each of us would be in to do the recording and set up a group chat via WhatsApp for coordination. Thanks to Roller, Patrick McAvoy, Calvin Hendryx-Parker, and all of the others who volunteered their time to run the video and streaming production at BSDCan, as well as all others who volunteered, even if it was just to carry a box. BSDCan couldn't happen without the army of volunteers. After the doc lounge, I visited the Hacker Lounge. There were already several tables full of people talking and working on various projects. In fact, there was a larger group of people who were collaborating on the new libtrue library that seemed to be having a great time. I did a little socializing and then got on my laptop and did some more work on the documentation using my new skills. I really enjoyed having a hacker lounge to go to at night. I want to give a big thank you to the FreeBSD Foundation for approving my travel grant. It was a great experience to meet the community and participate in discussions. I'm very grateful that I was able to attend my first BSDCan. After visiting the doc lounge a few times, I managed to get comfortable using the tools required to edit the documentation. By the end of the conference, I had submitted two documentation patches to the FreeBSD Bugzilla with several patches still in progress. Prior to the conference I expected that I would be spending a lot of time working on my Onion Omega and Edge Router Lite projects that I had with me, but I actually found that there was always something fun going on that I would rather do or work on. I can always work on those projects at home anyway. I had a good time working with the FreeBSD community and will continue working with them by editing the documentation and working with Bugzilla. One of the things I enjoy about these trip reports is when they help convince other people to make the trip to their first conference. Hopefully by sharing their experience, it will convince you to come to the next conference: vBSDCon in Virginia, USA: Sept 7-9 EuroBSDCon in Paris, France: Sept 21-24 BSDTW in Taipei, Taiwan: November 11-12 (CFP ends July 31st) *** BSDCan 2017 - Trip report double-p (http://undeadly.org/cgi?action=article&sid=20170629150641) Prologue Most overheard in Tokyo was "see you in Ottawaaaaah", so with additional "personal item" being Groff I returned home to plan the trip to BSDCan. Dan was very helpful with getting all the preparations (immigration handling), thanks for that. Before I could start, I had to fix something: the handling of the goat. With a nicely created harness, I could just hang it along my backpack. Done that it went to the airport of Hamburg and check-in for an itinerary of HAM-MUC-YUL. While the feeder leg was a common thing, boarding to YUL was great - cabin-crew likes Groff :) Arriving in Montreal was like entering a Monsoon zone or something, sad! After the night the weather was still rain-ish but improving and i shuttled to Dorval VIARail station to take me to Ottawa (ever avoid AirCanada, right?). Train was late, but the conductor (or so) was nice to talk to - and wanted to know about Groff's facebook page :-P. Picking a cab in Ottawa to take me to "Residence" was easy at first - just that it was the wrong one. Actually my fault and so I had a "nice, short" walk to the actual one in the rain with wrong directions. Eventually I made it and after unpacking, refreshment it was time to hit the Goat BOF! Day 1 Since this was my first BSDCan I didnt exactly knew what to expect from this BOF. But it was like, we (Keeper, Dan, Allan, ..) would talk about "who's next" and things like that. How mistaken I was :). Besides the sheer amount of BSD people entering the not-so-yuuge Oak some Dexter sneaked in camouflage. The name-giver got a proper position to oversee the mess and I was glad I did not leave him behind after almost too many Creemores. Day 2 Something happened it's crystal blue on the "roof" and sun is trying its best to wake me up. To start the day, I pick breakfast at 'Father+Sons' - I can really recommend that. Very nice home made fries (almost hashbrowns) and fast delivery! Stuffed up I trott along to get to phessler's tutorial about BGP-for-sysadmins-and-developers. Peter did a great job, but the "lab" couldn't happen, since - oh surprise - the wifi was sluggish as hell. Must love the first day on a conference every time. Went to Hackroom in U90 afterwards, just to fix stuff "at home". IPsec giving pains again. Time to pick food+beer afterwards and since it's so easy to reach, we went to the Oak again. Having a nice backyard patio experience it was about time to meet new people. Cheers to Tom, Aaron, Nick, Philip and some more, we'd an awesome night there. I also invited some not-really-computer local I know by other means who was completly overwhelmed by what kind of "nerds" gather around BSD. He planned to stay "a beer" - and it was rather some more and six hours. Looks like "we" made some impression on him :). Day 3 Easy day, no tutorials at hand, so first picking up breakfast at F+S again and moving to hackroom in U90. Since I promised phessler to help with an localized lab-setup, I started to hack on a quick vagrant/ansible setup to mimic his BGP-lab and went quickly through most of it. Plus some more IPsec debugging and finally fixing it, we went early in the general direction of the Red Lion to pick our registration pack. But before that could happen it was called to have shawarma at 3brothers along. Given a tight hangover it wasn't the brightest idea to order a poutine m-(. Might be great the other day, it wasn't for me at the very time and had to throw away most of it :(. Eventually passing on to the Red Lion I made the next failure with just running into the pub - please stay at the front desk until "seated". I never get used to this concept. So after being "properly" seated, we take our beers and the registration can commence after we had half of it. So I register myself; btw it's a great idea to grant "not needed" stuff to charity. So dont pick "just because", think about it if you really need this or that gadget. Then I register Groff - he really needs badges - just to have Dru coming back to me some minutes later one to hand me the badge for Henning. That's just "amazing"; I dont know IF i want to break this vicious circle the other day, since it's so funny. Talked to Theo about the ongoing IPsec problems and he taught me about utrace(2) which looks "complicated" but might be an end of the story the other day. Also had a nice talk to Peter (H.) about some other ideas along books. BTW, did I pay for ongoing beers? I think Tom did - what a guy :). Arriving at the Residence, I had to find my bathroom door locked (special thing).. crazy thing is they dont have a master key at the venue, but to have to call in one from elsewhere. Short night shortened by another 30minutes :(. Day 4 Weather is improving into beach+sun levels - and it's Conference Day! The opening keynote from Geist was very interesting ("citation needed"). Afterwards I went to zfs-over-ssh, nothing really new (sorry Allan). But then Jason had a super interesting talk on how about to apply BSD for the health-care system in Australia. I hope I can help him with the last bits (rdomain!) in the end. While lunch I tried to recall my memories about utrace(2) while talking to Theo. Then it was about to present my talk and I think it was well perceipted. One "not so good" feedback was about not taking the audience more into account. I think I was asking every other five slides or so - but, well. The general feedback (in spoken terms) was quite good. I was a bit "confused" and I did likely a better job in Tokyo, but well. Happened we ended up in the Oak again.. thanks to mwl, shirkdog, sng, pitrh, kurtm for having me there :) Day 5 While the weather had to decide "what next", I rushed to the venue just to gather Reyk's talk about vmd(8). Afterwards it was MSTP from Paeps which was very interesting and we (OpenBSD) should look into it. Then happened BUG BOF and I invite all "coastal Germans" to cbug.de :) I had to run off for other reasons and came back to Dave's talk which was AWESOME. Following was Rod's talk.. well. While I see his case, that was very poor. The auction into closing was awesome again, and I spend $50 on a Tshirt. :) + Epilogue I totally got the exit dates wrong. So first cancel a booking of an Hotel and then rebook the train to YUL. So I have plenty of time "in the morning" to get breakfast with the local guy. After that he drives me to VIARail station and I dig into "business" cussions. Well, see you in Ottawa - or how about Paris, Taipei? Bind Broker (http://www.tedunangst.com/flak/post/bind-broker) Ted Unangst writes about an interesting idea he has He has a single big server, and lots of users who would like to share it, many want to run web servers. This would be great, but alas, archaic decisions made long ago mean that network sockets aren't really files and there's this weird concept of privileged ports. Maybe we could assign each user a virtual machine and let them do whatever they want, but that seems wasteful. Think of the megabytes! Maybe we could setup nginx.conf to proxy all incoming connections to a process of the user's choosing, but that only works for web sites and we want to be protocol neutral. Maybe we could use iptables, but nobody wants to do that. What we need is a bind broker. At some level, there needs to be some kind of broker that assigns IPs to users and resolves conflicts. It should be possible to build something of this nature given just the existing unix tools we have, instead of changing system design. Then we can deploy our broker to existing systems without upgrading or disrupting their ongoing operation. The bind broker watches a directory for the creation, by users, of unix domain sockets. Then it binds to the TCP port of the same name, and transfers traffic between them. A more complete problem specification is as follows. A top level directory, which contains subdirectories named after IP addresses. Each user is assigned a subdirectory, which they have write permission to. Inside each subdirectory, the user may create unix sockets named according to the port they wish to bind to. We might assign user alice the IP 10.0.0.5 and the user bob the IP 10.0.0.10. Then alice could run a webserver by binding to net/10.0.0.5/80 and bob could run a mail server by binding to net/10.0.0.10/25. This maps IP ownership (which doesn't really exist in unix) to the filesystem namespace (which does have working permissions). So this will be a bit different than jails. The idea is to use filesystem permissions to control which users can bind to which IP addresses and ports The broker is responsible for watching each directory. As new sockets are created, it should respond by binding to the appropriate port. When a socket is deleted, the network side socket should be closed as well. Whenever a connection is accepted on the network side, a matching connection is made on the unix side, and then traffic is copied across. A full set of example code is provided There's no completely portable way to watch a directory for changes. I'm using a kevent extension. Otherwise we might consider a timeout and polling with fstat, or another system specific interface (or an abstraction layer over such an interface). Otherwise, if one of our mappings is ready to read (accept), we have a new connection to handle. The first half is straightforward. We accept the connection and make a matching connect call to the unix side. Then I broke out the big cheat stick and just spliced the sockets together. In reality, we'd have to set up a read/copy/write loop for each end to copy traffic between them. That's not very interesting to read though. The full code, below, comes in at 232 lines according to wc. Minus includes, blank lines, and lines consisting of nothing but braces, it's 148 lines of stuff that actually gets executed by the computer. Add some error handling, and working read/write code, and 200 lines seems about right. A very interesting idea. I wonder about creating a virtual file system that would implement this and maybe do a bit more to fully flesh out this idea. What do you think? *** News Roundup Daemons and friendly Ninjas (https://euroquis.nl/bobulate/?p=1600) There's quite a lot of software that uses CMake as a (meta-)buildsystem. A quick count in the FreeBSD ports tree shows me 1110 ports (over a thousand) that use it. CMake generates buildsystem files which then direct the actual build — it doesn't do building itself. There are multiple buildsystem-backends available: in regular usage, CMake generates Makefiles (and does a reasonable job of producing Makefiles that work for GNU Make and for BSD Make). But it can generate Ninja, or Visual Studio, and other buildsystem files. It's quite flexible in this regard. Recently, the KDE-FreeBSD team has been working on Qt WebEngine, which is horrible. It contains a complete Chromium and who knows what else. Rebuilding it takes forever. But Tobias (KDE-FreeBSD) and Koos (GNOME-FreeBSD) noticed that building things with the Ninja backend was considerably faster for some packages (e.g. Qt WebEngine, and Evolution data-thingy). Tobias wanted to try to extend the build-time improvements to all of the CMake-based ports in FreeBSD, and over the past few days, this has been a success. Ports builds using CMake now default to using Ninja as buildsystem-backend. Here's a bitty table of build-times. These are one-off build times, so hardly scientifically accurate — but suggestive of a slight improvement in build time. Name Size GMake Ninja liblxt 50kB 0:32 0:31 llvm38 1655kB * 19:43 musescore 47590kB 4:00 3:54 webkit2-gtk3 14652kB 44:29 37:40 Or here's a much more thorough table of results from tcberner@, who did 5 builds of each with and without ninja. I've cut out the raw data, here are just the average-of-five results, showing usually a slight improvement in build time with Ninja. Name av make av ninj Delta D/Awo compiler-rt 00:08 00:07 -00:01 -14% openjpeg 00:06 00:07 +00:01 +17% marble 01:57 01:43 -00:14 -11% uhd 01:49 01:34 -00:15 -13% opencacscade 04:08 03:23 -00:45 -18% avidemux 03:01 02:49 -00:12 – 6% kdevelop 01:43 01:33 -00:10 – 9% ring-libclient 00:58 00:53 -00:05 – 8% Not everything builds properly with Ninja. This is usually due to missing dependencies that CMake does not discover; this shows up when foo depends on bar but no rule is generated for it. Depending on build order and speed, bar may be there already by the time foo gets around to being built. Doxygen showed this, where builds on 1 CPU core were all fine, but 8 cores would blow up occasionally. In many cases, we've gone and fixed the missing implicit dependencies in ports and upstreams. But some things are intractable, or just really need GNU Make. For this, the FreeBSD ports infrastructure now has a knob attached to CMake for switching a port build to GNU Make. Normal: USES=cmake Out-of-source: USES=cmake:outsource GNU Make: USES=cmake:noninja gmake OoS, GMake: USES=cmake:outsource,noninja gmake Bad: USES=cmake gmake For the majority of users, this has no effect, but for our package-building clusters, and for KDE-FreeBSD developers who build a lot of CMake-buildsystem software in a day it may add up to an extra coffee break. So I'll raise a shot of espresso to friendship between daemons and ninjas. Announcing the pkgsrc-2017Q2 release (http://mail-index.netbsd.org/pkgsrc-users/2017/07/10/msg025237.html) For the 2017Q2 release we welcome the following notable package additions and changes to the pkgsrc collection: Firefox 54 GCC 7.1 MATE 1.18 Ruby 2.4 Ruby on Rails 4.2 TeX Live 2017 Thunderbird 52.1 Xen 4.8 We say goodbye to: Ruby 1.8 Ruby 2.1 The following infrastructure changes were introduced: Implement optional new pkgtasks and init infrastructure for pkginstall. Various enhancements and fixes for building with ccache. Add support to USE_LANGUAGES for newer C++ standards. Enhanced support for SSP, FORTIFY, and RELRO. The GitHub mirror has migrated to https://github.com/NetBSD/pkgsrc In total, 210 packages were added, 43 packages were removed, and 1,780 package updates were processed since the pkgsrc-2017Q1 release. *** OpenBSD changes of note 624 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-624) There are a bunch, but here are a few that jump out: Start plugging some leaks. Compile kernels with umask 007. Install them minus read permissions. Pure preprocessor implementation of the roff .ec and .eo requests, though you are warned that very bad things will happen to anybody trying to use these macros in OpenBSD manuals. Random linking for arm64. And octeon. And alpha. And hppa. There's some variation by platform, because every architecture has the kernel loaded with different flavors of initial physical and virtual mappings. And landisk. And loongson. And sgi. And macppc. And a gap file for sparc64, but nobody yet dares split locore. And arm7. Errata for perl File::Path race condition. Some fixes for potential link attacks against cron. Add pledge violations to acct reporting. Take random linking to the next stage. More about KARL - kernel address randomized link. As noted, a few difficulties with hibernate and such, but the plan is coming together. Add a new function reorder_kernel() that relinks and installs the new kernel in the background on system startup. Add support for the bootblocks to detect hibernate and boot the previous kernel. Remove the poorly described “stuff” from ksh. Replace usage of TIOCSTI in csh using a more common IO loop. Kind of like the stuff in ksh, but part of the default command line editing and parsing code, csh would read too many characters, then send the ones it didn't like back into the terminal. Which is weird, right? Also, more importantly, eliminating the code that uses TIOCSTI to inject characters into ttys means that maybe TIOCSTI can be removed. Revamp some of the authentication logging in ssh. Add a verbose flag to rm so you can panic immediately upon seeing it delete the wrong file instead of waiting to discover your mistake after the fact. Update libexpat to version 2.2.1 which has some security fixes. Never trust an expat, that's my motto. Update inteldrm to code based on Linux 4.4.70. This brings us support for Skylake and Cherryview and better support for Broadwell and Valleyview. Also adds MST support. Fun times for people with newish laptops. *** OPNsense 17.1.9 released (https://opnsense.org/opnsense-17-1-9-released/) firewall: move gateway switching from system to firewall advanced settings firewall: keep category selection when changing tabs firewall: do not skip gateway switch parsing too early (contributed by Stephane Lesimple) interfaces: show VLAN description during edit firmware: opnsense-revert can now handle multiple packages at once firmware: opnsense-patch can now handle permission changes from patches dnsmasq: use canned –bogus-priv for noprivatereverse dnsmasq: separate log file, ACL and menu entries dynamic dns: fix update for IPv6 (contributed by Alexander Leisentritt) dynamic dns: remove usage of CURLAUTH_ANY (contributed by Alexander Leisentritt) intrusion detection: suppress “fast mode available” boot warning in PCAP mode openvpn: plugin framework adaption unbound: add local-zone type transparent for PTR zone (contributed by Davide Gerhard) unbound: separate log file, ACL and menu entries wizard: remove HTML from description strings mvc: group relation to something other than uuid if needed mvc: rework “item in” for our Volt templates lang: Czech to 100% translated (contributed by Pavel Borecki) plugins: zabbix-agent 1.1 (contributed by Frank Wall) plugins: haproxy 1.16 (contributed by Frank Wall) plugins: acme-client 1.8 (contributed by Frank Wall) plugins: tinc fix for switch mode (contributed by Johan Grip) plugins: monit 1.3 (contributed by Frank Brendel) src: support dhclient supersede statement for option 54 (contributed by Fabian Kurtz) src: add Intel Atom Cherryview SOC HSUART support src: add the ID for the Huawei ME909S LTE modem src: HardenedBSD Stack Clash mitigations[1] ports: sqlite 3.19.3[2] ports: openvpn 2.4.3[3] ports: sudo 1.8.20p2[4] ports: dnsmasq 2.77[5] ports: openldap 2.4.45[6] ports: php 7.0.20[7] ports: suricata 3.2.2[8] ports: squid 3.5.26[9] ports: carootnss 3.31 ports: bind 9.11.1-P2[10] ports: unbound 1.6.3[11] ports: curl 7.54.1[12] *** Beastie Bits Thinkpad x230 - trying to get TrackPoint / Touchpad working in X (http://lists.dragonflybsd.org/pipermail/users/2017-July/313519.html) FreeBSD deprecates all r-cmds (rcp, rlogin, etc.) (http://marc.info/?l=freebsd-commits-all&m=149918307723723&w=2) Bashfill - art for your terminal (https://max.io/bash.html) Go 1.9 release notes: NetBSD support is broken, please help (https://github.com/golang/go/commit/32002079083e533e11209824bd9e3a797169d1c4) Jest, A ReST api for creating and managing FreeBSD jails written in Go (https://github.com/altsrc-io/Jest) *** Feedback/Questions John - zfs send/receive (http://dpaste.com/3ANETHW#wrap) Callum - laptops (http://dpaste.com/11TV0BJ) & An update (http://dpaste.com/3A14BQ6#wrap) Lars - Snapshot of VM datadisk (http://dpaste.com/0MM37NA#wrap) Daryl - Jail managers (http://dpaste.com/0CDQ9EK#wrap) ***
George Neville-Neil is a computer scientist, software engineer, and author. He is FreeBSD Core Team Member and President of The FreeBSD Foundation. He is the co-author (with Marshall Kirk McKusick) of the textbook The Design and Implementation of the FreeBSD Operating System. He has coded and published several pieces of open source software, including work on the FreeBSD Operating System, Precision Time Protocol daemon, Conductor, Packet Construction Set and Packet Debugger. He also specializes in building High-speed, low-latency systems for customers in the financial services sector. For the last 10 years, Neville-Neil has served on ACM Queue's magazine editorial board, where he developed the column Kode Vicious, which has been a regular feature in both Queue and Communications of the ACM for the last 10 years. He is also Editor in Chief at The FreeBSD Journal.
A pledge of love to OpenBSD, combating ransomware like WannaCry with OpenZFS, and using PFsense to maximize your non-gigabit Internet connection This episode was brought to you by Headlines ino64 project committed to FreeBSD 12-CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=318736) The ino64 project has been completed and merged into FreeBSD 12-CURRENT Extend the inot, devt, nlinkt types to 64-bit ints. Modify struct dirent layout to add doff, increase the size of dfileno to 64-bits, increase the size of dnamlen to 16-bits, and change the required alignment. Increase struct statfs fmntfromname[] and fmntonname[] array length MNAMELEN to 1024 This means the length of a mount point (MNAMELEN) has been increased from 88 byte to 1024 bytes. This allows longer ZFS dataset names and more nesting, and generally improves the usefulness of nested jails It also allow more than 4 billion files to be stored in a single file system (both UFS and ZFS). It also deals with a number of NFS problems, such as Amazon's EFS (cloud NFS), which uses 64 bit IDs even with small numbers of files. ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and by employing other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat around are broken in backward and forward incompatible ways. A bug in poudriere that may cause some packages to not rebuild is being fixed. Many packages like perl will need to be rebuilt after this change Update note: strictly follow the instructions in UPDATING. Build and install the new kernel with COMPAT_FREEBSD11 option enabled, then reboot, and only then install new world. So you need the new GENERIC kernel with the COMPAT_FREEBSD11 option, so that your old userland will work with the new kernel, and you need to build, install, and reboot onto the new kernel before attempting to install world. The usual process of installing both and then rebooting will NOT WORK Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kmoore) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib). Sponsored by: The FreeBSD Foundation (emaste, kib) Why I love OpenBSD (https://medium.com/@h3artbl33d/why-i-love-openbsd-ca760cf53941) Jeroen Janssen writes: I do love open source software. Oh boy, I really do love open source software. It's extendable, auditable, and customizable. What's not to love? I'm astonished by the idea that tens, hundreds, and sometimes even thousands of enthusiastic, passionate developers collaborate on an idea. Together, they make the world a better place, bit by bit. And this leads me to one of my favorite open source projects: the 22-year-old OpenBSD operating system. The origins of my love affair with OpenBSD From Linux to *BSD The advantages of OpenBSD It's extremely secure It's well documented It's open source > It's neat and clean My take on OpenBSD ** DO ** Combating WannaCry and Other Ransomware with OpenZFS Snapshots (https://www.ixsystems.com/blog/combating-ransomware/) Ransomware attacks that hold your data hostage using unauthorized data encryption are spreading rapidly and are particularly nefarious because they do not require any special access privileges to your data. A ransomware attack may be launched via a sophisticated software exploit as was the case with the recent “WannaCry” ransomware, but there is nothing stopping you from downloading and executing a malicious program that encrypts every file you have access to. If you fail to pay the ransom, the result will be indistinguishable from your simply deleting every file on your system. To make matters worse, ransomware authors are expanding their attacks to include just about any storage you have access to. The list is long, but includes network shares, Cloud services like DropBox, and even “shadow copies” of data that allow you to open previous versions of files. To make matters even worse, there is little that your operating system can do to prevent you or a program you run from encrypting files with ransomware just as it can't prevent you from deleting the files you own. Frequent backups are touted as one of the few effective strategies for recovering from ransomware attacks but it is critical that any backup be isolated from the attack to be immune from the same attack. Simply copying your files to a mounted disk on your computer or in the Cloud makes the backup vulnerable to infection by virtue of the fact that you are backing up using your regular permissions. If you can write to it, the ransomware can encrypt it. Like medical workers wearing hazmat suits for isolation when combating an epidemic, you need to isolate your backups from ransomware. OpenZFS snapshots to the rescue OpenZFS is the powerful file system at the heart of every storage system that iXsystems sells and of its many features, snapshots can provide fast and effective recovery from ransomware attacks at both the individual user and enterprise level as I talked about in 2015. As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks. OpenZFS snapshots in practice While OpenZFS is available on a number of desktop operating systems such as TrueOS and macOS, the most effective way to bring the benefits of OpenZFS snapshots to the largest number of users is with a network of iXsystems TrueNAS, FreeNAS Certified and FreeNAS Mini unified NAS and SAN storage systems. All of these can provide OpenZFS-backed SMB, NFS, AFP, and iSCSI file and block storage to the smallest workgroups up through the largest enterprises and TrueNAS offers available Fibre Channel for enterprise deployments. By sharing your data to your users using these file and block protocols, you can provide them with a storage infrastructure that can quickly recover from any ransomware attack thrown at it. To mitigate ransomware attacks against individual workstations, TrueNAS and FreeNAS can provide snapshotted storage to your VDI or virtualization solution of choice. Best of all, every iXsystems TrueNAS, FreeNAS Certified, and FreeNAS Mini system includes a consistent user interface and the ability to replicate between one another. This means that any topology of individual offices and campuses can exchange backup data to quickly mitigate ransomware attacks on your organization at all levels. Join us for a free webinar (http://www.onlinemeetingnow.com/register/?id=uegudsbc75) with iXsystems Co-Founder Matt Olander and learn more about why businesses everywhere are replacing their proprietary storage platforms with TrueNAS then email us at info@ixsystems.com or call 1-855-GREP-4-IX (1-855-473-7449), or 1-408-493-4100 (outside the US) to discuss your storage needs with one of our solutions architects. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @twitter (https://twitter.com/mwlauthor) Books, conferences, and how these two combine + BR: Welcome back. Tell us what you've been up to since the last time we interviewed you regarding books and such. + AJ: Tell us a little bit about relayd and what it can do. + BR: What other books do you have in the pipeline? + AJ: What are your criteria that qualifies a topic for a mastery book? + BR: Can you tell us a little bit about these writing workshops that you attend and what happens there? + AJ: Without spoiling too much: How did you come up with the idea for git commit murder? + BR: Speaking of BSDCan, can you tell the first timers about what to expect in the http://www.bsdcan.org/2017/schedule/events/890.en.html (Newcomers orientation and mentorship) session on Thursday? + AJ: Tell us about the new WIP session at BSDCan. Who had the idea and how much input did you get thus far? + BR: Have you ever thought about branching off into a new genre like children's books or medieval fantasy novels? + AJ: Is there anything else before we let you go? News Roundup Using LLDP on FreeBSD (https://tetragir.com/freebsd/networking/using-lldp-on-freebsd.html) LLDP, or Link Layer Discovery Protocol allows system administrators to easily map the network, eliminating the need to physically run the cables in a rack. LLDP is a protocol used to send and receive information about a neighboring device connected directly to a networking interface. It is similar to Cisco's CDP, Foundry's FDP, Nortel's SONMP, etc. It is a stateless protocol, meaning that an LLDP-enabled device sends advertisements even if the other side cannot do anything with it. In this guide the installation and configuration of the LLDP daemon on FreeBSD as well as on a Cisco switch will be introduced. If you are already familiar with Cisco's CDP, LLDP won't surprise you. It is built for the same purpose: to exchange device information between peers on a network. While CDP is a proprietary solution and can be used only on Cisco devices, LLDP is a standard: IEEE 802.3AB. Therefore it is implemented on many types of devices, such as switches, routers, various desktop operating systems, etc. LLDP helps a great deal in mapping the network topology, without spending hours in cabling cabinets to figure out which device is connected with which switchport. If LLDP is running on both the networking device and the server, it can show which port is connected where. Besides physical interfaces, LLDP can be used to exchange a lot more information, such as IP Address, hostname, etc. In order to use LLDP on FreeBSD, net-mgmt/lldpd has to be installed. It can be installed from ports using portmaster: #portmaster net-mgmt/lldpd Or from packages: #pkg install net-mgmt/lldpd By default lldpd sends and receives all the information it can gather , so it is advisable to limit what we will communicate with the neighboring device. The configuration file for lldpd is basically a list of commands as it is passed to lldpcli. Create a file named lldpd.conf under /usr/local/etc/ The following configuration gives an example of how lldpd can be configured. For a full list of options, see %man lldpcli To check what is configured locally, run #lldpcli show chassis detail To see the neighbors run #lldpcli show neighbors details Check out the rest of the article about enabling LLDP on a Cisco switch experiments with prepledge (http://www.tedunangst.com/flak/post/experiments-with-prepledge) Ted Unangst takes a crack at a system similar to the one being designed for Capsicum, Oblivious Sandboxing (See the presentation at BSDCan), where the application doesn't even know it is in the sandbox MP3 is officially dead, so I figure I should listen to my collection one last time before it vanishes entirely. The provenance of some of these files is a little suspect however, and since I know one shouldn't open files from strangers, I'd like to take some precautions against malicious malarkey. This would be a good use for pledge, perhaps, if we can get it working. At the same time, an occasional feature request for pledge is the ability to specify restrictions before running a program. Given some untrusted program, wrap its execution in a pledge like environment. There are other system call sandbox mechanisms that can do this (systrace was one), but pledge is quite deliberately designed not to support this. But maybe we can bend it to our will. Our pledge wrapper can't be an external program. This leaves us with the option of injecting the wrapper into the target program via LD_PRELOAD. Before main even runs, we'll initialize what needs initializing, then lock things down with a tight pledge set. Our eventual target will be ffplay, but hopefully the design will permit some flexibility and reuse. So the new code is injected to override the open syscall, and reads a list of files from an environment variable. Those files are opened and the path and file descriptor are put into a linked list, and then pledge is used to restrict further access to the file system. The replacement open call now searches just that linked list, returning the already opened file descriptors. So as long as your application only tries to open files that you have preopened, it can function without modification within the sandbox. Or at least that is the goal... ffplay tries to dlopen() some things, and because of the way dlopen() works, it doesn't go via the libc open() wrapper, so it doesn't get overridden ffplay also tries to call a few ioctl's, not allowed After stubbing both of those out, it still doesn't work and it is just getting worse Ted switches to a new strategy, using ffmpeg to convert the .mp3 to a .wav file and then just cat it to /dev/audio A few more stubs for ffmpeg, including access(), and adding tty access to the list of pledges, and it finally works This point has been made from the early days, but I think this exercise reinforces it, that pledge works best with programs where you understand what the program is doing. A generic pledge wrapper isn't of much use because the program is going to do something unexpected and you're going to have a hard time wrangling it into submission. Software is too complex. What in the world is ffplay doing? Even if I were working with the source, how long would it take to rearrange the program into something that could be pledged? One can try using another program, but I would wager that as far as multiformat media players go, ffplay is actually on the lower end of the complexity spectrum. Most of the trouble comes from using SDL as an abstraction layer, which performs a bunch of console operations. On the flip side, all of this early init code is probably the right design. Once SDL finally gets its screen handle setup, we could apply pledge and sandbox the actual media decoder. That would be the right way to things. Is pledge too limiting? Perhaps, but that's what I want. I could have just kept adding permissions until ffplay had full access to my X socket, but what kind of sandbox is that? I don't want naughty MP3s scraping my screen and spying on my keystrokes. The sandbox I created had all the capabilities one needs to convert an MP3 to audible sound, but the tool I wanted to use wasn't designed to work in that environment. And in its defense, these were new post hoc requirements. Other programs, even sed, suffer from less than ideal pledge sets as well. The best summary might be to say that pledge is designed for tomorrow's programs, not yesterday's (and vice versa). There were a few things I could have done better. In particular, I gave up getting audio to work, even though there's a nice description of how to work with pledge in the sio_open manual. Alas, even going back and with a bit more effort I still haven't succeeded. The requirements to use libsndio are more permissive than I might prefer. How I Maximized the Speed of My Non-Gigabit Internet Connection (https://medium.com/speedtest-by-ookla/engineer-maximizes-internet-speed-story-c3ec0e86f37a) We have a new post from Brennen Smith, who is the Lead Systems Engineer at Ookla, the company that runs Speedtest.net, explaining how he used pfSense to maximize his internet connection I spend my time wrangling servers and internet infrastructure. My daily goals range from designing high performance applications supporting millions of users and testing the fastest internet connections in the world, to squeezing microseconds from our stack —so at home, I strive to make sure that my personal internet performance is running as fast as possible. I live in an area with a DOCSIS ISP that does not provide symmetrical gigabit internet — my download and upload speeds are not equal. Instead, I have an asymmetrical plan with 200 Mbps download and 10 Mbps upload — this nuance considerably impacted my network design because asymmetrical service can more easily lead to bufferbloat. We will cover bufferbloat in a later article, but in a nutshell, it's an issue that arises when an upstream network device's buffers are saturated during an upload. This causes immense network congestion, latency to rise above 2,000 ms., and overall poor quality of internet. The solution is to shape the outbound traffic to a speed just under the sending maximum of the upstream device, so that its buffers don't fill up. My ISP is notorious for having bufferbloat issues due to the low upload performance, and it's an issue prevalent even on their provided routers. They walk through a list of router devices you might consider, and what speeds they are capable of handling, but ultimately ended up using a generic low power x86 machine running pfSense 2.3 In my research and testing, I also evaluated IPCop, VyOS, OPNSense, Sophos UTM, RouterOS, OpenWRT x86, and Alpine Linux to serve as the base operating system, but none were as well supported and full featured as PFSense. The main setting to look at is the traffic shaping of uploads, to keep the pipe from getting saturated and having a large buffer build up in the modem and further upstream. This build up is what increases the latency of the connection As with any experiment, any conclusions need to be backed with data. To validate the network was performing smoothly under heavy load, I performed the following experiment: + Ran a ping6 against speedtest.net to measure latency. + Turned off QoS to simulate a “normal router”. + Started multiple simultaneous outbound TCP and UDP streams to saturate my outbound link. + Turned on QoS to the above settings and repeated steps 2 and 3. As you can see from the plot below, without QoS, my connection latency increased by ~1,235%. However with QoS enabled, the connection stayed stable during the upload and I wasn't able to determine a statistically significant delta. That's how I maximized the speed on my non-gigabit internet connection. What have you done with your network? FreeBSD on 11″ MacBook Air (https://www.geeklan.co.uk/?p=2214) Sevan Janiyan writes in his tech blog about his experiences running FreeBSD on an 11'' MacBook Air This tiny machine has been with me for a few years now, It has mostly run OS X though I have tried OpenBSD on it (https://www.geeklan.co.uk/?p=1283). Besides the screen resolution I'm still really happy with it, hardware wise. Software wise, not so much. I use an external disk containing a zpool with my data on it. Among this data are several source trees. CVS on a ZFS filesystem on OS X is painfully slow. I dislike that builds running inside Terminal.app are slow at the expense of a responsive UI. The system seems fragile, at the slightest push the machine will either hang or become unresponsive. Buggy serial drivers which do not implement the break signal and cause instability are frustrating. Last week whilst working on Rump kernel (http://rumpkernel.org/) builds I introduced some new build issues in the process of fixing others, I needed to pick up new changes from CVS by updating my copy of the source tree and run builds to test if issues were still present. I was let down on both counts, it took ages to update source and in the process of cross compiling a NetBSD/evbmips64-el release, the system locked hard. That was it, time to look what was possible elsewhere. While I have been using OS X for many years, I'm not tied to anything exclusive on it, maybe tweetbot, perhaps, but that's it. On the BSDnow podcast they've been covering changes coming in to TrueOS (formerly PC-BSD – a desktop focused distro based on FreeBSD), their experiments seemed interesting, the project now tracks FreeBSD-CURRENT, they've replaced rcng with OpenRC as the init system and it comes with a pre-configured desktop environment, using their own window manager (Lumina). Booting the USB flash image it made it to X11 without any issue. The dock has a widget which states the detected features, no wifi (Broadcom), sound card detected and screen resolution set to 1366×768. I planned to give it a try on the weekend. Friday, I made backups and wiped the system. TrueOS installed without issue, after a short while I had a working desktop, resuming from sleep worked out of the box. I didn't spend long testing TrueOS, switching out NetBSD-HEAD only to realise that I really need ZFS so while I was testing things out, might as well give stock FreeBSD 11-STABLE a try (TrueOS was based on -CURRENT). Turns out sleep doesn't work yet but sound does work out of the box and with a few invocations of pkg(8) I had xorg, dwm, firefox, CVS and virtuabox-ose installed from binary packages. VirtualBox seems to cause the system to panic (bug 219276) but I should be able to survive without my virtual machines over the next few days as I settle in. I'm considering ditching VirtualBox and converting the vdi files to raw images so that they can be written to a new zvol for use with bhyve. As my default keyboard layout is Dvorak, OS X set the EFI settings to this layout. The first time I installed FreeBSD 11-STABLE, I opted for full disk encryption but ran into this odd issue where on boot the keyboard layout was Dvorak and password was accepted, the system would boot and as it went to mount the various filesystems it would switch back to QWERTY. I tried entering my password with both layout but wasn't able to progress any further, no bug report yet as I haven't ruled myself out as the problem. Thunderbolt gigabit adapter –bge(4) (https://www.freebsd.org/cgi/man.cgi?query=bge) and DVI adapter both worked on FreeBSD though the gigabit adapter needs to be plugged in at boot to be detected. The trackpad bind to wsp(4) (https://www.freebsd.org/cgi/man.cgi?query=wsp), left, right and middle clicks are available through single, double and tripple finger tap. Sound card binds to snd_hda(4) (https://www.freebsd.org/cgi/man.cgi?query=snd_hda) and works out of the box. For wifi I'm using a urtw(4) (https://www.freebsd.org/cgi/man.cgi?query=urtw) Alfa adapter which is a bit on the large side but works very reliably. A copy of the dmesg (https://www.geeklan.co.uk/files/macbookair/freebsd-dmesg.txt) is here. Beastie Bits OPNsense - call-for-testing for SafeStack (https://forum.opnsense.org/index.php?topic=5200.0) BSD 4.4: cat (https://www.rewritinghistorycasts.com/screencasts/bsd-4.4:-cat) Continuous Unix commit history from 1970 until today (https://github.com/dspinellis/unix-history-repo) Update on Unix Architecture Evolution Diagrams (https://www.spinellis.gr/blog/20170510/) “Relayd and Httpd Mastery” is out! (https://blather.michaelwlucas.com/archives/2951) Triangle BSD User Group Meeting -- libxo (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) *** Feedback/Questions Carlos - ASUS Tinkerboard (http://dpaste.com/1GJHPNY#wrap) James - Firewall question (http://dpaste.com/0QCW933#wrap) Adam - ZFS books (http://dpaste.com/0GMG5M2#wrap) David - Managing zvols (http://dpaste.com/2GP8H1E#wrap) ***
This week on BSD Now we cover the latest FreeBSD Status Report, a plan for Open Source software development, centrally managing bhyve with Ansible, libvirt, and pkg-ssh, and a whole lot more. This episode was brought to you by Headlines FreeBSD Project Status Report (January to March 2017) (https://www.freebsd.org/news/status/report-2017-01-2017-03.html) While a few of these projects indicate they are a "plan B" or an "attempt III", many are still hewing to their original plans, and all have produced impressive results. Please enjoy this vibrant collection of reports, covering the first quarter of 2017. The quarterly report opens with notes from Core, The FreeBSD Foundation, the Ports team, and Release Engineering On the project front, the Ceph on FreeBSD project had made considerable advances, and is now usable as the net/ceph-devel port via the ceph-fuse module. Eventually they hope to have a kernel RADOS block device driver, so fuse is not required CloudABI update, including news that the Bitcoin reference implementation is working on a port to CloudABI eMMC Flash and SD card updates, allowing higher speeds (max speed changes from ~40 to ~80 MB/sec). As well, the MMC Stack can now also be backed by the CAM framework. Improvements to the Linuxulator More detail on the pNFS Server plan B that we discussed in a previous week Snow B.V. is sponsoring a dutch translation of the FreeBSD Handbook using the new .po system *** A plan for open source software maintainers (http://www.daemonology.net/blog/2017-05-11-plan-for-foss-maintainers.html) Colin Percival describes in his blog “a plan for open source software maintainers”: I've been writing open source software for about 15 years now; while I'm still wet behind the ears compared to FreeBSD greybeards like Kirk McKusick and Poul-Henning Kamp, I've been around for long enough to start noticing some patterns. In particular: Free software is expensive. Software is expensive to begin with; but good quality open source software tends to be written by people who are recognized as experts in their fields (partly thanks to that very software) and can demand commensurate salaries. While that expensive developer time is donated (either by the developers themselves or by their employers), this influences what their time is used for: Individual developers like doing things which are fun or high-status, while companies usually pay developers to work specifically on the features those companies need. Maintaining existing code is important, but it is neither fun nor high-status; and it tends to get underweighted by companies as well, since maintenance is inherently unlikely to be the most urgent issue at any given time. Open source software is largely a "throw code over the fence and walk away" exercise. Over the past 15 years I've written freebsd-update, bsdiff, portsnap, scrypt, spiped, and kivaloo, and done a lot of work on the FreeBSD/EC2 platform. Of these, I know bsdiff and scrypt are very widely used and I suspect that kivaloo is not; but beyond that I have very little knowledge of how widely or where my work is being used. Anecdotally it seems that other developers are in similar positions: At conferences I've heard variations on "you're using my code? Wow, that's awesome; I had no idea" many times. I have even less knowledge of what people are doing with my work or what problems or limitations they're running into. Occasionally I get bug reports or feature requests; but I know I only hear from a very small proportion of the users of my work. I have a long list of feature ideas which are sitting in limbo simply because I don't know if anyone would ever use them — I suspect the answer is yes, but I'm not going to spend time implementing these until I have some confirmation of that. A lot of mid-size companies would like to be able to pay for support for the software they're using, but can't find anyone to provide it. For larger companies, it's often easier — they can simply hire the author of the software (and many developers who do ongoing maintenance work on open source software were in fact hired for this sort of "in-house expertise" role) — but there's very little available for a company which needs a few minutes per month of expertise. In many cases, the best support they can find is sending an email to the developer of the software they're using and not paying anything at all — we've all received "can you help me figure out how to use this" emails, and most of us are happy to help when we have time — but relying on developer generosity is not a good long-term solution. Every few months, I receive email from people asking if there's any way for them to support my open source software contributions. (Usually I encourage them to donate to the FreeBSD Foundation.) Conversely, there are developers whose work I would like to support (e.g., people working on FreeBSD wifi and video drivers), but there isn't any straightforward way to do this. Patreon has demonstrated that there are a lot of people willing to pay to support what they see as worthwhile work, even if they don't get anything directly in exchange for their patronage. It seems to me that this is a case where problems are in fact solutions to other problems. To wit: Users of open source software want to be able to get help with their use cases; developers of open source software want to know how people are using their code. Users of open source software want to support the the work they use; developers of open source software want to know which projects users care about. Users of open source software want specific improvements; developers of open source software may be interested in making those specific changes, but don't want to spend the time until they know someone would use them. Users of open source software have money; developers of open source software get day jobs writing other code because nobody is paying them to maintain their open source software. I'd like to see this situation get fixed. As I envision it, a solution would look something like a cross between Patreon and Bugzilla: Users would be able sign up to "support" projects of their choosing, with a number of dollars per month (possibly arbitrary amounts, possibly specified tiers; maybe including $0/month), and would be able to open issues. These could be private (e.g., for "technical support" requests) or public (e.g., for bugs and feature requests); users would be able to indicate their interest in public issues created by other users. Developers would get to see the open issues, along with a nominal "value" computed based on allocating the incoming dollars of "support contracts" across the issues each user has expressed an interest in, allowing them to focus on issues with higher impact. He poses three questions to users about whether or not people (users and software developers alike) would be interested in this and whether payment (giving and receiving, respectively) is interesting Check out the comments (and those on https://news.ycombinator.com/item?id=14313804 (reddit.com)) as well for some suggestions and discussion on the topic *** OpenBSD vmm hypervisor: Part 2 (http://www.h-i-r.net/2017/04/openbsd-vmm-hypervisor-part-2.html) We asked for people to write up their experience using OpenBSD's VMM. This blog post is just that This is going to be a (likely long-running, infrequently-appended) series of posts as I poke around in vmm. A few months ago, I demonstrated some basic use of the vmm hypervisor as it existed in OpenBSD 6.0-CURRENT around late October, 2016. We'll call that video Part 1. Quite a bit of development was done on vmm before 6.1-RELEASE, and it's worth noting that some new features made their way in. Work continues, of course, and I can only imagine the hypervisor technology will mature plenty for the next release. As it stands, this is the first release of OpenBSD with a native hypervisor shipped in the base install, and that's exciting news in and of itself To get our virtual machines onto the network, we have to spend some time setting up a virtual ethernet interface. We'll run a DHCP server on that, and it'll be the default route for our virtual machines. We'll keep all the VMs on a private network segment, and use NAT to allow them to get to the network. There is a way to directly bridge VMs to the network in some situations, but I won't be covering that today. Create an empty disk image for your new VM. I'd recommend 1.5GB to play with at first. You can do this without doas or root if you want your user account to be able to start the VM later. I made a "vmm" directory inside my home directory to store VM disk images in. You might have a different partition you wish to store these large files in. Boot up a brand new vm instance. You'll have to do this as root or with doas. You can download a -CURRENT install kernel/ramdisk (bsd.rd) from an OpenBSD mirror, or you can simply use the one that's on your existing system (/bsd.rd) like I'll do here. The command will start a VM named "test.vm", display the console at startup, use /bsd.rd (from our host environment) as the boot image, allocate 256MB of memory, attach the first network interface to the switch called "local" we defined earlier in /etc/vm.conf, and use the test image we just created as the first disk drive. Now that the VM disk image file has a full installation of OpenBSD on it, build a VM configuration around it by adding the below block of configuration (with modifications as needed for owner, path and lladdr) to /etc/vm.conf I've noticed that VMs with much less than 256MB of RAM allocated tend to be a little unstable for me. You'll also note that in the "interface" clause, I hard-coded the lladdr that was generated for it earlier. By specifying "disable" in vm.conf, the VM will show up in a stopped state that the owner of the VM (that's you!) can manually start without root access. Let us know how VMM works for you *** News Roundup openbsd changes of note 621 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-621) More stuff, more fun. Fix script to not perform tty operations on things that aren't ttys. Detected by pledge. Merge libdrm 2.4.79. After a forced unmount, also unmount any filesystems below that mount point. Flip previously warm pages in the buffer cache to memory above the DMA region if uvm tells us it is available. Pages are not automatically promoted to upper memory. Instead it's used as additional memory only for what the cache considers long term buffers. I/O still requires DMA memory, so writing to a buffer will pull it back down. Makefile support for systems with both gcc and clang. Make i386 and amd64 so. Take a more radical approach to disabling colours in clang. When the data buffered for write in tmux exceeds a limit, discard it and redraw. Helps when a fast process is running inside tmux running inside a slow terminal. Add a port of witness(4) lock validation tool from FreeBSD. Use it with mplock, rwlock, and mutex in the kernel. Properly save and restore FPU context in vmm. Remove KGDB. It neither compiles nor works. Add a constant time AES implementation, from BearSSL. Remove SSHv1 from ssh. and more... *** Digging into BSD's choice of Unix group for new directories and files (https://utcc.utoronto.ca/~cks/space/blog/unix/BSDDirectoryGroupChoice) I have to eat some humble pie here. In comments on my entry on an interesting chmod failure, Greg A. Woods pointed out that FreeBSD's behavior of creating everything inside a directory with the group of the directory is actually traditional BSD behavior (it dates all the way back to the 1980s), not some odd new invention by FreeBSD. As traditional behavior it makes sense that it's explicitly allowed by the standards, but I've also come to think that it makes sense in context and in general. To see this, we need some background about the problem facing BSD. In the beginning, two things were true in Unix: there was no mkdir() system call, and processes could only be in one group at a time. With processes being in only one group, the choice of the group for a newly created filesystem object was easy; it was your current group. This was felt to be sufficiently obvious behavior that the V7 creat(2) manpage doesn't even mention it. Now things get interesting. 4.1c BSD seems to be where mkdir(2) is introduced and where creat() stops being a system call and becomes an option to open(2). It's also where processes can be in multiple groups for the first time. The 4.1c BSD open(2) manpage is silent about the group of newly created files, while the mkdir(2) manpage specifically claims that new directories will have your effective group (ie, the V7 behavior). This is actually wrong. In both mkdir() in sysdirectory.c and maknode() in ufssyscalls.c, the group of the newly created object is set to the group of the parent directory. Then finally in the 4.2 BSD mkdir(2) manpage the group of the new directory is correctly documented (the 4.2 BSD open(2) manpage continues to say nothing about this). So BSD's traditional behavior was introduced at the same time as processes being in multiple groups, and we can guess that it was introduced as part of that change. When your process can only be in a single group, as in V7, it makes perfect sense to create new filesystem objects with that as their group. It's basically the same case as making new filesystem objects be owned by you; just as they get your UID, they also get your GID. When your process can be in multiple groups, things get less clear. A filesystem object can only be in one group, so which of your several groups should a new filesystem object be owned by, and how can you most conveniently change that choice? One option is to have some notion of a 'primary group' and then provide ways to shuffle around which of your groups is the primary group. Another option is the BSD choice of inheriting the group from context. By far the most common case is that you want your new files and directories to be created in the 'context', ie the group, of the surrounding directory. If you fully embrace the idea of Unix processes being in multiple groups, not just having one primary group and then some number of secondary groups, then the BSD choice makes a lot of sense. And for all of its faults, BSD tended to relatively fully embrace its changes While it leads to some odd issues, such as the one I ran into, pretty much any choice here is going to have some oddities. Centrally managed Bhyve infrastructure with Ansible, libvirt and pkg-ssh (http://www.shellguardians.com/2017/05/centrally-managed-bhyve-infrastructure.html) At work we've been using Bhyve for a while to run non-critical systems. It is a really nice and stable hypervisor even though we are using an earlier version available on FreeBSD 10.3. This means we lack Windows and VNC support among other things, but it is not a big deal. After some iterations in our internal tools, we realised that the installation process was too slow and we always repeated the same steps. Of course, any good sysadmin will scream "AUTOMATION!" and so did we. Therefore, we started looking for different ways to improve our deployments. We had a look at existing frameworks that manage Bhyve, but none of them had a feature that we find really important: having a centralized repository of VM images. For instance, SmartOS applies this method successfully by having a backend server that stores a catalog of VMs and Zones, meaning that new instances can be deployed in a minute at most. This is a game changer if you are really busy in your day-to-day operations. The following building blocks are used: The ZFS snapshot of an existing VM. This will be our VM template. A modified version of oneoff-pkg-create to package the ZFS snapshots. pkg-ssh and pkg-repo to host a local FreeBSD repo in a FreeBSD jail. libvirt to manage our Bhyve VMs. The ansible modules virt, virtnet and virtpool. Once automated, the installation process needs 2 minutes at most, compared with the 30 minutes needed to manually install VM plus allowing us to deploy many guests in parallel. NetBSD maintainer in the QEMU project (https://blog.netbsd.org/tnf/entry/netbsd_maintainer_in_the_qemu) QEMU - the FAST! processor emulator - is a generic, Open Source, machine emulator and virtualizer. It defines state of the art in modern virtualization. This software has been developed for multiplatform environments with support for NetBSD since virtually forever. It's the primary tool used by the NetBSD developers and release engineering team. It is run with continuous integration tests for daily commits and execute regression tests through the Automatic Test Framework (ATF). The QEMU developers warned the Open Source community - with version 2.9 of the emulator - that they will eventually drop support for suboptimally supported hosts if nobody will step in and take the maintainership to refresh the support. This warning was directed to major BSDs, Solaris, AIX and Haiku. Thankfully the NetBSD position has been filled - making NetBSD to restore official maintenance. Beastie Bits OpenBSD Community Goes Gold (http://undeadly.org/cgi?action=article&sid=20170510012526&mode=flat&count=0) CharmBUG's Tor Hack-a-thon has been pushed back to July due to scheduling difficulties (https://www.meetup.com/CharmBUG/events/238218840/) Direct Rendering Manager (DRM) Driver for i915, from the Linux kernel to Haiku with the help of DragonflyBSD's Linux Compatibility layer (https://www.haiku-os.org/blog/vivek/2017-05-05_[gsoc_2017]_3d_hardware_acceleration_in_haiku/) TomTom lists OpenBSD in license (https://twitter.com/bsdlme/status/863488045449977864) London Net BSD Meetup on May 22nd (https://mail-index.netbsd.org/regional-london/2017/05/02/msg000571.html) KnoxBUG meeting May 30th, 2017 - Introduction to FreeNAS (http://knoxbug.org/2017-05-30) *** Feedback/Questions Felix - Home Firewall (http://dpaste.com/35EWVGZ#wrap) David - Docker Recipes for Jails (http://dpaste.com/0H51NX2#wrap) Don - GoLang & Rust (http://dpaste.com/2VZ7S8K#wrap) George - OGG feed (http://dpaste.com/2A1FZF3#wrap) Roller - BSDCan Tips (http://dpaste.com/3D2B6J3#wrap) ***
This week we have a FreeBSD Foundation development update, tell you about sprinkling in the TrueOS project, Dynamic WDS & a whole lot more! This episode was brought to you by Headlines OpenSSH Removes SSHv1 Support (http://undeadly.org/cgi?action=article&sid=20170501005206) In a series of commits starting here (http://marc.info/?l=openbsd-cvs&m=149359384905651&w=2) and ending with this one (http://marc.info/?l=openbsd-cvs&m=149359530105864&w=2), Damien Miller completed the removal of all support for the now-historic SSHv1 protocol from OpenSSH (https://www.openssh.com/). The final commit message, for the commit that removes the SSHv1 related regression tests, reads: Eliminate explicit specification of protocol in tests and loops over protocol. We only support SSHv2 now. Dropping support for SSHv1 and associated ciphers that were either suspected to or known to be broken has been planned for several releases, and has been eagerly anticipated by many in the OpenBSD camp. In practical terms this means that starting with OpenBSD-current and snapshots as they will be very soon (and further down the road OpenBSD 6.2 with OpenSSH 7.6), the arcane options you used with ssh (http://man.openbsd.org/ssh) to connect to some end-of-life gear in a derelict data centre you don't want to visit anymore will no longer work and you will be forced do the reasonable thing. Upgrade. FreeBSD Foundation April 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/april-2017-development-projects-update/) FreeBSD runs on many embedded boards that provide a USB target or USB On-the-Go (OTG) interface. This allows the embedded target to act as a USB device, and present one or more interfaces (USB device classes) to a USB host. That host could be running FreeBSD, Linux, Mac OS, Windows, Android, or another operating system. USB device classes include audio input or output (e.g. headphones), mass storage (USB flash drives), human interface device (keyboards, mice), communications (Ethernet adapters), and many others. The Foundation awarded a project grant to Edward Tomasz Napierała to develop a USB mass storage target driver, using the FreeBSD CAM Target Layer (CTL) as a backend. This project allows FreeBSD running on an embedded platform, such as a BeagleBone Black or Raspberry Pi Zero, to emulate a USB mass storage target, commonly known as a USB flash stick. The backing storage for the emulated mass storage target is on the embedded board's own storage media. It can be configured at runtime using the standard CTL configuration mechanism – the ctladm(8) utility, or the ctl.conf(5) file. The FreeBSD target can now present a mass storage interface, a serial interface (for a console on the embedded system), and an Ethernet interface for network access. A typical usage scenario for the mass storage interface is to provide users with documentation and drivers that can be accessed from their host system. This makes it easier for new users to interact with the embedded FreeBSD board, especially in cases where the host operating system may require drivers to access all of the functionality, as with Windows and OS X. They provide instructions on how to configure a BeagleBone Black to act as a flash memory stick attached to a host computer. +Check out the article, test, and report back your experiences with the new USB OTG interface. *** Spring cleaning: Hardware Update and Preview of upcoming TrueOS changes (https://www.trueos.org/blog/spring-cleaning-hardware-update-preview-upcoming-trueos-changes/) The much-abused TrueOS build server is experiencing some technical difficulties, slowing down building new packages and releasing updates. After some investigation, one problem seemed to be a bug with the Poudriere port building software. After updating builders to the new version, some of the instability is resolved. Thankfully, we won't have to rely on this server so much, because… We're getting new hardware! A TrueOS/Lumina contributor is donating a new(ish) server to the project. Special thanks to TrueOS contributor/developer q5sys for the awesome new hardware! Preview: UNSTABLE and Upcoming TrueOS STABLE update A fresh UNSTABLE release is dropping today, with a few key changes: Nvidia/graphics driver detection fixes. Boot environment listing fix (FreeBSD boot-loader only) Virtual box issues fixed on most systems. There appears to be a regression in VirtualBox 5.1 with some hardware. New icon themes for Lumina (Preferences -> Appearance -> Theme). Removal of legacy pc-diskmanager. It was broken and unmaintained, so it is time to remove it. Installer/.iso Changes (Available with new STABLE Update): The text installer has been removed. It was broken and unmaintained, so it is time to remove it. There is now a single TrueOS install image. You can still choose to install as either a server or desktop, but both options live in a single install image now. This image is still available as either an .iso or .img file. The size of the .iso and .img files is reduced about 500 Mb to around 2Gb total. We've removed Firefox and Thunderbird from the default desktop installation. These have been replaced with Qupzilla and Trojita. Note you can replace Qupzilla and Trojita with Firefox and Thunderbird via the SysAdm Appcafe after completing the TrueOS install. Grub is no longer an installation option. Instead, the FreeBSD boot-loader is always used for the TrueOS partition. rEFInd is used as the master boot-loader for multi-booting; EFI partitioning is required. Qpdfview is now preinstalled for pdf viewing. Included a slideshow during the installation with tips and screenshots. Interview - Patrick M. Hausen - hausen@punkt.de (mailto:hausen@punkt.de) Founder of Punkt.de HAST - Highly Available Storage (https://wiki.freebsd.org/HAST) News Roundup (finally) investigating how to get dynamic WDS (DWDS) working in FreeBSD! (http://adrianchadd.blogspot.com/2017/04/finally-investigating-how-to-get.html) Adrian Chadd writes in his blog: I sat down recently to figure out how to get dynamic WDS working on FreeBSD-HEAD. It's been in FreeBSD since forever, and it in theory should actually have just worked, but it's extremely not documented in any useful way. It's basically the same technology in earlier Apple Airports (before it grew into what the wireless tech world calls "Proxy-STA") and is what the "extender" technology on Qualcomm Atheros chipsets implement. A common question I get from people is "why can't I bridge multiple virtual machines on my laptop and have them show up over wifi? It works on ethernet!" And my response is "when I make dynamic WDS work, you can just make this work on FreeBSD devices - but for now, use NAT." That always makes people sad. + Goes on to explain that normal station/access point setups have up to three addresses and depending on the packet type, these can vary. There are a couple of variations in the addresses, which is more than the number of address fields in a normal 802.11 frame. The big note here is that there's not enough MAC addresses to say "please send this frame to a station MAC address, but then have them forward it to another MAC address attached behind it in a bridge." That would require 4 mac addresses in the 802.11 header, which we don't get. .. except we do. There's a separate address format where from-DS and to-DS bits in the header set to 1, which means "this frame is coming from distribution system to a distribution system", and it has four mac addresses. The RA is then the AP you're sending to, and then a fourth field indicates the eventual destination address that may be an ethernet device connected behind said STA. If you don't configure up WDS, then when you send frames from a station from a MAC address that isn't actually your 802.11 interface MAC address, the system would be confused. The STA wouldn't be able to transmit it easily, and the AP wouldn't know how to get back to your bridged ethernet addresses. The original WDS was a statically configured thing. [...] So for static configurations, this works great. You'd associate your extender AP as a station of the central AP, it'd use wpa_supplicant to setup encryption, then anything between that central AP and that extender AP (as a station) would be encrypted as normal station traffic (but, 4-address frame format.) But that's not very convenient. You have to statically configure everything, including telling your central AP about all of your satellite extender APs. If you want to replace your central AP, you have to reprogram all of your extenders to use the new MAC addresses. So, Sam Leffler came up with "dynamic WDS" - where you don't have to explicitly state the list of central/satellite APs. Instead, you configure a central AP as "dynamic WDS", and when a 4-address frame shows up from an associated station, it "promotes" it to a WDS peer for you. On the satellite AP, it will just find an AP to communicate to, and then assume it'll do WDS and start using 4-address frames. It's still a bit clunky (there's no beacon, probe request, etc IEs that say "I do dynamic WDS!" so you'd better make ALL your central APs a different SSID!) but it certainly is better than what we had. Firstly, there are scripts in src/tools/tools/net80211/ - setup.wdsmain and setup.wdsrelay. These scripts are .. well, the almost complete documentation on a dynamic WDS setup. The manpage doesn't go into anywhere near enough information. So I dug into it. It turns out that dynamic WDS uses a helper daemon - 'wlanwds' - which listens for dynamic WDS configuration changes and will do things for you. This is what runs on the central AP side. Then it started making sense! So far, so good. I followed that script, modified it a bit to use encryption, and .. well, it half worked. Association worked fine, but no traffic was passing. A little more digging showed the actual problem - the dynamic WDS example scripts are for an open/unencrypted network. If you are using an encrypted network, the central AP side needs to enable privacy on the virtual interfaces so traffic gets encrypted with the parent interface encryption keys. Now, I've only done enough testing to show that indeed it is working. I haven't done anything like pass lots of traffic via iperf, or have a mix of DWDS and normal STA peers, nor actually run it for longer than 5 minutes. I'm sure there will be issues to fix. However - I do need it at home, as I can't see the home AP from the upstairs room (and now you see why I care about DWDS!) and so when I start using it daily I'll fix whatever hilarity ensues. Why don't schools teach debugging? (https://danluu.com/teach-debugging/) A friend of mine and I couldn't understand why some people were having so much trouble; the material seemed like common sense. The Feynman Method was the only tool we needed. Write down the problem Think real hard Write down the solution The Feynman Method failed us on the last project: the design of a divider, a real-world-scale project an order of magnitude more complex than anything we'd been asked to tackle before. I understand now why half the class struggled with the earlier assignments. Without an explanation of how to systematically approach problems, anyone who didn't intuitively grasp the correct solution was in for a semester of frustration. People who were, like me, above average but not great, skated through most of the class and either got lucky or wasted a huge chunk of time on the final project. I've even seen people talented enough to breeze through the entire degree without ever running into a problem too big to intuitively understand; those people have a very bad time when they run into a 10 million line codebase in the real world. The more talented the engineer, the more likely they are to hit a debugging wall outside of school. It's one of the most fundamental skills in engineering: start at the symptom of a problem and trace backwards to find the source. It takes, at most, half an hour to teach the absolute basics – and even that little bit would be enough to save a significant fraction of those who wash out and switch to non-STEM majors. Why do we leave material out of classes and then fail students who can't figure out that material for themselves? Why do we make the first couple years of an engineering major some kind of hazing ritual, instead of simply teaching people what they need to know to be good engineers? For all the high-level talk about how we need to plug the leaks in our STEM education pipeline, not only are we not plugging the holes, we're proud of how fast the pipeline is leaking. FreeBSD: pNFS server for testing (https://lists.freebsd.org/pipermail/freebsd-fs/2017-April/024702.html) Rick Macklem has issued a call for testing his new pNFS server: I now have a pNFS server that I think is ready for testing/evaluation. It is basically a patched FreeBSD-current kernel plus nfsd daemon. If you are interested, some very basic notes on how it works and how to set it up are at: http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt (http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt) A Plan B pNFS service consists of a single MetaData Server (MDS) and K Data Servers (DS), all of which would be recent FreeBSD systems. Clients will mount the MDS as they would a single NFS server. When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size == 0. Each of these files will also have two extended attributes in the system attribute name space: pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file on a DS for this file. pnfsd.dsattr - This extended attribute stores the Size, ModifyTime and Change attributes for the file. For each regular (VREG) file, the MDS creates a data storage file on one of the K DSs, in one of the dsNN directories. The name of this file is the file handle of the file on the MDS in hexadecimal. The DSs use 20 subdirectories named "ds0" to "ds19" so that no one directory gets too large. At this time, the MDS generates File Layout layouts to NFSv4.1 clients that know how to do pNFS. For NFS clients that do not support NFSv4.1 pNFS, there will be a performance hit, since the IO RPCs will be proxied by the MDS for the DS server the data storage file resides on. The current setup does not allow for redundant servers. If the MDS or any of the K DS servers fail, the entire pNFS service will be non-functional. Looking at creating mirrored DS servers is planned, but it may be a year or more before that is implemented. I am planning on using the Flex File Layout for this, since it supports client side mirroring, where the client will write to all mirrors concurrently. Beastie Bits Openbsd changes of note 620 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-620) Why Unix commands are short (http://www.catonmat.net/blog/why-unix-commands-are-short/) OPNsense 17.1.5 released (https://opnsense.org/opnsense-17-1-5-released/) Something for Apple dual-GPU users (http://lists.dragonflybsd.org/pipermail/commits/2017-April/625847.html) pkgsrcCon 2017 CFT (https://mail-index.netbsd.org/netbsd-advocacy/2017/05/01/msg000735.html) TrueOS/Lumina Dev Q&A: May 5th 2017 (https://discourse.trueos.org/t/trueos-lumina-dev-q-a-5-4-17/1347) Feedback/Questions Peter - Jails (http://dpaste.com/0J14HGJ#wrap) Andrew - Languages and University Courses (http://dpaste.com/31AVFSF#wrap) JuniorJobs (https://wiki.freebsd.org/JuniorJobs) Steve - TrueOS and Bootloader (http://dpaste.com/1BXVZSY#wrap) Ben - ZFS questions (http://dpaste.com/0R7AW2T#wrap) Steve - Linux Emulation (http://dpaste.com/3ZR7NCC#wrap)
This week on BSDNow, reports from AsiaBSDcon, TrueOS and FreeBSD news, Optimizing IllumOS Kernel, your questions and more. This episode was brought to you by Headlines AsiaBSDcon Reports and Reviews () AsiaBSDcon schedule (https://2017.asiabsdcon.org/program.html.en) Schedule and slides from the 4th bhyvecon (http://bhyvecon.org/) Michael Dexter's trip report on the iXsystems blog (https://www.ixsystems.com/blog/ixsystems-attends-asiabsdcon-2017) NetBSD AsiaBSDcon booth report (http://mail-index.netbsd.org/netbsd-advocacy/2017/03/13/msg000729.html) *** TrueOS Community Guidelines are here! (https://www.trueos.org/blog/trueos-community-guidelines/) TrueOS has published its new Community Guidelines The TrueOS Project has existed for over ten years. Until now, there was no formally defined process for interested individuals in the TrueOS community to earn contributor status as an active committer to this long-standing project. The current core TrueOS developers (Kris Moore, Ken Moore, and Joe Maloney) want to provide the community more opportunities to directly impact the TrueOS Project, and wish to formalize the process for interested people to gain full commit access to the TrueOS repositories. These describe what is expected of community members and committers They also describe the process of getting commit access to the TrueOS repo: Previously, Kris directly handed out commit bits. Now, the Core developers have provided a small list of requirements for gaining a TrueOS commit bit: Create five or more pull requests in a TrueOS Project repository within a single six month period. Stay active in the TrueOS community through at least one of the available community channels (Gitter, Discourse, IRC, etc.). Request commit access from the core developers via core@trueos.org OR Core developers contact you concerning commit access. Pull requests can be any contribution to the project, from minor documentation tweaks to creating full utilities. At the end of every month, the core developers review the commit logs, removing elements that break the Project or deviate too far from its intended purpose. Additionally, outstanding pull requests with no active dissension are immediately merged, if possible. For example, a user submits a pull request which adds a little-used OpenRC script. No one from the community comments on the request or otherwise argues against its inclusion, resulting in an automatic merge at the end of the month. In this manner, solid contributions are routinely added to the project and never left in a state of “limbo”. The page also describes the perks of being a TrueOS committer: Contributors to the TrueOS Project enjoy a number of benefits, including: A personal TrueOS email alias: @trueos.org Full access for managing TrueOS issues on GitHub. Regular meetings with the core developers and other contributors. Access to private chat channels with the core developers. Recognition as part of an online Who's Who of TrueOS developers. The eternal gratitude of the core developers of TrueOS. A warm, fuzzy feeling. Intel Donates 250.000 $ to the FreeBSD Foundation (https://www.freebsdfoundation.org/news-and-events/latest-news/new-uranium-level-donation-and-collaborative-partnership-with-intel/) More details about the deal: Systems Thinking: Intel and the FreeBSD Project (https://www.freebsdfoundation.org/blog/systems-thinking-intel-and-the-freebsd-project/) Intel will be more actively engaging with the FreeBSD Foundation and the FreeBSD Project to deliver more timely support for Intel products and technologies in FreeBSD. Intel has contributed code to FreeBSD for individual device drivers (i.e. NICs) in the past, but is now seeking a more holistic “systems thinking” approach. Intel Blog Post (https://01.org/blogs/imad/2017/intel-increases-support-freebsd-project) We will work closely with the FreeBSD Foundation to ensure the drivers, tools, and applications needed on Intel® SSD-based storage appliances are available to the community. This collaboration will also provide timely support for future Intel® 3D XPoint™ products. Thank you very much, Intel! *** Applied FreeBSD: Basic iSCSI (https://globalengineer.wordpress.com/2017/03/05/applied-freebsd-basic-iscsi/) iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered. The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it. Then, enable ctld and start it: sysrc ctld_enable=”YES” service ctld start You can use the ctladm command to see what is going on: root@bsdtarget:/dev # ctladm lunlist (7:0:0/0): Fixed Direct Access SPC-4 SCSI device (7:0:1/1): Fixed Direct Access SPC-4 SCSI device root@bsdtarget:/dev # ctladm devlist LUN Backend Size (Blocks) BS Serial Number Device ID 0 block 10485760 512 MYSERIAL 0 MYDEVID 0 1 block 10485760 512 MYSERIAL 1 MYDEVID 1 Now, let's configure the client side: In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. sysrc iscsid_enable=”YES” service iscsid start Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session: + iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget You should now have a new device (check dmesg), in this case, da1 The guide them walks through partitioning the disk, and laying down a UFS file system, and mounting it This it walks through how to disconnect iscsi, incase you don't want it anymore This all looked nice and easy, and it works very well. Now lets see what happens when you try to mount the iSCSI from Windows Ok, that wasn't so bad. Now, instead of sharing an entire space disk on the host via iSCSI, share a zvol. Now your windows machine can be backed by ZFS. All of your problems are solved. Interview - Philipp Buehler - pbuehler@sysfive.com (mailto:pbuehler@sysfive.com) Technical Lead at SysFive, and Former OpenBSD Committer News Roundup Half a dozen new features in mandoc -T html (http://undeadly.org/cgi?action=article&sid=20170316080827) mandoc (http://man.openbsd.org/mandoc.1)'s HTML output mode got some new features Even though mdoc(7) is a semantic markup language, traditionally none of the semantic annotations were communicated to the reader. [...] Now, at least in -T html output mode, you can see the semantic function of marked-up words by hovering your mouse over them. In terminal output modes, we have the ctags(1)-like internal search facility built around the less(1) tag jump (:t) feature for quite some time now. We now have a similar feature in -T html output mode. To jump to (almost) the same places in the text, go to the address bar of the browser, type a hash mark ('#') after the URI, then the name of the option, command, variable, error code etc. you want to jump to, and hit enter. Check out the full report by Ingo Schwarze (schwarze@) and try out these new features *** Optimizing IllumOS Kernel Crypto (http://zfs-create.blogspot.com/2014/05/optimizing-illumos-kernel-crypto.html) Sašo Kiselkov, of ZFS fame, looked into the performance of the OpenSolaris kernel crypto framework and found it lacking. The article also spends a few minutes on the different modes and how they work. Recently I've had some motivation to look into the KCF on Illumos and discovered that, unbeknownst to me, we already had an AES-NI implementation that was automatically enabled when running on Intel and AMD CPUs with AES-NI support. This work was done back in 2010 by Dan Anderson.This was great news, so I set out to test the performance in Illumos in a VM on my Mac with a Core i5 3210M (2.5GHz normal, 3.1GHz turbo). The initial tests of “what the hardware can do” were done in OpenSSL So now comes the test for the KCF. I wrote a quick'n'dirty crypto test module that just performed a bunch of encryption operations and timed the results. KCF got around 100 MB/s for each algorithm, except half that for AES-GCM OpenSSL had done over 3000 MB/s for CTR mode, 500 MB/s for CBC, and 1000 MB/s for GCM What the hell is that?! This is just plain unacceptable. Obviously we must have hit some nasty performance snag somewhere, because this is comical. And sure enough, we did. When looking around in the AES-NI implementation I came across this bit in aes_intel.s that performed the CLTS instruction. This is a problem: 3.1.2 Instructions That Cause VM Exits ConditionallyCLTS. The CLTS instruction causes a VM exit if the bits in position 3 (corresponding to CR0.TS) are set in both the CR0 guest/host mask and the CR0 read shadow. The CLTS instruction signals to the CPU that we're about to use FPU registers (which is needed for AES-NI), which in VMware causes an exit into the hypervisor. And we've been doing it for every single AES block! Needless to say, performing the equivalent of a very expensive context switch every 16 bytes is going to hurt encryption performance a bit. The reason why the kernel is issuing CLTS is because for performance reasons, the kernel doesn't save and restore FPU register state on kernel thread context switches. So whenever we need to use FPU registers inside the kernel, we must disable kernel thread preemption via a call to kpreemptdisable() and kpreemptenable() and save and restore FPU register state manually. During this time, we cannot be descheduled (because if we were, some other thread might clobber our FPU registers), so if a thread does this for too long, it can lead to unexpected latency bubbles The solution was to restructure the AES and KCF block crypto implementations in such a way that we execute encryption in meaningfully small chunks. I opted for 32k bytes, for reasons which I'll explain below. Unfortunately, doing this restructuring work was a bit more complicated than one would imagine, since in the KCF the implementation of the AES encryption algorithm and the block cipher modes is separated into two separate modules that interact through an internal API, which wasn't really conducive to high performance (we'll get to that later). Anyway, having fixed the issue here and running the code at near native speed, this is what I get: AES-128/CTR: 439 MB/s AES-128/CBC: 483 MB/s AES-128/GCM: 252 MB/s Not disastrous anymore, but still, very, very bad. Of course, you've got keep in mind, the thing we're comparing it to, OpenSSL, is no slouch. It's got hand-written highly optimized inline assembly implementations of most of these encryption functions and their specific modes, for lots of platforms. That's a ton of code to maintain and optimize, but I'll be damned if I let this kind of performance gap persist. Fixing this, however, is not so trivial anymore. It pertains to how the KCF's block cipher mode API interacts with the cipher algorithms. It is beautifully designed and implemented in a fashion that creates minimum code duplication, but this also means that it's inherently inefficient. ECB, CBC and CTR gained the ability to pass an algorithm-specific "fastpath" implementation of the block cipher mode, because these functions benefit greatly from pipelining multiple cipher calls into a single place. ECB, CTR and CBC decryption benefit enormously from being able to exploit the wide XMM register file on Intel to perform encryption/decryption operations on 8 blocks at the same time in a non-interlocking manner. The performance gains here are on the order of 5-8x.CBC encryption benefits from not having to copy the previously encrypted ciphertext blocks into memory and back into registers to XOR them with the subsequent plaintext blocks, though here the gains are more modest, around 1.3-1.5x. After all of this work, this is how the results now look on Illumos, even inside of a VM: Algorithm/Mode 128k ops AES-128/CTR: 3121 MB/s AES-128/CBC: 691 MB/s AES-128/GCM: 1053 MB/s So the CTR and GCM speeds have actually caught up to OpenSSL, and CBC is actually faster than OpenSSL. On the decryption side of things, CBC decryption also jumped from 627 MB/s to 3011 MB/s. Seeing these performance numbers, you can see why I chose 32k for the operation size in between kernel preemption barriers. Even on the slowest hardware with AES-NI, we can expect at least 300-400 MB/s/core of throughput, so even in the worst case, we'll be hogging the CPU for at most ~0.1ms per run. Overall, we're even a little bit faster than OpenSSL in some tests, though that's probably down to us encrypting 128k blocks vs 8k in the "openssl speed" utility. Anyway, having fixed this monstrous atrocity of a performance bug, I can now finally get some sleep. To made these tests repeatable, and to ensure that the changes didn't break the crypto algorithms, Saso created a crypto_test kernel module. I have recently created a FreeBSD version of crypto_test.ko, for much the same purposes Initial performance on FreeBSD is not as bad, if you have the aesni.ko module loaded, but it is not up to speed with OpenSSL. You cannot directly compare to the benchmarks Saso did, because the CPUs are vastly different. Performance results (https://wiki.freebsd.org/OpenCryptoPerformance) I hope to do some more tests on a range of different sized CPUs in order to determine how the algorithms scale across different clock speeds. I also want to look at, or get help and have someone else look at, implementing some of the same optimizations that Saso did. It currently seems like there isn't a way to perform addition crypto operations in the same session without regenerating the key table. Processing additional buffers in an existing session might offer a number of optimizations for bulk operations, although in many cases, each block is encrypted with a different key and/or IV, so it might not be very useful. *** Brendan Gregg's special freeware tools for sysadmins (http://www.brendangregg.com/specials.html) These tools need to be in every (not so) serious sysadmins toolbox. Triple ROT13 encryption algorithm (beware: export restrictions may apply) /usr/bin/maybe, in case true and false don't provide too little choice... The bottom command lists you all the processes using the least CPU cycles. Check out the rest of the tools. You wrote similar tools and want us to cover them in the show? Send us an email to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** A look at 2038 (http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/) I remember the Y2K problem quite vividly. The world was going crazy for years, paying insane amounts of money to experts to fix critical legacy systems, and there was a neverending stream of predictions from the media on how it's all going to fail. Most didn't even understand what the problem was, and I remember one magazine writing something like the following: Most systems store the current year as a two-digit value to save space. When the value rolls over on New Year's Eve 1999, those two digits will be “00”, and “00” means “halt operation” in the machine language of many central processing units. If you're in an elevator at this time, it will stop working and you may fall to your death. I still don't know why they thought a computer would suddenly interpret data as code, but people believed them. We could see a nearby hydropower plant from my parents' house, and we expected it to go up in flames as soon as the clock passed midnight, while at least two airplanes crashed in our garden at the same time. Then nothing happened. I think one of the most “severe” problems was the police not being able to open their car garages the next day because their RFID tokens had both a start and end date for validity, and the system clock had actually rolled over to 1900, so the tokens were “not yet valid”. That was 17 years ago. One of the reasons why Y2K wasn't as bad as it could have been is that many systems had never used the “two-digit-year” representation internally, but use some form of “timestamp” relative to a fixed date (the “epoch”). The actual problem with time and dates rolling over is that systems calculate timestamp differences all day. Since a timestamp derived from the system clock seemingly only increases with each query, it is very common to just calculate diff = now - before and never care about the fact that now could suddenly be lower than before because the system clock has rolled over. In this case diff is suddenly negative, and if other parts of the code make further use of the suddenly negative value, things can go horribly wrong. A good example was a bug in the generator control units (GCUs) aboard Boeing 787 “Dreamliner” aircrafts, discovered in 2015. An internal timestamp counter would overflow roughly 248 days after the system had been powered on, triggering a shut down to “safe mode”. The aircraft has four generator units, but if all were powered up at the same time, they would all fail at the same time. This sounds like an overflow caused by a signed 32-bit counter counting the number of centiseconds since boot, overflowing after 248.55 days, and luckily no airline had been using their Boing 787 models for such a long time between maintenance intervals. The “obvious” solution is to simply switch to 64-Bit values and call it day, which would push overflow dates far into the future (as long as you don't do it like the IBM S/370 mentioned before). But as we've learned from the Y2K problem, you have to assume that computer systems, computer software and stored data (which often contains timestamps in some form) will stay with us for much longer than we might think. The years 2036 and 2038 might be far in the future, but we have to assume that many of the things we make and sell today are going to be used and supported for more than just 19 years. Also many systems have to store dates which are far in the future. A 30 year mortgage taken out in 2008 could have already triggered the bug, and for some banks it supposedly did. sysgettimeofday() is one of the most used system calls on a generic Linux system and returns the current time in form of an UNIX timestamp (timet data type) plus fraction (susecondst data type). Many applications have to know the current time and date to do things, e.g. displaying it, using it in game timing loops, invalidating caches after their lifetime ends, perform an action after a specific moment has passed, etc. In a 32-Bit UNIX system, timet is usually defined as a signed 32-Bit Integer. When kernel, libraries and applications are compiled, the compiler will turn this assumption machine code and all components later have to match each other. So a 32-Bit Linux application or library still expects the kernel to return a 32-Bit value even if the kernel is running on a 64-Bit architecture and has 32-Bit compatibility. The same holds true for applications calling into libraries. This is a major problem, because there will be a lot of legacy software running in 2038. Systems which used an unsigned 32-Bit Integer for timet push the problem back to 2106, but I don't know about many of those. The developers of the GNU C library (glibc), the default standard C library for many GNU/Linux systems, have come up with a design for year 2038 proofness for their library. Besides the timet data type itself, a number of other data structures have fields based on timet or the combined struct timespec and struct timeval types. Many methods beside those intended for setting and querying the current time use timestamps 32-Bit Windows applications, or Windows applications defining _USE32BITTIMET, can be hit by the year 2038 problem too if they use the timet data type. The _time64t data type had been available since Visual C 7.1, but only Visual C 8 (default with Visual Studio 2015) expanded timet to 64 bits by default. The change will only be effective after a recompilation, legacy applications will continue to be affected. If you live in a 64-Bit world and use a 64-Bit kernel with 64-Bit only applications, you might think you can just ignore the problem. In such a constellation all instances of the standard time_t data type for system calls, libraries and applications are signed 64-Bit Integers which will overflow in around 292 billion years. But many data formats, file systems and network protocols still specify 32-Bit time fields, and you might have to read/write this data or talk to legacy systems after 2038. So solving the problem on your side alone is not enough. Then the article goes on to describe how all of this will break your file systems. Not to mention your databases and other file formats. Also see Theo De Raadt's EuroBSDCon 2013 Presentation (https://www.openbsd.org/papers/eurobsdcon_2013_time_t/mgp00001.html) *** Beastie Bits Michael Lucas: Get your name in “Absolute FreeBSD 3rd Edition” (https://blather.michaelwlucas.com/archives/2895) ZFS compressed ARC stats to top (https://svnweb.freebsd.org/base?view=revision&revision=r315435) Matthew Dillon discovered HAMMER was repeating itself when writing to disk. Fixing that issue doubled write speeds (https://www.dragonflydigest.com/2017/03/14/19452.html) TedU on Meaningful Short Names (http://www.tedunangst.com/flak/post/shrt-nms-fr-clrty) vBSDcon and EuroBSDcon Call for Papers are open (https://www.freebsdfoundation.org/blog/submit-your-work-vbsdcon-and-eurobsdcon-cfps-now-open/) Feedback/Questions Craig asks about BSD server management (http://pastebin.com/NMshpZ7n) Michael asks about jails as a router between networks (http://pastebin.com/UqRwMcRk) Todd asks about connecting jails (http://pastebin.com/i1ZD6eXN) Dave writes in with an interesting link (http://pastebin.com/QzW5c9wV) > applications crash more often due to errors than corruptions. In the case of corruption, a few applications (e.g., Log-Cabin, ZooKeeper) can use checksums and redundancy to recover, leading to a correct behavior; however, when the corruption is transformed into an error, these applications crash, resulting in reduced availability. ***
This week on the show, we've got FreeBSD quarterly Status reports to discuss, OpenBSD changes to the installer, EC2 and IPv6 and more. Stay This episode was brought to you by Headlines OpenBSD changes of note 6 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-6) OpenBSD can now be cross built with clang. Work on this continues Build ld.so with -fno-builtin because otherwise clang would optimize the local versions of functions like dlmemset into a call to memset, which doesn't exist. Add connection timeout for ftp (http). Mostly for the installer so it can error out and try something else. Complete https support for the installer. I wonder how they handle certificate verification. I need to look into this as I'd like to switch the FreeBSD installer to this as well New ocspcheck utility to validate a certificate against its ocsp responder. net lock here, net lock there, net lock not quite everywhere but more than before. More per cpu counters in networking code as well. Disable and lock Silicon Debug feature on modern Intel CPUs. Prevent wireless frame injection attack described at 33C3 in the talk titled “Predicting and Abusing WPA2/802.11 Group Keys” by Mathy Vanhoef. Add support for multiple transmit ifqueues per network interface. Supported drivers include bge, bnx, em, myx, ix, hvn, xnf. pledge now tracks when a file as opened and uses this to permit or deny ioctl. Reimplement httpd's support for byte ranges. Fixes a memory DOS. FreeBSD 2016Q4 Status Report (https://www.freebsd.org/news/status/report-2016-10-2016-12.html) An overview of some of the work that happened in October - December 2016 The ports tree saw many updates and surpassed 27,000 ports The core team was busy as usual, and the foundation attended and/or sponsored a record 24 events in 2016. CEPH on FreeBSD seems to be coming along nicely. For those that do not know, CEPH is a distributed filesystem that can sit on top of another filesystem. That is, you can use it to create a clustered filesystem out of a bunch of ZFS servers. Would love to have some viewers give it a try and report back. OpenBSM, the FreeBSD audit framework, got some updates Ed Schouten committed a front end to export sysctl data in a format usable by Prometheus, the open source monitoring system. This is useful for other monitoring software too. Lots of updates for various ARM boards There is an update on Reproducible Builds in FreeBSD, “ It is now possible to build the FreeBSD base system (kernel and userland) completely reproducibly, although it currently requires a few non-default settings”, and the ports tree is at 80% reproducible Lots of toolchain updates (gcc, lld, gdb) Various updates from major ports teams *** Amazon rolls out IPv6 support on EC2 (http://www.daemonology.net/blog/2017-01-26-IPv6-on-FreeBSD-EC2.html) A few hours ago Amazon announced that they had rolled out IPv6 support in EC2 to 15 regions — everywhere except the Beijing region, apparently. This seems as good a time as any to write about using IPv6 in EC2 on FreeBSD instances. First, the good news: Future FreeBSD releases will support IPv6 "out of the box" on EC2. I committed changes to HEAD last week, and merged them to the stable/11 branch moments ago, to have FreeBSD automatically use whatever IPv6 addresses EC2 makes available to it. Next, the annoying news: To get IPv6 support in EC2 from existing FreeBSD releases (10.3, 11.0) you'll need to run a few simple commands. I consider this unfortunate but inevitable: While Amazon has been unusually helpful recently, there's nothing they could have done to get support for their IPv6 networking configuration into FreeBSD a year before they launched it. You need the dual-dhclient port: pkg install dual-dhclient And the following lines in your /etc/rc.conf: ifconfigDEFAULT="SYNCDHCP acceptrtadv" ipv6activateallinterfaces="YES" dhclientprogram="/usr/local/sbin/dual-dhclient" + It is good to see FreeBSD being ready to use this feature on day 0, not something we would have had in the past Finally, one important caveat: While EC2 is clearly the most important place to have IPv6 support, and one which many of us have been waiting a long time to get, this is not the only service where IPv6 support is important. Of particular concern to me, Application Load Balancer support for IPv6 is still missing in many regions, and Elastic Load Balancers in VPC don't support IPv6 at all — which matters to those of us who run non-HTTP services. Make sure that IPv6 support has been rolled out for all the services you need before you start migrating. Colin's blog also has the details on how to actually activate IPv6 from the Amazon side, if only it was as easy as configuring it on the FreeBSD side *** FreeBSD's George Neville-Neil tries valiantly for over an hour to convince a Linux fan of the error of their ways (https://www.youtube.com/watch?v=cofKxtIO3Is) In today's episode of the Lunduke Hour I talk to George Neville-Neil -- author and FreeBSD advocate. He tries to convince me, a Linux user, that FreeBSD is better. + They cover quite a few topics, including: + licensing, and the motivations behind it + vendor relations + community + development model + drivers and hardware support + George also talks about his work with the FreeBSD Foundation, and the book he co-authored, “The Design and Implementation of the FreeBSD Operating System, 2nd Edition” News Roundup An interactive script that makes it easy to install 50+ desktop environments following a base install of FreeBSD 11 (https://github.com/rosedovell/unixdesktops) And I thought I was doing good when I wrote a patch for the installer that enables your choice of 3 desktop environments... This is a collection of scripts meant to install desktop environments on unix-like operating systems following a base install. I call one of these 'complete' when it meets the following requirements: + A graphical logon manager is presented without user intervention after powering on the machine + Logging into that graphical logon manager takes the user into the specified desktop environment + The user can open a terminal emulator I need to revive my patch, and add Lumina to it *** Firefox 51 on sparc64 - we did not hit the wall yet (https://blog.netbsd.org/tnf/entry/firefox_51_on_sparc64_we) A NetBSD developers tells the story of getting Firefox 51 running on their sparc64 machine It turns out the bug impacted amd64 as well, so it was quickly fixed They are a bit less hopeful about the future, since Firefox will soon require rust to compile, and rust is not working on sparc64 yet Although there has been some activity on the rust on sparc64 front, so maybe there is hope The post also look at a few alternative browsers, but it not hopeful *** Introducing Bloaty McBloatface: a size profiler for binaries (http://blog.reverberate.org/2016/11/07/introducing-bloaty-mcbloatface.html) I'm very excited to announce that today I'm open-sourcing a tool I've been working on for several months at Google. It's called Bloaty McBloatface, and it lets you explore what's taking up space in your .o, .a, .so, and executable binary files. Bloaty is available under the Apache 2 license. All of the code is available on GitHub: github.com/google/bloaty. It is quick and easy to build, though it does require a somewhat recent compiler since it uses C++11 extensively. Bloaty primarily supports ELF files (Linux, BSD, etc) but there is some support for Mach-O files on OS X too. I'm interested in expanding Bloaty's capabilities to more platforms if there is interest! I need to try this one some of the boot code files, to see if there are places we can trim some fat We've been using Bloaty a lot on the Protocol Buffers team at Google to evaluate the binary size impacts of our changes. If a change causes a size increase, where did it come from? What sections/symbols grew, and why? Bloaty has a diff mode for understanding changes in binary size The diff mode looks especially interesting. It might be worth setting up some kind of CI testing that alerts if a change results in a significant size increase in a binary or library *** A BSD licensed mdns responder (https://github.com/kristapsdz/mdnsd) One of the things we just have to deal with in the modern world is service and system discovery. Many of us have fiddled with avahi or mdnsd and related “mdns” services. For various reasons those often haven't been the best-fit on BSD systems. Today we have a github project to point you at, which while a bit older, has recently been updated with pledge() support for OpenBSD. First of all, why do we need an alternative? They list their reasons: This is an attempt to bring native mdns/dns-sd to OpenBSD. Mainly cause all the other options suck and proper network browsing is a nice feature these days. Why not Apple's mdnsd ? 1 - It sucks big time. 2 - No BSD License (Apache-2). 3 - Overcomplex API. 4 - Not OpenBSD-like. Why not Avahi ? 1 - No BSD License (LGPL). 2 - Overcomplex API. 3 - Not OpenBSD-like 4 - DBUS and lots of dependencies. Those already sound like pretty compelling reasons. What makes this “new” information again is the pledge support, and perhaps it's time for more BSD's to start considering importing something like mdnsd into their base system to make system discovery more “automatic” *** Beastie Bits Benno Rice at Linux.Conf.Au: The Trouble with FreeBSD (https://www.youtube.com/watch?v=Ib7tFvw34DM) State of the Port of VMS to x86 (http://vmssoftware.com/pdfs/State_of_Port_20170105.pdf) Microsoft Azure now offers Patent Troll Protection (https://thestack.com/cloud/2017/02/08/microsoft-azure-now-offers-patent-troll-ip-protection/) FreeBSD Storage Summit 2017 (https://www.freebsdfoundation.org/news-and-events/event-calendar/freebsd-storage-summit-2017/) If you are going to be in Tokyo, make sure you come to (http://bhyvecon.org/) Feedback/Questions Farhan - Laptops (http://pastebin.com/bVqsvM3r) Hjalti - rclone (http://pastebin.com/7KWYX2Mg) Ivan - Jails (http://pastebin.com/U5XyzMDR) Jungle - Traffic Control (http://pastebin.com/sK7uEDpn) ***
This week on BSDNow, we've got all sorts of post-holiday goodies to share. New OpenSSL APIs, Dtrace, OpenBSD This episode was brought to you by Headlines OpenSSL 1.1 API migration path, or the lack thereof (https://www.mail-archive.com/tech@openbsd.org/msg36437.html) As many of you will already be aware, the OpenSSL 1.1.0 release intentionally introduced significant API changes from the previous release. In summary, a large number of data structures that were previously publically visible have been made opaque, with accessor functions being added in order to get and set some of the fields within these now opaque structs. It is worth noting that the use of opaque data structures is generally beneficial for libraries, since changes can be made to these data structures without breaking the ABI. As such, the overall direction of these changes is largely reasonable. However, while API change is generally necessary for progression, in this case it would appear that there is NO transition plan and a complete disregard for the impact that these changes would have on the overall open source ecosystem. So far it seems that the only approach is to place the migration burden onto each and every software project that uses OpenSSL, pushing significant code changes to each project that migrates to OpenSSL 1.1, while maintaining compatibility with the previous API. This is forcing each project to provide their own backwards compatibility shims, which is practically guaranteeing that there will be a proliferation of variable quality implementations; it is almost a certainty that some of these will contain bugs, potentially introducing security issues or memory leaks. I think this will be a bigger issue for other operating systems that do not have the flexibility of the ports tree to deliver a newer version of OpenSSL. If a project switches from the old API to the new API, and the OS only provides the older branch of OpenSSL, how can the application work? Of course, this leaves the issue, if application A wants OpenSSL 1.0, and application B only works with OpenSSL 1.1, how does that work? Due to a number of factors, software projects that make use of OpenSSL cannot simply migrate to the 1.1 API and drop support for the 1.0 API - in most cases they will need to continue to support both. Firstly, I am not aware of any platform that has shipped a production release with OpenSSL 1.1 - any software that supported OpenSSL 1.1 only, would effectively be unusable on every platform for the time being. Secondly, the OpenSSL 1.0.2 release is supported until the 31st of December 2019, while OpenSSL 1.1.0 is only supported until the 31st of August 2018 - any LTS style release is clearly going to consider shipping with 1.0.2 as a result. Platforms that are attempting to ship with OpenSSL 1.1 are already encountering significant challenges - for example, Debian currently has 257 packages (out of 518) that do not build against OpenSSL 1.1. There are also hidden gotchas for situations where different libraries are linked against different OpenSSL versions and then share OpenSSL data structures between them - many of these problems will be difficult to detect since they only fail at runtime. It will be interesting to see what happens with OpenSSL, and LibreSSL Hopefully, most projects will decide to switch to the cleaner APIs provided by s2n or libtls, although they do not provide the entire functionality of the OpenSSL API. Hacker News comments (https://news.ycombinator.com/item?id=13284648) *** exfiltration via receive timing (http://www.tedunangst.com/flak/post/exfiltration-via-receive-timing) Another similar way to create a backchannel but without transmitting anything is to introduce delays in the receiver and measure throughput as observed by the sender. All we need is a protocol with transmission control. Hmmm. Actually, it's easier (and more reliable) to code this up using a plain pipe, but the same principle applies to networked transmissions. For every digit we want to “send” back, we sleep a few seconds, then drain the pipe. We don't care about the data, although if this were a video file or an OS update, we could probably do something useful with it. Continuously fill the pipe with junk data. If (when) we block, calculate the difference between before and after. This is a our secret backchannel data. (The reader and writer use different buffer sizes because on OpenBSD at least, a writer will stay blocked even after a read depending on the space that opens up. Even simple demos have real world considerations.) In this simple example, the secret data (argv) is shared by the processes, but we can see that the writer isn't printing them from its own address space. Nevertheless, it works. Time to add random delays and buffering to firewalls? Probably not. An interesting thought experiment that shows just how many ways there are to covertly convey a message *** OpenBSD Desktop in about 30 Minutes (https://news.ycombinator.com/item?id=13223351) Over at hackernews we have a very non-verbose, but handy guide to getting to a OpenBSD desktop in about 30 minutes! First, the guide will assume you've already installed OpenBSD 6.0, so you'll need to at least be at the shell prompt of your freshly installed system to begin. With that, now its time to do some tuning. Editing some resource limits in login.conf will be our initial task, upping some datasize tunables to 2GB Next up, we will edit some of the default “doas” settings to something a bit more workable for desktop computing Another handy trick, editing your .profile to have your PKG_PATH variables set automatically will make One thing some folks may overlook, but disabling atime can speed disk performance (which you probably don't care about atime on your desktop anyway), so this guide will show you what knobs to tweak in /etc/fstab to do so After some final WPA / Wifi configuration, we then drop to “mere mortal” mode and begin our package installations. In this particular guide, he will be setting up Lumina Desktop (Which yes, it is on OpenBSD) A few small tweaks later for xscreensaver and your xinitrc file, then you are ready to run “startx” and begin your desktop session! All in all, great guide which if you are fast can probably be done in even less than 30 minutes and will result in a rock-solid OpenBSD desktop rocking Lumina none-the-less. *** How DTrace saved Christmas (https://hackernoon.com/dtrace-at-home-145ba773371e) Adam Leventhal, one of the co-creators of DTrace, wrote up this post about how he uses DTrace at home, to save Christmas I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I'm glad I don't have to do 2016 again. It was just before midnight and either I'd make the card that night or leave an empty space on our friends' refrigerators. Adobe Illustrator had other ideas: “Unable to set maximum number of files to be opened” I'm not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions were working for me, and — inspired by Sara Mauskopf's excellent post (https://medium.com/startup-grind/how-to-start-a-company-with-no-free-time-b70fbe7b918a#.uujdblxc6) — I was rapidly running out of the time bounds for the project. Enough; I'd just DTrace it. A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can't imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I'm forever grateful to the Apple team that ported it to Mac OS X) Illustrator is failing on setrlimit(2) and blowing up as result. Let's confirm that it is in fact returning -1:$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }' dtrace: description 'syscall::setrlimit:return' matched 1 probe CPU ID FUNCTION:NAME 0 532 setrlimit:return -1 1 There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let's confirm that it is in fact setting the files limit and check the value to which it's being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process's address space: $ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlimcur, this->r.rlimmax); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe CPU ID FUNCTION:NAME 0 531 setrlimit:entry 1008 2800 7fffffffffffffff Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMITNOFILE | _RLIMITPOSIX_FLAG) The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy: $ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlimt *)alloca(sizeof (rlimt)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlimt), sizeof (rlimt)); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe dtrace: could not enable tracing: Permission denied Oh right. Thank you SIP (System Integrity Protection). This is a new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it's new I haven't yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It's easy enough to disable, but I'm reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb + After using DTrace to get the address of the setrlimit function, Adam used lldb to change the result before it got back to the application: (lldb) break set -n _init Breakpoint 1: 47 locations. (lldb) run … (lldb) di -s 0x1006e5b72 -c 1 0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit (lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90 (lldb) di -s 0x1006e5b72 -c 4 0x1006e5b72: xorl %eax, %eax 0x1006e5b74: nop 0x1006e5b75: nop 0x1006e5b76: nop Next I just did a process detach and got on with making that holiday card… DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn't write a program doesn't mean you can't fix it. News Roundup Say my Blog's name! (https://functionallyparanoid.com/2016/12/22/say-my-blogs-name/) Brian Everly over at functionally paranoid has a treat for us today. Let us give you a moment to get the tin-foil hats on… Ok, done? Let's begin! He starts off with a look at physical security. He begins by listing your options: BIOS passwords – Not something I'm typically impressed with. Most can be avoided by opening up the machine, closing a jumper and powering it up to reset the NVRAM to factory defaults. I don't even bother with them. Full disk encryption – This one really rings my bell in a positive way. If you can kill power to the box (either because the bad actor has to physically steal it and they aren't carrying around a pile of car batteries and an inverter or because you can interrupt power to it some other way), then the disk will be encrypted. The other beauty of this is that if a drive fails (and they all do eventually) you don't have to have any privacy concerns about chucking it into an electronics recycler (or if you are a bad, bad person, into a landfill) because that data is effectively gibberish without the key (or without a long time to brute force it). Two factor auth for logins – I like this one as well. I'm not a fan of biometrics because if your fingerprint is compromised (yes, it can happen – read (https://www.washingtonpost.com/news/federal-eye/wp/2015/07/09/hack-of-security-clearance-system-affected-21-5-million-people-federal-authorities-say/) about the department of defense background checks that were extracted by a bad agent – they included fingerprint images) you can't exactly send off for a new finger. Things like the YubiKey (https://www.yubico.com/) are pretty slick. They require that you have the physical hardware key as well as the password so unless the bad actor lifted your physical key, they would have a much harder time with physical access to your hardware. Out of those options, Brian mentions that he uses disk encryption and yubi-key for all his secure network systems. Next up is network segmentation, in this case the first thing to do is change your admin password for any ISP supplied modem / router. He goes on to scare us of javascript attacks being used not against your local machine, but instead non WAN exposed router admin interface. Scary Stuff! For added security, naturally he firewalls the router by plugging in the LAN port to a OpenBSD box which does the 2nd layer of firewall / router protection. What about privacy and browsing? Here's some more of his tips: I use Unbound as my DNS resolver on my local network (with all UDP port 53 traffic redirected to it by pf so I don't have to configure anything on the clients) and then forward the traffic to DNSCrypt Proxy, caching the results in Unbound. I notice ZERO performance penalty for this and it greatly enhances privacy. This combination of Unbound and DNSCrypt Proxy works very well together. You can even have redundancy by having multiple upstream resolvers running on different ports (basically run the DNSCrypt Proxy daemon multiple times pointing to different public resolvers). I also use Firefox exclusively for my web browsing. By leveraging the tips on this page (https://www.privacytools.io/), you can lock it down to do a great job of privacy protection. The fact that your laptop's battery drain rate can be used to fingerprint your browser completely trips me out but hey – that's the world we live in.' What about the cloud you may ask? Well Brian has a nice solution for that as well: I recently decided I would try to live a cloud-free life and I'll give you a bit of a synopsis on it. I discovered a wonderful Open Source project called FreeNAS (http://www.freenas.org/). What this little gem does is allow you to install a FreeBSD/zfs file server appliance on amd64 hardware and have a slick administrative web interface for managing it. I picked up a nice SuperMicro motherboard and chassis that has 4 hot swap drive bays (and two internal bays that I used to mirror the boot volume on) and am rocking the zfs lifestyle! (Thanks Alan Jude!) One of the nicest features of the FreeNAS is that it provides the ability to leverage the FreeBSD jail functionality in an easy to use way. It also has plugins but the security on those is a bit sketchy (old versions of libraries, etc.) so I decided to roll my own. I created two jails – one to run OwnCloud (yeah, I know about NextCloud and might switch at some point) and the other to run a full SMTP/IMAP email server stack. I used Lets Encrypt (https://letsencrypt.org/) to generate the SSL certificates and made sure I hit an A on SSLLabs (https://www.ssllabs.com/) before I did anything else. His post then goes in to talk about Backups and IoT devices, something else you need to consider in this truely paranoid world we are forced to live in. We even get a nice shout-out near the end! Enter TarSnap (http://www.tarsnap.com/) – a company that advertises itself as “Online Backups for the Truly Paranoid”. It brings a tear to my eye – a kindred spirit! :-) Thanks again to Alan Jude and Kris Moore from the BSD Now podcast (http://www.bsdnow.tv/) for turning me onto this company. It has a very easy command syntax (yes, it isn't a GUI tool – suck it up buttercup, you wanted to learn the shell didn't you?) and even allows you to compile the thing from source if you want to.” We've only covered some of the highlights here, but you really should take a few moments of your time today and read this top to bottom. Lots of good tips here, already thinking how I can secure my home network better. The open source book: “Producing Open Source Software” (http://producingoss.com/en/producingoss.pdf) “How to Run a Successful Free Software Project” by Karl Fogel 9 chapters and over 200 pages of content, plus many appendices Some interesting topics include: Choosing a good name version control bug tracking creating developer guidelines setting up communications channels choosing a license (although this guide leans heavily towards the GPL) setting the tone of the project joining or creating a Non-Profit Organization the economics of open source release engineering, packaging, nightly builds, etc how to deal with forks A lot of good information packaged into this ebook This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License *** DTrace Flamegraphs for node.js on FreeBSD (http://www.venshare.com/dtrace-flamegraphs-for-freebsd-and-node-js-2/) One of the coolest tools built on top of DTrace is flamegraphs They are a very accurate, and visual way to see where a program is spending its time, which can tell you why it is slow, or where it could be improved. Further enhancements include off-cpu flame graphs, which tell you when the program is doing nothing, which can also be very useful > Recently BSD UNIXes are being acknowledged by the application development community as an interesting operating system to deploy to. This is not surprising given that FreeBSD had jails, the original container system, about 17 years ago and a lot of network focused businesses such as netflix see it as the best way to deliver content. This developer interest has led to hosting providers supporting FreeBSD. e.g. Amazon, Azure, Joyent and you can get a 2 months free instance at Digital Ocean. DTrace is another vital feature for anyone who has had to deal with production issues and has been in FreeBSD since version 9. As of FreeBSD 11 the operating system now contains some great work by Fedor Indutny so you can profile node applications and create flamegraphs of node.js processes without any additional runtime flags or restarting of processes. This is one of the most important things about DTrace. Many applications include some debugging functionality, but they require that you stop the application, and start it again in debugging mode. Some even require that you recompile the application in debugging mode. Being able to attach DTrace to an application, while it is under load, while the problem is actively happening, can be critical to figuring out what is going on. In order to configure your FreeBSD instance to utilize this feature make the following changes to the configuration of the server. Load the DTrace module at boot Increase some DTrace limits Install node with the optional DTrace feature compiled in Follow the generic node.js flamegraph tutorial (https://nodejs.org/en/blog/uncategorized/profiling-node-js/) > I hope you find this article useful. The ability to look at a runtime in this manor has saved me twice this year and I hope it will save you in the future too. My next post on freeBSD and node.js will be looking at some scenarios on utilising the ZFS features. Also check out Brendan Gregg's ACM Queue Article (http://queue.acm.org/detail.cfm?id=2927301) “The Flame Graph: This visualization of software execution is a new necessity for performance profiling and debugging” SSHGuard 2.0 Call for Testing (https://sourceforge.net/p/sshguard/mailman/message/35580961/) SSHGuard is a tool for monitoring brute force attempts and blocking them It has been a favourite of mine for a while because it runs as a pipe from syslogd, rather than reading the log files from the disk A lot of work to get SSHGuard working with new log sources (journalctl, macOS log) and backends (firewalld, ipset) has happened in 2.0. The new version also uses a configuration file. Most importantly, SSHGuard has been split into several processes piped into one another (sshg-logmon | sshg-parser | sshg-blocker | sshg-fw). sshg-parser can run with capsicum(4) and pledge(2). sshg-blocker can be sandboxed in its default configuration (without pid file, whitelist, blacklisting) and has not been tested sandboxed in other configurations. Breaking the processes up so that the sensitive bits can be sandboxes is very nice to see *** Beastie Bits pjd's 2007 paper from AsiaBSDCon: “Porting the ZFS file system to the FreeBSD operating system” (https://2007.asiabsdcon.org/papers/P16-paper.pdf) A Message From the FreeBSD Foundation (https://vimeo.com/user60888329) Remembering Roger Faulkner, Unix Champion (http://thenewstack.io/remembering-roger-faulkner/) and A few HN comments (including Bryan Cantrill) (https://news.ycombinator.com/item?id=13293596) Feedback/Questions Peter - TrueOS Network (http://pastebin.com/QtyJeHMk) Chris - Remote Desktop (http://pastebin.com/ru726VTV) Goetz - Geli on Serial (http://pastebin.com/LQZPgF5g) Joe - BGP (http://pastebin.com/jFeL8zKX) Alejandro - BSD Router (http://pastebin.com/Xq9cbmfn) ***
This week on BSDNow! We've got Netflix + FreeBSD news to discuss, always a crowd pleaser, that plus EuroBSDCon is just around the corner. Stick around for your place This episode was brought to you by Headlines Protecting Netflix Viewing Privacy at Scale, with FreeBSD (http://techblog.netflix.com/search/label/FreeBSD) This blog post from Netflix tells the story of how Netflix developed in-kernel TLS to speed up delivery of video via HTTPS Since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs - from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016. We contribute to this effort on the software side by optimizing every aspect of the software for our unique use case - in particular, focusing on the open source FreeBSD operating system and the NGINX web server that run on the OCAs. In the modern internet world, we have to focus not only on efficiency, but also security. There are many state-of-the-art security mechanisms in place at Netflix, including Transport Level Security (TLS) encryption of customer information, search queries, and other confidential data. We have always relied on pre-encoded Digital Rights Management (DRM) to secure our video streams. Over the past year, we've begun to use Secure HTTP (HTTP over TLS or HTTPS) to encrypt the transport of the video content as well. This helps protect member privacy, particularly when the network is insecure - ensuring that our members are safe from eavesdropping by anyone who might want to record their viewing habits. The goal is to ensure that your government, ISP, and wifi sniffing neighbour cannot tell which Netflix videos you are watching Netflix Open Connect serves over 125 million hours of content per day, all around the world. Given our scale, adding the overhead of TLS encryption calculations to our video stream transport had the potential to greatly reduce the efficiency of our global infrastructure. We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-GCM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously - as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method. All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance. We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L). Netflix and NGINX had previously worked together to improve our HTTP client request and response time via the use of sendfile calls to perform a zero-copy data flow from storage (HDD or SSD) to network socket, keeping the data in the kernel memory address space and relieving some of the CPU burden. The Netflix team specifically added the ability to make the sendfile calls asynchronous - further reducing the data path and enabling more simultaneous connections. However, TLS functionality, which requires the data to be passed to the application layer, was incompatible with the sendfile approach. To retain the benefits of the sendfile model while adding TLS functionality, we designed a hybrid TLS scheme whereby session management stays in the application space, but the bulk encryption is inserted into the sendfile data pipeline in the kernel. This extends sendfile to support encrypting data for TLS/SSL connections. We tested the BoringSSL and ISA-L AES-GCM implementations with our sendfile improvements against a baseline of OpenSSL (with no sendfile changes), under typical Netflix traffic conditions on three different OCA hardware types. Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline - increasing performance by up to 30%, depending on the OCA hardware version. We chose the ISA-L cipher implementation, which had slightly better results. With these improvements in place, we can continue the process of adding TLS to our video streams for clients that support it, without suffering prohibitive performance hits. If you would like more detail, check out the papers from AsiaBSDCon 2015 (https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf) and the updated one from 2016 (https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf) *** OpenBSD on HP Stream 7 (http://www.tedunangst.com/flak/post/OpenBSD-on-HP-Stream-7) Recent events have rocked the mobile computing world to its core. OpenBSD retired the zaurus port, leaving users in desperate need of a new device. And not long before that, Microsoft released the Anniversary Update to Windows 10, but with free space requirements such that it's nigh impossible to install on cheap 32GB eMMC equipped devices such as the HP Stream series, leaving users searching for a new lightweight operating system. With necessity as both mother and father, the scene is set for a truly epic pairing. OpenBSD on the HP Stream 7. The HP Stream line is a series of budget computers in a couple form factors. The Stream 11 is a fairly typical netbook. However, the Stream 7 and 8 are tablets. They look like cheap Android devices, but inside the case, they're real boys, er PCs, with Intel Atom CPUs. To install OpenBSD on such a device, we need a few parts. Obviously, the tablet itself. There's a dearth of ports on these things, but there is a micro USB port. Attaching anything useful requires an OTG “on the go” cable that creates a type A port. Attaching more than one useful thing requires a mini hub. And completing the install requires one each USB stick, keyboard, and network adapter. First, we need to prep the machine to boot from USB. Actually, before doing anything, make sure you have a full charge. It's going to be battery only from here on out. Plug everything in. Flash drive, keyboard, and network into the hub, hub into the OTG cable, cable into the port on top of the Stream. Turn on the machine while holding the volume down button. This launches a mini menu from which we can enter the BIOS. There's a little on screen keyboard in the corner, so this can be done even without a keyboard attached, but the USB keyboard should work. We need to change two settings in the boot section. First, turn off secure boot. Second, switch boot order to prefer USB. Save and exit. The first reboot reveals a confirmation screen checking that we really want to disable secure boot. We must enter a PIN and press enter. Enter the PIN shown on the screen and press enter. And we are go. Then boot up OpenBSD from the USB drive Ted then works there a number of kernel panics and device driver issues, but after disabling ACPI and IntelDRM, the device boots OpenBSD. Of course, there's no X at this point. And definitely no touch screen. And no internal networking. However, by keeping our USB hub attached, we can drive the console and access the network. At least until the battery is depleted, even if we have no way of knowing how long that will be since we disabled all the ACPI devices, which also means no suspend or resume. With some xorg.conf hacking, he did get Xorg working *** DragonflyBSD steps towards base LibreSSL (http://lists.dragonflybsd.org/pipermail/commits/2016-September/624493.html) Project: DragonFlyBSD / Switch base to use private LibreSSL libraries (http://freshbsd.org/commit/dfbsd/304ca408000cd34559ef5319b4b5a6766d6eb35b) DragonFly BSD adopts uses of LibreSSL (http://undeadly.org/cgi?action=article&sid=20160911231651) The number of projects beginning to switch over to LibreSSL is growing and it appears we can now throw DragonFly into that camp. Following something that sounds vaguely familiar (Allan!) DFLY is now creating “private” LibreSSL libraries which are only linked against by base system binaries. For the moment OpenSSL is still built, primarily so that various ports and 3rd party apps can continue to function as before. A NO_OPENSSL option has also been added, but doesn't really do much (yet), since it'll still build and install headers / libraries even if set. *** OpenBSD g2k16 Hackathon g2k16 Hackathon Report: Antoine Jacoutot on Binary Patches (http://undeadly.org/cgi?action=article&sid=20160911012316) g2k16 Hackathon Report: Matthieu Herrb on xenodm (http://undeadly.org/cgi?action=article&sid=20160911231712) g2k16 Hackathon Report: Vincent Gross on iked(8), armv7 and sys/netinet[6] (http://undeadly.org/cgi?action=article&sid=20160911000337) g2k16 Hackathon Report: Florian Obser on httpd, networking, acme-client, and more (http://undeadly.org/cgi?action=article&sid=20160911000052) g2k16 Hackathon Report: Jasper Lievisse Adriaanse on ddb(4) and more (http://undeadly.org/cgi?action=article&sid=20160909012520) g2k16 Hackathon Report: Christian Weisgerber on gettext progress, RTC work, removing kernel cruft (http://undeadly.org/cgi?action=article&sid=20160908002430) g2k16 Hackathon Report: Brent Cook on Chromebooks, crypto, and more (http://undeadly.org/cgi?action=article&sid=20160907131655) g2k16 Hackathon Report: Ted Unangst on doas, signify, code removal (http://undeadly.org/cgi?action=article&sid=20160906230610) g2k16 Hackathon Report: Marc Espie on package signing evolution (http://undeadly.org/cgi?action=article&sid=20160905235911) g2k16 Hackathon Report: Adam Wolk on ports, wireless drivers and more (http://undeadly.org/cgi?action=article&sid=20160906004915) g2k16 Hackathon Report: Mike Larkin on vmm + vmd progress (http://undeadly.org/cgi?action=article&sid=20160905134009&mode=expanded) *** News Roundup OpenBSD (with encrypted softraid) on the Chromebook Pixel (https://jcs.org/notaweblog/2016/08/26/openbsd_chromebook/) Looking for a Laptop to make your OpenBSD road-warrior? If so, we have a great blog tutorial on getting OpenBSD setup on the Chromebook Pixel with encrypted softraid! Author Joshua Stein gives us a very verbose look at how to install and dial-in the laptop perfectly. But first for those wondering about the hardware in the pixel: The Chromebook Pixel LS (2015) has an Intel Core i7 processor (Broadwell) at 2.4Ghz, 16Gb of RAM, a 2560x1700 400-nit IPS screen (239ppi), and Intel 802.11ac wireless. It has a Kingston 64Gib flash chip, of which about 54Gib can be used by OpenBSD when dual-booting with a 1Gb Chrome OS partition. Due to this being a chromebook with seaBIOS, some manual key-press trickery will be required to initially get the OpenBSD Installer up and running. From here you'll want to pay special close attention to the disk partitioning. In particular Joshua will show us how to shrink the existing encrypted /home that ChromeOS uses, keeping the dual-boot intact. This will become important if you ever plan on updating the device. From here, we move back to a more traditional setup, but with the added bonus of doing a soft-raid setup. But the fun isn't over yet! If you want to make OpenBSD the default boot, that'll require cracking the lid on the device and removing a special pink write-protect screw. And of course if you want to remove the default splash-screen image, Joshua has you covered as well, although some flashrom magic will be required. At this point you are nearly done. Final details on enabling specific bits of hardware are discussed. Most things work, apart from Audio and Bluetooth as of right now. *** doas mastery (http://www.tedunangst.com/flak/post/doas-mastery) “doas” mastery - Paging MWL! Our buddy Ted Unangst has written up a great ‘mastery' guide of the doas command, which can come in handy if you are among the un-initiated in doas land. UNIX systems have two classes of user, the super user and regular users. The super user is super, and everybody else is not. This concentration of power keeps things simple, but also means that often too much power is granted. Usually we only need super user powers to perform one task. We would rather not have such power all the time. Think of the responsibility that would entail! Like the sudo command, doas allows for subdivision of super user privileges, granting them only for specific tasks. He starts with the basic doas.conf setup, which starts with an empty config file The doas config is much like a pf ruleset, the default is to block everything > We add the root rule second because doas evaluates rules in a last match manner. root is in the wheel group, so the first rule will match, and then we need to override that with a second rule. Remember to always start with general rules, then make them more specific. *** iXsystems iXsystems to host MeetBSD (https://www.ixsystems.com/blog/ixsystems-host-meetbsd-california-2016-uc-berkeley/) FreeBSD Foundation Welcomes New Board Members New Board Members (https://www.freebsdfoundation.org/blog/freebsd-foundation-welcomes-new-board-members/) The FreeBSD Foundation has added two new board members Interview with Kylie Liang (https://www.freebsdfoundation.org/blog/new-board-member-interview-kylie-liang/) Kylie will focus on representing FreeBSD at conferences and businesses in China I live in China. There, I can act as a bridge between Chinese companies and the FreeBSD community to help drive FreeBSD adoption. Through my leadership role in the FreeBSD Foundation, I will help promote FreeBSD in China and also represent the Foundation at conferences and events in my region. Kylie leads the team the ensures FreeBSD runs well on Hyper-V and Azure, including providing commercial support for customers who run FreeBSD or FreeBSD based appliances on the Azure Cloud I joined Microsoft and started to lead the project called FreeBSD Integration Service to get FreeBSD running well on Hyper-V and Azure. To promote our work and to understand the FreeBSD ecosystem, I started to participate in FreeBSD events where I was inspired by this technical community. Interview with Philip Paeps (https://www.freebsdfoundation.org/blog/new-board-member-interview-philip-paeps/) Philip started with FreeBSD in the early 2000s and got his commit bit in 2004 The patches I submitted to make ACPI and input devices work on that laptop led to a src commit bit in 2004. While I haven't worked on ACPI or input devices since, I have been contributing to different areas of the kernel. Taking up maintainership of some ports I cared about also got me a ports commit bit after some time. Philip will continue to help run EuroBSDCon, but is also spreading the word about FreeBSD in India and Africa Primarily, I think I can be useful! I attend (and organize) a number of conferences around the world every year, particularly in regions that have a mostly “stealthy” FreeBSD community. While I clearly don't need to be on the FreeBSD Foundation board to advocate for FreeBSD, joining as a director will provide an additional asset when working in areas of the world where organizational affiliations are meaningful. Philip has also developed network drivers and various other bits and pieces, and has extensive experience working with and for hardware vendors and appliance vendors Despite intending to eventually contribute their code to the FreeBSD Project as open source, many hardware vendors still find it very difficult to engage directly with the FreeBSD development community. The Foundation helps bridge that gap and helps facilitate collaboration between commercial vendors and the FreeBSD community. I hope to make FreeBSD more visible in regions of the world where it is historically under-represented. I expect I will be attending even more conferences and getting myself invited to even more organizations. more, less, and a story of typical Unix fossilization (https://utcc.utoronto.ca/~cks/space/blog/unix/MoreAndUnixFossilization) Chris Siebenmann from the University of Toronto digs into the history of the difference between ‘less' and ‘more' In the beginning, by which we mean V7, Unix didn't have a pager at all. That was okay; Unix wasn't very visual in those days, partly because it was still sort of the era of the hard copy terminal. Then along came Berkeley and BSD. People at Berkeley were into CRT terminals, and so BSD Unix gave us things like vi and the first pager program, more (which showed up quite early, in 3BSD, although this isn't as early as vi, which appears in 2BSD). Calling a pager more is a little bit odd but it's a Unix type of name and from the beginning more prompted you with '--More--' at the bottom of the screen. All of the Unix vendors that based their work on BSD Unix (like Sun and DEC) naturally shipped versions of more along with the rest of the BSD programs, and so more spread around the BSD side of things. However, more was by no means the best pager ever; as you might expect, it was actually a bit primitive and lacking in features. So fairly early on Mark Nudelman wrote a pager with somewhat more features and it wound up being called less as somewhat of a joke. In a sane world, Unix vendors would have either replaced their version of more with the clearly superior less or at least updated their version of more to the 4.3 BSD version. Maybe less wouldn't have replaced more immediately, but certainly over say the next five years, when it kept on being better and most people kept preferring it when they had a choice.” + “This entire history has led to a series of vaguely absurd outcomes on various modern Unixes. On Solaris derivatives more is of course the traditional version with source code that can probably trace itself all the way back to 3BSD, carefully updated to SUS compliance. Solaris would never dream of changing what more is, not even if the replacement is better. Why, it might disturb someone. Oddly, FreeBSD has done the most sensible thing; they've outright replaced more with less. There is a /usr/bin/more but it's the same binary as less and as you can see the more manpage is just the less manpage. OpenBSD has done the same thing but has a specific manpage for more instead of just giving you the less manpage. So, now you can see why I say that less is more, or more, or both, at several levels. less is certainly more than more, and sometimes less literally is more (or rather more is less, to put it the right way around). Beastie Bits PC-BSD listed in the top 8 'best' alternatives to Windows 10 (http://www.computerworlduk.com/galleries/operating-systems/-free-alternatives-windows-10-3639433/) Creating a quick DNS server with a Rapsberry Pi2 and FreeBSD 11.0-RC1 (http://bsdimp.blogspot.co.uk/2016/08/creating-quick-dns-server-with.html) Dual Boot OpenBSD and Linux + UEFI (https://bsdlaptops.wordpress.com/2016/03/07/vaio-pro-11-part-2/) DesktopBSD 2.0 various versions available (Gnome, Lumina, KDE, LXDE) (http://desktopbsd.boards.net/board/10/announcements) FreeBSD gets new ZFS features including: Compressed ARC (https://svnweb.freebsd.org/base?view=revision&revision=305323) and ZFS Allocation Throttle (https://svnweb.freebsd.org/base?view=revision&revision=305331) One Floppy NetBSD Distribution (https://github.com/user340/fdgw2) A Compendium of BUGs (https://github.com/q5sys/BUGtracker) Feedback/Questions Galahad - OpenBSD X setup (http://pastebin.com/b7W6NHqs) Tang - Subtitles (http://pastebin.com/P4MUs3Pa) Ivan - Zpool Options (http://pastebin.com/LQ8yTp0G) Brad - Replication Issue (http://pastebin.com/XTK5gXMU) MJ - HBA (http://pastebin.com/TdYTMSj9) ***
Today on the show, we are going to be chatting with Michael Dexter about a variety of topics, but of course including bhyve! That plus This episode was brought to you by Headlines NetBSD Introduction (https://bsdmag.org/netbsd_intr/) We start off today's episode with a great new NetBSD article! Siju Oommen George has written an article for BSDMag, which provides a great overview of NetBSD's beginnings and what it is today. Of course you can't start an article about NetBSD without mentioning where the name came from: “The four founders of the NetBSD project, Chris Demetriou, Theo de Raadt, Adam Glass, and Charles Hannum, felt that a more open development model would benefit the project: one centered on portable, clean and correct code. They aimed to produce a unified, multi-platform, production-quality, BSD-based operating system. The name “NetBSD” was suggested by de Raadt, based on the importance and growth of networks, such as the Internet at that time, the distributed and collaborative nature of its development.” From there NetBSD has expanded, and keeping in line with its motto “Of course it runs NetBSD” it has grown to over 57 hardware platforms, including “IA-32, Alpha, PowerPC,SPARC, Raspberry pi 2, SPARC64 and Zaurus” From there topics such as pkgsrc, SMP, embedded and of course virtualization are all covered, which gives the reader a good overview of what to expect in the modern NetBSD today. Lastly, in addition to mentioning some of the vendors using NetBSD in a variety of ways, including Point-Of-Sale systems, routers and thin-clients, you may not have known about the research teams which deploy NetBSD: NASA Lewis Research Center – Satellite Networks and Architectures Branch use NetBSD almost exclusively in their investigation of TCP for use in satellite networks. KAME project – A research group for implementing IPv6, IPsec and other recent TCP/IP related technologies into BSD UNIX kernels, under BSD license. NEC Europe Ltd. established the Network Laboratories in Heidelberg, Germany in 1997, as NEC's third research facility in Europe. The Heidelberg labs focus on software-oriented research and development for the next generation Internet. SAMS-II Project – Space Acceleration Measurement System II. NASA will be measuring the microgravity environment on the International Space Station using a distributed system, consisting of NetBSD.“ My condolences, you're now the maintainer of a popular open source project (https://runcommand.io/2016/06/26/my-condolences-youre-now-the-maintainer-of-a-popular-open-source-project/) A presentation from a Wordpress conference, about what it is like to be the maintainer of a popular open source project The presentation covers the basics: Open Source is more than just the license, it is about community and involvement The difference between Maintainers and Contributors It covers some of the reasons people do not open up their code, and other common problems people run into: “I'm embarrassed by my code” (Hint: so is everyone else, post it anyway, it is the best way to learn) “I'm discouraged that I can't finish releases on time” “I'm overwhelmed by the PR backlog” “I'm frustrated when issues turn into flamewars” “I'm overcommitted on my open source involvement” “I feel all alone” Each of those points is met with advice and possible solutions So, there you have it. Open up your code, or join an existing project and help maintain it *** FreeBSD Committer Allan Jude Discusses the Advantages of FreeBSD and His Role in Keeping Millions of Servers Running (http://www.hostingadvice.com/blog/freebsd-project-under-the-hood/) An interesting twist on our normal news-stories today, we have an article featuring our very own Allan Jude, talking about why FreeBSD and the advantages of working on an open-source project. “When Allan started his own company hosting websites for video streaming, FreeBSD was the only operating system he had previously used with other hosts. Based on his experience and comfort with it, he trusted the system with the future of his budding business.A decade later, the former-SysAdmin went to a conference focused on the open-source operating system, where he ran into some of the folks on its documentation team. “They inspired me,” he told our team in a recent chat. He began writing documentation but soon wanted to contribute improvements beyond the docs.Today, Allan sits as a FreeBSD Project Committer. It's rare that you get to chat with someone involved with a massive-scale open-source project like this — rare and awesome.” From there Allan goes into some of the reasons “Why” FreeBSD, starting with Code Organization being well-maintained and documented: “The FreeBSD Project functions like an extremely well-organized world all its own. Allan explained the environment: “There's a documentation page that explains how the file system's laid out and everything has a place and it always goes in that place.”” + In addition, Allan gives us some insight into his work to bring Boot-Environments to the loader, and other reasons why FreeBSD “just makes sense” + In summary Allan wraps it up quite nicely: “An important take-away is that you don't have to be a major developer with tons of experience to make a difference in the project,” Allan said — and the difference that devs like Allan are making is incredible. If you too want to submit the commit that contributes to the project relied on by millions of web servers, there are plenty of ways to get involved! We're especially talking to SysAdmins here, as Allan noted that they are the main users of FreeBSD. “Having more SysAdmins involved in the actual build of the system means we can offer the tools they're looking for — designed the way a SysAdmin would want them designed, not necessarily the way a developer would think makes the most sense” A guide to saving electricity and time with poudriere and bhyve (http://justinholcomb.me/blog/2016/07/03/poudriere-in-bhyve-and-bare-metal.html) “This article goes over running poudriere to built packages for a Raspberry Pi with the interesting twist of running it both as a bhyve guest and then switching to running on bare metal via Fiber Channel via ctld by sharing the same ZFS volume.” “Firstly, poudriere can build packages for different architectures such as ARM. This can save hours of build time compared to building ports from said ARM device.” “Secondly, let's say a person has an always-on device (NAS) running FreeBSD. To save power, this device has a CPU with a low clock-rate and low core count. This low clock-rate and core count is great for saving power but terrible for processor intensive application such as poudriere. Let's say a person also has another physical server with fast processors and a high CPU count but draws nearly twice the power and a fan noise to match.” “To get the best of both worlds, the goal is to build the packages on the fast physical server, power it down, and then start the same ZFS volume in a bhyve environment to serve packages from the always-on device.” The tutorial walks through setting up ‘ahost', the always on machine, ‘fhost' the fast but noisy build machine, and a raspberry pi It also includes creating a zvol, configuring iSCSI over fibre channel and exporting the zvol, booting an iSCSI volume in bhyve, plus installing and setting up poudriere This it configures booting over fibre channel, and cross-building armv6 (raspberry pi) packages on the fast build machine Then the fast machine is shut down, and the zvol is booted in bhyve on the NAS Everything you need to know to make a hybrid physical/virtual machine The same setup could also work to run the same bhyve VM from either ahost or fhost bhyve does not yet support live migration, but when it does, having common network storage like the zvol will be an important part of that *** Interview - Michael Dexter - editor@callfortesting.org (mailto:editor@callfortesting.org) / @michaeldexter (https://twitter.com/michaeldexter) The RoloDexter *** iXSystems Children's Minnesota Star Studio Chooses iXsystems' TrueNAS Storage (https://www.youtube.com/watch?v=FFbdQ_05e-0) *** News Roundup FreeBSD Foundation June 2016 Update (https://www.freebsdfoundation.org/wp-content/uploads/2016/06/FreeBSD-Foundation-June-2016-Update.pdf) The FreeBSD Foundation's June newsletter is out Make sure you submit the FreeBSD Community Survey (https://www.surveymonkey.com/r/freebsd2016) by July 7th: In addition to the opening message from the executive director of the foundation, the update includes details to sponsored work on the FreeBSD VM system, reports from a number of conferences the Foundation attended, including BSDCan The results of the foundation's yearly board meeting People the foundation recognized for their contributions to FreeBSD at BSDCan And an introduction to their new “Getting Started with FreeBSD” project *** [How-To] Building the FreeBSD OS from scratch (http://www.all-nettools.com/forum/showthread.php?34422-Building-the-FreeBSD-OS-from-scratch) A tutorial over at the All-NetTools.com forums that walks through building FreeBSD from scratch I am not sure why anyone would want to build Xorg from source, but you can It covers everything in quite a bit of detail, from the installation process through adding Xorg and a window manager from source It also includes tweaking some device node permissions for easier operation as a non-root user, and configuring the firewall *** Window Systems Should Be Transparent (http://doc.cat-v.org/bell_labs/transparent_wsys/) + Rob Pike of AT&T Labs writes about why Window Systems should be transparent This is an old paper (undated, but I think from the late 80s), but may contain some timeless insights “UNIX window systems are unsatisfactory. Because they are cumbersome and complicated, they are unsuitable companions for an operating system that is appreciated for its technical elegance” “A good interface should clarify the view, not obscure it” “Mux is one window system that is popular and therefore worth studying as an example of good design. (It is not commercially important because it runs only on obsolete hardware.) This paper uses mux as a case study to illustrate some principles that can help keep a user interface simple, comfortable, and unobtrusive. When designing their products, the purveyors of commercial window systems should keep these principles in mind.” There are not many commercial window systems anymore, but “open source” was not really a big thing when this paper was written *** Roger Faulkner, of Solaris fame passed away (http://permalink.gmane.org/gmane.comp.standards.posix.austin.general/12877) “RIP Roger Faulkner: creator of the One and True /proc, slayer of the M-to-N threading model -- and the godfather of post-AT&T Unix” @bcantrill: Another great Roger Faulkner story (https://twitter.com/bcantrill/status/750442169807171584) The story of how pgrep -w saved a monitor -- if not a life (https://news.ycombinator.com/item?id=4306515) @bcantrill: With Roger Faulkner, Tim led an engineering coup inside Sun that saved Solaris circa 2.5 (https://twitter.com/bcantrill/status/750442169807171584) *** Beastie Bits: Developer Ed Maste is requesting information from those who are users of libvgl. (https://lists.freebsd.org/pipermail/freebsd-stable/2016-June/084843.html) HEADS UP: DragonFly 4.5 world reneeds rebuilding (http://lists.dragonflybsd.org/pipermail/users/2016-June/249748.html) Chris Buechler is leaving the pfSense project, the entire community thanks you for your many years of service (https://blog.pfsense.org/?p=2095) GhostBSD 10.3-BETA1 now available (http://ghostbsd.org/10.3_BETA1) DragonFlyBSD adds nvmectl (http://lists.dragonflybsd.org/pipermail/commits/2016-June/500671.html) OPNsense 16.1.18 released (https://opnsense.org/opnsense-16-1-18-released/) bhyve_graphics hit CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=302332) BUG Update FreeBSD Central Twitter account looking for a new owner (https://twitter.com/freebsdcentral/status/750053703420350465) NYCBUG meeting : Meet the Smallest BSDs: RetroBSD and LiteBSD, Brian Callahan (http://lists.nycbug.org/pipermail/talk/2016-July/016732.html) NYCBUG install fest @ HOPE (http://lists.nycbug.org/pipermail/talk/2016-June/016694.html) SemiBUG is looking for presentations for September and beyond (http://lists.nycbug.org/pipermail/semibug/2016-June/000107.html) Caleb Cooper is giving a talk on Crytpo at KnoxBUG on July 26th (http://knoxbug.org/content/2016-07-26) Feedback/Questions Leif - ZFS xfer (http://pastebin.com/vvASr64P) Zach - Python3 (http://pastebin.com/SznQHq7n) Dave - Versioning (http://pastebin.com/qkpjKEr0) David - Encrypted Disk Images (http://pastebin.com/yr7BUmv2) Eli - TLF in all the wrong places (http://pastebin.com/xby81NvC) ***
On this episode of BSDNow, we will be talking to Glen Barber and Peter Wemm of the FreeBSD RE and Cluster Admin teams! That plus our This episode was brought to you by Headlines 2016 FreeBSD Community Survey (https://www.surveymonkey.com/r/freebsd2016) We often get comments from our listeners, “I'm not a developer, how can I help out”? Well today is your chance to do something. The FreeBSD Foundation has its 2016 Community Survey online, where they are asking for feedback from you! I just did the survey, it'll take you about 5 minutes, but gives you a chance to provide valuable feedback to the foundation about things that are important to you. Be sure to answer in as much detail as possible and the foundation will review and use this feedback for its operations going forward. *** ART, OpenBSDs new routing table, single thread performances (http://www.grenadille.net/post/2016/06/17/ART-single-thread-performances) OpenBSD has changed the way routes are looked up in the kernel as part of their path to an SMP networking stack The “Allotment Routing Table” (ART) is a performance tradeoff, where more memory is used to store the routing table, in exchange for faster lookups With this new arrangement, a full BGP routing table will grow from 130MB to 180MB of memory “ART is a free multibit trie based routing table. To keep it simple, it can be seen as using more memory for fewer CPU cycles. In other words, we get a faster lookup by wasting memory. The original paper (http://www.hariguchi.org/art/art.pdf) presents some performance comparisons between two ART configurations and the BSD Radix. But how does this apply to OpenBSD?” “I asked Hrvoje Popovski to run his packet forwarding test on his Xeon box (E5-2620 v2 @ 2.10GHz, 2400.34 MHz) with ix(4) (82599) interfaces. The test setup consist of three machines with the OpenBSD box in the middle” “The simulations have been performed with an OpenBSD -current from June 9th. The machine is configured with pf(4) disabled in order to force a single route lookup for every IPv4 packet. Based on the result of the lookup the kernel decide if it should forward, deliver or drop the packet” *** BSDCan 2016 Playlist (https://www.youtube.com/playlist?list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC) The complete set of videos from BSDCan is online and ready to be consumed Remember the good-ole days where we would wait months (or years) to get videos posted from conferences? Well, who are we kidding, some conferences STILL do that, but we can't count BSDCan among them. Only two weeks out from this years exciting BSDCan, and all the videos have now landed on YouTube. Granted, this is no substitute for actually being at the conference, but even if you attended you probably missed quite a few of the talks. There are no videos of the hallway track, which is the best part of the conference Except the dinner discussion of course. and don't forget the hacker lounge *** Should you be scared of Unix signals? (http://jvns.ca/blog/2016/06/13/should-you-be-scared-of-signals/) Do you know much about UNIX Signals? Are you afraid of their complexity? Do you know there are signals other than SIGKILL? This article talks about the practical implications of signals from a programming perspective The things you need to consider when dealing with signals Basically, you register a “signal handler”, the function that will be run when a signal arrives As you program is running, if a signal arrives, your program will be interrupted. Its current state will be saved and any system calls in progress will return EINTR (Error, Interrupted), then your signal handler will be run. Once the signal handler is complete, the state of your application will be restored, and execution will resume As long as your program properly handles this interruption, and errors that might result from it (getting EINTR from a read() call, instead of the data you expected), then everything should be fine. Of course, you need to be careful what you do inside your signal handler, as if you modify any variables or state in your application, it might be very confused when it resumes. *** Interview - Glen and Peter- News Roundup Unik - The Unikernel Compilation and Deployment Platform (uses NetBSD's Rump) (https://github.com/emc-advanced-dev/unik) We've talked a bit about NetBSD's RUMP (unikernel) in the past, including articles on how to deploy services using it. Now we have an interesting project which makes the process super-easy, and dare-we-say almost “Docker-Like?” The Unik project has a fairly complete walkthrough right on their GitHub project page, including details on installation and creating your own unikernel containers. In addition, it provides instructions on boot-strapping your own Go/Node.js/Python/Java applications, and supports out of Box VCenter / AWS / Qemu / VirtualBox providers. *** PkgSrc 50th Release Highlights () pkgsrc is celebrating its 50th release, and to highlight this, they have posted a series of interviews from people who have been active in the project pkgsrc 50th release interviews - Jonathan Perkin (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_jonathan) pkgsrc 50th release interviews - Ryo ONODERA (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_ryo) pkgsrc 50th release interviews - Joerg Sonnenberg (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interview_with) pkgsrc 50th release interviews - Sevan Janiyan (https://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_sevan) *** Migrating to FreeBSD from Solaris 11 (http://justinholcomb.me/blog/2016/02/28/migration-to-freebsd-part1.html) Part 2 (http://justinholcomb.me/blog/2016/03/12/migration-to-freebsd-part2.html) Part 3 (http://justinholcomb.me/blog/2016/03/19/migration-to-freebsd-part3.html) Part 4 (http://justinholcomb.me/blog/2016/03/26/migration-to-freebsd-part4.html) Part 5 (http://justinholcomb.me/blog/2016/04/03/migration-to-freebsd-part5.html) *** How to chroot www/firefox on NetBSD (https://github.com/alnsn/localpkgsrc/tree/master/firefox-chroot) Looking for a jail-like method of running FireFox on NetBSD? (Or possibly other BSDs?) We have a github repo with details on how to setup and run FireFox using a chroot using a “webuser” account for safety. Think of this as a jail alternative, may be useful on systems with no jail support. Of interest is the method used to do X forwarding. It uses Xorg TCP listen option (which is often off by default for security reasons). Perhaps SSH X forwarding would be a better alternative. (Or nullfs mounts of /tmp) *** Beastie Bits Tredly - V1 Release Candidate (https://github.com/tredly/tredly/releases/tag/v1.0.0-rc.1) Call for Testing - ypldap testing against OpenLDAP and Microsoft Active Directory (http://lists.freebsd.org/pipermail/freebsd-current/2016-June/061775.html) BSD Magazine, June 2016 Out Now (https://bsdmag.org/) Hammer2 - Add xxhash to H2 and throw in debug stuff for performance testing (http://lists.dragonflybsd.org/pipermail/commits/2016-June/500610.html) chyves pre-announcement (http://justinholcomb.me/blog/2016/06/14/chyves-project-preannouncement.html) *** Feedback/Questions Michael - Versioning (http://pastebin.com/1hpGrmuL) Michael - Removing Encryption (http://pastebin.com/2PkrMGGx) Bostjan - PC-BSD Questions (http://pastebin.com/q5VdmNxG) Fong - ZFS Rollback (http://pastebin.com/2aedLV7d) Jochen - Docker on FBSD (http://pastebin.com/dneVZkXc) ***
Kris is on vacation this week, so allan flies solo, provides a recap of BSDCan & cover's a boatload of news including Microsoft This episode was brought to you by Headlines BSDCan Recap and Live Stream Videos (http://www.bsdcan.org/2016/) OpenBSD BSDCan 2016 papers now available (http://www.openbsd.org/papers) Allan's slides (http://allanjude.com/bsd/BSDCan2016_-_GELIBoot.pdf) and Paper (http://allanjude.com/bsd/AsiaBSDCon2016_geliboot_pdf1a.pdf) Michael W Lucas presents Allan with a gift (https://www.youtube.com/watch?v=LFgxAHkrSTg) “FreeBSD Mastery: Advanced ZedFS” (http://blather.michaelwlucas.com/archives/2698) Highlighted Tweets: Groff Arrives at BSDCan (https://twitter.com/Keltounet/status/740344735194320896) FreeBSD Foundation recognizes the contributions of Bryan Drewery, Rod Grimes, Warren Block, & Gleb Smirnoff (https://twitter.com/freebsdfndation/status/742456950676393984) A moment of silence and shots in memory in Benjamin Perrault @creepingfur (https://twitter.com/__briancallahan/status/741854476340858880) @gvnn3 sells the FreeBSD Foundation shirt off of his back for Charity (https://twitter.com/Keltounet/status/741763867471155201) Michael W. Lucas asks Matt Ahrens how to pronounce ZFS, “You can pronounce ZFS however you like, but if you pronounce it 'reiserfs', people might be confused.” (https://twitter.com/cperciva/status/741375414967410688) Sysadmin T-Shirt (https://twitter.com/BSDCan/status/741420633007874050) FreeBSD Dev Summit ran out of room on the chalkboards listing accomplishments of 11.0 (https://twitter.com/SeanChittenden/status/740904105388978176) List of things people have or want for FreeBSD 12 (https://twitter.com/Keltounet/status/740928627471159296) Matt Ahrens signing Allan's ZFS book (https://twitter.com/kprovst/status/741322268480049152?cn=bWVudGlvbg%3D%3D&refsrc=email) FreeBSD's new marketing strategy (https://twitter.com/cperciva/status/741707948469157889) Charity Auction: systemd whoopie cushion (https://twitter.com/HippyWizard/status/741768670704066560) Embarass OpenBSD's @HenningBrauer by donating $10 to charity for a selfie with him wearing a Linux t-shirt (https://twitter.com/juliefriday/status/741948048788586496) @GroffTheBSDGoat changes handlers, from @HenningBrauer to @GavinAtkinson (https://twitter.com/GroffTheBSDGoat/status/742415390798716928) Day 1 Video (https://www.youtube.com/watch?v=AOidjSS7Hsg) Day 2 Video (https://www.youtube.com/watch?v=z7pDnBO5wSM) Allan's GELIBoot talk (day 2) (https://www.youtube.com/watch?v=z7pDnBO5wSM&feature=youtu.be&list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC&t=4440) *** Media Coverage of Microsoft + FreeBSD story (https://azure.microsoft.com/en-us/blog/freebsd-now-available-in-azure-marketplace/) Microsoft has released their own custom image of FreeBSD 10.3 for the Azure Cloud “This means that not only can you quickly bring-up a FreeBSD VM in Azure, but also that in the event you need technical support, Microsoft support engineers can assist.” “Microsoft is the publisher of the FreeBSD image in the marketplace rather than the FreeBSD Foundation. The FreeBSD Foundation is supported by donations from the FreeBSD community, including companies that build their solutions on FreeBSD. They are not a solution provider or an ISV with a support organization but rather rely on a very active community that support one another. In order to ensure our customers have an enterprise SLA for their FreeBSD VMs running in Azure, we took on the work of building, testing, releasing and maintaining the image in order to remove that burden from the Foundation. We will continue to partner closely with the Foundation as we make further investments in FreeBSD on Hyper-V and in Azure.” "It's quite a significant milestone for FreeBSD community and for Microsoft to publish a supported FreeBSD image on Azure Marketplace. We really appreciate Microsoft's commitment and investment in FreeBSD project". - Justin T. Gibbs, President of FreeBSD Foundation Microsoft took a FreeBSD 10.3-RELEASE image and added additional patches, most of which they have upstreamed but that were too late for the regular 10.3 release cycle. Rather than requiring users to use a snapshot of the stable/10 branch, which would complicate the user experience, and complicate the job of the Microsoft support engineers, they created their own “certified” release This allows Microsoft to selectively deploy errata fixes to the image as well It is not clear how this affects update mechanisms like freebsd-update(8) The Register (http://www.theregister.co.uk/2016/06/09/microsoft_freebsd/) The Inquirer (http://www.theinquirer.net/inquirer/news/2461070/microsoft-creates-own-distribution-of-freebsd-for-azure-developers) Infoworld (http://www.infoworld.com/article/3082090/open-source-tools/is-microsoft-publishing-its-own-freebsd-yes-and-no.html) The Hacker News (http://thehackernews.com/2016/06/microsoft-azure-freebsd.html) Windows Report (http://windowsreport.com/microsoft-freebsd-10-3-ready-made-vm-image-azure/) Windows Club (http://news.thewindowsclub.com/microsoft-freebsd-operating-system-84375/) *** Select works poorly (http://www.tedunangst.com/flak/post/select-works-poorly) “At the bottom of the OpenBSD man page for select is a little note. “Internally to the kernel, select() and pselect() work poorly if multiple processes wait on the same file descriptor.” There's a similar warning in the poll man page. Where does this warning come from and what does it mean?” Ted found that at first glance, OpenBSD's select() appears to be quite bad: “whenever some data gets written, we call wakeup(&selwait);. Based on what we've seen so far, one can conclude that this is likely to be inefficient. Every time any socket has some data available, we wake up every selecting process in the system. Works poorly indeed.” After further investigation, it turns out to not be quite as bad When the select() is first setup, the PID of the process that cares about the FD is recorded in the selinfo struct If a second process runs select() on the same FD, the SI_COLL (Select Collision) flag is set on the selinfo struct When selwakeup() is called, if SI_COLL is set, all select()ing processes are woken up, and the sysctl kern.nselcoll is incremented. If the flag is not set, and only a single PID is waiting for activity on that FD, only that process is woken up “This is not an intractable problem. kevent avoids it entirely. Other implementations may too. But practically, does it need to be solved? My laptop says it's happened 43 times. A server with substantially more uptime says 0. Doesn't seem so bad.” *** Interview - Hans Petter Selasky - hps@freebsd.org (mailto:hps@freebsd.org) / @twitter (https://twitter.com/user) Designing FreeBSD's USB drivers, hooking up a piano to FreeBSD & more! *** News Roundup Timeline of libexpat random vulnerability (http://www.tedunangst.com/flak/post/timeline-of-libexpat-random-vulnerability) Do you use FreeBSD as web server? Why or why not? (https://news.ycombinator.com/item?id=11804565) 20 years of NetBSD code Bloat (http://kristerw.blogspot.sg/2016/05/20-years-of-netbsd-code-bloat.html) HP Chromebook 13 now booting OpenBSD (https://jcs.org/statuses/2016/06/08/740606952149942272/) UNIX for Poets (https://web.stanford.edu/class/cs124/lec/124-UnixForPoets.pdf) Comparing live version upgrade methods (https://distrowatch.com/weekly.php?issue=20160530#upgrades) My life with FreeBSD on a Thinkpad X220 (https://www.reddit.com/r/BSD/comments/4n3flx/my_life_with_freebsd_on_a_thinkpad_x220/)
This week, Allan is out of town, but since when has that ever stopped us from bringing you a new episode of BSDNow? We have news, This episode was brought to you by Headlines Unix's file durability problem (https://utcc.utoronto.ca/~cks/space/blog/unix/FileSyncProblem) Another article by Chris Siebenmann from the University of Toronto This time, the issue was a lost comment on his Python based blog which uses files on disk rather than a database After an unexpected restart of the system, a recently posted comment no longer existed The post goes on to investigate what the ‘right way' to ensure file durability is The answer, as you might expect, is “it depends…” Normally, fsync() should work, but it seems with ext4 and some other file systems, you must also fsync() the directory where the file was created, or it might not be possible to find the file after a crash Do you need to fsync() the parent of that directory too? Then what is fdatasync() for? What about just calling sync()? “One issue is that unlike many other Unix API issues, it's impossible to test to see if you got it all correct and complete. If your steps are incomplete, you don't get any errors; your data is just silently sometimes at risk. Even with a test setup to create system crashes or abrupt power loss (which VMs make much easier), you need uncommon instrumentation to know things like if your OS actually issued disk flushes or just did normal buffered writes. And straightforward testing can't tell you if what you're doing will work all the time, because what is required varies by Unix, kernel version, and the specific filesystem involved.” Second post by author: How I'm trying to do durable disk writes (https://utcc.utoronto.ca/~cks/space/blog/python/HowISyncDataDWiki) Additional Discussion on Hacker News (https://news.ycombinator.com/item?id=11511269) The discussion on HN also gets into AIO and other more complicated facilities, but even those seem to be vague about when your data is actually safe At least ZFS ensures you never get half of your new data, and half of your old data. *** Build a FreeBSD 10.3-release Openstack Image with bsd-cloudinit (https://raymii.org/s/tutorials/FreeBSD_10.3-release_Openstack_Image.html) Are you using FreeBSD and OpenStack or would you like to be? We next have a great tutorial which explains the ins-and-outs of doing exactly that. Remy van Elst brings us a great walkthrough on his site on how to get started, and hint it involves just a few ‘pip' commands. After getting the initial Python tools bootstrapped, next he shows us how to save our OpenStack settings in a sourceable shell command, which comes in handy before doing admin on a instance. Next the ‘glance' and ‘cinder' tools are used to upload the target OS ISO file and then create a volume for it to install onto. Next the VM is started and some specific steps are outlined on getting FreeBSD 10.3 installed into the instance. It includes some helpful hints as how to fix a mountroot error, if you installed to ada0, but need to mount via vtdb0 instead now. After the installation is finished, the prep for ‘cloudinit' is done, and the resulting image is compressed and made ready for deployment. We've kinda stepped through some of the more gory steps here, but if OpenStack is something you work with, this tutorial should be at the top of your “must read” list. *** Undeadly and HTTPS (http://undeadly.org/cgi?action=article&sid=20160411201504) Undeadly, the OpenBSD journal, is thinking of moving to HTTPS only In order to do this, they would like some help rewriting part of the site Currently, when you login to post comments, this is done over HTTPS, but to an stunnel instance running a custom script that gives you a cookie, and sends you back to the non-HTTPS site They would like to better integrate the authentication system, and otherwise improve the code for the site There is some pushback as well, questioning whether it makes sense to block users who are unable to use HTTPS for one reason or another I think it makes sense to have the site default to HTTPS, but, maybe HTTPS only doesn't make sense. There is nothing private on the site, other than the authentication system which is optional, not required to post a comment. There is also some discussion about the code for the site, including the fact that when the code was released, the salt for the password database was included This is not actually a security problem, but the discussion may be interesting to some viewers *** FreeBSD Journal March/April Edition (https://www.freebsdfoundation.org/journal/browser-based-edition/) The next issue of the FreeBSD Journal is here, and this time it is about Teaching with Operating Systems In addition to the usual columns, including: svn update, the ports report, a conference report from FOSDEM, a meetup report from PortsCamp Taipei, A book review of "The Algorithm Design Manual", and the Events Calendar; there are a set of feature articles about teaching Teaching with FreeBSD through Tracing, Analysis, and Experimentation CHERI: Building a foundation for secure, trusted computing bases A brief history of Fast Filesystems There is also an interview with Gleb Smirnoff, a member of the Core team, release engineering, and the deputy security officer, as well as a senior software developer at Netflix Get the latest issue from your favourite mobile store, or the “Desktop Edition” directly in your browser from the FreeBSD Foundation's website *** Interview - Brooks Davis - brooks@FreeBSD.org (mailto:brooks@FreeBSD.org) / @brooksdavis (https://twitter.com/brooksdavis) CHERI and Capabilities *** TrueNAS Three-Peats!!! (https://www.ixsystems.com/blog/truenas-three-peats/) News Roundup UbuntuBSD Is Looking To Become An Official Ubuntu Flavor (http://linux.softpedia.com/blog/ubuntubsd-is-looking-to-become-an-official-ubuntu-flavor-502746.shtml) You may recall a few weeks back that we were a bit surprised by the UbuntuBSD project and its longevity / goals. However the project seems to be pushing forward, with news on softpedia.com that they are now seeking to become an ‘official' Ubuntu Flavor. They've already released a forth beta, so it seems the project currently has some developers pushing it forward: "I would like to contribute all my work to Ubuntu Community and, if you think it is worthy, make ubuntuBSD an official Ubuntu project like Xubuntu or Edubuntu," said Jon Boden. "If you're interested, please let me know how would you like me to proceed." It's Just Bits (http://blog.appliedcompscilab.com/its_just_bits/index.html) We have next an interesting blog post talking about the idea that “It's just all bits!” The author then takes us down the idea of no matter how old or mysterious the code may be, in the end it is ending up as bits arranged a certain way. Then the article transitions and takes us through the idea that old bits, and bits that have grown too large should often be good candidates for replacement by “simpler” bits, using OpenBSD as an example. “The OpenBSD community exemplifies this in many ways by taking existing solutions and simplifying them. Processing man pages is as old as Unix, and even in the 21st century OpenBSD has taken the time to rewrite the existing solution to be simpler and safer. It's just bits that need to be turned into other bits. Similarly, OpenBSD has introduced doas as an alternative to sudo. While not replacing sudo entirely, doas makes the 99.99% case of what people use sudo for easier and safer. They are just bits that need to be authenticated. “ All in all, a good read, and it reinforces the point that nothing is really truly “finished”. As computing advances and new technologies / practices are made available, sometimes it makes a lot of sense to go back and re-write things in order to simplify the complexity that has snuck in over time. *** Disk IO limiting is coming to FreeBSD (https://lists.freebsd.org/pipermail/svn-src-head/2016-April/084288.html) A much requested feature for both Jails and VM's on FreeBSD has just landed with experimental support in -HEAD, Disk IO limiting! The Commit message states as follows: “Add four new RCTL resources - readbps, readiops, writebps and writeiops, for limiting disk (actually filesystem) IO. Note that in some cases these limits are not quite precise. It's ok, as long as it's within some reasonable bounds. Testing - and review of the code, in particular the VFS and VM parts - is very welcome.” Well, what are you waiting for? This is a fantastic new feature which I'm sure will get incorporated into other tools for controlling jails and VM's down the road. If you give it a spin, be sure to report back bugs so they can get quashed in time for 11. *** BeastieBits PC-BSD 10.3 Is the Last in the Series, PC-BSD 11.0 Arrives Later This Year (http://news.softpedia.com/news/pc-bsd-10-3-is-the-last-in-the-series-pc-bsd-11-0-arrives-later-this-year-502570.shtml) ASLR now on by default in NetBSD amd64 (http://mail-index.netbsd.org/source-changes/2016/04/10/msg073939.html) Daniel Bilik's fix for hangs on Baytrail (http://lists.dragonflybsd.org/pipermail/users/2016-April/228682.html) Don't forget about PGCon 2016 (http://www.pgcon.org/2016/) Get your paper in for EuroBSDCon 2016, deadline is May 8th (https://2016.eurobsdcon.org/call-for-papers/) Feedback/Questions John - Destroy all Dataset (http://pastebin.com/QdGWn0TW) Thomas - Misc Questions (http://pastebin.com/43YkwBjP) Ben - ZFS Copy (http://pastebin.com/gdi3pswe) Bryson - SysV IPC (http://pastebin.com/E9n938D1) Drin - IPSEC (http://pastebin.com/bgGTmbDG) ***
This week on the show, we will be talking to Benedict Reushling about his role with the FreeBSD foundation and the journey that took him This episode was brought to you by Headlines HardenedBSD introduces full PIE support (https://hardenedbsd.org/article/shawn-webb/2016-04-15/introducing-full-pie-support) PIE base for amd64 and i386 Only nine applications are not compiled as PIEs Tested PIE base on several amd64 systems, both virtualized and bare metal Hoped to be to enabled it for ARM64 before or during BSDCan. Shawn will be bringing ten Raspberry Pi 3 devices (which are ARM64) with to BSDCan, eight of which will be given out to lucky individuals. “We want the BSD community to hack on them and get ARM64/Aarch64 fully functional on them.” *** Lessons learned from 30 years of MINIX (http://m.cacm.acm.org/magazines/2016/3/198874-lessons-learned-from-30-years-of-minix/fulltext) Eat your own dog food. By not relying on idiosyncratic features of the hardware, one makes porting to new platforms much easier. The Internet is like an elephant; it never forgets. When standards exist (such as ANSI Standard C) stick to them. Even after you have adopted a strategy, you should nevertheless reexamine it from time to time. Keep focused on your real goal, Einstein was right: Things should be as simple as possible but not simpler. *** pfSense 2.3 released (https://blog.pfsense.org/?p=2008) Rewrite of the webGUI utilizing Bootstrap TLS v1.0 disabled for the GUI Moved to a FreeBSD 10.3-RELEASE base PHP Upgraded to 5.6 The "Full Backup" feature has been deprecated Closed 760 total tickets of which 137 are fixed bugs Known Regressions OpenVPN topology change IP aliases with CARP IP parent lose their parent interface association post-upgrade IPsec IPComp does not work. IGMP Proxy does not work with VLAN interfaces. Many other updates and changes *** OPNsense 16.1.10 released (https://opnsense.org/opnsense-16-1-10-released/) openvpn: revive windows installer binaries system: improved config history and backup pages layout system: increased backup count default from 30 to 60 system: /var /tmp MFS awareness for crash dumps added trust: add “IP security IKE intermediate” to server key usage firmware: moved reboot, halt and defaults pages to new home languages: updates to Russian, French, German and Japanese Many other updates and changes *** Interview - Benedict Reuschling - bcr@freebsd.org (mailto:bcr@freebsd.org) FreeBSD Foundation in Europe *** News Roundup Write opinionated workarounds (http://www.daemonology.net/blog/2016-04-11-write-opinionated-workarounds.html) Colin Percival has written a great blog post this past week, specifically talking about his policy of writing “opinionated workarounds”. The idea came about due to his working on multi-platform software, and the frustrations of dealing with POSIX violations The crux of the post is how he deals with these workarounds. Specifically by only applying them to the particular system in which it was required. And doing so loudly. This has some important benefits. First, it doesn't potentially expose other systems to bugs / security flaws when a workaround doesn't “work” on a system for which it wasn't designed. Secondly it's important to complain. Loudly. This lets the user know that they are running on a system that doesn't adhere to POSIX compliance, and maybe even get the attention of a developer who could remedy the situation. *** Privilege escalation in calendar(1) (http://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2016-003.txt.asc) File this one under “Ouch that hurts” a new security vuln has been posted, this time against NetBSD's ‘calendar' command. Specifically it looks like some of the daily scripts uses the ‘-a' flag, which requires super-user privs in order to process all users calendar files and mail the results. However the bug occurred because the calendar command didn't drop priv properly before executing external commands (whoops!) To workaround you can set run_calendar=NO in the daily.conf file, or apply the fixed binary from upstream. *** PGCon 2016 (http://www.pgcon.org/2016/) PGCon 2016 is now only 4 weeks away The conference will be held at the University of Ottawa (same venue as BSDCan) from May 17th to 20th Tutorials: 17-18 May 2016 (Tue & Wed) Talks: 19-20 May 2016 (Thu-Fri) Wednesday is a developer unconference. Saturday is a user unconference. “PGCon is an annual conference for users and developers of PostgreSQL, a leading relational database, which just happens to be open source. PGCon is the place to meet, discuss, build relationships, learn valuable insights, and generally chat about the work you are doing with PostgreSQL. If you want to learn why so many people are moving to PostgreSQL, PGCon will be the place to find out why. Whether you are a casual user or you've been working with PostgreSQL for years, PGCon will have something for you.” New to PGSQL? Just a user? Long time developers? This conference has something for you. A great lineup of talks (https://www.pgcon.org/2016/schedule/events.en.html), plus unconference days focused on both users and developers *** CfP EuroBSDCon 2016 (https://2016.eurobsdcon.org/call-for-papers/) The call for papers has been issued for EuroBSDCon 2016 in Belgrade, Serbia The conference will be held from the 22nd to 25th of September, 2016 The deadline for talk submissions is: Sunday the 8th of May, 2016 Submit your talk or tutorial proposal before it is too late *** Beastie Bits “FreeBSD Mastery: Advanced ZFS” has officially been released (https://www.michaelwlucas.com/nonfiction/fmaz) Support of OpenBSD pledge(2) in programming Languages (https://gist.github.com/ligurio/f6114bd1df371047dd80ea9b8a55c104) pkgsrcCon 2016 -Call for Presentations (http://daemonforums.org/showthread.php?t=9781) Christos Zoulas talks about blacklistd (http://blog.netbsd.org/tnf/entry/talks_about_blacklistd) Penguicon 2016 Lucas Track Schedule (http://blather.michaelwlucas.com/archives/2617) Feedback/Questions Peter - NVME (http://pastebin.com/HiiDpGcT) Jeremy - Wireless Gear (http://pastebin.com/L5XeVS1H) Ted - Rpi2 Packages (http://pastebin.com/yrCEnkWt) - Cross Building Wiki (https://wiki.freebsd.org/FreeBSD/arm/crossbuild) Geoff - Jail Failover (http://pastebin.com/pYFC1vdQ) Zach - Graphical Bhyve? (http://pastebin.com/WEgN0ZVw) ***
This week on the show, we will be interviewing GNN of the FreeBSD project to talk about the new TeachBSD initiative. That plus the latest BSD headlines, all coming your way right now! This episode was brought to you by Headlines FreeBSD 10.3-RELEASE Announcement (https://www.freebsd.org/releases/10.3R/announce.html) FreeBSD 10.3 has landed, with extended support until April 30, 2018 This is likely to be the last extended support release, as starting with 11, the new support model will encourage upgrading to the latest minor version by ending support for the previous minor version approximately 2 months after each point release. The Major version / stable branch will still be supported for the same 5 year term. This will allow the FreeBSD project to move forward more quickly, while still providing the same level of long term support The UEFI boot loader is much improved, and now supports booting root-on-ZFS, and the beastie menu The beastie menu itself has been updated with support for ZFS Boot Environments The CAM Target Layer (CTL) now supports High Availability, allowing the construction of much more advanced storage systems The 64bit Linux Emulation Layer was backported Reroot support was added, allowing the system to boot off of a minimal image, such as a mfsroot and then reload all of userland from a different root file system (such as iSCSI, NFS, etc) The version of xz(1) has been updated to support multi-threaded compression sesutil(8) has been introduced, making it easier to manage large storage nodes Various ZFS updates As usual, a huge number of driver updates are also included *** How to use OpenBSD with Libreboot: detailed instructions (https://lists.nongnu.org/archive/html/libreboot/2016-04/msg00010.html) This tutorial covers installing OpenBSD on a Thinkpad X200 using Libreboot, a replacement for the traditional BIOS/firmware that comes from the manufacturer “Since 5.9, OpenBSD supports EFI boot mode, which means that it also have had to support framebuffer out of the box, so lack of proprietary VGA BIOS blob is no longer a problem and you can boot it with unmodified Libreboot binary release 20150518.” “In order to install OpenBSD on such a machine you will need someadditional preparations, since regular install59.fs won't work because bsd.rd doesn't have a framebuffer console.” A few extra steps are required to get it going, but they are outlined in the post This may be very interesting to those who prefer not to depend on binary blobs *** Linking the FreeBSD base system with lld -- status update (http://lists.llvm.org/pipermail/llvm-dev/2016-March/096449.html) The FreeBSD Foundation's Ed Maste provides an update on the LLVM mailing list about the progress of replacing the GNU linker with the lld in the FreeBSD base system “I'm pleased to report that I can now build a runnable FreeBSD system using lld as the linker (for buildworld), with a few workarounds and work-in-progress patches. I have not yet extensively tested the result but it is possible to login to the resulting system, and basic sanity tests I've tried are successful. Note that the kernel is still linked with ld.bfd.” Outstanding Issues Symbol version support (PR 23231). FreeBSD uses symbol versioning for backwards compatibility Linker script expression support (PR 26731). The FreeBSD kernel linker scripts contain expressions not currently supported by lld Library search paths. GNU LD automatically searches /lib, and lld does not the -N flag makes the text and data sections RW and does not page-align data. It is used by boot loader components. The -dc flag assigns space to common symbols when producing relocatable output (-r). It is used by the /rescue build, which is a single binary assembled from a collection of individual tools (sh, ls, fsck, ...) -Y adds a path to the default library search path. It is used by the lib32 build, which provides i386 builds of the system libraries for compatibility with i386 applications. With the ongoing work, it might be possible for FreeBSD 11 to use lld by default, although it might be best to wait to throw that particular switch *** Your favorite billion user company using BSD just flipped on encryption for all their users -- and it took 15 Engineers to do it (http://www.wired.com/2016/04/forget-apple-vs-fbi-whatsapp-just-switched-encryption-billion-people/) With the help of Moxie Marlinspike's Open Whisper Systems, WhatsApp has integrated the ‘Signal' encryption system for all messages, class, pictures, and videos sent between individuals or groups It uses public key cryptography, very similar to GPG, but with automated public key servers It also includes a system of QR codes to verify the identity of individuals in person, so you can be sure the person you are talking to is actually the person you met with WhatsApp runs their billion user network, using FreeBSD, with only about 50 engineers Only 15 of those engineers we needed to work on the project that has now deployed complete end-to-end encryption across the entire network The Wired article is very detailed and well worth the read *** Interview - George Neville-Neil - gnn@freebsd.org (mailto:gnn@freebsd.org) / @gvnn3 (https://twitter.com/gvnn3) Teaching BSD with Tracing News Roundup Faces of FreeBSD 2016: Scott Long (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2016-scott-long/) It's been awhile since we've had a new entry into the “Faces of FreeBSD” series, but due to popular demand it's back! This installment features developer Scott Long, who currently works at NetFlix, previously at Yahoo and Adaptec. Scott got a very early start into BSD, first discovering i386BSD 0.1 on a FTP server at Berkeley, back at 1992. From there on it's been a journey, following along with FreeBSD since version 1.0 in 1993. So what stuff can we blame Scott for? In his own words: I've been a source committer since 2000. I got my start by taking over maintainership of the Adaptec ‘aac' RAID driver. From 2002-2006 I was the Release Engineer and was responsible for the 5.x and 6.x releases. Though the early 5.x releases were not great, they were necessary stepping stones to the success of FreeBSD 6.x and beyond. I'm exceptionally proud of my role in helping FreeBSD move forward during that time. I authored and maintained the ‘mfi' and ‘mps' storage drivers, the ‘udf' filesystem driver, and several smaller sound and USB drivers. I've maintained, or at least touched, most of the storage device drivers in the system to some extent, and I implemented medium-grained locking on the CAM storage stack. Recently I've been working on overall system scalability and performance. ASCII Flow (http://asciiflow.com/) A website that lets to draw and share ASCII diagrams Great for network layout maps, rack diagrams, protocol analysis etc Use it in your presentations and slides Sample (https://drive.google.com/open?id=0BynxTTJrNUOKeWxCVm1ERExrNkU) *** System Under Test: FreeBSD (http://lowlevelbits.org/system-under-test-freebsd/) Part of a series looking at testing across a number of projects Outlines the testing framework of FreeBSD Provides a mini-tutorial on how to run the tests There are some other tests that are now covered, but this is due to a lack of documentation on the fact that the tests exist, and how to run them There is much ongoing work in this area *** Worst April Fools Joke EVER! (http://www.rhyous.com/2016/04/01/microsoft-announces-it-is-acquiring-freebsd-for-300-million/) While a bad April Fool's joke, it also shows some common misconceptions The FreeBSD Foundation does not own the source repository, it is only the care taken of the trademark, and other things that require a single legal entity OpenBSD and NetBSD are not ‘sub brands' of FreeBSD Bash was not ported to Windows, but rather Windows gained a system similar to FreeBSD's linux_compat It would be nice to have ZFS on Windows *** Beastie Bits Credit where credit's due... (https://forums.freebsd.org/threads/55642/) M:Tier's OpenBSD packages and binpatches updated for 5.9 (https://stable.mtier.org/) NYC BUG Meeting (2016-04-06) - Debugging with LLVM, John Wolfe (http://www.nycbug.org/index.cgi) Need to create extremely high traffic loads? kq_sendrecv is worth checking out (http://lists.dragonflybsd.org/pipermail/commits/2016-March/459651.html) If you're in the Maryland region, CharmBug has a meetup next week (http://www.meetup.com/CharmBUG/events/230048300/) How to get a desktop on DragonFly (https://www.dragonflybsd.org/docs/how_to_get_to_the_desktop/) Linux vs BSD Development Models (https://twitter.com/q5sys/status/717509675630084096) Feedback/Question Paulo - ZFS Setup (http://pastebin.com/raw/GrM0jKZK) Jonathan - Installation (http://pastebin.com/raw/13KCkhMU) Andrew - Career / School (http://pastebin.com/wsx90L2m)
This week on the show, Allan and I have gotten a bit more sleep since AsiaBSDCon, which is excellent since there is a LOT of news to cover. That plus our interview with Ports SecTeam member Mark Felder. So keep it This episode was brought to you by Headlines FreeNAS 9.10 Released (http://lists.freenas.org/pipermail/freenas-announce/2016-March/000028.html) OS: The base OS version for FreeNAS 9.10 is now FreeBSD 10.3-RC3, bringing in a huge number of OS-related bug fixes, performance improvements and new features. +Directory Services: You can now connect to large AD domains with cache disabled. +Reporting: Add the ability to send collectd data to a remote graphite server. +Hardware Support: Added Support for Intel I219-V & I219-LM Gigabit Ethernet Chipset Added Support for Intel Skylake architecture Improved support for USB devices (like network adapters) USB 3.0 devices now supported. +Filesharing: Samba (SMB filesharing) updated from version 4.1 to 4.3.4 Added GUI feature to allow nfsv3-like ownership when using nfsv4 Various bug fixes related to FreeBSD 10. +Ports: FreeBSD ports updated to follow the FreeBSD 2016Q1 branch. +Jails: FreeBSD Jails now default to a FreeBSD 10.3-RC2 based template. Old jails, or systems on which jails have been installed, will still default to the previous FreeBSD 9.3 based template. Only those machinesusing jails for the first time (or deleting and recreating their jails dataset) will use the new template. +bhyve: ++In the upcoming 10 release, the CLI will offer full support for managing virtual machines and containers. Until then, the iohyve command is bundled as a stop-gap solution to provide basic VM management support - *** Ubuntu BSD's first Beta Release (https://sourceforge.net/projects/ubuntubsd/) Under the category of “Where did this come from?”, we have a first beta release of Ubuntu BSD. Specifically it is Ubuntu, respun to use the FreeBSD kernel and ZFS natively. From looking at the minimal information up on sourceforge, we gather that is has a nice text-based installer, which supports ZFS configuration and iSCSI volume creation setups. Aside from that, it includes the XFCE desktop out of box, but claims to be suitable for both desktops and servers alike right now. We will keep an eye on this, if anybody listening has already tested it out, maybe drop us a line on your thoughts of how this mash-up works out. *** FreeBSD - a lesson in poor defaults (http://vez.mrsk.me/freebsd-defaults.txt) Former BSD producer, and now OpenBSD developer, TJ, writes a post detailing the defaults he changes in a fresh FreeBSD installation Maybe some of these should be the defaults While others are definitely a personal preference, or are not as security related as they seem A few of these, while valid criticisms, but some are done for a reason Specifically, the OpenSSH changes. So, you're a user, you install FreeBSD 10.0, and it comes with OpenSSH version X, which has some specific defaults As guaranteed by the FreeBSD Project, you will have a nice smooth upgrade path to any version in the 10.x branch Just because OpenSSH has released version Y, doesn't mean that the upgrade can suddenly remove support for DSA keys, or re-adding support for AES-CBC (which is not really weak, and which can be hardware accelerated, unlikely most of the replacements) “FreeBSD is the team trying to increase the risk.” Is incorrect, they are trying to reduce the impact on the end user Specifically, a user upgrading from 10.x to 10.3, should not end up locked out of their SSH server, or otherwise confronted by unexpected errors or slowdowns because of upstream changes I will note again, (and again), that the NONE cipher can NOT allow a user to “shoot themselves in the foot”, encryption is still used during the login phase, it is just disabled for the file transfer phase. The NONE cipher will refuse to work for an interactive session. While the post states that the NONE cipher doesn't improve performance that much, it infact does In my own testing, chacha20-poly1305 1.3 gbps, aes128-gcm (fastest) 5.0 gbps, NONE cipher 6.3 gbps That means that the NONE cipher is an hour faster to transfer 10 TB over the LAN. The article suggests just removing sendmail with no replacement. Not sure how they expect users to deliver mail, or the daily/weekly reports Ports can be compiled as a regular user. Only the install phase requires root for ntpd, it is not clear that there is an acceptable replacement yet, but I will not that it is off by default In the sysctl section, I am not sure I see how enabling tcp blackhole actually increases security at all I am not sure that linking to every security advisory in openssl since 2001 is actually useful Encrypted swap is an option in bsdinstall now, but I am not sure it is really that important FreeBSD now uses the Fortuna PRNG, upgraded to replace the older Yarrow, not vanilla RC4. “The resistance from the security team to phase out legacy options makes mewonder if they should be called a compatibility team instead.” I do not think this is the choice of the security team, it is the ABI guarantee that the project makes. The stable/10 branch will always have the same ABI, and a program or driver compiled against it will work with any version on that branch The security team doesn't really have a choice in the matter. Switching the version of OpenSSL used in FreeBSD 9.x would likely break a large number of applications the user has installed Something may need to be done differently, since it doesn't look like any version of OpenSSL, (or OpenSSH), will be supported for 5 years ever again *** ZFS Raidz Performance, Capacity and Integrity (https://calomel.org/zfs_raid_speed_capacity.html) An updated version of an article comparing the performance of various ZFS vdev configurations The settings users in the test may not reflect your workload If you are benchmarking ZFS, consider using multiple files across different datasets, and not making all of the writes synchronous Also, it is advisable to run more than 3 runs of each test Comparing the numbers from the 12 and 24 disk tests, it is surprising to see that the 12 mirror sets did not outperform the other configurations. In the 12 drive tests, the 6 mirror sets had about the same read performance as the other configurations, it is not clear why the performance with more disks is worse, or why it is no longer in line with the other configurations More investigation of this would be required There are obviously so other bottlenecks, as 5x SSDs in RAID-Z1 performed the same as 17x SSDs in RAID-Z1 Interesting results none the less *** iXSystems FreeNAS Mini Review (http://www.nasanda.com/2016/03/ixsystems-freenas-mini-nas-device-reviewed/) Interview - Mark Felder - feld@freebsd.org (mailto:feld@freebsd.org) / @feldpos (https://twitter.com/feldpos) Ports, Ports and more Ports DigitalOcean Digital Ocean's guide to setting up an OpenVPN server (https://www.digitalocean.com/community/tutorials/how-to-configure-and-connect-to-a-private-openvpn-server-on-freebsd-10-1) News Roundup AsiaBSDCon OpenBSD Papers (http://undeadly.org/cgi?action=article&sid=20160316153158&mode=flat&count=0) + Undeadly.org has compiled a handy list of the various OpenBSD talks / papers that were offered a few weeks ago at AsiaBSDCon 2016. Antoine Jacoutot (ajacoutot@) - OpenBSD rc.d(8) (slides | paper) Henning Brauer (henning@) - Running an ISP on OpenBSD (slides) Mike Belopuhov (mikeb@) - Implementation of Xen PVHVM drivers in OpenBSD (slides | paper) Mike Belopuhov (mikeb@) - OpenBSD project status update (slides) Mike Larkin (mlarkin@) - OpenBSD vmm Update (slides) Reyk Floeter (reyk@) - OpenBSD vmd Update (slides) Each talk provides slides, and some the papers as well. Also included is the update to ‘vmm' discussed at bhyveCon, which will be of interest to virtualization enthusiasts. *** Bitcoin Devs could learn a lot from BSD (http://bitcoinist.net/bitcoin-devs-could-learn-a-lot-from-bsd/) An interesting article this week, comparing two projects that at first glance may not be entirely related, namely BitCoin and BSD. The article first details some of the woes currently plaguing the BitCoin development community, such as toxic community feedback to changes and stakeholders with vested financial interests being unable to work towards a common development purpose. This leads into the crux or the article, about what BitCoin devs could learn from BSD: First and foremost, the way code is developed needs change to stop the current negative trend in Bitcoin. The FreeBSD project has a rigid internal hierarchy of people with write access to their codebase, which the various Bitcoin implementations also have, but BSD does this in a way that is very open to fresh eyes on their code, allowing parallel problem solving without the petty infighting we see in Bitcoin. Anyone can propose a commit publicly to the code, make it publicly available, and democratically decide which change ends up in the codebase. FreeBSD has a tiny number of core developers compared to the size of their codebase, but at any point, they have a huge community advancing their project without hard forks popping up at every small disagreement. Brian Armstrong commented recently on this flaw with Bitcoin development, particularly with the Core Devs: “Being high IQ is not enough for a team to succeed. You need to make reasonable tradeoffs, collaborate, be welcoming, communicate, and be easy to work with. Any team that doesn't have this will be unable to attract top talent and will struggle long term. In my opinion, perhaps the biggest risk in Bitcoin right now is, ironically, one of the things which has helped it the most in the past: the Bitcoin Core developers.” A good summary of the culture that could be adopted is summed up as follows: The other thing Bitcoin devs could learn from is the BSD community's adoption of the Unix Design philosophy. Primarily “Worse is Better,” The rule of Diversity, and Do One Thing and Do It Well. “Worse is Better” emphasizes using extant functional solutions rather than making more complex ones, even if they would be more robust. The Rule of Diversity stresses flexibility of the program being developed, allowing for modification and different implementations without breaking. Do one Thing and Do it well is a mantra of the BSD and Unix Communities that stresses modularity and progress over “perfect” solutions. Each of these elements help to make BSD a wildly successful open source project with a healthy development community and lots of inter-cooperation between the different BSD systems. While this is the opposite of what we see with Bitcoin at present, the situation is salvageable provided changes like this are made, especially by Core Developers. All in all, a well written and interesting take on the FreeBSD/BSD project. We hope the BitCoin devs can take something useful from it down the road. *** FreeBSD cross-compiling with gcc and poudriere (http://ben.eficium.net/2016/03/freebsd-cross-compiling-with-gcc.html) Cross-Compiling, always a challenge, has gotten easier using poudriere and qemu in recent years. However this blog post details some of the particular issues still being face when trying to compile some certain ports for ARM (I.E. rPi) that don't play nicely with FreeBSD's default CLANG compiler. The writer (Ben Slack) takes us through some of the work-arounds he uses to build some troublesome ports, namely lsof and libatomic_ops. Note this is not just an issue with cross compile, the above mentioned ports also don't build with clang on the Pi directly. After doing the initial poudriere/qemu cross-compile setup, he then shows us the minor tweaks to adjust which compiler builds specific ports, and how he triggers the builds using poudriere. With the actual Makefile adjustment being so minor, one wonders if this shouldn't just be committed upstream, with some if (ARM) - USE_GCC=yes type conditional. *** Nvidia releases new Beta graphics driver for FreeBSD (https://devtalk.nvidia.com/default/topic/925607/unix-graphics-announcements-and-news/linux-solaris-and-freebsd-driver-364-12-beta-/) Added support for the following GPUs: GeForce 920MX & GeForce 930MX Added support for the Vulkan API version 1.0. Fixed a bug that could cause incorrect frame rate reporting on Quadro Sync configurations with multiple GPUs. Added a new RandR property, CscMatrix, which specifies a 3x4 color-space conversion matrix. Improved handling of the X gamma ramp on GF119 and newer GPUs. On these GPUs, the RandR gamma ramp is always 1024 entries and now applies to the cursor and VDPAU or workstation overlays in addition to the X root window. Fixes for bugs and added several other EGL extensions *** Beastie Bits New TN Bug started (http://knoxbug.org/) DragonFlyBSD Network/TCP Performance's gets a bump (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/4a43469a10cef8c17553c342aab9d73611ea7bc8?utm_source=anzwix) FreeBSD Foundation introduces a new website and logo (https://www.freebsdfoundation.org/blog/introducing-a-new-look-for-the-foundation/) Our producer made these based on the new logo: http://q5sys.sh/2016/03/a-new-freebsd-foundation-logo-means-its-time-for-some-new-wallpapers/ http://q5sys.sh/2016/03/pc-bsd-and-lumina-desktop-wallpapers/ https://github.com/pcbsd/lumina/commit/60314f46247b7ad6e877af503b3814b0be170da8 IPv6 errata for 5.7/5.8, pledge errata for 5.9 (http://undeadly.org/cgi?action=article&sid=20160316190937&mode=flat) Sponsoring “PAM Mastery” (http://blather.michaelwlucas.com/archives/2577) A visualization of FreeBSD commits on GitHub for 2015 (https://rocketgraph.com/s/v89jBkKN4e-) The VAX platform is no more (http://undeadly.org/cgi?action=article&sid=20160309192510) Feedback/Questions Hunter - Utils for Blind (http://slexy.org/view/s20KPYDOsq) Chris - ZFS Quotas (http://slexy.org/view/s2EHdI3z3L) Anonymous - Tun, Tap and Me! (http://slexy.org/view/s21Nx1VSiU) Andrew - Navigating the BSDs (http://slexy.org/view/s2ZKK2DZTL) Brent - Wifi on BSD (http://slexy.org/view/s20duO29mN) ***
This week on BSDNow, Allan is back from the Storage Summit in Silicon Valley! We are going to get his thoughts on how the conference went, plus bring you the latest ZFS info discussed. That plus the usual BSD news is This episode was brought to you by Headlines OpenBSD website operators urged to fix mind-alteringly bad bug (http://www.theregister.co.uk/2016/02/21/openbsd_website_operator_patch_now_for_the_sake_of_your_sanity/?mt=1456206806399) We start off a bit light-hearted this week, with the important, breaking news that finally a long-standing OpenBSD bug has been addressed for the HTTP daemon. Specifically? It changes the default 404 page fonts away from Comic Sans, to a bit more crowd-pleasing alternative: “For some reason the httpd status pages (e.g. 404) use the Comic Sans typeface. This patch removes comic sans and sets the typeface to the default sans-serif typeface of the client. “This lowers the number of people contacting website maintainers with typeface complaints bordering on harassment”. Operators running HTTPD are highly encouraged to update their systems to the latest code, right now……... No seriously, we are waiting for you. Get it done now and then we'll continue with the show. Registration for AsiaBSDCon 2016 is now open + Talk Schedule (https://2016.asiabsdcon.org/registration/?lang=en) After a few delays, the registration for AsiaBSDCon has now opened! The conference starts in less than two weeks! now, so be sure to get signed up ASAP. In addition the schedule has been posted, and here's some of the highlights of this year's conference. In addition to FreeBSD and NetBSD dev summits on the first two days, we have some excellent tutorials being given this year by Kirk, Gnn, Dru and more! (https://2016.asiabsdcon.org/program.html.en) The regular paper talks also have lots of good ones this year, including this crazy encrypted boot loader one given by our very own Allan Jude! *** OPENBSD ON AWS : AN UNEXPECTED JOURNEY (http://blog.d2-si.fr/2016/02/15/openbsd-on-aws/?hn) We have a blog post from Antoine Jacoutot, talking about the process of getting OpenBSD up and running in AWS It starts with his process of creating an AMI from scratch, which ended up not being that bad: create and loopback-mount a raw image containing a UFS filesystem extract the OpenBSD base sets (which are just regular tarballs) and kernel enable console output (so that one could “aws ec2 get-console-output”) install the boot loader on the image then use the ec2 tools to import the RAW image to S3, convert it into a volume (ec2-import-volume) which we can snapshot (ec2-create-snapshot) and create an AMI from (ec2-register) The blog post also has a link to a script which automates this process, so don't be daunted if you didn't quite follow all of that. Thanks to the recently landed DomU support, the final pieces of the puzzle fell into place, allowing OpenBSD to function as a proper guest (with networking!) Next it details the process of injecting a public SSH key into the instances for instant remote access. An ec2-init.sh script was created (also on github) which does the following: setting the hostname installing the provided SSH public key to /root/.ssh/authorized_keys executing user-data (if it starts with a shebang) displaying the host SSH fingerprints on the console (to match cloud-init) With that done, OpenBSD is pretty much AWS ready! He then gives a brief walkthrough of setting up nginx for new users, but if you've already done this before then the instance is ready for you to hacking on. Start thinking of ideas for things with FreeBSD for Google's 2016 Summer of Code (https://wiki.freebsd.org/SummerOfCodeIdeas) Students and Developers, listen up! It's time to start thinking about GSoC again, and FreeBSD is looking to update its project ideas page. There's some good ones on the list, plus ones that should be pruned (such as GELI boot), but now is the time to start adding new ones before we get too deep into the process. This goes for the other BSD's as well, start thinking about your proposals, or if you are developer, which projects would be a good fit for mentoring. (Improving the Linux Compat layer is one I think should be done!) Guide to getting started with kernel hacking (https://wiki.freebsd.org/Graphics/Getting%20started%20with%20kernel%20projects) One of the things that's been asked frequently is how to contribute towards the efforts to bring updated DRM / X drivers to the FreeBSD kernel. Jean-Sébastien Pédron has started a great guide on the Wiki which details how to get started with the porting effort, and that developers need not be afraid of helping. *** Storage Summit Roundup Earlier this week a number of developers from FreeBSD, as well as various vendors that use FreeBSD, or provide products used with FreeBSD met for a Storage Summit (https://wiki.freebsd.org/201602StorageSummit), to discuss the future of these technologies The summit was co-located with the USENIX FAST (Filesystems And Storage Technologies) conference The summit was sponsored by the FreeBSD Foundation and FlightAware After a short introduction, the event opened with a Networking Synergy panel The focus of this panel was to see if there were techniques and lessons learned in improving the networking stack over the last 10 years that could be applied to improving the storage stack A lot of time was spent discussing issues like multi-queue support, CPU scheduling, and ways to modernize the stack CAM Scheduling & Locking Revamp (https://wiki.freebsd.org/201602StorageSummit/CAM) No notes posted User Space Storage Stack (https://wiki.freebsd.org/201602StorageSummit/UserSpace) One of the user space storage stacks discussed was Diskmap Like netmap, but for disks (diskmap) Kernel bypass for accessing disks Ilias Marinos, who is working on diskmap at Cambridge University, described diskmap to the group A design discussion then followed in which the memory management was covered as that's an issue for any sort of "IO" map system Action Items: Discuss with Luigi the idea of code merges Need a reset path API Kernel buffer mapping for reliability Support for other interfaces (SATA/SCSI) GEOM layer adaptation Adapting to New Storage Technologies (https://wiki.freebsd.org/201602StorageSummit/NewStorageTechnologies) This working group was led by Adrian Palmer, from Seagate SMR Persistent Memory Session 1: Device Identification and the structural requirements Agenda: We'll look over the Identification nuances and what needs to change to support the structure. Support for IO order guarantees, forward-write only requirements, new commands and topology. Dig into CAM and GEOM layers. Solutions should be fast and have as few code paths as possible Results: Small audience. We talked about zoned characteristics, and how it can be used in various workloads, projected to be implemented in years Session 2: Information dissemination and consumption Agenda: Where and how will information from the report_zones command be gathered, stored, combined and used. This will include userspace storage and multi-volume management. Will CAM store this data, or will GEOM? How frequently will this need to be queried/updated/verified from the drive? Results: Merged with ZFS working group to discuss SMR. Came up with idea that could be implemented as circular buffer zone type. Began to discuss solutions among developers ZFS (https://wiki.freebsd.org/201602StorageSummit/ZFS) During the first session we discussed how to improve dedup support + A dedup throttle or cap was discussed. When the size of the DDT grows beyond this size, new entries would not be deduped. An alternative to this was also discussed, where when the DDT reached the cap size, it would remove a random entry with only a single reference from the DDT to make room for the new entry. When a block is going to be freed, if it is not found in the DDT, it is assumed to have only 1 reference, and removed. There was also discussion of replacing the DDT with an in-memory hash table and a “log” of increment/decrement operations, that is periodically compacted. The hash table is recreated from the log at pool import time. This would reduce the in-memory footprint of the DDT, as well as speed up all write operations as adding an entry to the dedup log will be less expensive than updating the DDT. There was also discussion of using dedicated device(s) for the DDT, either using the DDT on SSD work by Nexenta, or the Metadata Classes work by Intel The first session also discussed Secure Delete and related things The desire for an implementation of TRIM that uses the “secure erase” functionality provided by some disks was expressed Overwriting sectors with patterns of garbage may be insufficient because SSDs may internally remap where a specific LBA physically resides The possibility of using something like the “eager zero” feature to periodically write zeros over all free blocks in the pool to erase any lingering data fragments Problems with the FreeBSD TRIM implementation were discussed, as well as looking at ways to implement the new ZFS TRIM implementation on FreeBSD ABD (ARC Buf Data) was discussed, a new design that lessens the requirement for contiguous memory. Only a small area of contiguous blocks is reserved at boot, and compressed ARC blocks are constructed of scatter-gather lists of individual pages The second session combined with the SMR group and talked about SMR support in ZFS Later in the second session ZFS Encryption was also discussed, mostly with a focus on what the use cases are The third session combined all of the groups for an overview of upcoming ZFS features including device removal and channel programs There was also a request for code review, for mostly finished projects like Persistent L2ARC, Writeback cache, and Large dnode support Hallway Track ZFS / VFS Interaction Adrian Palmer has been a FreeBSD hobbyist since FreeBSD 7, and I think I managed to convince him to start contributing *** News Roundup One Week with NetBSD 7.0: Back to Unix basics (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-back-to-unix.html) The author of this blog series is sending a week using NetBSD 7.0, following a previous series on Solaris 10 “This is actually familiar territory, as I've been using BSD variants almost exclusively since 2006. My recent SunOS explorations were triggered last summer by OpenBSD having choked on my current laptop's NVIDIA card, and from what I could see at the time, FreeBSD had the same problem, although I now know NVIDIA drivers exist for that system. The thing that keeps me from going all-in with FreeBSD 10.x, however, is the fact that Firefox crashes and leaves "core dump" messages in its wake, and I'm just not a Chrome kinda guy.” “For those with a catholic taste in Unix, NetBSD is a keg party at the Vatican. If you're an absolute Unix beginner, or have been living on Ubuntu-based Linux distros for too long, then you may feel stranded at first by NetBSD's sparseness. You'll find yourself staring into the abyss and seeing only a blinking cursor staring back. If you have the presence of mind to type startx, you'll be greeted by twm, a window manager offering little more than an xterm window with the same blinking cursor until you learn how to configure the .twmrc file to include whatever applications you want or need in the right-click menu.” “As for NetBSD itself, I can't think of any major productivity applications that can't be installed, and most multimedia stuff works fine.” Issues the author hopes to sort out in later posts: Audio playback (youtube videos in Firefox) Wireless Flash Digital Camera SD Card readability, video playback Audacity A “fancy” desktop like Gnome 2, KDE, or xfce In a follow-up post (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-libreoffice.html), the author got LibreOffice installed and sorted out the audio issues they were having In a later follow-up (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-mixed-review-of.html) XFCE is up and running as well *** ZFS is for Containers in Ubuntu 16.04 (http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html) As you may have heard, Ubuntu 16.04 will include ZFS -- baked directly into Ubuntu -- supported by Canonical “ZFS one of the most beloved features of Solaris, universally coveted by every Linux sysadmin with a Solaris background. To our delight, we're happy to make to OpenZFS available on every Ubuntu system.” What does “supported by Canonical” mean? “You'll find zfs.ko automatically built and installed on your Ubuntu systems. No more DKMS-built modules” “The user space zfsutils-linux package will be included in Ubuntu Main, with security updates provided by Canonical” The article then provides a quick tutorial for setting up Linux Containers (LXC) backed by ZFS In the example, ZFS is backed by a file on the existing disk, not by a real disk, and with no redundancy However, the setup script seems to support using real block devices The Software Freedom Conservancy (https://sfconservancy.org/) is expected to issue a statement detailing their opinion on the legalities and licensing issues of bundling ZFS with Linux. *** Polling is a Hack: Server Sent Events (EventSource) with gevent, Flask, nginx, and FreeBSD (http://hypatia.software/2016/01/29/polling-is-a-hack-server-sent-events-eventsource-with-gevent-flask-nginx-and-freebsd/) A tutorial on setting up ‘Server-Sent Events', also know as EventSource in javascript, to notify website clients of new data, rather than having the javascript constantly poll for new data. The setup uses FreeBSD, nginx, gevent, Python, and the Flask framework The tutorial walks through setting a basic Python application using the Flask framework Then setting up the client side in Javascript Then for the server side setup, it covers installing and configuring nginx, and py-supervisor on FreeBSD The tutorial also includes links to additional resources and examples, including how to rate limit the Flash application *** Why FreeBSD? (http://www.aikchar.me/blog/why-freebsd.html) An excellent article written by Hamza Sheikh, discussing why FreeBSD is now his clear choice for learning UNIX. The article is pretty well written and lengthy, but has some great parts which we wanted to share with you: There were many rough edges in the Linux world and some of them exist even today. Choosing the right distribution (distro) for the task at hand is always the first and most difficult decision to make. While this is a strength of the Linux community it is also its weakness. This is exacerbated with the toxic infighting within the community in the last few years. A herd of voices believes it is their right to bring down a distro community because it is not like their distro of choice. Forking upstream projects has somehow become taboo. Hurling abuse in mailing lists is acceptable. Helping new users is limited to lambasting their distro of choice. Creating conspiracy theories over software decisions is the way to go. Copyleft zealots roam social media declaring non-copyleft free software heretic abominations. It all boils down to an ecosystem soured by the presence of maniacs who have the loudest voices and they seem to be everywhere you turn. Where is the engineering among all this noise? Btrfs - baking for a long time - is still nowhere near ZFS in stability or feature parity. systemd is an insatiable entity that feeds on every idea in sight and just devours indiscriminately. Wayland was promised years ago and its time has yet to arrive. Containers are represented by Docker that neither securely contains applications nor makes them easy to manage in production. Firewalling is dithering between firewalld, nftables, etc. SystemTap cannot match DTrace. In the same time span what do various BSDs offer? pf, CARP, ZFS, Hammer, OpenSSH, jails, pkgsrc, (software) ports, DTrace, hardware portability; just to name a few. Few would deny that BSDs have delivered great engineering with free software licenses to the entire world. To me they appear to be better flag bearers of free software with engineering to back it. He then goes through some of the various BSD's and the specifics on why FreeBSD was the logical choice for his situation. But at the end has a great summary on the community as a whole: Finally - and maybe repeating myself here - I have nothing but praise for the community. Be it BSD Now, mailing lists, Reddit, Twitter, LFNW, or SeaGL, people have encouraged me, answered my questions, and filed bugs for me. I have been welcomed and made a part of the community with open arms. These reasons are (good) enough for me to use FreeBSD and contribute to it. BeastieBits OPNsense 16.1.3 released (https://opnsense.org/opnsense-16-1-3-released/) Copies of "FreeBSD Mastery: Specialty Filesystems" seen in the wild (https://twitter.com/Savagedlight/status/700001944547491842) pfsense training available in Europe (http://www.netgate.com/training/) LiteBSD now has 50 ports in its ports tree (https://github.com/ibara/LiteBSD-Ports) Ports tree locked for OpenBSD 5.9 (http://marc.info/?l=openbsd-ports&m=145615281431064&w=2) “FreeBSD Filesystem Fun” at March semibug (http://blather.michaelwlucas.com/archives/2556) Event #46 — Embedded Platforms (BSD, OpenWRT, Plan 9 & Inferno) (http://oshug.org/event/46) Feedback/Questions Frank - ZFS RAM? (http://slexy.org/view/s21lcCKrSB) David - ARM Porting (http://slexy.org/view/s204lxjvlq) Johnny - Lumina Default? (http://slexy.org/view/s2xMiSNLYn) Adam - PC-BSD Install and Q's (http://slexy.org/view/s214gJbLwD) Jeremy - Video Card Q (http://slexy.org/view/s20UNyzEeh) ***
This week on BSDNow, we are going to be talking to Ken Moore about the Lumina desktop environment, where it stands now & looking ahead. Then Allan turns the tables & interviews both Kris & Ken about new ongoings in PC-BSD land. Stay tuned, lots of exciting show is coming your way right now on BSDNow, the place to B...SD! This episode was brought to you by Headlines Linuxvoice reviews six NAS designed OSes and states that FreeNAS has the largest amount of features (https://www.linuxvoice.com/group-test-nas-distros/) The review compares the features of: FreeNAS, NAS4Free, Open Media Vault, Openfiler Community Edition, EasyNAS, and Turnkey Linux File Server “Many NAS solutions can do a lot more than just back up and restore files – you can extend them with plugins to do a variety of tasks. Some enable you to stream media to computers and others devices. Others can hook up with apps and services and allow them to use the NAS for storing and retrieving data” Open Media Vault: 4/5, “A feature-rich NAS distro that's easy to deploy and manage”. Many plugins, good UI Turnkey Linux File Server: 2/5, “A no-fuss distro that'll set up a fully functional file sharing server in no time”. No RAID, LVM must be down manually Openfiler Community Edition: 1/5, “There is a target segment for Openfiler, but we can't spot it”. In the middle of rebasing on CentOS, lacking documentation, confusing UI EasyNAS: 3/5, “A simple NAS distro that balances the availability of features with reasonable assumptions”. Major updates require reinstall, lacks advanced features and advanced protocols FreeNAS: 3/5, “FreeNAS The most feature-rich NAS distribution requires some getting used to”. Best documentation, best snapshot management, most plugins, jailed plugins, most enterprise features NAS4Free: 3/5, “NAS4Free An advanced NAS distro that's designed for advanced users”, additional flexibility with disk layout (partition the first disk to install the OS there, use remaining space for data storage) “If we had to award this group test to the distro with the biggest number of features then the top two challengers would have been FreeNAS and its protegée NAS4Free. While both of these solutions pitch themselves to users outside the corporate environment, they'd simply be overkill for most home users. Furthermore, their FreeBSD base and the ZFS filesystem, while a boon to enterprise users, virtually makes them alien technology to the average Linux household.” It is not clear why they gave NAS4Free and FreeNAS the same score when they wrote a list of reasons why FreeNAS was better. It seems the goal of their rundown was to find the best Linux NAS, not the best NAS. *** FreeBSD based Snort IPS (http://www.unixmen.com/freebsd-snort-ips/) UnixMen.com provides a new tutorial on setting up Snort, the IPS (Intrusion Prevention system) on FreeBSD Install Apache, PHP, and MySQL, then Snort Download the latest Snort rules from the official website Disable the Packet Filter on the USB interfaces to avoid issues with Snort Install oinkmaster and barnyard2, and configure them Then install the Snorby WEB interface, which will give you a nice overview of the data generated by the IPS Then install SnortSAM, and connect it to ipfw Now when Snort detects a potential intrusion, it will be displayed in Snorby, and automatically blocked with IPFW *** Opensource.com features two BSD developers as examples of how open source can help your career (https://opensource.com/life/16/1/3-new-open-source-contributors-share-their-experiences) “When contributing to open source projects and communities, one of the many benefits is that you can improve your tech skills. In this article, hear from three contributors on how their open source helped them get a job or improved their career.” Alexander Yurchenko, an OpenBSD developer who now works at Yandex says: “Participating in such a project yields colossal experience. A good, large open source project has everything that is typically required from a developer at job interviews: good planning, good coding, use of versioning systems and bug trackers, peer reviews, teamwork, and such. So, after stewing in such an environment for a year or two, you have a good opportunity to grow to a senior developer level.” “That is, in fact, what happened to me. I was hired as a senior developer without having any formal work experience on my service record. After the first week, my probation period was reduced from three months to zero.” While you may not have “formal work experience”, you do have a body of work, a (code/documentation/etc) portfolio, you can point to Having spent a year working somewhere may say something about you, but showing some code you wrote that other people use every day, is usually more valuable Alexander Polyakov, a DragonFly contributor, worked on updating support for other languages and on ACPI. “I even made some money in the process—a customer found me via git log. He wanted to use DragonFlyBSD in production and needed better ACPI support and some RAID driver or something.” “In a nutshell, contributing to various open source projects is how you gain great experience. Don't be afraid to send in bad code (happens to the best of us), keep calm (while being scolded for sending in that bad code), and choose projects you are really interested in. Then you'll both gain experience and have fun while you doing it.” Kirill Gorkunov talks about his experience with turning open source into a career: “For a few years, I've been fixing the code, sending patches, getting scolded for bad code and complimented for good code. That experience was priceless. And you can be sure that as soon as you get good at it, job offers will follow. This is, in fact, how I met the kernel developers working on OpenVZ. Together, we decided to continue working on the OpenVZ kernel and related stuff as well” When you contribute to open source, you end up being the person who wrote “Foo”, and this can often turn into work, when someone wants to build something with “Foo”, or like “Foo” This same point was focus of a panel the FreeBSD Foundation organized at the womENcourage conference in Sweden last year: Open Source as a Career Path (https://www.youtube.com/watch?v=p7PW1E3IJvY) *** FreeBSD, LibreSSL and LetsEncrypt oh my! (https://wiki.freebsd.org/BernardSpil/LetsEncrypt) Over on the FreeBSD Wiki, Bernard Spil (whom we've interviewed before) has started a walkthrough talking about how he uses LibreSSL and LetsEncrypt, without using the heavy python client The article provides detailed instructions on prepping the system and automating the process of updating the SSL certificates If you've used the “official” letsencrypt client in the past, you'll note some differences in his method, which keeps all the ‘acme-challenge' files in a single-directory, which is aliased into domains. Using this method also drops the requirement to run the letsencrypt auth as root, and allows you to run it as the unprivileged “letsencrypt” user instead. He mentions that the bash/zsh scripts used may be added to ports at some point as well *** Interview - Ken Moore & Kris Moore - ken@pcbsd.org (mailto:ken@pcbsd.org) / @pcbsdkris (https://twitter.com/pcbsdkris) PC-BSD's new SysAdm Project and Lumina Update *** News Roundup DragonFly Intel i915 support to match what's in the Linux 4.1 kernel (http://lists.dragonflybsd.org/pipermail/commits/2016-January/459241.html) In DragonFly's ongoing quest for DRM awesomeness, they have now merged changes to bring them up to Linux 4.1 kernel features. Some of the notables include that “Valleyview” support is greatly improved, and not considered preliminary anymore Skylake got some support improvements as well, including runtime power management, and that turbo and sleep states should be functional. Some great improvements to power usage have been added, such as setting GPU frequencies to hardware minimum and enabling of DRRS (Dynamic Refresh Rate Switching) being enabled by default They've even begun importing some of the prelim work for Broxton, the upcoming Atom SOC *** FreeNAS Home Server Build (https://ramsdenj.github.io/server/2016/01/01/FreeNAS-Server-Build.html) We have a nice article to share with you this week by John Ramsden, which walks us through his home-brew FreeNAS server setup. As is typical with most home users, he will be using the system to both serve media, and as a backup target for other systems. His hardware setup is pretty impressive for a home-brew, made up of the following: Fractal Design Node 804 Chassis Supermicro X10SL7-F Motherboard Xeon E3-1231 v3 CPU 4x Samsung DDR3 1.35v-1600 M391B1G73QH0 RAM 2x 32GB SATA III SMC DOM Boot Drive SeaSonic G-550 Power Supply Cyberpower CP1500PFCLCD 1500VA 900W PFC UPS 6x Western Digital 6TB Red HDD 2 x ENERMAX T.B. Silence UCTB12P Case Fan 3x Noctua NF-P14s redux-1200 Case Fan The SATA DOM was neat to see in use, in his case in a mirror He then walks us through his burn-in process, which involved memory testing for 46 hours, and then disk testing with the smartctl long tests. There is even details on how the fan thresholds were set up, which may be of use to other DiY'ers out there. The SATA DOM was neat to see in use, in his case in a mirror He then walks us through his burn-in process, which involved memory testing for 46 hours, and then disk testing with the smartctl long tests. There is even details on how the fan thresholds were set up, which may be of use to other DiY'ers out there. claviger manages your SSH authorized_keys files for you (https://github.com/bwesterb/claviger) An application to manage your SSH authorized_keys files for you Make a list of your keys (laptop, desktop, work) Then a list of your ssh accounts List which keys should be present, and which should be absent Optional setting to keep all “other” keys, such as those added by other users Optional list of specific “other” keys to allow (does not add them, but does not remove them if they are present) You say say ‘server2 like server1', and it will inherit all of the settings from that server There is a “default” server, that all others inherit *** FreeBSD 9.2 x64 OpenVPN AD authentication with crypt (http://www.unixmen.com/openvpn-ad-authentication-with-crypt/) A few days back unixmen.com posted a nice tutorial walkthrough of a OpenVPN setup on FreeBSD 9.2 using Active Directory for auth In this particular setup, FreeBSD is running the gateway / OpenVPN server, the client desktops are running Windows 7 and domain controller on Windows 2008 The setup on FreeBSD pretty straightforward, thanks to the openvpn-auth-ldap port. (Unknown why they didn't use the package) In addition to showing the details on how configuration was done on BSD, what makes this walkthrough nice is the addition of so many screenshots of how the windows configuration was done. Part of the walkthrough will also detail how they created their .ovpn files for importing on the OpenVPN clients. *** Beastie Bits dtrace included by default in NetBSD (http://cvsweb.netbsd.org/bsdweb.cgi/src/share/mk/bsd.own.mk.diff?r1=1.883&r2=1.884&only_with_tag=MAIN&f=h) FOSDEM16 is approaching, get ready to follow the BSD devroom (https://fosdem.org/2016/schedule/track/bsd/) Call for testing: Concurrent: malloc(3) calls (to speed up Firefox) (http://undeadly.org/cgi?action=article&sid=20160123165549) "With the PV drivers in -CURRENT, it is now possible to run OpenBSD within AWS." (http://daemonforums.org/showthread.php?p=57767) PC-BSD Handbook in Spanish (http://www.pcbsd.org/doc-archive/10.2/html-es/pcbsd.html) Feedback/Questions Clint - ZIL on Partition (http://pastebin.com/WLpHzz3F) Federico - LibreSSL and DMA (http://pastebin.com/1QFZU2Bz) Ghislain - FreeBSD vs Linux vs Illumos (http://pastebin.com/aesVaKG4) Cary - ZFS - Caching - Replication (http://pastebin.com/x4DRHP0i) ***
This week on BSDNow, we will be talking shop with Josh Paetzel of FreeNAS fame, hearing about his best do's and do-nots of using ZFS in production. Also, a quick iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** FreeNAS Logo Design Contest (https://www.ixsystems.com/freenas-logo-contest/) Rules and Requirements (https://forums.freenas.org/index.php?threads/freenas-logo-design-contest.39968/) For those of you curious about Kris' new lighting here are the links to what he is using. Softbox Light Diffuser (http://smile.amazon.com/gp/product/B00OTG6474?psc=1&redirect=true&ref_=oh_aui_detailpage_o01_s00&pldnSite=1) Full Spectrum 5500K CFL Bulb (http://smile.amazon.com/gp/product/B00198U6U6?psc=1&redirect=true&ref_=oh_aui_detailpage_o06_s00) *** This episode was brought to you by Headlines A Brief look back at 2015 (http://fossforce.com/2015/12/bsd-brief-look-back-2015/) As we start the show this week, we begin with a brief look back at BSD in 2015, brought to us by Larry at FOSS force. Aside from his issue with tap-to-click on the touchpad, his PC-BSD experience has been pretty good. (Larry, if you hear this, jump on #pcbsd on FreeNode and we will lend a hand) He mentions that this really isn't his first time running BSD, apparently back in ye-olden days he got NetBSD up and running on a PowerBook G3, until an update brought that experience to abrupt ending. He gives a shout-out to the FreeBSD Foundation as being a great go-to source for wrapup on the previous year in FreeBSD land, while also mentioning the great 4.4 release of DragonFly, and some of the variants, such as RetroBSD and LiteBSD He leaves us with a tease for 2016 that work is ongoing on Twitter to port over Mopidy, a python based extensible music server *** A look forward at BSD events throughout 2016 (http://www.bsdevents.org/scheduler/) After a quick look back at 2015, now its time to start planning your 2016 schedule. The BSDEvents site has a calendar of all the upcoming conferences / shows where BSD will have a presence this year. There are quite a few items on the agenda, including non BSD specific conferences, such as SCALE / Fosdem and more. Take a look and see, you may be able to find something close your location where you can come hang out with other BSD developers. (or better yet), if a linux conference is coming to your town, think about submitting a BSD talk! Additionally, if getting BSD Certification is something on your 2016 resolutions, you can often take the test at one of these shows, avoiding the need to travel to a testing center. *** The 'Hidden' Cost of Using ZFS for Your Home NAS (http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html) An article was recently posted that seems to be trying to dissuade people from using ZFS for their home NAS It points out what experienced users already know, but many newcomers are not strictly aware of: Expanding a ZFS pool is not always as straightforward as you think it should be ZFS was designed to be expanded, and it handled this very well However, a ZFS pool is made up of VDEVs, and it is these VDEVs that provide the redundancy. RAID-Z VDEVs cannot be changed once they are created. You can replace each disk individually, and the VDEV will grow to its new larger size, but you cannot add additional disks to a RAID-Z VDEV At this point, your option is to add an additional VDEV, although best practises dictate that the new VDEV should use an equal number of disks, to avoid uneven performance So, if you started with a 6 disk RAID-Z2, having to add 6 more disks to grow the pool does seem excessive For the best flexibility, use mirrors. If you had used 6 disks as 3 mirrors of 2 disks each, you could then just add 2 more disks at a time. The downside is that using 2TB disks, you'd only have 6TB of usable space, versus the 8TB you would get from those disks in a RAID-Z2 This is the trade-off, mirrors give you better performance and flexibility, but less space efficiency It is important to note that the diagrams in this article make it appear as if all parity information is stored on specific drives. In ZFS parity is spread across all drives. Often times, the data written to the drive is not of a size that can evenly be split across all drives, so the data actually ends up looking like this (http://blog.delphix.com/matt/files/2014/06/RAIDZ.png) The errors as I see it in the original article are: It notes that the hidden cost of ZFS is that if you add a second RAID-Z VDEV, you will have a whole second set of parity drives. While this is a cost, it is the cost of making sure your data is safe. If you had an array with more than 12 drives, it is likely that you would to be able to withstand the failure of the larger number of drives The article does not consider the resilver time. If you did create a configuration with a very wide RAID-Z stripe, the failure of a disk would leave the pool degraded for a much longer time, leaving your pool at risk for that longer period. The article does not consider performance. Two RAID-Z2 VDEVs of 6 disks each will give much better performance than a single VDEV of 10 or 12 disks, especially when it comes to IOPS. *** ZFS Boot Enviroments now availble in the FreeBSD bootloader (https://svnweb.freebsd.org/base?view=revision&revision=293001) It's been in phabricator for a while (and PC-BSD), but the support for Boot-Environments has now landed upstream in -CURRENT This work was helped by cross-project collaboration when an IllumOS Developer, Toomas Soome, started porting the FreeBSD loader to IllumOS to replace GRUB there This gives Beastie menu the ability to look at the ZFS disk, and dynamically list boot-environments that it finds. (Much nicer than GRUB, which required a pre-written configuration file) This work was extended further, when Toomas Soome also ported the Beastie Menu to the UEFI loader (https://svnweb.freebsd.org/base?view=revision&revision=293233) which is now enabled by default for UEFI (https://svnweb.freebsd.org/base?view=revision&revision=293234) All of these changes are scheduled to be merged back in time for FreeBSD 10.3 as well. There is also a patch being worked on to support booting from ZFS in UEFI (https://reviews.freebsd.org/D4515) This is exciting times for doing neat things with ZFS on root, these plus Allans forthcoming GELI support (https://reviews.freebsd.org/D4593) will negate the necessity for GRUB on PC-BSD for example (Kris is very happy) *** Interview - Josh Paetzel - email@email (mailto:email@email) / @bsdunix4ever (https://twitter.com/bsdunix4ever) ZFS Support *** News Roundup RetroBSD being tested on ESP32 (http://retrobsd.org/viewtopic.php?f=1&t=37470) More hardware news for RetroBSD and LiteBSD I don't know much about this hardware, but there is a lot of discussion in the forum threads about it Not sure what you are supposed to accomplish with only 400kb of ram LITEBSD Brings 4.4BSD to PIC32 (https://hackaday.com/2016/01/04/litebsd-brings-4-4bsd-to-pic32/) It is interesting to see these super-small boards with only 512kb of memory, but will crypto offload support It is also interesting to see talk of 140mbps WiFi, can the processor actually handle that much traffic? BSD Unix-like OS is Resurrected for Embedded IoT Market (http://thevarguy.com/open-source-application-software-companies/bsd-unix-os-resurrected-embedded-iot-market) Related to the above stories, we also have an article about BSD making a resurgence on various Internet of things devices, which mentions both RetroBSD and LiteBSD The article mentions that this is an exciting development for embedded vars who now have an alternative licensed open-source OS to potentially use *** HardenedBSD's new Binary Updater (https://hardenedbsd.org/article/shawn-webb/2015-12-31/introducing-hardenedbsds-new-binary-updater) It looks like there is now another way to update your FreeBSD(hardened) system The post by Shawn Web, details how the new updater will work in future releases of HBSD Right now it looks fairly straight-forward, creating both the base.txz and kernel.txz, along with some data for etcupdate It includes a nice option for the kernel name in the update, allowing different kernels to be installed / updated at will Everything is cryptographically signed and verified using the base system openssl The build system is fairly simple, only requiring “sh/git/openssl” to create the binary updates Planned features also include updating of jails, and ZFS boot-environments *** Sometimes, processors need (BSD) love too (http://functionallyparanoid.com/2016/01/02/sometimes-processors-need-love-too/) We have a blog post from Brian Everly, talking about his long journey into legacy processors and the plans for the future to work on better supporting them on OpenBSD ports He begins with the story of his UNIX journey to today, and why this fostered his love for many of these old (and not so old) architectures, such as Sparc64, PPC32, i386. This journey ended up with the purchase of some legacy hardware (ebay is your friend), and the creation of a database listing the major port blockers on each platform This is the great kind of thing folks can do to step up and help a project, even as a weekend hobby it's great to run some hardware and help test / fix up issues that other developers maybe don't interact with as much anymore. *** Beastie Bits The standard MWL disclaimer (http://blather.michaelwlucas.com/archives/2510) PC-BSD 11.0-CURRENTJAN2016 Available (http://lists.pcbsd.org/pipermail/testing/2016-January/010350.html) NetBSD pkgsrc-2015Q3 statistics (http://mail-index.netbsd.org/tech-pkg/2015/12/28/msg016193.html) NetBSD pkgsrc-2015Q4 released (http://mail-index.netbsd.org/tech-pkg/2016/01/01/msg016213.html) First Reproducible builds conference in Athens (http://blog.netbsd.org/tnf/entry/reproducible_builds_conference_in_athens) The creator of the original ThinkPad design passes away (http://www.theregister.co.uk/2016/01/06/thinkpad_designer_obituary) Feedback/Questions Andrew - High Contrast (http://slexy.org/view/s213iCKLwn) John - FreeNAS followup (http://slexy.org/view/s21ClGePLP) Giorgio - Custom Install (http://slexy.org/view/s21527pkO1) Don - ZFS Slowdowns (http://slexy.org/view/s2jOlCsjkU) Fred - Dual Boot PC-BSD/Linux (http://slexy.org/view/s21uaB0FDU) ***
This week, Allan is out of town at another Developer Summit, but we have a great episode coming This episode was brought to you by iX Systems Mission Complete (https://www.ixsystems.com/missioncomplete/) Submit your story of how you accomplished a mission with FreeBSD, FreeNAS, or iXsystems hardware, and you could win monthly prizes, and have your story featured in the FreeBSD Journal! *** Headlines WhatsApp founder, on how it got so HUGE (http://www.wired.com/2015/10/whatsapps-co-founder-on-how-the-iconoclastic-app-got-huge/) Wired has interviewed WhatsApp co-founder Brian Acton, about the infrastructure behind WhatsApp WhatsApp manages 900 million users with a team of 50, while Twitter needs around 4,000 employees to manage 300 million users. “FreeBSD has a nicely tuned network stack and extremely good reliability. We find managing FreeBSD installations to be quite straightforward.” “Linux is a beast of complexity. FreeBSD has the advantage of being a single distribution with an extraordinarily good ports collection.” “To us, it has been an advantage as we have had very few problems that have occurred at the OS level. With Linux, you tend to have to wrangle more and you want to avoid that if you can.” “FreeBSD happened because both Jan and I have experience with FreeBSD from Yahoo!.” Additional Coverage (http://uk.businessinsider.com/whatsapp-built-using-erlang-and-freebsd-2015-10) *** User feedback in the SystemD vs BSD init (https://www.textplain.net/blog/2015/problems-with-systemd-and-why-i-like-bsd-init/) We have a very detailed blog post this week from Randy Westlund, about his experiences on Linux and BSD, contrasting the init systems. What he finds is that while, it does make some things easier, such as writing a service file once, and having it run everywhere, the tradeoff comes in the complexity and lack of transparency. Another area of concern was the reproducibility of boots, how in his examples on servers, there can often be times when services start in different orders, to save a few moments of boot-time. His take on the simplicity of BSD's startup scripts is that they are very easy to hack on and monitor, while not introducing the feature creep we have seen in sysd. It will be interesting to see NextBSD / LaunchD and how it compares in the future! *** Learn to embrace open source, or get buried (http://opensource.com/business/15/10/ato-interview-jim-salter) At the recent “All Things Open” conference, opensource.com interviewed Jim Salter He describes how he first got started using FreeBSD to host his personal website He then goes on to talk about starting FreeBSDWiki.net and what its goals were The interview then talks about using Open Source at solve customers' problems at his consulting firm Finally, the talks about his presentation at AllThingsOpen: Move Over, Rsync (http://allthingsopen.org/talks/move-over-rsync/) about switching to ZFS replication *** HP's CTO Urges businesses to avoid permissive licenses (http://lwn.net/Articles/660428/) Martin Fink went on a rant about the negative effects of license proliferation While I agree that having too many new licenses is confusing and adds difficulty, I didn't agree with his closing point “He then ended the session with an extended appeal to move the open-source software industry away from permissive licenses like Apache 2.0 and toward copyleft licenses like the GPL” “The Apache 2.0 license is currently the most widely used "permissive" license. But the thing that developers overlook when adopting it, he said, is that by using Apache they are also making a choice about how much work they will have to put into building any sort of community around the project. If you look at Apache-licensed projects, he noted, "you'll find that they are very top-heavy with 'governance' structures." Technical committees, working groups, and various boards, he said, are needed to make such projects function. But if you look at copyleft projects, he added, you find that those structures simply are not needed.” There are plenty of smaller permissively licensed projects that do not have this sort of structure, infact, most of this structure comes from being an Apache run project, rather than from using the Apache or any other permissive license Luckily, he goes on to state that the “OpenSwitch code is released under the Apache 2.0 license, he said, because the other partner companies viewed that as a requirement.” “HP wanted to get networking companies and hardware suppliers on board. In order to get all of the legal departments at all of the partners to sign on to the project, he said, HP was forced to go with a permissive license” Hopefully the trend towards permissive licenses continues Additionally, in a separate LWN post: RMS Says: “I am not saying that competitors to a GNU package are unjust or bad -- that isn't necessarily so. The pertinent point is that they are competitors. The goal of the GNU Project is for GNU to win the competition. Each GNU package is a part of the GNU system, and should contribute to the success of the GNU Project. Thus, each GNU package should encourage people to run other GNU packages rather than their competitors -- even competitors which are free software.” (http://lwn.net/Articles/659757/) Never thought I'd see RMS espousing vendor lock-in *** Interview - Brian Callahan - bcallah@devio.us (mailto:bcallah@devio.us) / @twitter (https://twitter.com/__briancallahan) The BSDs in Education *** News Roundup Digital Libraries in Africa making use of DragonflyBSD and HAMMER (http://lists.dragonflybsd.org/pipermail/users/2015-October/228403.html) In the international development context, we have an interesting post from Michael Wilson of the PeerCorps Trust Fund. They are using DragonFlyBSD and FreeBSD to support the Tanzanian Digital Library Initiative in very resource-limited settings. They cite among the most important reasons for using BSD as the availability and quality of the documentation, as well as the robustness of the filesystems, both ZFS and HAMMER. Their website is now online over at (http://www.tandli.com/) , check it out to see exactly how BSD is being used in the field *** netflix hits > 65gbps from a single freebsd box (https://twitter.com/ed_maste/status/655120086248763396) A single socket server, with a high end Xeon E5 processor and a dual ported Chelsio T580 (2x 40 Gbps ports) set a netflix record pushing over 65 Gbps of traffic from a single machine The videos were being pushed from SSDs and some new high end NVMe devices The previous record at Netflix was 52 Gbps from a single machine, but only with very experimental settings. The current work is under much more typical settings By the end of that night, traffic surged to over 70 Gbps Only about 10-15% of that traffic was encrypted with the in-kernel TLS engine that Netflix has been working on with John-Mark Gurney It was reported that the machine was only using about 65% cpu, and had plenty of head room If I remember the discussion correctly, there were about 60,000 streams running off the machine *** Lumina Desktop 0.8.7 has been released (http://lumina-desktop.org/lumina-desktop-0-8-7-released/) A very large update has landed for PC-BSD's Lumina desktop A brand new “Start” menu has been added, which enables quick launch of favorite apps, pinning to desktop / favorites and more. Desktop icons have been overhauled, with better font support, and a new Grid system for placement of icons. Support for other BSD's such as DragonFly has been improved, along with TONS of internal changes to functionality and backends. Almost too many things to list here, but the link above will have full details, along with screenshots. *** A LiveUSB for NetBSD has been released by Jibbed (http://www.jibbed.org/) After a three year absence, the Jibbed project has come back with a Live USB image for NetBSD! The image contains NetBSD 7.0, and is fully R/W, allowing you to run the entire system from a single USB drive. Images are available for 8Gb and 4Gb sticks (64bit and 32bit respectively), along with VirtualBox images as well For those wanting X, it includes both X and TWM, although ‘pkgin' is available, so you can quickly add other desktops to the image *** Beastie Bits After recent discussions of revisiting W^X support in Mozilla Firefox, David Coppa has flipped the switch to enable it for OpenBSD users running -current. (http://undeadly.org/cgi?action=article&sid=20151021191401&mode=expanded) Using the vt(4) driver to change console resolution (http://lme.postach.io/post/changing-console-resolution-in-freebsd-10-with-vt-4) The FreeBSD Foundation gives a great final overview of the Grace Hopper Conference (http://freebsdfoundation.blogspot.com/2015/10/conference-recap-grace-hopper.html) A dialog about Compilers in the (BSD) base system (https://medium.com/@jmmv/compilers-in-the-bsd-base-system-1c4515a18c49) One upping their 48-core work from July, The Semihalf team shows off their the 96-core SMP support for FreeBSD on Cavium ThunderX (ARMv8 architecture (https://www.youtube.com/watch?v=1q5aDEt18mw) NYC Bug's November meeting will be featuring a talk by Stephen R. Bourne (http://lists.nycbug.org/pipermail/talk/2015-October/016384.html) New not-just-BSD postcast, hosted by two OpenBSD devs Brandon Mercer and Joshua Stein (http://garbage.fm/) Feedback/Questions Stefan (http://slexy.org/view/s21wjbhCJ4) Zach (http://slexy.org/view/s21TbKS5t0) Jake (http://slexy.org/view/s20AkO1i1R) Corey (http://slexy.org/view/s2nrUMatU5) Robroy (http://slexy.org/view/s2pZsC7arX) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
Coming up on this weeks episode, we have BSD news, tidbits and articles out the wazoo to share. Also, be sure to stick around for our interview with Brandon Mercer as he tells us about OpenBSD being used in the healthcare industry. This episode was brought to you by Headlines NetBSD 7.0 Release Announcement (http://www.netbsd.org/releases/formal-7/NetBSD-7.0.html) DRM/KMS support brings accelerated graphics to x86 systems using modern Intel and Radeon devices (Linux 3.15) Multiprocessor ARM support. Support for many new ARM boards, including the Raspberry Pi 2 and BeagleBone Black Major NPF improvements: BPF with just-in-time (JIT) compilation by default support for dynamic rules support for static (stateless) NAT support for IPv6-to-IPv6 Network Prefix Translation (NPTv6) as per RFC 6296 support for CDB based tables (uses perfect hashing and guarantees lock-free O(1) lookups) Multiprocessor support in the USB subsystem. GPT support in sysinst via the extended partitioning menu. Lua kernel scripting GCC 4.8.4, which brings support for C++11 Experimental support for SSD TRIM in wd(4) and FFS tetris(6): Add colours and a 'down' key, defaulting to 'n'. It moves the block down a line, if it fits. *** CloudFlare develops interesting new netmap feature (https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/) Normally, when Netmap is enabled on an interface, the kernel is bypassed and all of the packets go to the Netmap consumers CloudFlare has developed a feature that allows all but one of the RX queues to remain connected to the kernel, and only a single queue be passed to Netmap The change is a simple modification to the nm_open API, allowing the application to open only a specific queue of the NIC, rather than the entire thing The RSS or other hashing must be modified to not direct traffic to this queue Then specific flows are directed to the netmap application for matching traffic For example under Linux: ethtool -X eth3 weight 1 1 1 1 0 1 1 1 1 1 ethtool -K eth3 lro off gro off ethtool -N eth3 flow-type udp4 dst-port 53 action 4 Directs all name server traffic to NIC queue number 4 Currently there is no tool like ethtool to accomplish this same under FreeBSD I wonder if the flows could be identified more specifically using something like ipfw-netmap *** Building your own OpenBSD based Mail server! (http://www.theregister.co.uk/2015/09/12/feature_last_post_build_mail_server/?mt=1442858572214) part 2 (http://www.theregister.co.uk/2015/09/19/feature_last_post_build_mailserver_part_2/) part 3 (http://www.theregister.co.uk/2015/09/26/feature_last_post_build_mailserver_part_3/) The UK Register gives us a great writeup on getting your own mail server setup specifically on OpenBSD 5.7 In this article they used a MiniPC the Acer Revo One RL85, which is a decently priced little box for a mail server (http://www.theregister.co.uk/2015/07/24/review_acer_revo_one_rl85_/) While a bit lengthy in 3 parts, it does provide a good walkthrough of getting OpenBSD setup, PostFix and DoveCot configured and working. In the final installment it also provides details on spam filtering and antivirus scanning. Getting started with the UEFI bootloader on OpenBSD (http://blog.jasper.la/openbsd-uefi-bootloader-howto/) If you've been listening over the past few weeks, you've heard about OpenBSD.s new UEFI boot-loader. We now have a blog post with detailed instructions on how to get setup with this on your own system. The initial setup is pretty straightforward, and should only take a few minutes at most. In involves the usual fdisk commands to create a FAT EFI partition, and placing the bootx64.efi file in the correct location. As a bonus, we even get instructions on how to enable the frame-buffer driver on systems without native Intel video support (ThinkPad x250 in this example) *** Recipe for building a 10Mpps FreeBSD based router (http://blog.cochard.me/2015/09/receipt-for-building-10mpps-freebsd.html) Olivier, (of FreeNAS and BSD Router Project fame) treats us this week to a neat blog post about building your own high-performance 10Mpps FreeBSD router As he first mentions, the hardware required will need to be beefy, no $200 miniPC here. In his setup he uses a 8 core Intel Xeon E5-2650, along with a Quad port 10 Gigabit Chelsio TS540-CR. He mentions that this doesn't work quite on stock FreeBSD yet, you will need to pull code in from the projects/routing (https://svnweb.freebsd.org/base/projects/routing/) which fixes an issue with scaling on cores, in this case he is shrinking the NIC queues down to 4 from 8. If you don't feel like doing the compiles yourself, he also includes links to experimental BSDRouter project images which he used to do the benchmarks Bonus! Nice graphic of the benchmarks from enabling IPFW or PF and what that does to the performance. *** Interview - Brandon Mercer - bmercer@openbsd.org (mailto:bmercer@openbsd.org) / @knowmercymod (https://twitter.com/knowmercymod) OpenBSD in Healthcare Sorry about the audio quality degradation. The last 7 or 8 minutes of the interview had to be cut, a problem with the software that captures the audio from skype and adds it to our compositor. My local monitor is analogue and did not experience the issue, so I was unaware of the issue during the recording *** News Roundup Nvidia releases new beta FreeBSD driver along with new kernel module (https://devtalk.nvidia.com/default/topic/884727/unix-graphics-announcements-and-news/linux-solaris-and-freebsd-driver-358-09-beta-/) Includes a new kernel module, nvidia-modeset.ko While this module does NOT have any user-settable features, it works with the existing nvidia.ko to provide kernel-mode setting (KMS) used by the integrated DRM within the kernel. The beta adds support for 805A and 960A nvidia cards Also fixes a memory leak and some regressions *** MidnightBSD 0.7-RELEASE (http://www.midnightbsd.org/pipermail/midnightbsd-users/Week-of-Mon-20150914/003462.html) We missed this while away at Euro and elsewhere, but MidnightBSD (A desktop-focused FreeBSD 6.1 Fork) has come out with a new 0.7 release This release primarily focuses on stability, but also includes important security fixes as well. It cherry-picks updates to a variety of FreeBSD base-system updates, and some important ZFS features, such as TRIM and LZ4 compression Their custom .mports. system has also gotten a slew of updates, with almost 2000 packages now available, including a WiP of Gnome3. It also brings support for starting / stopping services automatically at pkg install or removal. They note that this will most likely be the last i386 release, joining the club of other projects that are going 64bit only. *** "Open Source as a Career Path" (http://media.medfarm.uu.se/play/video/5400) The FreeBSD Project held a panel discussion (http://www.cb.uu.se/~kristina/WomENcourage/2014/2015-09-25_Friday/2015-09-25%20113238.JPG) of why Open Source makes a good career path at the ACM.s womENcourage conference in Uppsala, Sweden, the weekend before EuroBSDCon The Panel was lead by Dru Lavigne, and consisted of Deb Goodkin, Benedict Reuschling, Dan Langille, and myself We attempted to provide a cross section of experiences, including women in the field, the academic side, the community side, and the business side During the question period, Dan gave a great answer (https://gist.github.com/dlangille/e262bccdea08b89b5360) to the question of .Why do open source projects still use old technologies like mailing lists and IRC. The day before, the FreeBSD Foundation also had a booth at the career fair. We were the only open source project that attended. Other exhibitors included: Cisco, Facebook, Intel, Google, and Oracle. The following day, Dan also gave a workshop (http://www.cb.uu.se/~kristina/WomENcourage/2014/2015-09-25_Friday/2015-09-25%20113238.JPG) on how to contribute to an open source project *** Beastie-Bits NetBSD 2015PkgSrc Freeze (http://mail-index.netbsd.org/pkgsrc-users/2015/09/12/msg022186.html) Support for 802.11N for RealTek USB in FreeBSD (https://github.com/freebsd/freebsd/commits/master/sys/dev/usb/wlan/if_rsu.c) Wayland ported to DragonFlyBSD (https://github.com/DragonFlyBSD/DeltaPorts/pull/123) OpenSMTPd developer debriefs on audit report (http://undeadly.org/cgi?action=article&sid=20151013161745) FreeBSD fixes issue with pf under Xen with TSO. Errata coming soon (https://svnweb.freebsd.org/base?view=revision&revision=289316) Xinuos funds the HardenedBSD project (http://slexy.org/view/s2EBjrxQ9M) Feedback/Questions Evan (http://slexy.org/view/s21PMmNFIs) Darin writes in (http://slexy.org/view/s20qH07ox0) Jochen writes in (http://slexy.org/view/s2d0SFmRlD) ***
Interview with Deb Goodkin, Director of Operations for the FreeBSD Foundation.The most recent newsletter, which talks about what the foundation did the last half of last year.http://www.freebsdfoundation.org/press/2011Dec-newsletter.shtml#LetterTheir blog:http://freebsdfoundation.blogspot.com/Facebook:https://www.facebook.com/FreeBSDFoundationLastly, but probably most importantly, people can donate here:http://www.freebsdfoundation.org/donate/File Info: 24Min, 12MB.Ogg Link:https://archive.org/download/bsdtalk211/bsdtalk211.ogg
In interview with Marshall Kirk McKusick and George Neville-Neil about the FreeBSD Foundation.More information at http://www.freebsdfoundation.org/File Info: 34Min, 16MB.Ogg link: https://archive.org/download/bsdtalk224/bsdtalk224.oggEnding audio is the opening to "Carp License" from the OpenBSD 3.5 release.
I'm having a great time at BSDCan 2008 in Ottawa.Interview with Justin Gibbs from the FreeBSD Foundation.File Info: 11Min, 5MB.Ogg Link:https://archive.org/download/bsdtalk149/bsdtalk149.ogg
This week, we have Nigel Williams here to bring us all sorts of info about Multipath TCP, what it is, how it works and the ongoing effort to bring it into FreeBSD. All that and of course the latest BSD news coming your way, right now! This episode was brought to you by Headlines Backing out changes doesn.t always pinpoint the problem (https://blog.crashed.org/dont-backout/) Peter Wemm brings us a fascinating look at debugging an issue which occurred on the FreeBSD build cluster recently. Bottom line? Backing out something isn.t necessarily the fix, rather it should be apart of the diagnostic process In this particular case, a change to some mmap() functionality ended up exposing a bug in the kernel.s page fault handler which existed since (wait for it.) 1997! As Peter mentions at the bottom of the Article, this bug had been showing up for years, but was sporadic and often written off as a networking hiccup. *** BSD Router Project benchmarks new routing changes to FreeBSD (https://github.com/ocochard/netbenchs/blob/master/Xeon_E5-2650-8Cores-Chelsio_T540-CR/nXxq10g/results/fbsd11-melifaro.r287531/README.md) A project branch of FreeBSD -CURRENT has been created with a number of optimizations to the routing code Alexander V. Chernikov (melifaro@).s routing branch (https://svnweb.freebsd.org/base/projects/routing/?view=log) The net result is an almost doubling of peak performance in packets per second Performance scales well with the number of NIC queues (2 queues is 88% faster than 1 queue, 3 is 270% faster). Unlike the previous code, when the number of queues hits 4, performance is down by only 10%, instead of being cut nearly in half Other Benchmark Results, and the tools to do your own tests (https://github.com/ocochard/netbenchs) *** When is SSL not SSL? (http://www.tedunangst.com/flak/post/the-peculiar-libretunnel-situation) Our buddy Ted has a good write-up on a weird situation related to licensing of stunnel and LibreSSL The problem exists due to stunnel being released with a different license, that is technically incompatible with the GPL, as well as linking against non-OpenSSL versions. The author has also decided to create specific named exceptions when the *SSL lib is part of the base operating system, but does not personally consider LibreSSL as a valid linking target on its own Ted points out that the LibreSSL team considers LibreSSL == OpenSSL, so this may be a moot concern *** Update on systembsd (http://darknedgy.net/files/systembsd.pdf) We.ve mentioned the GSoC project to create a SystemD shim in OpenBSD before. Now we have the slides from Ian Sutton talking about this project. As a refresher, this project is to take DBUS and create daemons emulating various systemd components, such as hostnamed, localed, timedated, and friends. Written from scratch in C, it was mainly created in the hopes of becoming a port, allowing Gnome and related tools to function on OpenBSD. This is a good read, especially for current or aspiring porters who want to bring over newer versions of applications which now depend upon SystemD. *** Interview - Nigel Williams - njwilliams@swin.edu.au (njwilliams@swin.edu.au) Multipath TCP News Roundup OpenBSD UEFI boot loader (http://marc.info/?l=openbsd-cvs&m=144115942223734&w=2) We.ve mentioned the ongoing work to bring UEFI booting to OpenBSD and it looks like this has now landed in the tree The .fdisk. utility has also been updated with a new -b flag, when used with .-i. will create the special EFI system partition on amd64/i386 . (http://marc.info/?l=openbsd-cvs&m=144139348416071&w=2) Some twitter benchmarks (https://twitter.com/mherrb/status/641004331035193344) *** FreeBSD Journal, July/August issue (https://www.freebsdfoundation.org/journal/vol2_no4/) The latest issue of the FreeBSD Journal has arrived As always, the Journal opens with a letter from the FreeBSD Foundation Feature Articles: Groupon's Deal on FreeBSD -- How to drive adoption of FreeBSD at your organization, and lessons learned in retraining Linux sysadmins FreeBSD: The Isilon Experience -- Mistakes not to make when basing a product on FreeBSD. TL;DR: track head Reflections on FreeBSD.org: Packages -- A status update on where we are with binary packages, what issues have been overcome, and which still remain Inside the Foundation -- An overview of some of the things you might not be aware that the FreeBSD Foundation is doing to support the project and attract the next generation of committers Includes a book review of .The Practise of System and Network Administration. As usual, various other reports are included: The Ports Report, SVN Update, A conference report, a report from the Essen hackathon, and the Event Calendar *** Building ARMv6 packages on FreeBSD, the easy way (http://blogs.freebsdish.org/brd/2015/08/25/building-arm-packages-with-poudriere-the-simple-way/) Previously we have discussed how to build ARMv6 packages on FreeBSD We also interviewed Sean Bruno about his work in this area Thankfully, over time this process has been simplified, and no longer requires a lot of manual configuration, or fussing with the .image activator. Now, you can just build packages for your Raspberry Pi or similar device, just as simply as you would build for x86, it just takes longer to build. *** New PC-BSD Release Schedule (http://blog.pcbsd.org/2015/09/new-release-schedule-for-pc-bsd/) The PC-BSD Team has announce an updated release schedule for beyond 10.2 This schedule follows more closely the FreeBSD schedules, with major releases only occurring when FreeBSD does the next point update, or major version bump. PC-BSD.s source tree has been split into master(current) and stable as well PRODUCTION / EDGE packages will be built from stable, with PRODUCTION updated monthly now. The -CURRENT monthly images will contain the master source builds. *** Feedback/Questions Joris writes in (http://slexy.org/view/s21cguSv7E) Anonymous (http://slexy.org/view/s217A5NNGg) Darin (http://slexy.org/view/s20HyiqJV0) ***
We're away at AsiaBSDCon this week, but we've still got a packed episode for you. First up is a sequel to the "PC-BSD tour" segment from a while back, highlighting how ZFS boot environments work. After that, Justin Gibbs joins us to talk about the FreeBSD foundation's 15th anniversary. We'll return next week with a normal episode of BSD Now - which is of course, the place to B.. SD. This episode was brought to you by Special segment Demystifying Boot Environments in PC-BSD Interview - Justin Gibbs - gibbs@freebsd.org (mailto:gibbs@freebsd.org) / @freebsdfndation (https://twitter.com/freebsdfndation) The FreeBSD foundation's 15th anniversary Discussion The story of PC-BSD
This week on the show, we sat down with John-Mark Gurney to talk about modernizing FreeBSD's IPSEC stack. We'll learn what he's adding, what needed to be fixed and how we'll benefit from the changes. As always, answers to your emails and all of this week's news, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines BSD panel at Phoenix LUG (https://www.youtube.com/watch?v=3AOF7fm-TJ0) The Phoenix, Arizona Linux users group had a special panel so they could learn a bit more about BSD It had one FreeBSD user and one OpenBSD user, and they answered questions from the organizer and the people in the audience They covered a variety of topics, including filesystems, firewalls, different development models, licenses and philosophy It was a good "real world" example of things potential switchers are curious to know about They closed by concluding that more diversity is always better, and even if you've got a lot of Linux boxes, putting a few BSD ones in the mix is a good idea *** Book of PF signed copy auction (http://bsdly.blogspot.com/2014/10/the-book-of-pf-3rd-edition-is-here.html) Peter Hansteen (who we've had on the show (http://www.bsdnow.tv/episodes/2014_04_30-puffy_firewall)) is auctioning off the first signed copy of the new Book of PF All the profits from the sale will go to the OpenBSD Foundation (http://www.openbsd.org/donations.html) The updated edition of the book includes all the latest pf syntax changes, but also provides examples for FreeBSD and NetBSD's versions (which still use ALTQ, among other differences) If you're interested in firewalls, security or even just advanced networking, this book is a great one to have on your shelf - and the money will also go to a good cause Michael Lucas (http://www.bsdnow.tv/episodes/2013_11_06-year_of_the_bsd_desktop) has challenged Peter (https://www.marc.info/?l=openbsd-misc&m=141429413908567&w=2) to raise more for the foundation than his last book selling - let's see who wins Pause the episode, go bid on it (http://www.ebay.com/itm/321563281902) and then come back! *** FreeBSD Foundation goes to EuroBSDCon (http://freebsdfoundation.blogspot.com/2014/10/freebsd-foundation-goes-to-eurobsdcon.html) Some people from the FreeBSD Foundation went to EuroBSDCon this year, and come back with a nice trip report They also sponsored four other developers to go The foundation was there "to find out what people are working on, what kind of help they could use from the Foundation, feedback on what we can be doing to support the FreeBSD Project and community, and what features/functions people want supported in FreeBSD" They also have a second report (http://freebsdfoundation.blogspot.com/2014/10/eurobsdcon-trip-report-kamil-czekirda.html) from Kamil Czekirda A total of $2000 was raised at the conference *** OpenBSD 5.6 released (http://www.openbsd.org/56.html) Note: we're doing this story a couple days early - it's actually being released on November 1st (this Saturday), but we have next week off and didn't want to let this one slip through the cracks - it may be out by the time you're watching this Continuing their always-on-time six month release cycle, the OpenBSD team has released version 5.6 It includes support for new hardware, lots of driver updates, network stack improvements (SMP, in particular) and new security features 5.6 is the first formal release with LibreSSL, their fork of OpenSSL, and lots of ports have been fixed to work with it You can now hibernate your laptop when using a fully-encrypted filesystem (see our tutorial (http://www.bsdnow.tv/tutorials/fde) for that) ALTQ, Kerberos, Lynx, Bluetooth, TCP Wrappers and Apache were all removed This will serve as a "transitional" release for a lot of services: moving from Sendmail to OpenSMTPD, from nginx to httpd (http://www.bsdnow.tv/episodes/2014_09_03-its_hammer_time) and from BIND to Unbound Sendmail, nginx and BIND will be gone in the next release, so either migrate to the new stuff between now and then or switch to the ports versions As always, 5.6 comes with its own song and artwork (http://www.openbsd.org/lyrics.html#56) - the theme this time was obviously LibreSSL Be sure to check the full changelog (http://www.openbsd.org/plus56.html) (it's huge) and pick up a CD or tshirt (http://www.openbsd.org/orders.html) to support their efforts If you don't already have the public key releases are signed with, getting a physical CD is a good "out of bounds" way to obtain it safely Here are some cool images of the set (https://imgur.com/a/5PtFe) After you do your installation or upgrade (http://www.openbsd.org/faq/upgrade56.html), don't forget to head over to the errata page (http://www.openbsd.org/errata56.html) and apply any patches listed there *** Interview - John-Mark Gurney - jmg@freebsd.org (mailto:jmg@freebsd.org) / @encthenet (https://twitter.com/encthenet) Updating FreeBSD's IPSEC stack News Roundup Clang in DragonFly BSD (https://www.dragonflydigest.com/2014/10/22/14942.html) As we all know, FreeBSD got rid of GCC in 10.0, and now uses Clang almost exclusively on i386/amd64 Some DragonFly developers are considering migrating over as well, and one of them is doing some work to make the OS more Clang-friendly We'd love to see more BSDs switch to Clang/LLVM eventually, it's a lot more modern than the old GCC most are using *** reallocarray(): integer overflow detection for free (http://lteo.net/blog/2014/10/28/reallocarray-in-openbsd-integer-overflow-detection-for-free/) One of the less obvious features in OpenBSD 5.6 is a new libc function: "reallocarray()" It's a replacement function for realloc(3) that provides integer overflow detection at basically no extra cost Theo and a few other developers have already started (https://secure.freshbsd.org/search?project=openbsd&q=reallocarray) a mass audit of the entire source tree, replacing many instances with this new feature OpenBSD's explicit_bzero was recently imported into FreeBSD, maybe someone could also port over this too *** Switching from Linux blog (http://bothsidesofthence.tumblr.com/) A listener of the show has started a new blog series, detailing his experiences in switching over to BSD from Linux After over ten years of using Linux, he decided to give BSD a try after listening to our show (which is awesome) So far, he's put up a few posts about his initial thoughts, some documentation he's going through and his experiments so far It'll be an ongoing series, so we may check back in with him again later on *** Owncloud in a FreeNAS jail (https://www.youtube.com/watch?v=z6VQwOl4wE4) One of the most common emails we get is about running Owncloud in FreeNAS Now, finally, someone made a video on how to do just that, and it's even jailed A member of the FreeNAS community has uploaded a video on how to set it up, with lighttpd as the webserver backend If you're looking for an easy way to back up and sync your files, this might be worth a watch *** Feedback/Questions Ernõ writes in (http://slexy.org/view/s2XEsQdggZ) David writes in (http://slexy.org/view/s21EizH2aR) Kamil writes in (http://slexy.org/view/s24SAJ5im6) Torsten writes in (http://slexy.org/view/s20ABZe0RD) Dominik writes in (http://slexy.org/view/s208jQs9c6) *** Mailing List Gold That's not our IP (https://mail-index.netbsd.org/source-changes/2014/10/17/msg059564.html) Is this thing on? (https://lists.freebsd.org/pipermail/freebsd-acpi/2014-June/008644.html) ***
Coming up on the show, we'll be showing you how to set up a secure, SSL-only webserver. There's also an interview with Eric Le Blan about community participation and FreeBSD's role in the commercial server space. All that and more, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Password gropers take spamtrap bait (http://bsdly.blogspot.com/2014/08/password-gropers-take-spamtrap-bait.html) Our friend Peter Hansteen (http://www.bsdnow.tv/episodes/2014_04_30-puffy_firewall), who keeps his eyes glued to his log files, has a new blog post He seems to have discovered another new weird phenomenon in his pop3 logs "yes, I still run one, for the same bad reasons more than a third of my readers probably do: inertia" Someone tried to log in to his service with an address that was known to be invalid The rest of the post goes into detail about his theory of why someone would use a list of invalid addresses for this purpose *** Inside the Atheros wifi chipset (https://www.youtube.com/watch?v=WOcYTqoSQ68) Adrian Chadd - sometimes known in the FreeBSD community as "the wireless guy" - gave a talk at the Defcon Wireless Village 2014 He covers a lot of topics on wifi, specifically on Atheros chips and why they're so popular for open source development There's a lot of great information in the presentation, including cool (and evil) things you can do with wireless cards Very technical talk; some parts might go over your head if you're not a driver developer The raw video file is also available to download (https://archive.org/download/WirelessVillageAtDefCon22/20-Atheros.mp4) on archive.org Adrian has also recently worked on getting Kismet and Aircrack-NG to work better with FreeBSD, including packet injection and other fun things *** Trip report and hackathon mini-roundup (http://freebsdfoundation.blogspot.com/2014/08/bsdcan-trip-report-mark-linimon.html) A few more (late) reports from BSDCan and the latest OpenBSD hackathon have been posted Mark Linimon mentions some of the future plans for FreeBSD's release engineering and ports Bapt also has a BSDCan report (http://freebsdfoundation.blogspot.com/2014/08/bsdcan-trip-report-baptiste-daroussin.html) detailing his work on ports and packages Antoine Jacoutot writes about (http://undeadly.org/cgi?action=article&sid=20140812064946) his work at the most recent hackathon, working with rc configuration and a new /etc/examples layout Peter Hessler, a latecomer to the hackathon, details his experience (http://undeadly.org/cgi?action=article&sid=20140806125308) too, hacking on the installer and built-in upgrade function Christian Weisgerber talks about (http://undeadly.org/cgi?action=article&sid=20140803122705) starting some initial improvements of OpenBSD's ports infrastructure *** DragonFly BSD 3.8.2 released (http://lists.dragonflybsd.org/pipermail/commits/2014-August/270573.html) Although it was already branched, the release media is now available for DragonFly 3.8.2 This is a minor update, mostly to fix the recent OpenSSL vulnerabilities It also includes some various other small fixes *** Interview - Eric Le Blan - info@xinuos.com (mailto:info@xinuos.com) Xinuos' recent FreeBSD integration, BSD in the commercial server space Tutorial Building a hardened, feature-rich webserver (http://www.bsdnow.tv/tutorials/nginx) News Roundup Defend your network and privacy, FreeBSD version (http://networkfilter.blogspot.com/2014/08/defend-your-network-and-privacy-vpn.html) Back in episode 39 (http://www.bsdnow.tv/episodes/2014_05_28-the_friendly_sandbox), we covered a blog post about creating an OpenBSD gateway - partly based on our tutorial (http://www.bsdnow.tv/tutorials/openbsd-router) This is a follow-up post, by the same author, about doing a similar thing with FreeBSD He mentions some of the advantages and disadvantages between the two operating systems, and encourages users to decide for themselves which one suits their needs The rest is pretty much the same things: firewall, VPN, DHCP server, DNSCrypt, etc. *** Don't encrypt all the things (http://www.tedunangst.com/flak/post/dont-encrypt-all-the-things) Another couple of interesting blog posts from Ted Unangst (http://www.bsdnow.tv/episodes/2014_02_05-time_signatures) about encryption It talks about how Google recently started ranking sites with HTTPS higher in their search results, and then reflects on how sometimes encryption does more harm than good After heartbleed, the ones who might be able to decrypt your emails went from just a three-letter agency to any script kiddie He also talks a bit about some PGP weaknesses and a possible future replacement He also has another, similar post entitled "in defense of opportunistic encryption (http://www.tedunangst.com/flak/post/in-defense-of-opportunistic-encryption)" *** New automounter lands in FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=270096) The work on the new automounter has just landed in 11-CURRENT With help from the FreeBSD Foundation, we'll have a new "autofs" kernel option Check the SVN viewer online to read over the man pages if you're not running -CURRENT You can also read a bit about it in the recent newsletter (https://www.freebsdfoundation.org/press/2014jul-newsletter#Project3) *** OpenSSH 6.7 CFT (https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-August/032810.html) It's been a little while since the last OpenSSH release, but 6.7 is almost ready Our friend Damien Miller (http://www.bsdnow.tv/episodes/2013_12_18-cryptocrystalline) issued a call for testing for the upcoming version, which includes a fair amount of new features It includes some old code removal, some new features and some internal reworkings - we'll cover the full list in detail when it's released This version also officially supports being built with LibreSSL now Help test it out and report any findings, especially if you have access to something a little more exotic than just a BSD system *** Feedback/Questions David writes in (http://slexy.org/view/s20yIP7VXa) Lachlan writes in (http://slexy.org/view/s2DeeUjAn6) Francis writes in (http://slexy.org/view/s216imwEb0) Frank writes in (http://slexy.org/view/s2oc8vavWe) Sean writes in (http://slexy.org/view/s20wL61sSr) ***
We kick off the first episode with the latest BSD news, show you how to avoid intrusion detection systems and talk to Peter Hessler about BGP spam blacklists! Headlines Radeon KMS commited (https://lists.freebsd.org/pipermail/svn-src-head/2013-August/050931.html) Committed by Jean-Sebastien Pedron Brings kernel mode setting to -CURRENT, will be in 10.0-RELEASE (ETA 12/2013) 10-STABLE is expected to be branched in October, to begin the process of stabilizing development Initial testing shows it works well May be merged to 9.X, but due to changes to the VM subsystem this will require a lot of work, and is currently not a priority for the Radeon KMS developer Still suffers from the syscons / KMS switcher issues, same as Intel video More info: https://wiki.freebsd.org/AMD_GPU *** VeriSign Embraces FreeBSD (http://www.eweek.com/enterprise-apps/verisign-embraces-open-source-freebsd-for-diversity/) "BSD is quite literally at the very core foundation of what makes the Internet work" Using BSD and Linux together provides reliability and diversity Verisign gives back to the community, runs vBSDCon "You get comfortable with something because it works well for your particular purposes and can find a good community that you can interact with. That all rang true for us with FreeBSD." *** fetch/libfetch get a makeover (http://freshbsd.org/commit/freebsd/r253680) Adds support for SSL certificate verification Requires root ca bundle (security/rootcanss) Still missing TLS SNI support (Server Name Indication, allows name based virtual hosts over SSL) *** FreeBSD Foundation Semi-Annual Newsletter (http://www.freebsdfoundation.org/press/2013Jul-newsletter) The FreeBSD Foundation took the 20th anniversary of FreeBSD as an opportunity to look at where the project is, and where it might want to go The foundation sets out some basic goals that the project should strive towards: Unify User Experience “ensure that knowledge gained mastering one task translates to the next” “if we do pay attention to consistency, not only will FreeBSD be easier to use, it will be easier to learn” Design for Human and Programmatic Use 200 machines used to be considered a large deployment, with high density servers, blades, virtualization and the cloud, that is not so anymore “the tools we provide for status reporting, configuration, and control of FreeBSD just do not scale or fail to provide the desired user experience” “The FreeBSD of tomorrow needs to give programmability and human interaction equal weighting as requirements” Embrace New Ways to Document FreeBSD More ‘Getting Started' sections in documentation Link to external How-Tos and other documentation “upgrade the cross-referencing and search tools built into FreeBSD, so FreeBSD, not an Internet search engine, is the best place to learn about FreeBSD” Spring Fundraising Campaign, April 17 - May 31, raised a total of $219,806 from 12 organizations and 365 individual donors. In the same period last year we raised a total of $23,422 from 2 organizations and 53 individuals Funds donated to the FreeBSD Foundation have been used on these projects recently: Capsicum security-component framework Transparent superpages support of the FreeBSD/ARM architecture Expanded and faster IPv6 Native in-kernel iSCSI stack Five New TCP Congestion Control Algorithms Direct mapped I/O to avoid extra memory copies Unified Extensible Firmware Interface (UEFI) boot environment Porting FreeBSD to the Genesi Efika MX SmartBook laptop (ARM-based) NAND Flash filesystem and storage stack Funds were also used to sponsor a number of BSD focused conferences: BSDCan, EuroBSDCon, AsiaBSDCon, BSDDay, NYCBSDCon, vBSDCon, plus Vendor summits and Developer summits It is important that the foundation receive donations from individuals, to maintain their tax exempt status in the USA. Even a donation of $5 helps make it clear that the FreeBSD Foundation is backed by a large community, not only a few vendors Donate Today (http://www.freebsdfoundation.org/donate) *** The place to B...SD Ohio Linuxfest, Sept. 13-15, 2013 (http://ohiolinux.org/schedule) Very BSD friendly Kirk McKusick giving the keynote BSD Certification on the 15th, all other stuff on the 14th Multiple BSD talks *** LinuxCon, Sept. 16-18, 2013 (http://events.linuxfoundation.org/events/linuxcon-north-america) Dru Lavigne and Kris Moore will be manning a FreeBSD booth Number of talks of interest to BSD users, including ZFS coop (http://linuxconcloudopenna2013.sched.org/event/b50b23f3ed3bd728fa0052b54021a2cc?iframe=yes&w=900&sidebar=yes&bg=no) EuroBSDCon, Sept. 26-29, 2013 (http://2013.eurobsdcon.org/eurobsdcon-2013/talks/) Tutorials on the 26 & 27th (plus private FreeBSD DevSummit) 43 talks spread over 3 tracks on the 28 & 29th Keynote by Theo de Raadt Hosted in the picturesque St. Julians Area, Malta (Hilton Conference Centre) *** Interview - Peter Hessler - phessler@openbsd.org (mailto:phessler@openbsd.org) / @phessler (https://twitter.com/phessler) Using BGP to distribute spam blacklists and whitelists Tutorial Using stunnel to hide your traffic from Deep Packet Inspection (http://www.bsdnow.tv/tutorials/stunnel) News Roundup NetBSD 6.1.1 released (https://blog.netbsd.org/tnf/entry/netbsd_6_1_1_released) First security/bug fix update of the NetBSD 6.1 release branch Fixes 4 security vulnerabilities Adds 4 new sysctls to avoid IPv6 DoS attacks Misc. other updates *** Sudo Mastery (http://blather.michaelwlucas.com/archives/1792) MWL is a well-known author of many BSD books Also does SSH, networking, DNSSEC, etc. Next book is about sudo, which comes from OpenBSD (did you know that?) Available for preorder now at a discounted price *** Documentation Infrastructure Enhancements (http://freebsdfoundation.blogspot.com/2013/08/new-funded-project-documentation.html) Gábor Kövesdán has completed a funded project to improve the infrastructure behind the documentation project Will upgrade documentation from DocBook 4.2 to DocBook 4.5 and at the same time migrate to proper XML tools. DSSSL is an old and dead standard, which will not evolve any more. DocBook 5.0 tree added *** FreeBSD FIBs get new features (https://svnweb.freebsd.org/base?view=revision&revision=254943) FIBs (as discussed earlier in the interview) are Forward Information Bases (technical term for a routing table) The FreeBSD kernel can be compiled to allow you to maintain multiple FIBs, creating separate routing tables for different processes or jails In r254943 ps(1) is extended to support a new column ‘fib', to display which routing table a process is using *** FreeNAS 9.1.0 and 9.1.1 released (http://www.ixsystems.com/resources/ix/news/ixsystems-announces-revolutionary-freenas-910-release.html) Many improvements in nearly all areas, big upgrade Based on FreeBSD 9-STABLE, lots of new ZFS features Cherry picked some features from 10-CURRENT New volume manager and easy to use plugin management system 9.1.1 released shortly thereafter to fix a few UI and plugin bugs *** BSD licensed "patch" becomes default (http://freshbsd.org/commit/freebsd/r253689) bsdpatch has become mature, does what GNU patch can do, but has a much better license Approved by portmgr@ for use in ports Added WITHGNUPATCH build option for people who still need it ***
© 2020-2025 Ivy Podcast Discovery LLC
