POPULARITY
16 episodes with Michael W Lucas
4 episodes with Michael W Lucas
3 episodes with Michael W Lucas
4 episodes with Michael W Lucas
4 episodes with Michael W Lucas
Guest Deb Goodkin Panelist Richard Littauer Show Notes This episode of Sustain features host Richard Littauer having a conversation with guest Deb Goodkin, the Executive Director of the FreeBSD Foundation. They discuss the history, development, and distinct characteristics of FreeBSD as an operating system independent from Linux, and the foundational support provided by the FreeBSD Foundation. The conversation delves into the organization's efforts in software development, security, community engagement, and plans to increase diversity. They also highlight the celebration of FreeBSD Day and Deb's passion for maintaining and promoting FreeBSD. Press download now to hear more! [00:00:55] Deb explains her role and clarifies that FreeBSD is an independent operating system, originally derived from Unix developed at Bell Labs. [00:02:27] Richard mentions “BSD” stands for “Berkeley Software Distribution.” Deb confirms the existence of the BSD license and its relevance. [00:03:27] Deb admits it's challenging to determine the number of FreeBSD users due to the nature of the BSD license but estimates hundreds of thousands based on device usage. [00:04:08] Deb distinguishes the FreeBSD Foundation from the FreeBSD Project, explaining the foundation's role in supporting the project and the community through funding, development, and advocacy. [00:06:47] Deb compares the FreeBSD Foundation to other foundations like the GNOME Foundation and the Linux Foundation, emphasizing FreeBSD's focus on software development and community support. [00:08:15] Deb reveals most of the foundation's funding comes from corporate sponsors. She highlights the foundation's increasing focus on security in response to global standards and government guidelines, mentioning FreeBSD's involvement in security research and development. [00:12:45] There's a discussion on the foundation's approach to policy and standards, noting the limited resources but emphasizing the importance of corporate and community input in deciding priorities and projects. [00:15:52] Richard questions Deb about feedback mechanisms used to gauge the FreeBSD community's concerns about security. He wonders about the participation dynamics. Deb details the communication channels within the FreeBSD community, mentions a recent community survey and what the responses revealed, and explains the 1% negative feedback. [00:19:56] The conversation shifts towards diversity and inclusion within FreeBSD, questioning the foundation's strategies to enhance geographical and demographic diversity. Deb discusses the foundation's efforts to increase gender diversity, mentioning initiatives targeting women, especially through involvement in computing events and universities. [00:24:05] Richard inquires about the prevalence of women in executive roles within open source foundations, sparking a conversation on gender dynamics in leadership and the intense dedication required in these roles. Deb shares her admiration for the passion and intelligence within the FreeBSD community and emphasizes the foundation's commitment to supporting the project and its people. [00:26:51] Why is Deb so passionate about FreeBSD? She reflects on her deep connection with the community, her continuous learning in the field, and her desire to see the project thrive long-term. [00:29:06] Richard acknowledges the upcoming FreeBSD Day, and Deb shares how it showcases the uses of FreeBSD with a week of activities. [00:31:26] Find out here you can learn more about FreeBSD and Deb's work online. {00:33:34] Richard asks about the FreeBSD logo, confused by the demon mascot. Deb clarifies that the mascot, named Beastie, is a playful representation related to the UNIX's background processes (daemons) and not an actual logo. Spotlight [00:35:34] Richard's spotlight is Philip Pullman's Trilogy book series. [00:36:12] Deb's spotlight is Michael Lucas's book, Absolute FreeBSD. Links SustainOSS (https://sustainoss.org/) SustainOSS Twitter (https://twitter.com/SustainOSS?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) SustainOSS Discourse (https://discourse.sustainoss.org/) podcast@sustainoss.org (mailto:podcast@sustainoss.org) SustainOSS Mastodon (https://mastodon.social/tags/sustainoss) Open Collective-SustainOSS (Contribute) (https://opencollective.com/sustainoss) Richard Littauer Socials (https://www.burntfen.com/2023-05-30/socials) Deb Goodkin X/Twitter (https://x.com/dgoodkin?lang=en) Deb Goodkin LinkedIn (https://www.linkedin.com/in/deb-goodkin-b282924a/) FreeBSD Foundation (https://freebsdfoundation.org/) 2024 FreeBSD Community Survey is Here-FreeBSD Foundation (https://freebsdfoundation.org/blog/2024-freebsd-community-survey-is-here/) FreeBSD Foundation YouTube (https://www.youtube.com/@freebsdfoundation2324) The FreeBSD Project (https://www.freebsd.org/) His Dark Materials by Philip Pullman-Wikipedia (https://en.wikipedia.org/wiki/His_Dark_Materials) Absolute FreeBSD by Michael W. Lucas (no starch press) (https://nostarch.com/absfreebsd3) Credits Produced by Richard Littauer (https://www.burntfen.com/) Edited by Paul M. Bahr at Peachtree Sound (https://www.peachtreesound.com/) Show notes by DeAnn Bahr Peachtree Sound (https://www.peachtreesound.com/) Special Guest: Deb Goodkin.
Author Michael W. Lucas joins us in this interview to talk about his latest book projects. Find out what he's up to regarding mail servers, conferences, his views on ChatGPT, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Interview - Michael W. Lucas - mwl@mwl.io (mailto:mwl@mwl.io) OpenBSD Mastery Filesystems Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. - Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Special Guest: Michael W Lucas.
In this episode, Martin and Chris host an eclectic panel of contributors to the *other* major FLOSS operating system family - you guessed it: the flavours of the Berkeley Software Distribution (aka BSD among friends). Disclaimer: you may be tempted to diverge from the Path of the Righteousness also known as Linux and give this alternative a spin. So this episode is *not* for the faint-hearted - listen at your own discretion! Also: the true defective nature of our beloved (?) hosts' past will be revealed - an episode not be missed despite the caveat! Plus a refresher on spaced-out operating system concepts including library operating systems and a rant on Android and friends. In addition to some cool BSD trolling... Links: OpenBSD: https://www.openbsd.org FeeBSD: https://www.freebsd.org NetBSD: https://www.netbsd.org DragonFlyBSD: https://www.dragonflybsd.org 386BSD: https://www.386bsd.org RUMP kernel: https://en.wikipedia.org/wiki/Rump_kernel Library operating systems: https://www.sigarch.org/leave-your-os-at-home-the-rise-of-library-operating-systems Free BSD Linux Compatibility: https://docs.freebsd.org/en/books/handbook/linuxemu BSD Jails (original paper): https://papers.freebsd.org/2000/phk-jails.files/sane2000-jail.pdf FeeBSD Ports: https://www.freebsd.org/ports NetBSD pkgsrc: https://www.netbsd.org/docs/pkgsrc ZFS: https://en.wikipedia.org/wiki/ZFS BSD's pledge: https://unix.stackexchange.com/questions/410056/what-is-openbsds-pledge-in-short FreeBSD and Netflix: https://papers.freebsd.org/2019/fosdem/looney-netflix_and_freebsd BSD Firewalls: https://docs.freebsd.org/en/books/handbook/firewalls Michael W. Lucas' "Savaged by Systemd": hhttps://www.goodreads.com/book/show/36190710-savaged-by-systemd Linux vs. Minix: https://www.oreilly.com/openbook/opensources/book/appa.html Pegasus spyware: https://en.wikipedia.org/wiki/Pegasus_(spyware) BSD History presentation: https://www.youtube.com/watch?v=DEEr6dT-4uQ The tragedy of systemd: https://www.youtube.com/watch?v=o_AIw9bGogo The Rise and Fall of Copyleft: https://archive.org/download/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3
In this episode, we interview Michael W. Lucas about his latest book projects including Git sync murder, TLS Mastery, getting paid for creative work, writing tools and techniques, and more. NOTES Interview - Michael W. Lucas - mwl@mwl.io (mailto:mwl@mwl.io) / @mwlauthor (https://twitter.com/mwlauthor) Cashflow for Creators (https://mwl.io/nonfiction/biz-craft) Charity Auction Against Human Trafficking (https://mwl.io/archives/12526) This is the rfc about what to not do. (https://datatracker.ietf.org/doc/html/rfc9049) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Special Guest: Michael W Lucas.
Mark interviews D.F. Hart about her journey through indie publishing, the mistakes she has made on that journey and the dramatic results of the decision to do a makeover on her thriller covers. Prior to the interview, Mark shares comments from recent episodes as well as a personal update. He also shares a word from this episode's sponsor.You can learn more about how you can get your work distributed to retailers and library systems around the world at starkreflections.ca/Findaway. In their conversation, DeAnna and Mark discuss: Why DeAnna writes under D.F. Hart and Faith Hart Her first experience publishing through a vanity press in 2010 How that first novel's idea came to her originally in 1989 How March of 2019 was Deanna's first experience publishing fully indie (as opposed to being done through a vanity press outfit) Reader reaction to the balance between law enforcement work and personal/home life in her main character How a supplemental character that Deanna felt sorry for and wanted to see her get a happy ending became the main character for the first book in her romance series Some of the other complicated ways the two series cross over with one another DeAnna's grandmother's advice to learn from the mistakes of others because you can't possibly make all those mistakes on your own The consult that DeAnna booked with Craig over at Hidden Gems and how that led to a discussion that helped her decide to change her covers Coming to the realization that, while her covers were pretty, they weren't right for the genre The book cover designer that DeAnna hired to re-do her thriller book covers The increase in sales and reviews/ratings on all the major retailers How December 2020 was DeAnna's first four figure month as an author, due to the cover update and the BookBub Feature Deal The uptake in newsletter subscribers and BookBub followers that has happened The importance of measuring results, analyzing data, and keeping a finger on the pulse of your business What DeAnna would have told her 2010 self How Vanity Press outfits prey on the lack of information of authors The importance of celebrating all of your successes along the way After the interview, Mark reflects on the decision DeAnna had to make to invest more in covers after having already invested in that in the past. Links of Interest DeAnna's Website Author Facebook Page D.F. Hart BookBub Goodreads Faith Hart BookBub Goodreads DeAnna's Article on Hidden Gems Rocking Book Covers Findaway Voices EP 170 - The Genre of Andrew Pyper EP 169 - The World Needs Your Book With Monica Leonelle EP 166 - Wording Around with Kathy Mac EP 165 - Rude Awakenings from Sleeping Rough EP 162 - Happy Accidents with Joshua Pantallaresco EP 138 - Cash Flow for Creators with Michael W. Lucas EP 127 - Consistency in Word Count with Blaze Ward EP 90 - Craig Martelle on Successful Indie Author Release Strategies Space: 1975 Anthology Wide Writer Survey Wide for the Win Submission Form Patreon for Stark Reflections The introductory, end, and bumper music for this podcast (“Laser Groove”) was composed and produced by Kevin MacLeod of www.incompetech.com and is Licensed under Creative Commons: By Attribution 3.0
FreeBSD Remote Process Plugin Final Milestone achieved, Tailscale for OpenBSD, macOS to FreeBSD migration, monitoring of our OpenBSD machines, OPNsense 20.7.6 released, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD Remote Process Plugin: Final Milestone Achieved (https://www.moritz.systems/blog/freebsd-remote-plugin-final-milestone-achieved/) Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are working on a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one. Tailscale on OpenBSD (https://rakhesh.com/linux-bsd/tailscale-on-openbsd/) I spent some time setting this up today evening and thought I’d post the steps here. Nothing fancy, just putting together various pieces actually. I assume you know what Tailscale is; if not check out their website. Basically it is a mesh network built on top of Wireguard. Using it you can have all your devices both within your LAN(s) and outside be on one overlay network as if they are all on the same LAN and can talk to each other. It’s my new favourite thing! News Roundup macOS to FreeBSD migration a.k.a why I left macOS (https://antranigv.am/weblog_en/posts/macos_to_freebsd/) This is not a technical documentation for how I migrated from macOS to FreeBSD. This is a high-level for why I migrated from macOS to FreeBSD. Not so long ago, I was using macOS as my daily driver. The main reason why I got a macbook was the underlying BSD Unix and the nice graphics it provides. Also, I have an iPhone. But they were also the same reasons for why I left macOS. Our monitoring of our OpenBSD machines, such as it is (as of November 2020 (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/OurOpenBSDMonitoring) We have a number of OpenBSD firewalls in service (along with some other OpenBSD servers for things like VPN endpoints), and I was recently asked how we monitor PF and overall network traffic on them. I had to disappoint the person who asked with my answer, because right now we mostly don't (although this is starting to change). OPNsense 20.7.6 released (https://opnsense.org/opnsense-20-7-6-released/) This update brings the usual mix of reliability fixes, plugin and third party software updates: FreeBSD, HardenedBSD, PHP, OpenSSH, StrongSwan, Suricata and Syslog-ng amongst others. Please note that Let's Encrypt users need to reissue their certificates manually after upgrading to this version to fix the embedded certificate chain issue with the current signing CA switch going on. NYC Bug Jan 2021 with Michael W. Lucas (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/nycbug) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions cy - .so files (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/cy%20-%20.so%20files) ben - mixer volume (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/ben%20-%20mixer%20volume) probono - live cds (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/383/feedback/probono%20-%20live%20cds) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Interview with Michael W. Lucas: SNMP and TLS book, cashflow for creators, book sale and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines Interview with Michael W. Lucas SNMP Book (https://mwl.io/nonfiction/networking#snmp) The Networknomicon (https://mwl.io/nonfiction/networking#networknomicon) Sponsor the TLS Book (https://www.tiltedwindmillpress.com/product-category/sponsor/) Cashflow for creators (https://mwl.io/nonfiction/biz-craft) Book sale (https://mwl.io/blog/9313) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** Special Guest: Michael W Lucas.
OpenBSD 6.8 has been released, NetBSD 9.1 is out, OpenZFS devsummit report, BastilleBSD’s native container management for FreeBSD, cleaning up old tarsnap backups, Michael W. Lucas’ book sale, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines OpenBSD 6.8 (https://www.openbsd.org/68.html) Released Oct 18, 2020. (OpenBSD's 25th anniversary) NetBSD 9.1 Released (https://www.netbsd.org/releases/formal-9/NetBSD-9.1.html) The NetBSD Project is pleased to announce NetBSD 9.1, the first update of the NetBSD 9 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements. OpenZFS Developer Summit 2020 (https://klarasystems.com/articles/openzfs-developer-summit-part-1/) As with most other conferences in the last six months, this year’s OpenZFS Developer’s Summit was a bit different than usual. Held via Zoom to accommodate for 2020’s new normal in terms of social engagements, the conference featured a mix of talks delivered live via webinars, and breakout sessions held as regular meetings. This helped recapture some of the “hallway track” that would be lost in an online conference. • After attending the conference, I wrote up some of my notes from each of the talks • Part 2 (https://klarasystems.com/articles/openzfs-developer-summit-part-2/) ZFS and FreeBSD Support Klara offers flexible Support Subscriptions for your ZFS and FreeBSD infrastructure, simply sign up for our monthly subscription! What's even better is that for the month of October we are giving away 3 months for free, for every yearly subscription, and one month free when you sign up for a 6-months subscription! Check it out on our website! (https://klarasystems.com/support/) News Roundup BastilleBSD - native container management for FreeBSD (https://fibric.hashnode.dev/bastillebsd-native-container-management-for-freebsd) Some time ago, I had the requirement to use FreeBSD in a project, and soon the question came up if Docker and Kubernetes can be used. On FreeBSD, Docker is not very well supported, and even if you can get it running, Linux is used in a Docker container. My experience with Docker on FreeBSD is awful, and so I started looking for alternatives. A quick search on one of the most significant online search engines led me to Jails and then to BastilleBSD. Tarsnap – cleaning up old backups (https://dan.langille.org/2020/09/10/tarsnap-cleaning-up-old-backups/) I use Tarsnap for my critical data. Case in point, I use it to backup my Bacula database dump. I use Bacula to backup my hosts. The database in question keeps track of what was backed up, from what host, the file size, checksum, where that backup is now, and many other items. Losing this data is annoying but not a disaster. It can be recreated from the backup volumes, but that is time consuming. As it is, the file is dumped daily, and rsynced to multiple locations. MWL - BookSale (https://mwl.io/archives/8009) For those interested in such things, I recently posted my 60,000th tweet. This prodded me to try an experiment I’ve been pondering for a while. Over at my ebookstore, two of my books are now on a “Name Your Own Price” sale. You can get git commit murder and PAM Mastery for any price you wish, with a minimum of $1. Beastie Bits Brian Kernighan: UNIX, C, AWK, AMPL, and Go Programming | Lex Fridman Podcast #109 (https://www.youtube.com/watch?v=O9upVbGSBFo) The UNIX Time-Sharing System - Dennis M. Ritchie and Ken Thompson - July 1974 (https://chsasank.github.io/classic_papers/unix-time-sharing-system.html#) Using a 1930 Teletype as a Linux Terminal (https://www.youtube.com/watch?v=2XLZ4Z8LpEE) *** ###Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions lars - infosec handbook (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/374/feedback/lars%20-%20infosec%20handbook.md) scott - zfs import (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/374/feedback/scott%20-%20zfs%20import.md) zhong - first episode (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/374/feedback/zhong%20-%20first%20episode.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
In this episode Mark interviews Rachael Herron about her writing, her multiple pseudonyms, being a hybrid author, and writing memoirs. Prior to the interview, Mark shares some comments from recent episodes and via Twitter, some information about an offer the Clark Chamberlain has for his Story Stuck Consultation, and a personal update. He also thanks new patron Johanna Rothman as well as all existing patrons for the podcast, and lets his patrons know that they were automatically entered in a chance to win a story stuck consultation from Clark. In their conversation, Mark and Rachael discuss: The thought that being a writer is like having homework for the rest of your life Rachael's thriller (Hush Little Baby) that she just turned into her editor at Penguin Random House and which comes out in May 2021 Getting over the "who cares" issues she was having working on this novel in the middle of a global pandemic and a world-wide drive to end systemic racism by recognizing that it's okay to delve deep into a focused project like this How willpower can be a finite, exhaustible resource How a good writer can take people out of their normal world and the feeling when a reader reaches out to let Rachael know that her book has moved them or helped them The feeling when you're reading a book that the person who wrote it "just gets you" The value of feeling seen and heard, of community, and of responding to reader emails Being involved in blogging from the early days, and the connections that can come out of that The fact that, despite appearing to share everything socially and on blogs, podcasts, etc, the content is actually curated - but that it's still authentic Rachael's book "Fast Draft Your Memoir: Write Your Life Story in 45 Hours" The importance of recognizing that there are plenty of people who want to write a memoir but who don't want to writers How there are so many genres within memoir One of the principal tenants of a good memoir - in that you treat yourself more critically than anyone else, and that you give and show grace to anyone who might have done you wrong How when you write about the most uncomfortable things how people really lean in to listen How Rachael's book "Letters to New Authors" is based on a series of emails she created to encourage other writers Being a fully hybrid author, and the mental shift involved when Rachael moves between traditional and indie publishing How not having control of little things can drive Rachael crazy with some of her books How Rachael's books, both, traditionally published and indie published, tend to sell better in eBook format The advice that Rachael would give to her younger self Ira Glass's theory of "the gap" And so much more . . . After the interview Mark reflects on writing about uncomfortable things. Links of Interest: Rachael Herron's Website Rachael's Resources for Writers (Including a weekly encouragement newsletter) Rachael's Essays on the Creative Life (Patreon) Rachael's Podcasts How Do You Write? The Writer's Well Rachael interviewed by Mark on the Draft2Digital Spotlight The Writer's Well EP 178 - If You Couldn't Write Books, How Would You Be Creative? (Mark guest co-host) Episode 137 - Dreaming, Singing, and Moving People with Ashley Joanisse Episode 138 - Cash Flow for Creators with Michael W. Lucas Mark's Stupid Dad Joke Short Film "The Things We Miss Most" Public Lending Right Program Canada Public Lending Right International Story Stuck Consulation Up, Up, And Away Superhero Storybundle Patreon for Stark Reflections Rachael Herron is the internationally bestselling author of more than two dozen books, including thrillers (under R.H. Herron), mainstream fiction, feminist romance, memoir, and nonfiction about writing. She received her MFA in writing from Mills College, Oakland, and she teaches writing extension workshops at both UC Berkeley and Stanford. She is a proud member of the NaNoWriMo Writer's Board and is the host and co-host of two awesome podcasts for writers. The introductory, end, and bumper music for this podcast (“Laser Groove”) was composed and produced by Kevin MacLeod of www.incompetech.com and is Licensed under Creative Commons: By Attribution 3.0
In this episode, Mark interviews Michael W. Lucas about he books he has written which range range from crime and SF thrillers to arcane tomes on how to build your own Verizon. They also chat quite a bit about Michael's latest book: Cash Flow for Creators. In the opening of the episode, Mark shares some comments from recent episodes, as well as a word from this episode's sponsor, Findaway Voices. You can learn more about how you can get your work distributed to retailers and library systems around the world at starkreflections.ca/Findaway. Immediately prior to the interview Mark shares a personal update about his recent appearance on the overnight radio program Coast to Coast with George Noory, and all the positive results that came from that. In their conversation, Michael and Mark talk about: How writing for Michael started when he was about 4 and realized that books were created by human beings and this was something people could grow up to do The evolution of getting to full time writer via rejection notices, selling more and more things, and working on the craft Michael's start in writing writing for table top games The styles of fiction that Michael writes and where his heart is in writing The many "what ifs" that Michael enjoys, such as "What if there were Orcs in 1927 Detroit running booze across the border?" What happened to make leaving the full-time role as a writer possible, which involved a lot of watching Michael's new book for writers called CALL FLOW FOR CREATORS and why it was written out of self-defense The reality that making a living is about cashflow How evening out cashflow helps reduce the highs and lows of the typical cycle of a writer's income The importance of looking at one's lifestyle, where they live, how they want to live, when making decisions regarding the amount of money they need to make, the expenses they are willing to live with, etc The special editions that Michael does of some of his books, made specifically for his readers How, if you can do something special that speaks to your people, how Michael would encourage you to give it a try The way that Michael does his crowd-funding a little differently, using disintermediation, rather than using IndieGoGo or Kickstarter How not only are all of Michael's books published wide everywhere, but that they are all also available for sale direct The sponsorships that Michael does that includes surprises, such as a gift he will send you upon the book's publication And more . . . After the interview, Mark reflects on a few things that Michael spoke about, including when you know your fans, you can feed them the right things that they value, and how people will support you and they WANT you to succeed - if you treat your readers well. Mark then shares details about how listeners can win a copy of Michael's book Cashflow for Creators. Links of Interest: Michael's Website Episode 137 - Action Sequences and Sex Scenes with Joshua Essoe Joshua's Kickstarter Project: Essoe's Guide to Writing: Action Sequences and Sex Scenes! Coast to Coast AM with George Noory (June 5, 2020) Patreon for Stark Reflections Michael W Lucas has written 30-odd books, ranging from crime and SF thrillers to arcane tomes on how to build your own Verizon. His latest book is "Cash Flow for Creators." The introductory, end, and bumper music for this podcast (“Laser Groove”) was composed and produced by Kevin MacLeod of www.incompetech.com and is Licensed under Creative Commons: By Attribution 3.0
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol. The post Heavy Networking 518: Learning To Live With SNMP appeared first on Packet Pushers.
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol. The post Heavy Networking 518: Learning To Live With SNMP appeared first on Packet Pushers.
Perhaps the most arcane technology in networking is the Simple Network Management Protocol (SNMP), which, despite its name, is one of the utmost confusing pieces of technology to actually use. Guests Michael W. Lucas and Allan Jude share tips and lessons on how to get the most you can out of this protocol. The post Heavy Networking 518: Learning To Live With SNMP appeared first on Packet Pushers.
In this episode, we interview Michael W. Lucas about his latest book projects, including the upcoming SNMP Mastery book.
In this episode, we interview Michael W. Lucas about his latest book projects, including the upcoming SNMP Mastery book. Interview - Michael Lucas Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag. Special Guest: Michael W Lucas.
In this episode, we interview Michael W. Lucas about his latest book projects, including the upcoming SNMP Mastery book.
vBSDcon 2019 recap, Unix at 50, OpenBSD on fan-less Tuxedo InfinityBook, humungus - an hg server, how to configure a network dump in FreeBSD, and more. Headlines vBSDcon Recap Allan and Benedict attended vBSDcon 2019, which ended last week. It was held again at the Hyatt Regency Reston and the main conference was organized by Dan Langille of BSDCan fame.The two day conference was preceded by a one day FreeBSD hackathon, where FreeBSD developers had the chance to work on patches and PRs. In the evening, a reception was held to welcome attendees and give them a chance to chat and get to know each other over food and drinks. The first day of the conference was opened with a Keynote by Paul Vixie about DNS over HTTPS (DoH). He explained how we got to the current state and what challenges (technical and social) this entails. If you missed this talk and are dying to see it, it will also be presented at EuroBSDCon next week John Baldwin followed up by giving an overview of the work on “In-Kernel TLS Framing and Encryption for FreeBSD” abstract (https://www.vbsdcon.com/schedule/2019-09-06.html#talk:132615) and the recent commit we covered in episode 313. Meanwhile, Brian Callahan was giving a separate session in another room about “Learning to (Open)BSD through its porting system: an attendee-driven educational session” where people had the chance to learn about how to create ports for the BSDs. David Fullard’s talk about “Transitioning from FreeNAS to FreeBSD” was his first talk at a BSD conference and described how he built his own home NAS setup trying to replicate FreeNAS’ functionality on FreeBSD, and why he transitioned from using an appliance to using vanilla FreeBSD. Shawn Webb followed with his overview talk about the “State of the Hardened Union”. Benedict’s talk about “Replacing an Oracle Server with FreeBSD, OpenZFS, and PostgreSQL” was well received as people are interested in how we liberated ourselves from the clutches of Oracle without compromising functionality. Entertaining and educational at the same time, Michael W. Lucas talk about “Twenty Years in Jail: FreeBSD Jails, Then and Now” closed the first day. Lucas also had a table in the hallway with his various tech and non-tech books for sale. People formed small groups and went into town for dinner. Some returned later that night to some work in the hacker lounge or talk amongst fellow BSD enthusiasts. Colin Percival was the keynote speaker for the second day and had an in-depth look at “23 years of software side channel attacks”. Allan reprised his “ELI5: ZFS Caching” talk explaining how the ZFS adaptive replacement cache (ARC) work and how it can be tuned for various workloads. “By the numbers: ZFS Performance Results from Six Operating Systems and Their Derivatives” by Michael Dexter followed with his approach to benchmarking OpenZFS on various platforms. Conor Beh was also a new speaker to vBSDcon. His talk was about “FreeBSD at Work: Building Network and Storage Infrastructure with pfSense and FreeNAS”. Two OpenBSD talks closed the talk session: Kurt Mosiejczuk with “Care and Feeding of OpenBSD Porters” and Aaron Poffenberger with “Road Warrior Disaster Recovery: Secure, Synchronized, and Backed-up”. A dinner and reception was enjoyed by the attendees and gave more time to discuss the talks given and other things until late at night. We want to thank the vBSDcon organizers and especially Dan Langille for running such a great conference. We are grateful to Verisign as the main sponsor and The FreeBSD Foundation for sponsoring the tote bags. Thanks to all the speakers and attendees! humungus - an hg server (https://humungus.tedunangst.com/r/humungus) Features View changes, files, changesets, etc. Some syntax highlighting. Read only. Serves multiple repositories. Allows cloning via the obvious URL. Supports go get. Serves files for downloads. Online documentation via mandoc. Terminal based admin interface. News Roundup OpenBSD on fan-less Tuxedo InfinityBook 14″ v2. (https://hazardous.org/archive/blog/openbsd/2019/09/02/OpenBSD-on-Infinitybook14) The InfinityBook 14” v2 is a fanless 14” notebook. It is an excellent choice for running OpenBSD - but order it with the supported wireless card (see below.). I’ve set it up in a dual-boot configuration so that I can switch between Linux and OpenBSD - mainly to spot differences in the drivers. TUXEDO allows a variety of configurations through their webshop. The dual boot setup with grub2 and EFI boot will be covered in a separate blogpost. My tests were done with OpenBSD-current - which is as of writing flagged as 6.6-beta. See Article for breakdown of CPU, Wireless, Video, Webcam, Audio, ACPI, Battery, Touchpad, and MicroSD Card Reader Unix at 50: How the OS that powered smartphones started from failure (https://arstechnica.com/gadgets/2019/08/unix-at-50-it-starts-with-a-mainframe-a-gator-and-three-dedicated-researchers/) Maybe its pervasiveness has long obscured its origins. But Unix, the operating system that in one derivative or another powers nearly all smartphones sold worldwide, was born 50 years ago from the failure of an ambitious project that involved titans like Bell Labs, GE, and MIT. Largely the brainchild of a few programmers at Bell Labs, the unlikely story of Unix begins with a meeting on the top floor of an otherwise unremarkable annex at the sprawling Bell Labs complex in Murray Hill, New Jersey. It was a bright, cold Monday, the last day of March 1969, and the computer sciences department was hosting distinguished guests: Bill Baker, a Bell Labs vice president, and Ed David, the director of research. Baker was about to pull the plug on Multics (a condensed form of MULTiplexed Information and Computing Service), a software project that the computer sciences department had been working on for four years. Multics was two years overdue, way over budget, and functional only in the loosest possible understanding of the term. Trying to put the best spin possible on what was clearly an abject failure, Baker gave a speech in which he claimed that Bell Labs had accomplished everything it was trying to accomplish in Multics and that they no longer needed to work on the project. As Berk Tague, a staffer present at the meeting, later told Princeton University, “Like Vietnam, he declared victory and got out of Multics.” Within the department, this announcement was hardly unexpected. The programmers were acutely aware of the various issues with both the scope of the project and the computer they had been asked to build it for. Still, it was something to work on, and as long as Bell Labs was working on Multics, they would also have a $7 million mainframe computer to play around with in their spare time. Dennis Ritchie, one of the programmers working on Multics, later said they all felt some stake in the success of the project, even though they knew the odds of that success were exceedingly remote. Cancellation of Multics meant the end of the only project that the programmers in the Computer science department had to work on—and it also meant the loss of the only computer in the Computer science department. After the GE 645 mainframe was taken apart and hauled off, the computer science department’s resources were reduced to little more than office supplies and a few terminals. Some of Allan’s favourite excerpts: In the early '60s, Bill Ninke, a researcher in acoustics, had demonstrated a rudimentary graphical user interface with a DEC PDP-7 minicomputer. Acoustics still had that computer, but they weren’t using it and had stuck it somewhere out of the way up on the sixth floor. And so Thompson, an indefatigable explorer of the labs’ nooks and crannies, finally found that PDP-7 shortly after Davis and Baker cancelled Multics. With the rest of the team’s help, Thompson bundled up the various pieces of the PDP-7—a machine about the size of a refrigerator, not counting the terminal—moved it into a closet assigned to the acoustics department, and got it up and running. One way or another, they convinced acoustics to provide space for the computer and also to pay for the not infrequent repairs to it out of that department’s budget. McIlroy’s programmers suddenly had a computer, kind of. So during the summer of 1969, Thompson, Ritchie, and Canaday hashed out the basics of a file manager that would run on the PDP-7. This was no simple task. Batch computing—running programs one after the other—rarely required that a computer be able to permanently store information, and many mainframes did not have any permanent storage device (whether a tape or a hard disk) attached to them. But the time-sharing environment that these programmers had fallen in love with required attached storage. And with multiple users connected to the same computer at the same time, the file manager had to be written well enough to keep one user’s files from being written over another user’s. When a file was read, the output from that file had to be sent to the user that was opening it. It was a challenge that McIlroy’s team was willing to accept. They had seen the future of computing and wanted to explore it. They knew that Multics was a dead-end, but they had discovered the possibilities opened up by shared development, shared access, and real-time computing. Twenty years later, Ritchie characterized it for Princeton as such: “What we wanted to preserve was not just a good environment in which to do programming, but a system around which a fellowship could form.” Eventually when they had the file management system more or less fleshed out conceptually, it came time to actually write the code. The trio—all of whom had terrible handwriting—decided to use the Labs’ dictating service. One of them called up a lab extension and dictated the entire code base into a tape recorder. And thus, some unidentified clerical worker or workers soon had the unenviable task of trying to convert that into a typewritten document. Of course, it was done imperfectly. Among various errors, “inode” came back as “eye node,” but the output was still viewed as a decided improvement over their assorted scribbles. In August 1969, Thompson’s wife and son went on a three-week vacation to see her family out in Berkeley, and Thompson decided to spend that time writing an assembler, a file editor, and a kernel to manage the PDP-7 processor. This would turn the group’s file manager into a full-fledged operating system. He generously allocated himself one week for each task. Thompson finished his tasks more or less on schedule. And by September, the computer science department at Bell Labs had an operating system running on a PDP-7—and it wasn’t Multics. By the summer of 1970, the team had attached a tape drive to the PDP-7, and their blossoming OS also had a growing selection of tools for programmers (several of which persist down to this day). But despite the successes, Thompson, Canaday, and Ritchie were still being rebuffed by labs management in their efforts to get a brand-new computer. It wasn’t until late 1971 that the computer science department got a truly modern computer. The Unix team had developed several tools designed to automatically format text files for printing over the past year or so. They had done so to simplify the production of documentation for their pet project, but their tools had escaped and were being used by several researchers elsewhere on the top floor. At the same time, the legal department was prepared to spend a fortune on a mainframe program called “AstroText.” Catching wind of this, the Unix crew realized that they could, with only a little effort, upgrade the tools they had written for their own use into something that the legal department could use to prepare patent applications. The computer science department pitched lab management on the purchase of a DEC PDP-11 for document production purposes, and Max Mathews offered to pay for the machine out of the acoustics department budget. Finally, management gave in and purchased a computer for the Unix team to play with. Eventually, word leaked out about this operating system, and businesses and institutions with PDP-11s began contacting Bell Labs about their new operating system. The Labs made it available for free—requesting only the cost of postage and media from anyone who wanted a copy. The rest has quite literally made tech history. See the link for the rest of the article How to configure a network dump in FreeBSD? (https://www.oshogbo.vexillium.org/blog/68/) A network dump might be very useful for collecting kernel crash dumps from embedded machines and machines with a larger amount of RAM then available swap partition size. Besides net dumps we can also try to compress the core dump. However, often this may still not be enough swap to keep whole core dump. In such situation using network dump is a convenient and reliable way for collecting kernel dump. So, first, let’s talk a little bit about history. The first implementation of the network dumps was implemented around 2000 for the FreeBSD 4.x as a kernel module. The code was implemented in 2010 with the intention of being part of FreeBSD 9.0. However, the code never landed in FreeBSD. Finally, in 2018 with the commit r333283 by Mark Johnston the netdump client code landed in the FreeBSD. Subsequently, many other commitments were then implemented to add support for the different drivers (for example r333289). The first official release of FreeBSD, which support netdump is FreeBSD 12.0. Now, let’s get back to the main topic. How to configure the network dump? Two machines are needed. One machine is to collect core dump, let’s call it server. We will use the second one to send us the core dump - the client. See the link for the rest of the article Beastie Bits Sudo Mastery 2nd edition is not out (https://mwl.io/archives/4530) Empirical Notes on the Interaction Between Continuous Kernel Fuzzing and Development (http://users.utu.fi/kakrind/publications/19/vulnfuzz_camera.pdf) soso (https://github.com/ozkl/soso) GregKH - OpenBSD was right (https://youtu.be/gUqcMs0svNU?t=254) Game of Trees (https://gameoftrees.org/faq.html) Feedback/Questions BostJan - Another Question (http://dpaste.com/1ZPCCQY#wrap) Tom - PF (http://dpaste.com/3ZSCB8N#wrap) JohnnyK - Changing VT without keys (http://dpaste.com/3QZQ7Q5#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
FreeBSD 11.3 has been released, OpenBSD workstation, write your own fuzzer for the NetBSD kernel, Exploiting FreeBSD-SA-19:02.fd, streaming to twitch using OpenBSD, 3 different ways of dumping hex contents of a file, and more. Headlines FreeBSD 11.3-RELEASE Announcement (https://www.freebsd.org/releases/11.3R/announce.html) The FreeBSD Release Engineering Team is pleased to announce the availability of FreeBSD 11.3-RELEASE. This is the fourth release of the stable/11 branch. Some of the highlights: The clang, llvm, lld, lldb, and compiler-rt utilities as well as libc++ have been updated to upstream version 8.0.0. The ELF Tool Chain has been updated to version r3614. OpenSSL has been updated to version 1.0.2s. The ZFS filesystem has been updated to implement parallel mounting. The loader(8) has been updated to extend geli(8) support to all architectures. The pkg(8) utility has been updated to version 1.10.5. The KDE desktop environment has been updated to version 5.15.3. The GNOME desktop environment has been updated to version 3.28. The kernel will now log the jail(8) ID when logging a process exit. Several feature additions and updates to userland applications. Several network driver firmware updates. Warnings for features deprecated in future releases will now be printed on all FreeBSD versions. Warnings have been added for IPSec algorithms deprecated in RFC 8221. Deprecation warnings have been added for weaker algorithms when creating geli(8) providers. And more... OpenBSD Is Now My Workstation (https://sogubsys.com/openbsd-is-now-my-workstation-operating-system/) Why OpenBSD? Simply because it is the best tool for the job for me for my new-to-me Lenovo Thinkpad T420. Additionally, I do care about security and non-bloat in my personal operating systems (business needs can have different priorities, to be clear). I will try to detail what my reasons are for going with OpenBSD (instead of GNU/Linux, NetBSD, or FreeBSD of which I’m comfortable using without issue), challenges and frustrations I’ve encountered, and what my opinions are along the way. Disclaimer: in this post, I’m speaking about what is my opinion, and I’m not trying to convince you to use OpenBSD or anything else. I don’t truly care, but wanted to share in case it could be useful to you. I do hope you give OpenBSD a shot as your workstation, especially if it has been a while. A Bit About Me and OpenBSD I’m not new to OpenBSD, to be clear. I’ve been using it off and on for over 20 years. The biggest time in my life was the early 2000s (I was even the Python port maintainer for a bit), where I not only used it for my workstation, but also for production servers and network devices. I just haven’t used it as a workstation (outside of a virtual machine) in over 10 years, but have used it for servers. Workstation needs, especially for a primary workstation, are greatly different and the small things end up mattering most. News Roundup Write your own fuzzer for NetBSD kernel! [Part 1] (https://blog.netbsd.org/tnf/entry/write_your_own_fuzzer_for) How Fuzzing works? The dummy Fuzzer. The easy way to describe fuzzing is to compare it to the process of unit testing a program, but with different input. This input can be random, or it can be generated in some way that makes it unexpected form standard execution perspective. The simplest 'fuzzer' can be written in few lines of bash, by getting N bytes from /dev/rand, and putting them to the program as a parameter. Coverage and Fuzzing What can be done to make fuzzing more effective? If we think about fuzzing as a process, where we place data into the input of the program (which is a black box), and we can only interact via input, not much more can be done. However, programs usually process different inputs at different speeds, which can give us some insight into the program's behavior. During fuzzing, we are trying to crash the program, thus we need additional probes to observe the program's behaviour. Additional knowledge about program state can be exploited as a feedback loop for generating new input vectors. Knowledge about the program itself and the structure of input data can also be considered. As an example, if the input data is in the form of HTML, changing characters inside the body will probably cause less problems for the parser than experimenting with headers and HTML tags. For open source programs, we can read the source code to know what input takes which execution path. Nonetheless, this might be very time consuming, and it would be much more helpful if this can be automated. As it turns out, this process can be improved by tracing coverage of the execution vBSDcon - CFP - Call for Papers ends July 19th (https://vbsdcon.com/) You can submit your proposal at https://easychair.org/conferences/?conf=vbsdcon2019 The talks will have a very strong technical content bias. Proposals of a business development or marketing nature are not appropriate for this venue. If you are doing something interesting with a BSD operating system, please submit a proposal. Whether you are developing a very complex system using BSD as the foundation, or helping others and have a story to tell about how BSD played a role, we want to hear about your experience. People using BSD as a platform for research are also encouraged to submit a proposal. Possible topics include: How we manage a giant installation with respect to handling spam, snd/or sysadmin, and/or networking, Cool new stuff in BSD, Tell us about your project which runs on BSD. Both users and developers are encouraged to share their experiences. Exploiting FreeBSD-SA-19:02.fd (https://secfault-security.com/blog/FreeBSD-SA-1902.fd.html) In February 2019 the FreeBSD project issued an advisory about a possible vulnerability in the handling of file descriptors. UNIX-like systems such as FreeBSD allow to send file descriptors to other processes via UNIX-domain sockets. This can for example be used to pass file access privileges to the receiving process. Inside the kernel, file descriptors are used to indirectly reference a C struct which stores the relevant information about the file object. This could for instance include a reference to a vnode which describes the file for the file system, the file type, or the access privileges. What really happens if a UNIX-domain socket is used to send a file descriptor to another process is that for the receiving process, inside the kernel a reference to this struct is created. As the new file descriptor is a reference to the same file object, all information is inherited. For instance, this can allow to give another process write access to a file on the drive even if the process owner is normally not able to open the file writable. The advisory describes that FreeBSD 12.0 introduced a bug in this mechanism. As the file descriptor information is sent via a socket, the sender and the receiver have to allocate buffers for the procedure. If the receiving buffer is not large enough, the FreeBSD kernel attempts to close the received file descriptors to prevent a leak of these to the sender. However, while the responsible function closes the file descriptor, it fails to release the reference from the file descriptor to the file object. This could cause the reference counter to wrap. The advisory further states that the impact of this bug is possibly a local privilege escalation to gain root privileges or a jail escape. However, no proof-of-concept was provided by the advisory authors. In the next section, the bug itself is analyzed to make a statement about the bug class and a guess about a possible exploitation primitive. After that, the bug trigger is addressed. It follows a discussion of three imaginable exploitation strategies - including a discussion of why two of these approaches failed. In the section before last, the working exploit primitive is discussed. It introduces a (at least to the author’s knowledge) new exploitation technique for these kind of vulnerabilities in FreeBSD. The stabilization of the exploit is addressed, too. The last section wraps everything up in a conclusion and points out further steps and challenges. The privilege escalation is now a piece of cake thanks to a technique used by kingcope, who published a FreeBSD root exploit in 2005, which writes to the file /etc/libmap.conf. This configuration file can be used to hook the loading of dynamic libraries if a program is started. The exploit therefore creates a dynamic library, which copies /bin/sh to another file and sets the suid-bit for the copy. The hooked library is libutil, which is for instance called by su. Therefore, a call to su by the user will afterwards result in a suid copy of /bin/sh. Streaming to Twitch using OpenBSD (https://dataswamp.org/~solene/2019-07-06-twitch.html) Introduction If you ever wanted to make a twitch stream from your OpenBSD system, this is now possible, thanks to OpenBSD developer thfr@ who made a wrapper named fauxstream using ffmpeg with relevant parameters. The setup is quite easy, it only requires a few steps and searching on Twitch website two informations, hopefully, to ease the process, I found the links for you. You will need to make an account on twitch, get your api key (a long string of characters) which should stay secret because it allow anyone having it to stream on your account. These same techniques should work for Twitch, YouTube Live, Periscope, Facebook, etc, including the live streaming service ScaleEngine provides free to BSD user groups. There is also an open source application called ‘OBS’ or Open Broadcaster Studio. It is in FreeBSD ports and should work on all of the other BSDs as well. It has a GUI and supports compositing and green screening. We use it heavily at ScaleEngine and it is also used at JupiterBroadcasting in place of WireCast, a $1000-per-copy commercial application. Beastie Bits Portland BSD Pizza Night - 2019-07-25 19:00 - Rudy's Gourmet Pizza (http://calagator.org/events/1250475868) KnoxBUG - Michael W. Lucas : Twenty Years in Jail (http://knoxbug.org/2019-07-29) Ohio Linuxfest - CFP - Closes August 17th (https://ohiolinux.org/call-for-presentations/) My college (NYU Tandon) is moving their CS department and I saw this on a shelf being moved (https://old.reddit.com/r/freebsd/comments/cdx8fp/my_college_nyu_tandon_is_moving_their_cs/) 3 different ways of dumping hex contents of a file (https://moopost.blogspot.com/2019/07/3-different-ways-of-dumping-hex.html) Feedback/Questions Sebastian - ZFS setup toward ESXi (http://dpaste.com/0DRKFH6#wrap) Christopher - Questions (http://dpaste.com/2YNN1SH) Ser - Bhyve and Microsoft SQL (http://dpaste.com/1F5TMT0#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
FreeBSD 11.3-beta 1 is out, BSDCan 2019 recap, OpenIndiana 2019.04 is out, Overview of ZFS Pools in FreeNAS, why open source firmware is important for security, a new Opnsense release, wireguard on OpenBSD, and more. Headlines FreeBSD 11.3-b1 is out BSDCan 2019 Recap We’re back from BSDCan and it was a packed week as always. It started with bhyvecon on Tuesday. Meanwhile, Benedict spent the whole day in productive meetings: annual FreeBSD Foundation board meeting and FreeBSD Journal editorial board meeting. On Wednesday, tutorials for BSDCan started as well as the FreeBSD Developer Summit. In the mornings, there were presentations in the big auditorium, while working groups about networking, failsafe bootcode, development web services, swap space management, and testing/CI were held. Friday had a similar format with an update from the FreeBSD core team and the “have, need, want” session for FreeBSD 13. In the afternoon, there were working groups about translation tools, package base, GSoC/Outreachy, or general hacking. Benedict held his Icinga tutorial in the afternoon with about 15 people attending. Devsummit presentation slides can be found on the wiki page and video recordings done by ScaleEngine are available on FreeBSD’s youtube channel. The conference program was a good mixture of sysadmin and tech talks across the major BSDs. Benedict saw the following talks: How ZFS snapshots really work by Matt Ahrens, 20 years in Jail by Michael W. Lucas, OpenZFS BOF session, the future of OpenZFS and FreeBSD, MQTT for system administrators by Jan-Piet Mens, and spent the rest of the time in between in the hallway track. Photos from the event are available on Ollivier Robert’s talegraph and Diane Bruce’s website for day 1, day 2, conference day 1, and conference day 2. Thanks to all the sponsors, supporters, organizers, speakers, and attendees for making this yet another great BSDCan. Next year’s BSDCan will be from June 2 - 6, 2020. OpenIndiana 2019.04 is out We have released a new OpenIndiana Hipster snapshot 2019.04. The noticeable changes: Firefox was updated to 60.6.3 ESR Virtualbox packages were added (including guest additions) Mate was updated to 1.22 IPS has received updates from OmniOS CE and Oracle IPS repos, including automatic boot environment naming Some OI-specific applications have been ported from Python 2.7/GTK 2 to Python 3.5/GTK 3 Quick Demo Video: https://www.youtube.com/watch?v=tQ0-fo3XNrg News Roundup Overview of ZFS Pools in FreeNAS FreeNAS uses the OpenZFS (ZFS) file system, which handles both disk and volume management. ZFS offers RAID options mirror, stripe, and its own parity distribution called RAIDZ that functions like RAID5 on hardware RAID. The file system is extremely flexible and secure, with various drive combinations, checksums, snapshots, and replication all possible. For a deeper dive on ZFS technology, read the ZFS Primer section of the FreeNAS documentation. SUGGEST LAYOUT attempts to balance usable capacity and redundancy by automatically choosing an ideal vdev layout for the number of available disks. The following vdev layout options are available when creating a pool: Stripe data is shared on two drives, similar to RAID0) Mirror copies data on two drives, similar to RAID1 but not limited to 2 disks) RAIDZ1 single parity similar to RAID5 RAIDZ2 double parity similar to RAID6 RAIDZ3 which uses triple parity and has no RAID equivalent Why OpenSource Firmware is Important for Security Roots of Trust The goal of the root of trust should be to verify that the software installed in every component of the hardware is the software that was intended. This way you can know without a doubt and verify if hardware has been hacked. Since we have very little to no visibility into the code running in a lot of places in our hardware it is hard to do this. How do we really know that the firmware in a component is not vulnerable or that is doesn’t have any backdoors? Well we can’t. Not unless it was all open source. Every cloud and vendor seems to have their own way of doing a root of trust. Microsoft has Cerberus, Google has Titan, and Amazon has Nitro. These seem to assume an explicit amount of trust in the proprietary code (the code we cannot see). This leaves me with not a great feeling. Wouldn’t it be better to be able to use all open source code? Then we could verify without a doubt that the code you can read and build yourself is the same code running on hardware for all the various places we have firmware. We could then verify that a machine was in a correct state without a doubt of it being vulnerable or with a backdoor. It makes me wonder what the smaller cloud providers like DigitalOcean or Packet have for a root of trust. Often times we only hear of these projects from the big three or five. OPNsense This update addresses several privilege escalation issues in the access control implementation and new memory disclosure issues in Intel CPUs. We would like to thank Arnaud Cordier and Bill Marquette for the top-notch reports and coordination. Here are the full patch notes: system: address CVE-2019-11816 privilege escalation bugs[1] (reported by Arnaud Cordier) system: /etc/hosts generation without interfacehasgateway() system: show correct timestamp in config restore save message (contributed by nhirokinet) system: list the commands for the pluginctl utility when n+ argument is given system: introduce and use userIsAdmin() helper function instead of checking for 'page-all' privilege directly system: use absolute path in widget ACLs (reported by Netgate) system: RRD-related cleanups for less code exposure interfaces: add EN DUID Generation using OPNsense PEN (contributed by Team Rebellion) interfaces: replace legacygetallinterface_addresses() usage firewall: fix port validation in aliases with leading / trailing spaces firewall: fix outbound NAT translation display in overview page firewall: prevent CARP outgoing packets from using the configured gateway firewall: use CARP net.inet.carp.demotion to control current demotion in status page firewall: stop live log poller on error result dhcpd: change rule priority to 1 to avoid bogon clash dnsmasq: only admins may edit custom options field firmware: use insecure mode for base and kernel sets when package fingerprints are disabled firmware: add optional device support for base and kernel sets firmware: add Hostcentral mirror (HTTP, Melbourne, Australia) ipsec: always reset rightallowany to default when writing configuration lang: say "hola" to Spanish as the newest available GUI language lang: updates for Chinese, Czech, Japanese, German, French, Russian and Portuguese network time: only admins may edit custom options field openvpn: call openvpnrefreshcrls() indirectly via plugin_configure() for less code exposure openvpn: only admins may edit custom options field to prevent privilege escalation (reported by Bill Marquette) openvpn: remove custom options field from wizard unbound: only admins may edit custom options field wizard: translate typehint as well plugins: os-freeradius 1.9.3 fixes string interpolation in LDAP filters (contributed by theq86) plugins: os-nginx 1.12[2] plugins: os-theme-cicada 1.17 (contributed by Team Rebellion) plugins: os-theme-tukan 1.17 (contributed by Team Rebellion) src: timezone database information update[3] src: install(1) broken with partially matching relative paths[4] src: microarchitectural Data Sampling (MDS) mitigation[5] ports: carootnss 3.44 ports: php 7.2.18[6] ports: sqlite 3.28.0[7] ports: strongswan custom XAuth generic patch removed wiregaurd on OpenBSD Earlier this week I imported a port for WireGuard into the OpenBSD ports tree. At the moment we have the userland daemon and the tools available. The in-kernel implementation is only available for Linux. At the time of writing there are packages available for -current. Jason A. Donenfeld (WireGuard author) has worked to support OpenBSD in WireGuard and as such his post on ports@ last year got me interested in WireGuard, since then others have toyed with WireGuard on OpenBSD before and as such I've used Ted's article as a reference. Note however that some of the options mentioned there are no longer valid. Also, I'll be using two OpenBSD peers here. The setup will be as follows: two OpenBSD peers, of which we'll dub wg1 the server and wg2 the client. The WireGuard service on wg1 is listening on 100.64.4.3:51820. Conclusion WireGuard (cl)aims to be easier to setup and faster than OpenVPN and while I haven't been able to verify the latter, the first is certainly true...once you've figured it out. Most documentation out there is for Linux so I had to figure out the wireguardgo service and the tun parameters. But all in all, sure, it's easier. Especially the client configuration on iOS which I didn't cover here because it's essentially pkgadd libqrencode ; cat client.conf | qrencode -t ansiutf8, scan the code with the WireGuard app and you're good to go. What is particularly neat is that WireGuard on iOS supports Always-on. Beastie Bits Serenity OS vkernels vs pmap Brian Kernighan interviews Ken Thompson Improvements in forking, threading, and signal code DragonFly 5.4.3 NetBSD on the Odroid C2 Feedback/Questions Paulo - Laptops A Listener - Thanks Bostjan - Extend a pool and lower RAM footprint Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.
This week we have a special episode with a Michael W. Lucas interview about his latest jail book that’s been released. We’re talking all things jails, writing, book sponsoring, the upcoming BSDCan 2019 conference, and more. ###Interview - Michael W. Lucas - mwl@mwl.io / @mwlauthor FreeBSD Mastery: Jails BR: Welcome back to the show and congratulations on your latest book. How many books did you have to write before you could start on FreeBSD Mastery: Jails? AJ: How much research did you have to do about jails? BR: The book talks about something called ‘incomplete’ jails. What do you mean by that? AJ: There are a lot of jail management frameworks out there. Why did you chose to write about iocage in the book? BR: How many jails do you run yourself? AJ: Can you tell us a bit about how you handle book sponsorship these days? BR: What other books (fiction and non-fiction) are you currently working on? AJ: Which talks are you looking forward to attend at the upcoming BSDCan conference? BR: How is the BSD user group going? AJ: Anything else you’d like to mention before we release you from our interview jail cell? Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.
This week we have a special episode with a Michael W. Lucas interview about his latest jail book that’s been released. We’re talking all things jails, writing, book sponsoring, the upcoming BSDCan 2019 conference, and more.
This week we have a special episode with a Michael W. Lucas interview about his latest jail book that’s been released. We’re talking all things jails, writing, book sponsoring, the upcoming BSDCan 2019 conference, and more.
FreeBSD 12.0 is finally here, partly-cloudy IPsec VPN, KLEAK with NetBSD, How to create synth repos, GhostBSD author interview, and more. ##Headlines FreeBSD 12.0 is available After a long release cycle, the wait is over: FreeBSD 12.0 is now officially available. We’ve picked a few interesting things to cover in the show, make sure to read the full Release Notes Userland: Group permissions on /dev/acpi have been changed to allow users in the operator GID to invoke acpiconf(8) to suspend the system. The default devfs.rules(5) configuration has been updated to allow mount_fusefs(8) with jail(8). The default PAGER now defaults to less(1) for most commands. The newsyslog(8) utility has been updated to reject configuration entries that specify setuid(2) or executable log files. The WITH_REPRODUCIBLE_BUILD src.conf(5) knob has been enabled by default. A new src.conf(5) knob, WITH_RETPOLINE, has been added to enable the retpoline mitigation for userland builds. Userland applications: The dtrace(1) utility has been updated to support if and else statements. The legacy gdb(1) utility included in the base system is now installed to /usr/libexec for use with crashinfo(8). The gdbserver and gdbtui utilities are no longer installed. For interactive debugging, lldb(1) or a modern version of gdb(1) from devel/gdb should be used. A new src.conf(5) knob, WITHOUT_GDB_LIBEXEC has been added to disable building gdb(1). The gdb(1) utility is still installed in /usr/bin on sparc64. The setfacl(1) utility has been updated to include a new flag, -R, used to operate recursively on directories. The geli(8) utility has been updated to provide support for initializing multiple providers at once when they use the same passphrase and/or key. The dd(1) utility has been updated to add the status=progress option, which prints the status of its operation on a single line once per second, similar to GNU dd(1). The date(1) utility has been updated to include a new flag, -I, which prints its output in ISO 8601 formatting. The bectl(8) utility has been added, providing an administrative interface for managing ZFS boot environments, similar to sysutils/beadm. The bhyve(8) utility has been updated to add a new subcommand to the -l and -s flags, help, which when used, prints a list of supported LPC and PCI devices, respectively. The tftp(1) utility has been updated to change the default transfer mode from ASCII to binary. The chown(8) utility has been updated to prevent overflow of UID or GID arguments where the argument exceeded UID_MAX or GID_MAX, respectively. Kernel: The ACPI subsystem has been updated to implement Device object types for ACPI 6.0 support, required for some Dell, Inc. Poweredge™ AMD® Epyc™ systems. The amdsmn(4) and amdtemp(4) drivers have been updated to attach to AMD® Ryzen 2™ host bridges. The amdtemp(4) driver has been updated to fix temperature reporting for AMD® 2990WX CPUs. Kernel Configuration: The VIMAGE kernel configuration option has been enabled by default. The dumpon(8) utility has been updated to add support for compressed kernel crash dumps when the kernel configuration file includes the GZIO option. See rc.conf(5) and dumpon(8) for additional information. The NUMA option has been enabled by default in the amd64 GENERIC and MINIMAL kernel configurations. Device Drivers: The random(4) driver has been updated to remove the Yarrow algorithm. The Fortuna algorithm remains the default, and now only, available algorithm. The vt(4) driver has been updated with performance improvements, drawing text at rates ranging from 2- to 6-times faster. Deprecated Drivers: The lmc(4) driver has been removed. The ixgb(4) driver has been removed. The nxge(4) driver has been removed. The vxge(4) driver has been removed. The jedec_ts(4) driver has been removed in 12.0-RELEASE, and its functionality replaced by jedec_dimm(4). The DRM driver for modern graphics chipsets has been marked deprecated and marked for removal in FreeBSD 13. The DRM kernel modules are available from graphics/drm-stable-kmod or graphics/drm-legacy-kmod in the Ports Collection as well as via pkg(8). Additionally, the kernel modules have been added to the lua loader.conf(5) module_blacklist, as installation from the Ports Collection or pkg(8) is strongly recommended. The following drivers have been deprecated in FreeBSD 12.0, and not present in FreeBSD 13.0: ae(4), de(4), ed(4), ep(4), ex(4), fe(4), pcn(4), sf(4), sn(4), tl(4), tx(4), txp(4), vx(4), wb(4), xe(4) Storage: The UFS/FFS filesystem has been updated to support check hashes to cylinder-group maps. Support for check hashes is available only for UFS2. The UFS/FFS filesystem has been updated to consolidate TRIM/BIO_DELETE commands, reducing read/write requests due to fewer TRIM messages being sent simultaneously. TRIM consolidation support has been enabled by default in the UFS/FFS filesystem. TRIM consolidation can be disabled by setting the vfs.ffs.dotrimcons sysctl(8) to 0, or adding vfs.ffs.dotrimcons=0 to sysctl.conf(5). NFS: The NFS version 4.1 server has been updated to include pNFS server support. ZFS: ZFS has been updated to include new sysctl(8)s, vfs.zfs.arc_min_prefetch_ms and vfs.zfs.arc_min_prescient_prefetch_ms, which improve performance of the zpool(8) scrub subcommand. The new spacemap_v2 zpool feature has been added. This provides more efficient encoding of spacemaps, especially for full vdev spacemaps. The large_dnode zpool feature been imported, allowing better compatibility with pools created under ZFS-on-Linux 0.7.x Many bug fixes have been applied to the device removal feature. This feature allows you to remove a non-redundant or mirror vdev from a pool by relocating its data to other vdevs. Includes the fix for PR 229614 that could cause processes to hang in zil_commit() Boot Loader Changes: The lua loader(8) has been updated to detect a list of installed kernels to boot. The loader(8) has been updated to support geli(8) for all architectures and all disk-like devices. The loader(8) has been updated to add support for loading Intel® microcode updates early during the boot process. Networking: The pf(4) packet filter is now usable within a jail(8) using vnet(9). The pf(4) packet filter has been updated to use rmlock(9) instead of rwlock(9), resulting in significant performance improvements. The SO_REUSEPORT_LB option has been added to the network stack, allowing multiple programs or threads to bind to the same port, and incoming connections load balanced using a hash function. Again, read the release notes for a full list, check out the errata notices. A big THANKS to the entire release engineering team and all developers involved in the release, much appreciated! ###Abandon Linux. Move to FreeBSD or Illumos If you use GNU/Linux and you are only on opensource, you may be doing it wrong. Here’s why. Is your company based on opensource based software only? Do you have a bunch of developers hitting some kind of server you have installed for them to “do their thing”? Being it for economical reasons (remember to donate), being it for philosophycal ones, you may have skipped good alternatives. The BSD’s and Illumos. I bet you are running some sort of Debian, openSuSE or CentOS. It’s very discouraging having entered into the IT field recently and discover many of the people you meet do not even recognise the name BSD. Naming Solaris seems like naming the evil itself. The problem being many do not know why. They can’t point anything specific other than it’s fading out. This has recently shown strong when Oracle officials have stated development for new features has ceased and almost 90 % of developers for Solaris have been layed off. AIX seems alien to almost everybody unless you have a white beard. And all this is silly. And here’s why. You are certainly missing two important features that FreeBSD and Illumos derivatives are enjoying. A full virtualization technology, much better and fully developed compared to the LXC containers in the Linux world, such as Jails on BSD, Zones in Solaris/Illumos, and the great ZFS file system which both share. You have probably heard of a new Linux filesystem named Btrfs, which by the way, development has been dropped from the Red Hat side. Trying to emulate ZFS, Oracle started developing Btrfs file system before they acquired Sun (the original developer of ZFS), and SuSE joined the effort as well as Red Hat. It is not as well developed as ZFS and it hasn’t been tested in production environments as extensively as the former has. That leaves some uncertainty on using it or not. Red Hat leaving it aside does add some more. Although some organizations have used it with various grades of success. But why is this anyhow interesting for a sysadmin or any organization? Well… FreeBSD (descendant of Berkeley UNIX) and SmartOS (based on Illumos) aglutinate some features that make administration easier, safer, faster and more reliable. The dream of any systems administrator. To start, the ZFS filesystem combines the typical filesystem with a volume manager. It includes protection against corruption, snapshots and copy-on-write clones, as well as volume manager. Jails is another interesting piece of technology. Linux folks usually associate this as a sort of chroot. It isn’t. It is somehow inspired by it but as you may know you can escape from a chroot environment with a blink of an eye. Jails are not called jails casually. The name has a purpose. Contain processes and programs within a defined and totally controlled environment. Jails appeared first in FreeBSD in the year 2000. Solaris Zones debuted on 2005 (now called containers) are the now proprietary version of those. There are some other technologies on Linux such as Btrfs or Docker. But they have some caveats. Btrfs hasn’t been fully developed yet and it’s hasn’t been proved as much in production environments as ZFS has. And some problems have arisen recently although the developers are pushing the envelope. At some time they will match ZFS capabilities for sure. Docker is growing exponentially and it’s one of the cool technologies of modern times. The caveat is, as before, the development of this technology hasn’t been fully developed. Unlike other virtualization technologies this is not a kernel playing on top of another kernel. This is virtualization at the OS level, meaning differentiated environments can coexist on a single host, “hitting” the same unique kernel which controls and shares the resources. The problem comes when you put Docker on top of any other virtualization technology such as KVM or Xen. It breaks the purpose of it and has a performance penalty. I have arrived into the IT field with very little knowledge, that is true. But what I see strikes me. Working in a bank has allowed me to see a big production environment that needs the highest of the availability and reliability. This is, sometimes, achieved by bruteforce. And it’s legitime and adequate. Redundancy has a reason and a purpose for example. But some other times it looks, it feels, like killing flies with cannons. More hardware, more virtual machines, more people, more of this, more of that. They can afford it, so they try to maintain the cost low but at the end of the day there is a chunky budget to back operations. But here comes reality. You’re not a bank and you need to squeeze your investment as much as possible. By using FreeBSD jails you can avoid the performance penalty of KVM or Xen virtualization. Do you use VMWare or Hyper-V? You can avoid both and gain in performance. Not only that, control and manageability are equal as before, and sometimes easier to administer. There are four ways to operate them which can be divided in two categories. Hardcore and Human Being. For the Hardcore use the FreeBSD handbook and investigate as much as you can. For the Human Being way there are three options to use. Ezjail, Iocage and CBSD which are frameworks or programs as you may call to manage jails. I personally use Iocage but I have also used Ezjail. How can you use jails on your benefit? Ever tried to configure some new software and failed miserably? You can have three different jails running at the same time with different configurations. Want to try a new configuration in a production piece of hardware without applying it on the final users? You can do that with a small jail while the production environment is on in another bigger, chunkier jail. Want to divide the hardware as a replica of the division of the team/s you are working with? Want to sell virtual machines with bare metal performance? Do you want to isolate some piece of critical software or even data in a more controlled environment? Do you have different clients and you want to use the same hardware but you want to avoid them seeing each other at the same time you maintain performance and reliability? Are you a developer and you have to have reliable and portable snapshots of your work? Do you want to try new options-designs without breaking your previous work, in a timeless fashion? You can work on something, clone the jail and apply the new ideas on the project in a matter of seconds. You can stop there, export the filesystem snapshot containing all the environment and all your work and place it on a thumbdrive to later import it on a big production system. Want to change that image properties such as the network stack interface and ip? This is just one command away from you. But what properties can you assign to a jail and how can I manage them you may be wondering. Hostname, disk quota, i/o, memory, cpu limits, network isolation, network virtualization, snapshots and the manage of those, migration and root privilege isolation to name a few. You can also clone them and import and export them between different systems. Some of these things because of ZFS. Iocage is a python program to manage jails and it takes profit from ZFS advantages. But FreeBSD is not Linux you may say. No it is not. There are no run levels. The systemd factor is out of this equation. This is so since the begginning. Ever wondered where did vi come from? The TCP/IP stack? Your beloved macOS from Apple? All this is coming from the FreeBSD project. If you are used to Linux your adaptation period with any BSD will be short, very short. You will almost feel at home. Used to packaged software using yum or apt-get? No worries. With pkgng, the package management tool used in FreeBSD has almost 27.000 compiled packages for you to use. Almost all software found on any of the important GNU/Linux distros can be found here. Java, Python, C, C++, Clang, GCC, Javascript frameworks, Ruby, PHP, MySQL and the major forks, etc. All this opensource software, and much more, is available at your fingertips. I am a developer and… frankly my time is money and I appreciate both much more than dealing with systems configuration, etc. You can set a VM using VMWare or VirtualBox and play with barebones FreeBSD or you can use TrueOS (a derivative) which comes in a server version and a desktop oriented one. The latter will be easier for you to play with. You may be doing this already with Linux. There is a third and very sensible option. FreeNAS, developed by iXSystems. It is FreeBSD based and offers all these technologies with a GUI. VMWare, Hyper-V? Nowadays you can get your hands off the CLI and get a decent, usable, nice GUI. You say you play on the cloud. The major players already include FreeBSD in their offerings. You can find it in Amazon AWS or Azure (with official Microsoft support contracts too!). You can also find it in DigitalOcean and other hosting providers. There is no excuse. You can use it at home, at the office, with old or new hardware and in the cloud as well. You can even pay for a support contract to use it. Joyent, the developers of SmartOS have their own cloud with different locations around the globe. Have a look on them too. If you want the original of ZFS and zones you may think of Solaris. But it’s fading away. But it really isn’t. When Oracle bouth Sun many people ran away in an stampide fashion. Some of the good folks working at Sun founded new projects. One of these is Illumos. Joyent is a company formed by people who developed these technologies. They are a cloud operator, have been recently bought by Samsung and have a very competent team of people providing great tech solutions. They have developed an OS, called SmartOS (based on Illumos) with all these features. The source from this goes back to the early days of UNIX. Do you remember the days of OpenSolaris when Sun opensourced the crown jewels? There you have it. A modern opensource UNIX operating system with the roots in their original place and the head planted on today’s needs. In conclusion. If you are on GNU/Linux and you only use opensource software you may be doing it wrong. And missing goodies you may need and like. Once you put your hands on them, trust me, you won’t look back. And if you have some “old fashioned” admins who know Solaris, you can bring them to a new profitable and exciting life with both systems. Still not convinced? Would you have ever imagined Microsoft supporting Linux? Even loving it? They do love now FreeBSD. And not only that, they provide their own image in the Azure Cloud and you can get Microsoft support, payed support if you want to use the platform on Azure. Ain’t it… surprising? Convincing at all? PS: I haven’t mentioned both softwares, FreeBSD and SmartOS do have a Linux translation layer. This means you can run Linux binaries on them and the program won’t cough at all. Since the ABI stays stable the only thing you need to run a Linux binary is a translation between the different system calls and the libraries. Remember POSIX? Choose your poison and enjoy it. ###A partly-cloudy IPsec VPN Audience I’m assuming that readers have at least a basic knowledge of TCP/IP networking and some UNIX or UNIX-like systems, but not necessarily OpenBSD or FreeBSD. This post will therefore be light on details that aren’t OS specific and are likely to be encountered in normal use (e.g., how to use vi or another text editor.) For more information on these topics, read Absolute FreeBSD (3ed.) by Michael W. Lucas. Overview I’m redoing my DigitalOcean virtual machines (which they call droplets). My requirements are: VPN Road-warrior access, so I can use private network resources from anywhere. A site-to-site VPN, extending my home network to my VPSes. Hosting for public and private network services. A proxy service to provide a public IP address to services hosted at home. The last item is on the list because I don’t actually have a public IP address at home; my firewall’s external address is in the RFC 1918 space, and the entire apartment building shares a single public IPv4 address.1 (IPv6? Don’t I wish.) The end-state network will include one OpenBSD droplet providing firewall, router, and VPN services; and one FreeBSD droplet hosting multiple jailed services. I’ll be providing access via these droplets to a NextCloud instance at home. A simple NAT on the DO router droplet isn’t going to work, because packets going from home to the internet would exit through the apartment building’s connection and not through the VPN. It’s possible that I could do work around this issue with packet tagging using the pf firewall, but HAProxy is simple to configure and unlikely to result in hard-to-debug problems. relayd is also an option, but doesn’t have the TLS parsing abilities of HAProxy, which I’ll be using later on. Since this system includes jails running on a VPS, and they’ve got RFC 1918 addresses, I want them reachable from my home network. Once that’s done, I can access the private address space from anywhere through a VPN connection to the cloudy router. The VPN itself will be of the IPsec variety. IPsec is the traditional enterprise VPN standard, and is even used for classified applications, but has a (somewhat-deserved) reputation for complexity, but recent versions of OpenBSD turn down the difficulty by quite a bit. The end-state network should look like: https://d33wubrfki0l68.cloudfront.net/0ccf46fb057e0d50923209bb2e2af0122637e72d/e714e/201812-cloudy/endstate.svg This VPN both separates internal network traffic from public traffic and uses encryption to prevent interception or tampering. Once traffic has been encrypted, decrypting it without the key would, as Bruce Schneier once put it, require a computer built from something other than matter that occupies something other than space. Dyson spheres and a frakton of causality violation would possibly work, as would mathemagical technology that alters the local calendar such that P=NP.2 Black-bag jobs and/or suborning cloud provider employees doesn’t quite have that guarantee of impossibility, however. If you have serious security requirements, you’ll need to do better than a random blog entry. ##News Roundup KLEAK: Practical Kernel Memory Disclosure Detection Modern operating systems such as NetBSD, macOS, and Windows isolate their kernel from userspace programs to increase fault tolerance and to protect against malicious manipulations [10]. User space programs have to call into the kernel to request resources, via system calls or ioctls. This communication between user space and kernel space crosses a security boundary. Kernel memory disclosures - also known as kernel information leaks - denote the inadvertent copying of uninitialized bytes from kernel space to user space. Such disclosed memory may contain cryptographic keys, information about the kernel memory layout, or other forms of secret data. Even though kernel memory disclosures do not allow direct exploitation of a system, they lay the ground for it. We introduce KLEAK, a simple approach to dynamically detect kernel information leaks. Simply said, KLEAK utilizes a rudimentary form of taint tracking: it taints kernel memory with marker values, lets the data travel through the kernel and scans the buffers exchanged between the kernel and the user space for these marker values. By using compiler instrumentation and rotating the markers at regular intervals, KLEAK significantly reduces the number of false positives, and is able to yield relevant results with little effort. Our approach is practically feasible as we prove with an implementation for the NetBSD kernel. A small performance penalty is introduced, but the system remains usable. In addition to implementing KLEAK in the NetBSD kernel, we applied our approach to FreeBSD 11.2. In total, we detected 21 previously unknown kernel memory disclosures in NetBSD-current and FreeBSD 11.2, which were fixed subsequently. As a follow-up, the projects’ developers manually audited related kernel areas and identified dozens of other kernel memory disclosures. The remainder of this paper is structured as follows. Section II discusses the bug class of kernel memory disclosures. Section III presents KLEAK to dynamically detect instances of this bug class. Section IV discusses the results of applying KLEAK to NetBSD-current and FreeBSD 11.2. Section V reviews prior research. Finally, Section VI concludes this paper. ###How To Create Official Synth Repo System Environment Make sure /usr/dports is updated and that it contains no cruft (git pull; git status). Remove any cruft. Make sure your ‘synth’ is up-to-date ‘pkg upgrade synth’. If you already updated your system you may have to build synth from scratch, from /usr/dports/ports-mgmt/synth. Make sure /etc/make.conf is clean. Update /usr/src to the current master, make sure there is no cruft in it Do a full buildworld, buildkernel, installkernel and installworld Reboot After the reboot, before proceeding, run ‘uname -a’ and make sure you are now on the desired release or development kernel. Synth Environment /usr/local/etc/synth/ contains the synth configuration. It should contain a synth.ini file (you may have to rename the template), and you will have to create or edit a LiveSystem-make.conf file. System requirements are hefty. Just linking chromium alone eats at least 30GB, for example. Concurrent c++ compiles can eat up to 2GB per process. We recommend at least 100GB of SSD based swap space and 300GB of free space on the filesystem. synth.ini should contain this. Plus modify the builders and jobs to suit your system. With 128G of ram, 30/30 or 40/25 works well. If you have 32G of ram, maybe 8/8 or less. ; Take care when hand editing! [Global Configuration] profileselected= LiveSystem [LiveSystem] Operatingsystem= DragonFly Directorypackages= /build/synth/livepackages Directoryrepository= /build/synth/livepackages/All Directoryportsdir= /build/synth/dports Directoryoptions= /build/synth/options Directorydistfiles= /usr/distfiles Directorybuildbase= /build/synth/build Directorylogs= /build/synth/logs Directoryccache= disabled Directorysystem= / Numberofbuilders= 30 Maxjobsperbuilder= 30 Tmpfsworkdir= true Tmpfslocalbase= true Displaywithncurses= true leverageprebuilt= false LiveSystem-make.conf should contain one line to restrict licensing to only what is allowed to be built as a binary package: LICENSESACCEPTED= NONE Make sure there is no other cruft in /usr/local/etc/synth/ In the example above, the synth working dirs are in “/build/synth”. Make sure the base directories exist. Clean out any cruft for a fresh build from-scratch: rm -rf /build/synth/livepackages/* rm -rf /build/synth/logs mkdir /build/synth/logs Run synth everything. I recommend doing this in a ‘screen’ session in case you lose your ssh session (assuming you are ssh’d into the build machine). (optionally start a screen session) synth everything A full synth build takes over 24 hours to run on a 48-core box, around 12 hours to run on a 64-core box. On a 4-core/8-thread box it will take at least 3 days. There will be times when swap space is heavily used. If you have not run synth before, monitor your memory and swap loads to make sure you have configured the jobs properly. If you are overloading the system, you may have to ^C the synth run, reduce the jobs, and start it again. It will pick up where it left off. When synth finishes, let it rebuild the database. You then have a working binary repo. It is usually a good idea to run synth several times to pick up any stuff it couldn’t build the first time. Each of these incremental runs may take a few hours, depending on what it tries to build. ###Interview with founder and maintainer of GhostBSD, Eric Turgeon Thanks you Eric for taking part. To start off, could you tell us a little about yourself, just a bit of background? How did you become interested in open source? When and how did you get interested in the BSD operating systems? On your Twitter profile, you state that you are an automation engineer at iXsystems. Can you share what you do in your day-to-day job? You are the founder and project lead of GhostBSD. Could you describe GhostBSD to those who have never used it or never heard of it? Developing an operating system is not a small thing. What made you decide to start the GhostBSD project and not join another “desktop FreeBSD” related project, such as PC-BSD and DesktopBSD at the time? How did you get to the name GhostBSD? Did you consider any other names? You recently released GhostBSD 18.10? What’s new in that version and what are the key features? What has changed since GhostBSD 11.1? The current version is 18.10. Will the next version be 19.04 (like Ubuntu’s version numbering), or is a new version released after the next stable TrueOS release Can you tell us something about the development team? Is it yourself, or are there other core team members? I think I saw two other developers on your Github project page. How about the relationship with the community? Is it possible for a community member to contribute, and how are those contributions handled? What was the biggest challenge during development? If you had to pick one feature readers should check out in GhostBSD, what is it and why? What is the relationship between iXsystems and the GhostBSD project? Or is GhostBSD a hobby project that you run separately from your work at iXsystems? What is the relationship between GhostBSD and TrueOS? Is GhostBSD TrueOS with the MATE desktop on top, or are there other modifications, additions, and differences? Where does GhostBSD go from here? What are your plans for 2019? Is there anything else that wasn’t asked or that you want to share? ##Beastie Bits dialog(1) script to select audio output on FreeBSD Erlang otp on OpenBSD Capsicum https://blog.grem.de/sysadmin/FreeBSD-On-rpi3-With-crochet-2018-10-27-18-00.html Introduction to µUBSan - a clean-room reimplementation of the Undefined Behavior Sanitizer runtime pkgsrcCon 2018 in Berlin - Videos Getting started with drm-kmod ##Feedback/Questions Malcolm - Show segment idea Fraser - Question: FreeBSD official binary package options Harri - BSD Magazine Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
OpenBSD 6.4 released, GhostBSD RC2 released, MeetBSD - the ultimate hallway track, DragonflyBSD desktop on a Thinkpad, Porting keybase to NetBSD, OpenSSH 7.9, and draft-ietf-6man-ipv6only-flag in FreeBSD. ##Headlines OpenBSD 6.4 released See a detailed log of changes between the 6.3 and 6.4 releases. See the information on the FTP page for a list of mirror machines. Have a look at the 6.4 errata page for a list of bugs and workarounds. signify(1) pubkeys for this release: base: RWQq6XmS4eDAcQW4KsT5Ka0KwTQp2JMOP9V/DR4HTVOL5Bc0D7LeuPwA fw: RWRoBbjnosJ/39llpve1XaNIrrQND4knG+jSBeIUYU8x4WNkxz6a2K97 pkg: RWRF5TTY+LoN/51QD5kM2hKDtMTzycQBBPmPYhyQEb1+4pff/H6fh/kA ###GhostBSD 18.10 RC2 Announced This second release candidate of GhostBSD 18.10 is the second official release of GhostBSD with TrueOS under the hood. The official desktop of GhostBSD is MATE. However, in the future, there might be an XFCE community release, but for now, there is no community release yet. What has changed since RC1 Removed drm-stable-kmod and we will let users installed the propper drm-*-kmod Douglas Joachin added libva-intel-driver libva-vdpau-driver to supports accelerated some video driver for Intel Issues that got fixed Bug #70 Cannot run Octopi, missing libgksu error. Bug #71 LibreOffice doesn’t start because of missing libcurl.so.4 Bug #72 libarchive is a missing dependency Again thanks to iXsystems, TrueOS, Joe Maloney, Kris Moore, Ken Moore, Martin Wilke, Neville Goddard, Vester “Vic” Thacker, Douglas Joachim, Alex Lyakhov, Yetkin Degirmenci and many more who helped to make the transition from FreeBSD to TrueOS smoother. Updating from RC1 to RC2: sudo pkg update -f sudo pkg install -f libarchive curl libgksu sudo pkg upgrade Where to download: All images checksum, hybrid ISO(DVD, USB) and torrent are available here: https://www.ghostbsd.org/download [ScreenShots] https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-2013-22-41.png https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-20_13-27-26.png ###OpenSSH 7.9 has been released and it has support for OpenSSL 1.1 Changes since OpenSSH 7.8 This is primarily a bugfix release. New Features ssh(1), sshd(8): allow most port numbers to be specified using service names from getservbyname(3) (typically /etc/services). ssh(1): allow the IdentityAgent configuration directive to accept environment variable names. This supports the use of multiple agent sockets without needing to use fixed paths. sshd(8): support signalling sessions via the SSH protocol. A limited subset of signals is supported and only for login or command sessions (i.e. not subsystems) that were not subject to a forced command via authorizedkeys or sshdconfig. bz#1424 ssh(1): support "ssh -Q sig" to list supported signature options. Also "ssh -Q help" to show the full set of supported queries. ssh(1), sshd(8): add a CASignatureAlgorithms option for the client and server configs to allow control over which signature formats are allowed for CAs to sign certificates. For example, this allows banning CAs that sign certificates using the RSA-SHA1 signature algorithm. sshd(8), ssh-keygen(1): allow key revocation lists (KRLs) to revoke keys specified by SHA256 hash. ssh-keygen(1): allow creation of key revocation lists directly from base64-encoded SHA256 fingerprints. This supports revoking keys using only the information contained in sshd(8) authentication log messages. Bugfixes ssh(1), ssh-keygen(1): avoid spurious "invalid format" errors when attempting to load PEM private keys while using an incorrect passphrase. bz#2901 sshd(8): when a channel closed message is received from a client, close the stderr file descriptor at the same time stdout is closed. This avoids stuck processes if they were waiting for stderr to close and were insensitive to stdin/out closing. bz#2863 ssh(1): allow ForwardX11Timeout=0 to disable the untrusted X11 forwarding timeout and support X11 forwarding indefinitely. Previously the behaviour of ForwardX11Timeout=0 was undefined. sshd(8): when compiled with GSSAPI support, cache supported method OIDs regardless of whether GSSAPI authentication is enabled in the main section of sshd_config. This avoids sandbox violations if GSSAPI authentication was later enabled in a Match block. bz#2107 sshd(8): do not fail closed when configured with a text key revocation list that contains a too-short key. bz#2897 ssh(1): treat connections with ProxyJump specified the same as ones with a ProxyCommand set with regards to hostname canonicalisation (i.e. don't try to canonicalise the hostname unless CanonicalizeHostname is set to 'always'). bz#2896 ssh(1): fix regression in OpenSSH 7.8 that could prevent public- key authentication using certificates hosted in a ssh-agent(1) or against sshd(8) from OpenSSH
We have a long interview with fiction and non-fiction author Michael W. Lucas for you this week as well as questions from the audience. ##Headlines Interview - Michael W. Lucas - mwlucas@michaelwlucas.com / @mwlauthor BR: [Welcome Back] AJ: What have you been doing since last we talked to you [ed, ssh, and af3e] BR: Tell us more about AF3e AJ: How did the first Absolute FreeBSD come about? BR: Do you have anything special planned for MeetBSD? AJ: What are you working on now? [FM:Jails, Git sync Murder] BR: What are your plans for next year? AJ: How has SEMIBug been going? Auction at https://mwl.io Patreon Link: ##Feedback/Questions Paul - Recent bhyve related videos (daemon) Michael - freebsd-update question Sigflup - pkg file search Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
We have a long interview with fiction and non-fiction author Michael W. Lucas for you this week as well as questions from the audience.
We have a long interview with fiction and non-fiction author Michael W. Lucas for you this week as well as questions from the audience.
Trip reports from the Essen Hackathon and BSDCam, CfT: ZFS native encryption and UFS trim consolidation, ZFS performance benchmarks on a FreeBSD server, how to port your OS to EC2, Vint Cerf about traceability, Remote Access console to an RPi3 running FreeBSD, and more. ##Headlines Essen Hackathon & BSDCam 2018 trip report Allan and Benedict met at FRA airport and then headed to the Air Rail terminal for our train to Essen where the Hackathon would happen over the weekend of Aug 10 - 12, 2018. Once there, we did not have to wait long until other early-arrivals would show up and soon we had about 10 people gathered for lunch. After buying some take-out pizzas and bringing it back to the Linuxhotel (there was a training still going on there so we could not get into our rooms yet), we sat in the sunny park and talked. More and more people arrived and soon, people started hacking on their laptops. Some people would not arrive until a few hours before midnight, but we already had a record appearance of 20 people in total. On Saturday, we gathered everyone in one of the seminar rooms that had rooms and chairs for us. After some organizational infos, we did an introductory round and Benedict wrote down on the whiteboard what people were interested in. It was not long until groups formed to talk about SSL in base, weird ZFS scrubs that would go over 100% completion (fixed now). Other people started working on ports, fixing bugs, or wrote documentation. The day ended in a BBQ in the Linuxhotel park, which was well received by everyone. On Sunday, after attendees packed up their luggage and stored it in the seminar room, we continued hacking until lunchtime. After a quick group picture, we headed to a local restaurant for the social event (which was not open on Saturday, otherwise we would have had it then). In the afternoon, most people departed, a good half of them were heading for BSDCam. Commits from the hackathon (the ones from 2018) Overall, the hackathon was well received by attendees and a lot of them liked the fact that it was close to another BSD gathering so they could nicely combine the two. Also, people thought about doing their own hackathon in the future, which is an exciting prospect. Thanks to all who attended, helped out here and there when needed. Special Thanks to Netzkommune GmbH for sponsoring the social event and the Linuxhotel for having us. Benedict was having a regular work day on Monday after coming back from the hackathon, but flew out to Heathrow on Tuesday. Allan was in London a day earlier and arrived a couple of hours before Benedict in Cambridge. He headed for the Computer Lab even though the main event would not start until Wednesday. Most people gathered at the Maypole pub on Tuesday evening for welcomes, food and drinks. On Wednesday, a lot of people met in the breakfast room of Churchill College where most people were staying and went to the Computer Lab, which served as the main venue for BSDCam, together. The morning was spend with introductions and collecting what most people were interested in talking. This unconference style has worked well in the past and soon we had 10 main sessions together for the rest of this and the following two days (full schedule). Most sessions took notes, which you can find on the FreeBSD wiki. On Thursday evening, we had a nice formal dinner at Trinity Hall. BSDCam 2018 was a great success with a lot of fruitful discussions and planning sessions. We thank the organizers for BSDCam for making it happen. A special mentions goes out to Robert Watson and his family. Even though he was not there, he had a good reason to miss it: they had their first child born at the beginning of the week. Congratulations and best wishes to all three of them! ###Call for Testing: ZFS Native Encryption for FreeBSD A port of the ZoL (ZFS-on-Linux) feature that provides native crypto support for ZFS is ready for testing on FreeBSD Most of the porting was done by sef@freebsd.org (Sean Eric Fagan) The original ZoL commit is here: https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 For an overview, see Tom Caputi’s presentation from the OpenZFS Developers Summit in 2016 Video: https://youtu.be/frnLiXclAMo Slides: https://drive.google.com/file/d/0B5hUzsxe4cdmU3ZTRXNxa2JIaDQ/view?usp=sharing WARNING: test in VMs or with spare disks etc, pools created with this code, or upgraded to this version, will no longer be importable on systems that do not support this feature. The on-disk format or other things may change before the final version, so you will likely have to ‘zfs send | zfs recv’ the data on to a new pool Thanks for testing to help this feature land in FreeBSD iXsystems ###Call for Testing: UFS TRIM Consolidation Kirk Mckusick posts to the FreeBSD mailing list looking for testers for the new UFS TRIM Consolidation code When deleting files on filesystems that are stored on flash-memory (solid-state) disk drives, the filesystem notifies the underlying disk of the blocks that it is no longer using. The notification allows the drive to avoid saving these blocks when it needs to flash (zero out) one of its flash pages. These notifications of no-longer-being-used blocks are referred to as TRIM notifications. In FreeBSD these TRIM notifications are sent from the filesystem to the drive using the BIODELETE command. Until now, the filesystem would send a separate message to the drive for each block of the file that was deleted. Each Gigabyte of file size resulted in over 3000 TRIM messages being sent to the drive. This burst of messages can overwhelm the drive’s task queue causing multiple second delays for read and write requests. This implementation collects runs of contiguous blocks in the file and then consolodates them into a single BIODELETE command to the drive. The BIODELETE command describes the run of blocks as a single large block being deleted. Each Gigabyte of file size can result in as few as two BIODELETE commands and is typically less than ten. Though these larger BIODELETE commands take longer to run, they do not clog the drive task queue, so read and write commands can intersperse effectively with them. Though this new feature has been throughly reviewed and tested, it is being added disabled by default so as to minimize the possibility of disrupting the upcoming 12.0 release. It can be enabled by running `sysctl vfs.ffs.dotrimcons=1’’. Users are encouraged to test it. If no problems arise, we will consider requesting that it be enabled by default for 12.0. This support is off by default, but I am hoping that I can get enough testing to ensure that it (a) works, and (b) is helpful that it will be reasonable to have it turned on by default in 12.0. The cutoff for turning it on by default in 12.0 is September 19th. So I am requesting your testing feedback in the near-term. Please let me know if you have managed to use it successfully (or not) and also if it provided any performance difference (good or bad). To enable TRIM consolidation usesysctl vfs.ffs.dotrimcons=1’ There is also a diff that adds additional statistics: https://lists.freebsd.org/pipermail/freebsd-current/2018-August/070798.html You can also watch the volume and latency of BIODELETE commands by running gstat with the -d flag ##News Roundup ZFS performance Aravindh Sampathkumar, a Performance Engineer and Sysadmin posts some simple benchmarks he did on a new ZFS server This is NOT an all-in post about ZFS performance. I built a FreeBSD+ZFS file server recently at work to serve as an offsite backup server. I wanted to run a few synthetic workloads on it and look at how it fares from performance perspective. Mostly for curiosity and learning purposes. As stated in the notes about building this server, performance was not one of the priorities, as this server will never face our active workload. What I care about from this server is its ability to work with rsync and keep the data synchronised with our primary storage server. With that context, I ran a few write tests to see how good our solution is and what to expect from it in terms of performance. The article then uses FIO to do some benchmarks. As the author did, make sure you match the FIO block size to the ZFS record size to avoid write amplification. Either tune FIO or adjust the recordsize property in ZFS You also want to consider compression and cache effects Write Performance: Incompressible: 1600-2600 MB/s, Compressible: 2500-6600 MB/s Another over 1200 MB/s is enough to keep your 10 gigabit network saturated The increased latency that is seen with higher number of writers working, may be the result of the ZFS backpressure system (the write throttle). There is some tuning that can be done there. Specifically, since this machine has 768 GB of ram, you might allow more than 4GB of dirty data, which would mean you’d be able to write larger batches and not have to push back while you wait for a transaction group to flush when dealing with gigabytes/sec of writes ###How to port your OS to EC2 Colin Percival reflects on his FreeBSD on EC2 maintainership efforts in his blog: I’ve been the maintainer of the FreeBSD/EC2 platform for about 7.5 years now, and as far as “running things in virtual machines” goes, that remains the only operating system and the only cloud which I work on. That said, from time to time I get questions from people who want to port other operating systems into EC2, and being a member of the open source community, I do my best to help them. I realized a few days ago that rather than replying to emails one by one it would be more efficient to post something publicly; so — for the benefit of the dozen or so people who want to port operating systems to run in EC2, and the curiosity of maybe a thousand more people who use EC2 but will never build AMIs themselves — here’s a rough guide to building EC2 images. Before we can talk about building images, there are some things you need: Your OS needs to run on x86 hardware. 64-bit (“amd64”, “x86-64”) is ideal, but I’ve managed to run 32-bit FreeBSD on “64-bit” EC2 instances so at least in some cases that’s not strictly necessary. You almost certainly want to have drivers for Xen block devices (for all of the pre-Nitro EC2 instances) or for NVMe disks (for the most recent EC2 instances). Theoretically you could make do without these since there’s some ATA emulation available for bootstrapping, but if you want to do any disk I/O after the kernel finishes booting you’ll want to have a disk driver. Similarly, you need support for the Xen network interface (older instances), Intel 10 GbE SR-IOV networking (some newer but pre-Nitro instances), or Amazon’s “ENA” network adapters (on Nitro instances), unless you plan on having instances which don’t communicate over the network. The ENA driver is probably the hardest thing to port, since as far as I know there’s no way to get your hands on the hardware directly, and it’s very difficult to do any debugging in EC2 without having a working network. Finally, the obvious: You need to have an AWS account, and appropriate API access keys. Building a disk image Building an AMI I wrote a simple tool for converting disk images into EC2 instances: bsdec2-image-upload. It uploads a disk image to Amazon S3; makes an API call to import that disk image into an EBS volume; creates a snapshot of that volume; then registers an EC2 AMI using that snapshot. To use bsdec2-image-upload, you’ll first need to create an S3 bucket for it to use as a staging area. You can call it anything you like, but I recommend that you Create it in a “nearby” region (for performance reasons), and Set an S3 “lifecycle policy” which deletes objects automatically after 1 day (since bsdec2-image-upload doesn’t clean up the S3 bucket, and those objects are useless once you’ve finished creating an AMI). Boot configuration Odds are that your instance started booting and got as far as the boot loader launching the kernel, but at some point after that things went sideways. Now we start the iterative process of building disk images, turning them into AMIs, launching said AMIs, and seeing where they break. Some things you’ll probably run into here: EC2 instances have two types of console available to them: A serial console and an VGA console. (Or rather, emulated serial and emulated VGA.) If you can have your kernel output go to both consoles, I recommend doing that. If you have to pick one, the serial console (which shows up as the “System Log” in EC2) is probably more useful than the VGA console (which shows up as “instance screenshot”) since it lets you see more than one screen of logs at once; but there’s a catch: Due to some bizarre breakage in EC2 — which I’ve been complaining about for ten years — the serial console is very “laggy”. If you find that you’re not getting any output, wait five minutes and try again. You may need to tell your kernel where to find the root filesystem. On FreeBSD we build our disk images using GPT labels, so we simply need to specify in /etc/fstab that the root filesystem is on /dev/gpt/rootfs; but if you can’t do this, you’ll probably need to have different AMIs for Nitro instances vs. non-Nitro instances since Xen block devices will typically show up with different device names from NVMe disks. On FreeBSD, I also needed to set the vfs.root.mountfrom kernel environment variable for a while; this also is no longer needed on FreeBSD but something similar may be needed on other systems. You’ll need to enable networking, using DHCP. On FreeBSD, this means placing ifconfigDEFAULT=“SYNCDHCP” into /etc/rc.conf; other systems will have other ways of specifying network parameters, and it may be necessary to specify a setting for the Xen network device, Intel SR-IOV network, and the Amazon ENA interface so that you’ll have the necessary configuration across all EC2 instance types. (On FreeBSD, ifconfigDEFAULT takes care of specifying the network settings which should apply for whatever network interface the kernel finds at boot time.) You’ll almost certainly want to turn on SSH, so that you can connect into newly launched instances and make use of them. Don’t worry about setting a password or creating a user to SSH into yet — we’ll take care of that later. EC2 configuration Now it’s time to make the AMI behave like an EC2 instance. To this end, I prepared a set of rc.d scripts for FreeBSD. Most importantly, they Print the SSH host keys to the console, so that you can veriy that they are correct when you first SSH in. (Remember, Verifying SSH host keys is more important than flossing every day.) Download the SSH public key you want to use for logging in, and create an account (by default, “ec2-user”) with that key set up for you. Fetch EC2 user-data and process it via configinit to allow you to configure the system as part of the process of launching it. If your OS has an rc system derived from NetBSD’s rc.d, you may be able to use these scripts without any changes by simply installing them and enabling them in /etc/rc.conf; otherwise you may need to write your own scripts using mine as a model. Firstboot scripts A feature I added to FreeBSD a few years ago is the concept of “firstboot” scripts: These startup scripts are only run the first time a system boots. The aforementioned configinit and SSH key fetching scripts are flagged this way — so if your OS doesn’t support the “firstboot” keyword on rc.d scripts you’ll need to hack around that — but EC2 instances also ship with other scripts set to run on the first boot: FreeBSD Update will fetch and install security and critical errata updates, and then reboot the system if necessary. The UFS filesystem on the “boot disk” will be automatically expanded to the full size of the disk — this makes it possible to specify a larger size of disk at EC2 instance launch time. Third-party packages will be automatically fetched and installed, according to a list in /etc/rc.conf. This is most useful if configinit is used to edit /etc/rc.conf, since it allows you to specify packages to install via the EC2 user-data. While none of these are strictly necessary, I find them to be extremely useful and highly recommend implementing similar functionality in your systems. Support my work! I hope you find this useful, or at very least interesting. Please consider supporting my work in this area; while I’m happy to contribute my time to supporting open source software, it would be nice if I had money coming in which I could use to cover incidental expenses (e.g., conference travel) so that I didn’t end up paying to contribute to FreeBSD. Digital Ocean https://do.co/bsdnow ###Traceability, by Vint Cerf A recent article from the August issue of the Communications of the ACM, for your contemplation: At a recent workshop on cybersecurity in the U.K., a primary topic of consideration was how to preserve the freedom and openness of the Internet while protecting against the harmful behaviors that have emerged in this global medium. That this is a significant challenge cannot be overstated. The bad behaviors range from social network bullying and misinformation to email spam, distributed denial of service attacks, direct cyberattacks against infrastructure, malware propagation, identity theft, and a host of other ills requiring a wide range of technical and legal considerations. That these harmful behaviors can and do cross international boundaries only makes it more difficult to fashion effective responses. In other columns, I have argued for better software development tools to reduce the common mistakes that lead to vulnerabilities that are exploited. Here, I want to focus on another aspect of response related to law enforcement and tracking down perpetrators. Of course, not all harms are (or perhaps are not yet) illegal, but discovering those who cause them may still be warranted. The recent adoption and implementation of the General Data Protection Regulation (GDPR) in the European Union creates an interesting tension because it highlights the importance and value of privacy while those who do direct or indirect harm must be tracked down and their identities discovered. In passing, I mention that cryptography has sometimes been blamed for protecting the identity or actions of criminals but it is also a tool for protecting privacy. Arguments have been made for “back doors” to cryptographic systems but I am of the opinion that such proposals carry extremely high risk to privacy and safety. It is not my intent to argue this question in this column. What is of interest to me is a concept to which I was introduced at the Ditchley workshop, specifically, differential traceability. The ability to trace bad actors to bring them to justice seems to me an important goal in a civilized society. The tension with privacy protection leads to the idea that only under appropriate conditions can privacy be violated. By way of example, consider license plates on cars. They are usually arbitrary identifiers and special authority is needed to match them with the car owners (unless, of course, they are vanity plates like mine: “Cerfsup”). This is an example of differential traceability; the police department has the authority to demand ownership information from the Department of Motor Vehicles that issues the license plates. Ordinary citizens do not have this authority. In the Internet environment there are a variety of identifiers associated with users (including corporate users). Domain names, IP addresses, email addresses, and public cryptography keys are examples among many others. Some of these identifiers are dynamic and thus ambiguous. For example, IP addresses are not always permanent and may change (for example, temporary IP addresses assigned at Wi-Fi hotspots) or may be ambiguous in the case of Network Address Translation. Information about the time of assignment and the party to whom an IP address was assigned may be needed to identify an individual user. There has been considerable debate and even a recent court case regarding requirements to register users in domain name WHOIS databases in the context of the adoption of GDPR. If we are to accomplish the simultaneous objectives of protecting privacy while apprehending those engaged in harmful or criminal behavior on the Internet, we must find some balance between conflicting but desirable outcomes. This suggests to me that the notion of traceability under (internationally?) agreed circumstances (that is, differential traceability) might be a fruitful concept to explore. In most societies today, it is accepted that we must be identifiable to appropriate authorities under certain conditions (consider border crossings, traffic violation stops as examples). While there are conditions under which apparent anonymity is desirable and even justifiable (whistle-blowing, for example) absolute anonymity is actually quite difficult to achieve (another point made at the Ditchley workshop) and might not be absolutely desirable given the misbehaviors apparent anonymity invites. I expect this is a controversial conclusion and I look forward to subsequent discussion. ###Remote Access Console using FreeBSD on an RPi3 Our friend, and FOSDEM Booth Neighbour, Jorge, has posted a tutorial on how he created a remote access console for his SmartOS server and other machines in his homelab Parts: Raspberry Pi 3 B+ NavoLabs micro POE Hat FT4232H based USB-to-RS232 (4x) adapter Official Raspberry Pi case (optional) Heat-sink kit (optional) USB-to-TTL adaptor (optional) Sandisk 16Gb microSD For the software I ended up using conserver. Below is a very brief tutorial on how to set everything up. I assume you have basic unix skills. Get an RPi3 image, make some minor modifications for RPi3+, and write it to the USB stick Configure FreeBSD on the RPi3 Load the ‘muge’ Ethernet Driver Load USB serial support Load the FTDI driver Enable SSHd and Conserver Configure Conserver Setup log rotation Start Conserver And you’re good to go A small bonus script I wrote to turn on the 2nd LED on the rPI once the system is booted, it will then blink the LED if someone is connected to any of the consoles. There is also a followup post with some additional tips: https://blackdot.be/2018/08/freebsd-uart-and-raspberry-pi-3-b/ ##Beastie Bits Annual Penguin Races Mscgen - Message Sequence Chart generator This patch makes FreeBSD boot 500 - 800ms faster, please test on your hardware FreeBSD’s arc4random() replaced with OpenBSD ChaCha20 implementation MeetBSD Devsummit open for registrations New Podcast interview with Michael W. Lucas Tarsnap ##Feedback/Questions We need more feedback emails. Please write to feedback@bsdnow.tv Additionally, we are considering a new segment to be added to the end of the show (to make it skippable), where we have a ~15 minute deep dive on a topic. Some initial ideas are on the Virtual Memory subsystem, the Scheduler, Capsicum, and GEOM. What topics would you like to get very detailed explanations of? Many of the explanations may have accompanying graphics, and not be very suitable for audio only listeners, that is why we are planning to put it at the very end of the episode. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
DragonflyBSD’s hammer1 encrypted master/slave setup, second part of our BSDCan recap, NomadBSD 1.1-RC1 available, OpenBSD adds an LDAP client to base, FreeBSD gets pNFS support, Intel FPU Speculation Vulnerability confirmed, and what some Unix command names mean. ##Headlines DragonflyBSD: Towards a HAMMER1 master/slave encrypted setup with LUKS I just wanted to share my experience with setting up DragonFly master/slave HAMMER1 PFS’s on top of LUKS So after a long time using an Synology for my NFS needs, I decided it was time to rethink my setup a little since I had several issues with it : You cannot run NFS on top of encrypted partitions easily I suspect I am having some some data corruption (bitrot) on the ext4 filesystem the NIC was stcuk to 100 Mbps instead of 1 Gbps even after swapping cables, switches, you name it It’s proprietary I have been playing with DragonFly in the past and knew about HAMMER, now I just had the perfect excuse to actually use it in production :) After setting up the OS, creating the LUKS partition and HAMMER FS was easy : kdload dm cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox newfs_hammer -L hammer1_secure_master /dev/mapper/fort_knox cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox_slave newfs_hammer -L hammer1_secure_slave /dev/mapper/fort_knox_slave Mount the 2 drives : mount /dev/mapper/fort_knox /fort_knox mount /dev/mapper_fort_know_slave /fort_knox_slave You can now put your data under /fort_knox Now, off to setting up the replication, first get the shared-uuid of /fort_knox hammer pfs-status /fort_knox Create a PFS slave “linked” to the master hammer pfs-slave /fort_knox_slave/pfs/slave shared-uuid=f9e7cc0d-eb59-10e3-a5b5-01e6e7cefc12 And then stream your data to the slave PFS ! hammer mirror-stream /fort_knox /fort_knox_slave/pfs/slave After that, setting NFS is fairly trivial even though I had problem with the /etc/exports syntax which is different than Linux There’s a few things I wish would be better though but nothing too problematic or without workarounds : Cannot unlock LUKS partitions at boot time afaik (Acceptable tradeoff for the added security LUKS gives me vs my old Synology setup) but this force me to run a script to unlock LUKS, mount hammer and start mirror-stream at each boot No S1/S3 sleep so I made a script to shutdown the system when there’s no network neighborgs to serve the NFS As my system isn’t online 24/7 for energy reasons, I guess will have to run hammer cleanup myself from time to time Some uncertainty because hey, it’s kind of exotic but exciting too :) Overall, I am happy, HAMMER1 and PFS are looking really good, DragonFly is a neat Unix and the community is super friendly (Matthew Dillon actually provided me with a kernel patch to fix the broken ACPI on the PC holding this setup, many thanks!), the system is still a “work in progress” but it is already serving my files as I write this post. Let’s see in 6 months how it goes in the longer run ! Helpful resources : https://www.dragonflybsd.org/docs/how_to_implement_hammer_pseudo_file_system__40___pfs___41___slave_mirroring_from_pfs_master/ ###BSDCan 2018 Recap As promised, here is our second part of our BSDCan report, covering the conference proper. The last tutorials/devsummit of that day lead directly into the conference, as people could pick up their registration packs at the Red Lion and have a drink with fellow BSD folks. Allan and I were there only briefly, as we wanted to get back to the “Newcomers orientation and mentorship” session lead by Michael W. Lucas. This session is intended for people that are new to BSDCan (maybe their first BSD conference ever?) and may have questions. Michael explained everything from the 6-2-1 rule (hours of sleep, meals per day, and number of showers that attendees should have at a minimum), to the partner and widowers program (lead by his wife Liz), to the sessions that people should not miss (opening, closing, and hallway track). Old-time BSDCan folks were asked to stand up so that people can recognize them and ask them any questions they might have during the conferences. The session was well attended. Afterwards, people went for dinner in groups, a big one lead by Michael Lucas to his favorite Shawarma place, followed by gelato (of course). This allowed newbies to mingle over dinner and ice cream, creating a welcoming atmosphere. The next day, after Dan Langille opened the conference, Benno Rice gave the keynote presentation about “The Tragedy of Systemd”. Benedict went to the following talks: “Automating Network Infrastructures with Ansible on FreeBSD” in the DevSummit track. A good talk that connected well with his Ansible tutorial and even allowed some discussions among participants. “All along the dwatch tower”: Devin delivered a well prepared talk. I first thought that the number of slides would not fit into the time slot, but she even managed to give a demo of her work, which was well received. The dwatch tool she wrote should make it easy for people to get started with DTrace without learning too much about the syntax at first. The visualizations were certainly nice to see, combining different tools together in a new way. ZFS BoF, lead by Allan and Matthew Ahrens SSH Key Management by Michael W. Lucas. Yet another great talk where I learned a lot. I did not get to the SSH CA chapter in the new SSH Mastery book, so this was a good way to wet my appetite for it and motivated me to look into creating one for the cluster that I’m managing. The rest of the day was spent at the FreeBSD Foundation table, talking to various folks. Then, Allan and I had an interview with Kirk McKusick for National FreeBSD Day, then we had a core meeting, followed by a core dinner. Day 2: “Flexible Disk Use in OpenZFS”: Matthew Ahrens talking about the feature he is implementing to expand a RAID-Z with a single disk, as well as device removal. Allan’s talk about his efforts to implement ZSTD in OpenZFS as another compression algorithm. I liked his overview slides with the numbers comparing the algorithms for their effectiveness and his personal story about the sometimes rocky road to get the feature implemented. “zrepl - ZFS replication” by Christian Schwarz, was well prepared and even had a demo to show what his snapshot replication tool can do. We covered it on the show before and people can find it under sysutils/zrepl. Feedback and help is welcome. “The Evolution of FreeBSD Governance” by Kirk McKusick was yet another great talk by him covering the early days of FreeBSD until today, detailing some of the progress and challenges the project faced over the years in terms of leadership and governance. This is an ongoing process that everyone in the community should participate in to keep the project healthy and infused with fresh blood. Closing session and auction were funny and great as always. All in all, yet another amazing BSDCan. Thank you Dan Langille and your organizing team for making it happen! Well done. Digital Ocean ###NomadBSD 1.1-RC1 Released The first – and hopefully final – release candidate of NomadBSD 1.1 is available! Changes The base system has been upgraded to FreeBSD 11.2-RC3 EFI booting has been fixed. Support for modern Intel GPUs has been added. Support for installing packages has been added. Improved setup menu. More software packages: benchmarks/bonnie++ DSBDisplaySettings DSBExec DSBSu mail/thunderbird net/mosh ports-mgmt/octopkg print/qpdfview security/nmap sysutils/ddrescue sysutils/fusefs-hfsfuse sysutils/fusefs-sshfs sysutils/sleuthkit www/lynx x11-wm/compton x11/xev x11/xterm Many improvements and bugfixes The image and instructions can be found here. ##News Roundup LDAP client added to -current CVSROOT: /cvs Module name: src Changes by: reyk@cvs.openbsd.org 2018/06/13 09:45:58 Log message: Import ldap(1), a simple ldap search client. We have an ldapd(8) server and ypldap in base, so it makes sense to have a simple LDAP client without depending on the OpenLDAP package. This tool can be used in an ssh(1) AuthorizedKeysCommand script. With feedback from many including millert@ schwarze@ gilles@ dlg@ jsing@ OK deraadt@ Status: Vendor Tag: reyk Release Tags: ldap_20180613 N src/usr.bin/ldap/Makefile N src/usr.bin/ldap/aldap.c N src/usr.bin/ldap/aldap.h N src/usr.bin/ldap/ber.c N src/usr.bin/ldap/ber.h N src/usr.bin/ldap/ldap.1 N src/usr.bin/ldap/ldapclient.c N src/usr.bin/ldap/log.c N src/usr.bin/ldap/log.h No conflicts created by this import ###Intel® FPU Speculation Vulnerability Confirmed Earlier this month, Philip Guenther (guenther@) committed (to amd64 -current) a change from lazy to semi-eager FPU switching to mitigate against rumored FPU state leakage in Intel® CPUs. Theo de Raadt (deraadt@) discussed this in his BSDCan 2018 session. Using information disclosed in Theo’s talk, Colin Percival developed a proof-of-concept exploit in around 5 hours. This seems to have prompted an early end to an embargo (in which OpenBSD was not involved), and the official announcement of the vulnerability. FPU change in FreeBSD Summary: System software may utilize the Lazy FP state restore technique to delay the restoring of state until an instruction operating on that state is actually executed by the new process. Systems using Intel® Core-based microprocessors may potentially allow a local process to infer data utilizing Lazy FP state restore from another process through a speculative execution side channel. Description: System software may opt to utilize Lazy FP state restore instead of eager save and restore of the state upon a context switch. Lazy restored states are potentially vulnerable to exploits where one process may infer register values of other processes through a speculative execution side channel that infers their value. · CVSS - 4.3 Medium CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:L/I:N/A:N Affected Products: Intel® Core-based microprocessors. Recommendations: If an XSAVE-enabled feature is disabled, then we recommend either its state component bitmap in the extended control register (XCR0) is set to 0 (e.g. XCR0[bit 2]=0 for AVX, XCR0[bits 7:5]=0 for AVX512) or the corresponding register states of the feature should be cleared prior to being disabled. Also for relevant states (e.g. x87, SSE, AVX, etc.), Intel recommends system software developers utilize Eager FP state restore in lieu of Lazy FP state restore. Acknowledgements: Intel would like to thank Julian Stecklina from Amazon Germany, Thomas Prescher from Cyberus Technology GmbH (https://www.cyberus-technology.de/), Zdenek Sojka from SYSGO AG (http://sysgo.com), and Colin Percival for reporting this issue and working with us on coordinated disclosure. iXsystems iX Ad Spot iX Systems - BSDCan 2018 Recap ###FreeBSD gets pNFS support Merge the pNFS server code from projects/pnfs-planb-server into head. This code merge adds a pNFS service to the NFSv4.1 server. Although it is a large commit it should not affect behaviour for a non-pNFS NFS server. Some documentation on how this works can be found at: Merge the pN http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt and will hopefully be turned into a proper document soon. This is a merge of the kernel code. Userland and man page changes will come soon, once the dust settles on this merge. It has passed a "make universe", so I hope it will not cause build problems. It also adds NFSv4.1 server support for the "current stateid". Here is a brief overview of the pNFS service: A pNFS service separates the Read/Write operations from all the other NFSv4.1 Metadata operations. It is hoped that this separation allows a pNFS service to be configured that exceeds the limits of a single NFS server for either storage capacity and/or I/O bandwidth. It is possible to configure mirroring within the data servers (DSs) so that the data storage file for an MDS file will be mirrored on two or more of the DSs. When this is used, failure of a DS will not stop the pNFS service and a failed DS can be recovered once repaired while the pNFS service continues to operate. Although two way mirroring would be the norm, it is possible to set a mirroring level of up to four or the number of DSs, whichever is less. The Metadata server will always be a single point of failure, just as a single NFS server is. A Plan B pNFS service consists of a single MetaData Server (MDS) and K Data Servers (DS), all of which are recent FreeBSD systems. Clients will mount the MDS as they would a single NFS server. When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size 0. Each of these files will also have two extended attributes in the system attribute name space: pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file(s) on DS(s) for this file. pnfsd.dsattr - This extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file, so that the MDS doesn't need to acquire the attributes from the DS for every Getattr operation. For each regular (VREG) file, the MDS creates a data storage file on one (or more if mirroring is enabled) of the DSs in one of the "dsNN" subdirectories. The name of this file is the file handle of the file on the MDS in hexadecimal so that the name is unique. The DSs use subdirectories named "ds0" to "dsN" so that no one directory gets too large. The value of "N" is set via the sysctl vfs.nfsd.dsdirsize on the MDS, with the default being 20. For production servers that will store a lot of files, this value should probably be much larger. It can be increased when the "nfsd" daemon is not running on the MDS, once the "dsK" directories are created. For pNFS aware NFSv4.1 clients, the FreeBSD server will return two pieces of information to the client that allows it to do I/O directly to the DS. DeviceInfo - This is relatively static information that defines what a DS is. The critical bits of information returned by the FreeBSD server is the IP address of the DS and, for the Flexible File layout, that NFSv4.1 is to be used and that it is "tightly coupled". There is a "deviceid" which identifies the DeviceInfo. Layout - This is per file and can be recalled by the server when it is no longer valid. For the FreeBSD server, there is support for two types of layout, call File and Flexible File layout. Both allow the client to do I/O on the DS via NFSv4.1 I/O operations. The Flexible File layout is a more recent variant that allows specification of mirrors, where the client is expected to do writes to all mirrors to maintain them in a consistent state. The Flexible File layout also allows the client to report I/O errors for a DS back to the MDS. The Flexible File layout supports two variants referred to as "tightly coupled" vs "loosely coupled". The FreeBSD server always uses the "tightly coupled" variant where the client uses the same credentials to do I/O on the DS as it would on the MDS. For the "loosely coupled" variant, the layout specifies a synthetic user/group that the client uses to do I/O on the DS. The FreeBSD server does not do striping and always returns layouts for the entire file. The critical information in a layout is Read vs Read/Writea and DeviceID(s) that identify which DS(s) the data is stored on. At this time, the MDS generates File Layout layouts to NFSv4.1 clients that know how to do pNFS for the non-mirrored DS case unless the sysctl vfs.nfsd.default_flexfile is set non-zero, in which case Flexible File layouts are generated. The mirrored DS configuration always generates Flexible File layouts. For NFS clients that do not support NFSv4.1 pNFS, all I/O operations are done against the MDS which acts as a proxy for the appropriate DS(s). When the MDS receives an I/O RPC, it will do the RPC on the DS as a proxy. If the DS is on the same machine, the MDS/DS will do the RPC on the DS as a proxy and so on, until the machine runs out of some resource, such as session slots or mbufs. As such, DSs must be separate systems from the MDS. *** ###[What does {some strange unix command name} stand for?](http://www.unixguide.net/unix/faq/1.3.shtml) + awk = "Aho Weinberger and Kernighan" + grep = "Global Regular Expression Print" + fgrep = "Fixed GREP". + egrep = "Extended GREP" + cat = "CATenate" + gecos = "General Electric Comprehensive Operating Supervisor" + nroff = "New ROFF" + troff = "Typesetter new ROFF" + tee = T + bss = "Block Started by Symbol + biff = "BIFF" + rc (as in ".cshrc" or "/etc/rc") = "RunCom" + Don Libes' book "Life with Unix" contains lots more of these tidbits. *** ##Beastie Bits + [RetroBSD: Unix for microcontrollers](http://retrobsd.org/wiki/doku.php) + [On the matter of OpenBSD breaking embargos (KRACK)](https://marc.info/?l=openbsd-tech&m=152910536208954&w=2) + [Theo's Basement Computer Paradise (1998)](https://zeus.theos.com/deraadt/hosts.html) + [Airport Extreme runs NetBSD](https://jcs.org/2018/06/12/airport_ssh) + [What UNIX shell could have been](https://rain-1.github.io/shell-2.html) *** Tarsnap ad *** ##Feedback/Questions + We need more feedback and questions. Please email feedback@bsdnow.tv + Also, many of you owe us BSDCan trip reports! We have shared what our experience at BSDCan was like, but we want to hear about yours. What can we do better next year? What was it like being there for the first time? + [Jason writes in](https://slexy.org/view/s205jU58X2) + https://www.wheelsystems.com/en/products/wheel-fudo-psm/ + [June 19th was National FreeBSD Day](https://twitter.com/search?src=typd&q=%23FreeBSDDay) *** - Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [feedback@bsdnow.tv](mailto:feedback@bsdnow.tv) ***
New ZFS features landing in FreeBSD, MAP_STACK for OpenBSD, how to write safer C code with Clang’s address sanitizer, Michael W. Lucas on sponsor gifts, TCP blackbox recorder, and Dell disk system hacking. Headlines [A number of Upstream ZFS features landed in FreeBSD this week] 9188 increase size of dbuf cache to reduce indirect block decompression With compressed ARC (6950) we use up to 25% of our CPU to decompress indirect blocks, under a workload of random cached reads. To reduce this decompression cost, we would like to increase the size of the dbuf cache so that more indirect blocks can be stored uncompressed. If we are caching entire large files of recordsize=8K, the indirect blocks use 1/64th as much memory as the data blocks (assuming they have the same compression ratio). We suggest making the dbuf cache be 1/32nd of all memory, so that in this scenario we should be able to keep all the indirect blocks decompressed in the dbuf cache. (We want it to be more than the 1/64th that the indirect blocks would use because we need to cache other stuff in the dbuf cache as well.) In real world workloads, this won't help as dramatically as the example above, but we think it's still worth it because the risk of decreasing performance is low. The potential negative performance impact is that we will be slightly reducing the size of the ARC (by ~3%). 9166 zfs storage pool checkpoint The idea of Storage Pool Checkpoint (aka zpool checkpoint) deals with exactly that. It can be thought of as a “pool-wide snapshot” (or a variation of extreme rewind that doesn’t corrupt your data). It remembers the entire state of the pool at the point that it was taken and the user can revert back to it later or discard it. Its generic use case is an administrator that is about to perform a set of destructive actions to ZFS as part of a critical procedure. She takes a checkpoint of the pool before performing the actions, then rewinds back to it if one of them fails or puts the pool into an unexpected state. Otherwise, she discards it. With the assumption that no one else is making modifications to ZFS, she basically wraps all these actions into a “high-level transaction”. More information 8484 Implement aggregate sum and use for arc counters In pursuit of improving performance on multi-core systems, we should implements fanned out counters and use them to improve the performance of some of the arc statistics. These stats are updated extremely frequently, and can consume a significant amount of CPU time. And a small bug fix authored by me: 9321 arcloancompressedbuf() can increment arcloanedbytes by the wrong value arcloancompressedbuf() increments arcloanedbytes by psize unconditionally In the case of zfscompressedarcenabled=0, when the buf is returned via arcreturnbuf(), if ARCBUFCOMPRESSED(buf) is false, then arcloanedbytes is decremented by lsize, not psize. Switch to using arcbufsize(buf), instead of psize, which will return psize or lsize, depending on the result of ARCBUF_COMPRESSED(buf). MAP_STACK for OpenBSD Almost 2 decades ago we started work on W^X. The concept was simple. Pages that are writable, should not be executable. We applied this concept object by object, trying to seperate objects with different qualities to different pages. The first one we handled was the signal trampoline at the top of the stack. We just kept making changes in the same vein. Eventually W^X came to some of our kernel address spaces also. The fundamental concept is that an object should only have the permissions necessary, and any other operation should fault. The only permission separations we have are kernel vs userland, and then read, write, and execute. How about we add another new permission! This is not a hardware permission, but a software permission. It is opportunistically enforced by the kernel. the permission is MAPSTACK. If you want to use memory as a stack, you must mmap it with that flag bit. The kernel does so automatically for the stack region of a process's stack. Two other types of stack occur: thread stacks, and alternate signal stacks. Those are handled in clever ways. When a system call happens, we check if the stack-pointer register points to such a page. If it doesn't, the program is killed. We have tightened the ABI. You may no longer point your stack register at non-stack memory. You'll be killed. This checking code is MI, so it works for all platforms. Since page-permissions are generally done on page boundaries, there is caveat that thread and altstacks must now be page-sized and page-aligned, so that we can enforce the MAPSTACK attribute correctly. It is possible that a few ports need some massaging to satisfy this condition, but we haven't found any which break yet. A syslog_r has been added so that we can identify these failure cases. Also, the faulting cases are quite verbose for now, to help identify the programs we need to repair. **iXsystems** Writing Safer C with the Clang Address Sanitizer We wanted to improve our password strength algorithm, and decided to go for the industry-standard zxcvbn, from the people at Dropbox. Our web front-end would use the default Javascript library, and for mobile and desktop, we chose to use the C implementation as it was the lowest common denominator for all platforms. Bootstrapping all of this together was done pretty fast. I had toyed around with a few sample passwords so I decided to run it through the test suite we had for the previous password strength evaluator. The test generates a large number of random passwords according to different rules and expects the strength to be in a given range. But the test runner kept crashing with segmentation faults. It turns out the library has a lot of buffer overflow cases that are usually "harmless", but eventually crash your program when you run the evaluator function too much. I started fixing the cases I could see, but reading someone else's algorithms to track down tiny memory errors got old pretty fast. I needed a tool to help me. That's when I thought of Clang's Address Sanitizer. AddressSanitizer is a fast memory error detector. It consists of a compiler instrumentation module and a run-time library Let's try the sanitizer on a simple program. We'll allocate a buffer on the heap, copy each character of a string into it, and print it to standard output. + The site walks through a simple example which contains an error, it writes past the end of a buffer + The code works as expected, and nothing bad happens. It must be fine… + Then they compile it again with the address sanitizer actived So what can we gather from that pile of hex? Let's go through it line by line. AddressSanitizer found a heap buffer overflow at 0x60200000ef3d, a seemingly valid address (not NULL or any other clearly faulty value). + ASAN points directly to the line of code that is causing the problem We're writing outside of the heap in this instruction. And AddressSanitizer isn't having it. This is definitely one of my favorite indications. In addition to telling which line in the code failed and where in the memory the failure happened, you get a complete description of the closest allocated region in memory (which is probably the region you were trying to access). + They then walk through combining this with lldb, the Clang debugger, to actually interactively inspect the state of the problem when an invalid memory access happens Back to my practical case, how did I put the address sanitizer to good use? I simply ran the test suite, compiled with the sanitizer, with lldb. Sure enough, it stopped on every line that could cause a crash. It turns out there were many cases where zxcvbn-c wrote past the end of allocated buffers, on the heap and on the stack. I fixed those cases in the C library and ran the tests again. Not a segfault in sight! I've used memory tools in the past, but they were usually unwieldy, or put such a toll on performance that they were useless in any real-life case. Clang's address sanitizer turned out to be detailed, reliable, and surprisingly easy to use. I've heard of the miracles of Valgrind but macOS hardly supports it, making it a pain to use on my MacBook Pro. Coupled with Clang's static analyzer, AddressSanitizer is going to become a mandatory stop for evaluating code quality. It's also going to be the first tool I grab when facing confusing memory issues. There are many more case where I could use early failure and memory history to debug my code. For example, if a program crashes when accessing member of a deallocated object, we could easily trace the event that caused the deallocation, saving hours of adding and reading logs to retrace just what happened. News Roundup On sponsor gifts Note the little stack of customs forms off to the side. It’s like I’ve learned a lesson from standing at the post office counter filling out those stupid forms. Sponsors should get their books soon. This seems like an apropos moment to talk about what I do for print sponsors. I say I send them “a gift,” but what does that really mean? The obvious thing to ship them is a copy of the book I’ve written. Flat-out selling print books online has tax implications, though. Sponsors might have guessed that they’d get a copy of the book. But I shipped them the hardcover, which isn’t my usual practice. That’s because I send sponsors a gift. As it’s a gift, I get to choose what I send. I want to send them something nice, to encourage them to sponsor another book. It makes no sense for me to send a sponsor a Singing Wedgie-O-Gram. (Well, maybe a couple sponsors. You know who you are.) The poor bastards who bought into my scam–er, sponsored my untitled book–have no idea what’s coming. As of right now, their sensible guesses are woefully incomplete. Future books? They might get a copy of the book. They might get book plus something. They might just get the something. Folks who sponsor the jails book might get a cake with a file in it. Who knows? It’s a gift. It’s my job to make that gift worthwhile. And to amuse myself. Because otherwise, what’s the point? TCP Blackbox Recorder ``` Add the "TCP Blackbox Recorder" which we discussed at the developer summits at BSDCan and BSDCam in 2017. The TCP Blackbox Recorder allows you to capture events on a TCP connection in a ring buffer. It stores metadata with the event. It optionally stores the TCP header associated with an event (if the event is associated with a packet) and also optionally stores information on the sockets. It supports setting a log ID on a TCP connection and using this to correlate multiple connections that share a common log ID. You can log connections in different modes. If you are doing a coordinated test with a particular connection, you may tell the system to put it in mode 4 (continuous dump). Or, if you just want to monitor for errors, you can put it in mode 1 (ring buffer) and dump all the ring buffers associated with the connection ID when we receive an error signal for that connection ID. You can set a default mode that will be applied to a particular ratio of incoming connections. You can also manually set a mode using a socket option. This commit includes only basic probes. rrs@ has added quite an abundance of probes in his TCP development work. He plans to commit those soon. There are user-space programs which we plan to commit as ports. These read the data from the log device and output pcapng files, and then let you analyze the data (and metadata) in the pcapng files. Reviewed by: gnn (previous version) Obtained from: Netflix, Inc. Relnotes: yes Differential Revision: https://reviews.freebsd.org/D11085 ``` **Digital Ocean** Outta the way, KDE4 KDE4 has been rudely moved aside on FreeBSD. It still installs (use x11/kde4) and should update without a problem, but this is another step towards adding modern KDE (Plasma 5 and Applications) to the official FreeBSD Ports tree. This has taken a long time mostly for administrative reasons, getting all the bits lined up so that people sticking with KDE4 (which, right now, would be everyone using KDE from official ports and packages on FreeBSD) don’t end up with a broken desktop. We don’t want that. But now that everything Qt4 and kdelibs4-based has been moved aside by suffixing it with -kde4, we have the unsuffixed names free to indicate the latest-and-greatest from upstream. KDE4 users will see a lot of packages moving around and being renamed, but no functional changes. Curiously, the KDE4 desktop depends on Qt5 and KDE Frameworks 5 — and it has for quite some time already, because the Oxygen icons are shared with KDE Frameworks, but primarily because FileLight was updated to the modern KDE Applications version some time ago (the KDE4 version had some serious bugs, although I can not remember what they were). Now that the names are cleaned up, we could consider giving KDE4 users the buggy version back. From here on, we’ve got the following things lined up: Qt 5.10 is being worked on, except for WebEngine (it would slow down an update way too much), because Plasma is going to want Qt 5.10 soon. CMake 3.11 is in the -rc stage, so that is being lined up. The kde5-import branch in KDE-FreeBSD’s copy of the FreeBSD ports tree (e.g. Area51) is being prepped and polished for a few big SVN commits that will add all the new bits. So we’ve been saying Real Soon Now ™ for years, but things are Realer Sooner Nower ™ now. Dell FS12-NV7 and other 2U server (e.g. C6100) disk system hacking A while back I reviewed the Dell FS12-NV7 – a 2U rack server being sold cheap by all and sundry. It’s a powerful box, even by modern standards, but one of its big drawbacks is the disk system it comes with. But it needn’t be. There are two viable solutions, depending on what you want to do. You can make use of the SAS backplane, using SAS and/or SATA drives, or you can go for fewer SATA drives and free up one or more PCIe slots as Plan B. You probably have an FS12 because it looks good for building a drive array (or even FreeNAS) so I’ll deal with Plan A first. Like most Dell servers, this comes with a Dell PERC RAID SAS controller – a PERC6/i to be precise. This ‘I’ means it has internal connectors; the /E is the same but its sockets are external. The PERC connects to a twelve-slot backplane forming a drive array at the front of the box. More on the backplane later; it’s the PERCs you need to worry about. The PERC6 is actually an LSI Megaraid 1078 card, which is just the thing you need if you’re running an operating system like Windows that doesn’t support a volume manager, striping and other grown-up stuff. Or if your OS does have these features, but you just don’t trust it. If you are running such an OS you may as well stick to the PERC6, and good luck to you. If you’re using BSD (including FreeNAS), Solaris or a Linux distribution that handles disk arrays, read on. The PERC6 is a solution to a problem you probably don’t have, but in all other respects its a turkey. You really want a straightforward HBA (Host Bus Adapter) that allows your clever operating system to talk directly with the drives. Any SAS card based on the 1078 (such as the PERC6) is likely to have problems with drives larger than 2Tb. I’m not completely sure why, but I suspect it only applies to SATA. Unfortunately I don’t have any very large SAS drives to test this theory. A 2Tb limit isn’t really such a problem when you’re talking about a high performance array, as lots of small drives are a better option anyway. But it does matter if you’re building a very large datastore and don’t mind slower access and very significant resilvering times when you replace a drive. And for large datastores, very large SATA drives save you a whole lot of cash. The best capacity/cost ratio is for 5Gb SATA drives Some Dell PERCs can be re-flashed with LSI firmware and used as a normal HBA. Unfortunately the PERC6 isn’t one of them. I believe the PERC6/R can be, but those I’ve seen in a FS12 are just a bit too old. So the first thing you’ll need to do is dump them in the recycling or try and sell them on eBay. There are actually two PERC6 cards in most machine, and they each support eight SAS channels through two SFF-8484 connectors on each card. Given there are twelve drives slots, one of the PERCs is only half used. Sometimes they have a cable going off to a battery located near the fans. This is used in a desperate attempt to keep the data in the card’s cache safe in order to avoid write holes corrupting NTFS during a power failure, although the data on the on-drive caches won’t be so lucky. If you’re using a file system like that, make sure you have a UPS for the whole lot. But we’re going to put the PERCs out of our misery and replace them with some nice new LSI HBAs that will do our operating system’s bidding and let it talk to the drives as it knows best. But which to pick? First we need to know what we’re connecting. Moving to the front of the case there are twelve metal drive slots with a backplane behind. Dell makes machines with either backplanes or expanders. A backplane has a 1:1 SAS channel to drive connection; an expander takes one SAS channel and multiplexes it to (usually) four drives. You could always swap the blackplane with an expander, but I like the 1:1 nature of a backplane. It’s faster, especially if you’re configured as an array. And besides, we don’t want to spend more money than we need to, otherwise we wouldn’t be hot-rodding a cheap 2U server in the first place – expanders are expensive. Bizarrely, HBAs are cheap in comparison. So we need twelve channels of SAS that will connect to the sockets on the backplane. The HBA you will probably want to go with is an LSI, as these have great OS support. Other cards are available, but check that the drivers are also available. The obvious choice for SAS aficionados is the LSI 9211-8i, which has eight internal channels. This is based on an LSI 2000 series chip, the 2008, which is the de-facto standard. There’s also four-channel -4i version, so you could get your twelve channels using one of each – but the price difference is small these days, so you might as well go for two -8i cards. If you want cheaper there are 1068-based equivalent cards, and these work just fine at about half the price. They probably won’t work with larger disks, only operate at 3Gb and the original SAS standard. However, the 2000 series is only about £25 extra and gives you more options for the future. A good investment. Conversely, the latest 3000 series cards can do some extra stuff (particularly to do with active cables) but I can’t see any great advantage in paying megabucks for one unless you’re going really high-end – in which case the NV12 isn’t the box for you anyway. And you’d need some very fast drives and a faster backplane to see any speed advantage. And probably a new motherboard…. Whether the 6Gb SAS2 of the 9211-8i is any use on the backplane, which was designed for 3Gb, I don’t know. If it matters that much to you you probably need to spend a lot more money. A drive array with a direct 3Gb to each drive is going to shift fast enough for most purposes. Once you have removed the PERCs and plugged in your modern-ish 9211 HBAs, your next problem is going to be the cable. Both the PERCs and the backplane have SFF-8484 multi-lane connectors, which you might not recognise. SAS is a point-to-point system, the same as SATA, and a multi-lane cable is simply four single cables in a bundle with one plug. (Newer versions of SAS have more). SFF-8484 multi-lane connectors are somewhat rare, (but unfortunately this doesn’t make them valuable if you were hoping to flog them on eBay). The world switched quickly to the SFF-8087 for multi-lane SAS. The signals are electrically the same, but the connector is not. Please generate and paste your ad code here. If left empty, the ad location will be highlighted on your blog pages with a reminder to enter your code. Mid-Post So there are two snags with this backplane. Firstly it’s designed to work with PERC controllers; secondly it has the old SFF-8484 connectors on the back, and any SAS cables you find are likely to have SFF-8087. First things first – there is actually a jumper on the backplane to tell it whether it’s talking to a PERC or a standard LSI HBA. All you need to do is find it and change it. Fortunately there are very few jumpers to choose from (i.e. two), and you know the link is already in the wrong place. So try them one at a time until it works. The one you want may be labelled J15, but I wouldn’t like to say this was the same on every variant. Second problem: the cable. You can get cables with an SFF-8087 on one end and an SFF-8484 on the other. These should work. But they’re usually rather expensive. If you want to make your own, it’s a PITA but at least you have the connectors already (assuming you didn’t bin the ones on the PERC cables). I don’t know what committee designed SAS cable connectors, but ease of construction wasn’t foremost in their collective minds. You’re basically soldering twisted pair to a tiny PCB. This is mechanically rubbish, of course, as the slightest force on the cable will lift the track. Therefore its usual to cover the whole joint in solidified gunk (technical term) to protect it. Rewiring SAS connectors is definitely not easy. I’ve tried various ways of soldering to them, none of which were satisfactory or rewarding. One method is to clip the all bare wires you wish to solder using something like a bulldog clip so they’re at lined up horizontally and then press then adjust the clamp so they’re gently pressed to the tracks on the board, making final adjustments with a strong magnifying glass and a fine tweezers. You can then either solder them with a fine temperature-controlled iron, or have pre-coated the pads with solder paste and flash across it with an SMD rework station. I’d love to know how they’re actually manufactured – using a precision jig I assume. The “easy” way is to avoid soldering the connectors at all; simply cut existing cables in half and join one to the other. I’ve used prototyping matrix board for this. Strip and twist the conductors, push them through a hole and solder. This keeps things compact but manageable. We’re dealing with twisted pair here, so maintain the twists as close as possible to the board – it actually works quite well. However, I’ve now found a reasonably-priced source of the appropriate cable so I don’t do this any more. Contact me if you need some in the UK. So all that remains is to plug your HBAs to the backplane, shove in some drives and you’re away. If you’re at this stage, it “just works”. The access lights for all the drives do their thing as they should. The only mystery is how you can get the ident LED to come on; this may be controlled by the PERC when it detects a failure using the so-called sideband channel, or it may be operated by the electronics on the backplane. It’s workings are, I’m afraid, something of a mystery still – it’s got too much electronics on board to be a completely passive backplane. Plan B: SATA If you plan to use only SATA drives, especially if you don’t intend using more than six, it makes little sense to bother with SAS at all. The Gigabyte motherboard comes with half a dozen perfectly good 3Gb SATA channels, and if you need more you can always put another controller in a PCIe slot, or even USB. The advantages are lower cost and you get to free up two PCIe slots for more interesting things. The down-side is that you can’t use the SAS backplane, but you can still use the mounting bays. Removing the backplane looks tricky, but it really isn’t when you look a bit closer. Take out the fans first (held in place by rubber blocks), undo a couple of screws and it just lifts and slides out. You can then slot and lock in the drives and connect the SATA connectors directly to the back of the drives. You could even slide them out again without opening the case, as long as the cable was long enough and you manually detached the cable it when it was withdrawn. And let’s face it – drives are likely to last for years so even with half a dozen it’s not that great a hardship to open the case occasionally. Next comes power. The PSU has a special connector for the backplane and two standard SATA power plugs. You could split these three ways using an adapter, but if you have a lot of drives you might want to re-wire the cables going to the backplane plug. It can definitely power twelve drives. And that’s almost all there is to it. Unfortunately the main fans are connected to the backplane, which you’ve just removed. You can power them from an adapter on the drive power cables, but there are unused fan connectors on the motherboard. I’m doing a bit more research on cooling options, but this approach has promising possibilities for noise reduction. Beastie Bits Adriaan de Groot’s post FOSDEM blog post My First FreeNAS smart(8) Call for Testing by Michael Dexter BSDCan 2018 Travel Grant Application Now Open BSD Developer Kristaps Dzonsons interviews Linus Torvalds, about diving Twitter vote - The secret to a faster FreeBSD default build world... tmate - Instant terminal sharing Tarsnap Feedback/Questions Vikash - Getting a port added Chris Wells - Quarterly Ports Branch FreeBSD-CI configs on Github Jenkins on the FreeBSD Wiki Gordon - Centralised storage suggestions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
New ZFS features landing in FreeBSD, MAP_STACK for OpenBSD, how to write safer C code with Clang’s address sanitizer, Michael W. Lucas on sponsor gifts, TCP blackbox recorder, and Dell disk system hacking.
New ZFS features landing in FreeBSD, MAP_STACK for OpenBSD, how to write safer C code with Clang’s address sanitizer, Michael W. Lucas on sponsor gifts, TCP blackbox recorder, and Dell disk system hacking.
We cover an interview about Unix Architecture Evolution, another vBSDcon trip report, how to teach an old Unix about backspace, new NUMA support coming to FreeBSD, and stack pointer checking in OpenBSD. This episode was brought to you by Headlines Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD (https://fosdem.org/2018/interviews/diomidis-spinellis/) Q: Could you briefly introduce yourself? I'm a professor of software engineering, a programmer at heart, and a technology author. Currently I'm also the editor in chief of the IEEE Software magazine. I recently published the book Effective Debugging, where I detail 66 ways to debug software and systems. Q: What will your talk be about, exactly? I will describe how the architecture of the Unix operating system evolved over the past half century, starting from an unnamed system written in PDP-7 assembly language and ending with a modern FreeBSD system. My talk is based, first, on a GitHub repository where I tried to record the system's history from 1970 until today and, second, on the evolution of documented facilities (user commands, system calls, library functions) across revisions. I will thus present the early system's defining architectural features (layering, system calls, devices as files, an interpreter, and process management) and the important ones that followed in subsequent releases: the tree directory structure, user contributed code, I/O redirection, the shell as a user program, groups, pipes, scripting, and little languages. Q: Why this topic? Unix stands out as a major engineering breakthrough due to its exemplary design, its numerous technical contributions, its impact, its development model, and its widespread use. Furthermore, the design of the Unix programming environment has been characterized as one offering unusual simplicity, power, and elegance. Consequently, there are many lessons that we can learn by studying the evolution of the Unix architecture, which we can apply to the design of new systems. I often see modern systems that suffer from a bloat of architectural features and a lack of clear form on which functionality can be built. I believe that many of the modern Unix architecture defining features are excellent examples of what we should strive toward as system architects. Q: What do you hope to accomplish by giving this talk? What do you expect? I'd like FOSDEM attendees to leave the talk with their mind full with architectural features of timeless quality. I want them to realize that architectural elegance isn't derived by piling design patterns and does not need to be expensive in terms of resources. Rather, beautiful architecture can be achieved on an extremely modest scale. Furthermore, I want attendees to appreciate the importance of adopting flexible conventions rather than rigid enforcement mechanisms. Finally, I want to demonstrate through examples that the open source culture was part of Unix from its earliest days. Q: What are the most significant milestones in the development of Unix? The architectural development of Unix follows a path of continuous evolution, albeit at a slowing pace, so I don't see here the most important milestones. I would however define as significant milestones two key changes in the way Unix was developed. The first occurred in the late 1970s when significant activity shifted from a closely-knit team of researchers at the AT&T Bell Labs to the Computer Science Research Group in the University of California at Berkeley. This opened the system to academic contributions and growth through competitive research funding. The second took place in the late 1980s and the 1990s when Berkeley open-sourced the the code it had developed (by that time a large percentage of the system) and enthusiasts built on it to create complete open source operating system distributions: 386BSD, and then FreeBSD, NetBSD, OpenBSD, and others. Q: In which areas has the development of Unix stalled? The data I will show demonstrate that there were in the past some long periods where the number of C library functions and system calls remained mostly stable. Nowadays there is significant growth in the number of all documented facilities with the exception of file formats. I'm looking forward to a discussion regarding the meaning of these growth patterns in the Q&A session after the talk. Q: What are the core features that still link the 1970 PDP-7 system to the latest FreeBSD 11.1 release, almost half a century apart? Over the past half-century the Unix system has grown by four orders of magnitude from a few thousand lines of code to many millions. Nevertheless, looking at a 1970s architecture diagram and a current one reveals that the initial architectural blocks are still with us today. Furthermore, most system calls, user programs, and C library functions of that era have survived until today with essentially similar functionality. I've even found in modern FreeBSD some lines of code that have survived unchanged for 40 years. Q: Can we still add innovative changes to operating systems like FreeBSD without breaking the ‘Unix philosophy'? Will there be a moment where FreeBSD isn't recognizable anymore as a descendant of the 1970 PDP-7 system? There's a saying that “form liberates”. So having available a time-tested form for developing operating system functionality allows you to innovate in areas that matter rather than reinventing the wheel. Such concepts include having commands act as a filter, providing manual pages with a consistent structure, supplying build information in the form of a Makefile, installing files in a well-defined directory hierarchy, implementing filesystems with an standardized object-oriented interface, and packaging reusable functions as a library. Within this framework there's ample space for both incremental additions (think of jq, the JSON query command) and radical innovations (consider the Solaris-derived ZFS and dtrace functionality). For this reason I think that BSD and Linux systems will always be recognizable as direct or intellectual descendants of the 1970s Research Unix editions. Q: Have you enjoyed previous FOSDEM editions? Immensely! As an academic I need to attend many scientific conferences and meetings in order to present research results and interact with colleagues. This means too much time spent traveling and away from home, and a limited number of conferences I'm in the end able to attend. Nevertheless, attending FOSDEM is an easy decision due to the world-changing nature of its theme, the breadth of the topics presented, the participants' enthusiasm and energy, as well as the exemplary, very efficient conference organization. Another vBSDCon trip report we just found (https://www.weaponizedawesome.com/blog/?cat=53) We just got tipped about another trip report from vBSDCon, this time from one of the first time speakers: W. Dean Freeman Recently I had the honor of co-presenting on the internals of FreeBSD's Kernel RNG with John-Mark Gurney at the 3rd biennial vBSDCon, hosted in Reston, VA hosted by Verisign. I've been in and out of the FreeBSD community for about 20 years. As I've mentioned on here before, my first Unix encounter was FreeBSD 2.2.8 when I was in the 7th or 8th grade. However, for all that time I've never managed to get out to any of the cons. I've been to one or two BUG meetings and I've met some folks from IRC before, but nothing like this. A BSD conference is a very different experience than anything else out there. You have to try it, it is the only way to truly understand it. I'd also not had to do a stand-up presentation really since college before this. So, my first BSD con and my first time presenting rolled into one made for an interesting experience. See, he didn't say terrifying. It went very well. You should totally submit a talk for the next conference, even if it is your first. That said, it was amazing and invigorating experience. I got to meet a few big names in the FreeBSD community, discuss projects, ideas for FreeBSD, etc. I did seem to spend an unusual amount of time talking about FIPS and Common Criteria with folks, but to me that's a good sign and indicative that there is interest in working to close gaps between FreeBSD and the current requirements so that we can start getting FreeBSD and more BSD-based products into the government and start whittling away the domination of Linux (especially since Oracle has cut Solaris, SPARC and the ZFS storage appliance business units). There is nothing that can match the high bandwidth interchange of ideas in person. The internet has made all kinds of communication possible, and we use it all the time, but every once in a while, getting together in person is hugely valuable. Dean then went on to list some of the talks he found most valuable, including DTrace, Capsicum, bhyve, *BSD security tools, and Paul Vixie's talk about gets() I think the talk that really had the biggest impact on me, however, was Kyle Kneisl's talk on BSD community dynamics. One of the key points he asked was whether the things that drew us to the BSD community in the first place would be able to happen today. Obviously, I'm not a 12 or 13 year old kid anymore, but it really got me thinking. That, combined with getting face time with people I'd previously only known as screen names has recently drawn me back into participating in IRC and rejoining mailing lists (wdf on freenode. be on the lookout!) Then Dean covered some thoughts on his own talk: JMG and my talk seems to have been well received, with people paying lots of attention. I don't know what a typical number of questions is for one of these things, but on day one there weren't that many questions. We got about 5 during our question time and spent most of the rest of the day fielding questions from interested attendees. Getting a “great talk!” from GNN after coming down from the stage was probably one of the major highlights for me. I remember my first solo talk, and GNN asking the right question in the middle to get me to explain a part of it I had missed. It was very helpful. I think key to the interest in our presentation was that JMG did a good job framing a very complicated topic's importance in terms everyone could understand. It also helped that we got to drop some serious truth bombs. Final Thoughts: I met a lot of folks in person for the first time, and met some people I'd never known online before. It was a great community and I'm glad I got a chance to expand my network. Verisign were excellent hosts and they took good care of both speakers (covering airfare, rooms, etc.) and also conference attendees at large. The dinners that they hosted were quite good as well. I'm definitely interested in attending vBSDCon again and now that I've had a taste of meeting IRL with the community on scale of more than a handful, I have every intention of finally making it to BSDCan next year (I'd said it in 2017, but then moved to Texas for a new job and it wasn't going to be practical). This year for sure, though! Teaching an Almost 40-year Old UNIX about Backspace (https://virtuallyfun.com/2018/01/17/teaching_an_almost_40-year_old_unix_about_backspace/) Introduction I have been messing with the UNIX® operating system, Seventh Edition (commonly known as UNIX V7 or just V7) for a while now. V7 dates from 1979, so it's about 40 years old at this point. The last post was on V7/x86, but since I've run into various issues with it, I moved on to a proper installation of V7 on SIMH. The Internet has some really good resources on installing V7 in SIMH. Thus, I set out on my own journey on installing and using V7 a while ago, but that was remarkably uneventful. One convenience that I have been dearly missing since the switch from V7/x86 is a functioning backspace key. There seem to be multiple different definitions of backspace: BS, as in ASCII character 8 (010, 0x08, also represented as ^H), and DEL, as in ASCII character 127 (0177, 0x7F, also represented as ^?). V7 does not accept either for input by default. Instead, # is used as the erase character and @ is used as the kill character. These defaults have been there since UNIX V1. In fact, they have been “there” since Multics, where they got chosen seemingly arbitrarily. The erase character erases the character before it. The kill character kills (deletes) the whole line. For example, “ba##gooo#d” would be interpreted as “good” and “bad line@good line” would be interpreted as “good line”. There is some debate on whether BS or DEL is the correct character for terminals to send when the user presses the backspace key. However, most programs have settled on DEL today. tmux forces DEL, even if the terminal emulator sends BS, so simply changing my terminal to send BS was not an option. The change from the defaults outlined here to today's modern-day defaults occurred between 4.1BSD and 4.2BSD. enf on Hacker News has written a nice overview of the various conventions Getting the Diff For future generations as well as myself when I inevitably majorly break this installation of V7, I wanted to make a diff. However, my V7 is installed in SIMH. I am not a very intelligent man, I didn't keep backup copies of the files I'd changed. Getting data out of this emulated machine is an exercise in frustration. In the end, I printed everything on screen using cat(1) and copied that out. Then I performed a manual diff against the original source code tree because tabs got converted to spaces in the process. Then I applied the changes to clean copies that did have the tabs. And finally, I actually invoked diff(1). Closing Thoughts Figuring all this out took me a few days. Penetrating how the system is put together was surprisingly fairly hard at first, but then the difficulty curve eased up. It was an interesting exercise in some kind of “reverse engineering” and I definitely learned something about tty handling. I was, however, not pleased with using ed(1), even if I do know the basics. vi(1) is a blessing that I did not appreciate enough until recently. Had I also been unable to access recursive grep(1) on my host and scroll through the code, I would've probably given up. Writing UNIX under those kinds of editing conditions is an amazing feat. I have nothing but the greatest respect for software developers of those days. News Roundup New NUMA support coming to FreeBSD CURRENT (https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068145.html) Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem, and vmpage*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VMNUMAALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VMNUMAALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff Stack pointer checking – OpenBSD (https://marc.info/?l=openbsd-tech&m=151572838911297&w=2) Stefan (stefan@) and I have been working for a few months on this diff, with help from a few others. At every trap and system call, it checks if the stack-pointer is on a page that is marked MAPSTACK. execve() is changed to create such mappings for the process stack. Also, libpthread is taught the new MAPSTACK flag to use with mmap(). There is no corresponding system call which can set MAP_FLAG on an existing page, you can only set the flag by mapping new memory into place. That is a piece of the security model. The purpose of this change is to twart stack pivots, which apparently have gained some popularity in JIT ROP attacks. It makes it difficult to place the ROP stack in regular data memory, and then perform a system call from it. Workarounds are cumbersome, increasing the need for far more gadgetry. But also the trap case -- if any memory experiences a demand page fault, the same check will occur and potentially also kill the process. We have experimented a little with performing this check during device interrupts, but there are some locking concerns and performance may then become a concern. It'll be best to gain experience from handle of syncronous trap cases first. chrome and other applications I use run fine! I'm asking for some feedback to discover what ports this breaks, we'd like to know. Those would be ports which try to (unconventionally) create their stacks in malloc()'d memory or inside another Data structure. Most of them are probably easily fixed ... Qt 5.9 on FreeBSD (https://euroquis.nl/bobulate/?p=1768) Tobias and Raphael have spent the past month or so hammering on the Qt 5.9 branch, which has (finally!) landed in the official FreeBSD ports tree. This brings FreeBSD back up-to-date with current Qt releases and, more importantly, up-to-date with the Qt release KDE software is increasingly expecting. With Qt 5.9, the Elisa music player works, for instance (where it has run-time errors with Qt 5.7, even if it compiles). The KDE-FreeBSD CI system has had Qt 5.9 for some time already, but that was hand-compiled and jimmied into the system, rather than being a “proper” ports build. The new Qt version uses a new build system, which is one of the things that really slowed us down from a packaging perspective. Some modules have been reshuffled in the process. Some applications depending on Qt internal-private headers have been fixed along the way. The Telegram desktop client continues to be a pain in the butt that way. Following on from Qt 5.9 there has been some work in getting ready for Clang 6 support; in general the KDE and Qt stack is clean and modern C++, so it's more infrastructural tweaks than fixing code. Outside of our silo, I still see lots of wonky C++ code being fixed and plenty of confusion between pointers and integers and strings and chars and .. ugh. Speaking of ugh, I'm still planning to clean up Qt4 on ARM aarch64 for FreeBSD; this boils down to stealing suitable qatomic implementations from Arch Linux. For regular users of Qt applications on FreeBSD, there should be few to no changes required outside the regular upgrade cycle. For KDE Plasma users, note that development of the ports has changed branches; as we get closer to actually landing modern KDE bits, things have been renamed and reshuffled and mulled over so often that the old plasma5 branch wasn't really right anymore. The kde5-import branch is where it's at nowadays, and the instructions are the same: the x11/kde5 metaport will give you all the KDE Frameworks 5, KDE Plasma Desktop and modern KDE Applications you need. Adding IPv6 to an Nginx website on FreeBSD / FreshPorts (https://dan.langille.org/2018/01/13/adding-ipv6-to-an-nginx-website-on-freebsd-freshports/) FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized. There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation. All of this took about 3 hours. We are using: FreeBSD 11.1 Bind 9.9.11 nginx 1.12.2 Fallout I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses. ZFS on TrueOS: Why We Love OpenZFS (https://www.trueos.org/blog/zfs-trueos-love-openzfs/) TrueOS was the first desktop operating system to fully implement the OpenZFS (Zettabyte File System or ZFS for short) enterprise file system in a stable production environment. To fully understand why we love ZFS, we will look back to the early days of TrueOS (formerly PC-BSD). The development team had been using the UFS file system in TrueOS because of its solid track record with FreeBSD-based computer systems and its ability to check file consistency with the built-in check utility fsck. However, as computing demands increased, problems began to surface. Slow fsck file verification on large file systems, slow replication speeds, and inconsistency in data integrity while using UFS logging / journaling began to hinder users. It quickly became apparent that TrueOS users would need a file system that scales with evolving enterprise storage needs, offers the best data protection, and works just as well on a hobbyist system or desktop computer. Kris Moore, the founder of the TrueOS project, first heard about OpenZFS in 2007 from chatter on the FreeBSD mailing lists. In 2008, the TrueOS development team was thrilled to learn that the FreeBSD Project had ported ZFS. At the time, ZFS was still unproven as a graphical desktop solution, but Kris saw a perfect opportunity to offer ZFS as a cutting-edge file system option in the TrueOS installer, allowing the TrueOS project to act as an indicator of how OpenZFS would fair in real-world production use. The team was blown away by the reception and quality of OpenZFS on FreeBSD-based systems. By its nature, ZFS is a copy-on-write (CoW) file system that won't move a block of data until it both writes the data and verifies its integrity. This is very different from most other file systems in use today. ZFS is able to assure that data stays consistent between writes by automatically comparing write checksums, which mitigates bit rot. ZFS also comes with native RaidZ functionality that allows for enterprise data management and redundancy without the need for expensive traditional RAID cards. ZFS snapshots allow for system configuration backups in a split-second. You read that right. TrueOS can backup or restore snapshots in less than a second using the ZFS file system. Given these advantages, the TrueOS team decided to use ZFS as its exclusive file system starting in 2013, and we haven't looked back since. ZFS offers TrueOS users the stable workstation experience they want, while simultaneously scaling to meet the increasing demands of the enterprise storage market. TrueOS users are frequently commenting on how easy it is to use ZFS snapshots with our built-in snapshot utility. This allows users the freedom to experiment with their system knowing they can restore it in seconds if anything goes wrong. If you haven't had a chance to try ZFS with TrueOS, browse to our download page and make sure to grab a copy of TrueOS. You'll be blown away by the ease of use, data protection functionality, and incredible flexibility of RaidZ. Beastie Bits Source Code Podcast Interview with Michael W Lucas (https://blather.michaelwlucas.com/archives/3099) Operating System of the Year 2017: NetBSD Third place (https://w3techs.com/blog/entry/web_technologies_of_the_year_2017) OPNsense 18.1-RC1 released (https://opnsense.org/opnsense-18-1-rc1-released/) Personal OpenBSD Wiki Notes (https://balu-wiki.readthedocs.io/en/latest/security/openbsd.html) BSD section can use some contribution (https://guide.freecodecamp.org/bsd-os/) The Third Research Edition Unix Programmer's Manual (now available in PDF) (https://github.com/dspinellis/unix-v3man) Feedback/Questions Alex - my first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) Brian - Brian - openbsd 6.2 and enlightenment .17 (http://dpaste.com/24DYF1J#wrap) ***
We’re talking writing this week with my good friend Michael W. Lucas, a fellow No Starch Press Author. We discussed how he became interested in writing and how his career as an author evolved alongside his technical career. Now a full-time writer, Michael has written dozens of books, including technical and fiction works. If you've ever been interested in the business, skillset, or process of authorship then you'll enjoy our conversation.
We take a look at two-faced Oracle, cover a FAMP installation, how Netflix works the complex stuff, and show you who the patron of yak shaving is. This episode was brought to you by Headlines Why is Oracle so two-faced over open source? (https://www.theregister.co.uk/2017/10/12/oracle_must_grow_up_on_open_source/) Oracle loves open source. Except when the database giant hates open source. Which, according to its recent lobbying of the US federal government, seems to be "most of the time". Yes, Oracle has recently joined the Cloud Native Computing Foundation (CNCF) to up its support for open-source Kubernetes and, yes, it has long supported (and contributed to) Linux. And, yes, Oracle has even gone so far as to (finally) open up Java development by putting it under a foundation's stewardship. Yet this same, seemingly open Oracle has actively hammered the US government to consider that "there is no math that can justify open source from a cost perspective as the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." That punch to the face was delivered in a letter to Christopher Liddell, a former Microsoft CFO and now director of Trump's American Technology Council, by Kenneth Glueck, Oracle senior vice president. The US government had courted input on its IT modernisation programme. Others writing back to Liddell included AT&T, Cisco, Microsoft and VMware. In other words, based on its letter, what Oracle wants us to believe is that open source leads to greater costs and poorly secured, limply featured software. Nor is Oracle content to leave it there, also arguing that open source is exactly how the private sector does not function, seemingly forgetting that most of the leading infrastructure, big data, and mobile software today is open source. Details! Rather than take this counterproductive detour into self-serving silliness, Oracle would do better to follow Microsoft's path. Microsoft, too, used to Janus-face its way through open source, simultaneously supporting and bashing it. Only under chief executive Satya Nadella's reign did Microsoft realise it's OK to fully embrace open source, and its financial results have loved the commitment. Oracle has much to learn, and emulate, in Microsoft's approach. I love you, you're perfect. Now change Oracle has never been particularly warm and fuzzy about open source. As founder Larry Ellison might put it, Oracle is a profit-seeking corporation, not a peace-loving charity. To the extent that Oracle embraces open source, therefore it does so for financial reward, just like every other corporation. Few, however, are as blunt as Oracle about this fact of corporate open-source life. As Ellison told the Financial Times back in 2006: "If an open-source product gets good enough, we'll simply take it. So the great thing about open source is nobody owns it – a company like Oracle is free to take it for nothing, include it in our products and charge for support, and that's what we'll do. "So it is not disruptive at all – you have to find places to add value. Once open source gets good enough, competing with it would be insane... We don't have to fight open source, we have to exploit open source." "Exploit" sounds about right. While Oracle doesn't crack the top-10 corporate contributors to the Linux kernel, it does register a respectable number 12, which helps it influence the platform enough to feel comfortable building its IaaS offering on Linux (and Xen for virtualisation). Oracle has also managed to continue growing MySQL's clout in the industry while improving it as a product and business. As for Kubernetes, Oracle's decision to join the CNCF also came with P&L strings attached. "CNCF technologies such as Kubernetes, Prometheus, gRPC and OpenTracing are critical parts of both our own and our customers' development toolchains," said Mark Cavage, vice president of software development at Oracle. One can argue that Oracle has figured out the exploitation angle reasonably well. This, however, refers to the right kind of exploitation, the kind that even free software activist Richard Stallman can love (or, at least, tolerate). But when it comes to government lobbying, Oracle looks a lot more like Mr Hyde than Dr Jekyll. Lies, damned lies, and Oracle lobbying The current US president has many problems (OK, many, many problems), but his decision to follow the Obama administration's support for IT modernisation is commendable. Most recently, the Trump White House asked for feedback on how best to continue improving government IT. Oracle's response is high comedy in many respects. As TechDirt's Mike Masnick summarises, Oracle's "latest crusade is against open-source technology being used by the federal government – and against the government hiring people out of Silicon Valley to help create more modern systems. Instead, Oracle would apparently prefer the government just give it lots of money." Oracle is very good at making lots of money. As such, its request for even more isn't too surprising. What is surprising is the brazenness of its position. As Masnick opines: "The sheer contempt found in Oracle's submission on IT modernization is pretty stunning." Why? Because Oracle contradicts much that it publicly states in other forums about open source and innovation. More than this, Oracle contradicts much of what we now know is essential to competitive differentiation in an increasingly software and data-driven world. Take, for example, Oracle's contention that "significant IT development expertise is not... central to successful modernization efforts". What? In our "software is eating the world" existence Oracle clearly believes that CIOs are buyers, not doers: "The most important skill set of CIOs today is to critically compete and evaluate commercial alternatives to capture the benefits of innovation conducted at scale, and then to manage the implementation of those technologies efficiently." While there is some truth to Oracle's claim – every project shouldn't be a custom one-off that must be supported forever – it's crazy to think that a CIO – government or otherwise – is doing their job effectively by simply shovelling cash into vendors' bank accounts. Indeed, as Masnick points out: "If it weren't for Oracle's failures, there might not even be a USDS [the US Digital Service created in 2014 to modernise federal IT]. USDS really grew out of the emergency hiring of some top-notch internet engineers in response to the Healthcare.gov rollout debacle. And if you don't recall, a big part of that debacle was blamed on Oracle's technology." In short, blindly giving money to Oracle and other big vendors is the opposite of IT modernisation. In its letter to Liddell, Oracle proceeded to make the fantastic (by which I mean "silly and false") claim that "the fact is that the use of open-source software has been declining rapidly in the private sector". What?!? This is so incredibly untrue that Oracle should score points for being willing to say it out loud. Take a stroll through the most prominent software in big data (Hadoop, Spark, Kafka, etc.), mobile (Android), application development (Kubernetes, Docker), machine learning/AI (TensorFlow, MxNet), and compare it to Oracle's statement. One conclusion must be that Oracle believes its CIO audience is incredibly stupid. Oracle then tells a half-truth by declaring: "There is no math that can justify open source from a cost perspective." How so? Because "the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." Which I guess is why Oracle doesn't use any open source like Linux, Kubernetes, etc. in its services. Oops. The Vendor Formerly Known As Satan The thing is, Oracle doesn't need to do this and, for its own good, shouldn't do this. After all, we already know how this plays out. We need only look at what happened with Microsoft. Remember when Microsoft wanted us to "get the facts" about Linux? Now it's a big-time contributor to Linux. Remember when it told us open source was anti-American and a cancer? Now it aggressively contributes to a huge variety of open-source projects, some of them homegrown in Redmond, and tells the world that "Microsoft loves open source." Of course, Microsoft loves open source for the same reason any corporation does: it drives revenue as developers look to build applications filled with open-source components on Azure. There's nothing wrong with that. Would Microsoft prefer government IT to purchase SQL Server instead of open-source-licensed PostgreSQL? Sure. But look for a single line in its response to the Trump executive order that signals "open source is bad". You won't find it. Why? Because Microsoft understands that open source is a friend, not foe, and has learned how to monetise it. Microsoft, in short, is no longer conflicted about open source. It can compete at the product level while embracing open source at the project level, which helps fuel its overall product and business strategy. Oracle isn't there yet, and is still stuck where Microsoft was a decade ago. It's time to grow up, Oracle. For a company that builds great software and understands that it increasingly needs to depend on open source to build that software, it's disingenuous at best to lobby the US government to put the freeze on open source. Oracle needs to learn from Microsoft, stop worrying and love the open-source bomb. It was a key ingredient in Microsoft's resurgence. Maybe it could help Oracle get a cloud clue, too. Install FAMP on FreeBSD (https://www.linuxsecrets.com/home/3164-install-famp-on-freebsd) The acronym FAMP refers to a set of free open source applications which are commonly used in Web server environments called Apache, MySQL and PHP on the FreeBSD operating system, which provides a server stack that provides web services, database and PHP. Prerequisites sudo Installed and working - Please read Apache PHP5 or PHP7 MySQL or MariaDB Install your favorite editor, ours is vi Note: You don't need to upgrade FreeBSD but make sure all patches have been installed and your port tree is up-2-date if you plan to update by ports. Install Ports portsnap fetch You must use sudo for each indivdual command during installations. Please see link above for installing sudo. Searching Available Apache Versions to Install pkg search apache Install Apache To install Apache 2.4 using pkg. The apache 2.4 user account managing Apache is www in FreeBSD. pkg install apache24 Confirmation yes prompt and hit y for yes to install Apache 2.4 This installs Apache and its dependencies. Enable Apache use sysrc to update services to be started at boot time, Command below adds "apache24enable="YES" to the /etc/rc.conf file. For sysrc commands please read ```sysrc apache24enable=yes Start Apache service apache24 start``` Visit web address by accessing your server's public IP address in your web browser How To find Your Server's Public IP Address If you do not know what your server's public IP address is, there are a number of ways that you can find it. Usually, this is the address you use to connect to your server through SSH. ifconfig vtnet0 | grep "inet " | awk '{ print $2 }' Now that you have the public IP address, you may use it in your web browser's address bar to access your web server. Install MySQL Now that we have our web server up and running, it is time to install MySQL, the relational database management system. The MySQL server will organize and provide access to databases where our server can store information. Install MySQL 5.7 using pkg by typing pkg install mysql57-server Enter y at the confirmation prompt. This installs the MySQL server and client packages. To enable MySQL server as a service, add mysqlenable="YES" to the /etc/rc.conf file. This sysrc command will do just that ```sysrc mysqlenable=yes Now start the MySQL server service mysql-server start Now run the security script that will remove some dangerous defaults and slightly restrict access to your database system. mysqlsecureinstallation``` Answer all questions to secure your newly installed MySQL database. Enter current password for root (enter for none): [RETURN] Your database system is now set up and we can move on. Install PHP5 or PHP70 pkg search php70 Install PHP70 you would do the following by typing pkg install php70-mysqli mod_php70 Note: In these instructions we are using php5.7 not php7.0. We will be coming out with php7.0 instructions with FPM. PHP is the component of our setup that will process code to display dynamic content. It can run scripts, connect to MySQL databases to get information, and hand the processed content over to the web server to display. We're going to install the modphp, php-mysql, and php-mysqli packages. To install PHP 5.7 with pkg, run this command ```pkg install modphp56 php56-mysql php56-mysqli Copy sample PHP configuration file into place. cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini Regenerate the system's cached information about your installed executable files rehash``` Before using PHP, you must configure it to work with Apache. Install PHP Modules (Optional) To enhance the functionality of PHP, we can optionally install some additional modules. To see the available options for PHP 5.6 modules and libraries, you can type this into your system pkg search php56 Get more information about each module you can look at the long description of the package by typing pkg search -f apache24 Optional Install Example pkg install php56-calendar Configure Apache to Use PHP Module Open the Apache configuration file vim /usr/local/etc/apache24/Includes/php.conf DirectoryIndex index.php index.html Next, we will configure Apache to process requested PHP files with the PHP processor. Add these lines to the end of the file: SetHandler application/x-httpd-php SetHandler application/x-httpd-php-source Now restart Apache to put the changes into effect service apache24 restart Test PHP Processing By default, the DocumentRoot is set to /usr/local/www/apache24/data. We can create the info.php file under that location by typing vim /usr/local/www/apache24/data/info.php Add following line to info.php and save it. Details on info.php info.php file gives you information about your server from the perspective of PHP. It' useful for debugging and to ensure that your settings are being applied correctly. If this was successful, then your PHP is working as expected. You probably want to remove info.php after testing because it could actually give information about your server to unauthorized users. Remove file by typing rm /usr/local/www/apache24/data/info.php Note: Make sure Apache / meaning the root of Apache is owned by user which should have been created during the Apache install is the owner of the /usr/local/www structure. That explains FAMP on FreeBSD. IXsystems IXsystems TrueNAS X10 Torture Test & Fail Over Systems In Action with the ZFS File System (https://www.youtube.com/watch?v=GG_NvKuh530) How Netflix works: what happens every time you hit Play (https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b) Not long ago, House of Cards came back for the fifth season, finally ending a long wait for binge watchers across the world who are interested in an American politician's ruthless ascendance to presidency. For them, kicking off a marathon is as simple as reaching out for your device or remote, opening the Netflix app and hitting Play. Simple, fast and instantly gratifying. What isn't as simple is what goes into running Netflix, a service that streams around 250 million hours of video per day to around 98 million paying subscribers in 190 countries. At this scale, providing quality entertainment in a matter of a few seconds to every user is no joke. And as much as it means building top-notch infrastructure at a scale no other Internet service has done before, it also means that a lot of participants in the experience have to be negotiated with and kept satiated?—?from production companies supplying the content, to internet providers dealing with the network traffic Netflix brings upon them. This is, in short and in the most layman terms, how Netflix works. Let us just try to understand how Netflix is structured on the technological side with a simple example. Netflix literally ushered in a revolution around ten years ago by rewriting the applications that run the entire service to fit into a microservices architecture?—?which means that each application, or microservice's code and resources are its very own. It will not share any of it with any other app by nature. And when two applications do need to talk to each other, they use an application programming interface (API)?—?a tightly-controlled set of rules that both programs can handle. Developers can now make many changes, small or huge, to each application as long as they ensure that it plays well with the API. And since the one program knows the other's API properly, no change will break the exchange of information. Netflix estimates that it uses around 700 microservices to control each of the many parts of what makes up the entire Netflix service: one microservice stores what all shows you watched, one deducts the monthly fee from your credit card, one provides your device with the correct video files that it can play, one takes a look at your watching history and uses algorithms to guess a list of movies that you will like, and one will provide the names and images of these movies to be shown in a list on the main menu. And that's the tip of the iceberg. Netflix engineers can make changes to any part of the application and can introduce new changes rapidly while ensuring that nothing else in the entire service breaks down. They made a courageous decision to get rid of maintaining their own servers and move all of their stuff to the cloud?—?i.e. run everything on the servers of someone else who dealt with maintaining the hardware while Netflix engineers wrote hundreds of programs and deployed it on the servers rapidly. The someone else they chose for their cloud-based infrastructure is Amazon Web Services (AWS). Netflix works on thousands of devices, and each of them play a different format of video and sound files. Another set of AWS servers take this original film file, and convert it into hundreds of files, each meant to play the entire show or film on a particular type of device and a particular screen size or video quality. One file will work exclusively on the iPad, one on a full HD Android phone, one on a Sony TV that can play 4K video and Dolby sound, one on a Windows computer, and so on. Even more of these files can be made with varying video qualities so that they are easier to load on a poor network connection. This is a process known as transcoding. A special piece of code is also added to these files to lock them with what is called digital rights management or DRM?—?a technological measure which prevents piracy of films. The Netflix app or website determines what particular device you are using to watch, and fetches the exact file for that show meant to specially play on your particular device, with a particular video quality based on how fast your internet is at that moment. Here, instead of relying on AWS servers, they install their very own around the world. But it has only one purpose?—?to store content smartly and deliver it to users. Netflix strikes deals with internet service providers and provides them the red box you saw above at no cost. ISPs install these along with their servers. These Open Connect boxes download the Netflix library for their region from the main servers in the US?—?if there are multiple of them, each will rather store content that is more popular with Netflix users in a region to prioritise speed. So a rarely watched film might take time to load more than a Stranger Things episode. Now, when you will connect to Netflix, the closest Open Connect box to you will deliver the content you need, thus videos load faster than if your Netflix app tried to load it from the main servers in the US. In a nutshell… This is what happens when you hit that Play button: Hundreds of microservices, or tiny independent programs, work together to make one large Netflix service. Content legally acquired or licensed is converted into a size that fits your screen, and protected from being copied. Servers across the world make a copy of it and store it so that the closest one to you delivers it at max quality and speed. When you select a show, your Netflix app cherry picks which of these servers will it load the video from> You are now gripped by Frank Underwood's chilling tactics, given depression by BoJack Horseman's rollercoaster life, tickled by Dev in Master of None and made phobic to the future of technology by the stories in Black Mirror. And your lifespan decreases as your binge watching turns you into a couch potato. It looked so simple before, right? News Roundup Moving FreshPorts (http://dan.langille.org/2017/11/15/moving-freshports/) Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice. In preparation for this move, I have: DNS TTL reduced to 60s Posted to Twitter Updated the status page Put the website put in offline mode: What was missed I turned off commit processing on the new server, but I did not do this on the old server. I should have: sudo svc -d /var/service/freshports That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless. The offline code Here is the basic stuff I used to put the website into offline mode. The main points are: header(“HTTP/1.1 503 Service Unavailable”); ErrorDocument 404 /index.php I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code. The dump The database dump just started (Sun Nov 5 17:07:22 UTC 2017). root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump That should take about 30 minutes. I have set a timer to remind me. Total time was: 1464.82 real 1324.96 user 37.22 sys The MD5 is: MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5 It is now: Sun Nov 5 17:34:07 UTC 2017 The rsync The rsync should take about 10-20 minutes. I have already done an rsync of yesterday's dump file. The rsync today should copy over only the deltas (i.e. differences). The rsync started at about Sun Nov 5 17:36:05 UTC 2017 That took 2m9.091s The MD5 matches. The restore The restore should take about 30 minutes. I ran this test yesterday. It is now Sun Nov 5 17:40:03 UTC 2017. $ createdb -T template0 -E SQL_ASCII freshports.testing $ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump Done. real 25m21.108s user 1m57.508s sys 0m15.172s It is now Sun Nov 5 18:06:22 UTC 2017. Insert break here About here, I took a 30 minute break to run an errand. It was worth it. Changing DNS I'm ready to change DNS now. It is Sun Nov 5 19:49:20 EST 2017 Done. And nearly immediately, traffic started. How many misses? During this process, XXXXX requests were declined: $ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log XXXXX That's it, we're done Total elapsed time: 1 hour 48 minutes. There are still a number of things to follow up on, but that was the transfers. The new FreshPorts Server (http://dan.langille.org/2017/11/17/x8dtu-3/) *** Using bhyve on top of CEPH (https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005876.html) Hi, Just an info point. I'm preparing for a lecture tomorrow, and thought why not do an actual demo.... Like to be friends with Murphy :) So after I started the cluster: 5 jails with 7 OSDs This what I manually needed to do to boot a memory stick Start een Bhyve instance rbd --dest-pool rbddata --no-progress import memstick.img memstick rbd-ggate map rbddata/memstick ggate-devvice is available on /dev/ggate1 kldload vmm kldload nmdm kldload iftap kldload ifbridge kldload cpuctl sysctl net.link.tap.uponopen=1 ifconfig bridge0 create ifconfig bridge0 addm em0 up ifconfig ifconfig tap11 create ifconfig bridge0 addm tap11 ifconfig tap11 up load the GGate disk in bhyve bhyveload -c /dev/nmdm11A -m 2G -d /dev/ggate1 FB11 and boot a single from it. bhyve -H -P -A -c 1 -m 2G -l com1,/dev/nmdm11A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap11 -s 4,ahci-hd,/dev/ggate1 FB11 & bhyvectl --vm=FB11 --get-stats Connect to the VM cu -l /dev/nmdm11B And that'll give you a bhyve VM running on an RBD image over ggate. In the installer I tested reading from the bootdisk: root@:/ # dd if=/dev/ada0 of=/dev/null bs=32M 21+1 records in 21+1 records out 734077952 bytes transferred in 5.306260 secs (138341865 bytes/sec) which is a nice 138Mb/sec. Hope the demonstration does work out tomorrow. --WjW *** Donald Knuth - The Patron Saint of Yak Shaves (http://yakshav.es/the-patron-saint-of-yakshaves/) Excerpts: In 2015, I gave a talk in which I called Donald Knuth the Patron Saint of Yak Shaves. The reason is that Donald Knuth achieved the most perfect and long-running yak shave: TeX. I figured this is worth repeating. How to achieve the ultimate Yak Shave The ultimate yak shave is the combination of improbable circumstance, the privilege to be able to shave at your hearts will and the will to follow things through to the end. Here's the way it was achieved with TeX. The recount is purely mine, inaccurate and obviously there for fun. I'll avoid the most boring facts that everyone always tells, such as why Knuth's checks have their own Wikipedia page. Community Shaving is Best Shaving Since the release of TeX, the community has been busy working on using it as a platform. If you ever downloaded the full TeX distribution, please bear in mind that you are downloading the amassed work of over 40 years, to make sure that each and every TeX document ever written builds. We're talking about documents here. But mostly, two big projects sprung out of that. The first is LaTeX by Leslie Lamport. Lamport is a very productive researcher, famous for research in formal methods through TLA+ and also known laying groundwork for many distributed algorithms. LaTeX is based on the idea of separating presentation and content. It is based around the idea of document classes, which then describe the way a certain document is laid out. Think Markdown, just much more complex. The second is ConTeXt, which is far more focused on fine grained layout control. The Moral of the Story Whenever you feel like “can't we just replace this whole thing, it can't be so hard” when handling TeX, don't forget how many years of work and especially knowledge were poured into that system. Typesetting isn't the most popular knowledge around programmers. Especially see it in the context of the space it is in: they can't remove legacy. Ever. That would break documents. TeX is also not a programming language. It might resemble one, but mostly, it should be approached as a typesetting system first. A lot of it's confusing lingo gets much better then. It's not programming lingo. By approaching TeX with an understanding for its history, a lot of things can be learned from it. And yes, a replacement would be great, but it would take ages. In any case, I hope I thoroughly convinced you why Donald Knuth is the Patron Saint of Yak Shaves. Extra Credits This comes out of a enjoyable discussion with [Arne from Lambda Island](https://lambdaisland.com/https://lambdaisland.com/, who listened and said “you should totally turn this into a talk”. Vincent's trip to EuroBSDCon 2017 (http://www.vincentdelft.be/post/post_20171016) My euroBSDCon 2017 Posted on 2017-10-16 09:43:00 from Vincent in Open Bsd Let me just share my feedback on those 2 days spent in Paris for the EuroBSDCon. My 1st BSDCon. I'm not a developer, contributor, ... Do not expect to improve your skills with OpenBSD with this text :-) I know, we are on October 16th, and the EuroBSDCon of Paris was 3 weeks ago :( I'm not quick !!! Sorry for that Arrival at 10h, I'm too late for the start of the key note. The few persons behind a desk welcome me by talking in Dutch, mainly because of my name. Indeed, Delft is a city in Netherlands, but also a well known university. I inform them that I'm from Belgium, and the discussion moves to the fact the Fosdem is located in Brussels. I receive my nice T-shirt white and blue, a bit like the marine T-shirts, but with the nice EuroBSDCon logo. I'm asking where are the different rooms reserved for the BSD event. We have 1 big on the 1st floor, 1 medium 1 level below, and 2 smalls 1 level above. All are really easy to access. In this entrance we have 4 or 5 tables with some persons representing their company. Those are mainly the big sponsors of the event providing details about their activity and business. I discuss a little bit with StormShield and Gandi. On other tables people are selling BSD t-shirts, and they will quickly be sold. "Is it done yet ?" The never ending story of pkg tools In the last Fosdem, I've already hear Antoine and Baptiste presenting the OpenBSD and FreeBSD battle, I decide to listen Marc Espie in the medium room called Karnak. Marc explains that he has rewritten completely the pkg_add command. He explains that, at contrario with other elements of OpenBSD, the packages tools must be backward compatible and stable on a longer period than 12 months (the support period for OpenBSD). On the funny side, he explains that he has his best idea inside his bath. Hackathons are also used to validate some ideas with other OpenBSD developers. All in all, he explains that the most time consuming part is to imagine a good solution. Coding it is quite straightforward. He adds that better an idea is, shorter the implementation will be. A Tale of six motherboards, three BSDs and coreboot After the lunch I decide to listen the talk about Coreboot. Indeed, 1 or 2 years ago I had listened the Libreboot project at Fosdem. Since they did several references to Coreboot, it's a perfect occasion to listen more carefully to this project. Piotr and Katazyba Kubaj explains us how to boot a machine without the native Bios. Indeed Coreboot can replace the bios, and de facto avoid several binaries imposed by the vendor. They explain that some motherboards are supporting their code. But they also show how difficult it is to flash a Bios and replace it by Coreboot. They even have destroyed a motherboard during the installation. Apparently because the power supply they were using was not stable enough with the 3v. It's really amazing to see that open source developers can go, by themselves, to such deep technical level. State of the DragonFly's graphics stack After this Coreboot talk, I decide to stay in the room to follow the presentation of Fran?ois Tigeot. Fran?ois is now one of the core developer of DrangonflyBSD, an amazing BSD system having his own filesystem called Hammer. Hammer offers several amazing features like snapshots, checksum data integrity, deduplication, ... Francois has spent his last years to integrate the video drivers developed for Linux inside DrangonflyBSD. He explains that instead of adapting this code for the video card to the kernel API of DrangonflyBSD, he has "simply" build an intermediate layer between the kernel of DragonflyBSD and the video drivers. This is not said in the talk, but this effort is very impressive. Indeed, this is more or less a linux emulator inside DragonflyBSD. Francois explains that he has started with Intel video driver (drm/i915), but now he is able to run drm/radeon quite well, but also drm/amdgpu and drm/nouveau. Discovering OpenBSD on AWS Then I move to the small room at the upper level to follow a presentation made by Laurent Bernaille on OpenBSD and AWS. First Laurent explains that he is re-using the work done by Antoine Jacoutot concerning the integration of OpenBSD inside AWS. But on top of that he has integrated several other Open Source solutions allowing him to build OpenBSD machines very quickly with one command. Moreover those machines will have the network config, the required packages, ... On top of the slides presented, he shows us, in a real demo, how this system works. Amazing presentation which shows that, by putting the correct tools together, a machine builds and configure other machines in one go. OpenBSD Testing Infrastructure Behind bluhm.genua.de Here Jan Klemkow explains us that he has setup a lab where he is able to run different OpenBSD architectures. The system has been designed to be able to install, on demand, a certain version of OpenBSD on the different available machines. On top of that a regression test script can be triggered. This provides reports showing what is working and what is not more working on the different machines. If I've well understood, Jan is willing to provide such lab to the core developers of OpenBSD in order to allow them to validate easily and quickly their code. Some more effort is needed to reach this goal, but with what exists today, Jan and his colleague are quite close. Since his company is using OpenBSD business, to his eyes this system is a "tit for tat" to the OpenBSD community. French story on cybercrime Then comes the second keynote of the day in the big auditorium. This talk is performed by the colonel of french gendarmerie. Mr Freyssinet, who is head of the Cyber crimes unit inside the Gendarmerie. Mr Freyssinet explains that the "bad guys" are more and more volatile across countries, and more and more organized. The small hacker in his room, alone, is no more the reality. As a consequence the different national police investigators are collaborating more inside an organization called Interpol. What is amazing in his talk is that Mr Freyssinet talks about "Crime as a service". Indeed, more and more hackers are selling their services to some "bad and temporary organizations". Social event It's now time for the famous social event on the river: la Seine. The organizers ask us to go, by small groups, to a station. There is a walk of 15 minutes inside Paris. Hopefully the weather is perfect. To identify them clearly several organizers takes a "beastie fork" in their hands and walk on the sidewalk generating some amazing reactions from some citizens and toursits. Some of them recognize the Freebsd logo and ask us some details. Amazing :-) We walk on small and big sidewalks until a small stair going under the street. There, we have a train station a bit like a metro station. 3 stations later they ask us to go out. We walk few minutes and come in front of a boat having a double deck: one inside, with nice tables and chairs and one on the roof. But the crew ask us to go up, on the second deck. There, we are welcome with a glass of wine. The tour Eiffel is just at few 100 meters from us. Every hour the Eiffel tower is blinking for 5 minutes with thousands of small lights. Brilliant :-) We see also the "statue de la libertee" (the small one) which is on a small island in the middle of the river. During the whole night the bar will be open with drinks and some appetizers, snacks, ... Such walking diner is perfect to talk with many different persons. I've discussed with several persons just using BSD, they are not, like me, deep and specialized developers. One was from Switzerland, another one from Austria, and another one from Netherlands. But I've also followed a discussion with Theo de Raadt, several persons of the FreeBSD foundation. Some are very technical guys, other just users, like me. But all with the same passion for one of the BSD system. Amazing evening. OpenBSD's small steps towards DTrace (a tale about DDB and CTF) On the second day, I decide to sleep enough in order to have enough resources to drive back to my home (3 hours by car). So I miss the 1st presentations, and arrive at the event around 10h30. Lot of persons are already present. Some faces are less "fresh" than others. I decide to listen to Dtrace in OpenBSD. After 10 minutes I am so lost into those too technical explainations, that I decide to open and look at my PC. My OpenBSD laptop is rarely leaving my home, so I've never had the need to have a screen locking system. In a crowded environment, this is better. So I was looking for a simple solution. I've looked at how to use xlock. I've combined it with the /ets/apm/suspend script, ... Always very easy to use OpenBSD :-) The OpenBSD web stack Then I decide to follow the presentation of Michael W Lucas. Well know person for his different books about "Absolute OpenBSD", Relayd", ... Michael talks about the httpd daemon inside OpenBSD. But he also present his integration with Carp, Relayd, PF, FastCGI, the rules based on LUA regexp (opposed to perl regexp), ... For sure he emphasis on the security aspect of those tools: privilege separation, chroot, ... OpenSMTPD, current state of affairs Then I follow the presentation of Gilles Chehade about the OpenSMTPD project. Amazing presentation that, on top of the technical challenges, shows how to manage such project across the years. Gilles is working on OpenSMTPD since 2007, thus 10 years !!!. He explains the different decisions they took to make the software as simple as possible to use, but as secure as possible, too: privilege separation, chroot, pledge, random malloc, ? . The development starts on BSD systems, but once quite well known they received lot of contributions from Linux developers. Hoisting: lessons learned integrating pledge into 500 programs After a small break, I decide to listen to Theo de Raadt, the founder of OpenBSD. In his own style, with trekking boots, shorts, backpack. Theo starts by saying that Pledge is the outcome of nightmares. Theo explains that the book called "Hacking blind" presenting the BROP has worried him since few years. That's why he developed Pledge as a tool killing a process as soon as possible when there is an unforeseen behavior of this program. For example, with Pledge a program which can only write to disk will be immediately killed if he tries to reach network. By implementing Pledge in the +-500 programs present in the "base", OpenBSD is becoming more secured and more robust. Conclusion My first EuroBSDCon was a great, interesting and cool event. I've discussed with several BSD enthusiasts. I'm using OpenBSD since 2010, but I'm not a developer, so I was worried to be "lost" in the middle of experts. In fact it was not the case. At EuroBSDCon you have many different type of enthusiasts BSD's users. What is nice with the EuroBSDCon is that the organizers foresee everything for you. You just have to sit and listen. They foresee even how to spend, in a funny and very cool attitude, the evening of Saturday. > The small draw back is that all of this has a cost. In my case the whole weekend cost me a bit more than 500euro. Based on what I've learned, what I've saw this is very acceptable price. Nearly all presentations I saw give me a valuable input for my daily job. For sure, the total price is also linked to my personal choice: hotel, parking. And I'm surely biased because I'm used to go to the Fosdem in Brussels which cost nothing (entrance) and is approximately 45 minutes of my home. But Fosdem is not the same atmosphere and presentations are less linked to my daily job. I do not regret my trip to EuroBSDCon and will surely plan other ones. Beastie Bits Important munitions lawyering (https://www.jwz.org/blog/2017/10/important-munitions-lawyering/) AsiaBSDCon 2018 CFP is now open, until December 15th (https://2018.asiabsdcon.org/) ZSTD Compression for ZFS by Allan Jude (https://www.youtube.com/watch?v=hWnWEitDPlM&feature=share) NetBSD on Allwinner SoCs Update (https://blog.netbsd.org/tnf/entry/netbsd_on_allwinner_socs_update) *** Feedback/Questions Tim - Creating Multi Boot USB sticks (http://dpaste.com/0FKTJK3#wrap) Nomen - ZFS Questions (http://dpaste.com/1HY5MFB) JJ - Questions (http://dpaste.com/3ZGNSK9#wrap) Lars - Hardening Diffie-Hellman (http://dpaste.com/3TRXXN4) ***
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
In this episode, we take a look at the reimplementation of NetBSD using a Microkernel, check out what makes DHCP faster, and see what high-process count support for DragonflyBSD has to offer, and we answer the questions you've always wanted to ask us. This episode was brought to you by Headlines A Reimplementation Of Netbsd Using a Microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Minix author Andy Tanenbaum writes in Part 1 of a-reimplementation-of-netbsd-using-a-microkernel (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-1-of-2/) Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project. Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running. The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this. The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project. Part 2 (http://theembeddedboard.review/a-reimplementation-of-netbsd-using-a-microkernel-part-2-of-2/) is also available. *** Rapid DHCP: Or, how do Macs get on the network so fast? (https://cafbit.com/post/rapid_dhcp_or_how_do/) One of life's minor annoyances is having to wait on my devices to connect to the network after I wake them from sleep. All too often, I'll open the lid on my EeePC netbook, enter a web address, and get the dreaded "This webpage is not available" message because the machine is still working on connecting to my Wi-Fi network. On some occasions, I have to twiddle my thumbs for as long as 10-15 seconds before the network is ready to be used. The frustrating thing is that I know it doesn't have to be this way. I know this because I have a Mac. When I open the lid of my MacBook Pro, it connects to the network nearly instantaneously. In fact, no matter how fast I am, the network comes up before I can even try to load a web page. My curiosity got the better of me, and I set out to investigate how Macs are able to connect to the network so quickly, and how the network connect time in other operating systems could be improved. I figure there are three main categories of time-consuming activities that occur during network initialization: Link establishment. This is the activity of establishing communication with the network's link layer. In the case of Wi-Fi, the radio must be powered on, the access point detected, and the optional encryption layer (e.g. WPA) established. After link establishment, the device is able to send and receive Ethernet frames on the network. Dynamic Host Configuration Protocol (DHCP). Through DHCP handshaking, the device negotiates an IP address for its use on the local IP network. A DHCP server is responsible for managing the IP addresses available for use on the network. Miscellaneous overhead. The operating system may perform any number of mundane tasks during the process of network initialization, including running scripts, looking up preconfigured network settings in a local database, launching programs, etc. My investigation thus far is primarily concerned with the DHCP phase, although the other two categories would be interesting to study in the future. I set up a packet capture environment with a spare wireless access point, and observed the network activity of a number of devices as they initialized their network connection. For a worst-case scenario, let's look at the network activity captured while an Android tablet is connecting: This tablet, presumably in the interest of "optimization", is initially skipping the DHCP discovery phase and immediately requesting its previous IP address. The only problem is this is a different network, so the DHCP server ignores these requests. After about 4.5 seconds, the tablet stubbornly tries again to request its old IP address. After another 4.5 seconds, it resigns itself to starting from scratch, and performs the DHCP discovery needed to obtain an IP address on the new network. In all fairness, this delay wouldn't be so bad if the device was connecting to the same network as it was previously using. However, notice that the tablet waits a full 1.13 seconds after link establishment to even think about starting the DHCP process. Engineering snappiness usually means finding lots of small opportunities to save a few milliseconds here and there, and someone definitely dropped the ball here. In contrast, let's look at the packet dump from the machine with the lightning-fast network initialization, and see if we can uncover the magic that is happening under the hood: The key to understanding the magic is the first three unicast ARP requests. It looks like Mac OS remembers certain information about not only the last connected network, but the last several networks. In particular, it must at least persist the following tuple for each of these networks: > 1. The Ethernet address of the DHCP server > 2. The IP address of the DHCP server > 3. Its own IP address, as assigned by the DHCP server During network initialization, the Mac transmits carefully crafted unicast ARP requests with this stored information. For each network in its memory, it attempts to send a request to the specific Ethernet address of the DHCP server for that network, in which it asks about the server's IP address, and requests that the server reply to the IP address which the Mac was formerly using on that network. Unless network hosts have been radically shuffled around, at most only one of these ARP requests will result in a response—the request corresponding to the current network, if the current network happens to be one of the remembered networks. This network recognition technique allows the Mac to very rapidly discover if it is connected to a known network. If the network is recognized (and presumably if the Mac knows that the DHCP lease is still active), it immediately and presumptuously configures its IP interface with the address it knows is good for this network. (Well, it does perform a self-ARP for good measure, but doesn't seem to wait more than 13ms for a response.) The DHCP handshaking process begins in the background by sending a DHCP request for its assumed IP address, but the network interface is available for use during the handshaking process. If the network was not recognized, I assume the Mac would know to begin the DHCP discovery phase, instead of sending blind requests for a former IP address as the Galaxy Tab does. The Mac's rapid network initialization can be credited to more than just the network recognition scheme. Judging by the use of ARP (which can be problematic to deal with in user-space) and the unusually regular transmission intervals (a reliable 1.0ms delay between each packet sent), I'm guessing that the Mac's DHCP client system is entirely implemented as tight kernel-mode code. The Mac began the IP interface initialization process a mere 10ms after link establishment, which is far faster than any other device I tested. Android devices such as the Galaxy Tab rely on the user-mode dhclient system (part of the dhcpcd package) dhcpcd program, which no doubt brings a lot of additional overhead such as loading the program, context switching, and perhaps even running scripts. The next step for some daring kernel hacker is to implement a similarly aggressive DHCP client system in the Linux kernel, so that I can enjoy fast sign-on speeds on my Android tablet, Android phone, and Ubuntu netbook. There already exists a minimal DHCP client implementation in the Linux kernel, but it lacks certain features such as configuring the DNS nameservers. Perhaps it wouldn't be too much work to extend this code to support network recognition and interface with a user-mode daemon to handle such auxillary configuration information received via DHCP. If I ever get a few spare cycles, maybe I'll even take a stab at it. You can also find other ways of optimizing the dhclient program and how it works in the dhclient tutorial on Calomel.org (https://calomel.org/dhclient.html). *** BSDCam Trip Report (https://www.freebsdfoundation.org/blog/bsdcam-2017-trip-report-michael-lucas/) Over the decades, FreeBSD development and coordination has shifted from being purely on-line to involving more and more in-person coordination and cooperation. The FreeBSD Foundation sponsors a devsummit right before BSDCan, EuroBSDCon, and AsiaBSDCon, so that developers traveling to the con can leverage their airfare and hammer out some problems. Yes, the Internet is great for coordination, but nothing beats a group of developers spending ten minutes together to sketch on a whiteboard and figuring out exactly how to make something bulletproof. In addition to the coordination efforts, though, conference devsummits are hierarchical. There's a rigid schedule, with topics decided in advance. Someone leads the session. Sessions can be highly informative, passionate arguments, or anything in between. BSDCam is… a little different. It's an invaluable part of the FreeBSD ecosystem. However, it's something that I wouldn't normally attend. But right now, is not normal. I'm writing a new edition of Absolute FreeBSD. To my astonishment, people have come to rely on this book when planning their deployments and operations. While I find this satisfying, it also increases the pressure on me to get things correct. When I wrote my first FreeBSD book back in 2000, a dozen mailing lists provided authoritative information on FreeBSD development. One person could read every one of those lists. Today, that's not possible—and the mailing lists are only one narrow aspect of the FreeBSD social system. Don't get me wrong—it's pretty easy to find out what people are doing and how the system works. But it's not that easy to find out what people will be doing and how the system will work. If this book is going to be future-proof, I needed to leave my cozy nest and venture into the wilds of Cambridge, England. Sadly, the BSDCam chair agreed with my logic, so I boarded an aluminum deathtrap—sorry, a “commercial airliner”—and found myself hurtled from Detroit to Heathrow. And one Wednesday morning, I made it to the William Gates building of Cambridge University, consciousness nailed to my body by a thankfully infinite stream of proper British tea. BSDCam attendance is invitation only, and the facilities can only handle fifty folks or so. You need to be actively working on FreeBSD to wrangle an invite. Developers attend from all over the world. Yet, there's no agenda. Robert Watson is the chair, but he doesn't decide on the conference topics. He goes around the room and asks everyone to introduce themselves, say what they're working on, and declare what they want to discuss during the conference. The topics of interest are tallied. The most popular topics get assigned time slots and one of the two big rooms. Folks interested in less popular topics are invited to claim one of the small breakout rooms. Then the real fun begins. I started by eavesdropping in the virtualization workshop. For two hours, people discussed FreeBSD's virtualization needs, strengths, and weaknesses. What needs help? What should this interface look like? What compatibility is important, and what isn't? By the end of the session, the couple dozen people had developed a reasonable consensus and, most importantly, some folks had added items to their to-do lists. Repeat for a dozen more topics. I got a good grip on what's really happening with security mitigation techniques, FreeBSD's cloud support, TCP/IP improvements, advances in teaching FreeBSD, and more. A BSDCan devsummit presentation on packaging the base system is informative, but eavesdropping on two dozen highly educated engineers arguing about how to nail down the final tidbits needed to make that a real thing is far more educational. To my surprise, I was able to provide useful feedback for some sessions. I speak at a lot of events outside of the FreeBSD world, and was able to share much of what I hear at Linux conferences. A tool that works well for an experienced developer doesn't necessarily work well for everyone. Every year, I leave BSDCan tired. I left BSDCam entirely exhausted. These intense, focused discussions stretched my brain. But, I have a really good idea where key parts of FreeBSD development are actually headed. This should help future-proof the new Absolute FreeBSD, as much as any computer book can be future-proof. Plus, BSDCam throws the most glorious conference dinner I've ever seen. I want to thank Robert Watson for his kind invitation, and the FreeBSD Foundation for helping defray the cost of this trip Interview - The BSDNow Crew As a kid, what did you dream of to become as an adult? JT: An Astronaut BR: I wanted to be a private detective, because of all the crime novels that I read back then. I didn't get far with it. However, I think the structured analysis skills (who did what, when, and such) help me in debugging and sysadmin work. AJ: Didn't think about it much How do you manage to stay organized day to day with so much things you're actively doing each day? (Day job, wife/girlfriend, conferences, hobbies, friends, etc.) JT: Who said I was organized? BR: A lot of stuff in my calendar as reminders, open browser tabs as “to read later” list. A few things like task switching when getting stuck helps. Also, focus on a single goal for the day, even though there will be distractions. Slowly, but steadily chip away at the things you're working on. Rather than to procrastinate and put things back to review later, get started early with easy things for a big task and then tackle the hard part. Often, things look totally chaotic and unmanageable, until you start working on them. AJ: I barely manage. Lots of Google Calendar reminders, and the entire wall of my office is covered in whiteboard sheet todo lists. I use pinboard.in to deal with finding and organizing bookmarks. Write things down, don't trust your memory. What hobbies outside of IT do you have? JT: I love photography, but I do that Professional part time, so I'm not sure if that counts as a hobby anymore. I guess it'd have to be working in the garage on my cars. BR: I do Tai Chi to relax once a week in a group, but can also do it alone, pretty much everywhere. Way too much Youtube watching and browsing the web. I did play some games before studying at the university and I'm still proud that I could control it to the bare minimum not to impact my studies. A few “lapses” from time to time, revisiting the old classics since the newer stuff won't run on my machines anyway. Holiday time is pretty much spent for BSD conferences and events, this is where I can relax and talk with like-minded people from around the world, which is fascinating. Plus, it gets me to various places and countries I never would have dared to visit on my own. AJ: I play a few video games, and I like to ski, although I don't go very often as most of my vacation time is spent hanging out with my BSD friends at various conferences How do you relax? JT: What is this word ‘relax' and what does it mean? BR: My Tai Chi plays a big part in it I guess. I really calms you and the constant stream of thoughts for a while. It also gives you better clarity of what's important in life. Watching movies, sleeping long. AJ: Usually watching TV or Movies. Although I have taken to doing most of my TV watching on my exercise bike now, but it is still mentally relaxing If FreeBSD didn't exist, which BSD flavour would you use? Why? JT: I use TrueOS, but if FreeBSD didn't exist, that project might not either… so… My other choice would be HardenedBSD, but since it's also based on FreeBSD I'm in the same dillema. BR: I once installed NetBSD to see what It can do. If FreeBSD wouldn't exist, I would probably try my luck with it. OpenBSD is also appealing, but I've never installed it. AJ: When I started using FreeBSD in 2000, the only other BSD I had heard of at the time was OpenBSD. If FreeBSD wasn't around, I don't think the world would look like it does, so it is hard to speculate. If any of the BSD's weren't around and you had to use Linux, which camp would belong to? (Redhat, SUSE, Debian, Ubuntu, Gentoo?) JT: I learned Linux in the mid 90s using Slackware, which I used consistently up until the mid 2000s, when I joined the PuppyLinux community and eventually became a developer (FYI, Puppy was/is/can be based on Slackware -- its complicated). So I'd go back to using either Slackware or PuppyLinux. BR: I tried various Linux distributions until I landed at Debian. I used is pretty extensively as my desktop OS at home, building custom kernels and packages to install them until I discovered FreeBSD. I ran both side by side for a few months for learning until one day I figured out that I had not booted Debian in a while, so I switched completely. AJ: The first Linux I played with was Slackware, and it is the most BSD like, but the bits of Linux I learned in school were Redhat and so I can somewhat wrap my head around it, although now that they are changing everything to systemd, all of that old knowledge is more harmful than useful. Are you still finding yourself in need to use Windows/Mac OS? Why? JT: I work part time as a professional Photographer, so I do use Windows for my photography work. While I can do everything I need to do in Linux, it comes down to being pragmatic about my time. What takes me several hours to accomplish in Linux I can accomplish in 20 minutes on Windows. BR: I was a long time Windows-only user before my Unix days. But back when Vista was about to come out and I needed a new laptop, my choice was basically learning to cope with Vistas awful features or learn MacOS X. I did the latter, it increased my productivity since it's really a good Unix desktop experience (at least, back then). I only have to use Windows at work from time to time as I manage our Windows Terminal server, which keeps the exposure low enough and I only connect to it to use a certain app not available for the Mac or the BSDs. AJ: I still use Windows to play games, for a lot of video conferencing, and to produce BSD Now. Some of it could be done on BSD but not as easily. I have promised myself that I will switch to 100% BSD rather than upgrade to Windows 10, so we'll see how that goes. Please describe your home networking setup. Router type, router OS, router hardware, network segmentation, wifi apparatus(es), other devices connected, and anything else that might be interesting about your home network. BR: Very simple and boring: Apple Airport Express base station and an AVM FritzBox for DNS, DHCP, and the link to my provider. A long network cable to my desktop machine. That I use less and less often. I just bought an RPI 3 for some home use in the future to replace it. Mostly my brother's and my Macbook Pro's are connected, our phones and the iPad of my mother. AJ: I have a E3-1220 v3 (dual 3.1ghz + HT) with 8 GB of ram, and 4x Intel gigabit server NICs as my router, and it runs vanilla FreeBSD (usually some snapshot of -current). I have 4 different VLANs, Home, Office, DMZ, and Guest WiFi. WiFi is served via a tiny USB powered device I bought in Tokyo years ago, it serves 3 different SSIDs, one for each VLAN except the DMZ. There are ethernet jacks in every room wired for 10 gigabit, although the only machines with 10 gigabit are my main workstation, file server, and some machines in the server rack. There are 3 switches, one for the house (in the laundry room), one for the rack, and one for 10gig stuff. There is a rack in the basement spare bedroom, it has 7 servers in it, mostly storage for live replicas of customer data for my company. How do guys manage to get your work done on FreeBSD desktops? What do you do when you need to a Linux or Windows app that isn't ported, or working? I've made several attempts to switch to FreeBSD, but each attempt failed because of tools not being available (e.g. Zoom, Dropbox, TeamViewer, Crashplan) or broken (e.g. VirtualBox). BR: I use VIrtualBox for everything that is not natively available or Windows-only. Unfortunately, that means no modern games. I mostly do work in the shell when I'm on FreeBSD and when it has to be a graphical application, then I use Fluxbox as the DE. I want to get work done, not look at fancy eye-candy that get's boring after a while. Deactivated the same stuff on my mac due to the same reason. I look for alternative software online, but my needs are relatively easy to satisfy as I'm not doing video editing/rendering and such. AJ: I generally find that I don't need these apps. I use Firefox, Thunderbird, OpenSSH, Quassel, KomodoEdit, and a few other apps, so my needs are not very demanding. It is annoying when packages are broken, but I usually work around this with boot environments, and being able to just roll back to a version that worked for a few days until the problem is solved. I do still have access to a windows machine for the odd time I need specific VPN software or access to Dell/HP etc out-of-band management tools. Which desktop environments are your favorite, and why? For example, I like i3, Xfce, and I'm drawn to Lumina's ethos, but so far always seem to end up back on Xfc because of its ease of use, flexibility, and dashing good looks. JT: As a Lumina Desktop developer, I think my preference is obvious. ;) I am also a long timeOpenBox user, so I have a soft place in my heart for that as well. BR: I use Fluxbox when I need to work with a lot of windows or an application demands X11. KDE and others are too memory heavy for me and I rarely use even 20% of the features they provide. AJ: I was a long time KDE user, but I have adopted Lumina. I find it fast, and that it gets out of my way and lets me do what I want. It had some annoyances early on, but I've nagged the developers into making it work for me. Which command-line shells do you prefer, why, and how (if at all) have you customised the environment or prompt? BR: I use zsh, but without all the fancy stuff you can find online. It might make you more productive, yes. But again, I try to keep things simple. I'm slowly learning tmux and want to work more in it in the future. I sometimes look at other BSD people's laptops and am amazed at what they do with window-management in tmux. My prompt looks like this: bcr@Voyager:~> 20:20 17-08-17 Put this in your .zshrc to get the same result: PROMPT='%n@%m:%~>' RPROMPT='%T %D' AJ: I started using tcsh early on, because it was the shell on the first box I had access to, and because one of the first things I read in “BSD Hacks” was how to enable ‘typo correction”, which made my life a lot better especially on dial up in the early days. My shell prompt looks like this: allan@CA-TOR1-02:/usr/home/allan% What is one thing (or more) missing in FreeBSD you would import from another project or community? Could be tech, process, etc. JT: AUFS from Linux BR: Nohup from Illumos where you can detach an already running process and put it in the background. I often forget that and I'm not in tmux when that happens, so I can see myself use that feature a lot. AJ: Zones (more complete Jails) from IllumOS how do you manage your time to learn about and work on FreeBSD? Does your work/employment enable what you do, or are your contributions mainly done in private time? JT: These days I'm mostly learning things I need for work, so it just falls into something I'm doing while working on work projects. BR: We have a lot of time during the semester holidays to learn on our own, it's part of the idea of being in a university to keep yourself updated, at least for me. Especially in the fast moving world of IT. I also read a lot in my free time. My interests can shift sometimes, but then I devour everything I can find on the topic. Can be a bit excessive, but has gotten me where I am now and I still need a lot to learn (and want to). Since I work with FreeBSD at work (my owndoing), I can try out many things there. AJ: My work means a spend a lot of time working with FreeBSD, but not that much time working ON it. My contributions are mostly done outside of work, but as I own the company I do get more flexibility to take time off for conferences and other FreeBSD related stuff. we know we can bribe Michael W Lucas with gelato (good gelato that is), but what can we use to bribe you guys? Like when I want to have Allan to work on fixing a bug which prevents me from running ZFS on this fancy rock64 board? BR: Desserts of various kinds. AJ: I am probably not the right person to look at your rock64 board. Most people in the project have taken to bribing me with chocolate. In general, my todo list is so long, the best way is a trade, you take this task and I'll take that task. Is your daily mobile device iOS, Android, Windows Mobile, or other? Why? JT: These days I'm using Android on my Blackberry Priv, but until recently I was still a heavy user of Sailfish OS. I would use SailfishOS everyday, if I could find a phone with a keyboard that I could run it on. BR: iOS on the iPhone 7 currently. Never used an Android phone, saw it on other people's devices and what they can do with it (much more). But the infrequent security updates (if any at all) keep me away from it. AJ: I have a Google Nexus 6 (Android 7.1). I wanted the ‘pure' Android experience, and I had been happy with my previous Nexus S. I don't run a custom OS/ROM or anything because I use the phone to verify that video streams work on an ‘average users device'. I am displeased that support for my device will end soon. I am not sure what device I will get next, but it definitely won't be an iPhone. News Roundup Beta Update - Request for (more) Testing (http://undeadly.org/cgi?action=article&sid=20170808065718&mode=flat&count=30) https://beta.undeadly.org/ has received an update. The most significant changes include: The site has been given a less antiquated "look". (As the topic icons have been eliminated, we are no longer seeking help with those graphics.) The site now uses a moderate amount of semantic HTML5. Several bugs in the HTML fragment validator (used for submissions and comments) have been fixed. To avoid generating invalid HTML, submission content which fails validation is no longer displayed in submission/comment previews. Plain text submissions are converted to HTML in a more useful fashion. (Instead of just converting each EOL to , the converter now generates proper paragraphs and interprets two or more consecutive EOLs as indicating a paragraph break.) The redevelopment remains a work-in-progress. Many thanks to those who have contributed! As before, constructive feedback would be appreciated. Of particular interest are reports of bugs in behaviour (for example, in the HTML validator or in authentication) that would preclude the adoption of the current code for the main site. High-process-count support added to master (http://lists.dragonflybsd.org/pipermail/users/2017-August/313552.html) We've fixed a number of bottlenecks that can develop when the number of user processes runs into the tens of thousands or higher. One thing led to another and I said to myself, "gee, we have a 6-digit PID, might as well make it work to a million!". With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. And, in fact, as today's machines start to ratchet up there in both memory capacity and core count, with fast storage (NVMe) and fast networking (10GigE and higher), even in consumer boxes, this is actually something that one might want to do. With AMD's threadripper and EPYC chips now out, the IntelAMD cpu wars are back on! Boasting up to 32 cores (64 threads) per socket and two sockets on EPYC, terabytes of ram, and motherboards with dual 10GigE built-in, the reality is that these numbers are already achievable in a useful manner. In anycase, I've tested these changes on a dual-socket xeon. I can in-fact start 900,000 processes. They don't get a whole lot of cpu and running 'ps' would be painful, but it works and the system is still responsive from the shell with all of that going on. xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 In fact, judging from the memory use, these minimal test processes only eat around 60KB each. 900,000 of them ate only 55GB on a 128GB machine. So even a million processes is not out of the question, depending on the cpu requirements for those processes. Today's modern machines can be stuffed with enormous amounts of memory. Of course, our PIDs are currently limited to 6 digits, so a million is kinda the upper limit in terms of discrete user processes (verses pthreads which are less restricted). I'd rather not go to 7 digits (yet). CFT: Driver for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens (https://lists.freebsd.org/pipermail/freebsd-current/2017-August/066783.html) Following patch [1] adds support for generic MS Windows 7/8/10 - compatible USB HID multi-touch touchscreens via evdev protocol. It is intended to be a native replacement of hid-multitouch.c driver found in Linux distributions and multimedia/webcamd port. Patch is made for 12-CURRENT and most probably can be applied to recent 11-STABLE and 11.1-RELEASE (not tested) How to test" 1. Apply patch [1] 2. To compile this driver into the kernel, place the following lines into your kernel configuration file: device wmt device usb device evdev Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5): wmt_load="YES" 3. Install x11-drivers/xf86-input-evdev or x11-drivers/xf86-input-libinput port 4. Tell XOrg to use evdev or libinput driver for the device: ``` Section "ServerLayout" InputDevice "TouchScreen0" "SendCoreEvents" EndSection Section "InputDevice" Identifier "TouchScreen0" Driver "evdev" # Driver "libinput" Option "Device" "/dev/input/eventXXX" EndSection ``` Exact value of "/dev/input/eventXXX" can be obtained with evemu-record utility from devel/evemu. Note1: Currently, driver does not support pens or touchpads. Note2: wmt.ko should be kld-loaded before uhid driver to take precedence over it! Otherwise uhid can be kld-unloaded after loading of wmt. wmt review: https://reviews.freebsd.org/D12017 Raw diff: https://reviews.freebsd.org/D12017.diff *** Beastie Bits BSDMag Programing Languages Infographic (https://bsdmag.org/programm_history/) t2k17 Hackathon Report: Bob Beck on buffer cache tweaks, libressl and pledge progress (http://undeadly.org/cgi?action=article&sid=20170815171854) New FreeBSD Journal (https://www.freebsdfoundation.org/past-issues/resource-control/) NetBSD machines at Open Source Conference 2017 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2017/08/10/msg000744.html) *** Feedback/Questions Dan - HDD question (http://dpaste.com/3H6TDJV) Benjamin - scrub of death (http://dpaste.com/10F086V) Jason - Router Opinion (http://dpaste.com/2D9102K) Sohrab - Thanks (http://dpaste.com/1XYYTWF) ***
This episode gives you the full dose of BSDCan 2017 recap as well as a blog post on conference speaking advice. Headlines Pre-conference activities: Goat BoF, FreeBSD Foundation Board Meeting, and FreeBSD Journal Editorial Board Meeting The FreeBSD Foundation has a new President as Justin Gibbs is busy this year with building a house, so George Neville-Neil took up the task to serve as President, with Justin Gibbs as Secretary. Take a look at the updated Board of Directors (https://www.freebsdfoundation.org/about/board-of-directors/). We also have a new staff member (https://www.freebsdfoundation.org/about/staff/): Scott Lamons joined the Foundation team as senior program manager. Scott's work for the Foundation will focus on managing and evangelizing programs for advanced technologies in FreeBSD including preparing project plans, coordinating resources, and facilitating interactions between commercial vendors, the Foundation, and the FreeBSD community. The Foundation also planned various future activities, visits of upcoming conferences, and finding new ways to support and engage the community. The Foundation now has interns in the form of co-op students from the University of Waterloo, Canada. This is described further in the May 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/may-2017-development-projects-update/). Both students (Siva and Charlie) were also the conference, helping out at the Foundation table, demonstrating the tinderbox dashboard. Follow the detailed instructions (https://www.freebsdfoundation.org/news-and-events/blog/blog-post/building-a-physical-freebsd-build-status-dashboard/) to build one of your own. The Foundation put out a call for Project Proposal Solicitation for 2017 (https://www.freebsdfoundation.org/blog/freebsd-foundation-2017-project-proposal-solicitation/). If you think you have a good proposal for work relating to any of the major subsystems or infrastructure for FreeBSD, we'd be happy to review it. Don't miss the deadlines for travel grants to some of the upcoming conferences. You can find the necessary forms and deadlines at the Travel Grant page (https://www.freebsdfoundation.org/what-we-do/travel-grants/travel-grants/) on the Foundation website. Pictures from the Goat BoF can be found on Keltia.net (https://assets.keltia.net/photos/BSDCan-2017/Royal%20Oak/index.html) Overlapping with the GoatBoF, members of the FreeBSD Journal editorial board met in a conference room in the Novotel to plan the upcoming issues. Topics were found, authors identified, and new content was discussed to appeal to even more readers. Check out the FreeBSD Journal website (https://www.freebsdfoundation.org/journal/) and subscribe if you like to support the Foundation in that way. FreeBSD Devsummit Day 1 & 2 (https://wiki.freebsd.org/DevSummit/201706) The first day of the Devsummit began with introductory slides by Gordon Tetlow, who organized the devsummit very well. Benno Rice of the FreeBSD core team presented the work done on the new Code of Conduct, which will become effective soon. A round of Q&A followed, with positive feedback from the other devsummit attendees supporting the new CoC. After that, Allan Jude joined to talk about the new FreeBSD Community Proposal (FCP) (https://github.com/freebsd/fcp) process. Modelled after IETF RFCs, Joyent RFDs, and Python PEP, it is a new way for the project to reach consensus on the design or implementation of new features or processes. The FCP repo contains FCP#0 that describes the process, and a template for writing a proposal. Then, the entire core team (except John Baldwin, who could not make it this year) and core secretary held a core Q&A session, Answering questions, gathering feedback and suggestions. After the coffee break, we had a presentation about Intel's QAT integration in FreeBSD. When the lunch was over, people spread out into working groups about BearSSL, Transport (TCP/IP), and OpenZFS. OpenZFS working group (https://pbs.twimg.com/media/DBu_IMsWAAId2sN.jpg:large): Matt Ahrens lead the group, and spent most of the first session providing a status update about what features have been recently committed, are out for review, on the horizon, or in the design phase. Existing Features Compressed ARC Compressed Send/Recv Recently Upstreamed A recent commit improved RAID-Z write speeds by declaring writes to padding blocks to be optional, and to always write them if they can be aggregated with the next write. Mostly impacts large record sizes. ABD (ARC buffer scatter/gather) Upstreaming In Progress Native Encryption Channel Programs Device Removal (Mirrors and Stripes) Redacted Send/recv Native TRIM Support (FreeBSD has its own, but this is better and applies to all ZFS implementations) Faster (mostly sequential) scrub/resilver DRAID (A great deal of time was spent explaining how this works, with diagrams on the chalk board) vdev metadata classes (store metadata on SSDs with data is on HDDs, or similar setups. Could also be modified to do dedup to SSD) Multi-mount protection (“safe import”, for dual-headed storage shelves) zpool checkpoint (rollback an entire pool, including zfs rename and zfs destroy) Further Out Import improvements Import with missing top-level vdevs (some blocks unreadable, but might let you get some data) Improved allocator performance -- vdev spacemap log ZIL performance Persistent L2ARC ZSTD Compression Day 2 Day two started with the Have/Want/Need session for FreeBSD 12.0. A number of features that various people have or are in the process of building, were discussed with an eye towards upstreaming them. Features we want to have in time for 12.0 (early 2019) were also discussed. After the break was the Vendor summit, which continued the discussion of how FreeBSD and its vendors can work together to make a better operating system, and better products based on it After lunch, the group broke up into various working groups: Testing/CI, Containers, Hardening UFS, and GELI Improvements Allan lead the GELI Improvements session. The main thrust of the discussions was fixing an outstanding bug in GELI when using both key slots with passphrases. To solve this, and make GELI more extensible, the metadata format will be extended to allow it to store more than 512 bytes of data (currently 511 bytes are used). The new format will allow arbitrarily large metadata, defined at creation time by selecting the number of user key slots desired. The new extended metadata format will contain mostly the same fields, except the userkey will no longer be a byte array of IV-key, Data-key, HMAC, but a struct that will contain all data about that key This new format will store the number of pkcs5v2 iterations per key, instead of only having a single location to store this number for all keys (the source of the original bug) A new set of flags per key, to control some aspects of the key (does it require a keyfile, etc), as well as possibly the role of the key. An auxdata field related to the flags, this would allow a specific key with a specific flag set, to boot a different partition, rather than decrypt the main partition. A URI to external key material is also stored per key, allowing GELI to uniquely identify the correct data to load to be able to use a specific decryption key And the three original parts of the key are stored in separate fields now. The HMAC also has a type field, allowing for a different HMAC algorithm to be used in the future. The main metadata is also extended to include a field to store the number of user keys, and to provide an overall HMAC of the metadata, so that it can be verified using the master key (provide any of the user keys) Other topics discussed: Ken Merry presented sedutil, a tool for managing Self Encrypting Drives, as may be required by certain governments and other specific use cases. Creating a deniable version of GELI, where the metadata is also encrypted The work to implemented GELI in the UEFI loader was discussed, and a number of developers volunteered to review and test the code Following the end of the Dev Summit, the “Newcomers orientation and mentorship” session was run by Michael W. Lucas, which attempts to pair up first time attendees with oldtimers, to make sure they always know a few people they can ask if they have questions, or if they need help getting introduced to the right people. News Roundup Conference Day 1 (http://www.bsdcan.org/2017/schedule/day_2017-06-09.en.html) The conference opened with some short remarks from Dan Langille, and then the opening keynote by Dr Michael Geist, a law professor at the University of Ottawa where he holds the Canada Research Chair in Internet and E-commerce Law. The keynote focused on what some of the currently issues are, and how the technical community needs to get involved at all levels. In Canada especially, contacting your representatives is quite effective, and when it does not happen, they only hear the other side of the story, and often end up spouting talking points from lobbyists as if they were facts. The question period for the keynote ran well overtime because of the number of good questions the discussion raised, including how do we fight back against large telcos with teams of lawyers and piles of money. Then the four tracks of talks started up for the day The day wrapped up with the Work In Progress (WIP) session. Allan Jude presented work on ZSTD compression in ZFS Drew Gallatin presented about work at Netflix on larger mbufs, to avoid the need for chaining and to allow more data to be pushed at once. Results in an 8% CPU time reduction when pushing 90 gbps of TLS encrypted traffic Dan Langille presented about letsencrypt (the acme.sh tool specifically), and bacula Samy Al Bahra presented about Concurrency Kit *** Conference Day 2 (http://www.bsdcan.org/2017/schedule/day_2017-06-10.en.html) Because Dan is a merciful soul, BSDCan starts an hour later on the second day Another great round of talks and BoF sessions over lunch The hallway track was great as always, and I spent most of the afternoon just talking with people Then the final set of talks started, and I was torn between all four of them Then there was the auction, and the closing party *** BSDCan 2017 Auction Swag (https://blather.michaelwlucas.com/archives/2962) Groff Fundraiser Pins: During the conference, You could get a unique Groff pin, by donating more than the last person to either the FreeBSD or OpenBSD foundation Michael W. Lucas and his wife Liz donated some interesting home made and local items to the infamous Charity Auction I donated the last remaining copy of the “Canadian Edition” of “FreeBSD Mastery: Advanced ZedFS”, and a Pentium G4400 (Skylake) CPU (Supports ECC or non-ECC) Peter Hessler donated his pen (Have you read “Git Commit Murder” yet?) Theo De Raadt donated his autographed conference badge David Maxwell donated a large print of the group photo from last years FreeBSD Developers Summit, which was purchased by Allan There was also a FreeBSD Dev Summit T-Shirt (with the Slogan: What is Core doing about it?) autographed by all of the attending members of core, with a forged jhb@ signature. Lastly, someone wrote “I
We're at BSDCan, but we have an interview with Michael W. Lucas which you don't want to miss. This episode was brought to you by Headlines We are off to BSDCan but we have an interview and news roundup for you. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @mwlauthor (https://twitter.com/mwlauthor) Books, conferences & how these two combine *** News Roundup In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play (http://bsdly.blogspot.no/2012/05/in-name-of-sane-email-setting-up-spamd.html) “The Grumpy BSD Guy”, Peter Hansteen is at it again, they have produced an updated version of a full recipe for OpenBSD's spamd for your primary AND secondary mail servers Recipes in our field are all too often offered with little or no commentary to help the user understand the underlying principles of how a specific configuration works. To counter the trend and offer some free advice on a common configuration, here is my recipe for a sane mail setup. Mailing lists can be fun. Most of the time the discussions on lists like openbsd-misc are useful, entertaining or both. But when your battle with spam fighting technology ends up blocking your source of information and entertainment (like in the case of the recent thread titled "spamd greylisting: false positives" - starting with this message), frustration levels can run high, and in the process it emerged that some readers out there place way too much trust in a certain site offering barely commented recipes (named after a rare chemical compound Cl-Hg-Hg-Cl). 4 easy steps: Make sure your MXes (both primary and secondary) are able to receive mail for your domains Set set up content filtering for all MXes, since some spambots actually speak SMTP Set up spamd in front of all MXes Set up synchronization between your spamds These are the basic steps. If you want to go even further, you can supplement your greylisting and publicly available blacklists with your own greytrapping, but greytrapping is by no means required. Once you have made sure that your mail exchangers will accept mail for your domains (checking that secondaries do receive and spool mail when you stop the SMTP service on the primary), it's time to start setting up the content filtering. The post provides links if you need help getting the basic mail server functionality going At this point you will more likely than not discover that any differences in filtering setups between the hosts that accept and deliver mail will let spam through via the weakest link. Tune accordingly, or at least until you are satisfied that you have a fairly functional configuration. As you will have read by now in the various sources I cited earlier, you need to set up rules to redirect traffic to your spamd as appropriate. Now let's take a peek at what I have running at my primary site's gateway. The articles provides a few different sets of rules The setup includes running all outgoing mail through spamd to auto-populate the whitelists, allowing replies to your emails to get through without greylisting At this point, you have seen how to set up two spamds, each running in front of a mail exchanger. You can choose to run with the default spamd.conf, or you can edit in your own customizations. There is also a link to Peter's spamd.conf if you want to use “what works for me” The fourth and final required step for a spamd setup with backup mail exchangers it to set up synchronization between the spamds. The synchronization keeps your greylists in sync and transfers information on any greytrapped entries to the partner spamds. As the spamd man page explains, the synchronization options -y and -Y are command line options to spamd. The articles steps through the process of configuring spamd to listen for synchronization, and to send synchronization messages to its peer With these settings in place, you have more or less completed step four of our recipe. The article also shows you how to configure spamd to log to a separate log file, to make the messages easier to find and consolidate between your mail servers After noting the system load on your content filtering machines, restart your spamds. Then watch the system load values on the content filterers and take a note of them from time to time, say every 30 minutes or so Step 4) is the last required step for building a multi-MX configuration. You may want to just leave the system running for a while and watch any messages that turn up in the spamd logs or the mail exchanger's logs The final embellishment is to set up local greytrapping. The principle is simple: If you have one or more addresses in your domain that you know will never be valid, you add them to your list of trapping addresses any host that tries to deliver mail to noreply@mydomain.nx will be added to the local blacklist spamd-greytrap to be stuttered at for as long as it takes. Greytrapping can be fun, you can search for posts here tagged with the obvious keywords. To get you started, I offer up my published list of trap addresses, built mainly from logs of unsuccessful delivery attempts here, at The BSDly.net traplist page, while the raw list of trap email addresses is available here. If you want to use that list in a similar manner for your site, please do, only remember to replace the domain names with one or more that you will be receiving mail for. Let us know how this affects your inbox *** Beastie Bits Status of FreeBSD's capsicum on Linux (http://www.capsicum-linux.org/) How to build a gateway, from 1979 (http://www.networksorcery.com/enp/ien/ien109.txt) Linux escapee Hamza Sheikh on “Why FreeBSD?” (https://bsdmag.org/why_freebsd/) UNIX is still as relevant as ever (https://blog.opengroup.org/2012/05/17/unix-is-still-as-relevant-as-ever/) Upcoming Summer 2017 FreeBSD Foundation Events (https://www.freebsdfoundation.org/blog/upcoming-summer-2017-freebsd-foundation-events/) ***
Today on the show, we've got a look at running OpenBSD on a APU, some BSD in your Android, managing your own FreeBSD cloud service with ansible and much more. Keep it turned on your place to B...SD! This episode was brought to you by Headlines OpenBSD on PC Engines APU2 (https://github.com/elad/openbsd-apu2) A detailed walkthrough of building an OpenBSD firewall on a PC Engines APU2 It starts with a breakdown of the parts that were purchases, totally around $200 Then the reader is walked through configuring the serial console, flashing the ROM, and updating the BIOS The next step is actually creating a custom OpenBSD install image, and pre-configuring its serial console. Starting with OpenBSD 6.0, this step is done automatically by the installer Installation: Power off the APU2 Insert the bootable OpenBSD installer USB flash drive to one of the USB slots on the APU2 Power on the APU2, press F10 to get to the boot menu, and choose to boot from USB (usually option number 1) At the boot> prompt, remember the serial console settings (see above) Also at the boot> prompt, press Enter to start the installer Follow the installation instructions The driver used for wireless networking is athn(4). It might not work properly out of the box. Once OpenBSD is installed, run fw_update with no arguments. It will figure out which firmware updates are required and will download and install them. When it finishes, reboot. Where the rubber meets the road… (part one) (https://functionallyparanoid.com/2016/11/29/where-the-rubber-meets-the-road-part-one/) A user describes their adventures installing OpenBSD and Arch Linux on a new Lenovo X1 Carbon (4th gen, skylake) They also detail why they moved away from their beloved Macbook, which while long, does describe a journey away from Apple that we've heard elsewhere. The journey begins with getting a new Windows laptop, shrinking the partition and creating space for a triple-boot install, of Windows / Arch / OpenBSD Brian then details how he setup the partitioning and performed the initial Arch installation, getting it tuned to his specifications. Next up was OpenBSD though, and that went sideways initially due to a new NVMe drive that wasn't fully supported (yet) The article is split into two parts (we will bring you the next installment at a future date), but he leaves us with the plan of attack to build a custom OpenBSD kernel with corrected PCI device identifiers. We wish Brian luck, and look forward to the “rest of the story” soon. *** Howto setup a FreeBSD jail server using iocage and ansible. (https://github.com/JoergFiedler/freebsd-ansible-demo) Setting up a FreeBSD jail server can be a daunting task. However when a guide comes along which shows you how to do that, including not exposing a single (non-jailed) port to the outside world, you know we had a take a closer look. This guide comes to us from GitHub, courtesy of Joerg Fielder. The project goals seem notable: Ansible playbook that creates a FreeBSD server which hosts multiple jails. Travis is used to run/test the playbook. No service on the host is exposed externally. All external connections terminate within a jail. Roles can be reused using Ansible Galaxy. Combine any of those roles to create FreeBSD server, which perfectly suits you. To get started, you'll need a machine with Ansible, Vagrant and VirtualBox, and your credentials to AWS if you want it to automatically create / destroy EC2 instances. There's already an impressive list of Anisible roles created for you to start with: freebsd-build-server - Creates a FreeBSD poudriere build server freebsd-jail-host - FreeBSD Jail host freebsd-jailed - Provides a jail freebsd-jailed-nginx - Provides a jailed nginx server freebsd-jailed-php-fpm - Creates a php-fpm pool and a ZFS dataset which is used as web root by php-fpm freebsd-jailed-sftp - Installs a SFTP server freebsd-jailed-sshd - Provides a jailed sshd server. freebsd-jailed-syslogd - Provides a jailed syslogd freebsd-jailed-btsync - Provides a jailed btsync instance server freebsd-jailed-joomla - Installs Joomla freebsd-jailed-mariadb - Provides a jailed MariaDB server freebsd-jailed-wordpress - Provides a jailed Wordpress server. Since the machines have to be customized before starting, he mentions that cloud-init is used to do the following: activate pf firewall add a pass all keep state rule to pf to keep track of connection states, which in turn allows you to reload the pf service without losing the connection install the following packages: sudo bash python27 allow passwordless sudo for user ec2-user “ From there it is pretty straight-forward, just a couple commands to spin up the VM's either locally on your VirtualBox host, or in the cloud with AWS. Internally the VM's are auto-configured with iocage to create jails, where all your actual services run. A neat project, check it out today if you want a shake-n-bake type cloud + jail solution. Colin Percival's bsdiff helps reduce Android apk bandwidth usage by 6 petabytes per day (http://android-developers.blogspot.ca/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html) A post on the official Android-Developers blog, talks about how they used bsdiff (and bspatch) to reduce the size of Android application updates by 65% bsdiff was developed by FreeBSD's Colin Percival Earlier this year, we announced that we started using the bsdiff algorithm (by Colin Percival). Using bsdiff, we were able to reduce the size of app updates on average by 47% compared to the full APK size. This post is actually about the second generation of the code. Today, we're excited to share a new approach that goes further — File-by-File patching. App Updates using File-by-File patching are, on average, 65% smaller than the full app, and in some cases more than 90% smaller. Android apps are packaged as APKs, which are ZIP files with special conventions. Most of the content within the ZIP files (and APKs) is compressed using a technology called Deflate. Deflate is really good at compressing data but it has a drawback: it makes identifying changes in the original (uncompressed) content really hard. Even a tiny change to the original content (like changing one word in a book) can make the compressed output of deflate look completely different. Describing the differences between the original content is easy, but describing the differences between the compressed content is so hard that it leads to inefficient patches. So in the second generation of the code, they use bsdiff on each individual file, then package that, rather than diffing the original and new archives bsdiff is used in a great many other places, including shrinking the updates for the Firefox and Chrome browsers You can find out more about bsdiff here: http://www.daemonology.net/bsdiff/ A far more sophisticated algorithm, which typically provides roughly 20% smaller patches, is described in my doctoral thesis (http://www.daemonology.net/papers/thesis.pdf). Considering the gains, it is interesting that no one has implemented Colin's more sophisticated algorithm Colin had an interesting observation (https://twitter.com/cperciva/status/806426180379230208) last night: “I just realized that bandwidth savings due to bsdiff are now roughly equal to what the total internet traffic was when I wrote it in 2003.” *** News Roundup Distrowatch does an in-depth review of NAS4Free (https://distrowatch.com/weekly.php?issue=20161114#nas4free) Jesse Smith over at DistroWatch has done a pretty in-depth review of Nas4Free. The review starts with mentioning that NAS4Free works on 3 platforms, ARM/i386/AMD64 and for the purposes of this review he would be using AMD64 builds. After going through the initial install (doing typical disk management operations, such as GPT/MBR, etc) he was ready to begin using the product. One concern originally observed was that the initial boot seemed rather slow. Investigation revealed this was due to it loading the entire OS image into memory, and the first (long) disk read did take some time, but once loaded was super responsive. The next steps involved doing the initial configuration, which meant creating a new ZFS storage pool. After this process was done, he did find one puzzling UI option called “VM” which indicated it can be linked to VirtualBox in some way, but the Docs didn't reveal its secrets of usage. Additionally covered were some of the various “Access” methods, including traditional UNIX permissions, AD and LDAP, and then various Sharing services which are typical to a NAS, Such as NFS / Samba and others. One neat feature was the built-in file-browser via the web-interface, which allows you another method of getting at your data when sometimes NFS / Samba or WebDav aren't enough. Jesse gives us a nice round-up conclusion as well Most of the NAS operating systems I have used in the past were built around useful features. Some focused on making storage easy to set up and manage, others focused on services, such as making files available over multiple protocols or managing torrents. Some strive to be very easy to set up. NAS4Free does pretty well in each of the above categories. It may not be the easiest platform to set up, but it's probably a close second. It may not have the prettiest interface for managing settings, but it is quite easy to navigate. NAS4Free may not have the most add-on services and access protocols, but I suspect there are more than enough of both for most people. Where NAS4Free does better than most other solutions I have looked at is security. I don't think the project's website or documentation particularly focuses on security as a feature, but there are plenty of little security features that I liked. NAS4Free makes it very easy to lock the text console, which is good because we do not all keep our NAS boxes behind locked doors. The system is fairly easy to upgrade and appears to publish regular security updates in the form of new firmware. NAS4Free makes it fairly easy to set up user accounts, handle permissions and manage home directories. It's also pretty straight forward to switch from HTTP to HTTPS and to block people not on the local network from accessing the NAS's web interface. All in all, I like NAS4Free. It's a good, general purpose NAS operating system. While I did not feel the project did anything really amazing in any one category, nor did I run into any serious issues. The NAS ran as expected, was fairly straight forward to set up and easy to manage. This strikes me as an especially good platform for home or small business users who want an easy set up, some basic security and a solid collection of features. Browsix: Unix in the browser tab (https://browsix.org/) Browsix is a research project from the PLASMA lab at the University of Massachusetts, Amherst. The goal: Run C, C++, Go and Node.js programs as processes in browsers, including LaTeX, GNU Make, Go HTTP servers, and POSIX shell scripts. “Processes are built on top of Web Workers, letting applications run in parallel and spawn subprocesses. System calls include fork, spawn, exec, and wait.” Pipes are supported with pipe(2) enabling developers to compose processes into pipelines. Sockets include support for TCP socket servers and clients, making it possible to run applications like databases and HTTP servers together with their clients in the browser. Browsix comprises two core parts: A kernel written in TypeScript that makes core Unix features (including pipes, concurrent processes, signals, sockets, and a shared file system) available to web applications. Extended JavaScript runtimes for C, C++, Go, and Node.js that support running programs written in these languages as processes in the browser. This seems like an interesting project, although I am not sure how it would be used as more than a toy *** Book Review: PAM Mastery (https://www.cyberciti.biz/reviews/book-review-pam-mastery/) nixCraft does a book review of Michael W. Lucas' “Pam Mastery” Linux, FreeBSD, and Unix-like systems are multi-user and need some way of authenticating individual users. Back in the old days, this was done in different ways. You need to change each Unix application to use different authentication scheme. Before PAM, if you wanted to use an SQL database to authenticate users, you had to write specific support for that into each of your applications. Same for LDAP, etc. So Open Group lead to the development of PAM for the Unix-like system. Today Linux, FreeBSD, MacOS X and many other Unix-like systems are configured to use a centralized authentication mechanism called Pluggable Authentication Modules (PAM). The book “PAM Mastery” deals with the black magic of PAM. Of course, each OS chose to implement PAM a little bit differently The book starts with the basic concepts about PAM and authentication. You learn about Multi-Factor Authentication and why use PAM instead of changing each program to authenticate the user. The author went into great details about why PAM is useful for developers and sysadmin for several reasons. The examples cover CentOS Linux (RHEL and clones), Debian Linux, and FreeBSD Unix system. I like the way the author described PAM Configuration Files and Common Modules that covers everyday scenarios for the sysadmin. PAM configuration file format and PAM Module Interfaces are discussed in easy to understand language. Control flags in PAM can be very confusing for new sysadmins. Modules can be stacked in a particular order, and the control flags determine how important the success or failure of a particular module. There is also a chapter about using one-time passwords (Google Authenticator) for your application. The final chapter is all about enforcing good password policies for users and apps using PAM. The sysadmin would find this book useful as it covers a common authentication scheme that can be used with a wide variety of applications on Unix. You will master PAM topics and take control over authentication for your organization IT infrastructure. If you are Linux or Unix sysadmin, I would highly recommend this book. Once again Michael W Lucas nailed it. The only book you may need for PAM deployment. get “PAM Mastery” (https://www.michaelwlucas.com/tools/pam) *** Reflections on Trusting Trust - Ken Thompson, co-author of UNIX (http://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html) Ken Thompson's "cc hack" - Presented in the journal, Communication of the ACM, Vol. 27, No. 8, August 1984, in a paper entitled "Reflections on Trusting Trust", Ken Thompson, co-author of UNIX, recounted a story of how he created a version of the C compiler that, when presented with the source code for the "login" program, would automatically compile in a backdoor to allow him entry to the system. This is only half the story, though. In order to hide this trojan horse, Ken also added to this version of "cc" the ability to recognize if it was recompiling itself to make sure that the newly compiled C compiler contained both the "login" backdoor, and the code to insert both trojans into a newly compiled C compiler. In this way, the source code for the C compiler would never show that these trojans existed. The article starts off by talking about a content to write a program that produces its own source code as output. Or rather, a C program, that writes a C program, that produces its own source code as output. The C compiler is written in C. What I am about to describe is one of many "chicken and egg" problems that arise when compilers are written in their own language. In this case, I will use a specific example from the C compiler. Suppose we wish to alter the C compiler to include the sequence "v" to represent the vertical tab character. The extension to Figure 2 is obvious and is presented in Figure 3. We then recompile the C compiler, but we get a diagnostic. Obviously, since the binary version of the compiler does not know about "v," the source is not legal C. We must "train" the compiler. After it "knows" what "v" means, then our new change will become legal C. We look up on an ASCII chart that a vertical tab is decimal 11. We alter our source to look like Figure 4. Now the old compiler accepts the new source. We install the resulting binary as the new official C compiler and now we can write the portable version the way we had it in Figure 3. The actual bug I planted in the compiler would match code in the UNIX "login" command. The replacement code would miscompile the login command so that it would accept either the intended encrypted password or a particular known password. Thus if this code were installed in binary and the binary were used to compile the login command, I could log into that system as any user. Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions. Next “simply add a second Trojan horse to the one that already exists. The second pattern is aimed at the C compiler. The replacement code is a Stage I self-reproducing program that inserts both Trojan horses into the compiler. This requires a learning phase as in the Stage II example. First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere. So now there is a trojan'd version of cc. If you compile a clean version of cc, using the bad cc, you will get a bad cc. If you use the bad cc to compile the login program, it will have a backdoor. The source code for both backdoors no longer exists on the system. You can audit the source code of cc and login all you want, they are trustworthy. The compiler you use to compile your new compiler, is the untrustworthy bit, but you have no way to know it is untrustworthy, and no way to make a new compiler, without using the bad compiler. The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect. Acknowledgment: I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics. I can- not find a more specific reference to this document. I would appreciate it if anyone who can supply this reference would let me know. Beastie Bits Custom made Beastie Stockings (https://www.etsy.com/listing/496638945/freebsd-beastie-christmas-stocking) Migrating ZFS from mirrored pool to raidz1 pool (http://ximalas.info/2016/12/06/migrating-zfs-from-mirrored-pool-to-raidz1-pool/) OpenBSD and you (https://home.nuug.no/~peter/blug2016/) Watson.org FreeBSD and Linux cross reference (http://fxr.watson.org/) OpenGrok (http://bxr.su/) FreeBSD SA-16:37: libc (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:37.libc.asc) -- A 26+ year old bug found in BSD's libc, all BSDs likely affected -- A specially crafted argument can trigger a static buffer overflow in the library, with possibility to rewrite following static buffers that belong to other library functions. HardenedBSD issues correction for libc patch (https://github.com/HardenedBSD/hardenedBSD/commit/fb823297fbced336b6beeeb624e2dc65b67aa0eb) -- original patch improperly calculates how many bytes are remaining in the buffer. From December the 27th until the 30th there the 33rd Chaos Communication Congress[0] is going to take place in Hamburg, Germany. Think of it as the yearly gathering of the european hackerscene and their overseas friends. I am one of the persons organizing the "BSD assembly (https://events.ccc.de/congress/2016/wiki/Assembly:BSD)" as a gathering place for BSD enthusiasts and waving the flag amidst the all the other projects / communities. Feedback/Questions Chris - IPFW + Wifi (http://pastebin.com/WRiuW6nn) Jason - bhyve pci (http://pastebin.com/JgerqZZP) Al - pf errors (http://pastebin.com/3XY5MVca) Zach - Xorg settings (http://pastebin.com/Kty0qYXM) Bart - Wireless Support (http://pastebin.com/m3D81GBW) ***
This week on BSDNow, we'll be talking to Diane Bruce about using it for Ham Radio Enthusiasts, the RPi3 and much more! That plus all the latest news from the week, This episode was brought to you by Headlines PC-BSD is now TrueOS (https://www.trueos.org/2016/09/01/pc-bsd-evolves-into-trueos/) If you've been watching this show the past few months, I've been dropping little hints about the upcoming rename of PC-BSD -> TrueOS. We've made that more official finally, and are asking folks to test out the software before a wider announcement this fall. For those wondering about the name change, it's been something discussed over the past few years at different times. With us beginning to move more aggressively with changes for 11.0 (and eventually 12-CURRENT), the time seemed right to have a fresh start, using it as a spring-board to introduce all the changes in both software, and development / release model. I'll be discussing more about this shift in a talk at MeetBSD2016 (Another reason for you to go), but here's some of the highlights. No longer tied to specific FreeBSD point-releases, TrueOS will instead follow a rolling-release model based upon FreeBSD -CURRENT. Special tooling and features (Such as boot-environments) make this a feasible option that we didn't have as easily in the early days of PC-BSD. In addition, TrueOS builds some things different from vanilla FreeBSD. Specifically Matt Macy's DRM and Linux Compat work, LibreSSL directly in base, built from External Toolchain (No clang in base system package) and much more. New tools have have replaced, and are in the process of replacing the legacy PC-BSD control panel as well, which allows remote operation, either via Qt GUI, or WebSockets / REST API's. I'll be talking about more as things unfold, but for now please feel free to test and let us have feedback while we push towards a more stable release. *** The Voicemail Scammers Never Got Past Our OpenBSD Greylisting (http://bsdly.blogspot.com/2016/08/the-voicemail-scammers-never-got-past.html) Peter Hansteen (That grumpy BSD guy) gives us an interesting look at how their OpenBSD grey-listing prevented spam from ever making it to their inbox. Specifically it looks like it occurred during Aug 23rd and 24th, with a particularly nasty ransomware payload destined to play havoc with Windows systems. Peter then walks us through their three-server mail setup, and how spamd is run in greylisting mode on each. The results? Nothing short of perfection: > “From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois). The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes. There were none. The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0. Zero. Not a single one made it through even to content filtering.” Not bad at all! Looks like spam-trap addresses + grey-listing is the way to go for stopping this kind of foolishness. Checkout Peter's blog post for more details, but perhaps this will encourage you to setup a similar-type system for your business. *** FreeBSD on a tiny system; what's missing (http://adrianchadd.blogspot.com/2016/08/freebsd-on-tiny-system-whats-missing.html) Adrian Chadd talks about some of the bits that are missing to make FreeBSD truly useful on small embedded devices Some of this stuff can be done now, but requires more work than it should “The first is a lack of real service management. FreeBSD doesn't have a service management daemon - the framework assumes that daemons implement their own background and monitoring. It would be much nicer if init or something similar to init could manage services and start/restart them where appropriate.” Of course, on a system with 32mb of memory, such a service manager would need to be very light weight “maybe I want to only start telnetd or dropbear/sshd whenever a connection comes in. But I'd also like to be able to add services for monitoring, such as dnsmasq and hostapd.” telnetd and sshd can be run from inetd, but often depend on special support from the daemon “The next is a lack of suitable syslog daemon. Yes, I'd like to be able to log some messages locally - even if it's only a couple hundred kilobytes of messages. I'd also like to be able to push messages to a remote service. Unfortunately the FreeBSD syslog daemon doesn't do log rotation or maximum log file sizes itself - it's done by "newsyslog" which runs out of cron. This isn't any good for real embedded systems with limited storage.” Syslog leaves much to be desired, especially in its configuration syntax, and filtering capabilities. Having it be able to detect with log files have grown beyond a reasonable size and fire off newsyslog would be very interesting “Then yes, there's a lack of a cron service. It'd be nice to have that integrated into the service management framework so things could be easily added/removed. I may just use cron, but that means cron is also always running which adds memory footprint (~1.3 megabytes) for something that is almost never actually active. When you have 32MB of RAM, that's quite a bit of wasted memory.” Systems have come back full circle, to where 32MB and 64MB are amounts of memory people expect to work with, while other people still want the system to perform well with 32 or 64 GB of memory It will be interesting to see how this balancing act plays out, trying to make the same codebase useful for extremely small and extremely large systems at the same time, while also running it on your middle of the road laptop. *** So I lost my OpenBSD FDE password (https://blog.filippo.io/so-i-lost-my-openbsd-fde-password/) “The other day I set up a new OpenBSD instance with a nice RAID array, encrypted with Full Disk Encryption. And promptly proceeded to forget part of the passphrase.” So they started a little project Goal: “We need to extract enough info from the encrypted disk and rebuild enough of the decryption algorithm to be able to rapidly try many passphrases.” The post walks through how they reverse engineered the encryption system from the source code and a hexdump of a small encrypted memory disk “Now that we know how to extract the data and how to try passphrases against it, it will be trivial to write a bruteforce tool to recover the part of passphrase I forgot.” So, rather than having to try every possible passphrase, they only had to try fuzzing around the known keyword that was their passphrase. “UPDATE: I found it! After fixing a bug or two in the brute force tool and almost losing hope, it found the right combination of forgotten word and (Italian) misspelling.” This work lead to the author recommending that OpenBSD consider strengthening the key derivation algorithm (http://marc.info/?l=openbsd-tech&m=147316661717410&w=2) used in its FDE. Rather than using a fixed number of rounds (8000 currently), do a small benchmark and determine how much work can be done in a reasonable amount of time This is what FreeBSD's GELI FDE does, targeting ‘over 2 million microseconds' of work. On my desktop i5-3570 this results in 974842 rounds. The number will likely not be the same twice because of minor variations in how long it will take in microseconds. *** Interview - Diane Bruce - db@freebsd.org (mailto:db@freebsd.org) / @Dianora_1 (https://twitter.com/Dianora_1) Ham Radio, RPi3 and more! News Roundup See Me (Michael W. Lucas) in 2016 (http://blather.michaelwlucas.com/archives/2739) Looking for a chance to interact with author Michael W Lucas in meat-space? (That sounds wrong) If so, he has posted a list of the up-coming conferences he'll be speaking at, starting with Ohio LinuxFest Oct 7-8, where he'll be giving an introduction to ZFS talk. Nov 8th, he'll also be at MUG (Michigan User Group) giving a PAM talk. Sadly, no MeetBSD for Michael this year [moment of silence], but if you are able to make it to one of the aforementioned gatherings, be sure to bring your books for autographs. We promise he doesn't bite. Much. *** It's hard work printing nothing (http://www.tedunangst.com/flak/post/its-hard-work-printing-nothing) “It all starts with a bug report to LibreSSL that the openssl tool crashes when it tries to print NULL. This bug doesn't manifest on OpenBSD because libc will convert NULL strings to ”(null)” when printing. However, this behavior is not required, and as observed, it's not universal. When snprintf silently accepts NULL, that simply leads to propagating the error.” “There's an argument to be made that silly error messages are better than crashing browsers, but stacking layers of sand seems like a poor means of building robust software in the long term.” “As soon as development for the next release of OpenBSD restarted, some developers began testing a patch that would remove this crutch from printf.” If you'd like to help with this work, see our call for volunteers from 2 weeks ago: opportunity to help: %s audit in mandoc (https://marc.info/?l=openbsd-misc&m=147059272201219&w=2) Of course, immediately things started to complain. The configure script for talloc does a number of checks (check out the additional interesting observations by TedU here) “The test checking that our snprintf function conforms to the C99 standard actually contains, at a minimum, 3 deviations from the standard. It should say “Checking for non-conformant vsnprintf”.” “Of course, we're dealing with NULL pointers, so all bets are off, but I wonder what people who expect printf NULL to work expect out of strlen? Does it return 0? Does it crash?” So, talloc decides that the system printf is no good, and it should use its own bundled implementation “After all the configure testing, eventually the build will fail, because somebody forgot to actually add the replacement object file to the Makefile.” “If the replacement function has never been used, that's hardly reassuring that it is actually better tested than the version we have in libc.” *** Revisiting W^X with OpenBSD 6.0 (http://blog.acumensecurity.net/revisiting-wx-with-openbsd-6-0/) OpenBSD 6.0 includes enforcement of W^X in user-land This prevents an application from being able to map a page of memory with both Write and Execute permissions (protecting mmap(2)) Once mapped a page of memory should not be able to have permissions escalated (protecting mprotect(2)) OpenBSD 6.0 enforces the strict W^X definition, and not the PaX/grsec “once write never execute” type of policy *** OpenBSD imports a letsencrypt client into the base system (http://undeadly.org/cgi?action=article&sid=20160901060733) We've mentioned letskencrypt before (A native C version of the letsencrypt client, developed by Kristaps). Looks like it's undergoing a name-change to “acme-client” and has made it's way into OpenBSD's base system! This should ensure first-class support for management of Let's Encrypt certificates, here's hoping the portable version continues to thrive as well. Congrats to Kristaps! *** Beastie Bits OpenBSD: Release Songs 6.0: "Goodbye" -- no more CD releases (https://www.openbsd.org/lyrics.html#60f) FreeBSD 101 Hacks (https://nanxiao.gitbooks.io/freebsd-101-hacks/content/) LibreSSL enabled by default in HardenedBSD (https://hardenedbsd.org/article/shawn-webb/2016-08-20/libressl-enabled-default) DragonflyBSD removes last bits of 32-bit Linux emulation and has no plans to implement 64-bit linux emulation (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624241.html) OpenBSD has sent 32bit sparc to the great bitbucket in the sky (https://twitter.com/phessler/status/771277693090467840) Front Range BSD User Group September Meeting (http://slexy.org/view/s2hm4HBkb2) KnoxBug TrueOS Wrap-up (http://knoxbug.org/content/going-with-the-flow) Feedback/Questions Cody - TrueOS Questions (http://pastebin.com/mVK8G1Vr) John - FreeNAS Backups (http://pastebin.com/xsUNUfCS) Herminio - PowerPC + OpenBSD (http://pastebin.com/nHkWuNkm) Dennis - pmake vs bmake (http://pastebin.com/NAh7r6Ed) Al - Upgrade conflicts (http://pastebin.com/8HaK7yJ6) ***
Kris is on vacation this week, so allan flies solo, provides a recap of BSDCan & cover's a boatload of news including Microsoft This episode was brought to you by Headlines BSDCan Recap and Live Stream Videos (http://www.bsdcan.org/2016/) OpenBSD BSDCan 2016 papers now available (http://www.openbsd.org/papers) Allan's slides (http://allanjude.com/bsd/BSDCan2016_-_GELIBoot.pdf) and Paper (http://allanjude.com/bsd/AsiaBSDCon2016_geliboot_pdf1a.pdf) Michael W Lucas presents Allan with a gift (https://www.youtube.com/watch?v=LFgxAHkrSTg) “FreeBSD Mastery: Advanced ZedFS” (http://blather.michaelwlucas.com/archives/2698) Highlighted Tweets: Groff Arrives at BSDCan (https://twitter.com/Keltounet/status/740344735194320896) FreeBSD Foundation recognizes the contributions of Bryan Drewery, Rod Grimes, Warren Block, & Gleb Smirnoff (https://twitter.com/freebsdfndation/status/742456950676393984) A moment of silence and shots in memory in Benjamin Perrault @creepingfur (https://twitter.com/__briancallahan/status/741854476340858880) @gvnn3 sells the FreeBSD Foundation shirt off of his back for Charity (https://twitter.com/Keltounet/status/741763867471155201) Michael W. Lucas asks Matt Ahrens how to pronounce ZFS, “You can pronounce ZFS however you like, but if you pronounce it 'reiserfs', people might be confused.” (https://twitter.com/cperciva/status/741375414967410688) Sysadmin T-Shirt (https://twitter.com/BSDCan/status/741420633007874050) FreeBSD Dev Summit ran out of room on the chalkboards listing accomplishments of 11.0 (https://twitter.com/SeanChittenden/status/740904105388978176) List of things people have or want for FreeBSD 12 (https://twitter.com/Keltounet/status/740928627471159296) Matt Ahrens signing Allan's ZFS book (https://twitter.com/kprovst/status/741322268480049152?cn=bWVudGlvbg%3D%3D&refsrc=email) FreeBSD's new marketing strategy (https://twitter.com/cperciva/status/741707948469157889) Charity Auction: systemd whoopie cushion (https://twitter.com/HippyWizard/status/741768670704066560) Embarass OpenBSD's @HenningBrauer by donating $10 to charity for a selfie with him wearing a Linux t-shirt (https://twitter.com/juliefriday/status/741948048788586496) @GroffTheBSDGoat changes handlers, from @HenningBrauer to @GavinAtkinson (https://twitter.com/GroffTheBSDGoat/status/742415390798716928) Day 1 Video (https://www.youtube.com/watch?v=AOidjSS7Hsg) Day 2 Video (https://www.youtube.com/watch?v=z7pDnBO5wSM) Allan's GELIBoot talk (day 2) (https://www.youtube.com/watch?v=z7pDnBO5wSM&feature=youtu.be&list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC&t=4440) *** Media Coverage of Microsoft + FreeBSD story (https://azure.microsoft.com/en-us/blog/freebsd-now-available-in-azure-marketplace/) Microsoft has released their own custom image of FreeBSD 10.3 for the Azure Cloud “This means that not only can you quickly bring-up a FreeBSD VM in Azure, but also that in the event you need technical support, Microsoft support engineers can assist.” “Microsoft is the publisher of the FreeBSD image in the marketplace rather than the FreeBSD Foundation. The FreeBSD Foundation is supported by donations from the FreeBSD community, including companies that build their solutions on FreeBSD. They are not a solution provider or an ISV with a support organization but rather rely on a very active community that support one another. In order to ensure our customers have an enterprise SLA for their FreeBSD VMs running in Azure, we took on the work of building, testing, releasing and maintaining the image in order to remove that burden from the Foundation. We will continue to partner closely with the Foundation as we make further investments in FreeBSD on Hyper-V and in Azure.” "It's quite a significant milestone for FreeBSD community and for Microsoft to publish a supported FreeBSD image on Azure Marketplace. We really appreciate Microsoft's commitment and investment in FreeBSD project". - Justin T. Gibbs, President of FreeBSD Foundation Microsoft took a FreeBSD 10.3-RELEASE image and added additional patches, most of which they have upstreamed but that were too late for the regular 10.3 release cycle. Rather than requiring users to use a snapshot of the stable/10 branch, which would complicate the user experience, and complicate the job of the Microsoft support engineers, they created their own “certified” release This allows Microsoft to selectively deploy errata fixes to the image as well It is not clear how this affects update mechanisms like freebsd-update(8) The Register (http://www.theregister.co.uk/2016/06/09/microsoft_freebsd/) The Inquirer (http://www.theinquirer.net/inquirer/news/2461070/microsoft-creates-own-distribution-of-freebsd-for-azure-developers) Infoworld (http://www.infoworld.com/article/3082090/open-source-tools/is-microsoft-publishing-its-own-freebsd-yes-and-no.html) The Hacker News (http://thehackernews.com/2016/06/microsoft-azure-freebsd.html) Windows Report (http://windowsreport.com/microsoft-freebsd-10-3-ready-made-vm-image-azure/) Windows Club (http://news.thewindowsclub.com/microsoft-freebsd-operating-system-84375/) *** Select works poorly (http://www.tedunangst.com/flak/post/select-works-poorly) “At the bottom of the OpenBSD man page for select is a little note. “Internally to the kernel, select() and pselect() work poorly if multiple processes wait on the same file descriptor.” There's a similar warning in the poll man page. Where does this warning come from and what does it mean?” Ted found that at first glance, OpenBSD's select() appears to be quite bad: “whenever some data gets written, we call wakeup(&selwait);. Based on what we've seen so far, one can conclude that this is likely to be inefficient. Every time any socket has some data available, we wake up every selecting process in the system. Works poorly indeed.” After further investigation, it turns out to not be quite as bad When the select() is first setup, the PID of the process that cares about the FD is recorded in the selinfo struct If a second process runs select() on the same FD, the SI_COLL (Select Collision) flag is set on the selinfo struct When selwakeup() is called, if SI_COLL is set, all select()ing processes are woken up, and the sysctl kern.nselcoll is incremented. If the flag is not set, and only a single PID is waiting for activity on that FD, only that process is woken up “This is not an intractable problem. kevent avoids it entirely. Other implementations may too. But practically, does it need to be solved? My laptop says it's happened 43 times. A server with substantially more uptime says 0. Doesn't seem so bad.” *** Interview - Hans Petter Selasky - hps@freebsd.org (mailto:hps@freebsd.org) / @twitter (https://twitter.com/user) Designing FreeBSD's USB drivers, hooking up a piano to FreeBSD & more! *** News Roundup Timeline of libexpat random vulnerability (http://www.tedunangst.com/flak/post/timeline-of-libexpat-random-vulnerability) Do you use FreeBSD as web server? Why or why not? (https://news.ycombinator.com/item?id=11804565) 20 years of NetBSD code Bloat (http://kristerw.blogspot.sg/2016/05/20-years-of-netbsd-code-bloat.html) HP Chromebook 13 now booting OpenBSD (https://jcs.org/statuses/2016/06/08/740606952149942272/) UNIX for Poets (https://web.stanford.edu/class/cs124/lec/124-UnixForPoets.pdf) Comparing live version upgrade methods (https://distrowatch.com/weekly.php?issue=20160530#upgrades) My life with FreeBSD on a Thinkpad X220 (https://www.reddit.com/r/BSD/comments/4n3flx/my_life_with_freebsd_on_a_thinkpad_x220/)
It's only one-week away from BSDCan, both Allan and I are excited to meet some of you in person! However, the show keeps on This episode was brought to you by Headlines dotSecurity 2016 - Theo de Raadt - Privilege Separation and Pledge (http://www.dotsecurity.io/) Video (https://www.youtube.com/watch?v=a_EYdzGyNWs) Slides (https://www.openbsd.org/papers/dot2016.pdf) Interested in Privilege Separation and security in general? If so, then you are in for a treat, we have both the video and slides from Theo de Raadt at dotSecurity 2016. Specifically the the talk starts off looking at Pledge (no copyright issues with the pictures I hope??) and how their NTP daemon uses it. After going through some internals, Theo reveals that around 10% of programs “pledged” so far were found to be trying to do actions outside of their security scope. On the future-work side, they mention going back and looking at OpenSSH privilege separation next, as well as working with other OS's that may want pledge support. *** bhyve now supports UEFI GOP (https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-May/004471.html) The log awaited UEFI GOP (Graphics Output Protocol (https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface#GOP)) features has landed in bhyve This provides emulated graphics via an internal VNC server, allowing users to have full graphical access to the guest OS This allows installation of Windows guests without needing to create a modified ISO with an unattended installation script The code has not actually landed in FreeBSD head yet, but has been committed to a project branch Following a few simple commands, you can compile the new bhyve binary on your -CURRENT system and get started right away This feature is expected to be included in the upcoming FreeBSD 11.0 This commit drop also brings with it: XHCI -- an emulated usb tablet device that provides exact mouse positioning in supported OSs PS2 mouse for fallback if the guest does not support XHCI (Windows 7) PS2 keyboard “The code has been tested with Windows 7/8/8.1/10 and Server 2k12/2k16, Ubuntu 15.10, and FreeBSD 10.3/11-CURRENT” “For VNC clients, TightVNC, TigherVNC, and RealVNC (aka VNC Viewer) have been tested on various hosts. The OSX VNC client is known not to work.” The VNC server supports an optional ‘wait' parameter, that causes the VM to not actually boot until the VNC client connects, allowing you to interrupt the boot process if need be Related user blog post (http://justinholcomb.me/blog/2016/05/28/bhyve-uefi-gop-support.html) SVN commit (https://svnweb.freebsd.org/base?view=revision&revision=300829) *** zfsd lands in FreeBSD HEAD, in time for 11.0-RELEASE (https://svnweb.freebsd.org/base?view=revision&revision=300906) zfsd has been committed to FreeBSD -CURRENT in time to be included in FreeBSD 11.0 zfsd is the missing piece required to make ‘hot spares' work properly in FreeBSD ZFS “zfsd attempts to resolve ZFS faults that the kernel can't resolve by itself. It listens to devctl(4) events, which is how the kernel notifies of events such as I/O errors and disk removals. Zfsd attempts to resolve these faults by activating or deactivating hotspares and onlining offline vdevs.” “The administrator never interacts with zfsd directly. Instead, he controls its behavior indirectly through zpool configuration. There are two ways to influence zfsd: assigning hotspares and setting pool properties. Currently, only the autoreplace property has any effect. See zpool(8) for details.” So, what example does it do? Device Removal: “When a leaf vdev disappears, zfsd will activate any available hotspare.” Device Arrival: “When a new GEOM device appears, zfsd will attempt to read its ZFS label, if any. If it matches a previously removed vdev on an active pool, zfsd will online it. Once resilvering completes, any active hotspare will detach automatically.” So if you disconnect a drive, then reconnect it, it will automatically be brought back online. Since ZFS is smart, the resilver will only have to copy data that has changed since the device went offline. “If the new device has no ZFS label but its physical path matches the physical path of a previously removed vdev on an active pool, and that pool has the autoreplace property set, then zfsd will replace the missing vdev with the newly arrived device. Once resilvering completes, any active hotspare will detach automatically.” If the new drive is in the same slot in your hot swap array as a failed device, it will be used as a replacement immediately. vdev degrade or fault events: “If a vdev becomes degraded or faulted, zfsd will activate any available hotspare. If a leaf vdev generates more than 50 I/O errors in a 60 second period, then zfsd will mark that vdev as FAULTED. zfs(4) will no longer issue any I/Os to it. zfsd will activate a hotspare if one is available.” Same for checksum errors. So if zfsd detects a drive is going bad, it brings the hotspare online before it is too late Spare addition: “If the system administrator adds a hotspare to a pool that is already degraded, zfsd will activate the spare.” Resilver complete: “zfsd will detach any hotspare once a permanent replacement finishes resilvering.” Physical path change: “If the physical path of an existing disk changes, zfsd will attempt to replace any missing disk with the same physical path, if its pool's autoreplace property is set.” In general, this tool means less reliance on the system administrator to keep the pool healthy *** W^X now mandatory in OpenBSD (http://undeadly.org/cgi?action=article&sid=20160527203200) We've talked a bit about W^X in the past. (Refresher: Memory being writable and executable at once) Well, this major security no-no is no-more on OpenBSD. Theo has committed a change which now prevents violations of this policy: “W^X violations are no longer permitted by default. A kernel log message is generated, and mprotect/mmap return ENOTSUP. If the sysctl(8) flag kern.wxabort is set then a SIGABRT occurs instead, for gdb use or coredump creation.” There are a few cases where you may still need W^X, which Theo points out can be enabled on a file-system basis. “W^X violating programs can be permitted on a ffs/nfs filesystem-basis, using the "wxallowed" mount option. One day far in the future upstream software developers will understand that W^X violations are a tremendously risky practice and that style of programming will be banished outright. Until then, we recommend most users need to use the wxallowed option on their /usr/local filesystem. At least your other filesystems don't permit such programs.” This is a great ability to grow, since now users can begin doing auditing of programs that violate this principle and making noise to upstream. *** Interview - Kristof Provost - kp@freebsd.org (mailto:kp@freebsd.org) @kprovst (https://twitter.com/kprovst) pf improvements on FreeBSD *** News Roundup GELI Support for the EFI Loader (https://ericmccorkleblog.wordpress.com/2016/05/28/freebsd-geli-support/) We've had Allan's work to bring GELI support to the GPT / BIOS / ZFS loader for a while now, but the missing piece has been support for EFI. No longer, Eric McCorkle has posted a blog entry (with relevant github links) introducing us to his work to bring GELI encryption support to EFI. First the bad-news. This won't make it into 11.0. (Maybe PC-BSD, TBD) Next he explains why this is more than just a new feature, but a re-factor of the EFI boot code: I have already written extensively about my EFI refactoring here. The reason for undertaking this effort, however, was driven by GELI support. Early in my work on this, I had implemented a non-EFI “providers” framework in boot1 in order to support the notion of disk partitions that may contain sub-partitions. This was deeply unsatisfying to me for several reasons: It implemented a lot of the same functionality that exists in the EFI framework. It involved implementing a GPT partition driver to deal with partition tables inside GELI partitions (GPT detection and support is guaranteed by the EFI spec). The interface between the EFI framework and the custom “providers” framework was awkward. The driver was completely boot1-specific, and exporting it to something like GRUB probably involved a total rewrite. Implementing it within loader was going to involve a lot of code duplication. There was no obvious was to pass keys between boot1, loader, and the kernel. With the issues known, Eric seems pleased with the results of the conversion so far: The GELI driver can be extracted from the FreeBSD codebase without too much trouble. While I was unable to go all the way to the EFI driver model, the only blocker is the bcache code, and once that is resolved, we can have hotplug support in the boot loader! The boot1 and loader codebases are now sharing all the backend drivers, and boot1 has been reduced to one very small source file. An interesting read, looking forward to playing with EFI more in the future! *** Faces of FreeBSD 2016: Michael W. Lucas (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2016-michael-lucas/) On this edition of “Faces of FreeBSD”, Michael W Lucas tells the story of how he got started with FreeBSD After an amusing re-telling of his childhood (The words “Purina Monkey Chow” were mentioned), he then tells us how he got into BSD. His being thrown into the project may sound familiar to many: I came in at 11 PM one night and was told “The DNS administrator just got walked out the door. You're the new lead DNS administrator. Make those servers work. Good luck.” From there (because he wanted more sleep), he began ripping out the systems that had been failing and waking him up at night. Good-bye UnixWare, Good-bye Solaris, hello BSD! A very amusing read, check it out! *** High Availability with PostgreSQL on FreeBSD (https://www.youtube.com/watch?v=ugct9-Mm7Ls) A talk by Sean Chittenden, who we interviewed previously on episode Episode 95 (http://www.bsdnow.tv/episodes/2015_06_24-bitrot_group_therapy) Explains how to setup Multi Data Center High Availability for PostgreSQL using consul Goes into how consul works, how it does the election, the gossip protocol, etc The HA setup uses DNS Failover, and the pros and cons of that approach are discussed Then he walks through the implementation details, and example configuration *** New FreeBSD i915 testing images (http://www.bsddesktop.com/images/) Still need users to test the Linux Kernel 4.6 DRM update to FreeBSD's graphics stack Download the test image and write it to a USB stick and boot from it It will not modify your installed system, it runs entirely off of the USB drive Allows you to test the updated drivers without having to install the development branch on your device you can tell them that ATI/AMD support will be coming shortly and that stability has been steadily improving and that I'll do another announcement as soon as I've had a chance to test the newest Xorg bits *** Beastie Bits Comfortable on the CLI: Series Part 1 (https://www.cotcli.com/post/The-Very-Basics/) FreeBSD Booting on the Netgate uFW, a smaller-than-a-raspberry-pi dual port firewall (https://gist.github.com/gonzopancho/8e7df7a826e9a2949b36ed2a9d30312e) Picture of uFW (https://twitter.com/gonzopancho/status/737874921435594753) uFW OpenSSL Benchmarks (https://gist.github.com/gonzopancho/8f20b50487a4f7de56e99448866a147d) ***
This week on the show, we interview author Michael W Lucas to discuss his new book in the FreeBSD This episode was brought to you by Headlines OpenBSD 5.9 Released early (http://undeadly.org/cgi?action=article&sid=20160329181346&mode=expanded) Finished ahead of schedule! OpenBSD 5.9 has officially landed We've been covering some of the ongoing changes as they landed in the tree, but with the official release it's time to bring you the final list of the new hotness which landed. First up: Pledge - Over 70%! Of the userland utilities have been converted to use it, and the best part, you probably didn't even notice UEFI - Laptops which are pre-locked down to boot UEFI only can now be installed and used - GPT support has also been greatly improved ‘Less' was replaced with a fork from Illumos, and has been further improved Xen DomU support - OpenBSD now plays nice in the cloud X11 - Broadwell and Bay Trail are now supported Initial work on making the network stack better support SMP has been added, this is still ongoing, but things are starting to happen 802.11N! Specifically for the iwn/iwm drivers In addition to support for UTF-8, most other locales have been ripped out, leaving only C and UTF-8 left standing in the wake All and all, sounds like a solid new release with plenty of new goodies to play with. Go grab a copy now! *** New routing table code (ART) enabled in -current (http://undeadly.org/cgi?action=article&sid=20160324093944) While OpenBSD 5.9 just landed, we also have some interesting work landing right now in -CURRENT as well. Specifically the new routing table code (ART) has landed: “I just enabled ART in -current, it will be the default routing table backend in the next snapshots. The plan is to squash the possible regressions with this new routing table backend then when we're confident enough, take its route lookup out of the KERNEL_LOCK(). Yes, this is one of the big steps for our network SMP improvements. In order to make progress, we need your help to make sure this new backend works well on your setup. So please, go download the next snapshot and report back. If you encounter any routing table regression, please make sure that you cannot reproduce it with your old kernel and include the output of # route -n show for the 2 kernels as well as the dmesg in your report. I know that simple dhclient(8) based setups work with ART, so please do not flood us too much. It's always great to know that things work, but it's also hard to keep focus ;) Thank your very much for your support!” + There you have it folks! If 5.9 is already too stale for you, time to move over to -CURRENT and give the new routing tables a whirl. fractal cells - FreeBSD-based All-In-One solution for software development startups (https://forums.freebsd.org/threads/55561/) Fractal Cells is a suite that transforms a stock FreeBSD installation into an instant “Startup Software Development Platform” It Integrates ZFS, PostgreSQL, OpenSMTPD, NGINX, OpenVPN, Redmine, Jenkins, Zabbix, Gitlab, and Ansible, all under OpenLDAP common authentication The suite is available under the 2-clause BSD license Provides all of the tools and infrastructure to build your application, including code review, issue tracking, continuous integration, and monitoring An interesting way to make it easier for people to start building new applications and startups on top of FreeBSD *** LinuxSecrets publishes guide on installing FreeBSD ezJail (http://www.linuxsecrets.com/blog/51freebsd/2016/02/29/1726-installing) Covers all of the steps of setting up ezjail on FreeBSD Includes the instructions for updating the version of the OS in the jail In a number of places the tutorial uses: > cat > /etc/rc.conf > setting=”value” Instead, use: sysrc setting=”value” It is safer, and easier to type When you create the jail, if you specify an IP address, it is expected that this IP address is already setup on the host machine If instead you specify: ‘em0|192.168.1.105' (where em0 is your network interface), the IP address will be added as an alias when the jail starts, and removed from the host when the jail is stopped You can also comma separate a list of addresses to have multiple IPs (possibly on different interfaces) in the jail Although recently posted, this appears as if it might be an update to a previous tutorial, as there are a few old references that have not been updated (pkg_add, rc.d/ezjail.sh), while the start of the article clearly covers pkg(8) *** Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @mwlauthor (https://twitter.com/mwlauthor) + New Book: “FreeBSD Mastery: Specialty Filesystems” News Roundup NetBSD on Dreamcast (https://github.com/fwbug/dreamcast-slides) Ahh the dreamcast, so much promise. So much potential. If you are still holding onto your beloved dreamcast hoping that someday Sega will re-enter the console market… Then give it up now! In the meantime, you can now do something more interesting with that box taking up space in the closet. We have a link to a GitHub repo where a user has uploaded his curses-based slide-show for the upcoming Fort-Wayne, Indiana meetup. Aside from the novelty of using a curses-based slide setup, the presenter will also be displaying them from his beloved dreamcast, which “of course” runs NetBSD 7 The slide source code is available, which you too can view / compile and find out details of getting NetBSD boot-strapped on the DC. *** OPNsense 16.1.7 Released (https://opnsense.org/opnsense-16-1-7-released/) captive portal: add session timeout to status info firewall: fix non-report of errors when filter reload errors couldn't be parsed proxy: adjust category visibility as not all of them were shown before firmware: fix an overzealous upgrade run when the package tool only changes options firmware: fixed the binary upgrade patch from 15.7.x in FreeBSD's package tool system: removed NTP settings from general settings access: let only root access status.php as it leaks too much info development: remove the automount features development: addition of “opnsense-stable” package on our way to nightly builds development: opnsense-update can now install locally available base and kernel sets *** “FreeBSD Mastery: Advanced ZFS” in tech review (http://blather.michaelwlucas.com/archives/2570) Most of the tech review is finished It was very interesting to hear from many ZFS experts that they learned something from reading the review copy of the book, I was not expecting this Many minor corrections and clarifications have been integrated The book is now being copy edited *** Why OpenBSD? (http://www.cambus.net/why-openbsd/) Frederic Cambus gives us a nice perspective piece today on what his particular reasons are for choosing OpenBSD. Frederic is no stranger to UNIX-Like systems, having used them for 20 years now. In particular starting on Slackware back in ‘96 and moving to FreeBSD from 2000-2005 (around the 4.x series) His adventure into OpenBSD began sometime after 2005 (specific time unknown), but a bunch of things left a very good impression on him throughout the years. First, was the ease of installation, with its very minimalistic layout, which was one of the fastest installs he had ever done. Second was the extensive documentation, which extends beyond just manpages, but into other forms of documentation, such as presentations and papers as well. He makes the point about an “ecosystem of quality” that surrounds OpenBSD: OpenBSD is an ecosystem of quality. This is the result of a culture of code auditing, reviewing, and a rigorous development process where each commit hitting the tree must be approved by other developers. It has a slower evolution pace and a more carefully planned development model which leads to better code quality overall. Its well deserved reputation of being an ultra secure operating system is the byproduct of a no compromise attitude valuing simplicity, correctness, and most importantly proactivity. OpenBSD also deletes code, a lot of code. Everyone should know that removing code and keeping the codebase modern is probably as important as adding new one. Quoting Saint-Exupery: "It seems that perfection is attained not when there is nothing more to add, but when there is nothing more to remove". The article then covers security mechanisms, as well as the defaults which are turned specifically with an eye towards security. All-in-all a good perspective piece about the reasons why OpenBSD is the right choice for Frederic, worth your time to read up on it if you want to learn more about OpenBSD's differences. *** BeastieBits Call for 2016Q1 quarterly status reports (https://docs.freebsd.org/cgi/getmsg.cgi?fetch=9011+0+current/freebsd-hackers) FreeBSD Mastery: Advanced ZFS” sponsorships ending soon (http://blather.michaelwlucas.com/archives/2593) Shawn Webb from HardenedBSD talking about giving away RPi3's at BSDCan and hacking on them to get FreeBSD working (https://docs.freebsd.org/cgi/getmsg.cgi?fetch=250105+0+archive/2016/freebsd-arm/20160306.freebsd-arm) xterm(1) now UTF-8 by default (http://undeadly.org/cgi?action=article&sid=20160308204011) Call For Artists: New Icon Theme (https://blog.pcbsd.org/2016/03/call-for-artists-new-icon-theme/) Happy 23rd Birthday, src! (http://blog.netbsd.org/tnf/entry/happy_23rd_birthday_src) Feedback/Questions Alison - Readahead and Wayland (http://slexy.org/view/s2oqRuXCYW) Kenny - Gear (http://slexy.org/view/s2sQ8MxNPh) Ben - IPFW2/3 (http://slexy.org/view/s20SRvXPZA) Brad - ZFS Writeback (http://slexy.org/view/s207mV2Ph1) Simon - BSD Toonz (http://slexy.org/view/s202loSWdf) ***
This week, Allan and I are away at AsiaBSDCon! (If you aren't there, you are missing out). We will be back with a live episode next week. However, we've been asked for Allan to tell us about ScaleEngine's This episode was brought to you by Interview - Allan Jude - allanjude@freebsd.org (mailto:allanjude@freebsd.org) / @allanjude (https://twitter.com/allanjude) Spotlight on ScaleEngine *** Beastie Bits NetBSD on an RPi Zero (https://github.com/ebijun/NetBSD/blob/master/dmesg/earmv6hf/RPI0) DragonFly tips for printing with CUPS (http://lists.dragonflybsd.org/pipermail/users/2016-February/228608.html) Fighting fraudulent networks using secure connections (SSL) blacklisting with OPNsense. Blocks known-bad certificates as listed at abuse.ch (https://opnsense.org/fighting-fraudulent-networks-using-secure-connections-ssl-with-opnsense/) Fix for running NetBSD/amd64 7.0 on kvm based virtual machines (https://imil.net/blog/2016/01/29/netbsdamd64-7-0-kvm/) Michael W. Lucas's new book, FreeBSD Mastery: Specialty Filesystems is now escaping (http://blather.michaelwlucas.com/archives/2537) The Penguicon Lucas Tech Track (http://blather.michaelwlucas.com/archives/2534) FreeBSD based nginx/ffmpeg camera recording and live streaming (http://www.unixmen.com/freebsd-nginx-ffmpeg-camera-recording-and-live-streaming/) CFT: New Jenkins Builder for FreeNAS / PC-BSD (https://github.com/iXsystems/ixbuild/) Status Update: PC-BSD's SysAdm Server (https://github.com/pcbsd/sysadm/) Status Update: PC-BSD's SysAdm Client UI (https://github.com/pcbsd/sysadm-ui-qt)
An interview with Allan Jude at vBSDCon in Reston, Virginia. We talk about the book FreeBSD Mastery: ZFS that Michael W. Lucas and he co-authored. You can find more information about the book at www.zfsbook.comFile Info: 16Min, 8MB.Ogg Link: https://archive.org/download/bsdtalk256/bsdtalk256.ogg
Interview with Michael W. Lucas at BSDCan 2008. We talk about some of his books and strategies for writing technical publications.File Info: 12Min, 6MB.Ogg Link:https://archive.org/download/bsdtalk153/bsdtalk153.ogg
Interview with Michael W. Lucas at BSDCan 2013. We talk about some of his recent books including Absolute OpenBSD 2nd edition and DNSSEC Mastery.You can find him at www.michaelwlucas.comFile Info: 28 Min, 14 MB.Ogg Link: https://archive.org/download/bsdtalk228/bsdtalk228.ogg
An interview with Michael W. Lucas at BSDCan 2012. We talk about his new book "SSH Mastery", the upcoming second edition of "Absolute OpenBSD", and general thoughts about the publishing industry.File Info: 16Min, 8MB.Ogg Link:https://archive.org/download/bsdtalk218/bsdtalk218.ogg
We have a packed show for you this week! We'll sit down for an interview with Chris Buechler, from the pfSense project, to learn just how easy it can be to deploy a BSD firewall. We'll also be showing you a walkthrough of the pfSense interface so you can get an idea of just how convenient and powerful it is. Answers to your questions and the latest headlines, here on BSD Now - the place to B.. SD. This episode was brought to you by Headlines EuroBSDCon and AsiaBSDCon (http://2014.eurobsdcon.org/calendar/call-for-papers/) This year, EuroBSDCon will be in September in Sofia, Bulgaria They've got a call for papers up now, so everyone can submit the talks they want to present There will also be a tutorial section of the conference AsiaBSDCon (http://2014.asiabsdcon.org/timetable.html.en) will be next month, in March! All the info about the registration, tutorials, hotels, timetable and location have been posted Check the link for all the details on the talks - if you plan on going to Tokyo next month, hang out with Allan and Kris and lots of BSD developers! *** FreeBSD 10 on Ubiquiti EdgeRouter Lite (http://rtfm.net/FreeBSD/ERL/) The Ubiquiti EdgeRouter Lite is a router that costs less than $100 and has a MIPS CPU This article goes through the process of installing and configuring FreeBSD on it to use as a home router Lots of good pictures of the hardware and specific details needed to get you set up It also includes the scripts to create your own images if you don't want to use the ones rolled by someone else For such a cheap price, might be a really fun weekend project to replace your shitty consumer router Of course if you're more of an OpenBSD guy, you can always see our tutorial (http://www.bsdnow.tv/tutorials/openbsd-router) for that too *** Signed pkgsrc package guide (http://blog.saveosx.org/signed-packages/) We got a request on IRC for more pkgsrc stuff on the show, and a listener provided a nice write-up It shows you how to set up signed packages with pkgsrc, which works on quite a few OSes (not just NetBSD) He goes through the process of signing packages with a public key and how to verify the packages when you install them The author also happens to be an EdgeBSD developer *** Big batch of OpenBSD hackathon reports (http://undeadly.org/cgi?action=article&sid=20140212083627) Five trip reports from the OpenBSD hackathon in New Zealand! In the first one, jmatthew details his work on fiber channel controller drivers, some octeon USB work and ARM fixes for AHCI In the second (http://undeadly.org/cgi?action=article&sid=20140213065843), ketennis gets into his work with running interrupt handlers without holding the kernel lock, some SPARC64 improvements and a few other things In the third (http://undeadly.org/cgi?action=article&sid=20140213173808), jsg updated libdrm and mesa and did various work on xenocara In the fourth (http://undeadly.org/cgi?action=article&sid=20140214070023), dlg came with the intention to improve SMP support, but got distracted and did SCSI stuff instead - but he talks a little bit about the struggle OpenBSD has with SMP and some of the work he's done In the fifth (http://undeadly.org/cgi?action=article&sid=20140214130039), claudio talks about some stuff he did for routing tables and misc. other things *** Interview - Chris Buechler - cmb@pfsense.com (mailto:cmb@pfsense.com) / @cbuechler (https://twitter.com/cbuechler) pfSense Tutorial pfSense walkthrough News Roundup FreeBSD challenge continues (http://www.thelinuxcauldron.com/2014/02/13/freebsd-challenge-day-13-30/) Our buddy from the Linux foundation continues his switching to BSD journey In day 13, he covers some tips for new users, mentions trying things out in a VM first In day 14 (http://www.thelinuxcauldron.com/2014/02/14/freebsd-challenge-day-14-30/), he starts setting up XFCE and X11, feels like he's starting over as a new Linux user learning the ropes again - concludes that ports are the way to go In day 15 (http://www.thelinuxcauldron.com/2014/02/14/freebsd-challenge-day-15-30/), he finishes up his XFCE configuration and details different versions of ports with different names, as well as learns how to apply his first patch In day 16 (http://www.thelinuxcauldron.com/2014/02/17/freebsd-challenge-day-16-30/), he dives into the world of FreeBSD jails (http://www.bsdnow.tv/tutorials/jails)! *** BSD books in 2014 (http://blather.michaelwlucas.com/archives/1962) BSD books are some of the highest quality technical writings available, and MWL has written a good number of them In this post, he details some of his plans for 2014 In includes at least one OpenBSD book, at least one FreeBSD book and... Very strong possibility of Absolute FreeBSD 3rd edition (watch our interview with him (http://www.bsdnow.tv/episodes/2013_11_06-year_of_the_bsd_desktop)) Check the link for all the details *** How to build FreeBSD/EC2 images (http://www.daemonology.net/blog/2014-02-16-FreeBSD-EC2-build.html) Our friend Colin Percival (http://www.bsdnow.tv/episodes/2014_01_22-tendresse_for_ten) details how to build EC2 images in a new blog post Most people just use the images he makes on their instances, but some people will want to make their own from scratch (https://svnweb.freebsd.org/base/user/cperciva/EC2-build/) You build a regular disk image and then turn it into an AMI It requires a couple ports be installed on your system, but the whole process is pretty straightforward *** PCBSD weekly digest (http://blog.pcbsd.org/2014/02/pc-bsd-weekly-feature-digest-17/) This time around we discuss how you can become a developer Kris also details the length of supported releases Expect lots of new features in 10.1 *** Feedback/Questions Sean writes in (http://slexy.org/view/s216xJoCVG) Jake writes in (http://slexy.org/view/s2gLrR3VVf) Niclas writes in (http://slexy.org/view/s21gfG3Iho) Steffan writes in (http://slexy.org/view/s2JNyw5BCn) Antonio writes in (http://slexy.org/view/s2kg3zoRfm) Chris writes in (http://slexy.org/view/s2ZwSIfRjm) ***
This week we'll be talking to Amitai Schlair of the NetBSD foundation about pkgsrc, NetBSD's future plans and much more. After that, if you've ever wondered what all this SSH stuff is about, today's tutorial has got you covered. We'll be showing you the basics of SSH, as well as how to combine it with tmux for persistent sessions. News, feedback and everything else, right here on BSD Now - the place to B.. SD. Headlines Faces of FreeBSD (http://freebsdfoundation.blogspot.com/2013/11/faces-of-freebsd-colin-percival.html) The FreeBSD foundation is publishing articles on different FreeBSD developers This one is about Colin Percival (cperciva@), the ex-security officer Tells the story of how he first found BSD, what he contributed back, how he eventually became the security officer Running series with more to come *** Lots of BSD presentation videos uploaded (http://www.freebsdnews.net/2013/11/14/eurobsdcon-2013-devsummit-video-recordings/) EuroBSDCon 2013 dev summit videos, AsiaBSDCon 2013 videos, MWL's presentation video Most of us never get to see the dev summit talks since they're only for developers AsiaBSDCon 2013 videos also up (https://www.youtube.com/user/bsdconferences) finally List of AsiaBSDCon presentation topics here (http://2013.asiabsdcon.org/papers/index.html) Our buddy Michael W Lucas gave an "OpenBSD for Linux users" talk (http://blather.michaelwlucas.com/archives/1879) at a Michigan Unix Users Group. He says "Among other things, I compare OpenBSD to Richard Stallman and physically assault an audience member. We also talk long long time, memory randomization, PF, BSD license versus GPL, Microsoft and other OpenBSD stuff" Really informative presentation, pretty long, answers some common questions at the end *** Call for Presentations: FOSDEM 2014 and NYCBSDCon 2014 (https://blog.netbsd.org/tnf/entry/call_for_presentations_bsd_devroom) FOSDEM 2014 will take place on 1–2 February, 2014, in Brussels, Belgium Just like in the last years, there will be both a BSD booth and a developer's room The topics of the devroom include all BSD operating systems. Every talk is welcome, from internal hacker discussion to real-world examples and presentations about new and shiny features. If you are in the area or want to go, check the show notes for details NYCBSDCon is also accepting papers (http://undeadly.org/cgi?action=article&sid=20131119053455). It'll be in New York City at the beginning of February 2014 If anyone wants to give a talk at one of these conferences, go ahead and send in your stuff! *** FreeBSD foundation's year-end fundraising campaign (https://lists.freebsd.org/pipermail/freebsd-announce/2013-November/001511.html) The FreeBSD foundation has been supporting the FreeBSD project and community for over 13 years As of today they have raised about half a million dollars, but still have a while to go Donations go towards new features, paying for the server infrastructure, conferences, supporting the community, hiring full-time staff members and promoting FreeBSD at events They are preparing the debut of a new online magazine, the FreeBSD Journal Typically big companies make their huge donations in December, like a couple of anonymous donors that gave around $250,000 each last year Make your donation today (http://www.freebsdfoundation.org/donate/) over at freebsdfoundation.org, every little bit helps Everyone involved with BSD Now made a donation last year and will do so again this year *** Interview - Amitai Schlair - schmonz@netbsd.org (mailto:schmonz@netbsd.org) / @schmonz (https://twitter.com/schmonz) The NetBSD Foundation, pkgsrc, future plans Tutorial Combining SSH and tmux (http://www.bsdnow.tv/tutorials/ssh-tmux) Note: there was a mistake in the video version of the tutorial, please consult the written version for the proper instructions. *** News Roundup PS4 released (http://www.theregister.co.uk/2013/11/16/sony_playstation_4_kernel) Sony's Playstation 4 is finally released As previously thought, its OS is heavily based on FreeBSD and uses the kernel among other things Link in the show notes contains the full list of BSD software they're using (http://www.scei.co.jp/ps4-license/) Always good to see BSD being so widespread *** BSD Mag November issue (http://bsdmag.org/magazine/1853-hast-on-freebsd-how-to-make-storage-highly-availble-by-using-hast) Free monthly BSD magazine publishes another issue This time their topics include: Configuring a Highly Available Service on FreeBSD, IT Inventory & Asset Management Automation, more FreeBSD Programming Primer, PfSense and Snort and a few others PDF linked in the show notes *** pbulk builds made easy (http://mail-index.netbsd.org/pkgsrc-users/2013/11/09/msg018881.html) NetBSD's pbulk tool (https://www.netbsd.org/docs/pkgsrc/bulk.html) is similar to poudriere (http://www.bsdnow.tv/tutorials/poudriere), but for pkgsrc While working on updating the documentation, a developer cleaned up quite a lot of code He wrote a script that automates pbulk deployment and setup The whole setup of a dedicated machine has been reduced to just three commands *** PCBSD weekly digest (http://blog.pcbsd.org/2013/11/pc-bsd-weekly-feature-digest-111513/) Over 200 PBIs have been populated in to the PC-BSD 10 Stable Appcafe Many PC-BSD programs received some necessary bug fixes and updates Some include network detection in the package and update managers, nvidia graphic detection, security updates for PCDM *** Feedback/Questions Peter writes in (http://slexy.org/view/s21oh3vP7t) Kjell-Aleksander writes in (http://slexy.org/view/s21zfqcWMP) Jordan writes in (http://slexy.org/view/s2ZmW77Odb) Christian writes in (http://slexy.org/view/s2BZq7xiyo) entransic writes in (http://slexy.org/view/s21xrk0M4k) ***
On this week's show, you'll be getting the full jail treatment. We'll show you how to create and deploy BSD jails, as well as chatting with Poul-Henning Kamp - the guy who actually invented them! There's lots of interesting news items to cover as well, so stay tuned to BSD Now - the place to B.. SD. Headlines FreeBSD turns it up to 11 (https://lists.freebsd.org/pipermail/svn-src-head/2013-October/052141.html) The -CURRENT branch is now known as 11 10 has been branched to -STABLE 10-BETA1 ISOs are available now Will be the next -RELEASE, probably next year *** Stopping the SSH bruteforce with BSD and pf (http://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html) The Hail Mary Cloud is an SSH bruteforce botnet that takes a different approach While most botnets pound port 22 rapidly, THMB does it very slowly and passively This makes prevention based on rate limiting more involved and complex Nice long blog post about some potential solutions (http://home.nuug.no/~peter/pf/en/bruteforce.html) and what we've learned *** ZFS and GELI in bsdinstall coming soon (http://freshbsd.org/commit/freebsd/r256343/) The man with the beard (http://www.bsdnow.tv/episodes/2013-09-25_teskeing_the_possibilities) strikes again, new patch allows for ZFS-on-root installs Supports GELI for disk encryption Might be the push we need to make Michael W Lucas update (https://twitter.com/mwlauthor/status/389524644983611392) his FreeBSD book *** AsiaBSDCon 2014 announced (http://2014.asiabsdcon.org/) Will be held in Tokyo, 13-16 March, 2014 The conference is for anyone developing, deploying and using systems based on FreeBSD, NetBSD, OpenBSD, DragonFlyBSD, Darwin and Mac OS X Call for papers can be found here (http://2014.asiabsdcon.org/cfp.html) *** Interview - Poul-Henning Kamp - phk@freebsd.org (mailto:phk@freebsd.org) / @bsdphk (https://twitter.com/bsdphk) FreeBSD beginnings, md5crypt, jails, varnish and his... telescope project? Tutorial Everything you need to know about Jails (http://www.bsdnow.tv/tutorials/jails) News Roundup New pf queue system (http://freshbsd.org/commit/openbsd/c4661054d1882026efca919b6cd9cf958a8698b4) Henning Brauer committed the new kernel-side bandwidth shaping subsystem Uses the HFSC algorithm behind the scenes ALTQ to be retired "in a release or two" - everyone should migrate soon *** Dragonfly imports FreeBSD KMS driver (http://lists.dragonflybsd.org/pipermail/commits/2013-October/198282.html) Hot on the trails of OpenBSD and later FreeBSD, Dragonfly gets AMD KMS Ported over from the FreeBSD port *** Get paid to hack OpenSSH (https://lists.mindrot.org/pipermail/openssh-unix-dev/2013-October/031695.html) Google has announced they will pay up to $3113.70 for security patches to OpenSSH Patches can fix security or improve security If you come up with something, send it to the OpenSSH guys *** Feedback/Questions Darren writes in (http://slexy.org/view/s24RmwvEvE) Kjell-Aleksander writes in (http://slexy.org/view/s2wFcFk9Yz) Ryan writes in (http://slexy.org/view/s23e920gNG) Alexander writes in (http://slexy.org/view/s2usxPqO9k) ***
In this episode: welcome to 2008; great audiobooks by Nathan Lowell, including Quarter Share, Half Share, Full Share, and South Coast, all available at Podiobooks.com; Hacker Public Radio; the KDE Release Event at the Google Campus in Mountain View, CA; a review of Absolute FreeBSD by Michael W. Lucas.
© 2020-2025 Ivy Podcast Discovery LLC
