POPULARITY
16 episodes with DTrace
12 episodes with DTrace
3 episodes with DTrace
2 episodes with DTrace
2 episodes with DTrace
2 episodes with DTrace
3 episodes with DTrace
2 episodes with DTrace
2 episodes with DTrace
We are digging into a superpower inside your Linux Kernel. How eBPF works, and how anyone can take advantage of it.Sponsored By:Tailscale: Tailscale is a programmable networking software that is private and secure by default - get it free on up to 100 devices! 1Password Extended Access Management: 1Password Extended Access Management is a device trust solution for companies with Okta, and they ensure that if a device isn't trusted and secure, it can't log into your cloud apps. River: River is the most trusted place in the U.S. for individuals and businesses to buy, sell, send, and receive Bitcoin. Support LINUX UnpluggedLinks:
What to say when Steve Jobs threatens to sue you. Original text by Jonathan Schwartz. More about Lighthouse Design's Concurrence courtesy of the Apple Wikia instance. Sun famously sued Microsoft over their incompatible Java implenentation variant in 1997. Microsoft settled by paying Sun a bunch of money. Please enjoy this Flash animation shown at JavaOne 2004 retelling the story. Steve Jobs quotes from Triumph of the Nerds, WWDC 1997 Q&A, and Macworld San Francisco 2003. In the mid-1990s, Sun Microsystems acquired StarDivision and its StarOffice product, which Sun open sourced and renamed OpenOffice. After some entirely predictable grief from Oracle, the community forked the project and delivered what we know today as LibreOffice. Apple adopted Sun's dynamic system-wide tracing and performance profiling framework DTrace, known as Instruments in Xcode's collection of tools. Apple announced Snow Leopard Server would ship with Sun's ZFS but that ultimately never happened for licensing and patent reasons. Whether Sun's soon-to-be-acquisition by Oracle and the Steve Jobs/Larry Ellison relationship would have helped or hindered this, we'll never know. Either way, Apple, I know you're reading this and I'd like APFS to checksum my data blocks too, not just the metadata. Thank you. Jonathan Schwartz and Scott McNealy quotes from Sun's NC03-Q3 (2003) keynote and JavaOne 2004. See Project Looking Glass in action.
Apple is finishing October with a bang. Monday brought the official release of Apple Intelligence with iOS 18.1 and also for iPads and Macs. Apple also announced a new iMac with an M4 processor on Monday. Tuesday brought the new, smaller, redesigned Mac Mini with M4. What will Wednesday bring? Good chance of MacBook Pros. Plus, there's other tech news to catch up on and some tips and picks to help you tech better! Watch on YouTube! INTRO (00:00) Vote (04:10) MAIN TOPIC: Trick or Treat? Apple Intelligence and new Macs (06:50) Apple Intelligence is available today on iPhone, iPad, and Mac iOS 18.2 with new Apple Intelligence features coming in December, Apple confirms Apple updates the iMac with new colors and an M4 chip Apple announces redesigned Mac Mini with M4 chip DAVE'S PRO-TIP OF THE WEEK: Select the camera type by long long-holding camera icon in the Control Center (22:20) JUST THE HEADLINES: (27:30) Britain to axe up to 1.5m lampposts Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said NASA reveals prototype telescope for gravitational wave observatory San Francisco's Muni's rail system will spend $212 million to upgrade from floppy disks Hacker returns $19.3 million to drained US government crypto wallet DTrace for Linux comes to Gentoo TAKES McDonald's restaurants finally have a solution to their busted McFlurry machine problem (29:20) Astropad Launches 'Bookcase' for Turning an iPhone Into an E-Reader (31:30) Vimeo brings spatial video sharing to everyone with new Apple Vision Pro app (35:00) Astronauts return from nearly eight months on ISS after Starliner problems (36:45) Adobe made its painting app completely free to take on Procreate (38:00) BONUS ODD TAKE: Kasso - Bonkers Japanese Skateboarding Show (39:45) PICKS OF THE WEEK: Dave: Mic Cover Foam Microphone Windscreen, Condenser Microphone Size A, 1 Pack (41:05) Nate: SmallRig 51" Cell Phone Selfie Stick Tripod with Wireless Remote, Aluminum Portable Cellphone Tripod Freely 10 Meters Bluetooth Remote Control Tripod for iPhone 16 Pro Max/15/14 Pro/14/13, Samsung S22/ S23 Ultra/Android Smartphone via TikTok (43:20) RAMAZON PURCHASE - Giveaway! (50:25)
This week Robie Basak joins Noah from the Ubuntu Summit and gives an introduction on how to get started contributing to Ubuntu. -- During The Show -- 01:26 HexOS - Craig Start with the Command Line When is a GUI appropriate Start with make a ZFS pool make a samba share Help us understand your goal What is HexOS Ubuntu and ZFS DKMS kABI Advantages of TrueNAS Snapshots Send/Receive 1 Click Re-silvering 17:08 Questions about HDMI switch - Andy Theater Receiver Decimator (https://www.amazon.com/Decimator-DMON-QUAD-SD-SDI-Multi-Viewer-Outputs/dp/B072NGFDMR) 21:14 News Wire SQLite 3.47.0 - sqlite.org (https://sqlite.org/releaselog/3_47_0.html) Peazip 10 - github.io (https://peazip.github.io) Jellyfin 10.10.0 - jellyfin.org (https://jellyfin.org/posts/jellyfin-release-10.10.0/) EasyOS 6.4 - puppylinux.com (https://forum.puppylinux.com/viewtopic.php?t=12973) Gnome 47.1 - gnome.org (https://discourse.gnome.org/t/gnome-47-1-released/24670) Tor Browser 14.0 - torproject.org (https://blog.torproject.org/new-release-tor-browser-140/) AlmaLinux Kitten 10 - almalinux.org (https://almalinux.org/blog/2024-10-22-introducing-almalinux-os-kitten/) Gentoo & DTrace 2.0 - gentoo.org (https://www.gentoo.org/news/2024/10/23/DTrace-for-Gentoo.html) NASA $15.6M Grant for Open Source Tools - spaceanddefense.io (https://spaceanddefense.io/nasa-awards-15-6-million-in-open-source-software-funding/) Open Source Printable Lathe - hackaday.com (https://hackaday.com/2024/10/23/a-3d-printed-open-source-lathe/) Thelio Astra - system76.com (https://system76.com/desktops/thelio-astra) Eight Nvidia High Severity Vulnerabilities - forbes.com (https://www.forbes.com/sites/daveywinder/2024/10/25/urgent-new-nvidia-security-warning-for-200-million-linux-and-windows-gamers/) OpenSSL 3.4 - github.com (https://github.com/openssl/openssl/releases/tag/openssl-3.4.0) IPS Snort v3.5 - github.com (https://github.com/snort3/snort3/releases) Parrot OS 6.2 - parrotsec.org (https://parrotsec.org/blog/2024-10-23-parrot-6.2-release-notes/) New Granite 3.0 - zdnet.com (https://www.zdnet.com/article/ibm-doubles-down-on-open-source-ai-with-new-granite-3-0-models/) HUGS - reuters.com (https://www.reuters.com/technology/startup-hugging-face-aims-cut-ai-costs-with-open-source-offering-2024-10-23/) SynthID Now Open Source - theverge.com (https://www.theverge.com/2024/10/23/24277873/google-artificial-intelligence-synthid-watermarking-open-source) Mochi 1 - venturebeat.com (https://venturebeat.com/ai/video-ai-startup-genmo-launches-mochi-1-an-open-source-model-to-rival-runway-kling-and-others/) Ubuntu Turns 20 - ubuntu.com (https://ubuntu.com/20years) 23:23 Robie Basak - Ubuntu Technical Council What drew you to Linux? Why did you decide to work for Canonical? What is the Ubuntu Technical Board? Difference between Ubuntu and Canonical The process of granting commit rights Conflict resolution Cloud init Unique ID Ubuntu Summit Range of interaction Membership Board Meeting Full Hour Long Meeting Recording YouTube (https://www.youtube.com/live/pyRcIZskKNE?si=frx3zrPhUoeLrHi8) 43:40 Fedora 41 Fedora 41 available early! New DNF bootc Plasma Mobile Spin (https://fedoramagazine.org/announcing-fedora-linux-41/) Fedora Magazine (https://fedoramagazine.org/announcing-fedora-linux-41/) Minisforum v3 (https://store.minisforum.com/products/minisforum-v3?) Steve, Fedora, hardware 50:30 Russian Kernel Maintainers Removed Greg Kroah-Hartman removed them due to "various compliance requirements" Removed developers Russian and not minor contributors We live in a world where decisions are made for political reasons zdnet.com (https://www.zdnet.com/article/why-remove-russian-maintainers-of-linux-kernel-heres-what-torvalds-says/) therecord.media (https://therecord.media/russia-separate-linux-community-kernel-maintainers-delisted) -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/413) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed)
George Cozma of Chips and Cheese joined Bryan, Adam, and the Oxide Friends to talk about AMD's new 5th generation EPYC processor, codename: Turin. What's new in Turin and how is Oxide's Turin-based platform coming along?In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest George Cozma, as well as Oxide colleagues Robert Mustacchi, Eric Aasen, Nathanael Huffman, and the quietly observant Aaron Hartwig.Some of the topics we hit on, in the order that we hit them:Chips and Cheese: AMD's Turin: 5th Gen EPYC LaunchedEnd of the Road: An Anandtech FarewellCentaur TechnologyAVX-512Zen5's AVX512 Teardown + More...Thermal Power Design (TDP)OxF: Rack Scale Networking (use of p4)P4AGESAOxF: The Network Behind the Network (Oxide server recovery)openSILphoronix: openSILPCB backdrillingOxF: AMD's MI300 (APUs)dtrace.conf(24) -- The DTrace unconference, December 11th, 2024If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Bryan Cantrill, Co-founder and CTO of Oxide Computer Company, joins Adam to share his journey from Sun to Oxide – from Sun and Fishworks, to DTrace, to ZFS, to Joyent and Node.js, and now working to build on-prem cloud servers as they should be at Oxide.
Bryan Cantrill, Co-founder and CTO of Oxide Computer Company, joins Adam to share his journey from Sun to Oxide – from Sun and Fishworks, to DTrace, to ZFS, to Joyent and Node.js, and now working to build on-prem cloud servers as they should be at Oxide.
This week Noah and Steve give you the latest on open source mobile operating systems. Windows is pushing people to Linux by way of charging them $244 per year for updates in the third year. -- During The Show -- 01:20 Hardware Protectli (https://protectli.com/) $170 Chinesium Device (https://www.newegg.com/p/22Z-007C-009Y8?Item=9SIAK3UJNH8968) Lenovo Thunderbolt Dock (https://www.amazon.com/ZoomSpeed-Universal-Thunder-40B00300US-DisplayPort/dp/B0B1T7PPGZ) CalDigit Thunderbolt Dock (https://www.caldigit.com/thunderbolt-station-4/) 09:19 Graphene OS - Craig GrapheneOS Moving away from phones JMP.Chat (https://jmp.chat/) Conversations (https://f-droid.org/en/packages/eu.siacs.conversations/) Gajim (https://gajim.org/) Linphone (https://linphone.org/) JMP.Chat phone service LineageOS (https://lineageos.org/) PostmarketOS (https://postmarketos.org/) SailfishOS (https://sailfishos.org/) Titan M chip Had pretty good luck on ebay NitroKey/NitroPhone (https://shop.nitrokey.com/shop?&search=nitrophone) 25:55 NixOS Thoughts - Jeremy W Where NixOS is useful "Productised" NixOS Set people up for success 32:58 Self Hosted Email - Jeremy H Write in and tell me about your self hosted email experiences 34:54 News Wire German State Moving to Linux - ARSTechnica (https://arstechnica.com/information-technology/2024/04/german-state-gov-ditching-windows-for-linux-30k-workers-migrating/) Kodi 21.0 - Kodi (https://kodi.tv/article/kodi-21-0-omega-release/) Nitrux - nxos.org (https://nxos.org/changelog/release-announcement-nitrux-3-4-0/) Ubuntu 24.04 Delayed - Toms Hardware (https://www.tomshardware.com/software/linux/ubuntu-2404-beta-delayed-due-to-malicious-code-in-xz-utils-other-linux-distros-are-also-affected) EndeavourOS ARM Discontinued - EndeavourOS (https://endeavouros.com/news/goodbye-endeavouros-arm/) Linux 6.7 EOL - Server Host (https://serverhost.com/blog/end-of-life-for-linux-kernel-6-7-urgent-call-for-users-to-upgrade-to-6-8/) QT Creator 13 - QT (https://www.qt.io/blog/qt-creator-13-released) FFmpeg 7.0 - FFmpeg (https://ffmpeg.org//index.html#pr7.0) Dtrace 2.0 - Phoronix (https://www.phoronix.com/news/D-Trace-2.0.0-1.14) AURORA-M - Mark Tech Post (https://www.marktechpost.com/2024/04/07/aurora-m-a-15b-parameter-multilingual-open-source-ai-model-trained-in-english-finnish-hindi-japanese-vietnamese-and-code/) Gretel AI Text-to-SQL - Mark Tech Post (https://www.marktechpost.com/2024/04/04/gretel-ai-releases-largest-open-source-text-to-sql-dataset-to-accelerate-artificial-intelligence-ai-model-training/) Viking Model Family - Mark Tech Post (https://www.marktechpost.com/2024/04/07/silo-ai-releases-new-viking-model-family-pre-release-an-open-source-llm-for-all-nordic-languages-english-and-programming-languages/) Framework Hiring - Phoronix (https://www.phoronix.com/news/Framework-OSS-Firmware-Hiring) 36:26 Bell Canada is deleting DVR/PVR recordings Steve hates Bell Canada Marriage photographer story HDCP (https://en.wikipedia.org/wiki/High-bandwidth_Digital_Content_Protection) HDCP Stripper (https://www.amazon.com/THWT-HDMI-EDID-Emulator-Model/dp/B0CRRWQ7XS) OREI HDMI splitter (https://www.amazon.com/THWT-HDMI-EDID-Emulator-Model/dp/B0CRRWQ7XS) 43:03 Windows Upgrades/Updates Windows 10 EOL October 2025 Extended Security Updates (ESUs) Microsoft will charge for updates ARS Technica (https://arstechnica.com/gadgets/2024/04/post-2025-windows-10-updates-for-businesses-start-at-61-per-pc-go-up-from-there/) 46:06 Germany Switching to Linux Linux solves "Windows high hardware requirements" Schleswig-Holstein developing open source directory service LiMux from Munich Steve's take Lxer.com (lxer.com/module/newswire/ext_link.php?rid=339628) ARS Technica (https://arstechnica.com/information-technology/2024/04/german-state-gov-ditching-windows-for-linux-30k-workers-migrating/) 51:10 American Privacy Rights Act Fusion Centers (lxer.com/module/newswire/ext_link.php?rid=339628) IQT (https://www.iqt.org/) Tax dollars funding data collection companies The Register (https://www.theregister.com/2024/04/09/us_federal_privacy_law_apra/) -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/384) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed)
Data visualization is an important--and overlooked!--tool in the software engineer's tool belt. Bryan describes a recent journey with gnuplot while Oxide colleague, Charlie Park, shares his own experience with data visualization and Adam offers a visual analysis of Simpsons episodes. Stay tuned to the end to find out about the Oxide and Friends book club coming up in May.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide Colleague, Charlie Park.(00:00) - Intro (13:39) - OODA (22:30) - Back to Bryan (24:27) - Flame Graphs (28:58) - Statemap (32:39) - Minard / Tufte (44:53) - thingskatedid (46:39) - DTrace aggregations (56:06) - ParaView (01:03:08) - Simpsons IMDb (01:05:16) - Survivorship Bias (01:15:03) - Kartlytics (01:18:15) - Kartlytics sample group (01:19:11) - Wrapping up (01:22:02) - OxF book club Some of the topics we hit on, in the order that we hit them:Bryan's rad gnuplotGitHub PR with Bryan's visualizationsTuftePronunciation of "Tufte" is /ˈtʌfti/Flame Graphsflamegraph-rsOODAThis American Life: A Little Bit of KnowledgeStatemapsMinard's diagramhttps://twitter.com/thingskatedid/status/1386077306381242371plot.awkVisualizing regular expressions and BNF grammars with GraphvizExample implementations of isvg and idotDTrace aggregationsRust crate ratatuiPrograms and libraries for plotting and other data visualizations:gnuplotMatplotlibggplot2ParaViewGLVisSimpsons IMDB visualizationAbraham Wald and the airplane diagram with red bullet holes – here's the origin storyKartlyticsHow Life Works by Philip BallIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Adam and the Oxide Friends follow Bryan on Mr. Nagle's Wild Ride as he investigates performance anomalies. Bryan used all manner of tool from gnuplot to DTrace-inspired bpftrace! If you have ever or plan to ever care about the latency of network-borne protocols, you won't want to miss this!We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from October 2nd, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers included Tom Lyon, James Tucker, Eliza Weisman, and Dan Ports.Some of the topics we hit on, in the order that we hit them: Latency Art: X marks the spot Latency Art: Rainbow Pterodactyl Nagle on Nagle Dan's tweet on Nagle Eliza's tweet on Nagle TCP_NODELAY or TCP? No, delay! Dr. Angela Collier on violin plots PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Bryan and Adam reminisce about the DTrace journey 20 years after first integrating the code into Solaris back in September 2003.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Josh Clulow.Some of the topics we hit on, in the order that we hit them: Hungry Jack's Bryan's other online dating profile The Sun E10000 (E10k), the world's worst router Leventhal's Conundrum DTrace as Half-Life 3, eternal vaporware More on SPARC, it's TLB, the %npc, and dtrace_fish from OxF May 2021 Solaris 9 was the completion of the Solaris 2.0 vision DTrace Kernel Technical Discussion (2002) Mr. Sparkle Firefox? Mozilla? Firebird! Dynamic Instrumentation of Production Systems, Usenix 2004 Graydon on DTrace in Rust Rust USDT crate DTrace on Windows Adam's blog: DTrace on macOS Adam's blog: DTrace for OEL PSARC cases from 2003 If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
We have a new show host, Understanding ZFS vdev Types, Don't abuse su for dropping user privileges, Dynamic Tracing on OpenBSD 7.3, new Libressl, Manual Jails on FreeBSD 12, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Host Introductions - Jason Tubnor - https://www.tubsta.com (https://www.tubsta.com) / @tubsta (https://twitter.com/tubsta) / @Tubsta@soc.feditime.com (https://soc.feditime.com) Headlines Understanding ZFS vdev Types (https://klarasystems.com/articles/openzfs-understanding-zfs-vdev-types/) Don't abuse su for dropping user privileges (https://jdebp.uk/FGA/dont-abuse-su-for-dropping-privileges.html) News Roundup Dynamic Tracing on OpenBSD 7.3 (https://blog.lambda.cx/posts/openbsd-dynamic-tracing/) new Libressl (https://undeadly.org/cgi?action=article;sid=20230528115900) Manual Jails on FreeBSD 12 (https://ogris.de/howtos/freebsd-jails.html) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Chris - questions (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/513/feedback/Chris%20-%20questions.md) Dan - zfs questions (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/513/feedback/Dan%20-%20zfs%20questions.md) Pablo - Jail question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/513/feedback/Pablo%20-%20Jail%20question.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
FreeBSD 13.2 Release, Using DTrace to find block sizes of ZFS, NFS, and iSCSI, Midnight BSD 3.0.1, Closing a stale SSH connection, How to automatically add identity to the SSH authentication agent, Pros and Cons of FreeBSD for virtual Servers, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines FreeBSD 13.2 Release Announcement (https://www.freebsd.org/releases/13.2R/announce/) Using DTrace to find block sizes of ZFS, NFS, and iSCSI (https://axcient.com/blog/using-dtrace-to-find-block-sizes-of-zfs-nfs-and-iscsi/) News Roundup Midnight BSD 3.0.1 (https://www.phoronix.com/news/MidnightBSD-3.0.1) Closing a stale SSH connection (https://davidisaksson.dev/posts/closing-stale-ssh-connections/) How to automatically add identity to the SSH authentication agent (https://sleeplessbeastie.eu/2023/04/10/how-to-automatically-add-identity-to-the-ssh-authentication-agent/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Dan - ZFS question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/504/feedback/Dan%20-%20ZFS%20question.md) Matt - Thanks (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/504/feedback/Matt%20-%20Thanks.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
Bryan and Adam and the Oxide Friends take on GPT and its implications for software engineering. Many aspiring programmers are concerned that the future of the profession is in jeopardy. Spoiler: the Oxide Friends see a bright future for human/GPT collaboration in software engineering.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 20th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers on MM DD included Josh Clulow, Keith Adams, Ashley Williams, and others. (Did we miss your name and/or get it wrong? Drop a PR!)Live chat from the show (lightly edited): ahl: John Carmack's tweet ahl: ...and the discussion Wizord: https://twitter.com/balajis/status/1636797265317867520 (the $1M bet on BTC, I take) dataphract: "prompt engineering" as in "social engineering" rather than "civil engineering" Grevian: I was surprised at how challenging getting good prompts could be, even if I wouldn't quite label it engineering TronDD: https://www.aiweirdness.com/search-or-fabrication/ MattCampbell: I tested ChatGPT in an area where I have domain expertise, and it got it very wrong. TronDD: Also interesting https://www.youtube.com/watch?v=jPhJbKBuNnA Wizord: the question is, when will it be in competition with people? Wizord: copilot also can review code and find bugs if you ask it in a right way ag_dubs: i suspect that a new job will be building tools that help make training sets better and i strongly suspect that will be a programming job. ai will need tools and data and content and there's just a whole bunch of jobs to build tools for AI instead of people Wizord: re "reading manual and writing DTrace scripts" I think it's possible, if done with a large enough token window. Wizord: (there are already examples of GPT debugging code, although trivial ones) flaviusb: The chat here is really interesting to me, as it seems to miss the point of the thing. ChatGPT does not and can not ever 'actually work' - and whether it works is kind of irrelevant. Like, the Jaquard Looms and Numerical Control for machining did not 'work', but that didn't stop the roll out. Columbus: Maybe it has read the dtrace manual
Bryan and Adam are joined by a number of members of the Oxide networking team to talk about the networking software that drives the Oxide rack. It turns out that rack-scale networking is hard... and has enormous benefits!We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 27th, 2023.In addition to Bryan Cantrill and Adam Leventhal, speakers included Ryan Goodfellow, Levon Tarver, Ben Naecker, and Arjen Roodselaar.Links Intel Tofino Series P4 (programming language) - Wikipedia p4lang/p4c: P4_16 reference compiler oxidecomputer/p4: A P4 compiler The quote crate: Rust quasi-quoting RIFT WG - Routing In Fat Trees | IETF Community Wiki Here's (much of) the live chat from the show: ahl https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.md ahl That's the Sidecar switch episode bcantrill https://p4.org/ admchl What does "at line rate" mean? Riking Line rate = As fast as the packets could possibly come. 1Gbit, 10Gbit, 100Gbit, etc admchl Do you need ASICs to hit that speed? I assume x86_64 is not going to be fast enough for these specialised operations? levon Yes, the Tofino 2 is the ASIC bcantrill You need ASICs bnaecker Yes, you really can't do these kinds of operations on a general purpose CPU. rng_drizzt Yeah, you need specialized silicon here. JustinAzoff Right, also often across all ports at the same time in both direction. a 48 port 10gbps switch will have a line rate of 960gbps (10 ** 48 ** 2) duckman So the advantage is being able to offload compute to the switch? bnaecker Yes, and specifically that you can separate the data plane (operations on the packets) from the control plane (decisions about what operations to allow or make). tahnok What's TCAM? levon Ternary Content Addressable Memory bnaecker https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMs ryaeng Sure beats logging into a number of Cisco switches and making changes at the console. admchl This is my favourite episode in a long time, this is all really fascinating. rng_drizzt the first Sidecar episode was nearly 1.5 years ago ü§Ø , right after we cut the first rev levon That episode blew my mind duckman This sounds like a big deal on the scale of ebpf duckman Or bigger bnaecker It is extremely useful for understanding the processing pipelines. As long as you only run single-packet integration tests üôÇ od0 just want to go out and find things to write P4 code for JustinAzoff yeah one way to think about that sort of thing is that xdp can be used to run little programs on a nic, where p4 is kind of like that, but running on effectively a nic with 48+ ports bcantrill https://github.com/oxidecomputer/p4 SyntheticGate sidecar is the "codename" of our switch box SyntheticGate "gimlet" is our server sled bcantrill https://github.com/oxidecomputer/propolis wmf So you have P4 and OPTE in the hypervisor at the same time? bnaecker OPTE is in the host kernel. arjenroodselaar The P4 runtime Ry described only exists in the test bed, where it high level simulates the switches. OPTE is part of the production environment. arjenroodselaar The rough difference between P4 and OPTE is that P4 works on individual packets without much concept of a session (so it can't reason about TCP streams, packet order etc, so no firewall like functionality), while OPTE aims to operate on streams of packets. JustinAzoff So you can run 100 VMs on a test system and wire them up to your virtual switch compiled by x4c? arjenroodselaar Correct. bcantrill OPTE == Oxide Packet Transformation Engine admchl Gimlet? rng_drizzt Compute server rng_drizzt The Sidecar switch is actually just a PCIe peripheral to a Gimlet. bnaecker The Gimlet managing the Sidecar is often called a "Scrimlet" for "Sidecar attached Gimlet" Riking and "how do i reconfigure this giant network without hosing my ability to reconfigure this giant network" ShaunO can identify with that - we seriously struggle to keep our own products inter-operating, let alone anyone else's levon It can feel like a Sisyphean task. a172 Setup a much smaller/simpler network in parallel that is accessible from "not your network" that gets you to the management interface. levon It's a whole new world when you can look at the actual table definitions in P4 rng_drizzt Owning all the layers here is immensely beneficial levon Those DTrace probes have been very helpful bnaecker Those probes turned out to be everywhere. They are are in: SQL queries, HTTP queries, log messages, Propolis hypervisor state, virtual storage system, networking protocol messages, the P4 emulator, and probably more that I'm forgetting about. levon For those unfamiliar with the DTrace tool, or the rationale behind leveraging DTrace over other tracing / debugging tools: https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdf bcantrill https://github.com/oxidecomputer/progenitor ahl some notes on rust codegen: https://github.com/ahl/codegen-template arjenroodselaar DDM! Bring us home! a172 it astonishes me how many "cloud" type architectures are built on v4 only or v4 first. a172 IPv6 is older than Wi-Fi a172 It solves real problems. PLEASE use it. nyanotech yessss finally someone realizes broadcast domains are also failure domains JustinAzoff the worst part of v6 is trying to run dual stack v4+v6, v6 only networks are fairly simple levon And the bigger the broadcast domain, the more irritating it is to troubleshoot it bcantrill "Hash and pray" arjenroodselaar FWIW while DDM is a cool thing we're building, one of the "simple" tasks Tofino does for us is NAT between the networks of our customers and their VPC networks they implement on our platform. arjenroodselaar Simple NAT is still surprisingly expensive and being able to do that at line rate is pretty nice. Riking TCP retransmits in steady state seems like an obvious observation point? arjenroodselaar Yes, you see TCP retransmits. arjenroodselaar But if you're running say Memcache over UDP and you get a sudden burst of incoming data as a result of a large number of cache queries you drop those packets (because the buffers can't keep up) and you see cache request timeouts. arjenroodselaar FB did some work on this about 10 years ago to avoid this ingest and dropped packets which hurt your p99 latency. Riking yeah smartnic is pushing the intelligence to the machine levon I know someone who basically polled all of the switches for buffer drops in an attempt to divine which paths were dropping packets due to micro-congestion admchl I feel like I'm in a secret society meeting learning The Hidden Truth behind Reality of The Network wmf I would argue if the entire hypervisor is on the smart NIC then you're no worse off than the Oxide architecture a172 I once stumbled on a bug where the vendor's custom protocol for monitoring (because snmp/syslog just cant keep up) had a trace log on the process, that could not be turned off. Some sort of race condition enabled it, and it happened on 1/3 of system boots. It was ~20k logs/s, iirc. a172 (im going to look up those numbers) levon I haven't worked with a SmartNIC fast enough to do this well JustinAzoff We use a FPGA Nic in our products for fast packet capturing. the service that bootstraps it had an issue that caused it to log an error... for every single packet... JustinAzoff that managed to log the same error something like 250,000 times a second arjenroodselaar The problem with SmartNICs is that their power features are way less advanced than the power scaling that x86 CPUs do. So you either run them or you don't, and they come with a 50-75W penalty. Unless you can really get useful work done for that 50W budget, a x86 CPU is much more flexible. arjenroodselaar What we really want is an AMD Epyc SoC with some amount of FPGA fabric That would let you build whatever makes sense there while still having much of the flexibility with respect to how/where you consume power. a172 It was enough to mess us up. 250k would have killed us even faster. JustinAzoff Yeah, it happily wrote that error message until the multi TB data array filled up. We reworked how log rate limiting and log rotation worked after that a172 I was mostly amused that the process that the process that existed because snmp/syslog couldn't keep up was getting a syslog for every iteration of a loop in the process a172 of course, if you are sending a packet for every packet you send, that sounds like it quickly becomes an exponential problem. JustinAzoff and to circle back around, this was code inside of the vendor SDK, that is not open source, that we couldn't fix ourselves. it's one of the only components of our system that we don't control. i wish we had our own NIC (that would probably run something like p4) levon And thus, this is how we become the way we are (at Oxide) a172 ours was on production network hardware (wireless controller). There is no hope of having source or any insight true observability into it. (edit: saying there was no insight is a little harsh) JustinAzoff one thing that came up before was if p4 was like ebpf.. there's actually a ebpf backend for p4 that supports some of the features: https://github.com/p4lang/p4c/blob/main/backends/ebpf/README.md bcantrill Thanks, all! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Today we have a special treat: Bryan Cantrill, co-founder and CTO of Oxide Computer! You may know Bryan from his work on DTrace. He worked at Sun for many years, then Oracle, and finally Joyent before starting Oxide. We dig deep into their company's mission/principles/values, hear how it it all started with a VC's blank check that turned out to be anything but, and learn how Oxide's integrated approach to hardware & software sets them up to compete with the established players by building servers as they should be.
Today we have a special treat: Bryan Cantrill, co-founder and CTO of Oxide Computer! You may know Bryan from his work on DTrace. He worked at Sun for many years, then Oracle, and finally Joyent before starting Oxide. We dig deep into their company's mission/principles/values, hear how it it all started with a VC's blank check that turned out to be anything but, and learn how Oxide's integrated approach to hardware & software sets them up to compete with the established players by building servers as they should be.
About LizLiz Rice is Chief Open Source Officer with cloud native networking and security specialists Isovalent, creators of the Cilium eBPF-based networking project. She is chair of the CNCF's Technical Oversight Committee, and was Co-Chair of KubeCon + CloudNativeCon in 2018. She is also the author of Container Security, published by O'Reilly.She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, and competing in virtual races on Zwift.Links: Isovalent: https://isovalent.com/ Container Security: https://www.amazon.com/Container-Security-Fundamental-Containerized-Applications/dp/1492056707/ Twitter: https://twitter.com/lizrice GitHub: https://github.com/lizrice Cilium and eBPF Slack: http://slack.cilium.io/ CNCF Slack: https://cloud-native.slack.com/join/shared_invite/zt-11yzivnzq-hs12vUAYFZmnqE3r7ILz9A TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the interesting things about hanging out in the cloud ecosystem as long as I have and as, I guess, closely tied to Amazon as I have been, is that you learned that you never quite are able to pronounce things the way that people pronounce them internally. In-house pronunciations are always a thing. My guest today is Liz Rice, the Chief Open Source Officer at Isovalent, and they're responsible for, among other things, the Cilium open-source project, which is around eBPF, which I can only assume is internally pronounced as ‘Ehbehpf'. Liz, thank you for joining me today and suffering my pronunciation slings and arrows.Liz: I have never heard ‘Ehbehpf' before, but I may have to adopt it. That's great.Corey: You also are currently—in a term that is winding down if I'm not misunderstanding—you were the co-chair of KubeCon and CloudNativeCon at the CNCF, and you are also currently on the technical oversight committee for the foundation.Liz: Yeah, yeah. I'm currently the chair, in fact, of the technical oversight committee.Corey: And now that Amazon has joined, I assumed that they had taken their horrible pronunciation habits, like calling AMIs ‘Ah-mies' and whatnot, and started spreading them throughout the ecosystem with wild abandon.Liz: Are we going to have to start calling CNCF ‘Ka'Nff' or something?Corey: Exactly. They're very frugal, by which I mean they never buy a vowel. So yeah, it tends to be an ongoing challenge. Joking and all the rest aside, let's start, I guess, at the macro view. The CNCF does an awful lot of stuff, where if you look at the CNCF landscape, for example, like, I think some of my jokes on the internet go a bit too far, but you look at this thing and last time I checked, there were something like four or 500 different players in various spaces.And it's a very useful diagram, don't get me wrong by any stretch of the imagination, but it also is one of those things that is so staggeringly vast that I've got a level with you on this one, given my old, ancient sysadmin roots, “The hell with it. I'm going to run some VMs in a three-tiered architecture just like grandma and grandpa used to do,” and call it good. Not really how the industry is evolved, but it's overwhelming.Liz: But that might be the right solution for your use case so, you know, don't knock it if it works.Corey: Oh, yeah. If it's a terrible architecture and it works, is it really that terrible of an architecture? One wonders.Liz: Yeah, yeah. I mean, I'm definitely not one of those people who thinks, you know, every solution has the same—you know, is solved by the same hammer, you know, all problems are not the same nail. So, I am a big fan of a lot of the CNCF projects, but that doesn't mean to say I think those are the only ways to deploy software. You know, there are plenty of things like Lambda are a really great example of something that is super useful and very applicable for lots of applications and for lots of development teams. Not necessarily the right solution for everything. And for other people, they need all the bells and whistles that something like Kubernetes gives them. You know, horses for courses.Corey: It's very easy for me to make fun of just about any company or service or product, but the thing that always makes me set that aside and get down to brass tacks has been, “Okay, great. You can build whatever you want. You can tell whatever glorious marketing narrative you wish to craft, but let's talk to a real customer because once we do that, then if you're solving a problem that someone is having in the wild, okay, now it's no longer just this theoretical exercise and PowerPoint. Now, let's actually figure out how things work when the rubber meets the road.”So, let's start, I guess, with… I'll leave it to you. Isovalent are the creators of the Cilium eBPF-based networking project.Liz: Yeah.Corey: And eBPF is the part of that I think I'm the most familiar with having heard the term. Would you rather start on the company side or on the eBPF side?Liz: Oh, I don't mind. Let's—why don't we start with eBPF? Yeah.Corey: Cool. So easy, ridiculous question. I know that it's extremely important because Brendan Gregg periodically gets on stage and tells amazing stories about this; the last time he did stuff like that, I went stumbling down into the rabbit hole of DTrace, and I have never fully regretted doing that, nor completely forgiven him. What is eBPF?Liz: So, it stands for extended Berkeley Packet Filter, and we can pretty much just throw away those words because it's not terribly helpful. What eBPF allows you to do is to run custom programs inside the kernel. So, we can trigger these programs to run, maybe because a network packet arrived, or because a particular function within the kernel has been called, or a tracepoint has been hit. There are tons of places you can attach these programs to, or events you can attach programs to.And when that event happens, you can run your custom code. And that can change the behavior of the kernel, which is, you know, great power and great responsibility, but incredibly powerful. So Brendan, for example, has done a ton of really great pioneering work showing how you can attach these eBPF programs to events, use that to collect metrics, and lo and behold, you have amazing visibility into what's happening in your system. And he's built tons of different tools for observing everything from, I don't know, memory use to file opens to—there's just endless, dozens and dozens of tools that Brendan, I think, was probably the first to build. And now this sort of new generations of eBPF-based tooling that are kind of taking that legacy, turning them into maybe more, going to say user-friendly interfaces, you know, with GUIs, and hooking them up to metrics platforms, and in the case of Cilium, using it for networking and hooking it into Kubernetes identities, and making the information about network flows meaningful in the context of Kubernetes, where things like IP addresses are ephemeral and not very useful for very long; I mean, they just change at any moment.Corey: I guess I'm trying to figure out what part of the stack this winds up applying to because you talk about, at least to my mind, it sounds like a few different levels all at once: You talk about running code inside of the kernel, which is really close to the hardware—it's oh, great. It's adventures in assembly is almost what I'm hearing here—but then you also talk about using this with GUIs, for example, and operating on individual packets to run custom programs. When you talk about running custom programs, are we talking things that are a bit closer to, “Oh, modify this one field of that packet and then call it good,” or are you talking, “Now, we launch Microsoft Word.”Liz: Much more the former category. So yeah, let's inspect this packet and maybe change it a bit, or send it to a different—you know, maybe it was going to go to one interface, but we're going to send it to a different interface; maybe we're going to modify that packet; maybe we're going to throw the packet on the floor because we don't—there's really great security use cases for inspecting packets and saying, “This is a bad packet, I do not want to see this packet, I'm just going to discard it.” And there's some, what they call ‘Packet of Death' vulnerabilities that have been mitigated in that way. And the real beauty of it is you just load these programs dynamically. So, you can change the kernel or on the fly and affect that behavior, just immediately have an effect.If there are processes already running, they get instrumented immediately. So, maybe you run a BPF program to spot when a file is opened. New processes, existing processes, containerized processes, it doesn't matter; they'll all be detected by your program if it's observing file open events.Corey: Is this primarily used from a security perspective? Is it used for—what are the common use cases for something like this?Liz: There's three main buckets, I would say: Networking, observability, and security. And in Cilium, we're kind of involved in some aspects of all those three things, and there are plenty of other projects that are also focusing on one or other of those aspects.Corey: This is where when, I guess, the challenge I run into the whole CNCF landscape is, it's like, I think the danger is when I started down this path that I'm on now, I realized that, “Oh, I have to learn what all the different AWS services do.” This was widely regarded as a mistake. They are not Pokémon; I do not need to catch them all. The CNCF landscape applies very similarly in that respect. What is the real-world problem space for which eBPF and/or things like Cilium that leverage eBPF—because eBPF does sound fairly low-level—that turn this into something that solves a problem people have? In other words, what is the problem that Cilium should be the go-to answer for when someone says, “I have this thing that hurts.”Liz: So, at one level, Cilium is a networking solution. So, it's Kubernetes CNI. You plug it in to provide connectivity between your applications that are running in pods. Those pods have to talk to each other somehow and Cilium will connect those pods together for you in a very efficient way. One of the really interesting things about eBPF and networking is we can bypass some of the networking stack.So, if we are running in containers, we're running our applications in containers in pods, and those pods usually will have their own networking namespace. And that means they've got their own networking stack. So, a packet that arrives on your machine has to go through the networking stack on that host machine, go across a virtual interface into your pod, and then go through the networking stack in that pod. And that's kind of inefficient. But with eBPF, we can look at the packet the moment it's come into the kernel—in fact in some cases, if you have the right networking interfaces, you can do it while it's still on the network interface card—so you look at that packet and say, “Well, I know what pod that's destined for, I can just send it straight there.” I don't have to go through the whole networking stack in the kernel because I already know exactly where it's going. And that has some real performance improvements.Corey: That makes sense. In my explorations—we'll call it—with Kubernetes, it feels like the universe—at least at the time I went looking into it—was, “Step One, here's how to wind up launching Kubernetes to run a blog.” Which is a bit like using a chainsaw to wind up cutting a sandwich. Okay, massively overpowered but I get the basic idea, like, “Okay, what's project Step Two?” It's like, “Oh, great. Go build Google.”Liz: [laugh].Corey: Okay, great. It feels like there's some intermediary steps that have been sort of glossed over here. And at the small-scale that I kicked the tires on, things like networking performance never even entered the equation; it was more about get the thing up and running. But yeah, at scale, when you start seeing huge numbers of containers being orchestrated across a wide variety of hosts that has serious repercussions and explains an awful lot. Is this the sort of thing that gets leveraged by cloud providers themselves, is it something that gets built in mostly on-prem environments, or is it something that rides in, almost, user-land for most of these use cases that customers coming to bringing to those environments? I'm sorry, users, not customers. I'm too used to the Amazonian phrasing of everyone as a customer. No, no, they are users in an open-source project.Liz: [laugh]. Yeah, so if you're using GKE, the GKE Dataplane V2 is using Cilium. Alibaba Cloud uses Cilium. AWS is using Cilium for EKS Anywhere. So, these are really, I think, great signals that it's super scalable.And it's also not just about the connectivity, but also about being able to see your network flows and debug them. Because, like you say, that day one, your blog is up and running, and day two, you've got some DNS issue that you need to debug, and how are you going to do that? And because Cilium is working with Kubernetes, so it knows about the individual pods, and it's aware of the IP addresses for those pods, and it can map those to, you know, what's the pod, what service is that pod involved with. And we have a component of Cilium called Hubble that gives you the flows, the network flows, between services. So, you know, we've probably all seen diagrams showing Service A talking to Service B, Service C, some external connectivity, and Hubble can show you those flows between services and the outside world, regardless of how the IP addresses may be changing underneath you, and aggregating network flows into those services that make sense to a human who's looking at a Kubernetes deployment.Corey: A running gag that I've had is that one of the drawbacks and appeals of Kubernetes, all at once, is that it lets you cosplay as a cloud provider, even if you don't happen to work for one of them. And there's a bit of truth to it, but let's be serious here, despite what a lot of the cloud providers would wish us to believe via a bunch of marketing, there's a tremendous number of data center environments out there, hybrid environments, and companies that are in those environments are not somehow laggards, or left behind technologically, or struggling to digitally transform. Believe it or not—I know it's not a common narrative—but large companies generally don't employ people who lack critical thinking skills and strategic insight. There's usually a reason that things are the way that they are and when you don't understand that my default approach is that, oh context that gets missing, so I want to preface this with the idea there is nothing wrong in those environments. But in a purely cloud-native environment—which means that I'm very proud about having no single points of failure as I have everything routing to a single credit card that pays the cloud providers—great. What is the story for Cilium if I'm using, effectively, the managed Kubernetes options that Name Any Cloud Provider will provide for me these days? Is it at that point no longer for me or is it something that instead expresses itself in ways I'm not seeing, yet?Liz: Yeah, so I think, as an open-source project—and it is the only CNI that's at incubation level or beyond, so you know, it's CNCF-supported networking solution; you can use it out of the box, you can use it for your tiny blog application if you've decided to run that on Kubernetes, you can do so—things start to get much more interesting at scale. I mean, that… continuum between you know, there are people purely on managed services, there are people who are purely in the cloud, hybrid cloud is a real thing, and there are plenty of businesses who have good reasons to have some things in their own data centers, something's in the public cloud, things distributed around the world, so they need connectivity between those. And Cilium will solve a lot of those problems for you in the open-source, but also, if you're telco scale and you have things like BGP networks between your data centers, then that's where the paid versions of Cilium, the enterprise versions of Cilium, can help you out. And, as Isovalent, that's our business model to have, like—we fully support or we contribute a lot of resources into the open-source Cilium, and we want that to be the best networking solution for anybody, but if you are an enterprise who wants those extra bells and whistles, and the kind of scale that, you know, a telco, or a massive retailer, or a large media organization, or name your vertical, then we have solutions for that as well. And I think it was one of the really interesting things about the eBPF side of it is that, you know, we're not bound to just Kubernetes, you know? We run in the kernel, and it just so happens that we have that Kubernetes interface for allocating IP addresses to endpoints that happened to be pods. But—Corey: So, back to my crappy pile of VMs—because the hell with all this newfangled container nonsense—I can still benefit from something like Cilium?Liz: Exactly, yeah. And there's plenty of people using it for just load-balancing, which, why not have an eBPF-based high-performance load balancer?Corey: Hang on, that's taking me a second to work my way through. What is the programming language for eBPF? It is something custom?Liz: Right. So, when you load your BPF program into the kernel, it's in the form of eBPF bytecode. There are people who write an eBPF bytecode by hand; I am not one of those people.Corey: There are people who used to be able to write Sendmail configs without running through the M four preprocessor, and I don't understand those people either.Liz: [laugh]. So, our choices are—well, it has to be a language that can be compiled into that bytecode, and at the moment, there are two options: C, and more recently, Rust. So, the C code, I'm much more familiar with writing BPF code in C, it's slightly limited. So, because these BPF programs have to be safe to run, they go through a verification process which checks that you're not going to crash the kernel, that you're not going to end up in some hardware loop, and basically make your machine completely unresponsive, we also have to know that BPF programs, you know, they'll only access memory that they're supposed to and that they can't mess up other processes. So, there's this BPF verification step that checks for example that you always check that a pointer isn't nil before you dereference it.And if you try and use a pointer in your C code, it might compile perfectly, but when you come to load it into the kernel, it gets rejected because you forgot to check that it was non-null before.Corey: You try and run it, the whole thing segfaults, you see the word ‘fault' there and well, I guess blameless just went out the window there.Liz: [laugh]. Well, this is the thing: You cannot segfault in the kernel, you know, or at least that's a bad [day 00:19:11]. [laugh].Corey: You say that, but I'm very bad with computers, let's be clear here. There's always a way to misuse things horribly enough.Liz: It's a challenge. It's pretty easy to segfault if you're writing a kernel module. But maybe we should put that out as a challenge for the listener, to try to write something that crashes the kernel from within an eBPF because there's a lot of very smart people.Corey: Right now the blood just drained from anyone who's listening, in the kernel space or the InfoSec space, I imagine.Liz: Exactly. Some of my colleagues at Isovalent are thinking, “Oh, no. What's she brought on here?” [laugh].Corey: What have you done? Please correct me if I'm misunderstanding this. So, eBPF is a very low-level tool that requires certain amounts of braining in order [laugh] to use appropriately. That can be a heavy lift for a lot of us who don't live in those spaces. Cilium distills this down into something that is all a lot more usable and understandable for folks, and then beyond that, you wind up with Isovalent, that winds up effectively productizing and packaging this into something that becomes a lot more closer to turnkey. Is that directionally accurate?Liz: Yes, I would say that's true. And there are also some other intermediate steps, like the CLI tools that Brendan Gregg did, where you can—I mean, a CLI is still fairly low-level, but it's not as low-level as writing the eBPF code yourself. And you can be quite in-dep—you know, if you know what things you want to observe in the kernel, you don't necessarily have to know how to write the eBPF code to do it, but if you've got these fairly low-level tools to do it. You're absolutely right that very few people will need to write their own… BPF code to run in the kernel.Corey: Let's move below the surface level of awareness; the same way that most of us don't need to know how to compile our own kernel in this day and age.Liz: Exactly.Corey: A few people very much do, but because of their hard work, the rest of us do not.Liz: Exactly. And for most of us, we just take the kernel for granted. You know, most people writing applications, it doesn't really matter if—they're just using abstractions that do things like open files for them, or create network connections, or write messages to the screen, you don't need to know exactly how that's accomplished through the kernel. Unless you want to get into the details of how to observe it with eBPF or something like that.Corey: I'm much happier not knowing some of the details. I did a deep dive once into Linux system kernel internals, based on an incredibly well-written but also obnoxiously slash suspiciously thick O'Reilly book, Linux Systems Internalsand it was one of those, like, halfway through, “Can I please be excused? My brain is full.” It's one of those things that I don't use most of it on a day-to-day basis, but it's solidified by understanding of what the computer is actually doing in a way that I will always be grateful for.Liz: Mmm, and there are tens of millions of lines of code in the Linux kernel, so anyone who can internalize any of that is basically a superhero. [laugh].Corey: I have nothing but respect for people who can pull that off.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.In your day job, quote-unquote—which is sort of a weird thing to say, given that you are working at an open-source company; in fact, you are the Chief Open Source Officer, so what you're doing in the community, what you're exploring on the open-source project side of things, it is all interrelated. I tend to have trouble myself figuring out where my job starts and stops most weeks; I'm sympathetic to it. What inspired you folks to launch a company that is, “Ah, we're going to be in the open-source space?” Especially during a time when there's been a lot of pushback, in some respects, about the evolution of open-source and the rise of large cloud providers, where is open-source a viable strategy or a tactic to get to an outcome that is pleasing for all parties?Liz: Mmm. So, I wasn't there at the beginning, for the Isovalent journey, and Cilium has been around for five or six years, now, at this point. I very strongly believe in open-source as an effective way of developing technology—good technology—and getting really good feedback and, kind of, optimizing the speed at which you can innovate. But I think it's very important that businesses don't think—if you're giving away your code, you cannot also sell your code; you have to have some other thing that adds value. Maybe that's some extra code, like in the Isovalent example, the enterprise-related enhancements that we have that aren't part of the open-source distribution.There's plenty of other ways that people can add value to open-source. They can do training, they can do managed services, there's all sorts of different—support was the classic example. But I think it's extremely important that businesses don't just expect that I can write a bunch of open-source code, and somehow magically, through building up a whole load of users, I will find a way to monetize that.Corey: A bunch of nerds will build my product for me on nights and weekends. Yeah, that's a bit of an outmoded way of thinking about these things.Liz: Yeah exactly. And I think it's not like everybody has perfect ability to predict the future and you might start a business—Corey: And I have a lot of sympathy for companies who originally started with the idea of, “Well, we are the project leads. We know this code the best, therefore we are the best people in the world to run this as a service.” The rise of the hyperscale cloud providers has called that into significant question. And I feel for them because it's difficult to completely pivot your business model when you're already a publicly-traded company. That's a very fraught and challenging thing to do. It means that you're left with a bunch of options, none of them great.Cilium as a project is not that old, neither is Isovalent, but it's new enough in the iterative process, that you were able to avoid that particular pitfall. Instead, you're looking at some level of making this understandable and useful to humans, almost the point where it disappears from their level of awareness that they need to think about. There's huge value in something like that. Do you think that there is a future in which projects and companies built upon projects that follow this model are similarly going to be having challenges with hyperscale cloud providers, or other emergent threats to the ecosystem—sorry, ‘threat' is an unfair and unkind word here—but changes to the ecosystem, as we see the world evolving in ways that most of us did not foresee?Liz: Yeah, we've certainly seen some examples in the last year or two, I guess, of companies that maybe didn't anticipate, and who necessarily has a crystal ball to anticipate how cloud providers might use their software? And I think in some cases, the cloud providers has not always been the most generous or most community-minded in their approach to how they've done that. But I think for a company, like Isovalent, our strong point is talent. It would be extremely rare to find the level of expertise in, you know, what is a pretty specialized area. You know, the people at Isovalent who are working on Cilium are also working on eBPF itself, and that level of expertise is, I think, pretty unrivaled.So, we're in such a new space with eBPF, we've only in the last year or so, got to the point where pretty much everyone is running a kernel that's new enough to use eBPF. Startups do have a kind of agility that I think gives them an advantage, which I hope we'll be able to capitalize on. I think sometimes when businesses get upset about their code being used, they probably could have anticipated it. You know, if it's open-source, people will use your software, and you have to think of that.Corey: “What do you mean you're using the thing we gave away for free and you're not paying us to use it?”Liz: Yeah.Corey: “Uh, did you hear what you just said?” Some of this was predictable, let's be fair.Liz: Yeah, and I think you really have to, as a responsible business, think about, well, what does happen if they use all the open-source code? You know, is that a problem? And as far as we're concerned, everybody using Cilium is a fantastic… thing. We fully welcome everyone using Cilium as their data plane because the vast majority of them would use that open-source code, and that would be great, but there will be people who need that extra features and the expertise that I think we're in a unique position to provide. So, I joined Isovalent just about a year ago, and I did that because I believe in the technology, I believe in the company, I believe in, you know, the foundations that it has in open-source.It's a very much an open-source first organization, which I love, and that resonates with me and how I think we can be successful. So, you know, I don't have that crystal ball. I hope I'm right, we'll find out. We should do this again, you know, a couple of years and see how that's panning out. [laugh].Corey: I'll book out the date now.Liz: [laugh].Corey: Looking back at our conversation just now, you talked about open-source, and business strategy and how that's going to be evolving. We talked about the company, we talked about an incredibly in-depth, technical product that honestly goes significantly beyond my current level of technical awareness. And at no point in any of those aspects of the conversation did you talk about it in a way that I did not understand, nor did you come off in any way as condescending. In fact, you wrote an O'Reilly book on Container Security that's written very much the same way. How did you learn to do that? Because it is, frankly, an incredibly rare skill.Liz: Oh, thank you. Yeah, I think I have never been a fan of jargon. I've never liked it when people use a complicated acronym, or really early days in my career, there was a bit of a running joke about how everything was TLAs. And you think, well, I understand why we use an acronym to shorten things, but I don't think we need to assume that everybody knows what everything stands for. Why can't we explain things in simple language? Why can't we just use ordinary terms?And I found that really resonates. You know, if I'm doing a presentation or if I'm writing something, using straightforward language and explaining things, making sure that people understand the, kind of, fundamentals that I'm going to build my explanation on. I just think that has a—it results in people understanding, and that's my whole point. I'm not trying to explain something to—you know, my goal is that they understand it, not that they've been blown away by some kind of magic. I want them to go away going, “Ah, now I understand how this bit fits with that bit,” or, “How this works.” You know?Corey: The reason I bring it up is that it's an incredibly undervalued skill because when people see it, they don't often recognize it for what it is. Because when people don't have that skill—which is common—people just write it off as oh, that person's a bad communicator. Which I think is a little unfair. Being able to explain complex things simply is one of the most valuable yet undervalued skills that I've found in this entire space.Liz: Yeah, I think people sometimes have this sort of wrong idea that vocabulary and complicated terms are somehow inherently smarter. And if you use complicated words, you sound smarter. And I just don't think that's accessible, and I don't think it's true. And sometimes I find myself listening to someone, and they're using complicated terms or analogies that are really obscure, and I'm thinking, but could you explain that to me in words of one syllable? I don't think you could. I think you're… hiding—not you [laugh]. You know, people—Corey: Yeah. No, no, that's fair. I'll take the accusation as [unintelligible 00:31:24] as I can get it.Liz: [laugh]. But I think people hide behind complex words because they don't really understand them sometimes. And yeah, I would rather people understood what I'm saying.Corey: To me—I've done it through conference talks, but the way I generally learn things is by building something with them. But the way I really learn to understand something is I give a conference talk on it because, okay, great. I can now explain Git—which was one of my early technical talks—to folks who built Git. Great. Now, how about I explain it to someone who is not immersed in the space whatsoever? And if I can make it that accessible, great, then I've succeeded. It's a lot harder than it looks.Liz: Yeah, exactly. And one of the reasons why I enjoy building a talk is because I know I've got a pretty good understanding of this, but by the time I've got this talk nailed, I will know this. I might have forgotten it in six months time, you know, but [laugh] while I'm giving that talk, I will have a really good understanding of that because the way I want to put together a talk, I don't want to put anything in a talk that I don't feel I could explain. And that means I have to understand how it works.Corey: It's funny, this whole don't give talks about things you don't understand seems like there's really a nouveau concept, but here we are, we're [working on it 00:32:40].Liz: I mean, I have committed to doing talks that I don't fully understand, knowing that—you know, with the confidence that I can find out between now and the [crosstalk 00:32:48]—Corey: I believe that's called a forcing function.Liz: Yes. [laugh].Corey: It's one of those very high-risk stories, like, “Either I'm going to learn this in the next three months, or else I am going to have some serious egg on my face.”Liz: Yeah, exactly, definitely a forcing function. [laugh].Corey: I really want to thank you for taking so much time to speak with me today. If people want to learn more, where can they find you?Liz: So, I am online pretty much everywhere as lizrice, and I am on Twitter. I'm on GitHub. And if you want to come and hang out, I am on the Cilium and eBPF Slack, and also the CNCF Slack. Yeah. So, come say hello.Corey: There. We will put links to all of that in the [show notes 00:33:28]. Thank you so much for your time. I appreciate it.Liz: Pleasure.Corey: Liz Rice, Chief Open Source Officer at Isovalent. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment containing an eBPF program that on every packet fires off a Lambda function. Yes, it will be extortionately expensive; almost half as much money as a Managed NAT Gateway.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk is about how I use several tools, technologies and processes to troubleshoot ClicHouse Performance. I will be talking about multiple Linux Toolkits, Trace Profilers like DTrace, BPF etc. and also ClickHouse System Tables. This talk also covers best practices/checklist / run-book for building "High Performance ClickHouse Infrastructure Operations" BIO Open Source Database Systems Geek in MySQL, MariaDB, PostgreSQL and ClickHouse with core expertise in performance, scalability, high availability and database reliability engineering, Shiv currently is the founder and principal of MinervaDB Inc., an enterprise-class 24*7 Consultative Support and Managed Services Provider for MySQL, MariaDB and PostgreSQL. Shiv also is the Founder and Principal of ChistaDATA Inc., an independent 24*7 Consultative Support and Managed Service Provider for ClickHouse. Shiv in the past worked for companies like MySQL AB, SUN Microsystems, AOL, eBay, PayPal, PalominoDB and Percona. Shiv also is a frequent speaker in open source conferences worldwide. KEY TAKE-AWAYS FROM THE TALK Troubleshooting ClickHouse Performance intuitively
About AnnaAnna has nearly ten years of experience researching and advising organizations on cloud adoption with a focus on security best practices. As a Gartner Analyst, Anna spent six years helping more than 500 enterprises with vulnerability management, security monitoring, and DevSecOps initiatives. Anna's research and talks have been used to transform organizations' IT strategies and her research agenda helped to shape markets. Anna is the Director of Thought Leadership at Sysdig, using her deep understanding of the security industry to help IT professionals succeed in their cloud-native journey.Anna holds a PhD in Materials Engineering from the University of Michigan, where she developed computational methods to study solar cells and rechargeable batteries.How do I adapt my security practices for the cloud-native world?How do I select and deploy appropriate tools and processes to address business needs?How do I make sense of new technology trends like threat deception, machine learning, and containers?Links: Sysdig: https://sysdig.com/ “2022 Cloud-Native Security and Usage Report”: https://sysdig.com/2022-cloud-native-security-and-usage-report/ Twitter: https://twitter.com/aabelak LinkedIn: https://www.linkedin.com/in/aabelak/ Email: anna.belak@sysdig.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I went to a conference talk at, basically, a user meetup. This was in the before times, when that wasn't quite as much of a deadly risk because of a pandemic, and mostly a deadly risk due to me shooting my mouth off when it wasn't particularly appreciated.At that talk, I wound up seeing a new open-source project that was presented to me, and it was called Sysdig. I wasn't quite sure on what it did at the time and I didn't know what it would be turning into, but here we are now, what is it, five years later. Well, it's turned into something rather interesting. This is a promoted episode brought to us by our friends at Sysdig and my guest today is their Director of Thought Leadership, Anna Belak. Anna, thank you for joining me.Anna: Hi, Corey. I'm very happy to be here. I'm a big fan.Corey: Oh, dear. So, let's start at the beginning. Well, we'll start with the title: Director of Thought Leadership. That is a lofty title, it sounds like you sit on the council of the Lords of Thought somewhere. Where does your job start and stop?Anna: I command the Council of the Lords of thought, actually. [laugh].Corey: Supply chain issues mean the robe wasn't available. I get it, I get it.Anna: There is a robe. I'm just not wearing it right now. So, the shortest way to describe the role is probably something that reports into engineering, interestingly, and it deals with product and marketing in a way that is half evangelism and half product strategy. I just didn't feel like being called any of those other things, so they were like, “Director of Thought Leadership you are.” And I was like, “That sounds awesome.”Corey: You know, it's one of those titles that people generally don't see a whole lot of, so if nothing else, I always liked those job titles that cause people to sit up and take notice as opposed to something that just people fall asleep by the time you get halfway through it because, in lieu of a promotion, people give you additional adjectives in your title. And we're going to go with it. So, before you wound up at Sysdig, you were at Gartner for a number of years.Anna: That's right, I spent about six years at Gartner, and there half the time I covered containers, Kubernetes, and DevOps from an infrastructure perspective, and half the time I spent covering security operations, actually, not specifically with respect to containers, or cloud, but broadly. And so my favorite thing is security operations, as it relates to containers and cloud-native workloads, which is kind of how I ended up here.Corey: I wouldn't call that my favorite thing. It's certainly something that is near and dear to the top of mind, but that's not because I like it, let's put it [laugh] that way. It's one of those areas where getting it wrong is catastrophic. Back in 2017, when I went to that meetup in San Francisco, Sysdig seemed really interesting to me because it looked like it tied together a whole bunch of different diagnostic tools, LSOF, strace, and the rest. Honestly—and I mean no slight to the folks who built out this particular tool—it felt like DTrace, only it understood the value of being accessible to its users without basically getting a doctorate in something.I like the idea, and it felt like it was very much aimed at an in-depth performance analysis story or an observability play. But today, it seems that you folks have instead gone in much more of a direction of DevSecOps, if the people listening to this, and you, will pardon the term. How did that happen? What was that product evolution like?Anna: Yeah, I think that's a fair assessment, actually. And again, no disrespect to DTrace of which I'm also a fan. So, we certainly started out in the container observability space, essentially because this whole Docker Kubernetes thing was exploding in popularity—I mean, before it was exploding, it was just kind of like, peaking out—and very quickly, our founder Loris, who is the co-founder of Wireshark, was like, “Hey, there's a visibility issue here. We can't see inside these things with the tools that we have that are built for host instrumentation, so I'm going to make a thing.” And he made a thing, and it was an awesome thing that was open-sourced.And then ultimately, what happened is, the ecosystem of containers and communities evolved, and more and more people started to adopt it. And so more people needed kind of a more, let's say, hefty, serious tool for observability, and then what followed was another tool for security because what we actually discovered was the data that we're able to collect from the system with Sysdig is incredibly useful for noticing security problems. So, that caused us to kind of expand into that space. And today we are very much a tool that still has an observability component that is quite popular, has a security component which is it's fairly broad: We cover CSPM use cases, we cover [CIEM 00:05:04] use cases, and we are very, kind of let's say, very strong and very serious about our detection response and runtime security use cases, which come from that pedigree of the original Sysdig as well.Corey: You can get a fairly accurate picture of what the future of technology looks like by taking a look at what my opinion of something is, and then doing the exact opposite of that. I was a big believer that virtualization, “Complete flash in the pan; who's going to use that?” Public cloud, “Are you out of your tree? No one's going to trust other companies with their holy of holies.” And I also spent a lot of time crapping on containers and not actually getting into them.Instead, I leapfrogged over into the serverless land, which I was a big fan of, which of course means that it's going to be doomed sooner or later. My security position has also somewhat followed similar tracks where, back when you're running virtual machines that tend to be persistent, you really have to care about security because you are running full-on systems that are persistent, and they run all kinds of different services simultaneously. Looking at Lambda Functions, for example, in the modern serverless world, I always find a lot of the tooling and services and offerings around security for that are a little overblown. They have a defined narrow input, they have a defined output, there usually aren't omnibus functions shoved in here where they have all kinds of different code paths. And it just doesn't have the same attack surface, so it often feels like it's trying to sell me something I don't need. Security in the container world is one of those areas I never had to deal with in anger, as a direct result. So, I have to ask, how bad is it?Anna: Well, I have some data to share with you, but I'll start by saying that I maybe was the opposite of you, so we'll see which one of us wins this one. I was an instant container fangirl from the minute I discovered them. But I crapped out—Corey: The industry shows you were right on that one. I think the jury [laugh] is pretty much in on this one.Anna: Oh, I will take it. But I did crap on Lambda Functions pretty hard. I was like, “Serverless? This is dumb. Like, how are we ever going to make that work?” So, it seems to be catching on a little bit, at least it. It does seem like serverless is playing the function of, like, the glue between bits, so that does actually make a lot of sense. In retrospect, I don't know that we're going to have—Corey: Well, it feels like it started off with a whole bunch of constraints around it, and over time, they've continued to relax those constraints. It used to be, “How do I package this?” It's, “Oh, simple. You just spent four days learning about all the ins and outs of this,” and now it's, “Oh, yeah. You just give it a Docker file?” “Oh. Well, that seems easier. I could have just been stubborn and waited.” Hindsight.Anna: Yeah, exactly. So, containers as they are today, I think are definitely much more usable than they were five-plus years ago. There are—again there's a lot of commercial support around these things, right? So, if you're, you know, like, a big enterprise client, then you don't really have time to fool around in open-source, you can go in, buy yourself a thing, and they'll come with support, and somebody will hold your hand as you figure it out, and it's actually quite, quite pleasant. Whether or not that has really gone mainstream or whether or not we've built out the entire operational ecosystem around it in a, let's say, safe and functional way remains to be seen. So, I'll share some data from our report, which is actually kind of the key thing I want to talk about.Corey: Yeah, I wanted to get into that. You wound up publishing this somewhat recently, and I regret that as of the time of this recording, I have not yet had time to go into it in-depth, and of course eviscerate it in my typical style on Twitter—although that may have been rectified by the time that this show airs, to be very clear—but it's the Sysdig “2022 Cloud-Native Security and Usage Report”.Anna: Please at me when you Twitter-shred it. [laugh].Corey: Oh, when I read through and screenshot it, and I'd make what observations that I imagine are witty. But I'm looking forward to it; I've done that periodically with the Flexera, “State of the Cloud” report for last few years, and every once in a while, whatever there's a, “We've done a piece of thought leadership, and written a report,” it's, “Oh, great. Let's make fun of it.” That's basically my default position on things. I am not a popular man, as you might imagine. But not having had the chance to go through it in-depth, what did this attempt to figure out when the study was built, and what did you learn that you found surprising?Anna: Yeah, so the first thing I want to point out because it's actually quite important is that this report is not a survey. This is actual data from our actual back end. So, we're a SaaS provider, we collect data for our customers, we completely anonymize it, and then we show in aggregate what in fact we see them doing or not doing. Because we think this is a pretty good indicator of what's actually happening versus asking people for their opinion, which is, you know, their opinion.Corey: Oh, I love that. My favorite lies that people tell are the lies they don't realize that they're telling. It's, I'll do an AWS bill analysis and, “Great. So, tell me about all these instances you have running over in Frankfurt.” “Oh, we don't have anything there.”I believe you're being sincere when you say this, however, the data does show otherwise, and yay, now we're in a security incident.Anna: Exactly.Corey: I'm a big believer of going to the actual source for things like this where it's possible.Anna: Exactly. So, I'll tell you my biggest takeaway from the whole thing probably was that I was surprised by the lack of… surprise. And I work in cloud-native security, so I'm kind of hoping every single day that people will start adopting these modern patterns of, like, discarding images, and deploying new ones when they found a vulnerability, and making ephemeral systems that don't run for a long time like a virtual machine in disguise, and so on. And it appears that that's just not really happening.Corey: Yeah, it's always been fun, more than a little entertaining, when I wind up taking a look at the aspirational plans that companies have. “Great, so when are you going to do”—“Oh, we're going to get to that after the next sprint.” “Cool.” And then I just set a reminder and I go back a year later, and, “How's that coming?” “Oh, yeah. We're going to get to that next sprint.”It's the big lie that we always tell ourselves that right after we finished this current project, then we're going to suddenly start doing smart things, making the right decisions, and the rest. Security, cost, and a few other things all tend to fall on the side of, you can spend infinite money and infinite time on these things, but it doesn't advance what your business is doing, but if you do none of those things, you don't really have a business anymore. So, it's always a challenge to get it prioritized by the strategic folks.Anna: Exactly. You're exactly right because what people ultimately do is they prioritize business needs, right? They are prioritizing whatever makes them money or creates the trinkets their selling faster or whatever it is, right? The interesting thing, though, is if you think about who our customers would be, like, who the people in this dataset are, they are all companies who are probably more or less born in the cloud or at least have some arm that is born in the cloud, and they are building software, right? So, they're not really just your average enterprises you might see in a Gartner client base which is more broad; they are software companies.And for software companies, delivering software faster is the most important thing, right, and then delivering secure software faster, should be the most important thing, but it's kind of like the other thing that we talk about and don't do. And that's actually what we found. We found that people do deliver software faster because of containers and cloud, but they don't necessarily deliver secure software faster because as is one of our data points, 75% of containers that run in production have critical or high vulnerabilities that have a patch available. So, they could have been fixed but they weren't fixed. And people ask why, right? And why, well because it's hard; because it takes time; because something else took priority; because I've accepted the risk. You know, lots of reasons why.Corey: One of the big challenges, I think, is that I can walk up and down the expo hall at the RSA Conference, which until somewhat recently, you were not allowed to present that or exhibit at unless you had the word ‘firewall' in your talk title, or wound up having certain amounts of FUD splattered across your banners at the show floor. It feels like there are 12 products—give or take—for sale there, but there are hundreds of booths because those products have different names, different messaging, and the rest, but it all feels like it distills down to basically the same general categories. And I can buy all of those things. And it costs an enormous pile of money, and at the end of it, it doesn't actually move the needle on what my business is doing. At least not in a positive direction, you know? We just set a giant pile of money on fire to make sure that we're secure.Well, great. Security is never an absolute, and on top of that, there's always the question of what are we trying to achieve as a business. As a goal—from a strategic perspective—security often looks a lot like, “Please let's not have a data breach that we have to report to people.” And ideally, if we have a lapse, we find out about it through a vector that is other than the front page of The New York Times. That feels like it's a challenging thing to get prioritized in a lot of these companies. And you have found in your report that there are significant challenges, of course, but also that some companies in some workloads are in fact getting it right.Anna: Right, exactly. So, I'm very much in line with your thinking about this RSA shopping spree, and the reality of that situation is that even if we were to assume that all of the products you bought at the RSA shopping center were the best of breed, the most amazing, fantastic, perfect in every way, you would still have to somehow build a program on top of them. You have to have a process, you have to have people who are bought into that process, who are skilled enough to execute on that process, and who are more or less in agreement with the people next door to them who are stuck using one of the 12 trinkets you bought, but not the one that you're using. So, I think that struggle persists into the cloud and may actually be worse in the cloud because now, not only are we having to create a processor on all these tools so that we can actually do something useful with them, but the platform in which we're operating is fundamentally different than what a lot of us learned on, right?So, the priorities in cloud are different; the way that infrastructure is built is a little different, like, you have to program a YAML file to make yourself an instance, and that's kind of not how we are used to doing it necessarily, right? So, there are lots of challenges in terms of skills gap, and then there's just this eternal challenge of, like, how do we put the right steps into place so that everybody who's involved doesn't have to suffer, right, and that the thing that comes out at the end is not garbage. So, our approach to it is to try to give people all the pieces they need within a certain scope, so again, we're talking about people developing software in a cloud-native world, we're focused kind of on containers and cloud workloads even though it's not necessarily containers. So that's, like, our sandbox, right? But whoever you are, right, the idea is that you need to look to the left—because we say ‘shift left'—but then you kind of have to follow that thread all the way to the right.And I actually think that the thing that people most often neglect is the thing on the right, right? They maybe check for compliance, you know, they check configurations, they check for vulnerabilities, they check, blah, blah, blah, all this checking and testing. They release their beautiful baby into the world, and they're like, okay, I wash my hands of it. It's fine. [laugh]. Right but—Corey: It has successfully been hurled over the fence. It is the best kind of problem, now: Someone else's.Anna: It's gone. Yeah. But it's someone else's—the attacker community, right, who are now, like, “Oh, delicious. A new target.” And like, that's the point at which the fun starts for a lot of those folks who are on the offensive side. So, if you don't have any way to manage that thing's security as it's running, you're kind of like missing the most important piece, right? [laugh].Corey: One of the challenges that I tend to see with a lot of programmatic analysis of this is that it doesn't necessarily take into account any of the context because it can't. If I have, for example, a containerized workload that's entire job is to take an image from S3, run some analysis or transformation on it then output the results of that to some data store, and that's all it's allowed to talk to you, it can't ever talk to the internet, having a system that starts shrieking about, “Ah, there's a vulnerability in one of the libraries that was used to build that container; fix it, fix it, fix it,” doesn't feel like it's necessarily something that adds significant value to what I do. I mean, I see this all the time with very purpose-built Lambda Functions that I have doing one thing and one thing only. “Ah, but one of the dependencies in the JSON processing library could turn into something horrifying.” “Yeah, except the only JSON it's dealing with is what DynamoDB returns. The only thing in there is what I've put in there.”That is not a realistic vector of things for me to defend against. The challenge then becomes when everything is screaming that it's an emergency when you know, due to context, that it's not, people just start ignoring everything, including the, “Oh, and by the way, the building is on fire,” as one of—like, on page five, that's just a small addendum there. How do you view that?Anna: The noise insecurity problem, I think, is ancient and forever. So, it was always bad, right, but in cloud—at least some containers—you would think it should be less bad, right, because if we actually followed these sort of cloud-native philosophy, of creating very purp—actually it's called the Unix philosophy from, like, I don't know, before I was born—creating things that are fairly purposeful, like, they do one thing—like you're saying—and then they disappear, then it's much easier to know what they're able to do, right, because they're only able to do what we've told them, they're able to do. So, if this thing is enabled to make one kind of network connection, like, I'm not really concerned about all the other network connections it could be making because it can't, right? So, that should make it easier for us to understand what the attack surface actually is. Unfortunately, it's fairly difficult to codify and productize the discovery of that, and the enrichment of the vulnerability information or the configuration information with that.That is something we are definitely focusing on as a vendor. There are other folks in the industry that are also working on this kind of thing. But you're exactly right, the prioritization of not just a vulnerability, but a vulnerability is a good example. Like, it's a vulnerability, right? Maybe it's a critical or maybe it's not.First of all, is it exposed to the outside world somehow? Like, can we actually talk to this system? Is it mitigated, right? Maybe there's some other controls in place that is mitigating that vulnerability. So, if you look at all this context, at the end of the day, the question isn't really, like, how many of these things can I ignore? The question is at the very least, which are the most important things that I actually can't ignore? So, like you're saying, like, the buildings on fire, I need to know, and if it's just, like, a smoldering situation, maybe that's not so bad. But I really need to know about the fire.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: It always becomes a challenge of prioritization, and that has been one of those things that I think, on some level, might almost cut against a tool that works at the level that Sysdig does. I mean, something that you found in your report, but I feel like, on some level, is one of those broadly known, or at least unconsciously understood things is, you can look into a lot of these tools that give incredibly insightful depth and explore all kinds of neat, far-future, bleeding edge, absolute front of the world, deep-dive security posture defenses, but then you have a bunch of open S3 buckets that have all of your company's database backups living in them. It feels like there's a lot of walk before you can run. And then that, on some level, leads to the wow, we can't even secure our S3 buckets; what's the point of doing anything beyond that? It's easy to, on some level, almost despair, want to give up, for some folks that I've spoken to. Do you find that is a common thing or am I just talking to people who are just sad all the time?Anna: I think a lot of security people are sad all the time. So, the despair is real, but I do think that we all end up in the same solution, right? The solution is defense in depth, the solution is layer control, so the reality is if you don't bother with the basic security hygiene of keeping your buckets closed, and like not giving admin access to every random person and thing, right? If you don't bother with those things, then, like, you're right, you could have all the tools in the world and you could have the most advanced tools in the world, and you're just kind of wasting your time and money.But the flipside of that is, people will always make mistakes, right? So, even if you are, quote-unquote, “Doing everything right,” we're all human, and things happen, and somebody will leave a bucket open on accident, or somebody will misconfigure some server somewhere, allowing it to make a connection it shouldn't, right? And so if you actually have built out a full pipeline that covers you from end-to-end, both pre-deployment, and at runtime, and for vulnerabilities, and misconfigurations, and for all of these things, then you kind of have checks along the way so that this problem doesn't make it too far. And if it does make it too far and somebody actually does try to exploit you, you will at least see that attack before they've ruined everything completely.Corey: One thing I think Sysdig gets very right that I wish this was not worthy of commenting on, but of course, we live in the worst timeline, so of course it is, is that when I pull up the website, it does not market itself through the whole fear, uncertainty, and doubt nonsense. It doesn't have the scary pictures of, “Do you know what's happening in your environment right now?” Or the terrifying statistics that show that we're all about to die and whatnot. Instead, it talks about the value that it offers its customers. For example, I believe its opening story is, “Run with confidence.” Like, great, you actually have some reassurance that it is not as bad as it could be. That is, on the one hand, a very uplifting message and two, super rare. Why is it that so much of the security industry resorts to just some of the absolute worst storytelling tactics in order to drive sales?Anna: That is a huge compliment, Corey, and thank you. We try very hard to be kind of cool in our marketing.Corey: It shows. I'm tired of the 1990s era story of, “Do you know where the hackers are?” And of course, someone's wearing, like, a ski mask and typing with gloves on—which is always how I break into things; I don't know about you—but all right, we have the scary clip art of the hacker person, and it just doesn't go anywhere positive.Anna: Yeah. I mean, I think there certainly was a trend for a while have this FUD approach. And it's still prevalent in the industry, in some circles more than others. But at the end of the day, Cloud is hard and security is hard, and we don't really want to add to the suffering; we would like to add to the solution, right? So, I don't think people don't know that security is hard and that hackers are out there.And you know, there's, like, ransomware on the news every single day. It's not exactly difficult to tell that there's a challenge there, so for us to have to go and, like, exacerbate this fear is almost condescending, I feel, which is kind of why we don't. Like, we know people have problems, and they know that they need to solve them. I think the challenge really is just making sure that A) can folks know where to start and how to build a sane roadmap for themselves? Because there are many, many, many things to work on, right?We were talking about context before, right? Like, so we actually try to gather this context and help people. You made a comment about how having a lot of telemetry might actually be a little bit counterproductive because, like, there's too much data, what do I do well—Corey: Here's the 8000 findings we found that you fail—great. Yeah. Congratulations, you're effectively the Nessus report as a company. Great. Here you go.Anna: Everything is over.Corey: Yeah.Anna: Well, no shit, Nessus, you know. Nessus did its thing. All right. [laugh].Corey: Oh, Nessus was fantastic. Nessus was—for those who are unaware, Nessus was an open-source scanner made by the folks at Tenable, and what was great about it was that you could run it against an environment, it would spit out all the things that it found. Now, one of the challenges, of course, is that you could white-label this and slap whatever logo you wanted on the top, and there were a lot of ‘security consultancies' that use the term incredibly… lightly, that would just run a Nessus report, drop off the thick print out. “Here's the 800 things you need to fix. Pay me.” And wander on off into the sunset.And when you have 800 things you need to fix, you fix none of them. And they would just sit there and atrophy on the shelf. Not to say that all those things weren't valid findings, but you know, the whole, you're using an esoteric, slightly deprecated TLS algorithm on one of your back-end services, versus your Elasticsearch database does not have a password set. Like, there are different levels of concern here. And that is the problem.Anna: Yeah. That is in fact one of the problems we're aggressively trying to solve, right? So, because we see so much of the data, we're actually able to piece together a lot of context to gives you a sense of risk, right? So, instead of showing all the data to the customer—the customer can see it if they want; like, it's all in there, you can look at it—one of the things we're really trying to do is collect enough information about the finding or the event or the vulnerability or whatever, so we can kind of tell you what to do.For example, one you can do this is super basic, but if you're looking at a specific vulnerability, like, let's say it's like Log4j or whatever, you type it in, and you can see all your systems affected by this thing, right? Then you can, in the same tool, like, click to the other tab, and you can see events associated with this vulnerability. So, if you can see the systems that the vulnerability is on and you can see there's weird activity on those systems, right? So, if you're trying to triage some weird thing in your environment, during the Log4j disaster, it's very easy for you to be like, “Huh. Okay, these are the relevant systems. This is the vulnerability. Like, here's all that I know about this stuff.”So, we kind of try to simplify as much as possible—my design team uses the word ‘easify,' which I love; it's a great word—to easify, the experience of the end-user so that they can get to whatever it is they're trying to do today. Like, what can I do today to make my company more secure as quickly as possible? So, that is sort of our goal. And all this huge wealth of information we gather, we try to package for the users in a way that is, in fact, digestible. And not just like, “Here's a deluge of suffering,” like, “Look.” [laugh]. You know?Corey: This is definitely complicated in the environment I tend to operate in which is almost purely AWS. How much more complex is get when people start looking into the multi-cloud story, or hybrid environments where they have data center is talking to things within AWS? Because then it's not just the expanded footprint, but the entire security model works slightly differently in all of those different environments as well, and it feels like that is not a terrific strategy.Anna: Yeah, this is tough. My feelings on multi-cloud are mostly negative, actually.Corey: Oh, thank goodness. It's not just me.Anna: I was going to say that, like, multi-cloud is not a strategy; it's just something that happens to you.Corey: Same with hybrid. No one plans to do hybrid. They start doing a cloud migration, realize halfway through some things are really hard to move, give up, plant the flag, declare victory, and now it's called hybrid.Anna: Basically. But my position—and again, as an analyst, you kind of, I think, end up in this position, you just have a lot of sympathy for the poor people who are just trying to get these stupid systems to run. And so I kind of understand that, like, nothing's ideal, and we're just going to have to work with it. So multi-cloud, I think is one of those things where it's not really ideal, we just have to work with it. There's certainly advantages to it, like, there's presumably some level of mythical redundancy or whatever. I don't know.But the reality is that if you're trying to secure a pile of junk in Azure and a pile of junk in AWS, like, it'd be nice if you had, like, one tool that told you what to do with both piles of junk, and sometimes we do do that. And in fact, it's very difficult to do that if you're not a third-party tool because if you're AWS, you don't have much incentive to, like, tell people how to secure Azure, right? So, any tool in the category of, like, third-party CSPM—Gartner calls them CWPP—kind of, cloud security is attempting to span those clouds because they always have to be relevant, otherwise, like, what's the point, right?Corey: Well, I would argue cynically there's also the VC model, where, “Oh, great. If we cover multiple cloud providers, that doubles or triples our potential addressable market.” And, okay, great, I don't have those constraints, which is why I tend to focus on one cloud provider where I tend to see the problems I know how to solve as opposed to trying to conquer the world. I guess I have my bias on that one.Anna: Fair. But there's—I think the barrier to entry is lower as a security vendor, right? Especially if you're doing things like CSPMs. Take an example. So, if you're looking at compliance requirements, right, if your team understands, like, what it means to be compliant with PCI, you know, like, [line three 00:28:14] or whatever, you can apply that to Azure and Amazon fairly trivially, and be like, “Okay, well, here's how I check in Azure, and here's how I check in Amazon,” right?So, it's not very difficult to, I think, engineer that once you understand the basic premise of what you're trying to accomplish. It does become complicated as you're trying to deal with more and more different cloud services. Again, if you're kind of trying to be a cloud security company, you almost have no choice. Like, you have to either say, “I'm only doing this for AWS,” which is kind of a weird thing to do because they're kind of doing their own half-baked thing already, or I have to do this for everybody. And so most default to doing it for everybody.Whether they do it equally well, for everybody, I don't know. From our perspective, like, there's clearly a roadmap, so we have done one of them first and then one of them second and one of the third, and so I guarantee you that we're better in some than others. So, I think you're going to have pluses and minuses no matter what you do, but ultimately what you're looking for is coverage of the tool's capabilities, and whether or not you have a program that is going to leverage that tool, right? And then you can check the boxes of like, “Okay. Does it do the AWS thing? Does it do this other AWS thing? Does it do this Azure thing?”Corey: I really appreciate your taking the time out of your day to speak with me. We're going to throw a link to the report itself in the [show notes 00:29:23], but other than that, if people want to learn more about how you view these things, where's the best place to find you?Anna: I am—rarely—but on Twitter at @aabelak. I am also on LinkedIn like everybody else, and in the worst case, you could find me by email, at anna.belak@sysdig.com.Corey: And we will of course put links to that in the [show notes 00:29:44]. Thank you so much for taking the time to speak with me today. I appreciate it.Anna: Thanks for having me, Corey. It's been fun.Corey: Anna Belak, Director of Thought Leadership at Sysdig. I'm Cloud Economist Corey Quinn and this is streaming on the cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me not only why this entire approach to security is awful and doomed to fail, but also what booth number I can find you at this year's RSA Conference.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Oxide and Friends Twitter Space: January 24th, 2022Taxonomy of HypeWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for January 24th, 2022.In addition to Bryan Cantrill and Adam Leventhal, speakers on January 24th included MattSci, Todd Gamblin, Aaron Goldman and Tom Lyon. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: The tweet about the topic: Johannes Klingebiel's (2022) The five Levels of Hype taxonomy [@8:24](https://youtu.be/qrWgmkBfn9s?t=504) Roko's Basilisk (slate.com) [@10:21](https://youtu.be/qrWgmkBfn9s?t=621) Cloud Computing [@12:09](https://youtu.be/qrWgmkBfn9s?t=729) Mobile, Wi-Fi (introduced in 1997) Adam broke his hand, but can still type dtrace with one hand [@15:14](https://youtu.be/qrWgmkBfn9s?t=914) Java Write once run anywhere Cross platform graphical interfaces Windows NT [@17:47](https://youtu.be/qrWgmkBfn9s?t=1067) Storage technology Dedup ZFS copies setting and redundant_metadata InfiniBand, iSCSI Extensions for RDMA (iSER), SCSI RDMA Protocol (SRP) [@26:15](https://youtu.be/qrWgmkBfn9s?t=1575) 3D XPoint (Intel Optane) wiki HP Memristor FAQ HP “The Machine” HP research's pure hype marketing pitch The (absolutely incredible) Star Trek crossover ad > I'm gonna provide you the emotion of a revolution, but not the technical detail to > support it, not yet, but it's coming. [@31:02](https://youtu.be/qrWgmkBfn9s?t=1862) Segway (wiki) Dean Kamen wiki Decoder Ring podcast (June 2021) Who Killed the Segway? ~40mins slate.com, Apple podcasts 2001 Good Morning America Segway unveiling, Diane Sawyer is underwhelmed > I'm tempted to say “that's it??” (nervous laughter) > But that can't be it!? [@34:29](https://youtu.be/qrWgmkBfn9s?t=2069) Maglev, Cold fusion Walter Isaacson (2021) The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race book Human Genome Project wiki Hype booms and busts Todd's story on working on fusion at a national lab, and the nature of gaining funding for large projects [@45:30](https://youtu.be/qrWgmkBfn9s?t=2730) Rust [@48:43](https://youtu.be/qrWgmkBfn9s?t=2923) DTrace [@52:14](https://youtu.be/qrWgmkBfn9s?t=3134) Nanotechnology K. Eric Drexler wiki Expert Systems, AR/VR [@56:23](https://youtu.be/qrWgmkBfn9s?t=3383) Chatbots Dan Olson (Jan 2022) Line Goes Up - The Problem with NFTs ~2hr video (worth every minute) [@59:11](https://youtu.be/qrWgmkBfn9s?t=3551) Serverless Itanium IA-64, Very long instruction word VLIW Fibre Channel over Ethernet FCoE, ATA over Ethernet AoE > A solution in search of a problem [@1:06:50](https://youtu.be/qrWgmkBfn9s?t=4010) Taligent wiki Tom Hormby (2014) Pink: Apple's First Stab at a Modern Operating System post Be Inc wikiBryan's Be whiteboard story [@1:13:47](https://youtu.be/qrWgmkBfn9s?t=4427) Docker Monetizing open source [@1:20:28](https://youtu.be/qrWgmkBfn9s?t=4828) 5G If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!
Oxide and Friends Twitter Space: November 22nd, 2021Talking TurkeysWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 22nd, 2021.In addition to Bryan Cantrill and Adam Leventhal, speakers on November 22nd included Rick Altherr, Ian, Simeon Miteff, MattSci, Nahum Shalman, Jason Ozolins, pgray, Bill Blum, Matt Ranney, Matt Campbell, FesterCluck, Rahul Saxena and Bartz the Man. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: [@4:26](https://youtu.be/U10SuAHV8kQ?t=266) Thanksgiving [@6:13](https://youtu.be/U10SuAHV8kQ?t=373) David Tolnay twitter and github Projects Serde Anyhow thiserror London hip hop musician Loyle Carner [@8:16](https://youtu.be/U10SuAHV8kQ?t=496) Adam is thankful for: ANTLR parser generator pest usdt DTrace probes [@11:35](https://youtu.be/U10SuAHV8kQ?t=695) Bryan is thankful for: build.rs Rust build scripts Saleae logic analyzers [@16:33](https://youtu.be/U10SuAHV8kQ?t=993) Ian: YubiKey [@19:09](https://youtu.be/U10SuAHV8kQ?t=1149) Matt Campbell: open source, Python accessibility Windows libraries from Chapel Hill [@23:52](https://youtu.be/U10SuAHV8kQ?t=1432) FesterCluck: Nodejs [@26:03](https://youtu.be/U10SuAHV8kQ?t=1563) Patrick: RabbitMQ [@28:19](https://youtu.be/U10SuAHV8kQ?t=1699) Nahum: WireGuard and Tailscale [@32:04](https://youtu.be/U10SuAHV8kQ?t=1924) Jason: truss by Roger Faulkner [@37:37](https://youtu.be/U10SuAHV8kQ?t=2257) Rahul: tldp.org Linux documentation [@42:11](https://youtu.be/U10SuAHV8kQ?t=2531) Simeon: sigrok, PulseView, Anyhow, thiserror [@44:35](https://github.com/dtolnay/thiserror) Adam: QMK, Magic Lantern by Trammell Hudson (twitter) [@47:36](https://youtu.be/U10SuAHV8kQ?t=2856) Matt: eBPF, (wiki) [@54:59](https://youtu.be/U10SuAHV8kQ?t=3299) MattSci: CUDA, Ethernet GPS John Bloom (2016) Eccentric Orbits book Differential GPS BeiDou Chinese satellites, GLONASS Russian satellites, and Galileo European Union satellites [@1:09:20](https://youtu.be/U10SuAHV8kQ?t=4160) Bartz: grep [@1:10:30](https://youtu.be/U10SuAHV8kQ?t=4230) Rick: Ghidra reverse engineering tool Interactive Disassembler IDA [@1:12:28](https://youtu.be/U10SuAHV8kQ?t=4348) Bill: Fastest Fourier Transform in the West FFTW, and gnuplot > I'm thankful that everywhere I look there's always something that hits my > sense of wonder. That's the thing I love about working in this industry. Adam appreciates spreadsheets as tools for analysis If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!
Oxide and Friends Twitter Space: November 15th, 2021The Wrath of KahnWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 15th, 2021.In addition to Bryan Cantrill and Adam Leventhal, speakers on November 15th included Dan Cross, Tom Lyon, Antranig Vartanian, Mat Trudel, Gabe Rudy, Simeon Miteff and bch. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: Severo Ornstein (2002) Computing in the Middle Ages: A View from the Trenches 1955-1983 book TX-2 computer in 1958 LINC Laboratory INstrument Computer in 1962 Wesley Clark IMP [@6:21](https://youtu.be/oft5i5RzIC8?t=381) Quote on paternity of ARPANET and the Internet [@7:51](https://youtu.be/oft5i5RzIC8?t=471) Bryan meets Knuth… briefly SOAP [@20:00](https://youtu.be/oft5i5RzIC8?t=1200) Quote from oral history of Bob Taylor (2008) [@21:37](https://youtu.be/oft5i5RzIC8?t=1297) Dan meets Knuth? [@25:23](https://youtu.be/oft5i5RzIC8?t=1523) The lone inventor [@26:40](https://youtu.be/oft5i5RzIC8?t=1600) The patent race with Alexander Graham Bell and Elisha Gray (wiki) “Inventor” of email [@30:49](https://youtu.be/oft5i5RzIC8?t=1849) Fathering and parenting (pioneers and settlers) Any lone inventors? Credit where credit is due. Teams as more than the sum of the parts. Turing Awards [@35:49](https://youtu.be/oft5i5RzIC8?t=2149) Science papers, teams [@37:14](https://youtu.be/oft5i5RzIC8?t=2234) Andy van Dam (wiki) “Hypertext '87 Keynote” address “Reflections on a Half Century of Hypertext” (2019) ~100mins presentation Ron Minnich (On the Metal podcast) [@39:11](https://youtu.be/oft5i5RzIC8?t=2351) Dennis Klatt and DECtalkDECtalk DTC01 used a 68000 and a TI 32010 DSP; DECtalk DTC03 used a 80186 and the same TI 32010. mame Doug Engelbart (wiki) [@44:37](https://youtu.be/oft5i5RzIC8?t=2677) Who's going to lead the charge? Michael Stonebraker (wiki) Seeing things through [@49:23](https://youtu.be/oft5i5RzIC8?t=2963) bch: communications and crediting [@50:53](https://youtu.be/oft5i5RzIC8?t=3053) DTrace, ZFS [@53:15](https://youtu.be/oft5i5RzIC8?t=3195) Mat: The Dream Machine M. Mitchell Waldrop (2001) “The Dream Machine: JCR Licklider and the Revolution that Made Computing Personal” book DARPA, private public research funding [@56:57](https://youtu.be/oft5i5RzIC8?t=3417) The hero narrative sells well If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!
Oxide and Friends Twitter Space: November 8th, 2021Supercomputers, Cray, and How Sun Picked SGI's PocketWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 8th, 2021.In addition to Bryan Cantrill and Adam Leventhal, speakers on November 8th included Tom Lyon, Shahin Khan, Darryl Ramm, Dan Cross, Courtney Malone, MattSci, Aaron Goldman, Simeon Miteff, and Jason Ozolins. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: Bryan's tweet about George Brown's recommending “The Supermen” Charles Murray (1997) “The Supermen: The story of Seymour Cray and the Technical Wizards Behind the Supercomputer” book [@1:28](https://youtu.be/y07PyBrrzMw?t=88) Tom's story meeting Boris Tom's tweet on meeting Boris Babayan Elbrus computers [@9:27](https://youtu.be/y07PyBrrzMw?t=567) Supercomputers and power [@15:16](https://youtu.be/y07PyBrrzMw?t=916) Cray designs Engineering Research Associates wiki Control Data Corporation wiki, CDC 1604 [@20:36](https://youtu.be/y07PyBrrzMw?t=1236) ETA Systems wiki [@23:57](https://youtu.be/y07PyBrrzMw?t=1437) On to the next big thing Steve Chen Cray X-MP [@29:37](https://youtu.be/y07PyBrrzMw?t=1777) Super computers as one-offs National Computational Infrastructure in Australia, NCI Gallium arsenide GPGPU [@33:47](https://youtu.be/y07PyBrrzMw?t=2027) Shahin on interconnects Jason on failure caused by a storm Cray C90 [@41:06](https://youtu.be/y07PyBrrzMw?t=2466) Courtney on bespoke toolchains and systems [@42:42](https://youtu.be/y07PyBrrzMw?t=2562) Influence of Cray on Sun 1996 Sun to purchase Cray Business Systems Division, hpcwire Floating Point Systems Inc wiki > Shahin: SGI really had no use for this system. They should have just killed it. [@50:10](https://youtu.be/y07PyBrrzMw?t=3010) Origin story of DTrace (2006 article) E10k [@56:14](https://youtu.be/y07PyBrrzMw?t=3374) Thinking Machines Corp, wiki [@57:36](https://youtu.be/y07PyBrrzMw?t=3456) Seymour Cray Les Davis “The ultimate team player” write up 2010 Oral history of Les Davis pdf [@1:00:08](https://youtu.be/y07PyBrrzMw?t=3608) Business Systems Division history, long road to Starfire [@1:04:20](https://youtu.be/y07PyBrrzMw?t=3860) SGI and Sun early history Non-uniform memory access NUMA [@1:10:40](https://youtu.be/y07PyBrrzMw?t=4240) Cray T3EMassively parallel MPP [@1:12:33](https://youtu.be/y07PyBrrzMw?t=4353) E10k stories boo.com wiki [@1:18:37](https://youtu.be/y07PyBrrzMw?t=4717) Cray, spooks, pop count [@1:20:45](https://youtu.be/y07PyBrrzMw?t=4845) Chen Cray X-MP and Y-MP Sequent [@1:24:04](https://youtu.be/y07PyBrrzMw?t=5044) An engineer sees his defunct machine being scrapped [@1:26:27](https://youtu.be/y07PyBrrzMw?t=5187) Jason's story of capacitors popping off the board The Capacitor plague If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!
DTrace network probes, next 50 years of shell programming, NetBSD on the Vortex86DX CPU, system CPU time in top, your filesystem as a dungeon, diving into toolchains, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines DTrace Network Probes (https://klarasystems.com/articles/dtrace-network-probes/) Unix Shell Programming: The Next 50 Years (https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s06-greenberg.pdf) News Roundup NetBSD on the Vortex86DX CPU (https://www.cambus.net/netbsd-on-the-vortex86dx-cpu/) System CPU time – ‘sys' time in top (https://blog.ycrash.io/2020/11/28/system-cpu-time-sys-time-in-top/) rpg-cli —your filesystem as a dungeon! (https://github.com/facundoolano/rpg-cli) Diving into toolchains (https://www.cambus.net/diving-into-toolchains/) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions • [Alfred - Advice](https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/409/feedback/Alfred%20-%20Advice) • [CY - Portable Patch Util](https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/409/feedback/CY%20-%20Portable%20Patch%20Util) • [Denis - State of ZFS Ecosystem](https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/409/feedback/Denis%20-%20State%20of%20ZFS%20Ecosystem) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
In this episode of ACM ByteCast, Rashmi Mohan hosts Bryan Cantrill, Co-founder and Chief Technology Officer at Oxide Computer Company and a past member of the ACM Queue Editorial Board. Previously, he was Vice President of Engineering and CTO at Joyent. He is known for his work on the award-winning DTrace software, a comprehensive dynamic tracing framework for which he was included in MIT Technology Review's TR35 (35 Top Young Innovators) list. Bryan describes discovering computing as a kid growing up in the 80s and falling in love with the challenge of solving difficult problems and getting hard programs to work. He talks about DTrace, which he first conceived as an undergraduate at Brown University and developed at Sun Microsystems (later acquired by Oracle). He also explains why he thinks open source will conquer every domain, his current challenge of designing a rack-scale computer for the enterprise, and much more.
Oxide and Friends Twitter Space: June 21, 2021What's a bug? What's a debugger?We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for June 21, 2021.In addition to Bryan Cantrill and Adam Leventhal, speakers on June 21st included Dan Cross, Sean Klein, Aram Hăvărneanu, and the mononymous Nate. (Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: Adam's toddler (being chased by a rooster) > Don't get me wrong, some of my best friends are three-year-olds. [@3:12](https://youtu.be/UOucW3F7nCg?t=192) Sy Brand's tutorial Writing a Debugger Lobsters – when HN isn't enough! [@4:34](https://youtu.be/UOucW3F7nCg?t=274) Bryan's debuggers MDB Modular Debugger > Adam: I think people are using cargo-cult debugging, rather than getting to the root cause > of these things, or being satisfied until they get to the root cause. > Bryan: I think with software systems, it's really hard to know what they're actually doing. Procedure Linkage Table aka “the plits” “Runtime Performance Analysis of the M-to-N Scheduling Model” (pdf) 1996 undergrad thesis (Brown CS dept website) [@6:29](https://youtu.be/UOucW3F7nCg?t=389) Threadmon website and 1997 paper (a retooling of the '96 paper) > When I built that tooling, it revealed this thing > is not doing at all what anyone thought it was doing. TNF Trace Normal Form > Part of the problem with debuggers… debuggers are historically written by compiler folks, > and not system folks. As a result, debuggers are designed to debug the problem that > compiler folks have the most familiarity with, and that's a compiler. > Debuggers are designed for reproducible problems, way too frequently. I view in situ breakpoint debugging as one sliver of debugging that's useful for one particular and somewhat unusual class of bugs. That's actually not the kind of debugger I want to use most of the time. Software breakpoints [@11:59](https://youtu.be/UOucW3F7nCg?t=719) > libdis was my intern project in 2000. The idea was to take the program text, > and interpret it in some structural form, and try to infer different things about the program. Ghidra: software reverse engineering tool Laura Abbott's Exploiting Undocumented Hardware Blocks in the LPC55S69 Volatility: the memory forensics framework Adam couldn't quite remember. [@14:59](https://youtu.be/UOucW3F7nCg?t=899) I meant this question earnestly, what is a debugger? The first bug > The term is somewhat regrettable… It implies a problem, when there may not be a problem. > It may just be I want to understand how the system is operating, independent of whether > it's doing it badly. Wikipedia on Observability (control theory) Oxide's embedded OS and companion debugger: Hubris and Humility [@19:01](https://youtu.be/UOucW3F7nCg?t=1141) Using DTrace to help customers understand their systems. > If you strings the DTrace binary, > you're not gonna find any mention of raincoats.Cliff Moon on Boundary [@22:13](https://youtu.be/UOucW3F7nCg?t=1333) Cardinal rule of debuggers: Don't kill the patient! (see also: Do No Harm) > Not killing the patient is really important, > this was always an Ur principle for us. The notion that the debugger has now become load bearing in the execution of the program, is a pretty grave responsibility. [@26:54](https://youtu.be/UOucW3F7nCg?t=1614) Post-mortem debugging > It is a tragedy of our domain that we do not debug post-mortem, routinely. Heisenbug (when the act of observing the problem, hides the problem) [@31:11](https://youtu.be/UOucW3F7nCg?t=1871) > What's going on in the system? It's not crashing, there's no core dump. > But the system is behaving in a way I didn't expect it to, and I want to know why. [@33:51](https://youtu.be/UOucW3F7nCg?t=2031) Pre-production reliability techniques > All of our pre-production work has gotten way better than it was, and I think that's > compensation for the fact we can't understand these systems when we deploy them. [@37:58](https://youtu.be/UOucW3F7nCg?t=2278) > The move to testing has in fact obviated some of the need for > what we consider traditional debuggers. > (Bryan audibly cringes) [@39:08](https://youtu.be/UOucW3F7nCg?t=2348) Automated and Algorithmic Debugging conference AADEBUG 2003 HOPL History of Programming Languages > There was a test suite of excellence when it comes to automated program debugging. > And it was some pile of C programs with known bugs, and you would throw your new > paper at it, and it would find 84% of the bugs, and there would be a lot of > slapping each other on the back on that. Really focused on the simplest of simple bugs. [@43:15](https://youtu.be/UOucW3F7nCg?t=2595) Bryan's Postmortem Object Type Identification paper > Who is my neighbor in memory? Because my neighbor just burned down my house basically. mdb's ::kgrep > I need to pause you there because it's so crazy, and I want to emphasize that > he means what he's saying. We look for the 64 bit value, and see where we find it. > This is a game of bingo across the entire address space. We can follow the pointers and propagate types. [@48:49](https://youtu.be/UOucW3F7nCg?t=2929) printf/println debugging – everyone's doing it > I think it's a mistake for people to denigrate printf debugging. > If you've got a situation that you can debug quickly with printf, you should do that. Early, sometimes student-friendly IDEs > These poor students are weeping in the Sun lab at two in the morning because they > can't debug their programs, because they're not allowed to use printf [@54:14](https://youtu.be/UOucW3F7nCg?t=3254) Research on statistical debugging from Ben Liblit [@57:32](https://youtu.be/UOucW3F7nCg?t=3452) > The disposition towards tooling changes once you've found your first bug with it. “I'm dealing with a house fire right now, it's not time for me to learn something new, my house is burning and I want to focus on that.” NOVA hypervisor debugging by inspecting registers > There's nothing quite like driving one of these unknown issues to the root cause; so satisfying. [@1:02:10](https://youtu.be/UOucW3F7nCg?t=3730) > I buy the argument that some of the lack of observability has been one of the strong motivators > for rooting out some of these problems earlier with CI/CD and test-driven development. [@1:03:04](https://youtu.be/UOucW3F7nCg?t=3784) > Dynamically instrumenting dynamic languages effectively requires VM cooperation. Perl 6 and Parrot book Raku née Perl 6 Adam's ten-year prediction: the end of Moore's Law will precipitate a culture of observability and debugging. If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!
Oxide and Friends Twitter Space: May 24, 2021from /proc to proc_macroWe've been holding a Twitter Space weekly on Mondays at 5p for about an hour. In addition to Bryan Cantrill and Adam Leventhal, speakers included Brian Cantrell (not making that one up!), Nima Johari, Joshua Clulow, Laura Abbott, and Tom Lyon. The recording is here.(Did we miss your name and/or get it wrong? Drop a PR!)Some of the topics we hit on, in the order that we hit them: The other Adam Leventhal [1] and the other AHL [2] [@3:16](https://youtu.be/85eApYSj3ic?t=196) Hockey Calder Cup Charlotte Checkers Grand Rapids Griffins [@4:02](https://youtu.be/85eApYSj3ic?t=242) Roger Faulkner invented the /proc filesystem Gerald Ford Presidential Library and MuseumGerald Ford inaugural address (including its most famous line, “our long national nightmare is over”) > I went in a Gerald Ford cynic, and came out a Gerald Ford super-fan Roger's “The Process File System and Process Model in UNIX System V” paper [@7:43](https://youtu.be/85eApYSj3ic?t=463) “I am on a mission from God to make programs debuggable” AVL trees and linked lists > Performance is the root of all evil. Trace Normal Form Watchpoints, libwatchmalloc > Watchpoints are magical, when they work. It feels like a superpower. [@11:37](https://youtu.be/85eApYSj3ic?t=697) > Roger made this incredible contribution about debugging infrastructure > being an attribute of a production system. strace, truss BONUS: 1986 USENIX: A System Call Tracer in UNIX The ptrace(2) system call ptrace's overloading of the wait(2) system call The German word that we're seeking: Misappropriation-of-mechanism-in-a-seemingly-clever way-but-is-ultimately-a-disaster > ptrace is the x86 of system calls [@16:45](https://youtu.be/85eApYSj3ic?t=1005) A long-coming apology.. Linux branded zones (LX) “Method and system for child-parent mechanism emulation via a general interface” patent > You have to be bug-for-bug compatible. LX vfork/signal bug that broke golang > vfork: unsafe at any speed, toxic in any quantity [@20:16](https://youtu.be/85eApYSj3ic?t=1216) Upstart's problematic use of ptrace(2) Celebrating Joshua getting ptrace correct for LX branded zones Stack shenanigans breaking LX Red zone, segmented stacks [@24:39](https://youtu.be/85eApYSj3ic?t=1479) The application was fishing in its own stack.. Clozure Common Lisp, mcontext > These kinds of lies just don't nest. Magic does not layer well. [@28:56](https://youtu.be/85eApYSj3ic?t=1736) Windows Subsystem for Linux WSL illumos on an M1? QEMU, ARM Cortex-M > It's hard to get the machine really properly emulated AWS Mac minis [@33:55](https://youtu.be/85eApYSj3ic?t=2035) It's kind of amazing that Apple has never had much interest in the server space. Apple Xserve CHRP The story of the stolen laptop. Little endian PowerPC OpenPOWER [@37:35](https://youtu.be/85eApYSj3ic?t=2255) Language H! NCR Language H: An informal overview ( part 1, part 2) The (other) D language [@39:12](https://youtu.be/85eApYSj3ic?t=2352) AADEBUG'03 Postmortem Object Type Identification [@41:31](https://youtu.be/85eApYSj3ic?t=2491) It all comes back to awk Bourne shell source code / Algol68 #defines Thompson shell Bryan's 2007 Dtrace review, Google TechTalk ~80mins [@48:07](https://youtu.be/85eApYSj3ic?t=2887) Dtrace language inspiration Dtrace clones > It was all based on us exploring some phenomenon, > something being kind of a pain in the ass or impossible, > and inventing something that was easy to use. Architectural review board: “This reminds us a lot of awk..” > What's the most powerful one-liner you can crank out with awk? CUDA, Bluespec [@52:35](https://youtu.be/85eApYSj3ic?t=3155) Rust proc_macros C preprocessor Rust macro_rules! > Reading about it for the first time, it felt like the forbidden fruit Tcl INTERCAL which might have been co-invented by Tom's brother?! Plan 9 (Did we miss anything? PRs always welcome!)Our next Twitter Space will be on May 31st, 2021 at 5p Pacific. This time for real: we'll be kicking off the discussion with Silicon Cowboys (the real-life Halt and Catch Fire) on the rise of Compaq – and their aspiration to be a different kind of company. Join us; we always love to hear from new speakers!
Ciclos de memoria, "core developers" y dataclasses https://podcast.jcea.es/python/12 En lo que sigue, cuando se habla de CPython, se refiere al intérprete de referencia de Python, que está escrito en lenguaje C: https://www.python.org/downloads/. Participantes: Eduardo Castro, email: info@ecdesign.es. Conectando desde A Guarda. Jesús Cea, email: jcea@jcea.es, twitter: @jcea, https://blog.jcea.es/, https://www.jcea.es/. Conectando desde Madrid. Javier, conectando desde Madrid. Víctor Ramírez, twitter: @virako, programador python y amante de vim, conectando desde Huelva. Juan Carlos. Audio editado por Pablo Gómez, twitter: @julebek. La música de la entrada y la salida es "Lightning Bugs", de Jason Shaw. Publicada en https://audionautix.com/ con licencia - Creative Commons Attribution 4.0 International License. [00:52] Seguimos hablando del bug comentado la semana pasada. bug bpo35930: "Raising an exception raised in a "future" instance will create reference cycles": https://bugs.python.org/issue35930. [02:17] El "bytecode" https://es.wikipedia.org/wiki/Bytecode que genera Python es muy mejorable. >>> import dis >>> def suma(valores): ... s=0 ... for i in valores: ... s+=i ... return s ... >>> dis.dis(suma) 2 0 LOAD_CONST 1 (0) 2 STORE_FAST 1 (s) 3 4 LOAD_FAST 0 (valores) 6 GET_ITER >> 8 FOR_ITER 12 (to 22) 10 STORE_FAST 2 (i) 4 12 LOAD_FAST 1 (s) 14 LOAD_FAST 2 (i) 16 INPLACE_ADD 18 STORE_FAST 1 (s) 20 JUMP_ABSOLUTE 8 5 >> 22 LOAD_FAST 1 (s) 24 RETURN_VALUE Inferencia de tipos: https://es.wikipedia.org/wiki/Inferencia_de_tipos. [08:32] Recogida de basuras. gc.set_threshold(): https://docs.python.org/3/library/gc.html#gc.set_threshold. gc.disable(): https://docs.python.org/3/library/gc.html#gc.disable. [11:27] Herramientas de monitorización: DTrace: http://dtrace.org/blogs/. Monitoriza el sistema operativo entero, incluyendo las aplicaciones, todo integrado, de forma segura y sin modificar el software. [13:32] Funcionalidades de auditoría de Python: PEP 551 -- Security transparency in the Python runtime https://www.python.org/dev/peps/pep-0551/. PEP 578 -- Python Runtime Audit Hooks https://www.python.org/dev/peps/pep-0578/. [16:47] Más herramientas de monitorización: SystemTap: https://es.wikipedia.org/wiki/SystemTap. eBPF: https://ebpf.io/. py-spy: https://github.com/benfred/py-spy. [17:52] Más sobre DTrace https://es.wikipedia.org/wiki/DTrace_(Sun_Microsystems) y Python: Añadir sondas DTrace al intérprete de Python: https://www.jcea.es/artic/python_dtrace.htm. [22:12] Tracemalloc. tracemalloc: https://docs.python.org/3/library/tracemalloc.html. [23:02] Seguimos hablando del bug comentado la semana pasada. bug bpo35930: "Raising an exception raised in a "future" instance will create reference cycles": https://bugs.python.org/issue35930. ¡Se ofrece una caja de cervezas! Brainstorming. Diagnóstico detallado. weakref — Weak references: https://docs.python.org/3/library/weakref.html. Se sube la apuesta a caja y media de cervezas :-). La excepción salta en un hilo y se "transporta" y almacena para que se pueda acceder desde otro hilo. Test reproducible. [36:42] Aviso legal. Machine learning para identificar los diferentes hablantes. [38:27] Las futuras notas de las grabaciones serán EXHAUSTIVAS (como estáis comprobando leyendo esto :). [39:17] Ideas para "cebar" las tertulias. Muchos temas recurrentes, se ve que hay temas "flotando" en el aire. [40:37] Cómo organizar las tertulias, diferentes intereses y profundidad. Dinámica de la tertulia. [42:32] ¿Cómo se organizan los "core developers"? El desarrollo se ha movido en github. Los bugs están a medio migrar, se va a integrar más en github. https://pyfound.blogspot.com/2020/05/pythons-migration-to-github-request-for.html PEP 581 -- Using GitHub Issues for CPython https://www.python.org/dev/peps/pep-0581/. Guía del desarrollador: https://devguide.python.org/. Backporting de bugs de cpython de la versión en desarrollo a las versiones estables. ¿Cómo se obtiene y se pierde el status de "core developer"? Steering council. PEP 8016: https://www.python.org/dev/peps/pep-8016/. Rol que cumple y cómo se elige. Desde que Guido no es BDFL, está muy activo en listas de correo y picando código. [52:22] ¡Víctor quiere más bugs para aprender! Bugs marcados como "easy", como forma de entrada a desarrolladores nuevos. [53:42] ¿Qué partes de CPython están escritas en C y cuáles en Python? Se escribe en C lo que no tiene más remedio, por rendimiento o porque interactúa con el sistema operativo. Más adelante de la conversación Jesús Cea explica cómo ver si un módulo concreto está en C o en Python sin tener que ir al código fuente. [57:32] PyPy https://www.pypy.org/. Intérprete de Python escrito en Python. RPython: https://rpython.readthedocs.io/en/latest/. [58:27] ¿Incluir otros lenguajes en la implementación de CPython? Rust: https://es.wikipedia.org/wiki/Rust_(lenguaje_de_programaci%C3%B3n). PyOxidizer: https://github.com/indygreg/PyOxidizer. Fragmentación. Jesús Cea estoy más centrado en la parte de C porque la mayor parte de los "core developers" no saben C. Añadir más lenguajes reduce el grupo de gente que puede mantener esas partes. Portabilidad de C. Bootstraping de un lenguaje con el propio lenguaje. Forth: https://en.wikipedia.org/wiki/Forth_(programming_language). [01:05:02] Python 3.9. Mejoras. Dificultades para utilizar la última versión de Python, en función de lo que tenga el cliente. [01:08:07] Dataclasses: https://docs.python.org/3/library/dataclasses.html. La dificultad para tener atributos opcionales. Algunas ideas. attrs: https://www.attrs.org/en/stable/. Usar valores "sentinel". DRY: https://es.wikipedia.org/wiki/No_te_repitas. [01:20:52] Pydantic: https://pydantic-docs.helpmanual.io/. [01:23:07] Horarios de las tertulias. Mucha discusión y algunas ideas. De momento no habrá cambios. Hace falta más feedback. Se agradecería que la gente que deje las tertulias, explicase por qué se ha ido. [01:30:27] Jesús Cea explica cómo ver si un módulo concreto está en C o en Python sin tener que ir al código fuente. [01:31:18] Más sobre la dinámica de las tertulias. Debate sobre presentarse o no en tertulias abiertas, o tener la cámara apagada. Va siendo necesario tener algun repositorio para que la gente de la tertulia pueda compartir cosas. ¿Lista de correo específica para las tertulias? [01:36:42] Actas de las tertulias y publicar las grabaciones de una puñetera vez. ¿Algún ingeniero de sonido en la sala? ¿Baratito? [01:39:08] El "nivel" de las listas de correo. ¿Dónde están las conversaciones interesantes? (aparte de la tertulia semanal :-). La maldición de lo básico e "introducción a". Igual para que haya conversación interesante, hay que hacer preguntas interesantes :-). Python-Madrid antes de que llegase Meetup. Jesús Cea sugiere listas como "python-ideas": https://mail.python.org/mailman3/lists/python-ideas.python.org/. También la lista de programación Python en español: python-es@python.org. Javier tiene intereses muy extraños :-). [01:54:52] Cierre. [01:56:42] Final.
Más de lo que nunca quisiste aprender sobre JIT, guardas y especialización https://podcast.jcea.es/python/11 En lo que sigue, cuando se habla de CPython, se refiere al intérprete de referencia de Python, que está escrito en lenguaje C: https://www.python.org/downloads/. Participantes: Eduardo Castro, email: info@ecdesign.es. Conectando desde A Guarda. Jesús Cea, email: jcea@jcea.es, twitter: @jcea, https://blog.jcea.es/, https://www.jcea.es/. Conectando desde Madrid. Javier, conectando desde Madrid. Víctor Ramírez, twitter: @virako, programador python y amante de vim, conectando desde Huelva. Miguel Sánchez, email: msanchez@uninet.edu, conectando desde Canarias. Audio editado por Pablo Gómez, twitter: @julebek. La música de la entrada y la salida es "Lightning Bugs", de Jason Shaw. Publicada en https://audionautix.com/ con licencia - Creative Commons Attribution 4.0 International License. [00:52] Aviso de que se está grabando. Temas legales. [01:52] Valor de publicar estos audios y las dificultades para hacerlo. [02:42] Métodos mágicos: __set_name__(). PEP 487: https://www.python.org/dev/peps/pep-0487/. [04:12] Problemas con PIP 20.3.2: https://github.com/pypa/pip/issues/9284. [05:52] ¿Actualizar a la última versión o esperar? Poder "echar atrás" fácil. Acumular cambios pendientes es deuda técnica. [10:42] Google caído https://www.theguardian.com/technology/2020/dec/14/google-suffers-worldwide-outage-with-gmail-youtube-and-other-services-down. [11:02] Generación de wheels en varios sistemas: https://pythonwheels.com/. auditwheel: https://pypi.org/project/auditwheel/. ¿Generación de Wheels en Microsoft Windows? [13:12] Caché local de PIP https://pip.pypa.io/en/stable/. [14:17] Event Sourcing https://dev.to/barryosull/event-sourcing-what-it-is-and-why-its-awesome. Módulo eventsourcing: https://pypi.org/project/eventsourcing/. [14:42] De momento se puede usar el viejo "resolver" de dependencias de PIP. Se puede usar la opción -use-deprecated=legacy-resolver. Esa opción se puede meter también en el fichero de configuración, para no tener que escribirlo en cada invocación. Jesús Cea comete el pecado de meter paquetes Python en el sistema operativo. [17:02] Batallitas de Jesús Cea. Jesús lleva dos años dándole vueltas a esto: bpo35930: "Raising an exception raised in a "future" instance will create reference cycles": https://bugs.python.org/issue35930. Explicación detallada del asunto. Brainstorming. [21:22] Visión a alto nivel del recolector de basuras de Python (cpython) Contador de referencias. Inmediato, pero no recoge ciclos. Si se crean instancias y no se destruyen, se llama a un recolector "pesado" que también recoge ciclos. Esto puede ser problemático al arrancar el programa, antes de que la creación/destrucción de objetos se "estabilice". gc.disable(): https://docs.python.org/3/library/gc.html#gc.disable. Jesús Cea "abusa" de los destructores y de que se ejecuten cuando él quiere. Lo práctico contra lo puro. Jesús ofrece cervezas. gc.collect(): https://docs.python.org/3/library/gc.html#gc.collect. Esto sirve tanto para recoger los ciclos como para comprobar si tu programa tiene ciclos de memoria o no. Futures: https://docs.python.org/3/library/concurrent.futures.html. [35:29] Módulo Manhole https://pypi.org/project/manhole/. Explorar un programa en producción. Tracemalloc: https://docs.python.org/3/library/tracemalloc.html. DTrace: http://dtrace.org/blogs/about/. py-spy: https://pypi.org/project/py-spy/. Pérdidas de memoria: Recordar lo hablado ya en tertulias anteriores. jemalloc: http://jemalloc.net/. MALLOC_PERTURB_: https://debarshiray.wordpress.com/2016/04/09/malloc_perturb_/. zswap: https://en.wikipedia.org/wiki/Zswap. [42:52] Micropython: https://micropython.org/. ESP8266: https://en.wikipedia.org/wiki/ESP8266. ESP32: https://en.wikipedia.org/wiki/ESP32. Bluetooth Low Energy: https://en.wikipedia.org/wiki/Bluetooth_Low_Energy. ¿Qué ventajas aporta usar Micropython? Velocidad de desarrollo y depuración. [52:42] ¿El futuro será mejor? O no. Desperdicio de recursos materiales porque realmente sobran. Python es mucho más lento que C y no digamos ensamblador. [57:17] Cambiar Python por un lenguaje más rápido. Go: https://en.wikipedia.org/wiki/Go_(programming_language). Rust: https://en.wikipedia.org/wiki/Rust_(programming_language). C++: https://en.wikipedia.org/wiki/C%2B%2B. [01:00:20] Python no pinta nada en móviles. Kivy: https://kivy.org/. [01:02:07] Acelerar Python. Subinterpreters: PEP 554: https://www.python.org/dev/peps/pep-0554/. Si los subintérpretes no compartiesen NADA, se podrían lanzar simultaneamente en varios núcleos de la CPU sin competir por un GIL https://en.wikipedia.org/wiki/Global_interpreter_lock único. JIT: https://es.wikipedia.org/wiki/Compilaci%C3%B3n_en_tiempo_de_ejecuci%C3%B3n. PYPY: https://www.pypy.org/. RPython: https://rpython.readthedocs.io/en/latest/. Numba: https://numba.pydata.org/. Cython: https://cython.org/. Python es "potencialmente" muy dinámico, pero en la práctica los programas no lo son. Jesús pone varios ejemplos. Conversación densa entre Jesús y Javier. Guardas para comprobar que la especialización sigue siendo correcta. Por ejemplo, para los diccionarios: PEP 509 Add a private version to dict: https://www.python.org/dev/peps/pep-0509/ "Tipado" más estricto. MYPY: http://mypy-lang.org/. Pydantic: https://pydantic-docs.helpmanual.io/. Comprobación de tipos en tiempo de ejecución. Descubrimiento de tipos en tiempo de ejecución, proporcionando "especialización". psyco: https://en.wikipedia.org/wiki/Psyco. Eduardo Castro entra y simplifica la discusión. Jesús explica qué hace "a+b" internamente. [01:29:22] PyParallel http://pyparallel.org/ Memoria transaccional: https://es.wikipedia.org/wiki/Memoria_transaccional. (nota de Jesús Cea): Los sistemas de persistencia Python descritos en tertulias anteriores pueden considerarse casos de memoria transaccional... si somos flexibles. "Colorear" objetos y que dos hilos no puedan acceder a objetos del mismo color simultaneamente o en transacciones concurrentes. [01:30:42] PYPY https://www.pypy.org/ es tan sofisticado que no lo entiende ni dios. Jesús Cea lo ha intentado y se ha rendido. psyco: https://en.wikipedia.org/wiki/Psyco. CFFI: https://cffi.readthedocs.io/en/latest/. [01:35:22] Compilar CPython a WebAssembly https://en.wikipedia.org/wiki/WebAssembly va más rápido que en C nativo. [01:36:02] Simplemente compilar código python con Cython https://cython.org/ sin declaración de tipos dobla la velocidad de ejecución. ¡CPython lo puede hacer mejor! [01:36:57] Subinterpreters: PEP 554: https://www.python.org/dev/peps/pep-0554/. Poder usar todos los núcleos de la CPU. [01:38:07] Seguimos hablando del asunto. [01:39:07] Un problema es que Python tiene la vocación de funcionar en todas partes, así que hay resistencia para implementar mejoras solo en ciertas plataformas. [01:40:17] Cierre. Dadle una pesada al bug bpo35930: "Raising an exception raised in a "future" instance will create reference cycles": https://bugs.python.org/issue35930. [01:41:13] Final.
Adventures in Freebernetes, tracing kernel functions, The better way of building FreeBSD networks, New beginnings: CDBUG virtual meetings, LibreSSL update in DragonFly, Signal-cli with scli on FreeBSD, and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines Adventures in Freebernetes: bhyve My Guest (https://productionwithscissors.run/2020/10/29/adventures-in-freebernetes-bhyve-my-guest/) Part 2 of experiments in FreeBSD and Kubernetes: Creating your first guest Tracing Kernel Functions: FBT stack() and arg (https://zinascii.com/2020/fbt-args-and-stack.html?s=03) In my previous post I described how FBT intercepts function calls and vectors them into the DTrace framework. That laid the foundation for what I want to discuss in this post: the implementation of the stack() action and built-in arg variables. These features rely on the precise layout of the stack, the details of which I touched on previously. In this post I hope to illuminate those details a bit more with the help of some visuals, and then guide you through the implementation of these two DTrace features as they relate to the FBT provider. News Roundup Dummynet: The Better Way of Building FreeBSD Networks (https://klarasystems.com/articles/dummynet-the-better-way-of-building-freebsd-networks/) Dummynet is the FreeBSD traffic shaper, packet scheduler, and network emulator. Dummynet allows you to emulate a whole set of network environments in a straight-forward way. It has the ability to model delay, packet loss, and can act as a traffic shaper and policer. Dummynet is roughly equivalent to netem in Linux, but we have found that dummynet is easier to integrate and provides much more consistent results. New beginnings: CDBUG virtual meetings (http://lists.nycbug.org/pipermail/cdbug-talk/2020-October/000901.html) I had overwhelmingly positive responses from the broader *BSD community about restarting CDBUG meetings as virtual, at least for now. Hopefully this works well and even when we're back to in-person meetings we can still find a way to bring in virtual attendees. LibreSSL update in DragonFly (https://www.dragonflydigest.com/2020/11/10/25143.html) DragonFly has a new version of libressl, noting cause it has a newer TLS1.3 implementation – something that may be necessary for you. Signal-cli with scli on FreeBSD (https://antranigv.am/weblog_en/posts/freebsd-signal-cli-scli/) So couple of days ago I migrated from macOS on Macbook Pro to FreeBSD on ThinkPad T480s. Beastie Bits Firefox is not paxctl safe for NetBSD (https://anonhg.netbsd.org/pkgsrc/rev/9386adbd052e) FreeBSD 12.2-RELEASE on Microsoft Azure Marketplace (https://azuremarketplace.microsoft.com/en-us/marketplace/apps/thefreebsdfoundation.freebsd-12_2?tab=Overview) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions carlos - BSD Now around the world (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/379/feedback/carlos%20-%20BSD%20Now%20around%20the%20world.md) paulo - freebsd on a Bananapi (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/379/feedback/paulo%20-%20freebsd%20on%20a%20Bananapi.md) paulo - followup (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/379/feedback/paulo%20-%20followup.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***
We review the Raspberry Pi 400. Then discover new features coming to Linux powered Dells. Plus an important Let's Encrypt update, and the next billion-dollar tech coming to Linux.
We review the Raspberry Pi 400. Then discover new features coming to Linux powered Dells. Plus an important Let's Encrypt update, and the next billion-dollar tech coming to Linux.
We review the Raspberry Pi 400. Then discover new features coming to Linux powered Dells. Plus an important Let's Encrypt update, and the next billion-dollar tech coming to Linux.
New Project: zedfs.com, TrueNAS CORE Ready for Deployment, IPC in FreeBSD 11: Performance Analysis, Unix Wildcards Gone Wild, Unix Wars, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines My New Project: zedfs.com (https://www.oshogbo.vexillium.org/blog/80/) Have you ever had an idea that keeps coming back to you over and over again? For a week? For a month? I know that feeling. My new project was born from this feeling. On this blog, I mix content a lot. I have written personal posts (not many of them, but still), FreeBSD development posts, development posts, security posts, and ZFS posts. This mixed content can be problematic sometimes. I share a lot of stuff here, and readers don’t know what to expect next. I am just excited by so many things, and I want to share that excitement with you! TrueNAS CORE is Ready for Deployment (https://www.ixsystems.com/blog/truenas-12-rc-1/) TrueNAS 12.0 RC1 was released yesterday and with it, TrueNAS CORE is ready for deployment. The merger of FreeNAS and TrueNAS into a unified software image can now begin its path into mainstream use. TrueNAS CORE is the new FreeNAS and is on schedule. The TrueNAS 12.0 BETA process started in June and has been the most successful BETA release ever with more than 3,000 users and only minor issues. Ars Technica provided a detailed technical walkthrough of the original BETA. There is a long list of features and performance improvements. During the BETA process, TrueNAS 12.0 demonstrated over 1.2 Million IOPS and over 23GB/s on a TrueNAS M60. News Roundup Interprocess Communication in FreeBSD 11: Performance Analysis (https://arxiv.org/pdf/2008.02145.pdf) Interprocess communication, IPC, is one of the most fundamental functions of a modern operating system, playing an essential role in the fabric of contemporary applications. This report conducts an investigation in FreeBSD of the real world performance considerations behind two of the most common IPC mechanisms; pipes and sockets. A simple benchmark provides a fair sense of effective bandwidth for each, and analysis using DTrace, hardware performance counters and the operating system’s source code is presented. We note that pipes outperform sockets by 63% on average across all configurations, and further that the size of userspace transmission buffers has a profound effect on performance — larger buffers are beneficial up to a point (∼ 32-64 KiB) after which performance collapses as a result of devastating cache exhaustion. A deep scrutiny of the probe effects at play is also presented, justifying the validity of conclusions drawn from these experiments. Back To The Future: Unix Wildcards Gone Wild (https://www.defensecode.com/public/DefenseCode_Unix_WildCards_Gone_Wild.txt) First of all, this article has nothing to do with modern hacking techniques like ASLR bypass, ROP exploits, 0day remote kernel exploits or Chrome's Chain-14-Different-Bugs-To-Get-There... Nope, nothing of the above. This article will cover one interesting old-school Unix hacking technique, that will still work nowadays in 2013. Unix Wars (https://www.livinginternet.com/i/iw_unix_war.htm) Dozens of different operating systems have been developed over the years, but only Unix has grown in so many varieties. There are three main branches. Four factors have facilitated this growth... Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Chris - installing FreeBSD 13-current (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/371/feedback/Chris%20-%20installing%20FreeBSD%2013-current.md) Dane - FreeBSD History Lesson (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/371/feedback/Dane%20-%20FreeBSD%20History%20Lesson.md) Marc - linux compat (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/371/feedback/Marc%20-%20linux%20compat.md) Mason - apropos battery (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/371/feedback/Mason%20-%20apropos%20battery.md) Paul - a topic idea (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/371/feedback/Paul%20-%20a%20topic%20idea.md) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
PHP Internals News: Episode 68: Observer API London, UK Thursday, September 17th 2020, 09:31 BST In this episode of "PHP Internals News" I chat with Levi Morrison (Twitter, GitHub) and Sammy Kaye Powers (Twitter, GitHub, Website) about the new Observer API. The RSS feed for this podcast is https://derickrethans.nl/feed-phpinternalsnews.xml, you can download this episode's MP3 file, and it's available on Spotify and iTunes. There is a dedicated website: https://phpinternals.news Transcript Derick Rethans 0:15 Hi, I'm Derick, and this is PHP internals news, a weekly podcast dedicated to demystifying the development of the PHP language. This is Episode 68. Today I'm talking with Levi Morrison, and Sammy Powers, about something called the observer API, which is something that is new in PHP eight zero. Now, we've already passed feature freeze, of course, but this snuck in at the last possible moment. What this is observer API going to solve? Levi Morrison 0:44 the observer API is primarily aimed at recording function calls in some way so it can also handle include, and require, and eval, and potentially in the future other things, but this is important because it allows you to write tools that automatically observe, hence the name, when a function begins or ends, or both. Derick Rethans 1:12 What would you use that for? Levi Morrison 1:13 So as an example, Xdebug can use this to know when functions are entered or or left, and other tools such as application performance monitoring, or APM tools like data dog, New Relic, tideways, instana so on, they can use these hooks too. Derick Rethans 1:38 From what I understand that is the point you're coming in from, because we haven't actually done a proper introduction, which I forgot about. I've been out of this for doing this for a while. So both you and Sammy you work for data dog and work on their APM tool, which made you start doing this, I suppose. Sammy Kaye Powers 1:54 Yeah, absolutely. One of the pain points of tying into the engine to to monitor things is that the hooks are insufficient in a number of different ways. The primary way that you would do function call interception is with a little hook called zend_execute_ex and this will hook all userland function calls. The problem is, it has an inherent stack bomb in it where if, depending on your stack size settings you, you're going to blow up your stack. At some point if you have a very very deeply deep call stack in PHP, PHP, technically has a virtually unlimited call stack. But when you use zend_execute_ex, it actually does limit your stack size to whatever your settings are your ulimit set stack size. One of the issues that this solves is that stack overflow issue that you can run into when intercepting userland calls but the other thing that it solves is the potential JIT issues that are coming with PHP eight, where the optimizations that it does could potentially optimize out a call to zend_execute_ex where a profiling or APM tracing kind of extension would not be able to enter set that call, because of the JIT. The Observer API enables to solve multiple issues with this. Not only that, there's more. there's more features to this thing, because zend_execute_ex by default will intercept all userland function calls, and you have no choice but to intercept every single call, whereas, this API is designed to also allow you to choose which function calls specifically you want to intercept, so there is on the very first call of a function call. And it'll basically send in the zend function. This is a little bit of a point we've been kind of going back and forth on what we actually send in on the initialisation. But at the moment it is a zend function so you can kind of look at that and say okay do I want to monitor this function or observe it. If your extension returns a handler and says this is my begin handler This is my end handler. Those handlers will fire at the beginning and end of every function call. So it gives you a little bit of fine grain sort of resolution on what you want to observe. The other really kind of baked in design, part of the design is, we wanted it to play well with neighbours, because some of the hooks, at the moment, well, pretty much all of the hooks. Aside from typical extension hooks. Whenever you tie into the engine it's very easy to be a noisy neighbor. It's very easy not to forward a hook along properly it's very easy to break something for another extension. This has like kind of neighbour considerations baked right in. So when you actually request a begin and end hook. It will manage on its side, how to actually call those, and so you don't have to forward along the hook to other other extensions to make it a little bit more stable in that sense. Derick Rethans 4:52 From working on Xdebug, this is definitely problem forwarding both zend_execute_ex and also zend_execute_internal, which is something I also override are of course. And I think there are similar issues with, with the error display as well and PHP eight will also have a different, or a new API for that as well. Also coming out of a different performance monitoring tool, which is interesting to see that all these things works. You mentioned the Zend function thing and I'm not sure how well versed the audiences and all this internal things what is this zend function thing? Levi Morrison 5:24 as any function in the engine is what represents a function so not the scope that it's called in but the scope that it's defined in. It represents both method calls and function calls. It's triggered whenever a user land function is in play. So it has the function name, the name of the class that it's associated with, it tells you how many parameters you have and things like this. It does not tell you the final object that it's called with, and this is partly why we are debating what exactly should get passed in here, because some people may care. Oh, I only want to observe this with particular inheritors or, or other things of that nature so there's a little bit of fine tuning in the design perhaps still but the basic idea is you'll know the name of the function. What class it's in, and it's bound late enough in the engine that you would also have access to whatever parents that class has, etc. Derick Rethans 6:33 Does it contain the arguments as well, that are being sent, or just a definition of the arguments? Levi Morrison 6:38 The Zend function only contains the definition of the arguments. The hook is split into three sections kind of so there's like initialisation and then begin and end. Initialisation only gives you the Zendo function but to begin and gives you access to what's called the Zend execute data which has more information, including the actual arguments being passed. Derick Rethans 7:03 Okay, so it's the idea of the initialisation, just to find out whether you actually want to intercept the function. And if you want remember that and if not it wouldn't ever bother are the trying to intercept that specific zend function either. Sammy Kaye Powers 7:17 Actually what we actually pass into that initialization function is has been sort of up for debate. The original implementations, that is plural. We've had many different implementations of this thing over the, over the year. Derick you did mention that this got squeezed in last minute it has been a work in progress for a very long time and it actually is fulfilling JIT work so there's a specific mention in the JIT RFC that that mentions an API that is going to be required to intercept some function calls that are optimized out so that's why we were able to sneak in a little bit past feature freeze on the actual merge I think. But what we actually sent into this initialization function is spin up two for debate based on how we've actually implemented it. One of the original early implementations actually called this initialization function during the runtime cache initialization, just basically kind of a cache that gets initialized before the execute data actually is created. We didn't have the option of sending in the execute data at that time, we did have the zend function. So we were sending that in. Later on this implementation get refactored to a number different ways. We have the option now to send an execute data if we wanted to, but it might be sufficient to send in the op array instead of the Zend function. The op array should be the full sort of set of opcodes that basically is a function definition from the perspective of of internals, but it also includes like includes and evals. Having additional information at initialisation might actually be handy. I think we're still kind of maybe thinking about that potentially changing I don't know what do you think Levi. Levi Morrison 8:59 Yeah, you can get the oparray from the function so it's a little pedantic on which one you pass in I guess, but yeah. The idea is that we don't want to intentionally restrict it. It's just that the implementations have changed over the year so we're not sure exactly what to pass in at the moment. I think a zend function's pretty safe, passing in a zend oparray is perhaps a better signal to what it's actually for, because it can measure function calls, but also include, require, eval. And the oparray technically does contain more information. Again, if you have zend function, you can check to see if it is an oparray and get the operate from the Zend function. So a little pedantic but maybe a little better in conveying the purpose and what exactly it targets. Derick Rethans 9:56 And you can also get the oparray from zend_execute_data. Levi Morrison 10:00 Yeah. Derick Rethans 10:01 If I want to make use of this observe API I will do that? I guess, you said only from extensions and not from userland. Sammy Kaye Powers 10:08 Exactly. At the moment you would as an extension during MINIT or startup, basically in the very early process with the actual PHP processes starting up, would basically register your initialization handler. And at that point, under the hood, the whole course of history is changed for PHP at that point, because there is a specific observer path that happens when an observer extension registers at MINIT or startup. At that point the initialization function will get called for every function call. The very first function call that that function called is called, I know that sounds confusing but if you think you have one function and it's called 100 times that initialization will run one time. That point you can return either a begin and or at an end handler. If you return null handlers it'll never, it'll never bother you again for that particular function, but it will continue to go on that is don't mentioned earlier for every new function that encounters every new function call and encounters, I should say. Derick Rethans 11:12 There is not much overhead, because the whole idea is that you want to do as little overhead as possible I suppose. Levi Morrison 11:19 Exactly, we have in our current design in pre PHP 8.0. You could hook into all function calls using that zend_execute_ex, but it has performance overhead just for doing the call. So let's imagine we're in a scenario where we have two extensions, say Xdebug and one APM product. Both of them aren't actually going to do anything on this particular call it will still call those hooks, which has overhead to it. So if nobody is interested in a function, the engine can very quickly determine this and avoid the overhead of doing nothing. This way we only pay significant costs, if there's something to be done. Derick Rethans 12:09 You're talking about not providing much overhead at all. Just having the observer API in place, was there any performance hits with that? Sammy Kaye Powers 12:17 That was actually one of the biggest sort of hurdles that we had to overcome specifically with Dmitri getting this thing, merged in because it does touch the VM and whenever you touch the VM like we're talking like any tiny little null check that you have in any of the handlers is probably going to have some sort of impact at least enough for Dmitri, who understandably cares about like very very very small overheads that are happening at the VM level, because these are happening for every function call. You know, this is, this is not something that's just happening, you know, one time during the request is happening a lot. In order to apeace Dmitri and get this thing merged in, it basically had to have zero overhead for the production version non observer, his production version but on the non observed version on the non observed path it had to basically reach zero on those benchmarks. That was quite a task to try to achieve. We went through four, about four or five different ways of tying into the engine, we got it down to about, like, two new checks for every function call. And that still was not sufficient, so we end up going with based on Dmitris excellent suggestion, went with the opcode specialization, to implement observers so that at compile time. We can look and see if there's an observer extension present and if there is, it will actually divert the function call related opcodes to specific handlers that are designed for observing and that way once, once you get past that point, the observer path is already determined at compile time and all the observer handlers fire. In a non observed environment, all of the regular handlers will fire without any observability checks in them. Derick Rethans 14:03 At the end of getting within the loss of zero or not? Levi Morrison 14:07 It is zero for certain things. Of course, there are other places besides the VM that you have to touch things here and there for, you know, keeping code tidy and code sane but it's effectively zero, for all intents and purposes. Goal achieved. I will say zero percent. Derick Rethans 14:30 I think the last version of the patch that I saw didn't have the code specialization in it yet. So I'm going to have to have a look at myself again. Levi Morrison 14:39 Yeah, the previous version had very low overhead, so low overhead that you couldn't really observe it through any time based things. But if you measured instructions retired or similar things from the CPU, then it was about point four to 1% reduction, and personally I would have said that's good enough because all of them would correctly branch predict, because you either have handlers in a request, or you don't. And so they would perfectly predict, every time. But still, those are extra instructions technically so that's why Dmitri pushed for specialization and those are no longer there. Derick Rethans 15:27 Does that mean there are new opcodes specifically for this, or is it just the specialization of the opcodes that is different? Sammy Kaye Powers 15:33 It's just this specialization. During the process of going, figuring out what exactly Dmitri needed to mergeable actually proposed an implementation that added basically an observer version of every kind of function call related opcode like do_fcall_observed, or observed_return or something like that. With, opcodes specialization, it reduces the amount of code that you have to write sort of at the VM level, it doesn't change the amount of code that's generated though because with opcode specialization, basically the definition file will get expanded by this, this php file that actually generates C code. When you add a specialization, to a handler that already has specializations on it, it will expand quite considerably. The PR at one point ended up being like 10,000 lines or something like that, so we had to do some serious reduction on the number of handlers that were automatically generated. Long story short, is there are no new opcodes but there are new opcode handlers to handle this specific path. Derick Rethans 16:40 Not sure what, if anything more to ask about the Observer API, do you have anything to ask yourself? Levi Morrison 16:45 I think it's worth repeating the merits of the observer API and where we're coming from. The key benefits in my opinion are that it allows you to target per function interception for observing. It allows you to do it in a way that's that plays nice with other people in the ecosystem and increasingly that's becoming more important. We've always had debuggers and some people occasionally need to turn debuggers on in production and other things like this. But increasingly, there are other products in this space; the number of APM products is growing over time. There are new security products that are also using these kinds of hooks. And I expect over time we will see more and more and more of these kinds of of tools, and so being able to play nicely is a very large benefit. At data dog where Sammy and I both work we've hit product incompatibilities a lot of times, and some people are better to work with than others. I know that Xdebug has done some work to be compatible with our product specifically but you know competitors aren't so interested in that. We care a lot about the community right, we want the whole community to have good tools, and I don't think we actually mentioned yet that we did collaborate with some other people and competitors in this space. That hopefully proves that that's not just words of mine that, you know, we actually met with the competitors who were willing to and discussed API design, and use cases, and making sure that we could all work together and compete on giving PHP good products rather than, you know, hoarding technical expertise and running over each other and causing incompatibilities with each other. So I think those are really important things. And then lastly, it does not have that stack overflow potential that the previous hooks you could use did. Derick Rethans 18:54 Yeah, which is still an issue for Xdebug but but I fixed that in a different way by setting an arbitrary limit on the amount of nested levels you can call, right. Levi Morrison 19:02 Yeah, and in practice that tends to work pretty well because most people don't have intentionally deep code. But for some people they do. And we can't as an APM product for instance say: sorry your code is just not good code, we can't observe a crash your your your thing and so we can't make that decision. And then the biggest con at the moment is that it doesn't work with JIT, but I want to specifically mention that, that's not a technical thing, that's just a not enough time has been put in that space yet because this was crunched to the last second trying to get it in. And so, some things didn't get as much focus yet. Hopefully by the time 8.0 gets released it will be compatible with JIT, or at least it will be only per function, so maybe a function that gets observed, maybe that can't be JIT compiled that specific function call, but all the other function calls that aren't observed would be able to. We'll see obviously there's still work to do there but that's our hope. Derick Rethans 20:10 What happens now, if, if you use the observer API and the JIT engine is active? Does it just disable the JIT engine. Sammy Kaye Powers 20:16 Yep. It just won't enable the JIT at all. In fact, it just goes ahead and disables it, if an observer extension is enabled and there is a little kind of debug message that's emitted inside of the OP cache specific logs that will will say specifically why the JIT isn't enabled just in case you're sitting here trying to turn the JIT on you're like, why isn't enabled, and it'll say there's an observer extension present so we can enable the JIT. Hopefully they'll be able to work a little bit, and maybe just change an optimization level or something in the future. I'd like to give a shout out to Benjamin Eberlei, who has been with us since the very beginning on this whole thing has been vetting the API on his products, has gotten xhprof on not only the original implementation but also on the newest implementation, and has just been a huge help in actually getting this thing pushed in, and was said some of the magic words that actually, this thing merged in, when it was looking like it wasn't gonna land for eight dot O and got it landed for eight dot one so Benjamin gets a huge thumbs up. So, Nikita Popov, Bob Weinand, and Joe Watkins really early on. These are awesome people from internals who have spent some time to help us vet the API, but also to help us with specific implementation details. It's been just a huge team effort from a lot of people and it was just like, really great to work across the board with everybody. Derick Rethans 21:35 Yeah, and the only thing right now of course is all the extensions that do observe things need to be compatible with this. Sammy Kaye Powers 21:43 Exactly. Derick Rethans 21:44 Which is also means there's work for me, or potentially. Sammy Kaye Powers 21:47 Absolutely. Levi Morrison 21:49 I guess one one minor point there is that if an extension does move to the new API, it is a little bit insulated from those that haven't moved to the new API. So, to some degree, it still benefits the people who haven't moved yet because the people who have moved have one less competitor in the same same hook, so it's just highlighting the fact that it plays nicely with other people. Derick Rethans 22:14 Is opcache itself actually going to use it or not? Levi Morrison 22:17 So this is focused only on userland functions; past iterations that was not the case. Dmitri kind of pushed back on having this API for internals and so that got dropped. I don't think at this stage there's any there's any value in opcache using it specifically, but there are some other built in things like Dtrace. I don't know how many people actually use Dtrace; I actually have used it once or twice, but Dtrace could use this hook in the future instead of having a custom code path and things like that. Derick Rethans 22:49 For Xdebug I still need to support PHP seven two and up, so I'm not sure how much work it is doing it right now, but definitely something to look into and move to in the future, I suppose. Well thank you very much to have a chat with me this morning. I can see that for Sammy the sun has now come up and I can see his face. Thanks for talking to me this morning. Sammy Kaye Powers 23:10 Thanks so much, Derick and I really appreciate all the hard work you put into this because I know firsthand experience how much work podcasts are so I really appreciate the determination to continue putting out episodes. It's a huge amount of work so thanks for being consistent. Levi Morrison 23:26 Yeah, thank you so much for having us Derick. Derick Rethans 23:30 Thanks for listening to this installment of PHP internals news, the weekly podcast dedicated to demystifying the development of the PHP language. I maintain a Patreon account for supporters of this podcast, as well as the Xdebug debugging tool. You can sign up for Patreon at https://drck.me/patroen. If you have comments or suggestions, feel free to email them to derick@phpinternals.news. Thank you for listening, and I'll see you next week. Show Notes Pull Request: https://github.com/php/php-src/pull/5857 Credits Music: Chipper Doodle v2 — Kevin MacLeod (incompetech.com) — Creative Commons: By Attribution 3.0
Introduction to dtrace. --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app Support this podcast: https://anchor.fm/jesu-tinto/support
스탠다드아웃 71번째 로그에서는 팟캐스트 녹음 환경, 아마존닷컴 반품 경험, gRPC 로드 밸런싱 대한 이야기를 나눴습니다. 참가자: @nacyo_t, @raccoonyy, @seapy, @ecleya 정기 후원 - stdout.fm are creating 프로그래머들의 팟캐스트 | Patreon 녹화 환경 소개 α6400 E-mount camera with APS-C Sensor | ILCE-6400 / ILCE-6400L / ILCE-6400M | Sony US FE 16–35 mm G Master Wide-Angle Zoom Lens | SEL1635GM | Sony US DSC-RX100 III Compact Digital Camera | Cyber-shot Pocket Camera | Sony US 4K HDR Camcorder with Fast Hybrid AF | 4K Handycam FDR-AX700 | Sony US FDR-AX60 | デジタルビデオカメラ Handycam ハンディカム | ソニー 홍진경 더 만두 리뷰를 빙자한 a6400 + SEL1635FE + MixPre-3 II + Shure SM57 - YouTube Good genes: Samsung NX500 review posted: Digital Photography Review 인사이트 번역가 모집 도서출판 인사이트 - BPF Performance Tools: Linux System and Application Observability 역자 모집(마감) 시스템 성능 분석과 최적화 - YES24 DTrace - Wikipedia Berkeley Packet Filter - Wikipedia Brendan Gregg’s Homepage 도서출판 인사이트 - Rust in Action 역자 모집(마감) 알라딘: 클린 아키텍처 알라딘: 클린 코드 Clean Code 기계는 어떻게 생각하는가? - YES24 ODK는 구인중 ODK Media HackerRank 유튜브 익스플로어 지원 중단 3월부터 인터넷 익스플로러로 유튜브 못 본다 : IT : 경제 : 뉴스 : 한겨레 WebP - Wikipedia VP9 - Wikipedia 오라클 신한은행 라이센스 분쟁 오라클 “신한은행, 수백억 내놔라” : 클리앙 AWS Outposts 개요 페이지 데이터 베이스 관리 시스템 | MySQL | Amazon Web Services Amazon Aurora 서버리스 - 온디맨드 Auto-scaling 관계형 데이터베이스 - AWS 판교 낙생지구 남판교에 1만가구 주거타운 뜬다…대장동 이어 낙생지구 개발 - 땅집고 > 투자리포트 아마존 반품 이야기 Dart: The World’s Smallest Laptop Adapter by FINsix — Kickstarter Wireless intrusion prevention system - Wikipedia 속도내는 스마트폰 열풍 구글 넥서스원 국내 첫 개통자나와 | 한경닷컴 아이패드 사용 불법이라더니..장관은 예외? - Chosunbiz > 테크 > ICT/미디어 해외 직구 되팔기 ‘불법’이라는 정부… 직구족들 “현실과 괴리” 불만 - 중앙일보 키크론 Keychron – 맥도 윈도우도 문제없다. 신고 안한 샤넬백, 공항서 걸릴 확률은? Smart Noise Cancelling Headphones 700 | Bose Alibaba.com: Manufacturers, Suppliers, Exporters & Importers from the world’s largest online B2B marketplace gRPC 로드밸런싱 gRPC 웹 서버 로드 밸런싱 | 서버 로드 밸런싱 | Amazon Web Services Envoy Proxy - Home Cloud Map – 클라우드 리소스를 위한 서비스 검색 AWS 라이트세일 Amazon Lightsail, 이제 EC2 업그레이드 경로 제공 DigitalOcean – The developer cloud Amazon EC2 Instance Comparison AWS Batch – 쉽고 효율적인 배치 컴퓨팅 기능 – AWS
LLDB Threading support now ready, Multiple IPSec VPN tunnels with FreeBSD, Netflix Optimized FreeBSD's Network Stack More Than Doubled AMD EPYC Performance, happy eyeballs with unwind(8), AWS got FreeBSD ARM 12, OpenSSH U2F/FIDO support, and more. Headlines LLDB Threading support now ready for mainline (https://blog.netbsd.org/tnf/entry/lldb_threading_support_now_ready) Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages. In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support, extending NetBSD's ptrace interface to cover more register types and fix compat32 issues and fixing watchpoint support. Then, I've started working on improving thread support which is taking longer than expected. You can read more about that in my September 2019 report. So far the number of issues uncovered while enabling proper threading support has stopped me from merging the work-in-progress patches. However, I've finally reached the point where I believe that the current work can be merged and the remaining problems can be resolved afterwards. More on that and other LLVM-related events happening during the last month in this report. Multiple IPSec VPN tunnels with FreeBSD (https://blog.socruel.nu/text-only/how-to-multiple-ipsec-vpn-tunnels-on-freebsd.txt) The FreeBSD handbook describes an IPSec VPN tunnel between 2 FreeBSD hosts (see https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ipsec.html) But it is also possible to have multiple, 2 or more, IPSec VPN tunnels created and running on a FreeBSD host. How to implement and configure this is described below. The requirements is to have 3 locations (A, B and C) connected with IPSec VPN tunnels using FreeBSD (11.3-RELEASE). Each location has 1 IPSec VPN host running FreeBSD (VPN host A, B and C). VPN host A has 2 IPSec VPN tunnels: 1 to location B (VPN host B) and 1 to location C (VPN host C). News Roundup Netflix Optimized FreeBSD's Network Stack More Than Doubled AMD EPYC Performance (https://www.phoronix.com/scan.php?page=news_item&px=Netflix-NUMA-FreeBSD-Optimized) Drew Gallatin of Netflix presented at the recent EuroBSDcon 2019 conference in Norway on the company's network stack optimizations to FreeBSD. Netflix was working on being able to deliver 200Gb/s network performance for video streaming out of Intel Xeon and AMD EPYC servers, to which they are now at 190Gb/s+ and in the process that doubled the potential of EPYC Naples/Rome servers and also very hefty upgrades too for Intel. Netflix has long been known to be using FreeBSD in their data centers particularly where network performance is concerned. But in wanting to deliver 200Gb/s throughput from individual servers led them to making NUMA optimizations to the FreeBSD network stack. Allocating NUMA local memory for kernel TLS crypto buffers and for backing files sent via sentfile were among their optimizations. Changes to network connection handling and dealing with incoming connections to Nginx were also made. For those just wanting the end result, Netflix's NUMA optimizations to FreeBSD resulted in their Intel Xeon servers going from 105Gb/s to 191Gb/s while the NUMA fabric utilization dropped from 40% to 13%. unwind(8); "happy eyeballs" (https://marc.info/?l=openbsd-tech&m=157475113130337&w=2) In case you are wondering why happy eyeballs: It's a variation on this: https://en.wikipedia.org/wiki/Happy_Eyeballs unwind has a concept of a best nameserver type. It considers a configured DoT nameserver to be better than doing it's own recursive resolving. Recursive resolving is considered to be better than asking the dhcp provided nameservers. This diff sorts the nameserver types by quality, as above (validation, resolving, dead...), and as a tie breaker it adds the median of the round trip time of previous queries into the mix. One other interesting thing about this is that it gets us past captive portals without a check URL, that's why this diff is so huge, it rips out all the captive portal stuff (please apply with patch -E): 17 files changed, 385 insertions(+), 1683 deletions(-) Please test this. I'm particularly interested in reports from people who move between networks and need to get past captive portals. Amazon now has FreeBSD ARM 12 (https://aws.amazon.com/marketplace/pp/B081NF7BY7) Product Overview FreeBSD is an operating system used to power servers, desktops, and embedded systems. Derived from BSD, the version of UNIX developed at the University of California, Berkeley, FreeBSD has been continually developed by a large community for more than 30 years. FreeBSD's networking, security, storage, and monitoring features, including the pf firewall, the Capsicum and CloudABI capability frameworks, the ZFS filesystem, and the DTrace dynamic tracing framework, make FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage systems. OpenSSH U2F/FIDO support in base (https://www.undeadly.org/cgi?action=article;sid=20191115064850) I just committed all the dependencies for OpenSSH security key (U2F) support to base and tweaked OpenSSH to use them directly. This means there will be no additional configuration hoops to jump through to use U2F/FIDO2 security keys. Hardware backed keys can be generated using "ssh-keygen -t ecdsa-sk" (or "ed25519-sk" if your token supports it). Many tokens require to be touched/tapped to confirm this step. You'll get a public/private keypair back as usual, except in this case, the private key file does not contain a highly-sensitive private key but instead holds a "key handle" that is used by the security key to derive the real private key at signing time. So, stealing a copy of the private key file without also stealing your security key (or access to it) should not give the attacker anything. Once you have generated a key, you can use it normally - i.e. add it to an agent, copy it to your destination's authorized_keys files (assuming they are running -current too), etc. At authentication time, you will be prompted to tap your security key to confirm the signature operation - this makes theft-of-access attacks against security keys more difficult too. Please test this thoroughly - it's a big change that we want to have stable before the next release. Beastie Bits DragonFly - git: virtio - Fix LUN scan issue w/ Google Cloud (http://lists.dragonflybsd.org/pipermail/commits/2019-November/719945.html) Really fast Markov chains in ~20 lines of sh, grep, cut and awk (https://0x0f0f0f.github.io/posts/2019/11/really-fast-markov-chains-in-~20-lines-of-sh-grep-cut-and-awk/) FreeBSD Journal Sept/Oct 2019 (https://www.freebsdfoundation.org/past-issues/security-3/) Michael Dexter is raising money for Bhyve development (https://twitter.com/michaeldexter/status/1201231729228308480) syscall call-from verification (https://marc.info/?l=openbsd-tech&m=157488907117170) FreeBSD Forums Howto Section (https://forums.freebsd.org/forums/howtos-and-faqs-moderated.39/) Feedback/Questions Jeroen - Feedback (http://dpaste.com/0PK1EG2#wrap) Savo - pfsense ports (http://dpaste.com/0PZ03B7#wrap) Tin - I want to learn C (http://dpaste.com/2GVNCYB#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
36 year old UFS bug fixed, a BSD for the road, automatic upgrades with OpenBSD, DTrace ext2fs support in FreeBSD, Dedicated SSH tunnel user, upgrading VMM VMs to OpenBSD 6.5, and more.
36 year old UFS bug fixed, a BSD for the road, automatic upgrades with OpenBSD, DTrace ext2fs support in FreeBSD, Dedicated SSH tunnel user, upgrading VMM VMs to OpenBSD 6.5, and more. Headlines 36+ year old bug in FFS/UFS discovered and patched This update eliminates a kernel stack disclosure bug in UFS/FFS directory entries that is caused by uninitialized directory entry padding written to the disk. When the directory entry is written to disk, it is written as a full 32bit entry, and the unused bytes were not initialized, so could possibly contain sensitive data from the kernel stack It can be viewed by any user with read access to that directory. Up to 3 bytes of kernel stack are disclosed per file entry, depending on the the amount of padding the kernel needs to pad out the entry to a 32 bit boundary. The offset in the kernel stack that is disclosed is a function of the filename size. Furthermore, if the user can create files in a directory, this 3 byte window can be expanded 3 bytes at a time to a 254 byte window with 75% of the data in that window exposed. The additional exposure is done by removing the entry, creating a new entry with a 4-byte longer name, extracting 3 more bytes by reading the directory, and repeating until a 252 byte name is created. This exploit works in part because the area of the kernel stack that is being disclosed is in an area that typically doesn't change that often (perhaps a few times a second on a lightly loaded system), and these file creates and unlinks themselves don't overwrite the area of kernel stack being disclosed. It appears that this bug originated with the creation of the Fast File System in 4.1b-BSD (Circa 1982, more than 36 years ago!), and is likely present in every Unix or Unix-like system that uses UFS/FFS. Amazingly, nobody noticed until now. This update also adds the -z flag to fsck_ffs to have it scrub the leaked information in the name padding of existing directories. It only needs to be run once on each UFS/FFS filesystem after a patched kernel is installed and running. Submitted by: David G. Lawrence dg@dglawrence.com So a patched kernel will no longer leak this data, and running the fsck_ffs -z command will erase any leaked data that may exist on your system OpenBSD commit with additional detail on mitigations The impact on OpenBSD is very limited: 1 - such stack bytes can be found in raw-device reads, from group operator. If you can read the raw disks you can undertake other more powerful actions. 2 - read(2) upon directory fd was disabled July 1997 because I didn't like how grep * would display garbage and mess up the tty, and applying vis(3) for just directory reads seemed silly. read(2) was changed to return 0 (EOF). Sep 2016 this was further changed to EISDIR, so you still cannot see the bad bytes. 3 - In 2013 when guenther adapted the getdents(2) directory-reading system call to 64-bit ino_t, the userland data format changed to 8-byte-alignment, making it incompatible with the 4-byte-alignment UFS on-disk format. As a result of code refactoring the bad bytes were not copied to userland. Bad bytes will remain in old directories on old filesystems, but nothing makes those bytes user visible. There will be no errata or syspatch issued. I urge other systems which do expose the information to userland to issue errata quickly, since this is a 254 byte infoleak of the stack which is great for ROP-chain building to attack some other bug. Especially if the kernel has no layout/link-order randomization ... NomadBSD, a BSD for the Road As regular It’s FOSS readers should know, I like diving into the world of BSDs. Recently, I came across an interesting BSD that is designed to live on a thumb drive. Let’s take a look at NomadBSD. NomadBSD is different than most available BSDs. NomadBSD is a live system based on FreeBSD. It comes with automatic hardware detection and an initial config tool. NomadBSD is designed to “be used as a desktop system that works out of the box, but can also be used for data recovery, for educational purposes, or to test FreeBSD’s hardware compatibility.” This German BSD comes with an OpenBox-based desktop with the Plank application dock. NomadBSD makes use of the DSB project. DSB stands for “Desktop Suite (for) (Free)BSD” and consists of a collection of programs designed to create a simple and working environment without needing a ton of dependencies to use one tool. DSB is created by Marcel Kaiser one of the lead devs of NomadBSD. Just like the original BSD projects, you can contact the NomadBSD developers via a mailing list. Version 1.2 Released NomadBSD recently released version 1.2 on April 21, 2019. This means that NomadBSD is now based on FreeBSD 12.0-p3. TRIM is now enabled by default. One of the biggest changes is that the initial command-line setup was replaced with a Qt graphical interface. They also added a Qt5 tool to install NomadBSD to your hard drive. A number of fixes were included to improve graphics support. They also added support for creating 32-bit images. Thoughts on NomadBSD I first discovered NomadBSD back in January when they released 1.2-RC1. At the time, I had been unable to install Project Trident on my laptop and was very frustrated with BSDs. I downloaded NomadBSD and tried it out. I initially ran into issues reaching the desktop, but RC2 fixed that issue. However, I was unable to get on the internet, even though I had an Ethernet cable plugged in. Luckily, I found the wifi manager in the menu and was able to connect to my wifi. Overall, my experience with NomadBSD was pleasant. Once I figured out a few things, I was good to go. I hope that NomadBSD is the first of a new generation of BSDs that focus on mobility and ease of use. BSD has conquered the server world, it’s about time they figured out how to be more user-friendly. News Roundup [OpenBSD automatic upgrade](https://www.tumfatig.net/20190426/openbsd-automatic-upgrade/) OpenBSD 6.5 advertises for an installer improvement: rdsetroot(8) (a build-time tool) is now available for general use. Used in combination with autoinstall.8, it is now really easy to do automatic upgrades of your OpenBSD instances. I first manually upgraded my OpenBSD sandbox to 6.5. Once that was done, I could use the stock rdsetroot(8) tool. The plan is quite simple: write an unattended installation response file, insert it to a bsd.rd 6.5 installation image and reboot my other OpenBSD instances using that image. Extra notes There must be a way to run onetime commands (in the manner of fw_update) to automatically run sysmerge and packages upgrades. As for now, I’d rather do it manually. This worked like a charm on two Synology KVM instances using a single sd0 disk, on my Thinkpad X260 using Encrypted root with Keydisk and on a Vultr instance using Encrypted root with passphrase. And BTW, the upgrade on the X260 used the (iwn0) wireless connection. I just read that florian@ has released the sysupgrade(8) utility which should be released with OpenBSD 6.6. That will make upgrades even easier! Until then, happy upgrading. FreeBSD Dtrace ext2fs Support Which logs were replaced by dtrace-probes: Misc printf's under DEBUG macro in the blocks allocation path. Different on-disk structures validation errors, now the filesystem will silently return EIO's. Misc checksum errors, same as above. The only debug macro, which was leaved is EXT2FSPRINTEXTENTS. It is impossible to replace it by dtrace-probes, because the additional logic is required to walk thru file extents. The user still be able to see mount errors in the dmesg in case of: Filesystem features incompatibility. Superblock checksum error. Create a dedicated user for ssh tunneling only I use ssh tunneling A LOT, for everything. Yesterday, I removed the public access of my IMAP server, it’s now only available through ssh tunneling to access the daemon listening on localhost. I have plenty of daemons listening only on localhost that I can only reach through a ssh tunnel. If you don’t want to bother with ssh and redirect ports you need, you can also make a VPN (using ssh, openvpn, iked, tinc…) between your system and your server. I tend to avoid setting up VPN for the current use case as it requires more work and more maintenance than running ssh server and a ssh client. The last change, for my IMAP server, added an issue. I want my phone to access the IMAP server but I don’t want to connect to my main account from my phone for security reasons. So, I need a dedicated user that will only be allowed to forward ports. This is done very easily on OpenBSD. The steps are: 1. generate ssh keys for the new user 2. add an user with no password 3. allow public key for port forwarding Obviously, you must allow users (or only this one) to make port forwarding in your sshd_config. That was easy. Some info on upgrading VMM VMs to 6.5 We're running dedicated vmm(4)/vmd(8) servers to host opinionated VMs. OpenBSD 6.5 is released! There are two ways you can upgrade your VM. Either do a manual upgrade or leverage autoinstall(8). You can take care of it via the console with vmctl(8). Upgrade yourself To get connected to the console you need to have access to the host your VM is running on. The same username and public SSH key, as provided for the VM, are used to create a local user on the host. When this is done you can use vmctl(8) to manage your VM. The options you have are: ```$ vmctl start id [-c]``` $ vmctl stop id [-fw]``` ```-w Wait until the VM has been terminated.``` -c Automatically connect to the VM console.``` See the Article for the rest of the guide Beastie Bits powerpc64 architecture support in FreeBSD ports GhostBSD 19.04 overview HardenedBSD will have two user selectable ASLR implementations NYCBUG 2016 Talk Shell-Fu Uploaded What is ZIL anyway? Feedback/Questions Quentin - Organize an Ada/BSD interview DJ - Update Patrick - Bhyve frontends A small programming note: After BSDNow episode 300, the podcast will switch to audio-only, using a new higher quality recording and production system. The live stream will likely still include video. Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.
36 year old UFS bug fixed, a BSD for the road, automatic upgrades with OpenBSD, DTrace ext2fs support in FreeBSD, Dedicated SSH tunnel user, upgrading VMM VMs to OpenBSD 6.5, and more.
Bryan Cantrill is the CTO of Joyent and well known for the development of DTrace at Sun Microsystems. Today on the podcast, Bryan discusses with Wes Reisz a bit about the origins of DTrace and then spends the rest of the time discussing why he feels Rust is the “biggest development in systems development in his career.” The podcast wraps with a bit about why Bryan feels we should be rewriting parts of the operating system in Rust. Why listen to the podcast: • DTrace came down to a desire to use Dynamic Program Text Modification to instrument running systems (much like debuggers do) and has its origins to when Bryan was an undergraduate. • When a programming language delivers something to you, it takes it from you in the runtime. The classic example of this is garbage collection. The programming language gives you the ability to use memory dynamically without thinking of how the memory is stored in the system, but then it’s going to exact a runtime cost. • One of the issues with C is that it just doesn’t compose well. You can’t just necessarily pull a library off the Internet and use it well. Everyone’s C is laden with some many idiosyncrasies on how it’s used and the contract on how memory is used. • Ownership is statically tracking who owns the structure. It’s ownership and the absence of GC that allows you to address the composability issues found in C. • It’s really easy in C to have integer overflow which leads to memory safety issues that can be exploited by an attacker. Rust makes this pretty much impossible because it’s very good at how it determines how you use signed vs unsigned types. • You don’t want people solving the same problems over and over again. You want composability. You want abstractions. What you don’t want is where you’ve removed so much developer friction that you develop code that is riddled with problems. For example, it slows a developer down to force them to run a linter, but it results in better artifacts. Rust effective builds a lot of that linter checking into the memory management/type checking system. • While there’s some learning curve to Rust. It’s not that bad if you realize there are several core concepts you need to understand to understand Rust. Rust is one of those languages that you really need to learn in a structured way. Sit down with a book and learn it. • Rust struggles when you have objects that are multiply owned (such as a Doubly Linked List). It’s because it doesn’t know who owns what. While Rust supports unsafe operations, you should resist the temptation to develop with a lot of unsafe operations if you want the benefits of what Rust offers developers. • Firmware is a great spot for growing Rust development in a process of replacing bits of what we think of as the operating system. More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2uZ5QHZ You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: https://bit.ly/2uZ5QHZ
We talk with Alessandro Vozza about the Cloud Native Computing Foundation. Dtrace on Windows. We review some salary data we've been collecting. And how much are ads costing you? News DTrace on Windows Open DTrace on GitHub Dynamic Tracing Guide DTrace on Windows Branch Ads cost you 10% of your users Pandora Learns the Cost of Ads and of Subscruptions Mozilla Research: The Effect of Ad Blocking on User Engagement with the Web My Salary Progression in Tech Form: http://msdv.sh/AnonymousPayForm Results: http://msdv.sh/AnonymousPayResults Allesandro Vozza works in Commercial Software Engineering at Microsoft. He's also a community organizer and participant, and an ambassador for the Cloud Native Computing Foundation. @bongo LinkedIn GitHub ------------------------------------------------------------------- cncf.io Members Ambassadors CNCF Github Graduation Criteria Kubecon Events Technical Oversight Committee CNCF Landscape
![Day[0] - Zero Days for Day Zero](https://ivyfm.s3.amazonaws.com/i320/820071.jpg)
00:00:30 Steam Client (CSGO) RCE 00:04:44 CS 1.6 Trojan.Belonard Malware Campaign 00:11:55 WebKit Structure ID Randomness Mitigation 00:20:48 Reuse Gadget Counts Whitepaper (ROP) 00:31:50 DTrace on Windows 00:38:20 Backdoor Attack in CNN's 00:55:05 DARPA's $10m Open Source Voting System 01:13:30 Vulnerability in Swiss E-Voting System
Which operating system is the best? Which programming language is the best? What text editor? Bryan Cantrill, CTO of Joyent says that is the wrong question. Languages, operating systems and communities have to make trade offs and they do that based on their values. So the right language is the one who's values align with you and your projects goals. This simple idea carries a lot of weight and I think has the potential to lift up technical discussions to a higher level of discourse. You will find it to be a helpful frame next time you need to make a technical decision. Bryan is also pretty excited about how the values of the rust community align with his values for system software. Also we cover Oberon, Clean and Simula 4, none of which I've never heard of and how IBM System/370 's Global Trace Facility doesn't hold a candle to Dtrace. Webpage for this episode Show Links: Software Values Slides The Design and Implementation of the FreeBSD Operating System Microsoft should buy github All Bryan's Talks Slack Channel for Site
OpenBSD 6.4 released, GhostBSD RC2 released, MeetBSD - the ultimate hallway track, DragonflyBSD desktop on a Thinkpad, Porting keybase to NetBSD, OpenSSH 7.9, and draft-ietf-6man-ipv6only-flag in FreeBSD. ##Headlines OpenBSD 6.4 released See a detailed log of changes between the 6.3 and 6.4 releases. See the information on the FTP page for a list of mirror machines. Have a look at the 6.4 errata page for a list of bugs and workarounds. signify(1) pubkeys for this release: base: RWQq6XmS4eDAcQW4KsT5Ka0KwTQp2JMOP9V/DR4HTVOL5Bc0D7LeuPwA fw: RWRoBbjnosJ/39llpve1XaNIrrQND4knG+jSBeIUYU8x4WNkxz6a2K97 pkg: RWRF5TTY+LoN/51QD5kM2hKDtMTzycQBBPmPYhyQEb1+4pff/H6fh/kA ###GhostBSD 18.10 RC2 Announced This second release candidate of GhostBSD 18.10 is the second official release of GhostBSD with TrueOS under the hood. The official desktop of GhostBSD is MATE. However, in the future, there might be an XFCE community release, but for now, there is no community release yet. What has changed since RC1 Removed drm-stable-kmod and we will let users installed the propper drm-*-kmod Douglas Joachin added libva-intel-driver libva-vdpau-driver to supports accelerated some video driver for Intel Issues that got fixed Bug #70 Cannot run Octopi, missing libgksu error. Bug #71 LibreOffice doesn’t start because of missing libcurl.so.4 Bug #72 libarchive is a missing dependency Again thanks to iXsystems, TrueOS, Joe Maloney, Kris Moore, Ken Moore, Martin Wilke, Neville Goddard, Vester “Vic” Thacker, Douglas Joachim, Alex Lyakhov, Yetkin Degirmenci and many more who helped to make the transition from FreeBSD to TrueOS smoother. Updating from RC1 to RC2: sudo pkg update -f sudo pkg install -f libarchive curl libgksu sudo pkg upgrade Where to download: All images checksum, hybrid ISO(DVD, USB) and torrent are available here: https://www.ghostbsd.org/download [ScreenShots] https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-2013-22-41.png https://www.ghostbsd.org/sites/default/files/Screenshotat2018-10-20_13-27-26.png ###OpenSSH 7.9 has been released and it has support for OpenSSL 1.1 Changes since OpenSSH 7.8 This is primarily a bugfix release. New Features ssh(1), sshd(8): allow most port numbers to be specified using service names from getservbyname(3) (typically /etc/services). ssh(1): allow the IdentityAgent configuration directive to accept environment variable names. This supports the use of multiple agent sockets without needing to use fixed paths. sshd(8): support signalling sessions via the SSH protocol. A limited subset of signals is supported and only for login or command sessions (i.e. not subsystems) that were not subject to a forced command via authorizedkeys or sshdconfig. bz#1424 ssh(1): support "ssh -Q sig" to list supported signature options. Also "ssh -Q help" to show the full set of supported queries. ssh(1), sshd(8): add a CASignatureAlgorithms option for the client and server configs to allow control over which signature formats are allowed for CAs to sign certificates. For example, this allows banning CAs that sign certificates using the RSA-SHA1 signature algorithm. sshd(8), ssh-keygen(1): allow key revocation lists (KRLs) to revoke keys specified by SHA256 hash. ssh-keygen(1): allow creation of key revocation lists directly from base64-encoded SHA256 fingerprints. This supports revoking keys using only the information contained in sshd(8) authentication log messages. Bugfixes ssh(1), ssh-keygen(1): avoid spurious "invalid format" errors when attempting to load PEM private keys while using an incorrect passphrase. bz#2901 sshd(8): when a channel closed message is received from a client, close the stderr file descriptor at the same time stdout is closed. This avoids stuck processes if they were waiting for stderr to close and were insensitive to stdin/out closing. bz#2863 ssh(1): allow ForwardX11Timeout=0 to disable the untrusted X11 forwarding timeout and support X11 forwarding indefinitely. Previously the behaviour of ForwardX11Timeout=0 was undefined. sshd(8): when compiled with GSSAPI support, cache supported method OIDs regardless of whether GSSAPI authentication is enabled in the main section of sshd_config. This avoids sandbox violations if GSSAPI authentication was later enabled in a Match block. bz#2107 sshd(8): do not fail closed when configured with a text key revocation list that contains a too-short key. bz#2897 ssh(1): treat connections with ProxyJump specified the same as ones with a ProxyCommand set with regards to hostname canonicalisation (i.e. don't try to canonicalise the hostname unless CanonicalizeHostname is set to 'always'). bz#2896 ssh(1): fix regression in OpenSSH 7.8 that could prevent public- key authentication using certificates hosted in a ssh-agent(1) or against sshd(8) from OpenSSH
We explain what eBPF is, how it works, and its proud BSD production legacy. eBPF is a technology that you’re going to be hearing more and more about. It powers low-overhead custom analysis tools, handles network security in a containerized world, and powers tools you use every day.
FreeBSD Foundation September Update, tiny C lib for programming Unix daemons, EuroBSDcon trip reports, GhostBSD tested on real hardware, and a BSD auth module for duress. ##Headlines FreeBSD Foundation Update, September 2018 MESSAGE FROM THE EXECUTIVE DIRECTOR Dear FreeBSD Community Member, It is hard to believe that September is over. The Foundation team had a busy month promoting FreeBSD all over the globe, bug fixing in preparation for 12.0, and setting plans in motion to kick off our 4th quarter fundraising and advocacy efforts. Take a minute to see what we’ve been up to and please consider making a donation to help us continue our efforts supporting FreeBSD! September 2018 Development Projects Update In preparation for the release of FreeBSD 12.0, I have been working on investigating and fixing a backlog of kernel bug reports. Of course, this kind of work is never finished, and we will continue to make progress after the release. In the past couple of months I have fixed a combination of long-standing issues and recent regressions. Of note are a pair of UNIX domain socket bugs which had been affecting various applications for years. In particular, Chromium tabs would frequently hang unless a workaround was manually applied to the system, and the bug had started affecting recent versions of Firefox as well. Fixing these issues gave me an opportunity to revisit and extend our regression testing for UNIX sockets, which, in turn, resulted in some related bugs being identified and fixed. Of late I have also been investigating reports of issues with ZFS, particularly, those reported on FreeBSD 11.2. A number of regressions, including a kernel memory leak and issues with ARC reclamation, have already been fixed for 12.0; investigation of other reports is ongoing. Those who closely follow FreeBSD-CURRENT know that some exciting work to improve memory usage on NUMA systems is now enabled by default. As is usually the case when new code is deployed in a diverse array of systems and workloads, a number of problems since have been identified. We are working on resolving them as soon as possible to ensure the quality of the release. I’m passionate about maintaining FreeBSD’s stability and dependability as it continues to expand and grow new features, and I’m grateful to the FreeBSD Foundation for sponsoring this work. We depend on users to report problems to the mailing lists and via the bug tracker, so please try running the 12.0 candidate builds and help us make 12.0 a great release. Fundraising Update: Supporting the Project It’s officially Fall here at Foundation headquarters and we’re heading full-steam into our final fundraising campaign of the year. We couldn’t even have begun to reach our funding goal of $1.25 million dollars without the support from the companies who have partnered with us this year. Thank you to Verisign for becoming a Silver Partner. They now join a growing list of companies like Xiplink, NetApp, Microsoft, Tarsnap, VMware, and NeoSmart Technologies that are stepping up and showing their commitment to FreeBSD! Funding from commercial users like these and individual users like yourself, help us continue our efforts of supporting critical areas of FreeBSD such as: Operating System Improvements: Providing staff to immediately respond to urgent problems and implement new features and functionality allowing for the innovation and stability you’ve come to rely on. Security: Providing engineering resources to bolster the capacity and responsiveness of the Security team providing your users with piece of mind when security issues arise. Release Engineering: Continue providing a full-time release engineer, resulting in timely and reliable releases you can plan around. Quality Assurance: Improving and increasing test coverage, continuous integration, and automated testing with a full-time software engineer to ensure you receive the highest quality, secure, and reliable operating system. New User Experience: Improving the process and documentation for getting new people involved with FreeBSD, and supporting those people as they become integrated into the FreeBSD Community providing the resources you may need to get new folks up to speed. Training: Supporting more FreeBSD training for undergraduates, graduates, and postgraduates. Growing the community means reaching people and catching their interest in systems software as early as possible and providing you with a bigger pool of candidates with the FreeBSD skills you’re looking for. Face-to-Face Opportunities: Facilitating collaboration among members of the community, and building connections throughout the industry to support a healthy and growing ecosystem and make it easier for you to find resources when questions emerge . We can continue the above work, if we meet our goal this year! If your company uses FreeBSD, please consider joining our growing list of 2018 partners. If you haven’t made your donation yet, please consider donating today. We are indebted to the individual donors, and companies listed above who have already shown their commitment to open source. Thank you for supporting FreeBSD and the Foundation! September 2018 Release Engineering Update The FreeBSD Release Engineering team continued working on the upcoming 12.0 RELEASE. At present, the 12.0 schedule had been adjusted by one week to allow for necessary works-in-progress to be completed. Of note, one of the works-in-progress includes updating OpenSSL from 1.0.2 to 1.1.1, in order to avoid breaking the application binary interface (ABI) on an established stable branch. Due to the level of non-trivial intrusiveness that had already been discovered and addressed in a project branch of the repository, it is possible (but not yet definite) that the schedule will need to be adjusted by another week to allow more time for larger and related updates for this particular update. Should the 12.0-RELEASE schedule need to be adjusted at any time during the release cycle, the schedule on the FreeBSD project website will be updated accordingly. The current schedule is available at: https://www.freebsd.org/releases/12.0R/schedule.html BSDCam 2018 Trip Report: Marie Helene Kvello-Aune I’d like to start by thanking the FreeBSD Foundation for sponsoring my trip to BSDCam(bridge) 2018. I wouldn’t have managed to attend otherwise. I’ve used FreeBSD in both personal and professional deployments since the year 2000, and over the last few years I have become more involved with development and documentation. I arrived in Gatwick, London at midnight. On Monday, August 13, I took the train to Cambridge, and decided to do some touristy activities as I walked from the train station to Churchill College. I ran into Allan outside the hotel right before the sky decided it was time for a heavy rainfall. Monday was mostly spent settling in, recouping after travel, and hanging out with Allan, Brad, Will and Andy later in the afternoon/evening. Read more… Continuous Integration Update The FreeBSD Foundation has sponsored the development of the Project’s continuous integration system, available at https://ci.FreeBSD.org, since June. Over the summer, we improved both the software and hardware infrastructure, and also added some new jobs for extending test coverage of the -CURRENT and -STABLE branches. Following are some highlights. New Hardware The Foundation purchased 4 new build machines for scaling up the computation power for the various test jobs. These newer, faster machines substantially speed up the time it takes to test amd64 builds, so that failing changes can be identified more quickly. Also, in August, we received a donation of 2 PINE A64-LTS boards from PINE64.org, which will be put in the hardware test lab as one part of the continuous tests. CI Staging Environment We used hardware from a previous generation CI system to build a staging environment for the CI infrastructure, which is available at https://ci-dev.freebsd.org. It executes the configurations and scripts from the “staging” branch of the FreeBSD-CI repository, and the development feature branches. We also use it to experiment with the new version of the jenkins server and plugins. Having a staging environment avoids affecting the production CI environment, reducing downtime. Mail Notification In July, we turned on failure notification for all the kernel and world build jobs. Committers will receive email containing the build information and failure log to inform them of possible problems with their modification on certain architectures. For amd64 of the -CURRENT branch, we also enabled the notification on failing regression test cases. Currently mail is sent only to the individual committers, but with help from postmaster team, we have created a dev-ci mailing list and will soon be also sending notifications there. New Test Job In August, we updated the embedded script of the virtual machine image. Originally it only executed pre-defined tests, but now this behavior can be modified by the data on the attached disk. This mechanism is used for adding new ZFS tests jobs. We are also working on analyzing and fixing the failing and skipped test cases. Work in Progress In August and September, we had two developer summits, one in Cambridge, UK and one in Bucharest, Romania. In these meetings, we discussed running special tests, such as ztest, which need a longer run time. We also planned the network testing for TCP/IP stack ###Daemonize - a Tiny C Library for Programming the UNIX Daemons Whatever they say, writing System-V style UNIX daemons is hard. One has to follow many rules to make a daemon process behave correctly on diverse UNIX flavours. Moreover, debugging such a code might be somewhat tricky. On the other hand, the process of daemon initialisation is rigid and well defined so the corresponding code has to be written and debugged once and later can be reused countless number of times. Developers of BSD UNIX were very aware of this, as there a C library function daemon() was available starting from version 4.4. The function, although non-standard, is present on many UNIXes. Unfortunately, it does not follow all the required steps to reliably run a process in the background on systems which follow System-V semantics (e.g. Linux). The details are available at the corresponding Linux man page. The main problem here, as I understand it, is that daemon() does not use the double-forking technique to avoid the situation when zombie processes appear. Whenever I encounter a problem like this one, I know it is time to write a tiny C library which solves it. This is exactly how ‘daemonize’ was born (GitHub mirror). The library consists of only two files which are meant to be integrated into the source tree of your project. Recently I have updated the library and realised that it would be good to describe how to use it on this site. If for some reason you want to make a Windows service, I have a battle tested template code for you as well. System-V Daemon Initialisation Procedure To make discussion clear we shall quote the steps which have to be performed during a daemon initialisation (according to daemon(7) manual page on Linux). I do it to demonstrate that this task is more tricky than one might expect. So, here we go: Close all open file descriptors except standard input, output, and error (i.e. the first three file descriptors 0, 1, 2). This ensures that no accidentally passed file descriptor stays around in the daemon process. On Linux, this is best implemented by iterating through /proc/self/fd, with a fallback of iterating from file descriptor 3 to the value returned by getrlimit() for RLIMITNOFILE. Reset all signal handlers to their default. This is best done by iterating through the available signals up to the limit of _NSIG and resetting them to SIGDFL. Reset the signal mask using sigprocmask(). Sanitize the environment block, removing or resetting environment variables that might negatively impact daemon runtime. Call fork(), to create a background process. In the child, call setsid() to detach from any terminal and create an independent session. In the child, call fork() again, to ensure that the daemon can never re-acquire a terminal again. Call exit() in the first child, so that only the second child (the actual daemon process) stays around. This ensures that the daemon process is re-parented to init/PID 1, as all daemons should be. In the daemon process, connect /dev/null to standard input, output, and error. In the daemon process, reset the umask to 0, so that the file modes passed to open(), mkdir() and suchlike directly control the access mode of the created files and directories. In the daemon process, change the current directory to the root directory (/), in order to avoid that the daemon involuntarily blocks mount points from being unmounted. In the daemon process, write the daemon PID (as returned by getpid()) to a PID file, for example /run/foobar.pid (for a hypothetical daemon “foobar”) to ensure that the daemon cannot be started more than once. This must be implemented in race-free fashion so that the PID file is only updated when it is verified at the same time that the PID previously stored in the PID file no longer exists or belongs to a foreign process. In the daemon process, drop privileges, if possible and applicable. From the daemon process, notify the original process started that initialization is complete. This can be implemented via an unnamed pipe or similar communication channel that is created before the first fork() and hence available in both the original and the daemon process. Call exit() in the original process. The process that invoked the daemon must be able to rely on that this exit() happens after initialization is complete and all external communication channels are established and accessible. The discussed library does most of the above-mentioned initialisation steps as it becomes immediately evident that implementation details for some of them heavily dependent on the internal logic of an application itself, so it is not possible to implement them in a universal library. I believe it is not a flaw, though, as the missed parts are safe to implement in an application code. The Library’s Application Programming Interface The generic programming interface was loosely modelled after above-mentioned BSD’s daemon() function. The library provides two user available functions (one is, in fact, implemented on top of the other) as well as a set of flags to control a daemon creation behaviour. Conclusion The objective of the library is to hide all the trickery of programming a daemon so you could concentrate on the more creative parts of your application. I hope it does this well. If you are not only interested in writing a daemon, but also want to make yourself familiar with the techniques which are used to accomplish that, the source code is available. Moreover, I would advise anyone, who starts developing for a UNIX environment to do that, as it shows many intricacies of programming for these platforms. ##News Roundup EuroBSDCon 2018 travel report and obligatory pics This was my first big BSD conference. We also planned - planned might be a big word - thought about doing a devsummit on Friday. Since the people who were in charge of that had a change of plans, I was sure it’d go horribly wrong. The day before the devsummit and still in the wrong country, I mentioned the hours and venue on the wiki, and booked a reservation for a restaurant. It turns out that everything was totally fine, and since the devsummit was at the conference venue (that was having tutorials that day), they even had signs pointing at the room we were given. Thanks EuroBSDCon conference organizers! At the devsummit, we spent some time hacking. A few people came with “travel laptops” without access to anything, like Riastradh, so I gave him access to my own laptop. This didn’t hold very long and I kinda forgot about it, but for a few moments he had access to a NetBSD source tree and an 8 thread, 16GB RAM machine with which to build things. We had a short introduction and I suggested we take some pictures, so here’s the ones we got. A few people were concerned about privacy, so they’re not pictured. We had small team to hold the camera :-) At the actual conference days, I stayed at the speaker hotel with the other speakers. I’ve attempted to make conversation with some visibly FreeBSD/OpenBSD people, but didn’t have plans to talk about anything, so there was a lot of just following people silently. Perhaps for the next conference I’ll prepare a list of questions to random BSD people and then very obviously grab a piece of paper and ask, “what was…”, read a bit from it, and say, “your latest kernel panic?”, I’m sure it’ll be a great conversation starter. At the conference itself, was pretty cool to have folks like Kirk McKusick give first person accounts of some past events (Kirk gave a talk about governance at FreeBSD), or the second keynote by Ron Broersma. My own talk was hastily prepared, it was difficult to bring the topic together into a coherent talk. Nevertheless, I managed to talk about stuff for a while 40 minutes, though usually I skip over so many details that I have trouble putting together a sufficiently long talk. I mentioned some of my coolest bugs to solve (I should probably make a separate article about some!). A few people asked for the slides after the talk, so I guess it wasn’t totally incoherent. It was really fun to meet some of my favourite NetBSD people. I got to show off my now fairly well working laptop (it took a lot of work by all of us!). After the conference I came back with a conference cold, and it took a few days to recover from it. Hopefully I didn’t infect too many people on the way back. ###GhostBSD tested on real hardware T410 – better than TrueOS? You might have heard about FreeBSD which is ultimately derived from UNIX back in the days. It is not Linux even though it is similar in many ways because Linux was designed to follow UNIX principles. Seeing is believing, so check out the video of the install and some apps as well! Nowadays if you want some of that BSD on your personal desktop how to go about? Well there is a full package or distro called GhostBSD which is based on FreeBSD current with a Mate or XFCE desktop preconfigured. I did try another package called TrueOS before and you can check out my blog post as well. Let’s give it a try on my Lenovo ThinkPad T410. You can download the latest version from ghostbsd.org. Creating a bootable USB drive was surprisingly difficult as rufus did not work and created a corrupted drive. You have to follow this procedure under Windows: download the 2.5GB .iso file and rename the extension to .img. Download Win32 Disk imager and burn the img file to an USB drive and boot from it. You will be able to start a live session and use the onboard setup to install GhostBSD unto a disk. I did encounter some bugs or quirks along the way. The installer failed the first time for some unknown reason but worked on the second attempt. The first boot stopped upon initialization of the USB3 ports (the T410 does not have USB3) but I could use some ‘exit’ command line magic to continue. The second boot worked fine. Audio was only available through headphones, not speakers but that could partially be fixed using the command line again. Lot’s of installed apps did not show up in the start menu and on goes the quirks list. Overall it is still better than TrueOS for me because drivers did work very well and I could address most of the existing bugs. On the upside: Free and open source FreeBSD package ready to go Mate or XFCE desktop (Mate is the only option for daily builds) Drivers work fine including LAN, WiFi, video 2D & 3D, audio, etc UFS or ZFS advanced file systems available Some downsides: Less driver and direct app support than Linux Installer and desktop have some quirks and bugs App-store is cumbersome, inferior to TrueOS ##Beastie Bits EuroBSDCon 2018 and NetBSD sanitizers New mandoc feature: -T html -O toc EuroBSDcon 2018 Polish BSD User Group garbage[43]: What year is it? The Demo @ 50 Microsoft ports DTrace from FreeBSD to Windows 10 OpenBSD joins Twitter NetBSD curses ripoffline improvements FCP-0101: Deprecating most 10/100 Ethernet drivers Announcing the pkgsrc-2018Q3 release Debian on OpenBSD vmd (without qemu or another debian system) A BSD authentication module for duress passwords (Joshua Stein) Disk Price/Performance Analysis ##Feedback/Questions DJ - Zombie ZFS Josua - arm tier 1? how to approach it -Gamah - 5ghz Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
FreeBSD and DragonflyBSD benchmarks on AMD’s Threadripper, NetBSD 7.2 has been released, optimized out DTrace kernel symbols, stuck UEFI bootloaders, why ed is not a good editor today, tell your BSD story, and more.
FreeBSD and DragonflyBSD benchmarks on AMD’s Threadripper, NetBSD 7.2 has been released, optimized out DTrace kernel symbols, stuck UEFI bootloaders, why ed is not a good editor today, tell your BSD story, and more.
FreeBSD and DragonflyBSD benchmarks on AMD’s Threadripper, NetBSD 7.2 has been released, optimized out DTrace kernel symbols, stuck UEFI bootloaders, why ed is not a good editor today, tell your BSD story, and more. ##Headlines FreeBSD & DragonFlyBSD Put Up A Strong Fight On AMD’s Threadripper 2990WX, Benchmarks Against Linux The past two weeks I have been delivering a great deal of AMD Threadripper 2990WX benchmarks on Linux as well as some against Windows and Windows Server. But recently I got around to trying out some of the BSD operating systems on this 32-core / 64-thread processor to see how they would run and to see whether they would have similar scaling issues or not like we’ve seen on the Windows side against Linux. In this article are FreeBSD and DragonFlyBSD benchmarks with the X399 + 2990WX compared to a few Linux distributions. The BSDs I focused my testing on were FreeBSD 11.2-STABLE and 12.0-CURRENT/ALPHA1 (the version in development) as well as iX System’s TrueOS that is tracking FreeBSD 12.0-CURRENT. Also included were DragonFlyBSD, with FreeBSD and DragonFlyBSD being tied as my favorite operating systems when it comes to the BSDs. When it came to FreeBSD 11.2-STABLE and 12.0-ALPHA1 on the Threadripper 2990WX, it worked out surprisingly well. I encountered no real issues during my two days of benchmarking on FreeBSD (and TrueOS). It was a great experience and FreeBSD was happy to exploit the 64 threads on the system. DragonFlyBSD was a bit of a different story… Last week when I started this BSD testing I tried DragonFly 5.2.2 as the latest stable release as well as a DragonFlyBSD 5.3 development snapshot from last week: both failed to boot in either BIOS or UEFI modes. But then a few days ago DragonFlyBSD lead developer Matthew Dillon bought himself a 2990WX platform. He made the necessary changes to get DragonFlyBSD 5.3 working and he ended up finding really great performance and potential out of the platform. So I tried the latest DragonFlyBSD 5.3 daily ISO on 22 August and indeed it now booted successfully and we were off to the races. Thus there are some DragonFlyBSD 5.3 benchmarks included in this article too. Just hours ago, Matthew Dillon landed some 2990WX topology and scheduler enhancements but that fell out of the scope of when DragonFly was installed on this system. But over the weekend or so I plan to re-test DragonFlyBSD 5.3 and see how those optimizations affect the overall 2990WX performance now on that BSD. DragonFlyBSD 5.4 stable should certainly be an interesting release on several fronts! With FreeBSD 11.2-STABLE and 12.0-ALPHA1 I ran benchmarks when using their stock compiler (LLVM Clang 6.0) as well as GCC 7.3 obtained via GCC 7.3. That was done to rule out compiler differences in benchmarking against the GCC-based Linux distributions. On DragonFlyBSD 5.3 it defaults to the GCC 5.4.1 but via pkg I also did a secondary run when upgraded to GCC 7.3. The hardware and BIOS/UEFI settings were maintained the same throughout the entire benchmarking process. The system was made up of the AMD Ryzen Threadripper 2990WX at stock speeds, the ASUS ROG ZENITH EXTREME motherboard, 4 x 8GB DDR4-3200MHz memory, Samsung 970 EVO 500GB NVMe SSD, and Radeon RX Vega 56 graphics card. All of these Linux vs. BSD benchmarks were carried out in a fully-automated and reproducible manner using the open-source Phoronix Test Suite benchmarking framework. While for the last of today’s BSD vs. Linux benchmarking on the Threadripper 2990WX, the Linux distributions came out slightly ahead of FreeBSD and DragonFlyBSD with GCC (another test having issues with Clang 6.0 on the BSDs). Overall, I was quite taken away by the BSD performance on the Threadripper 2990WX – particularly FreeBSD. In a surprising number of benchmarks, the BSDs were outperforming the tested Linux distributions though often by incredibly thin margins. Still, quite an accomplishment for these BSD operating systems and considering how much better Linux is already doing than Windows 10 / Windows Server on this 32-core / 64-thread processor. Then again, the BSDs like Linux have a long history of running on high core/thread-count systems, super computers, and other HPC environments. It will be interesting to see how much faster DragonFlyBSD can run given today’s commit to its kernel with scheduler and topology improvements for the 2990WX. Those additional DragonFlyBSD benchmarks will be published in the coming days once they are completed. ###NetBSD 7.2 released The NetBSD Project is pleased to announce NetBSD 7.2, the second feature update of the NetBSD 7 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements. General Security Note The NetBSD 7.2 release is a maintenance release of the netbsd-7 branch, which had it's first major release, NetBSD 7.0 in September 2015. A lot of security features have been added to later NetBSD versions, and for new installations we highly recommend using our latest release, NetBSD 8.0 instead. Some highlights of the 7.2 release are: Support for USB 3.0. Enhancements to the Linux emulation subsystem. Fixes in binary compatibility for ancient NetBSD executables. iwm(4) driver for Intel Wireless 726x, 316x, 826x and 416x series added. Support for Raspberry Pi 3 added. Fix interrupt setup on Hyper-V VMs with Legacy Network Adapter. SVR4 and IBCS2 compatibility subsystems have been disabled by default (besides IBCS2 on VAX). These subsystems also do not auto-load their modules any more. Various USB stability enhancements. Numerous bug fixes and stability improvements. Complete source and binaries for NetBSD 7.2 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at https://www.NetBSD.org/mirrors/. We encourage users who wish to install via ISO or USB disk images to download via BitTorrent by using the torrent files supplied in the images area. A list of hashes for the NetBSD 7.2 distribution has been signed with the well-connected PGP key for the NetBSD Security Officer: https://cdn.NetBSD.org/pub/NetBSD/security/hashes/NetBSD-7.2_hashes.asc NetBSD is free. All of the code is under non-restrictive licenses, and may be used without paying royalties to anyone. Free support services are available via our mailing lists and website. Commercial support is available from a variety of sources. More extensive information on NetBSD is available from our website: ##News Roundup Including optimized-out kernel symbols in dtrace on FreeBSD Have you ever had dtrace(1) on FreeBSD fail to list a probe that should exist in the kernel? This is because Clang will optimize-out some functions. The result is ctfconvert(1) will not generate debugging symbols that dtrace(1) uses to identify probes. I have a quick solution to getting those probes visible to dtrace(1). In my case, I was trying to instrument on ieee80211_ioctl_get80211, whose sister function ieee80211_ioctl_set80211 has a dtrace(1) probe in the generic FreeBSD 11 and 12 kernels. Both functions are located in /usr/src/sys/net80211/ieee80211_ioctl.c. My first attempt was to add to /etc/make.conf as follows and recompile the kernel. CFLAGS+=-O0 and -fno-inline-functions This failed to produce the dtrace(1) probe. Several other attempts failed and I was getting inconsistent compilation results (Is it me or is ieee80211_ioctl.c compiled with different flags if NO_CLEAN=1 is set?). When I manually compiled the object file by copying the compilation line for the object file and adding -O0 -fno-inline-functions, nm(1) on both the object file and kernel demonstrated that the symbol was present. I installed the kernel, rebooted and it was listed as a dtrace probe. Great! But as I continued to debug my WiFi driver (oh yeah, I’m very slowly extending rtwn(4)), I found myself rebuilding the kernel several times and frequently rebooting. Why not do this across the entire kernel? After hacking around, my solution was to modify the build scripts. My solution was to edit /usr/src/sys/conf/kern.pre.mk and modify all optimization level 2 to optimization level 0. The following is my diff(1) on FreeBSD 12.0-CURRENT. A few thoughts: This seems like a hack rather than a long-term solution. Either the problem is with the hard-coded optimization flags, or the inability to overwrite them in all places in make.conf. Removing optimizations is only something I would do in a non-production kernel, so its as if I have to choose between optimizations for a production kernel or having dtrace probes. But dtrace explicitly markets itself as not impactful on production. Using the dtrace pony as your featured image on WordPress does not render properly and must be rotated and modified. Blame Bryan Cantrill. If you have a better solution, please let me know and I will update the article, but this works for me! ###FreeBSD: UEFI Bootloader stuck on BootCurrent/BootOrder/BootInfo on Asus Motherboards (and fix!) Starting with FreeBSD CURRENT from about a few weeks of posting date, but including FreeBSD 12 alpha releases (not related to DEC Alpha), I noticed one thing: When I boot FreeBSD from UEFI on a homebuilt desktop with a Asus H87M-E motherboard, and have Root on ZFS, the bootloader gets stuck on lines like BootCurrent, BootOrder, and BootInfo. This issue occurs when I try to boot directly to efibootbootx64.efi. One person had a similar issue on a Asus H87I-PLUS motherboard. This issue may or may not exist on other Asus motherboards, desktops, or laptops. This may be specific to Asus motherboards for Intel’s Haswell, but may also exist on newer systems (e.g. Skylake) or older (e.g. Ivy Bridge) with Asus motherboards, as well as Asus desktops or laptops. There are two solutions to this problem: Use Legacy BIOS mode instead of UEFI mode Install a FreeBSD UEFI Boot entry Keep in mind that I am not going to talk about this issue and third-party UEFI boot managers such as rEFInd here. The first option is rather straightforward: you need to make sure your computer has “Secure Boot” disabled and “Legacy Boot” or “CSM” enabled. Then, you need to make sure FreeBSD is installed in BIOS mode. However, this solution is (in my opinion) suboptimal. Why? Because: You won’t be able to use hard drives bigger than 2TB You are limited to MBR Partitioning on Asus motherboards with UEFI as Asus motherboards refuse to boot GPT partitioned disks in BIOS mode Legacy BIOS mode may not exist on future computers or motherboards (although those systems may not have this issue, and this issue may get fixed by then) The second option, however, is less straightforward, but will let you keep UEFI. Many UEFI systems, including affected Asus motherboards described here, include a boot manager built into the UEFI. FreeBSD includes a tool called efibootmgr to manage this, similar to the similarly-named tool in Linux, but with a different syntax. ###Why ed(1) is not a good editor today I’ll start with my tweet: Heretical Unix opinion time: ed(1) may be the 'standard Unix editor', but it is not a particularly good editor outside of a limited environment that almost never applies today. There is a certain portion of Unixdom that really likes ed(1), the ‘standard Unix editor’. Having actually used ed for a not insignificant amount of time (although it was the friendlier ‘UofT ed’ variant), I have some reactions to what I feel is sometimes overzealous praise of it. One of these is what I tweeted. The fundamental limitation of ed is that it is what I call an indirect manipulation interface, in contrast to the explicit manipulation interfaces of screen editors like vi and graphical editors like sam (which are generally lumped together as ‘visual’ editors, so called because they actually show you the text you’re editing). When you edit text in ed, you have some problems that you don’t have in visual editors; you have to maintain in your head the context of what the text looks like (and where you are in it), you have to figure out how to address portions of that text in order to modify them, and finally you have to think about how your edit commands will change the context. Copious use of ed’s p command can help with the first problem, but nothing really deals with the other two. In order to use ed, you basically have to simulate parts of ed in your head. Ed is a great editor in situations where the editor explicitly presenting this context is a very expensive or outright impossible operation. Ed works great on real teletypes, for example, or over extremely slow links where you want to send and receive as little data as possible (and on real teletypes you have some amount of context in the form of an actual printout that you can look back at). Back in the old days of Unix, this described a fairly large number of situations; you had actual teletypes, you had slow dialup links (and later slow, high latency network links), and you had slow and heavily overloaded systems. However, that’s no longer the situation today (at least almost all of the time). Modern systems and links can easily support visual editors that continually show you the context of the text and generally let you more or less directly manipulate it (whether that is through cursoring around it or using a mouse). Such editors are easier and faster to use, and they leave you with more brainpower free to think about things like the program you’re writing (which is the important thing). If you can use a visual editor, ed is not a particularly good editor to use instead; you will probably spend a lot of effort (and some amount of time) on doing by hand something that the visual editor will do for you. If you are very practiced at ed, maybe this partly goes away, but I maintain that you are still working harder than you need to be. The people who say that ed is a quite powerful editor are correct; ed is quite capable (although sadly limited by only editing a single file). It’s just that it’s also a pain to use. (They’re also correct that ed is the foundation of many other things in Unix, including sed and vi. But that doesn’t mean that the best way to learn or understand those things is to learn and use ed.) This doesn’t make ed a useless, vestigial thing on modern Unix, though. There are uses for ed in non-interactive editing, for example. But on modern Unix, ed is a specialized tool, much like dc. It’s worth knowing that ed is there and roughly what it can do, but it’s probably not worth learning how to use it before you need it. And you’re unlikely to ever be in a situation where it’s the best choice for interactive editing (and if you are, something has generally gone wrong). (But if you enjoy exploring the obscure corners of Unix, sure, go for it. Learn dc too, because it’s interesting in its own way and, like ed, it’s one of those classical old Unix programs.) ##Beastie Bits Is there any interest in a #BSD user group in #Montreal? Tell your BSD story Finishing leftover tasks from Google Summer of Code Fuzzing the OpenBSD Kernel ARM - any Tier-1 *BSD options? ##Feedback/Questions Chris - byhve question Paulo - Topic suggestion Bostjan - How data gets to disk Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
Mitigating Spectre/Meltdown on HP Proliant servers, omniOS installation setup, debugging a memory corruption issue on OpenBSD, CfT for OpenZFS native encryption, Asigra TrueNAS backup appliance shown at VMworld, NetBSD 6 EoL, and more. ##Headlines How to mitigate Spectre and Meltdown on an HP Proliant server with FreeBSD As recently announced in a previous article I wanted to write a couple of guides on how to mitigate Spectre and Meltdown vulnerabilities in GNU/Linux and UNIX environments. It is always a good and I hope a standard practice to have your systems patched and if they aren’t for whatever the reason (that legacy thing you’re carrying on for ages) you may take the necessary extra steps to protect your environment. I never planned to do any article on patching anything. Nowadays it’s a no brainer and operating systems have provided the necessary tools for this to be easy and as smooth as possible. So why this article? Spectre and Meltdown are both hardware vulnerabilities. Major ones. They are meaningful for several reasons among them the world wide impact since they affect Intel and AMD systems which are ubiquitous. And second because patching hardware is not as easy, for the manufacturer and for the users or administrators in charge of the systems. There is still no known exploit around left out in the open hitting servers or desktops anywhere. The question is not if it will ever happen. The question is when will it happen. And it may be sooner than later. This is why big companies, governments and people in charge of big deployments are patching or have already patched their systems. But have you done it to your system? I know you have a firewall. Have you thought about CVE-2018-3639? This particular one could make your browser being a vector to get into your system. So, no, there is no reason to skip this. Patching these set of vulnerabilities implies some more steps and concerns than updating the operating system. If you are a regular Windows user I find rare you to be here and many of the things you will read may be foreign to you. I am not planning to do a guide on Windows systems since I believe someone else has or will do it and will do it better than me since I am not a pro Windows user. However there is one basic and common thing for all OS’s when dealing with Spectre and Meltdown and that is a microcode update is necessary for the OS patches to effectively work. What is microcode? You can read the Wikipedia article but in short it is basically a layer of code that allows chip manufacturers to deal with modifications on the hardware they’ve produced and the operating systems that will manage that hardware. Since there’s been some issues (namely Spectre and Meltdown) Intel and AMD respectively have released a series of microcode updates to address those problems. First series did come with serious problems and some regressions, to the point GNU/Linux producers stopped releasing the microcode updates through their release channels for updates and placed the ball on Intel’s roof. Patching fast does always include risks, specially when dealing with hardware. OS vendors have resumed their microcode update releases so all seems to be fine now. In order to update the microcode we’re faced with two options. Download the most recent BIOS release from our vendor, provided it patches the Spectre and Meltdown vulnerabilities, or patch it from the OS. If your hardware vendor has decided not to provide support on your hardware you are forced to use the latter solution. Yes, you can still keep your hardware. They usually come accompanied with a “release notes” file where there are some explanatory notes on what is fixed, what is new, etc. To make the search easy for you a news site collected the vendors list and linked the right support pages for anyone to look. In some scenarios it would be desirable not to replace the whole BIOS but just update the microcode from the OS side. In my case I should update an HP Proliant ML110 G7 box and the download link for that would be this. Instead of using the full blown BIOS update path we’ll use the inner utilities to patch Spectre and Meltdown on FreeBSD. So let’s put our hands on it See the article for the technical breakdown ###A look beyond the BSD teacup: OmniOS installation Five years ago I wrote a post about taking a look beyond the Linux teacup. I was an Arch Linux user back then and since there were projects like ArchBSD (called PacBSD today) and Arch Hurd, I decided to take a look at and write about them. Things have changed. Today I’m a happy FreeBSD user, but it’s time again to take a look beyond the teacup of operating systems that I’m familiar with. Why Illumos / OmniOS? There are a couple of reasons. The Solaris derivatives are the other big community in the *nix family besides Linux and the BSDs and we hadn’t met so far. Working with ZFS on FreeBSD, I now and then I read messages that contain a reference to Illumos which certainly helps to keep up the awareness. Of course there has also been a bit of curiosity – what might the OS be like that grew ZFS? Also the Ravenports project that I participate in planned to support Solaris/Illumos right from the beginning. I wanted to at least be somewhat “prepared” when support for that platform would finally land. So I did a little research on the various derivatives available and settled on the one that I had heard a talk about at last year’s conference of the German Unix Users Group: “OmniOS – Solaris for the Rest of Us”. I would have chosen SmartOS as I admire what Bryan Cantrill does but for getting to know Illumos I prefer a traditional installation over a run-from-RAM system. Of course FreeBSD is not run by corporations, especially when compared to the state of Linux. And when it comes to sponsoring, OpenBSD also takes the money… When it comes to FreeBSD developers, there’s probably some truth to the claim that some of them are using macOS as their desktop systems while OpenBSD devs are more likely to develop on their OS of choice. But then there’s the statement that “every innovation in the past decade comes from Solaris”. Bhyve alone proves this wrong. But let’s be honest: Two of the major technologies that make FreeBSD a great platform today – ZFS and DTrace – actually do come from Solaris. PAM originates there and a more modern way of managing services as well. Also you hear good things about their zones and a lot of small utilities in general. In the end it was a lack of time that made me cheat and go down the easiest road: Create a Vagrantfile and just pull a VM image of the net that someone else had prepared… This worked to just make sure that the Raven packages work on OmniOS. I was determined to return, though – someday. You know how things go: “someday” is a pretty common alias for “probably never, actually.” But then I heard about a forum post on the BSDNow! podcast. The title “Initial OmniOS impressions by a BSD user” caught my attention. I read that it was written by somebody who had used FreeBSD for years but loathed the new Code of Conduct enough to leave. I also oppose the Conduct and have made that pretty clear in my February post [ ! -z ${COC} ] && exit 1. As stated there, I have stayed with my favorite OS and continue to advocate it. I decided to stop reading the post and try things out on my own instead. Now I’ve finally found the time to do so. What’s next? That’s it for part one. In part two I’ll try to make the system useful. So far I have run into a problem that I haven’t been able to solve. But I have some time now to figure things out for the next post. Let’s see if I manage to get it working or if I have to report failure! ###What are all these types of memory in top(1)? Earlier this week I convinced Mark Johnston, one of the FreeBSD VM experts to update a page on the FreeBSD wiki that I saw was being referenced on stackoverflow and similar sites Mark updated the explanations to be more correct, and to include more technical detail for inquiring minds He also added the new type that appeared in FreeBSD somewhat recently Active - Contains memory “actively” (recently) being used by applications Inactive - Contains memory that has not been touched recently, or was released from the Buffer Cache Laundry - Contains memory that Inactive but still potentially contains useful data that needs to be stored before this memory can be used again Wired - Memory that cannot be swapped out, including the kernel, network stack, and the ZFS ARC Buf - Buffer Cache, used my UFS and most filesystems except ZFS (which uses the ARC) Free - Memory that is immediately available for use by the rest of the system ##News Roundup OpenBSD saves me again! — Debug a memory corruption issue Yesterday, I came across a third-part library issue, which crashes at allocating memory: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f594a5a9b6b in _int_malloc () from /usr/lib/libc.so.6 (gdb) bt #0 0x00007f594a5a9b6b in _int_malloc () from /usr/lib/libc.so.6 #1 0x00007f594a5ab503 in malloc () from /usr/lib/libc.so.6 #2 0x00007f594b13f159 in operator new (sz=5767168) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50 It is obvious that the memory tags are corrupted, but who is the murder? Since the library involves a lot of maths computation, it is not an easy task to grasp the code quickly. So I need to find another way: (1) Open all warnings during compilation: -Wall. Nothing found. (2) Use valgrind, but unfortunately, valgrind crashes itself: valgrind: the 'impossible' happened: Killed by fatal signal host stacktrace: ==43326== at 0x58053139: get_bszB_as_is (m_mallocfree.c:303) ==43326== by 0x58053139: get_bszB (m_mallocfree.c:315) ==43326== by 0x58053139: vgPlain_arena_malloc (m_mallocfree.c:1799) ==43326== by 0x5800BA84: vgMemCheck_new_block (mc_malloc_wrappers.c:372) ==43326== by 0x5800BD39: vgMemCheck___builtin_vec_new (mc_malloc_wrappers.c:427) ==43326== by 0x5809F785: do_client_request (scheduler.c:1866) ==43326== by 0x5809F785: vgPlain_scheduler (scheduler.c:1433) ==43326== by 0x580AED50: thread_wrapper (syswrap-linux.c:103) ==43326== by 0x580AED50: run_a_thread_NORETURN (syswrap-linux.c:156) sched status: running_tid=1 (3) Change compiler, use clang instead of gcc, and hope it can give me some clues. Still no effect. (4) Switch Operating System from Linux to OpenBSD, the program crashes again. But this time, it tells me where the memory corruption occurs: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000014b07f01e52d in addMod (r=, a=4693443247995522, b=28622907746665631, I figure out the issue quickly, and not bother to understand the whole code. OpenBSD saves me again, thanks! ###Native Encryption for ZFS on FreeBSD (Call for Testing) To anyone with an interest in native encryption in ZFS please test the projects/zfs-crypto-merge-0820 branch in my freebsd repo: https://github.com/mattmacy/networking.git git clone https://github.com/mattmacy/networking.git -b projects/zfs-crypto-merge-0820 The UI is quite close to the Oracle Solaris ZFS crypto with minor differences for specifying key location. Please note that once a feature is enabled on a pool it can’t be disabled. This means that if you enable encryption support on a pool you will never be able to import it in to a ZFS without encryption support. For this reason I would strongly advise against using this on any pool that can’t be easily replaced until this change has made its way in to HEAD after the freeze has been lifted. By way of background the original ZoL commit can be found at: https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 ###VMworld 2018: Showcasing Hybrid Cloud, Persistent Memory and the Asigra TrueNAS Backup Appliance During its last year in Las Vegas before moving back to San Francisco, VMworld was abuzz with all the popular buzzwords, but the key focus was on supporting a more agile approach to hybrid cloud. Surveys of IT stakeholders and analysts agree that most businesses have multiple clouds spanning both public cloud providers and private data centers. While the exact numbers vary, well over half of businesses have a hybrid cloud strategy consisting of at least three different clouds. This focus on hybrid cloud provided the perfect timing for our announcement that iXsystems and Asigra are partnering to deliver the Asigra TrueNAS Backup Appliance, which combines Asigra Cloud Backup software backed by TrueNAS storage. Asigra TrueNAS Backup Appliances provide a self-healing and ransomware-resistent OpenZFS backup repository in your private cloud. The appliance can simultaneously be used as general-purpose file, block, and object storage. How does this tie in with the hybrid cloud? The Asigra Cloud Backup software can backup data from public cloud repositories – G Suite, Office 365, Salesforce, etc. – as well as intelligently move backed-up data to the public cloud for long-term retention. Another major theme at the technical sessions was persistent memory, as vSphere 6.7 added support for persistent memory – either as a storage tier or virtualized and presented to a guest OS. As detailed in our blog post from SNIA’s Persistent Memory Summit 2018, persistent memory is rapidly becoming mainstream. Persistent memory bridges the gap between memory and flash storage – providing near-memory latency storage that persists across reboots or power loss. vSphere allows both legacy and persistent memory-aware applications to leverage this ultra-fast storage tier. We were excited to show off our newly-introduced TrueNAS M-Series at VMworld, as all TrueNAS M40 and M50 models leverage NVDIMM persistent memory technology to provide a super-fast write cache, or SLOG, without any of the limitations of Flash technology. The iXsystems booth’s theme was “Enterprise Storage, Open Source Economics”. iXsystems leverages the power of Open Source software, combined with our enterprise-class hardware and support, to provide incredibly low TCO storage for virtualization environments. Our TrueNAS unified storage and server offerings are an ideal solution for your organization’s private cloud infrastructure. Combined with VMware NSX Hybrid Connect – formerly known as VMware Hybrid Cloud Extension – you can seamlessly shift running systems into a public cloud environment for a true hybrid cloud solution. Another special treat at this year’s booth was iXsystems Vice President of Engineering Kris Moore giving demos of an early version of “Project TrueView”, a single-pane of glass management solution for administration of multiple FreeNAS and TrueNAS systems. In addition to simplified administration and enhanced monitoring, Project TrueView will also provide Role-Based Access Control for finer-grained permissions management. A beta version of Project TrueView is expected to be available at the end of this year. Overall, we had a great week at VMworld 2018 with lots of good conversations with customers, press, analysts, and future customers about TrueNAS, the Asigra TrueNAS Backup Appliance, iXsystems servers, Project TrueView, and more – our booth was more popular than ever! ###End of life for NetBSD 6.x In keeping with NetBSD’s policy of supporting only the latest (8.x) and next most recent (7.x) major branches, the recent release of NetBSD 8.0 marks the end of life for NetBSD 6.x. As in the past, a month of overlapping support has been provided in order to ease the migration to newer releases. As of now, the following branches are no longer maintained: netbsd-6-1 netbsd-6-0 netbsd-6 This means: There will be no more pullups to those branches (even for security issues) There will be no security advisories made for any those branches The existing 6.x releases on ftp.NetBSD.org will be moved into /pub/NetBSD-archive/ May NetBSD 8.0 serve you well! (And if it doesn’t, please submit a PR!) ##Beastie Bits Blast from the past: OpenBSD 3.7 CD artwork People are asking about scale of BSD projects. Let’s figure it out… Tuesday, 21 August 18: me, on ed(1), at SemiBUG arm64 gains RETGUARD Call for participation FreeBSD-UPB/bhyvearm64-utils ##Feedback/Questions Eric - FreeNAS for Vacation Patrick - Long Live Unix Jason - Jason - Full MP3 Recordings Bostjan - Question about jails and kernel Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
Fanless server setup with FreeBSD, NetBSD on pinebooks, another BSDCan trip report, transparent network audio, MirBSD's Korn Shell on Plan9, static site generators on OpenBSD, and more. ##Headlines Silent Fanless FreeBSD Desktop/Server Today I will write about silent fanless FreeBSD desktop or server computer … or NAS … or you name it, it can have multiple purposes. It also very low power solution, which also means that it will not overheat. Silent means no fans at all, even for the PSU. The format of the system should also be brought to minimum, so Mini-ITX seems best solution here. I have chosen Intel based solutions as they are very low power (6-10W), if you prefer AMD (as I often do) the closest solution in comparable price and power is Biostar A68N-2100 motherboard with AMD E1-2100 CPU and 9W power. Of course AMD has even more low power SoC solutions but finding the Mini-ITX motherboard with decent price is not an easy task. For comparison Intel has lots of such solutions below 6W whose can be nicely filtered on the ark.intel.com page. Pity that AMD does not provide such filtration for their products. I also chosen AES instructions as storage encryption (GELI on FreeBSD) today seems as obvious as HTTPS for the web pages. Here is how the system look powered up and working This motherboard uses Intel J3355 SoC which uses 10W and has AES instructions. It has two cores at your disposal but it also supports VT-x and EPT extensions so you can even run Bhyve on it. Components Now, an example system would look like that one below, here are the components with their prices. $49 CPU/Motherboard ASRock J3355B-ITX Mini-ITX $14 RAM Crucial 4 GB DDR3L 1.35V (low power) $17 PSU 12V 160W Pico (internal) $11 PSU 12V 96W FSP (external) $5 USB 2.0 Drive 16 GB ADATA $4 USB Wireless 802.11n $100 TOTAL The PSU 12V 160W Pico (internal) and PSU 12V 96W FSP can be purchased on aliexpress.com or ebay.com for example, at least I got them there. Here is the 12V 160W Pico (internal) PSU and its optional additional cables to power the optional HDDs. If course its one SATA power and one MOLEX power so additional MOLEX-SATA power adapter for about 1$ would be needed. Here is the 12V 96W FSP (external) PSU without the power cord. This gives as total silent fanless system price of about $120. Its about ONE TENTH OF THE COST of the cheapest FreeNAS hardware solution available – the FreeNAS Mini (Diskless) costs $1156 also without disks. You can put plain FreeBSD on top of it or Solaris/Illumos distribution OmniOSce which is server oriented. You can use prebuilt NAS solution based on FreeBSD like FreeNAS, NAS4Free, ZFSguru or even Solaris/Illumos based storage with napp-it appliance. ###An annotated look at a NetBSD Pinebook’s startup Pinebook is an affordable 64-bit ARM notebook. Today we’re going to take a look at the kernel output at startup and talk about what hardware support is available on NetBSD. Photo Pinebook comes with 2GB RAM standard. A small amount of this is reserved by the kernel and framebuffer. NetBSD uses flattened device-tree (FDT) to enumerate devices on all Allwinner based SoCs. On a running system, you can inspect the device tree using the ofctl(8) utility: Pinebook’s Allwinner A64 processor is based on the ARM Cortex-A53. It is designed to run at frequencies up to 1.2GHz. The A64 is a quad core design. NetBSD’s aarch64 pmap does not yet support SMP, so three cores are disabled for now. The interrupt controller is a standard ARM GIC-400 design. Clock drivers for managing PLLs, module clock dividers, clock gating, software resets, etc. Information about the clock tree is exported in the hw.clk sysctl namespace (root access required to read these values). # sysctl hw.clk.sun50ia64ccu0.mmc2 hw.clk.sun50ia64ccu0.mmc2.rate = 200000000 hw.clk.sun50ia64ccu0.mmc2.parent = pllperiph02x hw.clk.sun50ia64ccu0.mmc2.parent_domain = sun50ia64ccu0 Digital Ocean http://do.co/bsdnow ###BSDCan 2018 Trip Report: Mark Johnston BSDCan is a highlight of my summers: the ability to have face-to-face conversations with fellow developers and contributors is invaluable and always helps refresh my enthusiasm for FreeBSD. While in a perfect world we would all be able to communicate effectively over the Internet, it’s often noted that locking a group of developers together in a room can be a very efficient way to make progress on projects that otherwise get strung out over time, and to me this is one of the principal functions of BSD conferences. In my case I was able to fix some kgdb bugs that had been hindering me for months; get some opinions on the design of a feature I’ve been working on for FreeBSD 12.0; hear about some ongoing usage of code that I’ve worked on; and do some pair-debugging of an issue that has been affecting another developer. As is tradition, on Tuesday night I dropped off my things at the university residence where I was staying, and headed straight to the Royal Oak. This year it didn’t seem quite as packed with BSD developers, but I did meet several long-time colleagues and get a chance to catch up. In particular, I chatted with Justin Hibbits and got to hear about the bring-up of FreeBSD on POWER9, a new CPU family released by IBM. Justin was able to acquire a workstation based upon this CPU, which is a great motivator for getting FreeBSD into shape on that platform. POWER9 also has some promise in the server market, so it’s important for FreeBSD to be a viable OS choice there. Wednesday morning saw the beginning of the two-day FreeBSD developer summit, which precedes the conference proper. Gordon Tetlow led the summit and did an excellent job organizing things and keeping to the schedule. The first presentation was by Deb Goodkin of the FreeBSD Foundation, who gave an overview of the Foundation’s role and activities. After Deb’s presentation, present members of the FreeBSD core team discussed the work they had done over the past two years, as well as open tasks that would be handed over to the new core team upon completion of the ongoing election. Finally, Marius Strobl rounded off the day’s presentations by discussing the state and responsibilities of FreeBSD’s release engineering team. One side discussion of interest to me was around the notion of tightening integration with our Bugzilla instance; at moment we do not have any good means to mark a given bug as blocking a release, making it easy for bugs to slip into releases and thus lowering our overall quality. With FreeBSD 12.0 upon us, I plan to help with the triage and fixes for known regressions before the release process begins. After a break, the rest of the morning was devoted to plans for features in upcoming FreeBSD releases. This is one of my favorite discussion topics and typically takes the form of have/need/want, where developers collectively list features that they’ve developed and intend to upstream (have), features that they are missing (need), and nice-to-have features (want). This year, instead of the usual format, we listed features that are intended to ship in FreeBSD 12.0. The compiled list ended up being quite ambitious given how close we are to the beginning of the release cycle, but many individual developers (including myself) have signed up to deliver work. I’m hopeful that most, if not all of it, will make it into the release. After lunch, I attended a discussion led by Matt Ahrens and Alexander Motin on OpenZFS. Of particular interest to me were some observations made regarding the relative quantity and quality of contributions made by different “camps” of OpenZFS users (illumos, FreeBSD and ZoL), and their respective track records of upstreaming enhancements to the OpenZFS project. In part due to the high pace of changes in ZoL, the definition of “upstream” for ZFS has become murky, and of late ZFS changes have been ported directly from ZoL. Alexander discussed some known problems with ZFS on FreeBSD that have been discovered through performance testing. While I’m not familiar with ZFS internals, Alexander noted that ZFS’ write path has poor SMP scalability on FreeBSD owing to some limitations in a certain kernel API called taskqueue(9). I would like to explore this problem further and perhaps integrate a relatively new alternative interface which should perform better. Friday and Saturday were, of course, taken up by BSDCan talks. Friday’s keynote was by Benno Rice, who provided some history of UNIX boot systems as a precursor to some discussion of systemd and the difficulties presented by a user and developer community that actively resist change. The rest of the morning was consumed by talks and passed by quickly. First was Colin Percival’s detailed examination of where the FreeBSD kernel spends time during boot, together with an overview of some infrastructure he added to track boot times. He also provided a list of improvements that have been made since he started taking measurements, and some areas we can further improve. Colin’s existing work in this area has already brought about substantial reductions in boot time; amusingly, one of the remaining large delays comes from the keyboard driver, which contains a workaround for old PS/2 keyboards. While there seems to be general agreement that the workaround is probably no longer needed on most systems, the lingering uncertainty around this prevents us from removing the workaround. This is, sadly, a fairly typical example of an OS maintenance burden, and underscores the need to carefully document hardware bug workarounds. After this talk, I got to see some rather novel demonstrations of system tracing using dwatch, a new utility by Devin Teske, which aims to provide a user-friendly interface to DTrace. After lunch, I attended talks on netdump, a protocol for transmitting kernel dumps over a network after the system has panicked, and on a VPC implementation for FreeBSD. After the talks ended, I headed yet again to the hacker lounge and had some fruitful discussions on early microcode loading (one of my features for FreeBSD 12.0). These led me to reconsider some aspects of my approach and saved me a lot of time. Finally, I continued my debugging session from Wednesday with help from a couple of other developers. Saturday’s talks included a very thorough account by Li-Wen Hsu of his work in organizing a BSD conference in Taipei last year. As one of the attendees, I had felt that the conference had gone quite smoothly and was taken aback by the number of details and pitfalls that Li-Wen enumerated during his talk. This was followed by an excellent talk by Baptiste Daroussin on the difficulties one encounters when deploying FreeBSD in new environments. Baptiste offered criticisms of a number of aspects of FreeBSD, some of which hit close to home as they involved portions of the system that I’ve worked on. At the conclusion of the talks, we all gathered in the main lecture hall, where Dan led a traditional and quite lively auction for charity. I managed to snag a Pine64 board and will be getting FreeBSD installed on it the first chance I get. At the end of the auction, we all headed to ByWard for dinner, concluding yet another BSDCan. Thanks to Mark for sharing his experiences at this years BSDCan ##News Roundup Transparent network audio with mpd & sndiod Landry Breuil (landry@ when wearing his developer hat) wrote in… I've been a huge fan of MPD over the years to centralize my audio collection, and i've been using it with the http output to stream the music as a radio on the computer i'm currently using… audio_output { type "sndio" name "Local speakers" mixer_type "software" } audio_output { type "httpd" name "HTTP stream" mixer_type "software" encoder "vorbis" port "8000" format "44100:16:2" } this setup worked for years, allows me to stream my home radio to $work by tunnelling the port 8000 over ssh via LocalForward, but that still has some issues: a distinct timing gap between the 'local output' (ie the speakers connected to the machine where MPD is running) and the 'http output' caused by the time it takes to reencode the stream, which is ugly when you walk through the house and have a 15s delay sometimes mplayer as a client doesn't detect the pauses in the stream and needs to be restarted i need to configure/start a client on each computer and point it at the sound server url (can do via gmpc shoutcast client plugin…) it's not that elegant to reencode the stream, and it wastes cpu cycles So the current scheme is: mpd -> http output -> network -> mplayer -> sndiod on remote machine | -> sndio output -> sndiod on soundserver Fiddling a little bit with mpd outputs and reading the sndio output driver, i remembered sndiod has native network support… and the mpd sndio output allows you to specify a device (it uses SIO_DEVANY by default). So in the end, it's super easy to: enable network support in sndio on the remote machine i want the audio to play by adding -L to sndiod_flags (i have two audio devices, with an input coming from the webcam): sndiod_flags="-L10.246.200.10 -f rsnd/0 -f rsnd/1" open pf on port 11025 from the sound server ip: pass in proto tcp from 10.246.200.1 to any port 11025 configure a new output in mpd: audio_output { type "sndio" name "sndio on renton" device "snd@10.246.200.10/0" mixer_type "software" } and enable the new output in mpd: $mpc enable 2 Output 1 (Local speakers) is disabled Output 2 (sndio on renton) is enabled Output 3 (HTTP stream) is disabled Results in a big win: no gap anymore with the local speakers, no reencoding, no need to configure a client to play the stream, and i can still probably reproduce the same scheme over ssh from $work using a RemoteForward. mpd -> sndio output 2 -> network -> sndiod on remote machine | -> sndio output 1 -> sndiod on soundserver Thanks ratchov@ for sndiod :) ###MirBSD’s Korn Shell on Plan9 Jehanne Let start by saying that I’m not really a C programmer. My last public contribution to a POSIX C program was a little improvement to the Snort’s react module back in 2008. So while I know the C language well enough, I do not know anything about the subtleness of the standard library and I have little experience with POSIX semantics. This is not a big issue with Plan 9, since the C library and compiler are not standard anyway, but with Jehanne (a Plan 9 derivative of my own) I want to build a simple, loosely coupled, system that can actually run useful free software ported from UNIX. So I ported RedHat’s newlib to Jehanne on top of a new system library I wrote, LibPOSIX, that provides the necessary emulations. I wrote several test, checking they run the same on Linux and Jehanne, and then I begun looking for a real-world, battle tested, application to port first. I approached MirBSD’s Korn Shell for several reason: it is simple, powerful and well written it has been ported to several different operating systems it has few dependencies it’s the default shell in Android, so it’s really battle tested I was very confident. I had read the POSIX standard after all! And I had a test suite! I remember, I thought “Given newlib, how hard can it be?” The porting begun on September 1, 2017. It was completed by tg on January 5, 2018. 125 nights later. Turn out, my POSIX emulation was badly broken. Not just because of the usual bugs that any piece of C can have: I didn’t understood most POSIX semantics at all! iXsystems ###Static site generator with rsync and lowdown on OpenBSD ssg is a tiny POSIX-compliant shell script with few dependencies: lowdown(1) to parse markdown, rsync(1) to copy temporary files, and entr(1) to watch file changes. It generates Markdown articles to a static website. It copies the current directory to a temporary on in /tmp skipping .* and _*, renders all Markdown articles to HTML, generates RSS feed based on links from index.html, extracts the first tag from every article to generate a sitemap and use it as a page title, then wraps articles with a single HTML template, copies everything from the temporary directory to $DOCS/ Why not Jekyll or “$X”? ssg is one hundred times smaller than Jekyll. ssg and its dependencies are about 800KB combined. Compare that to 78MB of ruby with Jekyll and all the gems. So ssg can be installed in just few seconds on almost any Unix-like operating system. Obviously, ssg is tailored for my needs, it has all features I need and only those I use. Keeping ssg helps you to master your Unix-shell skills: awk, grep, sed, sh, cut, tr. As a web developer you work with lots of text: code and data. So you better master these wonderful tools. Performance 100 pps. On modern computers ssg generates a hundred pages per second. Half of a time for markdown rendering and another half for wrapping articles into the template. I heard good static site generators work—twice as fast—at 200 pps, so there’s lots of performance that can be gained. ;) ###Why does FreeBSD have virtually no (0%) desktop market share? Because someone made a horrible design decision back in 1984. In absolute fairness to those involved, it was an understandable decision, both from a research perspective, and from an economic perspective, although likely not, from a technology perspective. Why and what. The decision was taken because the X Window System was intended to run on cheap hardware, and, at the time, that meant reduced functionality in the end-point device with the physical display attached to it. At the same time, another force was acting to also limit X displays to display services only, rather than rolling in both window management and specific widget instances for common operational paradigms. Mostly, common operational paradigms didn’t really exist for windowing systems because they also simply didn’t exist at the time, and no one really knew how people were going to use the things, and so researchers didn’t want to commit future research to a set of hard constraints. So a decision was made: separate the display services from the application at the lowest level of graphics primitives currently in use at the time. The ramifications of this were pretty staggering. First, it guaranteed that all higher level graphics would live on the host side of the X protocol, instead of on the display device side of the protocol. Despite a good understanding of Moore’s law, and the fact that, since no X Terminals existed at the time as hardware, but were instead running as emulations on workstations that had sufficient capability, this put the higher level GUI object libraries — referred to as “widgets” — in host libraries linked into the applications. Second, it guaranteed that display organization and management paradigms would also live on the host side of the protocol — assumed, in contradiction to the previous decision, to be running on the workstation. But, presumably, at some point, as lightweight X Terminals became available, to migrate to a particular host computer managing compute resource login/access services. Between these early decisions reigned chaos. Specifically, the consequences of these decisions have been with us ever since: Look-and-feel are a consequence of the toolkit chosen by the application programmer, rather than a user decision which applies universally to all applications. You could call this “lack of a theme”, and — although I personally despise the idea of customizing or “theming” desktops — this meant that one paradigm chosen by the user would not apply universally across all applications, no matter who had written them. Window management style is a preference. You could call this a more radical version of “theming” — which you will remember, I despise — but a consequence to this is that training is not universal across personnel using such systems, nor is it transferrable. In other words, I can’t send someone to a class, and have them come back and use the computers in the office as a tool, with the computer itself — and the elements not specific to the application itself — disappearing into the background. Both of these ultimately render an X-based system unsuitable for desktops. I can’t pay once for training. Training that I do pay for does not easily and naturally translate between applications. Each new version may radically alter the desktop management paradigm into unrecognizability. Is there hope for the future? Well, the Linux community has been working on something called Wayland, and it is very promising… …In the same way X was “very promising” in 1984, because, unfortunately, they are making exactly the same mistakes X made in 1984, rather than correcting them, now that we have 20/20 hindsight, and know what a mature widget library should look like. So Wayland is screwing up again. But hey, it only took us, what, 25 years to get from X in 1987 to Wayland in in 2012. Maybe if we try again in 2037, we can get to where Windows was in 1995. ##Beastie Bits New washing machine comes with 7 pages of open source licenses! BSD Jobs Site FreeBSD Foundation Update, May 2018 FreeBSD Journal looking for book reviewers zedenv ZFS Boot Environment Manager Tarsnap ##Feedback/Questions Wouter - Feedback Efraim - OS Suggestion kevr - Raspberry Pi2/FreeBSD/Router on a Stick Vanja - Interview Suggestion Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
FreeBSD 11.2 has been released, setting up an MTA behind Tor, running pfsense on DigitalOcean, one year of C, using OpenBGPD to announce VM networks, the power to serve, and a BSDCan trip report. ##Headlines FreeBSD 11.2-RELEASE Available FreeBSD 11.2 was released today (June 27th) and is ready for download Highlights: OpenSSH has been updated to version 7.5p1. OpenSSL has been updated to version 1.0.2o. The clang, llvm, lldb and compiler-rt utilities have been updated to version 6.0.0. The libarchive(3) library has been updated to version 3.3.2. The libxo(3) library has been updated to version 0.9.0. Major Device driver updates to: cxgbe(4) – Chelsio 10/25/40/50/100 gigabit NICs – version 1.16.63.0 supports T4, T5 and T6 ixl(4) – Intel 10 and 40 gigabit NICs, updated to version 1.9.9-k ng_pppoe(4) – driver has been updated to add support for user-supplied Host-Uniq tags New drivers: + drm-next-kmod driver supporting integrated Intel graphics with the i915 driver. mlx5io(4) – a new IOCTL interface for Mellanox ConnectX-4 and ConnectX-5 10/20/25/40/50/56/100 gigabit NICs ocs_fc(4) – Emulex Fibre Channel 8/16/32 gigabit Host Adapters smartpqi(4) – HP Gen10 Smart Array Controller Family The newsyslog(8) utility has been updated to support RFC5424-compliant messages when rotating system logs The diskinfo(8) utility has been updated to include two new flags, -s which displays the disk identity (usually the serial number), and -p which displays the physical path to the disk in a storage controller. The top(1) utility has been updated to allow filtering on multiple user names when the -U flag is used The umount(8) utility has been updated to include a new flag, -N, which is used to forcefully unmount an NFS mounted filesystem. The ps(1) utility has been updated to display if a process is running with capsicum(4) capability mode, indicated by the flag ‘C’ The service(8) utility has been updated to include a new flag, -j, which is used to interact with services running within a jail(8). The argument to -j can be either the name or numeric jail ID The mlx5tool(8) utility has been added, which is used to manage Connect-X 4 and Connect-X 5 devices supported by mlx5io(4). The ifconfig(8) utility has been updated to include a random option, which when used with the ether option, generates a random MAC address for an interface. The dwatch(1) utility has been introduced The efibootmgr(8) utility has been added, which is used to manipulate the EFI boot manager. The etdump(1) utility has been added, which is used to view El Torito boot catalog information. The linux(4) ABI compatibility layer has been updated to include support for musl consumers. The fdescfs(5) filesystem has been updated to support Linux®-specific fd(4) /dev/fd and /proc/self/fd behavior Support for virtio_console(4) has been added to bhyve(4). The length of GELI passphrases entered when booting a system with encrypted disks is now hidden by default. See the configuration options in geli(8) to restore the previous behavior. In addition to the usual CD/DVD ISO, Memstick, and prebuilt VM images (raw, qcow2, vhd, and vmdk), FreeBSD 11.2 is also available on: Amazon EC2 Google Compute Engine Hashicorp/Atlas Vagrant Microsoft Azure In addition to a generic ARM64 image for devices like the Pine64 and Raspberry Pi 3, specific images are provided for: GUMSTIX BANANAPI BEAGLEBONE CUBIEBOARD CUBIEBOARD2 CUBOX-HUMMINGBOARD RASPBERRY PI 2 PANDABOARD WANDBOARD Full Release Notes ###Setting up an MTA Behind Tor This article will document how to set up OpenSMTPD behind a fully Tor-ified network. Given that Tor’s DNS resolver code does not support MX record lookups, care must be taken for setting up an MTA behind a fully Tor-ified network. OpenSMTPD was chosen because it was easy to modify to force it to fall back to A/AAAA lookups when MX lookups failed with a DNS result code of NOTIMP (4). Note that as of 08 May 2018, the OpenSMTPD project is planning a configuration file language change. The proposed change has not landed. Once it does, this article will be updated to reflect both the old language and new. The reason to use an MTA behing a fully Tor-ified network is to be able to support email behind the .onion TLD. This setup will only allow us to send and receive email to and from the .onion TLD. Requirements: A fully Tor-ified network HardenedBSD as the operating system A server (or VM) running HardenedBSD behind the fully Tor-ified network. /usr/ports is empty Or is already pre-populated with the HardenedBSD Ports tree Why use HardenedBSD? We get all the features of FreeBSD (ZFS, DTrace, bhyve, and jails) with enhanced security through exploit mitigations and system hardening. Tor has a very unique threat landscape and using a hardened ecosystem is crucial to mitigating risks and threats. Also note that this article reflects how I’ve set up my MTA. I’ve included configuration files verbatim. You will need to replace the text that refers to my .onion domain with yours. On 08 May 2018, HardenedBSD’s version of OpenSMTPD just gained support for running an MTA behind Tor. The package repositories do not yet contain the patch, so we will compile OpenSMTPD from ports. Steps Installation Generating Cryptographic Key Material Tor Configuration OpenSMTPD Configuration Dovecot Configuration Testing your configuration Optional: Webmail Access iXsystems https://www.forbes.com/sites/forbestechcouncil/2018/06/21/strings-attached-knowing-when-and-when-not-to-accept-vc-funding/#30f9f18f46ec https://www.ixsystems.com/blog/self-2018-recap/ ###Running pfSense on a Digital Ocean Droplet I love pfSense (and opnSense, no discrimination here). I use it for just about anything, from homelab to large scale deployments and I’ll give out on any fancy for a pfSense setup on a decent hardware. I also love DigitalOcean, if you ever used them, you know why, if you never did, head over and try, you’ll understand why. . Unfortunately, while DO offers tremendous amount of useful distros and applications, pfSense isn’t one of them. But, where there’s a will, there’s a way, and here’s how to get pfSense up and running on DO so you can have it as the gatekeeper to your kingdom. Start by creating a FreeBSD droplet, choose your droplet size (for modest setups, I find the 5$ to be quite awesome): There are many useful things you can do with pfSense on your droplet, from OpenVPN, squid, firewalling, fancy routing, url filtering, dns black listing and much much more. One note though, before we wrap up: You have two ways to initiate the initial setup wizard of the web-configurator: Spin up another droplet, log into it and browse your way to the INTERNAL ip address of the internal NIC you’ve set up. This is the long and tedious way, but it’s also somewhat safer as it eliminates the small window of risk the second method poses. or Once your WAN address is all setup, your pfSense is ready to accept https connection to start the initial web-configurator setup. Thing is, there’s a default, well known set of credential to this initial wizard (admin:pfsense), so, there is a slight window of opportunity that someone can swoop in (assuming they know you’ve installed pfsense + your wan IP address + the exact time window between setting up the WAN interface and completing the wizard) and do . I leave it up to you which of the path you’d like to go, either way, once you’re done with the web-configurator wizard, you’ll have a shiny new pfSense installation at your disposal running on your favorite VPS. Hopefully this was helpful for someone, I hope to get a similar post soon detailing how to get FreeNAS up and running on DO. Many thanks to Tubsta and his blogpost as well as to Allan Jude, Kris Moore and Benedict Reuschling for their AWESOME and inspiring podcast, BSD Now. ##News Roundup One year of C It’s now nearly a year that I started writing non-trivial amounts of C code again (the first sokol_gfx.h commit was on the 14-Jul-2017), so I guess it’s time for a little retrospective. In the beginning it was more of an experiment: I wanted to see how much I would miss some of the more useful C++ features (for instance namespaces, function overloading, ‘simple’ template code for containers, …), and whether it is possible to write non-trivial codebases in C without going mad. Here are all the github projects I wrote in C: sokol: a slowly growing set of platform-abstraction headers sokol-samples - examples for Sokol chips - 8-bit chip emulators chips-test - tests and examples for the chip- emulators, including some complete home computer emulators (minus sound) All in all these are around 32k lines of code (not including 3rd party code like flextGL and HandmadeMath). I think I wrote more C code in the recent 10 months than any other language. So one thing seems to be clear: yes, it’s possible to write a non-trivial amount of C code that does something useful without going mad (and it’s even quite enjoyable I might add). Here’s a few things I learned: Pick the right language for a problem C is a perfect match for WebAssembly C99 is a huge improvement over C89 The dangers of pointers and explicit memory management are overrated Less Boilerplate Code Less Language Feature ‘Anxiety’ Conclusion All in all my “C experiment” is a success. For a lot of problems, picking C over C++ may be the better choice since C is a much simpler language (btw, did you notice how there are hardly any books, conferences or discussions about C despite being a fairly popular language? Apart from the neverending bickering about undefined behaviour from the compiler people of course ;) There simply isn’t much to discuss about a language that can be learned in an afternoon. I don’t like some of the old POSIX or Linux APIs as much as the next guy (e.g. ioctl(), the socket API or some of the CRT library functions), but that’s an API design problem, not a language problem. It’s possible to build friendly C APIs with a bit of care and thinking, especially when C99’s designated initialization can be used (C++ should really make sure that the full C99 language can be used from inside C++ instead of continuing to wander off into an entirely different direction). ###Configuring OpenBGPD to announce VM’s virtual networks We use BGP quite heavily at work, and even though I’m not interacting with that directly, it feels like it’s something very useful to learn at least on some basic level. The most effective and fun way of learning technology is finding some practical application, so I decided to see if it could help to improve networking management for my Virtual Machines. My setup is fairly simple: I have a host that runs bhyve VMs and I have a desktop system from where I ssh to VMs, both hosts run FreeBSD. All VMs are connected to each other through a bridge and have a common network 10.0.1/24. The point of this exercise is to be able to ssh to these VMs from desktop without adding static routes and without adding vmhost’s external interfaces to the VMs bridge. I’ve installed openbgpd on both hosts and configured it like this: vmhost: /usr/local/etc/bgpd.conf AS 65002 router-id 192.168.87.48 fib-update no network 10.0.1.1/24 neighbor 192.168.87.41 { descr "desktop" remote-as 65001 } Here, router-id is set vmhost’s IP address in my home network (192.168.87/24), fib-update no is set to forbid routing table update, which I initially set for testing, but keeping it as vmhost is not supposed to learn new routes from desktop anyway. network announces my VMs network and neighbor describes my desktop box. Now the desktop box: desktop: /usr/local/etc/bgpd.conf AS 65001 router-id 192.168.87.41 fib-update yes neighbor 192.168.87.48 { descr "vmhost" remote-as 65002 } It’s pretty similar to vmhost’s bgpd.conf, but no networks are announced here, and fib-update is set to yes because the whole point is to get VM routes added. Both hosts have to have the openbgpd service enabled: /etc/rc.conf.local openbgpdenable="YES" Conclusion As mentioned already, similar result could be achieved without using BGP by using either static routes or bridging interfaces differently, but the purpose of this exercise is to get some basic hands-on experience with BGP. Right now I’m looking into extending my setup in order to try more complex BGP schema. I’m thinking about adding some software switches in front of my VMs or maybe adding a second VM host (if budget allows). You’re welcome to comment if you have some ideas how to extend this setup for educational purposes in the context of BGP and networking. As a side note, I really like openbgpd so far. Its configuration file format is clean and simple, documentation is good, error and information messages are clear, and CLI has intuitive syntax. Digital Ocean ###The Power to Serve All people within the IT Industry should known where the slogan “The Power To Serve” is exposed every day to millions of people. But maybe too much wishful thinking from me. But without “The Power To Serve” the IT industry today will look totally different. Companies like Apple, Juniper, Cisco and even WatsApp would not exist in their current form. I provide IT architecture services to make your complex IT landscape manageable and I love to solve complex security and privacy challenges. Complex challenges where people, processes and systems are heavily interrelated. For this knowledge intensive work I often run some IT experiments. When you run experiments nowadays you have a choice: Rent some cloud based services or DIY (Do IT Yourself) on premise Running your own developments experiments on your own infrastructure can be time consuming. However smart automation saves time and money. And by creating your own CICD pipeline (Continuous Integration, Continuous Deployment) you stay on top of core infrastructure developments. Even hands-on. Knowing how things work from a technical ‘hands-on’ perspective gives great advantages when it comes to solving complex business IT problems. Making a clear distinguish between a business problem or IT problem is useless. Business and IT problems are related. Sometimes causal related, but more often indirect by one or more non linear feedback loops. Almost every business depends of IT systems. Bad IT means often that your customers will leave your business. One of the things of FeeBSD for me is still FreeBSD Jails. In 2015 I had luck to attend to a presentation of the legendary hacker Poul-Henning Kamp . Check his BSD bio to see what he has done for the FreeBSD community! FreeBSD jails are a light way to visualize your system without enormous overhead. Now that the development on Linux for LXD/LXD is more mature (lxd is the next generation system container manager on linux) there is finally again an alternative for a nice chroot Linux based system again. At least when you do not need the overhead and management complexity that comes with Kubernetes or Docker. FreeBSD means control and quality for me. When there is an open source package I need, I want to install it from source. It gives me more control and always some extra knowledge on how things work. So no precompiled binaries for me on my BSD systems! If a build on FreeBSD fails most of the time this is an alert regarding the quality for me. If a complex OSS package is not available at all in the FreeBSD ports collection there should be a reason for it. Is it really that nobody on the world wants to do this dirty maintenance work? Or is there another cause that running this software on FreeBSD is not possible…There are currently 32644 ports available on FreeBSD. So all the major programming language, databases and middleware libraries are present. The FreeBSD organization is a mature organization and since this is one of the largest OSS projects worldwide learning how this community manages to keep innovation and creates and maintains software is a good entrance for learning how complex IT systems function. FreeBSD is of course BSD licensed. It worked well! There is still a strong community with lots of strong commercial sponsors around the community. Of course: sometimes a GPL license makes more sense. So beside FreeBSD I also love GPL software and the rationale and principles behind it. So my hope is that maybe within the next 25 years the hard battle between BSD vs GPL churches will be more rationalized and normalized. Principles are good, but as all good IT architects know: With good principles alone you never make a good system. So use requirements and not only principles to figure out what OSS license fits your project. There is never one size fits all. June 19, 1993 was the day the official name for FreeBSD was agreed upon. So this blog is written to celebrate 25th anniversary of FreeBSD. ###Dave’s BSDCan trip report So far, only one person has bothered to send in a BSDCan trip report. Our warmest thanks to Dave for doing his part. Hello guys! During the last show, you asked for a trip report regarding BSDCan 2018. This was my first time attending BSDCan. However, BSDCan was my second BSD conference overall, my first being vBSDCon 2017 in Reston, VA. Arriving early Thursday evening and after checking into the hotel, I headed straight to the Red Lion for the registration, picked up my badge and swag and then headed towards the ‘DMS’ building for the newbies talk. The only thing is, I couldn’t find the DMS building! Fortunately I found a BSDCan veteran who was heading there themselves. My only suggestion is to include the full building name and address on the BSDCan web site, or even a link to Google maps to help out with the navigation. The on-campus street maps didn’t have ‘DMS’ written on them anywhere. But I digress. Once I made it to the newbies talk hosted by Dan Langille and Michael W Lucas, it highlighted places to meet, an overview of what is happening, details about the ‘BSDCan widow/widower tours’ and most importantly, the 6-2-1 rule! The following morning, we were present with tea/coffee, muffins and other goodies to help prepare us for the day ahead. The first talk, “The Tragedy of systemd” covered what systemd did wrong and how the BSD community could improve on the ideas behind it. With the exception of Michael W Lucas, SSH Key Management and Kirk McKusick, The Evolution of FreeBSD Governance talk, I pretty much attended all of the ZFS talks including the lunchtime BoF session, hosted by Allan Jude. Coming from FreeNAS and being involved in the community, this is where my main interest and motivation lies. Since then I have been able to share some of that information with the FreeNAS community forums and chatroom. I also attended the “Speculating about Intel” lunchtime BoF session hosted by Theo de Raddt, which proved to be “interesting”. The talks ended with the wrap up session with a few words from Dan, covering the record attendance and made very clear there “was no cabal”. Followed by the the handing over of Groff the BSD goat to a new owner, thank you’s from the FreeBSD Foundation to various community committers and maintainers, finally ending with the charity auction, where a things like a Canadian $20 bill sold for $40, a signed FreeBSD Foundation shirt originally worn by George Neville-Neil, a lost laptop charger, Michael’s used gelato spoon, various books, the last cookie and more importantly, the second to last cookie! After the auction, we all headed to the Red Lion for food and drinks, sponsored by iXsystems. I would like to thank the BSDCan organizers, speakers and sponsors for a great conference. I will certainly hope to attend next year! Regards, Dave (aka m0nkey) Thanks to Dave for sharing his experiences with us and our viewers ##Beastie Bits Robert Watson (from 2008) on how much FreeBSD is in Mac OS X Why Intel Skylake CPUs are sometimes 50% slower than older CPUs Kristaps Dzonsons is looking for somebody to maintain this as mentioned at this link camcontrol(8) saves the day again! Formatting floppy disks in a USB floppy disk drive 32+ great indie games now playable on OpenBSD -current; 7 currently on sale! Warsaw BSD User Group. June 27 2018 18:30-21:00, Wheel Systems Office, Aleje Jerozolimskie 178, Warsaw Tarsnap ##Feedback/Questions Ron - Adding a disk to ZFS Marshall - zfs question Thomas - Allan, the myth perpetuator Ross - ZFS IO stats per dataset Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
DragonflyBSD’s hammer1 encrypted master/slave setup, second part of our BSDCan recap, NomadBSD 1.1-RC1 available, OpenBSD adds an LDAP client to base, FreeBSD gets pNFS support, Intel FPU Speculation Vulnerability confirmed, and what some Unix command names mean. ##Headlines DragonflyBSD: Towards a HAMMER1 master/slave encrypted setup with LUKS I just wanted to share my experience with setting up DragonFly master/slave HAMMER1 PFS’s on top of LUKS So after a long time using an Synology for my NFS needs, I decided it was time to rethink my setup a little since I had several issues with it : You cannot run NFS on top of encrypted partitions easily I suspect I am having some some data corruption (bitrot) on the ext4 filesystem the NIC was stcuk to 100 Mbps instead of 1 Gbps even after swapping cables, switches, you name it It’s proprietary I have been playing with DragonFly in the past and knew about HAMMER, now I just had the perfect excuse to actually use it in production :) After setting up the OS, creating the LUKS partition and HAMMER FS was easy : kdload dm cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox newfs_hammer -L hammer1_secure_master /dev/mapper/fort_knox cryptsetup luksFormat /dev/serno/ cryptsetup luksOpen /dev/serno/ fort_knox_slave newfs_hammer -L hammer1_secure_slave /dev/mapper/fort_knox_slave Mount the 2 drives : mount /dev/mapper/fort_knox /fort_knox mount /dev/mapper_fort_know_slave /fort_knox_slave You can now put your data under /fort_knox Now, off to setting up the replication, first get the shared-uuid of /fort_knox hammer pfs-status /fort_knox Create a PFS slave “linked” to the master hammer pfs-slave /fort_knox_slave/pfs/slave shared-uuid=f9e7cc0d-eb59-10e3-a5b5-01e6e7cefc12 And then stream your data to the slave PFS ! hammer mirror-stream /fort_knox /fort_knox_slave/pfs/slave After that, setting NFS is fairly trivial even though I had problem with the /etc/exports syntax which is different than Linux There’s a few things I wish would be better though but nothing too problematic or without workarounds : Cannot unlock LUKS partitions at boot time afaik (Acceptable tradeoff for the added security LUKS gives me vs my old Synology setup) but this force me to run a script to unlock LUKS, mount hammer and start mirror-stream at each boot No S1/S3 sleep so I made a script to shutdown the system when there’s no network neighborgs to serve the NFS As my system isn’t online 24/7 for energy reasons, I guess will have to run hammer cleanup myself from time to time Some uncertainty because hey, it’s kind of exotic but exciting too :) Overall, I am happy, HAMMER1 and PFS are looking really good, DragonFly is a neat Unix and the community is super friendly (Matthew Dillon actually provided me with a kernel patch to fix the broken ACPI on the PC holding this setup, many thanks!), the system is still a “work in progress” but it is already serving my files as I write this post. Let’s see in 6 months how it goes in the longer run ! Helpful resources : https://www.dragonflybsd.org/docs/how_to_implement_hammer_pseudo_file_system__40___pfs___41___slave_mirroring_from_pfs_master/ ###BSDCan 2018 Recap As promised, here is our second part of our BSDCan report, covering the conference proper. The last tutorials/devsummit of that day lead directly into the conference, as people could pick up their registration packs at the Red Lion and have a drink with fellow BSD folks. Allan and I were there only briefly, as we wanted to get back to the “Newcomers orientation and mentorship” session lead by Michael W. Lucas. This session is intended for people that are new to BSDCan (maybe their first BSD conference ever?) and may have questions. Michael explained everything from the 6-2-1 rule (hours of sleep, meals per day, and number of showers that attendees should have at a minimum), to the partner and widowers program (lead by his wife Liz), to the sessions that people should not miss (opening, closing, and hallway track). Old-time BSDCan folks were asked to stand up so that people can recognize them and ask them any questions they might have during the conferences. The session was well attended. Afterwards, people went for dinner in groups, a big one lead by Michael Lucas to his favorite Shawarma place, followed by gelato (of course). This allowed newbies to mingle over dinner and ice cream, creating a welcoming atmosphere. The next day, after Dan Langille opened the conference, Benno Rice gave the keynote presentation about “The Tragedy of Systemd”. Benedict went to the following talks: “Automating Network Infrastructures with Ansible on FreeBSD” in the DevSummit track. A good talk that connected well with his Ansible tutorial and even allowed some discussions among participants. “All along the dwatch tower”: Devin delivered a well prepared talk. I first thought that the number of slides would not fit into the time slot, but she even managed to give a demo of her work, which was well received. The dwatch tool she wrote should make it easy for people to get started with DTrace without learning too much about the syntax at first. The visualizations were certainly nice to see, combining different tools together in a new way. ZFS BoF, lead by Allan and Matthew Ahrens SSH Key Management by Michael W. Lucas. Yet another great talk where I learned a lot. I did not get to the SSH CA chapter in the new SSH Mastery book, so this was a good way to wet my appetite for it and motivated me to look into creating one for the cluster that I’m managing. The rest of the day was spent at the FreeBSD Foundation table, talking to various folks. Then, Allan and I had an interview with Kirk McKusick for National FreeBSD Day, then we had a core meeting, followed by a core dinner. Day 2: “Flexible Disk Use in OpenZFS”: Matthew Ahrens talking about the feature he is implementing to expand a RAID-Z with a single disk, as well as device removal. Allan’s talk about his efforts to implement ZSTD in OpenZFS as another compression algorithm. I liked his overview slides with the numbers comparing the algorithms for their effectiveness and his personal story about the sometimes rocky road to get the feature implemented. “zrepl - ZFS replication” by Christian Schwarz, was well prepared and even had a demo to show what his snapshot replication tool can do. We covered it on the show before and people can find it under sysutils/zrepl. Feedback and help is welcome. “The Evolution of FreeBSD Governance” by Kirk McKusick was yet another great talk by him covering the early days of FreeBSD until today, detailing some of the progress and challenges the project faced over the years in terms of leadership and governance. This is an ongoing process that everyone in the community should participate in to keep the project healthy and infused with fresh blood. Closing session and auction were funny and great as always. All in all, yet another amazing BSDCan. Thank you Dan Langille and your organizing team for making it happen! Well done. Digital Ocean ###NomadBSD 1.1-RC1 Released The first – and hopefully final – release candidate of NomadBSD 1.1 is available! Changes The base system has been upgraded to FreeBSD 11.2-RC3 EFI booting has been fixed. Support for modern Intel GPUs has been added. Support for installing packages has been added. Improved setup menu. More software packages: benchmarks/bonnie++ DSBDisplaySettings DSBExec DSBSu mail/thunderbird net/mosh ports-mgmt/octopkg print/qpdfview security/nmap sysutils/ddrescue sysutils/fusefs-hfsfuse sysutils/fusefs-sshfs sysutils/sleuthkit www/lynx x11-wm/compton x11/xev x11/xterm Many improvements and bugfixes The image and instructions can be found here. ##News Roundup LDAP client added to -current CVSROOT: /cvs Module name: src Changes by: reyk@cvs.openbsd.org 2018/06/13 09:45:58 Log message: Import ldap(1), a simple ldap search client. We have an ldapd(8) server and ypldap in base, so it makes sense to have a simple LDAP client without depending on the OpenLDAP package. This tool can be used in an ssh(1) AuthorizedKeysCommand script. With feedback from many including millert@ schwarze@ gilles@ dlg@ jsing@ OK deraadt@ Status: Vendor Tag: reyk Release Tags: ldap_20180613 N src/usr.bin/ldap/Makefile N src/usr.bin/ldap/aldap.c N src/usr.bin/ldap/aldap.h N src/usr.bin/ldap/ber.c N src/usr.bin/ldap/ber.h N src/usr.bin/ldap/ldap.1 N src/usr.bin/ldap/ldapclient.c N src/usr.bin/ldap/log.c N src/usr.bin/ldap/log.h No conflicts created by this import ###Intel® FPU Speculation Vulnerability Confirmed Earlier this month, Philip Guenther (guenther@) committed (to amd64 -current) a change from lazy to semi-eager FPU switching to mitigate against rumored FPU state leakage in Intel® CPUs. Theo de Raadt (deraadt@) discussed this in his BSDCan 2018 session. Using information disclosed in Theo’s talk, Colin Percival developed a proof-of-concept exploit in around 5 hours. This seems to have prompted an early end to an embargo (in which OpenBSD was not involved), and the official announcement of the vulnerability. FPU change in FreeBSD Summary: System software may utilize the Lazy FP state restore technique to delay the restoring of state until an instruction operating on that state is actually executed by the new process. Systems using Intel® Core-based microprocessors may potentially allow a local process to infer data utilizing Lazy FP state restore from another process through a speculative execution side channel. Description: System software may opt to utilize Lazy FP state restore instead of eager save and restore of the state upon a context switch. Lazy restored states are potentially vulnerable to exploits where one process may infer register values of other processes through a speculative execution side channel that infers their value. · CVSS - 4.3 Medium CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:L/I:N/A:N Affected Products: Intel® Core-based microprocessors. Recommendations: If an XSAVE-enabled feature is disabled, then we recommend either its state component bitmap in the extended control register (XCR0) is set to 0 (e.g. XCR0[bit 2]=0 for AVX, XCR0[bits 7:5]=0 for AVX512) or the corresponding register states of the feature should be cleared prior to being disabled. Also for relevant states (e.g. x87, SSE, AVX, etc.), Intel recommends system software developers utilize Eager FP state restore in lieu of Lazy FP state restore. Acknowledgements: Intel would like to thank Julian Stecklina from Amazon Germany, Thomas Prescher from Cyberus Technology GmbH (https://www.cyberus-technology.de/), Zdenek Sojka from SYSGO AG (http://sysgo.com), and Colin Percival for reporting this issue and working with us on coordinated disclosure. iXsystems iX Ad Spot iX Systems - BSDCan 2018 Recap ###FreeBSD gets pNFS support Merge the pNFS server code from projects/pnfs-planb-server into head. This code merge adds a pNFS service to the NFSv4.1 server. Although it is a large commit it should not affect behaviour for a non-pNFS NFS server. Some documentation on how this works can be found at: Merge the pN http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt and will hopefully be turned into a proper document soon. This is a merge of the kernel code. Userland and man page changes will come soon, once the dust settles on this merge. It has passed a "make universe", so I hope it will not cause build problems. It also adds NFSv4.1 server support for the "current stateid". Here is a brief overview of the pNFS service: A pNFS service separates the Read/Write operations from all the other NFSv4.1 Metadata operations. It is hoped that this separation allows a pNFS service to be configured that exceeds the limits of a single NFS server for either storage capacity and/or I/O bandwidth. It is possible to configure mirroring within the data servers (DSs) so that the data storage file for an MDS file will be mirrored on two or more of the DSs. When this is used, failure of a DS will not stop the pNFS service and a failed DS can be recovered once repaired while the pNFS service continues to operate. Although two way mirroring would be the norm, it is possible to set a mirroring level of up to four or the number of DSs, whichever is less. The Metadata server will always be a single point of failure, just as a single NFS server is. A Plan B pNFS service consists of a single MetaData Server (MDS) and K Data Servers (DS), all of which are recent FreeBSD systems. Clients will mount the MDS as they would a single NFS server. When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size 0. Each of these files will also have two extended attributes in the system attribute name space: pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file(s) on DS(s) for this file. pnfsd.dsattr - This extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file, so that the MDS doesn't need to acquire the attributes from the DS for every Getattr operation. For each regular (VREG) file, the MDS creates a data storage file on one (or more if mirroring is enabled) of the DSs in one of the "dsNN" subdirectories. The name of this file is the file handle of the file on the MDS in hexadecimal so that the name is unique. The DSs use subdirectories named "ds0" to "dsN" so that no one directory gets too large. The value of "N" is set via the sysctl vfs.nfsd.dsdirsize on the MDS, with the default being 20. For production servers that will store a lot of files, this value should probably be much larger. It can be increased when the "nfsd" daemon is not running on the MDS, once the "dsK" directories are created. For pNFS aware NFSv4.1 clients, the FreeBSD server will return two pieces of information to the client that allows it to do I/O directly to the DS. DeviceInfo - This is relatively static information that defines what a DS is. The critical bits of information returned by the FreeBSD server is the IP address of the DS and, for the Flexible File layout, that NFSv4.1 is to be used and that it is "tightly coupled". There is a "deviceid" which identifies the DeviceInfo. Layout - This is per file and can be recalled by the server when it is no longer valid. For the FreeBSD server, there is support for two types of layout, call File and Flexible File layout. Both allow the client to do I/O on the DS via NFSv4.1 I/O operations. The Flexible File layout is a more recent variant that allows specification of mirrors, where the client is expected to do writes to all mirrors to maintain them in a consistent state. The Flexible File layout also allows the client to report I/O errors for a DS back to the MDS. The Flexible File layout supports two variants referred to as "tightly coupled" vs "loosely coupled". The FreeBSD server always uses the "tightly coupled" variant where the client uses the same credentials to do I/O on the DS as it would on the MDS. For the "loosely coupled" variant, the layout specifies a synthetic user/group that the client uses to do I/O on the DS. The FreeBSD server does not do striping and always returns layouts for the entire file. The critical information in a layout is Read vs Read/Writea and DeviceID(s) that identify which DS(s) the data is stored on. At this time, the MDS generates File Layout layouts to NFSv4.1 clients that know how to do pNFS for the non-mirrored DS case unless the sysctl vfs.nfsd.default_flexfile is set non-zero, in which case Flexible File layouts are generated. The mirrored DS configuration always generates Flexible File layouts. For NFS clients that do not support NFSv4.1 pNFS, all I/O operations are done against the MDS which acts as a proxy for the appropriate DS(s). When the MDS receives an I/O RPC, it will do the RPC on the DS as a proxy. If the DS is on the same machine, the MDS/DS will do the RPC on the DS as a proxy and so on, until the machine runs out of some resource, such as session slots or mbufs. As such, DSs must be separate systems from the MDS. *** ###[What does {some strange unix command name} stand for?](http://www.unixguide.net/unix/faq/1.3.shtml) + awk = "Aho Weinberger and Kernighan" + grep = "Global Regular Expression Print" + fgrep = "Fixed GREP". + egrep = "Extended GREP" + cat = "CATenate" + gecos = "General Electric Comprehensive Operating Supervisor" + nroff = "New ROFF" + troff = "Typesetter new ROFF" + tee = T + bss = "Block Started by Symbol + biff = "BIFF" + rc (as in ".cshrc" or "/etc/rc") = "RunCom" + Don Libes' book "Life with Unix" contains lots more of these tidbits. *** ##Beastie Bits + [RetroBSD: Unix for microcontrollers](http://retrobsd.org/wiki/doku.php) + [On the matter of OpenBSD breaking embargos (KRACK)](https://marc.info/?l=openbsd-tech&m=152910536208954&w=2) + [Theo's Basement Computer Paradise (1998)](https://zeus.theos.com/deraadt/hosts.html) + [Airport Extreme runs NetBSD](https://jcs.org/2018/06/12/airport_ssh) + [What UNIX shell could have been](https://rain-1.github.io/shell-2.html) *** Tarsnap ad *** ##Feedback/Questions + We need more feedback and questions. Please email feedback@bsdnow.tv + Also, many of you owe us BSDCan trip reports! We have shared what our experience at BSDCan was like, but we want to hear about yours. What can we do better next year? What was it like being there for the first time? + [Jason writes in](https://slexy.org/view/s205jU58X2) + https://www.wheelsystems.com/en/products/wheel-fudo-psm/ + [June 19th was National FreeBSD Day](https://twitter.com/search?src=typd&q=%23FreeBSDDay) *** - Send questions, comments, show ideas/topics, or stories you want mentioned on the show to [feedback@bsdnow.tv](mailto:feedback@bsdnow.tv) ***
OpenZFS and DTrace updates in NetBSD, NetBSD network security stack audit, Performance of MySQL on ZFS, OpenSMTP results from p2k18, legacy Windows backup to FreeNAS, ZFS block size importance, and NetBSD as router on a stick. ##Headlines ZFS and DTrace update lands in NetBSD merge a new version of the CDDL dtrace and ZFS code. This changes the upstream vendor from OpenSolaris to FreeBSD, and this version is based on FreeBSD svn r315983. r315983 is from March 2017 (14 months ago), so there is still more work to do in addition to the 10 years of improvements from upstream, this version also has these NetBSD-specific enhancements: dtrace FBT probes can now be placed in kernel modules. ZFS now supports mmap(). This brings NetBSD 10 years forward, and they should be able to catch the rest of the way up fairly quickly ###NetBSD network stack security audit Maxime Villard has been working on an audit of the NetBSD network stack, a project sponsored by The NetBSD Foundation, which has served all users of BSD-derived operating systems. Over the last five months, hundreds of patches were committed to the source tree as a result of this work. Dozens of bugs were fixed, among which a good number of actual, remotely-triggerable vulnerabilities. Changes were made to strengthen the networking subsystems and improve code quality: reinforce the mbuf API, add many KASSERTs to enforce assumptions, simplify packet handling, and verify compliance with RFCs. This was done in several layers of the NetBSD kernel, from device drivers to L4 handlers. In the course of investigating several bugs discovered in NetBSD, I happened to look at the network stacks of other operating systems, to see whether they had already fixed the issues, and if so how. Needless to say, I found bugs there too. A lot of code is shared between the BSDs, so it is especially helpful when one finds a bug, to check the other BSDs and share the fix. The IPv6 Buffer Overflow: The overflow allowed an attacker to write one byte of packet-controlled data into ‘packetstorage+off’, where ‘off’ could be approximately controlled too. This allowed at least a pretty bad remote DoS/Crash The IPsec Infinite Loop: When receiving an IPv6-AH packet, the IPsec entry point was not correctly computing the length of the IPv6 suboptions, and this, before authentication. As a result, a specially-crafted IPv6 packet could trigger an infinite loop in the kernel (making it unresponsive). In addition this flaw allowed a limited buffer overflow - where the data being written was however not controllable by the attacker. The IPPROTO Typo: While looking at the IPv6 Multicast code, I stumbled across a pretty simple yet pretty bad mistake: at one point the Pim6 entry point would return IPPROTONONE instead of IPPROTODONE. Returning IPPROTONONE was entirely wrong: it caused the kernel to keep iterating on the IPv6 packet chain, while the packet storage was already freed. The PF Signedness Bug: A bug was found in NetBSD’s implementation of the PF firewall, that did not affect the other BSDs. In the initial PF code a particular macro was used as an alias to a number. This macro formed a signed integer. NetBSD replaced the macro with a sizeof(), which returns an unsigned result. The NPF Integer Overflow: An integer overflow could be triggered in NPF, when parsing an IPv6 packet with large options. This could cause NPF to look for the L4 payload at the wrong offset within the packet, and it allowed an attacker to bypass any L4 filtering rule on IPv6. The IPsec Fragment Attack: I noticed some time ago that when reassembling fragments (in either IPv4 or IPv6), the kernel was not removing the MPKTHDR flag on the secondary mbufs in mbuf chains. This flag is supposed to indicate that a given mbuf is the head of the chain it forms; having the flag on secondary mbufs was suspicious. What Now: Not all protocols and layers of the network stack were verified, because of time constraints, and also because of unexpected events: the recent x86 CPU bugs, which I was the only one able to fix promptly. A todo list will be left when the project end date is reached, for someone else to pick up. Me perhaps, later this year? We’ll see. This security audit of NetBSD’s network stack is sponsored by The NetBSD Foundation, and serves all users of BSD-derived operating systems. The NetBSD Foundation is a non-profit organization, and welcomes any donations that help continue funding projects of this kind. DigitalOcean ###MySQL on ZFS Performance I used sysbench to create a table of 10M rows and then, using export/import tablespace, I copied it 329 times. I ended up with 330 tables for a total size of about 850GB. The dataset generated by sysbench is not very compressible, so I used lz4 compression in ZFS. For the other ZFS settings, I used what can be found in my earlier ZFS posts but with the ARC size limited to 1GB. I then used that plain configuration for the first benchmarks. Here are the results with the sysbench point-select benchmark, a uniform distribution and eight threads. The InnoDB buffer pool was set to 2.5GB. In both cases, the load is IO bound. The disk is doing exactly the allowed 3000 IOPS. The above graph appears to be a clear demonstration that XFS is much faster than ZFS, right? But is that really the case? The way the dataset has been created is extremely favorable to XFS since there is absolutely no file fragmentation. Once you have all the files opened, a read IOP is just a single fseek call to an offset and ZFS doesn’t need to access any intermediate inode. The above result is about as fair as saying MyISAM is faster than InnoDB based only on table scan performance results of unfragmented tables and default configuration. ZFS is much less affected by the file level fragmentation, especially for point access type. ZFS stores the files in B-trees in a very similar fashion as InnoDB stores data. To access a piece of data in a B-tree, you need to access the top level page (often called root node) and then one block per level down to a leaf-node containing the data. With no cache, to read something from a three levels B-tree thus requires 3 IOPS. The extra IOPS performed by ZFS are needed to access those internal blocks in the B-trees of the files. These internal blocks are labeled as metadata. Essentially, in the above benchmark, the ARC is too small to contain all the internal blocks of the table files’ B-trees. If we continue the comparison with InnoDB, it would be like running with a buffer pool too small to contain the non-leaf pages. The test dataset I used has about 600MB of non-leaf pages, about 0.1% of the total size, which was well cached by the 3GB buffer pool. So only one InnoDB page, a leaf page, needed to be read per point-select statement. To correctly set the ARC size to cache the metadata, you have two choices. First, you can guess values for the ARC size and experiment. Second, you can try to evaluate it by looking at the ZFS internal data. Let’s review these two approaches. You’ll read/hear often the ratio 1GB of ARC for 1TB of data, which is about the same 0.1% ratio as for InnoDB. I wrote about that ratio a few times, having nothing better to propose. Actually, I found it depends a lot on the recordsize used. The 0.1% ratio implies a ZFS recordsize of 128KB. A ZFS filesystem with a recordsize of 128KB will use much less metadata than another one using a recordsize of 16KB because it has 8x fewer leaf pages. Fewer leaf pages require less B-tree internal nodes, hence less metadata. A filesystem with a recordsize of 128KB is excellent for sequential access as it maximizes compression and reduces the IOPS but it is poor for small random access operations like the ones MySQL/InnoDB does. In order to improve ZFS performance, I had 3 options: Increase the ARC size to 7GB Use a larger Innodb page size like 64KB Add a L2ARC I was reluctant to grow the ARC to 7GB, which was nearly half the overall system memory. At best, the ZFS performance would only match XFS. A larger InnoDB page size would increase the CPU load for decompression on an instance with only two vCPUs; not great either. The last option, the L2ARC, was the most promising. ZFS is much more complex than XFS and EXT4 but, that also means it has more tunables/options. I used a simplistic setup and an unfair benchmark which initially led to poor ZFS results. With the same benchmark, very favorable to XFS, I added a ZFS L2ARC and that completely reversed the situation, more than tripling the ZFS results, now 66% above XFS. Conclusion We have seen in this post why the general perception is that ZFS under-performs compared to XFS or EXT4. The presence of B-trees for the files has a big impact on the amount of metadata ZFS needs to handle, especially when the recordsize is small. The metadata consists mostly of the non-leaf pages (or internal nodes) of the B-trees. When properly cached, the performance of ZFS is excellent. ZFS allows you to optimize the use of EBS volumes, both in term of IOPS and size when the instance has fast ephemeral storage devices. Using the ephemeral device of an i3.large instance for the ZFS L2ARC, ZFS outperformed XFS by 66%. ###OpenSMTPD new config TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentioned at EuroBSDCon the one-line per rule config format was a design error. A new configuration grammar is almost ready and the underlying structures are simplified. Refactor removes ~750 lines of code and solves _many issues that were side-effects of the design error. New features are going to be unlocked thanks to this. Anatomy of a design error OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time. The initial configuration format was very different, I was inspired by pyr@’s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets. When I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed. It helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later. That being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong. One-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals. To get to the point: we should move to two-line rules :-) Anatomy of a design error OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time. The initial configuration format was very different, I was inspired by pyr@’s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets. When I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed. It helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later. That being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong. One-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals. To get to the point: we should move to two-line rules :-) The problem with one-line rules OpenSMTPD decides to accept or reject messages based on one-line rules such as: accept from any for domain poolp.org deliver to mbox Which can essentially be split into three units: the decision: accept/reject the matching: from any for domain poolp.org the (default) action: deliver to mbox To ensure that we meet the requirements of the transactions, the matching must be performed during the SMTP transaction before we take a decision for the recipient. Given that the rule is atomic, that it doesn’t have an identifier and that the action is part of it, the two only ways to make sure we can remember the action to take later on at delivery time is to either: save the action in the envelope, which is what we do today evaluate the envelope again at delivery And this this where it gets tricky… both solutions are NOT ok. The first solution, which we’ve been using for a decade, was to save the action within the envelope and kind of carve it in stone. This works fine… however it comes with the downsides that errors fixed in configuration files can’t be caught up by envelopes, that delivery action must be validated way ahead of time during the SMTP transaction which is much trickier, that the parsing of delivery methods takes place as the _smtpd user rather than the recipient user, and that envelope structures that are passed all over OpenSMTPD carry delivery-time informations, and more, and more, and more. The code becomes more complex in general, less safe in some particular places, and some areas are nightmarish to deal with because they have to deal with completely unrelated code that can’t be dealt with later in the code path. The second solution can’t be done. An envelope may be the result of nested rules, for example an external client, hitting an alias, hitting a user with a .forward file resolving to a user. An envelope on disk may no longer match any rule or it may match a completely different rule If we could ensure that it matched the same rule, evaluating the ruleset may spawn new envelopes which would violate the transaction. Trying to imagine how we could work around this leads to more and more and more RFC violations, incoherent states, duplicate mails, etc… There is simply no way to deal with this with atomic rules, the matching and the action must be two separate units that are evaluated at two different times, failure to do so will necessarily imply that you’re either using our first solution and all its downsides, or that you are currently in a world of pain trying to figure out why everything is burning around you. The minute the action is written to an on-disk envelope, you have failed. A proper ruleset must define a set of matching patterns resolving to an action identifier that is carved in stone, AND a set of named action set that is resolved dynamically at delivery time. Follow the link above to see the rest of the article Break ##News Roundup Backing up a legacy Windows machine to a FreeNAS with rsync I have some old Windows servers (10 years and counting) and I have been using rsync to back them up to my FreeNAS box. It has been working great for me. First of all, I do have my Windows servers backup in virtualized format. However, those are only one-time snapshops that I run once in a while. These are classic ASP IIS web servers that I can easily put up on a new VM. However, many of these legacy servers generate gigabytes of data a day in their repositories. Running VM conversion daily is not ideal. My solution was to use some sort of rsync solution just for the data repos. I’ve tried some applications that didn’t work too well with Samba shares and these old servers have slow I/O. Copying files to external sata or usb drive was not ideal. We’ve moved on from Windows to Linux and do not have any Windows file servers of capacity to provide network backups. Hence, I decided to use Delta Copy with FreeNAS. So here is a little write up on how to set it up. I have 4 Windows 2000 servers backing up daily with this method. First, download Delta Copy and install it. It is open-source and pretty much free. It is basically a wrapper for cygwin’s rsync. When you install it, it will ask you to install the Server services which allows you to run it as a Rsync server on Windows. You don’t need to do this. Instead, you will be just using the Delta Copy Client application. But before we do that, we will need to configure our Rsync service for our Windows Clients on FreeNAS. In FreeNAS, go under Services , Select Rsync > Rsync Modules > Add Rsync Module. Then fill out the form; giving the module a name and set the path. In my example, I simply called it WIN and linked it to a user called backupuser. This process is much easier than trying to configure the daemon rsyncd.conf file by hand. Now, on the Windows Client, start the DeltaCopy Client. You will create a new Profile. You will need to enter the IP of the Rsync server (FreeNAS) and specify the module name which will be called “Virtual Directory Name.” When you pull the select menu, the list of Rsync Modules you created earlier in FreeNAS will populate. You can set authentication. On the server, you can restrict by IP and do other things to lock down your rsync. Next, you will add folders (and/or files) you want to synchronize. Once the paths are set up, you can run a sync by right clicking the profile name. Here, I made a test sync to a home folder of a virtualized windows box. As you can see, I mounted the rsync volume on my mac to see the progress. The rsync worked beautifully. DeltaCopy did what it was told. Once you get everything working. The next thing to do is set schedules. If you done tasks schedules in Windows before, it is pretty straightforward. DeltaCopy has a link in the application to directly create a new task for you. I set my backups to run nightly and it has been working great. There you have it. Windows rsync to FreeNAS using DeltaCopy. The nice thing about FreeNAS is you don’t have to modify /etc/rsyncd.conf files. Everything can be done in the web admin. iXsystems ###How to write ATF tests for NetBSD I have recently started contributing to the amazing NetBSD foundation. I was thinking of trying out a new OS for a long time. Switching to the NetBSD OS has been a fun change. My first contribution to the NetBSD foundation was adding regression tests for the Address Sanitizer (ASan) in the Automated Testing Framework(ATF) which NetBSD has. I managed to complete it with the help of my really amazing mentor Kamil. This post is gonna be about the ATF framework that NetBSD has and how to you can add multiple tests with ease. Intro In ATF tests we will basically be talking about test programs which are a suite of test cases for a specific application or program. The ATF suite of Commands There are a variety of commands that the atf suite offers. These include : atf-check: The versatile command that is a vital part of the checking process. man page atf-run: Command used to run a test program. man page atf-fail: Report failure of a test case. atf-report: used to pretty print the atf-run. man page atf-set: To set atf test conditions. We will be taking a better look at the syntax and usage later. Let’s start with the Basics The ATF testing framework comes preinstalled with a default NetBSD installation. It is used to write tests for various applications and commands in NetBSD. One can write the Test programs in either the C language or in shell script. In this post I will be dealing with the Bash part. Follow the link above to see the rest of the article ###The Importance of ZFS Block Size Warning! WARNING! Don’t just do things because some random blog says so One of the important tunables in ZFS is the recordsize (for normal datasets) and volblocksize (for zvols). These default to 128KB and 8KB respectively. As I understand it, this is the unit of work in ZFS. If you modify one byte in a large file with the default 128KB record size, it causes the whole 128KB to be read in, one byte to be changed, and a new 128KB block to be written out. As a result, the official recommendation is to use a block size which aligns with the underlying workload: so for example if you are using a database which reads and writes 16KB chunks then you should use a 16KB block size, and if you are running VMs containing an ext4 filesystem, which uses a 4KB block size, you should set a 4KB block size You can see it has a 16GB total file size, of which 8.5G has been touched and consumes space - that is, it’s a “sparse” file. The used space is also visible by looking at the zfs filesystem which this file resides in Then I tried to copy the image file whilst maintaining its “sparseness”, that is, only touching the blocks of the zvol which needed to be touched. The original used only 8.42G, but the copy uses 14.6GB - almost the entire 16GB has been touched! What’s gone wrong? I finally realised that the difference between the zfs filesystem and the zvol is the block size. I recreated the zvol with a 128K block size That’s better. The disk usage of the zvol is now exactly the same as for the sparse file in the filesystem dataset It does impact the read speed too. 4K blocks took 5:52, and 128K blocks took 3:20 Part of this is the amount of metadata that has to be read, see the MySQL benchmarks from earlier in the show And yes, using a larger block size will increase the compression efficiency, since the compressor has more redundant data to optimize. Some of the savings, and the speedup is because a lot less metadata had to be written Your zpool layout also plays a big role, if you use 4Kn disks, and RAID-Z2, using a volblocksize of 8k will actually result in a large amount of wasted space because of RAID-Z padding. Although, if you enable compression, your 8k records may compress to only 4k, and then all the numbers change again. ###Using a Raspberry Pi 2 as a Router on a Stick Starring NetBSD Sorry we didn’t answer you quickly enough A few weeks ago I set about upgrading my feeble networking skills by playing around with a Cisco 2970 switch. I set up a couple of VLANs and found the urge to set up a router to route between them. The 2970 isn’t a modern layer 3 switch so what am I to do? Why not make use of the Raspberry Pi 2 that I’ve never used and put it to some good use as a ‘router on a stick’. I could install a Linux based OS as I am quite familiar with it but where’s the fun in that? In my home lab I use SmartOS which by the way is a shit hot hypervisor but as far as I know there aren’t any Illumos distributions for the Raspberry Pi. On the desktop I use Solus OS which is by far the slickest Linux based OS that I’ve had the pleasure to use but Solus’ focus is purely desktop. It’s looking like BSD then! I believe FreeBSD is renowned for it’s top notch networking stack and so I wrote to the BSDNow show on Jupiter Broadcasting for some help but it seems that the FreeBSD chaps from the show are off on a jolly to some BSD conference or another(love the show by the way). It looks like me and the luvverly NetBSD are on a date this Saturday. I’ve always had a secret love for NetBSD. She’s a beautiful, charming and promiscuous lover(looking at the supported architectures) and I just can’t stop going back to her despite her misgivings(ahem, zfs). Just my type of grrrl! Let’s crack on… Follow the link above to see the rest of the article ##Beastie Bits BSD Jobs University of Aberdeen’s Internet Transport Research Group is hiring VR demo on OpenBSD via OpenHMD with OSVR HDK2 patch runs ed, and ed can run anything (mentions FreeBSD and OpenBSD) Alacritty (OpenGL-powered terminal emulator) now supports OpenBSD MAP_STACK Stack Register Checking Committed to -current EuroBSDCon CfP till June 17, 2018 Tarsnap ##Feedback/Questions NeutronDaemon - Tutorial request Kurt - Question about transferability/bi-directionality of ZFS snapshots and send/receive Peter - A Question and much love for BSD Now Peter - netgraph state Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
OpenZFS and DTrace updates in NetBSD, NetBSD network security stack audit, Performance of MySQL on ZFS, OpenSMTP results from p2k18, legacy Windows backup to FreeNAS, ZFS block size importance, and NetBSD as router on a stick.
OpenZFS and DTrace updates in NetBSD, NetBSD network security stack audit, Performance of MySQL on ZFS, OpenSMTP results from p2k18, legacy Windows backup to FreeNAS, ZFS block size importance, and NetBSD as router on a stick.
Dropbox made $1.1bn last year, which is mind-blowing. What can we learn from the way Dropbox wiggled it’s way into so many people’s lives (11m paying users, it seems) versus competitors like Box? Well, probably a lot more than where Apple, Spotify, and Dropbox run their stuff in - or out! - of the cloud, a topic we also discuss. Also, sheep-skin shoes are hot, too hot. Also, something about dtrace and zfs, I don’t know - just listen to it. This episode brought to you by: Datadog! This episode is sponsored by Datadog, a monitoring platform for cloud-scale infrastructure and applications. Built by engineers, for engineers, Datadog provides visibility into more than 200 technologies, including AWS, Chef, and Docker, with built-in metric dashboards and automated alerts. With end-to-end request tracing, Datadog provides visibility into your applications and their underlying infrastructure—all in one place. Sign up for a free trial (https://www.datadoghq.com/lpgs/?utm_source=Advertisement&utm_medium=Advertisement&utm_campaign=SoftwareDefinedTalk-Tshirt-Native). Datadog also offers Forecast Alerts (https://www.datadoghq.com/blog/forecasts-datadog/), which makes it easy to get notified of potential problems before they cause outages. Read more at: https://www.datadoghq.com/blog/forecasts-datadog/ (https://www.datadoghq.com/blog/forecasts-datadog/) What even is a “Dropbox”? Now that we know they generated $1.1bn in revenue (http://tomtunguz.com/dropbox-s-1/) in CY2017 (with -10% op margins, translating to a loss of $111.7m (https://www.barrons.com/articles/dropbox-files-s1-with-1b-revenue-lots-of-restricted-stock-1519420407) and actual cash flow)…we should probably contemplate how they fit in. 451 estimates a valuation at $8bn+ (https://451research.com/report-short?entityId=94433&type=mis&alertid=1161&contactid=0033200001wgKCKAA2&utm_source=sendgrid&utm_medium=email&utm_campaign=market-insight&utm_content=newsletter&utm_term=94433-Dropbox+opens+up+to+public+markets). More: “Dropbox has taken just over 10 years to go public since its founding in 2007, which we attribute to anxiety over its high private valuation, a sizable profitability gap, and the dour outlook often associated with the EFSS segment.” More from 451 (https://451research.com/report-short?entityId=94433&type=mis&alertid=1161&contactid=0033200001wgKCKAA2&utm_source=sendgrid&utm_medium=email&utm_campaign=market-insight&utm_content=newsletter&utm_term=94433-Dropbox+opens+up+to+public+markets): “Its net loss ($111m) shrank by nearly half from 2016 – a faster pace than its topline growth. Its relative sales and marketing costs are lower than most of its peers. The vendor spent 28% of its revenue on sales and marketing – half the level of Box, a fellow FSS compatriot that's half the size as Dropbox." Man, think of the shit-per diem an travel policies for last year. E.g., who knew they were so widely used by normals?! 11m+ paying users, they says (https://www.barrons.com/articles/dropbox-files-s1-with-1b-revenue-lots-of-restricted-stock-1519420407). (But (https://451research.com/report-short?entityId=94433&type=mis&alertid=1161&contactid=0033200001wgKCKAA2&utm_source=sendgrid&utm_medium=email&utm_campaign=market-insight&utm_content=newsletter&utm_term=94433-Dropbox+opens+up+to+public+markets), it’s 45 to 1 free to pay.) Do we think GDrive/G Suite is this big? I mean, it must be at least once you throw in Docs and GMail. Gartner’s 2016 estimates (https://www.reuters.com/article/us-alphabet-gsuite/googles-g-suite-is-no-microsoft-killer-but-still-winning-converts-idUSKBN1FL3ZX): “$1.3 billion in G Suite sales ranked a distant No. 2 behind Office’s $13.8 billion, according to 2016 data from Gartner.” Checks out (https://medium.com/@raj_57679/how-big-is-g-suite-within-alphabets-other-revenue-q1-2017-9823783abc97). Tech Co.’s using Google Cloud Apple on Google (https://techcrunch.com/2018/02/27/apple-now-relies-on-google-cloud-platform-and-amazon-s3-for-icloud-data/) for iCloud (but also AWS for the same), Spotify on Google (http://www.zdnet.com/article/with-spotify-google-cloud-platform-gets-its-anchor-all-in-customer/), Dropbox on their own cloud (https://www.geekwire.com/2018/dropbox-saved-almost-75-million-two-years-building-tech-infrastructure/) (see Ben’s “turns out!” analysis (https://stratechery.com/2018/dropboxs-cost-of-revenue-cost-of-revenue-and-churn-cloudy-dropbox/)). Does this matter for normals? “Dropbox is likely an outlier with its successful cloud data migration off AWS.” (http://searchaws.techtarget.com/blog/AWS-Cloud-Cover/Dropbox-is-likely-an-outlier-with-its-successful-cloud-data-migration-off-AWS) Wired’s write-up on the migration (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/) from 2016 Relevant to your interests Pants are sized wrong (https://www.esquire.com/style/mens-fashion/a8386/pants-size-chart-090710/). DTrace going GPL-compatible (https://www.theregister.co.uk/2018/02/19/oracle_open_sources_dtrace_changes_licence_to_gpl/) Arrested DevOps talking with Andrew Shafer and Bryan Cantrill (https://www.arresteddevops.com/yelling-at-cloud/) The ongoing nothingburger of blockchain-beyond-bitcoin (https://www.geekwire.com/2018/microsofts-guthrie-claims-azure-seeing-blockchain-momentum-thats-not-moving-cloud-needle-time-soon/): “The only mass-market use of blockchain technology right now is bitcoin, and you can certainly debate just how widespread a market that really is. Lots of people are interested in blockchain’s distributed ledger system as a potential way to cut out the middleman in transactions between manufacturers or retailers and their suppliers, but the number of people actually using blockchain technology for those types of services right now is quite small.” More: “The entire market for blockchain services in 2017 — and not necessarily cloud vendor-provided blockchain services — sits at $708 million, according to a report from WinterGreen Research cited by The Information. By comparison, Gartner said last September that it expects cloud services revenue will have reached $260.2 billion in 2017.” WinterGreen, sittin’ in hot tubs, smokin’ those L’s: “In that report, WinterGreen also predicts astounding growth of 757 percent in that blockchain market by 2024 to $60.7 billion, which is among the most dramatic forward-looking statements I’ve seen in a while.” The concept of the millennial is dead (http://www.channelfutures.com/channel-futures/why-are-we-still-talking-about-millennial-problem-workforce). Time to start complaining and belly-aching about how simpering and fucked up the current generation of The Kids are. Conferences, et. al. March 9th to 13th, SXSW (https://www.sxsw.com/) - Brandon in Austin giving out stickers. Coté needs excuses to expense meals and drinks. March 22-23 - DevOps Talks Conference (http://www.devopstalks.com/), Melbourne Matt speaking April 26-27, DevOpsDays Jakarta (http://devopsdays.org/events/2018-jakarta/) - Matt (https://twitter.com/agilecircleindo/status/969511498287493120) is keynoting (https://twitter.com/agilecircleindo/status/969511498287493120), and Coté will be speaking too (https://twitter.com/agilecircleindo/status/969511498287493120). May 15th to 18th, 2018 - Coté talking EA at Continuous Lifecycle London (https://continuouslifecycle.london/sessions/the-death-of-enterprise-architecture-defeating-the-devops-microservices-and-cloud-native-assassins/). May 22-25, ChefConf 2018 (https://chefconf.chef.io/), in Chicago. SDT news & hype Check out Software Defined Interviews (http://www.softwaredefinedinterviews.com/), our new podcast. Pretty self-descriptive, plus the #exegesis podcast we’ve been doing, all in one, for free. Keep up with the weekly newsletter (https://us1.campaign-archive.com/home/?u=ce6149b4008d62a08093a4fa6&id=5877922e21). Join us in Slack (http://www.softwaredefinedtalk.com/slack). Buy some t-shirts (https://fsgprints.myshopify.com/collections/software-defined-talk)! DISCOUNT CODE: SDTFSG (20% off) Send your name and address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you a sticker. Recommendations Matt: Bose QuietComfort 20 headphones (https://www.amazon.com/dp/B00X9KVVQK/) Brandon: Version Control (https://www.audible.com/pd/Sci-Fi-Fantasy/Version-Control-Audiobook/B01BKY8A8I). Coté: Starbuck’s Blonde roast; low-sodium Kirkland bacon - turns out! - no sugar.
We explain the physics behind ZFS, DTrace switching to the GPL, Emacs debugging, syncookies coming to PF & FreeBSD's history on EC2. This episode was brought to you by Headlines 128 bit storage: Are you high? (https://blogs.oracle.com/bonwick/128-bit-storage:-are-you-high) For people who have heard about ZFS boiling oceans and wonder where that is coming from, we dug out this old piece from 2004 on the blog of ZFS co-creator Jeff Bonwick, originally from the Sun website. 64 bits would have been plenty ... but then you can't talk out of your ass about boiling oceans then, can you? Well, it's a fair question. Why did we make ZFS a 128-bit storage system? What on earth made us think it's necessary? And how do we know it's sufficient? Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 2^64 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. If 64 bits isn't enough, the next logical step is 128 bits. That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. But it does raise the question: what are the theoretical limits to storage capacity? Although we'd all like Moore's Law to continue forever, quantum mechanics imposes some fundamental limits on the computation rate and information capacity of any physical device. In particular, it has been shown that 1 kilogram of matter confined to 1 liter of space can perform at most 10^51 operations per second on at most 10^31 bits of information [see Seth Lloyd, "Ultimate physical limits to computation." Nature 406, 1047-1054 (2000)]. A fully-populated 128-bit storage pool would contain 2^128 blocks = 2^137 bytes = 2^140 bits; therefore the minimum mass required to hold the bits would be (2^140 bits) / (10^31 bits/kg) = 136 billion kg. That's a lot of gear. To operate at the 1031 bits/kg limit, however, the entire mass of the computer must be in the form of pure energy. By E=mc^2, the rest energy of 136 billion kg is 1.2x1028 J. The mass of the oceans is about 1.4x1021 kg. It takes about 4,000 J to raise the temperature of 1 kg of water by 1 degree Celcius, and thus about 400,000 J to heat 1 kg of water from freezing to boiling. The latent heat of vaporization adds another 2 million J/kg. Thus the energy required to boil the oceans is about 2.4x106 J/kg * 1.4x1021 kg = 3.4x1027 J. Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. Best part of all: you don't have to understand any of this to use ZFS. Rest assured that you won't hit any limits with that filesystem for a long time. You still have to buy bigger disks over time, though... *** dtrace for Linux, Oracle relicenses dtrace (https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-linux-oracle-does-the-right-thing/) At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom. Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port: ``` commit e1744f50ee9bc1978d41db7cc93bcf30687853e6 Author: Tomas Jedlicka tomas.jedlicka@oracle.com Date: Tue Aug 1 09:15:44 2017 -0400 dtrace: Integrate DTrace Modules into kernel proper This changeset integrates DTrace module sources into the main kernel source tree under the GPLv2 license. Sources have been moved to appropriate locations in the kernel tree. ``` That is right, dtrace dropped the CDDL and switched to the GPL! The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out). The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory. The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL. Thank you Oracle for making everyone's life easier by waving your magic relicensing wand! Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest. Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don't always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all. Debugging Emacs or: How I Learned to Stop Worrying and Love DTrace (http://nullprogram.com/blog/2018/01/17/) For some time Elfeed was experiencing a strange, spurious failure. Every so often users were seeing an error (spoiler warning) when updating feeds: “error in process sentinel: Search failed.” If you use Elfeed, you might have even seen this yourself. From the surface it appeared that curl, tasked with the responsibility for downloading feed data, was producing incomplete output despite reporting a successful run. Since the run was successful, Elfeed assumed certain data was in curl's output buffer, but, since it wasn't, it failed hard. Unfortunately this issue was not reproducible. Manually running curl outside of Emacs never revealed any issues. Asking Elfeed to retry fetching the feeds would work fine. The issue would only randomly rear its head when Elfeed was fetching many feeds in parallel, under stress. By the time the error was discovered, the curl process had exited and vital debugging information was lost. Considering that this was likely to be a bug in Emacs itself, there really wasn't a reliable way to capture the necessary debugging information from within Emacs Lisp. And, indeed, this later proved to be the case. A quick-and-dirty work around is to use condition-case to catch and swallow the error. When the bizarre issue shows up, rather than fail badly in front of the user, Elfeed could attempt to swallow the error — assuming it can be reliably detected — and treat the fetch as simply a failure. That didn't sit comfortably with me. Elfeed had done its due diligence checking for errors already. Someone was lying to Elfeed, and I intended to catch them with their pants on fire. Someday. I'd just need to witness the bug on one of my own machines. Elfeed is part of my daily routine, so surely I'd have to experience this issue myself someday. My plan was, should that day come, to run a modified Elfeed, instrumented to capture extra data. I would have also routinely run Emacs under GDB so that I could inspect the failure more deeply. For now I just had to wait to hunt that zebra. Bryan Cantrill, DTrace, and FreeBSD Over the holidays I re-discovered Bryan Cantrill, a systems software engineer who worked for Sun between 1996 and 2010, and is most well known for DTrace. My first exposure to him was in a BSD Now interview in 2015. I had re-watched that interview and decided there was a lot more I had to learn from him. He's become a personal hero to me. So I scoured the internet for more of his writing and talks. Some interesting operating system technology came out of Sun during its final 15 or so years — most notably DTrace and ZFS — and Bryan speaks about it passionately. Almost as a matter of luck, most of it survived the Oracle acquisition thanks to Sun releasing it as open source in just the nick of time. Otherwise it would have been lost forever. The scattered ex-Sun employees, still passionate about their prior work at Sun, along with some of their old customers have since picked up the pieces and kept going as a community under the name illumos. It's like an open source flotilla. Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn't have these Sun technologies available. The main reason is license incompatibility. Sun released its code under the CDDL, which is incompatible with the GPL. Ubuntu does infamously include ZFS, but other distributions are unwilling to take that risk. Porting DTrace is a serious undertaking since it's got its fingers throughout the kernel, which also makes the licensing issues even more complicated. Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that. Rather than adopt ZFS and DTrace, they've been reinvented from scratch: btrfs instead of ZFS, and a slew of partial options instead of DTrace. Normally I'm most interested in system call tracing, and my go to is strace, though it certainly has its limitations — including this situation of debugging curl under Emacs. Another famous example of NIH is Linux's epoll(2), which is a broken version of BSD kqueue(2). So, if I want to try these for myself, I'll need to install a different operating system. I've dabbled with OmniOS, an OS built on illumos, in virtual machines, using it as an alien environment to test some of my software (e.g. enchive). OmniOS has a philosophy called Keep Your Software To Yourself (KYSTY), which is really just code for “we don't do packaging.” Honestly, you can't blame them since they're a tiny community. The best solution to this is probably pkgsrc, which is essentially a universal packaging system. Otherwise you're on your own. There's also openindiana, which is a more friendly desktop-oriented illumos distribution. Still, the short of it is that you're very much on your own when things don't work. The situation is like running Linux a couple decades ago, when it was still difficult to do. If you're interested in trying DTrace, the easiest option these days is probably FreeBSD. It's got a big, active community, thorough documentation, and a huge selection of packages. Its license (the BSD license, duh) is compatible with the CDDL, so both ZFS and DTrace have been ported to FreeBSD. What is DTrace? I've done all this talking but haven't yet described what DTrace really is. I won't pretend to write my own tutorial, but I'll provide enough information to follow along. DTrace is a tracing framework for debugging production systems in real time, both for the kernel and for applications. The “production systems” part means it's stable and safe — using DTrace won't put your system at risk of crashing or damaging data. The “real time” part means it has little impact on performance. You can use DTrace on live, active systems with little impact. Both of these core design principles are vital for troubleshooting those really tricky bugs that only show up in production. There are DTrace probes scattered all throughout the system: on system calls, scheduler events, networking events, process events, signals, virtual memory events, etc. Using a specialized language called D (unrelated to the general purpose programming language D), you can dynamically add behavior at these instrumentation points. Generally the behavior is to capture information, but it can also manipulate the event being traced. Each probe is fully identified by a 4-tuple delimited by colons: provider, module, function, and probe name. An empty element denotes a sort of wildcard. For example, syscall::open:entry is a probe at the beginning (i.e. “entry”) of open(2). syscall:::entry matches all system call entry probes. Unlike strace on Linux which monitors a specific process, DTrace applies to the entire system when active. To run curl under strace from Emacs, I'd have to modify Emacs' behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That's a big deal. So, when it comes to this Elfeed issue, FreeBSD is much better poised for debugging the problem. All I have to do is catch it in the act. However, it's been months since that bug report and I'm not really making this connection yet. I'm just hoping I eventually find an interesting problem where I can apply DTrace. Bryan Cantrill: Talks I have given (http://dtrace.org/blogs/bmc/2018/02/03/talks/) *** News Roundup a2k18 Hackathon preview: Syncookies coming to PF (https://undeadly.org/cgi?action=article;sid=20180207090000) As you may have heard, the a2k18 hackathon is in progress. As can be seen from the commit messages, several items of goodness are being worked on. One eagerly anticipated item is the arrival of TCP syncookies (read: another important tool in your anti-DDoS toolset) in PF. Henning Brauer (henning@) added the code in a series of commits on February 6th, 2018, with this one containing the explanation: ``` syncookies for pf. when syncookies are on, pf will blindly answer each and every SYN with a syncookie-SYNACK. Upon reception of the ACK completing the 3WHS, pf will reconstruct the original SYN, shove it through pf_test, where state will be created if the ruleset permits it. Then massage the freshly created state (we won't see the SYNACK), set up the sequence number modulator, and call into the existing synproxy code to start the 3WHS with the backend host. Add an - somewhat basic for now - adaptive mode where syncookies get enabled if a certain percentage of the state table is filled up with half-open tcp connections. This makes pf firewalls resilient against large synflood attacks. syncookies are off by default until we gained more experience, considered experimental for now. see http://bulabula.org/papers/2017/bsdcan/ for more details. joint work with sashan@, widely discussed and with lots of input by many ``` The first release to have this feature available will probably be the upcoming OpenBSD 6.3 if a sufficient number of people test this in their setups (hint, hint). More info is likely to emerge soon in post-hackathon writeups, so watch this space! [Pale Moon] A Perfect example of how not to approach OS developers/packagers Removed from OpenBSD Ports due to Licensing Issues (https://github.com/jasperla/openbsd-wip/issues/86) FreeBSD Palemoon branding violation (https://lists.freebsd.org/pipermail/freebsd-ports/2018-February/112455.html) Mightnight BSD's response (https://twitter.com/midnightbsd/status/961232422091280386) *** FreeBSD EC2 History (http://www.daemonology.net/blog/2018-02-12-FreeBSD-EC2-history.html) A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years. Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments. August 25, 2006: Amazon EC2 launches. It supports a single version of Ubuntu Linux; FreeBSD is not available. December 13, 2010: I manage to get FreeBSD running on EC2 t1.micro instances. March 22, 2011: I manage to get FreeBSD running on EC2 "cluster compute" instances. July 8, 2011: I get FreeBSD 8.2 running on all 64-bit EC2 instance types, by marking it as "Windows" in order to get access to Xen/HVM virtualization. (Unfortunately this meant that users had to pay the higher "Windows" hourly pricing.) January 16, 2012: I get FreeBSD 9.0 running on 32-bit EC2 instances via the same "defenestration" trick. (Again, paying the "Windows" prices.) August 16, 2012: I move the FreeBSD rc.d scripts which handle "EC2" functionality (e.g., logging SSH host keys to the console) into the FreeBSD ports tree. October 7, 2012: I rework the build process for FreeBSD 9.1-RC1 and later to use "world" bits extracted from the release ISOs; only the kernel is custom-built. Also, the default SSH user changes from "root" to "ec2-user". October 31, 2012: Amazon launches the "M3" family of instances, which support Xen/HVM without FreeBSD needing to pay the "Windows" tax. November 21, 2012: I get FreeBSD added to the AWS Marketplace. October 2, 2013: I finish merging kernel patches into the FreeBSD base system, and rework the AMI build (again) so that FreeBSD 10.0-ALPHA4 and later use bits extracted from the release ISOs for the entire system (world + kernel). FreeBSD Update can now be used for updating everything (because now FreeBSD/EC2 uses a GENERIC kernel). October 27, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA2 and later AMIs will run FreeBSD Update when they first boot in order to download and install any critical updates. December 1, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA4 and later AMIs bootstrap the pkg tool and install packages at boot time (by default, the "awscli" package). December 9, 2013: I add configinit to FreeBSD 10.0-RC1 and later to allow systems to be easily configured via EC2 user-data. July 1, 2014: Amazon launches the "T2" family of instances; now the most modern family for every type of EC2 instance (regular, high-memory, high-CPU, high-I/O, burstable) supports HVM and there should no longer be any need for FreeBSD users to pay the "Windows tax". November 24, 2014: I add code to FreeBSD 10.2 and later to automatically resize their root filesystems when they first boot; this means that a larger root disk can be specified at instance launch time and everything will work as expected. April 1, 2015: I integrate the FreeBSD/EC2 build process into the FreeBSD release building process; FreeBSD 10.2-BETA1 and later AMIs are built by the FreeBSD release engineering team. January 12, 2016: I enable Intel 82599-based "first generation EC2 Enhanced Networking" in FreeBSD 11.0 and later. June 9, 2016: I enable the new EC2 VGA console functionality in FreeBSD 11.0 and later. (The old serial console also continues to work.) June 24, 2016: Intel 82599-based Enhanced Networking works reliably in FreeBSD 11.0 and later thanks to discovering and working around a Xen bug. June 29, 2016: I improve throughput on Xen blkfront devices (/dev/xbd*) by enabling indirect segment I/Os in FreeBSD 10.4 and later. (I wrote this functionality in July 2015, but left it disabled by default a first because a bug in EC2 caused it to hurt performance on some instances.) July 7, 2016: I fix a bug in FreeBSD's virtual memory initialization in order to allow it to support boot with 128 CPUs; aka. FreeBSD 11.0 and later support the EC2 x1.32xlarge instance type. January 26, 2017: I change the default configuration in FreeBSD 11.1 and later to support EC2's IPv6 networking setup out of the box (once you flip all of the necessary switches to enable IPv6 in EC2 itself). May 20, 2017: In collaboration with Rick Macklem, I make FreeBSD 11.1 and later compatible with the Amazon "Elastic File System" (aka. NFSv4-as-a-service) via the newly added "oneopenown" mount option (and lots of bug fixes). May 25, 2017: I enable support for the Amazon "Elastic Network Adapter" in FreeBSD 11.1 and later. (The vast majority of the work — porting the driver code — was done by Semihalf with sponsorship from Amazon.) December 5, 2017: I change the default configuration in FreeBSD 11.2 and later to make use of the Amazon Time Sync Service (aka. NTP-as-a-service). The current status The upcoming FreeBSD release (11.2) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial VGA), and every EC2 instance type (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances). Colin's Patreon' page if you'd like to support him (https://www.patreon.com/cperciva) X network transparency X's network transparency has wound up mostly being a failure (https://utcc.utoronto.ca/~cks/space/blog/unix/XNetworkTransparencyFailure) I was recently reading Mark Dominus's entry about some X keyboard problems, in which he said in passing (quoting himself): I have been wondering for years if X's vaunted network transparency was as big a failure as it seemed: an interesting idea, worth trying out, but one that eventually turned out to be more trouble than it was worth. [...] My first reaction was to bristle, because I use X's network transparency all of the time at work. I have several programs to make it work very smoothly, and some core portions of my environment would be basically impossible without it. But there's a big qualification on my use of X's network transparency, namely that it's essentially all for text. When I occasionally go outside of this all-text environment of xterms and emacs and so on, it doesn't go as well. X's network transparency was not designed as 'it will run xterm well'; originally it was to be something that should let you run almost everything remotely, providing a full environment. Even apart from the practical issues covered in Daniel Stone's slide presentation, it's clear that it's been years since X could deliver a real first class environment over the network. You cannot operate with X over the network in the same way that you do locally. Trying to do so is painful and involves many things that either don't work at all or perform so badly that you don't want to use them. In my view, there are two things that did in general X network transparency. The first is that networks turned out to not be fast enough even for ordinary things that people wanted to do, at least not the way that X used them. The obvious case is web browsers; once the web moved to lots of images and worse, video, that was pretty much it, especially with 24-bit colour. (It's obviously not impossible to deliver video across the network with good performance, since YouTube and everyone else does it. But their video is highly encoded in specialized formats, not handled by any sort of general 'send successive images to the display' system.) The second is that the communication facilities that X provided were too narrow and limited. This forced people to go outside of them in order to do all sorts of things, starting with audio and moving on to things like DBus and other ways of coordinating environments, handling sophisticated configuration systems, modern fonts, and so on. When people designed these additional communication protocols, the result generally wasn't something that could be used over the network (especially not without a bunch of setup work that you had to do in addition to remote X). Basic X clients that use X properties for everything may be genuinely network transparent, but there are very few of those left these days. (Not even xterm is any more, at least if you use XFT fonts. XFT fonts are rendered in the client, and so different hosts may have different renderings of the same thing, cf.) < What remains of X's network transparency is still useful to some of us, but it's only a shadow of what the original design aimed for. I don't think it was a mistake for X to specifically design it in (to the extent that they did, which is less than you might think), and it did help X out pragmatically in the days of X terminals, but that's mostly it. (I continue to think that remote display protocols are useful in general, but I'm in an usual situation. Most people only ever interact with remote machines with either text mode SSH or a browser talking to a web server on the remote machine.) PS: The X protocol issues with synchronous requests that Daniel Stone talks about don't help the situation, but I think that even with those edges sanded off X's network transparency wouldn't be a success. Arguably X's protocol model committed a lesser version of part of the NeWS mistake. X's network transparency was basically free at the time (https://utcc.utoronto.ca/~cks/space/blog/unix/XFreeNetworkTransparency) I recently wrote an entry about how X's network transparency has wound up mostly being a failure for various reasons. However, there is an important flipside to the story of X's network transparency, and that is that X's network transparency was almost free at the time and in the context it was created. Unlike the situation today, in the beginning X did not have to give up lots of performance or other things in order to get network transparency. X originated in the mid 1980s and it was explicitly created to be portable across various Unixes, especially BSD-derived ones (because those were what universities were mostly using at that time). In the mid to late 1980s, Unix had very few IPC methods, especially portable ones. In particular, BSD systems did not have shared memory (it was called 'System V IPC' for the obvious reasons). BSD had TCP and Unix sockets, some System V machines had TCP (and you could likely assume that more would get it), and in general your safest bet was to assume some sort of abstract stream protocol and then allow for switchable concrete backends. Unsurprisingly, this is exactly what X did; the core protocol is defined as a bidirectional stream of bytes over an abstracted channel. (And the concrete implementation of $DISPLAY has always let you specify the transport mechanism, as well as allowing your local system to pick the best mechanism it has.) Once you've decided that your protocol has to run over abstracted streams, it's not that much more work to make it network transparent (TCP provides streams, after all). X could have refused to make the byte order of the stream clear or required the server and the client to have access to some shared files (eg for fonts), but I don't think either would have been a particularly big win. I'm sure that it took some extra effort and care to make X work across TCP from a different machine, but I don't think it took very much. (At the same time, my explanation here is probably a bit ahistorical. X's initial development seems relatively strongly tied to sometimes having clients on different machines than the display, which is not unreasonable for the era. But it doesn't hurt to get a feature that you want anyway for a low cost.) I believe it's important here that X was intended to be portable across different Unixes. If you don't care about portability and can get changes made to your Unix, you can do better (for example, you can add some sort of shared memory or process to process virtual memory transfer). I'm not sure how the 1980s versions of SunView worked, but I believe they were very SunOS dependent. Wikipedia says SunView was partly implemented in the kernel, which is certainly one way to both share memory and speed things up. PS: Sharing memory through mmap() and friends was years in the future at this point and required significant changes when it arrived. Beastie Bits Grace Hopper Celebration 2018 Call for Participation (https://www.freebsdfoundation.org/news-and-events/call-for-papers/grace-hopper-celebration-2018-call-for-participation/) Google Summer of Code: Call for Project Ideas (https://www.freebsdfoundation.org/blog/google-summer-of-code-call-for-project-ideas/) The OpenBSD Foundation 2018 Fundraising Campaign (https://undeadly.org/cgi?action=article;sid=20180129190641) SSH Mastery 2/e out (https://blather.michaelwlucas.com/archives/3115) AsiaBSDcon 2018 Registration is open (https://2018.asiabsdcon.org/) Tarsnap support for Bitcoin ending April 1st; and a Chrome bug (http://mail.tarsnap.com/tarsnap-announce/msg00042.html) Feedback/Questions Todd - Couple Questions (http://dpaste.com/195HGHY#wrap) Seth - Tar Snap (http://dpaste.com/1N7NQVQ#wrap) Alex - sudo question (http://dpaste.com/3D9P1DW#wrap) Thomas - FreeBSD on ARM? (http://dpaste.com/24NMG47#wrap) Albert - Austria BSD User Group (http://dpaste.com/373CRX7#wrap)
We cover an interview about Unix Architecture Evolution, another vBSDcon trip report, how to teach an old Unix about backspace, new NUMA support coming to FreeBSD, and stack pointer checking in OpenBSD. This episode was brought to you by Headlines Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD (https://fosdem.org/2018/interviews/diomidis-spinellis/) Q: Could you briefly introduce yourself? I'm a professor of software engineering, a programmer at heart, and a technology author. Currently I'm also the editor in chief of the IEEE Software magazine. I recently published the book Effective Debugging, where I detail 66 ways to debug software and systems. Q: What will your talk be about, exactly? I will describe how the architecture of the Unix operating system evolved over the past half century, starting from an unnamed system written in PDP-7 assembly language and ending with a modern FreeBSD system. My talk is based, first, on a GitHub repository where I tried to record the system's history from 1970 until today and, second, on the evolution of documented facilities (user commands, system calls, library functions) across revisions. I will thus present the early system's defining architectural features (layering, system calls, devices as files, an interpreter, and process management) and the important ones that followed in subsequent releases: the tree directory structure, user contributed code, I/O redirection, the shell as a user program, groups, pipes, scripting, and little languages. Q: Why this topic? Unix stands out as a major engineering breakthrough due to its exemplary design, its numerous technical contributions, its impact, its development model, and its widespread use. Furthermore, the design of the Unix programming environment has been characterized as one offering unusual simplicity, power, and elegance. Consequently, there are many lessons that we can learn by studying the evolution of the Unix architecture, which we can apply to the design of new systems. I often see modern systems that suffer from a bloat of architectural features and a lack of clear form on which functionality can be built. I believe that many of the modern Unix architecture defining features are excellent examples of what we should strive toward as system architects. Q: What do you hope to accomplish by giving this talk? What do you expect? I'd like FOSDEM attendees to leave the talk with their mind full with architectural features of timeless quality. I want them to realize that architectural elegance isn't derived by piling design patterns and does not need to be expensive in terms of resources. Rather, beautiful architecture can be achieved on an extremely modest scale. Furthermore, I want attendees to appreciate the importance of adopting flexible conventions rather than rigid enforcement mechanisms. Finally, I want to demonstrate through examples that the open source culture was part of Unix from its earliest days. Q: What are the most significant milestones in the development of Unix? The architectural development of Unix follows a path of continuous evolution, albeit at a slowing pace, so I don't see here the most important milestones. I would however define as significant milestones two key changes in the way Unix was developed. The first occurred in the late 1970s when significant activity shifted from a closely-knit team of researchers at the AT&T Bell Labs to the Computer Science Research Group in the University of California at Berkeley. This opened the system to academic contributions and growth through competitive research funding. The second took place in the late 1980s and the 1990s when Berkeley open-sourced the the code it had developed (by that time a large percentage of the system) and enthusiasts built on it to create complete open source operating system distributions: 386BSD, and then FreeBSD, NetBSD, OpenBSD, and others. Q: In which areas has the development of Unix stalled? The data I will show demonstrate that there were in the past some long periods where the number of C library functions and system calls remained mostly stable. Nowadays there is significant growth in the number of all documented facilities with the exception of file formats. I'm looking forward to a discussion regarding the meaning of these growth patterns in the Q&A session after the talk. Q: What are the core features that still link the 1970 PDP-7 system to the latest FreeBSD 11.1 release, almost half a century apart? Over the past half-century the Unix system has grown by four orders of magnitude from a few thousand lines of code to many millions. Nevertheless, looking at a 1970s architecture diagram and a current one reveals that the initial architectural blocks are still with us today. Furthermore, most system calls, user programs, and C library functions of that era have survived until today with essentially similar functionality. I've even found in modern FreeBSD some lines of code that have survived unchanged for 40 years. Q: Can we still add innovative changes to operating systems like FreeBSD without breaking the ‘Unix philosophy'? Will there be a moment where FreeBSD isn't recognizable anymore as a descendant of the 1970 PDP-7 system? There's a saying that “form liberates”. So having available a time-tested form for developing operating system functionality allows you to innovate in areas that matter rather than reinventing the wheel. Such concepts include having commands act as a filter, providing manual pages with a consistent structure, supplying build information in the form of a Makefile, installing files in a well-defined directory hierarchy, implementing filesystems with an standardized object-oriented interface, and packaging reusable functions as a library. Within this framework there's ample space for both incremental additions (think of jq, the JSON query command) and radical innovations (consider the Solaris-derived ZFS and dtrace functionality). For this reason I think that BSD and Linux systems will always be recognizable as direct or intellectual descendants of the 1970s Research Unix editions. Q: Have you enjoyed previous FOSDEM editions? Immensely! As an academic I need to attend many scientific conferences and meetings in order to present research results and interact with colleagues. This means too much time spent traveling and away from home, and a limited number of conferences I'm in the end able to attend. Nevertheless, attending FOSDEM is an easy decision due to the world-changing nature of its theme, the breadth of the topics presented, the participants' enthusiasm and energy, as well as the exemplary, very efficient conference organization. Another vBSDCon trip report we just found (https://www.weaponizedawesome.com/blog/?cat=53) We just got tipped about another trip report from vBSDCon, this time from one of the first time speakers: W. Dean Freeman Recently I had the honor of co-presenting on the internals of FreeBSD's Kernel RNG with John-Mark Gurney at the 3rd biennial vBSDCon, hosted in Reston, VA hosted by Verisign. I've been in and out of the FreeBSD community for about 20 years. As I've mentioned on here before, my first Unix encounter was FreeBSD 2.2.8 when I was in the 7th or 8th grade. However, for all that time I've never managed to get out to any of the cons. I've been to one or two BUG meetings and I've met some folks from IRC before, but nothing like this. A BSD conference is a very different experience than anything else out there. You have to try it, it is the only way to truly understand it. I'd also not had to do a stand-up presentation really since college before this. So, my first BSD con and my first time presenting rolled into one made for an interesting experience. See, he didn't say terrifying. It went very well. You should totally submit a talk for the next conference, even if it is your first. That said, it was amazing and invigorating experience. I got to meet a few big names in the FreeBSD community, discuss projects, ideas for FreeBSD, etc. I did seem to spend an unusual amount of time talking about FIPS and Common Criteria with folks, but to me that's a good sign and indicative that there is interest in working to close gaps between FreeBSD and the current requirements so that we can start getting FreeBSD and more BSD-based products into the government and start whittling away the domination of Linux (especially since Oracle has cut Solaris, SPARC and the ZFS storage appliance business units). There is nothing that can match the high bandwidth interchange of ideas in person. The internet has made all kinds of communication possible, and we use it all the time, but every once in a while, getting together in person is hugely valuable. Dean then went on to list some of the talks he found most valuable, including DTrace, Capsicum, bhyve, *BSD security tools, and Paul Vixie's talk about gets() I think the talk that really had the biggest impact on me, however, was Kyle Kneisl's talk on BSD community dynamics. One of the key points he asked was whether the things that drew us to the BSD community in the first place would be able to happen today. Obviously, I'm not a 12 or 13 year old kid anymore, but it really got me thinking. That, combined with getting face time with people I'd previously only known as screen names has recently drawn me back into participating in IRC and rejoining mailing lists (wdf on freenode. be on the lookout!) Then Dean covered some thoughts on his own talk: JMG and my talk seems to have been well received, with people paying lots of attention. I don't know what a typical number of questions is for one of these things, but on day one there weren't that many questions. We got about 5 during our question time and spent most of the rest of the day fielding questions from interested attendees. Getting a “great talk!” from GNN after coming down from the stage was probably one of the major highlights for me. I remember my first solo talk, and GNN asking the right question in the middle to get me to explain a part of it I had missed. It was very helpful. I think key to the interest in our presentation was that JMG did a good job framing a very complicated topic's importance in terms everyone could understand. It also helped that we got to drop some serious truth bombs. Final Thoughts: I met a lot of folks in person for the first time, and met some people I'd never known online before. It was a great community and I'm glad I got a chance to expand my network. Verisign were excellent hosts and they took good care of both speakers (covering airfare, rooms, etc.) and also conference attendees at large. The dinners that they hosted were quite good as well. I'm definitely interested in attending vBSDCon again and now that I've had a taste of meeting IRL with the community on scale of more than a handful, I have every intention of finally making it to BSDCan next year (I'd said it in 2017, but then moved to Texas for a new job and it wasn't going to be practical). This year for sure, though! Teaching an Almost 40-year Old UNIX about Backspace (https://virtuallyfun.com/2018/01/17/teaching_an_almost_40-year_old_unix_about_backspace/) Introduction I have been messing with the UNIX® operating system, Seventh Edition (commonly known as UNIX V7 or just V7) for a while now. V7 dates from 1979, so it's about 40 years old at this point. The last post was on V7/x86, but since I've run into various issues with it, I moved on to a proper installation of V7 on SIMH. The Internet has some really good resources on installing V7 in SIMH. Thus, I set out on my own journey on installing and using V7 a while ago, but that was remarkably uneventful. One convenience that I have been dearly missing since the switch from V7/x86 is a functioning backspace key. There seem to be multiple different definitions of backspace: BS, as in ASCII character 8 (010, 0x08, also represented as ^H), and DEL, as in ASCII character 127 (0177, 0x7F, also represented as ^?). V7 does not accept either for input by default. Instead, # is used as the erase character and @ is used as the kill character. These defaults have been there since UNIX V1. In fact, they have been “there” since Multics, where they got chosen seemingly arbitrarily. The erase character erases the character before it. The kill character kills (deletes) the whole line. For example, “ba##gooo#d” would be interpreted as “good” and “bad line@good line” would be interpreted as “good line”. There is some debate on whether BS or DEL is the correct character for terminals to send when the user presses the backspace key. However, most programs have settled on DEL today. tmux forces DEL, even if the terminal emulator sends BS, so simply changing my terminal to send BS was not an option. The change from the defaults outlined here to today's modern-day defaults occurred between 4.1BSD and 4.2BSD. enf on Hacker News has written a nice overview of the various conventions Getting the Diff For future generations as well as myself when I inevitably majorly break this installation of V7, I wanted to make a diff. However, my V7 is installed in SIMH. I am not a very intelligent man, I didn't keep backup copies of the files I'd changed. Getting data out of this emulated machine is an exercise in frustration. In the end, I printed everything on screen using cat(1) and copied that out. Then I performed a manual diff against the original source code tree because tabs got converted to spaces in the process. Then I applied the changes to clean copies that did have the tabs. And finally, I actually invoked diff(1). Closing Thoughts Figuring all this out took me a few days. Penetrating how the system is put together was surprisingly fairly hard at first, but then the difficulty curve eased up. It was an interesting exercise in some kind of “reverse engineering” and I definitely learned something about tty handling. I was, however, not pleased with using ed(1), even if I do know the basics. vi(1) is a blessing that I did not appreciate enough until recently. Had I also been unable to access recursive grep(1) on my host and scroll through the code, I would've probably given up. Writing UNIX under those kinds of editing conditions is an amazing feat. I have nothing but the greatest respect for software developers of those days. News Roundup New NUMA support coming to FreeBSD CURRENT (https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068145.html) Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem, and vmpage*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VMNUMAALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VMNUMAALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff Stack pointer checking – OpenBSD (https://marc.info/?l=openbsd-tech&m=151572838911297&w=2) Stefan (stefan@) and I have been working for a few months on this diff, with help from a few others. At every trap and system call, it checks if the stack-pointer is on a page that is marked MAPSTACK. execve() is changed to create such mappings for the process stack. Also, libpthread is taught the new MAPSTACK flag to use with mmap(). There is no corresponding system call which can set MAP_FLAG on an existing page, you can only set the flag by mapping new memory into place. That is a piece of the security model. The purpose of this change is to twart stack pivots, which apparently have gained some popularity in JIT ROP attacks. It makes it difficult to place the ROP stack in regular data memory, and then perform a system call from it. Workarounds are cumbersome, increasing the need for far more gadgetry. But also the trap case -- if any memory experiences a demand page fault, the same check will occur and potentially also kill the process. We have experimented a little with performing this check during device interrupts, but there are some locking concerns and performance may then become a concern. It'll be best to gain experience from handle of syncronous trap cases first. chrome and other applications I use run fine! I'm asking for some feedback to discover what ports this breaks, we'd like to know. Those would be ports which try to (unconventionally) create their stacks in malloc()'d memory or inside another Data structure. Most of them are probably easily fixed ... Qt 5.9 on FreeBSD (https://euroquis.nl/bobulate/?p=1768) Tobias and Raphael have spent the past month or so hammering on the Qt 5.9 branch, which has (finally!) landed in the official FreeBSD ports tree. This brings FreeBSD back up-to-date with current Qt releases and, more importantly, up-to-date with the Qt release KDE software is increasingly expecting. With Qt 5.9, the Elisa music player works, for instance (where it has run-time errors with Qt 5.7, even if it compiles). The KDE-FreeBSD CI system has had Qt 5.9 for some time already, but that was hand-compiled and jimmied into the system, rather than being a “proper” ports build. The new Qt version uses a new build system, which is one of the things that really slowed us down from a packaging perspective. Some modules have been reshuffled in the process. Some applications depending on Qt internal-private headers have been fixed along the way. The Telegram desktop client continues to be a pain in the butt that way. Following on from Qt 5.9 there has been some work in getting ready for Clang 6 support; in general the KDE and Qt stack is clean and modern C++, so it's more infrastructural tweaks than fixing code. Outside of our silo, I still see lots of wonky C++ code being fixed and plenty of confusion between pointers and integers and strings and chars and .. ugh. Speaking of ugh, I'm still planning to clean up Qt4 on ARM aarch64 for FreeBSD; this boils down to stealing suitable qatomic implementations from Arch Linux. For regular users of Qt applications on FreeBSD, there should be few to no changes required outside the regular upgrade cycle. For KDE Plasma users, note that development of the ports has changed branches; as we get closer to actually landing modern KDE bits, things have been renamed and reshuffled and mulled over so often that the old plasma5 branch wasn't really right anymore. The kde5-import branch is where it's at nowadays, and the instructions are the same: the x11/kde5 metaport will give you all the KDE Frameworks 5, KDE Plasma Desktop and modern KDE Applications you need. Adding IPv6 to an Nginx website on FreeBSD / FreshPorts (https://dan.langille.org/2018/01/13/adding-ipv6-to-an-nginx-website-on-freebsd-freshports/) FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized. There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation. All of this took about 3 hours. We are using: FreeBSD 11.1 Bind 9.9.11 nginx 1.12.2 Fallout I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses. ZFS on TrueOS: Why We Love OpenZFS (https://www.trueos.org/blog/zfs-trueos-love-openzfs/) TrueOS was the first desktop operating system to fully implement the OpenZFS (Zettabyte File System or ZFS for short) enterprise file system in a stable production environment. To fully understand why we love ZFS, we will look back to the early days of TrueOS (formerly PC-BSD). The development team had been using the UFS file system in TrueOS because of its solid track record with FreeBSD-based computer systems and its ability to check file consistency with the built-in check utility fsck. However, as computing demands increased, problems began to surface. Slow fsck file verification on large file systems, slow replication speeds, and inconsistency in data integrity while using UFS logging / journaling began to hinder users. It quickly became apparent that TrueOS users would need a file system that scales with evolving enterprise storage needs, offers the best data protection, and works just as well on a hobbyist system or desktop computer. Kris Moore, the founder of the TrueOS project, first heard about OpenZFS in 2007 from chatter on the FreeBSD mailing lists. In 2008, the TrueOS development team was thrilled to learn that the FreeBSD Project had ported ZFS. At the time, ZFS was still unproven as a graphical desktop solution, but Kris saw a perfect opportunity to offer ZFS as a cutting-edge file system option in the TrueOS installer, allowing the TrueOS project to act as an indicator of how OpenZFS would fair in real-world production use. The team was blown away by the reception and quality of OpenZFS on FreeBSD-based systems. By its nature, ZFS is a copy-on-write (CoW) file system that won't move a block of data until it both writes the data and verifies its integrity. This is very different from most other file systems in use today. ZFS is able to assure that data stays consistent between writes by automatically comparing write checksums, which mitigates bit rot. ZFS also comes with native RaidZ functionality that allows for enterprise data management and redundancy without the need for expensive traditional RAID cards. ZFS snapshots allow for system configuration backups in a split-second. You read that right. TrueOS can backup or restore snapshots in less than a second using the ZFS file system. Given these advantages, the TrueOS team decided to use ZFS as its exclusive file system starting in 2013, and we haven't looked back since. ZFS offers TrueOS users the stable workstation experience they want, while simultaneously scaling to meet the increasing demands of the enterprise storage market. TrueOS users are frequently commenting on how easy it is to use ZFS snapshots with our built-in snapshot utility. This allows users the freedom to experiment with their system knowing they can restore it in seconds if anything goes wrong. If you haven't had a chance to try ZFS with TrueOS, browse to our download page and make sure to grab a copy of TrueOS. You'll be blown away by the ease of use, data protection functionality, and incredible flexibility of RaidZ. Beastie Bits Source Code Podcast Interview with Michael W Lucas (https://blather.michaelwlucas.com/archives/3099) Operating System of the Year 2017: NetBSD Third place (https://w3techs.com/blog/entry/web_technologies_of_the_year_2017) OPNsense 18.1-RC1 released (https://opnsense.org/opnsense-18-1-rc1-released/) Personal OpenBSD Wiki Notes (https://balu-wiki.readthedocs.io/en/latest/security/openbsd.html) BSD section can use some contribution (https://guide.freecodecamp.org/bsd-os/) The Third Research Edition Unix Programmer's Manual (now available in PDF) (https://github.com/dspinellis/unix-v3man) Feedback/Questions Alex - my first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) Brian - Brian - openbsd 6.2 and enlightenment .17 (http://dpaste.com/24DYF1J#wrap) ***
We walk through dumping a PS4 kernel in only 6 days, tell you the news that NetBSD 7.1.1 has been released, details on how to run FreeBSD on a Thinkpad T470, and there's progress in OpenBSD's pledge. This episode was brought to you by Headlines NetBSD 7.1.1 released (http://www.netbsd.org/releases/formal-7/NetBSD-7.1.1.html) The NetBSD Project is pleased to announce NetBSD 7.1.1, the first security/critical update of the NetBSD 7.1 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. Complete source and binaries for NetBSD 7.1.1 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at https://www.NetBSD.org/mirrors/. We encourage users who wish to install via ISO or USB disk images to download via BitTorrent by using the torrent files supplied in the images area. A list of hashes for the NetBSD 7.1.1 distribution has been signed with the well-connected PGP key for the NetBSD Security Officer: https://ftp.NetBSD.org/pub/NetBSD/security/hashes/NetBSD-7.1.1_hashes.asc NetBSD is free. All of the code is under non-restrictive licenses, and may be used without paying royalties to anyone. Free support services are available via our mailing lists and website. Commercial support is available from a variety of sources. More extensive information on NetBSD is available from our website: NetBSD website (www.NetBSD.org) +Changes Between 7.1 and 7.1.1 Below is an abbreviated list of changes in this release. The complete list can be found in the CHANGES-7.1.1 file in the top level directory of the NetBSD 7.1.1 release tree. Security Advisory Fixes The following security advisories were fixed: NetBSD-SA2017-004 buffer overflow via cmap for 4 graphics drivers. NetBSD-SA2017-005 x86: vulnerabilities in context handling. NetBSD-SA2017-006 Vnode reference leak in the openat system call. NetBSD-SA2018-001 Several vulnerabilities in context handling NetBSD-SA2018-002 Local DoS in virecover Note: Advisories prior to NetBSD-SA2017-004 do not affect NetBSD 7.1.1. Userland changes dhcrelay(8): Fix bug that prevented proper operation when run in the background. Heimdal: Update to 7.1. Fix CVE-2017-11103. mtree(8): Don't modify strings stored in hash, otherwise filling up of directory hierarchy stops if the same hash value occurs in directory and leaf. ping(8): Fix cksum calculation for clearing the cached route. resize_ffs(8): Fix numerous overflow errors which can lead to superblock corruption on large filesystems. rtadvd(8): Fix the default value of rltime. PR bin/51994. Update BIND to 9.10.5-P2. Update expat to 2.2.1. Update ntp to 4.2.8p10. Update root.cache to 2017102400. Update tzdata to 2017c. vi(1): Don't garble display when when resizing nvi in xterm. wpa_supplicant/hostapd: Update to 2.6. Apply fixes for CVEs 2017-13077 through 2017-13082 and CVEs 2017-13086 through 2017-13088. X: Apply fixes for CVEs 2017-12176 through 2017-12187, 2017-10971, 2017-10972, 2017-13722, 2017-13720, 2017-16611, and 2017-16612. *** ###Dumping a PS4 Kernel in "Only" 6 Days (https://fail0verflow.com/blog/2017/ps4-crashdump-dump/) > What if a secure device had an attacker-viewable crashdump format? What if that same device allowed putting arbitrary memory into the crashdump? Amazingly, the ps4 tempted fate by supporting both of these features! Let's see how that turned out… Crashdumps on PS4 The crash handling infrastructure of the ps4 kernel is interesting for 2 main reasons: It is ps4-specific code (likely to be buggy) If the crashdump can be decoded, we will gain very useful info for finding bugs and creating reliable exploits On a normal FreeBSD system, a kernel panic will create a dump by calling kernreboot with the RBDUMP flag. This then leads to doadump being called, which will dump a rather tiny amount of information about the kernel image itself to some storage device. On ps4, the replacement for doadump is mdbgrundump, which can be called from panic or directly from trapfatal. The amount of information stored into the dump is gigantic by comparison - kernel state for all process, thread, and vm objects are included, along with some metadata about loaded libraries. Other obvious changes from the vanilla FreeBSD method are that the mdbgrun_dump encodes data recorded into the dump on a field-by-field basis and additionally encrypts the resulting buffer before finally storing it to disk. Dumping Anything Let's zoom in to a special part of mdbgrundump - where it iterates over all process' threads and tries to dump some pthread state: dumpstate is a temporary buffer which will eventually make it into the crashdump. To summarize, sysdump_internalcallreaduser can be made to function as a read-anywhere oracle. This is because fsbase will point into our (owned) webkit process' usermode address space. Thus, even without changing the actual fsbase value, we may freely change the value of tcbthread, which is stored at fsbase + 0x10. Further, sysdump_internalcall_readuser will happily read from a kernel address and put the result into the dump. We can now put any kernel location into the dump, but we still need to decrypt and decode it… Aside from that, there's also the issue that we may only add 0x10 bytes per thread in this manner… Further reading: Crashdump Crypto Crashdump Decoding Crashdump Automation Triggering the Vulnerability The Fix (Kind of…) Fin Appendix Crashdump Decryptor NXDP Decoder *** ###BSDTW 2017 Conference Recap: Li-Wen Hsu (https://www.freebsdfoundation.org/blog/bsdtw-2017-conference-recap-li-wen-hsu/) BSDTW 2017 Conference Recap: Li-Wen Hsu 12/28/2017 > Last month, we held BSDTW 2017 on November 11-12th, 2017 in Taipei, Taiwan. It was the second largest BSD conference in Taiwan and the first one in this decade. In 2004, the first AsiaBSDCon was also held in Taipei. Then all of the following AsiaBSDCon conferences were held in Tokyo, Japan. (AsiaBSDCon 2018 will be in Tokyo again next year, please submit your talk proposal by December 31th 2017, and attend the conference on March 8th-11th) > We wanted to start small with the first BSDTW because we were not sure how much sponsorship or how many volunteers we might have. BSDTW 2017 was a single track, two-day conference with 11 selected 50 minute presentations and 1 WIP/lightning talk session consisting of 8 short talks. I do regret that we did not have any local presenters this year. It is also a similar problem at AsiaBSDCon. Unsurprisingly, as with AsiaBSDcon, the travel reimbursement took up a large part of the whole conference budget. We do have many good people that work in Asia, but we still need to encourage people to present their work more. > We had over 130 registered attendees, with 30% of them coming from outside of Taiwan. To our knowledge, in recent years, this is the only open source conference in Taiwan to be held entirely in English, and to have such a large portion of international attendees. This is also the first open source conference in Taiwan to focus entirely on operating systems. The attendees included students, professors, engineers or CTOs, and CEOs from technology companies. This is also the first time that GroffTheBSDGoat visited Taiwan! We were surprised that after the silence for so many years, there are still so many people that use and love BSD near us. We saw many old friends, who had “disappeared” for a long time, came back, and were glad to meet many new friends at the conference. I am really happy that this conference was able to bring together these people, from local and abroad. After attending BSD conferences around the world for many years, I feel that the friendship between BSD users is the most important thing in the BSD community, and one of the main reasons people stay. It has been my pleasure to bring this community back to my friends in my homeland. > After the two-day event, I truly understand that bootstrapping a new conference is a very hard job. One with many aspects that you don't even imagine until you're really in the process of planning an event. I now have an even greater respect for all of the conference organizers and realize that we need to have more people help them, to keep these conferences continue to get better and better. Plus, there will always be room for a new conference! > Thanks to the FreeBSD Foundation for being the biggest sponsor of BSDTW 2017 and always being the strongest backend of our community. We are excited about the many local companies and organizations that helped us whether with people, materials or financially. We even had 21 personal sponsors, more than two times the number of other big open source conferences in Taiwan. > As I said in the closing session, I'm not sure if there will be 2nd BSDTW next year. It still depends on the amount of sponsorship and number of volunteers. However, we will definitely hold more smaller meetups in the next year to keep building up the local BSD community. > Finally, in the beginning of this month, we had a “post-conference media workshop” for organizing the media files we collected in the BSDTW 2017. Here are the review article in Traditional Chinese and the photos: https://medium.com/@bsdtw/bsdtw-2017-總回顧-a402788daede (https://medium.com/@bsdtw/bsdtw-2017-總回顧-a402788daede) && https://www.flickr.com/photos/bsdtw/albums/72157689410035911 (https://www.flickr.com/photos/bsdtw/albums/72157689410035911) *** ##News Roundup ###Running FreeBSD on a Lenovo T470s (https://blog.grem.de/pages/t470s.html) Running FreeBSD on the Lenovo T470s ThinkPad > Installing FreeBSD on this machine was super easy. As I couldn't find a comprehensive/encouraging how-to about installing FreeBSD on a recent ThinkPad, I just wrote up the one below. It includes details about my personal setup, which are not required to run FreeBSD on this model, but which are more to my own taste. I still think this can be a quite useful inspiration for others who want to run their own customized configurations. Specs > The system I use has these specifications: Type: 20JS-001EGE CPU: Intel Core i7-6600U, 2x 2.60GHz RAM: 20GB DDR4 SSD: 512GB NVMe Graphics: Intel HD Graphics 520 (IGP), 1x HDMI 1.4 Display: 14", 1920x1080, non-glare, IPS Ports: 3x USB-A 3.0, 1x Thunderbolt 3, 1x Gb LAN Wireless: WLAN 802.11a/b/g/n/ac, Bluetooth 4.1, LTE (Micro-SIM) Cardreader: SD/SDHC/SDXC/MMC Webcam: 0.9 Megapixel Extras: MIL-STD-810G, Pointing Stick, Fingerprint-Reader, Docking port Things that work > Basically everything I care about: Accelerated video Keyboard Touchpad/ClickPad (like expected in a modern laptop) SSD WiFi Sound HDMI out Suspend to RAM Webcam Things that don't work Fingerprint reader Potentially anything I didn't test Battery life is okay, but could be better. Installation of the base system > I used a snapshot release of 12-CURRENT as the basis of my installation, particularly the one of 13th of December 2017. > I dd'ed it onto a memory stick and boot the laptop. I started a standard installation and created an encrypted ZFS pool on nvme0, using encryption, swap encryption and partition scheme "GPT (UEFI)". > After installation, it boots straight up. Ports tree used > All work is based on a head ports tree from about Dec 18, 22:15 CET, which should be more or less r456672. Preferred ClickPad configuration > As I'm not a fan of the the pointing stick, I disabled it in the bios. My final ClickPad configuration will be: Click to click (not tap), no middle button, right button in the lower right corner. As the old synaptics driver doesn't provide good thumb detection, libinput will be used. Check out the laptop list on the FreeBSD wiki for compatibility: (https://wiki.freebsd.org/Laptops/) *** ###FreeBSD desktop LiveCD creator (https://github.com/pkgdemon/comet) Introduction > The purpose of this tool is quickly generate bloat free images containing stock FreeBSD, and supported desktop environments. Features FreeBSD 11.1-RELEASE AMD64 Gnome & KDE desktop environments Hybrid DVD/USB image Screenshots [Gnome LiveCD])https://github.com/pkgdemon/comet/raw/master/screenshots/gnome-livecd.png?raw=true) KDE LiveCD (https://github.com/pkgdemon/comet/raw/master/screenshots/kde-livecd.png?raw=true) System Requirements FreeBSD 11.1, or higher for AMD64 20GB of free disk space 1GB of free memory UFS, or ZFS Initial Setup Install the required packages: pkg install git grub2-pcbsd grub2-efi xorriso Clone the repo: git clone https://www.github.com/pkgdemon/comet Enter the directory for running the LiveCD creator: cd comet/src Credentials for live media > User: liveuser > Password: freebsd *** ###iXsystems StorageCrypter Ransomware: Security Threat or Clickbait? (https://www.ixsystems.com/blog/storagecrypter/) ###pledge() work in progress (https://undeadly.org/cgi?action=article;sid=20171208082246) > I wanted to give an update that a two pledge-related changes are being worked on. The semantics and integration are complicated so it is taking some time. > One is execpromises. This will become the 2nd argument of pledge(). This allows one to set the pledge for the new image after pledge "exec"-allowed execve(). A warning though: utilizing this in software isn't as easy as you might think! The fork+exec + startup sequences needed to be studied quite carefully to ensure the newly-executed child doesn't ask for more than the parent's execpromises. In my experiments such a circumstance is exceedingly common, so the problem is eased by introducing a new pledge feature which allows pledge violations to return ENOSYS or such rather than killing the process. > This feature also needs to be used with great caution (especially in privileged programs) because programs which fail to observe errors may continue operating forward very incorrectly; you've lost the ability to catch it failing, and provide care by fixing the problem. > The other is pledgepaths. The semantics are still being tuned a bit. Before the first call to pledge() in a process, one can pledgepath() directories. Then later after pledge(), file access operations only work if the traversal of the path crosses one of those pre-declared directories (but better make sure you don't move a directory, because the kernel remembers and reasons about the vnode of the directory rather than the path). Something similar is being worked on for files, but we are still adjusting that, as well as a flag parameter for the pledgepath() call which may constrain the operations done on such files. > As such, pledgepath() will become a filesystem containment mechanism unlike chroot() because paths will still be based upon true /. > Patience. *** ###The anatomy of tee program on OpenBSD (http://nanxiao.me/en/the-anatomy-of-tee-program-on-openbsd/) > The tee command is used to read content from standard input and displays it not only in standard output but also saves to other files simultaneously. The source code of tee in OpenBSD is very simple, and I want to give it an analysis: > (1) tee leverages Singlely-linked List defined in sys/queue.h to manage outputted files (including standard output): struct list { SLIST_ENTRY(list) next; int fd; char *name; }; SLIST_HEAD(, list) head; ...... static void add(int fd, char *name) { struct list *p; ...... SLIST_INSERT_HEAD(&head, p, next); } int main(int argc, char *argv[]) { struct list *p; ...... SLIST_INIT(&head); ...... SLIST_FOREACH(p, &head, next) { ...... } } > To understand it easily, I extract the macros from sys/queue.h and created a file which utilizes the marcos: #define SLIST_HEAD(name, type) struct name { struct type *slh_first; /* first element */ } #define SLIST_ENTRY(type) struct { struct type *sle_next; /* next element */ } #define SLIST_FIRST(head) ((head)->slh_first) #define SLIST_END(head) NULL #define SLIST_EMPTY(head) (SLIST_FIRST(head) == SLIST_END(head)) #define SLIST_NEXT(elm, field) ((elm)->field.sle_next) #define SLIST_FOREACH(var, head, field) for((var) = SLIST_FIRST(head); (var) != SLIST_END(head); (var) = SLIST_NEXT(var, field)) #define SLIST_INIT(head) { SLIST_FIRST(head) = SLIST_END(head); } #define SLIST_INSERT_HEAD(head, elm, field) do { (elm)->field.sle_next = (head)->slh_first; (head)->slh_first = (elm); } while (0) struct list { SLIST_ENTRY(list) next; int fd; char *name; }; SLIST_HEAD(, list) head; int main(int argc, char *argv[]) { struct list *p; SLIST_INIT(&head); SLIST_INSERT_HEAD(&head, p, next); SLIST_FOREACH(p, &head, next) { } } > Then employed gcc‘s pre-processing function: # gcc -E slist.c # 1 "slist.c" # 1 "" # 1 "" # 1 "slist.c" # 30 "slist.c" struct list { struct { struct list *sle_next; } next; int fd; char *name; }; struct { struct list *slh_first; } head; int main(int argc, char *argv[]) { struct list *p; { ((&head)->slh_first) = NULL; }; do { (p)->next.sle_next = (&head)->slh_first; (&head)->slh_first = (p); } while (0); for((p) = ((&head)->slh_first); (p) != NULL; (p) = ((p)->next.sle_next)) { } } > It becomes clear now! The head node in list contains only 1 member: slhfirst, which points to the first valid node. For the elements in the list, it is embedded with next struct which uses slenext to refer to next buddy. > (2) By default, tee will overwrite the output files. If you want to append it, use -a option, and the code is as following: while (*argv) { if ((fd = open(*argv, O_WRONLY | O_CREAT | (append ? O_APPEND : O_TRUNC), DEFFILEMODE)) == -1) { ...... } ...... } > (3) The next part is the skeleton of saving content to files: while ((rval = read(STDIN_FILENO, buf, sizeof(buf))) > 0) { SLIST_FOREACH(p, &head, next) { n = rval; bp = buf; do { if ((wval = write(p->fd, bp, n)) == -1) { ...... } bp += wval; } while (n -= wval); } } > We need to iterates every opened file descriptor and write contents into it. > (4) Normally, theinterrupt signal will cause tee exit: # tee fdkfkdfjk fdkfkdfjk ^C # > To disable this feature, use -i option: # tee -i fdhfhd fdhfhd ^C^C > The corresponding code is like this: ...... case 'i': (void)signal(SIGINT, SIG_IGN); break; *** ##Beastie Bits What I learned from reading the OpenBSD's network stack source code (https://bijanebrahimi.github.io/blog/openbsds-network-stack-part-1.html) Broadcom BCM43224 and BCM43225 Wi-Fi cards now supported by bwn(4) (https://github.com/freebsd/freebsd/commit/888843e26a4e393f405c1c6cbdfc5b701670d363) Ingo details searching man pages (https://marc.info/?l=openbsd-misc&m=151320195122669&w=2) DTrace & ZFS Being Updated On NetBSD, Moving Away From Old OpenSolaris Code (https://www.phoronix.com/scan.php?page=news_item&px=NetBSD-ZFS-DTrace-Updating) Linux Professional Institute and BSD Certification Group Join Efforts (http://www.lpi.org/articles/linux-professional-institute-and-bsd-certification-group-join-efforts) The FreeBSD Foundation thanks Donors (https://www.freebsdfoundation.org/blog/thank-you-2/) ##Feedback/Questions Alex - My first freebsd bug (http://dpaste.com/3DSV7BC#wrap) John - Suggested Speakers (http://dpaste.com/2QFR4MT#wrap) Todd - Two questions (http://dpaste.com/2FQ450Q#wrap) Matthew - CentOS to FreeBSD (http://dpaste.com/3KA29E0#wrap) ***
Picking a compiler for debuggability, how to port Rust apps to FreeBSD, what the point of Docker is on FreeBSD/Solaris, another EuroBSDcon recap, and network manager control in OpenBSD This episode was brought to you by Headlines Compile once, Debug twice: Picking a compiler for debuggability, part 1 of 3 (https://backtrace.io/blog/compile-once-debug-twice-picking-a-compiler-for-debuggability-1of3/) An interesting look into why when you try to debug a crash, you can often find all of the useful information has been ‘optimized out' Have you ever had an assert get triggered only to result in a useless core dump with missing variable information or an invalid callstack? Common factors that go into selecting a C or C++ compiler are: availability, correctness, compilation speed and application performance. A factor that is often neglected is debug information quality, which symbolic debuggers use to reconcile application executable state to the source-code form that is familiar to most software engineers. When production builds of an application fail, the level of access to program state directly impacts the ability for a software engineer to investigate and fix a bug. If a compiler has optimized out a variable or is unable to express to a symbolic debugger how to reconstruct the value of a variable, the engineer's investigation process is significantly impacted. Either the engineer has to attempt to recreate the problem, iterate through speculative fixes or attempt to perform prohibitively expensive debugging, such as reconstructing program state through executable code analysis. Debug information quality is in fact not proportionally related to the quality of the generated executable code and wildly varies from compiler to compiler. Different compilers emit debug information at varying levels of quality and accuracy. However, certain optimizations will certainly impact any debugger's ability to generate accurate stack traces or extract variable values. In the above program, the value of argv is extracted and then the program is paused. The ckprloadptr function performs a read from the region of memory pointed to by argv, in a manner that prevents the compiler from performing optimization on it. This ensures that the memory access occurs and for this reason, the value of argv must be accessible by the time ckprloadptr is executed. When compiled with gcc, the debugger fails to find the value of the variable. The compiler determines that the value of argv is no longer needed after the ckprload_ptr operation and so doesn't bother paying the cost of saving the value. Some optimizations generate executable code whose call stack cannot be sufficiently disambiguated to reconcile a call stack that mirrors that of the source program. Two common culprits for this are tail call optimization and basic block commoning. In another example If the program receives a first argument of 1, then function is called with the argument of "a". If the program receives a first argument of 2, then function is called with the argument of "b". However, if we compile this program with clang, the stack traces in both cases are identical! clang informs the debugger that the function f invoked the function("b") branch where x = 2 even if x = 1. Though some optimizations will certainly impact the accuracy of a symbolic debugger, some compilers simply lack the ability to generate debug information in the presence of certain optimizations. One common optimization is induction variable elimination. A variable that's incremented or decremented by a constant on every iteration of a loop or derived from another variable that follows this pattern, is an induction variable. Coupled with other optimizations, the compiler is then able to generate code that doesn't actually rely on a dedicated counter variable “i” for maintaining the current offset into “buffer”. As you can see, i is completely optimized out. The compiler determines it doesn't have to pay the cost of maintaining the induction variable i. It maintains the pointer in the register %rdi. The code is effectively rewritten to something closer to this: So the for loop, changes into a while loop, with a condition of the end of the input We have shown some common optimizations that may get in the way of the debuggability of your application and demonstrated a disparity in debug information quality across two popular compilers. In the next blog post of this series, we will examine how gcc and clang stack up with regards to debug information quality across a myriad of synthetic applications and real world applications. Looking forward to part 2 *** This is how you can port your rust application to FreeBSD (https://medium.com/@andoriyu/this-is-how-you-can-port-your-rust-application-to-freebsd-7d3e9f1bc3df) This is how you can port your rust application to FreeBSD The FreeBSD Ports Collection is the way almost everyone installs applications (“ports”) on FreeBSD. Like everything else about FreeBSD, it is primarily a volunteer effort. It is important to keep this in mind when reading this document. In FreeBSD, anyone may submit a new port, or volunteer to maintain an existing unmaintained port. No special commit privilege is needed. For this guide I will use fd tool written by David Peter as example project. Prerequisites FreeBSD installation (VM is fine) Local ports tree (done via svn) portlint (located at devel/portlint) poudriere (located at ports-mgmt/poudriere)[optional] Getting ports tree When you install FreeBSD opt-out of the ports tree. Install svn: pkg install svn svn checkout https://svn.freebsd.org/ports/head /usr/ports Poudriere Sometimes you might get asked to show poudriere build log, sometimes you won't. It's good to have anyway. If you choose to use poudriere, use ZFS. There are plenty of guides on the subject. FreeBSD Porter's Handbook is the most complete source of information on porting to FreeBSD. Makefile Whole porting process in most cases is writing one Makefile. I recommend doing something like this. Here is the one I wrote for fd: Port metadata Each port must have one primary category in case of fd it will be sysutils, therefore it's located in /usr/ports/systuils/fd. PORTNAME= fd CATEGORIES= sysutils Since this port conflicts with other util named fd I specified package suffix as: PKGNAMESUFFIX= -find and indicate conflict: CONFLICTS_INSTALL= fd-[0-9]*. That means to install it from packages user will have to type: pkg install fd-find Licenses This section is different for every port, but in case of fd it's pretty straightforward: LICENSE= MIT APACHE20 LICENSE_COMB= dual Since fd includes the text of licenses you should do this as well: LICENSE_FILE_MIT= ${WRKSRC}/LICENSE-MIT LICENSE_FILE_APACHE20= ${WRKSRC}/LICENSE-APACHE Distfiles FreeBSD has a requirement that all ports must allow offline building. That means you have specified which files are needed to be downloaded. Luckily we now have helpers to download GitHub sources directly from GitHub: USE_GITHUB= yes GH_ACCOUNT= sharkdp Since PORTNANE is fd it will try to download sources for sharkdp/fd. By default it's going to download tag: ${DISTVERSIONPREFIX}${DISTVERSION}${DISTVERSIONSUFFIX} fd uses v as the prefix, therefore we need to specify: DISTVERSIONPREFIX= v. It's also possible to specify GH_TAGNAME in case tag name doesn't match that pattern. Extra packages There are very few rust projects that are standalone and use no crates dependencies. It's used to be PITA to make it work offline, but now cargo is a first class citizen in ports: USES= cargo CARGO_CRATES= aho-corasick-0.6.3 atty-0.2.3 # and so goes on Yes, you have to specify each dependency. Luckily, there is a magic awk script that turns Cargo.lock into what you need. Execute make cargo-crates in the port root. This will fail because you're missing checksum for the original source files: make makesum make cargo-crates This will give you what you need. Double check that result is correct. There is a way to ignore checksum error, but I can't remember… Execute make makesum again. CARGO_OUT If. build.rs relies on that you have to change it. fd allows you to use SHELLCOMPLETIONSDIR to specify where completions go, while ripgrep doesn't. In our case we just specify SHELLCOMPLETIONSDIR: SHELL_COMPLETIONS_DIR= ${WRKDIR}/shell-completions-dir CARGO_ENV= SHELL_COMPLETIONS_DIR=${SHELL_COMPLETIONS_DIR} PLIST FreeBSD is very strict about files it's installing and it won't allow you to install random files that get lost. You have to specify which files you're installing. In this case, it's just two: PLIST_FILES= bin/fd man/man1/fd.1.gz Note that sources for fd have uncompressed man file, while here it's listed as compressed. If port installs a lot of files, specify them in pkg-plist like here. To actually install them: post-install: @${STRIP_CMD} ${STAGEDIR}${PREFIX}/bin/fd ${INSTALL_MAN}${WRKSRC}/doc/fd.1 ${STAGEDIR}${MAN1PREFIX}/man/man1 Shell completions clap-rs can generate shell completions for you, it's usually handled by build.rs script. First, we need to define options: OPTIONS_DEFINE= BASH FISH ZSH # list options OPTIONS_DEFAULT= BASH FISH ZSH # select them by default BASH_PLIST_FILES= etc/bash_completion.d/fd.bash-completion FISH_PLIST_FILES= share/fish/completions/fd.fish ZSH_PLIST_FILES= share/zsh/site-functions/_fd To actually install them: post-install-BASH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/etc/bash_completion.d ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/fd.bash-completion ${STAGEDIR}${PREFIX}/etc/bash_completion.d post-install-FISH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/share/fish/completions ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/fd.fish ${STAGEDIR}${PREFIX}/share/fish/completions post-install-ZSH-on: @${MKDIR} ${STAGEDIR}${PREFIX}/share/zsh/site-functions ${INSTALL_DATA} ${SHELL_COMPLETIONS_DIR}/_fd ${STAGEDIR}${PREFIX}/share/zsh/site-functions Bonus round - Patching source code Sometimes you have to patch it and send the patch upstream. Merging it upstream can take awhile, so you can patch it as part of the install process. An easy way to do it: Go to work/ dir Copy file you want to patch and add .orig suffix to it Edit file you want to patch Execute make makepatch in port's root Submitting port First, make sure portlint -AC doesn't give you any errors or warnings. Second, make sure poudriere can build it on both amd64 and i386. If it can't?—?you have to either fix it or mark port broken for that arch. Follow this steps like I did steps. If you have any issues you can always ask your question in freebsd-ports on freenode try to find your answer in porter's handbook before asking. Conference Recap: EuroBSDCon 2017 Recap (https://www.freebsdfoundation.org/blog/conference-recap-eurobsdcon-2017-recap/) The location was wonderful and I loved sneaking out and exploring the city when I could. From what I heard, it was the largest BSD conference in history, with over 320 attendees! Each venue is unique and draws many local BSD enthusiasts, who normally wouldn't be able to travel to a conference. I love having the chance to talk to these people about how they are involved in the projects and what they would like to do. Most of the time, they are asking me questions about how they can get more involved and how we can help. Magical is how I would describe the conference social event. To stand in front of the dinner cruise on the Seine, with the Eiffel Tower standing tall, lit up in the night, while working – talking to our community members, was incredible. But, let me start at the beginning. We attend these conferences to talk to our community members, to find out what they are working on, determine technologies that should be supported in FreeBSD, and what we can do to help and improve FreeBSD. We started the week with a half-day board meeting on Wednesday. BSD conferences give us a chance to not only meet with community members around the world, but to have face-to-face meetings with our team members, who are also located around the world. We worked on refining our strategic direction and goals, determining what upcoming conferences we want FreeBSD presence at and who can give FreeBSD talks and workshops there, discussed current and potential software development projects, and discussed how we can help raise awareness about and increase the use of FreeBSD in Europe. Thursday was the first day of the FreeBSD developer summit, led by our very own Benedict Reuschling. He surprised us all by having us participate in a very clever quiz on France. 45 of us signed into the software, where he'd show the question on the screen and we had a limited amount of time to select our answers, with the results listed on the screen. It was actually a lot of fun, especially since they didn't publicize the names of the people who got the questions wrong. The lucky or most knowledgeable person on France, was des@freebsd.org. Some of our board members ran tutorials in parallel to the summit. Kirk McKusick gave his legendary tutorial, An Introduction to the FreeBSD Open-Source Operating System , George Neville-Neil gave his tutorial, DTrace for Developers, and Benedict Reuschling gave a tutorial on, Managing BSD systems with Ansible. I was pleased to have two chairs from ACM-W Europe run an “Increasing Diversity in the BSDs” BoF for the second year in a row. We broke up into three groups to discuss different gender bias situations, and what we can do to address these types of situations, to make the BSD projects more diverse, welcoming, and inclusive. At the end, people asked that we continue these discussions at future BSD conferences and suggested having an expert in the field give a talk on how to increase the diversity in our projects. As I mentioned earlier, the social dinner was on a boat cruising along the Seine. I had a chance to talk to community members in a more social environment. With the conference being in France, we had a lot of first time attendees from France. I enjoyed talking to many of them, as well as other people I only get to see at the European conferences. Sunday was full of more presentations and conversations. During the closing session, I gave a short talk on the Foundation and the work we are doing. Then, Benedict Reuschling, Board Vice President, came up and gave out recognition awards to four FreeBSD contributors who have made an impact on the Project. News Roundup Playing with the pine64 (https://chown.me/blog/playing-with-the-pine64.html) Daniel Jakots writes in his blog about his experiences with his two pine64 boards: Finding something to install on it 6 weeks ago, I ordered two pine64 units. I didn't (and still don't) have much plan for them, but I wanted to play with some cheap boards. I finally received them this week. Initially I wanted to install some Linux stuff on it, I didn't have much requirement so I thought I would just look what seems to be easy and/or the best supported systemd flavour. I headed over their wiki. Everything seems either not really maintained, done by some random people or both. I am not saying random people do bad things, just that installing some random things from the Internet is not really my cup of tea. I heard about Armbian (https://www.armbian.com/pine64/) but the server flavour seems to be experimental so I got scared of it. And sadly, the whole things looks like to be alot undermanned. So I went for OpenBSD because I know the stuff and who to har^Wkindly ask for help. Spoiler alert, it's boring because it just works. Getting OpenBSD on it I downloaded miniroot62.fs, dd'ed it on the micro SD card. I was afraid I'd need to fiddle with some things like sysutils/dtb because I don't know what I would have needed to do. That's because I don't know what it does and for this precise reason I was wrong and I didn't need to do anything. So just dd the miniroot62.fs and you can go to next checkpoint. I plugged an HDMI cable, ethernet cable and the power, it booted, I could read for 10 seconds but then it got dark. Of course it's because you need a serial console. Of course I didn't have one. I thought about trying to install OpenBSD blindly, I could have probably succeeded with autoinstall buuuuuut… Following some good pieces of advice from OpenBSD people I bought some cp2102 (I didn't try to understand what it was or what were the other possibilities, I just wanted something that would work :D). I looked how to plug the thing. It appears you can plug it on two different places but if you plug it on the Euler bus it could power a bit the board so if you try to reboot it, it would then mess with the power disruption and could lead a unclean reboot. You just need to plug three cables: GND, TXD and RXD. Of course, the TXD goes on the RXD pin from the picture and the RXD goes on the TXD pin. Guess why I'm telling you that! That's it Then you can connect with the usual $ cu -dl /dev/cuaU0 -s 115200 What's the point of Docker on FreeBSD or Solaris? (http://blog.frankleonhardt.com/2017/whats-the-point-of-docker-on-freebsd-or-solaris/) Penguinisters are very keen on their docker, but for the rest of us it may be difficult to see what the fuss is all about – it's only been around a few years and everyone's talking about it. And someone asked again today. What are we missing? Well docker is a solution to a Linux (and Windows) problem that FreeBSD/Solaris doesn't have. Until recently, the Linux kernel only implemented the original user isolation model involving chroot. More recent kernels have had Control Groups added, which are intended to provide isolation for a group of processes (namespaces). This came out of Google, and they've extended to concept to include processor resource allocation as one of the knobs, which could be a good idea for FreeBSD. The scheduler is aware of the JID of the process it's about to schedule, and I might take a look in the forthcoming winter evenings. But I digress. So if isolation (containerisation in Linux terms) is in the Linux kernel, what is Docker bringing to the party? The only thing I can think of is standardisation and an easy user interface (at the expense of having Python installed). You might think of it in similar terms to ezjail – a complex system intended to do something that is otherwise very simple. To make a jail in FreeBSD all you need do is copy the files for your system to a directory. This can even be a whole server's system disk if you like, and jails can run inside jails. You then create a very simple config file, giving the jail a name, the path to your files and an what IP addresses to pass through (if any) and you're done. Just type “service jail nameofjal start”, and off it goes. Is there any advantage in running Docker? Well, in a way, there is. Docker has a repository of system images that you can just install and run, and this is what a lot of people want. They're a bit like virtual appliances, but not mind-numbingly inefficient. You can actually run docker on FreeBSD. A port was done a couple of years ago, but it relies on the 64-bit Linux emulation that started to appear in 10.x. The newer the version of FreeBSD the better. Docker is in ports/sysutils/docker-freebsd. It makes uses of jails instead of Linux cgroups, and requires ZFS rather than UFS for file system isolation. I believe the Linux version uses Union FS but I could be completely wrong on that. The FreeBSD port works with the Docker hub repository, giving you access to thousands of pre-packaged system images to play with. And that's about as far as I've ever tested it. If you want to run the really tricky stuff (like Windows) you probably want full hardware emulation and something like Xen. If you want to deploy or migrate FreeBSD or Solaris systems, just copy a new tarball in to the directory and go. It's a non-problem, so why make it more complicated? Given the increasing frequency Docker turns up in conversations, it's probably worth taking seriously as Linux applications get packaged up in to images for easy access. Jails/Zones may be more efficient, and Docker images are limited to binary, but convenience tends to win in many environments. Network Manager Control for OpenBSD (http://www.vincentdelft.be/post/post_20171023) I propose you a small script allowing you to easily manage your networks connections. This script is integrated within the openbox dynamic menus. Moreover, it allow you to automatically have the connections you have pre-defined based. I was frustrated to not be able to swap quickly from one network interface to an another, to connect simply and quickly to my wifi, to my cable connection, to the wifi of a friend, ... Every time you have to type the ifconfig commands, .... This is nice, but boring. Surely, when you are in a middle of a presentation and you just want a quick connection to your mobile in tethering mode. Thanks to OpenBSD those commands are not so hard, but this frustrate me to not be able to do it with one click. Directly from my windows environment. Since I'm using Openbox, from a menu of openbox. So, I've looked around to see what is currently existing. One tool I've found was netctl (https://github.com/akpoff/netctl). The idea is to have a repository of hostname.if files ready to use for different cases. The idea sounds great, but I had some difficulties to use it. But what annoys me the most, is that it modify the current hostname.if files in /etc. To my eyes, I would avoid to modify those files because they are my working basis. I want to rely on them and make sure that my network will be back to a normal mode after a reboot. Nevertheless, if I've well understood netctl, you have a feature where it will look for the predefined network config matching the environment where you are. Very cool. So, after having played with netctl, look for alternative on internet, I've decided to create nmctl. A small python script which just perform the mandatory network commands. 1. nmctl: a Network Manager Control tool for OpenBSD Nmctl a small tool that allow you to manage your network connections. Why python ? Just because it's the easiest programming language for me. But I should maybe rewrite it in shell, more standard in the OpenBSD world than python. 1.1. download and install I've put nmctl on my sourceforge account here (https://sourceforge.net/p/nmctl/code/ci/master/tree/) You can dowload the last version here (https://sourceforge.net/p/nmctl/code/ci/master/tarball) To install you just have to run: make install (as root) The per-requists are: - having python2.7 installed - Since nmctl must be run as root, I strongly recommend you to run it via doas (http://man.openbsd.org/doas.conf.5). 1.2. The config file First you have to create a config and store it in /etc/nmctl.conf. This file must respect few rules: Each block must starts with a line having the following format: ''':''' Each following lines must start by at least one space. Those lines have more or less the same format as for hostname.if. You have to create a block with the name "open". This will be used to establish a connection to the Open Wifi around you (in restaurant for example) The order of those elements is important. In case you use the -restart option, nmctl will try each of those network configs one after one until it can ping www.google.com. (if you wan to ping something else, you can change it in the python script if you want). You can use external commands. Just preced them with the "!". You have macors. Macros allow you to perform some actions. The 2 currently implemented are '''''' and ''''''. You can use keywords. Currently the only one implemented is "dhcp" Basically you can put all commands that nmctl will apply to the interface to which those commands are referring to. So, you will always have "ifconfig ". Check the manpage of ifconfig to see how flexible command is. You have currently 2 macros: - which refers to the "nwid " when you select an Open Wifi with the -open option of nmctl. - is a macro generating a random mac address. This is useful test a dhcp server for example. The keyword "dhcp" will trigger a command like "dhclient ". 1.3. Config file sample. Let me show you one nmctl.conf example. It speaks by itself. ``` # the name open is required for Open wifi. # this is the interface that nmctl will take to establish a connection # We must put the macro . This is where nmctl will put the nwid command # and the selected openwifi selected by the parameter --open open:iwn0 !route flush -wpa dhcp cable:em0 !route flush dhcp lgg4:iwn0 !route flush nwid LGG4s_8114 wpakey aanotherpassword dhcp home:iwn0 !route flush nwid Linksys19594 wpakey apassword dhcp college:iwn0 !route flush nwid john wpakey haahaaaguessme dhcp cable_fixip:em0 !route flush inet 192.168.3.3 netmask 255.255.255.0 !route add -host default 192.168.3.1 # with this network interface I'm using the macro # which will do what you guess it will do :-) cable_random:em0 !route flush lladdr dhcp ``` In this config we have several cable's networks associated with my interface "em0" and several wifi networks associated with my wireless interface "iwn0". You see that you can switch from dhcp, to fixed IP and even you can play with the random mac address macro. Thanks to the network called "open", you can connect to any open wifi system. To do that, just type ''' nmctl --open ''' So, now, with just one command you can switch from one network configuration to an another one. That's become cool :-). 2. Integration with openbox Thanks to the dynamic menu feature of oenbox[sic], you can have your different pre-defined networks under one click of your mouse. For that, you just have to add, at the most appropriate place for you, the following code in your ./config/openbox/menu.xml In this case, you see the different networks as defined in the config file just above. 3. Automatically identify your available connection and connect to it in one go But the most interesting part, is coming from a loop through all of your defined networks. This loop is reachable via the -restart option. Basically the idea is to loop from the first network config to the last and test a ping for each of them. Once the ping works, we break the loop and keep this setting. Thus where ever you are, you just have to initiate a nmctl -restart and you will be connected to the network you have defined for this place. There is one small exception, the open-wifis. We do not include them in this loop exercise. Thus the way you define your config file is important. Since the network called "open" is dedicated to "open wifi", it will not be part of this scan exercise. I propose you keep it at the first place. Then, in my case, if my mobile, called lgg4, is open and visible by my laptop, I will connect it immediately. Second, I check if my "home wifi" is visible. Third, if I have a cable connected on my laptop, I'm using this connection and do a dhcp command. Then, I check to see if my laptop is not viewing the "college" wifi. ? and so on until a ping command works. If you do not have a cable in your laptop and if none of your pre-defined wifi connections are visible, the scan will stop. 3.1 examples No cable connected, no pre-defined wifi around me: t420:~$ time doas nmctl -r nwids around you: bbox2-d954 0m02.97s real 0m00.08s user 0m00.11s system t420:~$ t420:~$ I'm at home and my wifi router is running: ``` t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 ifconfig em0 down: 0 default fw done fw 00:22:4d:ac:30:fd done nas link#2 done route flush: 0 ifconfig iwn0 nwid Linksys19594 ...: 0 iwn0: no link ........... sleeping dhclient iwn0: 0 Done. PING www.google.com (216.58.212.164): 56 data bytes 64 bytes from 216.58.212.164: icmp_seq=0 ttl=52 time=12.758 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 12.758/12.758/12.758/0.000 ms ping -c1 -w2 www.google.com: 0 0m22.49s real 0m00.08s user 0m00.11s system t420:~$ ``` I'm at home but tethering is active on my mobile: ``` t420:~$ t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 LGG4s8114 ifconfig em0 down: 0 default fw done fw 00:22:4d:ac:30:fd done nas link#2 done route flush: 0 ifconfig iwn0 nwid LGG4s8114 ...: 0 iwn0: DHCPDISCOVER - interval 1 iwn0: DHCPDISCOVER - interval 2 iwn0: DHCPOFFER from 192.168.43.1 (a0:91:69:be:10:49) iwn0: DHCPREQUEST to 255.255.255.255 iwn0: DHCPACK from 192.168.43.1 (a0:91:69:be:10:49) iwn0: bound to 192.168.43.214 -- renewal in 1800 seconds dhclient iwn0: 0 Done. ping: Warning: www.google.com has multiple addresses; using 173.194.69.99 PING www.google.com (173.194.69.99): 56 data bytes 64 bytes from 173.194.69.99: icmp_seq=0 ttl=43 time=42.863 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 42.863/42.863/42.863/0.000 ms ping -c1 -w2 www.google.com: 0 0m13.78s real 0m00.08s user 0m00.13s system t420:~$ ``` Same situation, but I cut the tethering just after the scan. Thus the dhcp command will not succeed. We see that, after timeouts, nmctl see that the ping is failing (return code 1), thus he pass to the next possible pre-defined network. ``` t420:~$ time doas nmctl -r nwids around you: Linksys19594 bbox2-d954 LGG4s8114 ifconfig em0 down: 0 default 192.168.43.1 done 192.168.43.1 a0:91:69:be:10:49 done route flush: 0 ifconfig iwn0 nwid LGG4s8114 ...: 0 iwn0: no link ........... sleeping dhclient iwn0: 0 Done. ping: no address associated with name ping -c1 -w2 www.google.com: 1 ifconfig em0 down: 0 192.168.43.1 link#2 done route flush: 0 ifconfig iwn0 nwid Linksys19594 ...: 0 iwn0: DHCPREQUEST to 255.255.255.255 iwn0: DHCPACK from 192.168.3.1 (00:22:4d:ac:30:fd) iwn0: bound to 192.168.3.16 -- renewal in 302400 seconds dhclient iwn0: 0 Done. PING www.google.com (216.58.212.164): 56 data bytes 64 bytes from 216.58.212.164: icmp_seq=0 ttl=52 time=12.654 ms --- www.google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 12.654/12.654/12.654/0.000 ms ping -c1 -w2 www.google.com: 0 3m34.85s real 0m00.17s user 0m00.20s system t420:~$ ``` OpenVPN Setup Guide for FreeBSD (https://www.c0ffee.net/blog/openvpn-guide) OpenVPN Setup Guide Browse securely from anywhere using a personal VPN with OpenVPN, LDAP, FreeBSD, and PF. A VPN allows you to securely extend a private network over the internet via tunneling protocols and traffic encryption. For most people, a VPN offers two primary features: (1) the ability to access services on your local network over the internet, and (2) secure internet connectivity over an untrusted network. In this guide, I'll describe how to set up a personal VPN using OpenVPN on FreeBSD. The configuration can use both SSL certificates and LDAP credentials for authentication. We'll also be using the PF firewall to NAT traffic from our VPN out to the internet. One important note about running your own VPN: since you are most likely hosting your server using a VPS or hosting provider, with a public IP address allocated specifically to you, your VPN will not give you any extra anonymity on the internet. If anything, you'll be making yourself more of a target, since all your activity can be trivially traced back to your server's IP address. So while your VPN will protect you from a snooping hacker on the free WiFi at Starbucks, it won't protect you from a federal investigation. This guide assumes you are running FreeBSD with the PF firewall. If you're using a different Unix flavor, I'll probably get you most of the way there—but you'll be on your own when configuring your firewall and networking. Finally, I've used example.com and a non-routable public IP address for all the examples in this guide. You'll need to replace them with your own domain name and public IP address. Beastie Bits BSDCan 2017 videos (https://www.youtube.com/channel/UCuQhwHMJ0yK2zlfyRr1XZ_Q/feed) Getting started with OpenBSD device driver development PDF (https://www.openbsd.org/papers/eurobsdcon2017-device-drivers.pdf) AWS CloudWatch Logs agent for FreeBSD (https://macfoo.wordpress.com/2017/10/27/aws-cloudwatch-logs-agent-for-freebsd/) FreeBSD Foundation November 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/november-2017-development-projects-update/) Schedule for the BSD Devroom at FOSDEM 2018 (https://fosdem.org/2018/schedule/track/bsd/) *** Feedback/Questions Matt - The show and Cantrill (http://dpaste.com/35VNXR5#wrap) Paulo - FreeBSD Question (http://dpaste.com/17E9Z2W#wrap) Steven - Virtualization under FreeBSD (http://dpaste.com/1N6F0TC#wrap) ***
We take a look at two-faced Oracle, cover a FAMP installation, how Netflix works the complex stuff, and show you who the patron of yak shaving is. This episode was brought to you by Headlines Why is Oracle so two-faced over open source? (https://www.theregister.co.uk/2017/10/12/oracle_must_grow_up_on_open_source/) Oracle loves open source. Except when the database giant hates open source. Which, according to its recent lobbying of the US federal government, seems to be "most of the time". Yes, Oracle has recently joined the Cloud Native Computing Foundation (CNCF) to up its support for open-source Kubernetes and, yes, it has long supported (and contributed to) Linux. And, yes, Oracle has even gone so far as to (finally) open up Java development by putting it under a foundation's stewardship. Yet this same, seemingly open Oracle has actively hammered the US government to consider that "there is no math that can justify open source from a cost perspective as the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." That punch to the face was delivered in a letter to Christopher Liddell, a former Microsoft CFO and now director of Trump's American Technology Council, by Kenneth Glueck, Oracle senior vice president. The US government had courted input on its IT modernisation programme. Others writing back to Liddell included AT&T, Cisco, Microsoft and VMware. In other words, based on its letter, what Oracle wants us to believe is that open source leads to greater costs and poorly secured, limply featured software. Nor is Oracle content to leave it there, also arguing that open source is exactly how the private sector does not function, seemingly forgetting that most of the leading infrastructure, big data, and mobile software today is open source. Details! Rather than take this counterproductive detour into self-serving silliness, Oracle would do better to follow Microsoft's path. Microsoft, too, used to Janus-face its way through open source, simultaneously supporting and bashing it. Only under chief executive Satya Nadella's reign did Microsoft realise it's OK to fully embrace open source, and its financial results have loved the commitment. Oracle has much to learn, and emulate, in Microsoft's approach. I love you, you're perfect. Now change Oracle has never been particularly warm and fuzzy about open source. As founder Larry Ellison might put it, Oracle is a profit-seeking corporation, not a peace-loving charity. To the extent that Oracle embraces open source, therefore it does so for financial reward, just like every other corporation. Few, however, are as blunt as Oracle about this fact of corporate open-source life. As Ellison told the Financial Times back in 2006: "If an open-source product gets good enough, we'll simply take it. So the great thing about open source is nobody owns it – a company like Oracle is free to take it for nothing, include it in our products and charge for support, and that's what we'll do. "So it is not disruptive at all – you have to find places to add value. Once open source gets good enough, competing with it would be insane... We don't have to fight open source, we have to exploit open source." "Exploit" sounds about right. While Oracle doesn't crack the top-10 corporate contributors to the Linux kernel, it does register a respectable number 12, which helps it influence the platform enough to feel comfortable building its IaaS offering on Linux (and Xen for virtualisation). Oracle has also managed to continue growing MySQL's clout in the industry while improving it as a product and business. As for Kubernetes, Oracle's decision to join the CNCF also came with P&L strings attached. "CNCF technologies such as Kubernetes, Prometheus, gRPC and OpenTracing are critical parts of both our own and our customers' development toolchains," said Mark Cavage, vice president of software development at Oracle. One can argue that Oracle has figured out the exploitation angle reasonably well. This, however, refers to the right kind of exploitation, the kind that even free software activist Richard Stallman can love (or, at least, tolerate). But when it comes to government lobbying, Oracle looks a lot more like Mr Hyde than Dr Jekyll. Lies, damned lies, and Oracle lobbying The current US president has many problems (OK, many, many problems), but his decision to follow the Obama administration's support for IT modernisation is commendable. Most recently, the Trump White House asked for feedback on how best to continue improving government IT. Oracle's response is high comedy in many respects. As TechDirt's Mike Masnick summarises, Oracle's "latest crusade is against open-source technology being used by the federal government – and against the government hiring people out of Silicon Valley to help create more modern systems. Instead, Oracle would apparently prefer the government just give it lots of money." Oracle is very good at making lots of money. As such, its request for even more isn't too surprising. What is surprising is the brazenness of its position. As Masnick opines: "The sheer contempt found in Oracle's submission on IT modernization is pretty stunning." Why? Because Oracle contradicts much that it publicly states in other forums about open source and innovation. More than this, Oracle contradicts much of what we now know is essential to competitive differentiation in an increasingly software and data-driven world. Take, for example, Oracle's contention that "significant IT development expertise is not... central to successful modernization efforts". What? In our "software is eating the world" existence Oracle clearly believes that CIOs are buyers, not doers: "The most important skill set of CIOs today is to critically compete and evaluate commercial alternatives to capture the benefits of innovation conducted at scale, and then to manage the implementation of those technologies efficiently." While there is some truth to Oracle's claim – every project shouldn't be a custom one-off that must be supported forever – it's crazy to think that a CIO – government or otherwise – is doing their job effectively by simply shovelling cash into vendors' bank accounts. Indeed, as Masnick points out: "If it weren't for Oracle's failures, there might not even be a USDS [the US Digital Service created in 2014 to modernise federal IT]. USDS really grew out of the emergency hiring of some top-notch internet engineers in response to the Healthcare.gov rollout debacle. And if you don't recall, a big part of that debacle was blamed on Oracle's technology." In short, blindly giving money to Oracle and other big vendors is the opposite of IT modernisation. In its letter to Liddell, Oracle proceeded to make the fantastic (by which I mean "silly and false") claim that "the fact is that the use of open-source software has been declining rapidly in the private sector". What?!? This is so incredibly untrue that Oracle should score points for being willing to say it out loud. Take a stroll through the most prominent software in big data (Hadoop, Spark, Kafka, etc.), mobile (Android), application development (Kubernetes, Docker), machine learning/AI (TensorFlow, MxNet), and compare it to Oracle's statement. One conclusion must be that Oracle believes its CIO audience is incredibly stupid. Oracle then tells a half-truth by declaring: "There is no math that can justify open source from a cost perspective." How so? Because "the cost of support plus the opportunity cost of forgoing features, functions, automation and security overwhelm any presumed cost savings." Which I guess is why Oracle doesn't use any open source like Linux, Kubernetes, etc. in its services. Oops. The Vendor Formerly Known As Satan The thing is, Oracle doesn't need to do this and, for its own good, shouldn't do this. After all, we already know how this plays out. We need only look at what happened with Microsoft. Remember when Microsoft wanted us to "get the facts" about Linux? Now it's a big-time contributor to Linux. Remember when it told us open source was anti-American and a cancer? Now it aggressively contributes to a huge variety of open-source projects, some of them homegrown in Redmond, and tells the world that "Microsoft loves open source." Of course, Microsoft loves open source for the same reason any corporation does: it drives revenue as developers look to build applications filled with open-source components on Azure. There's nothing wrong with that. Would Microsoft prefer government IT to purchase SQL Server instead of open-source-licensed PostgreSQL? Sure. But look for a single line in its response to the Trump executive order that signals "open source is bad". You won't find it. Why? Because Microsoft understands that open source is a friend, not foe, and has learned how to monetise it. Microsoft, in short, is no longer conflicted about open source. It can compete at the product level while embracing open source at the project level, which helps fuel its overall product and business strategy. Oracle isn't there yet, and is still stuck where Microsoft was a decade ago. It's time to grow up, Oracle. For a company that builds great software and understands that it increasingly needs to depend on open source to build that software, it's disingenuous at best to lobby the US government to put the freeze on open source. Oracle needs to learn from Microsoft, stop worrying and love the open-source bomb. It was a key ingredient in Microsoft's resurgence. Maybe it could help Oracle get a cloud clue, too. Install FAMP on FreeBSD (https://www.linuxsecrets.com/home/3164-install-famp-on-freebsd) The acronym FAMP refers to a set of free open source applications which are commonly used in Web server environments called Apache, MySQL and PHP on the FreeBSD operating system, which provides a server stack that provides web services, database and PHP. Prerequisites sudo Installed and working - Please read Apache PHP5 or PHP7 MySQL or MariaDB Install your favorite editor, ours is vi Note: You don't need to upgrade FreeBSD but make sure all patches have been installed and your port tree is up-2-date if you plan to update by ports. Install Ports portsnap fetch You must use sudo for each indivdual command during installations. Please see link above for installing sudo. Searching Available Apache Versions to Install pkg search apache Install Apache To install Apache 2.4 using pkg. The apache 2.4 user account managing Apache is www in FreeBSD. pkg install apache24 Confirmation yes prompt and hit y for yes to install Apache 2.4 This installs Apache and its dependencies. Enable Apache use sysrc to update services to be started at boot time, Command below adds "apache24enable="YES" to the /etc/rc.conf file. For sysrc commands please read ```sysrc apache24enable=yes Start Apache service apache24 start``` Visit web address by accessing your server's public IP address in your web browser How To find Your Server's Public IP Address If you do not know what your server's public IP address is, there are a number of ways that you can find it. Usually, this is the address you use to connect to your server through SSH. ifconfig vtnet0 | grep "inet " | awk '{ print $2 }' Now that you have the public IP address, you may use it in your web browser's address bar to access your web server. Install MySQL Now that we have our web server up and running, it is time to install MySQL, the relational database management system. The MySQL server will organize and provide access to databases where our server can store information. Install MySQL 5.7 using pkg by typing pkg install mysql57-server Enter y at the confirmation prompt. This installs the MySQL server and client packages. To enable MySQL server as a service, add mysqlenable="YES" to the /etc/rc.conf file. This sysrc command will do just that ```sysrc mysqlenable=yes Now start the MySQL server service mysql-server start Now run the security script that will remove some dangerous defaults and slightly restrict access to your database system. mysqlsecureinstallation``` Answer all questions to secure your newly installed MySQL database. Enter current password for root (enter for none): [RETURN] Your database system is now set up and we can move on. Install PHP5 or PHP70 pkg search php70 Install PHP70 you would do the following by typing pkg install php70-mysqli mod_php70 Note: In these instructions we are using php5.7 not php7.0. We will be coming out with php7.0 instructions with FPM. PHP is the component of our setup that will process code to display dynamic content. It can run scripts, connect to MySQL databases to get information, and hand the processed content over to the web server to display. We're going to install the modphp, php-mysql, and php-mysqli packages. To install PHP 5.7 with pkg, run this command ```pkg install modphp56 php56-mysql php56-mysqli Copy sample PHP configuration file into place. cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini Regenerate the system's cached information about your installed executable files rehash``` Before using PHP, you must configure it to work with Apache. Install PHP Modules (Optional) To enhance the functionality of PHP, we can optionally install some additional modules. To see the available options for PHP 5.6 modules and libraries, you can type this into your system pkg search php56 Get more information about each module you can look at the long description of the package by typing pkg search -f apache24 Optional Install Example pkg install php56-calendar Configure Apache to Use PHP Module Open the Apache configuration file vim /usr/local/etc/apache24/Includes/php.conf DirectoryIndex index.php index.html Next, we will configure Apache to process requested PHP files with the PHP processor. Add these lines to the end of the file: SetHandler application/x-httpd-php SetHandler application/x-httpd-php-source Now restart Apache to put the changes into effect service apache24 restart Test PHP Processing By default, the DocumentRoot is set to /usr/local/www/apache24/data. We can create the info.php file under that location by typing vim /usr/local/www/apache24/data/info.php Add following line to info.php and save it. Details on info.php info.php file gives you information about your server from the perspective of PHP. It' useful for debugging and to ensure that your settings are being applied correctly. If this was successful, then your PHP is working as expected. You probably want to remove info.php after testing because it could actually give information about your server to unauthorized users. Remove file by typing rm /usr/local/www/apache24/data/info.php Note: Make sure Apache / meaning the root of Apache is owned by user which should have been created during the Apache install is the owner of the /usr/local/www structure. That explains FAMP on FreeBSD. IXsystems IXsystems TrueNAS X10 Torture Test & Fail Over Systems In Action with the ZFS File System (https://www.youtube.com/watch?v=GG_NvKuh530) How Netflix works: what happens every time you hit Play (https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b) Not long ago, House of Cards came back for the fifth season, finally ending a long wait for binge watchers across the world who are interested in an American politician's ruthless ascendance to presidency. For them, kicking off a marathon is as simple as reaching out for your device or remote, opening the Netflix app and hitting Play. Simple, fast and instantly gratifying. What isn't as simple is what goes into running Netflix, a service that streams around 250 million hours of video per day to around 98 million paying subscribers in 190 countries. At this scale, providing quality entertainment in a matter of a few seconds to every user is no joke. And as much as it means building top-notch infrastructure at a scale no other Internet service has done before, it also means that a lot of participants in the experience have to be negotiated with and kept satiated?—?from production companies supplying the content, to internet providers dealing with the network traffic Netflix brings upon them. This is, in short and in the most layman terms, how Netflix works. Let us just try to understand how Netflix is structured on the technological side with a simple example. Netflix literally ushered in a revolution around ten years ago by rewriting the applications that run the entire service to fit into a microservices architecture?—?which means that each application, or microservice's code and resources are its very own. It will not share any of it with any other app by nature. And when two applications do need to talk to each other, they use an application programming interface (API)?—?a tightly-controlled set of rules that both programs can handle. Developers can now make many changes, small or huge, to each application as long as they ensure that it plays well with the API. And since the one program knows the other's API properly, no change will break the exchange of information. Netflix estimates that it uses around 700 microservices to control each of the many parts of what makes up the entire Netflix service: one microservice stores what all shows you watched, one deducts the monthly fee from your credit card, one provides your device with the correct video files that it can play, one takes a look at your watching history and uses algorithms to guess a list of movies that you will like, and one will provide the names and images of these movies to be shown in a list on the main menu. And that's the tip of the iceberg. Netflix engineers can make changes to any part of the application and can introduce new changes rapidly while ensuring that nothing else in the entire service breaks down. They made a courageous decision to get rid of maintaining their own servers and move all of their stuff to the cloud?—?i.e. run everything on the servers of someone else who dealt with maintaining the hardware while Netflix engineers wrote hundreds of programs and deployed it on the servers rapidly. The someone else they chose for their cloud-based infrastructure is Amazon Web Services (AWS). Netflix works on thousands of devices, and each of them play a different format of video and sound files. Another set of AWS servers take this original film file, and convert it into hundreds of files, each meant to play the entire show or film on a particular type of device and a particular screen size or video quality. One file will work exclusively on the iPad, one on a full HD Android phone, one on a Sony TV that can play 4K video and Dolby sound, one on a Windows computer, and so on. Even more of these files can be made with varying video qualities so that they are easier to load on a poor network connection. This is a process known as transcoding. A special piece of code is also added to these files to lock them with what is called digital rights management or DRM?—?a technological measure which prevents piracy of films. The Netflix app or website determines what particular device you are using to watch, and fetches the exact file for that show meant to specially play on your particular device, with a particular video quality based on how fast your internet is at that moment. Here, instead of relying on AWS servers, they install their very own around the world. But it has only one purpose?—?to store content smartly and deliver it to users. Netflix strikes deals with internet service providers and provides them the red box you saw above at no cost. ISPs install these along with their servers. These Open Connect boxes download the Netflix library for their region from the main servers in the US?—?if there are multiple of them, each will rather store content that is more popular with Netflix users in a region to prioritise speed. So a rarely watched film might take time to load more than a Stranger Things episode. Now, when you will connect to Netflix, the closest Open Connect box to you will deliver the content you need, thus videos load faster than if your Netflix app tried to load it from the main servers in the US. In a nutshell… This is what happens when you hit that Play button: Hundreds of microservices, or tiny independent programs, work together to make one large Netflix service. Content legally acquired or licensed is converted into a size that fits your screen, and protected from being copied. Servers across the world make a copy of it and store it so that the closest one to you delivers it at max quality and speed. When you select a show, your Netflix app cherry picks which of these servers will it load the video from> You are now gripped by Frank Underwood's chilling tactics, given depression by BoJack Horseman's rollercoaster life, tickled by Dev in Master of None and made phobic to the future of technology by the stories in Black Mirror. And your lifespan decreases as your binge watching turns you into a couch potato. It looked so simple before, right? News Roundup Moving FreshPorts (http://dan.langille.org/2017/11/15/moving-freshports/) Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice. In preparation for this move, I have: DNS TTL reduced to 60s Posted to Twitter Updated the status page Put the website put in offline mode: What was missed I turned off commit processing on the new server, but I did not do this on the old server. I should have: sudo svc -d /var/service/freshports That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless. The offline code Here is the basic stuff I used to put the website into offline mode. The main points are: header(“HTTP/1.1 503 Service Unavailable”); ErrorDocument 404 /index.php I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code. The dump The database dump just started (Sun Nov 5 17:07:22 UTC 2017). root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump That should take about 30 minutes. I have set a timer to remind me. Total time was: 1464.82 real 1324.96 user 37.22 sys The MD5 is: MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5 It is now: Sun Nov 5 17:34:07 UTC 2017 The rsync The rsync should take about 10-20 minutes. I have already done an rsync of yesterday's dump file. The rsync today should copy over only the deltas (i.e. differences). The rsync started at about Sun Nov 5 17:36:05 UTC 2017 That took 2m9.091s The MD5 matches. The restore The restore should take about 30 minutes. I ran this test yesterday. It is now Sun Nov 5 17:40:03 UTC 2017. $ createdb -T template0 -E SQL_ASCII freshports.testing $ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump Done. real 25m21.108s user 1m57.508s sys 0m15.172s It is now Sun Nov 5 18:06:22 UTC 2017. Insert break here About here, I took a 30 minute break to run an errand. It was worth it. Changing DNS I'm ready to change DNS now. It is Sun Nov 5 19:49:20 EST 2017 Done. And nearly immediately, traffic started. How many misses? During this process, XXXXX requests were declined: $ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log XXXXX That's it, we're done Total elapsed time: 1 hour 48 minutes. There are still a number of things to follow up on, but that was the transfers. The new FreshPorts Server (http://dan.langille.org/2017/11/17/x8dtu-3/) *** Using bhyve on top of CEPH (https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005876.html) Hi, Just an info point. I'm preparing for a lecture tomorrow, and thought why not do an actual demo.... Like to be friends with Murphy :) So after I started the cluster: 5 jails with 7 OSDs This what I manually needed to do to boot a memory stick Start een Bhyve instance rbd --dest-pool rbddata --no-progress import memstick.img memstick rbd-ggate map rbddata/memstick ggate-devvice is available on /dev/ggate1 kldload vmm kldload nmdm kldload iftap kldload ifbridge kldload cpuctl sysctl net.link.tap.uponopen=1 ifconfig bridge0 create ifconfig bridge0 addm em0 up ifconfig ifconfig tap11 create ifconfig bridge0 addm tap11 ifconfig tap11 up load the GGate disk in bhyve bhyveload -c /dev/nmdm11A -m 2G -d /dev/ggate1 FB11 and boot a single from it. bhyve -H -P -A -c 1 -m 2G -l com1,/dev/nmdm11A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap11 -s 4,ahci-hd,/dev/ggate1 FB11 & bhyvectl --vm=FB11 --get-stats Connect to the VM cu -l /dev/nmdm11B And that'll give you a bhyve VM running on an RBD image over ggate. In the installer I tested reading from the bootdisk: root@:/ # dd if=/dev/ada0 of=/dev/null bs=32M 21+1 records in 21+1 records out 734077952 bytes transferred in 5.306260 secs (138341865 bytes/sec) which is a nice 138Mb/sec. Hope the demonstration does work out tomorrow. --WjW *** Donald Knuth - The Patron Saint of Yak Shaves (http://yakshav.es/the-patron-saint-of-yakshaves/) Excerpts: In 2015, I gave a talk in which I called Donald Knuth the Patron Saint of Yak Shaves. The reason is that Donald Knuth achieved the most perfect and long-running yak shave: TeX. I figured this is worth repeating. How to achieve the ultimate Yak Shave The ultimate yak shave is the combination of improbable circumstance, the privilege to be able to shave at your hearts will and the will to follow things through to the end. Here's the way it was achieved with TeX. The recount is purely mine, inaccurate and obviously there for fun. I'll avoid the most boring facts that everyone always tells, such as why Knuth's checks have their own Wikipedia page. Community Shaving is Best Shaving Since the release of TeX, the community has been busy working on using it as a platform. If you ever downloaded the full TeX distribution, please bear in mind that you are downloading the amassed work of over 40 years, to make sure that each and every TeX document ever written builds. We're talking about documents here. But mostly, two big projects sprung out of that. The first is LaTeX by Leslie Lamport. Lamport is a very productive researcher, famous for research in formal methods through TLA+ and also known laying groundwork for many distributed algorithms. LaTeX is based on the idea of separating presentation and content. It is based around the idea of document classes, which then describe the way a certain document is laid out. Think Markdown, just much more complex. The second is ConTeXt, which is far more focused on fine grained layout control. The Moral of the Story Whenever you feel like “can't we just replace this whole thing, it can't be so hard” when handling TeX, don't forget how many years of work and especially knowledge were poured into that system. Typesetting isn't the most popular knowledge around programmers. Especially see it in the context of the space it is in: they can't remove legacy. Ever. That would break documents. TeX is also not a programming language. It might resemble one, but mostly, it should be approached as a typesetting system first. A lot of it's confusing lingo gets much better then. It's not programming lingo. By approaching TeX with an understanding for its history, a lot of things can be learned from it. And yes, a replacement would be great, but it would take ages. In any case, I hope I thoroughly convinced you why Donald Knuth is the Patron Saint of Yak Shaves. Extra Credits This comes out of a enjoyable discussion with [Arne from Lambda Island](https://lambdaisland.com/https://lambdaisland.com/, who listened and said “you should totally turn this into a talk”. Vincent's trip to EuroBSDCon 2017 (http://www.vincentdelft.be/post/post_20171016) My euroBSDCon 2017 Posted on 2017-10-16 09:43:00 from Vincent in Open Bsd Let me just share my feedback on those 2 days spent in Paris for the EuroBSDCon. My 1st BSDCon. I'm not a developer, contributor, ... Do not expect to improve your skills with OpenBSD with this text :-) I know, we are on October 16th, and the EuroBSDCon of Paris was 3 weeks ago :( I'm not quick !!! Sorry for that Arrival at 10h, I'm too late for the start of the key note. The few persons behind a desk welcome me by talking in Dutch, mainly because of my name. Indeed, Delft is a city in Netherlands, but also a well known university. I inform them that I'm from Belgium, and the discussion moves to the fact the Fosdem is located in Brussels. I receive my nice T-shirt white and blue, a bit like the marine T-shirts, but with the nice EuroBSDCon logo. I'm asking where are the different rooms reserved for the BSD event. We have 1 big on the 1st floor, 1 medium 1 level below, and 2 smalls 1 level above. All are really easy to access. In this entrance we have 4 or 5 tables with some persons representing their company. Those are mainly the big sponsors of the event providing details about their activity and business. I discuss a little bit with StormShield and Gandi. On other tables people are selling BSD t-shirts, and they will quickly be sold. "Is it done yet ?" The never ending story of pkg tools In the last Fosdem, I've already hear Antoine and Baptiste presenting the OpenBSD and FreeBSD battle, I decide to listen Marc Espie in the medium room called Karnak. Marc explains that he has rewritten completely the pkg_add command. He explains that, at contrario with other elements of OpenBSD, the packages tools must be backward compatible and stable on a longer period than 12 months (the support period for OpenBSD). On the funny side, he explains that he has his best idea inside his bath. Hackathons are also used to validate some ideas with other OpenBSD developers. All in all, he explains that the most time consuming part is to imagine a good solution. Coding it is quite straightforward. He adds that better an idea is, shorter the implementation will be. A Tale of six motherboards, three BSDs and coreboot After the lunch I decide to listen the talk about Coreboot. Indeed, 1 or 2 years ago I had listened the Libreboot project at Fosdem. Since they did several references to Coreboot, it's a perfect occasion to listen more carefully to this project. Piotr and Katazyba Kubaj explains us how to boot a machine without the native Bios. Indeed Coreboot can replace the bios, and de facto avoid several binaries imposed by the vendor. They explain that some motherboards are supporting their code. But they also show how difficult it is to flash a Bios and replace it by Coreboot. They even have destroyed a motherboard during the installation. Apparently because the power supply they were using was not stable enough with the 3v. It's really amazing to see that open source developers can go, by themselves, to such deep technical level. State of the DragonFly's graphics stack After this Coreboot talk, I decide to stay in the room to follow the presentation of Fran?ois Tigeot. Fran?ois is now one of the core developer of DrangonflyBSD, an amazing BSD system having his own filesystem called Hammer. Hammer offers several amazing features like snapshots, checksum data integrity, deduplication, ... Francois has spent his last years to integrate the video drivers developed for Linux inside DrangonflyBSD. He explains that instead of adapting this code for the video card to the kernel API of DrangonflyBSD, he has "simply" build an intermediate layer between the kernel of DragonflyBSD and the video drivers. This is not said in the talk, but this effort is very impressive. Indeed, this is more or less a linux emulator inside DragonflyBSD. Francois explains that he has started with Intel video driver (drm/i915), but now he is able to run drm/radeon quite well, but also drm/amdgpu and drm/nouveau. Discovering OpenBSD on AWS Then I move to the small room at the upper level to follow a presentation made by Laurent Bernaille on OpenBSD and AWS. First Laurent explains that he is re-using the work done by Antoine Jacoutot concerning the integration of OpenBSD inside AWS. But on top of that he has integrated several other Open Source solutions allowing him to build OpenBSD machines very quickly with one command. Moreover those machines will have the network config, the required packages, ... On top of the slides presented, he shows us, in a real demo, how this system works. Amazing presentation which shows that, by putting the correct tools together, a machine builds and configure other machines in one go. OpenBSD Testing Infrastructure Behind bluhm.genua.de Here Jan Klemkow explains us that he has setup a lab where he is able to run different OpenBSD architectures. The system has been designed to be able to install, on demand, a certain version of OpenBSD on the different available machines. On top of that a regression test script can be triggered. This provides reports showing what is working and what is not more working on the different machines. If I've well understood, Jan is willing to provide such lab to the core developers of OpenBSD in order to allow them to validate easily and quickly their code. Some more effort is needed to reach this goal, but with what exists today, Jan and his colleague are quite close. Since his company is using OpenBSD business, to his eyes this system is a "tit for tat" to the OpenBSD community. French story on cybercrime Then comes the second keynote of the day in the big auditorium. This talk is performed by the colonel of french gendarmerie. Mr Freyssinet, who is head of the Cyber crimes unit inside the Gendarmerie. Mr Freyssinet explains that the "bad guys" are more and more volatile across countries, and more and more organized. The small hacker in his room, alone, is no more the reality. As a consequence the different national police investigators are collaborating more inside an organization called Interpol. What is amazing in his talk is that Mr Freyssinet talks about "Crime as a service". Indeed, more and more hackers are selling their services to some "bad and temporary organizations". Social event It's now time for the famous social event on the river: la Seine. The organizers ask us to go, by small groups, to a station. There is a walk of 15 minutes inside Paris. Hopefully the weather is perfect. To identify them clearly several organizers takes a "beastie fork" in their hands and walk on the sidewalk generating some amazing reactions from some citizens and toursits. Some of them recognize the Freebsd logo and ask us some details. Amazing :-) We walk on small and big sidewalks until a small stair going under the street. There, we have a train station a bit like a metro station. 3 stations later they ask us to go out. We walk few minutes and come in front of a boat having a double deck: one inside, with nice tables and chairs and one on the roof. But the crew ask us to go up, on the second deck. There, we are welcome with a glass of wine. The tour Eiffel is just at few 100 meters from us. Every hour the Eiffel tower is blinking for 5 minutes with thousands of small lights. Brilliant :-) We see also the "statue de la libertee" (the small one) which is on a small island in the middle of the river. During the whole night the bar will be open with drinks and some appetizers, snacks, ... Such walking diner is perfect to talk with many different persons. I've discussed with several persons just using BSD, they are not, like me, deep and specialized developers. One was from Switzerland, another one from Austria, and another one from Netherlands. But I've also followed a discussion with Theo de Raadt, several persons of the FreeBSD foundation. Some are very technical guys, other just users, like me. But all with the same passion for one of the BSD system. Amazing evening. OpenBSD's small steps towards DTrace (a tale about DDB and CTF) On the second day, I decide to sleep enough in order to have enough resources to drive back to my home (3 hours by car). So I miss the 1st presentations, and arrive at the event around 10h30. Lot of persons are already present. Some faces are less "fresh" than others. I decide to listen to Dtrace in OpenBSD. After 10 minutes I am so lost into those too technical explainations, that I decide to open and look at my PC. My OpenBSD laptop is rarely leaving my home, so I've never had the need to have a screen locking system. In a crowded environment, this is better. So I was looking for a simple solution. I've looked at how to use xlock. I've combined it with the /ets/apm/suspend script, ... Always very easy to use OpenBSD :-) The OpenBSD web stack Then I decide to follow the presentation of Michael W Lucas. Well know person for his different books about "Absolute OpenBSD", Relayd", ... Michael talks about the httpd daemon inside OpenBSD. But he also present his integration with Carp, Relayd, PF, FastCGI, the rules based on LUA regexp (opposed to perl regexp), ... For sure he emphasis on the security aspect of those tools: privilege separation, chroot, ... OpenSMTPD, current state of affairs Then I follow the presentation of Gilles Chehade about the OpenSMTPD project. Amazing presentation that, on top of the technical challenges, shows how to manage such project across the years. Gilles is working on OpenSMTPD since 2007, thus 10 years !!!. He explains the different decisions they took to make the software as simple as possible to use, but as secure as possible, too: privilege separation, chroot, pledge, random malloc, ? . The development starts on BSD systems, but once quite well known they received lot of contributions from Linux developers. Hoisting: lessons learned integrating pledge into 500 programs After a small break, I decide to listen to Theo de Raadt, the founder of OpenBSD. In his own style, with trekking boots, shorts, backpack. Theo starts by saying that Pledge is the outcome of nightmares. Theo explains that the book called "Hacking blind" presenting the BROP has worried him since few years. That's why he developed Pledge as a tool killing a process as soon as possible when there is an unforeseen behavior of this program. For example, with Pledge a program which can only write to disk will be immediately killed if he tries to reach network. By implementing Pledge in the +-500 programs present in the "base", OpenBSD is becoming more secured and more robust. Conclusion My first EuroBSDCon was a great, interesting and cool event. I've discussed with several BSD enthusiasts. I'm using OpenBSD since 2010, but I'm not a developer, so I was worried to be "lost" in the middle of experts. In fact it was not the case. At EuroBSDCon you have many different type of enthusiasts BSD's users. What is nice with the EuroBSDCon is that the organizers foresee everything for you. You just have to sit and listen. They foresee even how to spend, in a funny and very cool attitude, the evening of Saturday. > The small draw back is that all of this has a cost. In my case the whole weekend cost me a bit more than 500euro. Based on what I've learned, what I've saw this is very acceptable price. Nearly all presentations I saw give me a valuable input for my daily job. For sure, the total price is also linked to my personal choice: hotel, parking. And I'm surely biased because I'm used to go to the Fosdem in Brussels which cost nothing (entrance) and is approximately 45 minutes of my home. But Fosdem is not the same atmosphere and presentations are less linked to my daily job. I do not regret my trip to EuroBSDCon and will surely plan other ones. Beastie Bits Important munitions lawyering (https://www.jwz.org/blog/2017/10/important-munitions-lawyering/) AsiaBSDCon 2018 CFP is now open, until December 15th (https://2018.asiabsdcon.org/) ZSTD Compression for ZFS by Allan Jude (https://www.youtube.com/watch?v=hWnWEitDPlM&feature=share) NetBSD on Allwinner SoCs Update (https://blog.netbsd.org/tnf/entry/netbsd_on_allwinner_socs_update) *** Feedback/Questions Tim - Creating Multi Boot USB sticks (http://dpaste.com/0FKTJK3#wrap) Nomen - ZFS Questions (http://dpaste.com/1HY5MFB) JJ - Questions (http://dpaste.com/3ZGNSK9#wrap) Lars - Hardening Diffie-Hellman (http://dpaste.com/3TRXXN4) ***
Allan reports on his trip to BSD Taiwan, new versions of Lumina and GhostBSD are here, a bunch of OpenBSD p2k17 hackathon reports. This episode was brought to you by Headlines Allan's Trip Report from BSD Taiwan (https://bsdtw.org/) BSD TW and Taiwan in general was a fun and interesting experience I arrived Thursday night and took the high speed train to Taipei main station, and then got on the Red line subway to the venue. The dorm rooms were on par with BSDCan, except the mattress was better. I spent Friday with a number of other FreeBSD developers doing touristy things. We went to Taipei 101, the world's tallest building from 2004 - 2010. It also features the world's fastest elevator (2004 - 2016), traveling at 60.6 km/h and transporting passengers from the 5th to 89th floor in 37 seconds. We also got to see the “tuned mass damper”, a 660 tonne steel pendulum suspended between the 92nd and 87th floors. This device resists the swaying of the building caused by high winds. There are interesting videos on display beside the damper, of its reaction during recent typhoons and earthquakes. The Taipei 101 building sits just 200 meters from a major fault line. Then we had excellent dumplings for lunch After walking around the city for a few more hours, we retired to a pub to escape the heat of the sunny Friday afternoon. Then came the best part of each day in Taipei, dinner! We continued our efforts to cause a nation wide shortage of dumplings Special thanks to Scott Tsai (https://twitter.com/scottttw) who took detailed notes for each of the presentations Saturday marked the start of the conference: Arun Thomas provided background and then a rundown of what is happening with the RISC-V architecture. Notes (https://docs.google.com/document/d/1yrnhNTHaMDr4DG-iviXN0O9NES9Lmlc7sWVQhnios6g/edit#heading=h.kcm1n3yzl35q) George Neville-Neil talked about using DTrace in distributed systems as an in-depth auditing system (who did what to whom and when). Notes (https://docs.google.com/document/d/1qut6tMVF8NesrGHd6bydLDN-aKBdXMgHx8Vp3_iGKjQ/edit#heading=h.qdghsgk1bgtl) Baptiste Daroussin presented Poudrière image, an extension of everyone's favourite package building system, to build custom images of FreeBSD. There was discussion of making this generate ZFS based images as well, making it mesh very well with my talk the next day. Notes (https://docs.google.com/document/d/1LceXj8IWJeTRHp9KzOYy8tpM00Fzt7fSN0Gw83B9COE/edit#heading=h.incfzi6bnzxr) Brooks Davis presented his work on an API design for a replacement for mmap. It started with a history of address space management in the BSD family of operating systems going all the way back to the beginning. This overview of the feature and how it evolved filled in many gaps for me, and showed why the newer work would be beneficial. The motivation for the work includes further extensions to support the CHERI hardware platform. Notes (https://docs.google.com/document/d/1LceXj8IWJeTRHp9KzOYy8tpM00Fzt7fSN0Gw83B9COE/edit#heading=h.incfzi6bnzxr) Johannes M Dieterich gave an interesting presentation about using FreeBSD and GPU acceleration for high performance computing. One of the slides showed that amd64 has taken almost the entire market for the top 500 super computers, and that linux dominates the list, with only a few remaining non-linux systems. Sadly, at the supercomputing conference the next week, it was announced that linux has achieved 100% saturation of the top 500 super computers list. Johannes detailed the available tools, what ports are missing, what changes should be made to the base system (mostly OpenMP), and generally what FreeBSD needs to do to become a player in the supercomputer OS market. Johannes' perspective is interesting, as he is a computational chemist, not a computer scientist. Those interested in improving the numerical libraries and GPU acceleration frameworks on FreeBSD should join the ports team. Notes (https://docs.google.com/document/d/1uaJiqtPk8WetST6_GnQwIV49bj790qx7ToY2BHC9zO4/edit#heading=h.nvsz1n6w3gyq) The final talk of the day was Peter Grehan, who spoke about how graphics support in bhyve came to be. He provided a history of how the feature evolved, and where it stands today. Notes (https://docs.google.com/document/d/1LqJQJUwdUwWZ0n5KwCH1vNI8jiWGJlI1j0It3mERN80/edit#heading=h.sgeixwgz7bjs) Afterwards, we traveled as a group to a large restaurant for dinner. There was even Mongolian Vodka, provided by Ganbold Tsagaankhuu of the FreeBSD project. Sunday: The first talk of the day Sunday was mine. I presented “ZFS: Advanced Integration”, mostly talking about how boot environments work, and the new libbe and be(1) tools that my GSoC student Kyle Kneitinger created to manage them. I talked about how they can be used for laptop and developer systems, but also how boot environments can be used to replace nanobsd for appliances (as already done in FreeNAS and pfSense). I also presented about zfsbootcfg (zfs nextboot), and some future extensions to it to make it even more useful in appliance type workloads. I also provided a rundown of new developments out of the ZFS developer summit, two weeks previous. Notes (https://docs.google.com/document/d/1Blh3Dulf0O91A0mwv34UnIgxRZaS_0FU2lZ41KRQoOU/edit#heading=h.gypim387e8hy) Theo de Raadt presented “Mitigations and other real Security Features”, and made his case for changing to a ‘fail closed' mode of interoperability. Computer's cannot actually self heal, so lets stop pretending that they can. Notes (https://docs.google.com/document/d/1fFHzlxJjbHPsV9t_Uh3PXZnXmkapAK5RkJsfaHki7kc/edit#heading=h.192e4lmbl70c) Ruslan Bukin talked about doing the port of FreeBSD for RISC-V and writing the Device Drivers. Ruslan walked through the process step by step, leading members of the audience to suggest he turn it into a developer's handbook article, explaining how to do the initial bringup on new hardware. Ruslan also showed off a FreeBSD/MIPS board he designed himself and had manufactured in China. Notes (https://docs.google.com/document/d/1kRhRr3O3lQ-0dS0kYF0oh_S0_zFufEwrdFjG1QLyk8Y/edit#heading=h.293mameym7w1) Mariusz Zaborski presented Case studies on sandboxing the base system with Capsicum. He discussed the challenges encountered as existing programs are modified to sandbox them, and recent advancements in the debugging tools available during that process. Mariusz also discussed the Casper service at length, including the features that are planned for 2018 and onwards. Notes (https://docs.google.com/document/d/1_0BpAE1jGr94taUlgLfSWlJOYU5II9o7Y3ol0ym1eZQ/edit#heading=h.xm9mh7dh6bay) The final presentation of the day was Mark Johnston on Memory Management Improvements in FreeBSD 12.0. This talk provided a very nice overview of the memory management system in FreeBSD, and then detailed some of the recent improvements. Notes (https://docs.google.com/document/d/1gFQXxsHM66GQGMO4-yoeFRTcmOP4NK_ujVFHIQJi82U/edit#heading=h.uirc9jyyti7w) The conference wrapped up with the Work-in-Progress session, including updates on: multi-device-at-once GELI attach, MP-safe networking on NetBSD, pkgsrc, NetBSD in general, BSD on Microsoft Azure, Mothra (send-pr for bugzilla), BSDMizer a machine learning compiler optimizer, Hyperledger Sawtooth (blockchain), and finally VIMAGE and pf testing on FreeBSD. Notes (https://docs.google.com/document/d/1miHZEPrqrpCTh8JONmUKWDPYUmTuG2lbsVrWDtekvLc/edit#heading=h.orhedpjis5po) Group Photo (https://pbs.twimg.com/media/DOh1txnVoAAFKAa.jpg:large) BSDTW was a great conference. They are still considering if it should be an annual thing, trade off every 2nd year with AsiaBSDCon, or something else. In order to continue, BSD Taiwan requires more organizers and volunteers. They have regular meetups in Taipei if you are interested in getting involved. *** Lumina 1.4.0 released (https://lumina-desktop.org/version-1-4-0-released/) The Lumina Theme Engine (and associated configuration utility) The Lumina theme engine is a new component of the “core” desktop, and provides enhanced theming capabilities for the desktop as well as all Qt5 applications. While it started out life as a fork of the “qt5ct” utility, it quickly grew all sorts of new features and functionality such as system-defined color profiles, modular theme components, and built-in editors/creators for all components. The backend of this engine is a standardized theme plugin for the Qt5 toolkit, so that all Qt5 applications will now present a unified appearance (if the application does not enforce a specific appearance/theme of it's own). Users of the Lumina desktop will automatically have this plugin enabled: no special action is required. Please note that the older desktop theme system for Lumina has been rendered obsolete by the new engine, but a settings-conversion path has already been implemented which should transition your current settings to the new engine the first time you login to Lumina 1.4.0. Custom themes for the older system may not be converted though, but it is trivial to copy/paste any custom stylesheets from the old system into the editor for the new theme engine to register/re-apply them as desired. Lumina-Themes Repository I also want to give a shout-out to the trueos/lumina-themes github repository contributors. All of the wallpapers in the 1.4.0 screenshots I posted come from that package, and they are working on making more wallpapers, color palettes, and desktop styles for use with the Lumina Theme Engine. If your operating system does not currently provide a package for lumina-themes, I highly recommend that you make one as soon as possible! The Lumina PDF Viewer (lumina-pdf) This is a new, stand-alone desktop utility for viewing/printing/presenting PDF documents. It uses the poppler-qt5 library in the backend for rendering the document, but uses multi-threading in many ways (such as to speed up the loading of pages) to give the user a nice, streamlined utility for viewing PDF documents. There is also built-in presentation functionality which allows users to easily cast the document to a separate screen without mucking about in system menus or configuration utilities. Lumina PDF Viewer (1.4.0) Important Packaging Changes One significant change of note for people who are packaging Lumina for their particular operating system is that the minimum supported versions of Qt for Lumina have been changed with this release: lumina-core: Qt 5.4+ lumina-mediaplayer: Qt 5.7+ Everything else: Qt 5.2+ Of course, using the latest version of the Qt5 libraries is always recommended. When packaging for Linux distributions, the theme engine also requires the availability of some of the “-dev” packages for Qt itself when compiling the theme plugin. For additional information (specifically regarding Ubuntu builds), please take a look at a recent ticket on the Lumina repository. + The new lumina-pdf utility requires the availability of the “poppler-qt5” library. The includes for this library on Ubuntu 17.10 were found to be installed outside of the normal include directories, so a special rule for it was added to our OS-Detect file in the Lumina source tree. If your particular operating system also places the the poppler include files in a non-standard place, please patch that file or send us the information and we can add more special rules for your particular OS. Other Changes of Note (in no particular order) lumina-config: Add a new page for changing audio theme (login, logout, low battery) Add option to replace fluxbox with some other WM (with appropriate warnings) Have the “themes” page redirect to launching the Lumina theme engine configuration utility. start-lumina-desktop: Auto-detect the active X11 displays and create a new display for the Lumina session (prevent conflict with prior graphical sessions). Add a process-failure counter & restart mechanism. This is particularly useful for restarting Fluxbox from time to time (such as after any monitor addition/removal) lumina-xconfig: Restart fluxbox after making any monitor changes with xrandr. This ensures a more reliable session. Implement a new 2D monitor layout mechanism. This allows for the placement of monitors anywhere in the X/Y plane, with simplification buttons for auto-tiling the monitors in each dimension based on their current location. Add the ability to save/load monitor profiles. Distinguish between the “default” monitor arrangement and the “current” monitor arrangement. Allow the user to set the current arrangement as the new default. lumina-desktop: Completely revamp the icon loading mechanisms so it should auto-update when the theme changes. Speed up the initialization of the desktop quite a bit. Prevent loading/probing files in the “/net/” path for existence (assume they exist in the interest of providing shortcuts). On FreeBSD, these are special paths that actually pause the calling process in order to mount/load a network share before resuming the process, and can cause significant “hangs” in the desktop process. Add the ability to take a directory as a target for the wallpaper. This will open/probe the directory for any existing image files that it can use as a wallpaper and randomly select one. Remove the popup dialog prompting about system updates, and replace it with new “Restart (with updates)” buttons on the appropriate menus/windows instead. If no wallpapers selection is provided, try to use the “lumina-nature” wallpaper directory as the default, otherwise fall back on the original default wallpaper if the “lumina-themes” package is not installed. lumina-open: Make the *.desktop parsing a bit more flexible regarding quoted strings where there should not be any. If selecting which application to use, only overwrite the user-default app if the option is explicitly selected. lumina-fileinfo: Significant cleanup of this utility. Now it can be reliably used for creating/registering XDG application shortcuts. Add a whole host of new ZFS integrations: If a ZFS dataset is being examined, show all the ZFS properties for that dataset. If the file being examined exists within ZFS snapshots, show all the snapshots of the file lumina-fm: Significant use of additional multi-threading. Makes the loading of directories much faster (particularly ones with image files which need thumbnails) Add detection/warning when running as root user. Also add an option to launch a new instance of lumina-fm as the root user. [FreeBSD/TrueOS] Fix up the detection of the “External Devices” list to also list available devices for the autofs system. Fix up some drag and drop functionality. Expose the creation, extraction, and insertion of files into archives (requires lumina-archiver at runtime) Expand the “Open With” option into a menu of application suggestions in addition to the “Other” option which runs “lumina-open” to find an application. Provide an option to set the desktop wallpaper to the selected image file(s). (If the running desktop session is Lumina). lumina-mediaplayer: Enable the ability to playback local video files. (NOTE: If Qt5 is set to use the gstreamer multimedia backend, make sure you have the “GL” plugin installed for smooth video playback). lumina-archiver: Add CLI flags for auto-archive and auto-extract. This allows for programmatic/scriptable interactions with archives. That is not mentioning all of the little bugfixes, performance tweaks, and more that are also included in this release. *** The strongest KASLR, ever? (https://blog.netbsd.org/tnf/entry/the_strongest_kaslr_ever) Re: amd64: kernel aslr support (https://mail-index.netbsd.org/tech-kern/2017/11/14/msg022594.html) So, I did it. Now the kernel sections are split in sub-blocks, and are all randomized independently. See my drawing [1]. What it means in practice, is that Kernel ASLR is much more difficult to defeat: a cache attack will at most allow you to know that a given range is mapped as executable for example, but you don't know which sub-block of .text it is; a kernel pointer leak will at most allow you to reconstruct the layout of one sub-block, but you don't know the layout and address of the remaining blocks, and there can be many. The size and number of these blocks is controlled by the split-by-file parameter in Makefile.amd64. Right now it is set to 2MB, which produces a kernel with ~23 allocatable (ie useful at runtime) sections, which is a third of the total number supported (BTSPACENSEGS = 64). I will probably reduce this parameter a bit in the future, to 1.5MB, or even 1MB. All of that leaves us with about the most advanced KASLR implementation available out there. There are ways to improve it even more, but you'll have to wait a few weeks for that. If you want to try it out you need to make sure you have the latest versions of GENERICKASLR / prekern / bootloader. The instructions are still here, and haven't changed. Initial design As I said in the previous episode, I added in October a Kernel ASLR implementation in NetBSD for 64bit x86 CPUs. This implementation would randomize the location of the kernel in virtual memory as one block: a random VA would be chosen, and the kernel ELF sections would be mapped contiguously starting from there. This design had several drawbacks: one leak, or one successful cache attack, could be enough to reconstruct the layout of the entire kernel and defeat KASLR. NetBSD's new KASLR design significantly improves this situation. New design In the new design, each kernel ELF section is randomized independently. That is to say, the base addresses of .text, .rodata, .data and .bss are not correlated. KASLR is already at this stage more difficult to defeat, since you would need a leak or cache attack on each of the kernel sections in order to reconstruct the in-memory kernel layout. Then, starting from there, several techniques are used to strengthen the implementation even more. Sub-blocks The kernel ELF sections are themselves split in sub-blocks of approximately 1MB. The kernel therefore goes from having: { .text .rodata .data .bss } to having { .text .text.0 .text.1 ... .text.i .rodata .rodata.0 ... .rodata.j ... .data ...etc } As of today, this produces a kernel with ~33 sections, each of which is mapped at a random address and in a random order. This implies that there can be dozens of .text segments. Therefore, even if you are able to conduct a cache attack and determine that a given range of memory is mapped as executable, you don't know which sub-block of .text it is. If you manage to obtain a kernel pointer via a leak, you can at most guess the address of the section it finds itself in, but you don't know the layout of the remaining 32 sections. In other words, defeating this KASLR implementation is much more complicated than in the initial design. Higher entropy Each section is put in a 2MB-sized physical memory chunk. Given that the sections are 1MB in size, this leaves half of the 2MB chunk unused. Once in control, the prekern shifts the section within the chunk using a random offset, aligned to the ELF alignment constraint. This offset has a maximum value of 1MB, so that once shifted the section still resides in its initial 2MB chunk: The prekern then maps these 2MB physical chunks at random virtual addresses; but addresses aligned to 2MB. For example, the two sections in Fig. A will be mapped at two distinct VAs: There is a reason the sections are shifted in memory: it offers higher entropy. If we consider a .text.i section with a 64byte ELF alignment constraint, and give a look at the number of possibilities for the location of the section in memory: The prekern shifts the 1MB section in its 2MB chunk, with an offset aligned to 64 bytes. So there are (2MB-1MB)/(64B)=214 possibilities for the offset. Then, the prekern uses a 2MB-sized 2MB-aligned range of VA, chosen in a 2GB window. So there are (2GB-2MB)/(2MB)=210-1 possibilities for the VA. Therefore, there are 214x(210-1)˜224 possible locations for the section. As a comparison with other systems: OS # of possibilities Linux 2^6 MacOS 2^8 Windows 2^13 NetBSD 2^24 Of course, we are talking about one .text.i section here; the sections that will be mapped afterwards will have fewer location possibilities because some slots will be already occupied. However, this does not alter the fact that the resulting entropy is still higher than that of the other implementations. Note also that several sections have an alignment constraint smaller than 64 bytes, and that in such cases the entropy is even higher. Large pages There is also a reason we chose to use 2MB-aligned 2MB-sized ranges of VAs: when the kernel is in control and initializes itself, it can now use large pages to map the physical 2MB chunks. This greatly improves memory access performance at the CPU level. Countermeasures against TLB cache attacks With the memory shift explained above, randomness is therefore enforced at both the physical and virtual levels: the address of the first page of a section does not equal the address of the section itself anymore. It has, as a side effect, an interesting property: it can mostly mitigate TLB cache attacks. Such attacks operate at the virtual-page level; they will allow you to know that a given large page is mapped as executable, but you don't know where exactly within that page the section actually begins. Strong? This KASLR implementation, which splits the kernel in dozens of sub-blocks, randomizes them independently, while at the same time allowing for higher entropy in a way that offers large page support and some countermeasures against TLB cache attacks, appears to be the most advanced KASLR implementation available publicly as of today. Feel free to prove me wrong, I would be happy to know! WIP Even if it is in a functional state, this implementation is still a work in progress, and some of the issues mentioned in the previous blog post haven't been addressed yet. But feel free to test it and report any issue you encounter. Instructions on how to use this implementation can still be found in the previous blog post, and haven't changed since. See you in the next episode! News Roundup GhostBSD 11.1 Finally Ready and Available! (http://www.ghostbsd.org/11.1_release_announcement) Screenshots (https://imgur.com/a/Mu8xk) After a year of development, testing, debugging and working on our software package repository, we are pleased to announce the release of GhostBSD 11.1 is now available on 64-bit(amd64) architecture with MATE and XFCE Desktop on direct and torrent download. With 11.1 we drop 32-bit i386 supports, and we currently maintain our software packages repository for more stability. What's new on GhostBSD 11.1 GhostBSD software repository Support VMware Workstation Guest Features New UFS full disk mirroring option on the installer New UFS full disk MBR and GPT option on the installer New UFS full disk swap size option on the installer Whisker Menu as default Application menu on XFCE All software developed by GhostBSD is now getting updated ZFS configuration for disk What has been fixed on 11.1? Fix XFCE sound plugin Installer ZFS configuration file setting Installer ZFS setup appears to be incomplete The installer was not listing ZFS disk correctly. The installer The partition list was not deleted when pressing back XFCE and MATE shutdown/suspend/hibernate randomly missing Clicking 'GhostBSD Bugs' item in the Main menu -> 'System Tools' brings up 'Server not found' page XFCE installation - incorrect keyboard layout Locale setting not filling correctly Update Station tray icon The image checksum's, hybrid ISO(DVD, USB) images are available at GhostBSD (http://www.ghostbsd.org/download). *** p2k17 Hackathon Reports p2k17 Hackathon Report: Matthias Kilian on xpdf, haskell, and more (https://undeadly.org/cgi?action=article;sid=20171107034258) p2k17 Hackathon Report: Herzliche grusse vom Berlin (espie@ on mandoc, misc packages progress) (https://undeadly.org/cgi?action=article;sid=20171107185122) p2k17 Hackathon Report: Paul Irofti (pirofti@) on hotplugd(8), math ports, xhci(4) and other kernel advancements (https://undeadly.org/cgi?action=article;sid=20171107225258) p2k17 Hackathon report: Jeremy Evans on ruby progress, postgresql and webdriver work (https://undeadly.org/cgi?action=article;sid=20171108072117) p2k17 Hackathon report: Christian Weisgerber on random devices, build failures and gettext (https://undeadly.org/cgi?action=article;sid=20171109171447) p2k17 Hackathon report: Sebastian Reitenbach on Puppet progress (https://undeadly.org/cgi?action=article;sid=20171110124645) p2k17 Hackathon Report: Anthony J. Bentley on firmware, games and securing pkg_add runs (https://undeadly.org/cgi?action=article;sid=20171110124656) p2k17 Hackathon Report: Landry Breuil on Mozilla things and much more (https://undeadly.org/cgi?action=article;sid=20171113091807) p2k17 Hackathon report: Florian Obser on network stack progress, kernel relinking and more (https://undeadly.org/cgi?action=article;sid=20171113235334) p2k17 Hackathon report: Antoine Jacoutot on ports+packages progress (https://undeadly.org/cgi?action=article;sid=20171120075903) *** TrueOS Talks Tech and Open Source at Pellissippi State (https://www.trueos.org/blog/trueos-talks-tech-open-source-pellissippi-state/) Ken Moore of the TrueOS project presented a talk to the AITP group at Pellissippi State today entitled “It's A Unix(-like) system? An Introduction to TrueOS and Open source”. Joshua Smith of the TrueOS project was also in attendance. We were happy to see a good attendance of about 40 individuals that came to hear more about TrueOS and how we continue to innovate along with the FreeBSD project. Many good questions were raised about development, snapshots, cryptocurrency, and cyber-security. We've included a copy of the slides if you'd like to have a look at the talk on open source. We'd like to offer a sincere thanks to everyone who attended and offer an extended invitation for you to join us at our KnoxBUG group on October 30th @ the iXsystems offices! We hope to see you soon! Open Source Talk – Slideshare PDF (https://web.trueos.org/wp-content/uploads/2017/10/Open-Source-Talk.pdf) KnoxBug - Lumina Rising : Challenging Desktop Orthodoxy (http://knoxbug.org/content/octobers-talk-available-youtube) Ken gave his talk about the new Lumina 2.0 Window Manager that he gave at Ohio LinuxFest 2017 KnoxBUG October 2017 (https://youtu.be/w3ZrqxLTnIU) (OLF 2017) Lumina Rising: Challenging Desktop Orthodoxy (https://www.slideshare.net/beanpole135/olf-2017-lumina-rising-challenging-desktop-orthodoxy) *** Official OpenBSD 6.2 CD set - the only one to be made! (https://undeadly.org/cgi?action=article;sid=20171118190325) Our dear friend Bob Beck (beck@) writes: So, again this release the tradition of making Theo do art has continued! Up for sale by auction to the highest bidder on Ebay is the only OpenBSD 6.2 CD set to be produced. The case and CD's feature the 6.2 artwork, custom drawn and signed by Theo. All proceeds to support OpenBSD Go have a look at the auction As with previous OpenBSD auctions, if you are not the successful bidder, we would like to encourage you to donate the equivalent of you highest bid to the project. The Auction (https://www.ebay.ca/itm/Official-OpenBSD-6-2-CD-Set/253265944606) *** Beastie Bits HAMMER2 userspace on Linux (http://lists.dragonflybsd.org/pipermail/users/2017-October/313646.html) OpenBSD Porting Workshop (now changed to January 3, 2018) (http://www.nycbug.org/index.cgi?action=view&id=10655) Matt Ahrens on when Native Encryption for ZFS will land (https://twitter.com/mahrens1/status/921204908094775296) The first successful build of OpenBSD base system (http://nanxiao.me/en/the-first-successful-build-of-openbsd-base-system/) KnoxBug November Meeting (https://www.meetup.com/KnoxBUG-BSD-Linux-and-FOSS-Users-Unite/events/245291204/) Absolute FreeBSD, 3rd Edition, pre-orders available (https://www.michaelwlucas.com/os/af3e) Feedback/Questions Jon - Jails and Networking (http://dpaste.com/2BEW0HB#wrap) Nathan - bhyve Provisioning (http://dpaste.com/1GHSYJS#wrap) Lian - OpenSSL jumping the Shark (http://dpaste.com/18P8D8C#wrap) Kim - Suggestions (http://dpaste.com/1VE0K9E#wrap) ***
FreeBSD 10.4-RELEASE is here, more EuroBSDcon travel notes, the KRACK attack, ZFS and DTrace on NetBSD, and pfsense 2.4. This episode was brought to you by Headlines FreeBSD 10.4-RELEASE Available (https://www.freebsd.org/releases/10.4R/announce.html) FreeBSD 10.4-RELEASE is out. The FreeBSD Project dedicates the FreeBSD 10.4-RELEASE to the memory of Andrey A. Chernov. Some of the highlights: 10.4-RELEASE is the first FreeBSD release to feature full support for eMMC storage, including eMMC partitions, TRIM and bus speed modes up to HS400. Please note, though, that availability of especially the DDR52, HS200 and HS400 modes requires support in the actual sdhci(4) front-end as well as by the hardware used. Also note, that the SDHCI controller part of Intel® Apollo Lake chipsets is affected by several severe silicon bugs. Apparently, it depends on the particular Apollo Lake platform whether the workarounds in place so far are sufficient to avoid timeouts on attaching sdhci(4) there. Also in case a GPT disk label is used, the fsckffs(8) utility now is able to find alternate superblocks. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for accelerating ipsec(4). Support for the Kaby Lake generation of Intel® i219(4)/ i219(5) devices has been added to the em(4) driver. The em(4) driver is now capable of enabling Wake On LAN (WOL) also for Intel® i217, i218 and i219 chips. Note that stale interface configurations from previous unsuccessful attempts to enable WOL for these devices now will actually take effect. For example, an ifconfig em0 wol activates all WOL variants including wolmcast, which might be undesirable. Support for WOL has been added to the igb(4) driver, which was not able to activate this feature on any device before. The same remark regarding stale WOL configurations as for the em(4) driver applies. Userland coredumps can now trigger events such as generating a human readable crash report via devd(8). This feature is off by default. The firmware shipping with the qlxgbe(4) driver has been updated to version 5.4.66. Additionally, this driver has received some TSO and locking fixes, performance optimizations as well as SYSCTLs providing MAC, RX and TX statistics. Mellanox® ConnectX-4 series adapters are now supported by the newly added mlx5ib(4) driver. OpenSSH received an update to version 7.3p1. GNOME has been updated to version 3.18. Xorg-Server has been updated to version 1.18.4. Check out the full release notes and upgrade your systems to 10.4-RELEASE. Thanks to the FreeBSD Release Engineering Team for their efforts. *** EuroBSDcon 2017: "travel notes" after the conference (https://blog.netbsd.org/tnf/entry/eurobsdcon_2017_travel_notes_after) Leonardo Taccari posted in the NetBSD blog about his experiences at EuroBSDcon 2017: Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings! Thursday (21/09): NetBSD developers dinner Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers! Friday (22/09): NetBSD developers summit On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla. NetBSD on Google Compute Engine -- Benny Siegert (bsiegert) Scripting DDB with Forth -- Valery Ushakov (uwe) News from the version control front -- Jörg Sonnenberger (joerg) Afternoon discussions and dinner After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidentally dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together. Saturday (23/09): First day of conference session and Social Event A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry) Hardening pkgsrc -- Pierre Pronchery (khorben) Reproducible builds on NetBSD -- Christos Zoulas (christos) Social event The social event on Saturday evening took place on a boat that cruised on the Seine river. It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community. + Sunday (24/09): Second day of conference session The school of hard knocks - PT1 -- Sevan Janiyan (sevan) The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil) What's in store for NetBSD 8.0? -- Alistair Crooks (agc) Sunday dinner After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers! Conclusion It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!). I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme committee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference! I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit! A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :) Thank you! *** WiFi Vulnerability in WPA2: KRACK (https://www.krackattacks.com/) “We discovered serious weaknesses in WPA2, a protocol that secures all modern protected Wi-Fi networks. An attacker within range of a victim can exploit these weaknesses using key reinstallation attacks (KRACKs). Concretely, attackers can use this novel attack technique to read information that was previously assumed to be safely encrypted. This can be abused to steal sensitive information such as credit card numbers, passwords, chat messages, emails, photos, and so on. The attack works against all modern protected Wi-Fi networks. Depending on the network configuration, it is also possible to inject and manipulate data. For example, an attacker might be able to inject ransomware or other malware into websites.” “Note that if your device supports Wi-Fi, it is most likely affected. During our initial research, we discovered ourselves that Android, Linux, Apple, Windows, OpenBSD, MediaTek, Linksys, and others, are all affected by some variant of the attacks. For more information about specific products, consult the database of CERT/CC, or contact your vendor.” FreeBSD Advisory (https://www.freebsd.org/security/advisories/FreeBSD-SA-17:07.wpa.asc) As of the date of this recording, a few weeks ahead of when this episode will air, the issue is fixed in FreeBSD 11.0 and 11.1, and a workaround has been provided for 10.3 and 10.4 (install newer wpa_supplicant from ports). A fix for 10.3 and 10.4 is expected soon. They will more than likely be out by time you are watching this. The fix for 10.3 and 10.4 is more complicated because the version of wpasupplicant included in the base system is 2.0, from January 2013, so is nearly 5 years old, so the patches do not apply cleanly. The security team is still considering if it will try to patch 2.0, or just replace the version of wpasupplicant with 2.5 from FreeBSD 11.x. OpenBSD was unwilling to wait when the embargo was extended on this vulnerability and stealth fixed the issue on Aug 30th (https://marc.info/?l=openbsd-cvs&m=150410571407760&w=2) stsp@openbsd.org ‘s Mastodon post (https://mastodon.social/@stsp/98837563531323569) Lobste.rs conversation about flaw and OpenBSD's reaction (https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz) “What happened is that he told me on July 15, and gave a 6 weeks embargo until end of August. We already complained back then that this was way too long and leaving people exposed. Then he got CERT (and, thus, US gov agencies) involved and had to extend the embargo even further until today. At that point we already had the ball rolling and decided to stick to the original agreement with him, and he gave us an agreeing nod towards that as well.” “In this situation, a request for keeping the problem and fix secret is a request to leave our users at risk and exposed to insiders who will potentially use the bug to exploit our users. And we have no idea who the other insiders are. We have to assume that information of this kind leaks and dissipates pretty fast in the security “community”.” “We chose to serve the needs of our users who are the vulnerable people in this drama. I stand by that choice.” As a result of this: “To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.” NetBSD: “patches for the WPA issues in KRACK Attacks were committed Oct 16th to HEAD & are pending pullup to 6/7/8 branches” (http://mail-index.netbsd.org/source-changes/2017/10/16/msg088877.html) As of this recording, Dragonfly appears to use wpa_supplicant 2.1 which they imported in 2014 and has not been touched in over a year (https://github.com/DragonFlyBSD/DragonFlyBSD/commits/master/contrib/wpa_supplicant) *** News Roundup NetBSD - dtrace and ZFS update (https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html) Chuck Silvers writes to the tech-kern mailing list of NetBSD: I've been working on updating netbsd's copy of the dtrace and zfs code to rebase from the existing ancient opensolaris version to a recent freebsd version. most of the freebsd changes are pretty close to what netbsd needs, so that seems like a more useful upstream for us. I have things working well enough now that I want to share the code in preparation for committing. this update improves upon our existing dtrace/zfs code in several ways: picks up all the upstream zfs fixes and enhancements from the last decade zfs now supports mmap on netbsd, so you can run executables stored in zfs dtrace fbt probes can now be used in kernel modules (such as zfs) A patch is provided here: http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012 which needs to be applied using “patch -E” as it adds and removes files. He provides the following summary for the diff: freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for netbsd. a few updates to our copy of freebsd's libproc. build system support for using -fno-omit-frame-pointer everywhere and disabling other compiler optimizations that confuse dtrace. sample kernel config changes for a couple evbarm configs (the ones I tested). module/ksyms enhancements for dtrace integration. genfs API enhancements to support zfs. an option to have mutexes not become no-ops during a panic. uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs). Known issues with the patch include: unloading the zfs module fails even with no zpools imported if you've done much with zfs since it was loaded. there's some refcounting problem that I haven't tracked down yet. the module refcounting for active fbt probes is bogus. currently module refcounting is protected by kernconfig_lock(), but taking that lock down in the bowels of dtrace seems likely to create deadlocks. I plan to do something fancier but haven't gotten to it yet. the dtrace uregs[] stuff is probably still wrong. the CTF typeid overflow problem is still there (more on this below). Unsupported features include: the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar" zvols ZFS ACLs (aka. NFSv4 ACLs) NFS exporting a ZFS file system setting dtrace probes in application code using ZFS as the root fs new crypto hashes SHA512_256, skein, and edonr (the last one is not in freebsd yet either) zio delay injection (used for testing zfs) dtrace support for platforms other than x86 and arm A more detailed description of the CTF typeid overflow is also provided. Check out the full thread with followups and try out the patch if you're on NetBSD. *** pfSense 2.4.0-RELEASE Now Available! (https://www.netgate.com/blog/pfsense-2-4-0-release-now-available.html) Jim Pingle writes about the new release: We are excited to announce the release of pfSense® software version 2.4, now available for new installations and upgrades! pfSense software version 2.4.0 was a herculean effort! It is the culmination of 18 months of hard work by Netgate and community contributors, with over 290 items resolved. According to git, 671 files were changed with a total 1651680 lines added, and 185727 lines deleted. Most of those added lines are from translated strings for multiple language support! + Highlights FreeBSD 11.1-RELEASE as the base Operating System New pfSense installer based on bsdinstall, with support for ZFS, UEFI, and multiple types of partition layouts (e.g. GPT, BIOS) Support for Netgate ARM devices such as the SG-1000 OpenVPN 2.4.x support, which brings features like AES-GCM ciphers, speed improvements, Negotiable Crypto Parameters (NCP), TLS encryption, and dual stack/multihome Translation of the GUI into 13 different languages! For more information on contributing to the translation effort, read our previous blog post and visit the project on Zanata WebGUI improvements, such as a new login page, improved GET/POST CSRF handling, significant improvements to the Dashboard and its AJAX handling Certificate Management improvements including CSR signing and international character support Captive Portal has been rewritten to work without multiple instances of ipfw Important Information: 32-bit x86 and NanoBSD have been deprecated and are not supported on pfSense 2.4. Read the full release notes and let them know how you like the new release. *** OpenBSD changes of note 629 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-629) Use getrusage to measure CPU time in md5 benchmarking. Add guard pages at the end of kernel stacks so overflows don't run into important stuff. This would be useful in FreeBSD, even just to detect the condition. I had all kinds of strange crashes when I was accidently overflowing the stack when working on the initial version of the ZSTD patches before ZSTD gained a working heap mode. Add dwxe driver for ethernet found on Allwinner A64, H3 and H5 SoCs. Fix a regression caused by removal of SIGIO from some devices. In malloc, always delay freeing chunks and change ‘F' option to perform a more extensive check for double free. Change sendsyslog prototype to take a string, since there's little point logging not strings. The config program tries to modify zero initialized variables. Previous versions of gcc were patched to place these in the data segment, instead of the bss, but clang has no such patches. Long long ago, this was the default behavior for compilers, which is why gcc was patched to maintain that existing behavior, but now we want a slightly less unusual toolchain. Fix the underlying issue for now by annotating such variables with a data section attribute. *** t2k17 Hackathon Report: Philip Guenther: locking and libc (https://undeadly.org/cgi?action=article;sid=20170824080132) Next up in our series of t2k17 hackathon reports is this one from Philip Guenther: I showed up at t2k17 with a couple hold-over diffs from e2k17 that weren't stable then and hadn't gotten much better since, so after a red-eye through Chicago I arrived in the hackroom, fired up my laptop and synced trees. Meanwhile, people trickled in and the best part of hackathons, the conversations and "what do you think about this?" chats started. Theo introduced me to Todd Mortimer (mortimer@), who's been hacking on clang to implement RETGUARD for C programs. Over the hackathon we discussed a few loose ends that cropped up and what the correct behavior should be for them as well as the mechanics of avoiding 0xc3 bytes (the RET opcode) embedded in the middle of other multi-byte x86 machine code. Fun stuff. Martin (mpi@) and I had a conversation about the desirability of being able to sleep while holding netlock and pretty much came down on "oof, the scheduler does need work before the underlying issue driving this question can be resolved enough to answer it". :-( After some final hammering I got in an enhancement to pool(9) to let a pool use (sleeping) rwlocks instead of (spinning) mutexes and then immediately used that for the per-CPU pool cache pool as well as the futex pool. Further pools are likely to be converted as well kernel upper-level locking changes are made. Speaking of, a larger diff I had been working on for said upper-level locking was still suffering deadlock issues so I took a stab at narrowing it down to just a lock for the process tree, mostly mirroring the FreeBSD proctreelock. That appears to be holding up much better and I just have some code arrangement issues around sysptrace() before that'll go out for final review. Then most of the way through the week, Bob (beck@) vocally complained that life would be easier for libressl if we had some version of pthread_once() and the pthread mutex routines in libc. This would make some other stuff easier too (c.f. /usr/X11R6/lib/libpthread-stubs.*) and the TIB work over the last couple years has basically eliminated the runtime costs of doing so, so I spent most the rest of the hackathon finding the right place to draw a line through libpthread and move everything on the one side of the line into libc. That code seems pretty stable and the xenocara and ports people seem to like—or at least accept—the effects, so it will almost certainly go in with the next libc bump. Lots of other random conversations, hacking, meals, and beer. Many thanks to Ken (krw@) and local conspirators for another excellent Toronto hackathon! Beastie Bits 2017 NetBSD Foundation Officers (https://blog.netbsd.org/tnf/entry/2017_netbsd_foundation_officers) New BSDMag is out - Military Grade Data Wiping in FreeBSD with BCWipe (https://bsdmag.org/download/military-grade-data-wiping-freebsd-bcwipe/) LibertyBSD 6.1 released (http://libertybsd.net/) *** Feedback/Questions Eddy - EuroBSDCon 2017 video and some help (http://dpaste.com/3WDNV05#wrap) Eric - ZFS monitoring (http://dpaste.com/2RP0S60#wrap) Tom - BSD Hosting (http://dpaste.com/31DGH3J#wrap) ***
We look at how Netflix serves 100 Gbps from an Open Connect Appliance, read through the 2nd quarter FreeBSD status report, show you a freebsd-update speedup via nginx reverse proxy, and customize your OpenBSD default shell. This episode was brought to you by Headlines Serving 100 Gbps from an Open Connect Appliance (https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99) In the summer of 2015, the Netflix Open Connect CDN team decided to take on an ambitious project. The goal was to leverage the new 100GbE network interface technology just coming to market in order to be able to serve at 100 Gbps from a single FreeBSD-based Open Connect Appliance (OCA) using NVM Express (NVMe)-based storage. At the time, the bulk of our flash storage-based appliances were close to being CPU limited serving at 40 Gbps using single-socket Xeon E5–2697v2. The first step was to find the CPU bottlenecks in the existing platform while we waited for newer CPUs from Intel, newer motherboards with PCIe Gen3 x16 slots that could run the new Mellanox 100GbE NICs at full speed, and for systems with NVMe drives. Fake NUMA Normally, most of an OCA's content is served from disk, with only 10–20% of the most popular titles being served from memory (see our previous blog, Content Popularity for Open Connect (https://medium.com/@NetflixTechBlog/content-popularity-for-open-connect-b86d56f613b) for details). However, our early pre-NVMe prototypes were limited by disk bandwidth. So we set up a contrived experiment where we served only the very most popular content on a test server. This allowed all content to fit in RAM and therefore avoid the temporary disk bottleneck. Surprisingly, the performance actually dropped from being CPU limited at 40 Gbps to being CPU limited at only 22 Gbps! The ultimate solution we came up with is what we call “Fake NUMA”. This approach takes advantage of the fact that there is one set of page queues per NUMA domain. All we had to do was to lie to the system and tell it that we have one Fake NUMA domain for every 2 CPUs. After we did this, our lock contention nearly disappeared and we were able to serve at 52 Gbps (limited by the PCIe Gen3 x8 slot) with substantial CPU idle time. After we had newer prototype machines, with an Intel Xeon E5 2697v3 CPU, PCIe Gen3 x16 slots for 100GbE NIC, and more disk storage (4 NVMe or 44 SATA SSD drives), we hit another bottleneck, also related to a lock on a global list. We were stuck at around 60 Gbps on this new hardware, and we were constrained by pbufs. Our first problem was that the list was too small. We were spending a lot of time waiting for pbufs. This was easily fixed by increasing the number of pbufs allocated at boot time by increasing the kern.nswbuf tunable. However, this update revealed the next problem, which was lock contention on the global pbuf mutex. To solve this, we changed the vnode pager (which handles paging to files, rather than the swap partition, and hence handles all sendfile() I/O) to use the normal kernel zone allocator. This change removed the lock contention, and boosted our performance into the 70 Gbps range. As noted above, we make heavy use of the VM page queues, especially the inactive queue. Eventually, the system runs short of memory and these queues need to be scanned by the page daemon to free up memory. At full load, this was happening roughly twice per minute. When this happened, all NGINX processes would go to sleep in vm_wait() and the system would stop serving traffic while the pageout daemon worked to scan pages, often for several seconds. This problem is actually made progressively worse as one adds NUMA domains, because there is one pageout daemon per NUMA domain, but the page deficit that it is trying to clear is calculated globally. So if the vm pageout daemon decides to clean, say 1GB of memory and there are 16 domains, each of the 16 pageout daemons will individually attempt to clean 1GB of memory. To solve this problem, we decided to proactively scan the VM page queues. In the sendfile path, when allocating a page for I/O, we run the pageout code several times per second on each VM domain. The pageout code is run in its lightest-weight mode in the context of one unlucky NGINX process. Other NGINX processes continue to run and serve traffic while this is happening, so we can avoid bursts of pager activity that blocks traffic serving. Proactive scanning allowed us to serve at roughly 80 Gbps on the prototype hardware. Hans Petter Selasky, Mellanox's 100GbE driver developer, came up with an innovative solution to our problem. Most modern NICs will supply an Receive Side Scaling (RSS) hash result to the host. RSS is a standard developed by Microsoft wherein TCP/IP traffic is hashed by source and destination IP address and/or TCP source and destination ports. The RSS hash result will almost always uniquely identify a TCP connection. Hans' idea was that rather than just passing the packets to the LRO engine as they arrive from the network, we should hold the packets in a large batch, and then sort the batch of packets by RSS hash result (and original time of arrival, to keep them in order). After the packets are sorted, packets from the same connection are adjacent even when they arrive widely separated in time. Therefore, when the packets are passed to the FreeBSD LRO routine, it can aggregate them. With this new LRO code, we were able to achieve an LRO aggregation rate of over 2 packets per aggregation, and were able to serve at well over 90 Gbps for the first time on our prototype hardware for mostly unencrypted traffic. So the job was done. Or was it? The next goal was to achieve 100 Gbps while serving only TLS-encrypted streams. By this point, we were using hardware which closely resembles today's 100GbE flash storage-based OCAs: four NVMe PCIe Gen3 x4 drives, 100GbE ethernet, Xeon E5v4 2697A CPU. With the improvements described in the Protecting Netflix Viewing Privacy at Scale blog entry, we were able to serve TLS-only traffic at roughly 58 Gbps. In the lock contention problems we'd observed above, the cause of any increased CPU use was relatively apparent from normal system level tools like flame graphs, DTrace, or lockstat. The 58 Gbps limit was comparatively strange. As before, the CPU use would increase linearly as we approached the 58 Gbps limit, but then as we neared the limit, the CPU use would increase almost exponentially. Flame graphs just showed everything taking longer, with no apparent hotspots. We finally had a hunch that we were limited by our system's memory bandwidth. We used the Intel® Performance Counter Monitor Tools to measure the memory bandwidth we were consuming at peak load. We then wrote a simple memory thrashing benchmark that used one thread per core to copy between large memory chunks that did not fit into cache. According to the PCM tools, this benchmark consumed the same amount of memory bandwidth as our OCA's TLS-serving workload. So it was clear that we were memory limited. At this point, we became focused on reducing memory bandwidth usage. To assist with this, we began using the Intel VTune profiling tools to identify memory loads and stores, and to identify cache misses. Because we are using sendfile() to serve data, encryption is done from the virtual memory page cache into connection-specific encryption buffers. This preserves the normal FreeBSD page cache in order to allow serving of hot data from memory to many connections. One of the first things that stood out to us was that the ISA-L encryption library was using half again as much memory bandwidth for memory reads as it was for memory writes. From looking at VTune profiling information, we saw that ISA-L was somehow reading both the source and destination buffers, rather than just writing to the destination buffer. We realized that this was because the AVX instructions used by ISA-L for encryption on our CPUs worked on 256-bit (32-byte) quantities, whereas the cache line size was 512-bits (64 bytes)?—?thus triggering the system to do read-modify-writes when data was written. The problem is that the the CPU will normally access the memory system in 64 byte cache line-sized chunks, reading an entire 64 bytes to access even just a single byte. After a quick email exchange with the ISA-L team, they provided us with a new version of the library that used non-temporal instructions when storing encryption results. Non-temporals bypass the cache, and allow the CPU direct access to memory. This meant that the CPU was no longer reading from the destination buffers, and so this increased our bandwidth from 58 Gbps to 65 Gbps. At 100 Gbps, we're moving about 12.5 GB/s of 4K pages through our system unencrypted. Adding encryption doubles that to 25 GB/s worth of 4K pages. That's about 6.25 Million mbufs per second. When you add in the extra 2 mbufs used by the crypto code for TLS metadata at the beginning and end of each TLS record, that works out to another 1.6M mbufs/sec, for a total of about 8M mbufs/second. With roughly 2 cache line accesses per mbuf, that's 128 bytes * 8M, which is 1 GB/s (8 Gbps) of data that is accessed at multiple layers of the stack (alloc, free, crypto, TCP, socket buffers, drivers, etc). At this point, we're able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack. However, the goalposts keep moving. We've found that when we use more advanced TCP algorithms, such as RACK and BBR, we are still a bit short of our goal. We have several ideas that we are currently pursuing, which range from optimizing the new TCP code to increasing the efficiency of LRO to trying to do encryption closer to the transfer of the data (either from the disk, or to the NIC) so as to take better advantage of Intel's DDIO and save memory bandwidth. FreeBSD April to June 2017 Status Report (https://www.freebsd.org/news/status/report-2017-04-2017-06.html) FreeBSD Team Reports FreeBSD Release Engineering Team Ports Collection The FreeBSD Core Team The FreeBSD Foundation The Postmaster Team Projects 64-bit Inode Numbers Capability-Based Network Communication for Capsicum/CloudABI Ceph on FreeBSD DTS Updates Kernel Coda revival FreeBSD Driver for the Annapurna Labs ENA Intel 10G Driver Update pNFS Server Plan B Architectures FreeBSD on Marvell Armada38x FreeBSD/arm64 Userland Programs DTC Using LLVM's LLD Linker as FreeBSD's System Linker Ports A New USES Macro for Porting Cargo-Based Rust Applications GCC (GNU Compiler Collection) GNOME on FreeBSD KDE on FreeBSD New Port: FRRouting PHP Ports: Help Improving QA Rust sndio Support in the FreeBSD Ports Collection TensorFlow Updating Port Metadata for non-x86 Architectures Xfce on FreeBSD Documentation Absolute FreeBSD, 3rd Edition Doc Version Strings Improved by Their Absence New Xen Handbook Section Miscellaneous BSD Meetups at Rennes (France) Third-Party Projects HardenedBSD DPDK, VPP, and the future of pfSense @ the DPDK Summit (https://www.pscp.tv/DPDKProject/1dRKZnleWbmKB?t=5h1m0s) The DPDK (Data Plane Development Kit) conference included a short update from the pfSense project The video starts with a quick introduction to pfSense and the company behind it It covers the issues they ran into trying to scale to 10gbps and beyond, and some of the solutions they tried: libuinet, netmap, packet-journey Then they discovered VPP (Vector Packet Processing) The video then covers the architecture of the new pfSense pfSense has launched of EC2, on Azure soon, and will launch support for the new Atom C3000 and Xeon hardware with built-in QAT (Quick-Assist crypto offload) in November The future: 100gbps, MPLS, VXLANs, and ARM64 hardware support *** News Roundup Local nginx reverse proxy cache for freebsd-update (https://wiki.freebsd.org/VladimirKrstulja/Guides/FreeBSDUpdateReverseProxy) Vladimir Krstulja has created this interesting tutorial on the FreeBSD wiki about a freebsd-update reverse proxy cache Either because you're a good netizen and don't want to repeatedly hammer the FreeBSD mirrors to upgrade all your systems, or you want to benefit from the speed of having a local "mirror" (cache, more precisely), running a freebsd update reverse proxy cache with, say, nginx is dead simple. 1. Install nginx somewhere 2. Configure nginx for a subdomain, say, freebsd-update.example.com 3. On all your hosts, in all your jails, configure /etc/freebsd-update.conf for new ServerName And... that's it. Running freebsd-update will use the ServerName domain which is your reverse nginx proxy. Note the comment about using a "nearby" server is not quite true. FreeBSD update mirrors are frequently slow and running such a reverse proxy cache significantly speeds things up. Caveats: This is a simple cache. That means it doesn't consider the files as a whole repository, which in turn means updates to your cache are not atomic. It'd be advised to nuke your cache before your update run, as its point is only to retain the files in a local cache for some short period of time required for all your machines to be updated. ClonOS is a free, open-source FreeBSD-based platform for virtual environment creation and management (https://clonos.tekroutine.com/) The operating system uses FreeBSD's development branch (12.0-CURRENT) as its base. ClonOS uses ZFS as the default file system and includes web-based administration tools for managing virtual machines and jails. The project's website also mentions the availability of templates for quickly setting up new containers and web-based VNC access to jails. Puppet, we are told, can be used for configuration management. ClonOS can be downloaded as a disk image file (IMG) or as an optical media image (ISO). I downloaded the ISO file which is 1.6GB in size. Booting from ClonOS's media displays a text console asking us to select the type of text terminal we are using. There are four options and most people can probably safely take the default, xterm, option. The operating system, on the surface, appears to be a full installation of FreeBSD 12. The usual collection of FreeBSD packages are available, including manual pages, a compiler and the typical selection of UNIX command line utilities. The operating system uses ZFS as its file system and uses approximately 3.3GB of disk space. ClonOS requires about 50MB of active memory and 143MB of wired memory before any services or jails are created. Most of the key features of ClonOS, the parts which set it apart from vanilla FreeBSD, can be accessed through a web-based control panel. When we connect to this control panel, over a plain HTTP connection, using our web browser, we are not prompted for an account name or password. The web-based interface has a straight forward layout. Down the left side of the browser window we find categories of options and controls. Over on the right side of the window are the specific options or controls available in the selected category. At the top of the page there is a drop-down menu where we can toggle the displayed language between English and Russian, with English being the default. There are twelve option screens we can access in the ClonOS interface and I want to quickly give a summary of each one: Overview - this page shows a top-level status summary. The page lists the number of jails and nodes in the system. We are also shown the number of available CPU cores and available RAM on the system. Jail containers - this page allows us to create and delete jails. We can also change some basic jail settings on this page, adjusting the network configuration and hostname. Plus we can click a button to open a VNC window that allows us to access the jail's command line interface. Template for jails - provides a list of available jail templates. Each template is listed with its name and a brief description. For example, we have a Wordpress template and a bittorrent template. We can click a listed template to create a new jail with a vanilla installation of the selected software included. We cannot download or create new templates from this page. Bhyve VMs - this page is very much like the Jails containers page, but concerns the creation of new virtual machines and managing them. Virtual Private Network - allows for the management of subnets Authkeys - upload security keys for something, but it is not clear for what these keys will be used. Storage media - upload ISO files that will be used when creating virtual machines and installing an operating system in the new virtual environment. FreeBSD Bases - I think this page downloads and builds source code for alternative versions of FreeBSD, but I am unsure and could not find any associated documentation for this page. FreeBSD Sources - download source code for various versions of FreeBSD. TaskLog - browse logs of events, particularly actions concerning jails. SQLite admin - this page says it will open an interface for managing a SQLite database. Clicking link on the page gives a file not found error. Settings - this page simply displays a message saying the settings page has not been implemented yet. While playing with ClonOS, I wanted to perform a couple of simple tasks. I wanted to use the Wordpress template to set up a blog inside a jail. I wanted a generic, empty jail in which I could play and run commands without harming the rest of the operating system. I also wanted to try installing an operating system other than FreeBSD inside a Bhyve virtual environment. I thought this would give me a pretty good idea of how quick and easy ClonOS would make common tasks. Conclusions ClonOS appears to be in its early stages of development, more of a feature preview or proof-of-concept than a polished product. A few of the settings pages have not been finished yet, the web-based controls for jails are unable to create jails that connect to the network and I was unable to upload even small ISO files to create virtual machines. The project's website mentions working with Puppet to handle system configuration, but I did not encounter any Puppet options. There also does not appear to be any documentation on using Puppet on the ClonOS platform. One of the biggest concerns I had was the lack of security on ClonOS. The web-based control panel and terminal both automatically login as the root user. Passwords we create for our accounts are ignored and we cannot logout of the local terminal. This means anyone with physical access to the server automatically gains root access and, in addition, anyone on our local network gets access to the web-based admin panel. As it stands, it would not be safe to install ClonOS on a shared network. Some of the ideas present are good ones. I like the idea of jail templates and have used them on other systems. The graphical Bhyve tools could be useful too, if the limitations of the ISO manager are sorted out. But right now, ClonOS still has a way to go before it is likely to be safe or practical to use. Customize ksh display for OpenBSD (http://nanxiao.me/en/customize-ksh-display-for-openbsd/) The default shell for OpenBSD is ksh, and it looks a little monotonous. To make its user-experience more friendly, I need to do some customizations: (1) Modify the “Prompt String” to display the user name and current directory: PS1='$USER:$PWD# ' (2) Install colorls package: pkg_add colorls Use it to replace the shipped ls command: alias ls='colorls -G' (3) Change LSCOLORS environmental variable to make your favorite color. For example, I don't want the directory is displayed in default blue, change it to magenta: LSCOLORS=fxexcxdxbxegedabagacad For detailed explanation of LSCOLORS, please refer manual of colorls. This is my final modification of .profile: PS1='$USER:$PWD# ' export PS1 LSCOLORS=fxexcxdxbxegedabagacad export LSCOLORS alias ls='colorls -G' DragonFly 5 release candidate (https://www.dragonflydigest.com/2017/10/02/20295.html) Commit (http://lists.dragonflybsd.org/pipermail/commits/2017-September/626463.html) I tagged DragonFly 5.0 (commit message list in that link) over the weekend, and there's a 5.0 release candidate for download (http://mirror-master.dragonflybsd.org/iso-images/). It's RC2 because the recent Radeon changes had to be taken out. (http://lists.dragonflybsd.org/pipermail/commits/2017-September/626476.html) Beastie Bits Faster forwarding (http://www.grenadille.net/post/2017/08/21/Faster-forwarding) DRM-Next-Kmod hits the ports tree (http://www.freshports.org/graphics/drm-next-kmod/) OpenBSD Community Goes Platinum (https://undeadly.org/cgi?action=article;sid=20170829025446) Setting up iSCSI on TrueOS and FreeBSD12 (https://www.youtube.com/watch?v=4myESLZPXBU) *** Feedback/Questions Christopher - Virtualizing FreeNAS (http://dpaste.com/38G99CK#wrap) Van - Tar Question (http://dpaste.com/3MEPD3S#wrap) Joe - Book Reviews (http://dpaste.com/0T623Z6#wrap) ***
We recap EuroBSDcon in Paris, tell the story behind a pf PR, and show you how to do screencasting with OpenBSD. This episode was brought to you by Headlines Recap of EuroBSDcon 2017 in Paris, France (https://2017.eurobsdcon.org) EuroBSDcon was held in Paris, France this year, which drew record numbers this year. With over 300 attendees, it was the largest BSD event I have ever attended, and I was encouraged by the higher than expected number of first time attendees. The FreeBSD Foundation held a board meeting on Wednesday afternoon with the members who were in Paris. Topics included future conferences (including a conference kit we can mail to people who want to represent FreeBSD) and planning for next year. The FreeBSD Devsummit started on Thursday at the beautiful Mozilla Office in Paris. After registering and picking up our conference bag, everyone gathered for a morning coffee with lots of handshaking and greeting. We then gathered in the next room which had a podium with microphone, screens as well as tables and chairs. After developers sat down, Benedict opened the devsummit with a small quiz about France for developers to win a Mogics Power Bagel (https://www.mogics.com/?page_id=3824). 45 developers participated and DES won the item in the end. After introductions and collecting topics of interest from everyone, we started with the Work in Progress (WIP) session. The WIP session had different people present a topic they are working on in 7 minute timeslots. Topics ranged from FreeBSD Forwarding Performance, fast booting options, and a GELI patch under review to attach multiple providers. See their slides on the FreeBSD wiki (https://wiki.freebsd.org/DevSummit/201709). After lunch, the FreeBSD Foundation gave a general update on staff and funding, as well as a more focused presentation about our partnership with Intel. People were interested to hear what was done so far and asked a few questions to the Intel representative Glenn Weinberg. After lunch, developers worked quietly on their own projects. The mic remained open and occasionally, people would step forward and gave a short talk without slides or motivated a discussion of common interest. The day concluded with a dinner at a nice restaurant in Paris, which allowed to continue the discussions of the day. The second day of the devsummit began with a talk about the CAM-based SDIO stack by Ilya Bakulin. His work would allow access to wifi cards/modules on embedded boards like the Raspberry Pi Zero W and similar devices as many of these are using SDIO for data transfers. Next up was a discussion and Q&A session with the FreeBSD core team members who were there (missing only Benno Rice, Kris Moore, John Baldwin, and Baptiste Daroussin, the latter being busy with conference preparations). The new FCP (FreeBSD community proposals) were introduced for those who were not at BSDCan this year and the hows and whys about it. Allan and I were asked to describe our experiences as new members of core and we encouraged people to run for core when the next election happens. After a short break, Scott Long gave an overview of the work that's been started on NUMA (Non-Uniform Memory Architecture), what the goals of the project are and who is working on it. Before lunch, Christian Schwarz presented his work on zrepl, a new ZFS replication solution he developed using Go. This sparked interest in developers, a port was started (https://reviews.freebsd.org/D12462) and people suggested to Christian that he should submit his talk to AsiaBSDcon and BSDCan next year. Benedict had to leave before lunch was done to teach his Ansible tutorial (which was well attended) at the conference venue. There were organized dinners, for those two nights, quite a feat of organization to fit over 100 people into a restaurant and serve them quickly. On Saturday, there was a social event, a river cruise down the Seine. This took the form of a ‘standing' dinner, with a wide selection of appetizer type dishes, designed to get people to walk around and converse with many different people, rather than sit at a table with the same 6-8 people. I talked to a much larger group of people than I had managed to at the other dinners. I like having both dinner formats. We would also like to thank all of the BSDNow viewers who attended the conference and made the point of introducing themselves to us. It was nice to meet you all. The recordings of the live video stream from the conference are available immediately, so you can watch the raw versions of the talks now: Auditorium Keynote 1: Software Development in the Age of Heroes (https://youtu.be/4iR8g9-39LM?t=179) by Thomas Pornin (https://twitter.com/BearSSLnews) Tuning FreeBSD for routing and firewalling (https://youtu.be/4iR8g9-39LM?t=1660) by Olivier Cochard-Labbé (https://twitter.com/ocochardlabbe) My BSD sucks less than yours, Act I (https://youtu.be/4iR8g9-39LM?t=7040) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) My BSD sucks less than yours, Act II (https://youtu.be/4iR8g9-39LM?t=14254) by Antoine Jacoutot (https://twitter.com/ajacoutot) and Baptiste Daroussin (https://twitter.com/_bapt_) Reproducible builds on NetBSD (https://youtu.be/4iR8g9-39LM?t=23351) by Christos Zoulas Your scheduler is not the problem (https://youtu.be/4iR8g9-39LM?t=26845) by Martin Pieuchot Keynote 2: A French story on cybercrime (https://youtu.be/4iR8g9-39LM?t=30540) by Éric Freyssinet (https://twitter.com/ericfreyss) Case studies of sandboxing base system with Capsicum (https://youtu.be/jqdHYEH_BQY?t=731) by Mariusz Zaborski (https://twitter.com/oshogbovx) OpenBSD's small steps towards DTrace (a tale about DDB and CTF) (https://youtu.be/jqdHYEH_BQY?t=6030) by Jasper Lievisse Adriaanse The Realities of DTrace on FreeBSD (https://youtu.be/jqdHYEH_BQY?t=13096) by George Neville-Neil (https://twitter.com/gvnn3) OpenSMTPD, current state of affairs (https://youtu.be/jqdHYEH_BQY?t=16818) by Gilles Chehade (https://twitter.com/PoolpOrg) Hoisting: lessons learned integrating pledge into 500 programs (https://youtu.be/jqdHYEH_BQY?t=21764) by Theo de Raadt Keynote 3: System Performance Analysis Methodologies (https://youtu.be/jqdHYEH_BQY?t=25463) by Brendan Gregg (https://twitter.com/brendangregg) Closing Session (https://youtu.be/jqdHYEH_BQY?t=29355) Karnak “Is it done yet ?” The never ending story of pkg tools (https://youtu.be/1hjzleqGRYk?t=71) by Marc Espie (https://twitter.com/espie_openbsd) A Tale of six motherboards, three BSDs and coreboot (https://youtu.be/1hjzleqGRYk?t=7498) by Piotr Kubaj and Katarzyna Kubaj State of the DragonFly's graphics stack (https://youtu.be/1hjzleqGRYk?t=11475) by François Tigeot From NanoBSD to ZFS and Jails – FreeBSD as a Hosting Platform, Revisited (https://youtu.be/1hjzleqGRYk?t=16227) by Patrick M. Hausen Bacula – nobody ever regretted making a backup (https://youtu.be/1hjzleqGRYk?t=20069) by Dan Langille (https://twitter.com/DLangille) Never Lose a Syslog Message (https://youtu.be/qX0BS4P65cQ?t=325) by Alexander Bluhm Running CloudABI applications on a FreeBSD-based Kubernetes cluster (https://youtu.be/qX0BS4P65cQ?t=5647) by Ed Schouten (https://twitter.com/EdSchouten) The OpenBSD web stack (https://youtu.be/qX0BS4P65cQ?t=13255) by Michael W. Lucas (https://twitter.com/mwlauthor) The LLDB Debugger on NetBSD (https://youtu.be/qX0BS4P65cQ?t=16835) by Kamil Rytarowski What's in store for NetBSD 8.0? (https://youtu.be/qX0BS4P65cQ?t=21583) by Alistair Crooks Louxor A Modern Replacement for BSD spell(1) (https://youtu.be/6Nen6a1Xl7I?t=156) by Abhinav Upadhyay (https://twitter.com/abhi9u) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development (https://youtu.be/6Nen6a1Xl7I?t=5874) by Cherry G. Mathew Hardening pkgsrc (https://youtu.be/6Nen6a1Xl7I?t=9343) by Pierre Pronchery (https://twitter.com/khorben) Discovering OpenBSD on AWS (https://youtu.be/6Nen6a1Xl7I?t=14874) by Laurent Bernaille (https://twitter.com/lbernail) OpenBSD Testing Infrastructure Behind bluhm.genua.de (https://youtu.be/6Nen6a1Xl7I?t=18639) by Jan Klemkow The school of hard knocks – PT1 (https://youtu.be/8wuW8lfsVGc?t=276) by Sevan Janiyan (https://twitter.com/sevanjaniyan) 7 years of maintaining firefox, and still looking ahead (https://youtu.be/8wuW8lfsVGc?t=5321) by Landry Breuil Branch VPN solution based on OpenBSD, OSPF, RDomains and Ansible (https://youtu.be/8wuW8lfsVGc?t=12385) by Remi Locherer Running BSD on AWS (https://youtu.be/8wuW8lfsVGc?t=15983) by Julien Simon and Nicolas David Getting started with OpenBSD device driver development (https://youtu.be/8wuW8lfsVGc?t=21491) by Stefan Sperling A huge thanks to the organizers, program committee, and sponsors of EuroBSDCon. Next year, EuroBSDcon will be in Bucharest, Romania. *** The story of PR 219251 (https://www.sigsegv.be//blog/freebsd/PR219251) The actual story I wanted Kristof to tell, the pf bug he fixed at the Essen Hackathon earlier this summer. As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out. It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic. This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included. Obviously not the desired behaviour, but it seems strange. The instances of pf in the jails are supposed to be separate. We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find. In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host. Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either. The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire. When rules are removed pfunlinkrule() adds then to the Vpfunlinkedrules list (more on that funny V prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we've got a lot of states, so pfpurgethread() breaks that up into smaller chuncks, iterating only part of the state table on every run. We iterate over all of our virtual pf instances (VNETFOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pfpurgeexpiredstates(). We start at state 'idx' and only process a certain number (determined by the PFTMINTERVAL setting) states. The pfpurgeexpiredstates() function returns a new idx value to tell us how far we got. So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNETSET() or CURVNETRESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many Vpfvnetactive variables as there are vnets: number of vnet jails plus one (for the host system). Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on. That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pfpurgeunlinkedrules() runs it can end up free()ing rules that actually are still in use because pfpurgethread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule. That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, Vpfpurge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple. As you might expect, finding the problem takes a lot more work that fixing it Thanks to Kristof for writing up this detailed post explaining how the problem was found, and what caused it. *** vBSDcon 2017: BSD at Work (https://www.ixsystems.com/blog/vbsdcon-2017-dexter/) The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems' MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn't found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts. The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand. The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments. The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam's accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford's talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford's real-world experiences are perfectly inline with vBSDcon's goal to be more production-oriented than other BSDCons. Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil's did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon's consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies. BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas' latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone! News Roundup FreeBSD 10.4-RC2 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087848.html) FreeBSD 10.4 will be released soon, this is the last chance to find bugs before the official release is cut. Noteworthy Changes Since 10.4-RC1: Given that the amd64 disc1 image was overflowing, more of the base components installed into the disc1 (live) file systems had to be disabled. Most notably, this removed the compiler toolchain from the disc1 images. All disabled tools are still available with the dvd1 images, though. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for ipsec(4). Support for netmap(4) by the ixgbe(4) driver has been brought into line with the netmap(4) API present in stable/10. Also, ixgbe(4) now correctly handles VFs in its netmap(4) support again instead of treating these as PFs. During the creation of amd64 and i386 VM images, etcupdate(8) and mergemaster(8) databases now are bootstrapped, akin to what happens along the extraction of base.txz as part of a new installation via bsdinstall(8). This change allows for both of these tools to work out-of-box on the VM images and avoids errors seen when upgrading these images via freebsd-update(8). If you are still on the stable/10 branch, you should test upgrading to 10.4, and make sure there are no problems with your workload Additional testing specifically of the features that have changed since 10.4-BETA1 would also be most helpful This will be the last release from the stable/10 branch *** OpenBSD changes of note 628 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-628) EuroBSDCon in two weeks. Be sure to attend early and often. Many and various documentation improvements for libcrypto. New man pages, rewrites, expanded bugs sections, and more. Only allow upward migration in vmd. There's a README for the syspatch build system if you want to run your own. Move the kernel relinking code from /etc/rc into a seperate script usable by syspatch. Kernel patches can now be reduced to just the necessary files. Make the callers of sogetopt() responsible for allocating memory. Now allocation and free occur in the same place. Use waitpid() instead of wait() in most programs to avoid accidentally collecting the wrong child. Have cu call isatty() before making assumptions. Switch mandoc rendering of mathematical symbols and greek letters from trying to imitate the characters' graphical shapes, which resulted in unintelligible renderings in many cases, to transliterations conveying the characters' meanings. Update libexpat to 2.2.4. Fix copying partial UTF-8 characters. Sigh, here we go again. Work around bug in F5's handling of the supported elliptic curves extension. RFC 4492 only defines elliptic_curves for ClientHello. However, F5 is sending it in ServerHello. We need to skip over it since our TLS extension parsing code is now more strict. After a first install, run syspatch -c to check for patches. If SMAP is present, clear PSL_AC on kernel entry and interrupt so that only the code in copy{in,out}* that need it run with it set. Panic if it's set on entry to trap() or syscall(). Prompted by Maxime Villard's NetBSD work. Errata. New drivers for arm: rktemp, mvpinctrl, mvmpic, mvneta, mvmdio, mvpxa, rkiic, rkpmic. No need to exec rm from within mandoc. We know there's exactly one file and directory to remove. Similarly with running cmp. Revert to Mesa 13.0.6 to hopefully address rendering issues a handful of people have reported with xpdf/fvwm on ivy bridge with modesetting driver. Rewrite ALPN extension using CBB/CBS and the new extension framework. Rewrite SRTP extension using CBB/CBS and the new extension framework. Revisit 2q queue sizes. Limit the hot queue to 1/20th the cache size up to a max of 4096 pages. Limit the warm and cold queues to half the cache. This allows us to more effectively notice re-interest in buffers instead of losing it in a large hot queue. Add glass console support for arm64. Probably not yet for your machine, though. Replace heaps of hand-written syscall stubs in ld.so with a simpler framework. 65535 is a valid port to listen on. When xinit starts an X server that listens only on UNIX socket, prefer DISPLAY=unix:0 rather than DISPLAY=:0. This will prevent applications from ever falling back to TCP if the UNIX socket connection fails (such as when the X server crashes). Reverted. Add -z and -Z options to apmd to auto suspend or hibernate when low on battery. Remove the original (pre-IETF) chacha20-poly1305 cipher suites. Add urng(4) which supports various USB RNG devices. Instead of adding one driver per device, start bundling them into a single driver. Remove old deactivated pledge path code. A replacement mechanism is being brewed. Fix a bug from the extension parsing rewrite. Always parse ALPN even if no callback has been installed to prevent leaving unprocessed data which leads to a decode error. Clarify what is meant by syslog priorities being ordered, since the numbers and priorities are backwards. Remove a stray setlocale() from ksh, eliminating a lot of extra statically linked code. Unremove some NPN symbols from libssl because ports software thinks they should be there for reasons. Fix saved stack location after resume. Somehow clang changed it. Resume works again on i386. Improve error messages in vmd and vmctl to be more informative. Stop building the miniroot installer for OMAP3 Beagleboards. It hasn't worked in over a year and nobody noticed. Have the callers of sosetopt() free the mbuf for symmetry. On octeon, let the kernel use the hardware FPU even if emulation is compiled in. It's faster. Fix support for 486DX CPUs by not calling cpuid. I used to own a 486. Now I don't. Merge some drm fixes from linux. Defer probing of floppy drives, eliminating delays during boot. Better handling of probes and beacons and timeouts and scans in wifi stack to avoid disconnects. Move mutex, condvar, and thread-specific data routes, pthreadonce, and pthreadexit from libpthread to libc, along with low-level bits to support them. Let's thread aware (but not actually threaded) code work with just libc. New POSIX xlocale implementation. Complete as long as you only use ASCII and UTF-8, as you should. Round and round it goes; when 6.2 stops, nobody knows. A peak at the future? *** Screencasting with OpenBSD (http://eradman.com/posts/screencasting.html) USB Audio Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART: uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2 uaudio0: audio rev 1.00, 8 mixer controls audio1 at uaudio0 audioctl can read off all of the specific characterisitcs of this device $ audioctl -f /dev/audio1 | grep record mode=play,record record.rate=48000 record.channels=1 record.precision=16 record.bps=2 record.msb=1 record.encoding=slinear_le record.pause=0 record.active=0 record.block_size=1960 record.bytes=0 record.errors=0 Now test the recording from the second audio device using aucat(1) aucat -f rsnd/1 -o file.wav If the device also has a headset audio can be played through the same device. aucat -f rsnd/1 -i file.wav Screen Capture using Xvfb The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it's often very slow. x11vnc will print an estimate of the banwidth for the system your running. x11vnc ... 09/05/2012 22:23:45 fb read rate: 7 MB/sec This is about 4fps. We can do much better by using a virtual framebuffer. Here I'm setting up a new screen, setting the background color, starting cwm and an instance of xterm Xvfb :1 -screen 0 720x540x16 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xterm +sb -fa Hermit -fs 14 & Much better! Now we're up around 20fps. x11vnc -display :1 & ... 11/05/2012 18:04:07 fb read rate: 168 MB/sec Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression. vncviewer localhost -encodings raw A test recording with sound then looks like this ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly. Screen Capture using Xephyr Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card's RAM, so it's reasonably fast. Xephyr -ac -br -noreset -screen 800x600 :1 & DISPLAY=:1 xsetroot -solid steelblue & DISPLAY=:1 cwm & DISPLAY=:1 xrdb -load ~/.Xdefaults & DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 & Capture works in exactally the same way. This command tries to maintain 12fps. ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session. Arranging Windows I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool DISPLAY=:1 midori "file:///pwd" & sleep 2 DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0 DISPLAY=:1 xdotool search --class "midori" windowmove 400 0 DISPLAY=:1 xdotool search --class "midori" windowsize 400 576 This will position the window precisely so that it appears to be in a tmux window on the right. Audio/Video Sync If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi Sharing a Terminal with tmux If you're trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session. tmux -2 -S /tmp/tmux0 Then on the remote side connect on the same socket tmux -2 -S /tmp/tmux0 attach Taking Screenshots Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick's import command DISPLAY=:1 import -window root screenshot.png Audio Processing and Video Transcoding The first step is to ensure that the clip begins and ends where you'd like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45 ffmpeg -i interactive-sql.avi -vcodec copy -acodec copy -ss 00:00:00 -t 00:09:45 interactive-sql-trimmed.avi mv interactive-sql-trimmed.avi interactive-sql.avi Setting the gain correctly is very important with an analog mixer, but if you're using a USB mic there may not be a gain option; simply record using it's built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav normalize audio.wav Next merge the audio back in again ffmpeg -i interactive-sql.avi -i audio.wav -map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy: ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M -c:a libvorbis -q:a 6 interactive-sql.webm H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP: ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 -preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart -c:a aac -q:a 6 interactive-sql.mp4 TrueOS @ Ohio Linuxfest '17! (https://www.trueos.org/blog/trueos-ohio-linuxfest-17/) Dru Lavigne and Ken Moore are both giving presentations on Saturday the 30th. Sit in and hear about new developments for the Lumina and FreeNAS projects. Ken is offering Lumina Rising: Challenging Desktop Orthodoxy at 10:15 am in Franklin A. Hear his thoughts about the ideas propelling desktop environment development and how Lumina, especially Lumina 2, is seeking to offer a new model of desktop architecture. Elements discussed include session security, application dependencies, message handling, and operating system integration. Dru is talking about What's New in FreeNAS 11 at 2:00 pm in Franklin D. She'll be providing an overview of some of the new features added in FreeNAS 11.0, including: Alert Services Starting specific services at boot time AD Monitoring to ensure the AD service restarts if disconnected A preview of the new user interface support for S3-compatible storage and the bhyve hypervisor She's also giving a sneak peek of FreeNAS 11.1, which has some neat features: A complete rewrite of the Jails/Plugins system as FreeNAS moves from warden to iocage Writing new plugins with just a few lines of code A brand new asynchronous middleware API Who's going? Attending this year are: Dru Lavigne (dlavigne): Dru leads the technical documentation team at iX, and contributes heavily to open source documentation projects like FreeBSD, FreeNAS, and TrueOS. Ken Moore (beanpole134): Ken is the lead developer of Lumina and a core contributor to TrueOS. He also works on a number of other Qt5 projects for iXsystems. J.T. Pennington (q5sys): Some of you may be familiar with his work on BSDNow, but J.T. also contributes to the TrueOS, Lumina, and SysAdm projects, helping out with development and general bug squashing. *** Beastie Bits Lumina Development Preview: Theme Engine (https://www.trueos.org/blog/lumina-development-preview-theme-engine/) It's happening! Official retro Thinkpad lappy spotted in the wild (https://www.theregister.co.uk/2017/09/04/retro_thinkpad_spotted_in_the_wild/) LLVM libFuzzer and SafeStack ported to NetBSD (https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_ported) Remaining 2017 FreeBSD Events (https://www.freebsdfoundation.org/news-and-events/event-calendar/2017-openzfs-developer-summit/) *** Feedback/Questions Andrew - BSD Teaching Material (http://dpaste.com/0YTT0VP) Seth - Switching to Tarsnap after Crashplan becomes no more (http://dpaste.com/1SK92ZX#wrap) Thomas - Native encryption in ZFS (http://dpaste.com/02KD5FX#wrap) Coding Cowboy - Coding Cowboy - Passwords and clipboards (http://dpaste.com/31K0E40#wrap) ***
We recap vBSDcon, give you the story behind a PF EN, reminisce in Solaris memories, and show you how to configure different DEs on FreeBSD. This episode was brought to you by Headlines [vBSDCon] vBSDCon was held September 7 - 9th. We recorded this only a few days after getting home from this great event. Things started on Wednesday night, as attendees of the thursday developer summit arrived and broke into smallish groups for disorganized dinner and drinks. We then held an unofficial hacker lounge in a medium sized seating area, working and talking until we all decided that the developer summit started awfully early tomorrow. The developer summit started with a light breakfast and then then we dove right in Ed Maste started us off, and then Glen Barber gave a presentation about lessons learned from the 11.1-RELEASE cycle, and comparing it to previous releases. 11.1 was released on time, and was one of the best releases so far. The slides are linked on the DevSummit wiki page (https://wiki.freebsd.org/DevSummit/20170907). The group then jumped into hackmd.io a collaborative note taking application, and listed of various works in progress and upstreaming efforts. Then we listed wants and needs for the 12.0 release. After lunch we broke into pairs of working groups, with additional space for smaller meetings. The first pair were, ZFS and Toolchain, followed by a break and then a discussion of IFLIB and network drivers in general. After another break, the last groups of the day met, pkgbase and secure boot. Then it was time for the vBSDCon reception dinner. This standing dinner was a great way to meet new people, and for attendees to mingle and socialize. The official hacking lounge Thursday night was busy, and included some great storytelling, along with a bunch of work getting done. It was very encouraging to watch a struggling new developer getting help from a seasoned veteran. Watching the new developers eyes light up as the new information filled in gaps and they now understood so much more than just a few minutes before, and they raced off to continue working, was inspirational, and reminded me why these conferences are so important. The hacker lounge shut down relatively early by BSD conference standards, but, the conference proper started at 8:45 sharp the next morning, so it made sense. Friday saw a string of good presentations, I think my favourite was Jonathan Anderson's talk on Oblivious sandboxing. Jonathan is a very energetic speaker, and was able to keep everyone focused even during relatively complicated explanations. Friday night I went for dinner at ‘Big Bowl', a stir-fry bar, with a largish group of developers and users of both FreeBSD and OpenBSD. The discussions were interesting and varied, and the food was excellent. Benedict had dinner with JT and some other folks from iXsystems. Friday night the hacker lounge was so large we took over a bigger room (it had better WiFi too). Saturday featured more great talks. The talk I was most interested in was from Eric McCorkle, who did the EFI version of my GELIBoot work. I had reviewed some of the work, but it was interesting to hear the story of how it happened, and to see the parallels with my own story. My favourite speaker was Paul Vixie, who gave a very interesting talk about the gets() function in libc. gets() was declared unsafe before the FreeBSD project even started. The original import of the CSRG code into FreeBSD includes the compile time, and run-time warnings against using gets(). OpenBSD removed gets() in version 5.6, in 2014. Following Paul's presentation, various patches were raised, to either cause use of gets() to crash the program, or to remove gets() entirely, causing such programs to fail to link. The last talk before the closing was Benedict's BSD Systems Management with Ansible (https://people.freebsd.org/~bcr/talks/vBSDcon2017_Ansible.pdf). Shortly after, Allan won a MacBook Pro by correctly guessing the number of components in a jar that was standing next to the registration desk (Benedict was way off, but had a good laugh about the unlikely future Apple user). Saturday night ended with the Conference Social, and excellent dinner with more great conversations On Sunday morning, a number of us went to the Smithsonian Air and Space Museum site near the airport, and saw a Concorde, an SR-71, and the space shuttle Discovery, among many other exhibits. Check out the full photo album by JT (https://t.co/KRmSNzUSus), our producer. Thanks to all the sponsors for vBSDcon and all the organizers from Verisign, who made it such a great event. *** The story behind FreeBSD-EN-17.08.pf (https://www.sigsegv.be//blog/freebsd/FreeBSD-EN-17.08.pf) After our previous deep dive on a bug in episode 209, Kristof Provost, the maintainer of pf on FreeBSD (he is going to hate me for saying that) has written the story behind a recent ERRATA notice for FreeBSD First things first, so I have to point out that I think Allan misremembered things. The heroic debugging story is PR 219251, which I'll try to write about later. FreeBSD-EN-17:08.pf is an issue that affected some FreeBSD 11.x systems, where FreeBSD would panic at startup. There were no reports for CURRENT. There's very little to go on here, but we do know the cause of the panic ("integer divide fault"), and that the current process was "pf purge". The pf purge thread is part of the pf housekeeping infrastructure. It's a housekeeping kernel thread which cleans up things like old states and expired fragments. The lack of mention of pf functions in the backtrace is a hint unto itself. It suggests that the error is probably directly in pfpurgethread(). It might also be in one of the static functions it calls, because compilers often just inline those so they don't generate stack frames. Remember that the problem is an "integer divide fault". How can integer divisions be a problem? Well, you can try to divide by zero. The most obvious suspect for this is this code: idx = pfpurgeexpiredstates(idx, pfhashmask / (Vpfdefaultrule.timeout[PFTMINTERVAL] * 10)); However, this variable is both correctly initialised (in pfattachvnet()) and can only be modified through the DIOCSETTIMEOUT ioctl() call and that one checks for zero. At that point I had no idea how this could happen, but because the problem did not affect CURRENT I looked at the commit history and found this commit from Luiz Otavio O Souza: Do not run the pf purge thread while the VNET variables are not initialized, this can cause a divide by zero (if the VNET initialization takes to long to complete). Obtained from: pfSense Sponsored by: Rubicon Communications, LLC (Netgate) That sounds very familiar, and indeed, applying the patch fixed the problem. Luiz explained it well: it's possible to use Vpfdefaultrule.timeout before it's initialised, which caused this panic. To me, this reaffirms the importance of writing good commit messages: because Luiz mentioned both the pf purge thread and the division by zero I was easily able to find the relevant commit. If I hadn't found it this fix would have taken a lot longer. Next week we'll look at the more interesting story I was interested in, which I managed to nag Kristof into writing *** The sudden death and eternal life of Solaris (http://dtrace.org/blogs/bmc/2017/09/04/the-sudden-death-and-eternal-life-of-solaris/) A blog post from Bryan Cantrill about the death of Solaris As had been rumored for a while, Oracle effectively killed Solaris. When I first saw this, I had assumed that this was merely a deep cut, but in talking to Solaris engineers still at Oracle, it is clearly much more than that. It is a cut so deep as to be fatal: the core Solaris engineering organization lost on the order of 90% of its people, including essentially all management. Of note, among the engineers I have spoken with, I heard two things repeatedly: “this is the end” and (from those who managed to survive Friday) “I wish I had been laid off.” Gone is any of the optimism (however tepid) that I have heard over the years — and embarrassed apologies for Oracle's behavior have been replaced with dismay about the clumsiness, ineptitude and callousness with which this final cut was handled. In particular, that employees who had given their careers to the company were told of their termination via a pre-recorded call — “robo-RIF'd” in the words of one employee — is both despicable and cowardly. To their credit, the engineers affected saw themselves as Sun to the end: they stayed to solve hard, interesting problems and out of allegiance to one another — not out of any loyalty to the broader Oracle. Oracle didn't deserve them and now it doesn't have them — they have been liberated, if in a depraved act of corporate violence. Assuming that this is indeed the end of Solaris (and it certainly looks that way), it offers a time for reflection. Certainly, the demise of Solaris is at one level not surprising, but on the other hand, its very suddenness highlights the degree to which proprietary software can suffer by the vicissitudes of corporate capriciousness. Vulnerable to executive whims, shareholder demands, and a fickle public, organizations can simply change direction by fiat. And because — in the words of the late, great Roger Faulkner — “it is easier to destroy than to create,” these changes in direction can have lasting effect when they mean stopping (or even suspending!) work on a project. Indeed, any engineer in any domain with sufficient longevity will have one (or many!) stories of exciting projects being cancelled by foolhardy and myopic management. For software, though, these cancellations can be particularly gutting because (in the proprietary world, anyway) so many of the details of software are carefully hidden from the users of the product — and much of the innovation of a cancelled software project will likely die with the project, living only in the oral tradition of the engineers who knew it. Worse, in the long run — to paraphrase Keynes — proprietary software projects are all dead. However ubiquitous at their height, this lonely fate awaits all proprietary software. There is, of course, another way — and befitting its idiosyncratic life and death, Solaris shows us this path too: software can be open source. In stark contrast to proprietary software, open source does not — cannot, even — die. Yes, it can be disused or rusty or fusty, but as long as anyone is interested in it at all, it lives and breathes. Even should the interest wane to nothing, open source software survives still: its life as machine may be suspended, but it becomes as literature, waiting to be discovered by a future generation. That is, while proprietary software can die in an instant, open source software perpetually endures by its nature — and thrives by the strength of its communities. Just as the existence of proprietary software can be surprisingly brittle, open source communities can be crazily robust: they can survive neglect, derision, dissent — even sabotage. In this regard, I speak from experience: from when Solaris was open sourced in 2005, the OpenSolaris community survived all of these things. By the time Oracle bought Sun five years later in 2010, the community had decided that it needed true independence — illumos was born. And, it turns out, illumos was born at exactly the right moment: shortly after illumos was announced, Oracle — in what remains to me a singularly loathsome and cowardly act — silently re-proprietarized Solaris on August 13, 2010. We in illumos were indisputably on our own, and while many outsiders gave us no chance of survival, we ourselves had reason for confidence: after all, open source communities are robust because they are often united not only by circumstance, but by values, and in our case, we as a community never lost our belief in ZFS, Zones, DTrace and myriad other technologies like MDB, FMA and Crossbow. Indeed, since 2010, illumos has thrived; illumos is not only the repository of record for technologies that have become cross-platform like OpenZFS, but we have also advanced our core technologies considerably, while still maintaining highest standards of quality. Learning some of the mistakes of OpenSolaris, we have a model that allows for downstream innovation, experimentation and differentiation. For example, Joyent's SmartOS has always been focused on our need for a cloud hypervisor (causing us to develop big features like hardware virtualization and Linux binary compatibility), and it is now at the heart of a massive buildout for Samsung (who acquired Joyent a little over a year ago). For us at Joyent, the Solaris/illumos/SmartOS saga has been formative in that we have seen both the ill effects of proprietary software and the amazing resilience of open source software — and it very much informed our decision to open source our entire stack in 2014. Judging merely by its tombstone, the life of Solaris can be viewed as tragic: born out of wedlock between Sun and AT&T and dying at the hands of a remorseless corporate sociopath a quarter century later. And even that may be overstating its longevity: Solaris may not have been truly born until it was made open source, and — certainly to me, anyway — it died the moment it was again made proprietary. But in that shorter life, Solaris achieved the singular: immortality for its revolutionary technologies. So while we can mourn the loss of the proprietary embodiment of Solaris (and we can certainly lament the coarse way in which its technologists were treated!), we can rejoice in the eternal life of its technologies — in illumos and beyond! News Roundup OpenBSD on the Lenovo Thinkpad X1 Carbon (5th Gen) (https://jcs.org/2017/09/01/thinkpad_x1c) Joshua Stein writes about his experiences running OpenBSD on the 5th generation Lenovo Thinkpad X1 Carbon: ThinkPads have sort of a cult following among OpenBSD developers and users because the hardware is basic and well supported, and the keyboards are great to type on. While no stranger to ThinkPads myself, most of my OpenBSD laptops in recent years have been from various vendors with brand new hardware components that OpenBSD does not yet support. As satisfying as it is to write new kernel drivers or extend existing ones to make that hardware work, it usually leaves me with a laptop that doesn't work very well for a period of months. After exhausting efforts trying to debug the I2C touchpad interrupts on the Huawei MateBook X (and other 100-Series Intel chipset laptops), I decided to take a break and use something with better OpenBSD support out of the box: the fifth generation Lenovo ThinkPad X1 Carbon. Hardware Like most ThinkPads, the X1 Carbon is available in a myriad of different internal configurations. I went with the non-vPro Core i7-7500U (it was the same price as the Core i5 that I normally opt for), 16Gb of RAM, a 256Gb NVMe SSD, and a WQHD display. This generation of X1 Carbon finally brings a thinner screen bezel, allowing the entire footprint of the laptop to be smaller which is welcome on something with a 14" screen. The X1 now measures 12.7" wide, 8.5" deep, and 0.6" thick, and weighs just 2.6 pounds. While not available at initial launch, Lenovo is now offering a WQHD IPS screen option giving a resolution of 2560x1440. Perhaps more importantly, this display also has much better brightness than the FHD version, something ThinkPads have always struggled with. On the left side of the laptop are two USB-C ports, a USB-A port, a full-size HDMI port, and a port for the ethernet dongle which, despite some reviews stating otherwise, is not included with the laptop. On the right side is another USB-A port and a headphone jack, along with a fan exhaust grille. On the back is a tray for the micro-SIM card for the optional WWAN device, which also covers the Realtek microSD card reader. The tray requires a paperclip to eject which makes it inconvenient to remove, so I think this microSD card slot is designed to house a card semi-permanently as a backup disk or something. On the bottom are the two speakers towards the front and an exhaust grille near the center. The four rubber feet are rather plastic feeling, which allows the laptop to slide around on a desk a bit too much for my liking. I wish they were a bit softer to be stickier. Charging can be done via either of the two USB-C ports on the left, though I wish more vendors would do as Google did on the Chromebook Pixel and provide a port on both sides. This makes it much more convenient to charge when not at one's desk, rather than having to route a cable around to one specific side. The X1 Carbon includes a 65W USB-C PD with a fixed USB-C cable and removable country-specific power cable, which is not very convenient due to its large footprint. I am using an Apple 61W USB-C charger and an Anker cable which charge the X1 fine (unlike HP laptops which only work with HP USB-C chargers). Wireless connectivity is provided by a removable Intel 8265 802.11a/b/g/n/ac WiFi and Bluetooth 4.1 card. An Intel I219-V chip provides ethernet connectivity and requires an external dongle for the physical cable connection. The screen hinge is rather tight, making it difficult to open with one hand. The tradeoff is that the screen does not wobble in the least bit when typing. The fan is silent at idle, and there is no coil whine even under heavy load. During a make -j4 build, the fan noise is reasonable and medium-pitched, rather than a high-pitched whine like on some laptops. The palm rest and keyboard area remain cool during high CPU utilization. The full-sized keyboard is backlit and offers two levels of adjustment. The keys have a soft surface and a somewhat clicky feel, providing very quiet typing except for certain keys like Enter, Backspace, and Escape. The keyboard has a reported key travel of 1.5mm and there are dedicated Page Up and Page Down keys above the Left and Right arrow keys. Dedicated Home, End, Insert, and Delete keys are along the top row. The Fn key is placed to the left of Control, which some people hate (although Lenovo does provide a BIOS option to swap it), but it's in the same position on Apple keyboards so I'm used to it. However, since there are dedicated Page Up, Page Down, Home, and End keys, I don't really have a use for the Fn key anyway. Firmware The X1 Carbon has a very detailed BIOS/firmware menu which can be entered with the F1 key at boot. F12 can be used to temporarily select a different boot device. A neat feature of the Lenovo BIOS is that it supports showing a custom boot logo instead of the big red Lenovo logo. From Windows, download the latest BIOS Update Utility for the X1 Carbon (my model was 20HR). Run it and it'll extract everything to C:driversflash(some random string). Drop a logo.gif file in that directory and run winuptp.exe. If a logo file is present, it'll ask whether to use it and then write the new BIOS to its staging area, then reboot to actually flash it. + OpenBSD support Secure Boot has to be disabled in the BIOS menu, and the "CSM Support" option must be enabled, even when "UEFI/Legacy Boot" is left on "UEFI Only". Otherwise the screen will just go black after the OpenBSD kernel loads into memory. Based on this component list, it seems like everything but the fingerprint sensor works fine on OpenBSD. *** Configuring 5 different desktop environments on FreeBSD (https://www.linuxsecrets.com/en/entry/51-freebsd/2017/09/04/2942-configure-5-freebsd-x-environments) This fairly quick tutorial over at LinuxSecrets.com is a great start if you are new to FreeBSD, especially if you are coming from Linux and miss your favourite desktop environment It just goes to show how easy it is to build the desktop you want on modern FreeBSD The tutorial covers: GNOME, KDE, Xfce, Mate, and Cinnamon The instructions for each boil down to some variation of: Install the desktop environment and a login manager if it is not included: > sudo pkg install gnome3 Enable the login manager, and usually dbus and hald: > sudo sysrc dbusenable="YES" haldenable="YES" gdmenable="YES" gnomeenable="YES"? If using a generic login manager, add the DE startup command to your .xinitrc: > echo "exec cinnamon" > ~/.xinitrc And that is about it. The tutorial goes into more detail on other configuration you can do to get your desktop just the way you like it. To install Lumina: > sudo pkg install lumina pcbsd-utils-qt5 This will install Lumina and the pcbsd utilities package which includes pcdm, the login manager. In the near future we hear the login manager and some of the other utilities will be split into separate packages, making it easier to use them on vanilla FreeBSD. > sudo sysrc pcdmenable=”YES” dbusenable="YES" hald_enable="YES" Reboot, and you should be greeted with the graphical login screen *** A return-oriented programming defense from OpenBSD (https://lwn.net/Articles/732201/) We talked a bit about RETGUARD last week, presenting Theo's email announcing the new feature Linux Weekly News has a nice breakdown on just how it works Stack-smashing attacks have a long history; they featured, for example, as a core part of the Morris worm back in 1988. Restrictions on executing code on the stack have, to a great extent, put an end to such simple attacks, but that does not mean that stack-smashing attacks are no longer a threat. Return-oriented programming (ROP) has become a common technique for compromising systems via a stack-smashing vulnerability. There are various schemes out there for defeating ROP attacks, but a mechanism called "RETGUARD" that is being implemented in OpenBSD is notable for its relative simplicity. In a classic stack-smashing attack, the attack code would be written directly to the stack and executed there. Most modern systems do not allow execution of on-stack code, though, so this kind of attack will be ineffective. The stack does affect code execution, though, in that the call chain is stored there; when a function executes a "return" instruction, the address to return to is taken from the stack. An attacker who can overwrite the stack can, thus, force a function to "return" to an arbitrary location. That alone can be enough to carry out some types of attacks, but ROP adds another level of sophistication. A search through a body of binary code will turn up a great many short sequences of instructions ending in a return instruction. These sequences are termed "gadgets"; a large program contains enough gadgets to carry out almost any desired task — if they can be strung together into a chain. ROP works by locating these gadgets, then building a series of stack frames so that each gadget "returns" to the next. There is, of course, a significant limitation here: a ROP chain made up of exclusively polymorphic gadgets will still work, since those gadgets were not (intentionally) created by the compiler and do not contain the return-address-mangling code. De Raadt acknowledged this limitation, but said: "we believe once standard-RET is solved those concerns become easier to address separately in the future. In any case a substantial reduction of gadgets is powerful". Using the compiler to insert the hardening code greatly eases the task of applying RETGUARD to both the OpenBSD kernel and its user-space code. At least, that is true for code written in a high-level language. Any code written in assembly must be changed by hand, though, which is a fair amount of work. De Raadt and company have done that work; he reports that: "We are at the point where userland and base are fully working without regressions, and the remaining impacts are in a few larger ports which directly access the return address (for a variety of reasons)". It can be expected that, once these final issues are dealt with, OpenBSD will ship with this hardening enabled. The article wonders about applying the same to Linux, but notes it would be difficult because the Linux kernel cannot currently be compiled using LLVM If any benchmarks have been run to determine the cost of using RETGUARD, they have not been publicly posted. The extra code will make the kernel a little bigger, and the extra overhead on every function is likely to add up in the end. But if this technique can make the kernel that much harder to exploit, it may well justify the extra execution overhead that it brings with it. All that's needed is somebody to actually do the work and try it out. Videos from BSDCan have started to appear! (https://www.youtube.com/playlist?list=PLeF8ZihVdpFfVEsCxNWGDmcATJfRZacHv) Henning Brauer: tcp synfloods - BSDCan 2017 (https://www.youtube.com/watch?v=KuHepyI0_KY) Benno Rice: The Trouble with FreeBSD - BSDCan 2017 (https://www.youtube.com/watch?v=1DM5SwoXWSU) Li-Wen Hsu: Continuous Integration of The FreeBSD Project - BSDCan 2017 (https://www.youtube.com/watch?v=SCLfKWaUGa8) Andrew Turner: GENERIC ARM - BSDCan 2017 (https://www.youtube.com/watch?v=gkYjvrFvPJ0) Bjoern A. Zeeb: From the outside - BSDCan 2017 (https://www.youtube.com/watch?v=sYmW_H6FrWo) Rodney W. Grimes: FreeBSD as a Service - BSDCan 2017 (https://www.youtube.com/watch?v=Zf9tDJhoVbA) Reyk Floeter: The OpenBSD virtual machine daemon - BSDCan 2017 (https://www.youtube.com/watch?v=Os9L_sOiTH0) Brian Kidney: The Realities of DTrace on FreeBSD - BSDCan 2017 (https://www.youtube.com/watch?v=NMUf6VGK2fI) The rest will continue to trickle out, likely not until after EuroBSDCon *** Beastie Bits Oracle has killed sun (https://meshedinsights.com/2017/09/03/oracle-finally-killed-sun/) Configure Thunderbird to send patch friendly (http://nanxiao.me/en/configure-thunderbird-to-send-patch-friendly/) FreeBSD 10.4-BETA4 Available (https://www.freebsd.org/news/newsflash.html#event20170909:01) iXsystems looking to hire kernel and zfs developers (especially Sun/Oracle Refugees) (https://www.facebook.com/ixsystems/posts/10155403417921508) Speaking of job postings, UnitedBSD.com has few job postings related to BSD (https://unitedbsd.com/) Call for papers USENIX FAST ‘18 - February 12-15, 2018, Due: September 28 2017 (https://www.freebsdfoundation.org/news-and-events/call-for-papers/usenix-fast-18-call-for-papers/) Scale 16x - March 8-11, 2018, Due: October 31, 2017 (https://www.freebsdfoundation.org/news-and-events/call-for-papers/scale-16x-call-for-participation/) FOSDEM ‘18 - February 3-4, 2018, Due: November 3 2017 (https://www.freebsdfoundation.org/news-and-events/call-for-papers/fosdem-18-call-for-participation/) Feedback/Questions Jason asks about cheap router hardware (http://dpaste.com/340KRHG) Prashant asks about latest kernels with freebsd-update (http://dpaste.com/2J7DQQ6) Matt wants know about VM Performance & CPU Steal Time (http://dpaste.com/1H5SZ81) John has config questions regarding Dell precision 7720, FreeBSD, NVME, and ZFS (http://dpaste.com/0X770SY) ***
DragonflyBSD 4.8.1 has been released, we explore how the X11 clipboard works, and look at OpenBSD gaming resources. This episode was brought to you by Headlines LLVM, Clang and compiler-rt support enhancements (https://blog.netbsd.org/tnf/entry/llvm_clang_and_compiler_rt) In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero. LLVM In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests. There were four threads within LLVM: Broken std::callonce with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::callonce. I've discovered that the llvm::callonce workaround isn't sufficient for the whole LLVM functionality, as std::callonce can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker. Unportable shell construct hardcoded in tests ">&". This has been fixed upstream. LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X. Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt. The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests. Clang I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt. compiler-rt The current compiler-rt tasks can be divided into: upstream sanitizer common code shared with POSIX platforms upstream sanitizer common code shared with Linux and FreeBSD upstream sanitizer common code shared with FreeBSD upstream sanitizer common code specific to NetBSD build, execute and pass tests for sanitizer common code in check-santizer This means that ubsan, asan and the rest of the specific sanitizers wait in queue. All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one. The last point with check-sanitizer unveiled so far two generic bugs on NetBSD: Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls. Honor PTHREADDESTRUCTORITERATIONS in libpthread. These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems. The fixes were introduced by Christos Zoulas and backported to NetBSD-8. Plan for the next milestone I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future. LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0). This work was sponsored by The NetBSD Foundation. The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: http://netbsd.org/donations/#how-to-donate *** DragonFly BSD 4.8.1 released (http://lists.dragonflybsd.org/pipermail/commits/2017-August/626150.html) +Updates by dev: + Antonio Huete Jimenez (1): + libc/gmon: Replace sbrk() with mmap() + Francois Tigeot (3): + drm: bring in Linux compability changes from master + drm/linux: make flushwork() more robust + drm/i915: Update to Linux 4.7.10 + Imre Vadász (4): + drm - Fix hrtimer, don't reset timer->function to NULL in timeout handler. + sound - Delete devfs clone handler for /dev/dsp and /dev/mixer on unload. + ifvtnet - Allocate struct vtnettxheader entries from a queue. + Make sure that cam(4)'s dashutdown handler runs before DEVICESHUTDOWN(). + Matthew Dillon (24): + kernel - MFC b48dd28447fc (sigtramp workaround) + kernel - Fix deadlock in sound system + kernel - Fix broken wakeup in crypto code + kernel - Add KERNPROCSIGTRAMP + gcc - Adjust the unwind code to use the new sigtramp probe sysctl + kernel - Implement NX + kernel - Implement NX (2) + kernel - Implement machdep.pmapnxenable TUNABLE + kernel - Implement NX (3) - cleanup + kernel - Temporarily set the default machdep.pmapnxenable to 0 + param - Change _DragonFlyversion to 400801 + kernel - Fix i915 deadlock + pthreads - Change PTHREADSTACKMIN + libc - Fix bug in rcmdsh() + ppp - Fix minor overflow in protocol search + libtelnet - Fix improper statement construction (not a bug in the binary) + libdevstat - Limit sscanf field, fix redundant condition + openssh - Fix a broken assignment + window - Fix Graphics capability enable test + kernel - Fix event preset + mfiutil - Fix static buffer overflow + mixer - Fix sscanf() overflow + gcore - fix overflow in sscanf + kernel - Fix improper parens + Sascha Wildner (17): + libkvm: Fix char pointer dereference. + Fix some cases where an index was used before its limits check. + Really ensure that our world/kernel are built under POSIX locale ("C"). + zoneinfo: Create a /usr/share/zoneinfo/UTC link. + kernel/cam: Add CAMSCSIITNEXUSLOST (in preparation for virtioscsi(4)). + kernel: Add FreeBSD's virtioscsi(4) driver. + ccdconfig(8): Add missing free(). + libpuffs: Fix two asserts. + kernel/acpi: Untangle the wakecode generation during buildkernel. + kernel/acpica: Better check AcpiOsPredefinedOverride()'s InitVal argument + kernel/acpica: ACPITHREADID is unsigned. + kernel/acpica: Return curthread as thread id from AcpiOsGetThreadId(). + kernel/acpica: Remove no longer needed #include. + kernel/acpi: Call AcpiInitializeSubsystem() before AcpiInitializeTables(). + kernel/urtwn: Add missing braces. + kernel/ieee80211: Add missing braces. + libthreadxu: Fix checking of pthreadbarrierinit()'s count argument. + Sepherosa Ziehau (7): + sound/hda: Sync device ID table with FreeBSD + inet6: Restore mbuf hash after defragmentation. + pf: Normalized, i.e. defragged, packets requiring rehash. + em: Enable MSI by default on devices has PCI advanced features capability. + sched: Change CPU_SETSIZE to signed int, same as FreeBSD/Linux. + usched: Allow process to change self cpu affinity + ix: Fixup TX/RX ring settings for X550, which supports 64/64 TX/RX rings. + zrj (1): + Revert "Always use unix line endings" Porting Unix to the 386: A Practical Approach (http://www.informatica.co.cr/unix-source-code/research/1991/0101.html) The University of California's Berkeley Software Distribution (BSD) has been the catalyst for much of the innovative work done with the UNIX operating system in both the research and commercial sectors. Encompassing over 150 Mbytes (and growing) of cutting-edge operating systems, networking, and applications software, BSD is a fully functional and nonproprietary complete operating systems software distribution (see Figure 1). In fact, every version of UNIX available from every vendor contains at least some Berkeley UNIX code, particularly in the areas of filesystems and networking technologies. However, unless one could pay the high cost of site licenses and equipment, access to this software was simply not within the means of most individual programmers and smaller research groups. The 386BSD project was established in the summer of 1989 for the specific purpose of porting BSD to the Intel 80386 microprocessor platform so that the tools this software offers can be made available to any programmer or research group with a 386 PC. In coordination with the Computer Systems Research Group (CSRG) at the University of California at Berkeley, we successively ported a basic research system to a common AT class machine (see, Figure 2), with the result that approximately 65 percent of all 32-bit systems could immediately make use of this new definition of UNIX. We have been refining and improving this base port ever since. By providing the base 386BSD port to CSRG, our hope is to foster new interest in Berkeley UNIX technology and to speed its acceptance and use worldwide. We hope to see those interested in this technology build on it in both commercial and noncommercial ventures. In this and following articles, we will examine the key aspects of software, strategy, and experience that encompassed a project of this magnitude. We intend to explore the process of the 386BSD port, while learning to effectively exploit features of the 386 architecture for use with an advanced operating system. We also intend to outline some of the tradeoffs in implementation goals which must be periodically reexamined. Finally, we will highlight extensions which remain for future work, perhaps to be done by some of you reading this article today. Note that we are assuming familiarity with UNIX, its concepts and structures, and the basic functions of the 386, so we will not present exhaustive coverage of these areas. In this installment, we discuss the beginning of our project and the initial framework that guided our efforts, in particular, the development of the 386BSD specification. Future articles will address specific topics of interest and actual nonproprietary code fragments used in 386BSD. Among the future areas to be covered are: 386BSD process context switching Executing the first 386BSD process on the PC 386BSD kernel interrupt and exception handling 386BSD INTERNET networking ISA device drivers and system support 386BSD bootstrap process *** X11: How does “the” clipboard work (https://www.uninformativ.de/blog/postings/2017-04-02/0/POSTING-en.html) > If you have used another operating system before you switched to something that runs X11, you will have noticed that there is more than one clipboard: > Sometimes, you can use the mouse to select some text, switch to another window, and then hit the middle mouse button to paste text. > Sometimes, you can select text, then hit some hotkey, e.g. Ctrl+C, switch to another window, hit another hotkey, e.g. Ctrl+V, and paste said text. > Sometimes, you can do both. > Selections as a form of IPC First things first, in X11 land, “clipboards” are called “selections”. Yes, there is more than one selection and they all work independently. In fact, you can use as many selections as you wish. In theory, that is. When using selections, you make different clients communicate with each other. This means that those clients have to agree on which selections to use. You can't just invent your own selection and then expect Firefox to be compatible with it. How are selections identified? There are three “standard” selection names: PRIMARY: The “middle mouse clipboard” SECONDARY: Virtually unused these days CLIPBOARD: The “Ctrl+C clipboard” Program 1: Query selection owners Content type and conversion Program 2: Get clipboard as UTF-8 Program 3: Owning a selection Program 4: Content type TARGETS Handling binary data using xclip Large amounts of data Clipboard managers Summary News Roundup TrueOS Documentation: A great way to give back! (https://www.trueos.org/blog/trueos-documentation-great-way-give-back/) The TrueOS project is always looking for community contribution. Documentation changes are a great way for users to not only make a solid contribution to the project, but learn more about it too! Over the last few months, many users have asked for both simple and detailed instructions on making documentation changes. These are now added to the TrueOS handbook in the Contributing to TrueOS section. If interested in making a small alteration to the TrueOS handbook, here are some instructions for submitting a patch through the GitHub website. These instructions are also applicable to the Lumina and SysAdm handbooks. Lumina documentation is in the the lumina-docs repository, and SysAdm guides are in sysadm-docs. Make a Doc change! A GitHub account is required to submit patches to the TrueOS docs. Open a web browser and sign in to GitHub or make a new account. When making a new account, be sure to use an often checked email address, as all communication regarding patches and pull requests are sent to this address. Navigate to the trueos-docs GitHub repository. Click on the trueos-handbook directory to view all the documentation files. Open the .rst file corresponding to the chapter needing an update. The chapter names are reflected in the title of the .rst files. For example, open install.rst to fix an error spotted in handbook chapter 3: “Install”. This first image shows the trueos-docs repository and the contents of the trueos-handbook directory Open the desired chapter file by clicking its entry in the list. The trueos.rst file is an index file and should be ignored. Begin editing the file by clicking the Pencil icon in the upper right corner above the file's text. The file moves to edit mode, where it is now possible to make changes, as the next image shows. Editing install.rst with GitHub When making a simple change, it is recommended to avoid adjusting the specific formatting elements and instead work within or around them. Once satisfied, scroll to the bottom of the page and write a detailed commit summary of the new changes. Click Propose file change (green button), then Create pull request to submit the changes to the project. GitHub then does an automated merge check. Click Create pull request again to submit the change to the repository. In the final step, a developer or project committer reviews the changes, merging them into the project or asking for more changes as necessary. Learn more about TrueOS documentation To learn more about the underlying structure of TrueOS documentation like the Sphinx Documentation Generator and reStructuredText markup, browse the Advanced Documentation Changes section of the TrueOS handbook. This section also contains instructions for forking the repository and configuring a local clone, build testing, updating the translation files, and other useful information. The Sphinx website is also a valuable resource. libHijack Revival (https://www.soldierx.com/news/Hijack-Revival) Over a decade ago, while standing naked and vulnerable in the comfort of my steaming hot shower, I gathered my thoughts as humans typically attempt to do in the wee hours of the morning. Thoughts of a post-exploitation exercise raced in my mind, the same thoughts that made sleeping the night before difficult. If only I could inject into Apache some code that would allow me to hook into its parsing engine without requiring persistance. Putting a file-backed entry into /proc/pid/maps would tip off the security team to a compromise. The end-goal was to be able to send Apache a special string and have Apache perform a unique action based on the special string. FelineMenace's Binary Protection Schemes whitepaper provided inspiration. Silvio Cesare paved the way into PLT/GOT redirection attacks. Various Phrack articles selflessly contributed to the direction I was to head. Alas, in the aforementioned shower, an epiphany struck me. I jumped as an awkward stereotypical geek does: like an elaborate Elaine Benes dance rehearsal in the air. If I used PTrace, ELF, and the PLT/GOT to my advantage, I could cause the victim application to allocate anonymous memory mappings arbitrarily. In the newly-created memory mapping, I could inject arbitrary code. Since a typical operating system treats debuggers as God-like applications, the memory mapping could be mapped without write access, but as read and execute only. Thus enabling the stealth that I sought. The project took a few years to develop in my spare time. I ended up creating several iterations, taking a rough draft/Proof-of-Concept style code and rewriting it to be more efficient and effective. I had toyed with FreeBSD off-and-on for over a decade by this point, but by-and-large I was still mostly using Linux. FreeBSD gained DTrace and ZFS support, winning me over from the Linux camp. I ported libhijack to FreeBSD, giving it support for both Linux and FreeBSD simultaneously. In 2013, I started work on helping Oliver Pinter with his ASLR implementation, which was originally destined to be upstreamed to FreeBSD. It took a lot of work, and my interest in libhijack faded. As a natural consequence, I handed libhijack over to SoldierX, asking the community to take it and enhance it. Over four years went by without a single commit. The project was essentially abandoned. My little baby was dead. This past week, I wondered if libhijack could even compile on FreeBSD anymore. Given that four years have passed by and major changes have happened in those four years, I thought libhijack would need a major overhaul just to compile, let alone function. Imagine my surprise when libhijack needed only a few fixups to account for changes in FreeBSD's RTLD. Today, I'm announcing the revival of libhijack. No longer is it dead, but very much alive. In order to develop the project faster, I've decided to remove support for Linux, focusing instead on FreeBSD. I've removed hundreds of lines of code over the past few days. Supporting both FreeBSD and Linux meant some code had to be ugly. Now the beautification process has begun. I'm announcing the availability of libhijack 0.7.0 today. The ABI and API should be considered unstable as they may change without notice. Note that HardenedBSD fully mitigates libhijack from working with two security features: setting security.bsd.unprivilegedprocdebug to 0 by default and the implementation of PaX NOEXEC. The security.bsd.unprivilegedprocdebug sysctl node prevents PTrace access for applications the debugger itself did not fork+execve for unprivileged (non-root) users. Privileged users (the root account) can use PTrace to its fullest extent. HardenedBSD's implementation of PaX NOEXEC prevents the creation of memory mappings that are both writable and executable. It also prevents using mprotect to toggle between writable and executable. In libhijack's case, FreeBSD grants libhijack the ability to write to memory mappings that are not marked writable. Debuggers do this to set breakpoints. HardenedBSD behaves differently due to PaX NOEXEC. Each memory mapping has a notion of a maximum protection level. When a memory mapping is created, if the write bit is set, then HardenedBSD drops the execute bit from the maximum protection level. When the execute bit is set at memory mapping creation time, then the write bit is dropped from the maximum protection level. If both the write and execute bits are set, then the execute bit is silently dropped from both the mapping creation request and the maximum protection level. The maximum protection level is always obeyed, even for debuggers. Thus we see that PaX NOEXEC is 100% effective in preventing libhijack from injecting code into a process. Here is a screenshot showing PaX NOEXEC preventing libhijack from injecting shellcode into a newly-created memory mapping. What's next for libhijack? Here's what we have planned, in no particular order: Python bindings Port to arm64 This requires logic for handling machine-dependent code. High priority. Finish anonymous shared object injection. This requires implementing a custom RTLD from within libhijack. More cleanups. Adhere to style(9). libhijack can be found on GitHub @ https://github.com/SoldierX/libhijack *** Contributing to FreeBSD (https://blather.michaelwlucas.com/archives/2988) I've talked to a whole bunch of folks who say things like “I'm a junior programmer. I'm looking for a way to help. I have no specific expertise, but I'm willing to learn.” Today, I present such junior programmers with an opportunity. An opportunity for you to learn skills that will be incredibly valuable to your career, and will simultaneously expand your career opportunities. For decades, FreeBSD has relied on its users for testing. They expect users to install pre-release versions of the OS and exercise them to identify regressions. That's necessary, but it's nowhere near enough. The FreeBSD Testing Project is building an automated test suite for the entire operating system. They have a whole mess of work to do. There's only four people on the team, so each additional person that contributes can have a serious impact. They have tutorials on how to write tests, and sample tests. There's a whole bunch of tests left to be written. You have an almost open field. They need tests for everything from ls(1) to bhyve. (Yes, ls(1) broke at one point in the last few years.) Everything needs testing. Learning to write, submit, and commit small tests is valuable experience for developing the big tests. What's more, learning to write tests for a system means learning the system. Developing tests will transform you into a FreeBSD expert. Once you've demonstrated your competence, worth, and ability to work within the project, other FreeBSD teams will solicit your help and advice. The Project will suck you in. Testing is perhaps the most valuable contribution anyone can make to an open source project. And this door into the FreeBSD Project is standing wide, wide open. OpenBSD Gaming Resource (https://mrsatterly.com/openbsd_games.html) > What isn't there to love about playing video games on your favorite operating system? OpenBSD and video games feels like a natural combination to me. My resource has software lists, links to free games not in ports, lists of nonfree games, and recommendations. The Table of Contents has these high-level items for you: > General Resources > OpenBSD Exclusive > Ports > Network Clients > Browser Games > Game Engines > Multiple Game Engines > Multiple System Emulation > Computer Emulation > Game Console Emulation > Live Media Emulation > Operating System Emulation > Games in Other Software Have fun with these games! *** Beastie Bits Dragonfly introduces kcollect(8) (https://www.dragonflydigest.com/2017/08/07/20061.html) The Faces of Open Source (http://facesofopensource.com/unix/) Edgemesh CEO, Jake Loveless and Joyent CTO, Bryan Cantrill join together for a fireside chat to discuss distributed caching at scale, Docker, Node.js, Mystery Science Theater 3000, and more! (https://www.joyent.com/blog/joyent-edgemesh-cache-me-if-you-can) UFS: Place the information needed to find alternate superblocks to the end of the area reserved for the boot block (https://svnweb.freebsd.org/base?view=revision&revision=322297) Let ‘localhost' be localhost (https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-04) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Morgan - btrfs deprecated (http://dpaste.com/0JEYE1K) Ben - UEFI, GELI, BEADM, and more (http://dpaste.com/2TP90HD) Brad - Hostname Clarification (http://dpaste.com/1MQH1BD) M Rod - BSD Laptop (http://dpaste.com/39C6PGN) Jeremy - Contributing to BSDs (http://dpaste.com/3SVP5SF) ***
We recap our devsummit experiences at BSDCambridge, share why memcmp is more complicated than expected, explore Docker on FreeBSD, and we look at a retro terminal. This episode was brought to you by Headlines BSDCam recap (https://wiki.freebsd.org/DevSummit/201708) The 2017 Cambridge DevSummit took place from 2-4 August 2017. The event took place over three days including a formal dinner at St John's College, and was attended by 55 registered developers and guests. Prior to the start of the conference, we had a doc hacking lounge, the computer lab provided a room where we could meet and try to spend some time on documentation. Sevan walked two interested people through the process of creating a documentation patch and submitting it for the first time. In the process, found ways to improve the documentation on how to write documentation. The event is run "un-conference style" in that we brainstorm the actual session schedule on the first morning, with a focus on interactive topics that reflect the interests and exploit the knowledge of the attendees. The idea is to maximize the amount of discussion and decisions that can be made while we are all in the same room The first morning, we all gather in the slightly too small, and even more slightly under air conditioned FW11 classroom. We go around the room introducing ourselves, and listing a few topics we would be interested in discussing. Eventually the whiteboard is full of topics, with various numbers of ticks beside them to indicate the number of interested people There are breakout rooms of all sizes, so even topics with only a small group of interested folks can get a lot accomplished The most difficult is trying to schedule the sessions, as there is much overlap and people usually want to be in concurrent sessions, or someone's schedule means they won't be available that day, etc. This years working groups: Toolchain (Compilers, Linkers, External Toolchain, Static analysis and sanitizers) Virtualization (bhyve, xen, jails, docker) Transport (TCP) and Network Performance Security and mitigations (W^X, noexec stack, CFI, ASLR, KASLR, Safe Stack, etc) Testing (Status, What to test, How to test, QA for releases) Capsicum (Automation with LLVM etc, Casper, Namespacing, “Services”, capsh) Desktop / WiFi (drm-next, drivers, resume, power, installer, desktop, OOB Experience) Tracing (Blackbox, DTrace, KTR, ptrace, truss, hardware tracing) Packaging and Packaged Base (Sets, Kernels, Ports & flavours, sub-packages, privlib) Architectural Security Features (CPU Features: SGX, PXN/PAN, Pointer Authentication, AMD Memory Encryption, Libcrunch, RISC-V, CheriABI) Architectures and Embedded systems (RISC-V, ARM, ARM64, MIPS(64), SPARC64) Teaching (Audiences, Objectives, Targets, Material, future directions) Provisioning and Management Tools (CfgMgmt tools, Image building, VM/bhyve orchestration, Preconfigured VMs for testing, Wishlist) Storage (ZFS status update, ZFS encryption infrastructure, ZFS Zero Copy / Sendfile, Acceleration of checksums and raidz parity calculations, sesutil, mpsutil) And that wasn't everything. We then had a series of short talklets: Enhancing and replacing mmap() SDIO support eBPF support for FreeBSD Tracing + Virtualization Practical DMA Attack Protection On Thursday night there was a special dinner at St John's College Overall it was a great DevSummit, and I even managed to get some of the work assigned to me finished. Shortly I will commit an update to the boot loader menu that will automatically populate the kernel selection menu with the automatically detected list of installed kernels. The list is also properly refreshed when you switch boot environments. *** Hosts/BSD – for when you need to run your BSD inside a penguin (https://wiki.qemu.org/index.php/Hosts/BSD) This wiki provides details on how to run each of the various BSDs under QEMU The target audience is Linux developers looking to test their apps etc under BSD The wiki is in need of some love, there are some option questions, and it lacks some polish There are instructions on building qemu from source, but it should likely mention the qemu-devel port There should probably also be instructions on using other architectures, like ARM/MIPS etc If you have used QEMU, or would like to spend the time to learn how, please help update this wiki *** memcmp -- more complicated than you might expect (http://trust-in-soft.com/memcmp-requires-pointers-to-fully-valid-buffers/) “A suspicious pattern in open-source software” One bug recently found by John using tis-interpreter on a widely used open-source library involved the comparison of strings with memcmp. The unexpected condition was that memcmp was, in one case, called with a pointer to a buffer shorter than the length passed as third argument, breaking one of the two symmetrical pre-conditions in the function's ACSL contract A reason that may have made this use of memcmp look okay to the developer is that the buffers being passed to it always differed before the end of the buffers were reached. a memcmp implementation based on stopping as soon as a difference is found, would not have caused any out-of-bounds read access The first question raised was whether the pattern memcmp("a", "bc", 3) was problematic according to the letter of the C standard. If it was, the second question was whether the busy maintainer of one of the Open Source packages that make the Internet tick should be bothered with a bug report. I would like to be able to say that memcmp's ACSL contract was the product of careful deliberation, but unfortunately this is not the case: many standard function contracts were written quickly in order to get most of the standard library covered, and have not been tested by time. Anyway, upon proofreading the relevant clause in the C11 standard, my feeling was that the ACSL formalization was, in this particular case, right, and that it was undefined behavior to pass as memcmp argument a buffer that wasn't fully valid, even if the implementation sort-of needs to read the buffer's characters in order for the purpose of finding the first mismatch. The post then goes on to look at the memcmp code in glibc There are two distinct optimizations for long buffers, one that applies when both buffers start at the same offset modulo the word size, memcmpcommonalignment, and one that applies when they don't, memcmpnotcommonalignment. The function memcmpcommonalignment is relatively well-behaved: it reads from the two buffers aligned word by aligned word, and thus reads the entire words that contain differing bytes. If the caller passed buffers that aren't valid after the differing byte, this amounts to reading out of bounds, but this sort of out-of-bounds access is not detected by the typical MMU, which works at the scale of the page. The “notcommon_alignment” case, however, tells a different story. When passed the carefully (mis-)aligned buffers t1 and (char*)t2+1, although these buffers differ in the 8th byte, Glibc's implementation of memcmp reads 8 bytes beyond the end of t1. By making the 16th byte differ instead of the 8th one, it is also possible to make Glibc's implementation of memcmp read 16 bytes beyond the end of t1. In conclusion, yes, some implementations of memcmp will crash when invoked with buffers that aren't valid for the full length, even if they differ early. The circumstances are rare (probably the reason this bug was still there to be found in a library that had already been tested with all the available techniques) but outside the programmer's control. The pattern described in this post should be reported as a bug when found. It is interesting to read the detailed analysis of a bug in such a basic libc feature *** News Roundup Docker on FreeBSD (http://daemon-notes.com/articles/network/docker) There are two approaches to running Docker on FreeBSD. First one was created back in 2015 and it was a native port of Docker engine to FreeBSD. It was an ambitious project but nobody stepped forward to continuously port the never-ending flow of upstream code to FreeBSD. So the port still exists (sysutils/docker-freebsd) but it wasn't updated since 2015 and it is Docker v1 (it is v17 as of 2017). The other approach is to use official way of running Docker on platforms other than Linux. Well, somewhat official as Docker still does not support FreeBSD as a host officially. This is docker-machine tool which in turn will use VirtualBox to run a virtual machine with Linux and Docker engine. docker utility on the host will communicate with the engine inside VB where all the work will be done. This article describes what needs to be done to start using it. Before we begin you need VirtualBox installed. Do not skip adding /boot/loader.conf and /etc/rc.conf lines mentioned on that page. You won't need user inteface or anything, docker-machine will do all the work, just make sure VirtualBox is present and ready to be used. `pkg install docker docker-machine docker-compose' Docker will store its stuff in ~/.docker. You might not want the virtual machine image files to live in your home, in this case just create a symlink: mkdir ~/.docker ln -s /storage/docker ~/.docker/machine docker-machine create --driver virtualbox --virtualbox-memory 2048 --virtualbox-cpu-count 2 --virtualbox-disk-size 102400 --virtualbox-hostonly-cidr "10.2.1.1/24" docker1 Here's the example. We are creating machine named docker1. It is using VirtualBox driver, the vm has 2G of memory, 2 cores and 100G of disk space. docker-machine setups VirtualBox to use host-only network adapter (it will create vboxnet0 interface on the host automatically) and we are instructing it to use 10.2.1.1/24 as the address of this adapter — change it to what suits your needs or omit this flag (default is 192.168.99.1/24). And basically that is all. Check if it is running: docker-machine ls If you do open VirtualBox interface you will find a virtual machine named docker1 running. You can start/stop/whatever your machine using docker-machine utility. Here's how you can connect to the machine: docker utility by default tries to talk to Docker engine running on the same host. However with specific environment variables you can instruct it to talk to other host. docker-machine can export these variables for you. eval docker-machine env docker1 docker run hello-world There was quite a bit of discussion about docker at the FreeBSD developers summit in Cambridge during the first week of August. Two docker developers who had worked on the Mac OS X port, one of whom is an OpenBSD advocate, explained how docker has evolved, and the linux-isms have been abstracted away such that a truly native docker solution for FreeBSD can be built and maintained with a lot less headache than before I look forward to seeing if we can't make that happen *** The POSIX Shell And Utilities (http://shellhaters.org/) The POSIX Shell And Utilities Compiled for The Shell Hater's Handbook *** PostgreSQL – logging to a file (http://dan.langille.org/2017/07/31/postgresql-logging-to-a-file/) These steps were carried out on FreeBSD 11.0 with PostgreSQL 9.6 (two of my favorite tools). I like logging. I like logging PostgreSQL. With logs, you can see what happened. Without, you can only guess. Setting up logging for PostgreSQL involves several parts, each of which must be completed or else I don't get what I want. This is not a criticism of PostgreSQL. It's a feature. I am documenting this because each time I configure a new PostgreSQL instance, it takes me more than one iteration to get it working. The goal: this post lets both you and me get it right the first time. The parts include: + Telling PostgreSQL to log via syslog + Telling FreeBSD to local postgres to /var/log/postgres.log (my preference). + Telling PostgreSQL the things you want logged. + Changes to postgresql.conf The file location varies with the version installed. For PostgreSQL 9.6 on FreeBSD, the file is /var/db/postgres/data96/postgresql.conf (adjust 96 according to the version installed). I made these changes to that file. log_destination = 'syslog' log_min_messages = notice log_min_error_statement = notice log_checkpoints = on log_lock_waits = on log_timezone = 'UTC' By default, PostgreSQL logs to the local0 facility and is controlled by the syslog_facility in postgresql.conf. This will be used in syslog.conf (see the next section of this post). The above mentioned changes require a reload: service postgresql reload Changes to /etc/syslog.conf Now that we have PostgreSQL logging to syslog, we want to tell syslog where to put those messages. I changed this line in /etc/syslog.conf:*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages With .notice pulling in some local0 messages, adding local0.none to the line will free the messages up for later use in the configuration file. Otherwise, the PostgreSQL messages will be in /var/log/messages. The changed line is: `.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local0.none /var/log/messages Then, to get the messages into my preferred location, I added this to the file: local0.* /var/log/postgresql.log` Log file rotation For rotating my log file, I added a new file: /usr/local/etc/newsyslog.conf.d/postgresql96 /var/log/postgresql.log pgsql:wheel 640 7 * $D0 GB /var/db/postgres/data96/postmaster.pid 30 Before restarting syslog, I did this, so the destination file existed. This isn't always/strictly necessary, but because the ownership is not chown root:wheel, I do it to get that part set. touch /var/log/postgresql.log chown pgsql:wheel Restarting syslog: sudo kill -HUP `sudo cat /var/run/syslog.pid ` That's it Now you should see PostgreSQL logging in /var/log/postgresql.log. mandoc-1.14.2 released (http://undeadly.org/cgi?action=article&sid=20170729122350) i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ (http://mandoc.bsd.lv/). ```From: Ingo Schwarze schwarze@usta.de Date: Fri, 28 Jul 2017 20:12:44 +0200 To: discuss@mandoc.bsd.lv Subject: mandoc-1.14.2 released Hi, i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ . All downstream maintainers are encouraged to update their ports and packages from 1.14.1 to 1.14.2. Mandoc 1.14.2 is a feature release introducing: a new -Tmarkdown output mode anchors for deep linking into -Thtml manual pages a superset of the functionality of the former mdoclint(1) utility a new -Wstyle message level with several new messages automatic line breaking inside individual tbl(7) cells a rewrite of the eqn(7) lexer, and some eqn(7) rendering improvements support for many additional low-level roff(7) features and various smaller features and bug fixes. For more details, see: http://mandoc.bsd.lv/NEWS With the improved mandoc features, only twenty-five out of the ten thousand software packages in the OpenBSD ports tree still need groff to format their manual pages. Since the project has been called "mandoc" rather than "mdocml" for several years now, the website, the distribution tarball, and the source extraction directory are now also called "mandoc" rather than "mdocml". The release was tested on the following systems: + OpenBSD-current and OpenBSD-stable + NetBSD-current + illumos + Debian Linux + Void Linux x86_64 glibc and musl + Crux Linux + SunOS 5.11.2, 5.10, and 5.9 As before, catman(8) and the regression suite cannot be used on SunOS 5.10 and SunOS 5.9. A big thanks to everybody who provided patches, bug reports, feature suggestions, advice, and help with testing! Yours, Ingo``` Beastie Bits A good looking terminal emulator which mimics the old cathode display. Available in x11/cool-retro-terminal (https://github.com/Swordfish90/cool-retro-term) Milestone Complete! OpenRC conversion (https://www.trueos.org/blog/milestone-complete-openrc-conversion/) Healthy developer interaction between FreeBSD and IllumOS re: mdb (https://illumos.topicbox.com/groups/developer/discussions/T5eae6079331c4df4) Large Batch of Kernel Errata Patches Released (http://undeadly.org/cgi?action=article&sid=20170804053102) opnsense 17.7 released (https://opnsense.org/opnsense-17-7-released/) Twitter Co-Founder and CEO states “FreeBSD rules them all” (https://twitter.com/jack/status/892605692317650944) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Dominik - Monitoring Software (http://dpaste.com/08971FQ) Darren - Wonderful Awk (http://dpaste.com/0YCS4DN) Andrew - Thanks (http://dpaste.com/0ZREKTV) Jens - Migration Questions (http://dpaste.com/1GVZNWN) ***
This week on BSDNow, we have “Weird Unix Things”, “Is it getting Steamy in here?” and an Interview about BSD Sockets API. (Those This episode was brought to you by Headlines playonbsd with TrueOS: It's Getting Steamy in Here and I've Had Too Much Wine (https://www.trueos.org/blog/playonbsd-trueos-getting-steamy-ive-much-wine/) We've done a couple of tutorials in the past on using Steam and Wine with PC-BSD, but now with the addition of playonbsd to the AppCafe library, you have more options than ever before to game on your TrueOS system. We're going to have a look today at playonbsd, how it works with TrueOS, and what you can expect if you want to give it a try on your own system. Let's dive right in! Once playonbsd is installed, go back to your blank desktop, right-click on the wallpaper, and select terminal. Playonbsd does almost all the configuring for you, but there are still a couple of simple options you'll want to configure to give yourself the best experience. In your open terminal, type: playonbsd. You can also find playonbsd by doing a fast search using Lumina's built-in search function in the start menu after it's been installed. Once opened, a graphical interface greets us with easy to navigate menus and even does most of the work for you. A nice graphical UI that hides the complexity of setting up WINE and Steam, and lets you pick select the game you want, and get it setup Start gaming quicker, without the headache If you're a PC gamer, you should definitely give playonbsd a try! You may be surprised at how well it works. If you want to know ahead of time if your games are well supported or not, head on over to WineHQ and do a search. Many people have tested and provided feedback and even solutions for potential problems with a large variety of video games. This is a great resource if you run into a glitch or other problem. Weird Unix thing: 'cd //' (https://jvns.ca/blog/2017/02/08/weird-unix-things-cd/) So why can you do ‘cd //tmp', and it isn't the same as ‘cd /tmp'? The spec says: An implementation may further simplify curpath by removing any trailing characters that are not also leading characters, replacing multiple non-leading consecutive characters with a single , and replacing three or more leading characters with a single . If, as a result of this canonicalization, the curpath variable is null, no further steps shall be taken. “So! We can replace “three or more leading / characters with a single slash”. That does not say anything about what to do when there are 2 / characters though, which presumably is why cd //tmp leaves you at //tmp.” A pathname that begins with two successive slashes may be interpreted in an implementation-defined manner So what is it for? Well, the blog did a bit of digging and came up with this stackoverflow answer (http://unix.stackexchange.com/questions/256497/on-what-systems-is-foo-bar-different-from-foo-bar/256569#256569) In cygwin and some other systems // is treated as a unix-ified version of , to access UNC windows file sharing paths like servershare Perforce, the vcs, uses // to denote a path relative to the depot It seems to have been used in the path for a bunch of different network file systems, but also for myriad other things Testing out snapshots in Apple's next-generation APFS file system (https://arstechnica.com/apple/2017/02/testing-out-snapshots-in-apples-next-generation-apfs-file-system/) Adam Leventhal takes his DTrace hammer to Apple's new file system to see what is going on Back in June, Apple announced its new upcoming file system: APFS, or Apple File System. There was no mention of it in the WWDC keynote, but devotees needed no encouragement. They picked over every scintilla of data from the documentation on Apple's developer site, extrapolating, interpolating, eager for whatever was about to come. In the WWDC session hall, the crowd buzzed with a nervous energy, eager for the grand unveiling of APFS. I myself badge-swapped my way into the conference just to get that first glimpse of Apple's first original filesystem in the 30+ years since HFS Apple's presentation didn't disappoint the hungry crowd. We hoped for a modern filesystem, optimized for next generation hardware, rich with features that have become the norm for data centers and professionals. With APFS, Apple showed a path to meeting those expectations. Dominic Giampaolo and Eric Tamura, leaders of the APFS team, shared performance optimizations, data integrity design, volume management, efficient storage of copied data, and snapshots—arguably the feature of APFS most directly in the user's control. It's 2017, and Apple already appears to be making good on its promise with the revelation that the forthcoming iOS 10.3 will use APFS. The number of APFS tinkerers using it for their personal data has instantly gone from a few hundred to a few million. Beta users of iOS 10.3 have already made the switch apparently without incident. They have even ascribed unscientifically-significant performance improvements to APFS. Previously Adam had used DTrace to find a new syscall introduced in OS X, fs_snapshot, but he had not dug into how to use it. Now it seems, the time has come Learning from XNU and making some educated guesses, I wrote my first C program to create an APFS snapshot. This section has a bit of code, which you can find in this Github repo (https://github.com/ahl/apfs) That just returned “fs_snapshot: Operation not permitted” So, being Adam, he used DTrace to figure out what the problem was Running this DTrace script in one terminal while running the snapshot program in another shows the code flow through the kernel as the program executes In the code flow, the privcheckcred() function jumps out as a good place to continue because of its name, the fact that fs_snapshot calls it directly, and the fact that it returns 1 which corresponds with EPERM, the error we were getting. Turns out, it just requires some sudo With a little more testing I wrote my own version of Apple's unreleased snapUtil command from the WWDC demo We figured out the proper use of the fssnapshot system call and reconstructed the WWDC snapUtil. But all this time an equivalent utility has been lurking on macOS Sierra. If you look in /System/Library/Filesystems/apfs.fs/Contents/Resources/, Apple has included a number of APFS-related utilities, including apfssnapshot (and, tantalizingly, a tool called hfs_convert). Snapshots let you preserve state to later peruse; we can also revert an APFS volume to a previous state to restore its contents. The current APFS semantics around rollback are a little odd. The revert operation succeeds, but it doesn't take effect until the APFS volume is next mounted Another reason Apple may not have wanted people messing around with snapshots is that the feature appears to be incomplete. Winding yourself into a state where only a reboot can clear a mounted snapshot is easy, and using snapshots seems to break some of the diskutil APFS output It is interesting to see what you can do with DTrace, as well as to see what a DTrace and ZFS developer things of APFS *** Interview - Tom Jones - tj@enoti.me (mailto:tj@enoti.me) Replacing the BSD Sockets API *** News Roundup FreeBSD rc.d script to map ethernet device names by MAC address (https://github.com/eborisch/ethname) Self-contained FreeBSD rc.d script for re-naming devices based on their MAC address. I needed it due to USB Ethernet devices coming up in different orders across OS upgrades. Copy ethname into /usr/local/etc/rc.d/ Add the following to rc.conf: ethnameenable="YES" ethnamedevices="em0 ue0 ue1" # Replace with desired devices to rename Create /usr/local/etc/ifmap in the following format: 01:23:45:67:89:ab eth0 01:23:45:67:89:ac eth1 That's it. Use ifconfig_="" settings in rc.conf with the new names. I know MFSBSD has something like this, but a polished up hybrid of the two should likely be part of the base system if something is not already available This would be a great “Junior Job”, if say, a viewer wanted to get started with their first FreeBSD patch *** Mog: A different take on the Unix tool cat (https://github.com/witchard/mog) Do you abuse cat to view files? Did you know cat is meant for concatenating files, meaning: cat part1 part2 part3 > wholething.txt mog is a tool for actually viewing files, and it adds quite a few nice features Syntax highlight scripts Print a hex dump of binary files Show details of image files Perform objdump on executables List a directory mog reads the $HOME/.mogrc config file which describes a series of operations it can do in an ordered manner. Each operation has a match command and an action command. For each file you give to mog it will test each match command in turn, when one matches it will perform the action. A reasonably useful config file is generated when you first run it. How Unix erases things when you type a backspace while entering text (https://utcc.utoronto.ca/~cks/space/blog/unix/HowUnixBackspaces) Yesterday I mentioned in passing that printing a DEL character doesn't actually erase anything. This raises an interesting question, because when you're typing something into a Unix system and hit your backspace key, Unix sure erases the last character that you entered. So how is it doing that? The answer turns out to be basically what you'd expect, although the actual implementation rapidly gets complex. When you hit backspace, the kernel tty line discipline rubs out your previous character by printing (in the simple case) Ctrl-H, a space, and then another Ctrl-H. Of course just backing up one character is not always the correct way of erasing input, and that's when it gets complicated for the kernel. To start with we have tabs, because when you (the user) backspace over a tab you want the cursor to jump all the way back, not just move back one space. The kernel has a certain amount of code to work out what column it thinks you're on and then back up an appropriate number of spaces with Ctrl-Hs. Then we have the case when you quoted a control character while entering it, eg by typing Ctrl-V Ctrl-H; this causes the kernel to print the Ctrl-H instead of acting on it, and it prints it as the two character sequence ^H. When you hit backspace to erase that, of course you want both (printed) characters to be rubbed out, not just the 'H'. So the kernel needs to keep track of that and rub out two characters instead of just one. Chris then provides an example, from IllumOS, of the kernel trying to deal with multibyte characters FreeBSD also handles backspacing a space specially, because you don't need to actually rub that out with a 'b b' sequence; you can just print a plain b. Other kernels don't seem to bother with this optimization. The FreeBSD code for this is in sys/kern/ttyttydisc.c in the ttydiscrubchar function PS: If you want to see the kernel's handling of backspace in action, you usually can't test it at your shell prompt, because you're almost certainly using a shell that supports command line editing and readline and so on. Command line editing requires taking over input processing from the kernel, and so such shells are handling everything themselves. My usual way to see what the kernel is doing is to run 'cat >/dev/null' and then type away. And you thought the backspace key would be simple... *** FreeBSD ports now have Wayland (http://www.freshports.org/graphics/wayland/) We've discussed the pending Wayland work, but we wanted to point you today to the ports which are in mainline FreeBSD ports tree now. First of all, (And I was wondering how they would deal with this) it has landed in the “graphics” category, since Wayland is the Anti-X11, putting it in x11/ didn't make a lot of sense. Couple of notes before you start installing new packages and expecting wayland to “just work” First, this does require that you have working DRM from the kernel side. You'll want to grab TrueOS or build from Matt Macy's FreeBSD branches on GitHub before testing on any kind of modern Intel GPU. Nvidia with modesetting should be supported. Next, not all desktops will “just work”. You may need to grab experimental Weston for compositor. KDE / Gnome (And Lumina) and friends will grow Wayland support in the future, so don't expect to just fire up $whatever and have it all work out of box. Feedback is needed! This is brand new functionality for FreeBSD, and the maintainers will want to hear your results. For us on the TrueOS side we are interested as well, since we want to port Lumina over to Wayland soon(ish) Happy Experimenting! *** Beastie Bits Faces of FreeBSD 2017: Joseph Kong (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2017-joseph-kong/) OPNsense 17.1 “Eclectic Eagle”, based on FreeBSD 11 Released (https://opnsense.org/opnsense-17-1-released/) Why you should start programming on UNIX (http://www.koszek.com/blog/2017/01/28/why-you-should-start-programming-on-unix/) OpenSMTPD Mail Filtering (http://eradman.com/posts/opensmtpd-filtering.html) Feedback/Questions Zane - Databases and Jails (http://pastebin.com/89AyGe5F) Mohammad - USB Install (http://pastebin.com/Te8sz9id) Chuck - Updating Jails (http://pastebin.com/G2SzahWL) David - Lumina / LXQt (http://pastebin.com/71ExJLpL) ***
This week on BSDNow, we've got all sorts of post-holiday goodies to share. New OpenSSL APIs, Dtrace, OpenBSD This episode was brought to you by Headlines OpenSSL 1.1 API migration path, or the lack thereof (https://www.mail-archive.com/tech@openbsd.org/msg36437.html) As many of you will already be aware, the OpenSSL 1.1.0 release intentionally introduced significant API changes from the previous release. In summary, a large number of data structures that were previously publically visible have been made opaque, with accessor functions being added in order to get and set some of the fields within these now opaque structs. It is worth noting that the use of opaque data structures is generally beneficial for libraries, since changes can be made to these data structures without breaking the ABI. As such, the overall direction of these changes is largely reasonable. However, while API change is generally necessary for progression, in this case it would appear that there is NO transition plan and a complete disregard for the impact that these changes would have on the overall open source ecosystem. So far it seems that the only approach is to place the migration burden onto each and every software project that uses OpenSSL, pushing significant code changes to each project that migrates to OpenSSL 1.1, while maintaining compatibility with the previous API. This is forcing each project to provide their own backwards compatibility shims, which is practically guaranteeing that there will be a proliferation of variable quality implementations; it is almost a certainty that some of these will contain bugs, potentially introducing security issues or memory leaks. I think this will be a bigger issue for other operating systems that do not have the flexibility of the ports tree to deliver a newer version of OpenSSL. If a project switches from the old API to the new API, and the OS only provides the older branch of OpenSSL, how can the application work? Of course, this leaves the issue, if application A wants OpenSSL 1.0, and application B only works with OpenSSL 1.1, how does that work? Due to a number of factors, software projects that make use of OpenSSL cannot simply migrate to the 1.1 API and drop support for the 1.0 API - in most cases they will need to continue to support both. Firstly, I am not aware of any platform that has shipped a production release with OpenSSL 1.1 - any software that supported OpenSSL 1.1 only, would effectively be unusable on every platform for the time being. Secondly, the OpenSSL 1.0.2 release is supported until the 31st of December 2019, while OpenSSL 1.1.0 is only supported until the 31st of August 2018 - any LTS style release is clearly going to consider shipping with 1.0.2 as a result. Platforms that are attempting to ship with OpenSSL 1.1 are already encountering significant challenges - for example, Debian currently has 257 packages (out of 518) that do not build against OpenSSL 1.1. There are also hidden gotchas for situations where different libraries are linked against different OpenSSL versions and then share OpenSSL data structures between them - many of these problems will be difficult to detect since they only fail at runtime. It will be interesting to see what happens with OpenSSL, and LibreSSL Hopefully, most projects will decide to switch to the cleaner APIs provided by s2n or libtls, although they do not provide the entire functionality of the OpenSSL API. Hacker News comments (https://news.ycombinator.com/item?id=13284648) *** exfiltration via receive timing (http://www.tedunangst.com/flak/post/exfiltration-via-receive-timing) Another similar way to create a backchannel but without transmitting anything is to introduce delays in the receiver and measure throughput as observed by the sender. All we need is a protocol with transmission control. Hmmm. Actually, it's easier (and more reliable) to code this up using a plain pipe, but the same principle applies to networked transmissions. For every digit we want to “send” back, we sleep a few seconds, then drain the pipe. We don't care about the data, although if this were a video file or an OS update, we could probably do something useful with it. Continuously fill the pipe with junk data. If (when) we block, calculate the difference between before and after. This is a our secret backchannel data. (The reader and writer use different buffer sizes because on OpenBSD at least, a writer will stay blocked even after a read depending on the space that opens up. Even simple demos have real world considerations.) In this simple example, the secret data (argv) is shared by the processes, but we can see that the writer isn't printing them from its own address space. Nevertheless, it works. Time to add random delays and buffering to firewalls? Probably not. An interesting thought experiment that shows just how many ways there are to covertly convey a message *** OpenBSD Desktop in about 30 Minutes (https://news.ycombinator.com/item?id=13223351) Over at hackernews we have a very non-verbose, but handy guide to getting to a OpenBSD desktop in about 30 minutes! First, the guide will assume you've already installed OpenBSD 6.0, so you'll need to at least be at the shell prompt of your freshly installed system to begin. With that, now its time to do some tuning. Editing some resource limits in login.conf will be our initial task, upping some datasize tunables to 2GB Next up, we will edit some of the default “doas” settings to something a bit more workable for desktop computing Another handy trick, editing your .profile to have your PKG_PATH variables set automatically will make One thing some folks may overlook, but disabling atime can speed disk performance (which you probably don't care about atime on your desktop anyway), so this guide will show you what knobs to tweak in /etc/fstab to do so After some final WPA / Wifi configuration, we then drop to “mere mortal” mode and begin our package installations. In this particular guide, he will be setting up Lumina Desktop (Which yes, it is on OpenBSD) A few small tweaks later for xscreensaver and your xinitrc file, then you are ready to run “startx” and begin your desktop session! All in all, great guide which if you are fast can probably be done in even less than 30 minutes and will result in a rock-solid OpenBSD desktop rocking Lumina none-the-less. *** How DTrace saved Christmas (https://hackernoon.com/dtrace-at-home-145ba773371e) Adam Leventhal, one of the co-creators of DTrace, wrote up this post about how he uses DTrace at home, to save Christmas I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I'm glad I don't have to do 2016 again. It was just before midnight and either I'd make the card that night or leave an empty space on our friends' refrigerators. Adobe Illustrator had other ideas: “Unable to set maximum number of files to be opened” I'm not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions were working for me, and — inspired by Sara Mauskopf's excellent post (https://medium.com/startup-grind/how-to-start-a-company-with-no-free-time-b70fbe7b918a#.uujdblxc6) — I was rapidly running out of the time bounds for the project. Enough; I'd just DTrace it. A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can't imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I'm forever grateful to the Apple team that ported it to Mac OS X) Illustrator is failing on setrlimit(2) and blowing up as result. Let's confirm that it is in fact returning -1:$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }' dtrace: description 'syscall::setrlimit:return' matched 1 probe CPU ID FUNCTION:NAME 0 532 setrlimit:return -1 1 There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let's confirm that it is in fact setting the files limit and check the value to which it's being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process's address space: $ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlimcur, this->r.rlimmax); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe CPU ID FUNCTION:NAME 0 531 setrlimit:entry 1008 2800 7fffffffffffffff Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMITNOFILE | _RLIMITPOSIX_FLAG) The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy: $ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlimt *)alloca(sizeof (rlimt)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlimt), sizeof (rlimt)); }' dtrace: description 'syscall::setrlimit:entry' matched 1 probe dtrace: could not enable tracing: Permission denied Oh right. Thank you SIP (System Integrity Protection). This is a new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it's new I haven't yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It's easy enough to disable, but I'm reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb + After using DTrace to get the address of the setrlimit function, Adam used lldb to change the result before it got back to the application: (lldb) break set -n _init Breakpoint 1: 47 locations. (lldb) run … (lldb) di -s 0x1006e5b72 -c 1 0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit (lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90 (lldb) di -s 0x1006e5b72 -c 4 0x1006e5b72: xorl %eax, %eax 0x1006e5b74: nop 0x1006e5b75: nop 0x1006e5b76: nop Next I just did a process detach and got on with making that holiday card… DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn't write a program doesn't mean you can't fix it. News Roundup Say my Blog's name! (https://functionallyparanoid.com/2016/12/22/say-my-blogs-name/) Brian Everly over at functionally paranoid has a treat for us today. Let us give you a moment to get the tin-foil hats on… Ok, done? Let's begin! He starts off with a look at physical security. He begins by listing your options: BIOS passwords – Not something I'm typically impressed with. Most can be avoided by opening up the machine, closing a jumper and powering it up to reset the NVRAM to factory defaults. I don't even bother with them. Full disk encryption – This one really rings my bell in a positive way. If you can kill power to the box (either because the bad actor has to physically steal it and they aren't carrying around a pile of car batteries and an inverter or because you can interrupt power to it some other way), then the disk will be encrypted. The other beauty of this is that if a drive fails (and they all do eventually) you don't have to have any privacy concerns about chucking it into an electronics recycler (or if you are a bad, bad person, into a landfill) because that data is effectively gibberish without the key (or without a long time to brute force it). Two factor auth for logins – I like this one as well. I'm not a fan of biometrics because if your fingerprint is compromised (yes, it can happen – read (https://www.washingtonpost.com/news/federal-eye/wp/2015/07/09/hack-of-security-clearance-system-affected-21-5-million-people-federal-authorities-say/) about the department of defense background checks that were extracted by a bad agent – they included fingerprint images) you can't exactly send off for a new finger. Things like the YubiKey (https://www.yubico.com/) are pretty slick. They require that you have the physical hardware key as well as the password so unless the bad actor lifted your physical key, they would have a much harder time with physical access to your hardware. Out of those options, Brian mentions that he uses disk encryption and yubi-key for all his secure network systems. Next up is network segmentation, in this case the first thing to do is change your admin password for any ISP supplied modem / router. He goes on to scare us of javascript attacks being used not against your local machine, but instead non WAN exposed router admin interface. Scary Stuff! For added security, naturally he firewalls the router by plugging in the LAN port to a OpenBSD box which does the 2nd layer of firewall / router protection. What about privacy and browsing? Here's some more of his tips: I use Unbound as my DNS resolver on my local network (with all UDP port 53 traffic redirected to it by pf so I don't have to configure anything on the clients) and then forward the traffic to DNSCrypt Proxy, caching the results in Unbound. I notice ZERO performance penalty for this and it greatly enhances privacy. This combination of Unbound and DNSCrypt Proxy works very well together. You can even have redundancy by having multiple upstream resolvers running on different ports (basically run the DNSCrypt Proxy daemon multiple times pointing to different public resolvers). I also use Firefox exclusively for my web browsing. By leveraging the tips on this page (https://www.privacytools.io/), you can lock it down to do a great job of privacy protection. The fact that your laptop's battery drain rate can be used to fingerprint your browser completely trips me out but hey – that's the world we live in.' What about the cloud you may ask? Well Brian has a nice solution for that as well: I recently decided I would try to live a cloud-free life and I'll give you a bit of a synopsis on it. I discovered a wonderful Open Source project called FreeNAS (http://www.freenas.org/). What this little gem does is allow you to install a FreeBSD/zfs file server appliance on amd64 hardware and have a slick administrative web interface for managing it. I picked up a nice SuperMicro motherboard and chassis that has 4 hot swap drive bays (and two internal bays that I used to mirror the boot volume on) and am rocking the zfs lifestyle! (Thanks Alan Jude!) One of the nicest features of the FreeNAS is that it provides the ability to leverage the FreeBSD jail functionality in an easy to use way. It also has plugins but the security on those is a bit sketchy (old versions of libraries, etc.) so I decided to roll my own. I created two jails – one to run OwnCloud (yeah, I know about NextCloud and might switch at some point) and the other to run a full SMTP/IMAP email server stack. I used Lets Encrypt (https://letsencrypt.org/) to generate the SSL certificates and made sure I hit an A on SSLLabs (https://www.ssllabs.com/) before I did anything else. His post then goes in to talk about Backups and IoT devices, something else you need to consider in this truely paranoid world we are forced to live in. We even get a nice shout-out near the end! Enter TarSnap (http://www.tarsnap.com/) – a company that advertises itself as “Online Backups for the Truly Paranoid”. It brings a tear to my eye – a kindred spirit! :-) Thanks again to Alan Jude and Kris Moore from the BSD Now podcast (http://www.bsdnow.tv/) for turning me onto this company. It has a very easy command syntax (yes, it isn't a GUI tool – suck it up buttercup, you wanted to learn the shell didn't you?) and even allows you to compile the thing from source if you want to.” We've only covered some of the highlights here, but you really should take a few moments of your time today and read this top to bottom. Lots of good tips here, already thinking how I can secure my home network better. The open source book: “Producing Open Source Software” (http://producingoss.com/en/producingoss.pdf) “How to Run a Successful Free Software Project” by Karl Fogel 9 chapters and over 200 pages of content, plus many appendices Some interesting topics include: Choosing a good name version control bug tracking creating developer guidelines setting up communications channels choosing a license (although this guide leans heavily towards the GPL) setting the tone of the project joining or creating a Non-Profit Organization the economics of open source release engineering, packaging, nightly builds, etc how to deal with forks A lot of good information packaged into this ebook This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License *** DTrace Flamegraphs for node.js on FreeBSD (http://www.venshare.com/dtrace-flamegraphs-for-freebsd-and-node-js-2/) One of the coolest tools built on top of DTrace is flamegraphs They are a very accurate, and visual way to see where a program is spending its time, which can tell you why it is slow, or where it could be improved. Further enhancements include off-cpu flame graphs, which tell you when the program is doing nothing, which can also be very useful > Recently BSD UNIXes are being acknowledged by the application development community as an interesting operating system to deploy to. This is not surprising given that FreeBSD had jails, the original container system, about 17 years ago and a lot of network focused businesses such as netflix see it as the best way to deliver content. This developer interest has led to hosting providers supporting FreeBSD. e.g. Amazon, Azure, Joyent and you can get a 2 months free instance at Digital Ocean. DTrace is another vital feature for anyone who has had to deal with production issues and has been in FreeBSD since version 9. As of FreeBSD 11 the operating system now contains some great work by Fedor Indutny so you can profile node applications and create flamegraphs of node.js processes without any additional runtime flags or restarting of processes. This is one of the most important things about DTrace. Many applications include some debugging functionality, but they require that you stop the application, and start it again in debugging mode. Some even require that you recompile the application in debugging mode. Being able to attach DTrace to an application, while it is under load, while the problem is actively happening, can be critical to figuring out what is going on. In order to configure your FreeBSD instance to utilize this feature make the following changes to the configuration of the server. Load the DTrace module at boot Increase some DTrace limits Install node with the optional DTrace feature compiled in Follow the generic node.js flamegraph tutorial (https://nodejs.org/en/blog/uncategorized/profiling-node-js/) > I hope you find this article useful. The ability to look at a runtime in this manor has saved me twice this year and I hope it will save you in the future too. My next post on freeBSD and node.js will be looking at some scenarios on utilising the ZFS features. Also check out Brendan Gregg's ACM Queue Article (http://queue.acm.org/detail.cfm?id=2927301) “The Flame Graph: This visualization of software execution is a new necessity for performance profiling and debugging” SSHGuard 2.0 Call for Testing (https://sourceforge.net/p/sshguard/mailman/message/35580961/) SSHGuard is a tool for monitoring brute force attempts and blocking them It has been a favourite of mine for a while because it runs as a pipe from syslogd, rather than reading the log files from the disk A lot of work to get SSHGuard working with new log sources (journalctl, macOS log) and backends (firewalld, ipset) has happened in 2.0. The new version also uses a configuration file. Most importantly, SSHGuard has been split into several processes piped into one another (sshg-logmon | sshg-parser | sshg-blocker | sshg-fw). sshg-parser can run with capsicum(4) and pledge(2). sshg-blocker can be sandboxed in its default configuration (without pid file, whitelist, blacklisting) and has not been tested sandboxed in other configurations. Breaking the processes up so that the sensitive bits can be sandboxes is very nice to see *** Beastie Bits pjd's 2007 paper from AsiaBSDCon: “Porting the ZFS file system to the FreeBSD operating system” (https://2007.asiabsdcon.org/papers/P16-paper.pdf) A Message From the FreeBSD Foundation (https://vimeo.com/user60888329) Remembering Roger Faulkner, Unix Champion (http://thenewstack.io/remembering-roger-faulkner/) and A few HN comments (including Bryan Cantrill) (https://news.ycombinator.com/item?id=13293596) Feedback/Questions Peter - TrueOS Network (http://pastebin.com/QtyJeHMk) Chris - Remote Desktop (http://pastebin.com/ru726VTV) Goetz - Geli on Serial (http://pastebin.com/LQZPgF5g) Joe - BGP (http://pastebin.com/jFeL8zKX) Alejandro - BSD Router (http://pastebin.com/Xq9cbmfn) ***
This week on BSDNow, we have a very special guest joining us to tell us a tale of the early days in BSD history. That plus some new OpenSSH goodness, shell scripting utilities and much more. Stay tuned for your place to B...SD! This episode was brought to you by Headlines Call For Testing: OpenSSH 7.4 (http://marc.info/?l=openssh-unix-dev&m=148167688911316&w=2) Getting ready to head into the holidays for for the end of 2016 means some of us will have spare time on our hands. What a perfect time to get some call for testing work done! Damien Miller has issued a public CFT for the upcoming OpenSSH 7.4 release, which considering how much we all rely on SSH I would expect will get some eager volunteers for testing. What are some of the potential breakers? “* This release removes server support for the SSH v.1 protocol. ssh(1): Remove 3des-cbc from the client's default proposal. 64-bit block ciphers are not safe in 2016 and we don't want to wait until attacks like SWEET32 are extended to SSH. As 3des-cbc was the only mandatory cipher in the SSH RFCs, this may cause problems connecting to older devices using the default configuration, but it's highly likely that such devices already need explicit configuration for key exchange and hostkey algorithms already anyway. sshd(8): Remove support for pre-authentication compression. Doing compression early in the protocol probably seemed reasonable in the 1990s, but today it's clearly a bad idea in terms of both cryptography (cf. multiple compression oracle attacks in TLS) and attack surface. Pre-auth compression support has been disabled by default for >10 years. Support remains in the client. ssh-agent will refuse to load PKCS#11 modules outside a whitelist of trusted paths by default. The path whitelist may be specified at run-time. sshd(8): When a forced-command appears in both a certificate and an authorized keys/principals command= restriction, sshd will now refuse to accept the certificate unless they are identical. The previous (documented) behaviour of having the certificate forced-command override the other could be a bit confusing and error-prone. sshd(8): Remove the UseLogin configuration directive and support for having /bin/login manage login sessions.“ What about new features? 7.4 has some of those to wake you up also: “* ssh(1): Add a proxy multiplexing mode to ssh(1) inspired by the version in PuTTY by Simon Tatham. This allows a multiplexing client to communicate with the master process using a subset of the SSH packet and channels protocol over a Unix-domain socket, with the main process acting as a proxy that translates channel IDs, etc. This allows multiplexing mode to run on systems that lack file- descriptor passing (used by current multiplexing code) and potentially, in conjunction with Unix-domain socket forwarding, with the client and multiplexing master process on different machines. Multiplexing proxy mode may be invoked using "ssh -O proxy ..." sshd(8): Add a sshdconfig DisableForwaring option that disables X11, agent, TCP, tunnel and Unix domain socket forwarding, as well as anything else we might implement in the future. Like the 'restrict' authorizedkeys flag, this is intended to be a simple and future-proof way of restricting an account. sshd(8), ssh(1): Support the "curve25519-sha256" key exchange method. This is identical to the currently-support method named "curve25519-sha256@libssh.org". sshd(8): Improve handling of SIGHUP by checking to see if sshd is already daemonised at startup and skipping the call to daemon(3) if it is. This ensures that a SIGHUP restart of sshd(8) will retain the same process-ID as the initial execution. sshd(8) will also now unlink the PidFile prior to SIGHUP restart and re-create it after a successful restart, rather than leaving a stale file in the case of a configuration error. bz#2641 sshd(8): Allow ClientAliveInterval and ClientAliveCountMax directives to appear in sshd_config Match blocks. sshd(8): Add %-escapes to AuthorizedPrincipalsCommand to match those supported by AuthorizedKeysCommand (key, key type, fingerprint, etc.) and a few more to provide access to the contents of the certificate being offered. Added regression tests for string matching, address matching and string sanitisation functions. Improved the key exchange fuzzer harness.“ Get those tests done and be sure to send feedback, both positive and negative. *** How My Printer Caused Excessive Syscalls & UDP Traffic (https://zinascii.com/2014/how-my-printer-caused-excessive-syscalls.html) “3,000 syscalls a second, on an idle machine? That doesn't seem right. I just booted this machine. The only processes running are those required to boot the SmartOS Global Zone, which is minimal.” This is a story from 2014, about debugging a machine that was being slowed down by excessive syscalls and UDP traffic. It is also an excellent walkthrough of the basics of DTrace “Well, at least I have DTrace. I can use this one-liner to figure out what syscalls are being made across the entire system.” dtrace -n 'syscall:::entry { @[probefunc,probename] = count(); }' “Wow! That is a lot of lwpsigmask calls. Now that I know what is being called, it's time to find out who is doing the calling? I'll use another one-liner to show me the most common user stacks invoking lwpsigmask.” dtrace -n 'syscall::lwp_sigmask:entry { @[ustack()] = count(); }' “Okay, so this mdnsd code is causing all the trouble. What is the distribution of syscalls for the mdnsd program?” dtrace -n 'syscall:::entry /execname == "mdnsd"/ { @[probefunc] = count(); } tick-1s { exit(0); }' “Lots of signal masking and polling. What the hell! Why is it doing this? What is mdnsd anyways? Is there a man page? Googling for mdns reveals that it is used for resolving host names in small networks, like my home network. It uses UDP, and requires zero configuration. Nothing obvious to explain why it's flipping out. I feel helpless. I turn to the only thing I can trust, the code.” “Woah boy, this is some messy looking code. This would not pass illumos cstyle checks. Turns out this is code from Darwin—the kernel of OSX.” “Hmmm…an idea pops into my computer animal brain. I wonder…I wonder if my MacBook is also experiencing abnormal syscall rates? Nooo, that can't be it. Why would both my SmartOS server and MacBook both have the same problem? There is no good technical reason to link these two. But, then again, I'm dealing with computers here, and I've seen a lot of strange things over the years—I switch to my laptop.” sudo dtrace -n 'syscall::: { @[execname] = count(); } tick-1s { exit(0); }' Same thing, except mdns is called discoverd on OS X “I ask my friend Steve Vinoski to run the same DTrace one-liner on his OSX machines. He has both Yosemite and the older Mountain Lion. But, to my dismay, neither of his machines are exhibiting high syscall rates. My search continues.” “Not sure what to do next, I open the OSX Activity Monitor. In desperation I click on the Network tab.” “ HOLE—E—SHIT! Two-Hundred-and-Seventy Million packets received by discoveryd. Obviously, I need to stop looking at code and start looking at my network. I hop back onto my SmartOS machine and check network interface statistics.” “Whatever is causing all this, it is sending about 200 packets a second. At this point, the only thing left to do is actually inspect some of these incoming packets. I run snoop(1M) to collect events on the e1000g0 interface, stopping at about 600 events. Then I view the first 15.” “ A constant stream of mDNS packets arriving from IP 10.0.1.8. I know that this IP is not any of my computers. The only devices left are my iPhone, AppleTV, and Canon printer. Wait a minute! The printer! Two days earlier I heard some beeping noises…” “I own a Canon PIXMA MG6120 printer. It has a touch interface with a small LCD at the top, used to set various options. Since it sits next to my desk I sometimes lay things on top of it like a book or maybe a plate after I'm done eating. If I lay things in the wrong place it will activate the touch interface and cause repeated pressing. Each press makes a beeping noise. If the object lays there long enough the printer locks up and I have to reboot it. Just such events occurred two days earlier.” “I fire up dladm again to monitor incoming packets in realtime. Then I turn to the printer. I move all the crap off of it: two books, an empty plate, and the title for my Suzuki SV650 that I've been meaning to sell for the last year. I try to use the touch screen on top of the printer. It's locked up, as expected. I cut power to the printer and whip my head back to my terminal.” No more packet storm “Giddy, I run DTrace again to count syscalls.” “I'm not sure whether to laugh or cry. I laugh, because, LOL computers. There's some new dumb shit you deal with everyday, better to roll with the punches and laugh. You live longer that way. At least I got to flex my DTrace muscles a bit. In fact, I felt a bit like Brendan Gregg when he was debugging why OSX was dropping keystrokes.” “I didn't bother to root cause why my printer turned into a UDP machine gun. I don't intend to either. I have better things to do, and if rebooting solves the problem then I'm happy. Besides, I had to get back to what I was trying to do six hours before I started debugging this damn thing.” There you go. The Internet of Terror has already been on your LAN for years. Making Getaddrinfo Concurrent in Python on Mac OS and BSD (https://emptysqua.re/blog/getaddrinfo-cpython-mac-and-bsd/) We have a very fun blog post today to pass along originally authored by “A. Jesse Jiryu Davis”. Specifically the tale of one man's quest to unify the Getaddrinfo in Python with Mac OS and BSD. To give you a small taste of this tale, let us pass along just the introduction “Tell us about the time you made DNS resolution concurrent in Python on Mac and BSD. No, no, you do not want to hear that story, my friends. It is nothing but old lore and #ifdefs. But you made Python more scalable. The saga of Steve Jobs was sung to you by a mysterious wizard with a fanciful nickname! Tell us! Gather round, then. I will tell you how I unearthed a lost secret, unbound Python from old shackles, and banished an ancient and horrible Mutex Troll. Let us begin at the beginning.“ Is your interest piqued? It should be. I'm not sure we could do this blog post justice trying to read it aloud here, but definetly recommend if you want to see how he managed to get this bit of code working cross platform. (And it's highly entertaining as well) “A long time ago, in the 1980s, a coven of Berkeley sorcerers crafted an operating system. They named it after themselves: the Berkeley Software Distribution, or BSD. For generations they nurtured it, growing it and adding features. One night, they conjured a powerful function that could resolve hostnames to IPv4 or IPv6 addresses. It was called getaddrinfo. The function was mighty, but in years to come it would grow dangerous, for the sorcerers had not made getaddrinfo thread-safe.” “As ages passed, BSD spawned many offspring. There were FreeBSD, OpenBSD, NetBSD, and in time, Mac OS X. Each made its copy of getaddrinfo thread safe, at different times and different ways. Some operating systems retained scribes who recorded these events in the annals. Some did not.” The story continues as our hero battles the Mutex Troll and quests for ancient knowledge “Apple engineers are not like you and me — they are a shy and secretive folk. They publish only what code they must from Darwin. Their comings and goings are recorded in no bug tracker, their works in no changelog. To learn their secrets, one must delve deep.” “There is a tiny coven of NYC BSD users who meet at the tavern called Stone Creek, near my dwelling. They are aged and fierce, but I made the Sign of the Trident and supplicated them humbly for advice, and they were kindly to me.” Spoiler: “Without a word, the mercenary troll shouldered its axe and trudged off in search of other patrons on other platforms. Never again would it hold hostage the worthy smiths forging Python code on BSD.” *** Using release(7) to create FreeBSD images for OpenStack (https://diegocasati.com/2016/12/13/using-release7-to-create-freebsd-images-for-openstack-yes-you-can-do-it/) Following a recent episode where we covered a walk through on how to create FreeBSD guest OpenStack images, we wondered if it would be possible to integrate this process into the FreeBSD release(7) process, so they images could be generated consistently and automatically Being the awesome audience that you are, one of you responded by doing exactly that “During a recent BSDNow podcast, Allan and Kris mentioned that it would be nice to have a tutorial on how to create a FreeBSD image for OpenStack using the official release(7) tools. With that, it came to me that: #1 I do have access to an OpenStack environment and #2 I am interested in having FreeBSD as a guest image in my environment. Looks like I was up for the challenge.” “Previously, I've had success running FreeBSD 11.0-RELEASE on OpenStack but more could/should be done. For instance, as suggested by Allan, wouldn't be nice to deploy the latest code from FreeBSD ? Running -STABLE or even -CURRENT ? Yes, it would. Also, wouldn't it be nice to customize these images for a specific need? I'd say ‘Yes' for that as well.” “After some research I found that the current openstack.conf file, located at /usr/src/release/tools/ could use some extra tweaks to get where I wanted. I've created and attached that to a bugzilla on the same topic. You can read about that here (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213396).” Steps: Fetch the FreeBSD source code and extract it under /usr/src Once the code is in place, follow the regular process of build(7) and perform a make buildworld buildkernel Change into the release directory (/usr/src/release) and perform a make cloudware make cloudware-release WITH_CLOUDWARE=yes CLOUDWARE=OPENSTACK VMIMAGE=2G “That's it! This will generate a qcow2 image with 1.4G in size and a raw image of 2G. The entire process uses the release(7) toolchain to generate the image and should work with newer versions of FreeBSD.” + The patch has already been committed to FreeBSD (https://svnweb.freebsd.org/base?view=revision&revision=310047) Interview - Rod Grimes - rgrimes@freebsd.org (mailto:rgrimes@freebsd.org) Want to help fund the development of GPU Passthru? Visit bhyve.org (http://bhyve.org/) *** News Roundup Configuring the FreeBSD automounter (http://blog.khubla.com/freebsd/configuring-the-freebsd-automounter) Ever had to configure the FreeBSD auto-mounting daemon? Today we have a blog post that walks us through a few of the configuration knobs you have at your disposal. First up, Tom shows us his /etc/fstab file, and the various UFS partitions he has setup with the ‘noauto' flag so they are not mounted at system boot. His amd.conf file is pretty basic, with just options enabled to restart mounts, and unmount on exit. Where most users will most likely want to pay attention is in the crafting of an amd.map file Within this file, we have the various command-foo which performs mounts and unmounts of targeted disks / file-systems on demand. Pay special attention to all the special chars, since those all matter and a stray or missing ; could be a source of failure. Lastly a few knobs in rc.conf will enable the various services and a reboot should confirm the functionality. *** l2k16 hackathon report: LibreSSL manuals now in mdoc(7) (http://undeadly.org/cgi?action=article&sid=20161114174451) Hackathon report by Ingo Schwarze “Back in the spring two years ago, Kristaps Dzonsons started the pod2mdoc(1) conversion utility, and less than a month later, the LibreSSL project began. During the general summer hackathon in the same year, g2k14, Anthony Bentley started using pod2mdoc(1) for converting LibreSSL manuals to mdoc(7).” “Back then, doing so still was a pain, because pod2mdoc(1) was still full of bugs and had gaping holes in functionality. For example, Anthony was forced to basically translate the SYNOPSIS sections by hand, and to fix up .Fn and .Xr in the body by hand as well. All the same, he speedily finished all of libssl, and in the autumn of the same year, he mustered the courage to commit his work.” “Near the end of the following winter, i improved the pod2mdoc(1) tool to actually become convenient in practice and started work on libcrypto, converting about 50 out of the about 190 manuals. Max Fillinger also helped a bit, converting a handful of pages, but i fear i tarried too much checking and committing his work, so he quickly gave up on the task. After that, almost nothing happened for a full year.” “Now i was finally fed up with the messy situation and decided to put an end to it. So i went to Toulouse and finished the conversion of the remaining 130 manual pages in libcrypto, such that you can now view the documentation of all functions” Interactive Terminal Utility: smenu (https://github.com/p-gen/smenu) Ok, I've made no secret of my love for shell scripting. Well today we have a new (somewhat new to us) tool to bring your way. Have you ever needed to deal with large lists of data, perhaps as the result of a long specially crafted pipe? What if you need to select a specific value from a range and then continue processing? Enter ‘smenu' which can help make your scripting life easier. “smenu is a selection filter just like sed is an editing filter. This simple tool reads words from the standard input, presents them in a cool interactive window after the current line on the terminal and writes the selected word, if any, on the standard output. After having unsuccessfully searched the NET for what I wanted, I decided to try to write my own. I have tried hard to made its usage as simple as possible. It should work, even when using an old vt100 terminal and is UTF-8 aware.“ What this means, is in your interactive scripts, you can much easier present the user with a cursor driven menu to select from a range of possible choices. (Without needing to craft a bunch of dialog flags) Take a look, and hopefully you'll be able to find creative uses for your shell scripts in the future. *** Ubuntu still isn't free software (http://mjg59.dreamwidth.org/45939.html) “Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries. This does not affect your rights under any open source licence applicable to any of the components of Ubuntu. If you need us to approve, certify or provide modified versions for redistribution you will require a licence agreement from Canonical, for which you may be required to pay. For further information, please contact us” “Mark Shuttleworth just blogged (http://insights.ubuntu.com/2016/12/01/taking-a-stand-against-unstable-risky-unofficial-ubuntu-images/) about their stance against unofficial Ubuntu images. The assertion is that a cloud hoster is providing unofficial and modified Ubuntu images, and that these images are meaningfully different from upstream Ubuntu in terms of their functionality and security. Users are attempting to make use of these images, are finding that they don't work properly and are assuming that Ubuntu is a shoddy product. This is an entirely legitimate concern, and if Canonical are acting to reduce user confusion then they should be commended for that.” “The appropriate means to handle this kind of issue is trademark law. If someone claims that something is Ubuntu when it isn't, that's probably an infringement of the trademark and it's entirely reasonable for the trademark owner to take action to protect the value associated with their trademark. But Canonical's IP policy goes much further than that - it can be interpreted as meaning[1] that you can't distribute works based on Ubuntu without paying Canonical for the privilege, even if you call it something other than Ubuntu. [1]: And by "interpreted as meaning" I mean that's what it says and Canonical refuse to say otherwise” “If you ask a copyright holder if you can give a copy of their work to someone else (assuming it doesn't infringe trademark law), and they say no or insist you need an additional contract, it's not free software. If they insist that you recompile source code before you can give copies to someone else, it's not free software. Asking that you remove trademarks that would otherwise infringe trademark law is fine, but if you can't use their trademarks in non-infringing ways, that's still not free software.” “Canonical's IP policy continues to impose restrictions on all of these things, and therefore Ubuntu is not free software.” Beastie Bits OPNsense 16.7.10 released (https://opnsense.org/opnsense-16-7-10-released/) OpenBSD Foundation Welcomes First Iridium Donor: Smartisan (http://undeadly.org/cgi?action=article&sid=20161123193708&mode=expanded&count=8) Jan Koum donates $500,000 to FreeBSD (https://www.freebsdfoundation.org/blog/foundation-announces-new-uranium-donor/) The Soviet Russia, BSD makes you (https://en.wikipedia.org/wiki/DEMOS) Feedback/Questions Jason - Value (http://pastebin.com/gRN4Lzy8) Hamza - Shell Scripting (http://pastebin.com/GZYjRmSR) Blog link (http://aikchar.me/blog/unix-shell-programming-lessons-learned.html) Dave - Migrating to FreeBSD (http://pastebin.com/hEBu3Drp) Dan - Which BSD? (http://pastebin.com/1HpKqCSt) Zach - AMD Video (http://pastebin.com/4Aj5ebns) ***
This week on BSDNow, Allen is away in the UK (For BSDCam), but we still have a full episode for you! Don't miss our interview with This episode was brought to you by Headlines My two year journey to becoming an OS Developer (http://zinascii.com/2016/going-to-joyent.html) A blog post by Ryan Zezeski about how he ended doing OS Development instead of working on application We have featured his posts before, including The illumos SYSCALL Handler (http://zinascii.com/2016/the-illumos-syscall-handler.html) It started in the summer of 2014: I had just left Basho after 3.5 years of working on Riak, when I decided I wanted to become an OS developer. I purchased Solaris Internals, cloned illumos-gate, fired up cscope, and got to work. I hardly knew any C, x86 might as well have been Brainfuck, and, frankly, I knew shit about operating systems. But I was determined. I've always learned best by beating my head against something until it makes sense. I'm not a fast learner; I'm persistent. What others have in ability I make up for in effort. And when it comes to OS internals it's all about work ethic. The more you look, the more you realize it's just another program. The main difference being: it's the program all the other programs run on. My strategy: to pick something, anything, that looked interesting, and write a post describing how it works. I wrote several of these posts in 2014 and 2015. More important, it put me in touch with Roger Faulkner: the creator of truss(1), the Solaris process model, and the real /proc filesystem. At the time I didn't like my interaction with Roger. He explained, in what I would later find out to be his typical gruff manner, that I was wrong; so I concluded he is a prick. But over the years I realized that I was being a brat—he was trying to teach me something and I let my ego get in the way. I've come to view that interaction as a blessing. I interacted with one of the greats, a mentor of my mentor's mentor (a Great Great Mentor). A couple of weeks later something even more surreal happened, at illumos Day 2014. Bryan Cantrill was the last speaker of the day. One of my mentors and someone I admire greatly. He was there to regale us with the story of Joyent's resurrection of lx-branded zones: Linux system call emulation on top of the illumos kernel. But before he would do that he decided to speak about me! I couldn't believe it. I was so overwhelmed that I don't remember most of what he said. I was too busy flipping shit—Bryan Cantrill is on stage, in front of other kernel developers I look up to, saying my name. I was in a dream. It turns out, unknown to me at the time, that he wrote the POSIX queue code for both Solaris and QNX, which I wrote about. He compared me to the great expository technical writers Elliott Organick and Richard Stevens. And it was at this moment that I knew I could do this: I could become an OS developer. Never underestimate the effect kind words can have on someone that looks up to you. There is a lot more to the story, and it is definitely worth the read The story then goes on to talk about his recent run in with Bryan Cantrill > A week from now my two year journey to become an OS developer comes to an end; and a new chapter begins. I don't know what specific things I'm going to work on, but I'm sure it will push me to the limit. I look forward to the challenge. *** Version 1.0 of the Lumina Desktop released (https://lumina-desktop.org/version-1-0-0-released/) After 4 years of development, Lumina Desktop has now hit version 1.0! This release brings with it a slew of new features and support: Completely customizable interface! Rather than having to learn how to use a new layout, change the desktop to suit you instead! Simple shortcuts for any application! The “favorites” system makes it easy to find and launch applications at any time. Extremely lightweight! Allows applications to utilize more of your system hardware and revitalizes older systems! Multiple-monitor support! Each monitor is treated as an independent entity – making it great for presentation systems which use a temporary monitor or for workstations which utilize an array of monitors for various tasks. While originally developed on PC-BSD, it already has been ported to a variety of different platforms, including OpenBSD, DragonFly, NetBSD, Debian and Gentoo Lumina has become the defacto desktop environment for TrueOS (Formerly PC-BSD), and looks like will provide a solid framework to continue growing desktop features. *** n2k16 hackathon report: Ken Westerback on dhclient, bridges, routing and more (http://undeadly.org/cgi?action=article&sid=20160804200232) Next up, we have a report from Ken Westerback talking about the recent OpenBSD hackathon in Prague He starts by telling us about the work in bpf: First order of business, stsp@'s weird setup involving bridges and multiple dhclient clients. A bit of bpf(4) programming to restrict dhclient to handling ethernet packets unicast to its interface worked. Cool. Unfortunately it turned out some lazy dhcp servers always use ethernet broadcasts just because some lesser, non-OpenBSD clients ignore unicast packets until they have configured IP. Classic chicken and egg. So this was backed out just before 6.0. Sigh. Next up, he talks about an idea he had on the flight over, specifically with regard to how DHCP leases are stored, and how keeping the SSID information with them could speed up re-connection times, by only trying leases for current SSID's connected. After a day or so of hacking, it was working! However for $REASONS it was shelved for post 6.0, bummer! He then discusses an on-going project with Peter Hessler on passing along relevant PIDs in response to routing messages generated by kernel from ioctl events. This is something they've been hacking at, in order to allow dhclient to recognize its own routing messages. Sounds like they are both still works-in-progress. However, Ken did get something in for 6.0: Diving back into dhclient code I discovered that in situations where multiple offers were received the unused offers were not being declined and discarded. Despite a clear comment saying that's what was being done! Thus dhclient might gradually use up more and more memory. And possibly be retrying offers that should have been discarded. The fix for this did make 6.0! Yay! In Memoriam Roger Faulkner (https://www.usenix.org/memoriam-roger-faulkner) USENIX has re-released Roger Faulkner's original paper on /proc as a free download The UNIX community recently lost one of its original pioneers, Roger Faulkner, whom one commenter described as “The godfather of post-AT&T UNIX” In his memory, the USENIX group as re-released his original paper on the /proc file-system from 1991. Roger worked in many area's of UNIX, however the process file system /proc was his special baby. “/proc began as a debugger interface superseding ptrace(2) but has evolved into a general interface to the process model.” The original /proc only had a file for each process, not a directory. "Data may be transferred from or to any valid locations in the process's address space by applying lseek(2) to position the file at the virtual address of interest followed by read(2) or write(2)." Processes could be controlled using IOCTLs on the file As the USENIX article states: Roger believed that terrible things were sometimes required to create beautiful abstractions, and his trailblazing work on /proc embodies this burden: the innards may be delicate and nasty ("vile," as Roger might say in his distinguished Carolinian accent)—but the resulting abstractions are breathtaking in their power, scope and robustness. RIP Roger, and thanks for the wonderful UNIX legacy you've left us all. Interview - Myke Geiger - myke@servernorth.net (mailto:myke@servernorth.net) / @mWare (https://twitter.com/mWare) Using FreeBSD at a DSL/Cable ISP *** News Roundup New options in bsdinstall - some sysctls and date/time settings (https://www.reddit.com/r/freebsd/comments/4vxnw3/new_options_in_bsdinstall_some_sysctls_and/) bsdinstall in FreeBSD 11.0 will feature a number of new menus. The first, well allow you to set the date and time. Often on computers that have been in storage, or some embedded type devices that have no RTC, the date will be wildly wrong, and ntpd will refuse to run until the date is correctly set. This feature makes it easy to enter the date and time using dialog(1) The second menu, inspired by the existing ‘services' menu, offers a number of ‘hardening' options This menu allows users to easily enable a number of security features, including: Hide processes running as other users/groups Disable reading the kernel message buffer and debugging processes for unprivileged users Randomize the PID of newly created processes Enable the stack guard Erase /tmp at boot Disable remote syslog Disable sendmail All of these options are off by default, so that an install done with the installer will be the same as an install from source, or an upgrade. A number of these options are candidates to become on-by-default in the future, so the hope is that this menu will get more users to test these features and find any negative interactions with applications or general use, so they can be fixed. *** Rawrite32: the NetBSD image writing tool (https://www.netbsd.org/~martin/rawrite32/) Martin of the NetBSD project has released a new version of his USB imaging tool, rawrite32 For those who've not used this tool before, it is a Windows Application that allows writing NetBSD images directly to USB media (other other disk media) This update brings with it support for writing .xz file, and binary signing This may come in handy for writing other OS images to memory sticks as well, especially for those locked into a windows environment who need to switch. *** ZFS-Snap-Diff -- A pretty interface for viewing what changed after a ZFS snapshot (https://github.com/j-keck/zfs-snap-diff) There are lots of nice little utilities to help create and maintain your ZFS snapshots. However today we have something unique to look at, ‘zfs-snap-diff'. What makes it unique, is that it ships with a built-in golang / angularjs GUI for snapshot management It looks very powerful, including a built-in diff utility, so you can even see the changes in text-files, in addition to downloading files, restoring old versions and more. Its nice to see so many ZFS utilities starting to take off, and evolve file-management further. *** Dtrace Conf 2016 Event Videos (https://www.joyent.com/about/events/2016/dtrace-conf) The videos from Dtrace.conf 2016 have been posted Some highlights: Useful DTrace Intro CTF Everywhere Distributed DTrace DTrace for Apps DTrace json() subroutine Implementing (or not) fds[] in FreeBSD OpenDTrace DTrace performance improvements with always-on instrumentation D Syntactic Sugar DTrace and Go, DTrace and Postgres dtrace.conf(16) wrap-up by Bryan Cantrill (https://www.joyent.com/blog/dtrace-conf-16-wrap-up) Once again, it was an eclectic mix of technologists — and once again, the day got kicked off with me providing an introduction to dtrace.conf and its history. (Just to save you the time filling out your Cantrill Presentation Bingo Card: you can find me punching myself at 16:19, me offering unsolicited personal medical history at 20:11, and me getting trolled by unikernels at 38:25.) The next DTrace.conf isn't until 2020 *** Beastie Bits The BSD Daemon features in Mexican candy packaging (https://www.reddit.com/r/BSD/comments/4vngmw/the_bsd_daemon_feature_in_mexican_candy_packaging/) Remove PG_ZERO and zeroidle (page-zeroing) entirely (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624202.html) OpenBSD: Release Songs: 6.0: "Black Hat" (https://www.openbsd.org/lyrics.html#60b) OpenBSD Gaming Resource (http://satterly.neocities.org/openbsd_games.html) LibreSSL 2.4.2 and 2.3.7 Released (http://bsdsec.net/articles/libressl-2-4-2-and-2-3-7-released) Feedback/Questions Pedja - Bhyve GUI (http://pastebin.com/LJcJmNsR) Tim - Jail Management (http://pastebin.com/259x94Rh) Don - X260 (http://pastebin.com/A86yHnzz) David - Updates (http://pastebin.com/wjtcuVSA) Ghislain - Jail Management (http://pastebin.com/DgH9G7p5) ***
Today on the show, we are going to be talking to Jim Brown (of BSD Cert Fame) about his home-brew sprinkler system… Wait for it… This episode was brought to you by Headlines Distrowatch reviews OpenBSD and PCBSD's live upgrade method (http://distrowatch.com/weekly.php?issue=20160620#upgrade) Upgrading… The bane of any sysadmin! Distrowatch has recently done a write-up on the in-place upgrading of various distros / BSDs including PC-BSD and OpenBSD. Lets look first at the PC-BSD attempt, which was done going from 9.2 -> 10. “I soon found trying to upgrade either the base system or pkg would fail. The update manager did not provide details as to what had gone wrong and so I decided to attempt a manual upgrade by following the FreeBSD Handbook as I had when performing a live upgrade of FreeBSD back in May. At first the manual process seemed to work, downloading the necessary patches for FreeBSD 10 and getting me to resolve conflicts between my existing configuration files and the new versions. Part way through, we are asked to reboot and then continue the upgrade process using the freebsd-update command utility. PC-BSD failed to reboot and, in fact, the boot loader no longer found any operating systems to run.” Ouch! I'm not sure on the particular commands used, but to lose the boot-loader indicates something went horribly wrong. There is good news in this though. After the pain experienced in the 9.X upgrade process, 11.0 has been vastly improved to help fix this going forward. The updater is also self-updating, which means future changes to tools such as package can be accounted for in previously released versions. Moving on to OpenBSD, Jesse had much better luck: > “The documentation provided explains how to upgrade OpenBSD 5.8 to version 5.9 step-by-step and the instructions worked exactly as laid out. Upgrading requires two reboots, one to initiate the upgrade process and one to boot into the new version of OpenBSD. Upgrading the base operating system took approximately ten minutes, including the two reboots. Upgrading the third-party packages took another minute or two. The only quirk I ran into was that I had to manually update my repository mirror information to gain access to the new packages available for OpenBSD 5.9. If this step is not done, then the pkg_add package manager will continue to pull in packages from the old repository we set up for OpenBSD 5.8. “ A good read, and they covered some Linux distros such as Mint and OpenMandriva as well, if you want to find out how they fared. *** A curated list of awesome DTrace books, articles, videos, tools and resources (http://awesome-dtrace.com/) The website awesome-dtrace.com compiles a list of resources, including books, articles, videos, tools, and other resources, to help you get the most out of DTrace The list of books includes 2 open source books that are available on the web, and of course Brendan Gregg's official DTrace book There are also cheat sheets, one-liner collections, and a set of DTrace war stories A breakdown of different PID providers and the userspace statically defined tracepoints The videos from DTrace.conf 2008, 2012, and soon 2016 And links to the tools to start using DTrace with your favourite programming language, including Erlang, Node.JS, Perl, PHP, Python, or Ruby There are also DTrace setups for MySQL/MariaDB, and PostreSQL Joyent has even written a mod_usdt DTrace module for the Apache web server This seems like a really good resource, and with the efforts of the new OpenDTrace project, to modernize the dtracetoolkit and make it more useful across the different supported operating systems, there has never been a better time to start learning DTrace *** Installing OpenBSD using a serial console with no external monitor (http://unix.stackexchange.com/questions/292891/how-can-i-install-openbsd-using-the-serial-console-without-external-monitor-wi) Have you found yourself needing to install OpenBSD from USB, but with a twist, as in no external monitor? Well somebody has and asked the question on stackexchange. The answer provided is quite well explained, but in a nut-shell the process involves downloading the USB image and making some tweaks before copying it to the physical media. Specifically with a couple of well-placed echo's into boot.conf, the serial-port can be enabled and ready for use: echo "stty com0 115200" > /mnt/etc/boot.conf echo "set tty com0" >> /mnt/etc/boot.conf + After that, simply boot the box and you are ready to access the serial console and drive the installation as normal! #bsdhacks GSoC 2016 Reports: Split debug symbols for pkgsrc builds (https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug) The NetBSD blog provides a status report on one of the GSoC projects that is nearing its midterm evaluation The project to split debugging data into separate pkgsrc packages, so that users can install the debugging symbols if they need them to debug a failing application The report is very detailed, and includes “A quick introduction to ELF and how debug information are stored/stripped off” It walks through the process of writing a simple example application, compiling it, and dealing with the debug data It includes a number of very useful diagrams, and a summary of what changes needed to be make to the pkgsrc makefile infrastructure With this as a recipe, someone should be able to do something quite similar for FreeBSD's ports tree *** iXsystems iXsystems' TrueNAS Firmware Update Delivers Compelling Performance, Replication, and Graphing Improvements (https://www.ixsystems.com/blog/ixsystems-truenas-firmware-update-delivers-compelling-performance-replication-graphing-improvements/) *** Interview - Jim Brown - jpb@jimby.name (mailto:jpb@jimby.name) FreeBSD+BBB Sprinkler System News Roundup From the past : A Research Unix Reader (http://www.cs.dartmouth.edu/~doug/reader.pdf) A paper by by Douglas McIlroy “Selected pages from the nine research editions of the UNIX® Programmer's Manual illustrate the development of the system” “Accompanying commentary recounts some of the needs, events, and individual contributions that shaped this evolution.” Interesting insight into the evolution of the origin UNIX operating system *** Evolution of C programming practices – Unix 1973–2015 (http://kristerw.blogspot.com/2016/06/evolution-of-c-programming-practices.html) From the author of the recent post we covered, “20 years of NetBSD code bloat”, comes a new post “I found a recent paper that also looks at how the BSD code base has evolved, but from a very different perspective compared to my code-size investigation.” The paper "The Evolution of C Programming Practices: A Study of the Unix Operating System 1973–2015" investigates coding style, and tests seven hypotheses by looking at metrics (line length, number of volatile in the source code, etc.) in 66 releases of Unix from 1973 to 2014. The hypotheses are: > + Programming practices reflect technology affordances (e.g. developers may be more liberal with screen space when using high resolution displays) > + Modularity increases with code size > + New language features are increasingly used to saturation point > + Programmers trust the compiler for register allocation > + Code formatting practices converge to a common standard > + Software complexity evolution follows self correction feedback mechanisms > + Code readability increases and the result is that they seem to be true, as interpreted through the metrics. > “The data points for the releases have somewhat random dates. One issue is that the paper use each release's mean file date (the average of the files' last modification time) instead of the release date (that is why the graphs stop at November 2010, even though FreeBSD 10 was released in 2014). The idea is that this better reflects the age of the code base, but this has the effect of compressing some of the data points (especially the clustering around 1993-1994), and it makes the spline fitting even more suspect.” > “One other problem is that the original data used by the researchers seems to have incorrect timestamps. For example, 4.3BSD Net/1 was released in 1989, but is listed as 1993-12-25 in the paper. The same is true for at least the Net/2 release too, which was released in 1991, but the paper list it as 1993-07-02.” *** [old release pictures] openbsd 2.1 - 5.9, straight from theo's bookshelf. (https://twitter.com/blakkheim/status/747540167112671232) Speaking of old releases, our Producer JT picked up this gem at Southeast Linuxfest this year (https://twitter.com/q5sys/status/748003859012984837) Noah Axon shares a scan of his NetBSD 1.4 disc (https://plus.google.com/+NoahAxon/posts/VsiQhUn3tHb) Jan van den broek shares a pic of his FreeBSD 2.2.5 set (https://plus.google.com/101232368324501316985/posts/4QsaJE2KxXh) *** FreeBSD: Just in Time (https://bsdmag.org/just_in_time/) Another BSDMag goodie this week, we have a small article written by Jonathan Garrido which details their experience switching to FreeBSD for a NTP server. The article is short, but a good read: > “A Few years ago we had a time problem. Suddenly our linux NTP server, for a reason that I still do not know, started to fail giving us a lot of issues within all the equipment and services within our network. After a quick and brief meeting with management, I found out that there was not sufficient budget left for a fancy and well-suited appliance. So, with no time (literally) and no money to spend, I decided to give it a try and utilized a homemade open source solution, and the operating system of choice was FreeBSD 10.0.” “Now, let's pause for a second. You may be thinking, why in the world is this guy doing this, when he has never installed a BSD machine in his life? The answer is very simple; here, in the Dominican Republic, in the heart of the Caribbean, FreeBSD has a very good reputation when it comes to reliability and security. In fact, there is some collective thought within the sysadmin community that says something like: “If you want to deal only once with a service, install it over FreeBSD.”” Jonathan then goes through some of the steps taken to initial deploy NTP services, but with that out of the way, he has a great summary: > “Fascinated with the whole experience, we migrate one of our internal dns servers to a second FreeBSD machine and at the moment of this writing we are testing haproxy, an open source load-balancing proxy into a another server with the same OS. > After all this, no time issues have been reported in the past 2 years, so at least for my environment, FreeBSD came just in time.“ *** Beastie Bits MiniBSD laptop computer (https://hackaday.io/project/643-minibsd-laptop-computer) The state of LibreSSL in FreeBSD (https://attilagyorffy.com/2016/07/02/the-state-of-libressl-in-freebsd/) Justin Sherrill is looking for someone willing to run a Go builder with DragonflyBSD (https://www.dragonflydigest.com/2016/07/01/18372.html) Tiny Unix tools for Windows (https://tinyapps.org/blog/windows/201606040700_tiny_unix_tools_windows.html) OpenBSD's doas added to the FreeBSD Ports Tree (http://www.freshports.org/security/doas/) ubuntuBSD 16.04 to feature a combo of BusyBox and OpenRC, no systemd (http://linux.softpedia.com/blog/ubuntubsd-16-04-will-feature-a-combination-of-busybox-and-openrc-but-no-systemd-505463.shtml) Syncast Podcast 4 : Curl, libcurl and the future of the web, with Daniel Stenberg (http://podcast.sysca.st/podcast/4-curl-libcurl-future-web-daniel-stenberg/) Feedback/Questions Harri - Using beadm / zfssnap (http://pastebin.com/qKeCd63F) Jonathan - bhyve vs Proxmox (http://pastebin.com/EhXDwbWQ) Mohammad - Bhyve gfx passthrough (http://pastebin.com/ZCNk4Bga) Jeremy - Shapshots and more Snapshots (http://pastebin.com/xp7nzEYa) Ron - Microphone (http://pastebin.com/H2xr53CR) ***
This week on the show, Allan and I are going to be showing you a very interesting interview we did talking about using FreeBSD to drive This episode was brought to you by Headlines FreeBSD Core Team Election (https://www.freebsd.org/administration.html#t-core) Core.9 has been elected, and will officially take over from Core.8 on Wednesday, 6 July 2016 Many thanks to the outgoing members of the core team for their service over the last 2 years 214 out of 325 eligible voters (65.8%) cast their votes in an election counting 14 candidates. The top nine candidates are, in descending order of votes received: 180 84.1% Ed Maste (incumbent) 176 82.2% George V. Neville-Neil (incumbent) 171 79.9% Baptiste Daroussin (incumbent) 168 78.5% John Baldwin 166 77.6% Hiroki Sato (incumbent) 147 68.7% Allan Jude 132 61.7% Kris Moore 121 56.5% Benedict Reuschling 108 50.5% Benno Rice There was no tie for ninth. BSDNow and the entire community would also like to extend their thanks to all those who stood for election to the core team Next week's core meeting will encompass the members of Core.8 and Core.9, as responsibility for any outstanding items will be passed from outgoing members of core to the new incoming members *** Why I run OpenBSD (http://deftly.net/posts/2016-05-31-why-i-run-openbsd.html) This week we have a good article / blog post talking about why the posted has moved to OpenBSD from Linux. “One thing I learned during my travels between OSs: consistency is everything. Most operating systems seem to, at least, keep a consistent interface between themselves and binaries / applications. They do this by keeping consistent APIs (Application Programming Interfaces) and ABIs (Application Binary Interfaces). If you take a binary from a really old version of Linux and run or build it on a brand-spanking new install of Linux, it will likely Just Work™. This is great for applications and developers of applications. Vendors can build binaries for distribution and worry less about their product working when it gets out in the wild (sure this binary built in 2016 will run on RedHat AS2.1!!).“ The author then goes through another important part of the consistency argument, with what he calls “UPI” or “User Program Interfaces”. In other words, while the ABI may be stable, what about the end-user tooling that the user directly has to interact with on a daily basis? “This inconsistency seems to have come to be when Linux started getting wireless support. For some reason someone (vendors, maybe?) decided that ifconfig wasn't a good place to let users interact with their wireless device. Maybe they felt their device was special? Maybe there were technical reasons? The bottom line is, someone decided to create a new utility to manage a wireless device… and then another one came along… pretty soon there was iwconfig(8), iw(8), ifconfig(8), some funky thing that let windows drivers interface with Linux.. and one called ip(8) I am sure there are others I am forgetting, but I prefer to forget. I have moved onto greener pastures and the knowledge of these programs no longer serves me.” The article then goes through the rundown of how he evaluated the various BSD's and ultimately settled on OpenBSD: “OpenBSD won the showdown. It was the most complete, simple, and coherent system. The documentation was thorough, the code was easy to follow and understand. It had one command to configure all of the network interfaces! I didn't have wireless, but I was able to find a cheap USB adapter that worked by simply running man -k wireless and reading about the USB entries. It didn't have some of the applications I use regularly, so I started reading about ports (intuitively, via man ports!).” The ultimate NetBSD Router (http://blog.tbrodel.me/2016/#netbsd-router) “So yesterday I spent the day setting up a new firewall at home here, based off of this BSD Now tutorial. Having set up a couple of OpenBSD routers before, either based on old laptops, bulky old power-sucking desktops or completely over-specced machines like the Intel NUC, I wanted to get some kind of BSD onto a low-powered ARM board and use that instead.” “I've had a couple of Cubietrucks lying around for a while now, I've used them in a couple of art installations, running Debian and Pure Data, but over all they've been a bit disappointing. It's more the manufacturer's fault but they require blobs for the graphics and audio, which Debian won't allow, so as a multimedia board they're dud for video, and only passable for audio work with a usb sound card. So they've been collecting dust.” “Only thing missing is a second NIC, luckily I had an Apple USB->Ethernet dongle lying around, which when I bought it was the cheapest thing I could find on eBay that OpenBSD definitely supported. There, and on NetBSD, it's supported by the axe(4) driver. USB 2.0 works fine for me as I live in Australia and my ISP can only give me 30Mbps, so this should do for the forseeable future.” + The article then walks through installing and configuring NetBSD + Configuration includes: pf, unbound, and dhcpd “This project has been really fun, I started with basically no experience with NetBSD and have finished with a really useful, low-powered and robust appliance. It's a testament to the simplicity of the NetBSD system, and the BSD design principles in general, that such a novice as myself could figure this out. The NetBSD project has easily the most polished experience on Allwinner ARM boards, even Debian doesn't make it this easy. It's been a joy running the system, it has the bits I love from OpenBSD; ksh(1), tmux(1), an http daemon in base and of course, pf(4). This is mixed with some of the pragmatism I see in FreeBSD; a willingness to accept blobs if that really is the only way to boot, or get audio, or a video console.” bhyve-Bootable Boot Environments (http://callfortesting.org/bhyve-boot-environments/) We have a lengthy article also today from our friend Michael Dexter, who asks the basic question “What if multibooting and OS upgrades weren't horrible?” No doubt if you've been a frequent listener to this show, you've heard Allan or Myself talking about ZFS Boot Environments, and how they can “change your life”. Well today Michael goes further into detail on how the BE's work, and how they can be leveraged to do neat things, like installing other versions of an operating system from the original running system. “If you are reading this, you have probably used a personal computer with a BSD or GNU/Linux operating system and at some point attempted to multiboot between multiple operating systems on the same computer. This goal is typically attempted with complex disk partitioning and a BSD or GNU/Linux boot loader like LILO or GRUB, plus several hours of frustrating experimentation and perhaps data loss. While exotic OS experimentation has driven my virtualization work since the late 1990s, there are very pragmatic reasons for multibooting the same OS on the same hardware, notable for updates and failback to "known good" versions. To its credit, FreeBSD has long had various strategies including the NanoBSD embedded system framework with primary and secondary root partitions, plus the nextboot(8) utility for selecting the "next" kernel with various boot parameters. Get everything set correctly and you can multiboot "with impunity". “That's a good start, and over time we have seen ZFS "boot environments" be used by PC-BSD and FreeNAS to allow for system updates that allow one to fall back to previous versions should something go wrong. Hats off to these efforts but they exist in essentially purpose-built appliance environments. I have long sensed that there is more fun to be had here and a wonderful thing happened with FreeBSD 10.3 and 11.0: Allan Jude added a boot environment menu to the FreeBSD loader” From here Michael takes us through the mechanical bits of actually creating a new ZFS dataset (BE) and performing a fresh FreeBSD 10.3 installation into this new boot-environment. The twist comes at the end, where he next sets up the BE to be a root NFS for booting in bhyve! This is interesting and gives you a way to test booting into your new environment via a VM, before rebooting the host directly into it. *** Interview - Edicarla Andrade & Vinícius Zavam - @egypcio (https://twitter.com/egypcio) BSD-Powered Robots News Roundup Tomohiro Kasumi explains what “@@” means, in the context of the Hammer filesystem (http://lists.dragonflybsd.org/pipermail/users/2016-June/249717.html) A post from the Dragonfly users' mailing list about what the @@ construct means in the Hammer filesystem “@@ represents the existence of a PFS which is logically separated pseudo filesystem space within HAMMER's B-Tree” “HAMMER only has 1 large B-Tree per filesystem (not per PFS), so all the PFS exist within that single B-Tree. PFS are separated by localization parameter which is one of the B-Tree keys used to lookup the tree.” Each substring in "@@-1:00001" means: "@@" means it's a PFS or snapshot. "-1" means it's a master. ":" is just a separator. "00001" means it's PFS#1, where PFS#0 is the default PFS created on newfs. There is no "00000" because that's what's mounted on /HAMMER. PFS# is used for localization parameter. “Localization parameter has the highest priority when inserting or looking up B-Tree elements, so fs elements that belong to the same PFS# tend to be localized (clustered) within the B-Tree” There is also a note about how snapshots are named: "@@0x00..." A user points out that having : in the path can confuse some applications, such as in the case of adding the current directory or a relative path to the $PATH environment variable, which is a colon delimited list of paths This seems quite a bit more confusing that the datasets created by ZFS, but they might have other useful properties *** FreeBSD 11.0 nearing RC1 (https://www.freebsd.org/releases/11.0R/schedule.html) We've all been eagerly awaiting the pending release of FreeBSD 11.0, and the schedule has now been updated! The first release candidate is slated for July 29th! If all goes well (and we stick to schedule) there will be another RC2 and possible RC3 release, before 11.0 officially drops near the end of August. Start playing with those builds folks, be sure to send your feedback to the team to make this the best .0 release ever! *** TensorFlow on FreeBSD (http://ecc-comp.blogspot.com/2016/06/tensorflow-on-freebsd.html) Next we have a blog post about the experience of a “new” FreeBSD user trying to deploy some non-ported software to his new system. Specifically he was interested in running TensorFlow, but not doing a port himself, because in his words: “First, I apologize for not supplying a port archive myself. After reading the FreeBSD handbook for creating a port, it's too complex of a task for me right now. I've only been using FreeBSD for two weeks. I would also not like to waste anyone's time giving them a terrible port archive and mess up their system.” First of all, good ports are often born out of bad ports! Don't let the porting framework daunt you, give it a go, since that's the only way you are going to learn how to write “good” ports over time. The porters-handbook is a good first place to start, plus the community usually is very helpful in providing feedback. He then walks us through the changes made to the TensorFlow code (starting with the assumption that OSX was a good “flavor” to begin porting from) and ultimately compiling. This ends up with the creation of a pip package which works! A good tutorial, and also very similar to what goes on in the porting process. With this write-up perhaps somebody will take up creating a port of it… hint hint! *** NetBSD: A New Beginning? (http://jamesdeagle.blogspot.ca/2016/06/netbsd-new-beginning.html) We don't get enough NetBSD news at times, but this post by James Deagle talks about his adventure with NetBSD 7.0 and making it his “new beginning” “After a few months of traipsing around the worlds of SunOS and Linux, I'm back to NetBSD for what I hope will be a lengthy return engagement. And while I'm enamored of NetBSD for all the previously-mentioned reasons, I'm already thinking ahead to some problems to solve, some of which have also been mentioned before.” He then goes through and lists some of the small nits he's still running into during the daily workflow YouTube audio - Specifically he mentions that no audio is playing, but wonders if Flash plays some part. (Ideally you're not using Flash though, in which case you need to check the audio backend FF is using. Try PulseAudio since it seems the best supported. If pulse is already enabled, install ‘pavucontrol' to make sure audio is playing to the correct sound device) Slow gaming performance (TuxKart and Celestia) - Check DRI / Xorg? Or is it CPU bound? Lastly some unspecified Wireless issues, which typically end up being driver related. (Or use another chipset) Beastie Bits Reproducible NetBSD? 77.7% of the way there (https://reproducible.debian.net/netbsd/netbsd.html) Create FreeBSD virtual machine using qemu. Run the VM using xhyve. (https://gist.github.com/zg/38a3afa112ddf7de4912aafc249ec82f) FreeBSD PowerPC 32bit pkg repository (unofficial). ~19,500 packages, more to come (https://joshcummings.net/pub/FreeBSD) NetBSD machines at Open Source Conference 2016 Gunma (http://mail-index.netbsd.org/netbsd-advocacy/2016/05/16/msg000706.html) Adam Leventhal (of ZFS and DTrace) does an analysis of APFS (http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/) SemiBug June meeting summary (http://lists.nycbug.org/pipermail/semibug/2016-June/000106.html) KnoxBug Meeting (http://knoxbug.org/content/2016-07-26) Feedback/Questions Andrew - iocage (http://pastebin.com/nuYTzaG6) Florian - Arm + GitHub (http://pastebin.com/PzY68hNS) Clint - Synth (http://pastebin.com/JESGZjLu) Leonardo - Translations (http://pastebin.com/b4LAiPs4) Zachary - Moving things to VMs (http://pastebin.com/VRc8fvBk) ***
This week on BSDNow, Allan is back from the Storage Summit in Silicon Valley! We are going to get his thoughts on how the conference went, plus bring you the latest ZFS info discussed. That plus the usual BSD news is This episode was brought to you by Headlines OpenBSD website operators urged to fix mind-alteringly bad bug (http://www.theregister.co.uk/2016/02/21/openbsd_website_operator_patch_now_for_the_sake_of_your_sanity/?mt=1456206806399) We start off a bit light-hearted this week, with the important, breaking news that finally a long-standing OpenBSD bug has been addressed for the HTTP daemon. Specifically? It changes the default 404 page fonts away from Comic Sans, to a bit more crowd-pleasing alternative: “For some reason the httpd status pages (e.g. 404) use the Comic Sans typeface. This patch removes comic sans and sets the typeface to the default sans-serif typeface of the client. “This lowers the number of people contacting website maintainers with typeface complaints bordering on harassment”. Operators running HTTPD are highly encouraged to update their systems to the latest code, right now……... No seriously, we are waiting for you. Get it done now and then we'll continue with the show. Registration for AsiaBSDCon 2016 is now open + Talk Schedule (https://2016.asiabsdcon.org/registration/?lang=en) After a few delays, the registration for AsiaBSDCon has now opened! The conference starts in less than two weeks! now, so be sure to get signed up ASAP. In addition the schedule has been posted, and here's some of the highlights of this year's conference. In addition to FreeBSD and NetBSD dev summits on the first two days, we have some excellent tutorials being given this year by Kirk, Gnn, Dru and more! (https://2016.asiabsdcon.org/program.html.en) The regular paper talks also have lots of good ones this year, including this crazy encrypted boot loader one given by our very own Allan Jude! *** OPENBSD ON AWS : AN UNEXPECTED JOURNEY (http://blog.d2-si.fr/2016/02/15/openbsd-on-aws/?hn) We have a blog post from Antoine Jacoutot, talking about the process of getting OpenBSD up and running in AWS It starts with his process of creating an AMI from scratch, which ended up not being that bad: create and loopback-mount a raw image containing a UFS filesystem extract the OpenBSD base sets (which are just regular tarballs) and kernel enable console output (so that one could “aws ec2 get-console-output”) install the boot loader on the image then use the ec2 tools to import the RAW image to S3, convert it into a volume (ec2-import-volume) which we can snapshot (ec2-create-snapshot) and create an AMI from (ec2-register) The blog post also has a link to a script which automates this process, so don't be daunted if you didn't quite follow all of that. Thanks to the recently landed DomU support, the final pieces of the puzzle fell into place, allowing OpenBSD to function as a proper guest (with networking!) Next it details the process of injecting a public SSH key into the instances for instant remote access. An ec2-init.sh script was created (also on github) which does the following: setting the hostname installing the provided SSH public key to /root/.ssh/authorized_keys executing user-data (if it starts with a shebang) displaying the host SSH fingerprints on the console (to match cloud-init) With that done, OpenBSD is pretty much AWS ready! He then gives a brief walkthrough of setting up nginx for new users, but if you've already done this before then the instance is ready for you to hacking on. Start thinking of ideas for things with FreeBSD for Google's 2016 Summer of Code (https://wiki.freebsd.org/SummerOfCodeIdeas) Students and Developers, listen up! It's time to start thinking about GSoC again, and FreeBSD is looking to update its project ideas page. There's some good ones on the list, plus ones that should be pruned (such as GELI boot), but now is the time to start adding new ones before we get too deep into the process. This goes for the other BSD's as well, start thinking about your proposals, or if you are developer, which projects would be a good fit for mentoring. (Improving the Linux Compat layer is one I think should be done!) Guide to getting started with kernel hacking (https://wiki.freebsd.org/Graphics/Getting%20started%20with%20kernel%20projects) One of the things that's been asked frequently is how to contribute towards the efforts to bring updated DRM / X drivers to the FreeBSD kernel. Jean-Sébastien Pédron has started a great guide on the Wiki which details how to get started with the porting effort, and that developers need not be afraid of helping. *** Storage Summit Roundup Earlier this week a number of developers from FreeBSD, as well as various vendors that use FreeBSD, or provide products used with FreeBSD met for a Storage Summit (https://wiki.freebsd.org/201602StorageSummit), to discuss the future of these technologies The summit was co-located with the USENIX FAST (Filesystems And Storage Technologies) conference The summit was sponsored by the FreeBSD Foundation and FlightAware After a short introduction, the event opened with a Networking Synergy panel The focus of this panel was to see if there were techniques and lessons learned in improving the networking stack over the last 10 years that could be applied to improving the storage stack A lot of time was spent discussing issues like multi-queue support, CPU scheduling, and ways to modernize the stack CAM Scheduling & Locking Revamp (https://wiki.freebsd.org/201602StorageSummit/CAM) No notes posted User Space Storage Stack (https://wiki.freebsd.org/201602StorageSummit/UserSpace) One of the user space storage stacks discussed was Diskmap Like netmap, but for disks (diskmap) Kernel bypass for accessing disks Ilias Marinos, who is working on diskmap at Cambridge University, described diskmap to the group A design discussion then followed in which the memory management was covered as that's an issue for any sort of "IO" map system Action Items: Discuss with Luigi the idea of code merges Need a reset path API Kernel buffer mapping for reliability Support for other interfaces (SATA/SCSI) GEOM layer adaptation Adapting to New Storage Technologies (https://wiki.freebsd.org/201602StorageSummit/NewStorageTechnologies) This working group was led by Adrian Palmer, from Seagate SMR Persistent Memory Session 1: Device Identification and the structural requirements Agenda: We'll look over the Identification nuances and what needs to change to support the structure. Support for IO order guarantees, forward-write only requirements, new commands and topology. Dig into CAM and GEOM layers. Solutions should be fast and have as few code paths as possible Results: Small audience. We talked about zoned characteristics, and how it can be used in various workloads, projected to be implemented in years Session 2: Information dissemination and consumption Agenda: Where and how will information from the report_zones command be gathered, stored, combined and used. This will include userspace storage and multi-volume management. Will CAM store this data, or will GEOM? How frequently will this need to be queried/updated/verified from the drive? Results: Merged with ZFS working group to discuss SMR. Came up with idea that could be implemented as circular buffer zone type. Began to discuss solutions among developers ZFS (https://wiki.freebsd.org/201602StorageSummit/ZFS) During the first session we discussed how to improve dedup support + A dedup throttle or cap was discussed. When the size of the DDT grows beyond this size, new entries would not be deduped. An alternative to this was also discussed, where when the DDT reached the cap size, it would remove a random entry with only a single reference from the DDT to make room for the new entry. When a block is going to be freed, if it is not found in the DDT, it is assumed to have only 1 reference, and removed. There was also discussion of replacing the DDT with an in-memory hash table and a “log” of increment/decrement operations, that is periodically compacted. The hash table is recreated from the log at pool import time. This would reduce the in-memory footprint of the DDT, as well as speed up all write operations as adding an entry to the dedup log will be less expensive than updating the DDT. There was also discussion of using dedicated device(s) for the DDT, either using the DDT on SSD work by Nexenta, or the Metadata Classes work by Intel The first session also discussed Secure Delete and related things The desire for an implementation of TRIM that uses the “secure erase” functionality provided by some disks was expressed Overwriting sectors with patterns of garbage may be insufficient because SSDs may internally remap where a specific LBA physically resides The possibility of using something like the “eager zero” feature to periodically write zeros over all free blocks in the pool to erase any lingering data fragments Problems with the FreeBSD TRIM implementation were discussed, as well as looking at ways to implement the new ZFS TRIM implementation on FreeBSD ABD (ARC Buf Data) was discussed, a new design that lessens the requirement for contiguous memory. Only a small area of contiguous blocks is reserved at boot, and compressed ARC blocks are constructed of scatter-gather lists of individual pages The second session combined with the SMR group and talked about SMR support in ZFS Later in the second session ZFS Encryption was also discussed, mostly with a focus on what the use cases are The third session combined all of the groups for an overview of upcoming ZFS features including device removal and channel programs There was also a request for code review, for mostly finished projects like Persistent L2ARC, Writeback cache, and Large dnode support Hallway Track ZFS / VFS Interaction Adrian Palmer has been a FreeBSD hobbyist since FreeBSD 7, and I think I managed to convince him to start contributing *** News Roundup One Week with NetBSD 7.0: Back to Unix basics (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-back-to-unix.html) The author of this blog series is sending a week using NetBSD 7.0, following a previous series on Solaris 10 “This is actually familiar territory, as I've been using BSD variants almost exclusively since 2006. My recent SunOS explorations were triggered last summer by OpenBSD having choked on my current laptop's NVIDIA card, and from what I could see at the time, FreeBSD had the same problem, although I now know NVIDIA drivers exist for that system. The thing that keeps me from going all-in with FreeBSD 10.x, however, is the fact that Firefox crashes and leaves "core dump" messages in its wake, and I'm just not a Chrome kinda guy.” “For those with a catholic taste in Unix, NetBSD is a keg party at the Vatican. If you're an absolute Unix beginner, or have been living on Ubuntu-based Linux distros for too long, then you may feel stranded at first by NetBSD's sparseness. You'll find yourself staring into the abyss and seeing only a blinking cursor staring back. If you have the presence of mind to type startx, you'll be greeted by twm, a window manager offering little more than an xterm window with the same blinking cursor until you learn how to configure the .twmrc file to include whatever applications you want or need in the right-click menu.” “As for NetBSD itself, I can't think of any major productivity applications that can't be installed, and most multimedia stuff works fine.” Issues the author hopes to sort out in later posts: Audio playback (youtube videos in Firefox) Wireless Flash Digital Camera SD Card readability, video playback Audacity A “fancy” desktop like Gnome 2, KDE, or xfce In a follow-up post (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-libreoffice.html), the author got LibreOffice installed and sorted out the audio issues they were having In a later follow-up (http://jamesdeagle.blogspot.com/2016/02/one-week-with-netbsd-70-mixed-review-of.html) XFCE is up and running as well *** ZFS is for Containers in Ubuntu 16.04 (http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html) As you may have heard, Ubuntu 16.04 will include ZFS -- baked directly into Ubuntu -- supported by Canonical “ZFS one of the most beloved features of Solaris, universally coveted by every Linux sysadmin with a Solaris background. To our delight, we're happy to make to OpenZFS available on every Ubuntu system.” What does “supported by Canonical” mean? “You'll find zfs.ko automatically built and installed on your Ubuntu systems. No more DKMS-built modules” “The user space zfsutils-linux package will be included in Ubuntu Main, with security updates provided by Canonical” The article then provides a quick tutorial for setting up Linux Containers (LXC) backed by ZFS In the example, ZFS is backed by a file on the existing disk, not by a real disk, and with no redundancy However, the setup script seems to support using real block devices The Software Freedom Conservancy (https://sfconservancy.org/) is expected to issue a statement detailing their opinion on the legalities and licensing issues of bundling ZFS with Linux. *** Polling is a Hack: Server Sent Events (EventSource) with gevent, Flask, nginx, and FreeBSD (http://hypatia.software/2016/01/29/polling-is-a-hack-server-sent-events-eventsource-with-gevent-flask-nginx-and-freebsd/) A tutorial on setting up ‘Server-Sent Events', also know as EventSource in javascript, to notify website clients of new data, rather than having the javascript constantly poll for new data. The setup uses FreeBSD, nginx, gevent, Python, and the Flask framework The tutorial walks through setting a basic Python application using the Flask framework Then setting up the client side in Javascript Then for the server side setup, it covers installing and configuring nginx, and py-supervisor on FreeBSD The tutorial also includes links to additional resources and examples, including how to rate limit the Flash application *** Why FreeBSD? (http://www.aikchar.me/blog/why-freebsd.html) An excellent article written by Hamza Sheikh, discussing why FreeBSD is now his clear choice for learning UNIX. The article is pretty well written and lengthy, but has some great parts which we wanted to share with you: There were many rough edges in the Linux world and some of them exist even today. Choosing the right distribution (distro) for the task at hand is always the first and most difficult decision to make. While this is a strength of the Linux community it is also its weakness. This is exacerbated with the toxic infighting within the community in the last few years. A herd of voices believes it is their right to bring down a distro community because it is not like their distro of choice. Forking upstream projects has somehow become taboo. Hurling abuse in mailing lists is acceptable. Helping new users is limited to lambasting their distro of choice. Creating conspiracy theories over software decisions is the way to go. Copyleft zealots roam social media declaring non-copyleft free software heretic abominations. It all boils down to an ecosystem soured by the presence of maniacs who have the loudest voices and they seem to be everywhere you turn. Where is the engineering among all this noise? Btrfs - baking for a long time - is still nowhere near ZFS in stability or feature parity. systemd is an insatiable entity that feeds on every idea in sight and just devours indiscriminately. Wayland was promised years ago and its time has yet to arrive. Containers are represented by Docker that neither securely contains applications nor makes them easy to manage in production. Firewalling is dithering between firewalld, nftables, etc. SystemTap cannot match DTrace. In the same time span what do various BSDs offer? pf, CARP, ZFS, Hammer, OpenSSH, jails, pkgsrc, (software) ports, DTrace, hardware portability; just to name a few. Few would deny that BSDs have delivered great engineering with free software licenses to the entire world. To me they appear to be better flag bearers of free software with engineering to back it. He then goes through some of the various BSD's and the specifics on why FreeBSD was the logical choice for his situation. But at the end has a great summary on the community as a whole: Finally - and maybe repeating myself here - I have nothing but praise for the community. Be it BSD Now, mailing lists, Reddit, Twitter, LFNW, or SeaGL, people have encouraged me, answered my questions, and filed bugs for me. I have been welcomed and made a part of the community with open arms. These reasons are (good) enough for me to use FreeBSD and contribute to it. BeastieBits OPNsense 16.1.3 released (https://opnsense.org/opnsense-16-1-3-released/) Copies of "FreeBSD Mastery: Specialty Filesystems" seen in the wild (https://twitter.com/Savagedlight/status/700001944547491842) pfsense training available in Europe (http://www.netgate.com/training/) LiteBSD now has 50 ports in its ports tree (https://github.com/ibara/LiteBSD-Ports) Ports tree locked for OpenBSD 5.9 (http://marc.info/?l=openbsd-ports&m=145615281431064&w=2) “FreeBSD Filesystem Fun” at March semibug (http://blather.michaelwlucas.com/archives/2556) Event #46 — Embedded Platforms (BSD, OpenWRT, Plan 9 & Inferno) (http://oshug.org/event/46) Feedback/Questions Frank - ZFS RAM? (http://slexy.org/view/s21lcCKrSB) David - ARM Porting (http://slexy.org/view/s204lxjvlq) Johnny - Lumina Default? (http://slexy.org/view/s2xMiSNLYn) Adam - PC-BSD Install and Q's (http://slexy.org/view/s214gJbLwD) Jeremy - Video Card Q (http://slexy.org/view/s20UNyzEeh) ***
Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers The post Containers with Bryan Cantrill from Joyent appeared first on Software Engineering Daily.
Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers The post Containers with Bryan Cantrill from Joyent appeared first on Software Engineering Daily.
Allan's away at BSDCam this week, but we've still got an exciting episode for you. We sat down with Bryan Cantrill, CTO of Joyent, to talk about a wide variety of topics: dtrace, ZFS, pkgsrc, containers and much more. This is easily our longest interview to date! This episode was brought to you by Interview - Bryan Cantrill - bryan@joyent.com (mailto:bryan@joyent.com) / @bcantrill (https://twitter.com/bcantrill) BSD and Solaris history, illumos, dtrace, Joyent, pkgsrc, various topics (and rants) Feedback/Questions Randy writes in (http://slexy.org/view/s2b6dA7fAr) Jared writes in (http://slexy.org/view/s2vABMHiok) Steve writes in (http://slexy.org/view/s2194ADVUL) ***
Software Engineering Radio - The Podcast for Professional Software Developers
Coming up on the show this week, we've got an interview with Brendan Gregg of Netflix. He's got a lot to say about performance tuning and benchmarks, and even some pretty funny stories about how people have done them incorrectly. As always, this week's news and answers to your emails, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines Even more BSD presentation videos (https://www.meetbsd.com/) More videos from this year's MeetBSD and OpenZFS devsummit were uploaded since last week Robert Ryan, At the Heart of the Digital Economy (https://www.youtube.com/watch?v=Rc9k1xEepWU) FreeNAS & ZFS, The Indestructible Duo - Except for the Hard Drives (https://www.youtube.com/watch?v=d1C6DELK7fc) Richard Yao, libzfs_core and ioctl stabilization (https://www.youtube.com/watch?v=PIC0dwLRBZU) OpenZFS, Company lightning talks (https://www.youtube.com/watch?v=LmbI7F7XTTc) OpenZFS, Hackathon Presentation and Awards (https://www.youtube.com/watch?v=gPbVPwScMGk) Pavel Zakharov, Fast File Cloning (https://www.youtube.com/watch?v=_lGOAZFXra8) Rick Reed, Half a billion unsuspecting FreeBSD users (https://www.youtube.com/watch?v=TneLO5TdW_M) Alex Reece & Matt Ahrens, Device Removal (https://www.youtube.com/watch?v=Xs6MsJ9kKKE) Chris Side, Channel Programs (https://www.youtube.com/watch?v=RMTxyqcomPA) David Maxwell, The Unix command pipeline (https://www.youtube.com/watch?v=CZHEZHK4jRc) Be sure to check out the giant list of videos from last week's episode (http://www.bsdnow.tv/episodes/2014_11_19-rump_kernels_revisited) if you haven't seen them already *** NetBSD on a Cobalt Qube 2 (http://www.jarredcapellman.com/2014/3/9/NetBSD-and-a-Cobalt-Qube-2) The Cobalt Qube was a very expensive networking appliance around 2000 In 2014, you can apparently get one of these MIPS-based machines for about forty bucks This blog post details getting NetBSD installed and set up on the rare relic of our networking past If you're an old-time fan of RISC or MIPS CPUs, this'll be a treat for you Lots of great pictures of the hardware too *** OpenBSD vs. AFL (https://www.marc.info/?l=openbsd-cvs&w=2&r=1&s=afl&q=b) In their never-ending security audit, some OpenBSD developers have been hitting various parts of the tree (https://twitter.com/damienmiller/status/534156368391831552) with a fuzzer If you're not familiar, fuzzing (https://en.wikipedia.org/wiki/Fuzz_testing) is a semi-automated way to test programs for crashes and potential security problems The program being subjected to torture gets all sorts of random and invalid input, in the hopes of uncovering overflows and other bugs American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/), in particular, has provided some interesting results across various open source projects recently So far, it's fixed some NULL pointer dereferences in OpenSSH, various crashes in tcpdump and mandoc (http://www.bsdnow.tv/episodes/2014_11_12-a_mans_man) and a few other things (https://www.marc.info/?l=openbsd-cvs&m=141646270127039&w=2) AFL has an impressive list of CVEs (vulnerabilities) that it's helped developers discover and fix It also made its way into OpenBSD ports, FreeBSD ports and NetBSD's pkgsrc very recently, so you can try it out for yourself *** GNOME 3 hits the FreeBSD ports tree (https://svnweb.freebsd.org/ports?view=revision&revision=372768) While you've been able to run GNOME 3 on PC-BSD and OpenBSD for a while, it hasn't actually hit the FreeBSD ports tree.. until now Now you can play with GNOME 3 and all its goodies (as well as Cinnamon 2.2, which this also brings in) on vanilla FreeBSD Be sure to check the commit message and /usr/ports/UPDATING (http://www.bsdnow.tv/tutorials/ports) if you're upgrading from GNOME 2 You might also want to go back and listen to our interview (http://www.bsdnow.tv/episodes/2014_02_26-port_authority) with Joe Marcus Clark about GNOME's portability *** Interview - Brendan Gregg - bgregg@netflix.com (mailto:bgregg@netflix.com) / @brendangregg (https://twitter.com/brendangregg) Performance tuning, benchmarks, debugging News Roundup DragonFlyBSD 4.0 released (http://www.dragonflybsd.org/release40/) A new major version of DragonFly, 4.0.1, was just recently announced This version includes support for Haswell GPUs, lots of SMP improvements (including some in PF) and support for up to 256 CPUs It's also the first release to drop support for i386, so it joins PCBSD in the 64 bit-only club Check the release notes for all the details, including networking and kernel improvements, as well as some crypto changes *** Can we talk about FreeBSD vs Linux (https://news.ycombinator.com/item?id=8645443) Hackernews had a recent thread about discussing Linux vs BSD, and the trolls stayed away for once Rather than rehashing why one is "better" than the other, it was focused on explaining some of the differences between ecosystems and communities If you're one of the many people who watch our show just out of curiosity about the BSD world, this might be a good thread to read Someone in the comments even gave bsdnow.tv a mention as a good resource to learn, thanks guy *** OpenBSD IPSEC tunnel guide (http://www.packetmischief.ca/openbsd-ipsec-tunnel-guide/) If you've ever wanted to connect two networks with OpenBSD gateways, this is the article for you It shows how to set up an IPSEC tunnel between destinations, how to lock it down and how to access all the machines on the other network just like they were on your LAN The article also explains some of the basics of IPSEC if you're not familiar with all the terminology, so this isn't just for experts Though the article itself is a few years old, it mostly still applies to the latest stuff today All the tools used are in the OpenBSD base system, so that's pretty handy too *** DragonFly starts work on IPFW2 (http://www.dragonflybsd.org/docs/ipfw2/) DragonFlyBSD, much like FreeBSD, comes with more than one firewall you can use Now it looks like you're going to have yet another choice, as someone is working on a fork of IPFW (which is actually already in its second version, so it should be "IPFW3") Not a whole lot is known yet; it's still in heavy development, but there's a brief roadmap (http://www.dragonflybsd.org/docs/ipfw2/#index6h1) page with some planned additions The guy who's working on this has already agreed to come on the show for an interview, but we're going to give him a chance to get some more work done first Expect that sometime next year, once he's made some progress *** Feedback/Questions Michael writes in (http://slexy.org/view/s2NYgVifXN) Samael writes in (http://slexy.org/view/s21X02saI3) Steven writes in (http://slexy.org/view/s21Dj7zImH) Remy writes in (http://slexy.org/view/s218lXg38C) Michael writes in (http://slexy.org/view/s20SEuKlaH) ***
Coming up in this week's episode, we'll be talking with one of OpenBSD's newest developers - Brent Cook - about the portable version of LibreSSL and how it's developed. We've also got some information about the FreeBSD port of LibreSSL you might not know. The latest news and your emails, on BSD Now - the place to B.. SD. This episode was brought to you by Headlines FreeBSD quarterly status report (https://www.freebsd.org/news/status/report-2014-04-2014-06.html) FreeBSD has gotten quite a lot done this quarter Changes in the way release branches are supported - major releases will get at least five years over their lifespan A new automounter is in the works, hoping to replace amd (which has some issues) The CAM target layer and RPC stack have gotten some major optimization and speed boosts Work on ZFSGuru continues, with a large status report specifically for that The report also mentioned some new committers, both source and ports It also covers GNATS being replaced with Bugzilla, the new core team, 9.3-RELEASE, GSoC updates, UEFI booting and lots of other things that we've already mentioned on the show "Foundation-sponsored work resulted in 226 commits to FreeBSD over the April to June period" *** A new OpenBSD HTTPD is born (http://undeadly.org/cgi?action=article&sid=20140724094043) Work has begun on a new HTTP daemon in the OpenBSD base system A lot of people are asking (http://www.reddit.com/r/BSD/comments/2b7azm/openbsd_gets_its_own_http_server/) "why?" since OpenBSD includes a chrooted nginx already - will it be removed? Will they co-exist? Initial responses seem to indicate that nginx is getting bloated, and is a bit overkill for just serving content (this isn't trying to be a full-featured replacement) It's partially based on the relayd codebase and also comes from the author of relayd, Reyk Floeter This has the added benefit of the usual, easy-to-understand syntax and privilege separation There's a very brief man page (http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/httpd.8) online already It supports vhosts and can serve static files, but is still in very active development - there will probably be even more new features by the time this airs Will it be named OpenHTTPD? Or perhaps... LibreHTTPD? (I hope not) *** pkgng 1.3 announced (https://lists.freebsd.org/pipermail/freebsd-ports-announce/2014-July/000084.html) The newest version of FreeBSD's second generation package management system (http://www.bsdnow.tv/tutorials/pkgng) has been released, with lots of new features It has a new "real" solver to automatically handle conflicts, and dynamically discover new ones (this means the annoying -o option is deprecated now, hooray!) Lots of the code has been sandboxed for extra security You'll probably notice some new changes to the UI too, making things more user friendly A few days later 1.3.1 (https://svnweb.freebsd.org/ports?view=revision&sortby=date&revision=362996) was released to fix a few small bugs, then 1.3.2 (https://svnweb.freebsd.org/ports?view=revision&revision=363108) shortly thereafter and 1.3.3 (https://svnweb.freebsd.org/ports?view=revision&revision=363363) yesterday *** FreeBSD after-install security tasks (http://twisteddaemon.com/post/92921205276/freebsd-installed-your-next-five-moves-should-be) A number of people have written in to ask us "how do I secure my BSD box after I install it?" With this blog post, hopefully most of their questions will finally be answered in detail It goes through locking down SSH with keys, patching the base system for security, installing packages and keeping them updated, monitoring and closing any listening services and a few other small things Not only does it just list things to do, but the post also does a good job of explaining why you should do them Maybe we'll see some more posts in this series in the future *** Interview - Brent Cook - bcook@openbsd.org (mailto:bcook@openbsd.org) / @busterbcook (https://twitter.com/busterbcook) LibreSSL's portable version and development News Roundup FreeBSD Mastery - Storage Essentials (https://www.tiltedwindmillpress.com/?product=freebsd-mastery-storage-essentials) MWL (http://www.bsdnow.tv/episodes/2013_11_06-year_of_the_bsd_desktop)'s new book about the FreeBSD storage subsystems now has an early draft available Early buyers can get access to an in-progress draft of the book before the official release, but keep in mind that it may go through a lot of changes Topics of the book will include GEOM, UFS, ZFS, the disk utilities, partition schemes, disk encryption and maximizing I/O performance You'll get access to the completed (e)book when it's done if you buy the early draft The suggested price is $8 *** Why BSD and not Linux? (http://www.reddit.com/r/BSD/comments/2buea5/why_bsd_and_not_linux_or_why_linux_and_not_bsd/) Yet another thread comes up asking why you should choose BSD over Linux or vice-versa Lots of good responses from users of the various BSDs Directly ripping a quote: "Features like Ports, Capsicum, CARP, ZFS and DTrace were stable on BSDs before their Linux versions, and some of those are far more usable on BSD. Features like pf are still BSD-only. FreeBSD has GELI and ipfw and is "GCC free". DragonflyBSD has HAMMER and kernel performance tuning. OpenBSD have upstream pf and their gamut of security features, as well as a general emphasis on simplicity." And "Over the years, the BSDs have clearly shown their worth in the nix ecosystem by pioneering new features and driving adoption of others. The most recent on OpenBSD were 2038 support and LibreSSL. FreeBSD still arguably rules the FOSS storage space with ZFS." Some other users share their switching experiences - worth a read *** More g2k14 hackathon reports (http://undeadly.org/cgi?action=article&sid=20140724161550) Following up from last week's huge list (http://www.bsdnow.tv/episodes/2014_07_23-des_challenge_iv) of hackathon reports, we have a few more Landry Breuil (http://undeadly.org/cgi?action=article&sid=20140724161550) spent some time with Ansible testing his infrastructure, worked on the firefox port and tried to push some of their patches upstream Andrew Fresh (http://undeadly.org/cgi?action=article&sid=20140728122850) enjoyed his first hackathon, pushing OpenBSD's perl patches upstream and got tricked into rewriting the adduser utility in perl Ted Unangst (http://undeadly.org/cgi?action=article&sid=20140729070721) did his usual "teduing" (removing of) old code - say goodbye to asa, fpr, mkstr, xstr, oldrdist, fsplit, uyap and bluetooth Luckily we didn't have to cover 20 new ones this time! *** BSDTalk episode 243 (http://bsdtalk.blogspot.com/2014/07/mandoc-with-ingo-schwarze.html) The newest episode of BSDTalk (http://www.bsdnow.tv/episodes/2014_03_05-bsd_now_vs_bsdtalk) is out, featuring an interview with Ingo Schwarze of the OpenBSD team The main topic of discussion is mandoc, which some users might not be familiar with mandoc is a utility for formatting manpages that OpenBSD and NetBSD use (DragonFlyBSD and FreeBSD include it in their source tree, but it's not built by default) We'll catch up to you soon, Will! *** Feedback/Questions Thomas writes in (http://slexy.org/view/s2xLRQytAZ) Stephen writes in (http://slexy.org/view/s21AYng20n) Sha'ul writes in (http://slexy.org/view/s2DwLRdQDS) Florian writes in (http://slexy.org/view/s2E05L31BC) Bob Beck writes in (http://slexy.org/view/s21Nmg3Jrk) - and note the "Caution" section that was added to libressl.org (http://www.libressl.org/) ***
Новости Padrino v.0.10.7. Семья по достоинству оценила список багфиксов и HTML5-хелперы. Ruby OpenSource Challenge Gem для тестироваия api DTrace for ruby Очереди в Rails 4 Пишем парсер на Ruby для самых маленьких Версия 1.0.0 R18n от Андрея Ситника В ряду rvm и rbenv прибыло ActiveModel для REST Специальный гость Юра Соколов (Funny Falcon) Твиттер Юры Первый патч (бинарный поиск) и Обнаруженный благодаря ему баг Кэширование “раскрытых” LOAD_PATH (основной патч, ускоряющий загрузку) Принятый патч, упаковывающий маленькие хэши Патч ускоряющий создание хэшей (issues те же) Объединение этих патчей применительно к 1.9.3-p194 Ускорение массива как очереди: issue, pull request. Версия к 1.9.3-p194 и тесты Sequel и Jeremy Evans
Was ist DTrace und warum will man es? Wie schreibt man eigene Probes? Und was fängt man mit Instruments an? Diese Session zeigt, welche Probleme man mit DTrace und Instruments analysieren und lösen kann und welche Rolle es bei der Entwicklung und beim Aufspüren von Kundenproblemen spielt. Session 2, Terrassensaal, Macoun 2008
Chris (UK), Dtrace on FreeBSD, Nauseous, Richard Stallman, Vista perfmon, Steve (UK), Google, Mozilla, Diggnation, Mozilla Custom Keywords, Chris (UK) Blog Posting on Free content, Leo Laporte, Apress, Open Source Sex, MacBook vent, No 12" MacBook Pro, Apple out of India, Bluetooth Car Security, U. Washington Crypto Course, Classroom Presenter
© 2020-2025 Ivy Podcast Discovery LLC
