Podcasts about OpenVPN

У нас было несколько IP-transit'ов, полдюжины серверов, пару кб скриптов на питоне и настроенный 15 лет назад FreeRADIUS. Не то чтобы это всё нам было необходимо, но в какой-то момент мы не смогли остановиться и теперь 30 тысяч одновременных соединений OpenVPN — это что-то из раздряда "ну да, а че такого-то?" Надо разобраться! Про что: Рассказ по потивам OpenVPN@Yandex: большое плавание однопоточного сервера Каково это - 17 лет работы в Яндексе и прикоснуться почти ко всему? С Cisco VPN на OpenVPN. Почему динамический фаервол? Использование сертификатов X509 Интеграция с RADIUS для динамической конфигурации прав доступа. И патчи в апстрим. Почему OpenVPN, а не WireGuard, IPSec? Сообщение telecom №145. Тридцать тысяч OpenVPN-ов появились сначала на linkmeup.

WireGuard and other overlay VPNs are the focus of today's podcast with guest Tom Lawrence from Lawrence Systems. We dig into differences between WireGuard and traditional IPSec VPNs, how WireGuard’s opinionated approach to crypto suites helps improve its performance, and how WireGuard compares to OpenVPN. We also look at the broader category of overlay VPNs... Read more »

WireGuard and other overlay VPNs are the focus of today's podcast with guest Tom Lawrence from Lawrence Systems. We dig into differences between WireGuard and traditional IPSec VPNs, how WireGuard’s opinionated approach to crypto suites helps improve its performance, and how WireGuard compares to OpenVPN. We also look at the broader category of overlay VPNs... Read more »

Apple @ Work is exclusively brought to you by Mosyle, the only Apple Unified Platform. Mosyle is the only solution that integrates in a single professional-grade platform all the solutions necessary to seamlessly and automatically deploy, manage & protect Apple devices at work. Over 45,000 organizations trust Mosyle to make millions of Apple devices work-ready with no effort and at an affordable cost. Request your EXTENDED TRIAL today and understand why Mosyle is everything you need to work with Apple. In this episode of Apple @ Work, I talk with Pete Membrey from ExpressVPN about OpenVPN, LightWave in Rust, and much more. Connect with Bradley Twitter LinkedIn Listen and subscribe Apple Podcasts Overcast Spotify Pocket Casts Castro RSS Listen to Past Episodes

HackerOne's co-founder, Michiel Prins walks us through the latest new offensive security service: AI red teaming. At the same time enterprises are globally trying to figure out how to QA and red team generative AI models like LLMs, early adopters are challenged to scale these tests. Crowdsourced bug bounty platforms are a natural place to turn for assistance with scaling this work, though, as we'll discuss on this episode, it is unlike anything bug hunters have ever tackled before. Segment Resources: https://www.hackerone.com/ai/snap-ai-red-teaming https://www.hackerone.com/thought-leadership/ai-safety-red-teaming This interview is a bit different from our norm. We talk to the founder and CEO of OpenVPN about what it is like to operate a business based on open source, particularly through trying times like the recent pandemic. How do you compete when your competitors are free to build products using your software and IP? It seems like an oxymoron, but an open source-based business actually has some significant advantages over the closed source commercial approach. In this week's enterprise security news, the first cybersecurity IPO in 3.5 years! new companies new tools the fate of CISA and the cyber safety review board things we learned about AI in 2024 is the humanless SOC possible? NGFWs have some surprising vulnerabilities what did generative music sound like in 1996? All that and more, on this episode of Enterprise Security Weekly. Visit https://www.securityweekly.com/esw for all the latest episodes! Show Notes: https://securityweekly.com/esw-391

HackerOne's co-founder, Michiel Prins walks us through the latest new offensive security service: AI red teaming. At the same time enterprises are globally trying to figure out how to QA and red team generative AI models like LLMs, early adopters are challenged to scale these tests. Crowdsourced bug bounty platforms are a natural place to turn for assistance with scaling this work, though, as we'll discuss on this episode, it is unlike anything bug hunters have ever tackled before. Segment Resources: https://www.hackerone.com/ai/snap-ai-red-teaming https://www.hackerone.com/thought-leadership/ai-safety-red-teaming This interview is a bit different from our norm. We talk to the founder and CEO of OpenVPN about what it is like to operate a business based on open source, particularly through trying times like the recent pandemic. How do you compete when your competitors are free to build products using your software and IP? It seems like an oxymoron, but an open source-based business actually has some significant advantages over the closed source commercial approach. In this week's enterprise security news, the first cybersecurity IPO in 3.5 years! new companies new tools the fate of CISA and the cyber safety review board things we learned about AI in 2024 is the humanless SOC possible? NGFWs have some surprising vulnerabilities what did generative music sound like in 1996? All that and more, on this episode of Enterprise Security Weekly. Visit https://www.securityweekly.com/esw for all the latest episodes! Show Notes: https://securityweekly.com/esw-391

This interview is a bit different from our norm. We talk to the founder and CEO of OpenVPN about what it is like to operate a business based on open source, particularly through trying times like the recent pandemic. How do you compete when your competitors are free to build products using your software and IP? It seems like an oxymoron, but an open source-based business actually has some significant advantages over the closed source commercial approach. Show Notes: https://securityweekly.com/esw-391

This interview is a bit different from our norm. We talk to the founder and CEO of OpenVPN about what it is like to operate a business based on open source, particularly through trying times like the recent pandemic. How do you compete when your competitors are free to build products using your software and IP? It seems like an oxymoron, but an open source-based business actually has some significant advantages over the closed source commercial approach. Show Notes: https://securityweekly.com/esw-391

Estas son las aplicaciones que he borrado del ipad en diciembre 2024: **Redes sociales y mensajería**: Facebook, X, LinkedIn, Messenger y Telegram, **Multimedia y edición:** Dazn, Mitele, iMovie, Shazam, HashPhotos, Linearity curve, fontmania, Color, Freeform, Match Triple 3D, Flikr**Juegos**: Magic y RetroArch**Otras:** Feedly, Pocket, Milanote, Trello, OpenVPN, OpenBank, Soulver, SpeedTest, Remote, Traductor de Google, Linky, Buy me a pie, Opener, RVNC ViewerYa hablé de los usos actuales que le doy a mi ipad en este capítulo: 61 - ¿Para qué uso mi iPad? https://podcasters.spotify.com/pod/show/al-daily-podcast/episodes/61---Para-qu-uso-mi-iPad-e2l9scn/a-abcsvf2Dime qué te ha parecido este capitulo y deja un comentario en ivoox o Spotify.Si lo prefieres, envíame un correo electrónico a la dirección de gmail almadailypodcast. En redes soy @almajefi y me encuentras en X / Twitter, Bluesky, Threads, Instagram y Telegram.

There's been a bit of a shakeup this week, with Torvalds criticizing Docker, Rustls dominating the TLS performance war, and Intel releasing a graphics card while "retiring" their CEO. Then, Flathub and KDE are working on their finances, OpenVPN has modernized its kernel driver, and Steam Machines may be back! Oh, and don't forget OBS 31 or the potential security issue with OpenWRT! For tips, we have eza as an ls replacement, pv for pipe progress viewing, IMSProg for EEPROM hacking, and HandlePowerKey for customizing what your machine does when you hit the power button. Grab the show notes at https://bit.ly/4gl1VtB and enjoy! Host: Jonathan Bennett Co-Hosts: Rob Campbell, David Ruggles, and Jeff Massie Want access to the video version and exclusive features? Become a member of Club TWiT today! https://twit.tv/clubtwit Club TWiT members can discuss this episode and leave feedback in the Club TWiT Discord.

www.iotusecase.com#CONDITION-MONITORING #5G #WASSERMANAGEMENT In der 148. Folge des IoT Use Case Podcasts gibt Jürgen Grauer, Sales Direktor EMEA bei Red Lion Controls, spannende Einblicke in die Modernisierung von 70 Regenwasser-Pumpstationen der Entwässerungsbetriebe Würzburg. Er erklärt, wie Red Lion durch den Einsatz moderner Kommunikationsstandards und 5G-Kompatibilität die Zukunftssicherheit dieser kritischen Infrastruktur gewährleistet.  Folge 148 auf einen Blick (und Klick):[08:43] Herausforderungen, Potenziale und Status quo – So sieht der Use Case in der Praxis aus[13:00] Lösungen, Angebote und Services – Ein Blick auf die eingesetzten TechnologienZusammenfassung der Podcastfolge: In dieser Episode geht es um die Modernisierung von 70 Regenwasser-Pumpstationen der Entwässerungsbetriebe Würzburg in Zusammenarbeit mit Red Lion Controls. Die Pumpstationen, ursprünglich auf 3G-Basis betrieben, wurden mit modernen Kommunikationsstandards und 5G-Kompatibilität ausgestattet, um eine zukunftssichere, störungsfreie Überwachung und Kontrolle der Infrastruktur zu gewährleisten. Hauptthemen und Herausforderungen: Technologie-Upgrade für kritische Infrastruktur: Red Lion unterstützt den Übergang von veralteten Steuerungen und 3G-Modems hin zu einem modernen System, das 4G/5G und OPC UA integriert. OPC UA und DNP3-Integration: Diese Protokolle ermöglichen eine nahtlose Kommunikation zwischen OT (Operational Technology) und IT (Information Technology), ein Schlüssel für die Datenerfassung und Echtzeitüberwachung. Nutzung von Crimson®: Red Lions Low-Code-Software Crimson® bietet eine einfache, grafische Bedienoberfläche zur Konfiguration und Konvertierung von Protokollen. Die Software kann kostenfrei heruntergeladen werden und unterstützt den OPC UA-Server und -Client ohne zusätzliche Lizenzgebühren. Datensicherheit und Echtzeit-Datenerfassung: Die Datenpufferung über die FlexEdge®-Plattform gewährleistet, dass im Falle von Verbindungsabbrüchen keine Daten verloren gehen. OpenVPN und weitere Sicherheitsfunktionen schützen die Daten. Flexible Cloud-Anbindung: Die Lösung erlaubt eine einfache Integration mit führenden Cloud-Plattformen wie AWS, Azure und Aveva über MQTT und REST-API, was die Datenanalyse und -optimierung vereinfacht. Die Modernisierung der Pumpstationen zeigt, wie durch gezielte Upgrades hohe Kosten für vollständige Systemaustausche vermieden werden können. Mit Red Lions Einstieg in die HMS-Gruppe werden zukünftig noch umfassendere Security- und Netzwerklösungen erwartet, die vor allem für kritische Infrastrukturen von Bedeutung sind. -----Relevante Folgenlinks:Madeleine: (https://www.linkedin.com/in/madeleine-mickeleit/)Jürgen: (linkedin.com/in/jürgen-grauer-4b81b91a8)IoT im Klärwerk Bad Pyrmont: (https://iotusecase.com/de/podcast/energiekosten-ausfaelle-reduzieren/)Jetzt IoT Use Case auf LinkedIn folgen

Wes got Mom a new Linux laptop, and he lets her pick the distro. Plus, we take a look at the new Ubuntu 24.10, and why we think this release might be a good sign for the future.Sponsored By:Jupiter Party Annual Membership: Put your support on automatic with our annual plan, and get one month of membership for free!Tailscale: Tailscale is a programmable networking software that is private and secure by default - get it free on up to 100 devices! 1Password Extended Access Management: 1Password Extended Access Management is a device trust solution for companies with Okta, and they ensure that if a device isn't trusted and secure, it can't log into your cloud apps. Support LINUX UnpluggedLinks:

DEFCON Hijinx, AMD, Ukraine, FreeBSD, OpenVPN, the Pwnie Awards, Josh Marpet, and more, on this Edition of the Security Weekly News. Visit https://www.securityweekly.com/swn for all the latest episodes! Show Notes: https://securityweekly.com/swn-406

DEFCON Hijinx, AMD, Ukraine, FreeBSD, OpenVPN, the Pwnie Awards, Josh Marpet, and more, on this Edition of the Security Weekly News. Show Notes: https://securityweekly.com/swn-406

DEFCON Hijinx, AMD, Ukraine, FreeBSD, OpenVPN, the Pwnie Awards, Josh Marpet, and more, on this Edition of the Security Weekly News. Visit https://www.securityweekly.com/swn for all the latest episodes! Show Notes: https://securityweekly.com/swn-406

DEFCON Hijinx, AMD, Ukraine, FreeBSD, OpenVPN, the Pwnie Awards, Josh Marpet, and more, on this Edition of the Security Weekly News. Show Notes: https://securityweekly.com/swn-406

[Referências do Episódio] Ligação a cobrar: como criminosos estão atacando atendentes de call centers usando engenharia social - https://sidechannel.blog/ligacao-a-cobrar-como-criminosos-estao-atacando-atendentes-de-call-centers/  QuickShell: Sharing Is Caring about an RCE Attack Chain on Quick Share - https://www.safebreach.com/blog/rce-attack-chain-on-quick-share  Researchers Uncover 10 Flaws in Google's File Transfer Tool Quick Share - https://thehackernews.com/2024/08/researchers-uncover-10-flaws-in-googles.html  Chained for attack: OpenVPN vulnerabilities discovered leading to RCE and LPE - https://www.microsoft.com/en-us/security/blog/2024/08/08/chained-for-attack-openvpn-vulnerabilities-discovered-leading-to-rce-and-lpe/  DEF CON Official Talk | AMD Sinkclose: Universal Ring-2 Privilege Escalation | Las Vegas, NV - https://ioactive.com/event/def-con-talk-amd-sinkclose-universal-ring-2-privilege-escalation/  Guest Memory Vulnerabilities - https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7014.html  Roteiro e apresentação: Carlos Cabral e Bianca Oliveira Edição de áudio: Paulo Arruzzo Narração de encerramento: Bianca Garcia

This week we deep-dive into one of the best vulnerabilities we've seen in a long time _regreSSHion_ - an unauthenticated, remote, root code-execution vulnerability in OpenSSH. Plus we cover updates for Plasma Workspace, Ruby, Netplan, FontForge, OpenVPN and a whole lot more.

GMOサイバーセキュリティ byイエラエは、同社が提供する自動脆弱性診断・ASMツール「GMOサイバー攻撃 ネットde診断」の診断機能を拡張したと発表した。

Gabriel Custodiet speaks for a second time with Viktor Viksei about the problems of the VPN industry, about how VPNs work, and about one of the few non-sellouts to privacy: IVPN.   First episode with Viktor (Episode 39): https://odysee.com/@WatchmanPrivacy:1/UncomfortableTruthsAboutVPNIndustry:5 Guest Links → https://twitter.com/vonthedock (Viktor Viksei) → https://www.ivpn.net/ → https://www.ivpn.net/blog/   Watchman Privacy → https://watchmanprivacy.com → https://twitter.com/watchmanprivacy → https://www.amazon.com/Watchman-Guide-Privacy-Financial-Lifestyle/dp/B08PX7KFS2   Privacy Courses (supports the show) → https://rpf.gumroad.com/l/privatebitcoin → https://rpf.gumroad.com/l/hackproof   Monero Donation (supports the show) →8829DiYwJ344peEM7SzUspMtgUWKAjGJRHmu4Q6R8kEWMpafiXPPNBkeRBhNPK6sw27urqqMYTWWXZrsX6BLRrj7HiooPAy   Bitcoin Donation (supports the show) →https://btcpay0.voltageapp.io/apps/3JDQDSj2rp56KDffH5sSZL19J1Lh/pos   Please subscribe to and rate this podcast wherever you can to help it thrive. Thank you! → https://www.youtube.com/@WatchmanPrivacy  →https://odysee.com/@WatchmanPrivacy   Timeline 0:00 – Introduction 1:47 – The problem with free VPNs 4:30 – Your VPN won't go to jail for you for $5 6:50 – How bad are the fake VPN review websites? 10:45 – VPN Industry consolidation 15:45 – Most important aspects of a VPN 18:19 – Has IVPN been targeted by VPN juggernauts? 21:54 – Any trends in VPNs that IVPN is focused on 25:24 – What is happening at the ground level of a VPN server 28:35 – How would a 3-letter agency track a VPN user? 34:00 – How can VPN companies mitigate 3-letter agency tracking? 39:08 – How can we test if a VPN is working? 41:10 – WireGuard vs OpenVPN 44:50 – Is random-generated username and no password of IVPN and Mullvad a problem? 46:35 – Anti-Tracker of IVPN 49:30 – Why does torrenting on US VPN servers matter? 52:38 – Removal of killswitch due to Apple problems 55:56 – V2Ray: what does it do for us? 59:22 – VPN discrimination solutions 1:05:00 – Final thoughts   #IVPN #VPNIndustry #WatchmanPrivacy

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

Privacy Badger blocks trackers on news sites and prevents browser exposure to unwanted domains like TikTok and Datadog. No major updates on EU's controversial Article 45 in eIDAS 2.0. Industry pushback continues as implementation would threaten encryption. Cryptocurrency exchange Poloniex lost $130M in a hot wallet hack, the 14th largest crypto theft. Decentralized finance platform Raft lost $3.3M due to an exploit. Crook operated website iotaseed.io to generate wallet seed phrases, then recorded and stole them. New Intel processor vulnerability called Downfall leaks encryption keys and sensitive data between users on shared systems. Russia moves to formally ban all VPN use in the country. Two new flaws found in OpenVPN software, one allowing memory access. SpinRite development paused as DOS and Windows versions are complete. Understanding assembly language helps malware analysis and exploit development, but high-level decompilers also useful. Quantum-safe symmetric cryptography is limited compared to asymmetric crypto. EU's Article 45 allows transparent decryption and traffic interception, supposedly for security purposes. "Windshield Barnacle" parking enforcement device uses suction cups and 1000 lbs of force to immobilize vehicles until parking tickets are paid. Sci-fi book series Aeon 14 by M.D. Cooper offers fun military space opera adventure. 27-year-old theoretical crypto attack now shown practical. Passive network observers can steal SSH RSA keys if faulty signature generated, allowing impersonation. Show Notes - https://www.grc.com/sn/SN-948-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to this show at https://twit.tv/shows/security-now. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit You can submit a question to Security Now at the GRC Feedback Page. For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: grc.com, also the home of the best disk maintenance and recovery utility ever written Spinrite 6. Sponsors: kolide.com/securitynow bitwarden.com/twit GO.ACILEARNING.COM/TWIT

На прошедшей неделе в России предприняли попытку заблокировать уже не просто популярные VPN-клиенты, а протоколы OpenVPN и WireGuard. Эти попытки оказались успешными, но спустя несколько дней блокировки были сняты. Что это было остается только догадываться. Обсуждаем это и другие новости недели. Тайминги: 00:00:00 — Вступление и новости проекта Собираем вопросы для выпуска с Покрас Ломпасом 00:05:59 — Блокировка протоколов VPN В России Ton VPN. 00:17:39 — Представлены новые устройства Яндекса: Станция ТВ и Станция ТВ Про BeardyCast 395 «

A short podcast updating listeners on the security news of the last few days, as prepared by Catalin Cimpanu and read by Kaitlyn Sawrey. You can find the newsletter version of this podcast here.

Complex DDoS attacks on the rise MI6 warns of Chinese data traps Microsoft expands cloud log access And now a word from our sponsor, OpenVPN Karim Hakim, CTO at Hakim Misr Paco, says that CloudConnexa has given him some long-sought peace of mind. “OpenVPN has helped my company to access remote nodes securely without worrying about security protocols,” he says. “My company has been looking for a similar solution for years, and we finally got what we were looking for.” Read more at the link in our show notes.

Singpolyma from JMP.chat joins Ask Noah to discuss the public launch of JMP.chat! -- During The Show -- 01:40 JMP.Chat 6 Years in Beta Lots of work Always over delivered Singpolyma from JMP.Chat The history of JMP.Chat Why is JMP.Chat for geeks? De-coupling number from a device Financial model Beta Price Extension Pricing Porting Multi Account Cheogram App Desktop Apps Movim (https://mov.im/login) Gajim (https://gajim.org/) Dino (https://dino.im/) Beagle (Mac) (https://beagle.im/) Snikket (https://snikket.org/) JMP.Chat Matrix Bridging/Integration Porting numbers out 46:44 News Wire Distro Box 1.5 - GitHub (https://github.com/89luca89/distrobox/releases/tag/1.5.0) Tails 5.14 - Tails (https://tails.boum.org/news/version_5.14/index.en.html) Nvidia Driver - Nvidea (https://www.nvidia.com/download/driverResults.aspx/205464/en-us/) SparkyLinux 7.0 - Sparky Linux (https://sparkylinux.org/sparky-7-0-orion-belt/) Ultra Marine Linux 38 - Ultra Marine (https://ultramarine-linux.org) OpenVPN 2.6.5 - OpenVPN (https://community.openvpn.net/openvpn/wiki/Downloads) Linux 6.3.8 - LWN (https://lwn.net/Articles/934621/) Rust Tool Chain - Phoronix (https://www.phoronix.com/news/Rust-For-Linux-6.5) Steam Linux Beta Client - Phoronix (https://www.phoronix.com/news/Steam-Force-Desktop-UI-Scaling) LLM Blender - Mark Tech Post (https://www.marktechpost.com/2023/06/19/meet-llm-blender-a-novel-ensembling-framework-to-attain-consistently-superior-performance-by-leveraging-the-diverse-strengths-of-multiple-open-source-large-language-models-llms/) MS ORCA - Indian Express (https://indianexpress.com/article/technology/artificial-intelligence/microsoft-orca-ai-model-8672524/) 48:20 Firefox Multi-account containers - Jeremy Multi Account Containers plugin (https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/) is great Thanks for the feedback 49:29 Open Source Cameras? - Ziggy Dedicated open source camera episode? Synology Disk Station Axis Cameras Zone Minder (https://zoneminder.com/) Motion Eye (https://github.com/motioneye-project/motioneye) -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/342) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed) • Ask Noah Show © CC-BY-ND 2021 •

Este finde me lié la manta a la cabeza y me puse a cambiar la interconexión VPN entre las distintas casas de la familia. He eliminado la configuración de OpenVPN para montarlo todo con WireGuard y todo es mucho más limpio y rápido. Os cuento cómo lo he hecho.

This week we dig into the Asus Ally. We'll tell you about an awesome all-in-one content production studio that works FLAWLESS on Linux, and of course your questions go to the front of the line! -- During The Show -- 03:30 Self Hosted IMAP backup - Sebastián Steve would call Noah Mail spooler Local Thunderbird 08:53 General Feedback - DJ Americas Mailbox for friendly CMRA/PMB privacy/opsec 11:42 News Wire FreeSpire 9.5 FreeSpire (https://www.freespire.net/2023/05/freespire-95-released.html) TinyCore 14 Tiny Core Linux (http://forum.tinycorelinux.net/index.php/topic,26201.0.html) Firefox 113 MOzilla (https://www.mozilla.org/en-US/firefox/113.0/releasenotes/) Haiku & RISC-V RISC-V (https://landscape.riscv.org/card-mode?selected=haiku) BcacheFS Kernel.org (https://lore.kernel.org/lkml/20230509165657.1735798-1-kent.overstreet@linux.dev/T/#mf171fd06ffa420fe1bcf0f49a2b44a361ca6ac44) OpenVPN 2.6.4 OpenVPN (https://community.openvpn.net/openvpn/wiki/Downloads) Royal Ransomware Group Data Center Knowledge (https://www.datacenterknowledge.com/security/royal-ransomware-expands-target-linux-vmware-esxi) BPFDoor Hacker News (https://thehackernews.com/2023/05/new-variant-of-linux-backdoor-bpfdoor.html) NetFilter Flaw Bleeping Computer (https://www.bleepingcomputer.com/news/security/new-linux-kernel-netfilter-flaw-gives-attackers-root-privileges/) MichaelKors RaaS Hacker News (https://thehackernews.com/2023/05/new-michaelkors-ransomware-as-service.html) CISA KEV Security Week (https://www.securityweek.com/cisa-several-old-linux-vulnerabilities-exploited-in-attacks/) New Intel CPU Microcode Phoronix (https://www.phoronix.com/news/Intel-12-May-2023-Microcode) OpenSSF Alpha-Omega Project Security Week (https://www.securityweek.com/openssf-receives-5-million-for-open-source-software-security-project/) ImageBind Peta Pixel (https://petapixel.com/2023/05/10/meta-unveils-open-source-multimodal-generative-ai-system/) Hugging Face Make Use Of (https://www.makeuseof.com/what-is-huggingchat/) European Parliament Vote FSFE.org (https://fsfe.org/news/2023/news-20230511-01.en.html) 15:00 RODECaster II Pro with Linux Fully integrated audio production studio Smart Pads Different mixers on faders Mix minus Type C ports Rode.com (https://rode.com/en-us/interfaces-and-mixers/rodecaster-series/rodecaster-pro-ii) Guitar Center (https://www.guitarcenter.com/RODE/RODECaster-PRO-II-Integrated-Audio-Production-Studio-1500000376142.gc?cntry=us) Sweetwater (https://www.sweetwater.com/store/detail/Rodecaster2--rode-rodecaster-pro-ii-podcast-production-console) 23:40 Willow Voice Assistant $50 peice of tech Hardware based on ESP-32 Creator wanted to jump start Home Assistant Voice Assistant Might only relay sound to Home Assistant YouTube (https://www.youtube.com/watch?v=8ETQaLfoImc) ARSTechnica (https://arstechnica.com/gadgets/2023/05/willow-is-a-faster-self-hosted-diy-voice-assistant-built-on-50-gadgets/) 32:40 ASUS ROG Ally ROG Ally Runs MS Windows not Linux More up to date (released 2 years after Steam Deck) Low wattage modes Planned Obsolescence Running Windows ChromeBook Ewaste Does windows give ROG Ally and edge? Target audiance ARSTechnica (https://arstechnica.com/gadgets/2023/05/the-asus-rog-ally-runs-windows-eats-battery-and-needs-time-to-cook/) 50:50 Perils of Unsupported FOSS In Enterprise Open Source is winning Open Source doesn't mean cheap/free Software needs to be maintained 3rd Party Support Community Support The Register (https://www.theregister.com/2023/05/15/the_perils_of_unsupported_opensource/) -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/337) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed)

All links and images for this episode can be found on CISO Series. This show was recorded in front of a live audience in New York City! This week's episode is hosted by me, David Spark (@dspark), producer of CISO Series, and a special guest host, Aaron Zollman, CISO & vp, platform engineering, Cedar. Our guest is Colin Ahern, chief cyber officer for the State of New York. Thanks to our podcast sponsor, OpenVPN, SlashNext & Votiro Take the cost and complexity out of secure networking with OpenVPN. Whether you choose our cloud-delivered or self-hosted solution, subscriptions are based on concurrent connections, so you pay for what you actually use. Start today with free connections, no credit card required, and scale to paid when you're ready. SlashNext, a leader in SaaS-based Integrated Cloud Messaging Security across email, web, and mobile has the industry's first artificial intelligence solution, HumanAI, that uses generative AI to defend against advanced business email compromise (BEC), supply chain attacks, executive impersonation, and financial fraud. Request a demo today. No matter what technology or training you provide, humans are still the greatest risk to your security. Votiro's API-centric product sanitizes every file before it hits the endpoint, so the files that your employees open are safe. This happens in milliseconds, so the business stays safe and never slows down. In this episode: If you hired someone today, how would you know in 3 months time that they were the right fit? Do you have any other questions you've heard from candidates that you think are better? What doesn't the government currently know about cloud providers that they should know?

Craig Peterson  Insider Show NotesDecember 5 to December 11, 2022 China… Apple Makes Plans to Move Production Out of China https://www.wsj.com/articles/apple-china-factory-protests-foxconn-manufacturing-production-supply-chain-11670023099 In recent weeks, Apple Inc. has accelerated plans to shift some of its production outside China, long the dominant country in the supply chain that built the world's most valuable company, say people involved in the discussions. It is telling suppliers to plan more actively for assembling Apple products elsewhere in Asia, particularly India and Vietnam, they say and looking to reduce dependence on Taiwanese assemblers led by Foxconn Technology Group. After a year of events that weakened China's status as a stable manufacturing center, the upheaval means Apple no longer feels comfortable having so much of its business tied up in one place, according to analysts and people in the Apple supply chain. Cybercrime… Spyware posing as VPN apps https://www.welivesecurity.com/videos/spyware-posing-vpn-apps-week-security-tony-anscombe/ Bahamut APT group targets Android users via trojanized versions of two legitimate VPN apps – SoftVPN and OpenVPN. Since January 2022, Bahamut has distributed at least eight malicious apps to pilfer sensitive user data and actively spy on victims' messaging apps. These apps were never available for download from Google Play; instead, they were distributed through a fake SecureVPN website. ++++++++ Darknet markets generate millions in revenue by selling stolen personal data https://arstechnica.com/tech-policy/2022/12/darknet-markets-generate-millions-in-revenue-selling-stolen-personal-data/ Stolen data products flow through a supply chain consisting of producers, wholesalers, and consumers.  The stolen data supply chain begins with producers—hackers who exploit vulnerable systems and steal sensitive information such as credit card numbers, bank account information, and Social Security numbers. Next, the stolen data is advertised by wholesalers and distributors who sell the data. Finally, the data is purchased by consumers who use it to commit various forms of fraud, including fraudulent credit card transactions, identity theft, and phishing attacks. ++++++++ Voice-scamming site “iSpoof” seized, 100s arrested in a massive crackdown https://nakedsecurity.sophos.com/2022/11/25/voice-scamming-site-ispoof-seized-100s-arrested-in-massive-crackdown/ Whether you call it Caller ID or CLI, it's no more use in identifying the caller's actual phone number than the “From:” header in an email is in identifying the sender of an email. As a cybersecurity measure to help you identify callers you do trust, [Caller-ID] has an extreme false negative problem, meaning that if a call pops up from Dad, or Auntie Gladys, or perhaps more significantly, from Your Bank… …then there's a significant risk that it's a scam call that's deliberately been manipulated to get past your “do I know the caller?” test. ++++++++ U.S. Govt. Apps Bundled Russian Code With Ties to Mobile Malware Developer https://krebsonsecurity.com/2022/11/u-s-govt-apps-bundled-russian-code-with-ties-to-mobile-malware-developer/ A recent scoop by Reuters revealed that mobile apps for the U.S. Army and the Centers for Disease Control and Prevention (CDC) were integrating software that sends visitor data to a Russian company called Pushwoosh, which claims to be based in the United States. But that story omitted a crucial historical detail about Pushwoosh: In 2013, one of its developers admitted to authoring the Pincer Trojan, malware designed to intercept and forward text messages from Android mobile devices surreptitiously. Reuters also learned that the company's address in California does not exist and that two LinkedIn accounts for Pushwoosh employees in Washington, D.C. were fake. Android… Samsung's Android app-signing key has leaked and is being used to sign malware https://arstechnica.com/gadgets/2022/12/samsungs-android-app-signing-key-has-leaked-is-being-used-to-sign-malware/ A developer's cryptographic signing key is one of the major linchpins of Android security. Any time Android updates an app, the old app's signing key on your phone must match the key of the update you're installing. If a developer's signing key leaked, anyone could distribute malicious app updates, and Android would happily install them, thinking they are legit. On Android, the app-updating process isn't just for apps downloaded from an app store; you can also update bundled-in system apps made by Google, your device manufacturer, and any other bundled apps. ZeroTrust… Cloud security starts with zero trust https://www.helpnetsecurity.com/2022/11/28/cloud-zero-trust/ Most organizations have outdated security systems that are generally based on-premises. These outdated systems often add an extra layer of complexity to shifting to the cloud, but this complexity does not mean organizations should hold off on this shift.

Xe Iaso is the Archmage of Infrastructure at Tailscale and previously worked at Heroku.This episode originally aired on Software Engineering Radio but includes some additional discussion about their blog near the end of the episode.Topics covered: Use cases for VPNs Simplifying service authentication by identifying users via IP Peer-to-peer vs centralized "Virtual Pain Networks" Tailscale's tech stack and why they forked the go compiler DERP relay servers Struggling with the iOS network extension size limit The surprisingly small amount of infrastructure required to run a VPN Running your company on your own product Working at Heroku vs Tailscale Using the socratic style of debate in technical blog posts Related Links @theprincessxena Xe's Blog ACL samples Go links origin story How Tailscale works Tailscale SSH How Tailscale assigns IP addresses Hey linker, can you spare a meg? My Blog is Hilariously Overengineered to the Point People Think it's a Static Site The Sheer Terror of PAM Transcript[00:00:00] Jeremy: Today I'm talking to Xe Iaso, they're the archmage of infrastructure at tailscale, and they also have a great blog everyone should check out. Xe, welcome to software engineering radio.[00:00:12] Xe: Thanks. It's great to be here. [00:00:14] Jeremy: I think the first thing we should start with, is what's a, a VPN, because I think some people they may have used it to remote into their workplace or something like that. But I think the, the scope of what it's good for and what it does is a lot broader than that. So maybe you could talk a little bit about that first.[00:00:31] Xe: Okay. a VPN is short for virtual private network. It's basically a fake network that's overlaid on top of existing networks. And then you can use that network to do whatever you would with a normal computer network. this term has been co-opted by companies that are attempting to get into the, like hide my ass style market, where, you know, you encrypt your internet information and keep it safe from hackers.But, uh, so it makes it really annoying and hard to talk about what a VPN actually is. Because tailscale, uh, the company I work for is closer to like the actual intent of a VPN and not just, you know, like hide your internet traffic. That's already encrypted anyway with another level of encryption and just make a great access point for, uh, three letter agencies.But are there, use cases, past that, like when you're developing a piece of software, why would you decide to use a VPN outside of just because I want my, you know, my workers to be able to get access to this stuff.[00:01:42] Xe: So something that's come up, uh, when I've been working at tailscale is that sometimes we'll make changes to something. And it'll be changes to like the user experience of something on the admin panel or something. So in a lot of other places I've worked in order to have other people test that, you know, you'd have to push it to the cloud.It would have to spin up a review app in Heroku or some terrifying terraform of abomination would have to put it out onto like an actual cluster or something. But with tail scale, you know, if your app is running locally, you just give like the name of your computer and the port number. And you know, other people are able to just see it and poke it and experience it.And that basically turns the, uh, feedback cycle from, you know, like having to wait for like the state of the world to converge, to, you know, make a change, press F five, give the URL to a coworker and be like, Hey, is this Gucci?they can connect to your app as if you were both connected to the same switch.[00:02:52] Jeremy: You don't have to worry about, pushing to a cloud service or opening ports, things like that.[00:02:57] Xe: Yep. It will act like it's in the same room, even when they're not it'll even work. if you're at both at Starbucks and the Starbucks has reasonable policies, like holy crap, don't allow devices to connect to each other directly. so you know, you're working on. Like your screenplay app at your Starbucks or something, and you have a coworker there and you're like, Hey, uh, check this out and, uh, give them the link.And then, you know, they're also seeing the screenplay editor.[00:03:27] Jeremy: in terms of security and things like that. I mean, I'm picturing it kind of like we were sitting in the same room and there's a switch and we both plugged in. Normally when you do something like that, you kind of have, full access to whatever else is on the switch. Uh, you know, provided that's not being blocked by a, a firewall.is there like a layer of security on top of that, that a VPN service like tailscale would provide.[00:03:53] Xe: Yes. Um, there are these things called access control lists, which are kind of like firewall rules, except you don't have to deal with like the nightmare of writing an IP tables rule that also works in windows firewall and whatever they use in Mac OS. The ACL rules are applied at the tailnet level for every device in the tailnet.So if you have like developer machines, you can put people into groups as things like developers and say that developer machines can talk to production, but not people in QA. They can only talk to testing and people on SRE have, you know, permissions to go everywhere and people within their own teams can connect to each other. you can make more complicated policies like that fairly easily.[00:04:44] Jeremy: And when we think about infrastructure for, for companies, you were talking about how there could be development, infrastructure, production, infrastructure, and you kind of separate it all out. when you're working with cloud infrastructure. A lot of times, there's the, I always forget what it stands for, but there's like IAM.There's like policies that you can set up with the cloud provider that says these users can access this, or these machines can access this. And, and I wonder from your perspective, when you would choose to use that versus use something at the, the network or the, the VPN level.[00:05:20] Xe: The way I think about it is that things like IAM enforce, permissions for like more granularly scoped things like can create EC2 instances or can delete EC2 instances or something like that. And that's just kind of a different level of thing. uh, tailscale, ACLs are more, you know, X is allowed to connect to Y or with tailscale, SSH X is allowed to connect as user Y.and that's really different than like arbitrary capability things like IAM offers.you could think about it as an IAM system, but the main permissions that it's exposing are can X connect to Y on Zed port.[00:06:05] Jeremy: What are some other use cases where if you weren't using a VPN, you'd have to do a lot more work or there's a lot more complexity, kind of what are some cases where it's like, okay, using a VPN here makes a lot of sense.(The quick and simple guide to go links https://www.trot.to/go-links) [00:06:18] Xe: There is a service internal to tailscale called go, which is a, clone of Google's so-called go links where it's basically a URL shortener that lives at http://go. And, you know, you have go/something to get to some internal admin service or another thing to get to like, you know, the company directory and notion or something, and this kind of thing you could do with a normal setup, you know, you could set it up and have to do OAuth challenges everywhere and, you know, have to put and make sure that everyone has the right DNS configuration so that, it shows up in the right place.And then you have to deal with HTTPS um, because OAuth requires HTTPS for understandable and kind of important reasons. And it's just a mess. Like there's so many layers of stuff like the, the barrier to get, you know, like just a darn URL, shortener up turns from 20 minutes into three days of effort trying to, you know, understand how these various arcane things work together.You need to have state for your OAuth implementation. You need to worry about what the hell a a JWT is (sigh) . It's it it's just bad. And I really think that something like tailscale with everybody has an IP address. In order to get into the network, you have to sign in with your, auth provider, your, a provider tells tailscale who you are.So transitively every IP address is tied to an owner, which means that you can enforce access permission based on the IP address and the metadata about it that you grab from the tailscale. daemon, it's just so much simpler. Like you don't have to think about, oh, how do I set up OAuth this time? What the hell is an oauth proxy?Um, what is a Kubernetes? That sort of thing you just think about like doing the thing and you just do it. And then everything else gets taken care of it. It's like kind of the ultimate network infrastructure, because it's both omnipresent and something you don't have to think about. And I think that's really the power of tailscale.[00:08:39] Jeremy: typically when you would spin up a, a service that you want your developers or your system admins, to be able to log into, you would have to have some way of authenticating and authorizing that user. And so you were talking about bringing in OAuth and having your, your service understand that.But I, I guess what you're saying is that when you have something like tailscale, that's kind of front loaded, I guess you, you authenticate with tail scale, you get onto the network, you get your IP. And then from that point on you can access all these different services that know like, Hey, because you're on the network, we know you're authenticated and those services can just maybe map that IP that's not gonna change to like users in some kind of table. Um, and not have to worry about figuring out how do I authenticate this user.[00:09:34] Xe: I would personally more suggest that you use the, uh, whois, uh, look up route in the tailscale daemon's local API, but basically, yeah, you don't really have to worry too much about like the authentication layer because the authentication layer has already been done. You know, you've already done your two factor with Gmail or whatever, and then you can just transitively push that property onto your other machines.[00:10:01] Jeremy: So when you talk about this, this whois daemon, can you give an example of I'm in the network now I'm gonna make a service call to an application. what, what am I doing with this? This whois daemon?[00:10:14] Xe: It's more of like a internal API call that we expose via tailscaled's, uh, Unix, socket. but basically you give it an IP address and a port, and it tells you who the person is. It's kind of like the Unix ident protocol in a way, except completely not. And at a high level, you know, if you have something like a proxy for Grafana, you have that proxy for Grafana, make a call to the local tailscale daemon, and be like, Hey, who was this person?And the tailscale, daemon will spit back at JSON object. Like, oh, it's this person on this device and there you can do additional logic like maybe you shouldn't be allowed to delete things from an iOS device, you know, crazy ideas like that. there's not really support for like arbitrary capabilities and tailscaled at the time of recording, but we've had some thoughts would be cool.[00:11:17] Jeremy: would that also include things like having roles, for example, even if it's just strings, um, that you get back so that your application would know, okay. This person, is supposed to have admin access to this service based on what I got back from, this, this service.[00:11:35] Xe: Not currently, uh, you can probably do it via convention or something, but what's currently implemented in the actual, like, source code and user experience that they, you can't do that right now. Um, it is something that I've been, trying to think about different ways to solve, but it's also a problem.That's a bit big for me personally, to tackle.[00:11:59] Jeremy: there's, there's so many, I guess, different ways of doing it. That it's kind of interesting to think of a solution that's kind of built into the, the network. Yeah.[00:12:10] Xe: Yeah. and when I describe that authentication thing to some people, it makes them recoil in shock because there's kind of a Stockholm syndrome type effect with security, for a lot of things where, the easy way to do something and the secure way to do something are, you know, like completely opposite and directly conflicting with each other in almost every way.And over time, people have come to associate security or like corporate VPNs as annoying, complicated, and difficult. And the idea of something that isn't annoying, complicated or difficult will make people reject it, like just on principle, because you know, they've been trained that, you know, VPN equals virtual pain network and it, it's hard to get that association outta people's heads because you know, a lot of VPNs are virtual pain networks.Like. I used to work for Salesforce and Salesforce had this corporate VPN where no matter what you did, all of your traffic would go out to the internet from their data center. I think it was in San Francisco or something. And I was in the Seattle area. So whenever I had the VPN on my latency to Google shot up by like eight times and being a software person, you know, I use Google the same way that others breathe and it, it was just not fun.And I only had the VPN on for the bare minimum of when I needed it. And, oh God, it was so bad.[00:13:50] Jeremy: like some people, when they picture a VPN, they picture exactly what you're describing, where all of my traffic is gonna get routed to some central point. It's gonna go connect to the thing for me and then send the result back. so maybe you could talk a little bit about why that's, that's maybe a wrong assumption, I guess, in the case of tailscale, or maybe in the case of just more modern VPN solutions.[00:14:13] Xe: Yeah. So the thing that I was describing is what I've been lovingly calling the, uh, single point of failure as a service type model of VPN, where, you know, you have like the big server somewhere, it concentrates all the connections and, you know, like does things to make the computer feel like they've teleported over there, but overall it's a single point of failure.And if that falls over, you know, like goodbye, VPN. everybody's just totally screwed. And in contrast, tailscale does a more peer-to-peer thing so that everyone is basically on equal footing. Everyone can send traffic directly to each other, and if it can't get directly to there, it'll use a network of, uh, relay servers, uh, lovingly called Derp and you don't have to worry about, your single point of failure in your cluster, because there's just no single point of failure.Everything will directly communicate as much as possible. And if it can't, it'll still communicate anyway.[00:15:18] Jeremy: let's say I start up my computer and I wanna connect to a server in a data center somewhere at the very beginning, am I connecting to some server hosted at tailscale? And then. There's some kind of negotiation process where after that I connect directly or do I just connect directly straight away?[00:15:39] Xe: If you just turn on your laptop and log in, you know, to it signs into tailscale and gets you on the tailnet and whatnot, then it will actually start all connections via Derp just so that it can negotiate the, uh, direct connection. And in case it can't, you know, it's already connected via Derp so it just continues the connection with Derp and this creates a kind of seamless magic type experience where doing things over Derp is slower.Yes, it is measurably slower because you know, like you're not going directly, you're doing TCP inside of TCP. And you know, that comes with a average minefield of lasers or whatever you call it. And it does work though. It's not ideal if you wanna do things like copy large amounts of data, but if you want just want ssh into prod and see the logs for what the heck is going on and why you're getting paged at 3:00 AM. it's pretty great.[00:16:40] Jeremy: What you, you were calling Derp is it where you have servers kind of all over the world and somehow it determines which one's, I guess, is it which one's closest to your destination or which one's closest to you. I'm kind of[00:16:54] Xe: It's really interesting. It's one of the most weird distributed systems, uh, type things that I've ever seen. It's the kind of thing that could only come outta the mind of an X Googler, but basically every tailscale, every tailscale node has a connection to all of the Derp servers and through process of, you know, latency testing.It figures out which connection is the fastest and the lowest latency. And it calls that it's home Derp but because it's connected to everything is connected to every Derp you can have two people with different home Derps getting their packets relayed too other clients from different Derps.So, you know, if you have a laptop in Ottawa and a laptop in San Francisco, the laptop in San Francisco will probably use the, uh, Derp that's closest to it. But the laptop in Ottawa will also use the Derp that's closest to it. So you get this sort of like asynchronous thing, and it actually works out a lot better in practice, than you're probably imagining.[00:17:52] Jeremy: And then these servers, what was the, the technical term for them? Are they like relays or what's[00:17:58] Xe: They're relays. Uh, they only really deal with encrypted wire guard packets, and there's, no way for us at tailscale, to see the contents of Derp messages, it is literally just a forwarder. It, it literally just forwards things based on the key ID.[00:18:17] Jeremy: I guess if tail scale isn't able to decrypt the traffic, is, is that because the, the keys are only on the user's devices, like it's on their laptop and on the server they're trying to reach, or[00:18:31] Xe: Yeah. The private keys are live and die with those devices or the devices they were minted on. And the public keys are given to the coordination server and the coordination server spreads those around to every device in your tailnet. It does some limiting so that like, if you don't have ACL access to something, you don't get the private key, you don't get the, uh, public key for it.The public key, not the private key, the public key, not the private key. And yeah. Then, you know, you just go that way and it'll just figure it out. It's pretty nice.[00:19:03] Jeremy: When we're kind of talking about situations where it can't connect directly, that's where you would use the relay. what are kind of the typical cases where that happens, where you, you aren't able to just connect directly?[00:19:17] Xe: Hotel, wifi and paranoid network security setups, hotel wifi is the most notorious one because you know, you have like an overpriced wifi connection. And if you bring, like, I don't know like, You you're recording a bunch of footage on your iPhone. And because in, 2022. The iPhone has the USB2 connection on it.And you know, you wanna copy that. You wanna use the network, but you can't. So you could just let it upload through iCloud or something, or, you know, do the bare minimum. You need to get the, to get the data off with Derp it wouldn't be ideal, but it would work. And ironically enough, that entire complexity involved with, you know, doing TCP inside of TCP to copy a video file over to your laptop might actually be faster than USB2, which is something that I did the math for a while ago.And I just started laughing.[00:20:21] Jeremy: Yeah, that that is pretty, pretty ridiculous [00:20:23] Xe: welcome to the future, man (laughs) .[00:20:27] Jeremy: in terms of connecting directly, usually when you have a computer on the internet, you don't have all your ports open, you don't necessarily allow, just anybody to send you traffic over UDP and so forth. let's say I wanna send, UDP data to a, a server on my network, but, you know, maybe it has some TCP ports open. I I'm assuming once I connect into the network via the VPN, I'm able to use other protocols and ports that weren't necessarily exposed. Is that correct?[00:21:01] Xe: Yeah, you can use UDP. you can do basically anything you would do on a normal network except multicast um, because multicast is weird.I mean, there's thoughts on how to handle multicast, but the main problem is that like wireguard, which is what is tail tailscale is built on top of, is, so called OSI model layer three network, where it's at like, you know, the IP address level and multicast is a layer two or data link layer type thing.And, those are different numbers and, you can't really easily put, you know, like broadcast packets into IP, uh, IPV4 thinks otherwise, but, uh, in practice, no people don't actually use the broadcast address.[00:21:48] Jeremy: so for someone who's, they, they have a project or their company wants to get started. I mean, what does onboarding look like? What, what do they have to do to get all these devices talking to one another?[00:22:02] Xe: basically you, install tail scale, you log in with a little GUI thing or on a Linux server, you run tailscale up, and then you all log to the, to a, like a G suite account with the same domain name. So, you know, if your domain is like example.com, then everybody logs in with their example.com G suite account.And, there is no step three, everything is allowed and everything can just connect and you can change the permissions from there. By default, the ACLs are set to a, you know, very permissive allow everyone to talk to everyone on any port. Uh, just so that people can verify that it's working, you know, you can ping to your heart's content.You can play Minecraft with others. You can, you know, host an HTTP server. You can SSH into your development box and and write blog post with emacs, whatever you want.[00:22:58] Jeremy: okay, you install the, the software on your servers, your workstations, your laptops, and so on. And then at, after that there's some kind of webpage or dashboard you would go in and say, I want these people to be able to access these things and [00:23:14] Xe: Mm-hmm [00:23:15] Jeremy: these ports and so on.[00:23:17] Xe: you, uh, can customize the access control rules with something that looks like JSON, but with trailing commas and comments allowed, and you can go from there to customize basically anything to your heart's content. you can set rules so that people on the DevOps team can access everything, but you know, maybe marketing doesn't need access to the production database.So you don't have to worry about that as much.[00:23:45] Jeremy: there's, there's kind of different options for VPNs. CloudFlare access, zero tier, there's, there's some kind of, I think it's Nebula from slack or something like that. so I was kind of curious from your perspective, what's the, difference between those kinds of services and, and tailscale.[00:24:04] Xe: I'm gonna lead this out by saying that I don't totally understand the differences between a lot of them, because I've only really worked with tailscale. I know things about the other options, but, uh, I have the most experience with tailscale but from what I've been able to tell, there are things that tailscale offers that others don't like reverse mapping of IP addresses to people, or, there's this other feature that we've been working on, where you can embed tail scale as a library inside your go application, and then write a internal admin service that isn't exposed to the internet, but it's only exposed over tailscale.And I haven't seen a way to do those things with those others, but again, I haven't done much research. Um, I understand that zero tier has some layer, two capabilities, but I've, I don't have enough time in the day to look into.[00:25:01] Jeremy: There's been different, I guess you would call them VPN protocols. I mean, there's people have probably worked with IP sec in some situations they may have heard of OpenVPN, wireguard. in the case of tailscale, I believe you chose to build it on top of wireguard.So I wonder if you could talk a little bit about why, you chose wireguard and, and maybe what makes it unique.[00:25:27] Xe: I wasn't on the team that initially wrote like the core of tailscale itself. But from what I understand, wire guard was chosen because, what overhead, uh, it's literally, you just encrypt the packets, you send it to the other server, the other server decrypts them. And you know, you're done. it's also based purely on the public key. Um, the key pairs involved. And from what I understand, like at the wireguard protocol level, there's no reason why you, why you would need an IP address at all in theory, but in practice, you kind of need an IP address because you know, everything sucks. But also wire guard is like UDP only, which I think it at it's like core implementation, which is a step up from like AnyConnect and OpenVPN where they have TCP modes.So you can experience the, uh, glorious, trash fire of TCP in TCP. And from what I understand with wireguard, you don't need to set up a certificate authority or figure out how the heck to revoke certificates. Uh, you just have key pairs and if a node needs to be removed, you delete the key pair and you're done.And I think that really matches up with a lot of the philosophy behind how tailscale networks work a lot better. You know, you have a list of keys and if the network changes the list of keys changes, that's, that's the end of the story.So maybe one of the big selling points was just What has the least amount of things I guess, to deal with, or what's the, the simplest, when you're using a component that you want to put into your own product, you kind of want the least amount of things that could go wrong, I guess.[00:27:14] Xe: Yeah. It's more like simple, but not like limiting. Like, for example, a set of tinker toys is simple in that, you know, you can build things that you don't have to worry too much about the material science, but a set of tinker toys is also limiting because you know, like they're little wooden, dowels and little circles made out of wind that you stick the dowels into, you know, you can only do so much with it.And I think that in comparison, wireguard is simple. You know, there's just key pairs. They're just encryption. And it's simple in it's like overall theory and it's implementation, but it's not limiting. Like you can do pretty much anything you want with it.inherently whenever we build something, that's what we want, but that's a, that's an interesting way of putting it. Yeah.[00:28:05] Xe: Yeah. It. It can be kind of annoyingly hard to figure out how to make things as simple as they need to be, but still allow for complexity to occur. So you don't have to like set up a keyboard macro to write if error not equals nil over and over.[00:28:21] Jeremy: I guess the next thing I'd like to talk a little bit about is. We we've covered it a little bit, but at a high level, I understand that that tailscale uses wireguard, which is the open source, VPN protocol, I guess you could call it. And then there's the client software. You're saying you need to install on each of the servers and workstations.But there's also a, a control plane. and I wonder if you could kind of talk a little bit about I guess at a high level, what are all the different components of, of tailscale?[00:28:54] Xe: There's the agent that you install in your devices. The agent is basically the same between all the devices. It's all written in go, and it turns out that go can actually cross compile fairly well. So you have. Your, you know, your implementation in go, that is basically the, the same code, more or less running on windows, MacOS, freeBSD, Android, ChromeOS, iOS, Linux.I think I just listed all the platforms. I'm not sure, but you have that. And then there's the sort of control plane on tailscale's side, the control plane is basically like control, uh, which is, uh, I think a get smart reference. and that is basically a key dropbox. So, you know, you You authenticate through there. That's where the admin panel's hosted. And that's what tells the different tailscale nodes uh, the keys of all the other machines on the tailnet. And also on tailscale side there's, uh, Derp which is a fleet of a bunch of different VPSs in various clouds, all over the world, both to try to minimize cost and to, uh, have resiliency because if both digital ocean and Vultr go down globally, we probably have bigger problems.[00:30:15] Jeremy: I believe you mentioned that the, the clients were written in go, are the control plane and the relay, the Derp portion. Are those also written in go or are they[00:30:27] Xe: They're all written and go, yeah,go as much as possible. Yeah.It's kind of what happens when you have some ex go team members is the core people involved in tail scale, like. There's a go compiler fork that has some additional patches that go upstream either can't accept, uh, won't accept or hasn't yet accepted, for a while. It was how we did things like trying to shave off by bites from binary size to attempt to fit it into the iOS network extension limit.Because for some reason they only allowed you to have 15 megabytes of Ram for both like your application and working Ram. And it turns out that 15 megabytes of Ram is way more than enough to do something like OpenVPN. But you know, when you have a peer-to-peer VPN engine, it doesn't really work that well.So, you know, that's a lot of interesting engineering challenge.[00:31:28] Jeremy: That was specifically for iOS. So to run it on an iPhone.[00:31:32] Xe: Yeah. Um, and amazingly after the person who did all of the optimization to the linker, trying to get the binary size down as much as possible, like replacing Unicode packages was something that's more coefficient, you know, like basically all but compressing parts of the binary to try to save space. Then the iOS, I think 15 beta dropped and we found out that they increased the network extension Ram limit to 50 megabytes and the look of defeat on that poor person's face. I feel very bad for him.[00:32:09] Jeremy: you got what you wanted, but you're sad about it,[00:32:12] Xe: Yeah.[00:32:14] Jeremy: so that's interesting too. you were using a fork of the go compiler [00:32:19] Xe: Basically everything that is built is built using, uh, the tailscale fork, of the go compiler.[00:32:27] Jeremy: Going forward is the sort of assumption is that's what you'll do, or is it you're, you're hoping you can get this stuff upstreamed and then eventually move off of it.[00:32:36] Xe: I'm pretty sure that, I, I don't know if I can really make a forward looking statement like that, but, I've come to accept the fact that there's a fork of the go compiler. And as a result, it allows a lot more experimentation and a bit more of control, a bit more control over what's going on. like I'm, I'm not like the most happy with it, but I've, I understand why it exists and I'm, I've made my peace with it.[00:33:07] Jeremy: And I suppose it, it helps somewhat that the people who are working on it actually originally worked on the, go compiler at Google. Is that right?[00:33:16] Xe: Oh yeah. If, uh, there weren't ex go team people working on that, then I would definitely feel way less comfortable about it. But I trust that the people that are working on it, know what they're doing at least enough.[00:33:30] Jeremy: I, I feel like, that's, that's kind of the position we put ourselves in with software in general, right? Is like, do we trust our ourselves enough to do this thing we're doing?[00:33:39] Xe: Yeah. And trust is a bitch.[00:33:44] Jeremy: um, I think one of the things that's interesting about tail scale is that it's a product that's kind of it's like network infrastructure, right? It's to connect you to your other devices. And that's a little different than somebody running a software as a service. And so. how do you test something that's like built to support a network and, and how is that different than just making a web app or something like that.[00:34:11] Xe: Um, well, it's a lot more complicated for one, especially when you have to have multiple devices in the mix with multiple different operating systems. And I was working on some integration tests, doing stuff for a while, and it was really complicated. You have to spin up virtual machines, you know, you have to like make sure the virtual machines are attempting to download the version of the tailscale client you wanna test and. It's it's quite a lot in practice.[00:34:42] Jeremy: I mean, do you have a, a lab, you know, with Android phones and iPhones and laptops and all this sort of stuff, and you have some kind of automated test suite to see like, Hey, if these machines are in Ottawa and, my servers in San Francisco, like you're mentioning before that I can get from my iPhone to this server and the data center over here, that kind of thing.[00:35:06] Xe: What's the right way to phrase this without making things look bad. Um, it's a work in progress. It it's, it's really a hard problem to solve, uh, especially when the company is fully remote and, uh, like. Address that's listed on the business records is literally one of the founders condos because you know, the company has no office.So that makes the logistics for a lot of this. Even more fun.[00:35:37] Jeremy: Probably any company that's in an early stage feels the same way where it's like, everything's a work in progress and we're just gonna, we're gonna keep going and we're gonna get there. And as long as everything keeps running, we're good.[00:35:50] Xe: Yeah. I, I don't like thinking about it in that way, because it kind of sounds like pessimistic or defeatist, but at some level it's, it, it really is a work in progress because it's, it's a hard problem and hard problems take a lot of time to solve, especially if you want a solution that you're happy with.[00:36:10] Jeremy: And, and I think it's kind of a unique case too, where it's not like if it goes down, it's like people can't do their job. Right. So it's yeah.[00:36:21] Xe: Actually, if tail scales like control plane goes down, I don't think people would notice until they tried to like boot up a, a reboot, a laptop, or connect a new device to their tailnet. Because once, once all the tailscale agents have all of the information they need from the control plate, you know, they just, they just continue on independently and don't have to care.Derp is also fairly independent of the, like the key dropbox component. And, you know, if that, if that goes down Derp doesn't care at all,[00:37:00] Jeremy: Oh, okay. So if the control plane is down, as long as you had authenticated earlier in the day, you can still, I don't know if it's cached or something, but you can still continue to reach the relay servers, the Derp servers or your, [00:37:15] Xe: other nodes. Yeah. I, I'm pretty sure that in most cases, the control plane could be down for several hours a day and nobody would notice unless they're trying to deal with the admin panel.[00:37:28] Jeremy: Got it. that's a little bit of a relief, I suppose, for, for all of you running it,[00:37:33] Xe: Yeah. Um, it's also kind of hard to sell people on the idea of here is a VPN thing. You don't need to self host it and they're like, what? Why? And yeah, it can be fun.[00:37:49] Jeremy: though, I mean, I feel like anybody who has, self-hosted a VPN, they probably like don't really wanna do it. I don't know. Maybe I'm wrong.[00:38:00] Xe: well, so a lot of the idea of wanting to self host it is, uh, I think it's more of like trying to be self-sufficient and not have to rely on other companies, failures dictating your company's downtime. And, you know, like from some level that's very understandable. And, you know, if, you know, like tail scale were to get bought out and the new owners would, you know, like basically kill the product, they'd still have something that would work for them.I don't know if like such a defeatist attitude is like productive. But it is certainly the opinion that I have received when I have asked people why they wanna self-host. other people, don't want to deal with identity providers or the, like, they wanna just use their, they wanna use their own identity provider.And what was hilarious was there was one, there was one thing where they were like our old VPN server died once and we got locked out of our network. So therefore we wanna, we wanna self-host tailscale in the future so that this won't happen again.And I'm like, buddy, let's, let's just, let's just take a moment and retrace our steps here. CAuse I don't think you mean what you think you mean.[00:39:17] Jeremy: yeah, yeah. [00:39:19] Xe: In general, like I suggest people that, you know, even if they're like way deep into the tailscale, Kool-Aid they still have at least one other method of getting into their servers. Ideally, two. I, I admit that I'm, I come from an SRE style background and I am way more paranoid than most, but it, I usually like having, uh, a backup just in case.[00:39:44] Jeremy: So I, I suppose, on, on that note, let's, let's talk a little bit about your role at tailscale. the title of the archmage of infrastructure is one of the, the coolest titles I've, uh, I've seen. So maybe you can go a little bit into what that entails at, at tailscale.[00:40:02] Xe: I started that title as a joke that kind of stuck, uh, my intent, my initial intent was that every time someone asked, I'd say, I'd have a different, you know, like mystic sounding title, but, uh, archmage of infrastructure kind of stuck. And since then, I've actually been pivoting more into developer relations stuff rather than pure software engineering.And, from the feedback that I've gotten at the various conferences I've spoken at, they like that title, even though it doesn't really fit with developer relations work at all, it it's like it fits because it doesn't. You know, that kind of coney kind of way.[00:40:40] Jeremy: I guess this would go more into the, the infrastructure side, but. What does the, the scale of your infrastructure look like? I mean, I, I think that you touched a little bit on the fact that you have relay servers all over the place and you've got this control plane, but I wonder if you could give people a little bit of perspective of what kind of undertaking this is.[00:41:04] Xe: I am pretty sure at this point we have more developer laptops and the like, than we do production servers. Um, I'm pretty sure that the scale of the production of production servers are in the tens, at most. Um, it turns out that computers are pretty darn and efficient and, uh, you don't really need like a lot of computers to do something amazing.[00:41:27] Jeremy: the part that I guess surprises me a little bit is, is the relay servers, I suppose, because, I would imagine there's a lot of traffic that goes through those. are you finding that just most of the time they just aren't needed and usually you can make a direct connection and that's why you don't need too many of these.[00:41:45] Xe: From what I understand. I don't know if we actually have a way to tell, like what percentage of data is going over the relays versus not. And I think that was an intentional decision, um, that may have been revisited I'm operating based off of like six to 12 month old information right now. But in general, like the only state that the relay servers has is in Ram.And whenever the relay, whenever you disconnect the server, the state is dropped.[00:42:18] Jeremy: Okay.[00:42:19] Xe: and even then that state is like, you know, this key is listening. It is, uh, connected, uh, in case you wanna send packets over here, I guess. it's a bit less bandwidth than you're probably thinking it's not like enough to max it out 24/7, but it is, you know, measurable and there are some, you know, costs associated with it. This is also why it's on digital ocean and vulture and not AWS. but in general, it's a lot less than you'd think. I'm pretty sure that like, if I had to give a baseless assumption, I'd say that probably about like 85% of traffic goes directly.And the remaining is like the few cases in the whole punching engine that we haven't figured out yet. Like Palo Alto fire walls. Oh God. Those things are a nightmare.[00:43:13] Jeremy: I see. So it's most of the traffic actually ends up. Being straight peer to peer. Doesn't have to go through your infrastructure. And, and therefore it's like, you don't need too many machines, uh, to, to make this whole thing work.[00:43:28] Xe: Yeah. it turns out that computers are pretty darn fast and that copying data is something that computers are really good at doing. Um, so if you have, you know, some pretty darn fast computers, basically just sitting there and copying data back and forth all day, like it, you can do a lot with shockingly little.Um, when I first started, I believe that the Derp VMs were using like sometimes as little as one core and 512 megabytes of Ram as like a primary Derp. And, you know, we only noticed when, there were some weird connection issues for people that were only on Derp because there were enough users that the machine had ran out of memory.So we just, you know, upped the, uh, virtual machine size and called it a day. But it's, it's truly remarkable how mu how far you can get with very little[00:44:23] Jeremy: And you mentioned the relay servers, the, the Derp servers were on services like digital ocean and Vultr. I'm assuming because of the, the bandwidth cost, for the control plane, is, is that on AWS or some other big cloud provider?[00:44:39] Xe: it's on AWS. I believe it's in EU central 1.[00:44:44] Jeremy: You're helping people connect from device to device and in a situation like that. what does monitoring look like in, in incidents? Like what are you looking for to determine like, Hey, something's not working.[00:44:59] Xe: there's monitoring with, you know, Prometheus, Grafana, all of that stuff. there are some external probing things. there's also some continuous functional testing for trying to connect to tailscale and like log in as an account. And if that fails like twice in a row, then, you know, something's very wrong and, you know, raise the alarm.But in general. A lot of our monitoring is kind of hard at some level because you know, we're tailscale at a tailscale can't always benefit from tailscale to help operate tail scale because you know, it's tailscale. Um, so it, it still trying to figure out how to detangle the chicken and egg situation.It's really annoying.there's the, the term dog fooding, right? Where they're saying like, oh, we, we run, um, our own development on our own platform or our own software. but I could see when your product is network infrastructure, VPNs, where that could be a little, little dicey.[00:46:06] Xe: Yeah, it is very annoying. But I I'm pretty sure we'll figure something out. It is just a matter of when, another thing that's come up is we've kind of wanted to use tailscale's SSH features, where you specify ACLs in your, you specify ACL rules to allow people to SSH, to other nodes as various users.but if that becomes your main access to production, then you know, like if tailscale is down and you're tailscale, like how do you get in, uh, then there's been various philosophical discussions about this. it's also slightly worse if you use what's called check mode in SSH, where, uh, tail scale, SSH without check mode, you know, you just, it, the, the server checks against the policy rules and the ACL and if it. if it's okay, it lets you in. And if not, it says no, but with check mode, there's also this like eight hour, there's this like eight hour quote unquote lifetime for you to have like sudo mode on GitHub, where you do an auth an auth challenge with your auth aprovider. And then, you know, you're given a, uh, Hey, this person has done this thing type verification.And if that's down and that goes through the control plane, and if the control plane is down and you're tailscale, trying to debug the control plane, and in order to get into the control plane over tailscale, you need to use the, uh, control plane. It, you know, that's like chicken and egg problem level 78,which is a mythical level of chicken egg problem that, uh, has only been foretold in the legends of yore or something.[00:47:52] Jeremy: at that point, it sounds like somebody just needs to, to drive to the data center and plug into the switch.[00:47:59] Xe: I mean, It's not, it's not going to, it probably wouldn't be like, you know, we need to get a person with an angle grinder off of Craigslist type bad. Like it was with the Facebook BGP outage, but it it's definitely a chicken and egg problem in its own right.it makes you do a lot of lateral thinking too, which is also kind of interesting.[00:48:20] Jeremy: When, when you say lateral thinking, I'm just kind of curious, um, if you have an example of what you mean.[00:48:27] Xe: I don't know of any example that isn't NDAed. Um, but basically, you know, tail scale is getting to the, to the point where tailscale is relying on tailscale to make tailscale function and you know, yeah. This is classic oroboros style problem.I've heard a, uh, a wise friend of mine said that that is an ideal problem to have, which sounds weird at face value. But if you're getting to that point, that means that you're successful enough that, you know, you're having that problem, which is in itself a good thing, paradoxically.[00:49:07] Jeremy: better to have that problem than to have nobody care about the product. Right.[00:49:12] Xe: Yeah.[00:49:13] Jeremy: kind of on that, that note, um, you mentioned you worked at, at Salesforce, uh, I believe that was working on Heroku. I wonder if you could talk a little about your experience working at, you know, tailscale, which is kind of more of a, you know, early startup versus, uh, an established company like Salesforce.[00:49:36] Xe: So at the time I was working at Heroku, it definitely didn't feel like I was working at Salesforce for the majority of it. It felt like I was working, you know, at Heroku, like on my resume, I listed as Heroku. When I talked about it to people, I said, I worked at Heroku and that sales force was this, you know, mythical, Ohana thing that I didn't have to deal with unless I absolutely had to.By the end of the time I was working at Heroku, uh, the salesforce, uh, sort of started to creep in and, you know, we moved from tracking issues in GitHub issues. Like we were used to, to using their, oh, what's the polite way to say this, their creation, which is, which was like the moral equivalent of JIRA implemented on top of Salesforce.You had to be behind the VPN for it. And, you know, every ticket had 20 fields and, uh, there were no templates. And in comparison with tail scale, you know, we just use GitHub issues, maybe some like things in notion for doing like longer term tracking or Kanban stuff, but it's nice to not have. you know, all of the pomp and ceremony of filling out 20 fields in a ticket for like two sentences of this thing is obviously wrong and it's causing X to happen.Please fix.[00:51:08] Jeremy: I, I like that, that phrase, the, the creation, that's a very, very diplomatic term.[00:51:14] Xe: I mean, I can think of other ways to describe it, but I'm pretty sure those ways wouldn't be allowed on the podcast. So[00:51:25] Jeremy: Um, but, but yeah, I, I know what you mean for sure where, it, it feels like there's this movement from, Hey, let's just do what we need. Like let's fill in the information that's actually relevant and don't do anything else to a shift to, we need to fill in these 10 fields because that's the thing we do.Yeah.[00:51:48] Xe: Yeah. and in the time I've been working for tail scale, I'm like employee ID 12. And, uh, tail scale has gone from a company where I literally know everyone to just recently to the point where I don't know everyone anymore. And it's a really weird feeling. I've never been in a, like a small stage startup that's gotten to this size before, and I've described some of my feelings to other people who have been there and they're like, yeah, welcome to the club. So I figure a lot of it is normal. from what I understand, though, there's a lot of intentionality to try to prevent tail skill from becoming, you know, like Google style, complexity, organizational complexity, unless that is absolutely necessary to do something.[00:52:36] Jeremy: it's a function of size, right? Like as you have more people, more teams, then more process comes in. that's a really tricky balance to, to grow and still keep that feeling of, I'm just doing the thing, I'm doing the work rather than all this other process stuff.[00:52:57] Xe: Yeah, but it, I've also kind of managed to pigeonhole myself off into a corner with devrel stuff. And that's been nice. I've been working a bunch with, uh, like marketing people and, uh, helping out with support occasionally and doing a, like a godawful amount of writing.[00:53:17] Jeremy: the, the writing, for our audience's benefit, I, I think they should, they should really check out your blog because I think that the way you write your, your articles is very thoughtful in terms of the balance of the actual example code or example scripts and the descriptions and, and some there's a little bit of a narrative sometimes too.So, [00:53:40] Xe: Um, I'm actually more of a prose writer just by like how I naturally write things. And a lot of the style of how I write things is, I will take elements from, uh, the Socratic style of dialogue where, you know, you have the student and the teacher. And, you know, sometimes the student will ask questions that the teacher will answer.And I found that that's a particularly useful way to help model understanding or, you know, like put side concepts off into their own little blurbs or other things like that. I also started doing those conversation things with, uh, furry art, specifically to dunk on a homophobe that was getting very angry at furry art being in, uh, another person's blog.And that's it, it's occasionally fun to go into the, uh, orange website of bad takes and see the comments when people complain about it. oh gosh, the bad takes are hilariously good. Sometimes.[00:54:45] Jeremy: it's good that you have like a, a positive, mindset around that. I know some people can read, uh, that sort of stuff and go, you know, just get really bummed out. [00:54:54] Xe: One of the ways I see it is that a lot of the time algorithms are based on like sheer numbers. So if you like get something that makes people argue in the comments, that number will go up and because there's more comments on it, it makes more people more likely to, to read the article and click on it.So, sometimes I have been known to sprinkle, what's the polite way to say this. I've been known to sprinkle like intentionally kind of things that will, uh, get people and make them want to argue about it in the comments. Purely to make the engagement numbers rise up, which makes more people likely to read the article.And, it's kind of a dirty practice, but you know, it makes more people read the article and more people benefit. So, you know, like it's kind of morally neutral, I guess.[00:55:52] Jeremy: usually that, that seems like, a sketchy thing. But I feel like if it's in service to, uh, like a technical blog post, I mean, why not? Right.[00:56:04] Xe: And a lot of the times I'll usually have the like, uh, kind of bad take, be in a little conversation blurb thing so that people will additionally argue about the characterization of, you know, the imaginary cartoon shark or whatever.[00:56:20] Jeremy: That's good. It's the, uh, it's the Xe Xe universe that they're, they're stepping into.[00:56:27] Xe: I've heard people describe it, uh, lovingly as the xeiaso.net cinematic universe.I've had some ideas on how to expand it in the future with more characters that have more different kind of diverse backgrounds. But, uh, it turns out that writing this stuff is hard. Like actually very hard because you have to get this right.You have to get the right balance of like snark satire, uh, like enlightenment. Andit's, it's surprisingly harder than you'd think. Um, but after a while, I've just sort of managed to like figure out as I'm writing where the side tangents come off and which ones I should keep and which ones I should, uh, prune and which ones can also help, Gain deeper understanding with a little like Socratic dialogue to start with a Mo like an incomplete assumption, like an incomplete picture.And then, you know, a question of, wait, what about this thing? Doesn't that conflict with that? And like, well, yes. technically it does, but realistically we don't have to worry about that as much. So we can think about it just in terms of this bigger model and, uh, that's okay. Like, uh, I mentioned the OSI model earlier, you know, like the seven layer OSI model it's, you know, genuinely overkill for basically everything, except it's a really great conceptual model for figuring out the difference between, you know, like an ethernet cable, an ethernet, like the ethernet card, the IP stack TCP and, you know, TLS or whatever.I have a couple talks that are gonna be up by the time this is published. Uh, one of them is my, uh, rustconf talk on my, or what was it called? I think it was called the surreal horrors of PAM or something where I discussed my experience, trying to bug a PAM module in rust, uh, for work. And, uh, it's the kind of story where, you know, it's bad when you have a break point on dlopen.[00:58:31] Jeremy: That sounds like a nightmare.[00:58:32] Xe: Oh yeah. Like part of the attempting to fix that process involved, going very deep. We're talking like an HTML frame set in the internet archive for sunOS documentation that was written around the time that PAM was used. Like it's things that are bad enough were like everything in the frame set, but the contents had eroded away through bit rot and you know, you're very lucky just to have what you do.[00:59:02] Jeremy: well, I'm, I'm glad it was. It was you and not me. we'll get to, to hear about it and, and not have to go through the, the suffering ourselves.[00:59:11] Xe: yeah. One of the things I've been telling people is that I'm not like a brilliant programmer. Like I know a bunch of people who are definitely way smarter than me, but what I am is determined and, uh, determination is a bit stronger of a force than you'd think.[00:59:27] Jeremy: Yeah. I mean, without it, nothing gets done. Right.[00:59:30] Xe: Yeah.[00:59:31] Jeremy: as we wrap up, is there anything we missed or anything else you wanna mention? [00:59:36] Xe: if you wanna look at my blog, it's on xeiaso.net. That's X, E I a S o.net. Um, that's where I post things. You can see, like the 280 something articles at time of recording. It's probably gonna get to 300 at some point, oh God, it's gonna get to 300 at some point. Um, and yeah, from, I try to post articles about weekly, uh, depending on facts and circumstances, I have a bunch of talks coming up, like one about the hilarious over engineering I did in my blog.And maybe some more. If I get back positive responses from calls for paper submissions,[01:00:21] Jeremy: Very cool. Well, Xe thank you so much for, for coming on software engineering radio.[01:00:27] Xe: Yeah. Thank you for having me. I hope you have a good day and, uh, try out tailscale, uh, note my bias, but I think it's great.

Audio from the July 27th, 2022 installment of “Polygon Alpha” with Travis Cannell - Head of Product at Orchid Protocol.LinkTree - https://linktr.ee/polygonalphapodcastPolygon Alpha Shorts - https://tinyurl.com/PolygonAlphaShortsYouTube - https://www.youtube.com/c/PolygonTVApple - Follow the show on Apple Podcast!Spotify - Follow the show on Spotify!RSS feed - https://api.substack.com/feed/podcast/863588.rssThe Orchid Network - Enables a decentralized virtual private network (VPN), allowing users to buy bandwidth from a decentralized pool of service providers. - Orchid uses an ERC-20 utility token called OXT, a new VPN protocol for token-incentivized bandwidth proxying, and smart-contracts with algorithmic advertising and payment functions. - Orchid's users connect to bandwidth sellers using a provider directory, and they pay using probabilistic nanopayments so Ethereum transaction fees on packets are acceptably low. - Orchid Accounts: Orchid accounts are the decentralized entities that store digital currency on a blockchain to pay for services through nanopayments. The nanopayment smart contract governs Orchid accounts. The Orchid client requires an account in order to pay for VPN service. - The Orchid Client: An open-source, Virtual Private Network (VPN) client that supports decentralized Orchid accounts, as well as WireGuard and OpenVPN connections. The client can string together multiple VPN tunnels in an onion route and can provide local traffic analysis. - The Orchid DApp: The Orchid dApp allows you to create and manage Orchid Accounts. The operations supported by the account manager are simply an interface to the decentralized smart contract that holds the funds and governs how they are added and removed.~~~~Thank you so much for listening & watching the video, if you've not subscribed to the channel please do! We'll continue to bring new videos to you!Polygon offers scalable, affordable, secure and carbon-neutral web3 infrastructure built on Ethereum. Our products offer developers to create user-friendly applications #onPolygon with low transaction fees and without ever sacrificing securityPolygon official channel:Website: polygon.technologyTwitter: twitter.com/0xPolygonTelegram Community: t.me/polygonofficialTelegram announcement: t.me/PolygonAnnouncementsReddit: www.reddit.com/r/0xPolygon/Discord: discord.com/invite/polygonFacebook: www.facebook.com/0xPolygon.Technology/Polygon Alpha Podcast This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit polygonalpha.substack.com

Hoy os detallo un poco más cómo tengo montada mi red local única entre las casas de la familia. Utilizando mis EdgeRouter y el protocolo OpenVPN, hago que esté donde esté siempre me encuentre en red local con mis cacharros, facilitándome mucho las cosas y teniendo gran seguridad.

About SharoneI'm Sharone Zitzman, a marketing technologist and open source community builder, who likes to work with engineering teams that are building products that developers love. Having built both the DevOps Israel and Cloud Native Israel communities from the ground up, today I spend my time finding the places where technology and people intersect and ensuring that this is an excellent experience. You can find my talks, articles, and employment experience at rtfmplease.dev. Find me on Twitter or Github as @shar1z.Links Referenced: Personal Twitter: https://twitter.com/shar1z Website: https://rtfmplease.dev LinkedIn: https://www.linkedin.com/in/sharonez/ @TLVCommunity: https://twitter.com/TLVcommunity @DevOpsDaysTLV: https://twitter.com/devopsdaystlv TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem as their cloud native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, competence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud/C-H-R-O-N-O-S-P-H-E-R-E.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I have been remiss by not having today's guest on years ago because back before I started this ridiculous nonsense that, well, whatever it is you'd call what I do for a living, I did other things instead. I did the DevOps, which means I was sad all the time. And the thing that I enjoyed was the chance to go and speak on conference stages. One of those stages, early on in my speaking career, was at DevOpsDays Tel Aviv.My guest today is Sharone Zitzman, who was an organizer of DevOpsDays Tel Aviv, who started convincing me to come back. And today is in fact, in the strong tradition here of making up your own job titles in ways that make people smile, she is the Chief Manual Reader at RTFM Please Ltd. Sharone, thank you for joining me.Sharone: Thank you for having me, Corey. Israelis love the name of my company, but Americans think it has a lot of moxie and chutzpah. [laugh].Corey: It seems a little direct and aggressive. It's like, oh, good, you are familiar with how this is going to go. There's something to be said for telling people what you do on the tin upfront. I've never been a big fan of trying to hide that. I mean, the first iteration of my company was the Quinn Advisory Group because I thought, you know, let's make it look boring and sedate and like I can talk to finance people. And yeah, that didn't last more than ten seconds of people talking to me.Also, in hindsight, the logo of a big stylized Q. Yeah, I would have had to change that anyway, for the whole QAnon nonsense because I don't want to be mistaken for that particular brand of nuts.Sharone: Yeah, I decided to do away with the whole formalities and upfront, just go straight [laugh]. For the core of who we are, Corey; you are very similar in that. So, yes. Being a dev first company, I thought the developers would appreciate such a title and name for my company. And I have to give a shout out here to Avishai Ish-Shalom, who's my friend from the community who you also know from the DevOpsDays community.Corey: Oh, yeah @nukemberg on Twitter—Sharone: Yes exactly.Corey: For those who are not familiar.Sharone: [laugh]. Yep. He coined the name.Corey: The problem that I found is that people when they start companies or they manage their careers, they don't bias for the things that they're really good at. And it took me a long time to realize this, I finally discovered, “Ah, what am I the best at? That's right, getting myself fired for my personality, so why don't I build a business where that stops being a liability?” So, I started my own company. And I can tell this heroic retcon of what happened, but no, it's because I had nowhere else to go at that point.And would you hire me? Think about this for a minute. You, on the other hand, had options. You are someone with a storied history in community building, in marketing to developers without that either coming across as insincere or that marked condescending accent that so many companies love to have of, “Oh, you're a developer. Let me look at you and get down on my hands and knees like we're going camping and tell a story in ways that actively and passively insult you.”No, you have always gotten that pitch-perfect. The world was your oyster. And for some godforsaken reason, you looked around and decided, “Ah, I'm going to go out independently because you know what I love? Worrying.” Because let's face it, running your own company is an exercise in finding new and exciting things to worry about that 20 minutes ago, you didn't know existed. I say this from my own personal experience. Why would you ever do such a thing?Sharone: [laugh]. That's a great question. It was a long one, but a good one. And I do a thing where I hit the mic a lot because I also have. I can't control my hand motions.Corey: I too speak with my hands. It's fine.Sharone: [laugh]. Yeah, so it's interesting because I wanted to be independent for a really long time. And I wasn't sure, you know, if it was something that I could do if I was a responsible enough adult to even run my own company, if I could make it work, if I could find the business, et cetera. And I left the job in December 2020, and it was the first time that I hadn't figured out what I was doing next yet. And I wanted to take some time off.And then immediately, like, maybe a week after I started to get a lot of, like, kind of people reaching out. And I started to interview places and I started to look into possibly being a co-founder at places and I started to look at all these different options. And then just, I was like, “Well…. This is an opportunity, right? Maybe I should finally—that thing that's gnawing at the back of my head to see if, like, you know if I should go for this dream that I've always wanted, maybe now I can just POC it and see if, you know, it'll work.”And it just, like, kind of exploded on me. It was like there was so much demand, like, I just put a little, like, signal out to the world that this is something that I'm interested in doing, and everyone was like, “Ahh, I need that.” [laugh]. I wanted to take a quarter off and I signed my first clients already on February 1st, which was, like, a month after. I left in December and that—it was crazy. And since then, I've been in business. So, yeah. So, and since then, it's also been a really crazy ride; I got to discover some really exciting companies. So.Corey: How did you get into this? I found myself doing marketing-adjacent work almost entirely by accident. I started the newsletter and this podcast, and I was talking to sponsors periodically and they'd come back with, “Here's the thing we want you to talk about in the sponsor read.” And it's, “Okay, you want to give people a URL to go to that has four sub-directories and entire UTM code… okay, have you considered, I don't know, not?” And because so much of what they were talking about did not resonate.Because I have the engineering background, and it was, I don't understand what your company does and you're spending all your time talking about you instead of my painful problem. Because as your target market, I don't give the slightest of shits about you, I care about my problem, so tell me how you're going to solve my problem and suddenly I'm all ears. Spend the whole time talking about you, and I could not possibly care less and I'll fast-forward through the nonsense. That was my path to it. How did you get into it?Sharone: How did I get into it? It's interesting. So, I started my journey in typical marketing, enterprise B2B marketing. And then at GigaSpaces, we kickstarted the open-source project Cloudify, and that's when I found myself leading this project as the open-source community team leader, building, kind of, the community from the ground floor. And I discovered a whole new world of, like, how to build experience into your marketing, kind of making it really experiential and making sure that everyone has a really, really easy and frictionless way of using your product, and that the product—putting the product at the center and letting it speak for itself. And then you discover this whole new world of marketing where it's—and today, you know, it has more of a name and a title, PLG, and people—it has a whole methodology and practice, but then it was like we were—Corey: PLG? I'm unfamiliar with the acronym. I thought tech was bad for acronyms.Sharone: Right? [laugh]. So, product-led growth. But then, you know, like, kind of wasn't solidified yet. And so, a lot of what we were doing was making sure that developers had a really great experience with the product then it kind of sold itself and marketed itself.And then you understood what they wanted to hear and how they wanted to consume the product and how they wanted it to be and to learn about it and to kind of educate themselves and get into it. And so, a lot of the things that I learned in the context of marketing was very guerilla, right, from the ground up and kind of getting in front of people and in the way they wanted to consume it. And that taught me a lot about how developers consume technology, the different channels that they're involved in, and the different tools that they need in order to succeed, and the different, you know, all the peripheral experience, that makes marketing really, really great. And it's not about what you're selling to somebody; it's making your product shine and making the experience shine, making them ensure that it's a really, really easy and frictionless experience. You know, I like how [Donald Bacon 00:08:00] says it; he calls it, like, mean time to hello world, and that to me is the best kind of marketing, right? When you enable people to succeed very, very quickly.Corey: Yeah, there's something to be said for the ring of authenticity and the rest. Periodically I'll promote guest episodes on this, where it's a sponsored episode where people get up and they talk about what they're working on. And they're like, “Great. So, here's the sales pitch I want to give,” and it's no you won't because first, it won't work. And secondly, I'm sorry, whether it's a promoted episode or not, I will not publish something that isn't good because I have a reputation to uphold here.And people run into challenges an awful lot when they're trying to effectively tell their story. If you have a startup that was founded by an engineer, for example, as so many of these technical startups were, the engineer is often so deeply and profoundly in love with this problem space and the solution and the rest, but if they talk about that, no one cares about the how. I mean, I fix AWS bills, and people don't care—as a general rule—how I do that at all if they're in my target market. They don't care if it's through clever optimization, amazing tooling, doing it on-site, or taking hostages in Seattle. They care about their outcome much more than they ever do about the how.The only people who care about the how are engineers who very often are going to want to build it themselves, or work for you, or start a competitor. And it doesn't resonate in quite the same way. It's weird because all these companies are in slightly different spaces; all of them tend to do slightly different things—or very different things—but so many of the challenges that I see in the way that they're articulating what they do to customers rhymes with one another.Sharone: Yeah. So, I agree completely that developers will talk often about how it works. How it works. How does it work under the hood? What are the bits and bytes, you know?Like, nobody cares about how it works. People care about how will this make my life better, right? How will this improve my life? How will this change my life? [laugh]. As an operations engineer, if I'm, you know, crunching through logs, how will this tool change that? What my days look like? What will my on-call rotation look like? What will—you know, how are you changing my life for the better?So, I think that that's the question. When you learn how to crystallize the answer to that question and you hit it right on the mark—you know, and it takes a long time to understand the market, and to understand the buying persona, and t—and there's so much that you have to do in the background, and so much research you have to do to understand who is that person that needs to have that question answered? But once you do and you crystallize that answer, it lands. And that's the fun part about marketing, really trying to understand the person who's going to consume your product and how you can help them understand that you will make their life better.Corey: Back when I was starting out as a consultant myself, I would tell stories that I had seen in the AWS billing environment, and I occasionally had clients reach out to me, “Hey, why don't you tell our story in public?” It's, “Because that wasn't your story. That was something I saw on six different accounts in the same month. It is something that everyone is feeling.” It's, people think that you're talking about them.So, with that particular mindset on this, without naming specific companies, what themes are you seeing emerging? What are companies getting wrong when they are attempting and failing to market effectively to developers?Sharone: So, exactly what we're talking about in terms of the product pitch, in that they're talking at developers from this kind of marketing speak and this business language that, you know, developers often—you know, unless a company does a really, really good job of translating, kind of, the business value—which they should do, by the way—to engineers, but oftentimes, it's a little bit far from them in the chain, and so it's very hard for them to understand the business fluff. If you talk to them in bits and bytes of this is what my day-to-day developer workflow looks like and if we do these things, it'll cut down the time that I'm working on these things, it'll make these things easier, it'll help streamline whatever processes that are difficult, remove these bottlenecks, and help them understand, like I said, how it improves their life.But the things that I've seen breakdown is also in the authenticity, right? So obviously, the world is built on a lot of the same gimmicks and it's just a matter of whether you're doing it right or not, right? So, there's so much content out there and webcasts and webinars, and I don't know what and podcasts and whatever it is, but a lot of the time, people, their most valuable asset is their time. And if you end up wasting their time, without it being, like, really deeply valuable—if you're going to write content, make sure that there is a valuable takeaway; if you're going to create a webinar, make sure that somebody learned something. That if they're investing their time to join your marketing activities, make sure that they come away with something meaningful and then they'll really appreciate you.And it's the same idea behind the whole DevOpsDays movement with the law of mobility and open spaces that people if they find value, they'll join this open space and they'll participate meaningfully and they'll be a part of your event, and they'll come back to your event from year to year. But if you're not going to provide that tangible value that somebody takes away, and it's like, okay, well, I can practically apply this in my specific tech stack without using your tool, without having to have this very deterministic or specific kind of tech stack that they're talking about. You want to give people something—or even if it is, but even how to do it with or without, or giving them, like, kind of practical tools to try it. Or if there's an open-source project that they can check out first, or some kind of lean utility that gives them a good indication of the value that this will give them, that's a lot more valuable, I think. And practically understandable to somebody who wants to eventually consume your product or use your products.Corey: The way that I see things, at least in the past couple of years, the pandemic has sharpened an awful lot of the messaging that needs to happen. Because in most environments, you're sitting at a DevOpsDays in the front row or whatnot, and it's time for the sponsor talks and someone gets up and starts babbling and wasting your time, most people are not going to get up and leave. Okay, they will in Israel, but in most places, they're not going to get up and leave, whereas in pandemic land, it's you are one tab away from something I actually want—Sharone: Exactly.Corey: To be doing, so if you become even slightly boring, it's not going to go well. So, you have to be on message, you have to be on point or no one cares. People are like, “Oh, well what if we say the wrong thing and people wind up yelling about us on Twitter?” It's like unless it is for something horrifying, you should be so lucky because people are then talking about you. The failure mode isn't that people don't like your product, it's no one talks about it.Sharone: Yeah. No such thing as bad publicity [crosstalk 00:14:32] [laugh]—Corey: Oh, there very much is such a thing is bad publicity. Like, “I could be tweeting about your product most days,” is apparently a version of that, according to some folks. But it's a hard problem to solve for. And one of the things that continually surprises me is the things I'm still learning about this entire industry. The reason that people sponsor this show—and the rates they pay, to be direct—have little bearing to the actual size of the audience—as best we can tell; lies, damn lies, and podcast statistics; if you're listening to this, let me know. I'd love to know if anyone listens to this nonsense—but when you see all of that coming out, why are we able to charge the rates that we do?It's because the long-term value of someone who is going to buy a long-term subscription or wind up rolling out something like ChaosSearch or whatnot that is going to be a fundamental tenet of their product, one prospect becoming a customer pays for anything, I can sell a company, it will sponsor—they can pay me to sponsor for the next ten years, as opposed to the typical mass-market audience where well, I'm here to sling Casper mattresses today or something. It's a different audience and there's a different perception there. People are starting to figure out the value of—in an age where tracking is getting harder and harder to do and attribution will drive you nuts, instead of go where your audience is. Go where the people who care about the problem that you have and will experience that problem are going to hang out. And it always is wild to me to see companies missing out on that.It's, “Okay, so you're going to do a $25 million billboard ad in spotted in airports around the world talking about your company… but looking at your billboard, it makes no sense. I don't understand what it's there for.” Even as a brand awareness play, it fails because your logo is tiny in the corner or something. It's you spent that much money on ads, and maybe a buck on messaging because it seems like with all that attention you just bought, you had nothing worthwhile to say. That's the cardinal sin to me at least.Sharone: Yeah. One thing that I found—and back to our community circuit and things that we've done historically—but that's one thing that, you know, as a person comes from community, I've seen so much value, even from the smaller events. I mean, today, like with Covid and the pandemic and everything has changed all the equilibrium and the way things are happening. But some meetups are getting smaller, face-to-face events are getting smaller, but I've had people telling me that even from small, 30 to 40 people events, they'll go up and they'll do a talk and great, okay, a talk; everybody does talks, but it's like, kind of, the hallway track or the networking that you do after the talk and you actually talk to real users and hear their real problems and you tap into the real community. And some people will tell me like, I had four concrete leads from a 30-person meet up just because they didn't even know that this was a real challenge, or they didn't know that there was a tool that solves this problem, or they didn't understand that this can actually be achieved today.Or there's so many interesting technologies and emerging technologies. I'm privileged to be able to be at the forefront of that and discover it all, and I if I could, I would drop names of all of the awesome companies that work for me, that I work with, and just give them a shout out. But really, there's so many amazing companies doing, like, developer metrics, and all kinds of troubleshooting and failure analysis that's, like, deeply intelligent—and you're going to love this one: I have a Git replacement client apropos to your closing keynote of DevOpsDays 2015—and tapping into the communities and tapping into the real users.And sometimes, you know, it's just a matter of really understanding how developers are working, what processes look like, what workflows look like, what teams look like, and being able to architect your products and things around real use cases. And that you can only discover by really getting in front of actual users, or potential users, and learning from them and feedback loops, and that's the little core behind DevRel and developer advocacy is really understanding your actual users and your consumers, and encouraging them to you know, give you feedback and try things, and beta programs and a million things that are a lot more experiential today that help you understand what your users need, eventually, and how to actually architect that into your products. And that's the important part in terms of marketing. And it's a whole different marketing set. It's a whole different skill set. It's not talking at people, it's actually… ingesting and understanding and hearing and implementing and bringing it into your products.Corey: And it takes time. And you have to make yourself synonymous with a painful problem. And those problems are invariably very point-in-time specific. I don't give a crap about log aggregation today, but in two weeks from now, when I'm trying to chase down 18 different Lambdas function trying to figure out what the hell's broken this week, I suddenly will care very much about log aggregation. Who was that company that's in that space that's doing interesting things? And maybe it's Cribl, for example; they do a lot of stuff in that space and they've been a good sponsor. Great.I start thinking about those things in that light because it is—when I started having these problems, it sticks in your head and it resonates. And there's value and validity to that, but you're never going to be able to attribute that either, which is where people often lose their minds. Because for anything even slightly complicated—you're going to be selling things to big bank—great, good on you. Most of those customers are not going to go and spin up a trial in the dead of night. They're going to hear about you somewhere and think, “Ohh, this is interesting.”They're going to talk about a meeting, they're going to get approval, and at that point, you have long since lost any tracking opportunity there. So, the problem is that by saying it like this, as someone who is a publisher, let's be very clear here, it sounds like you're trying to justify your entire business model. I feel like that half the time, but I've been reassured by people who are experts in doing these things, like, oh, yeah, we have data on this; it's working. So, the alternative is either I accept that they're right or I sit here and arrogantly presume I know more about marketing than people who've devoted their entire careers to it. I'm not that bold. I am a white guy in tech, but not that much.Sharone: Yeah, I mean, the DevRel measurement problem is a known problem. We have people like [unintelligible 00:20:21] who have written about it. We have [Sarah Drasner 00:20:23], we have a million people that have written really, really great content about how do you really measure DevRel and the quality. And one of the things that I liked, Philipp Krenn, the dev advocate at Elastic once said in one of his talks that, you know, “If you're measuring your developer advocates on leads, you're a marketing organization. If you're measuring them on revenue, you're a sales organization. It's about reach, engagement, and awareness, and a lot of things that it's much, much harder to measure.”And I can say that, like, once upon a time, I used to try and attribute it at Cloudify. Like, I remember thinking, like, “Okay, maybe I could really track this back to, you know, the first touch that I actually had with this user.” It's really, really difficult, but I do remember, like, when we used to go out into the events and we were really active in the OpenStack community, in the DevOps community, and many other things, and I remember, like, even after events, like, you get all those lead gen emails. All I would say now is, like, “Hey, if you missed us at the booth, you know, and you want still want a t-shirt, you know, reach out and I'll ship it to you.” And some of those eventually, after we continued the relationship, and we, you know, when we were friends and community friends, six months later, when they moved to their next role at their next job, they were like, “Oh, now I have an opportunity to use Cloudify and I'm going to check it out.”And it's very long relationship that you have to cultivate. It has to be, you know, mutual. You have to be, you have to give be giving something and eventually is going to come back to you. Good deeds come back to you. So, I—that's my credo, by the way, good deeds come back to you. I believe in that and I try to live by that.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: So, I have one last question for you and it is pointed and the reason I buried it this deep in the episode is so that if I open with it, I will get letters and I'm hoping to get fewer of them. But I met you, again, at DevOpsDays Tel Aviv, and it was glorious. And then you said, “This is fun. Come help me organize it next year.”And I, like an idiot said, “Sure, that sounds awesome because I love going to conferences and it's great. So, what's involved?” “Oh, a whole bunch of meetings.” “Okay, great.” “And planning”—things I'm terrible at—“Okay.” And then the big day finally arrives where, “Great, when do we get to get on stage and tell a story?” Like, “That's the neat part. We don't.” So, I have to ask, given that it is all behind-the-scenes work that is fairly thankless unless you really screw it up because then it's very visible, what is the point of being so involved in the community?Sharone: Wow, that's a big question, Corey.Corey: It really is.Sharone: [laugh].Corey: Because you've been involved in community for a long time and you're very good at it.Sharone: It's true. It's true. Appreciate it, thank you. So, for me, first of all, I enjoy, kind of, the people aspect of it, absolutely. And that people aspect of it actually has played out in so many different ways.Corey: Oh, you mean great people, and also me.Sharone: [laugh]. Particularly you, Corey, and we will bring you back. [laugh]. And we will make sure you chop wood and carry water because eventually it'll fill your soul, you'll see. [laugh] one of the things that really I have had the privilege and honor, and having come out of, like, kind of all my community work is really the network I've built and the people that I've met.And I've learned so much and I've grown so much, but I've also had the opportunity to connect people, connect things that you wouldn't imagine, un—seemingly-related things. So, there are so many friends of mine that have grown up with me in this community, it's been already ten years now, and a lot of folks have now been going on to new adventures and are looking to kickstart their new startup and I can connect them to this investor, I can connect them to this other person who is maybe a good, you know, partner for their startup, and hiring opportunities, and something—I've had this, like, privilege of kind of being able to connect Israel to the outer world and other things and the global kind of community, and also bring really intelligent folks into the community. And this has just created this amazing flywheel of opportunity that I'm really happy to be at the center of. And I think I've grown as a person, I think our community has grown, has learned, and there's a lot of value in that, I think, yeah. We got to meet wonderful folks like you, Corey. [laugh].Corey: It has its moments. Again, you're one of those rarities in that it's almost become a trope in VC land where VCs always like, “How may I be useful?” And it's this self-serving transparent thing. Every single time you have deigned to introduce me to someone, it's been a productive conversation and I'm always glad I took the meeting. That is no small thing.A lot of people say, “I'm good at community,” which is sort of cover for, “I'm not good at anything,” but in your case, it—Sharone: [laugh]. [I'm an entrepreneur 00:24:48].—Corey: Is very much not true. Oh, yeah. I'm a big believer that ‘entrepreneur' and ‘hero' and other terms like that are things people call you; you don't call yourself that. It always feels weird for, “Oh, he's an entrepreneur.” It's like, that's a pretty lofty word for shitposting, but okay, we'll roll with it.It doesn't work that way. You've clearly invested long-term in a building reputation for yourself by building a name for yourself in the space, and I know that whenever you reach out to me as a result, you are not there to waste my time or shill some bullshit. It is always something that is going to, even if I don't love every aspect of it or agree with the core of the message you're sending, great, it is never not going to be worth my time, which is why I'm so glad I got the chance to talk to you this show.Sharone: I appreciate that. It's something that I really believe in, I don't want to waste people's time and I really only will connect folks or only really will reach out to someone if I do think that there's something meaningful for both sides. It's never only what's in it for me, also. I also want to make sure that there's something in it for the other person and it's something that makes sense and it's meaningful for both sides. I've had the opportunity of meeting such interesting folks, and sometimes it's just like, “You must meet. [laugh]. You will love each other.” You will have so much to do together or it's so much collaboration opportunity.And so yeah, I really am that type of person. And I'll even say from a personal perspective, you know, I know a lot of people, and I've even been asked from the flip side, “Okay, is this a toxic manager? Or is this a, you know, a good hire? Is this”—and I tried to provide really authentic input so people make the right decisions, or make, you know, the right contacts, or make—and that's something I really value. And I managed to build trust with a lot of really great folks—Corey: And also me—Sharone: —and it's come back to me, also. And—[laugh] and particularly you, again. [laugh].Corey: If people want to learn more about how you see the world and the space and otherwise bask in your wisdom, where's the best place to find you?Sharone: So, I'm on Twitter as @shar1z, which is SharoneZ. Basically, everyone thinks it's such a smart, or I don't know what, like, or an esoteric screen name. And I'm like, no, it's just my name, I just—the O-N-E is… the one. [laugh].So yes, shar1z on Twitter, but also my website, rtfmplease.dev, you can reach out, there's a contact form there. You can find me on the web anywhere—LinkedIn. Reach out, I answer almost all my DMs when I can. It's very rare that I don't answer DMs. Maybe there'll be a slight lag, but I do. And I really do like when folks reach out to me. I do like it when people try and make contact.Corey: And you can also be found, of course, wherever find DevOps products are sold, on stage apparently.Sharone: [laugh]. The DevOps community, that's right. @TLVCommunity, @DevOpsDaysTLV—don't out me. All those are—yes, those are also handles that I run on Twitter, it's true.Corey: Excellent.Sharone: So, when you see them all retweeting the same tweet, yes, it's happening within same five minutes, it's me.Corey: Oh, that would have made it way easier to go viral. My God, I should have just thought of that earlier.Sharone: [laugh].Corey: Thank you so much for your time. I appreciate it.Sharone: Thank you, Corey, for having me. It's been a privilege and honor being on your show and I really do think that you are doing wonderful things in the cloud space. You're teaching us, and we're all learning, and you—keep up the good work.Corey: Well, thank you. I appreciate that.Sharone: I also want to add that on proposed marketing and whatever, I do actually listen to all of your openings of all of your shows because they're not fluffy and I like that you do, like, kind of a deep explanation, a deep technical explanation of what your sponsoring product does, and it gives a lot more insight into why is this important. So, I think you're doing that right. So, anybody who's sponsoring this show, listen. Corey knows what he's doing.Corey: Well, thank you. I appreciate that. Yay, “I know what I'm doing.” That one's going in the testimonial kit. My God.Sharone: [laugh]. That's the name of this episode, “Corey knows what he's doing.”Corey: We're going to roll with it, you know. No take-backsies. Sharone Zitzman, Chief Manual Reader at RTFM Please. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review of your podcast platform of choice, or if it's on the YouTubes smash the like and subscribe buttons, whereas if you've hated this show, exact same thing—five-star review wherever you happen to find it, smash both the buttons—but also leave an insulting comment telling me that I'm completely wrong which then devolves into an 18-page diatribe about exactly how your nonsense, bullshit product is built and works.Sharone: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About RafalRafal is Serverless Engineer at Stedi by day, and Dynobase founder by night - a modern DynamoDB UI client. When he is not coding or answering support tickets, he loves climbing and tasting whiskey (not simultaneously).Links Referenced:Company Website: https://dynobase.dev TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's not too often that I wind up building an episode here out of a desktop application. I've done it once or twice, and I'm sure that the folks at Microsoft Excel are continually hoping for an invite to talk about things. But we're going in a bit of a different direction today. Rafal Wilinski is a serverless engineer at Stedi and, in apparently what is the job requirement at Stedi, he also has a side project that manifests itself as a desktop app. Rafal, thank you for joining me today. I appreciate it.Rafal: Yeah. Hi, everyone. Thanks for having me, Corey.Corey: I first heard about you when you launched Dynobase, which is awesome. It sounds evocative of dinosaurs unless you read it, then it's D-Y-N-O, and it's, “Ah, this sounds a lot like DynamoDB. Let me see what it is.” And sure enough, it was. As much as I love misusing things as databases, DynamoDB is actually a database that is decent and good at what it does.And please correct me if I get any of this wrong, but Dynobase is effectively an Electron app that you install, at least on a Mac, in my case; I don't generally use other desktops, that's other people's problems. And it provides a user-friendly interface to DynamoDB that is not actively hostile to the customer.Rafal: Yeah, exactly. That was the goal. That's how I envisioned it, and I hope I executed correctly.Corey: It was almost prescient in some ways because they recently redid the DynamoDB console in AWS to actively make it worse, to wind up working with individual items, to modify things. It feels like they are validating your market for you by, “Oh, we really like Dynobase. How do we drive more traffic to it? We're going to make this thing worse.” But back then when you first created this, the console was his previous version. What was it that inspired you to say, “You know what I'm going to build? A desktop application for a cloud service.” Because on the surface, it seems relatively close to psychotic, but it's brilliant.Rafal: [laugh]. Yeah, sure. So, a few years ago, I was freelancing on AWS. I was jumping between clients and my side projects. That also involved jumping between regions, and AWS doesn't have a good out-of-the-box solution for switching your accounts and switching your regions, so when you want it to work on your client table in Australia and simultaneously on my side project in Europe, there was no other solution than to have two browser windows open or to, even, browsers open.And it was super frustrating. So, I was like, hey, “DynamoDB has SDK. Electron is this thing that allows you to make a desktop application using HTML and JS and some CSS, so maybe I can do something with it.” And I was so naive to think that it's going to be a trivial task because it's going to be—come on, it's like, a couple of SDK calls, displaying some lists and tables, and that's pretty much it, right?Corey: Right. I use Retool as my system to build my newsletter every week, and that is the front-end I use to interact with DynamoDB. And it's great. It has a table component that just—I run a query that, believe it or not, is a query, not a scan—I know, imagine that, I did something slightly right this one time—and it populates things for the current issue into it, and then I basically built a CRUD API around it and have components that let me update, delete, remove, the usual stuff. And it's great, it works for my purposes, and it's fine.And that's what I use most of the time until I, you know, hit an edge case or a corner case—because it turns out, surprise everyone, I'm bad at programming—and I need to go in and tweak the table myself manually. And that's where Dynobase, at least for my use case, really comes into its own.Rafal: Good to hear. Good to hear. Yeah, that was exactly same case why I built it because yeah, I was also, a few years ago, I started working on some project which was really crazy. It was before AppSync times. We wanted to have GraphQL serverless API using single table design and testing principles [unintelligible 00:04:38] there.So, we've been verifying many things by just looking at the contents of the table, and sometimes fixing them manually. So, that was also the thing that motivated me to make the editing experience a little bit better.Corey: One thing I appreciate about the application is that it does things right. I mean, there's no real other way to frame that. When I fire up the application myself and I go to the account that I've been using it with—because in this case, there's really only one account that I have that contains the data that I spent that my time working with—and I get access to it on my machine via Granted, which because it's a federated SSO login. And it says, “Ah, this is an SSL account. Click here to open the browser tab and do the thing.”I didn't have to configure Dynobase. It is automatically reading my AWS config file in my user directory. It does a lot of things right. There's no duplication of work. From my perspective. It doesn't freak out because it doesn't know how SSO works. It doesn't have run into these obnoxious edge case problems that so many early generation desktop interfaces for AWS things seem to.Rafal: Wow, it seems like it works for you even better than for me. [laugh].Corey: Oh, well again, how I get into accounts has always been a little weird. I've ranted before about Granted, which is something that Common Fate puts out. It is a binary utility that winds up logging into different federated SSO accounts, opens them in Firefox containers so you could have you know, two accounts open, side-by-side. It's some nice affordances like that. But it still uses the standard AWS profile syntax which Dynobase does as well.There are a bunch of different ways I've logged into things, and I've never experienced friction [unintelligible 00:06:23] using Dynobase for this. To be clear, you haven't paid me a dime. In fact, just the opposite. I wind up paying my monthly Dynobase subscription with a smile on my face. It is worth every penny, just because on those rare moments when I have to work with something odd in DynamoDB, it's great having the tool.I want to be very clear here. I don't recall what the current cost on this is, but I know for a fact it is more than I spend every month on DynamoDB itself, which is fine. You pay for utility, not for the actual raw cost of the underlying resources on it. Some people tend to have issues with that and I think it's the wrong direction to go in.Rafal: Yeah, exactly. So, my logic was that it's a productivity improvement. And a lot of programmers are simply obsessed with productivity, right? We tend to write those obnoxious nasty Bash and Python scripts to automate boring tasks in our day jobs. So, if you can eliminate this chore of logging to different AWS accounts and trying to find them, and even if it takes, like, five or ten seconds, if I can shave that five or ten seconds every time you try to do something, that over time accumulates into a big number and it's a huge time investment. So, even if you save, like, I don't know, maybe one hour a month or one hour a quarter, I think it's still a fair price.Corey: Your pricing is very interesting, and the reason I say that is you do not have a free tier as such, you have a free seven-day trial, which is great. That is the way to do it. You can sign up with no credit card, grab the thing, and it's awesome. Dynobase.dev for folks who are wondering.And you have a solo yearly plan, which is what I'm on, which is $9 a month. Which means that you end up, I think, charging me $108 a year billed annually. You have a solo lifetime option for 200 bucks—and I'm going to fight with you about that one in a second; we're going to come back to it—then you have a team plan that is for I think for ten licenses at 79 bucks a month, and for 20 licenses it's 150 bucks a month. Great. And then you have an enterprise option for 250 a month, the end. Billed annually. And I have problems with that, too.So, I like arguing with pricing, I [unintelligible 00:08:43] about pricing with people just because I find that is one of those underappreciated aspects of things. Let's start with my own decisions on this, if I may. The reason that I go for the solo yearly plan instead of a lifetime subscription of I buy this and I get to use it forever in perpetuity. I like the tool but, like, the AWS service that underlies it, it's going to have to evolve in the fullness of time. It is going to have to continue to support new DynamoDB functionality, like the fact that they have infrequent access storage classes now, for tables, as an example. I'm sure they're coming up with other things as well, like, I don't know, maybe a sane query syntax someday. That might be nice if they ever built one of those.Some people don't like the idea of a subscription software. I do just because I like the fact that it is a continual source of revenue. It's not the, “Well, five years ago, you paid me that one-off thing and now you expect feature enhancements for the rest of time.” How do you think about that?Rafal: So, there are a couple of things here. First thing is that the lifetime support, it doesn't mean that I will be always implementing to my death all the features that are going to appear in DynamoDB. Maybe there is going to be a some feature and I'm not going to implement it. For instance, it's not possible to create the global tables via Dynobase right now, and it won't be possible because we think that majority of people dealing with cloud are using infrastructure as a code, and creating tables via Dynobase is not a super useful feature. And we also believe that it's not going to break even without support. [laugh]. I know it sounds bad; it sounds like I'm not going to support it at some point, but don't worry, there are no plans to discontinue support [crosstalk 00:10:28]—Corey: We all get hit by buses from time to time, let's be clear.Rafal: [laugh].Corey: And I want to also point out as well that this is a graphical tool that is a front-end for an underlying AWS service. It is extremely convenient, there is tremendous value in it, but it is not critical path as if suddenly I cannot use Dynobase, my production app is down. It doesn't work that way, in the sense—Rafal: Yes.Corey: Of a SaaS product. It is a desktop application. And huge fan of that as well. So, please continue.Rafal: Yeah, exactly—Corey: I just want to make sure that I'm not misleading people into thinking it's something it's not here. It's, “Oh, that sounds dangerous if that's critical pa”—yeah, it's not designed to be. I imagine, at least. If so it seems like a very strange use case.Rafal: Yeah. Also, you have to keep in mind that AWS isn't basically introducing breaking changes, especially in a service that is so popular as DynamoDB. I cannot imagine them, like, announcing, like, “Hey, in a month, we are going to deprecate this API, so you'd better start, you know, using this new API because this one is going to be removed.” I think that's not going to happen because of the millions of clients using DynamoDB actively. So, I think that makes Dynobase safe. It's built on a rock-solid foundation that is going to change only additively. No features are going to be just being removed.Corey: I think that there's a direction in a number of at least consumer offerings where people are upset at the idea of software subscriptions, the idea of why should I pay in perpetuity for a thing? And I want to call out my own bias here. For something like this, where you're charging $9 a month, I do not care about the price, truly I don't. I am a price inflexible customer. It could go and probably as high as 50 bucks a month and I would neither notice nor care.That is probably not the common case customer, and it's certainly not over in consumer-land. I understand that I am significantly in a privileged position when it comes to being able to acquire the tools that I need. It turns out compared to the AWS bill I have to deal with, I don't have to worry about the small stuff, comparatively. Not everyone is in that position, so I am very sympathetic to that. Which is why I want to deviate here a little bit because somewhat recently, Dynobase showed up on the AWS Marketplace.And I can go into the Marketplace now and get a yearly subscription for a single seat for $129. It is slightly more than buying it directly through your website, but there are some advantages for many folks in getting it on the Marketplace. AWS is an approved vendor, for example, so there's no procurement dance. It counts toward your committed spend on contracts if someone is trying to wind up hitting certain levels of spend on their EDP. It provides a centralized place to manage things, as far as those licenses go when people are purchasing it. What was it that made you decide to put this on the Marketplace?Rafal: So, this decision was pretty straightforward. It's just, you know, yet another distribution channel for us. So, imagine you're a software engineer that works for a really, really big company and it's super hard to approve some kind of expense using traditional credit card. You basically cannot go to my site and check out with a company credit card because of the processes, or maybe it takes two years. But maybe it's super easy to click this subscribe on your AWS account. So yeah, we thought that, hey, maybe it's going to unlock some engineers working at those big corporations, and maybe this is the way that they are going to start using Dynobase.Corey: Are you seeing significant adoption yet? Or is it more or less a—it's something that's still too early to say? And beyond that, are you finding that people are discovering the product via the AWS Marketplace, or is it strictly just a means of purchasing it?Rafal: So, when it comes to discovering, I think we don't have any data about it yet, which is supported by the fact that we also have zero subscriptions from the Marketplace yet. But it's also our fault because we haven't actually actively promoted the fact, apart from me sending just a tweet on Twitter, which is in [crosstalk 00:14:51]—Corey: Which did not include a link to it as well, which means that Google was our friend for this because let's face it, AWS Marketplace search is bad.Rafal: Well, maybe. I didn't know. [laugh]. I was just, you know, super relieved to see—Corey: No, I—you don't need to agree with that statement. I'm stating it as a fact. I am not a fan of Marketplace search. It irks me because for whatever reason whenever I'm in there looking for something, it does not show me the things I'm looking for, it shows me the biggest partners first that AWS has and it seems like the incentives are misaligned. I'm sure someone is going to come on the show to yell about me. I'm waiting for your call.Rafal: [laugh].Corey: Do you find that if someone is going to purchase it, do you have a preference that they go directly, that they go through the Marketplace? Is there any direction for you that makes more sense than another?Rafal: So ideally, would like to continue all the customers to purchase the software using the classical way, using the subscriptions for our website because it's just one flow, one system, it's simpler, it's cleaner, but we want it to give that option and to have more adoption. We'll see if that's going to work.Corey: I was going to say there were two issues I had with the pricing. That was one of them. The other is at the high end, the enterprise pricing being $250 a month for unlimited licenses, that doesn't feel like it is the right direction, and the reason I say that is a 50-person company would wind up being able to spend 250 bucks a month to get this for their entire team, and that's great and they're happy. So, could AWS or Coca-Cola, and at that very high level, it becomes something that you are signing up for significant amount of support work, in theory, or a bunch of other directions.I've always found that from where I stand, especially dealing with those very large companies with very specific SLA requirements and the rest, the pricing for enterprise that I always look for as the right answer for my mind is ‘click here to contact us.' Because procurement departments, for example, we want this, this, this, this, and this around data guarantees and indemnities and all the rest. And well, yeah, that's going to be expensive. And well, yeah. We're a procurement company at a Fortune 50. We don't sign contracts that don't have two commas in them.So, it feels like there's a dialing it in with some custom optionality that feels like it is signaling to the quote-unquote, ‘sophisticated buyer,' as patio11 likes to say on Twitter from time to time, that might be the right direction.Rafal: That's really good feedback. I haven't thought about it this way, but you really opened my eyes on this issue.Corey: I'm glad it was helpful. The reason I think about it this way is that more and more I'm realizing that pricing is one of the most key parts of marketing and messaging around something, and that is not really well understood, even by larger companies with significant staff and full marketing teams. I still see the pricing often feels like an afterthought, but personally, when I'm trying to figure out is this tool for me, the first thing I do is—I don't even read the marketing copy of the landing page; I look for the pricing tab and click because if the only prices ‘call for details,' I know, A, it's going to be expensive, be it's going to be a pain in the neck to get to use it because it's two in the morning; I'm trying to get something done. I want to use it right now. If I had to have a conversation with your sales team first, that's not going to be cheap and it's not going to be something I'm going to be able to solve my problem this week. And that is the other end of it. I yell at people on both sides on that one.Rafal: Okay.Corey: Again, none of this stuff is intuitive; all of this stuff is complicated, and the way that I tend to see the world is, granted, a little bit different than the way that most folks who are kicking around databases and whatnots tend to view the world. Do you have plans in the future to extend Dynobase beyond strictly DynamoDB, looking to explore other fine database options like Redis, or MongoDB, or my personal favorite Route 53 TXT records?Rafal: [laugh]. Yeah. So, we had plans. Oh, we had really big plans. We felt that we are going to create a second JetBrains company. We started analyzing the market when it comes to MongoDB, when it comes to Cassandra, when it comes to Redis. And our first pick was Cassandra because it seemed, like, to have really, really similar structure of the table.I mean, it's also no secret it also has a primary index, secondary global indexes, and things like that. But as always, reality surprises us over the amount of detail that we cannot see from the very top. And it isn't as simple as just an install AWS SDK and install Cassandra Connector on—or Cassandra SDK and just roll with that. It requires a really big and significant investment. And we decided to focus just on one thing and nail this one thing and do this properly.It's like, if you go into the cloud, you can try to build a service that is agnostic, it's not using the best features of the cloud. And you can move your containers, for instance, across the clouds and say, “Hey, I'm cloud-agnostic,” but at the same time, you're missing out all the best features. And this is the same way we thought about Dynabase. Hey, we can provide an agnostic core, but then the agnostic application isn't going to be as good and as sophisticated as something tailored specifically for the needs of this database and user using this exact database.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Some of the things that you do just make so much sense that I get actively annoyed that there aren't better ways to do it and other places for other things. For example, when I fire up a table in a particular region within Dynobase, first it does a scan, which, okay, that's not terrible. But on some big tables, that can get really expensive. But you cap it automatically to a thousand items. And okay, great.Then it tells me, how long did it take? In this case because, you know, I am using on-demand and the rest and it's a little bit of a pokey table, that scan took about a second-and-a-half. Okay. You scanned a thousand items. Well, there's a lot more than a thousand items in this table. Ah, you limited it, so you didn't wind up taking all that time.It also says that it took 51-and-a-half RCUs—or Read Credit Units—because you know, why use normal numbers when you're AWS and doing pricing dimensions on this stuff.Rafal: [laugh].Corey: And to be clear, I forget the exact numbers for reads, but it's something like a million read RCUs cost me a dollar or something like that. It is trivial; it does not matter, but because it is consumption-based pricing, I always live in a little bit of a concern that, okay, if I screw up and just, like, scan the entire 10-megabyte table every time I want to make an operation here, and I make a lot of operations in the course of a week, that's going to start showing up in the bill in some really unfortunate ways. This sort of tells me as an ongoing basis of what it is that I'm going to wind up encountering.And these things are all configurable, too. The initial stream limit that you have configured as a thousand. I can set that to any number I want if I think that's too many or too few. You have a bunch of pagination options around it. And you also help people build out intelligent queries, [unintelligible 00:22:11] can export that to code. It's not just about the graphical interface clickety and done—because I do love my ClickOps but there are limits to it—it helps formulate what kind of queries I want to build and then wind up implementing in code. And that is no small thing.Rafal: Yeah, exactly. This is how we also envision that. The language syntax in DynamoDB is really… hard.Corey: Awful. The term is awful.Rafal: [laugh]. Yeah, especially for people—Corey: I know, people are going to be mad at me, but they're wrong. It is not intuitive, it took a fair bit of wrapping my head around. And more than once, what I found myself doing is basically just writing a thin CRUD API in Lambda in front of it just so I can query it in a way that I think about it as opposed to—now I'm not even talking changing the query modeling; I just want better syntax. That's all it is.Rafal: Yeah. You also touch on modeling; that's also very important thing, especially—or maybe even scan or query. Suppose I'm an engineer with tens years of experience. I come to the DynamoDB, I jump straight into the action without reading any of the documentation—at least that's my way of working—and I have no idea what's the difference between a scan and query. So, in Dynobase, when I'm going to enter all those filtering parameters into the UI, I'm going to hit scan, Dynobase is automatically going to figure out for you what's the best way to query—or to scan if query is not possible—and also give you the code that actually was behind that operation so you can just, like, copy and paste that straight to your code or service or API and have exactly the same result.So yeah, we want to abstract away some of the weird things about DynamoDB. Like, you know, scan versus query, expression attribute names, expression attribute values, filter, filtering conditions, all sorts of that stuff. Also the DynamoDB JSON, that's also, like, a bizarre thing. This JSON-type thing we should get out of the box, we also take care of that. So, yeah. Yeah, that's also our mission to make the DynamoDB as approachable as possible. Because it's a great database, but to truly embrace it and to truly use it, it's hard.Corey: I want to be clear, just for folks who are not seeing some of the benefits of it the way that I've described it thus far. Yes, on some level, it basically just provides a attractive, usable interface to wind up looking at items in a DynamoDB table. You can also use it to wind up refining queries to look at very specific things. You can export either a selection or an entire table either to a local file—or to S3, which is convenient—but it goes beyond on that because once you have the query dialed in and you're seeing the things you want to see, there's a generate code button that spits it out in—for Python, for JavaScript, for Golang.And there are a few things that the AWS CLI is coming soon, according to the drop-down itself. Java; ooh, you do like pain. And Golang for example, it effectively exports the thing you have done by clicking around as code, which is, for some godforsaken reason, anathema to most AWS services. “Oh, you clicked around to the console to do a thing. Good job. Now, throw it all away and figure out how to do it in code.” As opposed to, “Here's how to do what you just did programmatically.” My God, the console could be the best IDE in the world, except that they don't do it for some reason.Rafal: Yeah, yeah.Corey: And I love the fact that Dynobase does.Rafal: Thank you.Corey: I'm a big fan of this. You can also import data from a variety of formats, export data, as well. And one of the more obnoxious—you talk about weird problems I have with DynamoDB that I wish to fix: I would love to move this table to a table in a different AWS account. Great, to do that, I effectively have to pause the service that is in front of this because I need to stop all writes—great—export the table, take the table to the new account, import the table, repoint the code to talk to that thing, and then get started again. Now, there are ways to do it without that, and they all suck because you have to either write a shim for it or you have to wind up doing a stream that winds up feeding from one to the other.And in many cases, well okay, I want to take the table here, I do a knife-edge cutover so that new rights go to the new thing, and then I just want to backfill this old table data into it. How do I do that? The official answer is not what you would expect it to be, the DynamoDB console of ‘import this data.' Instead, it's, “Oh, use AWS Glue to wind up writing an ETL function to do all of this.” And it's… what? How is that the way to do these things?There are import and export buttons in Dynobase that solve this problem beautifully without having to do all of that. It really is such a different approach to thinking about this, and I am stunned that this had to be done as a third party. It feels like you were using the native tooling and the native console the same way the rest of us do, grousing about it the same way the rest of us do, and then set out to fix it like none of us do. What was it that finally made you say, “You know, I think there's a better way and I'm going to prove it.” What pushed you over the edge?Rafal: Oh, I think I was spending, just, hours in the console, and I didn't have a really sophisticated suite of tests, which forced me [unintelligible 00:27:43] time to look at the data a lot and import data a lot and edit it a lot. And it was just too much. I don't know, at some point I realized, like, hey, there's got to be a better way. I browsed for the solutions on the internet; I realized that there is nothing on the market, so I asked a couple of my friends saying like, “Hey, do you also have this problem? Is this also a problem for you? Do you see the same challenges?”And basically, every engineer I talked to said, “Yeah. I mean, this really sucks. You should do something about it.” And that was the moment I realized that I'm really onto something and this is a pain that I'm not alone. And so… yeah, that gave me a lot of motivation. So, there was a lot of frustration, but there was also a lot of motivation to push me to create a first product in my life.Corey: It's your first product, but it does follow an interesting pattern that seems to be emerging, Cloudash—Tomasz and Maciej—wound up doing that as well. They're also working at Stedi and they have their side project which is an Electron-based desktop application that winds up, we're interfacing with AWS services. And it's. What are your job requirements over at Stedi, exactly?People could be forgiven for seeing these things and not knowing what the hell EDI is—which guilty—and figure, “Ah, it's just a very fancy term for a DevRels company because they're doing serverless DevRel as a company.” It increasingly feels an awful lot like that.j, what's going on over there where that culture just seems to be an emergent property?Rafal: So, I feel like Stedi just attracts a lot of people that like challenges and the people that have a really strong sense of ownership and like to just create things. And this is also how it feels inside. There is plenty of individuals that basically have tons of energy and motivation to solve so many problems not only in Stedi, but as you can see also outside of Stedi, which is a result—Cloudash is a result, the mapping tool from Zack Charles is also a result, and Michael Barr created a scheduling service. So, yeah, I think the principles that we have at Stedi basically attract top-notch builders.Corey: It certainly seems so. I'm going to have to do a little more digging and see what some of those projects are because they're new to me. I really want to thank you for taking so much time to speak with me about what you're building. If people want to learn more or try to kick the tires on Dynobase which I heartily recommend, where should they go?Rafal: Go to dynobase.dev, and there's a big download button that you cannot miss. You download the software, you start it. No email, no credit card required. You just run it. It scans your credentials, profiles, SSOs, whatever, and you can play with it. And that's pretty much it.Corey: Excellent. And we will put a link to that in the [show notes 00:30:48]. Thank you so much for your time. I really appreciate it.Rafal: Yeah. Thanks for having me.Corey: Rafal Wilinski, serverless engineer at Stedi and creator of Dynobase. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—or a thumbs up and like and subscribe buttons on the YouTubes if that's where you're watching it—whereas if you've hated this podcast, same thing—five-star review, hit the buttons and such—but also leave an angry, bitter comment that you're not going to be able to find once you write it because no one knows how to put it into DynamoDB by hand.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

During the weekend i setup whats called a RoadWarrior VPN. The name comes from the early 2000 when remote workers, usually travelling, on the road, sales people needed a way to safely connect back to the office and place orders and handle administration. It is essentially a client to server connection. One specific computer gets access to the network behind the firewall compared to a site-to-site VPN where two networks can be connected through a VPN-tunnel. I created this so that I can connect home and use my NAS even if I'm not at home. This is a quite interesting setup. I connect to my home through the VPN tunnel and then I'm routed through another VPN-tunnel to the office where my NAS is. The reason for this is that my office doesn't have a fixed IP that I control the firewall for. It was not straightforward to setup. But after a bit of tinkering it started to work just fine. The pfsense VPN-setup-wizard didn't work for me so I had to do it manually. But it is just a handful of steps. Step one is to setup a new Certification Authority, a CA, if there already is one, you do not have to do this of course. Step two is to create a server certificate - this is probably where my problem arose with the wizard. The CN (common name) of the cert should correspond to the adress your clients should connect too. Step three is to create a client certificate - this you can put what ever you want as the CN. Step four is to create the OpenVPN server. This is pretty straight forward. I choose to route all traffic through the tunnel - this compared to only send through the traffic that should go to the network behind the VPN. The main reason I choose this approach is that it makes my routing a slight easier. And since I am the only user that is using the setup I am not really concerned about the extra traffic usage. The fifth and last step is to configure te clients. The clients for me is my two laptops (Mac and Surface), and my iPad and my iPhone. I installed the client export plugin into my pfsense that could create ovnp-files for me simplifying the configuration. On my iPad, iPhone and Surface i used the OpenVPN client. It was a super simple setup. Install the client and import the ovpn file and it just worked. For my Mac I decided to use tunnelblick an OpenSource VPN client that I have used in the past. This was as easy to get working. I got a few complaints that a few of the features I used in my setup will be obsoleted in an upcomming releas (due to changes in OpenSSL). Today I was working out side of the office and my home and needed to get hold of a few files on the nas. And this setup up worked really well. I uploaded and downloaded roughly 2GB of data (video/screen recordings) - and it worked so well.

It's an off-by-one error in the podcast this week as we bring you part 4 of Camila's 3-part Ubuntu hardening series, plus we look at security updates for Thunderbird, OpenVPN, Python, Paramiko and more.

About RickI lead the developer relations team for strategic accounts at MongoDB. My responsibilities include defining technical standards for the global strategic accounts team and consulting with the largest customers and opportunities for the business. My role spans technology sectors and as part of my engagements I routinely provide guidance on industry best practices, technology transformation, distributed systems implementation, cloud migration, and more. I led the architecture and design effort at Amazon for migrating thousands of relational workloads from RDBMS to NoSQL and built the center of excellence team responsible for defining the best practices and design patterns used today by thousands of Amazon internal service teams and AWS customers. I currently operate as the technical leader for our global strategic account teams to build the market for MongoDB technology by facilitating center of excellence capabilities within our customer organizations through training, evangelism, and direct design consultation activities.30+ years of software and IT expertise.9 patents in Cloud Virtualization, Complex Event Processing, Root Cause Analysis, Microprocessor Architecture, and NoSQL Database technology.Links: MongoDB: https://www.mongodb.com/ Twitter: https://twitter.com/houlihan_rick TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A year or two before the pandemic hit, I went on a magical journey to a mythical place called Australia. I know, I was shocked as anyone to figure out that this was in fact real. And while I was there, I gave the opening keynote at a conference that was called Latency Conf, which is great because there's a heck of a timezone shift, and I imagine that's what it's talking about.The closing keynote was delivered by someone I hadn't really heard of before, and he started talking about single table design with respect to DynamoDB, which, okay, great; let's see what he's got to say. And the talk started off engaging and entertaining and a high-level overview and then got deeper and deeper and deeper and I felt, “Can I please be excused? My brain is full.” That talk was delivered by Rick Houlihan, who now is the Director of Developer Relations for Strategic Accounts over at MongoDB, and I'm fortunate enough to be able to get him here to more or less break down some of what he was saying back then, catch up with what he's been up to, and more or less suffer my slings and arrows. Rick, thank you for joining me.Rick: Great. Thanks, Corey. I really appreciate—you brought back some memories, you know, trip down memory lane there. And actually, interestingly enough, that was the world's introduction to single table design was that. That was my dry-run rehearsal for re:Invent 2018 is where I delivered that talk, and it has become since the most positive—Corey: This was two weeks before re:Invent, which was just a great thing. I'd been invited to go; why not? I figured I'd see a couple of clients I had out in that direction. And I learned things like Australia is a big place. So, doing a one-week trip, including Sydney, Melbourne, and Perth. Don't do that.Rick: I had no idea that it took so long to fly from one side to the other, right? I mean, that's a long plane [laugh] [crosstalk 00:02:15]—Corey: Oh, yeah. And you were working at AWS at the time—Rick: Absolutely.Corey: —so I can only assume that they basically stuffed you into a dog kennel and threw you underneath the seating area, given their travel policy?Rick: Well, you know, I have the—[clear throat] actually at the time, they just upgraded the policy to allow the intermediate seating, right? So, if you wanted to get the—Corey: Ohhh—Rick: I know—Corey: Big spender. Big spender.Rick: Yes, yes. I can get a little bit extra legroom, so I didn't have my knees shoved into some of these back. But it was good.Corey: So, let's talk about, I guess… we'll call it the elephant in the room. You were at MongoDB, where you were a big proponent of the whole no-SQL side of the world. Then you went to go work at AWS and you carried the good word of DynamoDB far and wide. It made an impression; I built my entire newsletter pipeline production system on top of DynamoDB. It has the same data in three different tables because I'm not good at listening or at computers.But now you're back at Mongo. And it's easy to jump to the conclusion of, “Oh, you're just shilling for whoever it is that happens to sign your paycheck.” And at this point, are you—what's the authenticity story? But I've been paying attention to what you've been saying, and I think that's a bad take because you have been saying the same things all along since before you were on the Dynamo side of it. I do some research for this show, and you've been advocating for outcomes and the right ways to do things. How do you view it?Rick: That's basically the story here, right? I've always been a proponent of NoSQL. You know, what I took—the knowledge—it was interesting, the knowledge I took from MongoDB evolved as I went to AWS and I delivered, you know, thousands of applications and deployed workloads that I'd never even imagined I would have my hands on before I went there. I mean, honestly, what a great place it was to cut your teeth on data modeling at scale, right? I mean, that's the—there is no greater scale.That's when you learn where things break. And honestly, a lot of the lessons I took from MongoDB, well, when I applied them at scale at AWS, they worked with varying levels of success, and we had to evolve those into the sets of design patterns, which I started to propose for DynamoDB customers, which had been highly effective. I still believe in all those patterns. I would never tell somebody that they need to drop everything and run to MongoDB, but, you know, again, all those patterns apply to MongoDB, too, right? A very—a lot—I wouldn't say all of them, but many of them, right?So, I'm a proponent of NoSQL. And I think we talked before the call a little bit about, you know, if I was out there hocking relational technology right now and saying RDBMS is the future, then everybody who criticizes anything I say, I would absolutely have to, you know, say that there's some validity there. But I'm not saying anything different I've ever said. MongoDB announced Serverless, if you remember, in July, and that was a big turning point for me because the API that we offer, the developer experience for MongoDB is unmatched, and this is what I talk to people now. And it's the patterns that I've always proposed, I still model data the same way, I don't do it any different, and I've always said, if you go back to my earlier sessions on NoSQL, it's all the same.It doesn't matter if it's MongoDB, DynamoDB, or any other technology. I've always shown people how to model their data and NoSQL and I don't care what database you're using, I've actually helped MongoDB customers do their job better over the years as well. So.Corey: Oh, yeah. And looking back at some of your early talks as well, you passed my test for, “Is this person a shill?” Because you wound up in those talks, addressing head-on when is a relational model the right thing to do? And then you put the answers up on a slide, and this—and what—it didn't distill down to, “If you're a fool.”Rick: [laugh].Corey: Because there are use cases where if you don't [unintelligible 00:05:48] your access patterns, if you have certain constraints and requirements, then yeah. That you have always been an advocate for doing the right thing for the workload. And in my experience, for my use cases, when I looked at MongoDB previously, it was not a fit for me. It was very much a you run this on an instance basis, you have to handle all this stuff. Like three—you kno, keeping it in triplicate in three different DynamoDB tables, my newsletter production pipeline now, including backups and the rest, of DynamoDB portion has climbed to the princely sum of $1.30 a month, give or take.Rick: A month. Yes, exactly.Corey: So, there's no answer for that there. Now that Mongo Serverless is coming out into the world, oh, okay, this starts to be a lot more compelling. It starts to be a lot more flexible.Rick: I was just going to say, for your use case there, Corey, you're probably looking at the very similar pricing experience now, with MongoDB Serverless. Especially when you look at the pricing model, it's very close to the on-demand table model. It actually has discounted tiering above it, which I haven't really broken it down yet against a provision capacity model, but you know, there's a lot of complexity in DynamoDB pricing. And they're working on this, they'll get better at it as well, but right now you have on-demand, you have provisioned throughput, you have [clear throat] reserved capacity allocations. And, you know, there's a time and place for all of those, but it puts the—again, it's just complexity, right?This is the problem that I've always had with DynamoDB. I just wish that we'd spent more time on improving the developer experience, right, enhancing the API, implementing some of these features that, you know, help. Let's make single table design a first-class citizen of the DynamoDB API. Right now it's a red—it's a—I don't want to say redheaded stepchild, I have two [laugh] I have two redhead children and my wife is redhead, but yeah. [laugh].Corey: [laugh]. That's—it's—Rick: That's the way it's treated, right? It's treated like a stepchild. You know, it's like, come on, we're fully funding the solutions within our own umbrella that are competing with ourselves, and at the same time, we're letting the DynamoDB API languish while our competitors are moving ahead. And eventually, it just becomes, you know, okay, guys, I want to work with the best tooling on the market, and that's really what it came down to. As long as DynamoDB was the king of serverless, yes, absolutely; best tooling on the market.And they still are [clear throat] the leader, right? There's no doubt that DynamoDB is ahead in the serverless landscape, that the MongoDB solution is in its nascency. It's going to be here, it's going to be great, that's part of what I'm here for. And that's again, getting back to why did you make the move, I want to be part of this, right? That's really what it comes down to.Corey: One of the things that I know that was my own bias has always been that if I'm looking at something like—that I'm looking at my customer environments to see what's there, I can see DynamoDB because it has its own line item in the bill. MongoDB is generally either buried in marketplace charges, or it's running on a bunch of EC2 instances, or it just shows up as data transfer. So, it's not as top-of-mind for the way that I view things in… through the lens of you know, billing. So, that does inform my perception, but I also know that when I'm talking to large-scale companies about what they're doing, when they're going all-in on AWS, a large number of them still choose things like Mongo. When I've asked them why that is, sometimes you get the answer of, “Oh, legacy. It's what we built on before.” Cool—Rick: Sure.Corey: —great. Other times, it's a, “We're not planning to leave, but if we ever wanted to go somewhere else, it's nice to not have to reimagine the entire data architecture and change the integration points start to finish because migrations are hard enough without that.” And there is validity to the idea of a strategic exodus being possible, even if it's not something you're actively building for all the time, which I generally advise people not to do.Rick: Yeah. There's a couple things that have occurred over the last, you know, couple of years that have changed the enterprise CIO and CTO's assessment of risk, right? Risk is the number one decision factor in a CTOs portfolio and a CIO's, you know, decision-making process, right? What is the risk? What is the impact of that risk? Do I need to mitigate that risk, or do I accept that risk? Okay?So, right now, what you've seen is with Covid, people have realized that you know, on-prem infrastructure is a risk, right? It used to be an asset; now it's a risk. Those personnel that have to run that on-prem infrastructure, hey, what happens when they're not available? The infrastructure is at risk. Okay.So, offloading that to cloud providers is the natural solution. Great. So, what happens when you offload to a cloud provider and IAD goes down, or you know, us-east-1 goes down—we call it IAD or we used to call it IAD internally at AWS when I was there because, you know, the regions were named by airport codes, but it's us-east-1—how many times has us-east-1 had problems? Do you want to really be the guy that every time us-east-1 goes down, you're in trouble? What happens when people in us-east-1 have trouble? Where do they go?Corey: Down generally speaking.Rick: [crosstalk 00:10:37]—well, if they're well-architected, right, if they're well-architected, what do they do? They go to us-west-2. How much infrastructure is us-west-2 have? So, if everybody in us-east-1 is well-architected, then they all go to us-west-2. What happens in us-west-2? And I guarantee you—and I've been warning about this at AWS for years, there's a cascade failure coming, and it's going to be coming because we're well-architecting everybody to failover from our largest region to our smaller regions.And those smaller regions, they cannot take the load and nobody's doing any of that planning, so, you know, sooner or later, what you're going to see is dominoes fall, okay? [clear throat]. And it's not just going to be us-east-1, it's going to be us-east-1 failed, and the rollover caused a cascade failure in us-west-2, which caused a cascade—Corey: Because everyone's failing over during—Rick: That's right. That's right.Corey: —this event the same way. And also—again, not to dunk on them unnecessarily, but when—Rick: No, I'm not dunking.Corey: —us-east-1 goes, down a lot of the control plane services freeze up—Rick: Oh, of course they do.Corey: —like [unintelligible 00:11:25].Rick: Exactly. Oh, we not single point of failure, right? Uh-huh, exactly. There you go, Route 53, now—and that actually surprised me is DynamoDB instead of Route 53 is your primary database. So, I'm actually must have had some impact on you—Corey: To move one workload off of Dynamo to Route 53 [crosstalk 00:11:39] issue number because I have to practice what I preach.Rick: That's right. Exactly.Corey: It was weird; they the thing slower and little bit less, uh—Rick: [laugh]. I love it when [crosstalk 00:11:45]—yeah, yeah—Corey: —and a little bit [crosstalk 00:11:45] cache-y. But yeah.Rick: —sure. Okay, I can understand that. [laugh].Corey: But it made the architecture diagram a little bit more head-scratching, and really, that's what it's all about. Getting a high score.Rick: Right. So, if you think about your data, right, I mean, would you rather be running on an infrastructure that's tied to a cloud provider that could experience these kinds of regional failures and cascade failures, or would you rather have your data infrastructure go across cloud providers so that when provider has problems, you can just go ahead and switch the light bulb over on the other one and ramp right back up, right? You know? And honestly, you're running active, active configurations and that kind of, [clear throat] you know, deployment, you know, design, and you're never going to go down. You're always going—Corey: The challenge I've had—Rick: —to be the one that stays up.Corey: The theory is sound, but the challenge I've had in production with trying these things is that one, the thing that winds up handling the failover piece is often causes more outage than the underlying stuff itself.Rick: Well, sure. Yeah.Corey: Two, when you're building something to run a workload to run in multiple cloud providers, you're forced to use a lot of—Rick: Lowest common denominator?Corey: Lowest common denominator stuff. Yeah.Rick: Yeah, yeah totally. I hear that all the time.Corey: Unless you're actively running it in both places, it looks like a DR Plan, which doesn't survive the next commit to the codebase. It's the—Rick: I totally buy that. You're talking about the stack, stack duplication, all that kind of—that's an overhead and complexity, I don't worry about at the data layer, right?Corey: Oh, yeah.Rick: The data layer—Corey: If you're talking about—Rick: —[crosstalk 00:12:58]Corey: —[crosstalk 00:12:58] data layer, oh, everything you're saying makes perfect sense.Rick: Makes perfect sense, right? And honestly, you know, let's put it this way: If this is what you want to do—Corey: What do you mean identity management and security handover working differently? Oh, that's a different team's problem. Oh, I miss those days.Rick: Yeah, you know, totally right. It's not ideal. But you know, I mean, honestly, it's not a deal that somebody wants to manage themselves, is moving that data around. The data is the lock-in. The data is the thing that ties you to—Corey: And the cost of moving it around in some cases, too.Rick: That's exactly right. You know, so you know, having infrastructure that spans providers and spans both on-prem and cloud, potentially, you know, that can span multiple on-prem locations, man, I mean, that's just that's power. And MongoDB provides that; I mean, DynamoDB can't. And that's really one of the biggest limitations that it will always have, right? And we talked about, and I still believe in the power of global tables, and multi-region deployments, and everything, it's all real.But these types of scenarios, I think this is the next generation of failure that the cloud providers are not really prepared for, they haven't experienced it, they don't know what it's even going to look like, and I don't think you want to be tied to a single provider when these things start happening, right, if you have a large amount of infrastructure deployed someplace. It just seems like [clear throat] that's a risk that you're running at these days, and you can mitigate that risk somewhat by going with a MongoDB Atlas. I agree, all those other considerations. But you know, I also heard—it's a lot of fun, too, right? There's a lot of fun in that, right?Because if you think about it, I can deploy technologies in ways on any cloud provider, they're going to be cloud provider agnostic, right? I can use, you know, containerized technologies, Kubernetes, I can use—hell, I'm not even afraid to use Lambda functions, and just, you know, put a wrapper around that code and deploy it both as a Lambda or a Cloud Function in GCP. The code's almost the same in many cases, right? What it's doing with the data, you can code this stuff in a way—I used to do it all the time—you abstract the data layer, right? Create a DAL. How about a CAL? A cloud [laugh] cloud access layer, right, you know? [laugh].Corey: I wish, on some level, we could go down some of these paths. And someone asked me once a while back of, “Well, you seem to have a lot of opinions on this. Do you think you could build a better cloud than AWS?” And my answer—Rick: Hell yes.Corey: —look them a bit by surprise of, “Absolutely. Step one, I want similar resources, so give me $20 billion to spend”—Rick: I was going to say, right?Corey: —”then I'm going to hire the smart people.” Not that we're somehow smarter or better or anything else than the people who built AWS originally, but now—Rick: We have all those lessons learned.Corey: —we have fifteen years of experience to fall back on.Rick: Exactly.Corey: “Oh. I wouldn't make that mistake again.”Rick: Exactly. Don't need to worry about that. Yeah exactly.Corey: You can't just turn off a cloud service and relaunch it with a completely different interface and API and the rest.Rick: People who criticize, you know, services like DynamoDB, like—and other AWS services—look, these things are like any kind of retooling of the services, it's like rebuilding the engine on the airplane while it's flying.Corey: Oh, yeah.Rick: And you have to do it with a level of service assurance that—I mean, come on. DynamoDB provides four nines out of the box, right? Five nines if you turn on global tables. And they're doing this at the same time as they have pipeline releases dropping regularly, right? So, you can imagine what kind of, you know, unit testing goes on there, what kind of Canary deployments are happening.It's just, it's an amazing infrastructure that they maintain, incredibly complex, you know? In some ways, these are lessons that we need to learn in MongoDB if we're going to be successful operating a shared backplane serverless, you know, processing fabric. We have to look at what DynamoDB does right. And we need to build our own infrastructure that mirrors those things, right? And in some ways, these things are there, in some ways, they're working on, in some ways, we got a long ways to go.But you know, I mean, it's this is the exciting part of that journey for me. Now, in my case, I focus on strategic accounts, right? Strategic accounts are big, you know, they're the potential to be our whale customers, right? These are probably not customers who would be all that interested in serverless, right? They're customers that would be more interested in provisioned infrastructure because they're the people that I talked to when I was at DynamoDB; I would be talking to customers who are interested in like, reserved capacity allocations, right? If you're talking about—Corey: Yeah, I wanted to ask you about that. You're developer advocacy—which I get—for strategic accounts.Rick: Right.Corey: And I'm trying to wrap my head around—Rick: Why [crosstalk 00:17:19]—Corey: [crosstalk 00:17:19] strategic accounts are the big ones, potential spend lots of stuff. Why do they need special developer advocacy?Rick: [laugh]. Well, yeah, it's funny because, you know, one of the reasons why it started talking to Mark Porter about this, you know, was the fact that, you know, the overlap is really around [clear throat] the engagements that I ran when I was doing the Amazon retail migration, right? When Amazon retail started to move to NoSQL, we deprecated 3000 Oracle server instances, we moved a large percentage of those workloads to NoSQL. The vast majority probably just were lift-and-shift into RDS and whatnot because they were too small, too old, not worth upgrading whatnot, but every single tier, what we call tier-one service, right, every money-making service was redesigned and redeployed on DynamoDB, right? So, we're talking about 25,000 developers that we had to ramp. This is back four years ago; now we have, like, 75,000.But back then we had 25,000 global developers, we had [clear throat] a technology shift, a fundamental paradigm shift between relational modeling and NoSQL modeling, and the whole entire organization needed to get up to speed, right? So, it was about creating a center of excellence, it was about operating as an office of the CTO within the organization to drive this technology into the DNA of our company. And so that exercise was actually incredibly informative, educational, in that process of executing a technology transformation in a major enterprise. And this is something that we want to reproduce. And it's actually what I did for Dynamo as well, really more than anything.Yes, I was on Twitter, I was on Twitch, I did a lot of these things that were kind of developer advocate, you know, activities, but my primary job at AWS was working with large strategic customers, enabling their teams, you know, teaching them how to model their data in NoSQL, and helping them cross the chasm, right, from relational. And that is advocacy, right? The way I do it is I use their workloads. [clear throat]. I use their—the customers, you know, project teams themselves, I break down their models, I break down their access patterns when I leave, essentially—with the whole day of design reviews, we'll walk through 12 or 15 workloads, and when I leave these guys have an idea: How would I do it if I wanted to use NoSQL, right?Give them enough breadcrumbs so that they can actually say, “Okay, if I want to take it to the next step, I can do it without calling up and say, ‘Hey, can we get a professional services team in here?'” right? So, it's kind of developer advocacy and it's kind of not, right? We're kind of recognizing that these are whales, these are customers with internal resources that are so huge, they could suck our Developer's Advocacy Team in and chew it up, right? So, what we're trying to do is form a focus team that can hit hard and move the needle inside the accounts. That's what I'm doing. Essentially, it's the same work I did for [clear throat] AWS for DynamoDB. I'm just doing it for, you know—they traded for a new quarterback. Let's put it that way. [laugh].Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: So, one thing that I find appealing about the approach maps to what I do in the world of cloud economics, where I—like, in my own environment, our AWS bill is creeping up again—we have 14 AWS accounts—and that's a little over $900 a month now. Which, yeah, big money, big money.Rick: [laugh].Corey: In the context of running a company, that no one notices or cares. And our customers spend hundreds of millions a year, pretty commonly. So, I see the stuff in the big accounts and I see the stuff in the tiny account here. Honestly, the more interesting stuff is generally in on the smaller side of the scale, just because you're not going to have a misconfiguration costing a third of your bill when a third of your bill is $80 million a year. So—Rick: That's correct. If you do then that's a real problem, right?Corey: Oh yeah.Rick: [laugh].Corey: It's very much a two opposite ends of a very broad spectrum. And advice for folks in one of those situations is often disastrous to folks on the other side of that.Rick: That's right. That's right. I mean, at some scale, managing granularity hurts you, right? The overhead of trying to keep your costs, you know, it—but at the same time, it's just different, a different measure of cost. There's a different granularity that you're looking at, right? I mean, things below a certain, you know, level stop becoming important when, you know, the budget start to get a certain scale or a certain size, right? Theoretically—Corey: Yeah, for there's certain workloads, things that I care about with my dollar-a-month Dynamo spend, if I were to move that to Mongo Serverless, great, but my considerations are radically different than a company that is spending millions a month on their database structure.Rick: That's right. Really, that's what it comes down to.Corey: Yeah, we don't care about the pennies. We care about is it going to work? How do we back it up? What's the replication factor?Rick: And that—but also, it's more than that. It's, you know, for me, from my perspective, it really comes down to that, you know, companies are spending millions of dollars a year in database services. These are companies that are spending ten times that, five times that, in you know, in developers, you know, expense, right? Building services, maintaining the code that runs—that the services run.You know, the biggest problem I had with MongoDB is the level of code complexity. It's a cut after cut after cut, right? And the way I kind of describe the experience—and other people have described it to me; I didn't come up with this analogy. I had a customer tell me this as they were leaving DynamoDB—“DynamoDB is death by a thousand cuts. You love it, you start using it, you find a little problem, you start fixing it. You start fixing it. You start fixing—you come up with a pattern. Talk to Rick, he'll come up with something. He'll tell you how to do that.” Okay?And you know, how many customers did I would do this with? You know, and it's honestly, they're 15-minute phone calls for me, but every single one of those 15-minute phone calls turns into eight hours of developer time writing the code, debugging it, deploying it over and over again, it's making sure it's going the way it's [crosstalk 00:23:02]—Corey: Have another 15-minute call with Rick, et cetera, et cetera. Yeah.Rick: Another 15—exactly. And it's like okay, that's you know—eventually, they just get tired of it, right? And I actually had a customer that tell me—a big customer—tell me flat out, “Yeah, you proved that the DynamoDB can support our workload and it'll probably do it cheaper, but I don't have a half-a-dozen Ricks on my team, right? I don't have any Ricks on my team. I can't be getting you in here every single time we have to do a complex data model overhaul, right?”And this was—granted, it was one of the more complex implementations that I've ever done. In order to make it work. I had to overload the fricking table with multiple access patterns on the partition key, something I never done in my life. I made it work, but it was just—honestly, that was an exercise to me that taught me something. If I have to do this, it's unnatural, okay?And that's—[laugh] you know what I mean? And honestly, there's API improvements that we could have done to make that less of a problem. It's not like we haven't known since the last, I don't know, I joined the company that a thousand WCUs per storage partition was pretty small. Okay? We've kind of known that for I don't know, since DynamoDB, was invented. As matter of fact is, from what I know, talking to people who were around back then, that was a huge bone of contention back in the day, right? A thousand WCUs, ten gigabytes, there were a lot of the PEs on the team that were going, “No way. No way. That's way too small.” And then there were other people that were like, “Nah, nobody's ever going to need more than that.” And you know, a lot of this was based on the analysis of [crosstalk 00:24:28]—Corey: Oh, nothing ever survives first contact from—Rick: Of course.Corey: —customer, particularly a customer who is not themselves deeply familiar with what's happening under the hood. Like, I had this problem back when I was traveling trainer for Puppet for a while. It was, “Great. Well, Puppet is obviously a piece of crap because everyone I talked to has problems with it.” So, I was one of the early developers behind SaltStack—Rick: Oh nice.Corey: —and, “Ah, this is going to be a thing of beauty and it'll be awesome.” And that lasted until the first time I saw what somebody's done with it in the wild. It was, “Oh, okay, that's an [unintelligible 00:25:00] choice.”Rick: Okay, that's how—“Yeah, I never thought about that,” right? Happy path. We all love the happy path, right? As we're working with technologies, we figure out how we like to use it, we all use it that way. Of course, you can solve any problem you want the way that you'd like to solve it. But as soon as someone else takes that clay, they mold a different statue and you go, “Oh, I didn't realize it could look like that.” Right, exactly.Corey: So, here's one for you that I've been—I still struggle with this from time to time, but why would I, if I'm building something out—well, first off, why on earth would I do that? I have people for that who are good at things—but if I'm building something out and it has a database layer, why would someone choose NoSQL over—Rick: Oh, sure.Corey: —over SQL?Rick: [crosstalk 00:25:38] question.Corey: —and let me be clear here—and I'm coming at this from the perspective of someone who, basically me a few years ago, who has no real understanding of what databases are. So, my mental model of a database is Microsoft Excel, where I can fire up a [unintelligible 00:25:51] table of these things—Rick: Sure. [laugh]. Hey, well then, you know what? Then you should love NoSQL because that's kind of the best analogy of what is NoSQL. It's like a spreadsheet, right? Whereas a relational database is like a bunch of spreadsheets, each with their own types of rows, right? So—[laugh].Corey: Oh, my mind was blown with relational stuff [unintelligible 00:26:07] wait, you could have multiple tables? It's, “What do you think relational meant there, buddy?” My map of NoSQL was always key and value, and that was it. And that's all it can be. And sure, for some things, that's what I use, but not everything.Rick: That's right. So, you know, the bottom line is, when you think about the relational database, it all goes back to, you know, the first paper ever written on the relational model, Edgar Codd—and I can't remember the exact title, but he wrote the distributed model, the data model for distributed systems, something like that. He discussed, you know, the concept of normalization, the power of normalization, why you would want this. And the reason why we wanted this, why he thought this was important, this actually kind of demonstrates how—boy, they used to write killer abstracts to papers, right? It's like the very first sentence, this is why I'm write in this paper. You read the first sentence, you know: “Future users of modern computer systems must have a way to be able to ask questions of the data without knowing how to write code.”I mean, I don't know if those were the words, but that was basically what he said, that was why he invented the normalized data model. Because, you know, with the hierarchical management systems at the time, everyone had to know everything about the data in order to be able to get any answers, right? And he was like, “No, I want to be able to just write a question and have the system answer that.” Now, at the time, a lot of people felt like that's great, and they agreed with his normalized model—it was elegant—but they all believe that the CPU overhead at the time was way too high, right? To generate these views of data on the fly, no freaking way. Storage is expensive. But it ain't that expensive, right?Well, this little thing called Moore's Law, right? Moore's Law balanced his checkbook for, like, 40 years, 50 years, it balanced the relational database checkbook, okay? So, as the CPUs got faster and faster, crunching, the data became less and less of a problem, okay? And so we crunched bigger and bigger data sets, we got very, very happy with this. Up until about 2014.At 2014, a really interesting thing happened. If you look at the top 500, which is the supercomputers, the top 500 supercomputing clusters around the world, and you look at their performance increases year-to-year after 2014, it went off a cliff. No longer beating Moore's Law. Ever since, they've been—and per-core performance, you know, CPU, you know, instructions executed per second, everything. It's just flattening. Those curves are flattening. Moore's Law is broken.Now, you'll get people argue about it, but the reality is, if it wasn't broken, the top 500 would still be cruising away. They're not. Okay? So, what this is telling us is that the relational database is losing its horsepower. Okay?Why is it happening? Because, you know, gate length has an absolute minimum, it's called zero, right? We can't have a logic gate that's the—with negative distance, right? [laugh]. So, you know, these things—but storage, storage, hey, it just keeps on getting cheaper and cheaper, right?We're going the other way with storage, right? It's gigabytes, it's terabytes, it's petabytes, you know, with CPU, we're going smaller and smaller and smaller, and the fab cost is increasing. There's just—it's going to take a next-generation CPU technology to get back on track with Moore's Law.Corey: Well, here's the challenge. Everything you're saying makes perfect sense from where your perspective is. I reiterate, you are working with strategic accounts, which means ‘big.' When I'm building something out in the evenings because I want to see if something is possible, performance considerations and that sort of characteristic does not factor into it. When I'm a very small-scale, I care about cost to some extent—sure, whatever—but the far more expensive aspect of it, in the ways that matter have been what is the expensive—what—the big expensive piece is—Rick: We've talked about it.Corey: —engineering time—Rick: That's what we just talked about, right?Corey: —where it's, “What I'm I familiar with?”Rick: As a developer, right, why would I use MongoDB over DynamoDB? Because the developer experience [crosstalk 00:29:33]—Corey: Exactly. Sure, down the road there are performance characteristics and yeah, at the time I have this super-large, scaled-out, complex workload, yeah, but most workloads will not get to that.Rick: Will not ever get there. Ever get there. [crosstalk 00:29:45]—Corey: Yeah, so optimizing for [crosstalk 00:29:45], how's it going to work when I'm Facebook-scale? It's—Rick: So, first of—no, exactly, Facebook scale is irrelevant here. What I'm talking about is actually a cost ratchet that's going to lever on midsize workloads soon, right? Within the next four to five years, you're going to see mid-level workloads start to suffer from significant performance cost deficiencies compared to NoSQL workloads running on the same. Now you—hell, you see it right now, but you don't really experience it, like you said, until you get to scale, right? But in midsize workloads, [clear throat] that's going to start showing up, right? This cost overhead cannot go away.Now, the other thing here that you got to understand is, just because it's new technology doesn't make it harder to use. Just because you don't know how to use something, right, doesn't mean that it's more difficult. And NoSQL databases are not more difficult than the relational database. I can express every single relationship in a NoSQL database that I express in a relational database. If you think about the modern OLTP applications, we've done the analysis, ad nauseum: 70% of access patterns are for a single object, a single row of data from a single table; another 20% are for a row of datas—a range of rows from a single table. Okay, that leaves only 10% of your access patterns involve any kind of complex table traversal or entity traversals. Okay?And most of those are simple one-to-many hierarchies. So, let's put those into perspective here: 99% of the access patterns in an OLTP application can be modeled without denormalization in a single table. Because single table doesn't require—just because I put all the objects in one place doesn't mean that it's denormalized. Denormalized requires strong redundancies in the stored set. Duplication of data. Okay?Edgar Codd himself said that the normalized data model does not depend on storage, that they are irrelevant. I could put all the objects in the same document. As long as there's no duplication of data, there's no denormalization. I know, I can see your head going, “Wow,” but it's true, right? Because as long as I can clearly express the relationships of the data without strong redundancies, it is a normalized data model.That's what most people don't understand. NoSQL does not require denormalization. That's a decision you make, and it usually happens when you have many-to-many relationships; then we need to start duplicating the data.Corey: In many cases, at least my own experience—because again, I am bad at computers—I find that the data model is not something that is sat out—that you sit down and consciously plan very often. It's rather something—Rick: Oh yeah.Corey: —happens to you instead. I mean—Rick: That's right. [laugh].Corey: —realistically, like, using DynamoDB for this is aspirational. I just checked, and if I look at—so I started this newsletter back in March of 2017. I spun up this DynamoDB table that backs it, and I know it's the one that's in production because it has the word ‘test' in its name, because of course it does. And I'm looking into it, and it has 8700 items in it now and it's 3.7 megabytes. It's—Rick: Sure, oh boy. Nothing, right?Corey: —not for nothing, this could have been just as easily and probably less complex for my level of understanding at the time, a CSV file that I—Rick: Right. Exactly, right.Corey: —grabbed from a Lambda out of S3, do the thing to it, and then put it back.Rick: [unintelligible 00:32:45]. Right.Corey: And then from a performance and perspective side on my side, it would make no discernible difference.Rick: That's right because you're not making high-velocity requests against the small object. It's just a single request every now and then. S3 performance would probably—you might even be less. It might even cost you less to use S3.Corey: Right. And 30 to 100 of the latest ones are the only things that are ever looked at in any given week, the rest of it is mostly deadstock that could be transitioned out elsewhere.Rick: Exactly.Corey: But again, like, now that they have their lower cost infrequent access storage, then great. It's not item level; it's table levels, so what's the point? I can knock that $1.30 a month down to, what, $1.10?Rick: Oh well, yeah, no, I mean, again, Corey for those small workloads, you know what? It's like, go with what you know. But the reality is, look, as a developer, we should always want to know more, and we should always want to know new things, and we should always be aware of where the industry is headed. And honestly, I've heard through—I'm an old, old school, relational guy, okay, I cut my teeth on—oh, God, I don't even know what version of MS SQL Server it was, but when I was, you know, interviewing at MongoDB. I was talking to Dan Pasette, about the old Enterprise Manager, where we did the schema designer and all this, and we were reminiscing about, you know, back in the day, right?Yeah, you know, reality of things are is that if you don't get tuned into the new tooling, then you're going to get left behind sooner or later. And I know a lot of people who that has happened to over the years. There's a reason why I'm 56 years old and still relevant in tech, okay? [laugh].Corey: Mainframes, right? I kid.Rick: Yes, mainframes.Corey: I kid. You're not that much older than I am, let's be clear here.Rick: You know what? I worked on them, okay? And some of my peers, they never stopped, right? They just kind of stayed there.Corey: I'm still waiting for AWS/400. We don't see them yet, but hope springs eternal.Rick: I love it. I love that. But no, one of the things that you just said that I think it hit me really, it's like the data model isn't something you think about. The data model is something that just happens, right? And you know what, that is a problem because this is exactly what developers today think. They think know the relational database, but they don't.You talk to any DBA out there who's coming in after the fact and cleaned up all the crappy SQL that people like me wrote, okay? I mean, honestly, I wrote some stuff in the day that I thought, “This is perfect. There's no way that could be anything better than this,” right? Nice derived table joins insi—and you know what? Then here comes the DBA when the server is running at 90% CPU and 100% percent memory utilization and page swapping like crazy, and you're saying we got to start sharding the dataset.And you know, my director of engineering at the time said, “No, no, no. What we need is somebody to come in and clean up our SQL.” I said, “What do you mean? I wrote that SQL.” He's like, “Like I said, we need someone to come and clean up our SQL.”I said, “Okay, fine.” We brought the guy in. 1500 bucks an hour, we paid this guy, I was like, “There's no way that this guy is going to be worth that.” A day and a half later, our servers are running at 50% CPU and 20% memory utilization. And we're thinking about, you know, canceling orders for additional hardware. And this was back in the day before cloud.So, you know, developers think they know what they're doing. [clear throat]. They don't know what they're doing when it comes to the database. And don't think just because it's a relational database and they can hack it easier that it's better, right? Yeah, it's, there's no substitute for knowing what you're doing; that's what it comes down to.So, you know, if you're going to use a relational database, then learn it. And honestly, it's a hell of a lot more complicated to learn a relational database and do it well than it is to learn how to model your data in NoSQL. So, if you sit two developers down, and you say, “You learn NoSQL, you learn relational,” two months later, this guy is still going to be studying. This guy's going to be writing code for seven weeks. Okay? [laugh]. So, you know, that's what it comes down to. You want to go fast, use NoSQL and you won't have any problems.Corey: I think that's a good place to leave it. If people want to learn more about how you view these things, where's the best place to find you?Rick: You know, always hit me up on Twitter, right? I mean, @houlihan_rick, that's my—underbar rick, that's my Twitter handle. And you know, I apologize to folks who have hit me up on Twitter and gotten no response. My Twitter as you probably have as well, my message request box is about 3000 deep.So, you know, every now and then I'll start going in there and I'll dig through, and I'll reply to somebody who actually hit me up three months ago if I get that far down the queue. It is a Last In, First Out, right? I try to keep things as current as possible. [laugh].Corey: [crosstalk 00:36:51]. My DMs are a trash fire. Apologies as well. And we will, of course, put links to it in the [show notes 00:36:55].Rick: Absolutely.Corey: Thank you so much for your time. I really do appreciate it. It's always an education talking to you about this stuff.Rick: I really appreciate being on the show. Thanks a lot. Look forward to seeing where things go.Corey: Likewise.Rick: All right.Corey: Rick Houlihan Director of Developer Relations, Strategic Accounts at MongoDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an upset comment talking about how we didn't go into the proper and purest expression of NoSQL non-relational data, DNS TXT records.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About RachelRachel leads product and technical marketing for Chronosphere. Previously, Rachel wore lots of marketing hats at CloudHealth (acquired by VMware), and before that, she led product marketing for cloud-integrated storage at NetApp. She also spent many years as an analyst at Forrester Research. Outside of work, Rachel tries to keep up with her young son and hyper-active dog, and when she has time, enjoys crafting and eating out at local restaurants in Boston where she's based.Links: Chronosphere: https://chronosphere.io Twitter: https://twitter.com/RachelDines Email: rachel@chronosphere.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A repeat guest joins me today, and instead of talking about where she works, instead we're going to talk about how she got there. Rachel Dines is the Head of Product and Technical Marketing at Chronosphere. Rachel, thank you for joining me.Rachel: Thanks, Corey. It's great to be here again.Corey: So, back in the early days of me getting started, well, I guess all this nonsense, I was an independent consultant working in the world of cloud cost management and you were over at CloudHealth, which was effectively the 800-pound gorilla in that space. I've gotten louder, and of course, that means noisier as well. You wound up going through the acquisition by VMware at CloudHealth, and now you're over at Chronosphere. We're going to get to all of that, but I'd rather start at the beginning, which, you know, when you're telling stories seems like a reasonable place to start. Your first job out of school, to my understanding, was as an analyst at Forrester is that correct?Rachel: It was yeah. Actually, I started as a research associate at Forrester and eventually became an analyst. But yes, it was Forrester. And when I was leaving school—you know, I studied art history and computer science, which is a great combination, makes a ton of sense—I can explain it another time—and I really wanted to go work at the equivalent of FAANG back then, which was just Google. I really wanted to go work at Google.And I did the whole song-and-dance interview there and did not get the job. Best thing that's ever happened to me because the next day a Forrester recruiter called. I didn't know what Forrester was—once again, I was right out of college—I said, “This sounds kind of interesting. I'll check it out.” Seven years later, I was a principal analyst covering, you know, cloud-to-cloud resiliency and backup to the cloud and cloud storage. And that was an amazing start to my career, that really, I'm credited a lot of the things I've learned and done since then on that start at Forrester.Corey: Well, I'll admit this: I was disturbingly far into my 30s before I started to realize what it is that Forrester and its endless brethren did. I'm almost certain you can tell that story better than I can, so what is it that Forrester does? What is its place in the ecosystem?Rachel: Forrester is one of the two or three biggest industry analyst firms. So, the people that work there—the analysts there—are basically paid to be, like, big thinkers and strategists and analysts, right? There's a reason it's called that. And so the way that we spent all of our time was, you know, talking to interesting large, typically enterprise IT, and I was in the infrastructure and operations group, so I was speaking to infrastructure, ops, precursors to DevOps—DevOps wasn't really a thing back in ye olden times, but we're speaking to them and learning their best practices and publishing reports about the technology, the people and the process that they dealt with. And so you know, over a course of a year, I would talk to hundreds of different large enterprises, the infrastructure and ops leaders at everyone from, like, American Express to Johnson & Johnson to Monsanto, learn from them, write research and reports, and also do things like inquiries and speaking engagements and that kind of stuff.So, the idea of industry analysts is that they're neutral, they're objective. You can go to them for advice, and they can tell you, you know, these are the shortlist of vendors you should consider and this is what you should look for in a solution.Corey: I love the idea of what that role is, but it took me a while as a condescending engineer to really wrap my head around it because I viewed it as oh, it's just for a cover your ass exercise so that when a big company makes a decision, they don't get yelled at later, and they said, “Well, it seemed like the right thing to do. You can't blame us.” And that is an overwhelmingly cynical perspective. But the way it was explained to me, it really was put into context—of all things—by way of using the AWS bill as a lens. There's a whole bunch of tools and scripts and whatnot on GitHub that will tell you different things about your AWS environment, and if I run them in my environment, yeah, they work super well.I run them in a client environment and the thing explodes because it's not designed to work at a scale of 10,000 instances in a single availability zone. It's not designed to do backing off so it doesn't exhaust rate limits across the board. It requires a rethinking at that scale. When you're talking about enterprise-scale, a lot of the Twitter zeitgeist, as it were, about what tools work well and what tools don't for various startups, they fail to cross over into the bowels of a regulated entity that has a bunch of other governance and management concerns that don't really apply. So, there's this idea of okay, now that we're a large, going entity with serious revenue behind this, and migrating to any of these things is a substantial lift. What is the right answer? And that is sort of how I see the role of these companies in the ecosystem playing out. Is that directionally correct?Rachel: I would definitely agree that that is directionally correct. And it was the direction that it was going when I was there at Forrester. And by the way, I've been gone from there for, I think, eight-plus years. So, you know, it's definitely evolved it this space—Corey: A lifetime in tech.Rachel: Literally feels like a lifetime. Towards the end of my time there was when we were starting to get briefings from this bookstore company—you might have heard of them—um, Amazon?Corey: Barnes and Noble.Rachel: Yes. And Barnes and Noble. Yes. So, we're starting to get briefings from Amazon, you know, about Amazon Web Services, and S3 had just been introduced. And I got really excited about Netflix and chaos engineering—this was 2012, right?—and so I did a bunch of research on chaos engineering and tried to figure out how it could apply to the enterprises.And I would, like, bring it to Capital One, and they were like, “Ya crazy.” Turns out I think I was just a little bit ahead of my time, and I'm seeing a lot more of the industry analysts now today looking at like, “Okay, well, yeah, what is Uber doing? Like, what is Netflix doing?” And figure out how that can translate to the enterprise. And it's not a one-to-one, right, just because the people and the structures and the process is so different, so the technology can't just, like, make the leap on its own. But yes, I would definitely agree with that, but it hasn't necessarily always been that way.Corey: Oh, yeah. Like, these days, we're seeing serverless adoption on some levels being driven by enterprises. I mean, Liberty Mutual is doing stuff there that is really at the avant-garde that startups are learning from. It's really neat to see that being turned on its head because you always see these big enterprises saying, “We're like a startup,” but you never see a startup saying, “We're like a big enterprise.” Because that's evocative of something that isn't generally compelling.“Well, what does that mean, exactly? You take forever to do expense reports, and then you get super finicky about it, and you have so much bureaucracy?” No, no, no, it's, “Now, that we're process bound, it's that we understand data sovereignty and things like that.” But you didn't stay there forever. You at some point decided, okay, talking to people who are working in this industry is all well and good, but time for you to go work in that industry yourself. And you went to, I believe, NetApp by way of Riverbed.Rachel: Yes, yeah. So, I left Forrester and I went over to Riverbed to work on their cloud storage solution as a product marketing. And I had an amazing six months at Riverbed, but I happened to join, unfortunately, right around the time they were being taken private, and they ended up divesting their storage product line off to NetApp. And they divested some of their other product lines to some other companies as part of the whole deal going private. So, it was a short stint at Riverbed, although I've met some people that I've stayed in touch with and are still my friends, you know, many years later.And so, yeah, ended up over at NetApp. And it wasn't necessarily what I had initially planned for, but it was a really fun opportunity to take a cloud-integrated storage product—so it was an appliance that people put in their data centers; you could send backups to it, and it shipped those backups on the back end to S3 and then to Glacier when that came out—trying to make that successful in a company that was really not overly associated with cloud. That was a really fun process and a fun journey. And now I look at NetApp and where they are today, and they've acquired Spot and they've acquired CloudCheckr, and they're, like, really going all-in in public cloud. And I like to think, like, “Hey, I was in the early days of that.” But yeah, so that was an interesting time in my life for multiple reasons.Corey: Yeah, Spot was a fascinating product, and I was surprised to see it go to NetApp. It was one of those acquisitions that didn't make a whole lot of sense to me at the time. NetApp has always been one of those companies I hold in relatively high regard. Back when I was coming up in the industry, a bit before the 2012s or so, it was routinely ranked as the number one tech employer on a whole bunch of surveys. And I don't think these were the kinds of surveys you can just buy your way to the top of.People who worked there seemed genuinely happy, the technology was fantastic, and it was, for example, the one use case in which I would run a database where its data store lived on a network file system. I kept whining at the EFS people over at AWS for years that well, EFS is great and all but it's no NetApp. Then they released NetApps on tap on FSX as a first-party service, in which case, okay, thank you. You have now solved every last reservation I have around this. Onward.And I still hold the system in high regard. But it has, on some level, seen an erosion. We're no longer in a world where I am hurling big money—or medium money by enterprise standards—off to NetApp for their filers. It instead is something that the cloud providers are providing, and last time I checked, no matter how much I spend on AWS they wouldn't let me shove a NetApp filer into us-east-1 without asking some very uncomfortable questions.Rachel: Yeah. The whole storage industry is changing really quickly, and more of the traditional on-premises storage vendors have needed to adapt or… not, you know, be very successful. I think that NetApp's done a nice job of adapting in recent years. But I'd been in storage and backup for my entire career at that point, and I was like, I need to get out. I'm done with storage. I'm done with backup. I'm done with disaster recovery. I had that time; I want to go try something totally new.And that was how I ended up leaving NetApp and joining CloudHealth. Because I'd never really done the startup thing. I done a medium-sized company at Riverbed; I'd done a pretty big company at NetApp. I've always been an entrepreneur at heart. I started my first business on the playground in second grade, and it was reselling sticks of gum. Like, I would go use my allowance to buy a big pack of gum, and then I sold the sticks individually for ten cents apiece, making a killer margin. And it was a subscription, actually. [laugh].Corey: Administrations generally—at least public schools—generally tend to turn a—have a dim view of those things, as I recall from my misspent youth.Rachel: Yeah. I was shut down pretty quickly, but it was a brilliant business model. It was—so you had to join the club to even be able to buy into getting the sticks of gum. I was, you know, all over the subscription business [laugh] back then.Corey: And area I want to explore here is you mentioned that you double-majored. One of those majors was computer science—art history was sort of set aside for the moment, it doesn't really align with either direction here—then you served as a research associate turned analyst, and then you went into product marketing, which is an interesting direction to go in. Why'd you do it?Rachel: You know, product marketing and industry analysts are there's a lot of synergy; there's a lot of things that are in common between those two. And in fact, when you see people moving back and forth from the analyst world to the vendor side, a lot of the time it is to product marketing or product management. I mean, product marketing, our whole job is to take really complex technical concepts and relate them back to business concepts and make them make sense of the broader world and tell a narrative around it. That's a lot of what an analyst is doing too. So, you know, analysts are writing, they're giving public talks, they're coming up with big ideas; that's what a great product marketer is doing also.So, for me, that shift was actually very natural. And by the way, like, when I graduated from school, I knew I was never going to code for a living. I had learned all I was going to learn and I knew it wasn't for me. Huge props, like, you know, all the people that do code for a living, I knew I couldn't do it. I wasn't cut out for it.Corey: I found somewhat similar discoveries on my own journey. I can configure things for a living, it's fun, but I still need to work with people, past a certain point. I know I've talked about this before on some of these shows, but for me, when starting out independently, I sort of assumed at some level, I was going to shut it down, and well, and then I'll go back to being an SRE or managing an ops team. And it was only somewhat recently that I had the revelation that if everything that I'm building here collapses out from under me or gets acquired or whatnot and I have to go get a real job again, I'll almost certainly be doing something in the marketing space as opposed to the engineering space. And that was an interesting adjustment to my self-image as I went through it.Because I've built everything that I've been doing up until this point, aligned at… a certain level of technical delivery and building things as an engineer, admittedly a mediocre one. And it took me a fair bit of time to get, I guess, over the idea of myself in that context of, “Wow, you're not really an engineer. Are you a tech worker?” Kind of. And I sort of find myself existing in the in-between spaces.Did you have similar reticence when you went down the marketing path or was it something that you had, I guess, a more mature view of it [laugh] than I did and said, “Yeah, I see the value immediately,” whereas I had to basically be dragged there kicking and screaming?Rachel: Well, first of all, Corey, congratulations for coming to terms with the fact that you are a marketer. I saw it in you from the minute I met you, and I think I've known you since before you were famous. That's my claim to fame is that I knew you before you were famous. But for me personally, no, I didn't actually have that stigma. But that does exist in this industry.I mean, I think people are—think they look down on marketing as kind of like ugh, you know, “The product sells itself. The product markets itself. We don't need that.” But when you're on the inside, you know you can have an amazing product and if you don't position it well and if you don't message it well, it's never going to succeed.Corey: Our consulting [sub-projects 00:14:31] are basically if you bring us in, you will turn a profit on the engaging. We are selling what basically [unintelligible 00:14:37] money. It is one of the easiest ROI calculations. And it still requires a significant amount of work on positioning even on the sales process alone. There's no such thing as an easy enterprise sale.And you're right, in fact, I think the first time we met, I was still running a DevOps team at a company and I was deploying the product that you were doing marketing for. And that was quite the experience. Honestly, it was one of the—please don't take this the wrong way at all—but you were at CloudHealth at the time and the entire point was that it was effectively positioned in such a way of, right, this winds up solving a lot of the problems that we have in the AWS bill. And looking at how some of those things were working, it was this is an annoying, obnoxious problem that I wish I could pay to make someone else's problem, just to make it go away. Well, that indirectly led to exactly where we are now.And it's really been an interesting ride, just seeing how that whole thing has evolved. How did you wind up finding yourself at CloudHealth? Because after VMware, you said it was time to go to a startup. And it's interesting because I look at where you've been now, and CloudHealth itself gets dwarfed by VMware, which is sort of the exact opposite of a startup, due to the acquisition. But CloudHealth was independent for years while you were there.Rachel: Yeah, it was. I was at CloudHealth for about three-plus years before we were acquired. You know, how did I end up there? It's… it's all hazy. I was looking at a lot of startups, I was looking for, like, you know, a Series B company, about 50 people, I wanted something in the public cloud space, but not storage—if I could get away from storage that was the dream—and I met the folks from CloudHealth, and obviously, I hadn't heard about—I didn't know about cloud cost management or cloud governance or FinOps, like, none of those were things back then, but I was I just was really attracted to the vision of the founders.The founders were, you know, Joe Kinsella and Dan Phillips and Dave Eicher, and I was like, “Hey, they've built startups before. They've got a great idea.” Joe had felt this pain when he was a customer of AWS in the early days, and so I was like—Corey: As have we all.Rachel: Right?Corey: I don't think you'll find anyone in this space who hasn't been a customer in that situation and realized just how painful and maddening the whole space is.Rachel: Exactly, yeah. And he was an early customer back in, I think, 2014, 2015. So yeah, I met the team, I really believed in their vision, and I jumped in. And it was really amazing journey, and I got to build a pretty big team over time. By the time we were acquired a couple of years later, I think we were maybe three or 400 people. And actually, fun story. We were acquired the same week my son was born, so that was an exciting experience. A lot of change happened in my life all at once.But during the time there, I got to, you know, work with some really, really cool large cloud-scale organizations. And that was during that time that I started to learn more about Kubernetes and Mesos at the time, and started on the journey that led me to where I am now. But that was one of the happiest accidents, similar to the happy accident of, like, how did I end up at Forrester? Well, I didn't get the job at Google. [laugh]. How did I end up at CloudHealth? I got connected with the founders and their story was really inspiring.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: It's amusing to me the idea that, oh, you're at NetApp if you want to go do something that is absolutely not storage. Great. So, you go work at CloudHealth. You're like, “All right. Things are great.” Now, to take a big sip of scalding hot coffee and see just how big AWS billing data could possibly be. Yeah, oops, you're a storage company all over again.Some of our, honestly, our largest bills these days are RDS, Athena, and of course, S3 for all of the bills storage we wind up doing for our customers. And it is… it is not small. And that has become sort of an eye-opener for me just the fact that this is, on some level, a big data problem.Rachel: Yeah.Corey: And how do you wind up even understanding all the data that lives in just the outputs of the billing system? Which I feel is sort of a good setup for the next question of after the acquisition, you stayed at VMware for a while and then matriculated out to where you are now where you're the Head of Product and Technical Marketing at Chronosphere, which is in the observability space. How did you get there from cloud bills?Rachel: Yeah. So, it all makes sense when I piece it together in my mind. So, when I was at CloudHealth, one of the big, big pain points I was seeing from a lot of our customers was the growth in their monitoring bills. Like, they would be like, “Okay, thanks. You helped us, you know, with our EC2 reservations, and we did right-sizing, and you help with this. But, like, can you help with our Datadog bill? Like, can you help with our New Relic bill?”And that was becoming the next biggest line item for them. And in some cases, they were spending more on monitoring and APM and like, what we now call some things observability, they were spending more on that than they were on their public cloud, which is just bananas. So, I would see them making really kind of bizarre and sometimes they'd have to make choices that were really not the best choices. Like, “I guess we're not going to monitor the lab anymore. We're just going to uninstall the agents because we can't pay this anymore.”Corey: Going down from full observability into sampling. I remember that. The New Relic shuffle is what I believe we call it at the time. Let's be clear, they have since fixed a lot of their pricing challenges, but it was the idea of great suddenly we're doing a lot more staging environments, and they come knocking asking for more money but it's a—I don't need that level of visibility in the pre-prod environments, I guess. I hate doing it that way because then you have a divergence between pre-prod and actual prod. But it was economically just a challenge. Yeah, because again, when it comes to cloud, architecture and cost are really one and the same.Rachel: Exactly. And it's not so much that, like—sure, you know, you can fix the pricing model, but there's still the underlying issue of it's not black and white, right? My pre-prod data is not the same value as my prod data, so I shouldn't have to treat it the same way, shouldn't have to pay for it the same way. So, seeing that trend on the one hand, and then, on the other hand, 2017, 2018, I started working on the container cost allocation products at CloudHealth, and we were—you know, this was even before that, maybe 2017, we were arguing about, like, Mesos and Kubernetes and which one was going to be, and I got kind of—got very interested in that world.And so once again, as I was getting to the point where I was ready to leave CloudHealth, I was like, okay, there's two key things I'm seeing in the market. One is people need a change in their monitoring and observability; what they're doing now isn't working. And two, cloud-native is coming up, coming fast, and it's going to really disrupt this market. So, I went looking for someone that was at the intersection of the two. And that's when I met the team at Chronosphere, and just immediately hit it off with the founders in a similar way to where I hit it off with the founders that CloudHealth. At Chronosphere, the founders had felt pain—Corey: Team is so important in these things.Rachel: It's really the only thing to me. Like, you spend so much time at work. You need to love who you work with. You need to love your—not love them, but, you know, you need to work with people that you enjoy working with and people that you learn from.Corey: You don't have to love all your coworkers, and at best you can get away with just being civil with them, but it's so much nicer when you can have a productive, working relationship. And that is very far from we're going to go hang out, have beers after work because that leads to a monoculture. But the ability to really enjoy the people that you work with is so important and I wish that more folks paid attention to that.Rachel: Yeah, that's so important to me. And so I met the team, the team was fantastic, just incredibly smart and dedicated people. And then the technology, it makes sense. We like to joke that we're not just taking the box—the observability box—and writing Kubernetes in Crayon on the outside. It was built from the ground up for cloud-native, right?So, it's built for this speed, containers coming and going all the time, for the scale, just how much more metrics and observability data that containers emit, the interdependencies between all of your microservices and your containers, like, all of that stuff. When you combine it makes the older… let's call them legacy. It's crazy to call, like, some of these SaaS solutions legacy but they really are; they weren't built for cloud-native, they were built for VMs and a more traditional cloud infrastructure, and they're starting to fall over. So, that's how I got involved. It's actually, as we record, it's my one-year anniversary at Chronosphere. Which is, it's been a really wild year. We've grown a lot.Corey: Congratulations. I usually celebrate those by having a surprise meeting with my boss and someone I've never met before from HR. They don't offer your coffee. They have the manila envelope of doom in front of them and hold on, it's going to be a wild meeting. But on the plus side, you get to leave work early today.Rachel: So, good thing you run in your own business now, Corey.Corey: Yeah, it's way harder for me to wind up getting surprise-fired. I see it coming [laugh]—Rachel: [laugh].Corey: —aways away now, and it looks like an economic industry trend.Rachel: [sigh]. Oh, man. Well, anyhow.Corey: Selfishly, I have to ask. You spent a lot of time working in cloud cost, to a point where I learned an awful lot from you as I was exploring the space and learning as I went. And, on some level, for me at least, it's become an aspect of my identity, for better or worse. What was it like for you to leave and go into an orthogonal space? And sure, there's significant overlap, but it's a very different problem aimed at different buyers, and honestly, I think it is a more exciting problem that you are in now, from a business strategic perspective because there's a limited amount of what you can cut off that goes up theoretically to a hundred percent of the cloud bill. But getting better observability means you can accelerate your feature velocity and that turns into something rather significant rather quickly. But what was it like?Rachel: It's uncomfortable, for sure. And I tend to do this to myself. I get a little bit itchy the same way I wanted to get out of storage. It's not because there's anything wrong with storage; I just wanted to go try something different. I tend to, I guess, do this to myself every five years ago, I make a slightly orthogonal switch in the space that I'm in.And I think it's because I love learning something new. The jumping into something new and having the fresh eyes is so terrifying, but it's also really fun. And so it was really hard to leave cloud cost management. I mean, I got to Chronosphere and I was like, “Show me the cloud bill.” And I was like, “Do we have Reserved Instances?” Like, “Are we doing Committed Use Discounts with Google?”I just needed to know. And then that helped. Okay, I got a look at the cloud bill. I felt a little better. I made a few optimizations and then I got back to my actual job which was, you know, running product marketing for Chronosphere. And I still love to jump in and just make just a little recommendation here and there. Like, “Oh, I noticed the costs are creeping up on this. Did we consider this?”Corey: Oh, I still get a kick out of that where I was talking to an Amazonian whose side project was 110 bucks a month, and he's like, yeah, I don't think you could do much over here. It's like, “Mmm, I'll bet you a drink I can.”—Rachel: Challenge accepted.Corey: —it's like, “All right. You're on.” Cut it to 40 bucks. And he's like, “How did you do that?” It's because I know what I'm doing and this pattern repeats.And it's, are the architectural misconfigurations bounded by contacts that turn into so much. And I still maintain that I can look at the AWS bill for most environments for last month and have a pretty good idea, based upon nothing other than that, what's going on in the environment. It turns out that maybe that's a relatively crappy observability system when all is said and done, but it tells an awful lot. I can definitely see the appeal of wanting to get away from purely cost-driven or cost-side information and into things that give a lot more context into how things are behaving, how they're performing. I think there's been something of an industry rebrand away from monitoring, alerting, and trending over time to calling it observability.And I know that people are going to have angry opinions about that—and it's imperative that you not email me—but it all is getting down to the same thing of is my site up or down? Or in larger distributed systems, how down is it? And I still think we're learning an awful lot. I cringe at the early days of Nagios when that was what I was depending upon to tell me whether my site was up or not. And oh, yeah, turns out that when the Nagios server goes down, you have some other problems you need to think about. It became this iterative, piling up on and piling up on and piling up on until you can get sort of good at it.But the entire ecosystem around understanding what's going on in your application has just exploded since the last time I was really running production sites of any scale, in anger. So, it really would be a different world today.Rachel: It's changing so fast and that's part of what makes it really exciting. And the other big thing that I love about this is, like, this is a must-have. This is not table stakes. This is not optional. Like, a great observability solution is the difference between conquering a market or being overrun.If you look at what our founders—our founders at Chronosphere came from Uber, right? They ran the observability team at Uber. And they truly believe—and I believe them, too—that this was a competitive advantage for them. The fact that you could go to Uber and it's always up and it's always running and you know you're not going to have an issue, that became an advantage to them that helped them conquer new markets. We do the same thing for our customers. Corey: The entire idea around how these things are talked about in terms of downtime and the rest is just sort of ludicrous, on some level, because we take specific cases as industry truths. Like, I still remember, when Amazon was down one day when I was trying to buy a pair of underwear. And by that theory, it was—great, I hit a 404 page and a picture of a dog. Well, according to a lot of these industry truisms, then, well, one day a week for that entire rotation of underpants, I should have just been not wearing any. But no here in reality, I went back an hour later and bought underpants.Now, counterpoint: If every third time I wound up trying to check out at Amazon, I wound up hitting that error page, I would spend a lot more money at Target. There is a point at which repeated downtime comes at a cost. But one-offs for some businesses are just fine. Counterpoint with if Uber is down when you're trying to get a ride, well, that ride [unintelligible 00:28:36] may very well be lost for them and there is a definitive cost. No one's going to go back and click on an ad as well, for example, and Amazon is increasingly an advertising company.So, there's a lot of nuance to it. I think we can generally say that across the board, in most cases, downtime bad. But as far as how much that is and what form that looks like and what impact that has on your company, it really becomes situationally dependent.Rachel: I'm just going to gloss over the fact that you buy your underwear on Amazon and really not make any commentary on that. But I mean—Corey: They sell everything there. And the problem, of course, is the crappy counterfeit underwear under the Amazon Basics brand that they ripped off from the good underwear brands. But that's a whole ‘nother kettle of wax for a different podcast.Rachel: Yep. Once again, not making any commentary on your—on that. Sorry, I lost my train of thought. I work in my dining room. My husband, my dog are all just—welcome to pandemic life here.Corey: No, it's fair. They live there. We don't, as a general rule.Rachel: [laugh]. Very true. Yeah. You're not usually in my dining room, all of you but—oh, so uptime downtime, also not such a simple conversation, right? It's not like all of Amazon is down or all of DoorDash is down. It might just be one individual service or one individual region or something that is—Corey: One service in one subset of one availability zone. And this is the problem. People complain about the Amazon status page, but if every time something was down, it reflected there, you'd see a never ending sea of red, and that would absolutely erode confidence in the platform. Counterpoint when things are down for you and it's not red. It's maddening. And there's no good answer.Rachel: No. There's no good answer. There's no good answer. And the [laugh] yeah, the Amazon status page. And this is something I—bringing me back to my Forrester days, availability and resiliency in the cloud was one of the areas I focused on.And, you know, this was once again, early days of public cloud, but remember when Netflix went down on Christmas Eve, and—God, what year was this? Maybe… 2012, and that was the worst possible time they could have had downtime because so many people are with their families watching their Doctor Who Christmas Specials, which is what I was trying to watch at the time.Corey: Yeah, now you can't watch it. You have to actually talk to those people, and none of us can stand them. And oh, dear Lord, yeah—Rachel: What a nightmare.Corey: —brutal for the family dynamic. Observability is one of those things as well that unlike you know, the AWS bill, it's very easy to explain to people who are not deep in the space where it's, “Oh, great. Okay. So, you have a website. It goes well. Then you want—it gets slow, so you put it on two computers. Great. Now, it puts on five computers. Now, it's on 100 computers, half on the East Coast, half on the West Coast. Two of those computers are down. How do you tell?”And it turns in—like, they start to understand the idea of understanding what's going on in a complex system. “All right, how many people work at your company?” “2000,” “Great. Three laptops are broken. How do you figure out which ones are broken?” If you're one of the people with a broken laptop, how do you figure out whether it's your laptop or the entire system? And it lends itself really well to analogies, whereas if I'm not careful when I describe what I do, people think I can get them a better deal on underpants. No, not that kind of Amazon bill. I'm sorry.Rachel: [laugh]. Yeah, or they started to think that you're some kind of accountant or a tax advisor, but.Corey: Which I prefer, as opposed to people at neighborhood block parties thinking that I'm the computer guy because then it's, “Oh, I'm having trouble with the printer.” It's, “Great. Have you tried [laugh] throwing away and buying a new one? That's what I do.”Rachel: This is a huge problem I have in my life of everyone thinking I'm going to fix all of their computer and cloud things. And I come from a big tech family. My whole family is in tech, yet somehow I'm the one at family gatherings doing, “Did you turn it off and turn it back on again?” Like, somehow that's become my job.Corey: People get really annoyed when you say that and even more annoyed when it fixes the problem.Rachel: Usually does. So, the thread I wanted to pick back up on though before I got distracted by my husband and dog wandering around—at least my son is not in the room with us because he'd have a lot to say—is that the standard industry definition of observability—so once again, people are going to write to us, I'm sure; they can write to me, not you, Corey, about observability, it's just the latest buzzword. It's just monitoring, or you know—Corey: It's hipster monitoring.Rachel: Hipster monitoring. That's what you like to call it. I don't really care what we call it. The important thing is it gets us through three phases, right? The first is knowing that something is wrong. If you don't know what's wrong, how are you supposed to ever go fix it, right? So, you need to know that those three laptops are broken.The next thing is you need to know how bad is it? Like, if those three laptops are broken is the CEO, the COO, and the CRO, that's real bad. If it's three, you know, random peons in marketing, maybe not so bad. So, you need to triage, you need to understand roughly, like, the order of magnitude of it, and then you need to fix it. [laugh].Once you fix it, you can go back and then say, all right, what was the root cause of this? How do we make sure this doesn't happen again? So, the way you go through that cycle, you're going to use metrics, you might use logs, you might use traces, but that's not the definition of observability. Observability is all about getting through that, know, then triage, then fix it, then understand.Corey: I really want to thank you for taking the time to speak with me today. If people do want to learn more, give you their unfiltered opinions, where's the best place to find you?Rachel: Well, you can find me on Twitter, I'm @RachelDines. You can also email me, rachel@chronosphere.io. I hope I don't regret giving out that email address. That's a good way you can come and argue with me about what is observability. I will not be giving advice on cloud bills. For that, you should go to Corey. But yeah, that's a good way to get in touch.Corey: Thank you so much for your time. I really appreciate it.Rachel: Yeah, thank you.Corey: Rachel Dines, Head of Product and Technical Marketing at Chronosphere. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, and castigate me with an angry comment telling me that I really should have followed the thread between the obvious link between art history and AWS billing, which is almost certainly a more disturbing Caravaggio.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AlexAlex holds a Ph.D. in Computer Science and Engineering from UC San Diego, and has spent over a decade building high-performance, robust data management and processing systems. As an early member of a couple fast-growing startups, he's had the opportunity to wear a lot of different hats, serving at various times as an individual contributor, tech lead, manager, and executive. Prior to joining the Duckbill Group, Alex spent a few years as a freelance data engineering consultant, helping his clients build, manage and maintain their data infrastructure. He lives in Los Angeles, CA.Links: Twitter: https://twitter.com/alexras/ Personal page: https://alexras.info Old Consulting website with blog: https://bitsondisk.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm the chief cloud economist at The Duckbill Group, which people are generally aware of. Today, I'm joined by our most recent principal cloud economist, Alex Rasmussen. Alex, thank you for joining me today, it is a pleasure to talk to you, as if we aren't talking to each other constantly, now that you work here.Alex: Thanks, Corey. It's great being here.Corey: So, I followed a more, I'd say traditional path for a cloud economist, but given that I basically had to invent the job myself, the more common path because imagine that you start building a role from scratch and the people you wind up looking for initially look a lot like you. And that is grumpy sysadmin, historically, turned into something, kind of begrudgingly, that looks like an SRE, which I still maintain are the same thing, but it is imperative people not email me about that. Yes, I know, you work at Google. But instead, what I found during my tenure as a sysadmin, is that I was working with certain things an awful lot, like web servers, and other things almost never, like databases and data warehouses. Because if you screw up a web server, we all have a good laugh, the site's down for a couple of minutes, life goes on, you have a shame trophy on your desk if that's your corporate culture, things continue.Mess up the data severely enough, and you don't have a company anymore. So, I was always told to keep my aura away from the expensive spendy things that power a company. You are sort of the first of a cloud economist subtype that doesn't resemble that. Before you worked here, you were effectively an independent consultant working on data engineering. Before that, you had a couple of jobs, but you had gotten a PhD in computer science, which means, first, you are probably one of the people in this world most qualified to pass some crappy job interview of solving a sorting algorithm on a whiteboard, but how did you get here from where you were?Alex: Great question. So, I like to joke that I kind of went to school until somebody told me that I had to stop. And I took that and went and started—or didn't start, but I was an early engineer at a startup and then was an executive at another early-stage one, and did a little bit of everything. And went freelance, did that for a couple of years, and worked with all kinds of different companies—vast majority of those being startups—helping them with data infrastructure problems. I've done a little bit of everything throughout my career.I've been, you know, IC, manager, manager, manager, IT guy, everything in between. I think on the data side of things, it just sort of happened, to be honest with you, it kind of started with the stuff that I did for my dissertation and parlayed that into a job back when the big data wave was starting to kind of truly crest. And I've been working on data infrastructure, basically my entire career. So, it wasn't necessarily something that was intentional. I've just been kind of taking the opportunity that makes the most sense for me it kind of every juncture. And my career path has been a little bit strange, both by academic and industrial standards. But I like where I'm at and I gained something really valuable from each of those experiences. So.Corey: It's been an interesting area of I won't say weakness here, but it's definitely been a bit of a challenge when we look at an AWS environment and even talking about a typical AWS customer without thinking of any of them in particular, I can already tell you a few things are likely to be true. For example, the number one most expensive line item in their bill is going to be EC2, and compute is the thing that powers it. Now, maybe that is they're running a bunch of instances the old-fashioned way. Maybe they're running Kubernetes but that's how it shows up. There's a lot of things that could be, and we look at what rounds that out.Now, the next item down should almost certainly not be data transfer and if so we should have a conversation, but data in one form or another is very often going to be number two. And that can mean a bunch of different things, historically. It could mean, “Oh, you have a whole bunch of stuff in S3. Let's talk about access patterns. Let's talk about lifecycle policies. Let's talk about making sure the really important stuff is backed up somewhere. Maybe you want to spend more on that particular aspect of it.”If it's on EBS volumes, that's interesting and definitely worth looking into and trying to understand the context of what's going on. Periodically we'll see a whole bunch of additional charges that speak to some of that EC2 charge in the form of EMR, AWS's Elastic MapReduce, which charges a per-hour instance charge, but also charges you for the instances that are running under the hood and under the EC2 line item. So, there's a lot of data lifecycle stuff, there's a lot of data ecosystem stories, that historically we've consulted out with experts in that particular space. And that's great, but we were starting to have to drag those people in on more and more engagements as we saw them. And we realized that was really something we had to build out as a core competency for ourselves.And we started out not intending to hire for someone with that specialty, but the more we talked to you, the more it became clear that this was a very real and very growing need that we and our customers have. How closely it is what you're doing now as far as AWS bill analysis and data pattern deep-dive align with what you were doing as a freelance consultant in the space?Alex: A lot more than you might expect. You know, I think that increasingly, what you're seeing now is that a company's core differentiator is its data, right, how much of it they have, what they do with it. And so, you know, to your point, I think when you look at any company's cloud spend, it's going to be pretty heavy on the data side in terms of, like, where have you put it? What are you doing to process it? Where is it going once it's been processed? And then how is that—Corey: And data transfer is a very important first word in that two-word sequence.Alex: Oh, sure is. And so I think that, like, in a lot of ways, the way that a customer's cloud architecture looks and the way that their bill looks kind of as a consequence of that is kind of a reification in a way of the way that the data flows from one place to another and what's done with it at each step along the way. I think what complicates this is that companies that have been around for a little while have lived through this kind of very amorphous, kind of, polyglot way that we're approaching data. You know, back when I was first getting started in the big data days, it was MapReduce, MapReduce, MapReduce, right? And we quickly [crosstalk 00:07:29]—Corey: Oh, yes. The MapReduce white paper out of Google, a beautiful April Fool's Day prank that the folks at Yahoo fell for hook, line, and sinker. They wrote Hadoop, and now we're all stuck with that pattern. Great gag, they really should have clarified they were kidding. Here we are.Alex: Exactly. So—Corey: I mostly kid.Alex: No, for sure. But I think especially when it comes to data, we tend to over-index on what the large companies do and then quickly realize that we've made a mistake and correct backwards, right? So, there was this big push toward MapReduce for everything until people realize that it was just a pain in the neck to operate and to build. And so then we moved into Spark, so kind of up-leveled a little bit. And then there was this kind of explosion of NoSQL and NewSQL databases that hit the market.And MongoDB inexplicably won that war and now we're kind of in this world where everything is cloud data warehouse, right? And now we're trying to wrestle with, like, is it actually a good idea to put everything in one warehouse and have SQL be the lingua franca on top of it? But it's all changing so rapidly. And when you come into a customer that's been around for 10 or 15 years, and has, you know, been in the cloud for a substantial—Corey: Yeah, one of those ancient customers. That is—Alex: I know, right?Corey: —basically old enough to almost get a driver's license? Oh, yeah.Alex: Right. It's one of those things where it's like, “Ah, yes, in startup years, you're, like, a hundred years old,” right? But still, you know, I think you see this, kind of—I wouldn't call it a graveyard of failed experiments, right, but it's a collection of, like, “Well, we tried this, and it kind of worked and we're keeping it around because the cost of moving this stuff around—the kind of data gravity, so to speak—is high enough that we're not going to bother transitioning it over.” But then you get into this situation where you have to bend over backwards to integrate anything with anything else. And we're still kind of in the early days of fixing that.Corey: And the AWS bill pattern that we see all the time across the board of those experiments were not successful and do not need to exist, but there's no context into that. The person that set them up left five years ago, the jobs are still running on time. What's happening with them? Well, we could stop them and see who screams, but very often, that's not the right answer either.Alex: And I think there's also something to note there, too, which is like, getting rid of data is very scary, right? I mean, if you resize a Kubernetes cluster from 15 nodes to 10, nobody's going to look at you sideways. But if you go, “Hey, we're just going to drop these tables.” The immediate reaction that you get, particularly from your data science team more often than not is, “Oh, God, what if we need that?” And so the conversation never really happens, and that causes this kind of snowball of data debt that persists in some cases for many, many years.Corey: Yeah, in some cases, what I found has been successful on those big unknown questions is don't delete the data, but restrict access to it for a few weeks and see what happens. Look into it a bit and make sure that it's not like, “Oh, cool. We just did for a month, and now we don't need that data. Let's get rid of it.” And then another month goes by it's like, “So, time to report quarterly earnings. Where's the data?”Oh, dear, that's not going to go well, for anyone. And understanding what's happening, the idea of cloning a petabyte of data so you can run an experiment on it. And okay, turns out the experiment wasn't needed. Do we still need to keep all of that?Alex: Yeah.Corey: The underlying platform advancements have been helpful toward this as well, a petabyte of data now in Glacier Deep Archive cost the princely sum of a thousand bucks a month, which is pretty close to the idea of why would I ever delete data ever again? I can get it back within a day if I need it, so let's just put it there instead.Alex: Right. You know, funny story. When I was in graduate school, we were dealing with, you know, 100 terabyte datasets on the regular that we had to generate every time because we only had 200 terabytes of raw storage. [laugh]. And this was before cloud was yet mature enough that we could get the kind of performance numbers that we wanted off of it.And we would end up having to delete the input data to make room for the output data. [laugh]. And thankfully, we don't need to do that anymore. But there are a lot of, kind of, anti-patterns that arise from that too, right? If data is easy to keep around forever, it stays around forever.And if it's easy to, let's say, run a SQL command against your Snowflake instance that scans 20 terabytes of data, you're just going to do it, and the exposure of that to you is so minimal that you can end up causing a whole bunch of problems for yourself by the fact that you don't have to deal with stuff at that low-level of abstraction anymore.Corey: It's always fun watching how this stuff manifests—because I'm dipping a toe into it from time to time—the easy, naive answer that we could give every customer but we don't is, “Huh. So, you have a whole bunch of EMR stuff? Well, you know, if you migrate that into something else, you'll save a whole bunch of money on that.” With no regard for the 500 jobs that run against that EMR cluster on a consistent basis that form is a key part of business process. “Yeah, if you could just do the entire flow of how data is operated with throughout your entire business that would be swell because you can save tens of thousands of dollars a month on that.” Yeah, how about we don't suggest things that are just absolute buffoonery.Alex: Well, and it's like, you know, you hit on a good point. Like, one of my least favorite words in the English language is the word ‘just.' And you know, I spent a few years as a freelance data consultant, and you know, a lot of what I would hear sometimes from customers is, “Well, why don't we ‘just' deprecate X?”Corey: “Why don't we just—” “I'm going to stop you there because there is no ‘just.'”Alex: Exactly.Corey: There's always context that we cannot have as outsiders.Alex: Precisely. Precisely. And digging into that really is—it's the fun part of the job, but it's also the hard part of the job.Corey: Before we created The Duckbill Group, which was really when I took Mike Julian on as business partner and CEO and formed the entity, I had something in common with you; I was freelancing for a couple of years beforehand. Now, I know why I wound up deciding, all right, we're going to turn this into a company, but what was it that I guess made you decide to, you know, freelancing is all well and good, but it's time to get something that looks a lot more like a quote-unquote, “Traditional job.”Alex: So, I think, on one level, I went freelance because I wasn't exactly sure what I wanted to do next. And I knew what I was good at. I knew what I had a lot of experience at, and I thought, “Well, I can just go out and kind of find a bunch of people that are willing to hire me to do what I'm good at doing, and then maybe eventually I'll find one of them that I like enough that I'll go and work for them. Or maybe I'll come up with some kind of a business model that I can repeat enough times that I don't have to worry that I wake up tomorrow and all of my clients are gone and then I have to go live in a van down by the river.”And I think when I heard about the opening at The Duckbill Group, I had been thinking for a little while about well, this has been going fine for a long time, but effectively what I've been doing is I've been you know, a staff-level data engineer for hire. And do I want to do something more than that, you know? Do I want to do something more comp—perhaps more sophisticated or more complex than that? And I rapidly came to the conclusion that in order to do that, I would have to have sales and marketing, and I would have to, you know, spend a lot of my time bringing in business. And that's just not something that I have really any experience in or I'm any good at.And, you know, I also recognize that, you know, I'm a relatively small fish in a relatively large pond, and if I wanted to get the kind of like, large scale people, the like the big, you know, Fortune 1000 company kind of customers, they may not pay attention to somebody like me. And so I think that ultimately, what I saw with The Duckbill Group was, number one, a group of people that were strongly aligned to the way that I wanted to keep doing this sort of work, right? Cultural alignment was really strong, good people, but also, you know, you folks have a thing that you figured out, and that puts you 10 to 15 steps ahead of where I was. And I was kind of staring down the barrel that, I'm like, am I going to have to take six months not doing client work so that I can figure out how to make this business sustain? And, you know, I think that ultimately, like, I just looked at it, and I said, this just makes sense to me, like, as a next step. And so here we all are.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: It's always fun seeing how people perceive what we've done from the outside. Like, “Oh, yeah, you just stumbled right onto the thing that works, and you've just been going, like, gangbusters ever since.” Then you come aboard, it's like, “Here, look at this pile of things that didn't pan out over here.” And it's, you get to see how the sausage is made in a way that we talk about from time to time externally, but surprisingly, most of our marketing efforts aren't really focused on, “And here's this other time we screwed up as well.” And we're honest about it, but it's not sort of the thing that we promote as the core message of what we do and who we are.A question I like to ask people during job interviews, and I definitely asked you this, and I'll ask you now, which is going to probably throw some folks for a loop because who talks to their current employees like this? But what's next for you? When it comes time for you to leave the Duckbill Group, what do you want to do after this job?Alex: That's a great question. So, I mean, as we've mentioned before, you know, my career trajectory has been very weird and circuitous. And, you know, I would be lying to you if I said that I had absolute certainty about what the rest of that looks like. I've learned a few things about myself in the course of my career, such as it is. In my kind of warm, gooey center, I build stuff. Like, that is what gives me joy, it is what makes me excited to wake up in the morning.I love looking at big, complicated things, breaking them down into pieces, and figuring out how to make the pieces work in a way that makes sense. And, you know, I've spent a long time in the data ecosystem. I don't know, necessarily, if that's something that I'm going to do forever. I'm not necessarily pigeonholing myself into that part of the space just yet, but as long as I get to kind of wake up in the morning, and say, “I'm going to go and build things and it's not going to actively make the world any worse,” I'm happy with that. And so that's really—you know, might go back to freelancing, might go and join another group, another company, big small, who knows. I'm kind of leaving that up to the winds of destiny, so to speak.Corey: One thing that I have found incredi—sorry. Let me just address that first. Like that—Alex: Sure.Corey: —is the right way to think about it. My belief has always been that you don't necessarily have, like, the ten-year plan, or the five-year plan or whatever it is because that's where you're going to go so much as it gives you direction and forces you to keep moving so you don't wind up sitting in the same place for five years with one year of experience repeated five times. It helps you remember the bigger picture. Because I've always despised this fiction that we see in job interviews where average tenure in our industry is 18 to 36 months, give or take, but somehow during the interviews, we all talk like this is now your forever job, and after 25 years, you'll retire. And yeah, let's be a little more realistic than that.My question is always what is next and how can we align in a way that helps you get to what's coming? That's the purpose behind the question, and that's—the only way to make that not just a drippingly insincere question is to mean it and to continue to focus on it from time to time of, great. What are you learning what's next? Now, at the time of this recording, you've been here, I believe three weeks if I'm not mistaken?Alex: I've—this is week two for me at time of recording.Corey: Excellent. Yes, my grasp of time is sort of hazy at the best of times. I have a—I do a lot of things.Alex: For sure.Corey: But yeah, it has been an eye-opening experience for me, not because, “Oh, wow, we have an employee.” Yeah, we've done that a few times before. But rather because of your background, you are asking different questions than we typically get during onboarding. I had a blog post go out recently—or will be by the time this airs—about a question that you asked about, “Wow, onboarding into our internal account structure for AWS is way more polished than I've ever seen it before. Is that something you built in-house? What is that?”And great. Oh, terrific, I'd forgotten that this is kind of a novel thing. No. What we're using is AWS's SSO offering, which is such a well-built, polished product that I can only assume that it's under NDA because Amazonians don't talk about it ever. But it's great.It has a couple of annoyances, but beyond that, it's something that I'm a big fan of, but I'd forgotten how transformative that is, compared to the usual approach of all right, here's your username, here's a password you're going to have to change, here are your IAM credentials to store on disk forever. It's the ability to look at what we're doing through the eyes of someone who is clearly deep into the technical weeds, but not as exposed to all of the minutiae of the 300-some-odd AWS services is really a refreshing thing for all of us, just because it helps us realize what it's like to see some of this stuff for the first time, as well as gives me content ideas because if it's new to you, I promise you are not the only person who's seeing it that way. And if you don't really understand something well enough to explain it, I would argue you don't really understand the thing, so it forces me to get more awareness around exactly how different facets work. It's been an absolutely fantastic experience so far, from my perspective.Alex: Thank you. Right back at you. I mean, spending so many years working with startups, my kind of level of expected sophistication is, “I'm going to write your password on the back of a napkin. I have fifteen other things to do. Go figure it out.” And so you know, it's always nice to see—particularly players like AWS that are such 800-pound gorillas—going in and trying to uplevel that experience in a way that feels like—because I mean, like, look, AWS could keep us with the, “Here's a CSV with your username and password. Good luck, have fun.” And you know, they would still make—Corey: And they're going to have to because so much automation is built around that—Alex: Oh yeah—Corey: In so many places.Alex: —so much.Corey: It's always net-additive, they never turn anything off, which is increasingly an operational burden.Alex: Yeah, absolutely. Absolutely. But yeah, it's nice to see them up-level this in a way that feels like they're paying attention to their customers' pain. And that's always nice to see.Corey: So, we met a few years ago—in the before times—at a mixer that we wound up throwing—slash meetup. It was in Southern California for some AWS event or another. You've been aware of who we are and what we do for a while now, so I'm very curious to know—and the joy of having these conversations is that I don't actually know what the answer is going to be, so this may never see the light of day if it goes to weird—Alex: [laugh].Corey: —in the wrong direction, but—no I'm kidding. What has been, I guess, the biggest points of dissonance or surprises based upon your perception of who we are and what we do externally, versus joining and seeing how the sausage is made?Alex: You know, I think the first thing is—um, well, how to put this. I think that a lot of what I was expecting, given how much work you all do and how big—well, ‘you all;' we do—and how big the list of clients is and how it gets bigger every day, I was expecting this to be, like, this very hyper put together, like, every little detail has been figured out kind of engagement where I would have to figure out how you all do this. And coming in and realizing that a lot of it is just having a lot of in-depth knowledge born from experience of a bunch of stuff inside of this ecosystem, and then the rest of it is kind of free jazz, is kind of encouraging. Because as someone that was you know, as a freelancer, right, who do you see, right? You see people who have big public presences or people who are giant firms, right?On the GCP side, SADA Systems is a great example. They're another local company for me here in Los Angeles, and—Corey: Oh, yes. [unintelligible 00:24:48] Miles has been a recurring guest on the show.Alex: Yeah. And he's great. And, like, they have this enormous company that's got, like, all these different specializations and they're basically kind of like the middleman for GCP on a lot of things. And, like, you see that, and then you kind of see the individual people that are like, “Yeah, you know, I'm not really going to tell you that I only have two clients and that if both of them go away, I'm screwed, but, like, I only have two clients, and if both of them go away, I'm screwed.” And so, you know, I think honestly seeing that, like, what you've built so far and what I hope to help you continue to build is, you know, you've got just enough structure around the thing so that it makes sense, and the rest of it, you're kind of admitting that no plan ever survives contact with the client, right, and that everybody's going to be different than that everybody's problems are going to be different.And that you can't just go in and say, “Here's a dashboard, here's a calculator, have fun, give me my money,” right? Because that feels like—in optimization spaces of any kind, be that cloud, or data or whatever, there's this, kind of, push toward, how do I automate myself out of a job, and the realization that you can't for something like this, and that ultimately, like, you're just going to have to go with what you know, is something that I kind of had a suspicion was the case, but this really made it clear to me that, like, oh, this is actually a reasonable way of going about this.Corey: We thought otherwise at one point. We thought that this was something could be easily addressed their software. We launched our DuckTools SaaS platform in beta and two months later, did the—our incredible journey has come to an end, and took it off of a public offering. Because it doesn't lend itself to solving these problems in software in any reasonable way. I am ever more convinced over time that the idea of being able to solve cloud cost optimization with software at VC-scale is a red herring.And yeah, it just isn't going to work because it's one size fits some. Our customers are, by definition, exceptional in many respects, and understanding the context behind why things are the way that they are mean that we can only go so far with process because then it becomes a let's have a conversation and let's be human. Otherwise, we try to overly codify the process, and congratulations, we just now look like really crappy software, but expensive because it's all people doing it. It doesn't work that way. We have tools internally that help smooth over a lot of those edges, but by and large, people who are capable of performing at especially at the principal level for a cloud economics role, inherently are going to find themselves stifled by too much process because they need to have the freedom to dig into the areas that are relevant to the customer.It's why we can't recraft all of our statements of work in ways that tend to shy away from explicitly defined deliverables. Because we deliver an outcome, but it's going to depend entirely, in most cases, up on what we discover along the way. Maybe a full-on report isn't the best way of presenting the data in the way that we see it. Maybe it's a small proof of concept script or something like that. Maybe it's, I don't know, an interpretive dance in front of the company's board.Alex: [laugh]. Right.Corey: I'm open to exploring opportunities. But it comes down to what is right for the customer. There's a reason we only ever charge a fixed fee for these things, and it's because at that point, great, we're giving you the advice that we'd implement ourselves. We have no partnerships with any vendor in the space just to avoid bias or the perception of same. It's important that we are the authoritative source around these things.Honestly, the thing that surprised me the most about all this is how true to that vision we've stayed as we've as we flushed out what works, what doesn't. And we can distantly fail to go out of business every month. I am ecstatic about that. I expected this to wind up cratering into a mountain four months after I went freelance. Not yet.Alex: Well, I mean, I think there's another aspect of this too, right? Because I've spent a lot of my career working inside of venture capital-backed companies. And there's a lot of positive things to be said about having ready access to that kind of cash, but it does something to your business the second you take it. And I've been in a couple of situations where, like, once you actually have that big bucket of money, the incentive is grow, right? Hire more people get more customers, go, go, go, go, go.And sometimes what you'll find is that you'll spend the time and the money on an initiative and it's clearly not working. And you just kind of have to keep doubling down because now you've got customers that are using this thing and now you have to maintain it, and before you know it, you've got this albatross hanging around your neck. And like one of the things that I really respect about the way that Duckbill Group is is handling this by not taking outside cash is, like, it frees you up to make these kinds of bets, and then two months later say, “Well, that didn't work,” and try something else. And you know, that's very difficult to do once you have to go and convince someone with, you know, money flowing out of their ears, that that's the right thing to do.Corey: We have to be intentional about what we're doing. One of the benefits of bringing you aboard is that one, it does improve our capacity for handling more engagements at the same time, but it also improves the quality of the engagements that we are delivering. Instead of basically doing a round-robin assignment policy we can—Alex: Right.Corey: —we consult with each other; we talk about specific areas in which we have specific expertise. You get dragged into a lot of data portions of existing engagements, and the rest of us get pulled into other areas in which you might not be as strong. For example, “What are all of these ridiculous services? I can't make heads or tails have the ridiculous naming side of it.” Surprise, that's not a you problem.It comes down to being able to work collaboratively and let each other shine in a way that doesn't mean we load people up with work. We're very strict about having a 40-hour or less work week, just because we're not rushing for an exit. We want to enjoy our time working, we want to enjoy what we're doing, and then we want to go home and don't think about work until it's time to come back and think about these things. Like, it's a lifestyle company, but that lifestyle doesn't need to be run, run, run, run, run all the time, and it doesn't need to be something that people barely tolerate.Alex: Yeah. And I think that, you know, especially coming from being an army of one in a lot of engagements, it is really refreshing to be able to—see because, you know, I'm fortunate enough, I have friends in the industry that I can go and say like, “I have no idea how to make heads or tails of X.” And you know, I can get help that way, but ultimately, like, the only other outlet that I have here is the customer and they're not bringing me in if they have those answers readily to hand. And so being able to bounce stuff off of other people inside of an organization like this has been really refreshing.Corey: One of the things I've appreciated about your tenure here so far is the questions that you ask are pitched at the perfect level, by which I mean, it is never something you could answer with a three-second visit to Google, but it's also not something that you've spent three days spinning your wheels on trying to understand. You do a bit of digging; it's a little unclear, especially since there are multiple paths to go down, and then you flag it for clarification. And there's really so much to be said for that. Really, when we're looking for markers of seniority in the interview process, it's admitting you don't know something, but then also talking about how you would go about getting the answer. And it's—because no one has all this stuff in their head. I spend a disturbing amount of time looking at search engines and trying to reformulate queries and to get answers that make sense.I don't have the entirety of AWS shoved into my head. Yet. I'm sure there's something at re:Invent that's going to be scary and horrifying that will claim to do it and basically have a poor user interface, but all right. When that comes, we'll reevaluate then because this industry is always changing.Alex: For sure. For sure. And I think it's, it's worth pointing out that, like, one of the things that having done this for a long time gives you is this kind of scaffolding in your head that you can hang things over. We're like, you don't need to have every single AWS service memorized, but if you've got that scaffold in your head going, “Oh, like, this thing sounds like it hangs over this part of the mental scaffold, and I've seen other things that do that, so I wonder if it does this and this and this,” right? And that's a lot of it, honestly.Because especially, like, when I was solely in the data space, there's a new data wareho—or a new, like, data catalog system coming out every other week. You know, there are a thousand different things that claim to do MLOps, right? And whenever, like, someone comes to me and says, “Do you have experience with such and such?” And the answer was usually, “Well if you hum a few bars, I can fake it.” And, you know, that tends to help a great deal.Corey: Yeah. “No, but I'll find out and get back to you,” the right answer. Making it up and being wrong is the best way to get rejected from an environment. That's not just consulting; that's employment, too. If 95% of the time, you give the right answer, but that one time and 20 you're going to just make it up, well, I have to validate the other 19 because I never know when someone's faking it or not. There's that level of earned trust that's important.Alex: Well, yeah. And you're being brought in to be the expert in the room. That doesn't necessarily mean that you are the all-seeing, all-knowing oracle of knowledge but, like, if you say a thing, people are just going to believe you. And so, you know, it's beholden on you—Corey: If not, we have a different problem.Alex: Well, yeah, exactly. Hopefully, right? But yeah, I mean, it's beholden on you to be honest with your customer at a certain point, I think.Corey: I really want to thank you for taking the time out of your day to got with me about this. And I would love to have you back on in a couple of months once you're fully up to speed and spinning at the proper RPMs and see what's happened then. I—Alex: Thank you. I'd—Corey: —really appreciate—Alex: —love to.Corey: —your time where's the best place for people to learn more about you if they haven't heard your name before?Alex: Well, let's see. I am @alexras on Twitter, A-L-E-X-R-A-S. My personal website is alexras.info.I've done some writing on data stuff, including a pretty big collection of blog posts on the data side of the AWS ecosystem that are still on my consulting page, bitsondisk.com. Other than that—I mean, yeah, Twitter is probably the best place to find me, so if you want to talk more about any weird, nerd data stuff, then please feel free to reach out there.Corey: And links to that will, of course, be in the [show notes 00:35:57]. Thanks again for your time. I really appreciate it.Alex: Thank you. It's been a pleasure.Corey: Alex Rasmussen, principal cloud economist here at The Duckbill Group. I am Corey Quinn, cloud economist to the stars, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that you then submit to three other podcast platforms just to make sure you have a backup copy of that particular piece of data.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About DavidDavid is an AWS expert who likes to design and build scalable solutions that are fully automated and take care of themselves. Now he is focusing on selling his own products on the AWS Marketplace.Links: 0x4447: https://0x4447.com/ Products page: https://products.0x4447.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is brought to us by 0x4447. And my guest today is David Gatti, their CEO. David, thank you for taking the time to speak with me today.David: Thank you for getting me on the show.Corey: One of the things that I find fascinating about what you do and where you come from is that for the last five years, you've been running an independent company that I would classify based upon our conversations as pretty close to a consultancy. However, you've gone down the path that I didn't when I set up my own consultancy, and started actually selling software—not just software: Solutions—as a packaged thing that you can wind up doling out to various customers, whereas I just went with the very high touch approach of, “Oh, let me come in and have a whole series of conversations with people.” Your scale is a heck of a lot more. So, do you view yourself these days as a software company, as a consultancy, or something else entirely?David: So, right now, I did put aside the consultancy because yeah, one thing that I realized, it's possible but it's very hard to scale, it's also hard to find people at the same level. So yeah, the scalability of the business is quite hard, whereas with software sold on the AWS Marketplace, that is much easier to scale than what I was doing before, and that's why I decided to take a break from consulting and focusing one hundred percent on the products that I sell on the AWS Marketplace to see how this goes and how it actually works, and can a business be built around it.Corey: The common wisdom that I've encountered is that consulting, especially when you're doing it yourself, is one of those things that is terrific when you find yourself in the position that I originally did of your employer showing up and, “Knock, knock,” “Who's there?” “Not you anymore. Get out.” And there's a somewhat, in my case, limited runway as far as how long I've got before I have to go find another job. With consulting, you can effectively go out and start talking to people, and provided that you can land a project, it starts throwing off revenue, basically immediately, whereas building software, building packages, things that you end up selling to people, it's almost like a real estate business on some level, where you have to take a lot of investment up front to wind up building the thing, where—because no one is, generally speaking, going to pay you spec work to go ahead and build something for 18 months and come back and hope that it works.David: Right.Corey: I also bias towards the services because I'm bad at writing code. You, on the other hand, write things that seem to actually work, which is another refreshing difference.David: Yes. So, I did that, but now I have a guy that is just a Linux expert. So, you were saying that there is a high investment in the beginning, but what actually—in my case what happened, I've been selling these products for the past three years basically as a hobby. So, when I was doing AWS consulting, I was seeing, like, a company has a problem, a repeating problem, so I was just creating a product, putting it on the Marketplace, and then sending it to them. So basically, they had a situation where I can manage those projects to update when there's a need to do an update, and there was always a standardization behind that, right?So, if they had, you know, five SFTP servers, and there was a need to make an update, I was making the update on my image, putting it on the Marketplace, and then updating all those servers in one go in a much quicker fashion then managing them one by one, right? And so I had this thing for three years. So now, when I started doing this full-time, I have a little bit of a leap on what's going on. So, I already had a bunch of clients that are using their products, so that actually helped me not to have to wait three years before I saw any revenue coming in.Corey: I always thought that the challenge behind building something like this was that well, you needed to actually be conversant in a programming language; that was the thing that you needed to package and build these things. But I take a look at what you have on the AWS Marketplace—and I will throw a link to this in the [show notes 00:04:39]—but you offer right now four different offerings: A Rsyslog server, a Samba server, VPN server, and an SFTP server, and every one of those four things, back in my DevOps days, I built and implemented on AWS, generally either from scratch or from something in the Marketplace—and I'll get to that in a bit—that didn't really meet a variety of needs. And every single time I built these things, it drove me up a wall because I had to do this without, like, solving a global problem locally, myself, to meet some pile of needs, then I had to worry about the maintenance of the thing, making sure that the care and feeding continued to work. And it just wasn't—it didn't work for me in the way that I wanted it to. It never occurred to me that I really could have just solved this whole thing once, [unintelligible 00:05:28] it on the Marketplace, and then just gone and grabbed the thing.David: Exactly. So, that was my exact thinking here. Especially when your work with the client, this [unintelligible 00:05:38] was also great [idea 00:05:39] because when you work with clients, they want to do things as fast as possible, right? So, can they say, “I need an SFTP server?” Of course, it takes, you know, half a day to set up something, but then they scream at you and say, like, “Hey, do the next thing. Do the next thing. Do the next thing.” And you never end up configuring the server that you're making a reliable way, sometimes you misconfigure it because, oh I forgot this option, and now everybody on the internet can access the server itself.Corey: Wait, screw up a server config? That doesn't sound like something I would do.David: Well, of course not.Corey: Yeah, no one [unintelligible 00:06:08] they're going to until oops.David: Yes. You're amazing and you're perfect, of course, but I'm not. And I was seeing, like, oh, you know, in the middle of the night, oh, I forgot this option. I forgot this. I forgot that.And so there was never a, basically, one place when the configuration just correct, right? And that was something that sparked my idea when I realized the Marketplace exists. It's like, oh, wait a moment, I can spend few weeks to do it, right, put it there and never worry about it again. And so if when a client says like, “Hey, I need this,” I can deploy it literally, in less than one minute. You have any of those products that actually I'm selling up and running, right?And of course, the VPN is going to be a little bit slower because it needs to generate all the certificates at the beginning, but for example, the SFTP one is just poof, you're deployment with our CloudFormation file, provide username and password, and you're up and running. And I see, for example, this thing with clients, which sometimes it's funny, when there's two clients that they use the SFTP server only once a day for one hour. So, every day is like one new instance created, then one instance removed, and one instance created and one instance removed. And so it keeps on going like that.Corey: The thing that always drove me nuts about building these things out was first I had to go and find something on those rare occasions where I used the Marketplace. Again, I wasn't really working in the same modern Marketplace that we think of today when we talk about the AWS Marketplace. It was very early on, the only way that it would deliver software was via, “Here's an AMI, grab the thing, and go ahead and deploy it, and it's going to have an additional hourly cost on. It the end.” And more or less the whole Henry Ford approach of, “Oh, you can get it in any color you want, as long as it's black.”So, back in those days, I would spin up an OpenVPN server—and I did this at several companies—I would go and find the thing on the Marketplace from I think it was the OpenVPN company behind the project. Great, I grabbed the thing, it had no additional cost through the Marketplace. I then had to go and get a custom license file from the vendor themselves, load the thing in, then start provisioning users. And this had no integration that I could discern with anything else we had going on, so all of this stuff was built through the web config on this thing, there was no facility for backing the thing up—certificate, material, et cetera, et cetera—so if something happened to that instance or that image, or we had to go through a DR exercise, well, time to reprovision everyone by hand again. And it was annoying because the money didn't matter. At a company scale, it really doesn't for something like this unless you're into the usurious ranges. It does not matter.It's the, I want to manage this simply and effectively in a way that makes sense, and in many cases in a way that is congruent with our on-prem environment. So, “Oh, there's a custom AWS service that offers something kind of like this. Use that instead.” It's, yeah, I don't like the idea, personally, of having to use a higher-level managed service that I'm very often going to need the most, right when things are getting wonky during an outage scenario. I want something that I understand and can work with.And I've always liked, even if I have all the latest whiz-bang accesses into an environment, in production environments, I spin up something like this anyway, just to give myself a backdoor in the event that everything else breaks. And I really like how you've structured your VPN server as far as backing up its config, sharing its configs, you can scale it to more than one instance—what a ridiculous concept that is—and so on and so forth.David: So, it's not more than one—I mean, yes, you can deploy to more than one time, but the thing that—because again, when you were saying, like, companies don't care about the cost, right? It's more about how annoying it is to use and set up, right? And so I'm one of those people that when I, for example, see things like I've been playing with servers since the '90s, right, and I was keeping rebuilding and recreating everything every single time from scratch.And, yeah, it was always painful. It took always a lot of time. For example, our server took six months to set up the right way. And also the pricing [unintelligible 00:10:11] the competition has is quite aggravating, I will say. Like, it's very hard to scale above a certain point, especially for the midsize companies.And the goal with the Marketplace is also, like, make it as simple as possible. Because AWS itself doesn't make it easy to be on the Marketplace, and it's almost, like, crazy how hard it is. So, for anybody who will like to—who might think, like, “Oh, I would like to try this AWS Marketplace thing,” I would say should do it, but be super patient. You cannot rush it because it's going to take you on average six months to understand how even the process of uploading anything and updating it and managing it is going to take it because their website that they've built has nothing to do with the console and it's a completely custom solution that is very clunky and still very old-fashioned, how you have to manage it.Corey: Tell me more about that. I've never gone through the process of putting something up on the Marketplace. To my understanding, you need to be an AWS partner in order to use the Marketplace, correct?David: No you don't have to.Corey: Okay.David: No. Thankfully not. I hope it's not going to do this thing is not going to change. [crosstalk 00:11:20]—Corey: Yeah. I wound up manifesting it into existence by saying that. Yeah. If you're on the Marketplace team listening to this, don't do that, please. I really don't want to get yelled at and have made things worse for people.David: Don't give them ideas. [laugh]. Okay?Corey: Exactly.David: No, it's anybody can do it. But yeah, how to add a new product. So, the process is you have to build an AMI first. And then you have to submit the AMI to AWS by first creating a special AMI role—sorry, I always get confused AMI, [IAM 00:11:51], I never—IAM is users. Okay.Corey: I think we have a few more acronyms that use most of the same letters. I think that's the right answer here.David: [laugh]. So, either IAM or AMI, whichever is responsible for roles, you have to create a special role to give AWS access to your AMI. Then you submit the image to AWS providing the role that they have to use. They scan it and they do simple checks to make sure that you don't for example, have SSH enabled with regular users, do some regular scanning to make sure that you're not using an image from ten years ago, right, of Linux. And once you pass that, you are able to actually create your first product.Then you have to write your title, description provide, for example, the ports that needs to be open, the URLs to separate resources, the pricing page, which takes on average one hour to fill up because let's say that you have 20 instances that you support, and for every instance, you have to write the price for that instance per one hour. Then if you want to have a discount of let's say 20%—because you can set it by the hour, or someone can pay you for the full year. And so for the full year, you might have a discount. So, you have to have also the price per hour discounted by the amount of percentage that you want, and then you have to repeat it 40 times. Because there is no way to upload that.Corey: That feels like the internal AWS billing system in some respects. “Well, if it's good enough for us it good enough for our customers.” And—David: [laugh]. Exactly.Corey: —now, I have empathy for the folks in the billing system internally; their job is very hard, but that doesn't mean that it's okay to wind up exposing those sharp edges to folks who are, you know, paying customers of these things.David: Right. And it'd be a simple thing like being able to import the CSV file with just two columns and that would be perfect. But no, you have to do it by hand. There is no other way. So hopefully—Corey: Or someone has to. Welcome to the crappiest internship of your life.David: Exactly.Corey: It feels like bringing people into data entry for stuff like that is cheating.David: Exactly. So, you do that and then I don't remember exactly what the other steps are to a new creating a completely new product because I did that three years ago, and so now, I'm been just updating those products, but yeah, then they have to review your submission, and once everything is okay, then your product is on the Marketplace, and you can—are already accept everything. If you, for example, want to have the image also available in some specific regions that are not the default ones, you have to enable this by hand. I don't remember anymore how, but it's not obvious.Corey: And you have to keep redoing this every time they launch a new region as well, I would imagine.David: So, they say that you can have enabled the option to automatically add it, but it still won't work. Well, it will work, but… let's say, so in my case, I'm using CloudFormation. I gave a complimentary CloudFormation file where if you want to deploy my product, you go to the documentation page, you click the orange button, and you basically provide the parameters, and you click next, next, next and the product is deployed within a few minutes.And in that CloudFormation file, I have a map of every AMI in every region. Okay? So, if they add a new region and they automatically add the AMI there, then if you don't get notified that there is a new region, you don't know that you have to update the CloudFormation file, and then someone might say, like, “Hey, David, why this product is not deployed in this region.” It's like, “Oops. I didn't know that they have to update the CloudFormation file with a new region.” Right?Corey: Yeah, I'm a big believer in ClickOps, the idea of doing things in the console, but everything you're talking about sounds like a fraught enough process that I'm guessing you have some form of automation that helps you with a lot of this.David: Yeah. So, I hate repeating anything more than once, so everything in my book is automated as much as possible. The documentation, for example, how I structure it, there is a section that tells you how to deploy it by just using CloudFormation file and clicking next, next, next, next until you have it. And then there's also the option if you want to deploy manually because you don't trust what the CloudFormation file is doing, right? Of course, you can see the source file if you wanted to, but sometimes people are a little bit wary about big CloudFormation files.In any case, I have this option, but they have this option as a separate thing. So, AWS has an option where you could add a CloudFormation file that goes with your product. The problem is to be able to submit a CloudFormation file natively so they will take care of it requires you to get Microsoft Office 365. Because they give you an Excel file that has, I think, a few thousand columns. And for example, numbers under [unintelligible 00:16:40], when you export, you save the final—or sorry, you export it, it will cut around 500 columns. So, you miss, like, two-thirds of what AWS will likely to send you. And why they do that, I have no idea. I don't know if they still do it after three years, but when I was doing it, they told me like, “Hey, this is the file. Fill it by hand.”Corey: About that time period, that was exactly how they did large-scale corporate discounts on custom contracts is that they would edit the AWS bill in Excel, or if not, the next closest thing to it because there were periodically errors that looked an awful lot like someone typo-ing something by hand.David: What—Corey: Computers are generally bad doing that, and it took an extra couple of weeks to get those bills, which is right around the speed of human.David: Wow.Corey: I see none of those problems anymore, which tells me, that's right, someone finally upgraded off of Microsoft Excel to the new level. Probably Airtable.David: [laugh]. Maybe. So, I don't know if that process is still there, but what they did, like, then I realized, oh, wait a moment, I can just have a CloudFormation file in S3 bucket publicly available and just use that instead of going through that process. Because I didn't want to pay on a yearly basis for a product that I'm going to use literally once a year. That didn't make any sense to me and so I decided I'm going to do it this way. That's why, yeah, if they add on a new region, I have to go out and update my own CloudFormation file because I maintain that myself, whereas they would maintain it for me, I guess.Corey: The way that I see all of the nuts and bolts of the engineering parts of getting all these things up and running on the Marketplace, it feels like it is finicky; it is sharp edges that AWS is basically known for in many respects, but without the impetus of making that meaningfully better, just because there's such an overriding business reason, that—it's not like there's a good competitor for something like this. So, if you want to sell things to AWS people in most frictionless way possible, it reflects on the AWS bill, causes discounting, counts for their spend commitments, and the rest, it's really the AWS Marketplace is the only game in town for a lot of that.David: Right. So, I don't know if they don't do it because they don't have enough competition or pressure because to me when I first started doing this AWS Marketplace, it felt to me like more Amazon than AWS, right? It feels more like an Amazon team was behind it and not people from AWS itself. It felt like completely something different. Not to mention, yeah, the console that they provide is something completely custom that has nothing to do with the typical AWS console.Corey: I've heard stories about the underpants store division's seller tools as well; very similar to the experience you're describing.David: Mmm. And also the support is different. So, it's not connected to the AWS console one. The good thing about it, it's free, but it's also only by email. And so yeah, it's a very weird, clunky situation where I mean, I'm someone that, I guess, loves the pain of AWS. [laugh].I don't know if that's a good thing or a bad thing. But when I started, I decided, you know what, I'm going to figure it out, and once I do, I'm going to feel happy that I was able to. Maybe that's their goal: It's to give us purpose in life. So, maybe that's the goal of AWS. I don't know.Corey: There are times I really wonder about that where it feels like it could be so much more than it is, but it's not. And, again, my experience with it is very similar to what you've described, where it's buying an AMI, the end. But now they're talking about selling SaaS subscriptions on it, they're talking about selling professional services—in some cases—on it. And effectively, it almost feels like it's trying to become the Marketplace through which all IT transacting starts to happen. And the tailwind that sort of is giving energy to a lot of those efforts is, if you have a multimillion-dollar spend commitment with AWS in return for discounting, you have to make sure you spend enough within the timeframe, 50% of all spend on the AWS Marketplace counts toward that.Now, other cloud providers, it's 100% of spend, but you know, AWS is nothing if not very tight with the dollar. So okay, fine, whatever. There's a reason for companies to go down that path. Talk to me a little bit about the business aspect of it because for me, it seems like the clear win, in the absence of anything else is—especially at larger companies—they already have a business relationship with AWS. The value to someone selling software on the Marketplace feels like it would be, first and foremost, an end-run around companies procurement departments.It's just oh, someone has to click a button and they're up and running, as opposed to going through the entire onboarding and contracting and all the rest, manual way. Other than the technical challenges of getting things up and running on it, how have you found that it works as far as getting in front of additional customers, as far as driving adoption? You could theoretically have—I imagine—have not gone down the Marketplace route at all and just sold this directly on your website, click here to buy a license file the way that a lot of stuff I used to as well, and would have cut out a lot of the painful building an AMI and putting it into the Marketplace story. What's the value to being in the Marketplace?David: Yeah, so in the beginning, the value was basically that it's on the Marketplace, as I was saying, I was using it with pre-existing clients, so it was easy for me because I knew AWS images were there. So, it was easy to just click my own CloudFormation file and tell the client after one minute, “Hey, it's up and running. You have a bunch of profiles for your VPN. Enjoy and have fun.” Right?That experience, once you have it on the Marketplace, it's nice because it just works. And you don't have to do much work. Then I realized that AWS, in the search bar in the console, when you were typing, for example, you know, you type EC2, S3, CloudFormation, to find the service, what they were doing originally is when you were typing in the search bar, you were getting the services of AWS, and then when there was nothing left, they were showing the results of the Marketplace, which was basically amazing because you have primetime in the console with your product, you had to do zero marketing, and you get every week, took new clients that are using our product. And the trend was growing pretty, pretty well.And that was a proposition that is just amazing. Like, nobody has that because you can have Fortune 500 companies using our product without doing anything. It just—is it simple to deploy? Yes. Does it provide value? Is the price great? And people were just using them. Fast forward now; what happened is AWS changed the console. And instead of showing, after the services, the Marketplace, like, now they show the sub-section of the services, they show the results from the blog, the articles, videos, whatever, I don't even know what they've put there—Corey: Originally, you could search my name in that search bar, and it would pop up a profile of me they did for re:Inforce in the security blog.David: [laugh]. There you go.Corey: “Meet Corey Quinn. A ‘cloud economist'—scare quotes and all—who does not work here. And it was glorious. Now, they've changed the algorithm so it pops up. “Oh, you want Corey Quinn, you must mean IoT Core.” So, that blog post is still there, but it's below the fold because of course they give precedence to a service that they have that nobody uses or understands. Because, Amazon.David: Yeah, of course. And so that was awful because suddenly I realized that, oh, I'm getting less and less new clients because you know, after six months, one year, people are shutting off their things because they're finished using them, and I will not getting new ones. But at that time, I was doing [AWS 00:24:06] consulting, so it's like, oh, maybe it was a glitch in the Matrix, whatever. I got lucky.But then after a few months, I realized, wait a moment. When I was working in AWS, I realized that the console results changed, and I went like, oh, that's what happened and that's why I'm getting less clients, right? So, in the beginning, that was a great thing and that's why I'm actually paying you to promote my business and my products because now there is no way to put the products in front of customers because AWS took it away. And so that's why I decided to actually go full-force on this to make sure that I promote as much as possible because that one cool feature that AWS was providing, they took it away for whatever reason because blog posts are more important than their partners, [laugh] I guess.Corey: Well, it depends on the partner and the tier of partner, and it feels like it's a matter—to be clear, full disclosure: I am not an AWS partner; I'm not partnered with any vendor in this space, for either real or perceived conflict of interest issues, so I don't have a particular horse in the race. But back when there were a small number of partners, the network really worked. Now, there are tens of thousands of partners, and well, what winds up being surfaced? Customers, as a result seem to be caring less about various partner statuses, unless they're trying to check a box on some contractual requirement. Instead, they just want the problem solved, and it's becoming increasingly challenging to differentiate just by the nature of how this works.I don't believe, in 2022, that you could build almost anything, and put it on the AWS Marketplace in isolation and expect that to suddenly drive adoption by the fact that you're there. It feels, to me, at least on the other side of the fence, that the Marketplace experience is all about, you go there and you look for the name of the thing that you already know that you want because you've heard about it from other means, and then you just click it and you go, and that's the end of it. It's a procurement story; it's not a discoverability story.David: Right. And yeah, so that's sort of a bit disappointing, and I even made a post on Reddit about it to just bring this up to AWS itself to say, like, “Hey, UI change is pretty severe.” Because I mean, they get a percentage of every hour, the products are running, so basically they shoot themselves in the foot by making less money because now they're getting less products are being shown to potential customers. So, yeah, that's a disappointing thing.When it comes to also you ask what other way there is to show their products to potential customers, so there is an option where AWS can help you out. And when I talked to them, I think last year, they said that if you reach $2 million in sales a year, then they will basically show you around other potential customers, right? Which is a little bit disappointing because especially if you're a small company like mine, it's pretty hard to get to that $2 million in a meaningful time. And if once you reach that point, you might go like, “Hmm, how is this going to help me if you now show me in front of other people?” So yeah.And of course, I understand them in a sense that if they show a product from the Marketplace to a big company and the product turns out to be of poor quality, then of course the client is going to tell AWS like, “Why you're showing us something that just doesn't do its job?” Right? But it'd be nice to have a [unintelligible 00:27:24] when you say, “Okay, you're starting out. After a few years, so we can show you to this midsize clients.” You don't have to go to, immediately, Fortune 500 companies. That doesn't make any sense, right?Corey: And I still—even the companies that are at that level, I've talked to them about how they've grown their business, and not a single one has ever credited anything AWS did to help them grow. Other than, “Well, they threw re:Invent, so we spent extortionate piles of money and set up a booth there, and the fact that we were allowed in the building to talk to people was helpful, I guess.” But it's all through their own works on this, I'm not convinced, to be very direct with you, that AWS knows how to effectively drive sales and adoption of things on their own Marketplace. That is an increasing source of concern.David: Right. And then there's no plan of what to do with a company that is starting on the Marketplace, once it's a few—or it's already a few years and established in the Marketplace and a big one. Yeah, they don't have any way to go about it, which is a bit disappointing. But again, I like a challenge. I like the misery of AWS, so I'm just doing it. [laugh].Corey: No, I hear you. Would you recommend other people in your position explore selling on the Marketplace, given the challenges and advantages both that you've experienced?David: So, if you were to start from scratch, it will take you, like, three years—maybe not three years, but it's not something that should be the primary revenue source of the business if you want to go into the AWS Marketplace situation because you have to have enough capital to do enough marketing to see if you can get in front of people. If you already do some consulting like me, where I did some stuff on the side, and then realized, oh, people are using it, people like it, they get some feedback, the want new features, like, “Oh, maybe I can start growing this bigger and bigger, right?” It's not something that's going to happen immediately. And especially the updating process that happens, it can get quite stressful because when you make an update—so you have a version of a product that's working and running, right? Now, you make an update and you have to spend at least a week or even sometimes two weeks to test that out to make sure that you didn't miss anything because you don't want people to update something and it stops working right?Corey: You can't break customer experiences on these things.David: Yeah. No.Corey: It becomes a nightmare.David: Because especially you don't know if, literally, a Fortune 500 company is using your product or, like, a tiny company that has only ten employees, right?Corey: Your update broke the file server with a VPN means it's unlikely that they're going to come back anytime soon, too.David: Right.Corey: You're also depending on AWS, in some respects, to steward the relationship because you're you don't have direct contact with your buyers.David: No. So, that's important thing. They don't give you access to the contacts; they give you access to the company information. So, I actually do have Fortune 500 companies using my products, but yeah, there's no way to get in touch with them. The only thing that you get is the company name, the address, the domain that they used to create an email. So, at least you can get a sense of, like, who this company is.But yeah, there is no way to get in touch if there is a problem. So, the only way that you can notify the customer that there's a new update is when you make an update, there is a text area that you can say what's new, what did you change, right? And that's the only communication that you get with the client. So if, for example, you do a big mistake, [laugh], you basically have that just little text box, and hopefully, someone reads it. But you know, AWS is known for sending 20 emails a week for every account that you open. Good luck getting through that noise.Corey: Hope that you don't miss the important ones as you go through. No—David: Exactly.Corey: —I hear you. These are problems that I think are on AWS's plate to solve. Hopefully, someone over there is listening to this and will at least reach out with a bit of a better story. I really want to thank you for taking the time to speak with me today. We'll include links, of course, to this in the [show notes 00:31:09]. Where else can people find you?David: They can find us basically on the product page of what we sell. So, we have products.0x4447.com/. That's where, basically, we keep all our products. We keep updating the page to provide more information about those products, how to get in touch with us, we provide training, demos, anything that you want. It's very easy to get in touch with us instead of—sometimes when it comes to AWS. So yeah, we are out there, pretty easy to find us. The domain—the company name is so unique that you either get our website or—Corey: Easy to find on Google.David: Yeah, so we're basically—the hex editor. And that's basically it. [laugh].Corey: Excellent. Well, we'll definitely put links to that in the show [notes 00:31:50]. Thank you so much for taking the time to speak with me today. I really appreciate it.David: Thank you very much.Corey: David Gatti, CEO of 0x4447. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that makes sure to mention exactly how long you've been working on the AWS Marketplace team.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About SethSeth Vargo is an engineer at Google. Previously he worked at HashiCorp, Chef Software, CustomInk, and some Pittsburgh-based startups. He is the author of Learning Chef and is passionate about reducing inequality in technology. When he is not writing, working on open source, teaching, or speaking at conferences, Seth advises non-profits.Links:Twitter: https://twitter.com/sethvargo TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I have a return guest today, though it barely feels like it qualifies because Seth Vargo was guest number three on this podcast. I've had a couple of folks on since then, and for better or worse, I'm no longer quite as scared of the microphone as I was back in those early days. Seth, thank you for joining me.Seth: Yeah, thank you so much for having me back, Corey. Really excited to figure out whatever we're talking about today.Corey: Well, let's start there because last time we spoke, you were if memory serves a developer advocate at Google Cloud.Seth: Correct.Corey: And you've changed jobs, but not companies—but kind of companies because, welcome to large environments—but over the past few years, you have remained at Google. You are no longer at Google Cloud and you're no longer a developer advocate. In fact, your title is simply ‘Engineer at Google.' And what you've been focusing on, to my understanding, is helping Alphabet companies, namely—you know, the Alphabet, always in parentheses in journalistic styles, Google's parent company because no one thinks of it in terms of Alphabet—is—you're effectively helping companies within the conglomerate umbrella securely and privately consume public cloud.Seth: Yes, that is correct. So, I used to work in what we call the Cloud PA—PA stands for product area. Other product areas are like Chrome and Android—and I moved to the Core PA where I'm helping lead and run an initiative that, like you said, is to help Alphabet companies to, you know, securely and privately use public cloud services.Corey: So, I am going to go out on a limb because my position on multi-cloud has always been pick a cloud—I don't particularly care which one—but pick one and focus on that. I'm going to go out on a limb and presume that given that you are not at Google Cloud anymore, but you are at Google, you probably have a slight preference as far as which public cloud these various companies within the umbrella should be consuming.Seth: Yeah. I mean, obviously, I think most viewers will think the answer is GCP. And if you said GCP, you would be, like, 95% correct.Corey: Well, you'd also be slightly less than that correct, because they're doing a whole rebrand and calling it Google Cloud in public, as opposed to GCP. You really don't work for the same org anymore. You're not up-to-date on the very latest messaging talking points.Seth: I missed—ugh, there's so many TLAs that you lose all your TLAs over time.Corey: Oh, yes.Seth: So, Google Cloud would be, like, 95% correct. But what you have to really understand is, Google has its own, you know, cloud—we didn't call it a cloud at the time, you might call it on-prem or legacy infrastructure, if you will—primarily built on a scheduling system called Borg, which is like Kubernetes version zero. And a lot of the Alphabet companies have workloads that run onboard. So, we're actually talking about hybrid cloud here, which, you know, you may not think of Google is like a hybrid cloud customer, but a workload that runs on our production infrastructure called Borg that needs to interact with a workload that runs on Google Cloud, that is hybrid cloud, it's no different than a customer who has their own data center that needs peering to a public cloud provider, you know, whether that's Google Cloud, or AWS, or Azure.I think the other thing is if you look at, like, the regulatory space, particularly a lot of the Alphabet companies operate in, say, like healthcare, or finance, or FinTech, where certain countries and certain jurisdictions have regulations around, like, you must be multi-cloud. You know, some people might say that means you have to run, you know, the same instance of the same app across clouds, or some people say your data can be here, but your workloads can be over there. That's to be interpreted, but you know, I would say 95% of GCP, but there is a—or sorry, 95% is Google Cloud—Corey: There we go.Seth: But there is a small percentage that is definitely going to be other cloud providers and hybrid cloud as well.Corey: My position on multi-cloud has often—people like to throw it in my face of, “See you gave this general guidance, and therefore whenever you say something that goes against it, you're a giant phony.” And it's yeah, Twitter doesn't do so well with the nuance. My position of pick a provider and go all-in is intended as general guidance for the common case. There are exceptions to this and any individual company or customer is going to have more context than that general guidance will. So, if you say you need to be in multiple clouds for certain reasons, you're probably correct.If you say you need to be in multiple clouds because your regulator demands it, you are certainly correct. I am not arguing against that in any way. I do want to disclaim my one of my biases here as well, and that is specifically that if I were building a startup today and I were not me—by which I mean having spent ten years in the AWS ecosystem learning, not just how it works, but how it breaks because that's important in production, and you know, also having a bunch of service owners at AWS on speed dial—and I, were approaching this from the naive, I need to pick a cloud, which one would I go with, my bias is for Google Cloud. And the reason behind that is the developer experience is spectacular as the primary but not only perspective on that. So, I am curious to know that as you're helping what are effectively internal customers move to Google Cloud, is their interaction with Google Cloud as a platform the same as it would be if I as a random outside customer, were using Google Cloud? Is there a bunch of internal backchannels? “Oh, you get the good kind of internal Google Cloud that most of us don't get access to?” Or something else?Seth: Yeah, so that's a great question. So first, you know, thank you for the kind words on the developer experience—Corey: They were honest words, to be clear. Let me be very direct with you, if I thought your developer experience was trash, I might not say it outright in their effort not to be, you know, actively antagonistic to someone I'm having on the show right now, but I would not say it if I didn't believe it.Seth: Yeah. And I totally—I know you, I've known you for many years. I totally believe you. But I do thank you for saying that because that was the team that I was on before this was largely responsible for that across the platform. But back to your original question around, like, what does the support experience look like? So, it's a little bit of both.So, Alphabet companies, they get a technical account manager, very similar to how, you know, reasonable-sized spend customer would get a technical account manager. That account manager has access to the Cloud support channels. So, all that looks the same. I think we're things look a little bit different is because myself and some of our other leads came from Cloud, you know, I generally don't like this phrase, but we know people. So, we tend not to go directly to Cloud when we can, right?We want Alphabet companies to really behave and act as if they were an external entity, but we're able to help the technical account manager navigate the support process a little bit better by saying like, “You need to ask for this person,” right? You need to say these words to get in front of the right person to get this ticket assigned to the right person. So, the process is still the same, but we're able to leverage our pre-existing knowledge with Cloud. The same way, if you had a [unintelligible 00:07:45] or an ex-Googler who worked for your company, would be able to kind of help move that support process along a little bit faster.Corey: I am quite sincere when I say that this is a problem that goes far beyond simply Google. A disturbing portion of my job as a cloud economist helping my clients consists of nothing other than introducing Amazonians to one another. And these are hard problems at scale. I work at a company with a dozen people in it. And it turns out that yeah, it's pretty easy to navigate who's responsible for what. When you have a hyperscale-size company in the trillion-dollar range, a lot of that breaks down super quickly.Seth: And there's just a lot of churn at all levels of the organization. And, you know, we talked about this when I first joined the show, like, I switched roles, I used to be in Cloud, and now I'm in what we call Core. I still get people who are reaching out to me, at Google and externally, who are saying, “Oh, can you answer this question? Hey, how do I do this?” And I, you know, I've gradually over the past couple of months, you know, convinced people that I don't work on that anymore, and I try to be helpful where I can, but the—Corey: You use the old name and everything. They're eventually going to learn, right?Seth: I know. They'll be like, “What do you call this? GCP? Okay, great. We don't need you anymore.” But it's true, right? Like, there's people leave the organization, people join the organization, there's reorgs, there's strategic changes, people, you know, switch roles within the org, and all of that leads to complexity with, you know, navigating, what is the size of a small nation, in some cases.Corey: Your line in your biography says that you enable Alphabet companies to securely and privately consume public cloud. Now, that would make perfect sense and I would really have no further questions based on what we've already said, except for the words securely and privately, and I want to dive into that, first. Let's work backwards with the second one first. What is ‘privately' mean in this context?Seth: So, privately means, like, privacy-preserving for both the Alphabet company and the users or customers that they have. So, when we look at that from the perspective of the Alphabet company, that means protecting their data from the eyes of the cloud provider. So, that's things like customer-managed encryption keys, you know, bring-your-own-encryption, that's making sure that you have things like, actually, transparency so that if at any point the cloud provider is accessing your data, even for a legitimate purpose, like submitting a support ticket or something—or diagnosing a support ticket, that you have visibility into that. Then the privacy-preserving side on the Alphabet company's customers is about providing that same level of visibility to their customers as well as making sure that any data that they're storing is, you know, private, it's not accessible to certain parties, it's following whether it's like, you know, actual legislation around how long data can be persisted, things like GDPR, or if it's just a general, like, data retention, insider risk management, all of that comes into this idea of, like, building a private system or privacy-preserving system.Corey: Let's be very clear that my position on it is that Google's relationship with privacy has been somewhat challenged, in due to no small part to the sheer scale of how large Google has grown. And let's be clear, I believe firmly that at certain points of scale, yeah, you deserve elevated levels of scrutiny. That is how we want society to function, by and large. And there are times where it feels a little odd on the cloud side. For example, as the time is recording, somewhat recently, there was a bug in some of the copyright detection stuff where Google Drive would start flagging files as having copyright challenges if they contained just the character ‘1' in them.Which, okay, clearly a bug, but it was a bit of a reminder for some folks that wait, but that's right, Google does tend to scan these things. Well, when you have a bunch of end-user customers and in the ways that Google does, that stuff is baked in and it shapes how you wind up seeing things. From Amazon's perspective, historically, they basically sold books and then later underpants. And doing e-commerce transactions was basically the extent of their data work with customers. They weren't really running large-scale, file sharing systems and abilities—in collaboration suites, at least not that really had any of those pesky things called customers.So, that is not built into their approach and their needs in the same way. To be clear, I am sympathetic to the problems, but it's also… it's a challenging problem, especially as you continue to evolve and move things into cloud, you absolutely must be able to trust your cloud provider, or you should not be working on that cloud provider, has been my approach.Seth: Yeah, I mean, there's certainly things that you can do to mitigate. But in general, like, there is some level of trust, forget the data, on the availability side, right? Like when the cloud provider says, “This is our SLA.” And you agree to that SLA, like, yeah, you get money back if they mess it up, but ultimately, you're trusting them to adhere to that SLA, right? And you get recompense if they fail to do so, but that's still, like, trust—trust is far more than just on the privacy side, right? It's on… the promise on the roadmap, it's on privacy, it's on the SLA, right?Corey: Yeah. And you see that concern expressed more articulately from enterprise customers, when there's a matter of trusting companies to do what they say, such as the continued investment that Alphabet slash Google is making in Google Cloud. It's easy to take the approach of well, you've turned off a bunch of consumer services, so therefore, you're going to turn off the cloud at some point, too. No, let me be very clear, for the record, I do not believe that you are going to one day flip a switch and turn off Google Cloud. And neither do your customers.Instead, the approach, the way that enterprises express this, it's not about you flipping the switch and turning it off—that's what contracts are for—their question, and they enshrine this in contracts, in some cases, in the event, not that you turn it off, but that you fail to appropriately continue to invest in the platform. Because at enterprise scale, this is how things tend to die. It is not through flipping a switch, in most cases, it's through, “We're just going to basically mothball it, keep it more or less exactly as it is until it slowly fades into irrelevance for a long period of time.” And when you're providing the infrastructure to run things for serious institutions, that part isn't okay. And credit where due, I have seen every indication that Google means it when they say this is an area of strategic and continued ongoing focus for us as a company.Seth: Yeah, I mean, Google is heavily investing in cloud. I mean, this is a brand new group that I'm working in and we're trying to get Alphabet companies onto cloud, so obviously there's some very high-level top-down executive support for this. I will say that the—a hundred percent agree with everything you're saying—the traditional enterprise approach of build this Java app—because let's be honest, it's always Java—build this Java app, compile it into a JAR and run it forever is becoming problematic. We saw this recently with, like, the log4j—Corey: Yeah, to be in a container. What the hell?Seth: [laugh].Corey: I'm kidding. I'm kidding. Please don't send me email, whatever you do.Seth: What's a container? I'm just kidding. Like, the idea of, like, software rotting is very real and it's becoming more and more of a risk to security, to privacy, to public cloud providers, to enterprises, where when you see something like log4j happen and you can't answer the question, like, do we have any code that uses that? Like, if getting the answer to that question takes you six weeks, [sigh] boy like, a lot of stuff can happen in six weeks while that particular thing is exploited. And you know, kind of gets into software supply chain a little bit, but I do agree that, like, secure, private, and stable APIs are super important, and it's an area where Google is investing. At the same time, I think the industry is moving, the enterprise industry is moving away a little bit from set-it-and-forget-it as a strategy.Corey: I want to talk about the security portion as well as far as securely consuming public cloud goes. And let me start off with a disclaimer here because I don't want people to misconstrue what I'm about to say. If you are migrating to one of the big three cloud providers, their security will be better than anything you will be able to achieve as a company yourself. Not you personally because Google is a bit of an asterisk to that statement, given what you have been doing and have been doing since the '90s in your on-prem world with Borg and the rest, but my philosophy on the relative positioning of the security of cloud providers relative to one another has changed. I spent four months beating the crap out of Azure forever having an issue where there was control plane access and then really saying nothing about it.And after I wound up finding—the day after I put out a blog post on that topic because I was tired of the lack of response, it came out that right at the same time AWS had a very similar problem and had not said anything themselves. And they went back and forth, apparently waiting to wind up doing a release until this happened, Orca Security wound up putting one out there, and it was frustrating on a couple of levels. First, the people at both of these companies who work in security are stars. There is no argument, no bones about that. Problems are going to happen, things are going to occur as a result, and the only saving grace then is the transparency and communication around it, and there was none of it from them.I'm also more than a little bit irked that my friends at AWS were aware of this, basically watched me drag Azure for four months knowing that they'd done the same thing and never bothered to say a word. But okay, that's a choice. I've been saying for a while that of the big three, Google's security posture is the most impressive. And it used to be a slight difference. Like, you nosed ahead of AWS in that respect, not by a huge margin, but by a bit.I don't think it's nearly as close these days, in my mind, and talking to other large companies about these things, and people who are paid to worry about these things all day long, I am very far from alone in that perspective. So, I guess my question for you is, as you look at moving the workload securely to Google Cloud, it feels like security is baked into everything that all aspects of your company have done. Why is that a specific area of focus? Or is that how it gets baked into everything you folks do?Seth: So, you kind of like set up the answer for this perfectly. I swear we didn't talk about this extensively beforehand.Corey: You didn't know any of that was coming, by the way, just to be very clear here. I don't sit here and feed, “All right, I'm going to say this. And here's the right res—” No, this is an impromptu, more or less ad hoc show every time I do it.Seth: Yeah. And I'm going to preface this by saying, like, I don't want this to sound, like, egotistical, but I have never found a company that has as rigorous security and privacy policies, reviews, and procedures as Google.Corey: I thought I had and I was wrong.Seth: Yeah. And—Corey: And I have a lot of apologizing to people to do as a result of that.Seth: And honestly, every time I interact with our internal security engineering teams, or our IP protection teams, I'm that Nathan Fillion meme, where he's like, what—you know, like, “Okay, I get it. I get it.” Right?Corey: And then facepalm it, uh, I should say some—I can't—yeah. Oh, yeah.Seth: The reason that it's hard for Alphabet companies to securely and privately move to cloud specifically for security, is because Alphabet's stance is so much more rigorous than anyone else in the industry, to the point where, in some cases, even our own cloud provider doesn't meet the bar for what we require for an internal workload. And that's really what it comes down to is, like, the reason that Google is the most secure cloud is because our bar is so high that sometimes we can't even meet it.Corey: I have to assume that the correct answer on this is that you then wind up talking to those product teams and figure out how to get them to a point where they can support that bar because the alternative is effectively, it's like, “Oh, yeah, this is Google Cloud and it's absolutely right for multinational banks to use, but you know, not Google workloads. That stuff's important.” And I don't think that is necessarily how you folks tend to view these things.Seth: So, it's a bidirectional stream, right? So, a lot of it is working with a product management team to figure out where we can add these additional security properties into the system—I should say, tri-directional. The second area is where the policy is so specific to Google that Google should actually build its own layer on top of it that adds the security because it's not generally applicable to even big, huge cloud customers. And then the third area is Google's a very big company. Sometimes we didn't write stuff down, and sometimes we have policies where no one can really articulate where that policy came from.And something that's new with this approach that we're taking now is, like, we're actually trying to figure out where that policy came from, and get at the impetus of what it was trying to protect against and make sure that it's still applicable. And I don't know if you've ever worked with governments or you know, large companies, right, they have this spreadsheet of hundreds of thousands of lines—Corey: You are basically describing my client list. Please continue.Seth: I mean, like, sometimes they have to use an Access database because they exhaust the number of rows in an Excel spreadsheet. And it's just checklist upon checklist upon checklist. And that's not how Google does security, right? Security is a very all-encompassing, kind of, 360 type of thing. But we do have policies that are difficult to articulate what they're actually protecting against, and we are constantly re-evaluating those, and saying, like, “This made sense on Borg. Does it actually make sense on Cloud?” And in some cases, it may not. We get the same protections using, say, a GCP-native service, and we can omit that requirement for this particular workload.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: I think that when it comes to things like policies that are intelligently crafted around security, you folks—and to be fair, the AWS security engineers as well—have been doing it right in that, okay, we're going to build a security control to make sure that a thing can't happen. That's not enough. Then there's the defense-in-depth. Okay, let's say that control fails for some variety of ways. Here are the other things we're going to do to prevent cross-account access, for example.And that in turn, winds up continuing to feed on itself and build into a culture of assuming that you can always continue to invest in security. How far is enough? Well, for most folks, they haven't gone far enough yet.Seth: Another way to put this is like, how well do you want to sleep at night? You know, there's folks on the Google security engineering team who are so smart, and they work on, like, our offensive security team, so their full-time job is to try to hack Google and then figure out how to prevent that. And, you know, so I've read some of the reports and some of the ways they think and I'm like, “How do you… how do you pick up a mobile phone and go to like, any website confidently knowing what you know?” Right? [laugh] and like, how do you—Corey: Who said anything about confidently? Yeah.Seth: Yeah. Yeah. How do you use self-checkout at a supermarket and, like, not just, like, wear your entire full-body tinfoil hat suit? But you know, I think the bigger risk is not knowing what the risks are. And this is a lot what we're seeing in software supply chain, too, is a lot of security is around threat modeling and not checklists. But we tend to, like, gravitate toward checklists because they're concrete.But you really have to ask yourself, like, do I need the same security properties on my static blog website that is stored on an S3 bucket or a GCS bucket that's public to the internet, that I do on my credit card processing service? And a lot of times we don't treat those differently, we don't apply a different threat model to them, and then everything has to have the same level of security.Corey: And then everything is in-scope for whatever it is you're trying to defend against. And that is a short path to madness.Seth: Yes. Yes. Your static HTML files and your GCS bucket are in scope for SOC 1 and 2 because you didn't have a way to say they weren't.Corey: Yeah. You've also done some—again, the nice thing about being at a company for a while—from what I can tell, given that I've never done until I started this place—is you move around and work on different projects. You were involved as well, personally, in the exposure notifications project, the joint collaboration thing between a number of companies in the somewhat early days of the pandemic that all of our phones talk to one another and anonymously and in a privacy-preserving way, let us know that hey, by the way, someone you were in close contact with has tested positive for Covid 19 in the previous fixed period of time. What did do you do over there?Seth: Yeah, so the exposure notifications project was a joint effort, primarily between Apple and Google to use Android and iOS devices to help stop the spread of Covid or reduce the spread of Covid as much as possible. The idea being because the incubation period is roughly 14 days, at least pre-Omicron, if we could tell you hey, you might have been exposed and get you to stay at home for three or four days, self-isolate, we could dramatically reduce the spread of Covid. And we know from some of the studies that have come out of, like, the UK and European region that, like, the technology actually reduced the spread of cases by, like, fourteen-hundred percent in some cases. I was one of the tech leads for the server-side. So, the way the system works is it uses the low-energy Bluetooth on iOS and Android devices to basically broadcast random IDs.So, I know this is Screaming into the Cloud, but if we can just quickly Screaming into the Void as a rebrand—Corey: Oh, yeah.Seth: —that's basically what's happening. [laugh]. You're generating these random identifiers, and just, like, yelling them, and there's other phones out there who are listening. And they collect these we'll call RPIs—or Rolling Indicators. They have no data in them.They're like literally, like, a UUID or 32 bytes of random data, they aren't at all, like, associated with your device or your person. So, then what happens is, like, let's say you're in a supermarket, you're near someone for, you know, every so often, and your phones exchange these IDs. If you then test positive, those IDs go up to a centralized server, the server again, also has no idea who you are, so the whole thing is privacy-preserving, end-to-end, then the server basically bundles all of what we call the TEKs, or the Temporary Exposure Keys—into a tarball that go up onto a CDN, and then every night, all of the devices that are participating in EN download this into a local key match. So, at no point does the server ever know that you were in a supermarket with someone else, only your phone knows that you came in contact with this TEK in the past 14 days—or 21 days in some jurisdictions—and it'll generate an exposure notification or an exposure alert, which says, like, “Hey, in the past 14 days, you've come in contact with someone who's confirmed positive for Covid.” And then there's guidance kind of varies by state and by health jurisdiction of, like, self-isolate, or go get tested, or whatever. But the idea—Corey: Or go to the bar in some places, apparently.Seth: Oh. Yeah. The server itself is actually—there's a verification component because ideally, like, we don't want people to just be like, oh, I'm Covid positive, and then like, all their friends get an alert, right? There needs to be some kind of verification mechanism where you either have a positive test, or you have a clinician or a physician who issues you code that you can put into your app so you can then release your keys. And then there's the actual key server component, which I kind of already described.So, it's a pretty complex system and actually is entirely serverless. So, the whole thing, including all, like, background job processing, it was designed to be serverless from the beginning. Total greenfield project, right, like, nothing like this exists, so we're really fortunate there. We made some fun and interesting design decisions to keep costs down while, you know, abusing slash using some of the features of serverless like auto-scaling and, you know, being able to fan out across multiple regions and things like that—Corey: And using DNS as a database. My personal favorite approach to things?Seth: We don't use DNS as a database. We do use Postgres—Corey: A missed opportunity.Seth: —a real database. But we do use DNS, just not for storing information.Corey: So, one question I have for you is that you've been at Google for a while and you've done an awful lot of things there, but previously, you've also done things that don't really directly aligne any of this stuff going on there. You were at HashiCorp and you were at Chef, neither of whom, to my understanding are technologies that Google makes extensive use of internally for their own stuff. It seems like—and even when you're at Google, you have been continually reinventing what it is that you do. I find that admirable because very often, when you see people at a company for a protracted period of time, they sort of get more or less pigeonholed into the role that looks fairly similar from year-to-year. You've been incredibly dynamic. Was it intentional and how do you do it?Seth: So, I have a diagnosed medical condition called Career-DHD. I'm just kidding, but I do. I get bored, and it's actually something that I'm really forward with my managers about. I've always been very straight with my managers and the people I work with it, like, 8 to 12 months from now, I will be doing something different. It will be different.Corey: I wish I'd figured that out earlier on. In my case, the way that I wound up solving for that is I've got to come in, I'm going to solve a interesting problem. When I'm done with that, the consulting engagement is over and then I'm going to go away and everyone knows the score going in. Works out way better than, and then I'm going to go cause problems on purpose in other people's parts of the org because I see problems there. That was where I always went off the rails.Seth: [laugh]. Yeah, I mean, I don't take a dissimilar approach. You know, I try to find high-priority, strategic things that also align with my interest. And it's important to me that there's things that I can provide and things that I can learn. I never like to be the smartest person in the room because you shouldn't be in that room anymore; there's no one for you to learn from. And it's great to share knowledge, but—Corey: I'm not convinced I'm the smartest person in the room right now, despite the fact that right now I'm the only person in the room that I'm sitting in.Seth: I mean, that Minecraft store is pretty intelligent.Corey: I saw Chihuahua wandering around here, too, a—Seth: [laugh].Corey: —minute ago, so there is that.Seth: But, you know, I think from, like, a career advice standpoint, I tell everyone, you should interview somewhere else at least once a year. You never know what's out there, and worst-case scenario, you kept your interview skills up to date.Corey: Keeping those skills in tune is so critically important just because it's a unique skill set that, for many folks, does not have a whole lot of applicability in their day-to-day job. So, if you suddenly have to find a new job, great, you're rusty at this, it's been years, and you're trying to remember, like, okay, when someone asks you what you're looking for in your next job, they're not trying to pick a fight. Don't respond as if they were. Like, the basic stuff. It's a skill, like anything else.Seth: Yeah. And, like, the common questions like, you know, “What do you want to do with your life?” Or like, “What accomplishment are you most proud of?” Like, having those not prepared, but like knowing in general what you want to say from those is very important when you're thinking about interviewing for other jobs. But even in a big company, like the transfer process is, pretty similar for, like, applying externally to other roles; like sometimes there's interviews—Corey: Do they make you code on whiteboards to solve algorithm problems?Seth: Not me. But—Corey: Good.Seth: —in general—Corey: Google has evolved its interview process since the last time I went through that particular brand of corporate hazing. Good, good, good.Seth: Yeah. The interview process has definitely been refactored a lot, especially with Covid and remote, but also just trying to be accessible to folks. I know one of the big changes Google has made is we no longer require, like, eight congruent hours of your time. You can split interviews out over multiple days, which has been really accommodating for folks that have, you know, already have a full-time job or have family obligations at home that don't let them just, like, take eight hours away and devote a hundred percent of their time to interviews. So, I think that is, you know, not a whole lot of positive things that come out of Covid, but the flexibility with, like, interviewing has enabled more people to participate in the interview process that otherwise would not have been able to do so.Corey: And there's something to be said, for making this more accessible to folks who come from backgrounds that don't all look identical. It's incredibly important.Seth: Yep.Corey: One thing that I definitely want to make sure we get to before the end of this is something you've been talking about that's a bit orthogonal, but maybe not entirely so, which is software supply chain security. That has been a common thread of discussion in some circles for a while. What is it, for those who are unfamiliar, like me sometimes, and what does it imply?Seth: Yeah, so I mean, in the past year—but if you look back, you'll find more cases of it—. We live in a world where no company—Google, Amazon, the US government—writes every line of code that they run. And even if you do, right, even if you could find a company that doesn't rely on any external dependencies, what language are they using? Did they write that language? Okay, let's say hypothetically, you write every single line of code and you wrote your own language, and only your employees contribute to that language.What operating system are you running on? Because I guarantee you, Linus probably contributed to it, or Gates contributed to it, and they don't work for you. But let's say you wrote your own operating system, right—so we're getting into, like, crazy Google things now, right? Like, only Google would write their own programming language and their own operating system, right? Who manufactured your CPU, right? Like, did you actually—Corey: There's always dependencies all the way down. We see this sometimes with companies talk about oh, yeah, we're going to go to multiple clouds or a different clouds so that we don't get impacted if there's another AWS outage in us-east-1. Cool, great. Power to you, but are you sure your payment providers not going to go down? Are they taking a dependency on us-east-1?Great, let's say that they're not. Are you sure that their vendors who are in the critical path are also not taking critical and core dependencies on that? And are you sure that they're aware of who all of those critical dependencies and those vendors are, and so on and so forth? It is a vast interconnected web. This is a problem. Dependency sprawl is real and I don't think that there's a good way to get to the bottom of it, particularly across company boundaries like that.Seth: Yeah. And this is where if you look at the non-software supply chain, like, if you look at construction, right? If you're working with a reputable construction agency, they're actually able to tell you, given a granite countertop or, you know, a quartz countertop, from what beach and what lot on what date the grains of sand in that countertop came from. That is a reality of that industry that is natural. You think about, like, automotive, like, VIN, the Vehicle Identification Numbers, like, they tell you exactly what manufacturer, and then there's records that show you exactly what human being on the line put that particular part in that machine.And we don't have that in software today. Like, we have some, you know, bastardized versions of, like, Software Bills of Material, or SBOM, but the simple fact of the matter is like because software has grown organically and because this wasn't ingrained in software from the beginning like it was from, you know, traditional manufacturing, you're going to have an insecure software supply chain for most of my life. Now, what does that actually mean, right—insecure has this negative connotation—it means that you need to make sure that you're aware of everything that you're depending on—which is kind of what you were saying is, like, both the technical dependencies and the process or the people dependencies—and you need to have a rigorous process for how you're going to respond to these incidents. And I think log4j was a really good eye-opening moment for folks when they realized that they didn't have a way to make a large-scale dependency update across their entire fleet of applications.Corey: Because who has to do that on a consistent basis? It happens rarely, but when it happens, it's super important.Seth: But I do think that more and more, we're going to see it happened more and more frequently. And ideally, you know, my opinion is that we're going to get to a point where this is inescapable, but ideally, we get to the point where it's like, “Oh, okay, this dependency is vulnerable. I have a playbook. I follow the playbook. Everything is patched in 30 minutes or less, and I can move on with my life.” And it's not a six-week fire drill with people working late and, you know, going super crazy, trying to mitigate these issues.You know, there's a lot of work happening in this space. We have, like, SLSA, which is an open standard—SLSA—for how you declare, kind of like, your software bill of materials and things like binary authorization and attestations. There's, like, Sigstore, there's Chainguard, there's some companies evolving in this space. Every time I talk to GitHub, I tell them, I'm like, “Hey, if this VP and that VP, like, talked together and, like, worked on something, you could do something amazing in this space.” But I think it's going to be quite a while until we get to a point where we can say the software supply chain is secure.Because like I was saying at the beginning, like, until you manufacture your own CPU, like, you're dependent on Intel and AMD. And until you write your own programming language, you're dependent on Ruby, Python, Go, whatever it might be. And until you take no dependencies on some external system—which by the way, might be a bad business decision, like, if someone did the work for you already in an open-source ecosystem, it's probably a better business decision to evaluate and use that than to build it yourself. Until we have the analysis on that supply chain, and we can in a dashboard, or the click of a button, or the run of a command, very easily see the security status of our supply chain—software supply chain—and determine if a particular vulnerability is or is not relevant, I think we're still going to be in this firefighting mode for at least another couple of years.Corey: And I want to say you're wrong, but I know you're not. And that's what, I guess, keeps a lot of us awake at night for unfortunate reasons. Seth, I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place to find you?Seth: I'm on Twitter. You can find me at—Corey: I'm sorry to hear that. So, am I. It's the experience.Seth: Yeah, you can find me at @sethvargo. If you say mean and hateful things to me, I actually exercise this finger, and you can click the block button real fast. But yeah, I mean, my DMs are open. If you have any questions, comments, complaints, concerns, you can throw the complaints away and come to me for everything else.Corey: Thank you so much for being so generous with your time. I really appreciate it.Seth: Yeah, thanks for having me. It's always a pleasure.Corey: Seth Vargo, engineer at Google. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment asking how dare I malign the good name of the other cloud provider that isn't Google that also just so coincidentally happens to employ you.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About BrookeBrooke is the Head of Enablement - AI/ML and Data at Blackbook.ai, an Australian based consulting firm and AWS Partner. Brooke has degrees in Mathematics and Data Engineering and they specialise in developing technically robust solutions that help “non-data people” harness the power of AI for their industry, and communicate this effectively.Outside of their 'day job', Brooke speaks at Data, AI, Software Engineering, UX and Business conferences and events to Australian and international audiences, and has guest lectured at the University of Queensland Business School and Griffith University. Brooke is proudly a volunteer member of the Queensland National Science Week Committee, and is always on the lookout for new ways to promote STEM pathways to young people, especially young women and members of the LGBTIQA+ community from regional Australia.Links: Blackbook: https://blackbook.ai/ Twitter: https://twitter.com/brooke_jamieson TikTok: https://www.tiktok.com/@brookebytes LinkedIn: https://www.linkedin.com/in/brookejamieson/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. As my 30s draw to a close, I am basically beating myself up emotionally by making myself feel tremendously, tremendously old. And there's no better way to do that than to go on TikTok where it pops up with, “Hey if you were born before 2004”—and then I just closed the video because it's ridiculous. It's more or less of a means of self-flagellation.But there are good parts to it. One of those good parts is I get to talk to people who I don't generally encounter in other areas of the giant cloud ecosystem, and my guest today is a shining example of someone who has been very prolific on TikTok but for some reason or other, hadn't really come across my radar previously. Brooke Jamieson is the Head of Enablement of AI and machine learning at Blackbook. Brooke, thank you for joining me today.Brooke: Thanks so much for having me. Welcome to 6 a.m. in Brisbane. [laugh].Corey: It was right before the pandemic that I did my first trip to Australia, discovered that was a real place. Like, “Oh, yeah. You're going to go to give a talk in Perth. What, are you taking a connection through Narnia?” No, no, it turns out it's a real place, unlike New Zealand.Brooke: Oh, yeah. New Zealand's fake.Corey: [laugh].Brooke: I booked a conference in Portugal soon, and it's going to take me 31 hours to get there from here. So. [laugh].Corey: I remember the days of international travel. Hopefully for me, they'll come back again, sooner or later.Brooke: Fingers crossed.Corey: What really struck my notice about a lot of your content is the way that you fold multiple things together. First and foremost, you talk an awful lot about machine learning, data engineering, et cetera, and you are the second person that I've encountered that really makes me think that there is something to all of this. The first being Emily Freeman, which I've discussed on the show previously, and on Twitter, and shouting from the rooftops because she works at AWS and is able to tell the story, which basically, I think makes her a heretic compared to most folks over in that org. But there's something about making incredibly complex things easily accessible, which is hard enough in its own right, but you also managed to do it basically via short-form video on TikTok. How did you discover all this?Brooke: Yeah, I have a very strange resume. [laugh]. It is sort of a layered Venn diagram is the way I normally talk about it if I'm doing a conference talk or something. So, I studied pure maths at university the first time, and then I went back and studied data engineering after. But then I also worked in fashion as a model internationally, and then I've also worked in things like user experience, doing lots of behavioral science, and everything even design-related around that.And then I've also done lots more work into cloud and AI and everything that happens. So overall, it's just being about educating people on this. Most of my role now is educating executives and showing them how they were lied to at various conferences so that they can actually make an informed decision. Because if I go to talk to a board, I know when I leave, they're going to have a conversation about what we talked about without me in the room, and I think executives keep making terrible decisions because they can't have that conversation as a group. They don't know what to do when the tour guide isn't there anymore because they don't have a shared vocabulary or a framework to talk about what they might like to do, or what they might like to prioritize to do first, things like that.So, so much of what I do is just really helping people to understand, conceptually from a high level what they're actually trying to do, so that then they can deliver on that rather than thinking, oh, I just really saw this cool model of a specific AI thing at a conference, and it was a cool animated slide. And I would like to purchase exactly one of those for my company, thank you.Corey: It's odd because you don't have a quote-unquote, “Traditional”—if there is such a thing—DevRel role: You're not an advocate, you're not an evangelist. And none of your content and talks that I have seen have been actively selling any product, but they very much been selling ideas and concepts. And it really strikes me that you have threaded the needle beautifully as far as understanding the assignment. You're trying to cause a shift in the audience, get them to see things in a way that they don't already without trying to push a particular product or a particular solution. How much of that was happy accident and how much of that was something you set out to do intentionally?Brooke: First, thanks so much. Second of all, I think this comes from studying maths. So, the number one skill you get from doing a pure maths degree is you have a toolbox with you, and then there's a number of things in that toolbox. There's different ways you can solve problems, and usually, there's a few different ways you can solve a given problem, but you just open up your toolbox that grows over time, and you can see what you can use in there to solve a problem. So, that's really how I've continued to exist, even working in user experience roles as well, just like what elements do we have to even work with here?And I brought that with me into the cloud as well because I think the really big thing with actually selling tech products is being confident enough to know that there are a number of things you can actually use instead of your product, but if you're confident enough in the product you have, it will be the obvious solution anyway, so instead, I just get people thinking about what they actually need it for, how they could use it solving a problem and give them ideas on how to apply it. And you would know this: In cloud, there's always ten million different ways to do something. [laugh]. And it's just, instead of getting them to think—because then you just get stuck in a thought vortex about, “This one or this one?” Or, “What am I doing,” but instead latch on to an idea of what you're trying to achieve, and then work out the most optimal way to do that for your underlying infrastructure as well. And even the training of staff that you have, is really important.Corey: There's a definite idea around selling—like, I think it's called ‘solution selling.' I don't know; I don't have a background in this stuff. I've basically stumbled into it. But periodically, I'll have folks come on this show, and I'll chat with them, “So, what is the outcome we're looking to have in the audience here?” Because again, telling a story with no real target in mind doesn't always go super well. And, “Oh, I want people to sign up for my product.” “Okay, how do you envision them doing that?”And their story is to sit there and pitch the whole time, and it's, yeah, that's going to be a really bad show, and I don't want to put that out. Instead, if you're active in a particular space, my approach has always been to talk about the painful problem that you solve and allude to what you do and a bit of how you do it. If you make the audience marinate in the painful problem, the folks who are experiencing that are going to sit up and self-select of, “Ooh, that sounds a lot like the problems we have. If they're talking about this, they might have some ideas and solutions.” It's a glimpse and a hook into reaching out to find out more.And to be clear, that's not the purpose of this show, but if someone wants to pitch a particular product or service, that's the way to do it because the other stuff just doesn't work. Giving away free t-shirts, for example, okay, you'll get a bunch of people clicking links and whatnot, but you're also effectively talking to people who are super willing to spend time filling out forms and talking to people to get a free t-shirt. I don't know that for many products, that's the best way to get qualified leads in.Brooke: Yeah, it's tricky. And I think it's just because everyone's doing what everyone told them to do. I love reading really terrible sales books. I started when I was younger, just because I could see people trying to use these tactics on me; and I just wanted to know everything there was to know about what it's like to be a used car salesman in the middle of [laugh] America. And so I've read all of these things, and lots of the strategies in them, they only work if you're in a very specific area that they're actually working in, and no one's getting to the problem of how do you actually like to be sold to? How can you improve the experience?And overall, for consulting, usually, it's someone—the best end game is someone has seen you around doing other things, and then they come back and they're like, “I've got a really weird problem. I didn't even know if this is what you can do. Can you help me with this?” And that is—the best client to have, they're the best—they're so open to ideas, they trust you because they've seen you do good work over time. And you would have seen this so many times, it's about someone just come to you with a really strange problem, and it may or may not even be what you've actually helped them with.Corey: Help me understand a bit what you do as a Head of Enablement? Because I've heard the term a few different ways, always at different companies. As far as day job goes, where do you start? Where do you stop?Brooke: Yeah, it's a very fake-sounding job title.Corey: [unintelligible 00:08:53]—“Oh, what are you?” “Oh, I'm an enabler.” Like, effectively standing behind someone who's debating relapsing into something, like, “Do it. Do it. Do it.” Now, I don't imagine that's what you do. But then again, AI and ML is a weird space. Maybe it is.Brooke: Just when my friends are online shopping, and they're not sure if they should buy something. I'm the one messaging them saying, “Yes, get it.” That's me. So no, what I do is I—there's really technical people in our teams, we've got about 150 consultants across Australia, and then there's very non-technical business executives who have a problem. And if you don't have a good conduit between those two groups, the business won't get what they need, and the technical people won't have the actual brief they need to solve the problem.Because so many times people will come to us with what they think is a problem, but it's actually a symptom, not the root cause, so you just need a really good understanding of overall how businesses work, how business processes work, as well, and then also just really good user experience, information architecture knowledge to go through that. But then all of that would only work if I also had the technical underpinnings so I can then make sure we have everything we need and then communicate that to the development team to make sure that everyone's getting what they need from it. Lots of places, my job doesn't exist in a lot of companies, and that's because they just try to mash [laugh] those two groups together with varying levels of success.Corey: Or it's sales enablement of, “Here's the pitch deck you use. I'm going to build slides all day,” et cetera. “Here's what the engineers are going to babble about. When they use this phrase, go ahead and repeat this talking point and they'll shut up and go away,” is often how it manifests. And I don't get that sense from you at all.I'm going to call you out slightly on this one. The way you just describe it like, “Well, there are some very technical people, and there are some non-technical people.” And you didn't actually put yourself into either one of those categories, but let's call out a bit of background on you. You have a degree in mathematics, but that wasn't enough, so you decided to go a little more technical than that; you also have a degree in data engineering. If you're listening to this, please don't take this the wrong way—Brooke: Definitely take it the wrong way. [laugh].Corey: —but you do not present as someone who is first and foremost like, “Code speaks. Code is everything,” the stereotypical technical person who gets lost in their absolute love of the technology to the exclusion of all else. You speak in a way that makes this stuff accessible. Never once in watching any of your content, have I come away feeling dumb as a result, and that's an incredibly rare thing. But make no mistake, you are profoundly technical on these things.Brooke: Yeah, making people not feel worried is my number one marketable skill when talking to executives because executives make bad decisions when they don't know how to have that conversation. But all of that is because they've been rising up in their organization for 20 to 30 years, and they didn't ask questions early on when tech was new, and then it's gotten to a point where they feel like they can't ask questions [laugh] anymore because they're the one in charge, and they're too nervous to admit they don't understand something. So, much of what I do that is successful when talking to executives is just really making sure that I'm never out to try and look like the smart one. So, I'm not ever just flexing technical knowledge to make people think that I am the God Almighty of all things tech.I don't care about that, so it's mostly about how can I make people really comfortable with something that they've been too scared to ask about probably for quite some time? So that then they can make an informed choice on that front and so they can actually be empowered by that knowledge that they now have. They probably were too scared to ask it the whole time. But it's just a way of getting through to them. And then you get so much trust from that as well, just because, as well, I'm always very confident to tell people if they've been given the wrong information by other parties, I will absolutely tell them immediately, or if they just don't know how to give success metrics for project, so they end up just forgetting that false negatives or false positives can exist. [laugh].So, educating them, even on accuracy and recall measures and things like that, as well and doing it in a way where they don't ever feel threatened is the number one key to success that no one ever tells you about as a thing because no one wants to even admit that people could possibly be threatened by this.Corey: A lot of the content that you wind up building is aimed around career advice, particularly for folks early on in their careers. And the reason I bring that up is that you are alluding to something that I see when I interview folks all the time—I went through it myself—where there was a time you're going through a technical interview, and you get the flop sweat where I don't know the answer to this question. And there are a few things you can do: You can give up and shut down, which okay, that is in many cases are natural inclination, but not particularly helpful in those environments; you can bluff your way through the answer, which I generally don't advise because when an interviewer is asking you a technical question, it's a reasonable guess that they know the right answer; but the mark of seniority that it took me a distressingly long time to learn this is I just sometimes laugh, I say, “I have absolutely no idea, but if I had to guess…” and then I'll speculate wildly. And that, in my experience, is the mark of the kind of person you generally want to have on your team. And there are elements of what you just said, threaded throughout that entire approach of not making people feel less than.On the other side of that interview table, when I'm sitting there as a candidate. I hated those interviews where someone sits there and tries to prove they're the smartest person in the room. Yeah, I too, am the smartest person in the room when I wrote the interview questions. But for me, it's a given Tuesday; for the person I'm interviewing, it's determining the next stage of their career. There's a power imbalance there.Brooke: Yeah. And this has always happened to me in job interviews as well. I have a very polarizing resume just because it's not traditional. I didn't do software engineering at university and then work as a software engineer. I just haven't gone through that linear pathway, so there's lots of people just trying to either figure me out or get to a gotcha moment where they can really just work out what's actually happening.I have no interest in it. I'm happy to say that I don't know something. And being able to openly say that you don't know something is so helpful, as you're saying. It's the number one skill I wish people would learn because it's fine to not [laugh] understand things.Corey: A couple of times, I was the first DevOps hire in startup that was basically being interviewed by a bunch of engineers. And I went through a lot of those interviews and took the job only a couple of times, and one of the key differentiators for me was when they sat down and looked sort of sheepish and asked me a question of the form, “Look, I know how to interview a software engineer, but I sort of get the sense that you're not going to do that super well.” Yeah, surprise; I'm not a software engineer. “What is the best way to interview you to really expose where you start and where you stop?” Which I think is such a great question, if you don't know.Now, in my world, the way that—now that I'm on the other side of the table, I bring in experts to help me evaluate people, otherwise I run the very real risk of hiring the person that sounds the most confident, and that doesn't generally end well in technical spaces. And really figure out what it is that makes people shine. Everyone talks about how to pass the technical interview, but there's very little discussion on the other side of it, which is what kind of training do most of us—are most of us given to effectively conduct a technical interview?Brooke: Yeah, and not even just interview skills, but leadership skills. Number one thing I always talk to you when I'm talking to university students is I let them know that probably the manager they will end up having, if they work in tech, probably has no leadership training or management training of any sort. So, [laugh] if you are just assuming that they will be, like, really just a straight, always making the best management decision or always doing something the most perfect way, they probably have no idea what they're doing as well. And that's really important for people to go into jobs and interviews knowing, is that it's fine if the other person doesn't know as well. Do they want you to win? I think that's the number one thing I always am left with after interviews.And even when I was interviewing—I worked as a marketing manager for a while, so even when I was interviewing for marketing jobs, you could tell whether the person on the other side of the table wanted you to do well or not. And especially when you're looking for an early career job, regardless of any other factor, if someone wants you to do well, that is a good job for you to have. It will just mean so much more to your momentum throughout your career.Corey: I'm a big believer in even if you decide not to continue with a particular candidate, my objective has always been that I want them to think well of the company, I want them to consider reapplying down the road when their skill set changes or what we're looking for changes. And I want them to walk away from the experience with a, “That was a very fair and honest experience. I might recommend applying there to other people I know.” And we've had some people come through that way, so we're definitely succeeding. Whereas I went through the Google SRE interview twice—the second time, I think, was in 2015—and I swore midway through the process that even if they offered me the job, I wouldn't accept it because they didn't want to work with a place where they were going to treat people like that.Full disclosure, I did not get the offer because I'm bad at solving, you know, coding challenges in a Google Doc. Who knew. But it was one of those, I will not put myself through that again. So yeah, it turns out now I've made myself completely unemployable by anyone, so problem solved. “Oh, yeah, I'm never going to put myself through one of those job interviews,” says man who made himself completely un-interviewable, any job ever, ever again. I'll have to change my name and enter witness protection if I want to [laugh] enter the industry after the nonsense I'm pulling.Brooke: Just wear a mustache. They'll never know.Corey: Oh, yeah, you joke but I—some of me wonders on that one. So, I am curious as to your adventures with TikTok. I know I started the show talking about that, but it's still a weird format for me. I thought I got weird comments on Twitter. Oh, no, no, no, not compared to some of the people responding to things on the TikToks.And it's a different format, it's a different audience, it feels like, but there's still a strong appetite for career discussions and for technical discussions as well. How did you stumble on the platform? And how did you figure out what you would be talking about there?Brooke: Yeah, I put off making a TikTok for so long. So, I worked as a model internationally before my current job—I did it during and after uni—and so I have a very fashion Instagram that's very polished… like, I put thought into the outfits that I'm wearing in photos, which means I just haven't posted a lot lately because I just don't have the energy. So, the idea of going on TikTok to do something that is very quick is horrible to me [laugh] as a thought. Also, I hid my fashion past, I was closeted for a long time in tech, just because it was actively negatively impacting career prospects. But one of the best gifts about moving to a leadership team in a management space is that people don't care about that as much anymore, which is really good.So, just it was a big move for me in terms of bringing my closeted to past back into what I'm actually doing in tech, just to get more people aware of the opportunities that are out there. Because there's so many people during Covid that wanted to work from home and they wanted to transition to a job that would allow them to work remotely with benefits and security and everything that goes along with that, and tech is a really good industry to get that in.Corey: Oh, there are millions of jobs now that didn't exist two years ago that empower full remote, either within a given country or globally. Just, do you have an internet connection wherever you happen to be? I mean, we have people here who are excited to go and do all kinds of traveling, and we have people who have—this has been challenging for them—but, on the paperwork on our side, just fill up the forms, but we've had to effectively open tax accounts with different states as they relocate during the course of the pandemic. And power to them; that's what administrative teams are for. But it's really nice to be able to empower stuff like that because for the longest time—I live in San Francisco, and it felt like the narrative was, “We are a disruptive industry that is changing the face of the world. And we are applying that disruption by taking a job that can be done from literally anywhere and creating a land crunch in eight square miles in an earthquake zone.”It really didn't seem like it was the most forward-thinking type of event. And I'm hoping—in fact, we're seeing evidence of it—that this is going to be one of the lasting changes of the pandemic. People don't want to go back into the crappy offices.Brooke: Yeah. And especially in Australia, as well. So, when you're talking about San Francisco being crunched into eight miles, Australia is like that, but the whole country. So, it's a very large space, but there's only a few capital cities dotted around that I think more than 90-something percent of people live in those big centers.Corey: Yeah. Yeah, I made that mistake by taking my week down there and visit—and giving talks in Sydney, Melbourne, and Perth. And it was, well, why not? Like, I'm flying all the way over there. How far apart could it be?It's, “What do you mean, there's that many time zones? And the flight is how many hours to go from one side to the other?” Yeah, on professional advice for people who are considering doing that: Don't.Brooke: Yeah, it's not for the faint-hearted. But it also means that there's so many people that don't live in capital cities that could now work remotely for tech companies. Or I know friends that they originally lived in a capital city, and then have gone to move into regional centers. And as someone that grew up in a regional center, that's so important to be able to spread the tech ecosystem out further. It's 1000 kilometers from where I live now, which I don't know what that is in miles in freedom units, but probably it's about a ten-hour drive if you're driving there; if you're driving without stopping.So, it's a really long way away. And that's just—it's not, like, something you can just drive a few times over for a meetup. There's just nothing around for quite a long time. So, being able to disperse technical knowledge throughout the country is something that's really important to me, especially just because it's opening up futures for more diverse groups, even the people that are using tech in the vast majority of geographically distributed Australia are completely ignored from making that tech. And that's something that's really a growing issue that is getting fixed as there are more opportunities to move remote jobs there. But people don't even know that these jobs exist, so it's just about getting out into the regions to show people that it's possible.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats V-U-L-T-R.com slash screaming.Corey: You were mentioning that you are about to embark on a 31-hour travel nightmare nonsense thing to go to Portugal to give a talk. What is the talk you're giving, and what's the venue?Brooke: It's for NDC Porto. And so, I've done NDC in Sydney, virtually, twice—Corey: NDC is… I'm sorry?Brooke: It's a really big conference. I think it started as… Norwegian something? Norwegian Developers Conference, someone will roast me in the comments of this.Corey: Well, not on this show. Generally, we get a pretty awesome audience base compared to, you know, the TikTok. So, I'm sure we'll excerpt parts of this for the TikToks, and then oh, then all hell is going to break loose.Brooke: [laugh]. That we'll find them. Yeah, it's a big conference series. So, they have them in Oslo, Copenhagen, Sydney, London, Melbourne, and Porto as well. So, it's quite a big—I think it's very eurocentric. I don't know if it would have be in any of the US audiences yet. But it's a really wholesome group.Last time I did the conference in Sydney, the segment before me was someone showing their pet llamas on camera. So… love that. [laugh]. But my talk is just about enterprise applications of AI and machine learning. So, it's mostly the same sessions that I give to executives; I just give it to software engineers, and then tell them about how I talk to executives while I'm doing it to show why it works.Corey: You also give periodic talks at universities as well. You have been very prolific on the speaking circuit. What's the common thread that winds up tying all of these disparate audiences together?Brooke: People ask me and I say yes. Um—[laugh].Corey: Hey, there we go.Brooke: Yeah. No, it's mostly about I just want to make this an easier pathway for other people. If you can see this art here—it's backwards, probably, but it says, “Be who you needed when you were younger.” I made this, and it's just how I go through my tech life. When I talk to high school students and university students, no one's ever honest with them about what it's actually like to have a job because everyone is just telling them how fantastic it is and how everyone will think it's so fantastic that they're a graduate of that institution, and they will get their dream job, and they'll ride home on a unicorn and everything will be perfect.And no adults are ever honest to people because everyone wants something from them. So, it is an absolute immense position of privilege to be able to go in and say, “Here is unfortunate realities of what you're about to step into.” Because my parents, neither of them worked in office, my mom teaches children with disabilities—so she's retired now—and my dad is a telephone technician, so like, I didn't know anyone working in office growing up, it wasn't part of what I did. So, I can't tell people that this is what networking actually is. That will be someone who you are in the room with right now who is extremely wealthy, and their parents own something, and they will sail through life. You need to work much harder than them. [laugh].And just being able to have these actual conversations with students—because it's so valuable—and it's guidance that I wish I had earlier on and that you can actually, if you're aware of what's happening in these systems, you can hedge against it. But it's just, I think it's doing students a disservice to not be honest to them, so I take a lot of pride in doing that. [laugh].Corey: I like doing that, but I'm also worried that I am going to send the wrong message if I do. Because let's be honest, I can get away with an awful lot of stuff based upon my perceived position in the industry, the fact that I am clearly self-employed—when you own the company, it turns out you can get away with a lot—and also I'm 15 to 20 years into my career, whereas if I pulled a lot of this nonsense fresh out of school in my first job, I would have been fired. No ‘would have' about it; I was fired and didn't even pull half of the jokes that I pull now. So, when I give interview advice on TikTok, like here's how you pick a fight with the interviewer. Yeah, if someone actually does that, it's not going to go well, so I live in fear of effectively giving the kind of advice that is actively harmful. If I'm going to do that, I at least try to put a disclaimer into it. But we'll see.Brooke: Yeah. And even just showing people that it is possible for an interviewer to not want you to do well. So, many people are not aware of that because the only idea they have of interviews is that… been something they've been told at whatever school or boot camp they went through that someone really wants them to succeed and will help them to develop their journey. That's not… it's not normal, so being able to actually decipher what is and isn't happening there is a really good skill. And people just aren't aware that things like that are possible, or even I don't know, as a… [unintelligible 00:27:33] person in STEM, I have a lot of sage advice to give to people about what it is and isn't like in reality.And that's where my history of very strange jobs comes into play as well. So, I worked at a car parts store for four years growing up, selling people different types of filters and fuel filters and sound systems for their car.Corey: And blinker fluid, depending on how sketchy the numbers look that month. Of course, of course.Brooke: Yeah. But people would come into the store and just ask for a man straightaway, or call up, and then I was the only one that had any physics education in the store, so some days if [my brother 00:28:08] wasn't working, so they would call up and ask for what type of resistance they needed for their car stereo, and I would tell them, but they would [unintelligible 00:28:17] put a man on. And then eventually they would be like, “I don't know. I have to ask Brooke.” And put me back on the phone, and I would just pretend to have never heard the start of it.But it's just, if you don't have a diverse background of jobs you've had, or different service jobs, it gives you more structure about how to actually talk about what it is like to work in tech because some bits are much worse than they appear, and some things are actually a lot better than they appear as well. It's just depending on who's talking about it in the media at a given day.Corey: And I've said it before—it's always worth repeating—this is what privilege looks like because it's easy for me to sit here and say, “Look, the stuff that I built, the company I've put together, the reputation for myself that I wound up establishing, well, I had to do it all myself. None of it was handed to me.” And that is true. However, I didn't have to fight against bullshit like that. I didn't have a headwind of people telling me that I was somehow unqualified or didn't belong in the place that I was in.When I made a pronouncement, even when it was wrong, it was presumed accurate until proven otherwise. So, there's a lot of stuff around this that just contributes to a terrible toxic environment. That is what privilege is, and you can't set that aside, you can't turn that away. And we all have privilege in different ways, but it's often considered to be controversial. I don't see it that way at all. It's one of those, “You were born on third base; you didn't hit a triple.”Brooke: And it's just about what you do with it, as well. There are some people who are immensely privileged and then they just do nothing to help anyone else. They don't let the ladder down for anyone else after them, so—Corey: “Send the elevator back down,” is what Stephen O'Grady over at RedMonk said, and is a phrase that's stuck with me for years now. And it's the perfect expression of it. It's as opposed to folks who wind up pulling up the rope behind them, “Well, screw you. I got mine.” No thanks.Brooke: Yeah.Corey: That's not how I want to be remembered.Brooke: And that's why it's so important to talk to people about this early in their career as well because these people will become a manager probably. So, then say, “Hey, when you are inevitably a manager, [laugh] you are in a position of power now. Here are things you can actively do that will be who you needed when you were younger.” That's what will actually help people, too. So, being able to really specifically say, once you are in an organization, you have the opportunity to make change, especially in graduate roles in organizations, I noticed they get so much bandwidth to actively make decisions because higher-ups are just so excited that there's someone young working there.So, being able to go through and look at what they are actually doing. And people trust you, then they trust your opinion, and they'll trust your opinion. Especially on issues like sustainability, everyone's just, “Oh, who's a child we can ask?” So, being able to then give them an answer that's helpful. Or say even, “I didn't know about this. Maybe you should ask someone that this affects.”Being able to then hand the microphone to someone else is a skill that is never actively taught to anyone. So, I think that's what's really—it's a slow part of diversity and inclusion changing over time, but it's a really important part of actively modeling that behavior of what it looks like to do a decent job.Corey: Brooke, I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place to find you?Brooke: I'm on Twitter as @brooke_jamieson; I'm on TikTok as BrookeBytes, and I'm on LinkedIn is probably the best place to—I check that inbox the most. And my name is just Brooke Jamieson, which will be in the show notes.Corey: And we will, of course, put links to all of that in the [show notes 00:31:38]. Thanks again for your time. I really appreciate it.Brooke: Thanks so much for having me.Corey: Brooke Jamieson, Head of Enablement for AI, ML, and data at Blackbook. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long, rambling, angry comment that says this is not the content that you expected, you were not happy with it at all, and if I really wanted to have these conversations, I should have instead first demonstrated both of our technical suitability by solving algorithm problems on a whiteboard.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

With Gareth Myles and Ted SalmonJoin us on Mewe RSS Link: https://techaddicts.libsyn.com/rss iTunes | Google Podcasts | Stitcher | Tunein | Spotify  Amazon | Pocket Casts | Castbox | PodHubUK Feedback and Contributions: Ian Barton on Tailscale If you have ever battled with the behemoth that is OpenVPN because you want to access your media server from outside your network you might want to try Tailscale. Tailscale is "A secure network that just works with Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere " Tailscale is built around the Wireguard VPN, which runs on almost any platform and is now built into the Linux kernel. You can use the basic product to connect up to 20 devices for nothing. I have it installed on my UNraid server which is running Tailscale inside a Docker container. Setting this up took all of 10 minutes. Next, I installed Tailscale on my Android phone and could use it to view my Emby media server via 4g. As it's a VPN all traffic between your computers is encrypted. There is an experimental MagicDNS that automatically registers DNS names for computers on your network, so you can use URLs that are easy to remember such as "mymediaserver" instead of having to remember its IP address. An additional plus point is that it somehow works on networks using CGNAT (mobile phones and Starlink) so you can access your phone from your computer via a web browser. Chris Kelly One for Ted and other vinyl lovers? A high quality turntable with built in aptX HD Bluetooth. ALVA TT V2 - Direct Drive Turntable with Bluetooth aptX HD | Cambridge Audio US Shaun Day As someone who spends a lot of time in the car and hates having the phone constantly plugged in and charging, this new Android Auto dongle will be fantastic. Can't wait for it to come to the UK! Motorola's New MA1 Adapter Makes Your Android Auto Wireless Hardline on the hardware: CES2022 TCL New Glasses 30% Lighter Six New TCL Tablets Samsung Galaxy S21 FE 5G vs Samsung Galaxy S20 FE 5G Samsung Freestyle projector launched: The smart TV made portable 55” Samsung Odyssey Ark Samsung's glare-free The Frame 2022 television is astounding Acer reveals three new Chromebooks, including one with a MediaTek SoC Nokia Mobile selling 4th year of security patch support for Nokia X10/X20 Nokia T20 to get a Bluetooth Keyboard Wallet case Razer refreshes Blade 14, 15, 17 laptops at CES 2022 Anker's CES lineup has you covered while at home and while on the go ASUS ROG launches new gaming laptops and peripherals at CES 2022 Lenovo Legion Y700 is an upcoming 8.8-inch 120Hz gaming Android tablet Using the Noveto N1 smart audio device is like 'wearing invisible headphones' Other stuff This portable Bluetooth speaker is powered by light Rolling Square's AirCard fits in your wallet, works just like an AirTag Drone carrying a defibrillator saves its first heart attack patient in Sweden Switch Sold More Units Than Every Other Console Combined In Japan Last Year Wi-Fi 6 Release 2 promises better upload performance Flap your trap about an App: BlackBerry Ends Service on Its Once-Ubiquitous Mobile Devices 22 Best Android Apps for Tablets Russia mandates local TV channels to be included in Netflix subscriptions Ceefax is (Hark) Back Stadia in 2022: I'm ready for cloud gaming, but Google isn't  Hark Back:  Encarta Bargain Basement: Best UK deals and tech on sale we have spotted Anker Wireless Charging 2 in 1 stand with Apple Watch charging holder - £15.00 Xiaomi 11T 5G 8+256GB - £349 from £549 (5x£69.80) - Specs ASUS VivoBook Full HD 14 Inch Laptop (Intel i5-1135G7, 16 GB RAM, 512 GB PCIe SSD, Windows 10 with Free Upgrade to Windows 11) - Price: £549.99 Elgato Stream Deck XL - £189 from £229 (really tempted!) Xiaomi Mi Pad 5 Wifi Only 6GB/128GB £303.00 SanDisk Extreme 1TB Portable NVMe SSD, USB-C, up to 1050MB/s Read and 1000MB/s Write Speed, Water and Dust-Resistant - 42% off, £128 from £222 Google Nest Wifi Router, Strong connection, Every direction - Was: £149.00 Now: £89.00 Anker USB C Hub, PowerExpand 11-in-1 USB C Hub Adapter, with 4K@60Hz HDMI and DP, 100W Power Delivery, USB-C and 3 USB-A Data ports, 1 Gbps Ethernet, 3.5mm Audio, microSD and SD Card Reader - £90 from £110 Acer Chromebook Spin 513 CP513-1H Convertible Laptop, Qualcomm Snapdragon Processor, 4GB RAM, 64GB eMMC, 13.3" Full HD Touchscreen, Blue - £329.99 Nokia XR20 £319 from £399 (or £63 x 5) Blue Microphones Yeti Professional USB Microphone for Recording, Streaming, Podcasting, Broadcasting, Gaming, Voiceovers, and More, Multi-Pattern, Plug 'n Play on PC and Mac - Silver - Was: £119.99 Now: £99.99 Main Show URL: http://www.techaddicts.uk | PodHubUK Contact:: gareth@techaddicts.uk | @techaddictsuk Gareth - @garethmyles | garethmyles.com Ted - tedsalmon.com | Ted's PayPal | Ted's Amazon | tedsalmon@post.com YouTube: Tech Addicts The PodHubUK PodcastsPodHubUK - Twitter - MeWe PSC Group - PSC Photos - PSC Classifieds - WhateverWorks - Camera Creations - TechAddictsUK - The TechBox - AAM - AAWP - Chewing Gum for the Ears - Projector Room - Coffee Time - Ted's Salmagundi - Steve's Rants'n'Raves - Ted's Amazon - Steve's Amazon - Buy Ted a Coffee