Podcasts about aws lambda

Cost is always top of mind when building in the cloud, and recently AWS has introduced some changes worth paying attention to. In this episode of AWS Bites, we explore a shift that caught many by surprise: the “free” INIT phase for Lambda's managed runtimes is going away. That cold start time that used to fly under the billing radar? It's now part of the cost. We dig into what this means for your workloads, who might feel the impact, and whether this gives languages like Rust and Go an extra edge. But it's not all bad news. AWS has also rolled out new pricing tiers for CloudWatch Logs, making it cheaper for high-volume accounts. On top of that, there are new options to send logs directly to S3 or Firehose, helping simplify pipelines and reduce costs. We close with a few tips to help you keep your Lambda and logging spend under control. If you're building on AWS and care about efficiency, this is one you won't want to miss.Big shoutout to fourTheorem for powering yet another episode of AWS Bites. At fourTheorem, we believe the cloud should be simple, scalable, and cost-effective, and we help teams do just that. Whether you're diving into containers, stepping into event-driven architecture, or scaling a global SaaS platform on AWS, or trying to keep cloud spend under control our team has your back. Visit https://fourTheorem.com to see how we can help you build faster, better, and with more confidence using AWS cloud!In this episode, we mentioned the following resources:AWS Blog – Tiered Pricing for AWS Lambda: https://aws.amazon.com/blogs/compute/introducing-tiered-pricing-for-aws-lambda/Luc van Donkersgoed – When is the Lambda INIT phase free and when is it billed?: https://lucvandonkersgoed.com/2022/04/09/when-is-the-lambda-init-phase-free-and-when-is-it-billed/AWS Bites – Explaining Lambda Runtimes (Episode 104): https://awsbites.com/104-explaining-lambda-runtimes/AWS Blog – Standardized Billing for Lambda INIT Phase: https://aws.amazon.com/blogs/compute/aws-lambda-standardizes-billing-for-init-phase/Lambda Cold Start Benchmarks by Maxim David: https://maxday.github.io/lambda-perf/Duckbill Group Blog – Lambda Logs Just Got Cheaper: https://www.duckbillgroup.com/blog/lambda-logs-just-got-cheaper/AWS Bites – Becoming a Logs Ninja with CloudWatch (Episode 35): https://awsbites.com/35-how-can-you-become-a-logs-ninja-with-cloudwatchDo you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠https://bsky.app/profile/eoin.sh⁠⁠⁠⁠ | ⁠⁠⁠⁠https://www.linkedin.com/in/eoins/⁠⁠⁠⁠- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠https://bsky.app/profile/loige.co⁠⁠⁠⁠ | ⁠⁠⁠⁠https://www.linkedin.com/in/lucianomammino/

Ashley Peacock, the author of Serverless Apps on Cloudflare, speaks with host Jeremy Jung about content delivery networks (CDNs). Along the way, they examine dependency injection with bindings, local development, serverless, cold starts, the V8 runtime, AWS Lambda vs Cloudflare workers, WebAssembly limitations, and core services such as R2, D1, KV, and Pages. Ashley suggests why most users use an external database and discusses eventually consistent data stores, S3-to-R2 migration strategies, queues and workflows, inter-service communication, durable objects, and describes some example projects. Brought to you by IEEE Computer Society and IEEE Software magazine.

Today's guest is AJ Stuyvenberg, a Staff Engineer at Datadog working on their Serverless observability project. He had a great article recently about how they rewrote their AWS Lambda extension in Rust. It's a really interesting look at a big, hard project, from thinking about when it's a good idea to do a rewrite to talking about their focus on performance and reliability above all else and what he thinks about the Rust ecosystem. Beyond that, AJ is just a learning machine, so I got his thoughts on all kinds of software development topics, from underrated AWS services and our favorite databases to the AWS Free Tier and the annoyances of a new AWS account. Finally, AJ dishes out some career advice for curious, ambitious developers.

In this conversation, Krish Palaniappan discusses the intricacies of deploying an API gateway on AWS, focusing on the management of API usage, reporting, and the challenges faced with certificate management. He elaborates on the deployment strategies across different environments, the debugging process for certificate issues, and the importance of understanding endpoint types and SSL certificates. The conversation also highlights the resolution of certificate chain issues and the necessary code adjustments to ensure smooth operation. In this conversation, Krish Palaniappan discusses the intricacies of optimizing AWS Lambda layers, the transition from AWS SDK version 2 to version 3, and the importance of efficient deployment strategies. He emphasizes the need for local development and testing using Express to enhance productivity and streamline the onboarding process for customers, including API key management and usage plans. Snowpal Products Backends as Services on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠AWS Marketplace⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Mobile Apps on ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠App Store⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ and ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Play Store⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Web App⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Education Platform⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ for Learners and Course Creators

In dieser Episode von Cloud Horizonte sprechen Oliver und Pavel auf dem AWS Summit in Berlin mit Frank Alsmann (VHV Versicherung) und Philipp Richter (Public Cloud Group) über die Cloud-Transformation der VHV. Die VHV migriert Anwendungen von On-Premise in die AWS Cloud – mit Fokus auf Serverless-Technologien wie AWS Lambda. Wir diskutieren die Herausforderungen einer solchen Migration, von Sicherheit und Kostenoptimierung bis hin zur Modernisierung des einer konkreten Anwendung, des Tarifrechners. Es geht nicht nur um Technik: Die Cloud-Reise ist auch eine kulturelle Veränderung. Wie nimmt man Mitarbeitende mit? Welche Rolle spielen Schulungen, Compliance und FinOps? Erfahren Sie, warum Cloud nicht nur Effizienz bringt, sondern auch neue Denkweisen in Unternehmen etabliert. Wir sprechen über Lessons Learned, den langfristigen Wandel in der IT und wie Unternehmen wie die VHV sich für die Zukunft aufstellen.

Managing security for a device that can autonomously interact with third-party services presents unique orchestration challenges that go beyond traditional IoT security models. In this episode of Detection at Scale, Matthew Domko, Head of Security at Rabbit, gives Jack an in-depth look at building security programs for AI-powered hardware at scale.   He details how his team achieved 100% infrastructure-as-code coverage while maintaining the agility needed for rapid product iteration. Matt also challenges conventional approaches to scaling security operations, advocating for a serverless-first architecture that has fundamentally changed how they handle detection engineering. His insights on using private LLMs via Amazon Bedrock to analyze security events showcase a pragmatic approach to AI adoption, focusing on augmentation of existing workflows rather than wholesale replacement of human analysis.  Topics discussed: How transitioning from reactive SIEM operations to a data-first security approach using AWS Lambda and SQS enabled Rabbit's team to handle complex orchestration monitoring without maintaining persistent infrastructure.  The practical implementation of LLM-assisted detection engineering, using Amazon Bedrock to analyze 15-minute blocks of security telemetry across their stack.  A deep dive into security data lake architecture decisions, including how their team addressed the challenge of cost attribution when security telemetry becomes valuable to other engineering teams.  The evolution from traditional detection engineering to a "detection-as-code" pipeline that leverages infrastructure-as-code for security rules, enabling version control, peer review, and automated testing of detection logic while maintaining rapid deployment capabilities. Concrete examples of integrating security into the engineering workflow, including how they use LLMs to transform security tickets to match engineering team nomenclature and communication patterns. Technical details of their data ingestion architecture using AWS SQS and Lambda, showing how two well-documented core patterns enabled the team to rapidly onboard new data sources and detection capabilities without direct security team involvement. A pragmatic framework for evaluating where generative AI adds value in security operations, focusing on specific use cases like log analysis and detection engineering where the technology demonstrably improves existing workflows rather than attempting wholesale process automation.  Listen to more episodes:  Apple  Spotify  YouTube Website

In this episode, we explore DuckDB, an open-source analytical database known for its speed and simplicity. Discover how DuckDB stands out in various applications and compare it to other tools like SQLite, Athena, Pandas, and Polars. We also demonstrate integrating DuckDB with AWS Lambda and Step Functions for serverless analytics.AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out ⁠fourtheorem.com⁠In this episode, we mentioned the following resources: Our `duck-query-lambda`, A Lambda runtime for DuckDB queries: https://github.com/fourTheorem/duck-query-lambda DuckDB's official website: https://duckdb.org/ LibSQL: https://github.com/tursodatabase/libsql Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠ | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/

In this episode, we discuss using AWS Lambda Powertools for Python to build serverless REST APIs with AWS Lambda. We cover the benefits of using Powertools for routing, validation, OpenAPI support, and more. Powertools provides an excellent framework for building APIs while maintaining Lambda best practices.In this episode, we mentioned the following resources: AWS Bites 41. How can Middy make writing Lambda functions easier? - ⁠https://awsbites.com/41-how-can-middy-make-writing-lambda-functions-easier⁠ AWS Bites 120. Lambda Best Practices - ⁠https://awsbites.com/120-lambda-best-practices/⁠ REST API - Powertools for AWS Lambda (Python) - ⁠https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/⁠ Hono - ⁠https://hono.dev/⁠ Fastify - ⁠https://fastify.dev/⁠ Axum - ⁠https://github.com/tokio-rs/axum⁠ FastAPI - ⁠https://fastapi.tiangolo.com/⁠Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on BlueSky or LinkedIn: https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/ https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/

AWS Community Builder and Software/Platform Engineer Ervin Szilágyi joins us today to talk about his project: creating a BlueSky Bot (for good, not evil

En este episodio especial del podcast de Charlas Técnicas AWS, cerramos oficialmente la Temporada 5 con el esperado re:Cap de re:Invent 2024. Y para hacerlo aún más especial, nos acompaña Marcia Villalba, co-host original del podcast.Hablamos de todo lo que dejó este re:Invent: los nuevos chips Graviton4, avances en inteligencia artificial con Amazon Q Developer y los nuevos modelos NOVA, mejoras en infraestructura como la fibra óptica de núcleo hueco, y la evolución de bases de datos con Aurora DSQL. Todo esto, mezclado con reflexiones sobre cómo AWS sigue apostando por la simplicidad sin perder su esencia.Si te perdiste el evento o quieres repasar los anuncios más importantes, este episodio es para ti.Temas destacados:[00:00] Introducción y regreso de Marcia Villalba.[01:35] Balance general de re:Invent 2024 y tendencias.[05:30] 10 años de AWS Lambda [08:20] Infraestructura avanzada: Hollow-Core Fiber y relojes atómicos.[09:45] Innovaciones en bases de datos: Aurora DSQL y DynamoDB Strong Consistency.[15:40] Mejoras en Amazon S3: S3 Metadata Search y S3 Tables.[17:20] Lanzamiento de Graviton4, Systolic Arrays y Neuron Kernel Interface (NKI).[21:07] Amazon Bedrock, agentes y novedades[26:17] Amazon Q Developer: integración con GitHub, .Net, VMware y Mainframe.[27:40] Amazon NOVA, nuevos modelos de IA Generativa[32:07] Herramientas para IA Generativa, prompt caching, inteligent routing.[38:20] Reflexiones sobre la keynote de Werner Vogels y la simplicidad en AWS.[44:30] Próximos pasos y sorpresas para la Temporada 6.[46:04] El regalo de Marcia.Nos vemos en la Temporada 6 con más contenido, historias de la comunidad y sorpresas. ¡Gracias por acompañarnos este año!

Join us on a serverless computing journey! Host Keith Townsend is with Amazon Web Services' Usman Khalid, Director, AWS Lambda on this episode of Six Five On The Road at AWS re:Invent. They look at a decade of evolution and what's in store for the future of serverless computing with AWS Lambda. Tune in for details

Sébastien Stormacq joins us to talk about AWS Lambda and Swift - what does "Serverless" mean, how deployment works, and how to get started.GuestSébastien ☁ Stormacq

This week, we discuss the relationship between DevOps and Platform Engineering, Gartner's take on Distributed Hybrid Infrastructure, and Nvidia's search for new use cases. Plus, a listener chimes in to clear up some Podman misconceptions. Watch the YouTube Live Recording of Episode (https://www.youtube.com/watch?v=DyjB-jmL0QQ) 495 (https://www.youtube.com/watch?v=DyjB-jmL0QQ) Runner-up Titles Prove me wrong AWS, prove me wrong Please turn off the lights Who's googling for “shift left”? I realized what they were talking about, it's computers They're talking but you're not listening Piling on the dead horse We gave this guy $5 billion dollars, check him out Podman is Pepsi Nobody's paying for that Niche Player Rundown Platform Engineering Is The New DevOps (https://www.forbes.com/sites/justinwarren/2024/11/21/platform-engineering-is-the-new-devops/) SRE Books (https://sre.google/books/) Magic Quadrant for Distributed Hybrid Infrastructure (https://www.gartner.com/doc/reprints?id=1-2J0PN9ZJ&ct=241007&st=sb&trk=0da8abef-e59d-40d4-b66b-ba96c755768b&sc_channel=el) AWS named as a leader again in the Gartner Magic Quadrant for Distributed Hybrid Infrastructure (https://aws.amazon.com/blogs/aws/aws-named-as-a-leader-again-in-the-gartner-magic-quadrant-for-distributed-hybrid-infrastructure/) Nvidia AI Easts the World — Benedict Evans (https://www.ben-evans.com/presentations) Nvidia revenue almost doubles on the year even as growth slows from previous quarter (https://on.ft.com/3Vpw2Z1) Nvidia's Huang Spreads the Gospel of AI in Search of More Customers (https://www.bloomberg.com/news/newsletters/2024-11-21/nvidia-s-huang-spreads-the-gospel-of-ai-in-search-of-more-customers?srnd=undefined) Amazon Updates Homegrown Chips, Even as It Grows Nvidia Ties (https://links.message.bloomberg.com/a/click?_t=f574328d4d0c4c359b90d8e49b10e21d&_m=e253c47d1776426cada2b989eb51ef3d&_e=BISdgjckKJ39RYZ5axUkOu4DkhEzj_0CzmZEdaLS3niAwih7Lch-yccqByy-SKSB_PawXlFTeOpypVo4aikKnrEHKgvZ1v2TyAeErFN65ZsdRhzpsl63CY7Ia4-4Y_AmaM8n0A6iEaAPInfkiRKNT3xf8OE6NLeC4L7EavGfLanwRXXmv773517sL7d2HT-Rcewoj4Ilv2S4WBW0l3E797KSeKHwZmNv3h9g8B7rUMFKXg8gnlDDRuYjGkBMn8m9-4yP3laYhYAwEeaW3arWkc1bzZFYO_N0fzB31aRoEEvMjvCyXvrv-fg1yhLbDHFZFK5xDr2cgqT8uxPoHajG8qPT7nzRt_56WNcg30HnKZ2OwDxnLJkIDzw47BuHXtk-BMsx5WG7Gn51NdUiPqUTAV5YHattNV9B5gmGwXtVZubp-eOJfFuCVKrLgVwrMLLqGMLEFhgI00D0RHwpXFbHDg%3D%3D) Nvidia Earnings, Strawberry and Video, The Networking Question (https://stratechery.com/2024/nvidia-earnings-strawberry-and-video-the-networking-question/) Podman: Podman in Action | Red Hat Developer (https://developers.redhat.com/e-books/podman-action) Kubernetes Podcast from Google: Episode 164 - Podman, with Daniel Walsh and Brent Baude (https://kubernetespodcast.com/episode/164-podman/) DevOps and Docker Talk: Cloud Native Interviews and Tooling | Podman In Action: Desktop, Machine, and more (https://podcast.bretfisher.com/episodes/podman-in-action-desktop-machine-and-more) Relevant to your Interests Microsoft Ignite 2024: Everything Revealed in 15 Minutes (https://www.youtube.com/watch?v=_4qsQ6OWZsM) Microsoft Ignite 2024: all the news from Microsoft's IT pro event (https://www.theverge.com/2024/11/19/24300001/microsoft-ignite-2024-news-ai-announcements-copilot-windows-azure-office) AWS Lambda turns ten – looking back and looking ahead | Amazon Web Services (https://aws.amazon.com/blogs/aws/aws-lambda-turns-ten-the-first-decade-of-serverless-innovation/) Kyndryl insiders claim new business is scarce (https://www.theregister.com/2024/11/20/kyndryl_little_new_business/) Snowflake snaps up data management company Datavolo (https://techcrunch.com/2024/11/20/snowflake-snaps-up-data-management-company-datavolo/) Northflank raises $22M to make Kubernetes work for your developers (https://northflank.com/blog/northflank-raises-22m-to-make-kubernetes-work-for-your-developers-ship-workloads-not-infrastructure) Overcast adds new listening stats and 48-hour undo features (https://9to5mac.com/2024/11/20/overcast-listening-history-undo-features/) Reddit was down — latest updates on major outage (https://www.tomsguide.com/news/live/reddit-down-live-updates-on-outage) Wiz acquires Dazz for $450M to expand its cybersecurity platform (https://techcrunch.com/2024/11/21/wiz-acquires-dazz-for-450m-to-expand-its-cybersecurity-platform/) Comcast is spinning off its cable TV business (https://www.theverge.com/2024/11/20/24301310/comcast-spinning-off-nbcuniversal-cable-tv-business) Snowflake's shares surge higher on blowout earnings, (https://t.co/alQ7p57V3y) Clouded Judgement 11.22.24 - Is Software Back? (https://cloudedjudgement.substack.com/p/clouded-judgement-112224-is-software?utm_source=post-email-title&publication_id=56878&post_id=151992794&utm_campaign=email-post-title&isFreemail=true&r=2l9&triedRedirect=true&utm_medium=email) WordPress.com owner Automattic snaps up grammar checker Harper (https://techcrunch.com/2024/11/21/wordpress-com-owner-automattic-snaps-up-grammar-checker-harper/) DHH Wants To Make Web Dev Easy Again, With Ruby on Rails (https://thenewstack.io/dhh-wants-to-make-web-dev-easy-again-with-ruby-on-rails/) Dear friend, you have built a Kubernetes (https://www.macchaffee.com/blog/2024/you-have-built-a-kubernetes/) Kamal 2.0 Released (https://dev.37signals.com/kamal-2/) 'I have no money': Thousands of Americans see their savings vanish in Synapse fintech crisis (https://www.cnbc.com/2024/11/22/synapse-bankruptcy-thousands-of-americans-see-their-savings-vanish.html) Pentagon audit highlights woeful ERP systems (https://www.thestack.technology/pentagon-audit-it-systems-erp/) Delivering 4K Video with Cloudflare R2 for $2.18 (https://screencasting.com/cheap-video-hosting) First Google Axion Processor Now Available: Claims Best Performance in Cloud Market (https://www.infoq.com/news/2024/11/google-axion-c4a/) Glassdoor Worklife Trends 2025 - Glassdoor US (https://www.glassdoor.com/blog/worklife-trends-2025/) Nonsense It looks like Backstage is working out. (https://www.threads.net/@derickevolved/post/DCmyJeYyWF3?xmt=AQGzEuuVG27rxo-L0IWbIfrdALmECac-SYLR2VaYkspHDw) European Showers (https://www.threads.net/@_yes_but/post/DCmOuqpyX-l?xmt=AQGzhm-KIrpChsW3eMGrDrVwzbijNEoRkz01Iin63gnOoQ) KFC's latest partnership is with Build-A-Bear Workshop (https://www.nrn.com/quick-service/kfc-s-latest-partnership-build-bear-workshop) Australia/Lord_Howe is the weirdest timezone | SSOReady (https://ssoready.com/blog/engineering/truths-programmers-timezones/) Listener Feedback Andrew created the Multiple Tab to PDF Printer (https://chromewebstore.google.com/detail/multiple-tab-to-pdf-print/anlocohdegpcbalhdigpjemapejhephi) Conferences CfgMgmtCamp (https://cfgmgmtcamp.org/ghent2025/), February 2-5, 2025. DevOpsDayLA (https://www.socallinuxexpo.org/scale/22x/events/devopsday-la) at SCALE22x (https://www.socallinuxexpo.org/scale/22x), March 6-9, 2025, discount code DEVOP SDT News & Community Join our Slack community (https://softwaredefinedtalk.slack.com/join/shared_invite/zt-1hn55iv5d-UTfN7mVX1D9D5ExRt3ZJYQ#/shared-invite/email) Email the show: questions@softwaredefinedtalk.com (mailto:questions@softwaredefinedtalk.com) Free stickers: Email your address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) Follow us on social media: Twitter (https://twitter.com/softwaredeftalk), Threads (https://www.threads.net/@softwaredefinedtalk), Mastodon (https://hachyderm.io/@softwaredefinedtalk), LinkedIn (https://www.linkedin.com/company/software-defined-talk/), BlueSky (https://bsky.app/profile/softwaredefinedtalk.com) Watch us on: Twitch (https://www.twitch.tv/sdtpodcast), YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured), Instagram (https://www.instagram.com/softwaredefinedtalk/), TikTok (https://www.tiktok.com/@softwaredefinedtalk) Book offer: Use code SDT for $20 off "Digital WTF" by Coté (https://leanpub.com/digitalwtf/c/sdt) Sponsor the show (https://www.softwaredefinedtalk.com/ads): ads@softwaredefinedtalk.com (mailto:ads@softwaredefinedtalk.com) Recommendations Brandon: Cursor (https://www.cursor.com/) Matt: iPhone Mirroring in macOS (https://www.google.com/search?client=safari&rls=en&q=iPhone+Mirroring+in+macOS+15&ie=UTF-8&oe=UTF-8) (https://www.google.com/search?client=safari&rls=en&q=iPhone+Mirroring+in+macOS+15&ie=UTF-8&oe=UTF-8)15 (https://www.google.com/search?client=safari&rls=en&q=iPhone+Mirroring+in+macOS+15&ie=UTF-8&oe=UTF-8) Coté: Insta360 Flow Pro gimbal (https://amzn.to/4g7t9UA) Photo Credits Header (https://unsplash.com/photos/clear-light-bulb-lot-PIrOQrqewLE) Artwork (https://unsplash.com/s/photos/grade-evaluation) Web 2.0 2FA Life Hacks (https://www.troyhunt.com/beyond-passwords-2fa-u2f-and-google-advanced-protection/)

Bret and Nirmal Mehta are joined by Ken Collins to dig into using AI for more than coding, and if we can build an AI assistant that knows us.They touch on a lot of tools and platforms. "We're bit all over the place on this one, from talking about AI features in our favorite note taking apps like Notion, to my journey of making an open AI assistant with all of my Q&A from my courses, thousands of questions and answers, to coding agents and more." Ken is a local friend in Virginia Beach and was on the show last year talking about AWS Lambda, and we've both been trying to find value in all of these AI tools for our day to day work.Be sure to check out the live recording of the complete show from October 24, 2024 on YouTube (Stream 279).★Topics★The Lifestyle Copilot Blog PostServerless AI Inference with Gemma 2 Blog Post Creators & Guests Cristi Cotovan - Editor Beth Fisher - Producer Bret Fisher - Host Ken Collins - Guest Nirmal Mehta - Host (00:00) - Intro (01:26) - AI in Recruitment at Torc (03:25) - AI for Day to Day Workflows (04:44) - Notion AI and RAG (07:20) - Creating Your Own AI Search Solution (13:59) - Choosing the Right LLM for the Job (20:55) - Personal AI and Long Context Windows (25:10) - Future of Personal Fine-Tuned Models (25:52) - AI Assistants in Meetings (27:34) - Temperature and AI Hallucinations (32:07) - Agents and Tool Integration (39:31) - Apple Intelligence and Personal AI (44:56) - AI Apps on Mobile (50:00) - LoRA You can also support my free material by subscribing to my YouTube channel and my weekly newsletter at bret.news!Grab the best coupons for my Docker and Kubernetes courses.Join my cloud native DevOps community on Discord.Grab some merch at Bret's Loot BoxHomepage bretfisher.com

An airhacks.fm conversation with Vadym Kazulkin (@VKazulkin) about: journey as a Java developer from the late 1990s to present, early experiences with Java and J2EE development, transition to cloud and serverless technologies, particularly AWS Lambda, discussion of Java performance on lambda compared to node.js, detailed explanation of AWS SnapStart technology for improving Java cold starts, pros and cons of "fat" Lambda functions versus microservices, challenges of using GraalVM with Lambda, importance of optimizing Lambda package size and dependencies, comparison of quarkus and Spring Boot on Lambda, benefits of serverless architecture for business logic focus, involvement with Java User Group Bonn and AWS Community Builder program, brief mention of asynchronous patterns in serverless architectures, importance of staying technically hands-on as a manager in the rapidly evolving cloud world Vadym Kazulkin on twitter: @VKazulkin

AWS Morning Brief for the week of November 4, with Corey Quinn. Links:Amazon CloudWatch now monitors EBS volumes exceeding provisioned performanceAmazon Q Developer announces support for inline chat to streamline the developer experienceAmazon Route 53 announces HTTPS, SSHFP, SVCB, and TLSA DNS resource record supportAmazon Virtual Private Cloud launches new security group sharing featuresAWS now accepts partial card paymentsAnnouncing AWS Amplify integration with Amazon S3 for static website hostingAWS CodeBuild now supports retrying builds automaticallyAWS Trust & Safety Center is now available on AWS re:Post2024 re:Invent Know Before You Go – Cloud Financial Management SessionsIntroducing an enhanced local IDE experience for AWS Lambda developers

AWS Morning Brief for the week of October 28, with Corey Quinn. Links:Amazon Aurora launches Global Database writer endpointAmazon Connect now offers screen sharingAmazon EKS endpoints now support connectivity over Internet Protocol version 6 (IPv6)AWS IAM Identity Center simplifies calls to AWS services with single identity contextEC2 Image Builder now supports building and testing macOS imagesIntroducing an enhanced in-console editing experience for AWS Lambda

De Java 23 à WebAssembly, en passant par l'IA et les design patterns, on a tout passé au crible #java #swift #webassembly #wordpress #webcomponents #llm #mongodb #keycloak #fairsource Enregistré le 18 octobre 2024 Téléchargement de l'épisode LesCastCodeurs-Episode–317.mp3 News Langages Java 23 est sorti ! InfoQ liste toutes les JEPs intégrées à la nouvelle version https://www.infoq.com/news/2024/09/java23-released/ Et FooJay plonge dans le détail https://foojay.io/today/java–23-has-arrived-and-it-brings-a-truckload-of-changes/ JEP 455: Primitive Types in Patterns, instanceof, and switch (Preview) JEP 466: Class-File API (Second Preview) JEP 467: Markdown Documentation Comments JEP 469: Vector API (Eighth Incubator) JEP 471: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal JEP 473: Stream Gatherers (Second Preview) JEP 474: ZGC: Generational Mode by Default JEP 476: Module Import Declarations (Preview) JEP 477: Implicitly Declared Classes and Instance Main Methods (Third Preview) JEP 480: Structured Concurrency (Third Preview) JEP 481: Scoped Values (Third Preview) JEP 482: Flexible Constructor Bodies (Second Preview) StringTemplate s'en va Un article sur l'API ClassFile qui sera un standard dans le JDK pour manipuler des classes (ala ASM) https://www.unlogged.io/post/class-file-api-not-your-everyday-java-api article long mais qui revient sur les raisons notamment parce que ASM est dans le JDK et qu'ils sont un problème de poule et d'oeuf et sur la forme de l'API a des exemples d'usage tout cela reste en preview dans le JDK des optimisation comme le lazy parsing et le constant pool sharing (en gros faire de la reference sur ce qui n'a pas changé Tip and Tail is back: cette fois une JEP https://openjdk.org/jeps/14 plus qu'une keynote provocative au language summit maintenant une JEP dite informative le language est un pu flou sur l'objectif entre regarder tip and tail pour vos librairies c'est bien et adoptons tous le meme tip du JDK jusqu'aux stack applicatives Apple annonce la sortie de son language Swift en version 6 https://www.swift.org/blog/announcing-swift–6/ Nouvelles plateformes : Swift 6 s'étend à de nouvelles plateformes (tous les grands OS déjà supportés), y compris les systèmes embarqués (sous ARM et Risc V). Swift Testing : Swift 6 introduit Swift Testing, une nouvelle bibliothèque de tests conçue pour Swift. Concurrence : Détection de data race en tant qu'erreur de compilation. Apple annonce travailler sur l'interopérabilité Swift / Java https://github.com/swiftlang/swift-java comme jextract mais dans l'autre sens The news Java https://www.infoq.com/news/2024/10/java-news-roundup-oct07–2024/ JDK 24 : Un calendrier pour la sortie de JDK 24 a été proposé. La première phase de réduction des fonctionnalités commencera le 5 décembre 2024. La version finale sera disponible le 18 mars 2025. JDK 24 introduira des mises à jour avec deux nouvelles API. La Vector API (JEP 489) facilitera les opérations sur des vecteurs, tandis que la Class-File API (JEP 484) permettra une manipulation plus efficace des fichiers de classes Java. Un changement de sécurité important est proposé avec JEP 486. Il prévoit de désactiver définitivement le Security Manager, qui a été déprécié. Cette décision signifie que cette fonctionnalité ne sera plus disponible dans les futures versions, car elle est considérée comme obsolète. Apache Tomcat et Cassandra : Les nouvelles versions de Tomcat (11.0.0) et de Cassandra (5.0.0) sont sorties. Elles incluent des améliorations et des corrections de bogues. Spring Framework : Des mises à jour pour Spring Framework (versions 3.4.0-M2, 3.3.3 et 3.2.8) ont été publiées. Elles intègrent le support d'une nouvelle API qui aide à la gestion de la mémoire. Quarkus : Red Hat a sorti la version 3.15 de Quarkus, qui apporte des corrections et des améliorations. Une nouvelle version, la 3.16, est prévue pour la fin octobre. Commonhaus Foundation : Une nouvelle organisation, la Commonhaus Foundation, a été créée pour aider les projets open source à être durables. Quarkus a rejoint cette fondation. Cassandra, Camel, Lamgchain, Micronaut, OpenLibery, JHipster, Ktor etc. Design patterns revisited: https://www.youtube.com/watch?v=kE5M6bwruhw Design and design patterns. Optional: patterns and anti-patterns. Iterator pattern. Lightweight Strategy. Factory Method using default methods. Laziness using Lambda Expressions. Decorator using Lambda Expressions. Creating Fluent interfaces. Execute Around Method Pattern. Creating a Closed Hierarchy with sealed classes. Popularité des langages de programmation https://www.techspot.com/news/105157-python-most-popular-coding-language-but-challengers-gaining.html Python reste le langage de programmation le plus populaire, surtout dans des domaines comme la science des données et le développement web. Il est apprécié pour sa simplicité et le grand nombre de bibliothèques disponibles, ce qui le rend facile à apprendre et à utiliser. De nombreuses entreprises, y compris des startups, utilisent Python pour diverses applications. Malgré sa dominance, d'autres langages comme JavaScript, Java et Go gagnent en popularité et pourraient défier la position de leader de Python. (Java est monté du poste 4 au 3, en 1 an) Les développeurs qui codent occasionnellement préfèrent Python, montrant ainsi son attrait au-delà des programmeurs professionnels. L'émergence d'outils comme ChatGPT facilite l'accès à la programmation, ce qui pourrait influencer les tendances futures en matière de langages de programmation. Librairies Paramétrer ses tests JUnit 5 avec @CsvSource https://mikemybytes.com/2021/10/19/parameterize-like-a-pro-with-junit–5-csvsource/ l'annotation permet d'avoir ses données de test au plus près de la méthode on écrit les données de test sous forme de CSV (éventuellement avec des délimiteurs de son choix pour plus de lisibilité, pour bien séparer les valeurs) par exemple -> ou maps to les valeurs peut être les paramètres de la method mai aussi les valeur de description du test Infrastructure Turbocharged Development: The Speed and Efficiency of WebAssembly par Danielle Lancashire https://devsummit.infoq.com/presentation/munich2024/turbocharged-development-speed-and-efficiency-webassembly L'utilisation de WebAssembly avec Serverless. Faire tourner des applications plus facilement dans le cloud.WebAssembly est rapide et sûr pour exécuter du code. Cela aide à déployer les applications plus rapidement et à utiliser moins de ressources. De nombreuses entreprises utilisent WebAssembly pour des tâches comme le traitement d'images et de données. Des plateformes comme Cloudflare Workers et AWS Lambda. La communauté autour de WebAssembly granèit. De nouveaux outils et bibliothèques sont créés. Cependant, il y a encore des défis à relever, comme la compatibilité et les performances. Malgré cela, l'avenir de WebAssembly est prometteur. Web C'est la guerre chez Wordpress https://techcrunch.com/2024/09/26/wordpress-vs-wp-engine-drama-explained/ une boite nommée WP Engine fait du hosting de WordPress mais ne contribue pas Automatic, les gens derrière WordPress leur onbt demandé de résoudre ce probleme, soit en payant des droits de trademark soit en contribuant de l'engineering upstream à auteur de 8% de leurs revenus WP Engine dit non Automatic coupe l'accès aux mises a jours de thèmes et de plugins à WP Engine mettant des sites à risque (securité) WP Engine dit que c'est un abus de position du CEO d'Automatic sur les accès WordPress.org Bref c'est le drame le CEO d'automatic propose à ses employés 6 mois de salaire si ils ne sont pas d'accord avec la stratégie https://www.cio.com/article/3550331/one-twelfth-of-automattic-staff-leave-over-wordpress-wp-engine-spat.html 8% ont pris l'offre Les WebComposants ne sont pas le fuitur https://dev.to/ryansolid/web-components-are-not-the-future–48bh un article d'un auteur proéminent de framework JavaScript Discute les avantages et les inconvenients de la standardisation qui permet d'élever le débat mais aussi bloque des avenues d'optimisations beaucoup d'exemples d'inovations en frameworks JS qui auraient été bloqués Les commentaires apres l'article sont interessants aussi (en contre perspective) mais tout le monde n'est pas d'accord avec cet article https://www.abeautifulsite.net/posts/web-components-are-not-the-future-they-re-the-present/ Data et Intelligence Artificielle Conseils et bonnes pratiques lors de l'intégration de LLM dans une application https://glaforge.dev/posts/2024/09/23/some-good-practices-when-integrating-an-llm-in-your-application/ management de prompt effectif versionnage et externalisation des prompts fixer la version des modèles optimisation et caching mettre en place des rails de sécurité évaluer et monitorer le comportement et la performance prioriser la sécurité des données privées Encore une nouvelle version de LangChain4j, avec la version 0.35 ! Guillaume couvre les nouveautés côté Gemini et Google Cloud https://glaforge.dev/posts/2024/09/29/lots-of-new-cool-gemini-stuff-in-langchain4j/ Support des toutes nouvelles versions de Gemini 1.5 (version 002) Un “document loader” pour charger des documents à partir de Google Cloud Storage Un “scoring model” qui permet de faire du “reranking” de résultat, pour trouver les résultats les plus pertinents pour une requête donnée Support de nouveaux paramètres des embedding models (choix de la dimensionalité des vecteurs, du troncage des textes en entrée) Ajout d'un “embedding model” pour le module Google AI Gemini Un estimateur de token pour Google AI Gemini Support des chat listeners Support des enums pour la sortie structurée JSON Et plein de mise à jour de la documentation pour refleter tous ces changements et aditions Self Correction Algo LLM https://www.infoq.com/news/2024/10/google-deepmind-score/ Google DeepMind a récemment publié SCoRe, une nouvelle méthode d'auto-correction pour les modèles de langage (LLM). Elle améliore la capacité des LLM à corriger leurs erreurs lorsqu'ils résolvent des problèmes de mathématiques ou de programmation. Contrairement aux méthodes antérieures, SCoRe utilise des données générées par le modèle lui-même pour créer des dialogues d'auto-correction. Cela permet au modèle de s'améliorer via un processus d'apprentissage par renforcement (RL) en deux étapes. Les modèles ajustés avec cette technique ont montré des améliorations significatives, surpassant les performances des modèles de base. Cette méthode pourrait ouvrir de nouvelles pistes pour rendre les LLM plus précis et robustes dans leurs réponses. MongoDB 8 est sorti https://www.mongodb.com/products/updates/version-release La version 8.0 est plus rapide, avec des lectures plus rapides, une meilleure gestion des mises à jour et des agrégations de séries temporelles jusqu'à 60 % plus rapides. De nouvelles fonctionnalités incluent le support des Query pour les données chiffrées, rendant le traitement des données sensibles plus facile. Beaucoup d'ameliorations pour la performance et scalabilité Guillaume explore les techniques avancées de Retrieval Augmented Generation pour améliorer la qualité des résultats de recherche dans ses propres documents, avec les LLMs https://glaforge.dev/talks/2024/10/14/advanced-rag-techniques/ Présentations et vidéos données lors de la conférence Devoxx Belgique Code des exemples disponibles sur Github Techniques de chunking : sliding window, hypothetical questions, semantic chunking, context retrieval chunking Techniques de retrieval : hypothetical document embedding, query compression, metadata filtering Outillage Article sur les cache alias en Infinispan https://infinispan.org/blog/2024/10/07/cache-aliases-redis-databases Explique comment on peut utiliser Infinispan pour remplacer Redis Explique la différence entre les database de Redis et les caches d'Infinispan Explique l'utilité des alias en général Explique comment on peut avoir un mapping des databases de Redis vers des caches d'Infinispan Sécurité Keycloak 26 est sorti: https://www.keycloak.org/2024/10/keycloak–2600-released Organizations feature: permet aux administrateurs de créer et gérer des structures organisationnelles, facilitant la gestion des rôles et des permissions. Persistent user sessions: Les sessions des utilisateurs sont maintenant stockées par default dans la base de donnée ce qui améliore la cohérence, surtout avec plusieurs instances. Login Theme: Offre un design plus propre et une option de mode sombre qui s'adapte aux préférences des utilisateurs. L'amélioration du déploiement multi-sites renforce la fiabilité et réduit le temps d'arrêt lors des demandes des utilisateurs. Admin recovery: une méthode simple pour récupérer l'accès administrateur si tous les comptes sont bloqués, en créant un compte temporaire via des variables d'environnement. Pour les utilisateurs qui migrent vers cette version, il est important de prêter attention aux changements liés à la gestion des caches et aux sessions persistantes. Loi, société et organisation Introduction des licences fair source https://techcrunch.com/2024/09/22/some-startups-are-going-fair-source-to-avoid-the-pitfalls-of-open-source-licensing/ Certaines startups utilisent des licences “fair source” pour partager leur code tout en protégeant leurs intérêts commerciaux. Les licences FSL (Functional Source License) et BUSL (Business Source License) permettent d'ouvrir le code après 2 ou 4 ans. Ces licences empêchent les concurrents de vendre des produits similaires tout de suite, offrant une protection temporaire. Certains critiques pensent que ces licences sont compliquées et pourraient limiter l'innovation, car elles ne sont pas totalement ouvertes. Le “fair source” est encore un concept nouveau, mais il pourrait devenir un bon compromis entre open source et logiciel privé. definition de fair source: code lisible publique, peut etre utilise et modifié avec des “restrictions minimales” pour proteger le business modele du producteur ; et devient open source de maniere deferée “any purpose other than a Competing Use. A Competing Use means use of the Software in or for a commercial product or service that competes with the Software or any other product or service we offer using the Software as of the date we make the Software available” Outils de l'épisode Un petit outil sympa pour les utilisateurs de Macs avec un écran “wide”, pour partager un écran virtuel : https://github.com/Stengo/DeskPad les écrans larges sont partagés entierement et ceui fait un rendu 16:9 pour les gens qui le voient cet écran acte comme un écran mais il est virtuel et on peut mettre les applications que l'on veut dedans on ne l'a pas testé Conférences La liste des conférences provenant de Developers Conferences Agenda/List par Aurélie Vache et contributeurs : 17–18 octobre 2024 : DevFest Nantes - Nantes (France) 17–18 octobre 2024 : DotAI - Paris (France) 30–31 octobre 2024 : Agile Tour Nantais 2024 - Nantes (France) 30–31 octobre 2024 : Agile Tour Bordeaux 2024 - Bordeaux (France) 31 octobre 2024–3 novembre 2024 : PyCon.FR - Strasbourg (France) 6 novembre 2024 : Master Dev De France - Paris (France) 7 novembre 2024 : DevFest Toulouse - Toulouse (France) 8 novembre 2024 : BDX I/O - Bordeaux (France) 13–14 novembre 2024 : Agile Tour Rennes 2024 - Rennes (France) 16–17 novembre 2024 : Capitole Du Libre - Toulouse (France) 20–22 novembre 2024 : Agile Grenoble 2024 - Grenoble (France) 21 novembre 2024 : DevFest Strasbourg - Strasbourg (France) 21 novembre 2024 : Codeurs en Seine - Rouen (France) 27–28 novembre 2024 : Cloud Expo Europe - Paris (France) 28 novembre 2024 : Who Run The Tech ? - Rennes (France) 2–3 décembre 2024 : Tech Rocks Summit - Paris (France) 3 décembre 2024 : Generation AI - Paris (France) 3–5 décembre 2024 : APIdays Paris - Paris (France) 4–5 décembre 2024 : DevOpsRex - Paris (France) 4–5 décembre 2024 : Open Source Experience - Paris (France) 5 décembre 2024 : GraphQL Day Europe - Paris (France) 6 décembre 2024 : DevFest Dijon - Dijon (France) 22–25 janvier 2025 : SnowCamp 2025 - Grenoble (France) 30 janvier 2025 : DevOps D-Day #9 - Marseille (France) 6–7 février 2025 : Touraine Tech - Tours (France) 25 mars 2025 : ParisTestConf - Paris (France) 3 avril 2025 : DotJS - Paris (France) 10–12 avril 2025 : Devoxx Greece - Athens (Greece) 16–18 avril 2025 : Devoxx France - Paris (France) 7–9 mai 2025 : Devoxx UK - London (UK) 12–13 juin 2025 : DevLille - Lille (France) 24 juin 2025 : WAX 2025 - Aix-en-Provence (France) 18–19 septembre 2025 : API Platform Conference - Lille (France) & Online 9–10 octobre 2025 : Volcamp - Clermont-Ferrand (France) Nous contacter Pour réagir à cet épisode, venez discuter sur le groupe Google https://groups.google.com/group/lescastcodeurs Contactez-nous via twitter https://twitter.com/lescastcodeurs Faire un crowdcast ou une crowdquestion Soutenez Les Cast Codeurs sur Patreon https://www.patreon.com/LesCastCodeurs Tous les épisodes et toutes les infos sur https://lescastcodeurs.com/

In this episode of the vBrownBag, Damian does a deep dive on getting started with PowerShell on AWS Lambda. He covers setting up a development environment, packaging & publishing PowerShell on Lambda, lessons learned, and more! 00:28 A quick overview of PowerShell & AWS Lambda

We'll explore 3 use cases for monitoring data. They are:* Analyzing long-term trends* Comparing over time or experiment groups* Conducting ad hoc retrospective analysis Analyzing long-term trends You can ask yourself a couple of simple questions as a starting point:* How big is my database?* How fast is the database growing? * How quickly is my user count growing?As you get comfortable with analyzing data for the simpler questions, you can start to analyze trends for less straightforward questions like:* How is the database performance evolving? Are there signs of degradation?* Is there consistent growth in data volume that may require future infrastructure adjustments?* How is overall resource utilization trending over time across different services?* How is the cost of cloud resources evolving, and what does that mean for budget forecasting?* Are there recurring patterns in downtime or service degradation, and what can be done to mitigate them?Sebastian mentioned that it's a part of observability he enjoys doing. I can understand why. It's exciting to see how components are changing over a period and working out solutions before you end up in an incident response nightmare.Getting to effectively analyze the trends requires the right level of data retention settings. Because if you're throwing out your logs, traces, and metrics too early, you will not have enough historical data to do this kind of work.Doing this right means having the right amount of data in place to be able to analyze those trends over time, and that will of course depend on your desired period. Comparing over time or experiment groupsGoogle's definitionYou're comparing the data results for different groups that you want to compare and contrast. Using a few examples from the SRE (2016) book:* Are your queries faster in this version of this database or this version of that database? * How much better is my memcache hit rate with an extra node and is my site slower than it was last week? You're comparing it to different buckets of time and different types of products.A proper use case for comparing groupsSebastian did this particular use case recently because he had to compare two different technologies for deploying code: AWS Lambda vs AWS Fargate ECS. He took those two services and played around with different memories and different virtual CPUs. Then he ran different amounts of requests against those settings and tried to figure out which one was the better technology option most cost-effectively.His need for this went beyond engineering work but enabling product teams with the right decision-making data. He wrote out a knowledge base article to give them guidance for a more educated decision on the right AWS service.Having the data to compare the two services allowed him to answer questions like:* When should you be using either of these technologies? * What use cases would either technology be more suitable for?This data-based decision support is based mainly on monitoring or observability data. The idea of using the monitoring data to compare tools and technologies for guiding product teams is something I think reliability folk can gain a lot of value from doing. Conducting ad hoc retrospective analysis (debugging)Debugging is a bread-and-butter responsibility for anyone who is a software engineer of any level. It's something that everybody should know a little bit more about than other tasks because there are very effective and also very ineffective ways of going about debugging. Monitoring data can help make the debugging process fall into the effective side.There are organizations where you have 10 different systems. In one system, you might get one fragmented piece of information. In another, you'll get another fragment. And so on for all the different systems. And then you have to correlate these pieces of information in your head and hopefully, you get some clarity out of the fragments to form some kind of insight. Monitoring data that are brought together into one datastream can help correlate and combine all these pieces of information. With it, you can:* Pinpoint slow-running queries or functions by analyzing execution times and resource usage, helping you identify inefficiencies in your code* Correlate application logs with infrastructure metrics to determine if a performance issue is due to code errors or underlying infrastructure problems* Track memory leaks or CPU spikes by monitoring resource usage trends, which can help you identify faulty code or services* Set up detailed error tracking that automatically flags code exceptions and matches them with infrastructure events, to get to the root cause faster* Monitor system load alongside application performance to see if scaling issues are related to traffic spikes or inefficient code pathsBeing able to do all this makes the insight part easier for you. And so your debugging approach becomes very different. It becomes much more effective. It becomes much less time-consuming. It potentially makes the debugging task fun.Because you get to the root cause of the thing that is not working much faster. Your monitoring/observability data setup can make it nice and fun to a certain degree, or it can make it downright miserable. If it's done well, it's just one of those things you don't even have to think about. It's just part of your job. You do it. It's very effective and you move on. Wrapping upSo we've covered three more use cases for monitoring data, other than the usual alerts and dashboards.They are once again:* analyzing long-term trends* comparing over time or experiment groups and* conducting ad hoc retrospective analysis, aka debuggingNext time your boss asks you what all these systems do, you now have three more reasons that you need to focus on your monitoring and be able to use it more effectively. Until next time, happy monitoring. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit read.srepath.com

This interview was recorded for GOTO Unscripted.https://gotopia.techRead the full transcription of this interview hereJulian Wood - Serverless Developer Advocate at AWSMarcia Villalba - Serverless Developer Advocate at AWSRESOURCESMarciahttps://twitter.com/mavi888uyhttps://www.linkedin.com/in/marciavillalbahttps://marcia.devhttps://github.com/mavi888Julianhttps://twitter.com/julian_woodhttp://www.wooditwork.comhttps://www.linkedin.com/in/julianrwoodLinkshttps://serverlessland.comhttps://aws.amazon.com/blogs/compute/optimizing-lambda-functions-packaged-as-container-imageshttps://serverlessland.com/patterns/s3-lambda-container-samhttps://aws.amazon.com/blogs/compute/building-php-lambda-functions-with-docker-container-imagesJulian & James: https://youtu.be/KDtsC7SjsdAMarcia: https://youtu.be/bC51iRtXlRwLiz: https://youtu.be/8fi7uSYlOdcJulian: https://youtu.be/4YeZf3HupQADESCRIPTIONExplore the seamless integration of container images with AWS Lambda! Marcia Villalba and Julian Wood unravel the intricacies of development, deployment, and optimization. Discover how containers revolutionize serverless computing, offering speed, efficiency, and scalability in the cloud. From demystifying cold starts to harnessing caching technologies, this is your chance to learn from the experts. [...]RECOMMENDED BOOKSAdam Bellemare • Building Event-Driven MicroservicesPeter Sbarski • Serverless Architectures on AWSMichael Stack • Event-Driven Architecture in GolangFord, Richards, Sadalage & Dehghani • Software Architecture: The Hard PartsGerardus Blokdyk • Event-Driven Architecture EDAJames Urquhart • Flow ArchitecturesTwitterInstagramLinkedInFacebookLooking for a unique learning experience?Attend the next GOTO conference near you! Get your ticket: gotopia.techSUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted daily!

In this episode, we discuss AWS Lambda provisioned concurrency. We start with a recap of Lambda cold starts and the different concurrency control options. We then explain how provisioned concurrency works to initialize execution environments in advance to avoid cold starts. We cover how to enable it, pricing details, common issues like over/under-provisioning, and alternatives like self-warming functions or using other services like ECS and Fargate.

Thank you to Hookdeck for sponsoring this episode. If you're looking to level-up your event-driven architecture, then check out their serverless event gateway at hookdeck.com/theburningmonk and help support this channel.AWS Serverless Hero Luciano Mammino shares the history of Middy, the popular middleware engine for AWS Lambda functions; why he's sold on writing Lambda functions in Rust and why you should too!Links from the episode:AWS Bites channelMiddyHow to sponsor MiddyCrafting Lambda Functions in RustEasy mode RustUsing Node.js ES modules and top-level await in AWS LambdaUsing Middy with TypescriptEp97 on LLRT (the superfast JavaScript runtime for Lambda)Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0

What are you doing differently today that you're stopping tomorrow's legacy? In this episode Ashish spoke to Adrian Asher, CISO and Cloud Architect at Checkout.com, to explore the journey from monolithic architecture to cloud-native solutions in a regulated fintech environment. Adrian shared his perspective on why there "aren't enough lambdas" and how embracing cloud-native technologies like AWS Lambda and Fargate can enhance security, scalability, and efficiency. Guest Socials:⁠ ⁠⁠Adrian's Linkedin ⁠ Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Podcast- Youtube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security BootCamp Questions asked: (00:00) Introduction (01:59) A bit about Adrian (02:47) Cloud Naive vs Cloud Native (03:54) Checkout's Cloud Native Journey (05:44) What is AWS Fargate? (06:52) There are not enough Lambdas (09:52) The evolution of the Security Function (12:15) Culture change for being more cloud native (15:23) Getting security teams ready for Gen AI (18:16) Where to start with Cloud Native? (19:14) Where you can connect with Adrian? (19:39) The Fun Section

In this episode, we discuss Luciano's new book project on using Rust to write AWS Lambda functions. We start with a recap on why Rust is a good fit for Lambda, including performance, efficiency, safety, and low cold start times. Luciano provides details on the book's progress so far, the intended audience, and the current published chapters covering Lambda internals, getting started with Rust Lambda, and building a URL shortener app with DynamoDB. We also explore the differences between traditional publishing and self-publishing, and why Luciano chose the self-publishing route for this book. Luciano shares insights into the writing process with AsciiDoc, code samples, SVG image generation, and using Gumroad for distribution. He invites feedback from listeners who have experience with Rust and Lambda.

How to secure AWS cloud using AWS Lambda? We spoke to Lily Chau from Roku at BSidesSF about her experience and innovative approach to tackling security issues in AWS environments. From deploying IAM roles to creating impactful playbooks with AWS Lambda, Lily shared her take on automating remediation processes. We spoke about the challenges of managing cloud security with tools like CSPM and CNAPP, and how Lily and her team took a different approach that goes beyond traditional methods to achieve real-time remediation. Guest Socials:⁠ ⁠⁠Lily Twitter Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Podcast- Youtube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security BootCamp Questions asked: (00:00) Introduction (01:56) A bit about Lily (02:27) What is Auto Remediation? (03:56) Example of Auto Remediation (05:19) CSPMs and Auto Remediation (06:58) Make Auto Remediation in Cloud work for you (09:49) Where to get started with Auto Remediation? (11:52) What defines a High Impact Playbook? (12:58) Auto Remediation for Lateral Movement (14:35) What is running in the background? (16:41) What skillset is required? (19:08) The Fun Section Resources for the episode: Lily's talk at BsidesSF

Scott and Wes chat with Richard Davison from AWS about LLRT, a new runtime tailored specifically for Lambda. They dive into the benefits of using LLRT, challenges with JavaScript in serverless, and why Rust was chosen for its development. Show Notes 00:00 Welcome to Syntax! 01:07 Who is Richard Davison? 05:11 What is LLRT and what's the motivation for building it? 08:25 AWS Lambda example. 11:20 What makes LLRT specifically tailored to Lambda? 14:55 Brought to you by Sentry.io. 15:22 Node.js in Lambda. 16:00 What are some challenges that people have with JavaScript in serverless? 17:20 Lambda memory configuration. 19:23 Managing cost of compute. 21:29 Simpler and faster than Node, Bun, Dino, but not a replacement. 22:31 The benchmarks. 27:00 Quick.js, the main reason for the performance gains. Fabrice Bellard QuickJS. 28:03 The Quick.js engine. 30:35 What was the reason behind creating Quick.js? 33:46 What made you pick Rust for LLRT? 36:34 Abstractions and the value of speed. 39:08 The JIT Compiler. 42:38 Compile cache. 43:27 De-optimizations. 44:59 Node.js Compat, what to use and avoid with LLRT. GitHub AWS Labs Compatibility Chart. 47:52 Will you target with WinterCG spec? 50:22 Streams API. 52:06 What about WebSockets? 53:10 Is this going to be promoted from a labs project? 54:49 Sick Picks + Shameless Plugs. Sick Picks Richard: QuickJS Engine, JSLinux. Shameless Plugs Richard: Javascript Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

On this episode of the Six Five Webcast - AWS Serverless Series, Keith Townsend is joined by Amazon Web Services' AWS Lambda Usman Khalid and Spencer Dillard for a conversation on leveraging AWS Serverless technologies to achieve rapid innovation without compromising security. Their discussion covers: The advantages of the Serverless operating model versus traditional application development Common security challenges in modern application development and how AWS addresses these The shared responsibility model for securing Serverless applications on AWS Built-in protections provided by AWS Serverless services like AWS Lambda and Amazon ECS with AWS Fargate How the ephemeral nature of Serverless resources contributes to security  

Poor air quality is a major global issue causing around 7 million premature deaths per year, disproportionately affecting low and middle income countries. OpenAQ is an organization dedicated to identifying solutions to this large issue. OpenAQ uses Amazon Web Services (AWS) to collect and harmonize open air quality data from government sources and low-cost sensors around the world, ingesting around 10-12 million measurements daily. To learn more, the Fix This team sat down with Russ Biggs, director of technology at OpenAQ. Russ shared how AWS Lambda helps OpenAQ scale its data collection and harmonization pipeline in a serverless manner. And helps scientists, environmental justice groups, and community organizations access OpenAQ's data to study air pollution impacts, advocate for regulations, and raise awareness.

What is the state of serverless computing and Python in 2024? What are some of the new tools and best practices? We are lucky to have Tony Sherman who has a lot of practical experience with serverless programming on the show. Episode sponsors Sentry Error Monitoring, Code TALKPYTHON Mailtrap Talk Python Courses Links from the show Tony Sherman on Twitter: twitter.com Tony Sherman: linkedin.com PyCon serverless talk: youtube.com AWS re:Invent talk: youtube.com Powertools for AWS Lambda: docs.powertools.aws.dev Pantsbuild: The ergonomic build system: pantsbuild.org aws-lambda-power-tuning: github.com import-profiler: github.com AWS Fargate: aws.amazon.com Run functions on demand. Scale automatically.: digitalocean.com Vercel: vercel.com Deft: deft.com 37 Signals We stand to save $7m over five years from our cloud exit: world.hey.com The Global Content Delivery Platform That Truly Hops: bunny.net Watch this episode on YouTube: youtube.com --- Stay in touch with us --- Subscribe to us on YouTube: youtube.com Follow Talk Python on Mastodon: talkpython Follow Michael on Mastodon: mkennedy

Welcome to part four in the AWS Certification Exam Prep Mini-Series! Whether you're an aspiring cloud enthusiast or a seasoned developer looking to deepen your architectural acumen, you've landed in the perfect spot. In this six-part saga, we're demystifying the pivotal role of a Solutions Architect in the AWS cloud computing cosmos. In this fourth episode, Caroline and Dave chat again with Anya Derbakova, a Senior Startup Solutions Architect at AWS, known for weaving social media magic, and Ted Trentler, a Senior AWS Technical Instructor with a knack for simplifying the complex. Together, we will step into the realm of performance, where we untangle the complexities of designing high-performing architectures in the cloud. We dissect the essentials of high-performing storage solutions, dive deep into elastic compute services for scaling and cost efficiency, and unravel the intricacies of optimizing database solutions for unparalleled performance. Expect to uncover: • The spectrum of AWS storage services and their optimal use cases, from Amazon S3's versatility to the shared capabilities of Amazon EFS. • How to leverage Amazon EC2, Auto Scaling, and Load Balancing to create elastic compute solutions that adapt to your needs. • Insights into serverless computing paradigms with AWS Lambda and Fargate, highlighting the shift towards de-coupled architectures. • Strategies for selecting high-performing database solutions, including the transition from on-premise databases to AWS-managed services like RDS and the benefits of caching with Amazon ElastiCache. • A real-world scenario where we'll navigate the challenge of processing hundreds of thousands of online votes in minutes, testing your understanding and application of high-performing AWS architectures. Whether you're dealing with vast amounts of data, requiring robust compute power, or ensuring your architecture can handle peak loads without a hitch, we've got you covered! Anya on LinkedIn: https://www.linkedin.com/in/annadderbakova/ Ted on Twitter: https://twitter.com/ttrentler Ted on LinkedIn: https://linkedin/in/tedtrentler Caroline on Twitter: https://twitter.com/carolinegluck Caroline on LinkedIn: https://www.linkedin.com/in/cgluck/ Dave on Twitter: https://twitter.com/thedavedev Dave on LinkedIn: https://www.linkedin.com/in/davidisbitski AWS SAA Exam Guide - https://d1.awsstatic.com/training-and-certification/docs-sa-assoc/AWS-Certified-Solutions-Architect-Associate_Exam-Guide.pdf Party Rock for Exam Study - https://partyrock.aws/u/tedtrent/KQtYIhbJb/Solutions-Architect-Study-Buddy All Things AWS Training - Links to Self-paced and Instructor Led https://aws.amazon.com/training/ AWS Skill Builder – Free CPE Course - https://explore.skillbuilder.aws/learn/course/134/aws-cloud-practitioner-essentials AWS Skill Builder – Learning Badges - https://explore.skillbuilder.aws/learn/public/learning_plan/view/1044/solutions-architect-knowledge-badge-readiness-path AWS Usergroup Communities: https://aws.amazon.com/developer/community/usergroups Subscribe: Spotify: https://open.spotify.com/show/7rQjgnBvuyr18K03tnEHBI Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-developers-podcast/id1574162669 Stitcher: https://www.stitcher.com/show/1065378 Pandora: https://www.pandora.com/podcast/aws-developers-podcast/PC:1001065378 TuneIn: https://tunein.com/podcasts/Technology-Podcasts/AWS-Developers-Podcast-p1461814/ Amazon Music: https://music.amazon.com/podcasts/f8bf7630-2521-4b40-be90-c46a9222c159/aws-developers-podcast Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zb3VuZGNsb3VkLmNvbS91c2Vycy9zb3VuZGNsb3VkOnVzZXJzOjk5NDM2MzU0OS9zb3VuZHMucnNz RSS Feed: https://feeds.soundcloud.com/users/soundcloud:users:994363549/sounds.rss

In this episode, I spoke with Ian Griffiths, a Technical Fellow at Endjin, a Microsoft MVP, and the author of O'Reilly's Programming C# 10.0.In this episode, we took a deep dive into Azure Functions, how it works and how it differs (significantly) from AWS Lambda.Links from the episode:Bye bye Azure Functions, hello Azure Container AppsWASI (WebAssembly system interface)EndjinIntroductions to Reactive Extensions for .NetOpening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0

On this episode of The Six Five, hosts Patrick Moorhead and Daniel Newman are joined by AWS's Holly Mesrobian, VP Serverless Compute for a conversation on the integration and future of AWS Serverless Compute, including Lambda, ECS, Fargate, and Event-Driven Architectures. Our discussion covers: Navigating Kubernetes complexities with AWS Serverless Celebrating 10 years of AWS Lambda and its impact on customers How AWS Serverless changes application architecture towards event-driven models Integration of applications within the AWS portfolio akin to iPaaS Meeting the demands of Generative AI workloads with serverless solutions Learn more at AWS.

Many people are using serverless these days, often without even really thinking about it because they are the default mode of operation for many popular services like NextJS with Vercel. But what is it? Should you use it? Join Richard and I as we discuss serverless, how it works, and how we consider using it.Join The Reactors! thereactshow.com/the-reactors-communityJoin our Discord! https://discord.gg/zXYggKUBC2My book: Foundations of High-Performance React https://www.thereactshow.com/bookMusic by DRKST DWN: https://soundcloud.com/drkstdwnSupport the show

An airhacks.fm conversation with Jose Paumard (@JosePaumard) about: Jose previously on "#271 From Image Recognition to CoffeeCast", Joses favourite Java 21 feature: virtual threads, Project Loom, Tomas Langer on airhacks.fm: "#58 Helidon: Never Block The Thread", the advantages of structured concurrency, scope values: immutability and passing, the challenges of debugging reactive code, replacing thread locals, Scala, concurrency and plain Java features, ThreadLocal vs. Scoped Values (JEP-464), AWS Lambda and virtual threads, running same code in serverless or threaded environments with quarkus Jose Paumard on twitter: @JosePaumard

On this episode, Heitor Lessa, Chief Architect at AWS, shares some insights on the journey of Powertools for AWS Lambda and the practices involved in growing and maintaining an open source community.

In this episode we join AWS Solutions Architect Vihang Shah to talk about strategies to optimize costs and performance with AWS Lambda. We will share insights into real world scenarios and provide actionable tips to strike a balance between performance efficiency and cost savings in the realm of Serverless computing. Its time to optimize your Serverless journey with AWS Lambda!AWS Hosts: Nolan Chen & Malini ChatterjeeEmail Your Feedback: rethinkpodcast@amazon.comLinks for the show: Learn about Serverless: https://aws.amazon.com/training/learn-about/serverless/Fundamentals of AWS Lambda: https://serverlessland.com/content/service/lambda/guides/aws-lambda-fundamentals/what-is-aws-lambdaServerless Optimization Workshop (Performance and Cost): https://catalog.us-east-1.prod.workshops.aws/workshops/2d960419-7d15-44e7-b540-fd3ebeb7ce2e/en-US

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack (https://www.dataengineeringpodcast.com/rudderstack) You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It's the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it's real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize (https://www.dataengineeringpodcast.com/materialize) today to get 2 weeks free! Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis functions enable developers and data engineers to build an organizational toolbox of functions to process, transform, and enrich ingested events “on the fly” in a serverless manner using AWS Lambda syntax, without boilerplate, orchestration, error handling, and infrastructure in almost any language, including Go, Python, JS, .NET, Java, SQL, and more. Go to dataengineeringpodcast.com/memphis (https://www.dataengineeringpodcast.com/memphis) today to get started! Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst (https://www.dataengineeringpodcast.com/starburst) and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino. Your host is Tobias Macey and today I'll be sharing an update on my own journey of building a data platform, with a particular focus on the challenges of tool integration and maintaining a single source of truth Interview Introduction How did you get involved in the area of data management? data sharing weight of history existing integrations with dbt switching cost for e.g. SQLMesh de facto standard of Airflow Single source of truth permissions management across application layers Database engine Storage layer in a lakehouse Presentation/access layer (BI) Data flows dbt -> table level lineage orchestration engine -> pipeline flows task based vs. asset based Metadata platform as the logical place for horizontal view Contact Info LinkedIn (https://linkedin.com/in/tmacey) Website (https://www.dataengineeringpodcast.com) Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning. Visit the site (https://www.dataengineeringpodcast.com) to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com (mailto:hosts@dataengineeringpodcast.com)) with your story. To help other people find the show please leave a review on Apple Podcasts (https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557) and tell your friends and co-workers Links Monologue Episode On Data Platform Design (https://www.dataengineeringpodcast.com/data-platform-design-episode-268) Monologue Episode On Leaky Abstractions (https://www.dataengineeringpodcast.com/abstractions-and-technical-debt-episode-374) Airbyte (https://airbyte.com/) Podcast Episode (https://www.dataengineeringpodcast.com/airbyte-open-source-data-integration-episode-173/) Trino (https://trino.io/) Dagster (https://dagster.io/) dbt (https://www.getdbt.com/) Snowflake (https://www.snowflake.com/en/) BigQuery (https://cloud.google.com/bigquery) OpenMetadata (https://open-metadata.org/) OpenLineage (https://openlineage.io/) Data Platform Shadow IT Episode (https://www.dataengineeringpodcast.com/shadow-it-data-analytics-episode-121) Preset (https://preset.io/) LightDash (https://www.lightdash.com/) Podcast Episode (https://www.dataengineeringpodcast.com/lightdash-exploratory-business-intelligence-episode-232/) SQLMesh (https://sqlmesh.readthedocs.io/) Podcast Episode (https://www.dataengineeringpodcast.com/sqlmesh-open-source-dataops-episode-380) Airflow (https://airflow.apache.org/) Spark (https://spark.apache.org/) Flink (https://flink.apache.org/) Tabular (https://tabular.io/) Iceberg (https://iceberg.apache.org/) Open Policy Agent (https://www.openpolicyagent.org/) The intro and outro music is from The Hug (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug) by The Freak Fandango Orchestra (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/) / CC BY-SA (http://creativecommons.org/licenses/by-sa/3.0/)

AWS Morning Brief for the week of November 13, 2023, with Corey Quinn. Show Notes:Links: Amazon Aurora Global Database for PostgreSQL now supports write forwarding Amazon SQS announces support for JSON protocol AWS Cost Management now provides purchase recommendations for Amazon MemoryDB Reserved Nodes Introducing the Generative AI Center of Excellence for AWS Partners: The Path to AI Expertise New – Block Public Sharing of Amazon EBS Snapshots New for Amazon Comprehend – Toxicity Detection AWS CodeBuild adds support for AWS Lambda compute mode An Overview of Bulk Sender Changes at Yahoo/Gmail Creating a correction of errors document Know Before You Go – AWS re:Invent 2023 | AWS Management Console Unhoused individuals gain shelter, prove their identity using AWS-powered solution Kiip How VMware partnered with AWS to nurture a culture of sustainability How to Stop Feeding AWS's AI With Your Data

In this episode, I spoke with Ran Isenberg, who is an AWS Serverless Hero and Principal Software Architect at CyberArk. Amongst other things, we talked about platform engineering at CyberArk, how they adopted CDK and how they approach testing and tenant isolations.Links from the episode:* Ran's blog* Open positions at CyberArk* cdk-nag* Ran's AWS Lambda cookbook* See Ran speak at re:Invent, session OPN305* My approach towards serverless testing* My course on serverless testing* Episode 85 with Matt Bonig about CDK dos & don'tsYou can find Ran on X as @IsenbergRan-----For more stories about real-world use of serverless technologies, please subscribe to the channel and follow me on X as @theburningmonk.And if you're hungry for more insights, best practices, and invaluable tips on building serverless apps, make sure to subscribe to our free newsletter and elevate your serverless game! https://theburningmonk.com/subscribeOpening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0

Bret and Nirmal are joined by Ken Collins, AWS Serverless Hero and Principal Engineer at Custom Ink to discuss all things Lambda and to dig into the details of running containers in serverless.

AWS Morning Brief for the week of September 25, 2023, with Corey Quinn. Links: Today Corey is hosting a drink-up at 6 PM in Seattle at Outer Planet Brewing. If you're in town / free, come on by; let him buy you a beer. Later this week Corey will be hosting an AMA on 9/27 @ noon PDT over on YouTube. Bring questions! Accenture Extends Generative AI Capabilities to Accelerate Adoption and Value on AWS  New – Amazon EC2 M2 Pro Mac Instances Built on Apple Silicon M2 Pro Mac Mini Computers  How Chime Financial uses AWS to build a serverless stream analytics platform and defeat fraudsters  Centralizing management of AWS Lambda layers across multiple AWS Accounts Handle traffic spikes with Amazon DynamoDB provisioned capacity Streamline interstate Department of Motor Vehicles collaboration with Private Blockchain  How to host your Unreal Engine game for under $1 per player with Amazon GameLift  How United Airlines built a cost-efficient Optical Character Recognition active learning pipeline How VirtuSwap accelerates their pandas ... -based trading simulations with an Amazon SageMaker Studio custom container and AWS GPU instances Provision sandbox accounts with budget limits to reduce costs using AWS Control Tower Reducing the Scope of Impact with Cell-Based Architecture - Reducing the Scope of Impact with Cell-Based Architecture From Massage Therapist to Cloud Associate with AWS Academy 

AWS Morning Brief for the week of September 18, 2023 with Corey Quinn. Links: Amazon SNS FIFO topics now support message delivery to Amazon SQS Standard queues Announcing API Gateway console refresh  Cost Anomaly Detection increases custom anomaly monitor limit to 500 Custom notifications are now available for AWS Chatbot  How to Integrate Amazon CloudWatch Alarms with Atlassian Confluence Knowledge Articles  Building a secure webhook forwarder using an AWS Lambda extension and Tailscale Deploy Generative AI Models on Amazon EKS Troubleshoot networking issues during database migration with the AWS DMS diagnostic support AMI  Using AWS CloudFormation and AWS Cloud Development Kit to provision multicloud resources Combining content moderation services with graph databases & analytics to reduce community toxicity AWS Private Certificate Authority Retail Partner Conversations: How Rokt is impacting the future of retail  Simplify access to internal information using Retrieval Augmented Generation and LangChain Agents  How to view Azure costs using Amazon QuickSight  Centralized Dashboard for AWS Config and AWS Security Hub  Benefits of Domain Registration with Amazon Route 53  Use Bring your own IP addresses (BYOIP) and RFC 8805 for localization of Internet content Using NAT Gateways with multiple-Amazon VPCs at scale  Navigating change: From ophthalmologist to AWS Cloud expert

Austin Parker, Community Maintainer at OpenTelemetry, joins Corey on Screaming in the Cloud to discuss OpenTelemetry's mission in the world of observability. Austin explains how the OpenTelemetry community was able to scale the OpenTelemetry project to a commercial offering, and the way Open Telemetry is driving innovation in the data space. Corey and Austin also discuss why Austin decided to write a book on OpenTelemetry, and the book's focus on the evergreen applications of the tool. About AustinAustin Parker is the OpenTelemetry Community Maintainer, as well as an event organizer, public speaker, author, and general bon vivant. They've been a part of OpenTelemetry since its inception in 2019.Links Referenced: OpenTelemetry: https://opentelemetry.io/ Learning OpenTelemetry early release: https://www.oreilly.com/library/view/learning-opentelemetry/9781098147174/ Page with Austin's social links: https://social.ap2.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Look, I get it. Folks are being asked to do more and more. Most companies don't have a dedicated DBA because that person now has a full time job figuring out which one of AWS's multiple managed database offerings is right for every workload. Instead, developers and engineers are being asked to support, and heck, if time allows, optimize their databases. That's where OtterTune comes in. Their AI is your database co-pilot for MySQL and PostgresSQL on Amazon RDS or Aurora. It helps improve performance by up to four x OR reduce costs by 50 percent – both of those are decent options. Go to ottertune dot com to learn more and start a free trial. That's O-T-T-E-R-T-U-N-E dot com.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's been a few hundred episodes since I had Austin Parker on to talk about the things that Austin cares about. But it's time to rectify that. Austin is the community maintainer for OpenTelemetry, which is a CNCF project. If you're unfamiliar with, we're probably going to fix that in short order. Austin, Welcome back, it's been a month of Sundays.Austin: It has been a month-and-a-half of Sundays. A whole pandemic-and-a-half.Corey: So, much has happened since then. I tried to instrument something with OpenTelemetry about a year-and-a-half ago, and in defense to the project, my use case is always very strange, but it felt like—a lot of things have sharp edges, but it felt like this had so many sharp edges that you just pivot to being a chainsaw, and I would have been at least a little bit more understanding of why it hurts so very much. But I have heard from people that I trust that the experience has gotten significantly better. Before we get into the nitty-gritty of me lobbing passive-aggressive bug reports at you have for you to fix in a scenario in which you can't possibly refuse me, let's start with the beginning. What is OpenTelemetry?Austin: That's a great question. Thank you for asking it. So, OpenTelemetry is an observability framework. It is run by the CNCF, you know, home of such wonderful award-winning technologies as Kubernetes, and you know, the second biggest source of YAML in the known universe [clear throat].Corey: On some level, it feels like that is right there with hydrogen as far as unlimited resources in our universe.Austin: It really is. And, you know, as we all know, there are two things that make, sort of, the DevOps and cloud world go around: one of them being, as you would probably know, AWS bills; and the second being YAML. But OpenTelemetry tries to kind of carve a path through this, right, because we're interested in observability. And observability, for those that don't know or have been living under a rock or not reading blogs, it's a lot of things. It's a—but we can generally sort of describe it as, like, this is how you understand what your system is doing.I like to describe it as, it's a way that we can model systems, especially complex, distributed, or decentralized software systems that are pretty commonly found in larg—you know, organizations of every shape and size, quite often running on Kubernetes, quite often running in public or private clouds. And the goal of observability is to help you, you know, model this system and understand what it's doing, which is something that I think we can all agree, a pretty important part of our job as software engineers. Where OpenTelemetry fits into this is as the framework that helps you get the telemetry data you need from those systems, put it into a universal format, and then ship it off to some observability back-end, you know, a Prometheus or a Datadog or whatever, in order to analyze that data and get answers to your questions you have.Corey: From where I sit, the value of OTel—or OpenTelemetry; people in software engineering love abbreviations that are impenetrable from the outside, so of course, we're going to lean into that—but what I found for my own use case is the shining value prop was that I could instrument an application with OTel—in theory—and then send whatever I wanted that was emitted in terms of telemetry, be it events, be it logs, be it metrics, et cetera, and send that to any or all of a curation of vendors on a case-by-case basis, which meant that suddenly it was the first step in, I guess, an observability pipeline, which increasingly is starting to feel like a milit—like an industrial-observability complex, where there's so many different companies out there, it seems like a good approach to use, to start, I guess, racing vendors in different areas to see which performs better. One of the challenges I've had with that when I started down that path is it felt like every vendor who was embracing OTel did it from a perspective of their implementation. Here's how to instrument it to—send it to us because we're the best, obviously. And you're a community maintainer, despite working at observability vendors yourself. You have always been one of those community-first types where you care more about the user experience than you do this quarter for any particular employer that you have, which to be very clear, is intended as a compliment, not a terrifying warning. It's why you have this authentic air to you and why you are one of those very few voices that I trust in a space where normally I need to approach it with significant skepticism. How do you see the relationship between vendors and OpenTelemetry?Austin: I think the hard thing is that I know who signs my paychecks at the end of the day, right, and you always have, you know, some level of, you know, let's say bias, right? Because it is a bias to look after, you know, them who brought you to the dance. But I think you can be responsible with balancing, sort of, the needs of your employer, and the needs of the community. You know, the way I've always described this is that if you think about observability as, like, a—you know, as a market, what's the total addressable market there? It's literally everyone that uses software; it's literally every software company.Which means there's plenty of room for people to make their numbers and to buy and sell and trade and do all this sort of stuff. And by taking that approach, by taking sort of the big picture approach and saying, “Well, look, you know, there's going to be—you know, of all these people, there are going to be some of them that are going to use our stuff and there are some of them that are going to use our competitor's stuff.” And that's fine. Let's figure out where we can invest… in an OpenTelemetry, in a way that makes sense for everyone and not just, you know, our people. So, let's build things like documentation, right?You know, one of the things I'm most impressed with, with OpenTelemetry over the past, like, two years is we went from being, as a project, like, if you searched for OpenTelemetry, you would go and you would get five or six or ten different vendor pages coming up trying to tell you, like, “This is how you use it, this is how you use it.” And what we've done as a community is we've said, you know, “If you go looking for documentation, you should find our website. You should find our resources.” And we've managed to get the OpenTelemetry website to basically rank above almost everything else when people are searching for help with OpenTelemetry. And that's been really good because, one, it means that now, rather than vendors or whoever coming in and saying, like, “Well, we can do this better than you,” we can be like, “Well, look, just, you know, put your effort here, right? It's already the top result. It's already where people are coming, and we can prove that.”And two, it means that as people come in, they're going to be put into this process of community feedback, where they can go in, they can look at the docs, and they can say, “Oh, well, I had a bad experience here,” or, “How do I do this?” And we get that feedback and then we can improve the docs for everyone else by acting on that feedback, and the net result of this is that more people are using OpenTelemetry, which means there are more people kind of going into the tippy-tippy top of the funnel, right, that are able to become a customer of one of these myriad observability back ends.Corey: You touched on something very important here, when I first was exploring this—you may have been looking over my shoulder as I went through this process—my impression initially was, oh, this is a ‘CNCF project' in quotes, where—this is not true universally, of course, but there are cases where it clearly—is where this is an, effectively, vendor-captured project, not necessarily by one vendor, but by an almost consortium of them. And that was my takeaway from OpenTelemetry. It was conversations with you, among others, that led me to believe no, no, this is not in that vein. This is clearly something that is a win. There are just a whole bunch of vendors more-or-less falling all over themselves, trying to stake out thought leadership and imply ownership, on some level, of where these things go. But I definitely left with a sense that this is bigger than any one vendor.Austin: I would agree. I think, to even step back further, right, there's almost two different ways that I think vendors—or anyone—can approach OpenTelemetry, you know, from a market perspective, and one is to say, like, “Oh, this is socializing, kind of, the maintenance burden of instrumentation.” Which is a huge cost for commercial players, right? Like, if you're a Datadog or a Splunk or whoever, you know, you have these agents that you go in and they rip telemetry out of your web servers, out of your gRPC libraries, whatever, and it costs a lot of money to pay engineers to maintain those instrumentation agents, right? And the cynical take is, oh, look at all these big companies that are kind of like pushing all that labor onto the open-source community, and you know, I'm not casting any aspersions here, like, I do think that there's an element of truth to it though because, yeah, that is a huge fixed cost.And if you look at the actual lived reality of people and you look at back when SignalFx was still a going concern, right, and they had their APM agents open-sourced, you could go into the SignalFx repo and diff, like, their [Node Express 00:10:15] instrumentation against the Datadog Node Express instrumentation, and it's almost a hundred percent the same, right? Because it's truly a commodity. There's no—there's nothing interesting about how you get that telemetry out. The interesting stuff all happens after you have the telemetry and you've sent it to some back-end, and then you can, you know, analyze it and find interesting things. So, yeah, like, it doesn't make sense for there to be five or six or eight different companies all competing to rebuild the same wheels over and over and over and over when they don't have to.I think the second thing that some people are starting to understand is that it's like, okay, let's take this a step beyond instrumentation, right? Because the goal of OpenTelemetry really is to make sure that this instrumentation is native so that you don't need a third-party agent, you don't need some other process or jar or whatever that you drop in and it instruments stuff for you. The JVM should provide this, your web framework should provide this, your RPC library should provide this right? Like, this data should come from the code itself and be in a normalized fashion that can then be sent to any number of vendors or back ends or whatever. And that changes how—sort of, the competitive landscape a lot, I think, for observability vendors because rather than, kind of, what you have now, which is people will competing on, like, well, how quickly can I throw this agent in and get set up and get a dashboard going, it really becomes more about, like, okay, how are you differentiating yourself against every other person that has access to the same data, right? And you get more interesting use cases and how much more interesting analysis features, and that results in more innovation in, sort of, the industry than we've seen in a very long time.Corey: For me, just from the customer side of the world, one of the biggest problems I had with observability in my career as an SRE-type for years was you would wind up building your observability pipeline around whatever vendor you had selected and that meant emphasizing the things they were good at and de-emphasizing the things that they weren't. And sometimes it's worked to your benefit; usually not. But then you always had this question when it got things that touched on APM or whatnot—or Application Performance Monitoring—where oh, just embed our library into this. Okay, great. But a year-and-a-half ago, my exposure to this was on an application that I was running in distributed fashion on top of AWS Lambda.So great, you can either use an extension for this or you can build in the library yourself, but then there's always a question of precedence where when you have multiple things that are looking at this from different points of view, which one gets done first? Which one is going to see the others? Which one is going to enmesh the other—enclose the others in its own perspective of the world? And it just got incredibly frustrating. One of the—at least for me—bright lights of OTel was that it got away from that where all of the vendors receiving telemetry got the same view.Austin: Yeah. They all get the same view, they all get the same data, and you know, there's a pretty rich collection of tools that we're starting to develop to help you build those pipelines yourselves and really own everything from the point of generation to intermediate collection to actually outputting it to wherever you want to go. For example, a lot of really interesting work has come out of the OpenTelemetry collector recently; one of them is this feature called Connectors. And Connectors let you take the output of certain pipelines and route them as inputs to another pipeline. And as part of that connection, you can transform stuff.So, for example, let's say you have a bunch of [spans 00:14:05] or traces coming from your API endpoints, and you don't necessarily want to keep all those traces in their raw form because maybe they aren't interesting or maybe there's just too high of a volume. So, with Connectors, you can go and you can actually convert all of those spans into metrics and export them to a metrics database. You could continue to save that span data if you want, but you have options now, right? Like, you can take that span data and put it into cold storage or put it into, like, you know, some sort of slow blob storage thing where it's not actively indexed and it's slow lookups, and then keep a metric representation of it in your alerting pipeline, use metadata exemplars or whatever to kind of connect those things back. And so, when you do suddenly see it's like, “Oh, well, there's some interesting p99 behavior,” or we're hitting an alert or violating an SLO or whatever, then you can go back and say, like, “Okay, well, let's go dig through the slow da—you know, let's look at the cold data to figure out what actually happened.”And those are features that, historically, you would have needed to go to a big, important vendor and say, like, “Hey, here's a bunch of money,” right? Like, “Do this for me.” Now, you have the option to kind of do all that more interesting pipeline stuff yourself and then make choices about vendors based on, like, who is making a tool that can help me with the problem that I have? Because most of the time, I don't—I feel like we tend to treat observability tools as—it depends a lot on where you sit in the org—but you certainly seen this movement towards, like, “Well, we don't want a tool; we want a platform. We want to go to Lowe's and we want to get the 48-in-one kit that has a bunch of things in it. And we're going to pay for the 48-in-one kit, even if we only need, like, two things or three things out of it.”OpenTelemetry lets you kind of step back and say, like, “Well, what if we just got, like, really high-quality tools for the two or three things we need, and then for the rest of the stuff, we can use other cheaper options?” Which is, I think, really attractive, especially in today's macroeconomic conditions, let's say.Corey: One thing I'm trying to wrap my head around because we all find when it comes to observability, in my experience, it's the parable of three blind people trying to describe an elephant by touch; depending on where you are on the elephant, you have a very different perspective. What I'm trying to wrap my head around is, what is the vision for OpenTelemetry? Is it specifically envisioned to be the agent that runs wherever the workload is, whether it's an agent on a host or a layer in a Lambda function, or a sidecar or whatnot in a Kubernetes cluster that winds up gathering and sending data out? Or is the vision something different? Because part of what you're saying aligns with my perspective on it, but other parts of it seem to—that there's a misunderstanding somewhere, and it's almost certainly on my part.Austin: I think the long-term vision is that you as a developer, you as an SRE, don't even have to think about OpenTelemetry, that when you are using your container orchestrator or you are using your API framework or you're using your Managed API Gateway, or any kind of software that you're building something with, that the telemetry data from that software is emitted in an OpenTelemetry format, right? And when you are writing your code, you know, and you're using gRPC, let's say, you could just natively expect that OpenTelemetry is kind of there in the background and it's integrated into the actual libraries themselves. And so, you can just call the OpenTelemetry API and it's part of the standard library almost, right? You add some additional metadata to a span and say, like, “Oh, this is the customer ID,” or, “This is some interesting attribute that I want to track for later on,” or, “I'm going to create a histogram here or counter,” whatever it is, and then all that data is just kind of there, right, invisible to you unless you need it. And then when you need it, it's there for you to kind of pick up and send off somewhere to any number of back-ends or databases or whatnot that you could then use to discover problems or better model your system.That's the long-term vision, right, that it's just there, everyone uses it. It is a de facto and du jour standard. I think in the medium term, it does look a little bit more like OpenTelemetry is kind of this Swiss army knife agent that's running on—inside cars in Kubernetes or it's running on your EC2 instance. Until we get to the point of everyone just agrees that we're going to use OpenTelemetry protocol for the data and we're going to use all your stuff and we just natively emit it, then that's going to be how long we're in that midpoint. But that's sort of the medium and long-term vision I think. Does that track?Corey: It does. And I'm trying to equate this to—like the evolution back in the Stone Age was back when I was first getting started, Nagios was the gold standard. It was kind of the original Call of Duty. And it was awful. There were a bunch of problems with it, but it also worked.And I'm not trying to dunk on the people who built that. We all stand on the shoulders of giants. It was an open-source project that was awesome doing exactly what it did, but it was a product built for a very different time. It completely had the wheels fall off as soon as you got to things were even slightly ephemeral because it required this idea of the server needed to know where all of the things that was monitoring lived as an individual host basis, so there was this constant joy of, “Oh, we're going to add things to a cluster.” Its perspective was, “What's a cluster?” Or you'd have these problems with a core switch going down and suddenly everything else would explode as well.And even setting up an on-call rotation for who got paged when was nightmarish. And a bunch of things have evolved since then, which is putting it mildly. Like, you could say that about fire, the invention of the wheel. Yeah, a lot of things have evolved since the invention of the wheel, and here we are tricking sand into thinking. But we find ourselves just—now it seems that the outcome of all of this has been instead of one option that's the de facto standard that's kind of terrible in its own ways, now, we have an entire universe of different products, many of which are best-of-breed at one very specific thing, but nothing's great at everything.It's the multifunction printer conundrum, where you find things that are great at one or two things at most, and then mediocre at best at the rest. I'm excited about the possibility for OpenTelemetry to really get to a point of best-of-breed for everything. But it also feels like the money folks are pushing for consolidation, if you believe a lot of the analyst reports around this of, “We already pay for seven different observability vendors. How about we knock it down to just one that does all of these things?” Because that would be terrible. What do you land on that?Austin: Well, as I intu—or alluded to this earlier, I think the consolidation in the observability space, in general, is very much driven by that force you just pointed out, right? The buyers want to consolidate more and more things into single tools. And I think there's a lot of… there are reasons for that that—you know, there are good reasons for that, but I also feel like a lot of those reasons are driven by fundamentally telemetry-side concerns, right? So like, one example of this is if you were Large Business X, and you see—you are an engineering director and you get a report, that's like, “We have eight different metrics products.” And you're like, “That seems like a lot. Let's just use Brand X.”And Brand X will tell you very, very happily tell you, like, “Oh, you just install our thing everywhere and you can get rid of all these other tools.” And usually, there's two reasons that people pick tools, right? One reason is that they are forced to and then they are forced to do a bunch of integration work to get whatever the old stuff was working in the new way, but the other reason is because they tried a bunch of different things and they found the one tool that actually worked for them. And what happens invariably in these sort of consolidation stories is, you know, the new vendor comes in on a shining horse to consolidate, and you wind up instead of eight distinct metrics tools, now you have nine distinct metrics tools because there's never any bandwidth for people to go back and, you know—you're Nagios example, right, Nag—people still use Nagios every day. What's the economic justification to take all those Nagios installs, if they're working, and put them into something else, right?What's the economic justification to go and take a bunch of old software that hasn't been touched for ten years that still runs and still does what needs to do, like, where's the incentive to go and re-instrument that with OpenTelemetry or anything else? It doesn't necessarily exist, right? And that's a pretty, I think, fundamental decision point in everyone's observability journey, which is what do you do about all the old stuff? Because most of the stuff is the old stuff and the worst part is, most of the stuff that you make money off of is the old stuff as well. So, you can't ignore it, and if you're spending, you know, millions of millions of dollars on the new stuff—like, there was a story that went around a while ago, I think, Coinbase spent something like, what, $60 million on Datadog… I hope they asked for it in real money and not Bitcoin. But—Corey: Yeah, something I've noticed about all the vendors, and even Coinbase themselves, very few of them actually transact in cryptocurrency. It's always cash on the barrelhead, so to speak.Austin: Yeah, smart. But still, like, that's an absurd amount of money [laugh] for any product or service, I would argue, right? But that's just my perspective. I do think though, it goes to show you that you know, it's very easy to get into these sort of things where you're just spending over the barrel to, like, the newest vendor that's going to come in and solve all your problems for you. And just, it often doesn't work that way because most places aren't—especially large organizations—just aren't built in is sort of like, “Oh, we can go through and we can just redo stuff,” right? “We can just roll out a new agent through… whatever.”We have mainframes [unintelligible 00:25:09], mainframes to thinking about, you have… in many cases, you have an awful lot of business systems that most, kind of, cloud people don't like, think about, right, like SAP or Salesforce or ServiceNow, or whatever. And those sort of business process systems are actually responsible for quite a few things that are interesting from an observability point of view. But you don't see—I mean, hell, you don't even see OpenTelemetry going out and saying, like, “Oh, well, here's the thing to let you know, observe Apex applications on Salesforce,” right? It's kind of an undiscovered country in a lot of ways and it's something that I think we will have to grapple with as we go forward. In the shorter term, there's a reason that OpenTelemetry mostly focuses on cloud-native applications because that's a little bit easier to actually do what we're trying to do on them and that's where the heat and light is. But once we get done with that, then the sky is the limit.[midroll 00:26:11]Corey: It still feels like OpenTelemetry is evolving rapidly. It's certainly not, I don't want to say it's not feature complete, which, again, what—software is never done. But it does seem like even quarter-to-quarter or month-to-month, its capabilities expand massively. Because you apparently enjoy pain, you're in the process of writing a book. I think it's in early release or early access that comes out next year, 2024. Why would you do such a thing?Austin: That's a great question. And if I ever figure out the answer I will tell you.Corey: Remember, no one wants to write a book; they want to have written the book.Austin: And the worst part is, is I have written the book and for some reason, I went back for another round. I—Corey: It's like childbirth. No one remembers exactly how horrible it was.Austin: Yeah, my partner could probably attest to that. Although I was in the room, and I don't think I'd want to do it either. So, I think the real, you know, the real reason that I decided to go and kind of write this book—and it's Learning OpenTelemetry; it's in early release right now on the O'Reilly learning platform and it'll be out in print and digital next year, I believe, we're targeting right now, early next year.But the goal is, as you pointed out so eloquently, OpenTelemetry changes a lot. And it changes month to month sometimes. So, why would someone decide—say, “Hey, I'm going to write the book about learning this?” Well, there's a very good reason for that and it is that I've looked at a lot of the other books out there on OpenTelemetry, on observability in general, and they talk a lot about, like, here's how you use the API. Here's how you use the SDK. Here's how you make a trace or a span or a log statement or whatever. And it's very technical; it's very kind of in the weeds.What I was interested in is saying, like, “Okay, let's put all that stuff aside because you don't necessarily…” I'm not saying any of that stuff's going to change. And I'm not saying that how to make a span is going to change tomorrow; it's not, but learning how to actually use something like OpenTelemetry isn't just knowing how to create a measurement or how to create a trace. It's, how do I actually use this in a production system? To my point earlier, how do I use this to get data about, you know, these quote-unquote, “Legacy systems?” How do I use this to monitor a Kubernetes cluster? What's the important parts of building these observability pipelines? If I'm maintaining a library, how should I integrate OpenTelemetry into that library for my users? And so on, and so on, and so forth.And the answers to those questions actually probably aren't going to change a ton over the next four or five years. Which is good because that makes it the perfect thing to write a book about. So, the goal of Learning OpenTelemetry is to help you learn not just how to use OpenTelemetry at an API or SDK level, but it's how to build an observability pipeline with OpenTelemetry, it's how to roll it out to an organization, it's how to convince your boss that this is what you should use, both for new and maybe picking up some legacy development. It's really meant to give you that sort of 10,000-foot view of what are the benefits of this, how does it bring value and how can you use it to build value for an observability practice in an organization?Corey: I think that's fair. Looking at the more quote-unquote, “Evergreen,” style of content as opposed to—like, that's the reason for example, I never wind up doing tutorials on how to use an AWS service because one console change away and suddenly I have to redo the entire thing. That's a treadmill I never had much interest in getting on. One last topic I want to get into before we wind up wrapping the episode—because I almost feel obligated to sprinkle this all over everything because the analysts told me I have to—what's your take on generative AI, specifically with an eye toward observability?Austin: [sigh], gosh, I've been thinking a lot about this. And—hot take alert—as a skeptic of many technological bubbles over the past five or so years, ten years, I'm actually pretty hot on AI—generative AI, large language models, things like that—but not for the reasons that people like to kind of hold them up, right? Not so that we can all make our perfect, funny [sigh], deep dream, meme characters or whatever through Stable Fusion or whatever ChatGPT spits out at us when we ask for a joke. I think the real win here is that this to me is, like, the biggest advance in human-computer interaction since resistive touchscreens. Actually, probably since the mouse.Corey: I would agree with that.Austin: And I don't know if anyone has tried to get someone that is, you know, over the age of 70 to use a computer at any time in their life, but mapping human language to trying to do something on an operating system or do something on a computer on the web is honestly one of the most challenging things that faces interface design, face OS designers, faces anyone. And I think this also applies for dev tools in general, right? Like, if you think about observability, if you think about, like, well, what are the actual tasks involved in observability? It's like, well, you're making—you're asking questions. You're saying, like, “Hey, for this metric named HTTPrequestsByCode,” and there's four or five dimensions, and you say, like, “Okay, well break this down for me.” You know, you have to kind of know the magic words, right? You have to know the magic promQL sequence or whatever else to plug in and to get it to graph that for you.And you as an operator have to have this very, very well developed, like, depth of knowledge and math and statistics to really kind of get a lot of—Corey: You must be at least this smart to ride on this ride.Austin: Yeah. And I think that, like that, to me is the real—the short-term win for certainly generative AI around using, like, large language models, is the ability to create human language interfaces to observability tools, that—Corey: As opposed to learning your own custom SQL dialect, which I see a fair number of times.Austin: Right. And, you know, and it's actually very funny because there was a while for the—like, one of my kind of side projects for the past [sigh] a little bit [unintelligible 00:32:31] idea of, like, well, can we make, like, a universal query language or universal query layer that you could ship your dashboards or ship your alerts or whatever. And then it's like, generative AI kind of just, you know, completely leapfrogs that, right? It just says, like, well, why would you need a query language, if we can just—if you can just ask the computer and it works, right?Corey: The most common programming language is about to become English.Austin: Which I mean, there's an awful lot of externalities there—Corey: Which is great. I want to be clear. I'm not here to gatekeep.Austin: Yeah. I mean, I think there's a lot of externalities there, and there's a lot—and the kind of hype to provable benefit ratio is very skewed right now towards hype. That said, one of the things that is concerning to me as sort of an observability practitioner is the amount of people that are just, like, whole-hog, throwing themselves into, like, oh, we need to integrate generative AI, right? Like, we need to put AI chatbots and we need to have ChatGPT built into our products and da-da-da-da-da. And now you kind of have this perfect storm of people that really don't ha—because they're just using these APIs to integrate gen AI stuff with, they really don't understand what it's doing because a lot you know, it is very complex, and I'll be the first to admit that I really don't understand what a lot of it is doing, you know, on the deep, on the foundational math side.But if we're going to have trust in, kind of, any kind of system, we have to understand what it's doing, right? And so, the only way that we can understand what it's doing is through observability, which means it's incredibly important for organizations and companies that are building products on generative AI to, like, drop what—you know, walk—don't walk, run towards something that is going to give you observability into these language models.Corey: Yeah. “The computer said so,” is strangely dissatisfying.Austin: Yeah. You need to have that base, you know, sort of, performance [goals and signals 00:34:31], obviously, but you also need to really understand what are the questions being asked. As an example, let's say you have something that is tokenizing questions. You really probably do want to have some sort of observability on the hot path there that lets you kind of break down common tokens, especially if you were using, like, custom dialects or, like, vectors or whatever to modify the, you know, neural network model, like, you really want to see, like, well, what's the frequency of the certain tokens that I'm getting they're hitting the vectors versus not right? Like, where can I improve these sorts of things? Where am I getting, like, unexpected results?And maybe even have some sort of continuous feedback mechanism that it could be either analyzing the tone and tenor of end-user responses or you can have the little, like, frowny and happy face, whatever it is, like, something that is giving you that kind of constant feedback about, like, hey, this is how people are actually like interacting with it. Because I think there's way too many stories right now people just kind of like saying, like, “Oh, okay. Here's some AI-powered search,” and people just, like, hating it. Because people are already very primed to distrust AI, I think. And I can't blame anyone.Corey: Well, we've had an entire lifetime of movies telling us that's going to kill us all.Austin: Yeah.Corey: And now you have a bunch of, also, billionaire tech owners who are basically intent on making that reality. But that's neither here nor there.Austin: It isn't, but like I said, it's difficult. It's actually one of the first times I've been like—that I've found myself very conflicted.Corey: Yeah, I'm a booster of this stuff; I love it, but at the same time, you have some of the ridiculous hype around it and the complete lack of attention to safety and humanity aspects of it that it's—I like the technology and I think it has a lot of promise, but I want to get lumped in with that set.Austin: Exactly. Like, the technology is great. The fan base is… ehh, maybe something a little different. But I do think that, for lack of a better—not to be an inevitable-ist or whatever, but I do think that there is a significant amount of, like, this is a genie you can't put back in the bottle and it is going to have, like, wide-ranging, transformative effects on the discipline of, like, software development, software engineering, and white collar work in general, right? Like, there's a lot of—if your job involves, like, putting numbers into Excel and making pretty spreadsheets, then ooh, that doesn't seem like something that's going to do too hot when I can just have Excel do that for me.And I think we do need to be aware of that, right? Like, we do need to have that sort of conversation about, like… what are we actually comfortable doing here in terms of displacing human labor? When we do displace human labor, are we doing it so that we can actually give people leisure time or so that we can just cram even more work down the throats of the humans that are left?Corey: And unfortunately, I think we might know what that answer is, at least on our current path.Austin: That's true. But you know, I'm an optimist.Corey: I… don't do well with disappointment. Which the show has certainly not been. I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?Austin: Welp, I—you can find me on most social media. Many, many social medias. I used to be on Twitter a lot, and we all know what happened there. The best place to figure out what's going on is check out my bio, social.ap2.io will give you all the links to where I am. And yeah, been great talking with you.Corey: Likewise. Thank you so much for taking the time out of your day. Austin Parker, community maintainer for OpenTelemetry. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment pointing out that actually, physicists say the vast majority of the universe's empty space, so that we can later correct you by saying ah, but it's empty whitespace. That's right. YAML wins again.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

AWS Morning Brief for the week of August 28, 2023, with Corey Quinn. Links: Amazon Aurora Global Database introduces Global Database Failover Amazon ElastiCache for Memcached simplifies creating new clusters in the AWS Management Console Improvements to multi-account management for Amazon GuardDuty AWS Certificate Manager introduces Enterprise Controls to help govern certificate issuance AWS Cost Explorer announces support for AWS Billing Conductor AWS Microservice Extractor now supports visualizing very large enterprise applications AWS re:Post launches an enhanced search experience Announcing AWS ROSA console support for the ROSA with hosted control planes preview EC2 Hibernate now supports Amazon EC2 M7i and M7i-flex instances Manage Cost Allocation Tags with Last-Updated and Last-Used timestamps  Protecting an AWS Lambda function URL with Amazon CloudFront and Lambda@Edge  Choose AWS Graviton and cloud storage for your Ethereum nodes infrastructure on AWS  How Amazon Finance Technologies built an event-driven and scalable remittance service using Amazon DynamoDB Upgrade from Amazon Aurora Serverless v1 to v2 with minimal downtime Next Big Things for Retail – Generative AI leads the pack but isn't alone Explain medical decisions in clinical settings using Amazon SageMaker Clarify Build a serverless store finder site using Amazon Location Service Configuring client IP address preservation with a Network Load Balancer in AWS Global Accelerator  How to use pulse-level control on OQC's superconducting quantum computer AWS Digital Sovereignty Pledge: Announcing new dedicated infrastructure options

Last week in security news: People are still discovering some effects of the latest Azure security breach, Introducing the first AWS Security Heroes, How to Receive Alerts When Your IAM Configuration Changes, and more!Links: Following the latest Azure breach, the CEO of Tenable says they can see banking customer credentials even now. Introducing the first AWS Security Heroes How to Receive Alerts When Your IAM Configuration Changes  Perform continuous vulnerability scanning of AWS Lambda functions with Amazon Inspector Recent Software-based Power Side-Channel Security Research  You can totally use AWS's SSM agent as post-exploitation RAT malware

Joël has been fighting a frustrating bug where he's integrating with a third-party database, and some queries just crash. Stephanie shares her own debugging story about a leaky stub that caused flaky tests. Additionally, they discuss the build vs. buy decision when integrating with third-party systems. They consider the time and cost implications of building their own integration versus using off-the-shelf components and conclude that the decision often depends on the specific needs and priorities of the project, including how quickly a solution is needed and whether the integration is core to the business's value proposition. Ruby class instance variables (https://www.codegram.com/blog/understanding-class-instance-variables-in-ruby/) Build vs Buy by Josh Clayton (https://thoughtbot.com/blog/build-vs-buy-considerations-for-new-products) Sustainable Rails (https://sustainable-rails.com/) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: My world has been kind of frustrating recently. I've been fighting a really frustrating bug where I'm integrating with a third-party database. And there are queries that just straight-up crash. Any query that instantiates an instance of an ActiveRecord object will just straight-up fail. And that's because before, we make the actual query, almost like a preflight query that fetches the schema of the database, particularly the list of tables that the database has, and there's something in this schema that the code doesn't like, and everything just crashes. Specifically, I'm using an ODBC connection. I forget exactly what the acronym stands for, Open Database connection, maybe? Which is a standard put up by Microsoft. The way I'm integrating it via Ruby is there's a gem that's a C extension. And somewhere deep in the C extension, this whole thing is crashing. So, I've had to sort of dust off some C a little bit to look through. And it's not super clear exactly why things are crashing. So, I've spent several days trying to figure out what's going on there. And it's been really cryptic. STEPHANIE: Yeah, that does sound frustrating. And it seems like maybe you are a little bit out of your depth in terms of your usual tools for figuring out a bug are not so helpful here. JOËL: Yeah, yeah. It's a lot harder to just go through and put in a print or a debug statement because now I have to recompile some C. And, you know, you can mess around with some things by passing different flags. But it is a lot more difficult than just doing, like, a bundle open and binding to RB in the code. My ultimate solution was asking for help. So, I got another thoughtboter to help me, and we paired on it. We got to a solution that worked. And then, right before I went to deploy this change, because this was breaking on the staging website, I refreshed the website just to make sure that everything was breaking before I pushed the fix to see that everything is working. This is a habit I've picked up from test-driven development. You always want to see your test break before you see it succeed. And this is a situation where this habit paid off because the website was just working. My changes were not deployed. It just started working again. Now it's gotten me just completely questioning whether my solution fixes anything. The difficulty is because I am integrating [inaudible 03:20] third-party database; it's non-deterministic. The schema on there is changing rather frequently. I think the reason things are crashing is because there's some kind of bad data or data that the ODBC adapter doesn't like in this third-party system. But it just got introduced one day; everything started breaking, and then somehow it got removed, and everything is working again without any input or code changes on my end. So, now I don't trust my fix. STEPHANIE: Oh no. Yeah, I would struggle with that because your reality has come crashing down, [laughs] or how you understood reality. That's tough. Where do you think you'll go from here? If it's no longer really an issue in this current state of the schema, is it worth pursuing further at this time? JOËL: So, that's interesting because it turns into a prioritization problem. And for this particular project, with the deadlines that we have, we've decided it's not worth it. I've opened up a PR with my fix, with some pretty in-depth documentation for why I thought that was the fix and what I think the underlying problem is. If this shows up again in the next few days, I'll have that PR that I can pull in and see if it fixes things, and if it doesn't, I'll probably just close that PR, but it'll be available for us if we ever run into this again. I've also looked at a few potential mitigating situations. Part of the problem is that this is a, like, massive system. The Rails app that I'm using really doesn't need to deal with this massive database. I think there's, you know, almost 1,000 tables, and I really only care about a subset of tables in, like, one underlying schema. And so, I think by reducing the permissions of my database user to only those tables that I care about, there's a lower chance of me triggering something like this. STEPHANIE: Interesting. What you mentioned about, you know, having that PR continue to exist will be really helpful for future folks who might come across the same problem, right? Because then they can see, like, all of the research and investigation you've already done. And you may have already done this, but if you do think it's a schema issue, I'm curious about whether the snapshot of the schema could be captured from when it was failing to when it has magically gotten fixed. And I wonder if there may be some clues there for some future investigator. JOËL: Yeah. I'm not sure what our backup situation is because this is a third-party system, so I'd have to figure out what things are like in the admin interface there. But yeah, if there is some kind of auditing, or snapshots, or backups, or something there, and I have rough, you know, if I know it's within a 24-hour period, maybe there's something there that would tell me what's happening. My best guess is that there's some string that is longer than expected or maybe being marked as a CHAR when it should be a VARCHAR, or maybe something that's not a non-UTF-8 encoded character, or something weird like that. So, I never know exactly what was wrong in the schema. There's some weird string thing happening that's causing the Ruby adapter to blow up. STEPHANIE: That also feels so unsatisfying [laughs] for you. I could imagine. JOËL: Yeah, there's no, like, clean resolution, right? It's a, well, the bug is gone for now. We're trying to make it less likely for it to pop up again in the future. I'm trying to leave some documentation for the next person who's going to come along, and I'm moving forward, fingers crossed. Is that something you've ever had to do on one of your projects? STEPHANIE: Given up? Yes. [laughter] I think I have definitely had to learn how to timebox debugging and have some action items for when I just can't figure it out. And, you know, like we mentioned, leaving some documentation for the next person to pick up, adding some additional logging so that maybe we can get more clues next time. But, you know, realizing that I do have to move on and that's the best that I can do is really challenging. JOËL: So, you used two words here to describe the situation: one was giving up, and the other one was timebox. I think I really like the idea of describing this as timeboxing. Giving up feels kind of like, defeatist. You know, there's so many things that we can do with our time, and we really have to be strategic with how we prioritize. So, I like the idea of describing this as a timeboxing situation. STEPHANIE: Yeah, I agree. Maybe I should celebrate every time that I successfully timebox something [laughs] according to how I planned to. [laughs] JOËL: There's always room to extend the timebox, right? STEPHANIE: [laughs] It's funny you bring up a debugging mystery because I have one of my own to share today. And I do have to say that it ended up being resolved, [chuckles] so it was a win in my book. But I will call this the case of the leaky stub. JOËL: That sounds slightly scary. STEPHANIE: It really was. The premise of what we were trying to figure out here was that we were having some flaky tests that were failing with a runtime error, so that was already kind of interesting. But it was quickly determined it was flaky because of the tests running in a certain order, so-- JOËL: Classic. STEPHANIE: Right. So, I knew something was happening, and any tests that came after it were running into this error. And I was taking a look, and I figured out how to recreate it. And we even isolated to the test itself that was running before everything else, that would then cause some problems. And so, looking into this test, I saw that it was stubbing the find method on an ActiveRecord model. JOËL: Interesting. STEPHANIE: Yeah. And the stubbed value that we were choosing to return ended up being referenced in the tests that followed. So, that was really strange to me because it went against everything I understood about how RSpec cleans up stubs between tests, right? JOËL: Yeah, that is really strange. STEPHANIE: Yeah, and I knew that it was referencing the stub value because we had set a really custom, like, ID value to it. So, when I was seeing this exact ID value showing up in a test that seemed totally unrelated, that was kind of a clue that there was some leakage happening. JOËL: So, what did you do next? STEPHANIE: The next discovery was that the error was actually raised in the factory setup for the failing tests and not even getting to running the examples at all. So, that was really strange. And digging into the factories was also its own adventure because there was a lot of complexity in the factories. A lot of them used hooks as well that then called some application code. And it was a wild goose chase. But ultimately, I realized that in the factory setup, we were calling some application code for that model where we had stubbed the find, and it had used the find method to memoize a class instance variable. JOËL: Oh no. I can see where this is going. STEPHANIE: Yeah. So, at some point, our model.find() returned our, you know, stub value that we had wanted in the previous test. And it got cached and just continued to leak into everything else that eventually would try to call that memoized method when it really should have tried to do that look-up for a separate record. JOËL: And class instance variables will persist between tests as long as they're on the same thread, right? STEPHANIE: Yeah, as far as I understand it. JOËL: That sounds like a really frustrating journey. And then that moment when you see the class instance variable, and you're like, oh no, I can't believe this is happening. STEPHANIE: Right? It was a real recipe for disaster, I think, where we had some, you know, really complicated factories. We had some sneaky caching issues, and this, you know, totally seemingly random runtime error that was being raised. And it was a real wild goose chase because there was not a lot of directness in going down the debugging path. I feel like I went around all over the codebase to get to the root of it. And, in the end, you know, we were trying to come up with some takeaways. And what was unfortunate was that you know, like, normally, stubbing find can be okay if you are, you know, really wanting to make sure that you are returning your mocked value that you may have, like, stubbed some other stuff on in your test. But because of all this, we were like, well, should we just not stub find on this really particular model? And that didn't seem particularly sustainable to make as a takeaway for other developers who want to avoid this problem. So, in the end, I think we scoped the stub to be a little more specific with the arguments that we wanted to target. And that was the way that we went forward with the particular flaky test at hand. JOËL: It sounds like the root cause of the problem was not so much the stub as it was the fact that this value is getting cached at the class level. Is that right? STEPHANIE: Yes and no. It seems like a real pain for running the tests. But I'm assuming that it was done for a good reason in production, maybe, maybe not. To be fair, I think we didn't need to cache it at all because it's calling a find, which is, you know, should be pretty quick and doesn't need to be cached. But who knows? It's hard to tell. It was really old code. And I think we were feeling also a little nervous to adjust something that we weren't sure what the impact would be. JOËL: I'm always really skeptical of caching. Caching has its place. But I think a lot of developers are a little too happy to introduce one, especially doing it preemptively that, oh well, we might need a cache here, so why not? Let's add that. Or even sometimes, just as a blind solution to any kind of slowness, oh, the site is slow; let's throw a cache here and hope for the best. And the, like, bedrock, like, rule zero of any kind of performance tuning is you've got to measure before and after and make sure that the change that you introduce actually makes things better. And then, also, is it better enough speed-wise that you're willing to pay any kind of costs associated to maintaining the code now that it's more complex? And a lot of caches can have some higher carrying costs. STEPHANIE: Yeah, that's a great point. This debugging mystery an example of one of them. JOËL: How long did it take you to figure out the solution here? STEPHANIE: So, like you, I actually was on a bit of the incorrect path for a little while. And it was only because this issue affected a different flaky test that someone else was investigating that they were able to connect the dots and be like, I think these, you know, two issues are related. And they were the ones who ultimately were able to point us out to the offending test if you will. So, you know, it took me a few days. And I imagine it took the other developer a few days. So, our combined effort was, like, over a week. JOËL: Yep. So, for all our listeners out there, you just heard that Stephanie and I [laughs] both went on multi-day debugging journeys. That happens to everyone. Just because we've been doing this job for years doesn't mean that every bug is, like, a thing that we figure out immediately. So, separately from this bug that I've been working on, a big issue that's been front of mind for me on this project has been the classic build versus buy decision. Because we're integrating with a third-party system, we have to look at either building our own integration or trying to use some off-the-shelf components. And there's a few different levels of this. There are some parts where you can actually, like, literally buy an integration and think through some of the decisions there. And then there's some situations where maybe there's an open-source component that we can use. And there's always trade-offs with both the commercial and the open-source situation. And we have to decide, are we willing to use this, or do we want to build our own? And those have been some really interesting discussions to have. STEPHANIE: Yeah. I think you actually expanded this decision-making problem into a build versus buy versus open source because they are kind of, you know, really different solutions with different outcomes in terms of, you know, maintenance and dependencies, right? And that all have, like, a little bit of a different way to engage with them. JOËL: Interesting. I think I tend to think of the buy category, including both like commercial off-the-shelf software and also open-source off-the-shelf software, things that we wouldn't build custom for ourselves but that are third-party components that we can pull in. STEPHANIE: Yeah, that's interesting because I had a bit of a different mental model because, in my head, when you're buying a commercial solution, you, you know, are maybe losing out on some opportunities for customization or even, like, forking it on your own. So, with an open-source solution, there could be an aspect of making it work for you. Whereas for a commercial solution, you really become dependent on that other company and whether they are willing to cater [laughs] to your needs or not. JOËL: That's fair. For something that's closed-source where you don't actually have access to the code, say it's more of a software as a service situation, then, yeah, you're kind of locked in and hoping that they can provide the needs that you have. On the flip side, you are generally paying for some level of support. The quality of that varies sometimes from one vendor to another. But if something goes wrong, usually, there's someone you can email, someone you can call, and they will tell you how to fix the problem, or they will fix it on their end. STEPHANIE: For the purposes of this conversation, should we talk about the differences, you know, building yourself or leaning on an existing built-out solution for you? JOËL: The project I'm working on is integrating with a Snowflake data warehouse, which is an external place that stores data accessible through something SQL-like. And one of the things that's attractive about this is that you can pull in data from a variety of different sources, transform it, and have it all stored in a kind of standardized structure that you can then integrate with. So, for pulling data in, you can build your own sort of ingestion pipeline, if you want, with code, and their APIs, and things. But there are also third-party vendors that will give you kind of off-the-shelf components that you can use for a lot of popular other data sources that you might want to pull. So, you're saying; I want to pull from this external service. They've probably got a pre-built connector for it. They can also do things like pull from an arbitrary Postgres database on some other server if that's something you have access to. It becomes really attractive because all you need to do is create an account on this website, plug in a few, like, API keys and URLs. And, all of a sudden, data is just flowing from one third-party system into your Snowflake data Warehouse, and it all just kind of works. And you don't have to bother with APIs, or ODBC, or any of that kind of stuff. STEPHANIE: Got it. Yeah, that does sound convenient. As you were talking about this, I was thinking about how if I were in the position of trying to decide how to make that integration happen, the idea of building it would seem kind of scary, especially if it's something that I don't have a lot of expertise in. JOËL: Yeah, so this was really interesting. In the beginning of the project, I looked into a little bit of what goes into building these, and it's fairly simple in terms of the architecture. You just need something that writes data files to typically something like an S3 bucket. And then you can point Snowflake to periodically pull from that bucket, and you write an import script to, you know, parse the columns and write them to the right tables in the structure that you want inside Snowflake. Where things get tricky is the actual integration on the other end. So, you have some sort of third-party service. And now, how do you sort of, on a timer maybe, pull data from that? And if there are data changes that you're synchronizing, is it just all append-only data? Or are you allowing the third-party service to say, "Hey, I deleted this record, and you should reflect that in Snowflake?" Or maybe dealing with an update. So, all of these things you have to think about, as well as synchronization. What you end up having to do is you probably boot up some kind of small service and, you know, maybe this is a small Ruby app that you have on Heroku, maybe this is, like, an AWS Lambda kind of thing. And you probably end up running this every so many seconds or so many minutes, do some work, potentially write some files to S3. And there's a lot of edge cases you have to think about to do it properly. And so, not having to think about all of those edge cases becomes really enticing when you're looking to potentially pay a third party to do this for you. STEPHANIE: Yeah, when you used the words new service, I bristled a little bit [laughs] because I've definitely seen this happen maybe on a bit of a bigger scale for a tool or solution for some need, right? Where some team is formed, or maybe we kind of add some more responsibilities to an existing team to spin up a new service with a new repo with its own pipeline, and it becomes yet another thing to maintain. And I have definitely seen issues with the longevity of that kind of approach. JOËL: The idea of maintaining a fleet of little services for each of our integrations seemed very unappealing to me, especially given that setting something like this up using the commercial approach probably takes 30 minutes per third-party service. There's no way I'm standing up an app and doing this whole querying every so many minutes, and getting data, and transforming it, and writing it to S3, and addressing all the edge cases in 30 minutes. And it's building something that's robust. And, you know, maybe if I want to go, like, really low tech, there's something fun I could do with, like, a Zapier hook and just, like, duct tape a few services together and make this, like, a no-code solution. I still don't know that it would have the robustness of the vendor. And I don't think that I could do it in the same amount of time. STEPHANIE: Yeah. I like the keyword robustness here because, at first, you were saying, like, you know, this looked relatively small in scope, right? The code that you had to write. But introducing all of the variables of things that could go wrong [laughs] beyond the custom part that you actually care about seemed quite cumbersome. JOËL: I think there's also, at this point, a lot of really interesting prioritization questions. There are money questions, but there are also time questions you have to think about. So, how much dev time do we want to devote upfront to building out these integrations? And if you're trying to move fast and get a proof of concept out, or even get, like, an MVP out in front of customers, it might be worth paying more money upfront to a third-party vendor because it allows you to ship something this week rather than next month. STEPHANIE: Yeah. The "How soon do you need it?" is a very good question to ask. Another one that I have learned to include in my arsenal of, you know, evaluating this kind of stuff comes from a thoughtbot blog by Josh Clayton, where he, you know, talks about the build versus buy problem. And his takeaway is that you should buy when your business is not dependent on it. JOËL: When it's not part of, like, the core, like, value-add that your business is doing. Why spend developer time on something that's not, like, the core thing that your product is when you can pay someone else to do it for you? And like we said earlier, a lot of that time ends up being sunk into edge cases and robustness and things like that to the point where now you have to build an expertise in a, like, secondary thing that your business doesn't really care about. STEPHANIE: Yeah, absolutely. I think this is also perhaps where very clear business goals or a vision would come in handy as well. Because if you're considering building something that doesn't quite support that vision, then it will likely end up continuing to be deprioritized over the long term until it becomes this thing that no one is accountable for maintaining and caring for. And just causes a lot of, honestly, morale issues is what I've seen when some service that was spun up to try to solve a particular problem is kind of on its last legs and has been really neglected, and no one wants to work on it. But it ends up causing issues for the rest of the development team. But then they're also really focused on initiatives that actually do provide the business value. That is a really hard balancing act that I've seen teams struggle with. JOËL: Earlier this year, we were talking about the book Sustainable Rails. And it really hammers home the idea of a carrying cost for the code, and I think that's exactly what we're talking about here. And that carrying cost can be time and money. But I like that you also mentioned the morale effects. You know, that's a carrying cost that just sort of depresses the productivity of your team when morale is low. STEPHANIE: Yeah, absolutely. I'm curious if we could discuss some of the carrying costs of buying a solution and where you've seen that become tricky. JOËL: The first thing to look at is the literal cost, the money aspect of things. And I think it's a really interesting situation for the business models for these types of Snowflake connectors because they typically charge by the amount of data that you're transmitting, so per row of data that you're transmitting. And so, that cost will fluctuate depending on whether the third-party service you're integrating with is, like, really chatty or not. When you contrast that to building, building typically has a relatively fixed cost. It's a big upfront cost, and then there's some maintenance cost to go with it. So, if I'm building some kind of integration for, let's say, Shopify, then there's the cost I need to build up front to integrate that. And if that takes me, I don't know, a week or two weeks, or however long it is, you know, that's a pretty big chunk of time. And my time is money. And so, you can actually do the math and say, "Well, if we know that we're getting so many rows per day at this rate from the commercial vendor, how many weeks do we have to pay for the commercial one before we break even and it becomes more expensive than building it upfront, just in terms of my time?" And sometimes you do that math, and you're like, wow, you know, we could be going on this commercial thing for, like, two years before we break even. In that case, from a purely financial point of view, it's probably worth paying for that connector. And so, now it becomes really interesting. You say, okay, well, which are the connectors that we have that are low volume, and which are the ones that are high volume? Because each of them is going to have a different break-even point. The ones that you break even after, you know, three or four weeks might be the ones that become more interesting to have a conversation about building. Whereas some of the others, it's clearly not worth our time to build it ourselves. STEPHANIE: The way you described this problem was really interesting to me because it almost sounds like you found the solution somewhere in the middle, potentially, where, you know, you may try building the ones that are highest priority, and you end up learning a lot from that experience, right? That could make it easier or at least, like, set you up to consider doing that moving forward in the future if you find, like, that is what is valuable. But it's interesting to me that you kind of have the best of both worlds of, like, getting the commercial solutions now for the things that are lower value and then doing what you can to get the most out of building a solution. JOËL: Yeah. So, my final recommendation ended up being, let's go all commercial for now. And then, once we've built out something, and because speed is also an issue here, once we've built out something and it's out with customers, and we're starting to see value from this, then we can start looking at how much are we paying per week for each of these connectors? And is it worth maybe going back and building our own for some of these higher-volume connectors? But starting with the commercial one for everything. STEPHANIE: Yeah, I actually think that's generally a pretty good path forward because then you are also learning about how you use the commercial solution and, you know, which features of it are critical so that if you do eventually find yourselves, like, maybe considering a shift to building in-house, like, you could start with a more clear MVP, right? Because you know how your team is using an existing product and can focus on the parts that your business are dependent on. JOËL: Yeah, it's that classic iterative development style. I think here it's also kind of inspired by a strategy I typically use for performance, which is make it work before you try to make it fast. And, actually, make it work, then profile, then measure, find the hotspots, and then focus on making those things fast. So, in this case, instead of speed, we're talking about money. So, it's make it work, then profile, find the parts that are expensive, and make the trade-off of, like, okay, is it worth investing into making that part less expensive in terms of resources? STEPHANIE: I like that as a framework a lot. JOËL: A lot of what we do as programmers is optimization, right? And sometimes, we're optimizing for execution time. Sometimes we're optimizing for memory cost, and sometimes we're optimizing for dollars. STEPHANIE: Yeah, that's really interesting because, with the buy solution, you know very clearly, like, how much the thing will cost. Whereas I've definitely seen teams go down the building route, and it always takes longer than expected [laughs], and that is money, right? In terms of the developer's time, for sure. JOËL: Yeah, definitely, like, add some kind of multiplier when you're budgeting out that build alternative because, quite likely, there are some edge cases that you haven't thought about that the commercial partner has, and you will have to spend more time on that than you expected. STEPHANIE: Yeah, in addition to whatever opportunity cost of not working on something that is driving revenue for the business right now. JOËL: Exactly. STEPHANIE: So, the direction of this conversation ended up going kind of towards, like, what is best for the team at, like, a product and company level. But I think that we make these decisions a lot more frequently, even when it comes to whether we pull in a gem or, you know, use an open-source tool or not. And I would be really interested in discussing more of that in another episode. JOËL: Yeah. That gets into some controversial takes, right? It's the evergreen topic of: do we build it ourselves, or do we pull in some kind of third-party package? STEPHANIE: Something for the future to look forward to. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeee!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

Welcome episode 220 of The Cloud Pod podcast - where the forecast is always cloudy! This week your hosts, Justin, Jonathan, Ryan, and Matthew discuss all things cloud, including virtual machines, an AI partnership between Microsoft and Meta for Llama 2, Lambda functions, Fargate, and lots of security updates including the Outlook breach and WORM protections. This and much more in our newest episode.  Titles we almost went with this week: Too Many Bees died for Honeycode Microsoft announces that AI will only cost you 3 arms and a leg.   The Cloud Pod also detects Recursive Loops in cloud news The cloud pod disables health checks bc who needs them A big thanks to this week's sponsor: Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

AWS Lambda proactively warms things up WormGPT Cybercrime Tool Heralds an Era of AI Malware vs. AI Defenses DirectorStorage on Windows now works for consumers and games White House Fills in Details of National Cybersecurity Strategy Hackers Say Generative AI Unlikely to Replace Human Cybersecurity Skills According to Bugcrowd Survey    Alfonso Santos and Larry Longhurst of Veridas talk about digital identity and biometric security Hosts: Louis Maresca and Curtis Franklin Guests: Alfonso U. Santos and Larry Longhurst Download or subscribe to this show at https://twit.tv/shows/this-week-in-enterprise-tech. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsors: canary.tools/twit - use code: TWIT GO.ACILEARNING.COM/TWIT