POPULARITY
16 episodes with appsync
3 episodes with appsync
3 episodes with appsync
3 episodes with appsync
2 episodes with appsync
2 episodes with appsync
2 episodes with appsync
2 episodes with appsync
In this pre-re:Invent 2024 episode, Luciano and Eoin discuss some of their favorite recent AWS announcements, including improvements to AWS Step Functions, Lambda runtime updates, DynamoDB price reductions, ALB header injection, Cognito enhancements, VPC public access blocking, and more. They share their thoughts on the implications of these new capabilities and look forward to seeing what else is announced at the conference. Overall, it's an exciting time for AWS developers with many new features to explore. Very important: no focus on GenAI in this episode :) AWS Bites is brought to you, as always, by fourTheorem! Sometimes, AWS is overwhelming and you might need someone to provide clear guidance in the fog of cloud offerings. That someone is fourTheorem. Check them out at fourtheorem.com In this episode, we mentioned the following resources: The repo containing the code of the AWS Bites website: https://github.com/awsbites/aws-bites-site Orama Search: https://orama.com/ JSONata in AWS Step Functions: https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/ EC2 Auto Scaling improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-ec2-auto-scaling-highly-responsive-scaling-policies/ Node.js 22 is available for Lambda: https://aws.amazon.com/blogs/compute/node-js-22-runtime-now-available-in-aws-lambda/ Python 3.13 runtime: https://aws.amazon.com/blogs/compute/python-3-13-runtime-now-available-in-aws-lambda/ Aurora Serverless V2 now scales to 0: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-aurora-serverless-v2-scaling-zero-capacity/ Episode 95 covering Mountpoint for S3: https://awsbites.com/95-mounting-s3-as-a-filesystem/ One Zone caching for Mountpoint for S3: https://aws.amazon.com/about-aws/whats-new/2024/11/mountpoint-amazon-s3-high-performance-shared-cache/ Appending to S3 objects: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html 1 million S3 Buckets per account: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-up-1-million-buckets-per-aws-account/ DynamoDB cost reduction: https://aws.amazon.com/blogs/database/new-amazon-dynamodb-lowers-pricing-for-on-demand-throughput-and-global-tables/ ALB Headers: https://aws.amazon.com/about-aws/whats-new/2024/11/aws-application-load-balancer-header-modification-enhanced-traffic-control-security/ Cognito Managed Login: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ Cognito Passwordless Authentication: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ VPC Block Public Access: https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-block-public-access/ Episode 88 where we talk about VPC Lattice: https://awsbites.com/88-what-is-vpc-lattice/ Direct integration between Lattice and ECS: https://aws.amazon.com/blogs/aws/streamline-container-application-networking-with-native-amazon-ecs-support-in-amazon-vpc-lattice/ Resource Control Policies: https://aws.amazon.com/blogs/aws/introducing-resource-control-policies-rcps-a-new-authorization-policy/ Episode 23 about EventBridge: https://awsbites.com/23-what-s-the-big-deal-with-eventbridge/ EventBridge latency improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-eventbridge-improvement-latency-event-buses/ AppSync web sockets: https://aws.amazon.com/blogs/mobile/announcing-aws-appsync-events-serverless-websocket-apis/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter: - https://twitter.com/eoins - https://twitter.com/loige
Erik Hanchett, senior developer advocate at AWS Amplify, explores the world of Fullstack TypeScript. He discusses the significance of end-to-end type safety, the tools to achieve it, and delves into the benefits and functionalities of AWS Amplify. Links https://www.programwitherik.com https://x.com/erikch https://www.youtube.com/c/programwitherik https://www.linkedin.com/in/erikhanchett https://trpc.io https://orval.dev https://docs.amplify.aws We want to hear from you! How did you find us? Did you see us on Twitter? In a newsletter? Or maybe we were recommended by a friend? Let us know by sending an email to our producer, Emily, at emily.kochanekketner@logrocket.com (mailto:emily.kochanekketner@logrocket.com), or tweet at us at PodRocketPod (https://twitter.com/PodRocketpod). Follow us. Get free stickers. Follow us on Apple Podcasts, fill out this form (https://podrocket.logrocket.com/get-podrocket-stickers), and we'll send you free PodRocket stickers! What does LogRocket do? LogRocket provides AI-first session replay and analytics that surfaces the UX and technical issues impacting user experiences. Start understand where your users are struggling by trying it for free at [LogRocket.com]. Try LogRocket for free today.(https://logrocket.com/signup/?pdr) Special Guest: Erik Hanchett.
Dive into the world of GraphQL APIs on AWS this week! We'll explore the recently launched feature in AppSync: asynchronous Lambda functions for GraphQL resolvers. But first, we'll break down the advantages of GraphQL over REST APIs and the limitations of synchronous calls in GraphQL. Then, we'll uncover the power of async Lambdas: stream data directly to your client for a more responsive experience and unlock innovative use cases, like generative AI-powered chatbots built with Lambdas. Curious how this can transform your applications? Tune in to learn more! With Derek Bingham, Developer Advocate, AWS https://www.linkedin.com/in/derekwbingham/ - Derek's blog about AppSync async Lambda resolvers https://community.aws/content/2hlqAp86YvckSS2DrVvZ1qdArqF/async-lambda-and-appsync?lang=en - AWS AppSync https://docs.aws.amazon.com/appsync/latest/devguide/what-is-appsync.html - AWS Lambda https://docs.aws.amazon.com/lambda/latest/dg/welcome.html - Streaming a response from a Lambda function https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html - AWS AppSync sample code https://github.com/aws-samples/aws-appsync-resolver-samples - Michael (App Sync Developer Advocate) YouTube channel https://www.youtube.com/@focusotter/videos
In this episode I am talking to Javier Ramirez from QuestDB, about everything QuestDB. This episode is a great resource to understand how QuestDB works, its architecture, what is it optimised for and whats upcoming as per the roadmap. If you have timeseries data and need a simple yet highly scalable solution, #QuestDB is a great option. Chapters: 00:00 Introduction 03:04 Understanding QuestDB: Origins and Use Cases 09:21 Deep Dive into QuestDB's Architecture and Data Ingestion 19:07 Optimizing Data Reads and Writes in QuestDB 28:40 Exploring Data Granularity and Partitioning in QuestDB 29:29 Optimizing Query Performance with Partition Strategies 30:26 Handling Data Ingestion and Query Efficiency 32:58 In-depth Look at Data Duplication and Ingestion Performance 34:55 Understanding Compression and Its Impact on Performance 38:51 Replication and Data Distribution Strategies 47:10 Observability and Metrics in QuestDB 50:57 Future Developments and Enhancements in QuestDB 58:45 Closing Remarks Links: QuestDB: https://questdb.io/ Github: https://github.com/questdb/questdb =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #questdb #sql #timeseries #timeseriesanalysis #databases #highscale #scaleup #performance #parquet #S3 #replication #writeaheadlog #wal #durability #columnstore
In this episode I talk to Andras Gerlits, who founded omniledger.io. Andras has a very interesting view on how Distributed Consistency should work that can get rid of several bottlenecks when it comes to maintaining Distributed consistency. He argues how getting rid of a global wall clock and using causality to approach Distributed consistency helps you build resilient, simple and performant systems. We have gone deeper into how that can be achieved and how the product works. Chapters: 00:00 Introduction 00:52 Andras's Journey into Distributed Consistency 03:04 The Evolution of Data Consistency in Banking and Beyond 08:04 Introducing Client-Centric Consistency 10:36 Exploring the Standard Model of Distributed Consistency 16:01 Redefining Strong Consistency with a Relativistic Approach 34:25 Practical Implications of Client-Centric Consistency in Banking 36:20 Mitigating Latencies and Partitions in Distributed Systems 41:08 Exploring System Reliability and Availability 41:52 Tuning System Properties for Specific Use Cases 43:07 Comparing Standard and New Models for Data Management 45:08 Understanding Local Progress and Mutex-Free Updates 47:23 Deep Dive into Token-Based Ordering and Global Calibration 58:30 Introducing OmniLedger: A New Approach to Distributed Consistency 01:02:41 Performance Optimizations and Tunable Consistency 01:08:20 Ideal Use Cases and Potential Limitations of OmniLedger 01:14:30 Future Directions and Closing Thoughts Links: Our website: https://omniledger.io A long-form essay on the thinking behind our model: https://medium.com/p/5e397cb12e63 A demo of transactionality https://www.youtube.com/watch?v=XJSSjY4szZE I think my blog in general might be interesting to some https://medium.com/@andrasgerlits The science-paper with all its mathematical rigour: https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #databases #sql #consistency #distributedsystems
In this video I talk to Semih Salihoglu about KuzuDB : A highly scalable, extremely fast, easy to use embeddable Graph Database. Chapters: 00:00 Introduction 00:40 The Genesis of KuzuDB: From Academic Research to Startup 06:40 Graph Databases 101: Understanding the Basics and Beyond 10:24 When to Opt for a Graph Database: Use Cases and Advantages 19:16 KuzuDB vs. Traditional Databases: A Comparative Analysis 24:39 Inside KuzuDB: Optimizations and Data Ingestion Explained 31:08 Exploring Query Optimizations in Graph Databases 31:34 The Relational Nature of Graph Databases 33:33 Factorization: A Key Optimization Technique 38:50 Integrating New Data Sources and Handling Joins 43:39 Optimizing Write Operations and Index Management 50:23 Comparing Kuzu with Other Graph Databases 58:50 Future Developments and Vision for Kuzu Important links: - History of DBMSs and the IDS, which is the first database in history, which had a graph-based model: https://dl.acm.org/doi/abs/10.1145/1147376.1147382 is a good paper by CS historian on this history and a must read for everyone interested in the birth of databases as a field. - https://blog.kuzudb.com/post/what-every-gdbms-should-do-and-vision/ blog on the what every GDBMS should do and vision of Kùzu. - The user survey paper that got Semih into GDBMSs. https://arxiv.org/pdf/1709.03188.pdf - Blog on factorization https://blog.kuzudb.com/post/factorization/ - Kùzu's RDFGraphs feature https://docs.kuzudb.com/rdf-graphs/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
In this video, I talk to Stephan Ewen from Restate, who is popularly known from the world of Apache Flink. We have talked about the problems in the world of Distributed systems and the complex solutions developers have to deal with. This complexity makes the architecture so complex that it eventually creates reliability, Observability and delivery velocity problems. Restate aims to solve it by making resilience and durability for your services, functions and RPC a lot simpler. Chapters: 00:00 Introduction 00:45 Introducing Restate: A Solution for Distributed System Challenges 01:22 Deep Dive into Restate with Stefan: From Apache Flink to Building Resilient Systems 06:04 The Complexities of Distributed Systems and How Restate Addresses Them 15:49 The Vision of Restate: Simplifying Developer Experience in Distributed Systems 24:42 Integrating Restate into Your Architecture: A User's Perspective 33:16 Exploring Restate: The Durable Service Mesh 33:32 The Power of Restate in Handling Transactions 34:26 Restate's Role in Service Communication and Durability 35:40 Deep Dive into Restate's Mechanisms and Benefits 38:04 Practical Example: Email Pipeline with Restate 39:40 Understanding Restate's Log and Event Handling 58:43 Restate's Unique Features and Programming Model 01:04:22 Final Thoughts on Restate's Impact and Deployment Restate: https://restate.dev/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #faulttolerance #reliability #resilience
In this episode of The GeekNarrator podcast, our host Kaivalya talks to Seeta Somagani from Volt Active Data, a low latency stream processing platform. They discuss fascinating topics about what low latency stream processing means, the different guarantees that Volt Active Data provides, and the various problems it can solve. They delve into the evolution of VoltDB to Volt Active Data, real-time data processing use cases, the high-level architecture, and how the platform effectively addresses high-concurrency challenges. This is a must-listen for anyone interested in understanding latency critical applications, data processing, and high performance computing. Chapters: 00:00 Welcome to The GeekNarrator Podcast with Special Guest from Volt Active Data 00:41 Introduction 01:45 The Evolution of VoltDB to Volt Active Data 06:13 Exploring Real-Time Data Processing and Use Cases 08:25 Addressing High-Concurrency Challenges in Various Industries 12:57 High-Level Architecture of Volt Active Data 19:26 Understanding Stored Procedures and Data Processing in Volt 22:48 Practical Application: Tracking Data Usage with Volt Active Data 25:16 Diving into Replicated and Partitioned Tables 25:44 Exploring Event Processing and Exporting 26:57 Understanding Stored Procedures and Performance 29:03 Partitioning Strategies and Recommendations 31:39 Ensuring Determinism in Stored Procedures 35:02 Handling Complex Requirements with Compound Procedures 37:25 Fault Tolerance and Data Replication Strategies 40:44 Exploring Use Cases for VoltActiveData 43:30 The Future of Streaming and VoltActiveData's Role 47:05 Closing Remarks and How to Learn More Volt Active Data: https://www.voltactivedata.com/use-cases/activesd-streaming-data/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #sql #streamprocessing #java #acid
In an enlightening episode of the GeekNarrator Podcast, host Kaivalya Apte and TigerBeetle's CEO, Joran, delve deep into the world of online transaction processing (OLTP). They discuss the origin, unique architecture, and innovative methodologies behind TigerBeetle, a database tailored to efficiently handle high-volume transaction systems. The podcast explores the system's key features such as efficient scalability, performance-oriented design, and optimized memory usage, demonstrating its robustness in handling business transactions and accounting. It also elucidates TigerBeetle's adaptability to various domains beyond finance, like energy management and gaming, while highlighting the rigorous testing it undergoes for impeccable quality assurance. Chapters: 00:00 Introduction 01:19 Joran's Journey into Databases 03:59 Understanding Financial Transaction Databases 07:41 The Evolution of OLTP and OLAP 16:13 The Need for a New Database: TigerBeetle 16:53 Performance and Safety Features of TigerBeetle 28:49 The Importance of Safety in Financial Transactions 36:49 Changing Developer Experience with TigerBeetle 41:43 Understanding the CPU and Memory Bandwidth 42:12 The Importance of Data Format Language 43:27 The Concept of Serialization and its Impact 46:23 The Architecture of TigerBeetle 46:29 The Role of Replicated State Machine 48:18 The Importance of Consensus in Replication 50:20 The Structure of TigerBeetle 50:37 The Importance of Log in Systems 50:51 Understanding the State in Replicated State Machine 52:55 The Role of LSM in TigerBeetle 53:55 The Impact of Compaction Process on Performance 57:06 The Importance of Predictability in Software 01:06:15 The Read and Write Path in TigerBeetle 01:14:46 Potential Use Cases for TigerBeetle 01:17:09 Understanding the Limitations of TigerBeetle =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #tigerbeetledb #databases #acid #olap #oltp #postgres #mysql
In this episode, we dive deep into the world of clean coding with none other than the master and pioneer of the field, Uncle Bob. We explore the nuances and the art behind writing effective and efficient scripts. This conversation covers the nitty-gritty of writing and editing scripts, from understanding how to break down large functions, to discussing principles like 'Single Responsibility Principle', 'Dependency Inversion Principle' and how to balance the 'DRY' (Don't Repeat Yourself) principles. Uncle Bob also shares valuable insights on testing, handling errors, naming conventions and how to work with different types of duplication in coding. He shares recommended resources and books that every coder should read. Chapters: 00:00 Introduction and Welcome 00:06 The Importance of Code Quality 00:29 Introducing Robert Martin (Uncle Bob) 01:39 Uncle Bob's Journey in Programming 02:34 Discussion on Functional Design and New Book 03:52 The Evolution of Software Development 04:28 Revisiting the Clean Code Book 04:49 The Impact of Hardware Changes on Software 06:13 The Evolution of Programming Languages 07:33 The Importance of Code Structure and Organization 09:07 The Impact of Microservices and Open Source 11:14 The Role of Modular Programming 22:07 The Importance of Naming in Code 26:31 The Role of Functions in Code 34:12 The Role of Switch Statements in Code 42:36 The Importance of Immutability 51:00 Dealing with Complex Steps in Programming 51:21 Implementing State Machines in Programming 51:46 The Pragmatic Approach to Programming 53:01 Understanding Error Handling in Programming 54:08 The Challenge of Exception Handling 57:27 The Importance of Log Messages in Debugging 01:03:05 The Dilemma of Code Duplication 01:05:51 The Intricacies of Error Handling 01:07:40 The Role of Abstraction in Programming 01:13:55 The Importance of Testing in Programming 01:19:43 The Challenges of Mocking in Testing 01:25:11 The Essence of Programming: Discipline, Ethics, and Standards Book Recommendations: Tidy First: https://www.oreilly.com/library/view/tidy-first/9781098151232/ Design Patterns: https://www.amazon.de/-/en/Erich-Gamma/dp/0201633612 Analysis Pattern: https://martinfowler.com/books/ap.html Structured Analysis and System Specification: https://www.amazon.de/-/en/Tom-Demarco/dp/0138543801 Fundamental Algorithms: https://www.amazon.com/Art-Computer-Programming-Vol-Fundamental/dp/0201896834 Sorting and Searching: https://www.amazon.de/-/en/Donald-Knuth/dp/0201896850 Structure and Interpretation of Computer Programs: https://web.mit.edu/6.001/6.037/sicp.pdf =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
In this episode of the Geek Narrator Podcast, Kaivalya Apte engages in an invigorating discussion with Dominik Tornow, the founder and CEO of Resonate. They explore Durable Async-Await, an interesting concept in distributed systems world, along with other nuances of distributed programming. Dominik also talks about the development and execution of Resonate to simplify distributed systems with a focus on observability, usability, and it's future direction. The conversation concludes with a discussion of different concurrency models and the future of distributed systems Chapters: 00:00 Introduction and Guest Background 02:44 Understanding Async Await 10:25 Challenges with Current Async Await Model 12:53 Introducing Resonate: A Solution for Distributed Async Await 13:34 Practical Application: E-commerce Example 24:57 Understanding the Role of the Platform in Distributed Systems 30:12 Dealing with Partial Failures in Distributed Systems 39:44 Getting Started with Resonate 40:40 Introduction to Resonate and its Simplicity 41:09 Getting Started with Resonate: Installation and Setup 42:22 Understanding the Durability Aspect of Resonate 42:49 Exploring the Resonate Durable Promise Server 44:10 Scaling Up: Introducing Workers into the System 48:35 The Importance of Open Standards in Resonate 50:17 Exploring the Integration Capabilities of Durable Promises 01:04:31 Understanding the Role of Timeouts in Durable Promises 01:07:29 The Future of Resonate: Challenges and Upcoming Features 01:13:04 Understanding the Limitations of Durable Promises 01:14:51 Wrapping Up: Final Thoughts on Resonate and Durable Promises References: A note on Distributed Systems: https://scholar.harvard.edu/files/waldo/files/waldo-94.pdf Thinking in Distributed Systems: https://dtornow.gumroad.com/l/distributed-systems McCarthy's paper: https://www-formal.stanford.edu/jmc/recursive/recursive.html =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
Join host Kaivalya Apte in this episode of The Geek Narrator Podcast as he discusses observability engineering with field CTO at Honeycomb, Liz Fong-Jones. They delve into the importance of observability for software engineers, the role of Honeycomb in popularizing this concept, and how observability has evolved over the years. Liz shares her experiences transitioning from being an SRE at Google to advocating for observability at Honeycomb and walking the journey from developer advocate to Field CTO. They discuss the definitions and misconceptions surrounding observability and elucidate on Service-Level Objectives (SLOs) & indicators (SLIs) and challenges they solve. Tune in for an informative and in-depth conversation on observability engineering. Chapters: 00:00 Introduction 00:08 Understanding Observability Engineering 00:37 Guest Introduction: Liz Fong Jones 00:53 Liz's Journey to Field CTO at Honeycomb 27:38 Understanding Site Reliability Workbook Materials 27:57 Identifying Critical User Journeys 29:49 Different Types of Services and Their SLOs 33:05 Setting Up SLOs: Granularity and Number 42:42 Understanding Service Level Indicators (SLIs) 50:26 Common Mistakes in Setting Up SLOs 52:09 Cultivating an Observability-Driven Development Culture References: Observability Engineering: https://www.oreilly.com/library/view/observability-engineering/9781492076438/ @Google SRE book: https://sre.google/books/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
In this video I talk about Apache Pulsar with Matteo Merli, CTO at StreamNative. This episode will provide you good insight about how Apache Pulsar works and more importantly differs with the most popular Pub/Sub and streaming platform Apache Kafka. Things like, what enables possibility of 1 million topics? Why is rebalancing not required? How does decoupled storage and compute architecture works? How it uses the concept of Subscriptions to avoid retaining data unnecessarily? And much more... Chapters: 00:00 Introduction and Guest Introduction 00:08 Understanding Apache Pulsar and its Origin 01:22 The Problem Apache Pulsar was Designed to Solve 02:35 The Evolution of Apache Pulsar 05:15 Understanding Basic Concepts of Apache Pulsar 09:27 Deep Dive into Apache Pulsar's Architecture 21:16 Understanding the Flow of Data in Apache Pulsar 28:54 Understanding Subscriptions in Apache Pulsar 31:57 Understanding End-to-End Latency and Subscription Creation 32:32 Broker's Role and Handling Metadata 33:05 Memory Management and Consumer Handling 34:07 Message Processing and Flow Control 34:32 Message Storage and Retrieval 36:00 Comparing Pulsar with Kafka 43:52 Understanding Multi-Tenancy in Pulsar 49:17 Exploring Tiered Storage and Future Developments Important links: StreamNative: https://streamnative.io/ Apache Pulsar: https://pulsar.apache.org/ Matteo Merli: https://twitter.com/merlimat =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
Deep Dive into Victoria Metrics with Alex and Roman Join the insightful discussion with Vitoriametrics creators, Alex and Roman, in the Geekneritor podcast hosted by Kaivalya Apte. This episode explores the internals of Victoria Metrics - a highly scalable monitoring solution and time series database. Discover the origins of Victoria Metrics, understand how it evolved, and learn about its unique architecture and functionality. From the concept of time series, the usage of consistent hashing in data distribution to real-world applications, it's all packed into this engaging conversation. 00:00 Introduction 01:52 The Genesis of VictoriaMetrics 02:18 The Journey from Postgres to Clickhouse 03:19 The Transition from Prometheus to Victoria Metrics 05:08 The Birth and Evolution of Victoria Metrics 13:01 The Architecture of Victoria Metrics 20:10 Data Ingestion and Integration in Victoria Metrics 29:15 Understanding the Vector Metric Architecture 30:30 Comparing Shared Storage and Object Store 31:00 Designing the VictoriaMetrics Architecture 32:01 The Role of Object Storage 36:15 The Importance of Indexing 43:19 Understanding the Ingestion Process 45:46 Exploring the Select Process 55:55 Future Plans for Victoria Metrics Important Links: 1. Architecture Overview: https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview 2. How ClickHouse Inspired Us to Build a High Performance Time Series Database https://altinity.com/wp-content/uploads/2021/11/How-ClickHouse-Inspired-Us-to-Build-a-High-Performance-Time-Series-Database.pdf 3. Frequently asked questions. https://docs.victoriametrics.com/FAQ.html =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!
In this video I discuss Database tuning and Optimisation with Andy Pavlo, OtterTune. Andy is an Associate Professor with Indefinite Tenure of Databaseology in the Computer Science Department at Carnegie Mellon University. My research interest is in database management systems, specifically main memory systems, self-driving / autonomous architectures, transaction processing systems, and large-scale data analytics. 00:00 Introduction and Welcome 01:31 Understanding Database Optimization 05:48 Understanding When Database Tuning is Needed 08:45 Understanding Database Optimization Difficulties 16:16 Understanding Default Settings in Databases 22:35 Role of Machine Learning in Database Tuning 22:38 Introduction to Ottertune 28:36 Data Collection for Machine Learning Model 35:25 Deployment and Data Collection Process 38:03 Admitting the Limitations of Current Model 38:53 Challenges in Predicting Performance Improvements 39:28 The Importance of Data Collection Over Time 39:52 Avoiding Weekend and Holiday Tuning 40:05 Introducing New Features for Database Comparison 42:09 Provisioning Recommendations and Performance Predictions 43:03 The Importance of Telemetry in Understanding Database Performance 44:01 Handling Dramatic Changes in Database Workloads 44:48 Preparing for Predictable Traffic Spikes 48:13 The Importance of Testing in Database Optimization 53:33 The Future of Database Optimization 55:50 Common Mistakes in Database Management 01:09:15 The Future of Holistic Database Tuning Links: Ottertune: https://ottertune.com/ Andy Pavlo: https://www.cs.cmu.edu/~pavlo/ CMU youtube: https://www.youtube.com/@UCHnBsf2rH-K7pn09rb3qvkA Resources: CMU: https://15799.courses.cs.cmu.edu/spring2022/schedule.html Ottertune blog: https://ottertune.com/blog =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Stay Curios! Keep Learning!
Deep Dive into DuckDB with CTO Mark Raasveldt Decode the insights of databases with Geek Narrator podcast. In this episode, host Kaivalya Apte converses with Mark Raasveldt, the CTO of DuckDB labs, discussing his journey from being a database enthusiast to creating DuckDB. They delve into how DuckDB, an analytical database, differs from other databases, the design decisions, its internal mechanisms, and much more. The episode also highlights the advantages of DuckDB in analytics, the motivation behind its ACID compliance, and how DuckDB handles ingestion, transaction isolation, mutations, and queries. Join in to learn how your data workloads can benefit from DuckDB. 00:00 Introduction and Guest Introduction 00:44 Guest's Journey into Databases 03:40 The Birth of DuckDB 04:30 Challenges with Existing Databases 05:15 Technical Difficulties 05:16 Why Existing Databases Fall Short for Data Scientists 09:16 The Role of SQLite and Its Limitations 13:59 Defining DuckDB 16:48 Comparing DuckDB with Other Analytical Databases 19:50 Deployment Models for DuckDB 22:47 Data ingestion into DuckDB 22:51 Data Ingestion in DuckDB 30:24 How DuckDB Handles Updates and Mutations 35:35 Understanding Column Granularity and Rewrites 35:58 Implications of Compression on Data Updates 36:38 Trade-offs in Row Group Size 37:32 Benefits of Column Storage Model 38:15 Row Groups and Parallelism 39:02 Choosing Row Group Size: An Experimental Approach 40:00 Handling Data Type Changes in Columns 41:00 Internal Data Structures in DuckDB 42:21 Reading Data: Point Lookups, Aggregations, and Joins 47:22 Optimization for Full Table Scans 53:49 Understanding ACID Compliance in DuckDB 55:49 Multi-Version Concurrency Control (MVCC) in DuckDB 59:50 Use Cases and Applications of DuckDB 01:01:42 The Story Behind DuckDB's Name 01:02:34 Future Vision for DuckDB References: DuckDB: https://duckdb.org/ Mark's blog: https://mytherin.github.io/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! Cheers, The GeekNarrator
In this episode we talk about ScyllaDB internals with Felipe Mendes. Chapters: 0:00 ScyllaDB internals with Felipe Mendes 07:51 Write Path - API to Storage 11:40 What makes it faster than Cassandra? 13:39 Optimisations: Sea Star, shard per core architecture 15:49 Optimisations: No Garbage collection and Custom Cache Implementation 18:15 Optimisations: Scheduling groups and IO priority classes 20:07 Optimisations: IO scheduler 22:55 Benefits of shard per core architecture 30:16 Write path - Hows is a coordinator chosen? 38:20 Read path 39:27 Read path optimisations - Index Caching 41:48 Shard vs Partition 43:10 Shard per core architecture tradeoff 44:03 Observability of Database References: ScyllaDB architecture: https://opensource.docs.scylladb.com/stable/architecture/ Sea star: https://seastar.io/ ScyllaDB Caching: https://www.scylladb.com/2018/07/26/how-scylla-data-cache-works/ Shard per core architecture: https://www.scylladb.com/product/technology/shard-per-core-architecture/ Database performance at Scale: https://www.scylladb.com/2023/10/02/introducing-database-performance-at-scale-a-free-open-source-book/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! Cheers, The GeekNarrator
In this episode I talk to Michael Hunger from Neo4j about Graph Database Internals (Neo4J) Chapters: 0:00 Introduction and historical context 20:51 Data Modelling 25:16 Problem with SQL for Graph Model 26:21 Cypher - Query Language 28:23 Write Path 31:36 Neo4J Storage Layer 33:51 Graph API on top of Relational Model vs Native Graph Databases 37:05 Create Node Relationships 40:42 What makes Graph Database's performance better? 46:00 Partitioning Strategy 53:20 Read path 59:27 Schema Migration 01:04:41 Graph database use cases =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! Cheers, The GeekNarrator
What's the difference between developer advocacy, tech evangelism, and solutions architecture? A few years ago, all these job titles would have sounded made up. But in the mad rush we know as "cloud development", each of them have distinct roles and importance. Join Michael "Focus Otter" Liendo and Allen Helton as they dive into what makes developer advocacy unique and relevant. The two discuss "mid-funnel" advocacy, how teams should utilize DA's, and touch on imposter syndrome. About Michael Michael is a senior developer advocate for AppSync at AWS. He has a passion for educating others and does so via his blog, videos, and speaking at conferences. Michael focuses on fullstack development and is particularly interested in Micro-SaaS applications. He also goes by Focus Otter, which is his highly recognizable and influential tech brand. Links Twitter - https://twitter.com/focusotter LinkedIn - https://www.linkedin.com/in/focusotter Focus Otter blog - https://blog.focusotter.com Hammock-driven development video -https://www.youtube.com/watch?v=f84n5oFoZBc --- Send in a voice message: https://podcasters.spotify.com/pod/show/readysetcloud/message Support this podcast: https://podcasters.spotify.com/pod/show/readysetcloud/support
In this episode I talk to Micah Wylde about why #Rust could be the best choice for writing distributed systems and how does it compare to #C++, #Java and #Go. Chapters: 00:00 Introduction 03:48 History of Systems Programming 09:42 Is C++ coming back? 13:31 Problems with C++ 16:24 Problems with Java 25:18 Problems with Go 31:21 Why did you choose Rust? 35:19 What makes Rust better? 41:49 Rust cannot save you from logical bugs 44:02 Problems in the context of Stream Processing 48:10 Challenges with Rust 51:28 Learning Rust 54:10 Future of Rust 56:41 A Summary Blog mentioned in the discussion: https://www.arroyo.dev/blog/rust-for-data-infra For the courses mentioned use the following links: Coupon code: "geeknarrator" Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! Cheers, The GeekNarrator
AWS AppSync is a serverless GraphQL and Pub/Sub APIs that simplify application development through a single endpoint to securely query, update, or publish data. Tune in to listen to Jillian Forde, Sr. Startup Solutions Architect and Ozioma Uzoegwu, Sr. Solutions Architect talk about the functionalities we have on AppSync that can help you implement enterprise level security on your GraphQL APIs on AppSync. Recently we launched the support of Private APIs on AppSync, the most requested feature on AppSync, so listen in to also learn about this new feature and how you can get started.
Welcome to the newest episode of The Cloud Pod podcast - where the forecast is always cloudy! Today your hosts Justin, Jonathan, and Matt discuss all things cloud and AI, as well as some really interesting forays into quantum computing, changes to Google domains, Google accusing Microsoft of cloud monopoly shenanigans, and the fact that Azure wants all your industry secrets. Also, Finops and all the logs you could hope for. Are your secrets safe? Better tune in and find out! Titles we almost went with this week: The Cloud Pod Adds Domains to the Killed by Google list The Cloud Pod Whispers it's Secrets to Azure OpenAI The Cloud Pod Accuses the Cloud of Being a Monopoly The Cloud Pod Does Not Pass Go and Does Not collect $200 A big thanks to this week's sponsor: Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.
Brooke Sargent, Software Engineer at Honeycomb, joins Corey on Screaming in the Cloud to discuss how she fell into the world of observability by adopting Honeycomb. Brooke explains how observability was new to her in her former role, but she quickly found it to enable faster learning and even a form of self care for herself as a developer. Corey and Brooke discuss the differences of working at a large company where observability is a new idea, versus an observability company like Honeycomb. Brooke also reveals the importance of helping people reach a personal understanding of what observability can do for them when trying to introduce it to a company for the first time. About BrookeBrooke Sargent is a Software Engineer at Honeycomb, working on APIs and integrations in the developer ecosystem. She previously worked on IoT devices at Procter and Gamble in both engineering and engineering management roles, which is where she discovered an interest in observability and the impact it can have on engineering teams.Links Referenced: Honeycomb: https://www.honeycomb.io/ Twitter: https://twitter.com/codegirlbrooke TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode—which is another way of saying sponsored episode—is brought to us by our friends at Honeycomb. And today's guest is new to me. Brooke Sargent is a software engineer at Honeycomb. Welcome to the show, Brooke.Brooke: Hey, Corey, thanks so much for having me.Corey: So, you were part of I guess I would call it the new wave of Honeycomb employees, which is no slight to you, but I remember when Honeycomb was just getting launched right around the same time that I was starting my own company and I still think of it as basically a six-person company versus, you know, a couple of new people floating around. Yeah, turns out, last I checked, you were, what, north of 100 employees and doing an awful lot of really interesting stuff.Brooke: Yeah, we regularly have, I think, upwards of 100 in our all-hands meeting, so definitely growing in size. I started about a year ago and at that point, we had multiple new people joining pretty much every week. So yeah, a lot of new people.Corey: What was it that drove you to Honeycomb? Before this, you spent a bit of time over Procter and Gamble. You were an engineering manager and now you're going—you went from IC to management and now you're IC again. There's a school of thought that I vehemently disagree with, that that's a demotion. I think they are orthogonal skill sets to my mind, but I'm curious to hear your journey through your story.Brooke: Yeah, absolutely. So yeah, I worked at Procter and Gamble, which is a big Cincinnati company. That's where I live and I was there for around four years. And I worked in both engineering and engineering management roles there. I enjoy both types of roles.What really drove me to Honeycomb is, my time at Procter and Gamble, I spent probably about a year-and-a-half, really diving into observability and setting up an observability practice on the team that I was on, which was working on connected devices, connected toothbrushes, that sort of thing. So, I set up an observability practice there and I just saw so much benefit to the engineering team culture and the way that junior and apprentice engineers on the team were able to learn from it, that it really caught my attention. And Honeycomb is what we were using and I kind of just wanted to spend all of my time working on observability-type of stuff.Corey: When you say software engineer, my mind immediately shortcuts to a somewhat outdated definition of what that term means. It usually means application developer, to my mind, whereas I come from the world of operations, historically sysadmins, which it still is except now, with better titles, you get more money. But that's functionally what SRE and DevOps and all the rest of the terms still currently are, which is, if it plugs into the wall, congratulations. It's your problem now to go ahead and fix that thing immediately. Were you on the application development side of the fence? Were you focusing on the SRE side of the world or something else entirely?Brooke: Yeah, so I was writing Go code in that role at P&G, but also doing what I call it, like, AWS pipe-connecting, so a little bit of both writing application code but also definitely thinking about the architecture aspects and lining those up appropriately using a lot of AWS serverless and managed services. At Honeycomb, I definitely find myself—I'm on the APIs and partnerships team—I find myself definitely writing a lot more code and focusing a lot more on code because we have a separate platform team that is focusing on the AWS aspects.Corey: One thing that I find interesting is that it is odd, in many cases, to see, first, a strong focus on observability coming from the software engineer side of the world. And again, this might be a legacy of where I was spending a lot of my career, but it always felt like getting the application developers to instrument whatever it was that they were building felt in many ways like it was pulling teeth. And in many further cases, it seemed that you didn't really understand the value of having that visibility or that perspective into what's going on in your environment, until immediately after. You really wished you had that perspective into what was going on in your environment, but didn't. It's similar to, no one is as zealous about backups as someone who's just suffered a data loss. Same operating theory. What was it that you came from the software engineering side to give a toss about the idea of observability?Brooke: Yeah, so working on the IoT—I was working on, like, the cloud side of things, so in Internet of Things, you're keeping a mobile application, firmware, and cloud synced up. So, I was focused on the cloud aspect of that triangle. And we got pretty close to launching this greenfield IoT cloud that we were working on for P&G, like, we were probably a few months from the initial go-live date, as they like to call it, and we didn't have any observability. We were just kind of sending things to CloudWatch logs. And it was pretty painful to figure out when something went wrong, from, like, you know, hearing from a peer on a mobile app team or the firmware team that they sent us some data and they're not seeing it reflected in the cloud that is, like, syncing it up.Figuring out where that went wrong, just using CloudWatch logs was pretty difficult and syncing up the requests that they were talking about to the specific CloudWatch log that had the information that we needed, if we had even logged the right thing. And I was getting a little worried about the fact that people were going to be going into stores and buying these toothbrushes and we might not have visibility into what could be going wrong or even being able to be proactive about what is going wrong. So, then I started researching observability. I had seen people talking about it as a best practice thing that you should think about when you're building a system, but I just hadn't had the experience with it yet. So, I experimented with Honeycomb a bit and ended up really liking their approach to observability. It fit my mental model and made a lot of sense. And so, I went full-steam ahead with implementing it.Corey: I feel what you just said is very key: the idea of finding an observability solution that keys into the mental model that someone's operating with. I found that a lot of observability talk sailed right past me because it did not align with that, until someone said, “Oh yeah, and then here's events.” “Well, what do you mean by event?” It distills down to logs. And oh, if you start viewing everything as a log event, then yeah, that suddenly makes a lot of sense, and that made it click for me in a way that, honestly, is a little embarrassing that it didn't before then.But I come from a world before containers and immutable infrastructure and certainly before the black boxes that are managed serverless products, where I'm used to, oh, something's not working on this Linux box. Well, I have root, so let's go ahead and fix that and see what's going on. A lot of those tools don't work, either at scale or in ephemeral environments or in scenarios where you just don't have the access to do the environment. So, there's this idea that if you're trying to diagnose something that happened and the container that it happened on stopped existing 20 minutes ago, your telemetry game has got to be on point or you're just guessing at that point. That is something that I think I did myself a bit of a disservice by getting out of hands-on keyboard operations roles before that type of architecture really became widespread.Brooke: Yeah, that makes a lot of sense. On the team that I was on, we were using a lot of AWS Lambda and similarly, tracking things down could be a little bit challenging. And emitting telemetry data also has some quirks [laugh] with Lambda.Corey: There certainly is. It's also one of those areas that, on some level, being stubborn to adopt it works to your benefit. Because when Lambda first came out, it was a platform that was almost entirely identified by its constraints. And Amazon didn't do a terrific job, at least in the way that I tend to learn, of articulating what those constraints are. So, you learn by experimenting and smacking face first into a lot of those things.What the hell do you mean you can't write to the file? Oh, it's a read-only file system. [except slash tap 00:08:39]. What do you mean, it's only half a gigabyte? Oh, that's the constraint there. Well, what do you mean, it automatically stops after—I think back in that point it was five or ten minutes; it's 15 these days. But—Brooke: Right.Corey: —I guess it's their own creative approach to solving the halting problem from computer science classes, where after 15 minutes, your code will stop executing, whether you want it to or not. They're effectively evolving these things as we go and once you break your understanding in a few key ways, at least from where I was coming from, it made a lot more sense. But ugh, that was a rough couple of weeks for me.Brooke: Yeah [laugh]. Agreed.Corey: So, a topic that you have found personally inspiring is that observability empowers junior engineers in a bunch of ways. And I do want to get into that, but beforehand, I am curious as to the modern-day path for SREs because it doesn't feel to me like there is a good answer for, “What does a junior SRE look like?” Because the answer is, “Oh, they don't.” It goes back to the old sysadmin school of thought, which is that, oh, you basically learn by having experience. I've lost count a number of startups I've encountered where you have a bunch of early-20-something engineers but the SRE folks are all generally a decade into what they're what they've been doing because the number-one thing you want to hear from someone in that role is, “Oh, the last time I saw it, here's what it was.” What is the observability story these days for junior engineers?Brooke: So, with SRE I agr—like, that's a conversation that I've had a lot of times on different teams that I've been on, is just can a junior SRE exist? And I think that they can.Corey: I mean, they have to because otherwise, it's well, where does it SRE come from? Oh, they spring—Brooke: [laugh].Corey: —fully formed from the forehead of some God out of mythology. It doesn't usually work that way.Brooke: Right. But you definitely need a team that is ready to support a junior SRE. You need a robust team that is interested in teaching and mentoring. And not all teams are like that, so making sure that you have a team culture that is receptive to taking on a junior SRE is step number one. And then I think that the act of having an observability practice on a team is very empowering to somebody who is new to the industry.Myself, I came from a self-taught background, learning to code. I actually have a music degree; I didn't go to school for computer science. And when I finally found my way to observability, it made so many, kind of, light bulbs go off of just giving me more visuals to go from, “I think this is happening,” to, “I know this is happening.” And then when I started mentoring juniors and apprentices and putting that same observability data in front of them, I noticed them learning so much faster.Corey: I am curious in that you went from implementing a lot of these things and being in a management role of mentoring folks on observability concepts to working for an observability vendor, which is… I guess I would call Honeycomb the observability vendor. They were the first to really reframe a lot of how we considered what used to be called monitoring and now it's called observability, or as I think of it, hipster monitoring.Brooke: [laugh].Corey: But I am curious as to when you look at this, my business partner wrote a book for O'Reilly, Practical Monitoring, and he loved it so much that by the end of that book, he got out of the observability monitoring space entirely and came to work on AWS bills with me. Did you find that going to Honeycomb has changed your perspective on observability drastically?Brooke: I had definitely consumed a lot of Honeycomb's blog posts, like, that's one of the things that I had loved about the company is they put out a lot of interesting stuff, not just about observability but about operating healthy teams, and like you mentioned, like, a pendulum between engineering management and being an IC and just interesting concepts within our industry overall as, like, software engineers and SREs. So, I knew a lot of the thought leadership that the company put out, and that was very helpful. It was a big change going from an enterprise like Procter and Gamble to a startup observability company like Honeycomb, just—and also, going from a company that very much believes in in-person work to remote-first work at Honeycomb, now. So, there were a lot of, like, cultural changes, but I think I kind of knew what I was getting myself into as far as the perspective that the company takes on observability.Corey: That is always the big, somewhat awkward question because of the answer goes a certain way, it becomes a real embarrassment, but I'm confident enough, having worked with Honeycomb as a recurring sponsor and having helped out on the AWS bill side of the world since you were a reference client on both sides of that business, I want to be very clear that I don't think I'm throwing you under a bus on this one. But do you find that the reality, now that you've been there for a year, has matched the external advertising and the ethos of the story they tell about Honeycomb from the outside?Brooke: I definitely think it matches up. One thing that is just different about working inside of a company like Honeycomb versus working at a company that doesn't have any observability at all yet, is that there are a lot of abstraction layers in our codebase and things like that. So, me being a software engineer and writing code Honeycomb compared to P&G, I don't have to think about observability as much because everybody in the company is thinking about observability and had thought about it before I joined and had put in a lot of thought to how to make sure that we consistently have telemetry data that we need to solve problems versus I was thinking about this stuff on the daily at P&G.Corey: Something I've heard from former employees of a bunch of different observability companies has a recurring theme to it, and that it's hard to leave. Because when you're at an observability company, everything is built with an eye toward observability. And there's always the dogfooding story of, we instrument absolutely everything we have with everything that we sell the customers. Now, in practice, you leave and go to a different company, that is almost never going to be true, if for no other reason than based on simple economics. Turning on every facet of every observability tool that a given company sells becomes extraordinarily expensive and is an investment decision, so companies say yes to some, no to others. Do you think you're going to have that problem if and when you decide it's time to move on to your next role, assuming of course, that it's not at a competing observability company?Brooke: I'm sure there will be some challenges if I decide to move on from working for observability platforms in the future. The one that I think would be the most challenging is joining a team where people just don't understand the value of observability and don't want to invest, like, the time and effort into actually instrumenting their code, and don't see why they need to do it, versus just, like, they haven't gotten there yet or they haven't had enough people hired to do it just yet. But if people are actively, like, kind of against the idea of instrumenting your code, I think that would be really challenging to kind of shift to especially after, over the last two-and-a-half years or so, being so used to having this, like, extra sense when I'm debugging problems and dealing with outages.Corey: I will say, it was a little surreal the first time I wound up taking a look at Honeycomb's environment—because I do believe that cost and architecture are fundamentally the same thing when it comes to cloud—and you had clear lines of visibility into what was going on in your AWS bill by way of Honeycomb as a product. And that's awesome. I haven't seen anyone else do that yet and I don't know that it would necessarily work as well because, as you said, there, everyone's thinking about it through this same shared vision, whereas in a number of other companies, it flat out does not work that way. There are certain unknowns and questions. And from the outside, and when you first start down this path, it feels like a ridiculous thing to do, until you get to a point of seeing the payoff, and yeah, this makes an awful lot of sense.I don't know that it would, for example, work as a generic solution for us to roll out to our various clients and say, oh, let's instrument your environment with this and see what's going on because first, we don't have that level of ability to make change in customer environments. We are read-only for some very good reasons. And further, it also seems like it's a, “Step one: change your entire philosophy around these sorts of things so we can help you spend less on AWS,” seems like a bit of a tall order.Brooke: Yeah, agreed. And yeah, on previous teams that I've been on, I definitely—and I think it's fair, absolutely fair, that there were things where, especially using AWS serverless services, I was trying to get as much insight as possible into adding some of these services to our traces, like, AppSync was one example where I could not for the life of me figure out how to get AppSync API requests onto my Honeycomb trace. And I spent a lot of time trying to figure it out. And I had team members that would just be, like, you know, “Let's timebox this; let's not, like, sink all of our time into it.” And so, I think as observability evolves, hopefully, carving out those patterns continues to get easier so that engineers don't have to spend all of their time, kind of, carving out those patterns.Corey: It feels like that's the hard part, is the shift in perspective. Instrumenting a given tool into an environment is not the heavy lift compared to appreciating the value of it. Do you find that that was an easy thing for you to overcome, back when you were at Procter and Gamble, as far as people already have bought in, on some level, to observability from having seen it in some kind of scenarios where it absolutely save folks' bacon? Or was it the problem of, first you have to educate people about the painful problem that they have before they realize it is in fact, A, painful, and B, a problem, and then C, that you have something to sell them that will solve that? Because that pattern is a very hard sales motion to execute in most spaces. But you were doing it at it, from the customer side first.Brooke: Yeah. Yeah, doing it from the customer side, I was able to get buy-in on the team that I was on, and I should also say, like, the team that I was on was considered an innovation team. We were in a separate building from, like, the corporate building and things like that, which I'm sure played into some of those cultural aspects and dynamics. But trying to educate people outside of our team and trying to build an observability practice within this big enterprise company was definitely very challenging, and it was a lot of spending time sharing information and talking to people about their stack and what languages and tools that they're using and how this could help them. I think until people have had that, kind of, magical moment of using observability data to solve a problem for themselves, it's very hard, it can be very hard to really make them understand the value.Corey: That was is always my approach because it feels like observability is a significant and sizable investment in infrastructure alone, let alone mental overhead, the teams to manage these things, et cetera, et cetera. And until you have a challenge that observability can solve, it feels like it is pure cost, similar to backups, where it's just a whole bunch of expense for no benefit until suddenly, one day, you're very glad you had it. Now, the world is littered with stories that are very clear about what happens when you don't have backups. Most people have a personal story around that, but it feels like it's less straightforward to point at a visceral story where not having observability really hobbled someone or something.It feels like—because in the benefit of perfect hindsight, oh yeah, like a disk filled up and we didn't know about that. Like, “Ah, if we just had the right check, we would have caught that early on.” Yeah, coulda, woulda shoulda, but it was a cascading failure that wasn't picked up until seven levels downstream. Do you think that that's the situation these days or am I misunderstanding how people are starting to conceive about this stuff?Brooke: Yeah. I mean, I definitely have a couple of stories of even once I was on the journey to observability adoption—which I call it a journey because you don't just—kind of—snap your fingers and have observability—I started with one service, instrumenting that and just, like, gradually, over sprint's would instrument more services and pull more team members in to do that as well. But when we were in that process of instrumenting services, there was one service which was our auth service—which maybe should have been the first one that we instrumented—that a code change was made and it was erroring every time somebody tried to sign up in the app. And if we had observability instrumentation in place for that service, it wouldn't have taken us, like, the four or five hours to find the problem of the one line of code that we had changed; we would have been able to see more clearly what error was happening and what line of code it was happening on and probably fix it within an hour.And we had a similar issue with a Redshift database that we were running more on the metrics side of things. We were using it to send analytics data to other people in the company and that Redshift database just got maxed out at a certain point. The CPU utilization was at, like, 98% and people in the company were very upset and [laugh] having a very bad time querying their analytics data.Corey: It's a terrific sales pitch for Snowflake, to be very direct, because you hear that story kind of a lot.Brooke: Yeah, it was not a fun time. But at that point, we started sending Redshift metrics data over to Honeycomb as well, so that we could keep a better pulse on what exactly was happening with that database.Corey: So, here's sort of the acid test: people tend to build software when they're starting out greenfield, in ways that emphasize their perspective on the world. For example, when I'm building something new, doesn't matter if it's tiny or for just a one-off shitposting approach, and it touches anything involving AWS, first thing I do out of the gate is I wind up setting tags so that I can do cost allocation work on it; someday, I'm going to wonder how much this thing cost. That is, I guess my own level of brokenness.Brooke: [laugh].Corey: When you start building something at work from scratch, I guess this is part ‘you,' part ‘all of Honeycomb,' do you begin from that greenfield approach of Hello World of instrumenting it for observability, even if it's not explicitly an observability-focused workload? Or is it something that you wind up retrofitting with observability insights later, once it hits a certain point of viability?Brooke: Yeah. So, if I'm at the stage of just kind of trying things out locally on my laptop, kind of outside of, like, the central repo for the company, I might not do observability data because I'm just kind of learning and trying things out on my laptop. Once I pull it into our central repo, there is some observability data that I am going to get, just in the way that we kind of have our services set up. And as I'm going through writing code to do this whatever new feature I'm trying to do, I'm thinking about what things, when this breaks—not if it breaks; when it breaks [laugh]—am I going to want to know about in the future. And I'll add those things, kind of, on the fly just to make things easier on myself, and that's just kind of how my brain works at this point of thinking about my future self, which is, kind of like, the same definition of self-care. So, I think of observability as self-care for developers.But later on, when we're closer to actually launching a thing, I might take another pass at just, like, okay, let's once again take a look at the error paths and how this thing can break and make sure that we have enough information at those points of error to know what is happening within a trace view of this request.Corey: My two programming languages that I rely on the most are enthusiasm and brute force, and I understand this is not a traditional software engineering approach. But I've always found that having to get observability in involved a retrofit, on some level. And it always was frustrating to me just because it felt like it was so much effort in various ways that I've just always kicked myself: I should have done this early on. But I've been on the other side of that, and it's like, should I instrument this with good observability? No, that sounds like work. I want to see if this thing actually works at all, or not first.And I don't know what side of the fence is the correct one to be on, but I always find that I'm on the wrong one. Like, I don't know if it's, like, one of those, there's two approaches and neither one works. I do see in client environments where observability is always, always, always something that has to be retrofit into what it is that they're doing. Does it stay that way once companies get past a certain point? Does observability of adoption among teams just become something that is ingrained into them or do people have to consistently relearn that same lesson, in your experience?Brooke: I think it depends, kind of, on the size of your company. If you are a small company with a, you know, smaller engineering organization where it's not, I won't say easy, but easier to get kind of full team buy-in on points of view and decisions and things like that, it becomes more built-in. If you're in a really big company like the one that I came from, I think it is continuously, like, educating people and trying to show the value of, like, why we are doing this—coming back to that why—and like, the magical moment of, like, stories of problems that have been solved because of the instrumentation that was in place. So, I guess, like most things, it's an, ‘it depends.' But the larger that your company becomes, I think the harder it gets to keep everybody on the same page.Corey: I am curious, in that I tend to see the world through AWS bills, which is a sad, pathetic way to live that I don't recommend to basically anyone, but I do see the industry, or at least my client base, forming a bit of a bimodal distribution. On one side, you have companies like Honeycomb, including, you know, Honeycomb, where the majority of your AWS spend is driven by the application that is Honeycomb, you know, the SaaS thing you sell to people to solve their problems. The other side of the world are companies that look a lot more like Procter and Gamble, presumably, where—because I think of oh, what does Procter and Gamble do? And the answer is, a lot. They're basically the definition of conglomerate in some ways.So, you look at that, a bill at a big company like that and it might be hundreds of millions of dollars, but the largest individual workload is going to be a couple million at best. So, it feels very much like it's this incredibly diffuse selection of applications. And in those environments, you have to start thinking a lot more about centralization things you can do, for example, for savings plan purchases and whatnot, whereas at Honeycomb-like companies, you can start looking at, oh, well, you have this single application that's the lion's share of everything. We can go very deep into architecture and start looking at micro-optimizations here that will have a larger impact. Having been an engineer at both types of companies, do you find that there's a different internal philosophy, or is it that when you're working in a larger company on a specific project, that specific project becomes your entire professional universe?Brooke: Yeah, definitely at P&G, for the most part, IoT was kind of the center of my universe. But one philosophy that I noticed as being different—and I think this is from being an enterprise in a startup—is just the way that thinking about cost and architecture choices, kind of, happened. So, at P&G, like I said, we were using a lot of Lambda, and pretty much any chance we got, we used a serverless or managed offering from AWS. And I think a big part of that reasoning was because, like I said earlier, P&G is very interested in in-person work. So, everybody that we hired her to be located in Cincinnati.And it became hard to hire for people who had Go and Terraform experience because a lot of people in the Midwest are much more comfortable in .NET and Java; there's just a lot more jobs using those technologies. So, we had a lot of trouble hiring and would choose—because P&G had a lot of money to spend—to give AWS that money because we had trouble finding engineers to hire, whereas Honeycomb really does not seem to have trouble hiring engineers. They hire remote employees and lots of people are interested in working at Honeycomb and they also do not have the bank account [laugh] that Procter and Gamble has, so just thinking about cost and architecture is kind of a different beast. So, at Honeycomb, we are building a lot more services versus just always choosing a serverless or easy, like, AWS managed option to think about it less.Corey: Yeah, at some level, it's an unfair question, just because it comes down to, in the fullness of time, even Honeycomb turns into something that looks a lot more like Procter and Gamble. Because, okay, you have the Honeycomb application. That's great, but as the company continues to grow and offer different things to different styles of customers, you start seeing a diffusion where, yeah, everything stills observability focused, but I can see a future where it becomes a bunch of different subcomponents. You make acquisitions of other companies that wind up being treated as separate environments and the rest. And in the fullness of time, I can absolutely see that that is the path that a lot of companies go down.So, it might also just be that I'm looking at this through a perspective lens of… just company stage, as opposed to what the internal story of the company is. I mean, Procter and Gamble's, what, a century old give or take? Whereas Honeycomb is an ancient tech company, by which I mean it's over 18 months old.Brooke: Yeah, P&G was founded in 1837. So that's—Corey: Almost 200 years old. Wonderful.Brooke: —quite old [laugh]. Yeah [laugh].Corey: And for some reason, they did not choose to build their first technical backbone on top of AWS back then. I don't understand why, for the life of me.Brooke: [laugh]. Yeah, but totally agree on your point that the kind of difference of thinking about cost and architecture definitely comes from company's stage rather than necessarily the industry.Corey: I really want to thank you for taking the time out of your day to talk with me about what you're up to and how you view these things. If people want to learn more, what's the best place for them to find you?Brooke: Yeah, so I think the main place that I still sometimes am, is Twitter: @codegirlbrooke is my username. But I'm only there sometimes, now [laugh].Corey: I feel like that's a problem a lot of us are facing right now. Like, I'm more active on Bluesky these days, but it's still invite only and it feels like it's too much of a weird flex to wind up moving people to just yet. I'm hoping that changes soon, but we'll see how it plays. We'll, of course, put links to that in the [show notes 00:31:53]. I really want to thank you for taking the time out of your day to talk with me.Brooke: Yeah, thanks so much for chatting with me. It was a good time.If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
In this episode, Emily and Dave catch up with Brice Pelle, Principal Product Manager for AWS Appsync. AppSync is a managed service that helps simplify data access, and the creation of scalable GraphQL APIs. Brice walks us through everything new in Appsync for developers post re:Invent 2022! This includes the ability to now write GraphQL APIs using JavaScript, the availability of two new libraries, and a new test API to simplify local development and testing. Brice on Twitter: https://twitter.com/BricePelle [DOCS] AWS AppSync Developer Guide - https://docs.aws.amazon.com/appsync/latest/devguide/what-is-appsync.html [DOCS] AWS AppSync JavaScript Resolver Developer Guide - https://docs.aws.amazon.com/appsync/latest/devguide/tutorials-js.html [GITHUB] AWS AppSync JavaScript SDK - https://github.com/awslabs/aws-mobile-appsync-sdk-js [PODCAST] AWS Developers Podcast - Episode 026 - AWS AppSync with Brice Pelle - https://open.spotify.com/episode/52FkI83nj99YwbNGGVIXOD [PODCAST] She's Off Script Podcast - https://www.serwaaadjeipelle.com/the-podcast/ [PORTAL] AWS AppSync - https://aws.amazon.com/appsync/ Subscribe: Amazon Music: https://music.amazon.com/podcasts/f8bf7630-2521-4b40-be90-c46a9222c159/aws-developers-podcast Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-developers-podcast/id1574162669 Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zb3VuZGNsb3VkLmNvbS91c2Vycy9zb3VuZGNsb3VkOnVzZXJzOjk5NDM2MzU0OS9zb3VuZHMucnNz Spotify: https://open.spotify.com/show/7rQjgnBvuyr18K03tnEHBI TuneIn: https://tunein.com/podcasts/Technology-Podcasts/AWS-Developers-Podcast-p1461814/ RSS Feed: https://feeds.soundcloud
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Michael #1: Jupyter Server 2.0 is released! Jupyter Server provides the core web server that powers JupyterLab and Jupyter Notebook. New Identity API: As Jupyter continues to innovate its real-time collaboration experience, identity is an important component. New Authorization API: Enabling collaboration on a notebook shouldn't mean “allow everyone with access to my Jupyter Server to edit my notebooks”. What if I want to share my notebook with e.g. a subset of my teammates? New Event System API: jupyter_events—a package that provides a JSON-schema-based event-driven system to Jupyter Server and server extensions. Terminals Service is now a Server Extension: Jupyter Server now ships the “Terminals Service” as an extension (installed and enabled by default) rather than a core Jupyter Service. pytest-jupyter: A pytest plugin for Jupyter Brian #2: Converting to pyproject.toml Last week, episode 314, we talked about “Tools for rewriting Python code” and I mentioned a desire to convert setup.py/setup.cfg to pyproject.toml Several of you, including Christian Clauss and Brian Skinn, let me know about a few tools to help in that area. Thank you. ini2toml - Automatically translates .ini/.cfg files into TOML “… can also be used to convert any compatible .ini/.cfg file to TOML.” “ini2toml comes in two flavours: “lite” and “full”. The “lite” flavour will create a TOML document that does not contain any of the comments from the original .ini/.cfg file. On the other hand, the “full” flavour will make an extra effort to translate these comments into a TOML-equivalent (please notice sometimes this translation is not perfect, so it is always good to check the TOML document afterwards).” pyproject-fmt - Apply a consistent format to pyproject.toml files Having a consistent ordering and such is actually quite nice. I agreed with most changes when I tried it, except one change. The faulty change: it modified the name of my project. Not cool. pytest plugins are traditionally named pytest-something. the tool replaced the - with _. Wrong. So, be careful with adding this to your tool chain if you have intentional dashes in the name. Otherwise, and still, cool project. validate-pyproject - Automated checks on pyproject.toml powered by JSON Schema definitions It's a bit terse with errors, but still useful. $ validate-pyproject pyproject.toml Invalid file: pyproject.toml [ERROR] `project.authors[{data__authors_x}]` must be object $ validate-pyproject pyproject.toml Invalid file: pyproject.toml [ERROR] Invalid value (at line 3, column 12) I'd probably add tox Don't forget to build and test your project after making changes to pyproject.toml You'll catch things like missing dependencies that the other tools will miss. Michael #3: aws-lambda-powertools-python Via Mark Pender A suite of utilities for AWS Lambda Functions that makes distributed tracing, structured logging, custom metrics, idempotency, and many leading practices easier Looks kinda cool if you prefer to work almost entirely in python and avoid using any 3rd party tools like Terraform etc to manage the support functions of deploying, monitoring, debugging lambda functions - Tracing: Decorators and utilities to trace Lambda function handlers, and both synchronous and asynchronous functions Logging - Structured logging made easier, and decorator to enrich structured logging with key Lambda context details Metrics - Custom Metrics created asynchronously via CloudWatch Embedded Metric Format (EMF) Event handler: AppSync - AWS AppSync event handler for Lambda Direct Resolver and Amplify GraphQL Transformer function Event handler: API Gateway and ALB - Amazon API Gateway REST/HTTP API and ALB event handler for Lambda functions invoked using Proxy integration Bring your own middleware - Decorator factory to create your own middleware to run logic before, and after each Lambda invocation Parameters utility - Retrieve and cache parameter values from Parameter Store, Secrets Manager, or DynamoDB Batch processing - Handle partial failures for AWS SQS batch processing Typing - Static typing classes to speedup development in your IDE Validation - JSON Schema validator for inbound events and responses Event source data classes - Data classes describing the schema of common Lambda event triggers Parser - Data parsing and deep validation using Pydantic Idempotency - Convert your Lambda functions into idempotent operations which are safe to retry Feature Flags - A simple rule engine to evaluate when one or multiple features should be enabled depending on the input Streaming - Streams datasets larger than the available memory as streaming data. Brian #4: How to create a self updating GitHub Readme Bob Belderbos Bob's GitHub profile is nice Includes latest Pybites articles, latest Python tips, and even latest Fosstodon toots And he includes a link to an article on how he did this. A Python script that pulls together all of the content, build-readme.py and fills in a TEMPLATE.md markdown file. It gets called through a GitHub action workflow, update.yml and automatically commits changes, currently daily at 8:45 This happens every day, and it looks like there are only commits if Note: We covered Simon Willison's notes on self updating README on episode 192 in 2020 Extras Brian: GitHub can check your repos for leaked secrets. Julia Evans has released a new zine, The Pocket Guide to Debugging Python Easter Eggs Includes this fun one from 2009 from Barry Warsaw and Brett Cannon >>> from __future__ import barry_as_FLUFL >>> 1 2 True >>> 1 != 2 File "[HTML_REMOVED]", line 1 1 != 2 ^ SyntaxError: invalid syntax Crontab Guru Michael: Canary Email AI 3.11 delivers First chance to try “iPad as the sole travel device.” Here's a report. Follow up from 306 and 309 discussions. Maps be free New laptop design Joke: What are clouds made of?
Two brothers discussing all things AWS every week. Hosted by Andreas and Michael Wittig presented by cloudonaut.
Listen to AWS.fm: https://aws.fm/episodes/episode-25-shawn-swyx-wangShawn joins Adam to discuss Amplify and its place in the developer ecosystem, whether we should care about Cloudflare, yet, and how to cope with the anxiety that can come with being extremely online. Also, it sounds like Adam is a tech bro and he's NOT happy about it.TranscriptAdam Elmore: Hey, everyone. Welcome to AWS FM, a podcast with guests from around the AWS community. I'm your host, Adam Elmore. And today, I'm joined by Shawn Swyx Wang. Hi, Shawn.Shawn Wang: Hey, Adam. How's it going?Adam Elmore: It's going well. I've been extremely excited. I've said this on a ton of podcasts, that I'm excited to get on with a guest, but this has been a long time because before I took my break, I was going to get on with you. Took a big, long break, and I've finally got you on. You're somebody, and I'm going to say a lot of things, I'm very dramatic, but you're somebody that I really admire in the online space. You have this ability to think about things, and distill them, and put them out there in a way that I admire greatly. I'm so excited to have you on here. It's going to be hard for me to stay on any one topic because I have just a list of questions I want to ask you, basically.Shawn Wang: [inaudible 00:00:52].Adam Elmore: First, could you tell everyone on this show who you are, just the short version of Shawn?Shawn Wang: Yeah. So I'm Shawn, born and raised in Singapore, went to The States for college and then spent my first career in finance where I did investment banking and hedge funds. Loved the coding part because every junior finance person starts to learn to code, and didn't like the stress of the finance part, so I pivoted to tech where I was a software engineer at Two Sigma and then I was in developer relations at Netlify, AWS, Temporal, and I've just joined Airbyte as head of developer experience.Adam Elmore: Oh, I did not know you weren't still at Temporal. So Airbyte, what is Airbyte?Shawn Wang: Airbyte is a data integration company, it basically has the largest community of open-source connectors for connecting to any SaaS API source into your data warehouse. So for anyone doing data engineering, the first task that you have to do is to get data from all the different silos of data in your business. Let's say you have a Salesforce being the source of truth for customers, Stripe being the source of truth for transactions, get all of them into a single data warehouse for you to do operations on. So the goal is to have the largest community of open-source developers for connecting all the data and liberating your data from all the silos that you have in your business.Adam Elmore: And how long ago did you start? How did I miss this?Shawn Wang: A couple weeks ago. I actually have not announced it on Twitter, which is why.Adam Elmore: Oh, there you go.Shawn Wang: I like to slow play it. So when I joined Temporal, I actually waited for six months to really understand Temporal and to practice my pitch before announcing it on Twitter. And that's how I like to do things because, well, partially I want to be fully up to speed before I represent something publicly.Adam Elmore: Yeah. So I want to talk about that. You get very up to speed in a way that I don't see a lot of people on Twitter. I don't see them understand things in the way that you do. So you obviously write, your blog is a huge source of information for me, and I've enjoyed it quite a lot, but it's not just that you write, it's the way you think about things. Does that come from your finance, your analytical background in finance, or were you like that before, your ability to see the whole forest, take in the way things are trending and the way things are moving, put it all together and distill it into these wonderful articles? Where does that come from?Shawn Wang: Oh, so first of all, thanks for the very kind words. I don't hear back from my readers that often, so it's really nice when I get to talk to someone like this. So yeah, I would say a lot of this stuff is actually from my finance days. This is the kind of analysis that you would have to do when you do an investment report or investment research on any stock or any industry. You want to get a perspective of what's going on, what the trends are, who the major players are, and form an opinion on where things are going. And I think taking that finance mindset into the bets I have, in terms of technologies, whether or not it's for using them personally in my personal stack or for joining them as a startup employee, I think is extremely underrated. And it's something I'm trying to model and hopefully teach people someday.Shawn Wang: Although I'm not sure about the teaching part, because if I say like, "Get rich by doing investment analysis stock on early stage startups," I would feel like a hustler. So maybe not that, but I just do like engaging in that. And probably it's an exercise for me to think things through clearly by writing it down. And I also get a lot of feedback from that, so I actually improve and learn a lot by learning in public. And that's the other thing that I am pretty well known for, so this is the application of the general purpose learning in public principle.Adam Elmore: Yeah. No, and I love your learning in public article. I hope more people see how you break down systems and the world around us and distill it. I hope more people do that because I'd love to have more sources of that kind of information. It's really fascinating and that's a lot of what I want to talk about today is your opinions on the future and where certain things are headed. First, I want to talk, you did work at AWS. How long were you at AWS?Shawn Wang: A year. AWS Amplify.Adam Elmore: Yeah. So I'd love to know, I guess what it was like working at AWS, what you took from that, but also more broadly, I want to get into Amplify and where it fits. You sort of live in that intersection. I feel like web, and cloud, and infrastructure, where things are trending, and I want to talk Amplify's place in that, but first, what was your role there like at AWS, at Amplify?Shawn Wang: Yeah, I was a senior dev advocate at Amplify, basically doing demos and talks for Amplify. And the fun thing about working at Amplify is that you are essentially also a developer advocate for all the underlying services. So amplify is essentially a roll up of DynamoDB, API Gateway, AWS AppSync, even file storage like S3. You could do some demos with that. And I did, I made like a DIY Dropbox clone. But it's focus on front-end engineers. And I think that was the first time that AWS had ever made a dedicated arm or products for front-end engineers. And it turned out to be a really good bet because AWS Amplify was one of the fastest growing AWS services, at least during the time that I was there. So I thought it was just really compelling to try it out and obviously everyone has very high regard for AWS. There's a bunch of services that I only experienced on the inside and I only learned about once I got on the inside, and I thought that was really interesting as well.Shawn Wang: A few things I'll point out. I really loved the AWS interview process, actually. I felt like it was very rigorous and I definitely haven't had as rigorous a process anywhere else. And they really got a good look at every single part of me before they made the decision. And fortunately for me, it was a unanimous, good decision, but I felt challenged. I felt like there was a lot of growth that I took away from that process as well. So I highly recommend going through it, even if you don't necessarily take the job.Shawn Wang: And once you're in, I think the other practice I really like was the weekly business reviews. Not everyone gets to be a part of, but I was, and essentially you have a P&L from the central AWS finance team that week to week tells you how well you're doing or not. And the PMs in particular, they'll put up highlights, they bring up topics of discussion, and the general manager would be grilling people on. And I thought that was just a fun way to run a business. It was a little bit stressful, sometimes a little bit dramatic, but hey, it forced you to take on the issues head on instead of ignoring them for three months to a year, which I've also seen happen.Shawn Wang: So I just really appreciated that directness, and everything that you've heard about on the outside about AWS culture applies, like they'll send out the memo and the first 10 minutes of the meeting will be spend in complete silence where you just read the memo.Adam Elmore: Just read the memo. Yeah, that's real. Well, what about the leadership principle? You talked about interviewing there. Did you feel like you started to embody those? Did those really become something you valued or was it sort of like, you're just doing it because that's what Amazon cares about?Shawn Wang: There are a few things here. So I think one, people are drawn to Amazon because of leadership principles, like literally is what the interview is for. So you can't really join without already having them ingrained in you. And then second, yes, it gets brought up a lot when decisions are being made or just behaviors being modeled or discussed, especially in the performance review stuff. So I think that is useful, that is helpful, but at the same time I have problems with some of the LPs myself. "Be right a lot." What the hell is that?Adam Elmore: So what is right?Shawn Wang: Yes, exactly. What is right, what is a lot? So I think that, for example, what is underdiscussed or just not on the table, just because it comes from so much up high and has so much baggage and history with it, is that sometimes you have to try to be wrong, to take more risks. And being right a lot means that you might be more conservative than you otherwise should be. It leads to very incrementalist thinking, which is like, "All right, what is the most obvious next step? What is the low-hanging fruit? What is the short thing?" You just pick that over something that is more risky, but potentially has higher impact.Adam Elmore: Yeah. No, that makes sense. I want to, I want to shift gears a little bit and talk about Amplify. Now that you're outside of AWS, you mentioned it was sort of the first example of AWS trying to go to the front-end developer and bundle up more of a developer experience. How do you feel? And you may have information from being there about traction and things like that. How do you feel about Amplify's return on investment and is Amazon doing a good job, I guess, with Amplify in terms of trying to package up their own experience? Do you see that resonating with developers?Shawn Wang: So I think Amazon is doing a good enough job at addressing the needs of AWS customers. And that's something that is Prime first and foremost, like excels at that. Amplify could be doing a lot better at competing with the other standalone front-end developer focused startups that are out there that don't have the AWS infrastructure, which should help, but actually sometimes hurts it a little bit. So my favorite example of this is, so there's another company Begin, begin.com with Brian LeRoux. It's a four-persons company, and they also do very similar things. They deploy on top of Amazon, they are entirely serverless, they have a smaller set of offerings that they have, but their deploy speeds are in order of magnitude, faster than Amplify. They can deploy faster to AWS than Amplify can.Shawn Wang: And that's because Amplify doesn't do some of the trickery that they do, like having a cold pool ready or anything like that. When people are not married to the AWS stack, just because that's the solution, that's the technology provider or cloud that their company has picked. When you have free choice, then you come with no baggage and just being from AWS doesn't give you any home ground advantage anymore. Therefore, you have to really, really, really compete on developer experience. And that's something that Amplify still needed to work on at the time that I left.Adam Elmore: Yeah. I'm glad you brought up Begin too. I'm curious how it fits into the landscape. I've seen you mention Begin within some of your articles, like the cloud distros article I think about, I want to talk about that, but how is Begin doing? I interact with Brian on Twitter, I generally like him a lot, I like what they're building, but it is sort of a thing you have to buy into. It's like a whole different way of building applications. Do you have any sense for how they fit as a player in all of this?Shawn Wang: They're tiny. I mean, they're not a rocket ship by any means, but they absolutely solve the problem for the serverless full stack minimalist aesthetic that they're going for.Adam Elmore: Those are all things I like, so.Shawn Wang: Right down to the API calls, having an inbuilt authentication solution that when you write the serverless function, you just have the user ID and it's all done for you with cookies in the background. That's just beautiful, that's [inaudible 00:12:58] mess with cognito or anything like that. Because it's very straightforward, that is the way that I would want to build serverless applications. If I didn't have some kind of big enterprise thing requirement, which maybe it's a premature optimization to try to glom that on in the first place, which is what you're required to do with AWS Amplify.Shawn Wang: So I don't think I have enough experience to really judge, are they the right technical choice in all aspects? But I think there's just a certain aesthetic that you try to optimize for. And if you have full stack needs, if you like serverless, if you like one of everything, essentially one story solution, one queuing solution, one database solution, then Begin is the right curation for you. And then Amplify is sort of the more fully loaded solution if you want an easy way to access, let's say API Gateway, even like the... Actually just before I left, they actually launched support for serverless containers with a AWS Fargate, which is also super interesting.Adam Elmore: Oh, I didn't even know Amplify supported that.Shawn Wang: Yeah, exactly. They're just different trade offs in the spectrum, like Begin is way more opinionated than Amplify. Amplify is way more opinionated than the full set of AWS services that are possibly out there. I think they serve front-end developers well in all different respects. Yeah. I think Amplify is definitely hitting its goals and probably exceeding its goals for adoption internally. Begin could do a better job at marketing and something that I should probably try to help them on just because I'm a friend of the company and so, I mean, I just really like the philosophy, but at the same time, there are other competitors out there, like CloudFlare Workers is essentially trying to become a Jamstack or a backend-as-a-service platform, because they have Workers KV and Durable Objects. And that's a very compelling solution for a particular type of audience.Shawn Wang: And it's weird because you have to be much more specific now. Like that's the thing, you have to figure out which part of the population you are in, in order to figure out which provider is best for you. There's no such thing as one provider fits all. It's really about like, "Okay, do you like the minimalist approach? Go with Begin. Do you like the edge-first approach? Maybe go with CloudFlare. Do you like the little bit more full stack, scalable, cloudy service? Maybe go with Amplify." There's a lot there. Like, "Do you like to self-host containers? Maybe go with Fly.io or Render.com. There's just a lot of options out there, but all of them happened to be built on top of AWS, which is why we had the cloud distros thesis.Adam Elmore: Yeah. And I've consumed a lot of your content on that front, like hosted back ends. I do wonder where it's all headed. Maybe the answer is that there's just going to be a lot of options, and because there's a lot of different use cases, I guess maybe narrowing it down. Like if I really don't care about enterprise stuff or big teams, if I just care about building stuff with small teams, startups, that's where I live. Do you have any predictions, I guess, for where ideal product building is headed? Is it hosted back ends to go with your hosted front ends on Vercel or whatever else? Is it learning AWS primitives and just good and good at building stuff? How do you see that forecasting into the future?Shawn Wang: What's the alternative to hosted back ends?Adam Elmore: I guess what I do right now is build... Like I kind of use all the Amplify services, I just don't use Amplify. So I build a lot of bespoke APIs with AppSync, and Dynamo, and whatever.Shawn Wang: So because you have that knowledge, that's the best thing for you, because you already have that knowledge. Like it's not a big deal for you to spin up another service, but for others it would be, because they would be new to that and sometimes a more friendly layer that abstracts it away for them would be helpful. So it's really hard to say which is going to win just because they're all going to win in some way, but some will be more winning than others. That's kind of how I view it.Adam Elmore: Yeah. Yeah.Shawn Wang: Because at the end of the day, like cloud is such a big deal, it's such a multi decade thing. It's going to take the rest of our lives to play out. That means that the vast majority of users of cloud haven't adopted it yet, still. This late into the game, they still haven't adopted it yet.Adam Elmore: It's so hard for me to wrap my brain around. It seems like it's been so long. And when you say the rest of our lives, I don't put it in that kind of perspective. I need to calm down trying to figure out what's going to happen in the next three years. Like it doesn't matter.Shawn Wang: Yeah. Yeah. Lambda is like seven years old. This is so early. The way that this looks 40, 50 years from now is going to be so different. AWS has like a million-something customers, imagine it having 10 million. When you have order of magnitude, when we start to think in terms of orders of magnitude, you start to really sweat the small details a lot less because you're like, "Whatever. Everyone's going to win."Adam Elmore: We all win. Yeah, I guess it's true. I don't know if you've talked about this, I'm sure you've thought about it, and maybe you have written about this, but it's the idea of scarcity versus abundance mentality, I guess. It's weird because all at the same time, I agree with the sentiment that if you're on Twitter or you're very online or whatever, you should have this mentality that we can all lift each other up and we can all succeed. But then on the other hand, you've got the climate and how much can the earth sustain in terms of everything can only grow so much. I just had that thought, that sort of raw stream of consciousness. So I don't know if you've got any refined response to that. Is that sort of totally different concepts that I shouldn't conflate?Shawn Wang: What, the limits to growth thesis?Adam Elmore: Oh, yeah. I guess that's what it's called. See, I knew you'd have a name for it or something. Like the idea that we can all succeed, but at the same time, we all need to do a lot less because the planet can't succeed if we all...Shawn Wang: I mean, this is about the offline-online shift. So we can still do a lot less and cloud can still grow because the mix of what we do in-cloud versus off-cloud is still very much imbalanced. So when you do things like pay attention to an Andy Jassy Keynote, and he'll talk about like, "Oh, cloud penetration is whatever, 20%, 30%." That is how low it is and it still takes a long time for people to adopt for whatever reason, institutional or just generational, or maybe our technology's not there yet. There's still a lot that needs to be developed to serve all kinds of markets that it hasn't penetrated. My favorite stat was that online shopping went from 10% to 20% in COVID.Adam Elmore: I can't believe it's only 20%. That's actually...Shawn Wang: Exactly, right?Adam Elmore: That's bonkers.Shawn Wang: So there's some version of the future where that is 70%, which means that you still have a long, long, long, long, long way to grow for every part of e-commerce and the planet can still win by maybe more efficient sorting or less retail outlets. I don't know. I don't know about that. I think I'm much more shakier ground there, but yeah, often the online transition, I think it is a very positive thing for the planet, especially because a lot of the major clouds are committing to net zero carbon footprints. I'm not sure if AWS has actually done that yet, but definitely Microsoft and Google have done it, which means AWS will eventually do it.Adam Elmore: And I know AWS, they've launched sustainability insights and stuff recently, where you can start to see the emissions impact of the services you're spinning up. I know Google's done that for some time, but AWS is now doing that, I think.Shawn Wang: Right. But we're actually measuring it now versus not measuring it before, so whatever. This is peanuts compared to like, "All right, are we moving to electric vehicles or something?" That is way more of an interesting concern than this stuff. Like invent a better battery and that will drastically accelerate the move to solar, and that will be much more meaningful than choosing paper straws. Sweating over the carbon footprint of your EC2 instance is the developer equivalent of choosing a paper straw. Really, look, I appreciate the effort, the spirit's, the heart's in the right place, but really if you want to make an impact, go work in the big things.Adam Elmore: I'm glad you said that because this is not on my notes, this is not something I planned to talk about, but this is the thing that I feel like to make an impact, I've really struggled, I'm 15 years into my career, I've been like a software engineer mostly early in my career, then I did a startup, and then I've mostly just been doing consulting. I feel like there are more possible things I could do with my time than ever. And it's so hard for me to decide what is worth spending time on.Adam Elmore: And I guess, do you have any thoughts on senior engineers, when you get to a point in your career where you have more flexibility and more opportunities, what is the most impactful thing? I've thought about making courses, I've thought about building products and just continuing with consulting. Is there a way to split your time that you're ever going to feel good about?Shawn Wang: Probably not.Adam Elmore: Okay. It's good to know. I can stop trying to find it.Shawn Wang: Yeah. The menu options is so high. I think just figure out what gives you energy and then try to spend more of your time and day on that than stuff that takes away energy from you, so it was just a very hippie thing for me to say.Adam Elmore: Yeah. No, that seems much simpler than I'm making it.Shawn Wang: There's a concept here that I do like to share about leverage. There's an inherent tension between productivity and leverage. I think we are trained from basically our days in school, that high productivity is the goal, which is you want to have a packed calendar, you want to be doing eight different things at once. You should feel bad if your efficiency went down 10% compared to last week or whatever, and you're not meeting your OKRs or whatever. And the exact opposite to that is leverage where you want to have one thing, you want to do one thing and just have a lot of impacts come out of that.Shawn Wang: And I think there's a movement, at least in VC circles, but also in sort of tech bro circles of waking up to the idea of slack in your life, and having peace and not having so much going on, and just doing high leverage activities that help you extend your reach without you necessarily putting more hours in or being super productive. Like being unproductive is fantastic. It's actually people who cannot figure out leverage who have to try to be productive. If you can figure out leverage, then productivity doesn't matter at all.Adam Elmore: Yeah. No, that's good stuff. I think I intuitively knew that. I just have a really hard time. I feel like I'm much more seeing the tree versus the forest, so I really appreciate talking with people like you that see the broader picture. I think I have a lot of thoughts and then I read an article of yours and it helps me put words to those thoughts that I couldn't really formalize in my head.Shawn Wang: I should really write about this more, but I feel like I haven't got it yet. You see me out there, you see me doing all sorts of random crap. So I haven't internalized it fully. I haven't let go of the sort of productivity mantra. Part of that is me being very risk-averse, part of that is me being doubting myself. Definitely, the stuff that you see from me has extremely high leverage. I think, okay... The other thing is I also have second thoughts or doubts about this whole leverage thing, that's why I have a very divisive tone about VCs and tech bros, because everyone wants to be high leverage, everyone wants to do the 80-20. Nobody wants to ship stuff, they just want to tweet thoughts, and then they think they're done. Right?Adam Elmore: Yeah.Shawn Wang: That's what they think high leverage is. But really the people who get shit done, swipe to find details and take things to the finish line. And guess what? Doing that last 10% is super low leverage. Like, "Oh man, I got to fix this stupid SEO description or the OG image isn't right, let me go fix that." That kind of small little details matter for the quality of the products and for shipping things, but all the high-leverage people feel like they're above that because it's not a good use of time.Adam Elmore: So are they the high-leverage people or you're saying the people that want to be high leverage, is that the VCs and the tech bros?Shawn Wang: Yeah, exactly.Adam Elmore: What is tech bro? I feel like I probably am a tech bro, and I don't want to be a tech bro, but I feel like I'm a white male that has a podcast, so I can't escape it.Shawn Wang: Yeah. Yeah. I'm a tech bro guy. I'm sort of reluctantly in that demographic. Yeah, the tech bro is a bro that's in tech.Adam Elmore: Okay. Yeah. Well.Shawn Wang: That is fully aware. Okay. I do like to have this mis-metric. If you're fully up to speed on the latest news, the gossip, you know all the new launches and new products, you're definitely a tech bro.Adam Elmore: Okay. Okay.Shawn Wang: If nothing surprises you, you're a tech bro. If you know what AUM is, if you know what ARR is, if you know all these acronyms without even blinking, you're a tech bro. Well, the real people who get shit done out there are wonderfully blissfully ignorant. They'll be like, "What is this whole Twitter kerfuffle, what's going on? I don't know. I just completely stayed out of the loop." But you being a tech bro, you would know the blow by blow of like Elon did this, twitter did that, Elon did other thing, twitter did other thing. It doesn't matter, the stuff doesn't matter to some extent and tech bros are so involved in their own filter bubble that they don't see their own forest for the trees, so.Adam Elmore: You said Twitter. I think I've been on Twitter actively for a year or so and I don't know that I'm better for it. I don't know that like... I know that I'm very influenced by that sphere and sort of feeling like, I think that's why it's so surprising to me when I hear about cloud adoption or I hear about online shopping. It just seems like everyone lives in this little community and it's very easy to just not really remember the people that are actually around me in my local community and what life is actually like. Is there a way to balance it? Is there a way to balance being very online, being a member of this Twitter community and still keep a good grasp on the real world?Shawn Wang: I don't think I personally have figured that out a lot, but I think it's basically the developer equivalent of go touch grass, which is go outside.Adam Elmore: Yeah, yeah, yeah.Shawn Wang: Have hobbies, have kids.Adam Elmore: That I was going to say, I've got two boys and they make me be outside a whole lot, so that probably helps, I guess, somewhat.Shawn Wang: Yeah, yeah, yeah.Adam Elmore: I think the biggest thing for me just career and in terms of the always online, the tech broness, I think giving my wife the opportunity to set some boundaries around the time that I am working, I think this stage of my career, I've been able to say I'm going to work less and just seeing her role and what her life looks like and realizing how it shouldn't be this different. Like we shouldn't have such a, I don't know, huge chasm in terms of our daily life. Like I get to go enjoy what I do all day. Yeah, that's helped. We've carved out a lot of time that's like, "This is time for family." I think yeah, but my online, my work life feels very homogenous, I guess. And it could be better.Shawn Wang: For me, it's like, "All right, figure out what is probably going to make your money and focus all your attention on that. Ignore everything else. Try to stick to, okay, what can you reasonably explain to your non-technical relatives? If you can't really justify it to them, then maybe have a second thought about like, 'All right, what am I really doing here?' Am I really making the world a better place by inventing a better form of infrastructure as code? Probably not." Unless you become a billionaire by creating HashiCorp, right?Adam Elmore: Yeah, I guess it happens in that very rare instance. Yeah.Shawn Wang: Right. But it can happen. You just have to be super clear on what you're trying to do here. And just like, yeah, be super intellectually honest about like, "Look, you're you're in this for the money, whatever you work on is probably going to be irrelevant in 10 years anyway. It doesn't matter, but you're at least going to have fun, you're going to build some relationships, you're going to make some people happy, create some jobs, whatever, and then spend the rest of your time with family and friends."Adam Elmore: That was a very succinct way of wrapping up a lot of the things I needed answered. So I don't know if anyone that listens to this podcast cares about any of this. I really appreciate the conversation we just had.Shawn Wang: No, no. I think yeah, this is very real and I really appreciate you bringing it up, because I don't get a lot of chance to talk about this.Adam Elmore: Yeah. No, I live in the Ozarks, so tech literacy here is super low. I think that's where getting into the Twitter community, it was like, "I have friends now that I can talk to about technology and things I care about." But yeah, finding that balance. I think it's really very practical of you, very wise of you to point out that ultimately this stuff doesn't necessarily matter in a decade, that whatever I think I'm working on that's so important is probably more about the people, more about what I'm kind of enjoying the process along the way and that it's making a living and that we're moving a little bit forward whatever parts we touch and what other people we can be involved with. That was very nice for me to hear.Shawn Wang: I will point out one thing. So humanity is kind of moving onto this metaverse. If there's anything that's actually real about the metaverse is that you have your community online that is dissociated from your physical community. You're so into AWS, or cloud, or anything like that, and no one else around you physically is, and it's fine. And this is something that actually the crypto bros, they probably got right. So I think Balaji Srinivasan, who is one of the crypto investors at Andreessen Horowitz, he released this book recently about building a digital nation, which is really compelling, which is like, essentially there's the world of physical nations, like the ones that country that've boundaries, but then there's the digital nations, which are formed online, and you're a member of the digital nation of probably tech Twitter, whatever.Adam Elmore: Yeah, yeah.Shawn Wang: Or AWS Twitter. And I kind of liken it to the difference between friends being the family that you choose versus the family that you have is the one that you're born with.Adam Elmore: Yeah, yeah, yeah, yeah.Shawn Wang: So where you're physically located is just the nation that you're born with or the nation that you have to live in for your family reasons, but the one that you do online, that's the nation that you choose, so you're member of a different nation online. And that nation is global, it's ephemeral, it's virtual, whatever that is. But it's something that you prefer to spend your time in as compared to your physical nation.Adam Elmore: Yeah. So I feel like since getting really active in Twitter and being involved with the AWS community, even outside of Twitter, it is so global. It's helped me see the perspective of America, where I live, so differently. Just getting all those other points of view and just knowing that when I interact with someone, it's not this base assumption that they understand the world through the lens of America like I do. I very much appreciate that. I feel like I'm, if anything, becoming more and more dissociated with the country I physically live in, because I just don't interact much with people outside of these walls. I don't know if it was COVID and being in all the time. I always have been kind of an at-home person.Shawn Wang: So that is dangerous. Right? That is dangerous.Adam Elmore: Yeah. It feels dangerous. Yeah, tell me why.Shawn Wang: Well, because if you don't care about the physical environment that you're in, then it's going to degrade, it's going to diverge away from your preference.Adam Elmore: Yeah.Shawn Wang: I don't know if that's inherently bad to me. Like there's definitely a physical element to humanity that we should keep around. We are not just brains plugged into the matrix. Essentially this leads to the matrix, that we might also just be plugged into something virtual online and spend zero time on a physical environment. Most people would not like to live that way, and that means we should care about what's going on around us. And we should try to have some physical presence that we're actually proud of and enjoy. And I think that there's a tension there that I think is sort of the modern humanistic existentialism, which is like, "How much of my life should I spend online versus how much should I spend in person?" And the fact that you have to choose is just nuts.Adam Elmore: Yeah. And I think my problem, like if I'm just being honest with myself and just thinking through this, I spend about as much time, I think, in the real world, but it's just with my family, at home, it's with my neighbor, I got a neighbor that I go for walks every week with. It's like my very, very hyper local community. But what's going on in the City of Nixa? It's like 10,000 people where I live. What's the local government doing? I don't know. I have no idea. What's the State of Missouri doing? Probably stuff I don't like.Shawn Wang: Exactly. And look, this has a very real impact on us because these people are making the laws that we have to follow. And we don't have a voice because we choose not to have a voice because we choose to not care. But hey, is it really our fault when the Supreme Court or the Congress makes a law that we don't like? Well, yeah. I mean, what did you expect? You didn't spend any time investing in that part of the world. It's like, "When are we going to have a software engineer in Congress?" That's really the big question.Adam Elmore: Yeah. There's not a lot of tech representation, is there? In government in the United States.Shawn Wang: No, because everyone hates politics, they love to dunk on it, they don't want to do a thing about it, but that's kind of the problem. I don't care which side of the bench you're on, like just the politicalness because you feel like you're not a member of the physical nation, you're a member of the digital nation. That is a problem for the physical nation, because at the end of the day, that's basically a reality.Adam Elmore: Yeah. Oh, I think of that, there was that Netflix documentary. I don't even know if it was just on Netflix, but there was that social. Well, I don't even remember what it was called, it was about social media and had all these people from Facebook and other places, or ex-Facebook, talking about just this impact that the very online nature of our generation, what it's doing to our brains and all that. This all sort of ties in my mind. Like I definitely need to do some more things that are yeah, going to impact my life, my kids' lives, sort of being more involved, I guess, outside of... Like I divide my time into I'm at work and I'm on a computer all day or I'm with my family and we're out in the yard playing. It's those two things. And I make no time for anything else, but that's probably not good. Not a good, long-term solution.Adam Elmore: Okay. Now I'm getting way off the rails. AWS FM, people literally listen to this for some good AWS bits. They've turned out long ago. I do have a couple more questions here, getting back to like I'm a developer, I like building full-stack web applications and I happen to like leveraging AWS. I'm going to ask you a few things. When should I care about CloudFlare? They announce all this cool stuff and it really is genuinely cool sounding, but there's so much of a barrier to adoption, like for me to change my day to day and start using a new thing. When should I care about CloudFlare?Shawn Wang: I have the article on this, about how CloudFlare is playing Go while AWS plays chess, so I highly recommend reading that up. Essentially, CloudFlare is a really good CDN. AWS has its own. I would think you can do up comparisons of CloudFront and CloudFlare all day long, but I would say that CloudFlare probably has much more of a security focus than CloudFront has, and that by default wins you the majority of the business and it happens to be very easily adoptable because you just need to configure some DNS, just is carrying a lot of weight there and it comes to DNS.Adam Elmore: If you're asking someone in the Ozarks around me, then what's DNS, first of all?Shawn Wang: So I think it basically starts from the outside in. You want to think about CloudFlare, you think about where your user's traffic is coming in. Maybe you want to protect those with CloudFlare and then you want to come in a little bit. CloudFlare has this S3 wrapper called R2, that basically reduces a lot of your outgoing bandwidth costs. And that seems like basically a Pareto optimal win. Pareto being you're no worse off in any dimension and you're better off in one dimension, which is cost. And that's just a function of CloudFlare.Shawn Wang: Like how many points of presence does AWS have? I think in the hundreds, maybe 100, 150, something like that. CloudFlare has tens of thousands, right?Adam Elmore: Oh, okay.Shawn Wang: It's just a much better edge network than AWS has. And so they just have a fundamentally different business model. And I think once you understand that from a fundamental physics and points of presence perspective, then you're understanding, "Okay, this is what I'm getting that AWS doesn't do." It's not a straight up one-to-one competitor, it's trying to tackle the cloud problem from a different way.Shawn Wang: So you do the cloud traffic protection, then you do the sort of egress charges, which are sort of the main sticking point of AWS. Then you get into the extra stuff that CloudFlare offers for application builders. And I focus on this because I'm an application builder. CloudFlare's other offerings for security that I have no idea, security and networking that I have no idea about, particularly if you need to wire a building or an office, they have a box that's pretty sweet for everything I heard. CloudFlare One is the name of it if you want to Google it.Adam Elmore: Okay. Yeah, I do.Shawn Wang: But for application developers, CloudFlare Workers, that team is the sort of primary team that's working on that. And that is, there's edge function service that would be a big leap to adopt because they don't run Node.js, they run V8 isolates, which are taken out of the Chrome V8 engine.Adam Elmore: Is it similar to like Lambda@Edge? Like the same kind of...?Shawn Wang: No, it is not.Adam Elmore: Oh, is Lambda@Edge node?Shawn Wang: Yes.Adam Elmore: Oh, it is.Shawn Wang: Yes.Adam Elmore: It is. Now, what is it similar to? It's similar to, I guess like Middleware and Next.js, that's that same kind of a limited runtime environment?Shawn Wang: I think so. Yeah, exactly, exactly. I would say it's more limited in Lambda@Edge and it's got different costs and criteria. Basically, there's just more of the open source ecosystem that it will be incompatible with CloudFlare Workers than it would be with Lambda@Edge. And that's the thing that you need to know because you're going to use...Adam Elmore: CloudFront Functions.Shawn Wang: Ah, okay. Yeah, that's the one I keep forgetting.Adam Elmore: I don't know who's using it, but that's what I was thinking of.Shawn Wang: Right. So I used to use this only for smart redirects, like looking at the headers of a request and saying, "If you're coming in with a header indicating you're from a certain region, certain IPS, certain language, then I'm going to route you to a different location than I would normally." Only for route, but now Edge Functions are becoming so capable that you might be able to do rendering on demands instead of just routing. And that actually is unlocking a few new things because on top of that, CloudFlare also has persistence solutions with Workers KV, which is their eventually consistent store, and Workers, and Durable Objects, which is their strongly consistent store. So either one of those combined with the ability to render, means that you can actually just host a site full stack with Front on the Edge. There's no origin server, there's no region, you just have everything everywhere all at once, which is a favorite phrase that I try to sneak in.Adam Elmore: Yeah. That's super compelling.Shawn Wang: So yeah, your latencies go down from like 300 milliseconds to nine, just because you're just pinging near a cell tower or something.Adam Elmore: Yeah, that's incredible. And they've just announced, I don't remember D1 or whatever. I don't know, I can't keep track of their product names, but they have like a distributed SQL offering as well that's coming or...Shawn Wang: SQLite. Yeah.Adam Elmore: Yeah. SQLite at the edge.Shawn Wang: I mean, everything's just built on top, it's just clearly built on top of the original persistence primitive that they have. And so once they got strongly consistent and eventually consistent, those are the two dimensions that you really care about. You can build any sort of solution on that, so the SQLite offering is just built on top of that.Adam Elmore: Yeah. Okay. So I don't know if I'm going to like jump on this stuff yet, but it does sound like there is a world where I could build side projects just on CloudFlare, like stuff runs all at the edge and I don't have to build up, I guess, is the interop, like if I want to still stand up a GraphQL API in AWS, like AppSync or something, is there interoping between the two services? You said their durable storage sits on top of S3, so it's actually, you're using an S3 bucket, you're just wrapping it with a CloudFlare thing?Shawn Wang: It's a proxy.Adam Elmore: Okay. Are people building hybrid CloudFlare, oh, I know they are, hybrid CloudFlare and AWS back ends today? I think I know of a couple at least. Is that a thing you recommend?Shawn Wang: I would say yeah, there are. I'd say this is definitely on the cutting edge. You do it because you feel like [inaudible 00:42:35].Adam Elmore: It's like Twitter, where you do it and you talk about it on Twitter and then everyone thinks...Shawn Wang: It's theoretically possible, it's just like probably not in any size.Adam Elmore: Doesn't make sense yet. Okay. So I'm going to say, I don't need to care about CloudFlare yet, that's what I'm going to say based on this conversation. I mean, I'm going to keep reading the articles, but.Shawn Wang: The only thing I'll point out is don't stop there because this is what they've achieved in the past three, four years, they clearly have a roadmap, they clearly are going to keep going, and just eating the cloud from outside in, which is the name of the article. What else of the functionality can be replicated in an-edge-first way? CloudFlare is probably going to do that. And so there's a whole roadmap that just consists of looking at the AWS console and just going, "That first, that first, that first comes [inaudible 00:43:17]."Adam Elmore: Yep. Yep. Yep.Shawn Wang: And then there's a question of just what kind of application are you building and do you really need the full set of AWS services, or can you just start from the edge first? That's how disruption happens. Disruption happens by taking a section on the market that nobody cared about and making that your entire thing, and then making it so capable over time that people see no use to use the old thing, but it takes a course of what, 10, 20 years to do that because AWS has just spent the past 20 years doing that in the first place.Adam Elmore: I just don't keep those time frames in mind. Like Twitter has warped my sense of when things are coming. And when you say 10, 20 years, it's like, I don't think about anything that's coming 10, 20 years from now. I think I'm thinking what's coming in the next 18 months.Shawn Wang: Right. But that's a problem for us, because that short-term mentality stops us from betting on big trends early. And I think to build anything of significance, you have to do it for 10 years.Adam Elmore: Yeah. I got to get off Twitter, that's what I'm coming to here.Shawn Wang: I think so. I think I'm going to do it in healthy amounts. So I actually, one of my longstanding wishlist projects is to actually build a Twitter client that has a time limit.Adam Elmore: Oh, nice. Yes.Shawn Wang: [inaudible 00:44:25] Client with a time limit. If you're going to have more time, you're going to have to pay to donate to your favorite charity or something.Adam Elmore: Oh, I love it.Shawn Wang: And that's in my wishlist.Adam Elmore: Yeah. I will use it. You've got your first user if you build it.Shawn Wang: I'll just say the only reason I don't do it is because nobody trusts the Twitter API.Adam Elmore: So one more, should I care about it yet or not? Because I see Brian LeRoux talk about this quite a bit. Deno. Should I care about Deno yet?Shawn Wang: I think so. I think it's there. I think it's there. So what is Deno? Dino is sort of the new runtime that the original creator of Node.js is saying, "All right, I'm going to do this over. Node.js has been around for 10 years. I see all the flaws of it, now I'm going to start over from scratch." I was very skeptical of Deno when it first came out, but it's been two years and it's really shown a lot of progress. And I think the governance is right, the funding model was right, and the adoption is growing. What is really compelling to me about Deno, just not from a technical perspective, from a business perspective, which feeds into a technical, the business side. There are companies so Superbase and Netlify, both launched edge functions powered by Deno, which means that their biggest products shipping capability announcement of the year of 2022 was someone else's product. It was a startup that's way younger than them, but they just have the right abstraction and the right cloud service that is already functional that they're launching. So it's weird.Shawn Wang: Deno's go-to-market strategy is just waiting for other people to wake up and go, "I need this. Deno's the only supplier in the market for this. And yeah, let's just bring it on and ship it as our thing." Where it really is Deno's thing, but they're just letting other people white label them. It's that's fantastic. So I mean, from that perspective alone in the past six months, I've really changed to, from like, "Okay, Node and Deno will coexist for the foreseeable future because there's such a huge install base of Node into every incremental app will probably be built in Deno."Adam Elmore: Well, that's... Yeah. No, that's what I needed to hear. I think I there's a lot of excitement. I see it all, but it's all Twitter, so I needed to hear it face to face that it's worth digging into.Adam Elmore: One last question. We do have a couple more minutes here. Do you have thoughts on the whole macro venture capital situation and how that might impact the next 5, 10 years? And I don't know if we're entering into some tightening cycle that we've never seen anything like the last 10 years, 13, whatever years, of government injecting so much capital into the system. And if that starts going away, do you have opinions or thoughts on all these startups that are making our lives better? Like I think of DevX startups where I don't know how financially sound they are yet, they've been living off the VC. Do you have thoughts on all that?Shawn Wang: Not fully formed ones, but I can give you a quick hit.Adam Elmore: Yeah. Yeah.Shawn Wang: So how bad did it get? It got to the point, so the average price of sales ratio of a publicly traded company would be in the range of 10 to 50. That's a very wide range, meaning your market capitalization, the total value of a company is 50 times your sales. In private markets, the price of sales ratios of funding rounds, series A and B, and all that, got up to 1,000 times.Adam Elmore: Oh my God.Shawn Wang: We had 1,500 at one of the startups that I was at and I heard of one startup that was 2,500.Adam Elmore: Wow.Shawn Wang: So that was the peak in November of last year. Those days are gone, people are now asking for 100X, which is very like 10X fall, like very, very big. That's why almost nobody's raising money. So that VC market is right up, I'll say it has different impact on different stages. And this is all to do with like, "Okay, would you invest in Stripe at 95 billion when Shopify used to be 100 billion and now it's worth 20 billion?" You probably want to buy the more quality asset that's already publicly listed than the very stable asset that is at a high valuation.Shawn Wang: So this is the deal making has just gone off. Like I think at the seed stage, people are completely unaffected. I think people are cognizant of the fact that economic cycles repeat or like, this is not going to... This is a recession. We are probably already in a recession right now, we are in a tightening cycle right now, but this is probably not one of those that's just going to drag out super long. And startup take 10 years to build anyway, so why should your early stage investing be affected at all by what the current level of the S&P is? It shouldn't.Adam Elmore: Yeah. No, it's true. I mean, so much of this conversation just echoes your bias towards long term versus short term, and I should have known that coming in. I'm asking all these questions that are very much like, there's a clear answer if you just think outside of the next year.Shawn Wang: Oh, I love training people to do that.Adam Elmore: Yeah. No, it's really nice.Shawn Wang: Take a long-term perspective in the history and then project it out to the future as well, and try to make decisions on that, so.Adam Elmore: Yeah, it's sort of refreshing, especially in this sort of anxiety-ridden digital space. I feel like when you zoom out things feel a lot less pressing or anxiety-laden, I guess. I don't know. Yeah, I appreciate that.Shawn Wang: It's weird because I think that's true, but at the same time, you're only here on this earth for so long. When you zoom out, that actually reduces the available number of decisions that you can possibly make, which means that each decision goes from being a two-way door into a one-way door because you want to make more substantial decisions. Therefore, for example, when I changed jobs, it took me like two months of agonizing to finally land on something, because I could have done any number of things and I think you have to really examine your beliefs as to what the long-term trends are going to be and trade that off versus being happy in the short run.Adam Elmore: Yeah. I'm going to be trying to do that. I think I'm in the middle of the agonizing stage right now, trying to figure out what's next, but I'm going to try and think a little more long term.Shawn Wang: The thing I'll point you to, you're talking about courses and stuff like that in leverage, I'll say definitely check out Eric Jorgenson, who is the book writer for Naval Ravikant. He wrote the Almanac of Naval Ravikant, and he's trying to build up a thesis or a body of knowledge around what leverage is and what leverage means. And then the other thing I'll point you to is Nathan Barry, who's the founder of ConvertKit who talked about the letters of wealth creation and how some things are more high leverage than others, so.Adam Elmore: Thank you so much for that. Again, this podcast may just be for me, but that's okay because I got a lot out of it. Thank you so much for taking the time, Shawn.Shawn Wang: [inaudible 00:50:58].Adam Elmore: I didn't know how much I'd get in on my... I think we covered half the things I thought about talking to you about. You're just a wealth of knowledge, you're sort of a wise sage in this community and it's been so great to pick your brain. Thanks for coming on.Shawn Wang: I think we're the same age.Adam Elmore: Oh, yeah. Well yeah, you've been using your time better, I guess. You've been doing more high-leverage things or something.Shawn Wang: Yeah. Thanks for having me around, but we can talk anytime. I really enjoyed this conversation.Adam Elmore: That sounds good. Thanks, Shawn.
About RafalRafal is Serverless Engineer at Stedi by day, and Dynobase founder by night - a modern DynamoDB UI client. When he is not coding or answering support tickets, he loves climbing and tasting whiskey (not simultaneously).Links Referenced:Company Website: https://dynobase.dev TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's not too often that I wind up building an episode here out of a desktop application. I've done it once or twice, and I'm sure that the folks at Microsoft Excel are continually hoping for an invite to talk about things. But we're going in a bit of a different direction today. Rafal Wilinski is a serverless engineer at Stedi and, in apparently what is the job requirement at Stedi, he also has a side project that manifests itself as a desktop app. Rafal, thank you for joining me today. I appreciate it.Rafal: Yeah. Hi, everyone. Thanks for having me, Corey.Corey: I first heard about you when you launched Dynobase, which is awesome. It sounds evocative of dinosaurs unless you read it, then it's D-Y-N-O, and it's, “Ah, this sounds a lot like DynamoDB. Let me see what it is.” And sure enough, it was. As much as I love misusing things as databases, DynamoDB is actually a database that is decent and good at what it does.And please correct me if I get any of this wrong, but Dynobase is effectively an Electron app that you install, at least on a Mac, in my case; I don't generally use other desktops, that's other people's problems. And it provides a user-friendly interface to DynamoDB that is not actively hostile to the customer.Rafal: Yeah, exactly. That was the goal. That's how I envisioned it, and I hope I executed correctly.Corey: It was almost prescient in some ways because they recently redid the DynamoDB console in AWS to actively make it worse, to wind up working with individual items, to modify things. It feels like they are validating your market for you by, “Oh, we really like Dynobase. How do we drive more traffic to it? We're going to make this thing worse.” But back then when you first created this, the console was his previous version. What was it that inspired you to say, “You know what I'm going to build? A desktop application for a cloud service.” Because on the surface, it seems relatively close to psychotic, but it's brilliant.Rafal: [laugh]. Yeah, sure. So, a few years ago, I was freelancing on AWS. I was jumping between clients and my side projects. That also involved jumping between regions, and AWS doesn't have a good out-of-the-box solution for switching your accounts and switching your regions, so when you want it to work on your client table in Australia and simultaneously on my side project in Europe, there was no other solution than to have two browser windows open or to, even, browsers open.And it was super frustrating. So, I was like, hey, “DynamoDB has SDK. Electron is this thing that allows you to make a desktop application using HTML and JS and some CSS, so maybe I can do something with it.” And I was so naive to think that it's going to be a trivial task because it's going to be—come on, it's like, a couple of SDK calls, displaying some lists and tables, and that's pretty much it, right?Corey: Right. I use Retool as my system to build my newsletter every week, and that is the front-end I use to interact with DynamoDB. And it's great. It has a table component that just—I run a query that, believe it or not, is a query, not a scan—I know, imagine that, I did something slightly right this one time—and it populates things for the current issue into it, and then I basically built a CRUD API around it and have components that let me update, delete, remove, the usual stuff. And it's great, it works for my purposes, and it's fine.And that's what I use most of the time until I, you know, hit an edge case or a corner case—because it turns out, surprise everyone, I'm bad at programming—and I need to go in and tweak the table myself manually. And that's where Dynobase, at least for my use case, really comes into its own.Rafal: Good to hear. Good to hear. Yeah, that was exactly same case why I built it because yeah, I was also, a few years ago, I started working on some project which was really crazy. It was before AppSync times. We wanted to have GraphQL serverless API using single table design and testing principles [unintelligible 00:04:38] there.So, we've been verifying many things by just looking at the contents of the table, and sometimes fixing them manually. So, that was also the thing that motivated me to make the editing experience a little bit better.Corey: One thing I appreciate about the application is that it does things right. I mean, there's no real other way to frame that. When I fire up the application myself and I go to the account that I've been using it with—because in this case, there's really only one account that I have that contains the data that I spent that my time working with—and I get access to it on my machine via Granted, which because it's a federated SSO login. And it says, “Ah, this is an SSL account. Click here to open the browser tab and do the thing.”I didn't have to configure Dynobase. It is automatically reading my AWS config file in my user directory. It does a lot of things right. There's no duplication of work. From my perspective. It doesn't freak out because it doesn't know how SSO works. It doesn't have run into these obnoxious edge case problems that so many early generation desktop interfaces for AWS things seem to.Rafal: Wow, it seems like it works for you even better than for me. [laugh].Corey: Oh, well again, how I get into accounts has always been a little weird. I've ranted before about Granted, which is something that Common Fate puts out. It is a binary utility that winds up logging into different federated SSO accounts, opens them in Firefox containers so you could have you know, two accounts open, side-by-side. It's some nice affordances like that. But it still uses the standard AWS profile syntax which Dynobase does as well.There are a bunch of different ways I've logged into things, and I've never experienced friction [unintelligible 00:06:23] using Dynobase for this. To be clear, you haven't paid me a dime. In fact, just the opposite. I wind up paying my monthly Dynobase subscription with a smile on my face. It is worth every penny, just because on those rare moments when I have to work with something odd in DynamoDB, it's great having the tool.I want to be very clear here. I don't recall what the current cost on this is, but I know for a fact it is more than I spend every month on DynamoDB itself, which is fine. You pay for utility, not for the actual raw cost of the underlying resources on it. Some people tend to have issues with that and I think it's the wrong direction to go in.Rafal: Yeah, exactly. So, my logic was that it's a productivity improvement. And a lot of programmers are simply obsessed with productivity, right? We tend to write those obnoxious nasty Bash and Python scripts to automate boring tasks in our day jobs. So, if you can eliminate this chore of logging to different AWS accounts and trying to find them, and even if it takes, like, five or ten seconds, if I can shave that five or ten seconds every time you try to do something, that over time accumulates into a big number and it's a huge time investment. So, even if you save, like, I don't know, maybe one hour a month or one hour a quarter, I think it's still a fair price.Corey: Your pricing is very interesting, and the reason I say that is you do not have a free tier as such, you have a free seven-day trial, which is great. That is the way to do it. You can sign up with no credit card, grab the thing, and it's awesome. Dynobase.dev for folks who are wondering.And you have a solo yearly plan, which is what I'm on, which is $9 a month. Which means that you end up, I think, charging me $108 a year billed annually. You have a solo lifetime option for 200 bucks—and I'm going to fight with you about that one in a second; we're going to come back to it—then you have a team plan that is for I think for ten licenses at 79 bucks a month, and for 20 licenses it's 150 bucks a month. Great. And then you have an enterprise option for 250 a month, the end. Billed annually. And I have problems with that, too.So, I like arguing with pricing, I [unintelligible 00:08:43] about pricing with people just because I find that is one of those underappreciated aspects of things. Let's start with my own decisions on this, if I may. The reason that I go for the solo yearly plan instead of a lifetime subscription of I buy this and I get to use it forever in perpetuity. I like the tool but, like, the AWS service that underlies it, it's going to have to evolve in the fullness of time. It is going to have to continue to support new DynamoDB functionality, like the fact that they have infrequent access storage classes now, for tables, as an example. I'm sure they're coming up with other things as well, like, I don't know, maybe a sane query syntax someday. That might be nice if they ever built one of those.Some people don't like the idea of a subscription software. I do just because I like the fact that it is a continual source of revenue. It's not the, “Well, five years ago, you paid me that one-off thing and now you expect feature enhancements for the rest of time.” How do you think about that?Rafal: So, there are a couple of things here. First thing is that the lifetime support, it doesn't mean that I will be always implementing to my death all the features that are going to appear in DynamoDB. Maybe there is going to be a some feature and I'm not going to implement it. For instance, it's not possible to create the global tables via Dynobase right now, and it won't be possible because we think that majority of people dealing with cloud are using infrastructure as a code, and creating tables via Dynobase is not a super useful feature. And we also believe that it's not going to break even without support. [laugh]. I know it sounds bad; it sounds like I'm not going to support it at some point, but don't worry, there are no plans to discontinue support [crosstalk 00:10:28]—Corey: We all get hit by buses from time to time, let's be clear.Rafal: [laugh].Corey: And I want to also point out as well that this is a graphical tool that is a front-end for an underlying AWS service. It is extremely convenient, there is tremendous value in it, but it is not critical path as if suddenly I cannot use Dynobase, my production app is down. It doesn't work that way, in the sense—Rafal: Yes.Corey: Of a SaaS product. It is a desktop application. And huge fan of that as well. So, please continue.Rafal: Yeah, exactly—Corey: I just want to make sure that I'm not misleading people into thinking it's something it's not here. It's, “Oh, that sounds dangerous if that's critical pa”—yeah, it's not designed to be. I imagine, at least. If so it seems like a very strange use case.Rafal: Yeah. Also, you have to keep in mind that AWS isn't basically introducing breaking changes, especially in a service that is so popular as DynamoDB. I cannot imagine them, like, announcing, like, “Hey, in a month, we are going to deprecate this API, so you'd better start, you know, using this new API because this one is going to be removed.” I think that's not going to happen because of the millions of clients using DynamoDB actively. So, I think that makes Dynobase safe. It's built on a rock-solid foundation that is going to change only additively. No features are going to be just being removed.Corey: I think that there's a direction in a number of at least consumer offerings where people are upset at the idea of software subscriptions, the idea of why should I pay in perpetuity for a thing? And I want to call out my own bias here. For something like this, where you're charging $9 a month, I do not care about the price, truly I don't. I am a price inflexible customer. It could go and probably as high as 50 bucks a month and I would neither notice nor care.That is probably not the common case customer, and it's certainly not over in consumer-land. I understand that I am significantly in a privileged position when it comes to being able to acquire the tools that I need. It turns out compared to the AWS bill I have to deal with, I don't have to worry about the small stuff, comparatively. Not everyone is in that position, so I am very sympathetic to that. Which is why I want to deviate here a little bit because somewhat recently, Dynobase showed up on the AWS Marketplace.And I can go into the Marketplace now and get a yearly subscription for a single seat for $129. It is slightly more than buying it directly through your website, but there are some advantages for many folks in getting it on the Marketplace. AWS is an approved vendor, for example, so there's no procurement dance. It counts toward your committed spend on contracts if someone is trying to wind up hitting certain levels of spend on their EDP. It provides a centralized place to manage things, as far as those licenses go when people are purchasing it. What was it that made you decide to put this on the Marketplace?Rafal: So, this decision was pretty straightforward. It's just, you know, yet another distribution channel for us. So, imagine you're a software engineer that works for a really, really big company and it's super hard to approve some kind of expense using traditional credit card. You basically cannot go to my site and check out with a company credit card because of the processes, or maybe it takes two years. But maybe it's super easy to click this subscribe on your AWS account. So yeah, we thought that, hey, maybe it's going to unlock some engineers working at those big corporations, and maybe this is the way that they are going to start using Dynobase.Corey: Are you seeing significant adoption yet? Or is it more or less a—it's something that's still too early to say? And beyond that, are you finding that people are discovering the product via the AWS Marketplace, or is it strictly just a means of purchasing it?Rafal: So, when it comes to discovering, I think we don't have any data about it yet, which is supported by the fact that we also have zero subscriptions from the Marketplace yet. But it's also our fault because we haven't actually actively promoted the fact, apart from me sending just a tweet on Twitter, which is in [crosstalk 00:14:51]—Corey: Which did not include a link to it as well, which means that Google was our friend for this because let's face it, AWS Marketplace search is bad.Rafal: Well, maybe. I didn't know. [laugh]. I was just, you know, super relieved to see—Corey: No, I—you don't need to agree with that statement. I'm stating it as a fact. I am not a fan of Marketplace search. It irks me because for whatever reason whenever I'm in there looking for something, it does not show me the things I'm looking for, it shows me the biggest partners first that AWS has and it seems like the incentives are misaligned. I'm sure someone is going to come on the show to yell about me. I'm waiting for your call.Rafal: [laugh].Corey: Do you find that if someone is going to purchase it, do you have a preference that they go directly, that they go through the Marketplace? Is there any direction for you that makes more sense than another?Rafal: So ideally, would like to continue all the customers to purchase the software using the classical way, using the subscriptions for our website because it's just one flow, one system, it's simpler, it's cleaner, but we want it to give that option and to have more adoption. We'll see if that's going to work.Corey: I was going to say there were two issues I had with the pricing. That was one of them. The other is at the high end, the enterprise pricing being $250 a month for unlimited licenses, that doesn't feel like it is the right direction, and the reason I say that is a 50-person company would wind up being able to spend 250 bucks a month to get this for their entire team, and that's great and they're happy. So, could AWS or Coca-Cola, and at that very high level, it becomes something that you are signing up for significant amount of support work, in theory, or a bunch of other directions.I've always found that from where I stand, especially dealing with those very large companies with very specific SLA requirements and the rest, the pricing for enterprise that I always look for as the right answer for my mind is ‘click here to contact us.' Because procurement departments, for example, we want this, this, this, this, and this around data guarantees and indemnities and all the rest. And well, yeah, that's going to be expensive. And well, yeah. We're a procurement company at a Fortune 50. We don't sign contracts that don't have two commas in them.So, it feels like there's a dialing it in with some custom optionality that feels like it is signaling to the quote-unquote, ‘sophisticated buyer,' as patio11 likes to say on Twitter from time to time, that might be the right direction.Rafal: That's really good feedback. I haven't thought about it this way, but you really opened my eyes on this issue.Corey: I'm glad it was helpful. The reason I think about it this way is that more and more I'm realizing that pricing is one of the most key parts of marketing and messaging around something, and that is not really well understood, even by larger companies with significant staff and full marketing teams. I still see the pricing often feels like an afterthought, but personally, when I'm trying to figure out is this tool for me, the first thing I do is—I don't even read the marketing copy of the landing page; I look for the pricing tab and click because if the only prices ‘call for details,' I know, A, it's going to be expensive, be it's going to be a pain in the neck to get to use it because it's two in the morning; I'm trying to get something done. I want to use it right now. If I had to have a conversation with your sales team first, that's not going to be cheap and it's not going to be something I'm going to be able to solve my problem this week. And that is the other end of it. I yell at people on both sides on that one.Rafal: Okay.Corey: Again, none of this stuff is intuitive; all of this stuff is complicated, and the way that I tend to see the world is, granted, a little bit different than the way that most folks who are kicking around databases and whatnots tend to view the world. Do you have plans in the future to extend Dynobase beyond strictly DynamoDB, looking to explore other fine database options like Redis, or MongoDB, or my personal favorite Route 53 TXT records?Rafal: [laugh]. Yeah. So, we had plans. Oh, we had really big plans. We felt that we are going to create a second JetBrains company. We started analyzing the market when it comes to MongoDB, when it comes to Cassandra, when it comes to Redis. And our first pick was Cassandra because it seemed, like, to have really, really similar structure of the table.I mean, it's also no secret it also has a primary index, secondary global indexes, and things like that. But as always, reality surprises us over the amount of detail that we cannot see from the very top. And it isn't as simple as just an install AWS SDK and install Cassandra Connector on—or Cassandra SDK and just roll with that. It requires a really big and significant investment. And we decided to focus just on one thing and nail this one thing and do this properly.It's like, if you go into the cloud, you can try to build a service that is agnostic, it's not using the best features of the cloud. And you can move your containers, for instance, across the clouds and say, “Hey, I'm cloud-agnostic,” but at the same time, you're missing out all the best features. And this is the same way we thought about Dynabase. Hey, we can provide an agnostic core, but then the agnostic application isn't going to be as good and as sophisticated as something tailored specifically for the needs of this database and user using this exact database.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Some of the things that you do just make so much sense that I get actively annoyed that there aren't better ways to do it and other places for other things. For example, when I fire up a table in a particular region within Dynobase, first it does a scan, which, okay, that's not terrible. But on some big tables, that can get really expensive. But you cap it automatically to a thousand items. And okay, great.Then it tells me, how long did it take? In this case because, you know, I am using on-demand and the rest and it's a little bit of a pokey table, that scan took about a second-and-a-half. Okay. You scanned a thousand items. Well, there's a lot more than a thousand items in this table. Ah, you limited it, so you didn't wind up taking all that time.It also says that it took 51-and-a-half RCUs—or Read Credit Units—because you know, why use normal numbers when you're AWS and doing pricing dimensions on this stuff.Rafal: [laugh].Corey: And to be clear, I forget the exact numbers for reads, but it's something like a million read RCUs cost me a dollar or something like that. It is trivial; it does not matter, but because it is consumption-based pricing, I always live in a little bit of a concern that, okay, if I screw up and just, like, scan the entire 10-megabyte table every time I want to make an operation here, and I make a lot of operations in the course of a week, that's going to start showing up in the bill in some really unfortunate ways. This sort of tells me as an ongoing basis of what it is that I'm going to wind up encountering.And these things are all configurable, too. The initial stream limit that you have configured as a thousand. I can set that to any number I want if I think that's too many or too few. You have a bunch of pagination options around it. And you also help people build out intelligent queries, [unintelligible 00:22:11] can export that to code. It's not just about the graphical interface clickety and done—because I do love my ClickOps but there are limits to it—it helps formulate what kind of queries I want to build and then wind up implementing in code. And that is no small thing.Rafal: Yeah, exactly. This is how we also envision that. The language syntax in DynamoDB is really… hard.Corey: Awful. The term is awful.Rafal: [laugh]. Yeah, especially for people—Corey: I know, people are going to be mad at me, but they're wrong. It is not intuitive, it took a fair bit of wrapping my head around. And more than once, what I found myself doing is basically just writing a thin CRUD API in Lambda in front of it just so I can query it in a way that I think about it as opposed to—now I'm not even talking changing the query modeling; I just want better syntax. That's all it is.Rafal: Yeah. You also touch on modeling; that's also very important thing, especially—or maybe even scan or query. Suppose I'm an engineer with tens years of experience. I come to the DynamoDB, I jump straight into the action without reading any of the documentation—at least that's my way of working—and I have no idea what's the difference between a scan and query. So, in Dynobase, when I'm going to enter all those filtering parameters into the UI, I'm going to hit scan, Dynobase is automatically going to figure out for you what's the best way to query—or to scan if query is not possible—and also give you the code that actually was behind that operation so you can just, like, copy and paste that straight to your code or service or API and have exactly the same result.So yeah, we want to abstract away some of the weird things about DynamoDB. Like, you know, scan versus query, expression attribute names, expression attribute values, filter, filtering conditions, all sorts of that stuff. Also the DynamoDB JSON, that's also, like, a bizarre thing. This JSON-type thing we should get out of the box, we also take care of that. So, yeah. Yeah, that's also our mission to make the DynamoDB as approachable as possible. Because it's a great database, but to truly embrace it and to truly use it, it's hard.Corey: I want to be clear, just for folks who are not seeing some of the benefits of it the way that I've described it thus far. Yes, on some level, it basically just provides a attractive, usable interface to wind up looking at items in a DynamoDB table. You can also use it to wind up refining queries to look at very specific things. You can export either a selection or an entire table either to a local file—or to S3, which is convenient—but it goes beyond on that because once you have the query dialed in and you're seeing the things you want to see, there's a generate code button that spits it out in—for Python, for JavaScript, for Golang.And there are a few things that the AWS CLI is coming soon, according to the drop-down itself. Java; ooh, you do like pain. And Golang for example, it effectively exports the thing you have done by clicking around as code, which is, for some godforsaken reason, anathema to most AWS services. “Oh, you clicked around to the console to do a thing. Good job. Now, throw it all away and figure out how to do it in code.” As opposed to, “Here's how to do what you just did programmatically.” My God, the console could be the best IDE in the world, except that they don't do it for some reason.Rafal: Yeah, yeah.Corey: And I love the fact that Dynobase does.Rafal: Thank you.Corey: I'm a big fan of this. You can also import data from a variety of formats, export data, as well. And one of the more obnoxious—you talk about weird problems I have with DynamoDB that I wish to fix: I would love to move this table to a table in a different AWS account. Great, to do that, I effectively have to pause the service that is in front of this because I need to stop all writes—great—export the table, take the table to the new account, import the table, repoint the code to talk to that thing, and then get started again. Now, there are ways to do it without that, and they all suck because you have to either write a shim for it or you have to wind up doing a stream that winds up feeding from one to the other.And in many cases, well okay, I want to take the table here, I do a knife-edge cutover so that new rights go to the new thing, and then I just want to backfill this old table data into it. How do I do that? The official answer is not what you would expect it to be, the DynamoDB console of ‘import this data.' Instead, it's, “Oh, use AWS Glue to wind up writing an ETL function to do all of this.” And it's… what? How is that the way to do these things?There are import and export buttons in Dynobase that solve this problem beautifully without having to do all of that. It really is such a different approach to thinking about this, and I am stunned that this had to be done as a third party. It feels like you were using the native tooling and the native console the same way the rest of us do, grousing about it the same way the rest of us do, and then set out to fix it like none of us do. What was it that finally made you say, “You know, I think there's a better way and I'm going to prove it.” What pushed you over the edge?Rafal: Oh, I think I was spending, just, hours in the console, and I didn't have a really sophisticated suite of tests, which forced me [unintelligible 00:27:43] time to look at the data a lot and import data a lot and edit it a lot. And it was just too much. I don't know, at some point I realized, like, hey, there's got to be a better way. I browsed for the solutions on the internet; I realized that there is nothing on the market, so I asked a couple of my friends saying like, “Hey, do you also have this problem? Is this also a problem for you? Do you see the same challenges?”And basically, every engineer I talked to said, “Yeah. I mean, this really sucks. You should do something about it.” And that was the moment I realized that I'm really onto something and this is a pain that I'm not alone. And so… yeah, that gave me a lot of motivation. So, there was a lot of frustration, but there was also a lot of motivation to push me to create a first product in my life.Corey: It's your first product, but it does follow an interesting pattern that seems to be emerging, Cloudash—Tomasz and Maciej—wound up doing that as well. They're also working at Stedi and they have their side project which is an Electron-based desktop application that winds up, we're interfacing with AWS services. And it's. What are your job requirements over at Stedi, exactly?People could be forgiven for seeing these things and not knowing what the hell EDI is—which guilty—and figure, “Ah, it's just a very fancy term for a DevRels company because they're doing serverless DevRel as a company.” It increasingly feels an awful lot like that.j, what's going on over there where that culture just seems to be an emergent property?Rafal: So, I feel like Stedi just attracts a lot of people that like challenges and the people that have a really strong sense of ownership and like to just create things. And this is also how it feels inside. There is plenty of individuals that basically have tons of energy and motivation to solve so many problems not only in Stedi, but as you can see also outside of Stedi, which is a result—Cloudash is a result, the mapping tool from Zack Charles is also a result, and Michael Barr created a scheduling service. So, yeah, I think the principles that we have at Stedi basically attract top-notch builders.Corey: It certainly seems so. I'm going to have to do a little more digging and see what some of those projects are because they're new to me. I really want to thank you for taking so much time to speak with me about what you're building. If people want to learn more or try to kick the tires on Dynobase which I heartily recommend, where should they go?Rafal: Go to dynobase.dev, and there's a big download button that you cannot miss. You download the software, you start it. No email, no credit card required. You just run it. It scans your credentials, profiles, SSOs, whatever, and you can play with it. And that's pretty much it.Corey: Excellent. And we will put a link to that in the [show notes 00:30:48]. Thank you so much for your time. I really appreciate it.Rafal: Yeah. Thanks for having me.Corey: Rafal Wilinski, serverless engineer at Stedi and creator of Dynobase. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—or a thumbs up and like and subscribe buttons on the YouTubes if that's where you're watching it—whereas if you've hated this podcast, same thing—five-star review, hit the buttons and such—but also leave an angry, bitter comment that you're not going to be able to find once you write it because no one knows how to put it into DynamoDB by hand.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Quelles sont les nouveautés AWS des deux dernières semaines ? Dans cet épisode, nous parlons de nouvelles instances EC2, de Elastic File System qui reduit la latence en lecture, de CodeGuru Reviewer qui enrichit ses librairies. Il sera question d'un anniversaire et de deux nouveautés attendues par les développeurs de fonctions Lambda ou d'API GraphQL avec AppSync.
Quelles sont les nouveautés AWS des deux dernières semaines ? Dans cet épisode, nous parlons de nouvelles instances EC2, de Elastic File System qui reduit la latence en lecture, de CodeGuru Reviewer qui enrichit ses librairies. Il sera question d'un anniversaire et de deux nouveautés attendues par les développeurs de fonctions Lambda ou d'API GraphQL avec AppSync.
In this episode, Dave chats with Brice Pelle, Principal Solutions Architect at AWS, about AWS AppSync. AppSync is a managed service that helps simplify data access, and the creation of scalable GraphQL APIs. Brice introduces AppSync, shares how developers can save time when managing mobile notifications, examples of how customers are using the service today, covers recent releases, and then what the AppSync team is looking to do next. Brice on Twitter: https://twitter.com/BricePelle Dave on Twitter: https://twitter.com/thedavedev AWS AppSync Service: https://go.aws/3Ao0Spa AWS AppSync Developer Documentation: https://go.aws/3nMx2FC AWS AppSync Customer Case Studies: https://go.aws/3AkCDYT GraphQL Website: https://bit.ly/3fJClBd
Listen to the Changelog: https://changelog.com/podcast/467Essays: https://www.swyx.io/LIP https://www.swyx.io/api-economy https://www.swyx.io/cloudflare-go TranscriptJerod Santo: So swyx, we have been tracking your work for years; well, you've been Learning in Public for years, so I've been (I guess) watching you learn, but we've never had you on the show, so welcome to The Changelog.Shawn Wang: Thank you. Long-time listener, first-time guest, I guess... [laughs]Adam Stacoviak: Yeah.Jerod Santo: Happy to have you here.Adam Stacoviak: Very excited to have you here.Jerod Santo: So tell us a little bit of your story, because I think it informs the rest of our conversation. We're gonna go somewhat deep into some of your ideas, some of the dots you've been connecting as you participate and watch the tech industry... But I think for this conversation it's probably useful to get to know you, and how you got to be where you are. Not the long, detailed story, but maybe the elevator pitch of your recent history. Do you wanna hook us up?Shawn Wang: For sure. For those who want the long history, I did a 2,5-hour podcast with Quincy Larson from FreeCodeCamp, so you can go check that out if you want. The short version is I'm born and raised in Singapore, came to the States for college, and was totally focused on finance. I thought people who were in the finance industry rules the world, they were masters of the universe... And I graduated just in time for the financial crisis, so not a great place to be in. But I worked my way up and did about 6-7 years of investment banking and hedge funds, primarily trading derivatives and tech stocks. And the more I covered tech stocks, the more I realized "Oh, actually a) the technology is taking over the world, b) all the value is being created pre-IPO, so I was investing in public stocks, after they were basically done growing... And you're kind of just like picking over the public remains. That's not exactly true, but...Jerod Santo: Yeah, tell that to Shopify...Shawn Wang: I know, exactly, right?Adam Stacoviak: And GitLab.Shawn Wang: People do IPO and have significant growth after, but that's much more of a risk than at the early stage, where there's a playbook... And I realized that I'd much rather be value-creating than investing. So I changed careers at age 30, I did six months of FreeCodeCamp, and after six months of FreeCodeCamp - you know, I finished it, and that's record time for FreeCodeCamp... But I finished it and felt not ready, so I enrolled myself in a paid code camp, Full Stack Academy in New York, and came out of it working for Two Sigma as a frontend developer. I did that for a year, until Netlify came along and offered me a dev rel job. I took that, and that's kind of been my claim to fame; it's what most people know me for, which is essentially being a speaker and a writer from my Netlify days, from speaking about React quite a bit.[04:13] I joined AWS in early 2020, lasted a year... I actually was very keen on just learning the entire AWS ecosystem. You know, a frontend developer approaching AWS is a very intimidating task... But Temporal came along, and now I'm head of developer experience at Temporal.Adam Stacoviak: It's an interesting path. I love the -- we're obviously huge fans of FreeCodeCamp, and Quincy, and all the work he's done, and the rest of the team has done to make FreeCodeCamp literally free, globally... So I love to see -- it makes you super-happy inside just to know how that work impacts real people.Like, you see things happen out there, and you think "Oh, that's impacting", but then you really meet somebody, and 1) you said you're a long-time listener, and now you're on the show, so it just really -- like, having been in the trenches so long, and just see all this over-time pay off just makes me really believe in that whole "Slow and steady, keep showing up, do what needs done", and eventually things happen. I just love that.Shawn Wang: Yeah. There's an infinite game mentality to this. But I don't want to diminish the concept of free, so... It bothers me a little, because Quincy actually struggles a lot with the financial side of things. He supports millions of people on like a 300k budget. 300k. If every single one of us who graduated at FreeCodeCamp and went on to a successful tech career actually paid for our FreeCodeCamp education - which is what I did; we started the hashtag. It hasn't really taken off, but I started a hashtag called #payitbackwards. Like, just go back, once you're done -- once you can afford it, just go back and pay what you thought it was worth. For me, I've paid 20k, and I hope that everyone who graduates FreeCodeCamp does that, to keep it going.Adam Stacoviak: Well, I mean, why not...?Shawn Wang: I'd also say one thing... The important part of being free is that I can do it on nights and weekends and take my time to decide if I want to change careers. So it's not just a free replacement to bootcamps, it actually is an async, self-guided, dip-your-toe-in-the-water, try-before-you-buy type of thing for people who might potentially change their lives... And that's exactly what happened for me. I kept my day job until the point I was like "Okay, I like enough of this... I'm still not good, but I like enough of this that I think I could do this full-time."Adam Stacoviak: I like the #payitbackwards hashtag. I wish it had more steam, I suppose.Jerod Santo: We should throw some weight behind that, Adam, and see if we can...Adam Stacoviak: Yeah. Well, you know, you think about Lambda School, for example - and I don't wanna throw any shade by any means, because I think what Austin has done with Lambda... He's been on Founders Talk before, and we talked deeply about this idea of making a CS degree cost nothing, and there's been a lot of movement on that front there... But you essentially go through a TL;DR of Lambda as you go through it, and you pay it after you get a job if you hit certain criteria, and you pay it based upon your earnings. So why not, right? Why not have a program like that for FreeCodeCamp, now that you actually have to commit to it... But it's a way. I love that you paid that back and you made that an avenue, an idea of how you could pay back FreeCodeCamp, despite the commitment not being there.Jerod Santo: Right.Shawn Wang: Yeah. And Quincy is very dedicated to it being voluntary. He thinks that people have different financial situations. I don't have kids, so I can afford a bit more. People should have that sort of moral obligation rather than legal obligation.I should mention that Lambda School is currently being accused of some fairly substantial fraud against its students...Jerod Santo: Oh, really?Shawn Wang: Yeah, it actually just came out like two days ago.Adam Stacoviak: I saw that news too, on Monday.Shawn Wang: Yeah. It's not evidenced in the court of law, it's one guy digging up dirt; let's kind of put this in perspective. But still, it's very serious allegations, and it should be investigated. That said, the business of changing careers and the business of teaching people to code, and this innovation of Income Share Agreements (ISA), where it actually makes financial sense for people to grow bootcamps and fund bootcamps - this is something I strongly support... Whether or not it should be a venture-funded thing, where you try to go for 10x growth every year - probably not... [laughs]Adam Stacoviak: Yeah...Jerod Santo: So after FreeCodeCamp you didn't feel quite ready, so you did do a bootcamp... Did you feel ready after that?Shawn Wang: [08:03] Yeah. [laughs] I did a reflection, by the way, of my first year of learning to code, so people can look it up... It's called "No zero days. My path to learning to code", and I think I posted it on Hacker News. And doing everything twice actually helped me a lot. Because before I came into my paid bootcamp, I had already spun up some React apps. I had already started to mess with WebPack, and I knew enough that I wasn't understanding it very much, I was just following the instructions. But the second time you do things, you have to space, to really try to experiment, to actually read the docs, which most people don't do, and actually try to understand what the hell it is you're doing. And I felt that I had an edge over the other people in my bootcamp because I did six months of FreeCodeCamp prior.Jerod Santo: So this other thing that you do, which not everybody does, is this Learning in Public idea... And you have this post, Learn in Public. You call it "The fastest way to learn", or the fastest way to build your expertise - networking, and second brain. I'm not sure what the second brain is, so help us out with that one... But also, why is learning in public faster than learning in private.Shawn Wang: Yeah. This is a reflection that came from me understanding the difference, qualitatively, between why I'm doing so well in my tech career versus my finance career. In finance, everything is private, meaning the investment memos that I wrote, the trade ideas that I had - they're just from a company; they're intellectual property of my company. In fact, I no longer own them. Some of my best work has been in that phase, and it's locked up in an email inbox somewhere, and I'll never see it again. And that's because tech is a fundamentally open and positive-sum industry, where if you share things, you don't lose anything; you actually gain from sharing things... Whereas in finance it's a zero-sum battle against who's got the secret first and who can act on it first.And I think when you're in tech, you should exploit that. I think that we have been trained our entire lives to be zero-sum, from just like the earliest days of our school, where we learn, we keep it to ourselves to try to pass the test, try to get the best scores, try to get the best jobs, the best colleges, and all that, because everything's positional. For you to win, others have to lose. But I don't see tech in that way, primarily because tech is still growing so fast. There's multiple ways for people to succeed, and that's just the fundamental baseline. You layer on top of that a bunch of other psychological phenomenon.I've been really fascinated by this, by what it is so effective. First of all, you have your skin in the game, meaning that a lot of times when your name is on the blog posts out there, or your name is on the talk that you gave, your face is there, and people can criticize you, you're just incentivized to learn better, instead of just "Oh, I'll read this and then I'll try to remember it." No, it doesn't really stick as much. So having skin in the game really helps.When you get something wrong in public, there are two effects that happen. First is people will climb over broken glass to correct you, because that's how the internet does. There's a famous XKCD comic where like "I can't go to bed yet." "Why?" "Someone's wrong on the internet. I have to correct them."Jerod Santo: Right.Shawn Wang: So people are incentivized to fix your flaws for you - and that's fantastic - if you have a small ego.Jerod Santo: I was gonna say, that requires thick skin.Shawn Wang: Yeah, exactly. So honestly -- and that's a barrier for a lot of people. They cannot get over this embarrassment. What I always say is you can learn so much on the internet, for the low, low price of your ego. If we can get over that, we can learn so much, just because you don't care. And the way to get over it is to just realize that the version that you put out today is the version you should be embarrassed about a year from now, because that shows that you've grown. So you divorce your identity from your work, and just let people criticize your work; it's fine, because it was done by you, before you knew what you know today. And that's totally fine.And then the second part, which is that once you've gotten something wrong in public, it's just so embarrassing that you just remember it in a much clearer fashion. [laughter] This built a feedback loop, because once you started doing this, and you show people that you respond to feedback, then it builds a feedback and an expectation that you'll do the next thing, and people respond to the next thing... It becomes a conversation, rather than a solitary endeavor of you just learning the source material.So I really like that viral feedback loop. It helps you grow your reputation... Because this is not just useful for people who are behind you; a lot of people, when they blog, when they write, when they speak, they're talking down. They're like "I have five years experience in this. Here's the intro to whatever. Here's the approach to beginners." They don't actually get much out of that.[12:17] That's really good, by the way, for beginners; that's really important, that experts in the field share their knowledge. They don't see this blogging or this speaking as a way to level up in terms of speaking to their experts in their fields. But I think it's actually very helpful. You can be helpful to people behind you, you can be helpful to people around you, but you can actually be helpful to people ahead of you, because you're helping to basically broadcast or personalize their message. They can check their messaging and see - if you're getting this wrong, then they're getting something wrong on their end, docs-wise, or messaging-wise. That becomes a really good conversation. I've interacted with mentors that way. That's much more how I prefer to interact with my mentors than DM-ing and saying "Hey, can you be my mentor?", which is an unspecified, unpaid, indefinitely long job, which nobody really enjoys. I like project-based mentorship, I like occasional mentorship... I really think that that develops when you learn in public.Adam Stacoviak: I've heard it say that "Today is the tomorrow you hope for."Shawn Wang: Wow.Adam Stacoviak: Because today is always tomorrow at some point, right? Like, today is the day, and today you were hoping for tomorrow to be better...Jerod Santo: I think by definition today is not tomorrow...Adam Stacoviak: No, today is the tomorrow that you hoped for... Meaning like "Seize your moment. It's here."Jerod Santo: Carpe diem. Gotcha.Adam Stacoviak: Yeah, kind of a thing like that.Shawn Wang: I feel a little shady -- obviously, I agree, but also, I feel a little shady whenever I venture into this territory, because then it becomes very motivational speaking-wise, and I'm not about that. [laughs]Adam Stacoviak: Kind of... But I think you're in the right place; keep showing up where you need to be - that kind of thing. But I think your perspective though comes from the fact that you had this finance career, and a different perspective on the way work and the way a career progressed. And so you have a dichotomy essentially between two different worlds; one where it's private, and one where it's open. That to me is pretty interesting, how you were able to tie those two together and see things differently. Because I think too often sometimes in tech, especially staying around late at night, correcting someone on the internet, you're just so deeply in one industry, and you have almost a bubble around you. You have one lens for which you see the world. And you've been able to have multi-faceted perspectives of this world, as well as others, because of a more informed career path.Jerod Santo: Yeah. When you talk about finance as a zero-sum game, I feel like there's actually been moves now to actually open up about finance as well; I'm not sure if either of you have tracked the celebrity rise of Cathie Wood and Ark Invest, and a lot of the moves that she's doing in public. They're an investment fund, and they will actually publish their moves at the end of every day. Like, "We sold these stocks. We bought these stocks." And people laughed at that for a while, but because she's been successful with early on Bitcoin, early with Tesla, she's very much into growth stocks - because of that, people started to follow her very closely and just emulate. And when she makes moves now, it makes news on a lot of the C-SPANs and the... Is C-SPAN the Congress one? What's the one that's the finance one...?Shawn Wang: CNBC?Jerod Santo: CNBC, not C-SPAN. And so she's very much learning in public. She's making her moves public, she's learning as she goes, and to a certain degree it's paid off, it's paid dividends in her career. Now, I'm not sure if everyone's doing that... When you look at crypto investors, like - okay, pseudonymous, but a lot of that stuff, public ledgers. So there's moves that are being made in public there as well. So I wonder if eventually some of that mentality will change. What do you think about that?Shawn Wang: [15:45] It's definitely changed for -- there's always been celebrity investors, and people have been copying the Buffett portfolio for 30 years. So none of that is new. What is new is that Cathie Wood is running an ETF, and just by way of regulation and by way of innovation, she does have to report those changes. [laughs] So mutual funds, hedge fund holdings - these have all been public, and people do follow them. And you're always incentivized to talk your book after you've established your position in your book...Jerod Santo: Right, but you establish it first.Shawn Wang: ...so none of that has changed. But yeah, Cathie has been leading an open approach...Jerod Santo: Is it the rate of disclosure perhaps that's new? Because it seems like it's more real-time than it has historically...Shawn Wang: Yeah. I mean, she's running an ETF, which is new, actually... Because most people just run mutual funds or hedge funds, and those are much more private. The other two I'll probably shout out is Patrick O'Shaughnessy who's been running I guess a fund of funds, and he's been fairly open. He actually adopted the "learn in public" slogan in the finance field, independently of me. And then finally, the other one is probably Ted Seides, who is on the institutional investor side of things. So he invests for universities, and teachers pensions, and stuff like that. So all these people - yeah, they've been leading that... I'm not sure if it's spreading, or they've just been extraordinarily successful in celebrity because of it.Adam Stacoviak: This idea of "in public" is happening. You see people too, like -- CopyAI is building in public... This idea of learning in public, or building in public, or exiting in public... Whatever the public might be, it's happening more and more... And I think it's definitely similar to the way that open source moves around. It's open, so it's visible to everyone. There's no barrier to see what's happening, whether it's positive or negative, with whatever it is in public. They're leveraging this to their advantage, because it's basically free marketing. And that's how the world has evolved to use social media. Social media has inherently been public, because it's social...Jerod Santo: Sure.Adam Stacoviak: Aside from Facebook being gated, with friends and stuff like that... Twitter is probably the most primary example of that, maybe even TikTok, where if I'm a creator on TikTok, I almost can't control who sees my contact. I assume it's for the world, and theoretically, controlled by the algorithm... Because if I live in Europe, I may not see content in the U.S, and the algorithm says no, or whatever. But it's almost like everybody is just in public in those spaces, and they're leveraging it to their advantage... Which is an interesting place to be at in the world. There was never an opportunity before; you couldn't do it at that level, at that scale, ten years ago, twenty years ago. It's a now moment.Jerod Santo: Yeah. Swyx, can you give us an example of something learned in public? Do you basically mean like blog when you've learned something, or ask questions? What does learning in public actually mean when it comes to -- say, take a technology. Maybe you don't understand Redux. I could raise my hand on that one... [laughter] How could I learn that in public?Shawn Wang: There are a bunch of things that you can try. You can record a livestream of you going through the docs, and that's useful to maintainers, understanding "Hey, is this useful or not?" And that's immediately useful. It's so tangible.I actually have a list -- I have a talk about this on the blog post as well... Just a suggestion of things you can do. It's not just blogging. You can speak, you can draw comics, cheatsheets are really helpful... I think Amy Hoy did a Ruby on Rails cheatsheet that basically everyone has printed out and stapled to their wall, or something... And if you can do a nice cheatsheet, I think that's also a way for you to internalize those things that you're trying to learn anyway, and it just so happens to benefit others.So I really like this idea that whatever content you're doing, it's learning exhaust, it's a side effect of you learning, and you just happen to put it out there; you understand what formats work for you, because you have abnormal talents. Especially if you can draw, do that. People love developers who can draw. And then you just put it out there, and you win anyway just by doing it. You don't need an audience. You get one if you do this long enough, but you don't need an audience right away. And you win whether or not people participate with you. It's a single-player game that can become a multiplayer game.Specifically for Redux - you know, go through source code, or go through the docs, build a sample app, do like a simple little YouTube video on it... Depending on the maturity, you may want to try to speak at a meetup, or whatever... You don't have to make everything a big deal. I'm trying to remove the perception from people that everything has to be this big step, like it has to be top of Hacker News, or something. No. It could just be helpful for one person. I often write blog posts with one persona in mind. I mean, I don't name that person, but if you focus on that target persona, actually often it does better than when you try to make some giant thesis that shakes the world...Adam Stacoviak: [20:22] Yeah. Too often we don't move because we feel like the weight of the move is just too much. It's like "How many people have to read this for me to make this a success for me?" You mentioned it's a learning exhaust... And this exhaust that you've put out before - has it been helpful really to you? Is that exhaust process very helpful to you? Is that ingrained in the learnings that you've just gone through, just sort of like synthesize "Okay, I learned. Here's actually what I learned"?Shawn Wang: Yeah. This is actually an opportunity to tie into that second brain concept which maybe you wanted to talk a little bit about. Everything that you write down becomes your second brain. At this point I can search Google for anything I've ever written on something, and actually come up on my own notes, on whatever I had. So I'm not relying on my memory for that. Your human brain, your first brain is not very good at storage, and it's not very good at search; so why not outsource that to computers? And the only way to do that is you have to serialize your knowledge down into some machine-readable format that's part of research. I do it in a number of places; right now I do it across GitHub, and my blog, and a little bit of my Discord. Any place where you find you can store knowledge, I think that's a really good second brain.And for Jerod, I'll give you an example I actually was gonna bring up, which is when I was trying to learn React and TypeScript - like, this goes all the way back to my first developer job. I was asked to do TypeScript, even though I'd never done it before. And honestly, my team lead was just like "You know TypeScript, right? You're a professional React dev, you have to know TypeScript." And I actually said no, and I started learning on day one.And what I did was I created the React to TypeScript cheatsheet, which literally was just copy-pasteable code of everything that I found useful and I wish I knew when I was starting out. And I've just built that over time. That thing's been live for three years now, it's got like 20,000 stars. I've taught thousands of developers from Uber, from Microsoft, React and TypeScript. And they've taught me - every time they send in a question or a PR... I think it's a very fundamental way of interacting, which is learning in public, but specifically this one - it's open source knowledge; bringing up our open source not just to code, but to everything else. I think that's a fundamental feedback loop that I've really enjoyed as well.Break: [22:31]Jerod Santo: One of the things I appreciate about you, swyx, is how you are always thinking, always writing down your thoughts... You've been watching and participating in this industry now for a while, and you've had some pretty (I think) insightful writings lately. The first one I wanna talk about is this API Economy post. The Light and Dark Side of the API Economy. You say "Developers severely underestimate the importance of this to their own career." So I figure if that's the case, we should hear more about it, right?Shawn Wang: [laughs] Happy to talk about it. So what is the API economy? The API economy is developers reshaping the world in their image. Very bold statement, but kind of true, in the sense that there is now an API for everything - API for cards, API for bank accounts, API for text, API for authentication, API for shipping physical goods... There's all sorts of APIs. And what that enables you to do as a developer is you can call an API - as long as you know REST or GraphQL these days, you know how to invoke these things and make these things function according to the rest of your program. You can just fit those things right in. They're a very powerful thing to have, because now the cost of developing one of these services just goes down dramatically, because there's another company doing that as a service for you.I wrote about it mainly because at Netlify we were pitching serverless, we were pitching static hosting, and we were pitching APIs. That's the A in JAMstack. But when I google "API economy", all the search results were terrible. Just horrible SEO, bland, meaningless stuff that did not speak to developers; it was just speaking to people who like tech buzzwords. So I wrote my own version. The people who coined it at Andreessen Horowitz, by the way, still to this day do not have a blog post on the API economy. They just have one podcast recording which nobody's gonna listen. So I just wrote my version.Jerod Santo: You're saying people don't listen to podcasts, or what?Shawn Wang: [laughs] When people are looking up a term, they are like "What is this thing?", and you give them a podcast, they're not gonna sit down and listen for 46 minutes on a topic. They just want like "Give me it, in one paragraph. Give me a visual, and I'm gonna move on with my day." So yeah, whenever I see an opportunity like that, I try to write it up. And that's the light side; a lot of people talk about the light side. But because it's a personal blog, I'm empowered to also talk about the dark side, which is that as much as it enables developers, it actually is a little bit diminishing the status of human expertise and labor and talent. So we can talk a little bit about that, but I'm just gonna give you time to respond.Jerod Santo: [28:05] Hm. I'm over here thinking now that you're not at Netlify, I'm curious - this is tangential, but what's your take on JAMstack now? I know you were a professional salesman there for a while, but... It seems like JAMstack - we've covered it for years, it's a marketing term, it's something we've already been doing, but maybe taking it to the next level... There's lots of players now - Netlify, Vercel etc. And yet, I don't see much out there in the real world beyond the people doing demos, "Here's how to build a blog, here's how to do this, here's my personal website", and I'm just curious... I'm not like down on JAMstack, but I just don't see it manifesting in the ways that people have been claiming it's going to... And maybe we're just waiting for the technology to catch up. I'd just love to hear what you think about it now.Shawn Wang: Yeah. I think that you're maybe not involved in that world, so you don't see this, but real companies are moving on to JAMstack. The phrasing that I like is that -- JAMstack has gone mainstream, and it's not even worth talking about these days, because it's just granted that that's an option for you... So PayPal.me is on the JAMstack, there's large e-commerce sites... Basically, anything that decouples your backend from your frontend, and your frontend is statically-hosted - that is JAMstack.I actually am blanking on the name, but if you go check out the recent JAMstack Conf, they have a bunch of examples of people who've not only moved to JAMstack, but obviously moved to Netlify, where they're trying to promote themselves.Jerod Santo: Sure, yeah.Shawn Wang: So yes, it's true that I'm no longer a professional spokesperson, but it's not true that JAMstack is no longer being applied in the enterprise, because it is getting adoption; it's moved on that boring phase where people don't talk about it.One thing I'll say - a thesis that I've been pursuing is that JAMstack is in its endgame. And what do I mean by that? There's a spectrum between the previous paradigm that JAMstack was pushing back on, which is the all-WordPress/server-render-everything paradigm, and then JAMstack is prerender-everything. And now people are filling in--Jerod Santo: In the middle.Shawn Wang: ...I'm gonna put my hands in the Zoom screen right now. People are filling that gap between fully dynamic and fully static. So that's what you see with Next.js and Gatsby moving into serverless rendering, partial rendering or incremental rendering... And there's a full spectrum of ways in which you can optimize your rendering for the trade-offs of updating your content, versus getting your data/content delivered as quickly as possible. There's always some amount of precompilation that you need to do, and there's always some amount of dynamicism that you have to do, that cannot be precompiled. So now there is a full spectrum between those.Why I say it's the end game is because that's it, there's nothing else to explore. It's full-dynamic, full-static, choose some mix in the middle, that's it. It's boring.Jerod Santo: Hasn't that always been the case though? Hasn't there always been sites that server-side render some stuff, and pre-render other things? You know, we cache, we pre-render, some people crawl their own websites once, and... I don't know it seems like maybe just a lot of excitement around a lot of things that we've been doing for many years.Shawn Wang: [laughs] So first of all, those are being remade in the React ecosystem of things, which a lot of us lost when a lot of the web development industry moved to React... So that's an important thing to get back.I mean, I agree, that's something that we've always had, pre-rendering, and services like that, caching at the CDN layer - we've always had that. There's some differences... So if you understand Netlify and why they're trying to push distributed persistent rendering (DVR), it's because caching is a hard problem, and people always end up turning off the cache. Because the first time you run into a bug, you're gonna turn off the cache. And the cache is gonna stay off.So the way that Netlify is trying to fix it is that we put the cache in Git, essentially. Git is the source of truth, instead of some other source of truth distributed somewhere between your CDN and your database and somewhere else. No, everything's in Git. I'm not sure if I've represented that well, to be honest... [laughter]Adam Stacoviak: Well, good thing you don't work for Netlify anymore. We're not holding you to the Netlify standard.Shawn Wang: [31:58] Exactly. All I can say is that to me now it's a good thing in the sense that it's boring. It's the good kind of boring, in the sense of like "Okay, there's a spectrum. There's all these techniques. Yes, there were previous techniques, but now these are the new hotness. Pick your choice." I can get into a technical discussion of why this technique, the first one, the others... But also, is it that interesting unless you're evaluating for your site? Probably not...Jerod Santo: Well, it does play into this API economy though, right? Because when you're full JAMstack, then the A is your most important thing, and when the A is owned by a bunch of companies that aren't yours - like, there's a little bit of dark side there, right? All of a sudden, now I'm not necessarily the proprietor of my own website, to a certain degree, because I have these contracts. I may or may not get cut off... There's a lot of concerns when everybody else is a dependency to your website.Shawn Wang: Yeah. So I don't consider that a dark side at all.Jerod Santo: No, I'm saying to me that seems like a dark side.Shawn Wang: Yeah, sure. This is the risk of lock-in; you're handing over your faith and your uptime to other people. So you have to trade that off, versus "Can you build this yourself? And are you capable of doing something like this, and are you capable of maintaining it?" And that is a very high upfront cost, versus the variable cost of just hiring one of these people to do it for you as a service.So what I would say is that the API economy is a net addition, because you as a startup - the startup cost is very little, and if you get big enough where it makes sense for you to build in-house - go ahead. But this is a net new addition for you to turn fixed costs into variable costs, and start with a small amount of investment. But I can hire -- like, Algolia was started by three Ph.D's in search, and I can hire them for cents to do search on my crummy little website. I will absolutely do that every single day, until I get to a big enough point where I cannot depend on them anymore, and I have to build my own search. Fine, I'll do that. But until then, I can just rely on them. That's a new addition there.Jerod Santo: One hundred percent. So what then do you think is the darker side? You mentioned it, but put a finer point on it.Shawn Wang: Yeah. The dark side is that there are people -- like, when I call an Uber ride, Uber is an API for teleportation, essentially. I'm here, I wanna go there. I press a button, the car shows up. I get in the car, get off, I'm there. What this papers over is that the API is calling real actual humans, who are being commoditized. I don't care who drives the car, I really don't. I mean, they may have some ratings, but I kind of don't care.Jerod Santo: That was the case with taxis though, wasn't it?Shawn Wang: That was the case with taxis, for sure. But there's a lot of people living below the API, who are economically constrained, and people who live above the API, developers, who have all the upside, essentially... Because the developers are unique, the labor is commoditized. My DoorDash pickers, my Instacart deliverers - all these are subsumed under the API economy. They're commodities forever, they know it, and there's no way out for them, unless they become developers themselves. There's a class system developing below and above the API. And the moment we can replace these people under the API with robots, you better believe we'll do that, because robots are way cheaper, and they complain less, they can work 24 hours, all this stuff.Jerod Santo: Yeah.Shawn Wang: So that's the dark side, which is, yeah, as a developer now - fantastic. I can control most parts of the economy with just a single API call. As a startup founder, I can develop an API for literally anything, and people will buy it. The downside is human talent is being commoditized, and I don't know how to feel about that. I think people are not talking enough about it, and I just wanna flag it to people.Jerod Santo: Yeah.Adam Stacoviak: So dark side could mean a couple things. One, it could mean literally bad; dark as synonymous with bad. Or dark as in shady. And we're not sure, it's obscured in terms of what's happening. And so let's use an Instacarter or a Dasher - to use their terminology. I happen to be a DoorDash user, so I know they're called Dashers; that's the only reason I know that. It's not a downplay, it's just simply what the terminology is...[35:59] You could say it's below the API, but I wonder, if you've spoken with these people, or people that live in what you call below the API, because I would imagine they're not doing that because they're being forced. Like, it's an opportunity for them.Shawn Wang: Oh, yeah.Adam Stacoviak: And I remember when I was younger and I had less opportunity because I had less "above the API" (so to speak) talent... And I do agree there's a class here, but I wonder if it's truly bad; that dark is truly bad, or if it's just simply obscure in terms of how it's gonna play out.Shawn Wang: This is about upside. They will never get to that six figures income with this thing.Adam Stacoviak: Not that job.Jerod Santo: No.Shawn Wang: It's really about the class system, which is the dark side. You don't want to have society splinter into like a serving class and whatever the non-serving class is. It's also about the upside - like, I don't see a way for these people to break out unless, they really just take a hard stop and just go to a completely different career track.Jerod Santo: Right.Adam Stacoviak: Here's where I have a hard time with that... I'm not pushing back on that you're wrong, I'm just wondering more deeply...Shawn Wang: Sure.Adam Stacoviak: I imagine at one point in my life I was a DoorDasher.Shawn Wang: Yeah.Adam Stacoviak: I washed dishes, I did definitely unique jobs at a young age before I had skill. And so the path is skill, and as long as we have a path to skill, which you've show-cased through FreeCodeCamp in your path, then I think that dark side is just simply shady, and not bad.Shawn Wang: Okay.Adam Stacoviak: And I'm just trying to understand it, because I was truly a DoorDasher before DoorDash was available. I washed dishes, delivered papers, I had servant-level things; I was literally a server at a restaurant before... And I loved doing that kind of work, but my talents have allowed me to go above that specific job, and maybe even the pay that came with that job. I've served in the military before, got paid terrible dollars, but I loved the United States military; it's great. And I love everybody who's served in our military. But the point is, I think the path is skill, and as long as we have a pathway to skill, and jobs that can house that skill and leverage that skill to create new value for the world, I just wonder if it's just necessary for society to have, I suppose, above and below API things.Jerod Santo: Until we have all the robots. Then there is nobody underneath. At that point it's all robots under the API.Shawn Wang: Yes, and that is true in a lot of senses, actually. Like, farming is mostly robots these days. You do have individual farmers, but they're much less than they used to be. I don't know what to say about that, shady or dark... I think it's just -- there's no career track. You have to go break out of that system yourself. Thank God there's a way to do it. But back in the day, you used to be able to go from the mailroom to the boardroom.Adam Stacoviak: I see.Shawn Wang: I see these stories of people who used to be janitors at schools become the principal. Companies used to invest in all their people and bring them up. But now we're just hiring your time, and then if you wanna break out of that system - good luck, you're on your own. I think that that lack of upward mobility is a problem, and you're not gonna see it today. It's a slow-moving train wreck. But it's gonna happen where you have society split in two, and bad things happen because of it.Adam Stacoviak: I mean, I could agree with that part there, that there definitely is no lateral movement from Dasher to CEO of DoorDash.Shawn Wang: It's just not gonna happen.Adam Stacoviak: Or VP of engineering at DoorDash. I think because there is no path, the path would be step outside of that system, because that system doesn't have a path. I could agree with that, for sure.Jerod Santo: Yeah. I mean, the good news is that we are creating -- there are paths. This is not like a path from X to Y through that system, but there are other alternate paths that we are creating and investing in, and as well as the API gets pushed further and further down in terms of reachability - we now have more and more access to those things. It's easier now, today, than it ever has been, because of what we were talking about, to be the startup founder, right? To be the person who starts at CEO because the company has one person in it, and they're the CEO. And to succeed in that case, and become the next DoorDash.Adam Stacoviak: True.Jerod Santo: So there are opportunities to get out, it's just not a clear line... And yeah, it takes perhaps some mentorship, perhaps ingenuity... A lot of the things that it takes to succeed anyway, so...Shawn Wang: [40:05] I'll give a closing note for developers who are listening, because you're already a developer... So the analogy is if you're above the API, you tell machines what to do; if you're below the API, machines tell you what to do. So here's the developer analogy, which is there's another division in society, which is the kanban board. If you're below the kanban board, the kanban board tells you what to do. If you're above it, you tell developers what to do. [laughs]Jerod Santo: There you go.Shawn Wang: So how do you break out of that class division? I'll leave it out to you, but just keep in mind, there's always layers.Jerod Santo: I love that.Adam Stacoviak: I love the discussion around it, but I'm also thankful you approached the subject by a way of a blog post, because I do believe that this is interesting to talk about, and people should talk about it, for sure. Because it provides introspection into, I guess, potentially something you don't really think about, like "Do I live below or above the APi?" I've never thought about that in that way until this very moment, talking to you, so... I love that.Break: [40:58]Jerod Santo: So another awesome post you have written lately is about Cloudflare and AWS. Go - not the language, the game Go... I know very little about the language, and I know even less about the game... And Chess... How Cloudflare is approaching things, versus how AWS and Google and others are... Given us the TL;DR of that post, and then we'll discuss.Shawn Wang: Okay. The TL;DR of that post is that Cloudflare is trying to become the fourth major cloud after AWS, Azure and GCP. The way they're doing it is fundamentally different than the other three, and the more I've studied them - I basically observed Cloudflare for the entire time since I joined Netlify. Netlify kind of is a competitor to Cloudflare, and it's always this uncomfortable debate between "Should you put Cloudflare in front of Netlify? Netlify itself is a CDN. Why would you put a CDN in front of another CDN?" Oh, because Netlify charges for bandwidth, and Cloudflare does not. [laughter]Jerod Santo: It's as simple as that.Shawn Wang: And then there's DDOS protection, all that stuff; very complicated. Go look up the Netlify blog post on why you should not put Cloudflare in front of Netlify, and decide for yourself. But Netlify now taking on AWS S3 - S3 is like a crown jewel of AWS. This is the eighth wonder of the world. It provides eleven nines of durability. Nothing less than the sun exploding will take this thing down... [laughs]Jerod Santo: Right? You know what's funny - I don't even consider us at Changelog AWS customers; I don't even think of us that way. But of course, we use S3, because that's what you do. So yeah, we're very much AWS customers, even though I barely even think about it, because S3 is just like this thing that of course you're gonna use.Shawn Wang: There's been a recent history of people putting out S3-compatible APIs, just because it's so dominant that it becomes the de-facto standard. Backblaze did it recently. But Cloudflare putting out R2 and explicitly saying "You can slurp up the S3 data, and by the way, here's all the cost-benefit of AWS egress charges that's what Matthew Prince wrote about in his blog post is all totally true, attacks a part of AWS that it cannot compromise on and just comes at the top three clouds from a different way, that they cannot respond to.[44:17] So I always like these analogies of how people play destruction games. I'm a student of destruction, and I study Ben Thompson and Clay Christensen, and that entire world, very quickly... So I thought this was a different model of destruction, where you're essentially embracing rather than trying to compete head-on. And wrapping around it is essentially what Go does versus chess, and I like -- you know, there's all these comparisons, like "You're playing 2D chess, I'm playing 3D chess. You're playing chess, I'm playing Go." So Cloudflare is playing Go by surrounding the S3 service and saying "Here is a strict superset. You're already a consumer of S3. Put us on, and magically your costs get lower. Nothing else about it changes, including your data still lives in AWS if you ever decide to leave us." Or if you want to move to Cloudflare, you've just gotta do the final step of cutting off S3.That is a genius, brilliant move that I think people don't really appreciate, and it's something that I study a lot, because I work at companies that try to become the next big cloud. I worked at Netlify, and a lot of people are asking, "Can you build a large public company on top of another cloud? Our second-layer cloud is viable." I think Vercel and Netlify are proving that partially it is. They're both highly valued. I almost leaked some info there... When does this go out? [laughs]Jerod Santo: Next week, probably...Shawn Wang: Okay, alright... So they're both highly valued, and - like, can they be hundred-billion-dollar companies? I don't know. We don't know the end state of cloud, but I think people are trying to compete there, and every startup -- I nearly joined Render.com as well. Every startup that's trying to pitch a second-layer cloud thesis is always working under the shadows of AWS. And this is the first real thesis that I've seen, that like "Oh, okay, you not only can credibly wrap around and benefit, you can actually come into your own as a fourth major cloud." So I'm gonna stop there... There's so many thoughts I have about Cloudflare.Jerod Santo: Yeah. So do you see that R2 then -- I think it's a brilliant move, as you described it... As I read your post, I started to appreciate, I think, the move, more than I did when I first read about it and I was like "Oh, they're just undercutting." But it seems they are doing more than just that. But do you think that this R2 then is a bit of a loss leader in order to just take a whole bunch of AWS customers, or do you think there's actually an economic -- is it economically viable as a standalone service, or do you think Cloudflare is using it to gain customers? What are your thoughts in their strategy of Why?Shawn Wang: This is the top question on Twitter and on Hacker News when they launch. They are going to make money on this thing, and the reason is because of all the peering agreements that they've established over the past five years. As part of the normal business strategy of Cloudflare, they have peering agreements with all of the ISPs; bandwidth is free for them. So... For them in a lot of cases. Again, I have to caveat all this constantly, because I should note to people that I am not a cloud or networking expert. I'm just learning in public, just like the rest of you, and here's what I have so far. So please, correct me if I'm wrong, and I'll learn from it.But yeah, I mean - straight on, it's not a loss leader. They plan to make money on it. And the reason they can is because they have worked so hard to make their cost structure completely different in AWS, and they've been a friend to all the other ISPs, rather than AWS consuming everything in its own world. Now you're starting to see the benefits of that strategy play out. And by the way, this is just storage, but also they have data store, also they have service compute, all following the same model.Jerod Santo: So what do you think is a more likely path over the next two years? Cloudflare --Adam Stacoviak: Prediction time!Jerod Santo: ...Cloudflare steals just massive swathes of AWS customers, or AWS slashes prices to compete?Shawn Wang: So I try not to do the prediction business, because I got out of that from the finance days... All I'm doing is nowcasting. I observe what I'm seeing now and I try to put out the clearest vision of it, so the others can follow.I think that it makes sense for them to be replicating the primitives of every other cloud service. So in 2017 they did service compute with Cloudflare Workers. In 2018 they did eventually consistent data store. In 2019 - website hosting; that's the Netlify competitor. In 2020 they did strongly-consistent data store, with Durable Objects. In 2021 object storage. What's next on that list? Go on to your AWS console and go shopping. And instead of seven different ways to do async messaging in AWS, probably they're gonna do one way in Cloudflare. [laughs]Adam Stacoviak: [48:34] A unified API, or something like that...Jerod Santo: Yeah, they'll just look at AWS' offerings, the ones they like the best, and do it that way, right?Shawn Wang: Yeah, just pick it up.Adam Stacoviak: Maybe the way to get a prediction out of you, swyx, might be rather than directly predict, maybe describe how you win Go.Shawn Wang: How you win Go...Adam Stacoviak: Yeah, what's the point of Go? How do you win Go? Because that might predict the hidden prediction, so to speak.Shawn Wang: Okay. For listeners who don't know Go, let me draw out the analogy as well. So most people are familiar with chess; individual chess pieces have different values and different points, and they must all support each other. Whenever you play chess, you need the Knight to support the pawns, something like that... Whereas in Go, you place your pieces everywhere, and they're all indistinguishable from each other. And it's more about claiming territory; at the end of the day, that's how you win Go, you claim the most territory compared to the others... And it's never a winner-take-all situation. Most likely, it's like a 60/40. You won 60% of the territory and your competitor has 40% of the territory. That's more likely a mapping of how cloud is gonna play out than chess, where winner-takes-all when you take the King. There's no King in the cloud, but--Jerod Santo: Are you sure...?Shawn Wang: ...there's a lot likely of territory claiming, and Cloudflare is really positioned very well for that. It's just part of the final realization that I had at the end of the blog post. And partially, how you take individual pieces of territory is that you surround all the pieces of the enemy and you place the final piece and you fill up all the gaps, such that the enemy is completely cut off from everything else and is surrounded. And that's what R2 does to S3 - it surrounds S3, and it's up to you to place that final piece. They call it, Atari, by the way, which is the name of the old gaming company, Atari. They have placed AWS S3 in Atari, and it's up to the customers to say "I'm gonna place that final piece. I'm gonna pay the cost of transferring all my data out of S3 and cut S3 off", and they cut off all the remaining liberties. So how do you win in Go? You claim the most amount of territory, and you surround the pieces of the enemy.Adam Stacoviak: Which, if you thought maybe that was oxygen, the territory, you might suck the oxygen away from them, so they can't live anymore, so to speak... And maybe you don't take it by killing it. Maybe you sort of suffocate it almost, if their space becomes small enough; if you take enough territory and it begins to shrink enough, it's kind of like checkmate, but not.Shawn Wang: Yeah. There's also a concept of sente in Go, which is that you make a move that the opponent has to respond to, which is kind of like a check, or checkmate -- actually, not; just the check, in chess. And right now, AWS doesn't feel the need to respond. Cloudflare is not big enough. Like, these are names to us, but let's just put things in numbers. Cloudflare's market cap is 36 billion, AWS' market cap is 1.6 trillion; this is Amazon's total market cap. Obviously, AWS is a subset of that.Jerod Santo: Sure.Shawn Wang: So your competitor is 40 times larger than you. Obviously, Cloudflare is incentivized to make a lot of noise and make themselves seem bigger than it is. But until AWS has to respond, this is not real.Adam Stacoviak: Nice.Jerod Santo: So as a developer, as a customer of potentially one or both of these... Let's say you have a whole bunch of stuff on S3 - I'm asking you personally now, swyx - and R2 becomes available... Is that a no-brainer for you, or is there any reason not to use that?Shawn Wang: You're just adding another vendor in your dependency tree. I think for anyone running silicon bandwidth, it is a no-brainer.Jerod Santo: Yeah. So over the course of n months, where n equals when they launch plus a certain number - I mean, I think this is gonna end up eventually on Amazon's radar, to where it's gonna start affecting some bottom lines that important people are gonna notice. So I just wonder - I mean, how much territory can Cloudflare grab before there's a counter-move? It's gonna be interesting to watch.Shawn Wang: [52:12] So Ben from Vantage actually did a cost analysis... Vantage is a startup that is made up former AWS Console people; they're trying to build a better developer experience on top of AWS. They actually did a cost analysis on the R2 move, and they said that there's probably a hundred billion dollars' worth of revenue at stake for Amazon. So if they start to have a significant dent in that, let's say like 40%, AWS will probably have to respond. But until then, there's nothing to worry about. That's literally how it is in Amazon; you have to see the numbers hit before you respond.Jerod Santo: Yeah. It hasn't even been a blip on the radar at this point, the key metrics to the people who are important enough to care are watching. You said you started watching all of these CDNs. Of course, you worked at Netlify... You take an interest in backends. There's something you mentioned in the break about frontenders versus backend, and where you've kind of been directing your career, why you're watching Cloudflare so closely, what you're up to now with your work... Do you wanna go there?Shawn Wang: Let's go there. So if you track my career, I started out as a frontend developer. I was developing design systems, I was working with Storybook, and React, and all that... Then at Netlify I was doing more serverless and CLI stuff. At AWS more storage and database and AppSync and GraphQL stuff... And now at Temporal I'm working on a workflow engine, pure backend. I just went to KubeCon two weeks ago...Jerod Santo: Nice!Shawn Wang: What is a frontend developer doing at KubeCon...?Adam Stacoviak: New territory.Shawn Wang: It's a frontend developer who realizes that there's a career ceiling for frontend developers. And it's not a polite conversation, and obviously there are exceptions to frontend developers who are VPs of engineering, frontend developers who are startup founders... And actually, by the way, there's a lot of VC funding coming from frontend developers, which is fantastic for all my friends. They're all getting funded, left, right and center. I feel left out. But there is a Career ceiling, in a sense that survey a hundred VPs of engineering, how many of them have backend backgrounds, and how many of them have frontend backgrounds? And given that choice, what's more likely for you and your long-term career progression? Do you want to specialize in frontend or do you want to specialize in backend? Different people have different interests, and I think that you can be successful in whatever discipline you pick. But for me, I've been moving towards the backend for that reason.Adam Stacoviak: Describe ceiling. What exactly do you mean when you say "ceiling"?Shawn Wang: Career ceiling. What's your terminal title.Jerod Santo: Like your highest role, or whatever. Highest salary, highest role, highest title...Adam Stacoviak: Gotcha.Shawn Wang: Like, straight up, how many VPs of engineering and CTOs have backend backgrounds versus frontend.Jerod Santo: Yeah. I mean, just anecdotally, I would agree with you that it's probably 8 or 9 out of 10 CTOs have -- is that what you said, 8 or 9?Shawn Wang: Yeah, yeah. So there's obviously an economic reasoning for this; it's because there's a bias in the industry that frontend is not real development, and backend is. And that has to be combated. But also, there's an economic reasoning, and I always go back to the economics part, because of my finance background... Which is that your value to the company, your value to the industry really depends on how many machines run through you. You as an individual unit of labor, how much money do you control, and how much machine process, or compute, or storage, or whatever runs through you. And just straight-up frontend doesn't take as much. [laughs] Yes, frontend is hard, yes, design is hard, yes, UX is crucially important, especially for consumer-facing products... But at the end of the day, your compute is being run on other people's machines, and people don't value that as much as the compute that I pay for, that I need to scale, and therefore I need an experienced leader to run that, and therefore that is the leader of my entire eng.Jerod Santo: I wonder if that changes at all for very product-focused orgs, where I think a lot of frontenders, the moves are into product design and architecture, and away from - not software architecture, but product design. And it seems like maybe if you compare - not VP of engineering, but VP of product, you'd see a lot of former frontenders.Shawn Wang: [56:03] Yeah.Jerod Santo: Maybe that's their path. Do you think that's --Shawn Wang: Totally. But you're no longer a frontend dev. You suddenly have to do mocks...Jerod Santo: Yeah, but when you're VP of engineering you're not a backend dev either.Shawn Wang: Yeah.Jerod Santo: So you're kind of both ascending to that degreeShawn Wang: Backends devs will never report to you, let's put it that way.Jerod Santo: Okay. Fair.Shawn Wang: [laughter] But somehow, frontend devs have to report to backend devs, for some reason; just because they're superior, or something. I don't know, it's just like an unspoken thing... It's a very impolite conversation, but hey, it's a reality, man.Jerod Santo: So do you see this personally, or do you see this by looking around?Shawn Wang: Yeah.Jerod Santo: Yeah. You felt like you had reached a ceiling.Shawn Wang: Well, again, this is very impolite; there's a ton of ways to succeed, and there are definitely exceptions. Emily Nakashima at Honeycomb - former frontend person, now VP of engineering. I don't know, I could have done that. I have interest in backend and I'm pursuing that. So I will say that - this is a soft ceiling, it's a permeable ceiling. It's not a hard ceiling.Jerod Santo: Sure.Shawn Wang: But there's a ceiling though, because you can see the numbers.Adam Stacoviak: What is it in particular the VP of engineering does that would make a frontender less likely to have that role? What specifically? I mean, engineering is one of the things, right? Commanding the software... Which is not necessarily frontend.Jerod Santo: Well, frontend is also an engineering discipline.Adam Stacoviak: I guess it kind of depends on the company, too. Honeycomb is probably a different example.Shawn Wang: I haven't been a VP of engineering, so I only have some theories. I suggest you just ask the next VP of engineering that you talk to, or CTO.Adam Stacoviak: Yeah.Jerod Santo: Yeah. That'd be a good one to start asking people.Adam Stacoviak: What do you do here? What is it you do here?Shawn Wang: What is it you do here?Jerod Santo: Exactly.Shawn Wang: [laughs]Adam Stacoviak: Well, I just wondered if there was a specific skillset that happens at that VP of engineering level that leads more towards a backender being more likely than a frontender to get hired into the role.Shawn Wang: I think there's some traditional baggage. Power structures persist for very long times... And for a long time UX and frontend was just not valued. And we're like maybe five years into the shift into that. It's just gonna take a long time.Jerod Santo: I agree with that. So tell us what you're up to now. You said you're doing workflows... I saw a quick lightning talk; you were talking about "React for the backend." So you're very much taking your frontend stuff into the backend here, with React for the backend. Tell us about that.Shawn Wang: Let's go for it. So at Netlify and at AWS I was essentially a developer advocate for serverless. So this is very cool - it does pay-as-you-go compute, and you can do a lot of cool stuff with it. But something that was always at the back of my mind bothering me, that serverless does not do well, is long-running jobs. It just does not do well. You have to chain together a bunch of stuff, and it's very brittle; you cannot test it... It's way more expensive than you would do in a normal environment.Jerod Santo: Yeah.Shawn Wang: And it made me realize that in this move to take apart everything and make everything as a service, we have gained scalability, but we've lost basically everything else. And what I was trying to do was "How do we reconstruct the experience of the monolith? What are the jobs to be done?" When you break it down, what does a computer do for you, and what is not adequately addressed by the ecosystem?I went through the exercise... I wrote a blog post called "Reconstructing the monolith, and I actually listed it out." So what are the jobs of cloud for a computer? You want static file serving, you want functions, you want gateway, you want socket management, job runners, queue, scheduler, cold storage, hot storage. There's meta jobs like error logging, usage logging, dashboarding, and then edge computing is like a unique to cloud thing. But everything else, you can kind of break it up and you can locate it on one machine, or you can locate it on multiple machines, some of them owned by you, some of them not owned by you.The thing that serverless -- that had a whole in the ecosystem was job running. Not good. Basically, as an AWS developer right now, the answer is you set a CloudWatch schedule function, and you pull an endpoint, and that should read some states from a database, and check through where you are, and compute until the 15-minute timeout for Lambda, and then save it back in, and then wait for the next pull, and start back up again. Super-brittle, and just a terrible experience; you would never want to go this way.[01:00:08.13] The AWS current response to that is AWS Step Functions, which is a JSON graph of what happens after the other, and this central orchestrator controls all of that. I think we could do better, and that's eventually what got me to temporal. So essentially, this blog post that I wrote - people found me through that, and hired both our head of product and myself from this single blog post. So it's probably the highest ROI blog post I've ever written.Jerod Santo: Wow. That's spectacular.Shawn Wang: It's just the VC that invested in Temporal. So what Temporal does is it helps you write long-running workflows in a doable fashion; every single state transition is persisted to a database, in idiomatic code. So idiomatic Java, idiomatic Go, idiomatic JavaScript, and PHP. This is different from other systems, because other systems force you to learn their language. For Amazon, you have to learn Amazon States Language. For Google Workflows - Google Workflows has a very long, very verbose JSON and YAML language as well.And these are all weird perversions of -- like, you wanna start simple; JSON is very simple, for doing boxes and arrows, and stuff like that... But you start ending up having to handwrite the AST of a general-purpose programming language, because you want variables, you want loops, you want branching, you want all that god stuff. And the best way to model asynchronous and dynamic business logic is with a general-purpose programming language, and that's our strong opinion there.So Temporal was created at Uber; it runs over 300 use cases at Uber, including driver onboarding, and marketing, and some of the trips stuff as well. It was open source, and adopted at Airbnb, and Stripe, and Netflix, and we have all those case studies on -- DoorDash as well, by the way, runs on the Uber version of Temporal.Jerod Santo: There you go, Adam.Shawn Wang: And yeah, they spun out to a company two years ago, and we're now trying to make it as an independent cloud company. And again, the
Yan joins Adam to discuss their shared love for AppSync & VTL and his experiences as a thought leader and content creator in the serverless space.
Un épisode sur deux du podcast est consacré à une brève revue des principales nouveautés AWS. Cette semaine, nous parlons de deux services important maintenant disponibles sur la région de Paris, de création automatique de pipeline de déploiements avec le CDK ou avec SAM, de EC2 classique qui prend sa retraite, de Lambda pour autoriser les appels d'API GraphQL avec AppSync et de Amplify qui s'intégère désormais avec Amazon Location Service.
Un épisode sur deux du podcast est consacré à une brève revue des principales nouveautés AWS. Cette semaine, nous parlons de deux services important maintenant disponibles sur la région de Paris, de création automatique de pipeline de déploiements avec le CDK ou avec SAM, de EC2 classique qui prend sa retraite, de Lambda pour autoriser les appels d'API GraphQL avec AppSync et de Amplify qui s'intégère désormais avec Amazon Location Service.
About Nader DabitNader Dabit is a web and mobile developer, author, and Developer Relations Engineer building the decentralized future at Edge and Node. Previously, he worked as a Developer Advocate at AWS Mobile working with projects like AWS AppSync and AWS Amplify. He is also the author and editor of React Native in Action and OpenGraphQL.Nader Dabit Twitter: @dabit3Edge and Node Twitter: @edgeandnodeGraph protocol Twitter: @graphprotocolEdge and Node: edgeandnode.com Everest: everest.link YouTube: YouTube.com/naderdabitWhat is Web3? The Decentralized Internet of the Future ExplainedWatch this episode on YouTube: https://youtu.be/pSv_cCQyCPQ This episode is sponsored by CBT Nuggets and Fauna. TranscriptJeremy: Hi everyone. I'm Jeremy Daly and this is Serverless Chats. Today I am joined again by Nader Dabit. Hey Nader, thanks for joining me.Nader: Hey Jeremy. Thanks for having me.Jeremy: You are now a developer relations engineer at Edge & Node. I would love it if you could tell the listeners a little bit about yourself. I think a lot of people probably know you already, but a little bit about your background and then what Edge & Node is.Nader: Yeah, totally. My name is Nader Dabit like you mentioned, and I've been a developer for about, I guess, nine or ten years now. A lot of people might know me from my work with AWS, where I worked with the Amplify team with the front end web and mobile team, doing a lot of full stack stuff there as well as serverless. I've been working as a developer relations person, developer advocate, actually, leading the front end web and mobile team at AWS for a little over three years I was there. I was a manager for the last year and I became really, really interested in serverless while I was there. It led to me writing a book, which is Full Stack Serverless. It also just led me down the rabbit hole of managed services and philosophy and all this stuff.It's been really, really cool to learn about everything in the space. Edge & Node is my next step, I would say, in doing work and what I consider maybe a serverless area, but it's an area that a lot of people might not associate with the traditional, I would say definition of serverless or the types of companies they often associate with serverless. But Edge & Node is a company that was spun off from a team that created a decentralized API protocol, which is called the Graph protocol. And the Graph protocol started being built in 2017. It was officially launched in a decentralized way at the end of 2020. Now we are currently finalizing that migration from a hosted service to a decentralized service actually this month.A lot of really exciting things going on. We'll talk a lot about that and what all that means. But Edge & Node itself, we do support the Graph protocol, that's part of what we do, but we also build out decentralized applications ourselves. We have a couple of applications that we're building as engineers. We're also doing a lot of work within the Web3 ecosystem, which is known as the decentralized web ecosystem by investing in different people and companies and supporting different things and spreading awareness around some of the things that are going on here because it does have a lot to do with maybe the work that people are doing in the Web2 space, which would be the traditional webspace, the space that I was in before.Jeremy: Right, right. Here I am. I follow you on Twitter. Love the videos that you do on your YouTube channel. You're like a shining example of what a really good developer relations dev advocate is. You just produce so much content, things like that, and you're doing all this stuff on serverless and I'm loving it. And then all of a sudden, I see you post this thing saying, hey, I'm leaving AWS Amplify. And you mentioned something about blockchain and I'm like, okay, wait a minute. What is this that Nader is now doing? Explain to me this, or maybe explain to me and hopefully the audience as well. What is the blockchain have to do with this decentralized applications or decentralized, I guess Web3?Nader: Web3 as defined by definition, what you might see if you do some research, would be what a lot of people are talking about as the next evolution of the web as we know it. In a lot of these articles and stuff that people are trying to formalize ideas and stuff, the original web was the read-only web where we were not creators, the only creators were maybe the developers themselves. Early on, I might've gone and read a website and been able to only interact with the website by reading information. The current version that we're currently experiencing might be considered as Web2 where everyone's a creator. All of the interfaces, all of the applications that we interact with are built specifically for input. I can actually create a comment, I can upload a video, I can share stuff, and I can write to the web. And I can read.And then the next evolution, a lot of people are categorizing, yes, is Web3. It's like taking a lot of the great things that we have today and maybe improving upon those. A lot of people and everyone kind of, this is just a really, a very old discussion around some of the trade-offs that we currently make in today's web around our data, around advertising, around the way a lot of business models are created for monetization. Essentially, they all come down to the manipulation of user data and different tricks and ways to steal people's data and use that essentially to create targeted advertising. Not only does this lead to a lot of times a negative experience. I just saw a tweet yesterday that resonated a lot with me that said, "YouTube is no longer a video platform, it's now an ad platform with videos in between." And that's the way I feel about YouTube. My kids ...Jeremy: Totally.Nader: ... I have kids that use YouTube and it's interesting to watch them because they know exactly what to do when the ads come up and exactly how to time it because they're used to, ads are just part of their experience. That's just what they're used to. And it's not just YouTube, it's every site that's out there, that's a social site, Instagram, LinkedIn. I think that that's not the original vision that people had, right, for the web. I don't think this was part of it. There have been a lot of people proposing solutions, but the core fundamental problem is how these applications are engineered, but also how the applications are paid for. How do these companies pay for developers to build. It's a really complex problem that, the simplest solution is just sell ads or maybe create something like a developer platform where you're charging a weekly or monthly or yearly or something like that.I would say a lot of the ideas around Web3 are aiming to solve this exact problem. In order to do that you have to rethink how we build applications. You have to rethink how we store data. You have to rethink about how we think about identity as well, because again, how do you build an application that deals with user data without making it public in some way? Right? How do we deal with that? A lot of those problems are the things that people are thinking about and building ways to address those in this decentralized Web3 world. It became really fascinating to me when I started looking into it because I'm very passionate about what I'm doing. I really enjoy being a developer and going out and helping other people, but I always felt there was something missing because I'm sitting here and I love AWS still.In fact, I would 100% go back and work there or any of these big companies, right? Because you can't really look at a company as, in my opinion, a black or white, good or bad thing, there's companies are doing good things and bad things at the same time. For instance, at AWS, I would meet a developer, teach them something at a workshop, a year later they would contact me and be like, hey, I got my first job or I created a business, or I landed my first client. So you're actually helping improve people's lives, at the same time you're reading these articles about Amazon in the news with some of the negative stuff going on. The way that I look at it is, I can't sit there and say any company is good or bad, but I felt a lot of the applications that people were building were also, at the end goal when you hear some of these VC discussions or people raising money, a lot of the end goal for some of the people I was working with were just selling advertising.And I'm like, is this really what we're here to do? It doesn't feel fulfilling anymore when you start seeing that over and over and over. I think the really thing that fascinated me was that people are actually building applications that are monetized in a different way. And then I started diving into the infrastructure that enabled this and realized that there was a lot of similarities between serverless and how developers would deploy and build applications in this way. And it was the entry point to my rabbit hole.Jeremy: I talked to you about this and I've been reading some of the stuff that you've been putting out and trying to educate myself on some of this. It seems very much so that show Silicon Valley on HBO, right? This decentralized web and things like that, but there's kind of, and totally correct me if I'm wrong here, but I feel there's two sides of this. You've got one side that is the blockchain, that I think some people are familiar with in the, I guess in the context of cryptocurrency, right? This is a very popular use of the blockchain because you have that redundancy and you have the agreement amongst multiple places, it's decentralized. And so you have that security there around that. But there's other uses for the blockchain as well.Especially things like banking and real estate and some of those other use cases that I'd like to talk about. And then there's another side of it that is this decentralized piece. Is the decentralized piece of it like building apps? How is that related to the blockchain or are those two separate things?Nader: Yeah, absolutely. I'm a big fan of Silicon Valley. Working in tech, it's almost like every single episode resonates with you if you've been in here long enough because you've been in one of those situations. The blockchain is part of the discussion. Crypto is part of the discussion, and those things never really interested me, to be honest. I was a speculator in crypto from 2015 until now. It's been fun, but I never really looked at crypto in any other way other than that. Blockchain had a really negative, I would say, association in my mind for a long time, I just never really saw any good things that people were doing with it. I just didn't do any research, maybe didn't understand what was going on.When I started diving into it originally what really got me interested is the Graph protocol, which is one of the things that we work on at Edge & Node. I started actually understanding, why does this thing exist? Why is it there? That led me to understanding why it was there and the fact that 90% of dApps, decentralized apps in the Ethereum ecosystem are using it. And billions of queries, companies with billions of dollars in transactions are all using this stuff. I'm like, okay, this whole world exists, but why does it exist? I guess to give you an example, I guess we can talk about the Graph protocol. And there are a lot of other web, I would say Web3 or decentralized infrastructure protocols that are out there that are similar, but they all are doing similar things in the sense of how they're actually built and how they allow participation and stuff like that.When you think of something like AWS, you think of, AWS has all of these different services. I want to build an app, I need storage. I need some type of authentication layer, maybe with Cognito, and then maybe I need someplace to execute some business logic. So maybe I'll spin up some serverless functions or create an EC2 instance, whatever. You have all these building blocks. Essentially what a lot of these decentralized protocols like the Graph are doing, are building out the same types of web infrastructure, but doing so in a decentralized way. Why does that even matter? Why is that important? Well, for instance, when you live, let's say for example in another country, I don't know, in South America and outside the United States, or even in the United States in the future, you never know. Let's say that you have some application and you've said something rude about maybe the president or something like that.Let's say that for whatever reason, somebody hacks the server that you're dealing with or whatever, at the end of the day, there is a single point of failure, right? You have your data that's controlled by the cloud provider or the government can come in and they can have control over that. The idea around some of, pretty much all of the decentralized protocols is that they are built and distributed in a way that there is no single point of failure, but there's also no single point of control. That's important when you're living in areas that have to even worry about stuff like that. So maybe we don't have to worry about that as much here, but in other countries, they might.Building something like a server is not a big deal, right? With AWS, but how would you build a server and make it available for anyone in the world to basically deploy and do so in a decentralized way? I think that's the problem that a lot of these protocols are trying to solve. For the Graph in particular, if you want to build an application using data that's stored on a blockchain. There's a lot of applications out there that are basically using the blockchain for mainly, right now it's for financial, transactional reasons because a lot of the transactions actually cost a lot of money. For instance, Uniswap is one of these applications. If you want to basically query data from a blockchain, it's not as easy as querying data from a traditional server or database.For us we are used to using something like DynamoDB, or some type of SQL database, that's very optimized for queries. But on the blockchain, you're basically having these blocks that add up every time. You create a transaction, you save it. And then someone comes behind them and they save another transaction. Over time you build up this data that's aggregated over time. But let's say you want to hit that database with the, quote-unquote, database with a query and you want to retrieve data over time, or you want to have some type of filtering mechanism. You can't do that. You can't just query blockchains the way you can from a regular database. Similar to how a database basically indexes data and stores it and makes it efficient for retrieval, the Graph protocol basically does that, but for blockchain data.Anyone that wants to build an application, one of these decentralized apps on top of blockchain data has a couple of options. They can either build their own indexing server and deploy it to somewhere like AWS. That takes away the whole idea of decentralization because then you have a single point of failure again. You can query data directly from the blockchain, from your client application, which takes a very long time. Both of those are not, I would say the most optimal way to build. But also if you're building your own indexing server, every time you want to come up with a new idea also, you have to think about the resources and time that go into it. Basically, I want to come up with a new idea and test it out, I have to basically build a server index, all this data, create APIs around it. It's time-intensive.What the Graph protocol allows you to do is, as a developer you can basically define a subgraph using YAML, similar to something like cloud formation or a very condensed version of that maybe more Serverless Framework where you're defining, I want to query data from this data source, and I want to save these entities and you deploy that to the network. And that subgraph will basically then go and look into that blockchain. And will look for all the transactions that have happened, and it will go ahead and save those and make those available for public retrieval. And also, again, one of the things that you might think of is, all of this data is public. All of the data that's on the blockchain is public.Jeremy: Right. Right. All right. Let me see if I could repeat what you said and you tell me if I'm right about this. Because this was one of those things where blockchain ... you're right. To me, it had a negative connotation. Why would you use the blockchain, unless you were building your own cryptocurrency? Right. That just seemed like that's what it was for. Then when AWS comes out with QLDB or they announced that or whatever it was. I'm like, okay, so this is interesting, but why would you use it, again, unless you're building your own cryptocurrency or something because that's the only thing I could think of you would use the blockchain for.But as you said, with these blockchains now, you have highly sensitive transactions that can be public, but a real estate transaction, for example, is something really interesting, where like, we still live in a world where if Bank of America or one of these other giant banks, JPMorgan Chase or something like that gets hacked, they could wipe out financial data. Right? And I know that's backed up in multiple regions and so forth, but this is the thing where if you're doing some transaction, that you want to make sure that transaction lives forever and isn't manipulated, then the blockchain is a good place to do that. But like you said, it's expensive to write there. But it's even harder to read off the blockchain because it's that ledger, right? It's just information coming in and coming in.So event storming or if you were doing event sourcing or something that, it's that idea. The idea with these indexers are these basically separate apps that run, and again, I'm assuming that these protocols, their software, and things that you don't have to build this yourself, essentially you can just deploy these things. Right? But this will read off of the blockchain and do that aggregation for you and then make that. Basically, it caches the blockchain. Right? And makes that available to you. And that you could deploy that to multiple indexers if you wanted to. Right? And then you would have access to that data across multiple providers.Nader: Right. No single point of failure. That's exactly right. You basically deploy a very concise configuration file that defines how you want your data stored and made available. And then it goes, and it just starts at the very beginning and it queries all those blocks or reads all those blocks, saves the data in a database, and then it keeps up with additional new updates. If someone writes a new transaction after that, it also saves that and makes it available for efficient retrieval. This is just for blockchain data. This is the data layer for, but it's not just a blockchain data in the future. You can also query from IPFS, which is a file storage layer, somewhat S3. You can query from other chains other than Ethereum, which is kind of like the main chamber.In the future really what we're hoping to have is a complete API on top of all public data. Anybody that wants to have some data set available can basically deploy a subgraph and index it and then anyone can then essentially query for it. It's like when you think of public data, we're not really used to thinking of data in this way. And also I think a good thing to talk about in a moment is the types of apps that you can build because you wouldn't want to store private messages on a blockchain or something like that. Right? The types of apps that people are building right now at least are not 100% in line with everything. You can't do everything I would say right now in Web3 that you can do in Web2.There are only certain types of applications, but those applications that are successful seem to be wildly successful and have a lot of people interested in them and using them. That's the general idea, is like you have this way to basically deploy APIs and the technology that we use to query is GraphQL. That was one of the reasons that I became interested as well. Right now the main data sources are blockchains like Ethereum, but in the future, we would like to make that available to other data sources as well.Jeremy: Right. You mentioned earlier too because there are apps obviously being built on this that you said are successful. And the problem though, I think right now, because I remember I speculated a little bit with Bitcoin and I bought a whole bunch of Ripple, so I'm still hanging on to it. Ripple XPR whatever, let's go. Anyways, but it was expensive to make a transaction. Right? Reading off of the blockchain itself, I think just connecting generally doesn't cost money, but if you're, and I know there's some costs with indexers and that's how that works. But in terms of the real cost, it's writing to the blockchain. I remember moving some Bitcoin at one point, I think cost me $30 to make one transaction, to move something like that.I can see if you're writing a $300,000 real estate transaction, or maybe some really large wire transfer or something that you want to record, something that makes sense where you could charge a fee of $30 or $40 in order to do that. I can't see you doing that for ... certainly not for web streaming or click tracking or something like that. That wouldn't make sense. But even for smaller things there might be writing more to it, $30 or whatever that would be ... seems quite expensive. What's the hope around that?Nader: That was one of the biggest challenges and that was one of the reasons that when I first, I would say maybe even considered this as a technology back in the day, that I would be considering as something that would possibly be usable for the types of applications I'm used to seeing. It just was like a no-brainer, like, no. I think right now, and that's one of the things that attracted me right now to some of the things that are happening, is a lot of those solutions are finally coming to fruition for fixing those sorts of things. There's two things that are happening right now that solve that problem. One of them is, they are merging in a couple of updates to the base layer, layer one, which would be considered something like Ethereum or Bitcoin. But Ethereum is the main one that a lot of the financial stuff that I see is happening.Basically, there are two different updates that are happening, I think the main one that will make this fee transactional price go down a little bit is sharding. Sharding is basically going to increase the number of, I believe nodes that are basically able to process the transactions by some number. Basically, that will reduce the cost somewhat, but I don't think it's ever going to get it down to a usable level. Instead what the solutions seem to be right now and one of the solutions that seems to actually be working, people are using it in production really recently, this really just started happening in the last couple of months, is these layer 2 solutions. There are a couple of different layer 2 solutions that are basically layers that run on top of the layer one, which would be something like Ethereum.And they treat Ethereum as the settlement layer. It's almost like when you interact with the bank and you're running your debit card. You're probably not talking to the bank directly and they are doing that. Instead, you have something like Visa who has this layer 2 on top of the banks that are managing thousands of transactions per second. And then they take all of those transactions and they settle those in an underlying layer. There's a couple different layer 2s that seem to be really working well right now in the Ethereum ecosystem. One of those is Arbitrum and then the other is I think Matic, but I think they have a different name now. Both of those seem to be working and they bring the cost of a transaction down to a fraction of a penny.You have, instead of paying $20 or $30 for a transaction, you're now paying almost nothing. But now that's still not cheap enough to probably treat a blockchain as a traditional database, a high throughput database, but it does open the door for a lot of other types of applications. The applications that you see building on layer one where the transactions really are $5 to $20 or $30 or typically higher value transactions. Things like governance, things like financial transactions, you've heard of NFTs. And that might make sense because if someone's going to spend a thousand bucks or 500 bucks, whatever ...Jeremy: NFTs don't make sense to me.Nader: They're not my thing either, the way they're being, I would say, talked about today especially, but I think in the future, the idea behind NFTs is interesting, but yeah, I'm in the same boat as you. But still to those people, if you're paying a thousand dollars for something then that 5 or 10 or 20 bucks might make sense, but it's not going to make sense if I just want to go to an e-commerce store and pay $5 for something. Right? I think that these layer 2s are starting to unlock those potential opportunities where people can start building these true financial applications that allow these transactions to happen at the same cost or actually a lot cheaper maybe than what you're paying for a credit card transaction, or even what those vendors, right? If you're running a store, you're paying percentages to those companies.The idea around decentralization comes back to this discussion of getting rid of the middleman, and a lot of times that means getting rid of the inefficiencies. If you can offload this business logic to some type of computer, then you've basically abstracted away a lot of inefficiencies. How many billions of dollars are spent every year by banks flying their people around the world and private jets and these skyscrapers and stuff. Now, where does that money come from? It comes from the consumer and them basically taking fees. They're taking money here and there. Right? That's the idea behind technology in general. They're like whenever something new and groundbreaking comes in, it's often unforeseen, but then you look back five years later and you're like, this is a no-brainer. Right?For instance Blockbuster and Netflix, there's a million of them. I don't have to go into that. I feel this is what that is for maybe the financial institutions and how we think about finance, especially in a global world. I think this was maybe even accelerated by COVID and stuff. If you want to build an application today, imagine limiting yourself to developers in your city. Unless you're maybe in San Francisco or New York, where that might still work. If I'm here in Mississippi and I want to build an application, I'm not going to just look for developers in a 30-mile radius. That is just insane. And I don't use that word mildly, it's just wild to think about that. You wouldn't do that.Instead, you want to look in your nation, but really you might want to look around the world because you now have things like Slack and Discord and all these asynchronous ways of doing work. And you might be able to find the best developer in the world for 25% or 50% of what you would typically find locally and an easy way to pay them might just be to just send them some crypto. Right? You don't have to go find out all their banking information and do all the wiring and all this other stuff. You just open your wallet, you send them the money and that's it. It's a done deal. But that's just one thing to think about. To me when I think about building apps in Web2 versus Web3, I don't think you're going to see the Facebook or Instagram use case anytime in the next year or two. I think the killer app for right now, it's going to be financial and e-commerce stuff.But I do think in maybe five years you will see someone crack that application for, something like a social media app where we're basically building something that we use today, but maybe in a better way. And that will be done using some off-chain storage solution. You're not going to be writing all these transactions again to a blockchain. You're going to have maybe a protocol like Graph that allows you to have a distributed database that is managed by one of these networks that you can write to. I think the ideas that we're talking about now are the things that really excite me anyway.Jeremy: Let's go back to GraphQL for a second, though. If you were going to build an app on top of this, and again, that's super exciting getting those transaction fees down, because I do feel every time you try to move money between banks or it's the $3 fee, if you go to a foreign ATM and you take money out of an ATM, they charge you. Everybody wants to take a cut somewhere along, and there's probably reasons for it, but also corporate jets cost money. So that makes sense as well. But in terms of the GraphQL protocol here, so if I wanted to build an application on top of it, and maybe my application doesn't write to the blockchain, it just reads from it, with one of these indexers, because maybe I'm summing up some financial transactions or something, or I've got an app we can look things up or whatever, I'm building something.I'm querying using the GraphQL, this makes sense. I have to use one of these indexers that's aggregating that data for me. But what if I did want to write to the blockchain, can I use GraphQL to do a mutation and actually write something to the blockchain? Or do I have to write to it directly?Nader: Yeah, that's actually a really, really good question. And that's one of the things that we are currently working on with the Graph. Right now if you want to write a transaction, you typically are going to be using one of these JSON RPC wallets and using some type of client library that interacts with the wallet and signs the transaction with the private key. And then that sends the transaction to the blockchain directly. And you're talking to the blockchain and you're just using something like the Graph to query. But I think what would be ideal and what we think would be ideal, is if someone could use a single technology, a single language, and a single abstraction to do everything, not only with reading and writing but also with subscriptions for real-time updates.That's where we think the whole idea for this will ultimately be, and that's what we're working on now. Right now you can only query. And if you want to write a transaction, you basically are still going to be using something like ethers.js or Web3 or one of these other libraries that allows you to sign a transaction using your wallet. But in the future and in fact, we're already building this right now as having an end-to-end GraphQL library that allows you to write transactions as well as read. That way someone just learns a single API and it's a lot easier. It would also make it easier for developers that are coming from a traditional web background to come in because there's a little bit of learning curve for understanding how to create one of these signed providers and write the transaction. It's not that much code, but it is a new way of thinking about things.Jeremy: Well I think both of us coming from the serverless space, we know that new way of thinking about things certainly can throw a wrench in the system when a new developer is trying to pick that stuff up.Nader: Yeah.Jeremy: All right. So that's the blockchain side of things with the data piece of it. I think people could wrap their head around that. I think it makes a lot of sense. But I'm still, the decentralized, the other things that you talked about. You mentioned an S3, something that's sort of an S3 type protocol that you can use. And what are some of the other ones? I think I've written some of them down here. Acash was one, Filecoin, Livepeer. These are all different protocols or services that are hosted by the indexers, or is this a different thing than the indexers? How does that work? And then how would you use that to save data, maybe save some blob, a blob storage or something like that?Nader: Let's talk about the tokenomics idea around how crypto fits into this and how it actually powers a protocol like this. And then we'll talk about some of those other protocols. How do people actually build all this stuff and do it for, are they getting paid for it? Is it free? How does that work and how does this network actually stay up? Because everything costs money, developers' time costs money, and so on and so forth. For something like the Graph, basically during the building phase of this protocol, basically, there was white papers and there was blog posts, and there was people in Discords talking about the ideas that were here. They basically had this idea to build this protocol. And this is a very typical life cycle, I would say.You have someone that comes up with an idea, they document some of it, they start building it. And the people that start building it are going to be basically part of essentially the founding team you could think of, in the sense of they're going to be having equity. Because at the end of the day, to actually launch one of these decentralized protocols, the way that crypto comes into it, there's typically some type of a token offering. The tokens need to be for a network like this, some type of utility token to keep the network running in the future. You're not just going to create some crypto and that's it like, right? I think that's the whole idea that I thought was going on when in reality, these tokens are typically used for powering the protocol.But let's say early on you have let's say 20 developers and they all build 5% of the system, whatever percentage that you want to talk about, whatever. Let's say you have these people helping out and then you actually build the thing and you want to go ahead and launch it and you have something that's working. A lot of times what people will do is they'll basically have a token offering, where they'll basically say, okay, let's go ahead and we're going to mint X number of tokens, and we're going to put these on the market and we're going to also pay these people that helped build this system, X number of tokens, and that's going to be their payment. And then they can go and sell those or keep those or trade those or whatever they would like to do.And then you have the tokens that are then put on the public market essentially. Once you've launched the protocol, you have to have tokens to basically continue to power the protocol and fund it. There are different people that interact with the protocol in different ways. You have the indexers themselves, which are basically software engineers that are deploying whatever infrastructure to something like AWS or GCP. These people are still using these cloud providers or they're maybe doing it at their house, whatever. All you basically need is a server and you want to basically run this indexer node, which is software that is open source, and you run this node. Basically, you can go ahead and say, okay, I want to start being an indexer and I want to be one of the different nodes on the network.To do that you basically buy some GRT, Graph Token, and in our case you stake it, meaning you are putting this money up to basically affirm that you are an indexer on the protocol and you are going to be accepting subgraph developers to deploy their subgraphs to your indexer. You stake that money and then when people use the API, they're basically paying money just like they might pay money to somewhere like API gateway or AppSync. Instead, they're paying money for their subgraph and that money is paid in GRT and it's distributed to the people in the ecosystem. Like me as a developer, I'm deploying the subgraph, and then if I have a million people using it, then I make some money. That's one way to use tokens in the system.Another way is basically to, as an outside person looking in, I can say, this indexer is really, really good. They know what they're doing. They're a very strong engineer. I'm going to basically put some money into their indexer and I'm basically backing them as an indexer. And then I will also share the money that comes in from the query fees. And then there are also people that are subgraph developers, which is the stuff that I've been working with mainly, where I can basically come up with a new API. I can be like, it'd be cool if I took data from this blockchain and this file system and merged it together, and I made this really cool API that people can use to build their apps with. I can deploy that. And basically, people can signal to this subgraph using tokens. And when people do that, they can say that they believe that this is a good subgraph to use.And then when people use that, I can also make money in that way. Basically, people are using tokens to be part of the system itself, but also to use that. If I'm a front end application like Uniswap and I want to basically use the Graph, I can basically say, okay, I'm going to put a thousand dollars in GRT tokens and I'm going to be using this API endpoint, which is a subgraph. And then all of the money that I have put up as someone that's using this, is going to be taken as the people start using it. Let's say I have a million queries and each query is one, 1000th of a cent, then after those million queries are up, I've spent $100 or something like that. Kind of similar to how you might pay AWS, you're now paying, you know, subgraph developers and indexers.Jeremy: Right. Okay. That makes sense. So then that's the payment method of that. So then these other protocols that get built on top of it, the Acash and Filecoin and Livepeer. So those ...Nader: They're all operating in a very similar fashion.Jeremy: Okay. All right. And so it's ...Nader: They have some type of node software that's run and people can basically run this node on some server somewhere and make it available as part of the network. And then they can use the tokens to participate. There's Filecoin for file storage. There's also IPFS, which is actually more of, it's a completely free service, but it's also not something that's as reliable as something like S3 or Filecoin. And then you have, like you mentioned, I believe Acash, which is a way to execute arbitrary code, business logic, and stuff like that. You have Ceramic Network, which is something that you can use for authentication. You have Livepeer which is something you use for live streaming. So you have all these ideas, these decentralized services fitting in these different niches.Jeremy: Right, right. Okay. So then now you've got a bunch of people. Now you mentioned this idea of, you could say, this is a good indexer. What about bad indexers? Right?Nader: That's a really good question.Jeremy: Yeah. You're relying on people to take data off of a public blockchain, and then you're relying on them to process it correctly and give you back good data. I'm assuming they could manipulate that data if they wanted to. I don't know why, but let's say they did. Is there a way to guarantee that you're getting the correct data?Nader: Yeah. That's a whole part of how the system works. There's this whole idea and this whole, really, really deep rabbit hole of crypto-economics and how these protocols are structured to incentivize and also disincentivize. In our protocol, basically, you have this idea of slashing and this is also a fairly known and used thing in the ecosystem and in the space. It's this idea of slashing. Basically, you incentivize people to go out and find people that are serving incorrect data. And if that person finds someone that's serving incorrect data, then the person that's serving the incorrect data is, quote-unquote, slashed. And that basically means that they're not only not going to receive the money from the queries that they were serving, but they also might lose the money that they put up to be a part of the network.I mentioned you have to actually put up money to deploy an indexer to the network, that money could also be at risk. You're very, very, very much so financially disincentivized to do that. And there's actually, again, incentives in the network for people to go and find those people. It's all-around incentives, game theory, and things like that.Jeremy: Which makes a ton of sense. That's good to know. You mentioned, you threw out the number, five years from now, somebody might build the killer app or whatever, they'll figure out some of these things. Where are we with this though? Because this sounds really early, right? There's still things that need to be figured out. Again, it's public data on the blockchain. How do you see this evolving? When do you think Web3 will be more accessible to the masses?Nader: Today people are actually building really, really interesting applications that are fitting the current technology stack, what are the things that you can build? People are already building those. But when you think about the current state of the web, where you have something like Twitter, or Facebook or Instagram, where I would say, especially maybe something like Facebook, that's extremely, extremely complex with a lot of UI interaction, a lot of private data, messages and stuff. I think to build something like that, yeah, it's going to be a couple of years. And then you might not even see certain types of applications being built. I don't think there is going to be this thing where there is no longer these types of applications. There are only these new types. I think it's more of a new type of application that people are going to be building, and it's not going to be a winner takes all just like in all tech in my opinion.I wouldn't say all but in many areas of tech where you're thinking of something as a zero-sum game where I don't think this is. But I do think that the most interesting stuff is around how Web3 essentially enables native payments and how people are going to use these native payments in interesting ways that maybe we haven't thought of yet. One of the ways that you're starting to see people doing, and a lot of venture capitalists are now investing in a lot of these companies, if you look at a lot of the companies coming out of YC and a lot of the new companies that these traditional venture capitalists are investing in, are a lot of TOMS crypto companies.When you think about the financial incentives, the things that we talked about early on, let's say you want to have the next version of YouTube and you don't want to have ads. How would that even work? Right? You still need to enable payments. But there's a couple of things that could happen there. Well, first of all, if you're building an application in the way that I've talked about, where you basically have these native payments or these native tokens that can be part of the whole process now, instead of waiting 10 years to do an IPO for an application that has been around for those 10 years and then paying back all his investors and all of those people that had been basically pulling money out their pockets to take part in.What if someone that has a really interesting idea and maybe they have a really good track record, they come out with a new application and they're basically saying, okay, if you want to own a piece of this, we're going to basically create a token and you can have ownership in it. You might see people doing these ICO's, initial coin offerings, or whatever, where basically they're offering portions of the company to anyone that wants to own it and then incentivizing people to basically use those, to govern how the application is built in the future. Let's say I own 1% of this company and a proposal is put up to do something new. I can basically say, I can use that portion of my ownership to vote on things. And then people that are speculating can say, this company is doing interesting things. I'm going to buy into it, therefore driving the price up or down.Kind of like the same way that you see the traditional stock market there, but without all of the regulation and friction that comes with that. I think that's interesting and you're already seeing companies doing that. You're not seeing the majority of companies doing that or anything like that, but you are starting to see those types of things happening. And that brings around the discussion of regulations. Is ... can you even do something like that in the United States? Well, maybe, maybe not. Does that mean people are going to start building these companies elsewhere? That's an interesting discussion as well. Right now if you want to build an application this way, you need to have some type of utility that these tokens are there for. You can't just do them purely on speculation, at least right now. But I think it's going to be interesting for sure, to watch.Jeremy: Right. And I think too that, I'm just thinking if you're a bank, right? And you maybe have a bunch of private transactions that you want to keep private. Because again, I don't even know how, I don't know how we get to private transactions on the blockchain. I could see you wanting to have some transactions that were public blockchain and some that were private and maybe a hybrid approach would make sense for some companies.Nader: I think the idea that we haven't really talked about at all is identity and how identity works compared to how we're used to identity. The way that we're used to identity working is, we basically go to a new website and we're like, this looks awesome. Let me try it out. And they're like, oh wait, we need your name, your email address, your phone number, and possibly your credit card and all this other stuff. We do that over and over and over, and over time we've now given our personal information to 500 people. And then you start getting these emails, your data has been breached, every week you get one of these emails, if you're someone like me, I don't know. Maybe I'm just signing up for too much stuff. Maybe not every week, but maybe every month or two. But you're giving out your personal data.But we're used to identity as being tied to our own physical name and address and things like that. But what if identity was something that was more abstract? And I think that that's the way that you typically see identity managed in Web3. When you're dealing with authentication mechanisms, one of the most interesting things that I think that is part of this whole discussion is this idea of a single sign-on mechanism, that you own your identity and you can transfer it across all the applications and no one else is in control of it. When you use something like an Ethereum wallet, like MetaMask, for example, it's an extension you can just download and put crypto in and basically make payments on the web with. When you create a wallet, you're given a wallet address. And the wallet address is basically created using public key cryptography, where basically you start with this private key, your public key is derived from the private key, and then your address is dropped from the public key.And when you send a transaction, you basically sign the transaction with your private key and you send your public key along with the transaction, and the person that receives that can decode the transaction with the public key to verify that that's who signed the transaction. Using this public key cryptography that only you can basically sign with your own address and your own password, it's all stored on the blockchain or in some decentralized manner. Actually in this case stored on the blockchain or it depends on how you use it really, I guess. But anyway, the whole idea here is that you completely own your identity. If you never decide to associate that identity with your name and your phone number, then who knows who's sending these transactions and who knows what's going on, because why would you need to associate your own name and phone number with all of these types of things, in these situations where you're making payments and stuff like that. Right?What is the idea of a user profile anyway, and why do you actually need it? Well, you might need it on certain applications. You might need it or want it on social network, or maybe not, or you might come up with a pseudonym, because maybe you don't want to associate yourself with whatever. You might want to in other cases, but that's completely up to you and you can have multiple wallet addresses. You might have a public wallet address that you associate your name with that you are using on social media. You might have a private wallet address that you're never associating with your name, that you're using for financial transactions. It's completely up to you, but no one can change that information. One of the applications that I recently built was called Decentralized Identity. I built it and release it a few days ago.And it's an implementation of this and it's using some of these Web3 technologies. One of them is IDX. One of them is Ceramic, which is a decentralized protocol similar to the Graph but for identity. And then it's using something called DIDs, which are decentralized identifiers, which are a way to have a completely unique ID based off of your address. And then you own the control over that. You can basically go in and make updates to that profile. And then any application across the web that you choose to use can then access that information. You're only dealing with it stored in one place. You have full control over it, at any time you can go in and delete that. You can go in and change it. No one has control over it except for you.The idea of identity is a mind-bending thing in this space because I think we're so used to just handing everybody our real names and our real phone numbers and all of our personal information and just having our fingers crossed, that we're just not used to anything else.Jeremy: It's all super interesting. You mentioned earlier about, would it be legal in the United States? I'm thinking of all these recent ransomware attacks and I think they were able to trace back some Bitcoin transaction, they were actually able to trace it back to the individual group that accepted the payment. It opens up a whole can of worms. I love this idea of being anonymous and not being tracked, but then it's also like, what could bad actors do with anonymous financial transactions and things like that? So ...Nader: There kind of has been anonymous transactional layer for a long time. Cash brought in, you can't really do a lot of illegal stuff these days without cash. So should we get rid of cash? I think with any technology ...Jeremy: No, but I mean, there's a limit though, right? You can't withdraw more than $10,000 worth of cash without the FBI being flagged and you can't deposit more, you know what I mean?Nader: You can't take a million dollars worth of Bitcoin that you've gotten from ransomware and turn it into cash either.Jeremy: That's also true. Right.Nader: Because it's all tracked on the blockchain, that's probably how they caught those people. Right? They somehow had their personal information tied to a transaction, because if you follow these transactions long enough, you're going to find some origination point. I agree though. There's definitely trade-offs with everything. I don't think I'm ever the type to argue that. There's good things and there's bad things. I think you have to look at the whole picture and decide for yourself, what you think. I'm the type that's like, let's lay out all of the ideas and let the market decide.Jeremy: Right. Yeah. I totally agree with that. All this stuff is fascinating, there is way too much more for me to learn at this point. I think my brain is filled at this point. Anything else about Edge & Node? Any cool things you're working on there or anything you want people to know?Nader: We're working on a couple of different projects. I can't really talk about some of them because they're not released yet, but we are working on a new version of something called Everest, and Everest is already out. If you want to check it out, it's at everest.link. It's basically a repository of a bunch of different applications that have already been built in the Web3 ecosystem. It also ties in a lot of the stuff that we talked about, like identity and stuff like that. You can basically sign in with your Ethereum wallet. You can basically interact with different applications and stuff, but you can also just see the types of stuff people are building. It's categorized into games, financial apps. If you've listened to this and you're like, this sounds cool, but are people actually building stuff? This is a place to see hundreds of apps that people have are already built and that are out there and successful.Jeremy: Awesome. All right. Well, listen, Nader, this was awesome. Thank you so much for sharing this with me. I know I learned a ton. I hope the listeners learned a ton. If people want to learn more about this or just follow you and keep up with what you're doing, what's the best way to do that?Nader: I would say check out Twitter, we're on Twitter @dabit3 for me, @edgeandnode for Edge & Node, and of course @graphprotocol for Graph protocol.Jeremy: Okay. And then edgeandnode.com. Your YouTube channel is just youtube.com/naderdabit, N-A-D-E-R D-A-B-I-T. And then you had an article on Web3 and I'll put it in the show notes.Nader: Yeah. Put it in the show notes. For freeCodeCamp, it's called what is Web3. And it's really a condensed version of a lot of the stuff we talked about. Maybe go into a little bit more depth around native payments and how people might build companies in the way that we've talked about here.Jeremy: Awesome. All right. Well, I will get all that stuff into the show notes. Thanks again, Nader.Nader: Thanks for having me. It was good to talk.
Silicon Valley is a bit quieter these days and big tech workers are looking for greener pastures. At the beginning of the pandemic, the tech workforce - and a significant percentage of the world's population - were asked to work from home. Nabis, a leading cannabis distribution platform in California, realizes that these employees are now both comfortable and productive while working remotely. Seizing the moment, Nabis is actively taking advantage of this shift in dynamics and recruiting tech workers to come to the cannabis industry. Vince Ning, CEO of Nabis, left his position as a software engineer with Microsoft to start his own business, an app development platform that he eventually sold and is now Amazon's AppSync. Following this Ning, and co-founder Jun S. Lee started Nabis in 2017 with the vision “To empower the world to discover cannabis by providing choice, access, and innovation.” https://www.cannabistech.com/webcasts/a-tech-moment/taking-advantage-of-the-big-tech-exodus/ --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app
About AviadAviad Mor is the Co-Founder & CTO at Lumigo. Lumigo’s SaaS platform helps companies monitor and troubleshoot serverless applications while providing actionable insights that prevent business disruptions. Aviad has over a decade of experience in technology leadership, heading the development of core products in Check Point from inception to wide adoption.Links:Lumigo: https://lumigo.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: This episode is sponsored in part by our friends at Lumigo. If you’ve built anything from serverless, you know that if there’s one thing that can be said universally about these applications, it’s that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications.It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You’ve created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I periodically talk about how I bolt together a whole bunch of different serverless tools in horrifying ways to write my newsletter every week. At last count, I was up to something like four API Gateways, twenty-nine Lambda functions, and counting. How do I figure out if something’s broken in there? Well, honestly, I just keep clicking the button until it works, which is a depressingly honest story.Now, that doesn’t work for everyone. Today’s promoted episode is brought to us by Lumigo. And my guest today is Aviad Mor, their CTO, and co-founder. Aviad, thanks for taking the time to suffer my slings and arrows.Aviad: Thank you, Corey. I’m very glad to be here today.Corey: So, let’s begin at, I guess, the very easy, high-level question: what is Lumigo and is ‘loom-ago’ an accepted alternate pronunciation?Aviad: [laugh]. So, Lumigo is a monitoring and debugging platform for serverless environments. And yes, you can call it whatever you want as long as it’s Lu-mi-go. What we do is we integrate with the customer’s AWS account, we do a very quick connection to its Lambdas, and then we’re able to show him exactly what’s going on in his system: what’s going well, what’s going wrong, and how we can fix it.Corey: So, let’s make sure that we hit a few points here at the beginning. It is AWS specific at this time?Aviad: Yes, it is. We’re not officially exclusive with AWS, but right now we see the most interesting serverless environments in AWS, so it’s a pretty easy call. But we are keeping our eye open to, you know, Google, Microsoft, even Oracle.Corey: Oh, Oracle Cloud has some phenomenally interesting serverless stories that I don’t think the world is paying enough attention to yet. But one of these days, I’m hoping that that’s going to change just because they have so much savvy locked up in that platform.Aviad: Right. They do have serverless functions. Yeah, so.Corey: They acquired the iron.io folks a while back, and those people were way ahead of Lambda at the time.Aviad: Right, right. So, we’re waiting for the big breakout of serverless in Oracle, and then we’ll build the best monitoring solution for them.Corey: So, something else, I think, that you have successfully navigated as far as, I guess, the traps that various observability tooling falls into, you also talk on your site about monitoring AWS Lambda as the center around which everything winds up being captured. You also, of course, integrate with the things that tied directly into it, such as API Gateway—or ‘opi-gateway,’ as I’m sure they mispronounce it at AWS—but that’s sort of where you stop. You don’t also show all of the container workloads that folks are running, and, “Oh, hey. While we have access to your API, here’s a whole story about ECS, and RDS, and all the rest.” And eventually, it feels like everything, in the fullness of time, tries to become Datadog version two.And that always drove me nuts because I want something that’s going to talk to me specifically about what it is that I’m looking at in a serverless application context, not trying to be all things to all people at once. Is that a fair assessment of the product strategy you folks have pursued?Aviad: Right. So, we’re very focused on serverless. We think there’s a lot of interesting things that we can do there, and we’re actually seeing more and more use cases of serverless. And it is important to say that when we say serverless, it’s very clear what is serverless. So Lambda, of course, and API Gateway, DynamoDB, S3, and so on.There’s a lot of services in data ecosystem, and seeing them all being tied together in a serverless cloud application, we’re able to do all of that to monitor it; not only monitor it at the high level, but also get into the details and show you things which are very specific because this is what we do all day, and sometimes all night. And then there are those boundaries of where do we go beyond serverless. So, there are some hybrid environments out there. And when I say ‘hybrid,’ the easy hybrid, which is you have two different applications which just happen to be on the same AWS account; one of them is completely serverless, and then the other one is EC2. So, that’s kind of hybrid.But the more interesting hybrid is those applications which start with an API Gateway in the Lambda, and then are directly connected to something else, which is maybe Fargate, ECS, EKS, and so on. So, we are very much focused on serverless, but we are getting also a lot of requests from our customers, “So, show us, also, the other parts.” We’re starting to look at that, but we’re not losing our focus. Our focus is still very much on the serverless while allowing you to tie together if you do have some other aspects in your environment to see them all together.Corey: So, you’ve done a number of things that I would consider best in class as you’ve gone through this. First and foremost, let’s begin with the easy stuff. It doesn’t appear that your dashboard, your tooling itself, is purely serverless itself. I can tell this because when I click around in your site, the site loads super quickly. It’s not waiting for cold starts or waiting for the latency inherent to the Lambda.It’s clear that you have not gone so far down the path of being, I guess, religiously correct around everything must be serverless all the times in favor of improving customer experience. That’s something that I’ve seen a number of different vendors fall into the trap of, of, “Why is the dashboard so slow to load?” “Ah, because everything is itself a Lambda function.” Is that accurate, or if you just found a way to improve Lambda [laugh] function performance in an ungodly way?Aviad: [laugh]. We are serverless—we call ourselves serverless first, but the customer is always—he’s really the first. So, if there’s a place where serverless is not the best solution, we’re going to use whatever is the best solution. But the truth is, we’re, I’d say, something like 99% serverless. And specifically, anything which is dashboard-facing customer-facing, that’s actually completely serverless.So, we did have to put in a lot of work, but also, I have to say that AWS went a very long way, like, in the last two years, allowing us to give much better latencies in different parts of the dashboard. So, all of that is serverless, and it goes together with the new features of Lambdas, and API Gateways, and a lot of small things we had to do in order to provide the best experience to the customer.Corey: The next thing that I think was interesting, as far as, I guess, capturing the way in which people use these things. One of the earliest problems I had, in the early days of these, I guess, new breed of serverless tools was getting everything instrumented correctly. It felt like it took in some cases more time to get the observability pieces working than it did to write the thing in the first place. So, you’re integrating out of the gate with a lot of the right things as best I can tell. Your website advertises that you integrate with the Serverless Framework, you integrate with a bunch of other [processes 00:07:52] as well. Chalice, which I haven’t seen used in anger too much, but okay; Terraform, which everyone’s using; Stackery, et cetera. Is AWS’s SAM on the list as well?Aviad: Yes, it actually is. And once we started seeing more and more users using SAM, we had to provide a way to allow them to easily do the integration. Because one of the things that we learned is, no, our users are developers and, just like you said, they don’t want to spend time on anything, which is not like doing the thing that they want to do. Especially in serverless, because the whole serverless premise is work on what you do best, and don’t spend time on everything else. So, we actually spend a lot of time ourselves in order to make the integrations as easy as possible, as quickly as possible, and that also means that working with a lot of different tools to fit all the different environments our users are using out there.Corey: It looks like you’re doing this through the application—judiciously—of a bunch of custom layers. In other words, whatever you wind up using winds up being built as an underpinning of the existing Lambda functions, so it’s super easy to wind up grabbing the necessary dependencies for anything that you folks support without having to go back and refactor existing applications. Is that directionally correct?Aviad: Right. That’s correct. We’re using layers in order to, on one hand, do this deep integration we do with the Lambda, allowing us to do different instrumentations, collecting the data that’s being passed into the Lambda, being passed out of the Lambda, on one hand. But on the other hand, so the developer doesn’t have to make any code changes, and he can do whatever changes he wants to do. Doesn’t have to think about Lumigo at any point, and serverless layer does everything for him automatically.Corey: How do you handle support of the Lambda@Edge functions, which seem an awful lot like regular Lambda functions, except they’re worse in nearly every single way, every time I’ve tried to use them? In fact, in my experience, the best practice has been to immediately rip out Lambda@Edge and replace it with something else. Recently, it was formally disclosed that they only ran in a subset of 13 regional cache locations, and they still took a full CloudFront distribution update cycle every time you did a deployment, which dramatically slowed everything down for deploying it; they were massively expensive to run at significant scale, and they would log to whatever region was closest so it was a constant game of whack a mole to figure out what was going on. But, you know, other than that, they were great. How do you approach those?Aviad: Lambda@Edge are not very easy to use, and they’re, like, let’s say they’re full of surprises [laugh] because not everything they do is exactly what you find in the documentation. But again, since our users are using them, we had to make sure that we give them proper support. And giving them the proper support—other than running and collecting the data—is things that you mentioned, like the fact that it will log to the specific region it’s running in, so you have to go and collect all this data from different places, and you don’t really know exactly where it’s going to run. So, the main thing here is just to make things easy. It’s a bit of a mess when you’re looking at it directly, and taking all the information, putting it in one place so you as a user can just go ahead and read it and you don’t care where it’s running and what it’s doing, that was the main challenge which we worked on and added to the product.Corey: So, across the board, it seems like you folks have been evolving in lockstep with the underlying platform itself. Have you had time to evaluate their new CloudFront Functions, I believe is what they’re calling it. Or is it CloudFront Workers? I can never quite keep it straight; between all the different providers, all the words start to sound alike. But the thing that only runs for a millisecond or two, only in JavaScript, only in the actual all the CloudFront edge locations, et cetera, et cetera. Rather than fixing Lambda@Edge, they decided to roll something completely different out, and I haven’t looked at anything approaching the observability story yet because I’m still too angry about it.Aviad: [laugh]. Right. So, there’s a lot of things coming out, and we’re also very close partners with AWS, so in many cases, we’re actually beta users of new services or new functionality in Lambda. And one of the hardest parts is—and then we cannot spend all our time checking everything new. So, this is one of the things which is still in the to-do list; we’re going to check it very close, in a very close time.I think it’s interesting to see how we can actually use it and is it as quickly as they say. What they say usually works; we’ll see if it works already today, or do we have to wait a little bit until it works exactly like they said. But no, that’s one of the things that are on my to-do list. I’m really looking forward to checking it out.Corey: So, it looks like once I set this up and it starts monitoring my account—or observing my account. I know I know, observability is just hipster monitoring, but no one agrees with me on that, so I’m going to keep rolling with it anyway just to irritate people—it looks like I can effectively more-or-less click a button, and suddenly, you will roll out an underlying Lambda layer to all of my existing Lambda functions. How does that get maintained whenever I wind up, for example, doing a new deployment with the serverless framework or something like it that isn’t aware of that underlying layer, so it—presumably—would revert that layer itself in the definition? Or am I misunderstanding how that works?Aviad: No, no. You’re actually getting it right. So, unless you, for example, are using a serverless plugin, so this is an integral part of your deployment, one of the things that we need to do is to automatically identify that a deployment is happening so we can automatically update the Lambda layer to be the right one, so you won’t miss anything. And this deep integration, which is happening without the user having to know anything about it, this is, I think, one of the most important parts because in serverless, as you know, you have so many components, and very easily you can reach, you know, hundreds of Lambdas, which are things that we’re seeing. So, if a user has to take care and maintain something across a hundred Lambdas or more, you can be sure that it won’t be maintained because he has, like, something much more important to do.So, behind the scenes, immediately as the deployment is happening, we can recognize that it’s happening, and then update the layer that’s required. And by the way, now the layers have a new part called extensions, which allow us and everybody else to do a lot more with those layers, basically allowing the code to run in parallel to the Lambda. So, this is a new thing that AWS has started to roll out, and we think will allow us to give even better experience to our users.Corey: Let’s have a look across the, I guess, the ecosystem of have different approaches to this stuff. One thing that has always annoyed me about a whole raft of observability and monitoring tools is they wind up charging me whatever it is they charge me; it’s generally fine—and I don’t really have a problem with that. You know, in advance going in what things are going to cost you. Incidentally, what is your pricing model?Aviad: So, our pricing model is according to the number of invocations you have. So, we have basically two models which we’re using right now, and each one can decide what he wants better. So, if you want to know in advance exactly how much you’re going to pay, you can go with the tiered model meaning, I want to pay for, let’s say, a million invocations each month, and then you’re sure that you’re paying exactly for what you have a budget for. And it’s always related to how much your AWS account is working, so similar to how much you’re paying for your Lambdas. And then there’s another way, which is dynamic pricing, which is very similar to serverless payment.So, it’s really according to the number of invocations you have; you don’t need to decide in advance, and at the end of each month, according to the number of invocations you have, you get the bill. And that way it’s not based on the invocation in general; it’s exactly according to the number of invocations.Corey: And let’s be clear, if I wind up exceeding the number of invocations under my plan, it just stops tracing and observing these things, it doesn’t break my app.Aviad: Yeah, right. [laugh].Corey: Always good to triple-check those things. It seems like that might hurt.Aviad: That’s very important. You’re totally correct. And, yeah, we never do anything bad to your Lambdas. That’s written on the top of our door: “Never hurt a Lambda.” And we make sure that nothing bad happens, we just stopped collecting data.And by the way, even as you pass your limit, we still collect the basic metrics so you can see what’s going on in your system. But you won’t be able to see the rich information, all the information that allowing you to do the debugging, or seeing the full traceability end-to-end of all the invocations and see how they’re connected to each other.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: So, the pricing makes perfect sense, and that is in line with what I would expect, but the thing that irritates me then is, “Great. I know what I’m going to be paying you folks on a monthly basis, and that’s fine.” And then I use the monitoring tool and it cost me over three times as much in AWS charges, both direct and indirect, where it’s, “Oh, now CloudWatch is going to suddenly be the largest component of my bill and data transfer for sending everything externally winds up spiking into the stratosphere.” What’s your experience been around that?Aviad: So, since we are collecting data and we are doing API calls, it will affect your AWS bill. But because we don’t want to irritate you, or anybody else, we are putting a lot of focus to see that we’re doing the absolute minimal possible effect on your system. So, for example, as we’re collecting data from your Lambda, we’re doing our best to add only milliseconds to the running time of your Lambda so you don’t end up paying a lot more for the runtime. Or for the API calls or data transfer, we have a lot of optimizations that we did, so the billing on your AWS account is really very, very small; it’s not something that you will notice. And sometimes when people do ask us, we go together with them into their account and show them exactly how their billing was affected by Lumigo so they’ll have assurance that nothing crazy is going on there.Corey: Which is I guess one of the fundamental problems of the underlying platform itself. I have a hard time blaming you for any of this stuff. This is the perpetual joyless story of getting to work with a variety of AWS services. It’s not something that I see that you folks have a good way around just on basis of how the underlying platform works.Aviad: Yeah. And then there are a lot of different prices for a lot of small things that you do, and you need to be able to collect it all in order to have the big picture of the effect. And yeah, we don’t have a silver bullet for it, but we can show exactly where we’re going, what we’re adding, to show how low it is.Corey: One of the things that I think is not well understood for folks who are not into the serverless ecosystem is just how these applications tend to look. In most organic environments, you’ll see a whole bunch of Lambda functions that are all tied together with basically spit and baling wire. They talk to each other, either directly on the back end—which is an anti-pattern in many respects, let’s not kid ourselves—or alternately, they’re approaching through a lens of, we’re going to now talk to each other through hardened REST APIs, which is generally preferred, but also a little finicky to get up and running. So, at some point, you have a request come in, and it winds up bouncing around through a whole bunch of different subsystems. Tracing, and a lot of the observability story around serverless is figuring out, all right, somewhere in that rat’s nest, it winds up failing.Where did it break? What was it that actually threw the exception? What was it that prevented something from working? Or alternately, adding latency: where is the bulk of the time serving that request being spent? And you would think that this is the sort of thing that AWS could handle itself.And they’ve tried with their X-Ray distributed tracing option, which more or less feels like a proof of concept demonstrating what not to do. And if you take a look from their application view, and all the rest, it is the best sales pitch I can possibly imagine for any of the serverless monitoring tools that I’ve seen because it is so badly articulated. You have to instrument all of your stuff by hand. There’s none of this, oh, I’ll look at it and figure out what it’s talking to and build an automated trace approach, the way that Lumigo does. And that’s always frustrated me because I shouldn’t have to turn every weird analysis into a murder mystery. Am I missing something obvious in how this could be done using native tools directly, or is it really as bad as I believe it is?Aviad: [laugh]. I won’t say it as bad as you’re saying it is. I think X-Ray is a great place to start with. So, if you have, like, just a few Lambdas; you’re starting to check out the serverless world, X-Ray can be good enough if you don’t want to start with a third-party tool right at the beginning. But then as it gets a little bit complex, it’s going to get hard, especially if you’re trying to do it yourself.That’s usually the wow part when people start using Lumigo when we show them a demo, is seeing how everything is tied together. So, once you see how everything is tied together: the whole system, which components are talking to each other, and how they’re affecting each other. And for example, if one of them goes down, does it mean that the whole system now is not working, or maybe, eh, wasn’t that important, and everything is working. I’ll fix it next week. But I think the most important part is actually what we call the transactions.So, as you said, there’s an API call at the very beginning with an API Gateway or AppSync, and then it can go through dozens of components. Some of them are not directly related, so it’s like, Lambda calling, putting something into a DynamoDB, which triggers a DynamoDB stream. And then another Lambda is being called, and so on, and so on. It’s crucial to be able to see how everything is connected, both very visually, so you can understand it. There’s only so much you can understand when looking at a list as a human being, right?You need to see it visually how everything is connected. But then after you understand how everything is connected in this specific transaction, if, for example, you have an issue in a specific invocation, you need to understand the story of that invocation. And maybe you’re looking at a Lambda which starts to throw an exception, and you didn’t change anything in its code today, yesterday, or the day before that, so take care of that exception, but the root cause is probably not in that Lambda, it’s probably upstream. So, you need to be able to understand exactly what was the chain of events, all the calls being made until that specific Lambda was called to see the data being passed, including the data that Lambda maybe passed to a third-party API—like Stripe or PayPal—and what it got in return. And only when you’re able to see all of that you’re able to solve an issue quickly, not a murder mystery like it might be. Time over time without having to think about how will I make sure that I make all the code changes in order to keep getting these transactions?Corey: So, taking a look at the somewhat crowded space—if I’m being perfectly honest with you—of the entirety of, let’s call it the serverless observability space—or ‘observerless,’ as I’m a big fan of calling it—what is it the differentiates Lumigo from a number of other offerings that people could wind up pulling out of the hat?Aviad: Right. So, that’s a great question. And every time somebody asks me, the first thing I can say is, the more I see people getting into this space, I think that that’s a great sign. Because that means there’s more serverless activity, there’s more companies doing serverless and it means that our serverless space is interesting. People see an opportunity there, and they want to try and solve the issues that we’re seeing there.And I think that there’s a few things: one of them is the serverless expertise. So, if you look at a lot of the big companies—like I’ll mention Datadog and New Relic—they’re doing a lot of great things, but in the end, in the serverless environment, there are very specific things which you need to know, have to do in order to be able to do that distributed tracing, the distributed tracing which allows you to correlate specific transactions together and then bring in those metrics which are relevant and bring in the logs which are relevant for a specific transaction. That’s a lot of hard work which we put in in order to be able to do the transactions with a distributed tracing in the best way possible, and then showing it to you in the simplest way possible. And today, I think that Lumigo does that in a very good way. And also, if we’re looking around at other players, which are not only the big ones, also players, which are doing more specifically serverless, I think that if we’re looking at companies which are very focused on serverless, and serverless is the thing that they do, you’ll still see that Lumigo is the one which is doing serverless the most, let’s call it.So, as serverless is expanding, we’re still not becoming generic—something that we mentioned before—and this allows us not only to do the best distributed tracing but also allow us to show you, out of the box, a lot of issues which might be hiding in your environment. So, it’s not only, “Okay, you have an exception here,” it’s also more specific things to serverless. Like for example, because it’s event-driven, so sometimes you’ll get duplicate events that Kinesis or SQL might send you over and over the same event. The fact that we can show you it automatically and put a spotlight on it can save you a lot of time in trying to understand why things are not working the way you think they should be working. And allowing us to scan your environment and show you misconfigurations which are specific to serverless, this is the kind of things that once you use Lumigo, you get automatically without having to do anything special and that can save you a lot of time.Corey: I think that’s a relatively astute position to take. I’m a big believer in getting the right tool for the right job. I don’t necessarily want the one single pane of glass to look at everything. I think that is something that is very often misunderstood. Yeah, I might be using three or four different clouds in different ways.I don’t need to see a roundup of all of them; I don’t necessarily care what the billing looks like on all of them; I don’t necessarily want to spend my time thinking about different aspects of these things juxtaposed to one another, and it’s a pain in the butt to have to sort through to find the thing I actually care about. So yeah, on some level, I definitely want there to be a specific tool. And let’s be clear, you have a terrific stack of things that you integrate with for alerting, for opening tickets, for remediation—or issues, as the case may be. Nomenclature is always a distraction. Don’t at me—but yeah, across the board, I see that you’re doing a lot of things right that if I were going to be entering the space, I would make a lot of those decisions very similarly. And then expect to hear it from the internet. You’ve been around for years now and are continuing to grow. What’s next for you, folks?Aviad: So, that’s a great question, which I asked myself every morning. I’ll actually take together the two things that you mentioned. One is how we’re focused on serverless, and the second is where do we want to grow from there? And when you do this great focus, you have to make sure that what you’re focusing on is big enough. So, as we’re growing, we’re very happy to see that the serverless is growing with us.We’re seeing more and more places using serverless. We see a lot more users, companies, developers going into serverless. And we see new types of users. So, it’s not only those bleeding-edge technologies that people want to use, and they are really trying to find out how they can use it. We’re seeing more and more places, for example, enterprises that had maybe one architect in the beginning that said, “Okay, I’m going to use serverless.”And now a year or two afterwards, they see that it’s working, and it saving them money. They’re able to build faster, and now it’s both spreading virally to other teams which are starting to use that, and also the initial project, which was started two years ago, is now growing and becoming bigger and more complex. And also, that team which was just starting with serverless two years ago now has maybe a second and third product. So, what we’re doing is we’re looking how we can give serverless better and better monitoring for the new services that are entering that field. And also, we’re very strong believers that developers today are doing much of that monitoring—or observability, you can choose whatever you want—and that means that it goes all the way into debugging.So, we think that doing those two together, bringing together the monitoring and debugging is a great opportunity for our users just to save them more time because it’s the same person who’s going to do both those things, and trying to keep being best of breed in serverless, and doing those two together, I think that’s going to be hard. And that’s exactly the challenge that we’re taking, and we want to see how we’re doing it to the best.Corey: And I think that that is probably the best way to approach it. If people want to learn more about what you’re up to, how you view these things, and ideally, kick the tires on Lumigo and see for themselves, where can they find you?Aviad: So, easiest thing you can do, just search for Lumigo in Google, you’ll get to lumigo.io. And from there, it’s very easy to try us out.Corey: And we will, of course, put links to that in the [show notes 00:31:24]. Thank you so much for taking the time to speak with me today. I really appreciate it.Aviad: Thank you, Corey. It was great fun and looking forward for the next time.Corey: Absolutely. Aviad Mor, co-founder and CTO at Lumigo. I’m Chief Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a long rambling comment telling me how very wrong I am on the wonder that is Lambda@Edge.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 11/19 に出たアップデート13件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ AWS Network Firewall が登場 AWS Parallel Cluster が NVIDIA GPU DirectRDMA をサポート Amazon EC2 Fleet がインスタントタイプのフリートの削除をサポート Amazon EC2 Auto Scaling が複数の起動テンプレートをサポート AWS IAM が IAM ユーザーのデフォルトパスワードポリシーを強化 Amazon Textract が AWS KMS に対応 Amazon Honeycode が Single Sign-On をサポート AWS Managed Microsoft AD が自動化されたマルチリージョンレプリケーションをサポート AppSync を用いたスポーツ情報のリアルタイムアップデートを追加 AWS Backup と AWS Organizations によるクロスアカウントバックアップ機能が登場 Amazon QuickSight がフィルター機能をアップデート Amazon Lex でコンテキスト管理機能を発表 Amazon Kinesis Data Analytics が Apache Flink ダッシュボードに対応 ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ
You can find Dale Salter on Twitter as @enepture.We spoke extensively about GraphQL in this episode. If you want to learn GraphQL and AppSync with a hands-on tutorial, then check out the AppSync Masterclass.For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.This episode is sponsored by ChaosSearch. Have you heard about ChaosSearch? It’s the fully managed log analytics platform that uses your Amazon S3 storage as the data store! Companies like Armor, HubSpot, Alert Logic and many more are already using ChaosSearch as a critical part of their infrastructure and processing terabytes of log data every day. Because ChaosSearch uses your Amazon S3 storage, there’s no moving data around, no data retention limits and you can save up to 80% vs other methods of log analysis. So if you’re sick and tired of your ELK Stack falling over, or having your data retention squeezed by increasing costs, then visit ChaosSearch.io today and join the log analysis revolution!Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0
You can find Ari Palo on Twitter as @aripalo and his LinkedIn profile here.Also, check out his new blog at aripalo.com.Here are links to the tools we mentioned in the show:lumigo-cliaws-lambda-power-tuningdazn-lambda-powertoolsAnd the previous episodes we touched on:#12: Serverless at LEGO with Nicole Yip and Sheen Brisals#3: Building a fully serverless insurance company with Joe EmisonNOTE: since we recorded the show, AppSync has added caching support for nested resolvers!For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0
AWS Amlify Guest Details Nader Dabit Senior Developer Advocate at Amazon Web Services Web and mobile developer specializing in cross-platform & cloud-enabled application development. Links https://www.linkedin.com/in/naderdabit/ https://twitter.com/dabit3 https://dev.to/dabit3/ Questions What is AWS Amplify? Do you see more people using AppSync vs. API Management? We have had a hard time differentiating DynamoDB Scan/Query from what actually gets filtered through AppSync, any perspectives on how to avoid common issues with this? Is there a way to debug using `amplify mock function` in VSCode with 4.18.0?? Do you recommend using Cloudfront to serve S3 images for your Amplify hosted site? Future I know the video plugin is coming along, are there any other cool features coming out for Amplify that people should know about? Purrfect Picks Nader https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393351599 https://www.amazon.com/Autobiography-Malcolm-Told-Alex-Haley/dp/0345350685 https://www.youtube.com/channel/UCv75sKQFFIenWHrprnrR9aA Alex Amplify Video Tutorial for VOD https://github.com/awslabs/unicornflix --- Send in a voice message: https://anchor.fm/purrfect-dev/message Support this podcast: https://anchor.fm/purrfect-dev/support
This is part 1 of my conversation with Scott Smethurst and Diana Ionita about their work at LifeWorks. We talked about the story of LifeWorks, a wellbeing startup that were acquired for $400M, in part, thanks to the heroics of Scott and Diana. They implemented business critical features using serverless technologies, delivered them under severe time constraints and created significant business value which played a part in the acquisition.We talked about how they made their systems multi-region and geo-partitioned their user data. We discussed why they opted for Apollo server instead of AppSync and the challenges of transitioning to a serverless way of doing things - including changing in mindset, local debugging and testing.Check out org-formation, an open-source tool to help you manage your AWS organization using infrastructure-as-code (IAC) and CloudFormation-like syntax.For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-monday/License: http://creativecommons.org/licenses/by/4.0/
Jeremy chats with Nader Dabit about the AWS Amplify team's philosophy around full-stack development, what the Amplify framework is empowering developers to do, and how new features like the Amplify Datastore are making it even easier to build full-scale serverless applications.
Jeremy chats with Marcia Villalba about the benefits of building applications with GraphQL, how to use AWS AppSync to build serverless applications with it, and some best practices for using it in your projects.
Nader Dabit is the author on React Native in Action, Host of React Native Radio, Educator, Speaker, and doing developer relations for AWS Cloud. Chantastic asks about Amplify and AppSync, where they fit into AWS offerings, why they make authentication and GraphQL server setup a breeze, and how we can start using them. They discuss the opportunity and difficulty in podcast, the challenges of author a book, and travel the world speaking and educating.
The Amazon mobile services group is one of the newer ones in their offerings. However, that does not mean it is something you want to skip over. These solutions are likely to be very valuable to your mobile application development efforts. They will help you increase productivity and ease your integration with the AWS infrastructure when you use it for the solution platform. AWS Amplify This helps you build and deploy mobile and web applications. Amplify is a development framework (libraries, components, etc.) and the CLI to help you build a mobile backend for your application. It is compatible with React, iOS, Android, and Web-native applications. This service frees you up to focus on your front-end and business logic rather than struggle through the integration points with AWS. Amazon API Gateway Prior post link is https://develpreneur.com/api-gateway-amazons-service-for-connecting-your-systems/ This service allows you to build, deploy, and manage API's. Thus, it can be thought of as a layer on top of your API to make it easy to maintain a static interface from the point of client applications and services. Amazon Pinpoint We discussed this in the customer engagement tools. As a reminder, it makes it easy to use Push notifications for mobile applications. Thus, it is your easy way to communicate with app users. AWS AppSync Any mobile application will need to handle the issue of disconnected state. You may not want the user to be able to access the application unless a network connection is present. However, that can make it very limited and turn away customers. The AppSync service provides an easy way to handle data synchronization across app and back-end through a framework and libraries. AWS Device Farm This allows you to test android, iOS, and web apps on real devices in a browser. The devices exist in the AWS cloud and provide a cost-effective way to test a large number of devices. The pricing is just seventeen cents per minute or $250 per month for unlimited devices/testing.
Panel: AJ O’Neal Aimee Knight Joe Eames Special Guests: Nader Dabit In this episode, the panel talks with programmer, Nader Dabit, who has been with Amazon’s AWS for the past six months. They discuss the new innovations that Amazon is currently working on, and the exciting new projects that Nader gets to be involved with. Check out this episode to hear all the latest! Show Topics: 1:45 – There are two main things that Nader works with. Check out this timestamp to see what they are. 3:29 – AJ to Nader: Tell me more about manage cloud. I am not sure about Cognito. 3:56 – Yes, Cognito is used by/through Amazon. 5:06 – What are the other manage cloud services that companies want to offer through the tools you have? 5:12 – Nader answers AJ’s question. 7:30 – Can you give me more specifics on the storage solutions you are offering? 8:03 – Nader answers AJ’s question. People store websites there for example. Frontend developers are using S3 buckets, and they are using the library, which is a storage solution. 9:10 – AJ and Nader are having a dialogue between different situations, and Nader is giving the solutions to those hypothetical situations. 10:17 – AJ: “I am interested in what you are talking about AppSync. Can you tell me how that works?” AJ is picking Nader’s brain about how AppSync works. 11:05 – Nader: “It is a single API layer for a point of entry. You can have multi-data sources.” Nader continues, in detail, answering AJ’s question. 12:36 – AJ: As a frontend developer, it sounds like I will have to become familiar with the backend, too. How is it providing the most value? What is it that I do not have to touch, because I am using this? 15:37 – How would these relations work? As a frontend developer, and I do not want to learn sequel, how would that might look like; currently or in the future? How do you extract that knowledge? 16:18 – Yes, it is not an easy solution to solve. Nader goes into detail about how he would approach this situation. 18:26 – AJ: Are these resolvers written in JavaScript? 22:04 – Acronym fun! 22:45 – Node 23:51 – Summarizing these pasts 20-some-minutes: Off-Storage, AppSync, Landis, and others are what people are using Amplify for. New Question/New Topic: Simplify. 25:45 – AWS MOBILE – is not mobile specific. 26:44 – If you are using Angular, we have a plugin in Angular to help you. We also have that for React and Vue as well. 27:52 – Advertisement 28:56 – What should we be talking about? 29:04 – Let’s talk about Amazon’s Lex, Chat Bot. Nader goes into full detail of this service. 33:52 – Apple T.V. 34:00 – AJ: Sounds like this is more platform/ more agnostic than getting different things to come together, and the Microsoft one is more hybrid and the Amazon one is more open? 35:13 – Joe, let’s go back to what you had to ask. 35:28 – Nader, you talked about PUSH notifications earlier. What is Pub/Sub? 36:30 – Is this like traditional hooks? Or custom? 37:25 – What is the “stuff” that gets you up in the morning and gets you excited to go to work at AWS? 38:40 – Nader: I really had no desire to change career paths, but it happened. 41:30 – AJ: I totally agree with the idea in that finding the common patterns, so that way someone on the lower-level can participate. AJ wants a platform that is open or purchase that can offer some of these benefits. It could be open-source or you used to buy the different tools. 43:27 AJ: What about for the hobbyist? 43:40 – Nader: I agree, that would be really nice. I can’t think of any free services that would be nice. 44:03 AJ – Not free in “free,” but “free” towards the idea of “free speech.” They would all be available and you get to choose what works well for you. 45:00 – SHOUTOUT to LISTENERS: Have an idea about this? Shoot the panel an e-mail! 45:33 – Hopefully this opens the listeners’ eyes to what’s out there. 45:48 – Cloud services. 46:55 – Innovation follows niche markets. When something gets big and established, innovation comes to a plateau. The innovation will develop in a new economic area like hydraulics. AJ thinks a niche will develop. 49:03 – Is there anything, Dabit, which you would like to talk about? 49:15 – Can we talk about AI as a service? 51:10 – Nader saw a demonstration recently. 52:26 – Hearing these implications is so cool, but when it comes to ML a panelist dabbled a little bit. He watched some videos, unless you want to devote a year or two to learning it then it’s too complex to put together. Do you have to be genius-level to get through? 53:29 – ML you are passing data. Nader is not quite sure. 56:00 Nader just did a blog post check-it-out! 56:49 – Let’s do Picks! 56:50 – Advertisement Links: Nader Dabit’s Twitter Nader Dabit’s Medium Nader Dabit’s LinkedIn Nader Dabit’s GitHub Nader Dabit’s Website Nader Dabit’s YouTube channel Nader Dabit’s Egg Head JavaScript Amazon’s Cognito AWS AppSyncNode Landis AWS Mobile Vue Angular Amazon’s Lex – Chat Bot Apple T.V. Push Notifications Pub/Sub AWS’ Artificial Intelligence (AI) Sponsors: Kendo UI Sentry Digital Ocean Picks: AJ O’Neal Blog / Thoughty 2’s Video: Pop Music The Innovator’s Solution / Book The Innovator’s Dilemma / Book Joe Eames Framework Summit - Tickets are still available! Movie: Equalizer 2 Nader Dabit Finland – Graph Talks Conference, October AWS – San Francisco - LOFT
Panel: AJ O’Neal Aimee Knight Joe Eames Special Guests: Nader Dabit In this episode, the panel talks with programmer, Nader Dabit, who has been with Amazon’s AWS for the past six months. They discuss the new innovations that Amazon is currently working on, and the exciting new projects that Nader gets to be involved with. Check out this episode to hear all the latest! Show Topics: 1:45 – There are two main things that Nader works with. Check out this timestamp to see what they are. 3:29 – AJ to Nader: Tell me more about manage cloud. I am not sure about Cognito. 3:56 – Yes, Cognito is used by/through Amazon. 5:06 – What are the other manage cloud services that companies want to offer through the tools you have? 5:12 – Nader answers AJ’s question. 7:30 – Can you give me more specifics on the storage solutions you are offering? 8:03 – Nader answers AJ’s question. People store websites there for example. Frontend developers are using S3 buckets, and they are using the library, which is a storage solution. 9:10 – AJ and Nader are having a dialogue between different situations, and Nader is giving the solutions to those hypothetical situations. 10:17 – AJ: “I am interested in what you are talking about AppSync. Can you tell me how that works?” AJ is picking Nader’s brain about how AppSync works. 11:05 – Nader: “It is a single API layer for a point of entry. You can have multi-data sources.” Nader continues, in detail, answering AJ’s question. 12:36 – AJ: As a frontend developer, it sounds like I will have to become familiar with the backend, too. How is it providing the most value? What is it that I do not have to touch, because I am using this? 15:37 – How would these relations work? As a frontend developer, and I do not want to learn sequel, how would that might look like; currently or in the future? How do you extract that knowledge? 16:18 – Yes, it is not an easy solution to solve. Nader goes into detail about how he would approach this situation. 18:26 – AJ: Are these resolvers written in JavaScript? 22:04 – Acronym fun! 22:45 – Node 23:51 – Summarizing these pasts 20-some-minutes: Off-Storage, AppSync, Landis, and others are what people are using Amplify for. New Question/New Topic: Simplify. 25:45 – AWS MOBILE – is not mobile specific. 26:44 – If you are using Angular, we have a plugin in Angular to help you. We also have that for React and Vue as well. 27:52 – Advertisement 28:56 – What should we be talking about? 29:04 – Let’s talk about Amazon’s Lex, Chat Bot. Nader goes into full detail of this service. 33:52 – Apple T.V. 34:00 – AJ: Sounds like this is more platform/ more agnostic than getting different things to come together, and the Microsoft one is more hybrid and the Amazon one is more open? 35:13 – Joe, let’s go back to what you had to ask. 35:28 – Nader, you talked about PUSH notifications earlier. What is Pub/Sub? 36:30 – Is this like traditional hooks? Or custom? 37:25 – What is the “stuff” that gets you up in the morning and gets you excited to go to work at AWS? 38:40 – Nader: I really had no desire to change career paths, but it happened. 41:30 – AJ: I totally agree with the idea in that finding the common patterns, so that way someone on the lower-level can participate. AJ wants a platform that is open or purchase that can offer some of these benefits. It could be open-source or you used to buy the different tools. 43:27 AJ: What about for the hobbyist? 43:40 – Nader: I agree, that would be really nice. I can’t think of any free services that would be nice. 44:03 AJ – Not free in “free,” but “free” towards the idea of “free speech.” They would all be available and you get to choose what works well for you. 45:00 – SHOUTOUT to LISTENERS: Have an idea about this? Shoot the panel an e-mail! 45:33 – Hopefully this opens the listeners’ eyes to what’s out there. 45:48 – Cloud services. 46:55 – Innovation follows niche markets. When something gets big and established, innovation comes to a plateau. The innovation will develop in a new economic area like hydraulics. AJ thinks a niche will develop. 49:03 – Is there anything, Dabit, which you would like to talk about? 49:15 – Can we talk about AI as a service? 51:10 – Nader saw a demonstration recently. 52:26 – Hearing these implications is so cool, but when it comes to ML a panelist dabbled a little bit. He watched some videos, unless you want to devote a year or two to learning it then it’s too complex to put together. Do you have to be genius-level to get through? 53:29 – ML you are passing data. Nader is not quite sure. 56:00 Nader just did a blog post check-it-out! 56:49 – Let’s do Picks! 56:50 – Advertisement Links: Nader Dabit’s Twitter Nader Dabit’s Medium Nader Dabit’s LinkedIn Nader Dabit’s GitHub Nader Dabit’s Website Nader Dabit’s YouTube channel Nader Dabit’s Egg Head JavaScript Amazon’s Cognito AWS AppSyncNode Landis AWS Mobile Vue Angular Amazon’s Lex – Chat Bot Apple T.V. Push Notifications Pub/Sub AWS’ Artificial Intelligence (AI) Sponsors: Kendo UI Sentry Digital Ocean Picks: AJ O’Neal Blog / Thoughty 2’s Video: Pop Music The Innovator’s Solution / Book The Innovator’s Dilemma / Book Joe Eames Framework Summit - Tickets are still available! Movie: Equalizer 2 Nader Dabit Finland – Graph Talks Conference, October AWS – San Francisco - LOFT
Panel: AJ O’Neal Aimee Knight Joe Eames Special Guests: Nader Dabit In this episode, the panel talks with programmer, Nader Dabit, who has been with Amazon’s AWS for the past six months. They discuss the new innovations that Amazon is currently working on, and the exciting new projects that Nader gets to be involved with. Check out this episode to hear all the latest! Show Topics: 1:45 – There are two main things that Nader works with. Check out this timestamp to see what they are. 3:29 – AJ to Nader: Tell me more about manage cloud. I am not sure about Cognito. 3:56 – Yes, Cognito is used by/through Amazon. 5:06 – What are the other manage cloud services that companies want to offer through the tools you have? 5:12 – Nader answers AJ’s question. 7:30 – Can you give me more specifics on the storage solutions you are offering? 8:03 – Nader answers AJ’s question. People store websites there for example. Frontend developers are using S3 buckets, and they are using the library, which is a storage solution. 9:10 – AJ and Nader are having a dialogue between different situations, and Nader is giving the solutions to those hypothetical situations. 10:17 – AJ: “I am interested in what you are talking about AppSync. Can you tell me how that works?” AJ is picking Nader’s brain about how AppSync works. 11:05 – Nader: “It is a single API layer for a point of entry. You can have multi-data sources.” Nader continues, in detail, answering AJ’s question. 12:36 – AJ: As a frontend developer, it sounds like I will have to become familiar with the backend, too. How is it providing the most value? What is it that I do not have to touch, because I am using this? 15:37 – How would these relations work? As a frontend developer, and I do not want to learn sequel, how would that might look like; currently or in the future? How do you extract that knowledge? 16:18 – Yes, it is not an easy solution to solve. Nader goes into detail about how he would approach this situation. 18:26 – AJ: Are these resolvers written in JavaScript? 22:04 – Acronym fun! 22:45 – Node 23:51 – Summarizing these pasts 20-some-minutes: Off-Storage, AppSync, Landis, and others are what people are using Amplify for. New Question/New Topic: Simplify. 25:45 – AWS MOBILE – is not mobile specific. 26:44 – If you are using Angular, we have a plugin in Angular to help you. We also have that for React and Vue as well. 27:52 – Advertisement 28:56 – What should we be talking about? 29:04 – Let’s talk about Amazon’s Lex, Chat Bot. Nader goes into full detail of this service. 33:52 – Apple T.V. 34:00 – AJ: Sounds like this is more platform/ more agnostic than getting different things to come together, and the Microsoft one is more hybrid and the Amazon one is more open? 35:13 – Joe, let’s go back to what you had to ask. 35:28 – Nader, you talked about PUSH notifications earlier. What is Pub/Sub? 36:30 – Is this like traditional hooks? Or custom? 37:25 – What is the “stuff” that gets you up in the morning and gets you excited to go to work at AWS? 38:40 – Nader: I really had no desire to change career paths, but it happened. 41:30 – AJ: I totally agree with the idea in that finding the common patterns, so that way someone on the lower-level can participate. AJ wants a platform that is open or purchase that can offer some of these benefits. It could be open-source or you used to buy the different tools. 43:27 AJ: What about for the hobbyist? 43:40 – Nader: I agree, that would be really nice. I can’t think of any free services that would be nice. 44:03 AJ – Not free in “free,” but “free” towards the idea of “free speech.” They would all be available and you get to choose what works well for you. 45:00 – SHOUTOUT to LISTENERS: Have an idea about this? Shoot the panel an e-mail! 45:33 – Hopefully this opens the listeners’ eyes to what’s out there. 45:48 – Cloud services. 46:55 – Innovation follows niche markets. When something gets big and established, innovation comes to a plateau. The innovation will develop in a new economic area like hydraulics. AJ thinks a niche will develop. 49:03 – Is there anything, Dabit, which you would like to talk about? 49:15 – Can we talk about AI as a service? 51:10 – Nader saw a demonstration recently. 52:26 – Hearing these implications is so cool, but when it comes to ML a panelist dabbled a little bit. He watched some videos, unless you want to devote a year or two to learning it then it’s too complex to put together. Do you have to be genius-level to get through? 53:29 – ML you are passing data. Nader is not quite sure. 56:00 Nader just did a blog post check-it-out! 56:49 – Let’s do Picks! 56:50 – Advertisement Links: Nader Dabit’s Twitter Nader Dabit’s Medium Nader Dabit’s LinkedIn Nader Dabit’s GitHub Nader Dabit’s Website Nader Dabit’s YouTube channel Nader Dabit’s Egg Head JavaScript Amazon’s Cognito AWS AppSyncNode Landis AWS Mobile Vue Angular Amazon’s Lex – Chat Bot Apple T.V. Push Notifications Pub/Sub AWS’ Artificial Intelligence (AI) Sponsors: Kendo UI Sentry Digital Ocean Picks: AJ O’Neal Blog / Thoughty 2’s Video: Pop Music The Innovator’s Solution / Book The Innovator’s Dilemma / Book Joe Eames Framework Summit - Tickets are still available! Movie: Equalizer 2 Nader Dabit Finland – Graph Talks Conference, October AWS – San Francisco - LOFT
Forrest breaks down AWS AppSync, the death of the 'middle tier', and the future of back-end development in the serverless world.
Forrest breaks down AWS AppSync, the death of the 'middle tier', and the future of back-end development in the serverless world.
There’s no clever title this week, just straight to the point of covering the highlights of AWS re:Invent this week. They got the kubernetes now! There’s a passel of releases as well. We also discuss some other news like Meg Whitman leaving HPE (on good standing), net neutrality, WeWork buying Meetup, and Arby’s. For reals! Pre-Roll SDT News SDT got a new logo! SDT got 1,000 logo stickers to give away! You can get a sticker but completing this survey (https://www.surveymonkey.com/r/SSCKN86) or sending us your address in Slack. US Addresses only until Matt can come and get some stickers. We’ll be doing a live show - probably - on Jan 16 at the CloudAustin Meetup (https://www.meetup.com/CloudAustin/events/244102686/). Check out the Software Defined Talk Members Only White-Paper Ex (https://www.patreon.com/sdt)e (https://www.patreon.com/sdt)g (https://www.patreon.com/sdt)esi (https://www.patreon.com/sdt)s (https://www.patreon.com/sdt) podcast Join us all in the SDT Slack (http://www.softwaredefinedtalk.com/slack). Upcoming SDT newsletter (http://eepurl.com/dbM2_X). Misc. news before re:Invent coverage Changing of the guard at HPE (https://www.theregister.co.uk/2017/11/21/hpe_meg_whitman/). WeWork buys MeetUp (https://www.wired.com/story/why-wework-is-buying-meetup/). Net Neutrality (https://www.wired.com/story/heres-how-the-end-of-net-neutrality-will-change-the-internet/) - I realize this is naive, but I feel like things already operate this way. EFF write-up (https://www.eff.org/deeplinks/2017/11/lump-coal-internets-stocking-fcc-poised-gut-net-neutrality-rules) Stratechery (https://stratechery.com/2017/light-touch-cable-and-dsl-the-broadband-tradeoff-the-importance-of-antitrust/) & follow-up (https://stratechery.com/2017/light-touch-cable-and-dsl-the-broadband-tradeoff-the-importance-of-antitrust/) This week in PE: OOOHH-OOOO! BARRACUDA (https://www.theregister.co.uk/2017/11/27/barracuda_private_equity/)! Also, Arby’s: eat all you want you’ll die anyway (https://www.cnbc.com/2017/11/28/roark-capital-to-buy-buffalo-wild-wings-for-2-point-9-billion.html). Work in tech? Time to ask for a raise. (https://www.wsj.com/articles/tech-boom-creates-new-order-for-world-markets-1511260200) Good overview (http://www.computerweekly.com/feature/Redefining-OpenStack-Addressing-the-identity-and-integration-for-enterprise-readiness) of the end of OpenStack’s big tent theory. AWS re:Invent AWS Business Update Amazon Web Services has an $18 billion revenue (https://www.channele2e.com/news/live-blog-amazon-web-services-ceo-andy-jassy/) run rate and the business is growing 42 percent year over year New AWS Services (100+ new total) Loosely break into themes of Containers, Databases, AI/ML, and IOT Amazon MQ (https://aws.amazon.com/blogs/aws/amazon-mq-managed-message-broker-service-for-activemq/) - Apache ActiveMQ as a Service (lunches eaten?) AppSync (https://aws.amazon.com/blogs/aws/introducing-amazon-appsync/) - GraphQL as a Service (lunches eaten?) Aurora Serverless (https://aws.amazon.com/blogs/aws/in-the-works-amazon-aurora-serverless/) - burst database consumption Comprehend (https://aws.amazon.com/blogs/aws/amazon-comprehend-continuously-trained-natural-language-processing/) - Natural Language Processing across 98 languages DeepLens (https://aws.amazon.com/blogs/aws/deeplens/) - video camera with AI embedded DynamoDB Global (https://aws.amazon.com/blogs/aws/new-for-amazon-dynamodb-global-tables-and-on-demand-backup/) - similar to Azure/Google initiatives EC2 Bare Metal Instances (https://aws.amazon.com/blogs/aws/new-amazon-ec2-bare-metal-instances-with-direct-access-to-hardware/) - lots of competitors try to differentiate on this (lunches eaten?) came out of the VMware work i3.metal instance types c5 AMIs can work too (new KVM-based instance type) EC2 Instance types, up to 25Gbps networking H1 (https://aws.amazon.com/blogs/aws/new-h1-instances-fast-dense-storage-for-big-data-applications/) - higher throughput to storage, replacing D2 instances M5 (https://aws.amazon.com/blogs/aws/m5-the-next-generation-of-general-purpose-ec2-instances/) - 1.15Gbps write to storage, encrypted at rest, multipurpose instances, new Nitro hypervisor Deep dive on EC2 virtualization/bare metal (http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html) T2 Unlimited (https://aws.amazon.com/blogs/aws/new-t2-unlimited-going-beyond-the-burst-with-high-performance/) - good for microservices, bursty workloads with a credit system Elastic Container Service for Kubernetes (https://aws.amazon.com/blogs/aws/amazon-elastic-container-service-for-kubernetes/) (EKS) - called it! upstream K8s automatically runs K8s with three masters across three AZs monitoring/healthchecks built in, managed service Fargate (https://aws.amazon.com/blogs/aws/aws-fargate/) - Containers on demand, no host/orchestrator needed similar to Azure Container Instances apparently Google has App Engine Flexible which is similar (thanks JP!) So, Matt: why would I use EKS instead of Fargate, etc.? Another write-up (https://www.enterprisetech.com/2017/11/30/kubernetes-momentum-builds-new-aws-tools/). FreeRTOS (https://aws.amazon.com/freertos/) - AWS bought(?) existing open source (https://www.freertos.org) IoT operating system vendor Glacier/S3 Select (https://aws.amazon.com/blogs/aws/s3-glacier-select/) - run SQL-like queries against your buckets and storage (CSV & JSON) GuardDuty (https://aws.amazon.com/blogs/aws/amazon-guardduty-continuous-security-monitoring-threat-detection/) - continuous security monitoring & threat detection (lunches eaten?) IoT Analytics (https://aws.amazon.com/blogs/aws/launch-presenting-aws-iot-analytics/) - MQTT processing, reporting & storage IoT Device Defender (https://aws.amazon.com/blogs/aws/in-the-works-aws-sepio-secure-your-iot-fleet/) - reporting, alerting & mitigation of existing IoT fleets IoT Device Management (https://aws.amazon.com/blogs/aws/aws-iot-device-management/) - lifecycle, management & monitoring of IoT devices Kinesis Video Streams (https://aws.amazon.com/blogs/aws/amazon-kinesis-video-streams-serverless-video-ingestion-and-storage-for-vision-enabled-apps/) - video ingestion/processing service Media Services (https://aws.amazon.com/blogs/aws/aws-media-services-process-store-and-monetize-cloud-based-video/) - YouTube as a Service, including monetization. Seems there should be an embeddable player somewhere. Neptune (https://aws.amazon.com/blogs/aws/amazon-neptune-a-fully-managed-graph-database-service/) - managed graph database service (lunches eaten?) Rekognition Video (https://aws.amazon.com/blogs/aws/launch-welcoming-amazon-rekognition-video-service/) - Rekognition now does video SageMaker (https://aws.amazon.com/blogs/aws/sagemaker/) - framework for building AI services Sumerian (https://aws.amazon.com/blogs/aws/launch-presenting-amazon-sumerian/) - VR/AR/3D IDE and platform? Systems Manager (https://aws.amazon.com/blogs/aws/aws-systems-manager/) - custom dashboards based off of tags, ties into AWS system management tools Time Sync Service (https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/) - AWS NTP Translate (https://aws.amazon.com/blogs/aws/introducing-amazon-translate-real-time-text-language-translation/) - Google & MS already have this Transcribe (https://aws.amazon.com/blogs/aws/amazon-transcribe-scalable-and-accurate-automatic-speech-recognition/) - speech recognition, we should use this! More: The New Stack (https://thenewstack.io/aws-takes-kubernetes-offers-serverless-database-service/), The Register (https://www.theregister.co.uk/2017/11/29/amazon_aws_kubernetes/). This kind of over-the-top analysis (https://blog.codeship.com/aws-reinvent-a-musical-review-of-the-2017-keynote/) is kinda our thing (https://www.patreon.com/sdt). BACK OFF, MAN! AWS Strategy Update On Hybrid Cloud: “In the fullness of time — I don’t know if it’s five, 10 or 15 years out — relatively few companies will own their own data centers. Those that do will have a much smaller footprint. It will be a transition and it won’t happen overnight.” Link (https://www.channele2e.com/news/live-blog-amazon-web-services-ceo-andy-jassy/) More: ‘Is Multi-Cloud Real?: “We certainly get asked about it a lot. Most enterprises, when they think about a plan for moving to the cloud, they think they will distribute workloads across a couple of cloud providers. But few actually make that decision because you have to standardize on lowest common denominator when you go multi-cloud. AWS is so far ahead and you don’t want to handicap developer teams. Asking developers to be fluent in multiple cloud platforms is a lot. And all the cloud providers have volume discounts. If you split workloads across multi-cloud, you’re diminishing those discounts. In practice, companies pick a predominate cloud provider for their workloads. And they may have a secondary cloud provider just in case they want to switch providers.’ AWS re:Invent Preview Review ✔SaaS lunches will be eaten? ✔Amazon Kubernetes Service? This Week in Kubernetes All about AWS this week! Well, GKS did get rid of billing for cluster managers Coté finished up this pile of crap (get a preview!) (https://docs.google.com/document/d/13JaEeN3Vww_Lu5FTgFgArl16HUQ2Oo4lIR1ua_7zZU4/edit?usp=sharing) and right after emailing it in was reminded that Ben wrote this up already, (https://stratechery.com/2016/how-google-cloud-platform-is-challenging-aws/) plus an update based on re:Invent this week (https://stratechery.com/2017/aws-fargate-and-kubernetes-support-embrace-and-extend-awss-execution-advantage/). End-roll Conferences Coté’s junk: NEXT WEEK, FOOLS! SpringOne Platform registration open (https://2017.springoneplatform.io/ehome/s1p/registration), Dec 4th to 5th. Use the code S1P200_Cote for $200 off registration (https://2017.springoneplatform.io/ehome/s1p/registration). Coté and many others speaking. Coté will be doing a tiny talk at CloudAustin on December 19th (https://www.meetup.com/CloudAustin/events/244459662/). Matt’s (not) on the Road! Taking it off for the Holidays. Recommendations Matt Ray: Art of War, backlaid by Wu Tang Clan (https://www.youtube.com/watch?v=qCk7ozsr428) Brandon: Hindenburg audio editor (https://hindenburg.com/). Coté: Programmed Inequality (http://amzn.to/2Aj4StV); drink after the kids go to bed; Mindhunter (https://www.netflix.com/title/80114855); Jim and Andy (https://www.netflix.com/title/80209608).
Prod. #163 YouTube - Mini NES - OCULUS - Jailbreak! 1. YouTube GO es real - http://bit.ly/YoutubeGo 2. Facebook está trabajando duramente en las Oculus y añade guantes para interactuar - http://bit.ly/2kFN2sH 3. Juega Snes, Sega y otras consolas en tu NES Classic - http://bit.ly/2kXEZ9j 4. Ford invertirá 1.000 millones de dólares para su coche autónomo - http://bit.ly/2kYjUeK 5. Comunidad del Jailbreak recibe la actualización de AppSync Unified para iOS 10 - http://bit.ly/AppSync10
Enabling our customer Business Applications is at the heart of everything we do. EMC AppSync is designed to simplify and automate the process of generating and consuming copies of production data. By providing an abstraction layer between applications and underlying storage replication technology, AppSync simplifies the process of Application Replica creation. This Integrated Copy Data Management (iCDM) tool provides an intelligent and simple to use interface that can provide application owners and VMware admins the ability to restore and clone their data, with or without leveraging the storage team. The AppSync team recently released version 3.0 of the product, and Boaz Michaely (@BoazMichaely) AppSync Product Manager, gives us the details this week on EMC The Source Podcast. Don’t miss “EMC The Source” app in the App Store. Be sure to subscribe to The Source Podcast on iTunes, Stitcher Radio or Google Play and visit the official blog at thesourceblog.emc.com EMC: The Source Podcast is hosted by Sam Marraccini (@SamMarraccini)
© 2020-2025 Ivy Podcast Discovery LLC
