Podcasts about t sql

Many announcements and interesting releases came out this past month, so we got a lot to talk about in this episode! Relevant links: What's New in SQL Server 2025 - SQL Server | Microsoft Learn SQL Server 2025 - AI ready enterprise database from ground to cloud | Microsoft Community Hub Announcing Public Preview of DiskANN in SQL Server 2025 | Microsoft Community Hub SQL Server 2025: introducing optimized Halloween protection | Microsoft Community Hub SQL Server 2025: introducing tempdb space resource governance | Microsoft Community Hub Unlocking the Power of Regex in SQL Server - Azure SQL Devs' Corner Announcing the General Availability (GA) of JSON data type and JSON aggregates | Microsoft Community Hub Announcing the Public Preview of JSON index in SQL Server 2025 | Microsoft Community Hub ZSTD compression in SQL Server 2025 | Microsoft Community Hub MSSQL Extension for VS Code: GitHub Copilot Preview + UI GA MSSQL Extension for VS Code: Introducing Schema Compare (Preview) - Azure SQL Devs' Corner SQL Server Management Studio (SSMS) 21 is now generally available (GA) | Microsoft Community Hub Copilot in SSMS preview Recently released: Updates to the SqlPackage and the DacFx ecosystem | Microsoft Community Hub Visual Studio 2022 Release Notes | Microsoft Learn Free SQL Managed Instance offer is now generally available Avoid T-SQL anti-patterns with the free T-SQL analysis tool - Azure SQL Devs' Corner

Guy talks about a recent migration project he was working on to move an on-prem SQL Server to AWS RDS. Eitan talks about a new AI-related feature in Azure SQL Databases. And we also discuss the new SNAPSHOT backup T-SQL command in SQL Server 2022. Relevant links: Importing and exporting SQL Server databases using native backup and restore - Amazon Relational Database Service Learn about Amazon RDS maintenance window | AWS re:Post LLM Structured Output from Azure SQL Explained - Azure SQL Devs' Corner Create a Transact-SQL snapshot backup - SQL Server | Microsoft Learn

Unify and secure your data—no matter where it lives—without sacrificing control using OneLake security, part of Microsoft Fabric. With granular permissions down to the row, column, and table level, you can confidently manage access across engines like Power BI, Spark, and T-SQL, all from one place. Discover, label, and govern your data with clarity using the integrated OneLake catalog that surfaces the right items fast. Aaron Merrill, Microsoft Fabric Principal Program Manager, shows how you can stay in control, from security to discoverability—owning, sharing, and protecting data on your terms. ► QUICK LINKS: 00:00 - OneLake & Microsoft Fabric core concepts 01:28 - Table level security 02:11 - Column level security 03:06 - Power BI report 03:28 - Row level security 04:23 - Data classification options 05:19 - OneLake catalog 06:22 - View and manage data 06:48 - Governance 07:36 - Microsoft Fabric integration 07:59 - Wrap up ► Link References Check out our blog at https://aka.ms/OneLakeSecurity Sign up for a 60-day free trial at https://fabric.microsoft.com ► Unfamiliar with Microsoft Mechanics? As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. • Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries • Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog • Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast ► Keep getting this insider knowledge, join us on social: • Follow us on Twitter: https://twitter.com/MSFTMechanics • Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ • Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ • Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics  

A domain is a set of possible values (among other definitions). I use this word a lot in my work, often with a problem domain (the thing you're trying to solve) or the domain of possible values (like the US States and Territories list). That last one is interesting, as this is often the set of data we stick in some reference or lookup table to use in a form on a screen. There is a domain as part of the SQL specification, which I never knew about. I was reading an article from Joe Celko on the CREATE DOMAIN statement. This doesn't exist in SQL Server and T-SQL, but it has been a part of PostgreSQL for quite a few versions. The article talks about the definition of a domain from a few very experienced database design people. Read the rest of A Domain for Data

In this livestream, Frank and Andy discuss the timeless nature of backend enterprise tech, that, much like a Christmas special from decades ago, is still very much celebrated.Moments00:00 Exploring SSIS future in a festive episode.08:28 Data engineering evolved from business intelligence systems.10:57 Social networks project before Facebook's popularity.19:19 SSIS training informed data engineering concepts teaching.24:56 Bill Gates moved project to immature Microsoft tooling.29:10 Data engineering possible in 2024 using T-SQL.35:23 Huge cloud companies surpass previous brick-and-mortar giants.40:10 Old technologies endure; misconceptions about their age.46:03 Evaluate change benefits: technical ease, business growth.52:30 Cloud departure interests rise, SSIS assistance sought.55:47 Big government agency utilizing diverse cloud platforms.01:00:59 Security is crucial; clients' preferences vary.01:08:56 Certification issues hinder software updates and compliance.01:10:02 People stick with older systems for reasons.01:15:15 Proper GPU driver drastically improved loading time.01:22:16 Repost increased engagement and communication with author.01:25:45 Data scientists should learn SQL for simplicity.01:31:06 Obsolete systems cause issues without quotes.

How do you test your database? While at NDC Porto, Richard chatted with Dan Mallott about building unit tests for transactional databases like SQL Server. Dan talks about using testing frameworks constructed for the purpose, like TSQL-T, to make it easier to test individual database elements, from stored procedures to column constraints. The conversation digs into the challenges around testing, tolerating the changes to the database, and tweaking how you write your T-SQL code to be more testable. But the power of getting database tests into your CI/CD pipeline is enormous - catch more problems in testing before they become problems in production!LinkstsqltDbFitRecorded October 17, 2024

I'm not the smartest developer or DBA. I find myself mystified at times by Itzik's posts on T-SQL queries and I'm amazed at times by the complex systems that I see the DCAC people put together. I can usually figure things out (sometimes by asking the authors a question), but it's not always easy to do. We have some truly gifted, incredibly intelligent people in this business. I am, however, effective. I have been very successful in my career at getting things done well enough, things that work well, meet the needs of my client/employer, and meeting deadlines. I don't just slap things together, but think about them, build them, test them (don't forget this), and then make sure they're working when they're deployed. Read the rest of Effective Engineering

I don't see a lot of SQL at The Daily WTF, but this one was great. It's a stored procedure that was likely just converted from embedded code, as noted by the poster. It's a strange set of code, that doesn't quite make sense to me, and I can't imagine why someone wrote it. Arguably, this is no better than having this code in a C# or ASP.NET application. Or is it? Read the rest of Bad Stored Procedures

Check out new AI integrations for your Azure SQL databases. With Retrieval Augmented Generation, you can bridge structured data with generative AI, enhancing natural language queries across applications. With advanced vector-based semantic search, discover precise insights tailored to your data, while Copilot in Azure streamlines troubleshooting and T-SQL query authoring. Optimize workflows, personalize responses, and unlock new levels of efficiency in SQL-driven AI applications. Accelerate performance troubleshooting and complex query authoring tasks with Copilot in Azure. Quickly diagnose database issues and receive expert recommendations for optimization, ensuring optimal performance and reliability. Seamlessly traverse hierarchies within tables and generate intricate queries with ease, saving time and resources. Bob Ward, Azure Principal Architect, shows how to unleash the full potential of your SQL data, driving innovation and intelligence across your applications.   ► QUICK LINKS: 00:00 - AI and Azure SQL 01:40 - Using T-SQL for search 02:30 - Using Azure AI Search 03:17 - Vector embeddings and skillsets 04:08 - Connect your SQL data to an AI app 05:44 - Test it in Azure OpenAI Studio playground 07:22 - Combine native JSON data type in SQL 08:30 - Hybrid search 09:56 - Copilot in Azure: Performance troubleshooting 11:11 - Copilot in Azure: Query authoring 12:24 - Permissions 12:40 - Wrap up   ► Link References For building AI apps, check out https://aka.ms/sqlai  Try out new copilot experiences at https://aka.ms/sqlcopilot   ► Unfamiliar with Microsoft Mechanics?  As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. • Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries • Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog • Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast   ► Keep getting this insider knowledge, join us on social: • Follow us on Twitter: https://twitter.com/MSFTMechanics  • Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ • Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ • Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics  

I saw this tweet recently, where Richie Rump asked what has changed in T-SQL since the SQL Server 2012 version. A few people from Microsoft responded that there were changes in all versions, and while I think some versions have few changes, I decided to look. SQL Server 2012 introduced the window functions with the OVER() clause to SQL Server. This was a huge change in that many aggregate queries were much easier to write without needing complex GROUP BY lists and subqueries or unions to join together different data. While I'm not an expert by any means, I find lots of queries for reporting easier to write with the window functions, and I've grown to enjoy using these in code. Read the rest of The Growth of T-SQL

It couldn't have happened to a worse set of people, and I'm glad it did, but I'll still take a positive lesson out of this. A hacker sent a typo in a command to a botnet and lost control. That's kind of funny, and I'm glad it happened. The less botnets, the better, IMHO. The coding in this software didn't have good error handling, which is a lesson in and of itself. Overall it seems many developers do a good job of error handling, but I still encounter more pieces of software that allow problematic input than I'd like. While we don't have great error handling in T-SQL, you can make some checks, and you should. Read the rest of Minimize Typing

EP#66 Willis Nana : Data Ingenieur En savoir plus sur moi : https://espresso-jobs.com/conseils-carriere/les-geeks-du-web-willis-nana/ "Creates simple solutions to complex problems" - Mes Skills Data Engineering

Datamart is a new self-service capability included with Power BI Premium that reduces the time it takes to do custom departmental reporting, from weeks to minutes. In just a few clicks, both business users and analysts can easily create and access their own mix of data from dozens of the most common data types and services, all backed by Azure SQL that brings elastic scale and ad-hoc analysis to Power BI. Charles Webb, from the data team at Microsoft, joins Jeremy Chapman to show you a Datamart in action. ► QUICK LINKS: 00:00 - Introduction 00:43 - What changes with Datamart? 02:21 - See Datamart in action 05:34 - Analyst experience 06:32 - Complex requests 09:24 - Pull Data with T-SQL 12:01 - Where to get started ► Link References: For more guidance, check out https://aka.ms/DatamartDocs ► Unfamiliar with Microsoft Mechanics? As the Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. ► Unfamiliar with Microsoft Mechanics? • As the Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries?sub_confirmation=1 • Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog • Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/website • To get the newest tech for IT in your inbox, subscribe to our newsletter: https://www.getrevue.co/profile/msftmechanics ► Keep getting this insider knowledge, join us on social: • Follow us on Twitter: https://twitter.com/MSFTMechanics • Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ • Enjoy us on Instagram: https://www.instagram.com/microsoftmechanics/ • Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics

Why would you use SSMS/T-SQL over PowerShell (PoSh)? When is T-SQL directly a better option than PoSh? That's a question I ask myself regularly as I see articles and blogs that discuss how to accomplish a particular task using one tool or the other. There is plenty of overlap in the capabilities for each language when it comes to working with SQL Server, so this is a decision I think about regularly. This is especially true if you use dbatools. There also appears to be a bias towards one tool or the other for each individual. Many people traditionally have used T-SQL to accomplish most database tasks, and they tend to always look for a solution with a script in SSMS. Others are excited by PoSh and I have seen plenty of questions on the SQL Server Central forums asking how to structure their code in that language. In both cases, there is no shortage of people that argue that you should use T-SQL instead of PoSh or vice versa. Read the rest of Using T-SQL over PoSh

SQL Server is constantly growing and changing, as are most database platforms. There are lots of platform changes, among them enhancements to the T-SQL language. Microsoft has added window functions, in-memory structures, the ability to execute code in other languages, and more. Some of these features are well built and some need more work. What's always interesting to me is what actually gets built and what doesn't. There was an article recently on evaluating features in an RDBMS, and the article uses the JSON data type in Google's Big Query as an example. The evaluation is interesting, examining whether the feature actually helps the user, or if it is mostly marketing. In this case, the feature is outside of the "normal" conventions of the platform,  but it is useful. Read the rest of The Usefulness of Database Features

Guy and Eitan discuss some interesting announcements and focus on the topic of Security in SQL Server. Relevant links: Announcing new features for the SQL IaaS Agent Extension! - Microsoft Tech Community Troubleshooting SQL Server made easy via Notebooks – seeking your feedback - Microsoft Tech Community Veeam security breach: Backup “Removal” Solutions - From Conti Ransomware With Love sp_helprotect is very useful in permission issue - Microsoft Tech Community Databases and Guests - Vitaly Bruk Performance Tuning Like a Pro - with Hypothetical Indexes - Eitan Blumin Dynamic Data Masking in SQL Server Madeira Toolbox Roundup: GUEST user with database permissions.sql PUBLIC role with excessive permissions.sql DBCC CLONEDATABASE (Transact-SQL) Audit trace file binary search by timestamp.sql Security_Audit_Default_Trace.sql SQLSecurityAudit.sql Detect Weak Passwords in SQL Server track_logins_in_use_extended_events.sql

Guy and Eitan talk about some important news, about Availability Sets in Azure, and several useful T-SQL scripts in our Toolbox. Relevant links: Azure Feedback is back! Troubleshooting SQL Server made easy via Notebooks – seeking your feedback Open sourcing the .NET 5 C# Language Extension for SQL Server Replication enhancements in the SQL Server 2019 CU13 release This episode was recorded on October 25th, before the important announcement about the release of SQL Server 2022. Check it out here. Availability Sets for Virtual Machines in Azure Madeira Toolbox Round-Up: Certificate Management Scripts Shrink All Shrinkable Transaction Logs.sql Ola Hallengren Recommended IndexOptimize Examples

Imke Feldmann is among the first few to have recognized the incredible value and potential of this thing called Power Pivot in Excel (which was the precursor to Power BI).  And did she ever run with it, launching quite the successful solo consultancy and training service!  She exemplifies the helpful nature of the data community through her blog, The BIccountant, where she shares her amazing Microsoft BI tool knowledge. Her background is in Finance and Accounting, but you'll quickly realize she knows a great deal more than just Finance and Accounting! Contact Imke: The BIccountant Imke's Twitter References in this Episode: Imke's Github MS Power BI Idea - Customizable Ribbon - Please Upvote :) MS Power BI Idea - Speed Up PQ By Breaking Refresh Chain - Please Upvote :) Episode Timeline: 3:00 - The value of outsourcing certain business functions, Imke's path to Power BI starts with Rob's blog, a multi-dimensional cube discussion breaks out! 19:45 - One of Power BI's strengths is collaboration, Imke LOVES her some Power Query and M and loves DAX not so much 33:45 - Imke has a BRILLIANT idea about how to improve Power Query and some other improvements that we'd like to see in PQ 52:30 -  Rob's VS code experience, how COVID has affected the consulting business, Staying solo vs growing a company and how Imke determines which clients she takes on Episode Transcript: Rob Collie (00:00:00): Hello friends. Today's guest is Imke Feldmann. We've been working for a long time, nearly a year to arrange the schedules to get her on the show, and I'm so glad that we finally managed to do it. For a moment, imagine that it's 2010, 2011, that era. During that timeframe, I felt not quite alone, but a member of a very slowly growing and small community of people who had glimpsed what Power Pivot could do. And for those of you who don't know what Power Pivot is, and that was the version of Power BI, the first version that was embedded only in Excel. And at the time, the way the community grew, we'll use a metaphor for this. Imagine that the community was a map of the world and the map is all dark, but slowly, you'd see these little dim lights lighting up like one over here in the UK, one in the Southwest corner of the United States, very faintly. Rob Collie (00:00:51): And these would be people who were just becoming aware of this thing, this Power Pivot thing, and you'd watch them. They'd sort of show up on the radar, very tentatively at first kind of dipping their toe, and then that light would get brighter, and brighter, and brighter over time, as they really leaned in, and they learned more and more, and they became more adept at it. And this was the way things went for a long time. And then in 2011, out of nowhere in Germany on the map, this light comes on at full intensity, brightly declaring itself as super talented and powerful. And that was what it felt like to come across Imke Feldmann. Rob Collie (00:01:27): Like all of our guests, there's a little bit of that accidental path in her career, but also a tremendous sense of being deliberate. When this stuff crossed her radar, she appreciated it immediately. And I didn't know this until this conversation, but she quit her corporate job in 2013, the same year that I founded P3 as a real company, and became a freelancer. So for eight plus years, she has been a full time Power BI professional. There truly aren't that many people who can say that in the world. Our conversation predictably wandered. At one point, we got pretty deep into the notion of M and Power Query and it's screaming need for more buttons on its ribbon. And Imke has some fantastic ideas on how they should be addressing that. Rob Collie (00:02:14): We also, of course, naturally talked about the differences between remaining a solo freelancer as she has, in contrast to the path that I chose, which is scaling up a consulting practice business. Along the way we reprised the old and completely pointless debate of DAX versus M, I even try to get Tom hooked on M as his new obsession. We'll see how well that goes. Most importantly though, it was just a tremendous pleasure to finally get to talk to Imke at length for the first time after all these years, we literally crossed paths 10 years ago. So it was a conversation 10 years in the making compress down to an hour and change. I hope you enjoy it as much as we did, so let's get into it. Announcer (00:02:56): Ladies and gentlemen, may I have your attention, please? Announcer (00:03:00): This is The Raw Data by P3 Adaptive podcast, with your host Rod Collie, and your cohost Thomas LaRock. Find out what the experts at P3 Adaptive can do for your business. Just go to P3adaptive.com. Raw Data by P3 Adaptive is data with the human element. Rob Collie (00:03:24): Welcome to the show Imke Feldmann. How are you today? Imke Feldmann (00:03:27): Thank you, Rob. Great. It's a great day here over in Germany. Rob Collie (00:03:30): We have been talking about doing this for the better part of a year. So I'm glad that we're landing the guest, Imke is here. I really appreciate you doing this. So why don't we start with the basics. What are you up to these days? What do you do for a living? Imke Feldmann (00:03:48): I have people building great Power BI solutions these days. Rob Collie (00:03:55): Ah, yes. Imke Feldmann (00:03:55): That's how I fill my days. Rob Collie (00:03:58): I hear that that's a good business. Imke Feldmann (00:03:58): Yeah, it is. Rob Collie (00:04:03): So, and your website is? Imke Feldmann (00:04:06): Thebiaccountant.com. Rob Collie (00:04:07): Is that what you are on Twitter as well? Imke Feldmann (00:04:08): Yes. That's also my Twitter handle theBIccountant without an A in the middle. I just replaced the A from accountant with a BI. Rob Collie (00:04:17): There you go. Imke Feldmann (00:04:18): Yeah. Rob Collie (00:04:18): That's right. So that means that I'm going to make a tremendous leap here, wait till you see these powers of observation and deduction. You must have an accounting background? Imke Feldmann (00:04:29): I do, yes. Rob Collie (00:04:30): See you look at that. That's why I make the money. Okay, let's start there, was accounting your first career out of school? Imke Feldmann (00:04:39): Yes. I went to university and studied some economics or business stuff there, they'll know it's translated into English. And then I worked as a business controller. After that, I took over a job to lead a bookkeeping departments or to work with an area where the numbers came from basically. And then after that, I worked as the finance director, where I was responsible for a whole bunch of areas, controlling bookkeeping, IT, HR, and production. So that was quite a job with a broad range of responsibilities. Rob Collie (00:05:18): So you mentioned, kind of slipped IT into that list, right? Imke Feldmann (00:05:23): Yeah. Rob Collie (00:05:23): There's all these things in that list of responsibilities that all seemed they belong together, right? Bookkeeping, accounting, control or finance, IT. We've run into this before, with actually a number of people, that a lot of times the accounting or finance function in a company kind of wins the job of IT by default. Imke Feldmann (00:05:45): Yeah. It seems quite common in Germany, at least I would say. Rob Collie (00:05:48): I get multiple examples, but one that I can absolutely point to is Trevor Hardy from the Canadian Football League, he is in accounting, accounting and finance. And just by default, well, that's close to computers. Imke Feldmann (00:06:00): Yes. Rob Collie (00:06:01): And so it just kind of pulls the IT function in. Now is that true at really large organizations in Germany or is it a mid market thing? Imke Feldmann (00:06:09): No I would say a mid market thing. Rob Collie (00:06:12): That's true here too. So when there isn't an IT org yet it ends up being, oftentimes it falls to the finance and accounting function. Hey, that's familiar. It's kind of funny when you think about it, but it's familiar. And isn't finance itself pretty different from accounting? How much of a leap is that? What was that transition like for you taking over the finance function as well? We tend to talk about these things, at least in the US, is like almost like completely separate functions at times. Imke Feldmann (00:06:43): It depends, but at least it had something to do with my former education, which wasn't the case with IT. So, I mean, of course on a certain management level, you are responsible for things that you're not necessarily familiar with in detail. You just have to manage the people that know the details and do the jobs for you. So that was not too big an issue I must admit. Rob Collie (00:07:10): My first job out of school was Microsoft, an organization of that size, I was hyper specialized in terms of what I did. At this company at P, we are nowhere near that scale, and there's a lot more of that multiple hat wearing. I've definitely been getting used to that over the last decade, the first decade plus of my career, not so much. Imke Feldmann (00:07:31): Yeah. That's interesting because I basically went completely the other way around. I see myself now as working as a technical specialist and as a freelancer, I don't have to manage any employees anymore. Rob Collie (00:07:47): Well, so now you wear all the hats? Imke Feldmann (00:07:49): Yes. In a certain way, yes. Rob Collie (00:07:51): Okay. There's no HR department necessarily, right, so it's just you. But marketing, sales, delivery, everything. Imke Feldmann (00:08:01): Yep, that's true. Yep. And when I first started, I tried to do everything by myself, but the test changed as well. So in the past I started to outsource more things, but to external companies, not internal staff. Rob Collie (00:08:17): So you're talking about outsourcing certain functions in your current business, is that correct? Imke Feldmann (00:08:22): Yes, yes. Rob Collie (00:08:22): So it's interesting, right? Even that comes with tremendous risk when you delegate a certain function to an outside party whose incentives and interests they are never going to be 100% aligned with yours. Even we have been taken for a ride multiple times by third-party consulting firms that we've hired to perform certain functions for us. Imke Feldmann (00:08:46): Oh, no I don't outsource and your services that I directly provide to my clients. Rob Collie (00:08:49): Oh, no, no. Imke Feldmann (00:08:50): No. Rob Collie (00:08:50): No, we don't either. But I'm saying for example, our Salesforce implementation for instance- Imke Feldmann (00:08:56): Okay, mm-hmm (affirmative). Rob Collie (00:08:57): ... Has been a tremendous money sink for us over the years. Where we're at is good, but the ROI on that spend has been pretty poor. It's really easy to throw a bunch of money at that and it just grinds and grinds and grinds. And so this contrast that I'm getting around to is really important because that's not what it's like to be a good Power BI consultant, right? You're not that kind of risk for your clients. But if you go out and hire out some sort of IT related services for example, like Salesforce development, we're exposed to that same sort of drag you out into the deep water and drown you business model, that's not how we operate. I'm pretty sure that's not how you operate either. And so anyway, when you start talking about outsourcing, I just thought, oh, we should probably talk about that. Have you outsourced anything for your own sort of back office? Imke Feldmann (00:09:52): Back office stuff, yeah. My blog, WordPress stuff, or computer stuff in the background. So security [inaudible 00:09:59] the stuff and things like that, things that are not my core, I hire consultants to help me out with things that I would formally Google, spend hours Googling with. Rob Collie (00:10:09): Yes. Imke Feldmann (00:10:10): Now I just hire consultants to do that. Or for example, for Power Automate, this is something that I wanted to learn and I saw the big potential for clients. And there I also did private training basically, or coaching, or how you called it, hire specialists. Rob Collie (00:10:27): To kind of getting you going? Imke Feldmann (00:10:29): Exactly, exactly. Rob Collie (00:10:30): And those things that you've outsourced for your back office, have there been any that felt like what I described you end up deep in the spend and deepen the project going, "What's going on here?" Imke Feldmann (00:10:41): I'm usually looking for freelancers on that. And I made quiet good experiences with it, I must say. Rob Collie (00:10:49): Well done. Well done. All right. So let's rewind a bit, we'll get to the point where you're in charge of the finance department, which of course includes IT. Imke Feldmann (00:10:58): Not necessarily so. I felt quite sad for the guys who I had to manage because I said, "Well, I'm really sorry, but you will hear a lot of questions from me, especially at the beginning of our journey," because I had to learn so much in order to be a good manager for them. So that was quite different situation compared to the management roles in finance that I had before, because there I had the impression that I knew something, but IT was basically blank. Rob Collie (00:11:30): I would imagine that that experience turned out to be very important, the good cross pollination, the exposure to the IT function and sort of like seeing it from their side of the table, how valuable is that turned out to be for your career? Imke Feldmann (00:11:45): I think it was a good learning and really interesting experience for me just to feel comfortable with saying that I have no clue and ask the people how things work and just feel relaxed about not being the expert in a certain area and just be open to ask, to get a general understanding of things. Rob Collie (00:12:09): That's definitely the way to do it, is to be honest and transparent and ask all the questions you need to do. It's easier said than done. I think a lot of people feel the need to bluff in those sorts of situations. And that usually comes back to haunt them, not always. Imke Feldmann (00:12:25): No, that's true. Rob Collie (00:12:27): Some people do get away with it, which is a little sad. So at what point did you discover Power BI? Imke Feldmann (00:12:35): I didn't discover Power BI, I discovered Power Pivot, for your blog of course. Rob Collie (00:12:41): Oh, really? Imke Feldmann (00:12:43): Yes, yes, yes, yes. I think it was in, must be 2011, something like that. Rob Collie (00:12:50): Early, yeah. Imke Feldmann (00:12:51): Yeah. Quite early. When I was building a multidimensional cube with a freelancer for our finance department, then I was just searching a bit what is possible, how we should approach this and things like that. So we started with multi-dimensional cube because that was something where I could find literature about and also find experts who could have me building that. But when doing so, I really liked the whole experience and it was a really excellent project that I liked very much. And so I just searched around in the internet and tried to find out what's going on in that area. And this is where I discovered your blog. Rob Collie (00:13:35): I have no idea. First of all, I had no idea that my old blog was where you first crossed paths with this. Imke Feldmann (00:13:42): I think [inaudible 00:13:43]. Rob Collie (00:13:44): And secondly, I had no idea that it was that early. I mean, I remember when you showed up on the radar, Scott [inaudible 00:13:51] had discovered your blog and said, "Hey, Rob, have you seen this? Have you seen what she is doing? She is amazing." That wasn't 2011, that was a little bit later. I don't remember when but... Imke Feldmann (00:14:06): No, I think we've met first. I think we met on the Mr. XR Forum on some crazy stuff I did there. I cannot even remember what that was, but I started blogging in 2015 and we definitely met before. Rob Collie (00:14:21): That's what it was. It was the forums. And Scott was the one that had stumbled upon what you were doing there and brought my attention to it. I was like, whoa. It was like... Imke Feldmann (00:14:34): That last really some crazy stuff. I think I was moving data models from one Excel file to another or something like that. Some crazy stuff with [inaudible 00:14:43] and so on. Rob Collie (00:14:44): You obviously remember a better than I do. But I just remember being jaw dropped, blown away, impressed, by what you were doing. And the thing is the world of Power Pivot interest at that point in time still seems so small. The community still seems so small that for you to emerge on our radar fully formed, already blowing our minds, that was the first thing we ever heard from you. That was a real outlier because usually the way the curve of awareness went with other members of the community is that like, you'd see something modest from them. And you'd sorta like witnessed their upward trajectory as they developed. Of course, you've continued to improve and learn and all of that since then. But as far as our experience of it, it was you just showed up already at the graduate level, just like where did she come from? So cool. So you said that you enjoyed the multi-dimensional cube project? Imke Feldmann (00:15:43): Mm-hmm (affirmative). Yes. I don't know MDX, but I totally enjoyed the project. So being able to build a reporting solution for my own company, basically then for the company I worked for, and doing it live with a consultant with a freelancer on my hand, discussing how things should look like and just seeing the thing form before my eyes and grow. And this was just such an enjoyable experience for me. Rob Collie (00:16:11): So the thing that's striking about that for me is, there's no doubt that the multi-dimensional product from Microsoft was a valuable product. It did good things. But I never have heard someone say that they really enjoyed the implementation process as a client, right? Imke Feldmann (00:16:31): Okay. Rob Collie (00:16:31): You had a freelancer doing the work. So something you said there really jumped out at me, it was, sort of like doing the project live. So the way that this worked traditionally, at least in the US, is the consultant would interview you about your requirements and write a big long requirements document and then disappear and go build a whole bunch of stuff and come back and show it to you, and it's completely not what anyone expected. It's almost like you're on completely different planets. Obviously, if you'd had that experience, you would not be saying that you enjoyed it. So there had to be something different about the way that you and that freelancer interacted. Do you remember what the workflow was like? Imke Feldmann (00:17:16): What we did is that we often met together and just looked at where we're at and what the next steps should be. And we definitely had specific targets in mind. So there were some reports that I had defined as a target, and around these reports I was aware that we needed something that a proper data model, because I also knew that I wanted to have some sort of a general set up that could be carried from Excel as well. So I knew about cube functions, and I knew that on one hand I needed these reports that had formerly been within our ERP system. Also, I wanted them to be in a separate solution that was under my control and independent from the ERP system. And on the other hand, I wanted some more. So I wanted the flexibility to be able to vary this data and for certain other purposes in the controlling department as well. So basically being able to do ad hoc analysis on it. Imke Feldmann (00:18:23): And we met often and I showed a certain interest in how the table logic was created. So I knew that the MDX was over my head at the time, but I showed a very strong interest in which table are created, how they relate to each other, and that was quite unusual. At least this is what the [inaudible 00:18:47] the freelancer told me. Rob Collie (00:18:49): I bet. Imke Feldmann (00:18:50): He said that he doesn't see that very often that clients showed this sort of interest. Rob Collie (00:18:56): Did he say, "Yeah. You really seem to be having fun with this. Most of my clients don't enjoy this." You said that you met very often, so were there times where he was writing MDX while you were in the room? Imke Feldmann (00:19:10): Sometimes yes, because I said, "Well, can we switch this a bit or make some changes?" And sometimes he said, "Well, I can try adjust now." Because he came over for one day or half a day, and then we spoke things through and defined further things. And if we were finishing early, he would just stay and do some coding there. But apart from that, he would work from home and do the big stuff. Rob Collie (00:19:37): OLAP originally it stands for online analytical processing, where online meant not batch, right? It meant you could ask a question and get the answer while you were still sitting there. Imke Feldmann (00:19:51): Okay. Oh, really? Rob Collie (00:19:53): That's what online meant. Imke Feldmann (00:19:54): It's interesting. Rob Collie (00:19:56): It basically meant almost like real time. It's a cousin of real time, that's what online meant at that point, as opposed to offline where you write a query and submit it and come back next week right? So that's what the online and OLAP comes from. Imke Feldmann (00:20:12): Oh, interesting. Rob Collie (00:20:13): We would pick a different terminology of OLAP were it invented today. So something interesting about, it sounds like your experience, and I did not anticipate drilling into your experience with multi-dimensional on this conversation, but I think it's really important is that at least some portion of that project that you sponsored and implemented with the freelancer, at least some portion of the work was similarly performed online. Meaning the two of you were sort of in real time communication as things evolved. And the old model and the vast majority of multidimensional solutions that have ever been built in the world, the MDX powered solutions, were built and an offline model, where the majority of the communication supposedly takes place in the form of a requirements document. Rob Collie (00:21:05): And that was a deeply, deeply, deeply flawed approach to the problem, that just doesn't actually work. So I guess it's not surprising to me that the one time I've ever heard someone say they really enjoyed that multi-dimensional project, that at least a portion of that multidimensional project was sort of almost like real-time collaboratively performed rather than completely asynchronous, right? I guess we want to be really geeky, we could say it was a synchronous model of communication as opposed to an asynchronous one. And Power BI really facilitates that kind of interaction. Imke Feldmann (00:21:41): Absolutely. Rob Collie (00:21:42): The reason why the MDX multi-dimensional model worked the way it did, or there was two reasons, one is a legitimate one on one of them is more cynical. So the legitimate reason is, is that it required reprocessing of the cube for every change, it's just too slow, right? The stakeholder, the business stakeholder doesn't typically have the time or the patience to sit there while the code's being written, because it's so long between even just implementing a formula change sometimes would be, well, we need to wait an hour. And so the attention span of the business person can't be held for good reason there, right? And so that sort of drove it into an asynchronous model. Rob Collie (00:22:23): The other reason is, is that that is asynchronous model turned out to be a really good business model for the consultants, because the fact that it didn't work meant that every project lasted forever. And so that's the cynical reason. But Power BI is not long delays. You change the measure formula, or you add an extra relationship, or heck even bringing in a new table, just a brand new table, bring it in, it wasn't even in the model, now it's in the model. End to end that can sometimes be measured in minutes or even seconds. And so you can retain engaged collaborative interest. Now it's not like you're always doing that, right? There's still room for offline asynchronous work in our business, but really critical portions of it can be performed the other way. And I think that makes a huge difference. Imke Feldmann (00:23:13): Yep. And that's what I like about it. So it's so great to be able to have, as a consultant, to perform really relatively large tasks without any further involvement of other people. Which, I mean, honestly, I don't call myself a team worker, not because I don't love other people also, but teamwork means you have to communicate with other people, make sure that they know what you're working on. So there are so many interfaces that have to be maintained if you're working with other people. And so I really laugh the way I work currently being able to deliver full solutions as a one woman show consultant. That is really a pleasure for me. That's really my preferred way of work, I must say. Because I can really focus on the things that have to be done and I'm able to deliver value in a relatively short time for the clients. Rob Collie (00:24:14): That's a really interesting concept. There are certain kinds of problems in which collaboration, a team collaboration is absolutely necessary. The magic of collaboration sometimes can beat problems that no individual could ever beat. At the same time though, there's this other dynamic, right, where having a team working on a problem is actually a real liability because the communication complexity between the people becomes the majority of the work. Here's a really hyper simplified example. There used to be sort of a three-person committee, if you will, that was running our company P3, me and two other people. Imke Feldmann (00:24:57): Mm-hmm (affirmative). Rob Collie (00:24:58): And so all leadership decisions were essentially handled at that level. Well, things change, people move on, right? And so we went from a three person committee to a two person committee. We didn't anticipate the two of us who stayed, right? We did not anticipate how much simpler that was going to make things. We thought, just do the math, right, it's going to be like, well, it's one less person to get on the same page. So it's going to be a one-third reduction in complexity. It was actually double that because we went from having three pairs of communication, right, the triangle has three sides, to a line that only has one side, right? So there was only one linkage that needed to be maintained as opposed to three geometrically, combinatorially, whatever we're going to say, right? It just became- Imke Feldmann (00:25:45): Exponential. Rob Collie (00:25:45): ... Exponetially simpler. And so for problems that can be soloed, you have this amazing savings in efficiency, in clarity, even, right? Imke Feldmann (00:25:59): Yup. Rob Collie (00:25:59): There's just so many advantages when you can execute as one person, then there's the other examples like our company at our size now, even ignoring the number of consultants that we need to do our business, just the back office alone, we need the difference in skills. We need the difference in talents and interests and everything. We simply could not exist without that kind of collaboration. However, when our consultants were working with a client, usually it's essentially a one-on-one type of thing, right? We don't typically put teams of consultants on the same project. We might have multiple consultants working for the same client and they might be building something that's somehow integrated, but it's still very similar, I think to your model, when you actually watch sort of the work being done, there's this amazing savings and complexities. Imke Feldmann (00:26:50): Yup, that's true. Of course I have a network in the background. So when big problems arise where I need brain input, of course, I have a network, but it's not a former company. Rob Collie (00:27:02): And that's how we work too, right? We have all kinds of internal Slack channels. For some reason we adopted Slack years ago before Teams was really a thing. So Slack is sort of like our internal social network. There's a lot of discussion of problems, and solutions, and a lot of knowledge sharing, and people helping each other out behind the scenes in that same way. Again, we do bring multiple consultants into particularly large projects, but it's not like there's three people working together on the same formula. In Power BI, the things that you do in ETL, the things that you do in power query are intimately interrelated with the data model and the decks that you need to create. And imagine parceling that out to three different people. You have one formula writer, one data modeler, one ETL specialist, you would never ever get anywhere in that kind of approach. Imke Feldmann (00:28:00): Not necessarily. I mean, the tax people are the person responsible for the data model. He could write down his requirements. He could define the tables basically. And then someone could try to get the data from the sources. But of course, then you get some feedback that the data isn't there or that the model has to be shaped in a different way. So it has two sides to it. But that's interesting to see that you have the same experience, that Power BI models or solutions of a certain size that can very well be handled by one person alone. And that really brings speed, and flexibility, and agility to the whole development process I think. Rob Collie (00:28:41): You communicate with yourself at what's above giga? Peta, petabit? you communicate with yourself at petabit speed and you communicate with others through a noisy 2,400 baud modem that's constantly breaking up. It's amazing what that can do for you sometimes. So there comes a point in your journey where you decide to go freelance. Imke Feldmann (00:29:07): Yup. Rob Collie (00:29:08): That's a courageous leap. When did that happen and what led you to that conclusion? Imke Feldmann (00:29:13): I made the decision in 2012 already to do that. Rob Collie (00:29:19): Wow. Imke Feldmann (00:29:20): And I just saw the light. I just saw the light in Power Pivot and then Power Query came along and I saw what Microsoft was after. And as I said, I enjoyed the building of the cube, getting my hands dirty, reading about the technologies behind it and so on. And this was what I felt passionate about. And I also had the idea that I needed some break from company politics. And so I just thought, well, I give it a try. And if it doesn't work, I can find a job after that or find a company where I work for at any time after that. So I just tried it and it worked. Rob Collie (00:30:05): So you decided in 2012, did you make the break in 2012 as well? Imke Feldmann (00:30:12): I prepared it, and then I just in 2013, I started solo. Rob Collie (00:30:18): Okay. 2013 is also when we formally formed our company. For 2010-2013, it was a blog. I had other jobs. I had other clients essentially, but I wasn't really hanging out the shingle so to speak, as you know, we're not an actual business really until 2013. And I guess it's not much accident that we both kind of did the same thing about the same time, it's that demand was finally sufficient I think in 2013 to support going solo. In 2012, there weren't enough clients to even support one consultant. And so, oh, that's great. And I think you really liked Power Query too, does M speak to you? Imke Feldmann (00:31:02): Yes. Yes. Yeah. Rob Collie (00:31:03): It does, doesn't it? Imke Feldmann (00:31:04): I really prefer Power Query or M over DAX, I must admit. It has been much more liable to me than DAX. Rob Collie (00:31:15): Oh, and I liked you so much before you said that. I'm team DAX all the way. Imke Feldmann (00:31:23): I know. I know. I know. I mean, of course I love to use DAX as well, but I really feel very, very strong about Power Query. And I mean, I had such a great journey with it. I mean, it was really [inaudible 00:31:35] work for me personally, that I did with it. And it was just a great journey to understand how things work. I mean, this has been the first coding language for me that I really learned. And it was just a great journey to learn all the things and starting to blog about it. And of course, I started basically helping people in the forum, that's where I basically built my knowledge about it, solving other people's problems. And this was just a great journey. And Polar Query has always been good to me than DAX. Rob Collie (00:32:14): This is really cool, right? So you fell in love with Power Pivot, so DAX and data model, right? There was no Power Query. Imke Feldmann (00:32:21): Mm-hmm (affirmative)-, that's true. Rob Collie (00:32:23): Okay. And because we had no Power Query, there were many, many, many things you couldn't do in Power Pivot unless your data source was a database. Imke Feldmann (00:32:30): Yup. Rob Collie (00:32:31): Because you needed views created that gave you the right shape tables, right? If your original data source didn't have a lookup table, a dimension table, you had to make one. And how are you going to make one without Power Query? It gets crazy, right? At least unbelievable. So try to mentally travel back for a moment to the point in time where you're willing to, and not just, it doesn't sound like you were just willing to, you were eager to go solo to become a freelancer, right, with just DAX and data modeling. And then after that, this thing comes along that you light up when you talk about. You didn't have this thing that you love, but you were already in, that doesn't happen very often. Imke Feldmann (00:33:18): It could be that loved DAX at the beginning, but it just started to disappoint me at sometimes. Rob Collie (00:33:29): Oh, okay. Thomas LaRock (00:33:29): It disappoints everyone. Rob Collie (00:33:29): I'm just devastated. Imke Feldmann (00:33:35): No, I mean, it's amazing what DAX can do, but I mean, we all know it looks easy at the beginning, but then you can really get trapped in certain situations. Rob Collie (00:33:46): Yeah. I described these two things is like the length and width of a rectangle, Power Query and DAX. Take your pick, which one's the width, which one's the length? I don't care. And then we ask which one is more responsible for the area of the rectangle, right? Neither. You can double the length of either of them and it doubles the area of the rectangle. So it's really ironic that I'm so sort of firmly on team DAX for a number of reasons. Number one, is that I'm really not actually that good at it compared to the people who've come along since. Like my book, for instance, I think, I look at it as this is the 100 and maybe the 200 level course at university, maybe the first in the second course, maybe, but it's definitely not the third course. The thing that you take in your third or fourth year of university, that's not covered in my book in terms of DAX. Rob Collie (00:34:44): And basically every one of the consultants at our company is better at DAX than I am. And that's great. That's really good. And the other thing that's ironic about my love of DAX over M, is if these two were in conflict, which they aren't. Imke Feldmann (00:35:00): No they are. Rob Collie (00:35:02): Is that I actually was trying for years to get a Power Query like project started on the Excel team. I knew how much time was being chewed up in the world just transforming data, not analyzing it even, just getting things ready for analysis. It's just ungodly amounts of time. And so I was obsessed with end-user ETL. When I was on the Excel team, it was like a running joke, someone would mention in a meeting, "Well, that's kind of like ETL," and other people would go, "Oh no, no, don't say that in front of Rob, he's going to get started and he won't shut up about it for the next 30 minutes." On the podcast with the Power Query team, I told them I'm really glad that no one ever agreed to fund my project on the Excel team because now that I see what Power Query is like I grossly underestimated how much work needed to go into something like that. And I'm glad that Microsoft isn't saddled with some old and completely inadequate solution to the Power Query space, because now that I've seen what the real thing looks like, I'm like, "Oh my gosh, we would've never been able to pull that off." Rob Collie (00:36:14): So the thing that I was most obsessed with is the thing that now that it's actually been built, for some reason, I just find M to be, I don't know, there's like a reverse gravity there that pushes me away. Imke Feldmann (00:36:26): What I actually would like to see is that there's less need to use M in the Power Query product. So first, the only thing I was dreaming about was finally to have a function library that can easily be shipped from then, or that you can download from internet or wherever, where you can use additional functions in your M code. So this was the first thing that I was really passionate about and thought that we should have such a thing in Power Query to be able to make more cool things, or group steps together. But now what I really think we should actually have and see in Power Query is the ability to build our own ribbons and to the query editor. Rob Collie (00:37:13): Yes. Imke Feldmann (00:37:13): Like we have in an Excel. So this is something that in my eyes would really bring a big push to the product and actually would make so much sense for the people who start using these products. I mean the whole Power platform can have so many benefits for finance department, all departments, but I mean, I'm passionate about finance departments. But have you counted how many low-code languages are in there, if you include Power Apps and Power Automate and all these things? Rob Collie (00:37:50): Low-code. Imke Feldmann (00:37:50): And honestly, in order to come up with any solution that makes sense in a business environment, I would say in all of these solutions, there is no way around the code at the end. I mean, you get quite far with clicky, clicky, but I haven't seen solutions where you get around the languages. And now imagine the typical finance people who really they know the Excel formulas and some of them might know VBA as well. And now their server uses new low-code, no-code word, and just get your head around about five or six new languages that you all have to know and learn in order to get something useful and so on. So I think that's just not feasible for people who have real jobs in the business to learn all that. Rob Collie (00:38:42): Well, that's what you're here for, right? That's what your business is for and that's what P3 is for. Imke Feldmann (00:38:48): We get them started and the products are great. And if there are people in the companies who have a drive to learn things and take the time they get their heads around it, but it could be easier. It could be easier with things like that, where we could provide additional user interfaces and just make it even easier for people to build great solutions for them or adapt solutions that consultants had build initially, but to maintain them by themselves and make adjustments to them if needed. Rob Collie (00:39:19): So [inaudible 00:39:20] has an old joke where he says, when he's doing a presentation or something, he says, "That's a good question. And I define good question as a question I know the answer to, right." And then he says, "But then a great question is a question that is covered by the very next slide." So there's a similar parallel joke to make here, which is that, that idea you just talked about with the ribbons and everything, right? So if I said, it's a smart idea, what I would mean is, again, this is a joke, right? I would mean that that's an idea that I agree with and have kind of already had. But if I say it's a brilliant idea- Imke Feldmann (00:39:55): Okay. Rob Collie (00:39:56): ... Then it's an even better version of an idea that I've already had that has never occurred to me. Your idea is a brilliant idea. Imke Feldmann (00:40:02): Okay. Rob Collie (00:40:06): It goes beyond. So I have been advocating privately behind the scenes with the Power Query team forever telling them that they need about three or four more ribbon tabs. There's just way too many commonly encountered problems for which you can imagine there being a button for, and there's no button. Imke Feldmann (00:40:28): Exactly. Rob Collie (00:40:29): And it's like, I don't understand. I used to be on teams like that, but I don't understand why they haven't gotten to this. Because it seems so low hanging fruit. They've already built the engine, they've built the language, right? The language can already handle this, but you actually had two brilliant ideas in there that had never occurred to me. First of all, I'm used to the idea that the community can't contribute libraries of functions, they can't do that for DAX. Imke Feldmann (00:40:57): Mm-hmm (affirmative). Rob Collie (00:40:58): That's not even like engineering possible for DAX. And the reason for it is, is that the DAX engine is so heavily optimized in so many ways that there'd be no way to plug in some new function that's unpredictable in terms of what it needs to do. All of these things, they're all inherently interrelated and they make changes in the storage and the query engine to make this function work better and vice versa, because it has to take advantage of the index compression scheme and all of that kind of stuff. It's actually not possible, is the wrong word, but it's actually orders of magnitude more difficult, if not impossible to allow DAX to have UDF, user-defined function type of feature. Rob Collie (00:41:42): I don't think Power Query is like that though. Maybe naively, because again, I'm not on the internals team on the Power Query side. But it does seem like a UDF capability is at least much more feasible- Imke Feldmann (00:41:53): Absolutely. Rob Collie (00:41:54): ... For Power Query, which does execute row by row essentially. Other languages have this, right? One of the reasons that R is so popular is not that R is so awesome, is that R has tremendous libraries of commonly solved problems that you can just go grab off the internet or off the shelf and plug into your solution. Imke Feldmann (00:42:14): I have my own library I've created. You can go to my GitHub and you'll see 50, 60 custom M functions. You can package them in a record and [inaudible 00:42:24] them as a library and your M code, or you could even connect live to them and run them with an execute statement. But this is too difficult, although it's just a couple of clicks, but it's too difficult or at least intimidating for the beginners, who really Power Query beginners who start with the products, I think there's so much potential to make their life easier. And that's not through some coding stuff, or I know this function, I know that function, that's really can only come in my eyes through user interface with buttons. Rob Collie (00:42:59): Yeah, I agree. And just as importantly for me, is that I might actually come around and be like, just as much team Power Query as team DAX. Honestly, my frustration is just the M language and just my total lack of desire to learn it. [crosstalk 00:43:16]. It is what it really comes down to. It's not about M, it's not about Power Query, it's about me. Whereas again, I know the need that it fills is massively important. So it's not that I think it's a bad mission, I think it's like the mission in a lot of ways. I was obsessed with it long before I ever crossed paths with business intelligence, I was obsessed with data transformation, end user data transformation. It's just a problem that's about as ubiquitous as it gets. So let's make it happen. We agree, the two of us, that's it, right? It's like we need to go provide a unified front. Imke Feldmann (00:43:52): I think that that's an idea in the idea forum, I might send the link that you can maybe post. Rob Collie (00:43:56): We want that thing up, voted to the moon. I'll even go figure out what my sign in is on the ideas side. Imke Feldmann (00:44:08): Oh, good luck with it. Rob Collie (00:44:09): Which is absolutely impossible. I have no idea which of the 14 counts. And then I'll try to create a new one and it'll go, "Nah, you're not allowed to. We know it's you, but we won't tell you who it is, what your email address is." So I completely agree. So there's so many problems. I always struggle to produce the list. It's like I need to be writing down the list of things that are crucial, but here's an example. Remove duplicates, but control which duplicate you keep. That's a problem that can't be solved in the GUI today. Imke Feldmann (00:44:48): And you need the intimidating type of buffer that you have to write by hand around it, which is just pain. Rob Collie (00:44:56): Remove dups and don't care which one you keep. Okay, fine. That's a great simple button. There should be an advanced section that allows you to specify, oh, but before you keep the dups, sort by this column or sort in the following manner. Imke Feldmann (00:45:10): Exactly. Rob Collie (00:45:10): And then keep the first one of each group. It's easy for us to say outside the team, but apparently that is a, we just make a joke, right? That's apparently a Manhattan project level of software to add that extra button. Anyway, we'll get that. Thomas LaRock (00:45:27): That doesn't make sense to me though. I'm fascinated by all of your conversation and you guys are a hundred miles away from me in a lot of this stuff, but I could listen to it all day. But no, the fact that Excel can't do the remove duplicates, except for like the first of each one of something, that's a simple group by. In my head, I sit there and go that's easily solvable because Excel and DAX does such great stuff that I would never want to do in TSQL, how the hell do we stumble across a thing that's been solved by straight up SQL language that somehow can't get into an Excel? Rob Collie (00:46:01): Well, let's explain the problem very clearly and see if we're on the same page as to what the problem is, but either way it'll be valuable. So let's say you have a whole bunch of orders, a table full of orders. That is a really wide Franken table. It's got things like customer ID, customer address, customer phone number, but also what product they ordered, and how much of it, and how much it cost. Okay, and a date, a date of the order. All right. And you've been given this table because the people that are responsible for this system, they think that what you want is a report and not a data source. And this is incredibly common. Okay. So you need to extract a customer's dimension or lookup table out of this. You need to create a customer's table so that you can build a good star schema model. Okay. And Power Query is right there to help you. Power Query will help you invent a customer's look up table where one wasn't provided, and that's awesome. Rob Collie (00:46:58): Okay. So you say, okay, see customer ID this column. I want to remove duplicates based on that column. Okay, great. But now it's just that the order that the data came in from the report file or the database or whatever that will determine which duplicate is kept. What you really want to do of course is take the most recent customer order of each customer ID because they've probably moved. They may have changed phone numbers, whatever, right? You want their most recent contact information. You don't want their contact information for 15 years ago. And the M language allows you to solve this problem essentially sort by date, and then keep the most recent, but only if you get into the code manually, and as Imke points out, it's not even if you go into the code, the things that you would want to do, if you do a sort, you can add a sort step to the Power Query with the buttons, with the GUI, and then you do the remove duplicates and it ignores the source. Imke Feldmann (00:47:59): Yes. Rob Collie (00:48:02): The GUI almost tries to tell you that it's impossible, but if you know about table dot buffer. Imke Feldmann (00:48:07): So the question is why do we have a sort command in Power Query when it doesn't give the sort order? I mean, that is the question to ask. But that's how it is. Rob Collie (00:48:16): It sorts the results. It sorts the results, it just doesn't sort for the intermediate steps. Imke Feldmann (00:48:20): Why? No, that's quite technical. But would just be great if such a common task could be done with buttons that is reliable at the end. I fully agree. Rob Collie (00:48:35): So Tom, I think this one's really just an example of, again, I truly think that M and Power Query, just like DAX and data modeling, the Power BI data modeling, both of these things belong in the software hall of fame of all time. It is amazing, Power Query, M, is just ridiculously amazing. It's one of the best things ever invented. Remember this is someone who's associated with being a critic of it. Imke Feldmann (00:49:04): Yeah, you're making progress, it's great to see. Rob Collie (00:49:07): And yet I'm telling you that it's one of the top five things ever invented probably. And I think there's a certain tendency when you've done something that amazing to lose track of the last mile. I think it's more of a human thing. Imke Feldmann (00:49:19): Maybe, but I mean, what I see is that they are investing quite a lot in data flows, which makes a lot of sense as well in my eyes. Rob Collie (00:49:27): All that really does though, as far as you and I are concerned, Imke, is it makes it even more important that they solve this problem. Because it's now exposed in two different usage scenarios. Imke Feldmann (00:49:37): Yeah, you're right. Rob Collie (00:49:39): And I want my data flow to be able to control which duplicates are kept too. So that's what I'm saying. There's all these big sort of infrastructural technical challenges that do tend to draw resources. And it's not a neglect thing. Imke Feldmann (00:49:54): No, no. Rob Collie (00:49:54): It isn't like a willful failure or anything like that, I don't want to paint that kind of negative of a picture. Imke Feldmann (00:49:59): No. Rob Collie (00:50:00): It's just that out here in reality, the inability to do, even if we just identified the top 10 things like this, addressing those top 10 things with GUI, with buttons, what have I think in the world, maybe even a bigger impact than the entire data flows project, right? Because you would expand the footprint of human beings that are advocates of this stuff and then you go build data flows. You don't have to think of it as either or, right? They should do both. It's just that I think it's hard to appreciate the impact of those 10 buttons when you're on the software team. It's easier to appreciate the impact of data flows, which is massive. I don't mean to denigrate that. I think it's crazy good. It's just that this other thing is of a similar magnitude in terms of benefit, but it's harder to appreciate when you're on the software team. It's easier to appreciate when you're out here in the trenches, living it every single day. And every time I run into a problem like this, I have to put my hand up and say to my own team, I have to say, " Help." Thomas LaRock (00:51:02): So a casual observation I have is that you wish for there to exist one tool that will handle all of your data janitorial needs. And that tool doesn't necessarily exist because life is dirty, so is your data and you're never going to anticipate everything possible. Now, should that sorting functionality exist in that duplicates, the scenario gave me? Yeah, probably. But there's always going to be something next. And that's why I go to you and I say, the thing that you've described to me is you need your data to be tidy so that it can be consumed and used by a lot of these features that we've talked about today. And in order to get to tidy data, there's no necessarily one tool. Thomas LaRock (00:51:48): You're a big fan of the ETL, Rob. You know that, hey, maybe I need to take the source data and run it through some Python scripts, or some M, or something first before it goes to this next thing. And that's the reality that we really have. What you're wishing for is the one tool, the one button to rule it all. And that's going to take a while before that ever comes around. Rob Collie (00:52:09): The thing is though, is that M is ridiculously complete. Imke Feldmann (00:52:14): Yeah. Rob Collie (00:52:15): You can do anything with it. And it's a language that's optimized for data transformation. So I know you can do anything with C++ too, right? But this is a data crunching, data transformation, specialized language that is really complete. And its UI is woefully under serving the capabilities of the engine. And so I suppose we could imagine and deliberately design a data transformation scenario that maybe M couldn't do it. Imke Feldmann (00:52:45): No. Rob Collie (00:52:46): I think that'd be a very difficult challenge considering how good M is. Imke Feldmann (00:52:49): I think in terms of logic, M can do anything, but in terms of performance, there is some room for improvements. So because there's a streaming semantic running in the background, and as long as the stream runs through all the steps, if you have complex queries, this can really slow things down. And currently there is no button or command in the M language to cut the stream and say, well, stop it here and buffer what you have calculated until here, and then continue from there. So if you have really complex stuff that would benefit from an intermediate buffer, then you can store that in an Azure blob or CSV, or whatever. Specifically if you're working with data flows, you can create some automatic processes that would enable this kind of buffering. Imke Feldmann (00:53:45): And then you will see that the speed of the whole process that can really increase dramatically because in some situations, the speed in M drops exponentially. And these are occasions where a buffer would really helped things, but we don't have it yet in the engine of Power Query. So this was what really be something else that would be fairly beneficial if we wouldn't have to make these work-arounds through things. Rob Collie (00:54:14): Tom, that just occurred to me, I can't believe this is the first time that this thought has crossed my mind. But I think that you might fall into an abyss of love with M. Thomas LaRock (00:54:28): Well, I'm a huge James Bond fan, but... Rob Collie (00:54:30): Oh, no. I think you would really, really just dig it. Thomas LaRock (00:54:38): I don't think I have time to take on a new relationship at this point. I'm still with Python and R, so I mean, I don't know. I'm not going to disagree, I'm just, please don't start a new addiction for me. Rob Collie (00:54:51): Think of the content though, that you could produce over time. The M versus SQL versus Python treatises. Thomas LaRock (00:54:59): Cookbook. Rob Collie (00:55:00): You were made for this mission Tom. Thomas LaRock (00:55:03): Okay. So we'll have to talk later about it. You can sweet talk me. You know I've let you sweet talk me into any [inaudible 00:55:08]. Rob Collie (00:55:08): That's right, that's right. Come on, Tom. Get into M, you know that thing that I have nothing but praise for, that I just love to death, you need to do that. Thomas LaRock (00:55:18): For you. That's what you want to do, is you want to learn it but [inaudible 00:55:21] through me. Rob Collie (00:55:22): Oh, that wouldn't work. I would be, "Oh yeah, well this is still M." Thomas LaRock (00:55:29): You're going to be like, "Tom, where's your latest blog post on M so I can read it and hate upon it even more?" Rob Collie (00:55:37): No, I would not read. Just as the first step. Thomas LaRock (00:55:42): I'm going to read it, but not leave a comment about how much I hate it. Rob Collie (00:55:45): Let's go back to talking about how we did a bunch of big fat Fisher-Price buttons for me to mash my thumbs in the UI. That's what I need. Thomas LaRock (00:55:54): You know what? I'll do that. I'll open up VS code and I'll just build this one big button, it's Rob's button. Rob Collie (00:56:00): Hey, you won't believe this, but I recently installed VS code. Thomas LaRock (00:56:03): I don't believe it, why? Rob Collie (00:56:05): Well, because I needed to edit, not even write, because I'm not capable of it. I needed to edit an interface, add on customization for World of Warcraft. And the only purpose of this World of Warcraft add on interface modification was to allow me to drop snarky comments into a particular channel of the conversation based on the button that I press. I needed a menu of snarky comments to drop at particular points in time. It's hard to type them out all the time, right? So it's just like, now here we go. I dropped one of those. I dropped one of those. Thomas LaRock (00:56:37): We got to get you a real job or something. You got way too much time on your hands. Rob Collie (00:56:42): That was my number one contribution to the World of Warcraft Guild. For a couple of months, there was the snarky rogue chat. Thomas LaRock (00:56:48): You know that is on brand. Rob Collie (00:56:56): It prefixed every comment in the chat with a prefix, you came from rogue chat 9,000. So that people who aren't on the joke were like, "Why is this guy, he's usually very quiet, become so obnoxious. Look at the things he's saying." Anyway. So VS code. And that also involved GitHub. Because my friend who wrote the stub, the shell of this add on for me is a vice president at GitHub. So of course he puts the code in GitHub and points me to it and then points me to VS code, and I'm like, "Oh, you're making me work now? Okay. But you wrote the shell for me, so okay. All right. I'll play ball." So it doesn't sound like you regret your decision to go solo. Imke Feldmann (00:57:40): Absolutely. Rob Collie (00:57:41): You're not looking to go back to corporate life. Imke Feldmann (00:57:43): Absolutely not. Rob Collie (00:57:44): Not missing that. So what can you tell us about the last year or two? What impact, if any, did COVID have on your business? Imke Feldmann (00:57:52): Business has grown especially the last year. So people needed more reports than ever and solutions. So it really, I don't know whether it was COVID effect or just the fact that Power BI is growing and growing. Rob Collie (00:58:07): I'm sure it's both. So the dynamic we saw during 2020. So 2020 would be the, if you're going to have a year that was negatively impacted by COVID, it would have been 2020. And what we saw in 2020 was that we were definitely not acquiring new clients. We weren't making new relationships at nearly the rate we had been people weren't taking risks on meeting a new BI firm. That wasn't something that there was as much appetite for as there had been. However, amongst the clients where we already had a good relationship, we'd already been working with them for a while, their needs for data work expanded as a result of COVID because it did, it created all kinds of new problems and it invalidated so many existing blueprints of tribal knowledge of how we run the business. When reality changes, you need new maps, you need new campuses. Rob Collie (00:59:04): And so on net, we ended up our overall business still grew modestly over the course of 2020, year over year compared to 2019. But then when the new clients started to become viable again, people started looking, we're interested in making new relationships, 2021 has been a very, very strong year of growth, not moderate, really kind of crazy. How do you keep up with increased demand as a one person shop? Imke Feldmann (00:59:35): Saying no. Rob Collie (00:59:36): You have to make your peace with saying no. At one point in my history, I faced sort of the same thing and I decided not to say no, and instead decided to grow the company. That brought an enormous amount of risk and stress- Imke Feldmann (00:59:55): I can imagine. Rob Collie (00:59:55): ... Into my life that I did not anticipate its magnitude. I'm sure I anticipated it, but I didn't anticipate the magnitude of it. I'm very grateful that I'd made that decision though, because where we are today is incredible. That's a rocky transition. So today everything runs like clockwork basically. We have a lot of growth ahead of us that seems almost like it's just going to happen, we're just going to keep growing for a long time. But we had to set the table we had to build our organism as a company into a very different form than what it had been when it was just me. And that molting process it was very painful. I don't pretend that the scaling decision is the right decision, it's very much a personal one. I've certainly lived that. If the version of me that made the decision to scale the company knew everything that was coming, it would have been a much harder decision to make. You kind of have to have a little bit of naive optimism even to make that leap. Imke Feldmann (01:00:57): I can imagine that once you get these things figured out and with the dynamic that the product has, that has a good chance to get it going into a very successful business, I believe. Rob Collie (01:01:10): Well, with your profile and with the growing demand for these sorts of services, the percentage of no that you have to say is just going to keep going up. Imke Feldmann (01:01:20): Yeah. But I made my decision and that's just fine. Rob Collie (01:01:25): I'm very supportive of that decision. I don't have any criticism of it, again, especially knowing what I know now. But if there's going to be come a point where you're going to be saying yes 1% of the time, and the answer to that is ultimately, well, you just raise your rates, which is also very difficult to do. In the end, it's almost like an auction for your services. You need to run yourself like Google. There's a 40 hour block of Imke time coming up for availability. We'll just put it on eBay. Imke Feldmann (01:01:59): I mean, it's just nice to be able to choose with whom you work with. That's just nice. And I earned enough money, so that's fine. So I'm happy with that. Rob Collie (01:02:12): How do you choose who you work with? Is it mostly based on industry? Is it mostly based on job function that you're helping? Or is it more about the specific people? There's all kinds of things that could... Let's say if I came to your website today, I filled out your contact form, what are the things that I could say in that contact for a message that would lead you to say no, versus leads you to say maybe? Imke Feldmann (01:02:37): What I really like to do is to work with finance directors. So basically not people exactly like me, but I like to see that the managers approached me and they have an interest in the product itself and also therefore an interest to push it into their departments. So this is for me, a very, very good starting point because it's an area I'm familiar with. I know that there's enough critical support to get the decisions that have to be made and maybe also push IT to help with certain things. This is really one of my favorite set ups, I would say. Rob Collie (01:03:19): Yeah, we do a lot of work with finance departments as well. How long does sort of your average relationship run with a client? How long do you end up working with the same organization on average? Imke Feldmann (01:03:31): That's hard to say, that's really completely different. It can be the initial five days kickoff where we set up a PNL statement connect all the finance data and they go along with that. And basically, never hear again, or just occasionally hear again, "Can you help me with this problem or that problem?" And it could also be going on for years, basically with breaks in between of course, but some customers, they come every now and then when they want to expand things. Now I have a customer that I'm working on some hours or even days ever week since over a year by now. Rob Collie (01:04:15): That sounds similar to my experience as a freelancer, when it was just me, less similar to our business today, a little bit less. I mean, I think it's still more similar than not. It's just that the dial has moved a little bit. Imke Feldmann (01:04:32): So how long are your engagements then, usually? Rob Collie (01:04:35): Most of our engagements are, if we start out doing kind of that kickoff you're talking about, we started like a project with people, that tends to not be the end. We don't typically have people just immediately vanish after that because that's usually the point at which, I mean, they've got something working already, very often after the first week or so of working with a client, they've usually got some really amazing things built already at that point. But at the same time, that's really just at the beginning of the appetite. Usually there are things that are

גיא ואיתן מדברים על כמה חדשות חשובות בנושא SQL SERVER, אירועים מעניינים, ופניני חוכמה אחרים כגון, למה לגבות זנב? ואיך רפליקציה עושה לעצמה דדלוק? קישורים רלוונטיים: PASS Data Community Summit 2021 (November 8-12) Development Lifecycle Basics for DBAs Group By Conference 2021 (October 26-27) SSDT Methodologies for DB DevOps Transferring and Manipulating Data between Azure SQL Databases SQL Server 2016 SP3: Last Service Pack for SQL Server What does Microsoft Extended Support include? Introducing the GREATEST and LEAST T-SQL functions Where does one report SQL Server or T-SQL bugs, now that UserVoice is gone? What is the replacement Microsoft as it relates to SQL Server? Why Backup the Tail? - Chad Callihan Replication Deadlocks - Erik Darling

There is one thing that Aaron Bertrand won't get mad at me for doing and that's creating an infinite loop with a WHILE (1=1) pattern. He wrote about this recently in a tip, noting that it's easy to create an infinite loop with code like this, something that never ends well. I have purposefully written infinite loops in the past. A long, long time ago Radio Shack would put out TRS-80s on display, and a few of us would type a quick BASIC program to perform some math work on the screen over and over with an infinite loop. It crashed at some point with an overflow, but for a period of time, it would look neat on the screen. Less neat to the salespeople who weren't always sure how to stop it and demonstrate something else without resetting the machine. Read the rest of To Inifinity and Beyond With 1=1

2021-09-07 Weekly News - Episode 118Watch the video version on YouTube at https://youtu.be/pzKWPhzBpqI Hosts:  Eric Peterson - Senior Developer for Ortus Solutions Brad Wood - Software Consultant for Ortus Solutions Thanks to our Sponsor - Ortus SolutionsThe makers of ColdBox, CommandBox, ForgeBox, TestBox and almost every other Box out there. A few ways  to say thanks back to Ortus Solutions: Like and subscribe to our videos on YouTube.  Sign up for a free or paid account on CFCasts, which is releasing new content every week Become an ITB Sponsor - https://www.intothebox.org/#sponsors-2021  Buy Ortus's new Book - 102 ColdBox HMVC Quick Tips and Tricks on GumRoad (http://gum.co/coldbox-tips) Patreon SupportWe have 39 patreons providing 100% of the funding for our Modernize or Die Podcasts via our Patreon site: https://www.patreon.com/ortussolutions. We are now 41% of the way to our next goal, fully funding the ForgeBox.io site.Now offering Annual Memberships, pay for the year and save 10% - great for businesses.News and EventsAdobe ColdFusion Summit 2021Dec 7-8, 2021 - VirtualRegister now for this year's ColdFusion Summit!https://cfsummit.vconfex.com/site/adobe-cold-fusion-summit-2021/1290Special Into the Box Announcement coming on Thursday, September 9th, 2021Adobe 1 Day Workshop - Adobe ColdFusion Workshop with Damien BruyndonckxWed, September 22, 202109:00 - 17:00 CEST EUROPEANJoin the Adobe ColdFusion Workshop to learn how you and your agency can leverage ColdFusion to create amazing web content. This one-day training will cover all facets of Adobe ColdFusion that developers need to build applications that can run across multiple cloud providers or on-premise.https://coldfusion-workshop.meetus.adobeevents.com/ Adobe Webinar for September - Head in the clouds: Installing ColdFusion on Azure and AWS with Mark TakataWed, September 22, 202109:00 - 10:00 PDTIf you've been thinking about getting your CF instance moved into the cloud but don't know where to start, this webinar will get you rolling. We will explore how to install, configure and update CF2021 installs in both Amazon AWS and Microsoft Azure and give you tips and tricks on how to get things off the ground.https://installing-coldfusion-on-azure-and-aws.meetus.adobeevents.com/ CFCasts Content Updateshttps://www.cfcasts.com Just ReleasedUp and Running with Quick The N+1 Problem and Eager Loading Step 7 Exercise Scopes Coming this weekMore Up and Running with QuickSend your suggestions at https://cfcasts.com/supportConferences and TrainingInto the Box 2021 - live in Person in Texas.September 23rd and 24th.No workshops this year.Speakers and Schedule Released!!https://intothebox.orghttps://itb2021.eventbrite.comAdobe ColdFusion Summit 2021December 7th and 8thVirtualhttps://cfsummit.vconfex.com/site/adobe-cold-fusion-summit-2021/1290ITB Latam 2021 - live in personDecember 2nd or 3rd 2021 (confirming dates asap)More conferencesNeed more conferences, this site has a huge list of conferences for almost any language/community.https://confs.tech/Blogs, Tweets and Videos of the WeekBlog - Mark Takata - An introduction to the ColdFusion CLIYou might have heard that ColdFusion has a command line interface (well… actually we have a couple!), but what can it do? Turns out, quite a lot! Let's get you introduced to the CF CLI!https://coldfusion.adobe.com/2021/08/introduction-coldfusion-cli/Blog - Ben Nadel - Using Relative File Paths To Configure Application Mappings In Lucee CFML 5.3.8.201Lucee CFML has a lot of really nice developer ergonomics; one of which is its attitude towards relative file paths. In general, relative file paths "just work". They work when performing file I/O; and, as we now know, they work when configuring ColdFusion application mappings.https://www.bennadel.com/blog/4106-using-relative-file-paths-to-configure-application-mappings-in-lucee-cfml-5-3-8-201.htmBlog - Ben Nadel - Maintaining Route Information During SPA (Single-Page Application) Authentication In Lucee CFMLThe InVision platform is composed of a series of SPAs - Single-Page Applications - that are rendered with AngularJS on the front-end and powered by Lucee CFML on the back-end. Within each SPA, the current route is stored in the URL fragment (aka, the hash), which allowed client-side routing to work in older browsers such as IE6. The biggest challenge presented by fragment-based routing is that the fragment is never sent to the server during navigation. This makes it hard to maintain proper routing during external login events such as Single Sign-On (SSO). To get around this, I've started to use URL query-string parameters when defining routes in the transactional emails sent out from our Lucee CFML application.https://www.bennadel.com/blog/4108-maintaining-route-information-during-spa-single-page-application-authentication-in-lucee-cfml.htmBlog - Ben Nadel - I Wish My Relational Database Tables Were NarrowerNaming things is one of the hardest parts of computer science. Which I believe is a big part of why data modeling is so hard: it's challenging to see how one concept can be decomposed into multiple concepts when you don't know what those smaller concepts are called. This is why my relational database tables tend to get wider over time (ie, gain additional columns). But, after many years of building data-driven application using SQL and ColdFusion, I've come to appreciate how powerful it can be to keep database tables narrower.https://www.bennadel.com/blog/4107-i-wish-my-relational-database-tables-were-narrower.htmCFML JobsSeveral positions available on https://www.getcfmljobs.com/Listing over 180 ColdFusion positions from 85 companies across 111 locations in 5 Countries.2 new jobs listedFull-Time - Senior Coldfusion Developer at Remote - United States Posted Sept 4 https://www.getcfmljobs.com/jobs/index.cfm/united-states/Senior-Coldfusion-Developer-LATAM-at-Colon-PA/11327Full-Time Contract - ColdFusion Software Developer - Mid to Senior Level at Remote - United States Posted Sept 4https://www.getcfmljobs.com/jobs/index.cfm/united-states/ColdFusion-Software-Developer-Mid-to-Senior-Level-Remote-Position-at-Denver-CO/11328ForgeBox Module of the WeekCBSecurity by Ortus SolutionsNow with Refresh Token support!Refresh token endpoint /cbsecurity/refreshToken for secure refresh token generationManual refresh token method on the JwtService : refreshToken( token )Auto refresh token header interceptions for JWT validatorshttps://forgebox.io/view/cbsecurityhttps://coldbox-security.ortusbooks.com/VS Code Hint Tips and Tricks of the WeekSQL ServerMicrosoft - 924,675 downloads - 5 starsWelcome to mssql for Visual Studio Code! An extension for developing Microsoft SQL Server, Azure SQL Database and SQL Data Warehouse everywhere with a rich set of functionalities, including: Connect to Microsoft SQL Server, Azure SQL Database and SQL Data Warehouses. Create and manage connection profiles and most recently used connections. Write T-SQL script with IntelliSense, Go to Definition, T-SQL snippets, syntax colorizations, T-SQL error validations and GO batch separator. Execute your scripts and view results in a simple to use grid. Save the result to json or csv file format and view in the editor. Customizable extension options including command shortcuts and more. https://marketplace.visualstudio.com/items?itemName=ms-mssql.mssqlhttps://docs.microsoft.com/en-us/sql/tools/visual-studio-code/sql-server-develop-use-vscode?view=sql-server-ver15Thank you to all of our Patreon SupportersThese individuals are personally supporting our open source initiatives to ensure the great toolings like CommandBox, ForgeBox, ColdBox,  ContentBox, TestBox and all the other boxes keep getting the continuous development they need, and funds the cloud infrastructure at our community relies on like ForgeBox for our Package Management with CommandBox. You can support us on Patreon here https://www.patreon.com/ortussolutionsNow offering Annual Memberships, pay for the year and save 10% - great for businesses. Bronze Packages and up, now get a ForgeBox Pro and CFCasts subscriptions as a perk for their Patreon Subscription. All Patreon supporters have a Profile badge on the Community Website All Patreon supporters have their own Private Forum access on the Community Website Patreons John Wilson - Synaptrix  Don Bellamy Eric Hoffman David Belanger Dean Maunder Gary Knight Giancarlo Gomez Jonathan Perret Mario Rodrigues Jeffry McGee - Sunstar Media Yogesh Mathur Joseph Lamoree Ben Nadel Brett DeLine Carl Von Stetten Charlie Arehart Dan Card Daniel Garcia Didier Lesnicki Edgardo Cabezas Jan Jannek Jason Daiger Jeff McClain Jeremy Adams Jonas Eriksson Jordan Clark Kai Koenig Laksma Tirtohadi Leon Seremelis Matthew Darby Matthew Clemente Mingo Hagen Patrick Flynn Ross Phillips Scott Steinbeck Shawn Oden Stephany Monge Steven Klotz You can see an up to date list of all sponsors on Ortus Solutions' Websitehttps://ortussolutions.com/about-us/sponsors ★ Support this podcast on Patreon ★

As database professionals, we all know how to manage SQL Server Agent jobs using SQL Server Management Studio (SSMS), TSQL, and PowerShell. With Azure Data Studio and SQL Server Agent extension, we have new tools to create, maintain and manage SQL Agent jobs using Notebook. These are called Notebook Jobs. Once you successfully execute a Notebook job manually or via schedule, each run's results are saved into a separate Notebook, which you can easily share with others. In this episode of Data Exposed: MVP Edition, Taiob Ali will walk you through every step of Notebook Jobs.[00:31] About Taiob Ali[01:34] Benefits of SQL Server Agent Jobs[04:50] Demo[T012:30] Getting StartedResources:Github About Taiob Ali:Taiob Ali, Microsoft Data Platform MVP, is an accomplished technical leader with a proven record of success. During his last 15 years, he has worked with the Microsoft Data Platform and MongoDB, both on-premises and cloud. His experience includes all three major business sectors: finance, e-commerce, and healthcare. Taiob has hands-on experience managing large database projects, including massive data migration, intricate process design, testing and deployment, performance tuning, and long-term capacity planning. Taiob is currently working at "GMO LLC" as Database Solutions Manager, focusing on cloud migration, automation, improving, and streamlining operational workflow. He is a regular speaker at local and virtual chapters, Data Saturdays, and Azure conferences. He is a board member of New England SQL Server User Group, founder of 'Database Professionals Virtual Meetup Group', and organizer of Boston Data Saturday.About MVPs:Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. They are always on the "bleeding edge" and have an unstoppable urge to get their hands on new, exciting technologies. They have a very deep knowledge of Microsoft products and services, while also being able to bring together diverse platforms, products, and solutions, to solve real-world problems. MVPs make up a global community of over 4,000 technical experts and community leaders across 90 countries/regions and are driven by their passion, community spirit, and the quest for knowledge. Above all and in addition to their amazing technical abilities, MVPs are always willing to help others - that's what sets them apart. Learn more.

As database professionals, we all know how to manage SQL Server Agent jobs using SQL Server Management Studio (SSMS), TSQL, and PowerShell. With Azure Data Studio and SQL Server Agent extension, we have new tools to create, maintain and manage SQL Agent jobs using Notebook. These are called Notebook Jobs. Once you successfully execute a Notebook job manually or via schedule, each run's results are saved into a separate Notebook, which you can easily share with others. In this episode of Data Exposed: MVP Edition, Taiob Ali will walk you through every step of Notebook Jobs.[00:31] About Taiob Ali[01:34] Benefits of SQL Server Agent Jobs[04:50] Demo[T012:30] Getting StartedResources:Github About Taiob Ali:Taiob Ali, Microsoft Data Platform MVP, is an accomplished technical leader with a proven record of success. During his last 15 years, he has worked with the Microsoft Data Platform and MongoDB, both on-premises and cloud. His experience includes all three major business sectors: finance, e-commerce, and healthcare. Taiob has hands-on experience managing large database projects, including massive data migration, intricate process design, testing and deployment, performance tuning, and long-term capacity planning. Taiob is currently working at "GMO LLC" as Database Solutions Manager, focusing on cloud migration, automation, improving, and streamlining operational workflow. He is a regular speaker at local and virtual chapters, Data Saturdays, and Azure conferences. He is a board member of New England SQL Server User Group, founder of 'Database Professionals Virtual Meetup Group', and organizer of Boston Data Saturday.About MVPs:Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. They are always on the "bleeding edge" and have an unstoppable urge to get their hands on new, exciting technologies. They have a very deep knowledge of Microsoft products and services, while also being able to bring together diverse platforms, products, and solutions, to solve real-world problems. MVPs make up a global community of over 4,000 technical experts and community leaders across 90 countries/regions and are driven by their passion, community spirit, and the quest for knowledge. Above all and in addition to their amazing technical abilities, MVPs are always willing to help others - that's what sets them apart. Learn more.

Many of us reading this are data professionals, and we likely know quite a bit about SQL in general. We may use T-SQL, specifically, with SQL Server, but many of the skills we have would be portable to other dialects, such as those use in Oracle, PostgreSQL, etc. We'd certainly need to brush up on best practices and which language constructs are better suited for a specific platform, but most of the knowledge transfers. I ran across an interesting post on the value of SQL in a career, mostly for someone that might be moving into the data science or machine learning/AI type role. The post notes that lots of these job descriptions mention SQL, but focus quite a bit on R, Python, modeling, etc. In this case, SQL matters, and I'd agree with the author that it matters a lot. Most of the work in those fields is data prep, and SQL makes this much easier at scale than other languages. Read the rest of Better SQL is a Good Career Investment

Atilla Yardımcı ve Ömer Çolakoğlu ile sohbet ettiğimiz bu bölümde odağımızı SQL'de ve hatta daha da çok T-SQL üzerinde tutuyoruz. Ek olarak iki konuğumuz sektör dinamiklerini ve Veri Bilimci adaylarına da tavsiyelerini ve deneyimlerini paylaşıyorlar. Linkedin: Atilla Yardımcı - Ömer Çolakoğlu

It's not every day that you can hear a great conversation with the Head of Product of Excel. Brian Jones sits down with us and talks about the past, present, and very promising future of Excel. Rob and Brian go way back, and the stories and laughs abound!   Check out this cool World Orca Day Excel template for kids!   Episode Timeline: 4:00 - Brian's lofty title is Head of Product at Excel, The importance and magic of Excel, and people's a-ha moments with Excel 20:25 - The difficulty of not seeing your projects' impact on the world and how the heck does Bluetooth fit into the story?!, Rob and Brian reminisce with some funny conference stories 32:00 - The XML file format and some very neat XML tricks that everyone should know about 51:25 - The birth of the Excel Web App and Rob can't believe some of the things that Brian's team has done with Excel 1:05:00 - How to onboard the Excel, VLOOKUP, and Pivot crowd into data modeling and Power BI, and the future of Excel most certainly includes the Lambda function (maybe!) Episode Transcript: Rob Collie (00:00:00): Hello, friends. Today's guest, Brian Jones, head of product for this thing you might've heard of called Microsoft Excel. Brian and I go back a long way. We were both youngsters at Microsoft at the same time, and we both worked on some early features of Office apps, and we're friends. Really, really have sincerely warm feelings about this guy, as you often do with people that you essentially grew up with. And that's what we did. When Brian and I first worked together, he was working on Word and I was working on Excel. But even though Brian was on Word at the time, he was already working on what we would today call citizen developer type of functionality in the Word application. So even though we were essentially on different sides of the aisle within the Office organization, we were already finding ourselves able to connect over this affinity for the citizen developer. Rob Collie (00:00:55): Now we have some laughs during this conversation about how in hindsight, the things he and I were working on at the time didn't turn out to be as significant as we thought they were in the moment. But those experiences were very valuable in shaping both of us for the initiatives that came later. Rob Collie (00:01:11): Like almost everyone at Microsoft, Brian has moved around a bit. He's worked on file formats for the entire Office suite, which ended up enabling Power Pivot version one to actually function the way that it should. He's worked on Office-wide extensibility and programmability, back to that citizen developer thing again. And in that light, it's only natural that Excel's gravity reeled him in. And in that light, it's only natural that someone like that, someone like Brian, found his way to Excel, and it really is a match made in heaven. And if you permit me the Excel joke, that turned out to be a great match. Rob Collie (00:01:50): We took the obligatory and entertaining, I hope, walk down memory lane. We spent a lot more time than I expected talking about file format. And the reason why is that file formats are actually a fascinating topic when you really get into it. Lot of history there, a lot of very interesting history and challenges we walked through. And of course, we do get around to talking about Excel, its current state, where it's headed, and also the amazing revelation for me that monthly releases actually mean a longer attention span for a product and how we ended up getting functionality now as a result of the monthly release cycle that would have never fit into the old multi-year release cycle. We were super grateful to have him on the show. And as usual, we learned things. I learned things. I have a different view of the world after having this conversation than I did before it, which is a huge gift. And I hope that you get the same sort of thing out of it. So let's get into it. Announcer (00:02:56): Ladies and gentlemen, may I have your attention please? Announcer (00:03:03): This is the Raw Data by P3 Adaptive podcast, with your host Rob Collie and your cohost Thomas Larock. Find out what the experts at P3 Adaptive can do for your business. Just go to p3adaptive.com. Raw Data by P3 Adaptive is data with the human element. Rob Collie (00:03:26): Welcome to the show. Brian Jones, how are you, sir? Brian Jones (00:03:30): I am fantastic. Thank you for having me, Rob. I'm excited. Rob Collie (00:03:33): So let's start here today. Well, you and I go way back, but today, what's your job title and what are your responsibilities? Brian Jones (00:03:42): So today, my job is I'm the head of product for the Excel team. So I lead the team of product managers that are tasked with or given the honor of deciding the future of Excel, where we go with Excel, what are the set of things that we go and build Rob Collie (00:03:59): Head of product. That's a title that we didn't have back when I was still at Microsoft. We did at one point have something called a product unit manager. Is it similar to that? How does that relate? Brian Jones (00:04:11): That's a good question. So we're continuing to evolve the way that we use titles internally. So internally, we have titles that still for most folks externally don't make any sense, like program manager, group program manager, program manager manager, director of program manager. And so for externally, whenever I'm on LinkedIn or if I do PR interviews, things like that, I use the term head of product. Internally, we don't have the term head of product. Rob Collie (00:04:37): Okay. All right. So that's a translation for us. Brian Jones (00:04:40): Yes, exactly. Trying to translate the Microsoft internal org chart to something that makes more sense to folks. Rob Collie (00:04:49): Yeah. So things like, if we use the word orthogonal, what we're really saying is that's not relevant. Brian Jones (00:04:53): Exactly. Rob Collie (00:04:54): That kind of decoder ring. Brian Jones (00:04:57): I didn't realize orthogonal [inaudible 00:04:59] until you said it and I'm like, " Oh yeah, no. Of course, that is completely a ridiculous term to use." Rob Collie (00:05:03): Or I don't know if they still do this, but an old joke that Dave [Gayner 00:05:07] and I used to have, it was all his joke at the time. It was big bet. Do we still talk about big bet? We're going to place a big bet. Brian Jones (00:05:14): Yep. Big bet or big rocks. Big rocks. You know the- Rob Collie (00:05:17): Big rocks. Whoa. Brian Jones (00:05:18): Yeah. It's kind of an analogy. You've got a jar and you want to fill it with the big rocks first, and then you let the sand fill in the rest of the space. So what are the big rocks? Rob Collie (00:05:26): Okay. Yeah. But big bet was one that we used to always make fun of. Brian Jones (00:05:31): Especially when there'd be, "Here are the big bets," and there's 20 of them. Rob Collie (00:05:34): Yeah. The joke I think we used to make was we would call something a big bet when we really didn't have any good reason for doing what we were doing. Anyway, all right. So you're head of product for Excel. That is a pretty heady job. That's pretty awesome. Brian Jones (00:05:52): It's a pretty fun job. Absolutely. Rob Collie (00:05:54): I mean, you're not lacking for eyeballs in that business, are you? We're all friends here. We're all on the same side of this story. I mean, it is the lingua franca of business, Excel. It is the business programming tool. People don't necessarily think of it as programming, but formulas are a programming language. To be head of product for the platform, you could call it an application, but really it's probably more accurate to call it a platform that is, I think, is the single most critical platform to business in the world. That's pretty amazing. Brian Jones (00:06:30): Absolutely. And that's usually the way that we talk about it internally. It depends on who your audience is externally when you're talking about it. But yeah, Excel is a programming language. I remember even before, back when I was on the Word team, but I would go and meet with PJ, who ran program manager for Office all up. And he'd always referred to Excel more as an IDE. And that didn't totally resonate with me at the time because to me, Excel was just a list app, an app for just tracking things. I didn't totally understand what he meant by that, but I'd nod cause he was super important and smart. And it wasn't really until I started working on the team that I was like, "Oh, I totally understand all these things that PJ used to reference." Rob Collie (00:07:06): This one of the things I had been dying to ask you is when you and I first met, I was working on the Excel team, but still had... Gosh, this was year 2000 maybe, maybe 2001. And even though I was nominally part of the Excel team at that point, I still didn't really know Excel, and you were working on Word. So the thing we both had in common at that point is that we didn't know Excel. So I wanted to get your perspective. I know that you've done some things other than Word, but we were already sort of teasing this. So let's just get into it. What's it like to come from "outside" Excel and how's that transition? How do you view Excel differently today versus what you did before? We already started talking about that. The list keeper. That's very common way for people to view it. Brian Jones (00:07:53): When I first started, yeah, I was on Word, although I was working on more kind of end user developer type of pieces of Word. That's how you and I first interacted because we were talking about XML. The first feature I owned was a feature called easy data binding to Excel. And the whole idea was when you could easily bring content from Excel into Word, but then create a link back so that the content in Word would stay live. And a lot of this stuff that I did while I was on Word was all about trying to make Word a little bit more of a structured tool so that people could actually program against it because Word is completely unstructured. It's just free-flowing text. So trying to write a solution against that is almost impossible because you can't predict anything. So we did a lot of work to add structure, whereas Excel out of the gate has all that structure. So it's just much easier to go and program. Brian Jones (00:08:39): If I had gone straight from Word to Excel, it would have been a little bit more of a shock, but I actually had about eight years in between where I was running our extensibility team. So a lot of the work we would do was revving the add-in model and extensibility for Excel. So I got some exposure there. When we did all of the file format stuff and the whole file format campaign, That was a couple of years where I was working really closely with a bunch of folks in Excel, like Dan [Badigan 00:09:06] and folks like that. So I had a bit of exposure, but I'll tell you when I first joined, I had a similar job, but it was for the Access team and we were building up some new tech. Brian Jones (00:09:17): Some of it still is there today. Office Forms came out of some of the investments that we were doing in Access. But when I showed up into Excel, I was very much in that mode of, "Why don't the Excel folks, get it? Everything should be a table with column headings." And like, "That's the model. And why do they stick with this grid? Clearly word of it is eventually going to go away from the printed page as the key medium. Excel's got to go away from the grid. And they've got to understand that this should just be all tables that can be related." And thankfully, I was responsible when I joined and didn't try and act like I knew everything. So I took some time to go and learn. Brian Jones (00:09:52): And it didn't take me long. We have some crazy financial modeling experts on the team and stuff like that, where I'd say it was maybe six months in that it clicked for me where I understood those two key pieces. The grid and formulas are really the soul and the IP of Excel. The fact that you can lay out information really easily on a grid, you have formulas that are your logic, and you can do this step-by-step set of processes where each cell is almost like another little debug point for you. [Cal captain sub 00:10:20] second, and it's the easiest way to go and learn logic and how to build logic. Brian Jones (00:10:25): I didn't get any of that at that time, but you pick it up pretty quickly when you start to look at all the solutions that people are building. And now, obviously, I've been on the team now for five years, so I'm super sold around it. But I'd say it took me a little while and I'm still learning. It takes a while to learn the whole thing. Rob Collie (00:10:41): Yeah. It's funny. Like you said, Word's completely unstructured. You're looking in from the outside and you're like, "Well, Excel is completely structured." Then you get close to it. You're like, "Oh no. And it's not, really." Brian Jones (00:10:52): No. Not at all. Rob Collie (00:10:53): I mean, it's got the cells. Rows and columns. You can't avoid those. But within that landscape, is it kind of deliberately wild west? You can do whatever you need to. You're right. Okay. So tables, yes. Tables are still very important. But you've got these parameters and assumptions and inputs. And what do you do with those? I mean, they're not make a table for those. Brian Jones (00:11:19): Yep. Absolutely. I think that the thing that I started to get really quickly was the beauty of that. Like you said, it's unstructured. You have nice reference points. So if you're trying to build logic, formulas, you can reference things. But there's no rule about whether or not things go horizontally, vertically, diagonally, whatever. You can take whatever's in your mind that you're trying to make a decision around and use that flexible grid to lay it out. It's like a mind map. If you think about the beauty, the flexibility of a mind map, that's what the grid is. You can go and lay out all the information however it makes the most sense to you. Brian Jones (00:11:53): Really, that's what makes Excel still so relevant today. If you think about the way business is evolving, people are getting more and more data, change is just more constant, business processes are changing all the time. So there are certain processes where people can say, "This thing is always going to work the same way." And so you can go and get a vertical railed solution. That's why we use the term rail. That's kind of like if I always know I'm going to take this cargo from LA to San Francisco, I can go and build some rails, and I got a train, it'll always go there and do the same thing. But if business is constantly changing, those rails are quickly going to break and you're going to have to go off the rails. Excel is more like a car than a train. You can go anywhere with it. And so as the business processes change, the people who are using Excel are the same people who are the ones changing those business processes. Those are the business folks. And so they can go and evolve and adapt it and they don't have to go and find another ISV to go and build them another solution based on that new process that's probably going to change again in six months. Thomas Larock (00:12:52): So Brian's been in charge for five years of Excel, and he's sitting there telling us how there's still more to learn. And two weeks ago, we all got renewed as MVPs. And so I was on the MVP website, and I'm going through all the DLs I can join because that's all a manual process these days. I'm like, "Oh, there's the Excel MVP DL. I don't know why I haven't joined this yet." So I click. I'm immediately flooded with 100 emails a day. 100 emails a day. Now, I don't believe I am a novice when it comes to Excel. I don't. I know I'm not on you all's level at all when it comes to it. You build and work and live the product. But I know my way around enough that I can explain things to others when they say, "I'm trying to do this thing." "Oh, I think it's possible." Thomas Larock (00:13:40): But I read these passionate MVPs that you have and the stuff that they highlight, and it's not complex stuff. It's like, "Hey, this title bar seems to be wider in this." And I'm like, I might not even notice this stuff. And I see these features that aren't a complex feature, but I'm like, "I didn't even know that was there. I didn't even know you could do that. Oh, you can do that too." There's so much. And like you said, it's a programming language. It's an IDE. It's all these things. As [Sinopski 00:14:10] said, "It's the killer app for Windows." To have the head of product say that, there's just so much. He really means it. There is a lot to it. And it is something that is malleable and usable by hundreds of millions of people a day. Brian Jones (00:14:25): Yeah. Rob Collie (00:14:26): My old joke is, if you want to know how good someone is at Excel, just ask them, "How good are you at Excel?" And then take their answer and invert it. Brian Jones (00:14:37): That's absolutely true. Rob Collie (00:14:38): If someone says, "Yeah, I'm really good at it," You know they don't have any clue because they haven't glimpsed the depth of that particular mine shaft. And once someone has been to the show, they know better than to oversell their knowledge because they know they can't know everything. Rob Collie (00:14:54): You say you're good at Excel. And then the very next question is one that you're not going to be able to answer. So you got to be careful. [inaudible 00:15:00] person views Excel as Word with a grid. And that's not obviously what it is, but that's the oversimplification for... I don't know... maybe 80% of humanity. Brian Jones (00:15:10): Yeah. And the thing is, there's a lot more that we're doing in the app now to try and make it, one, more approachable, because there's a set of folks that just find it really intimidating, for sure. You open it up and it's this huge, dense grid. Like, "Hey, where do I start? What should I go and do? I've never even heard of this thing before." In the past, a lot of stuff that we would do, we never really thought about those first steps of using the app because we were always like, "Well, everybody knows our app. We're going to go and do the things for everybody that knows our app." And I think we're doing a better job now trying to think, "Well, there's a bunch of people who don't know about our app. Let's go and figure out what the experience should be like for them." Brian Jones (00:15:43): But we've done a lot with AI where we're trying to get a little bit better about... We look at your data. Recommend things to you. So we'll say, "If you've got a table of data, hey, here's a pivot table." You may not have even heard of the pivot table before. So really more like, "Hey, here's a summary of your data." You want to go and insert that. Brian Jones (00:16:00): In fact, those tests are always fun because then we get to work with people who've really haven't ever used a pivot table. So it's always fun to hear the words that they use to describe what a pivot table is. It's like, "Oh wow, you grouped my data for me." Or stuff like that like, "Wow. That's a nice name for it too." So we're trying to do more of that to expose people to really those higher-end things. But those things where for those of us that use it, once you discover that stuff, you're even more hooked on the product. You're like, man, that first experience of somebody built a pivot table for you and you realize, "Oh my God, I didn't know I could do this with my data. Look how much easier it is for me to see what's going on," and trying to get more people to experience that kind of magical moment. Thomas Larock (00:16:39): Now imagine being me and only knowing pivot through T-SQL and that magical day when you meet Rob and he's like, "You just pivot table [inaudible 00:16:49]." And you're like, "How many hours have I wasted? Why didn't someone tell me?" Brian Jones (00:16:56): Yeah. We get that a lot when we'll go and show stuff. Oftentimes, the reaction is more frustration. "I can't believe I didn't know about this for the past five years." Rob Collie (00:17:05): We get that all the time now with Power Pivot and Power Query and Power BI in general. The target audience for that stuff hasn't been really effectively addressed by Microsoft marketing. But even back, just regular pivot tables, such a powerful tool, and so poorly named. You weren't around on the Excel team, Brian, when I waged a six-month campaign to try to rename pivot table to summary table. Brian Jones (00:17:31): Oh really? Rob Collie (00:17:31): Yeah. Brian Jones (00:17:31): How long ago was that? Rob Collie (00:17:33): Oh, well, it was a long time ago. I mean [crosstalk 00:17:35]- Brian Jones (00:17:36): Pivot tables had already been out for quite a while. Rob Collie (00:17:37): Oh God. Yeah. I mean, they were long established. They were in the product. I didn't even know what they did. Believe it or not, I worked on the Excel team for probably about a year before I actually figured what pivot tables could do. People would just throw it around all the time on the team like, "Well, once you have the data, then you can chart it. You can pivot it," blah, blah, blah, blah, blah. And so I would fit in- Brian Jones (00:17:58): You would nod? Rob Collie (00:17:59): I would fit in... I would also author sentences like that, that had the word pivot in it. It was a pretty safe thing to do. There was no downside to it. But believe it or not, the time that I discovered what pivot tables are for... you'll love this... I was trying to figure out how to skill balance the four different fantasy football leagues that I had organized within the Excel and Access team. I wanted to spread it out. Levels of experience. I've got this table of data with the person's name and their level of experience and my tentative league assignment. And just this light bulb went on. I'm like, "Oh my God, I bet this is what pivot tables are for." Total expertise by league. Like, "Oh, look at that. It's totally it." That was a big change for me. That was during the release, Brian, where you and I were working together. Brian Jones (00:18:54): I think I played on one of those fantasy football leagues. Rob Collie (00:18:56): You might have. Brian Jones (00:18:57): I was one of the people with zero experience. I remember going into the draft not knowing... I knew football, but I didn't know anything about fantasy football. Rob Collie (00:19:03): That's right. We did loop you in. So let's do that way back machine for a moment. That release when you and I met was the first release on Excel. I was the lead at that point. It was my first time being a lead. It was the first time I was in charge of a feature set, and it was really my baby, this XML thing we were doing. And the reason for that was because no one was paying any attention. That was this weird release. For a whole release, Office went and tried to do cloud services without having any idea what that really was going to mean. And so we stripped all of the applications down to skeleton crews. And this is really the only reason why on the Excel side, some youngster like me was allowed to be a lead and come up with a feature, because no one cared. No one was paying any attention. There was no one minding the store. Rob Collie (00:19:48): I remember being so wild-eyed enthusiastic about how much this was going to change the world, this XML import export future. And I mean, you might as well just take it out. I can't imagine it's being used hardly at all today. I bet Power View is used more often than the XML import export feature. You all have done a pretty good job of hiding it. So kudos. But it was a good thing to cut my teeth on. I learned a lot of valuable lessons on that release. Rob Collie (00:20:24): How do you feel about the XML structure document work that you were doing in Word at the time? Do you kind of have the same feeling looking back at it that I do? Brian Jones (00:20:33): It was a similar thing. In fact, we did rip it out a couple of years later. I think that when you and I would talk about it, we would talk about these scenarios that were super righteous and great. And then we just start geeking out on tech. And then we would get way too excited about the tech and we kind of forget about those initial scenarios. We wouldn't stop and think, "Wait a minute. These users we're talking about, are they actually going to go and create XML files?" Because you need one of those to start with before any of this stuff makes sense. And no, of course, they're not. But for me, a lot of it started from that. Like I said, one of my first features was that easy data binding to Excel feature. And we thought, "Hey, maybe XML would be a good tech for us to use as a way of having Word and Excel talk to each other," because clearly they have different views on what formatting is and how to present information, but the underlying semantic information, that could be shared. Brian Jones (00:21:20): And so I could have a set of products show up in Excel as a table. And when they come into Word, they look more like a catalog of products. That totally makes sense. And we just did a lot of assumptions that people would make, do all the glue that was really necessary. And of course, they didn't. So I had the exact same experience. The other big thing that was different back then for us was we would plan something, meet with customers for six months, but then it'd take three years to go and build it. We had no way of validating that stuff with customers because we couldn't get them any of the builds. And then even after we shipped it, they weren't actually going to deploy it for another three-plus years. And so the reality is from when you had the idea to where you actually can see that it's actually not working and people aren't using it is probably about six years. So you've probably moved on to something else by then. Brian Jones (00:22:04): The only way you really as a PM got validation that your feature was great was whether or not leadership and maybe press got excited about your thing, but you didn't get a whole lot of signal from actual customers whether or not the thing was working, which is obviously completely different now, thank goodness. Rob Collie (00:22:18): Yeah. That Is true. It took some of the fun out of being done too, now looking back at it, like the day of the ship party, when we were done with the three-year release. "Okay, fine." We'd dunk each other in fountains and there'd be hijinks and stuff. But the world did not experience us being done. That was purely just us feeling done. And then it was like you take a week off maybe, and then the next week, you're right back to the grind at the very beginning. You never got the payoff. Even if you built something really good, by the time the world discovered it and it was actually really helping people at any significant scale, you're no longer even working on that product. Brian Jones (00:22:57): Yeah. You're doing something completely different. Rob Collie (00:22:59): You might be in a different division, both finding out the things in real time that Rob Collie (00:23:03): [inaudible 00:23:00] Both finding out the things, sort of in real time, that aren't working. That's the obvious advantage, right? But there's also this other emotional thing. Like you never got the satisfaction when you actually did succeed. Brian Jones (00:23:11): Right. You didn't see it actually get picked up, adopted. Millions and millions of people using it, which is what the team gets now. We no longer pick a project and say, "Okay, how many people and how long is this going to take?" You really just try and figure out what's critical mass for that project. And then you just let them run. And you'd be really clear around what are the goals and outcomes they're trying to drive. And they just keep going until they actually achieve that. Or we realized that we were wrong, right? And we say, "Hey, we thought people are going to be excited about this. It's not even an implementation thing. We were just wrong. We misread what people really were trying to do. Let's stop. Let's kind of figure out a way of moving off of that and go and figure out what the next thing is we should go and do." Rob Collie (00:23:50): That era that we're talking about right now. The 2003 release of Office. I was still very much a computer science graduate and amateur human. That's exactly backwards, it turns out, if you're trying to design a tool that's going to be used by humanity. Brian Jones (00:24:08): Well, it's what leads you to get really Excited about XML? Rob Collie (00:24:12): That's right. Yeah. That's right. Tech used to have such a power in my life. I'm exactly the opposite now. Every time I hear about some new tech, I'm like, "Yeah, prove it." I am not going to believe in this new radical thing until it actually changes the world around me. I'm not going to be trying to catch that wave. But XML did that to me. It was almost a threat. If we don't take this seriously, we're going to get outflanked. It got really egregious. Rob Collie (00:24:42): I had a coworker one time in that same release in the middle of one of my presentations asked me. This guy wasn't particularly, in the final analysis, looking back, not one of the stronger members of the team, but he had a lot of sibling rivalry essentially in his DNA. And he'd asked me in front of his crowded room, "Well, what are you going to do about Bluetooth?" And, we didn't know what Bluetooth was yet, right? It was like, unless I had an answer for what we were going to do about Bluetooth and Excel, right? Then I was not up on things. You know, the thing that we use to connect our headphones. At the time, Bluetooth was one of those things that might just disrupt everything. Brian Jones (00:25:29): It was funny. It was at that same time, I was asked to give a presentation to the Word team about Bluetooth. We were all assigned things to go and research as part of planning and that was one of the ones I was asked. And I gave a presentation that was just very factual. Here's what it is. And I was given really bad feedback that like, "Hey, I wasn't actually talking about it strategically and how it was going to affect Word. I was just being very factual." And I was like, "I don't understand. I don't understand what success looks like in this task." Right. Rob Collie (00:25:59): I remember going, a couple of years later, going into an offsite, those offsite big, I don't know if you all still do those things, big offsite, blue sky brainstorming sessions. There was this really senior development lead that was there with me. And he and I were kind of buddies. At one point, halfway through the day, he just leans over to me and says, "Hey, I'm going to the restroom and I'm not coming back." And I looked at him in horror, almost like "Thou dost dishonor the offsite!?" And he's like, "Yeah, you know, I've never really believed that much in this particular phase of the product cycle. It's never really meant anything to me. It's all just BS." It was just devastating. I just knew it was right. He was... Brian Jones (00:26:46): But you didn't want to, you didn't want to believe that. Rob Collie (00:26:52): I mean, I felt so special. I was invited to the offsite, the big wigs and everything. Brian Jones (00:26:57): They have nice catering too, Rob Collie (00:26:59): Yeah and he was totally right to leave. Brian Jones (00:27:04): I always remember getting super nervous to present stuff for those. Once it was actually, it was one of our XML ones where I was trying to convince, it was my attempt to get us to create an XML file format, which actually ended up, obviously, happening. But I got an engineer to go to work with and we had Word through an add-in, start to write to XML. And it was just a basic XML format. And then I built all of these... it was like asp.net tools that would go and then create an HTML version of the Word doc that was editable. And it also even created, I think it was called WHAP, I don't remember, like a tech for phones. It was back when you didn't have the rich feature phones, but these basic ones. Brian Jones (00:27:41): And so I created this thing that was almost like a SharePoint site. So you could take all your Word docs, go through this add-in, and then you could actually get an HTML view of them to edit it and a phone view of them to go and edit it. Brian Jones (00:27:51): I think it was probably 2002 or 2001, but I was so excited to go and show that at the offsite because I was like, "Okay, this is where I make it, man. Everybody's going to be so excited about me." But I don't know. I think everybody was excited about Bluetooth at that point or something. Yeah. Rob Collie (00:28:05): Oh yeah Bluetooth, WHAP was so 15 minutes ago. So there's a few, irresistibly funny or interesting things I want to zero in on from that era before we come back to present, and we're definitely going to come back to present, for sure. Rob Collie (00:28:21): First of all, we went to a conference like some W3C sponsor. I don't think it was necessarily W3C affiliated, but it was the XML conference. Brian Jones (00:28:31): The one in Baltimore? Rob Collie (00:28:32): Yes. Rob Collie (00:28:33): Okay. Now two very, very, very memorable things happened at that conference. I bet you already know one of them. But the other one was, and we're just going to make this all this anonymous person's fault. Okay. We're not going to abdicate any responsibility. And we're just going to talk about our one coworker from Eastern Europe who brought his wife and they had vodka in their hotel fridge, or freezer, or something like that. And every day I would wake up and say, "I am not going to get suckered into that again." Rob Collie (00:29:12): And then the next day I would wake up and say the same thing. That was a tough trip. Brian Jones (00:29:16): I definitely remember that. Rob Collie (00:29:18): Even on my young, relatively young, body at the time that... Trying to keep up with that, that was difficult. But the single most outstanding memory from that conference, and we will also leave this person anonymous. But there was an executive at Microsoft who was hotter on XML than either you or I, which is hard to believe, right. And we ended up with the sponsored after hours session at this conference. You remember this? You see... Brian Jones (00:29:45): I do. Rob Collie (00:29:46): You know where we're going. Okay. So this was a 30 minute sponsored by Dell or something. Right. It was a 30 minute session, at 5:00 PM, at the end of a conference day where everyone's trying to go back and get to the bars or whatever, right.? But, it's a Microsoft executive, it's Dell sponsored, we'll show up. And the plan was at the end of this 30 minute talk given by this executive, he was going to bring all of us up on the stage to show everyone the team that had done all of this, right? Great plan. Except it was the worst presentation in history. I remember it running for two hours. It was so bad that we started off with 200 people in the room and at the end of it, and I'm just like an agony the whole time cause like I'm associated with this, right? Rob Collie (00:30:31): At the end of these two hours, or what felt like two hours anyway, it was easily 90 minutes. There's five people left in this room of 200 and it's not like the presentation is adapted to the fact that it's a smaller audience. It's just continued to drone on exactly as if everyone was there, right? And I'm sitting here thinking, "Okay, he's not going to call us all up on this stage. There's been more people on the stage than in the audience. If he does this, he's clearly not going to do that." And then he did and we all had to parade up there and stand there like the biggest dodos. I've never been more professionally embarrassed I don't think, than that moment. Rob Collie (00:31:14): And we're all looking at each other as we get up out of our seats like, "Oh my god." Brian Jones (00:31:19): I definitely remember this. Rob Collie (00:31:22): I don't see how you could have forgotten. Brian Jones (00:31:23): Well, yeah. And the person that we're talking about is actually one of my favorite people on the planet. I totally... I love this guy. I view him as like a mentor and everything, but... Which makes me remember it even more. Brian Jones (00:31:34): I think it was just, there was so much excitement. There'd been so much build up to this and this was like a kind of crescendo right? Of bringing this stuff. We probably should have had it a little bit shorter. Rob Collie (00:31:46): I mean when it reaches the point where clueless, mid twenties, Rob Collie is going, "Oh no, this is not the emotional, this not the move." You don't do it. Brian Jones (00:31:58): I'm no longer excited about being called up. Rob Collie (00:32:04): So from my perspective, you kind of parlayed that experience of the XML and all that kind of stuff. I think you did a really fantastic job of everything you guys did on that product. Again, it was the relevance that ultimately fell flat for both of us right. I guess in the end, the excitement with XML wasn't really all that appreciably different from the excitement about Bluetooth. I mean, it's everywhere, right? XML is everywhere. Bluetooth is everywhere and neither one of them really changed things in terms of what Excel or Word should be doing. It seemed like you played that into this file format second act. And I think very, very, very effectively, actually there was a little bit of controversy. Rob Collie (00:32:43): Let's set the stage for people. This was the 2007 release of Office where all the file formats got radically overhauled. This is when the extra X appeared on the end of all the file names, right? Brian Jones (00:32:58): Yeah. Rob Collie (00:32:58): There was a controversy internally. Kind of starting with Bill actually. That we shouldn't make well-documented transparent file format specs, right. There was this belief that the opaque file formats of the previous decades was in some sense, some big moat against competition. And of course, a lot of our competitors agreed. Tailor out in the public saying, "Yes, this is a barrier to competition. It's a monopolistic, blah, blah, blah." We, Microsoft had just gotten its ass kicked in the Anna Truss case. So it was really interesting. I credit Brian, your crew, with really advocating this very effectively. That's a difficult ship to turn. First of all, you got all these teams to buy into all this extra work, which no one wants to do. But when it's not even clear whether you have top level executive support, in fact, you might actually have C-suite antagonism towards an idea. To get it done. That's a career making achievement. I'm sure you remember all of that. Right. But what are your reactions to that controversy? Do you remember being in the midst of that? Brian Jones (00:34:12): I do. It was definitely a long running project. It evolved over quite a number of years. The beginning of it was, in that previous release, the XML stuff you and I were talking about was more about what we called "Custom XML". Right? So people would go and create for themselves. But in that same release, we had Word, we outputted an XML format that was our definition, which we called "Word ML" and Excel did a similar thing. Words' we try to make full fidelity. So you could save any word document in the XML format. Excel's was kind of a tailored down, it was less about formatting, it was more, "Hey, here's like..." It's almost like, "Here's a better version of CSV, right. But we're going to do it as XML." And so we already had a little bit of that. Brian Jones (00:34:53): And the whole reason we were looking at that was, on the Word side, for instance, a lot of the customer issues that we'd get where people would have corrupt files, they were corrupt because they there'd be some add-in that they had running or some third party app that was reading and writing word files. The files were fairly brittle and complex. The binary format... The binary format was written back in the days of floppy disks, right? So the top priority was how quickly can you write to a floppy disk and read from a floppy disc, right? It wasn't about, how easy is this for other people to go and read and write? Not because it was on purpose, make it hard. It was just the primary bid is let's get this thing so it's really easy to read and write from floppy, right? Brian Jones (00:35:31): And so in Word, we were like, "Wow, I think that there's a bigger opportunity here for an ecosystem around Word if we make it easy for people to read and write Word docs and build solutions around them." And so then the next release, the Excel team was looking at doing some big changes around a lot of the limitations, like how many rows you could have in columns, right. Lengths of like formulas and things like that. Right. And so there was this thing where the Excel team was like, "We are going to need to create a new file format." And on the word side, we thought this XML thing was great. We want to move to that as our new format. Brian Jones (00:36:01): And so everything kind of came together and it was clear. Hey, this is going to be the release that we are going to go and rev our file format, which we hadn't done in a while. This is also the release of the ribbon. So there were two really big major changes in that product, right? It was the new file format and the ribbon. It's funny. I still refer to it as the new file format, even though it's 15 years old. Rob Collie (00:36:23): Yeah. It's the new file format it's still new, yeah. Brian Jones (00:36:25): I still call it that, which is kind of nuts. But I think that the controversies you were talking about was really more of a... Boy, this is a really big deal for the product. We had changed file formats before in the past and not necessarily gotten it right. And there were a lot of challenges around compatibility and stuff. And so there was just a lot of worry of let's make sure you all have your stuff together here, right? Like let's make sure that this doesn't in any way break, stop people from wanting to upgrade to the new version. But it went really well. The whole goal of it was let's get something that we think third parties can go and read and write, and this is going to help build an ecosystem. And a new ecosystem run Office. Office already had big ecosystem with VBA and COMM add-ons and stuff like that, right.? But we won't have this new ecosystem around our file formats as a thing. That's why we chose... There's a packaging layer, which is all zip based. So if people haven't played around with it that XLSX, you can just put a .zip at the end and double click it. And it's just a zip file. And you can see a whole bunch of stuff inside of it. Right? Rob Collie (00:37:23): Yeah. If you're listening, you haven't done that go right now, run don't walk, grab an Excel file or a Word file, whatever. Go and rename the XLSX or BPTX, go ahead and rename it so that it ends in .zip and then open it up and you'll be blown away. Thomas Larock (00:37:38): PowerPoint is my favorite when I have to find some unknown setting that I need and I can just search through the whole thing. Yep. Rob Collie (00:37:45): Or all the images. You want to get all the images out of the PowerPoint file. It's just a zip file that has a bunch of images in it. Right. Brian Jones (00:37:50): So I also did this for backpack. It's the same thing. You can crack open the backpack by renaming a zip file... Thomas Larock (00:37:58): An actual physical backpack? What are we... what are we talking about here? Brian Jones (00:38:03): Ah yeah. Rob Collie (00:38:03): This is the digital acetate that is over the top of the entire physical world that you aren't aware of. Thomas Larock (00:38:08): Digital acetate, that's it? That's it. That's where the podcast peaks. Right? Those two words. We're all going home now. Brian Jones (00:38:19): Yeah. No. A SQL server, there's DAC pack, which is just the, say database schema. Then there's a backpack which has the data and the schema combined. But you can, if you rename them . zip, you can crack them open to see the XML that makes up those forms. So it's not just office products. Rob Collie (00:38:37): We ended up standardizing the entire thing, but that packaging format, it was called OPC, Open Packaging Convention, or something like that. It was something that we did in partnership with a Windows team. It's part of the final ISO standard for our file format. And then there were a lot of other folks that went and used that exact same standard. Because it's a really easy way of you have a zip package. You can have a whole bunch of pieces inside of it, which are XML. And then there's this convention for how you can do relationships between the different pieces. So I can have a slide. That's an XML and it can declare relationships to all the images that it uses. And that way it's really quick, easy to know, okay, here's all the content I need to grab if I want to move pieces of it outside of the file. Rob Collie (00:39:16): So the single coolest thing I've ever done with, we'll just call it your file format Brian. We'll just pretend that it was only you working on that. Brian Jones (00:39:23): Just me yeah, I was pretty busy, but yeah. Rob Collie (00:39:27): So the very, very first version of Power Pivot, first of all, your file format, the new file format made Power Pivot possible. We needed to go and add this gigantic binary stream of compressed data and everything, everything about Power Pivot needed to be saved in the file. At the beginning of the project, everyone was saying, "Oh, no, we're going to save it as two separate files." And I'm like, "Are you guys kidding?" The Pivot cache, for instance, is saved in the same file. You can't throw a multi file solution at people and expect it to... This was actually like Manhattan project, just to get that stream saved into the same file. It was pretty crazy. However, when it was done, there was something really awesome I wasn't aware of until the very end, which was, first of all, you could open up a zip file and just tunnel down and you would find a file in there called item one.data. Rob Collie (00:40:21): Okay. That was the Power Pivot blob. That was everything about the Power Pivot thing. And it was by far the biggest thing in the file, like it was like 99% of the file size was what was there. However, as this backup, someone had decided, I had nothing to do with this, to save all of the instructions. I think it's called XML for analysis XMLA. All of the instructions that would be required to rebuild exactly that file, but without any of the actual binary data in it. So it was a very, very small amount of XML. Okay. So here's what we would do because there were no good automation, no interfaces, no APIs. If we needed to add like 500 formulas to a Power Pivot file, you could go through the UI and write those 500 formulas, type, click, type, click, type, click. Rob Collie (00:41:08): Okay. So what we would do, and my first job outside of Microsoft, is we would go in there and we would edit that XML backup and add all the formulas we wanted in it. And by the way, I would use Excel to write these formulas. I would use string concatenation and all of that kind of stuff to write these things. It was very, very, very sensitive, one character out of place in the whole thing fails. So you make those changes. You save the file, reopen it, nothing happens because it's just the backup. Okay. So then you've got to go and you've got to create a zero byte item one.data file on your desktop and you copy it into the zip file and overwrite the real item one.data, therefore deliberately corrupting the primary copy. So when you reopen the file it triggers the backup process and it rehydrates with all of your stuff, it was awesome. Rob Collie (00:41:57): And then a couple of releases of Power Pivot later, suddenly that didn't work anymore and I was really pissed. But it just really shows you, it opens up so many opportunities that you never would have expected. And even a hack like that, that's not the kind that you'd be really looking for, but the fact that something like that even happens as a result of this is really indicative of what a success it was. Brian Jones (00:42:19): Yeah. I mean, there's a lot of those things where, I love building platforms, like that's my favorite part of the job. It's all those things that you see people do that you never would have predicted. Right? That's just so exciting. PowerPoint had this huge group of folks that would go and build things like doc assembly stuff, right. Where they go and automatically build PowerPoint decks on demand, right? Based on who you're going to go and present to cause they've just shredded the thing. In fact, when we did the ISO standardization, it was a 6,000 page doc that we had to go. And we built and reviewed with a standards body and we did it over about a year. Which sounds nuts, a 6,000 page doc in about a year. And the way that we were able to do that is there was never really a 6,000 page doc. There's a database where there's a row for every single element and attribute in this, in the whole schema, that would then have the column which is the description, which would just be the word XML. Brian Jones (00:43:09): And so we could, on demand, at any point, generate whatever view or part of the doc we wanted. So we'd say, "Hey, we're going to go in now, review everything that has to do with formatting across Word, Excel and PowerPoint." And so we just click a couple buttons and the database would spit out a Word doc that was just that part. Everybody could go and edit it cause we were using the structured elements we'd added to Word, which is called content controls, which was the next version of that XML stuff that we had to deprecate. And then the process, as soon as you'd finish editing that Word doc, we just submit it back. The process would go back and shred that Word doc again and put it all back in the database. And so we really used the file format to bootstrap documenting the file format. Rob Collie (00:43:48): And then when you dump a 6,000 page document on someone, they have no choice. But to just say, yep, it looks good to us. Brian Jones (00:43:55): Well, there was a pretty, incredibly thorough review still. It was just pretty impressive. The final vote that we had in Geneva, the process leading up to that, the amount of feedback that we got. Cause basically the ISO, you can kind of think of it like the UN, you go and show up and every country has a seat, right? I mean, not everybody participates, but anybody that wants to can. And so yeah, we had to respond to thousands of comments around different pieces, things that people wanted to see changed. Rob Collie (00:44:22): Yeah. I can imagine, right. Think about it. You just said at the final vote in Geneva. That's a heavy moment man. Thomas Larock (00:44:29): Yeah. That threw me off for a second. I thought, for sure, you were talking Switzerland, but now thinking that was just a code name. Rob Collie (00:44:38): No, I think, I think he was actually in Switzerland. Brian Jones (00:44:40): In Switzerland. Rob Collie (00:44:41): Have you seen the chamber where they do these votes? It looks just like the Senate from episode one of Star Wars. It's just like that. It's pretty heavy. Brian Jones (00:44:51): The little levitating... Rob Collie (00:44:53): The floating lift. Yeah. I think they call that digital acetate. I think that's what they call that. By the way on the Excel team, the way I came to look at the new file format and the open architecture of it, again, this this will show you how quickly I had turned into the more cynic side of things. Well, okay. We're going to be changing file formats. And we're doing that for our benefit because we didn't have enough bits allocated in the 1980s version of the file format that was saved to floppy disc, as you pointed out, right. Who could ever imagine having more than 64,000 rows, it's just inconceivable or 250 columns or whatever, right.? Because we hadn't allocated that. We'd made an engineering mistake, essentially, we hadn't future-proofed. So we need to make a file format change for our benefit, right. To undo one of our mistakes. And the way I looked at it was, "Ooh, all this open file format stuff, that'll be like the 'Look, squirrel!'" To distract people and to sort of justify, while we went and did this other thing, which, ultimately it actually went pretty well. The transition for the customers actually wasn't nearly as bad, because we actually Took it seriously. Rob Collie (00:46:03): The transition for the customers actually wasn't nearly as bad because we actually took it seriously. We didn't cut any corners. We did all the right things. Brian Jones (00:46:07): Well, there were several benefits too. We were talking about all the kind of ecosystem development benefits, but the fact that the file was zipped and compressed right, it meant that the thing was smaller. And that was all of a sudden, it was no longer about floppy discs. People are sharing files on networks. And so actually being able to go and have a file that's easier to share, send over network because it's smaller was a thing. Brian Jones (00:46:26): There were a couple of things that we were able to go and highlight. There's also a pretty nice thing where it was actually more robust because it was XML, and we split it into multiple pieces of XML. It meant that even if you had bit rot, you would only lose one little piece of the file, whereas with old the binary format, you had some bit rot and the whole thing is impossible to open up.There are a couple of things that were in user benefits too, which helped. Rob Collie (00:46:50): And ultimately, on the Excel side, the user got a million row spreadsheet format and the ability to use a hell of a lot more than like 14 colors that could be used in a single spreadsheet or something. It was .like a power of two minus two, so many bizarre things. Like Excel had more colors than that, but you couldn't use more than a certain subset in a- Brian Jones (00:47:10): At a time, yeah. Rob Collie (00:47:10): -In a single file. So yeah, there were a lot of benefits. They just weren't- Brian Jones (00:47:15): It's not like it's an explicit choice. It's just that at the time somebody is implementing something, you're right in a way, assuming, "Oh, this is fine. This is enough. I'll never have to worry with this issue." Rob Collie (00:47:25): Why waste the whole byte on that? When you can cram four different settings into a single byte. If you read the old stories about Gates and Allen programming up at Harvard, they had these vicious head-to-head competitions to see who could write the compiler or the section of basic in the fewest bytes possible. This was still very much hanging over Microsoft, even the vestiges of it were still kind of hanging over us even when I arrived. But certainly in the '80s when the Excel file format was being designed for that rev, it was still very much like, "Why waste all those bits in a byte?" "Let's cap it at four bits". Thomas Larock (00:48:05): In that blog series from Sinofsky, he talks a lot about that at the early start. And I'm at a point now where he's talking a lot about the code reuse because the Excel team, the Word team, I guess PowerPoint, but all these other teams, were all dealing with, say, text. And they were all doing their own code for how that text would be displayed and shown. And Bill would be the one being like, " This is ridiculous". "We should be able to reuse the code between these products". And to me, that would just be common sense. But these groups, Microsoft just grew so rapidly so quickly, they were off on their own, and they have to ship. I ain't got time to wait around for this, for somebody to build an API, things like that. I'll just write it myself. Brian Jones (00:48:50): It's a general thing that you get as you get larger where the person in charge that can oversee everything is like, "Well, these are all my resources", and, "Wow, I don't want three groups all building the same thing". But then when you get down, there's also a reality of we're just going to have a very different view on text and text layout than Excel. And Excel is not going to say, "I want all of that code that Word uses to lay out all of their content to be running for every single cell". Right? That's just suboptimal. And so it's always this fun conversation back and forth around where do you have shared code and reuse and where do you say it's okay for this specific app to have this more optimized thing that might look the same, but in reality, it's not really the same. Rob Collie (00:49:33): Brian, do you remember the ... I'm sure you do, but I don't remember what company they were from. But at one point in this file format effort, these really high priced consultants showed up and went around and interviewed us a couple of times. Do you remember that phase? It was like- Brian Jones (00:49:51): Was that towards the end? There was a couple summary stories that were pulled together just to talk about the overall processes. It was actually after the standardization. Rob Collie (00:49:58): I remember this being at the point in time where it was still kind of a question. whether we should do it. Brian Jones (00:50:02): I don't remember that. Rob Collie (00:50:04): The thing I remember really vividly is a statement that Chris Pratley would make over and over again, this encapsulated it for me. I came around to seeing it his way, which was the file format isn't the thing. That's not the moat. The thing that makes Office unique is the behaviors of the application. It's not the noun of the file format. It's the verb of what happens in the app. It's instructed to think that even if you took exactly the Excel team today, every single person that's already worked on it, and said, "Hey, you have to go rebuild Excel exactly". There's no way that version of Excel would be compatible with the one we have now. It would drift so much. Rob Collie (00:50:43): You could even have access to all the same specs. We would even cheat and say, "Look, you can have access to every single spec ever written". So? It was clearly someone had thought it was time to bring in like a McKinsey. They were all well dressed. They were all attractive. They were all a little too young to be the ones sort of making these decisions. It was just really weird to have them show up, three people in your office. Like, "Okay, I'll tell you what's going on". Brian Jones (00:51:11): I can totally imagine. It's funny I don't remember that. There were several rounds of analysis on how we were doing it, what we're doing and making sure we were doing it the right way. But yeah, Chris is spot on. I mean, your point about rebuilding it, that's essentially what we've been going through for the past five plus years around our web app. It's a lot of work. Unfortunately, we can't let it drift. The expectation from everybody is, "Hey, I learned the Wind 32 version. When I go to the web, I want it to feel the same. I don't want to feel like I'm now using some different app." Rob Collie (00:51:44): What an amazing, again, like a Manhattan project type of thing, this notion of rewriting Excel to run on the web and be compatible. Brian Jones (00:51:55): Yeah, with 30 years of innovation. Rob Collie (00:51:56): Yeah. That started in the 2007 release. Excel services, the first release of Excel services was 2007. And this whole thing about shared code, like what features, what functions of Excel, what pieces of it were going to be rewritten to be quote unquote "shared code"? And shared code meant it was actually server safe, which none of regular desktop Excel written in the early '80s, still carrying around assembly in certain places, assembly code of all things, right? Excel was not server safe. It was about as far from server safe as you could get. And so to rewrite this so ambitious without breaking anything. Oh my God. What a massive ... This dates back, gosh, more than 15 years. Brian Jones (00:52:45): Yeah. I'd say like the first goals around it were a bit different, right? It wasn't a web version of Excel. It was like BI scenarios and how can we have dashboards and Excel playing a role in dashboards. But yeah, I'd say since I joined, it was probably maybe a half a year or a year into when I joined, we just made the decision to shift a huge chunk of our funding to the web app. It was just clear that we need to make even more rapid progress. If you go, we have a site where you can go and see all the features that are rolling out there. It's incredible. And it's just because of the depth of the product. "Wow that's so many features you've done. You must be almost done". But then you look at everything else that's still isn't done yet. Brian Jones (00:53:23): Now thankfully, we're getting to the point where we can look at telemetry and say, "Hey, we've got most people covered." Most users, when we look at what they do in Windows, they could use the web app and shouldn't notice a difference. But there still is a set of things that we're going to keep churning through. So that'll continue to be a huge, huge investment for us. But yeah, the shared code strategy, we have an iPhone version, an iPad version an Android version. We've got Excel across all platforms. And because of the shared code, when we add new features, the feature crew that's working on that, they need to have a plan for how they're going to roll out across all those platforms, clearly levered shared code. But they also need to think through user experience and stuff like that too. Clearly a feature on a phone is going to behave differently than it's going to behave on a desktop. Rob Collie (00:54:05): Part of me, just like, kind of wants to just say, "I don't even believe that you've pulled that off, there's no way". It's kind of like, I've never looked at the Android version, and until I look at the Android version, I'm just going to assume it's not real. This is why it's one of the hardest things imaginable to have a single code base with all these different user experience, just fundamental paradigms of difference between these platforms. Like really? Come on. Brian Jones (00:54:34): It was a massively ambitious project. Mac shifted over maybe three years ago. And that's when, all of a sudden, in addition to a bunch of just features that people have been asking for that we'd never been able to get to, the massive one there was we were able to roll out the co-authoring multiplayer mode for Excel. Rob Collie (00:54:50): Multiplayer. Brian Jones (00:54:52): That's the term I like for co-authoring. It's more fun. Rob Collie (00:54:55): Yeah. It's like MMO for spreadsheets. Brian Jones (00:54:57): Yes. We were able to get that for the Mac. I mean, all of our platforms. One of us can be on an iPad, an iPhone, the web app, and we'll all see what we're doing in real time, making edits and all of that stuff. That alone, if you want to talk about massive projects, 30 years of features and innovation, basically that means we had to go and teach Excel how to communicate to another version of Excel and be very specific about, "This is what I did." "Here's the action I took." And that is massive. There are thousands and thousands of things you can do in the product. So getting it so that all of those versions are in sync the entire time, and so we're all seeing the exact same results of calc and all of that. That itself was a huge, massive project. Rob Collie (00:55:37): Take this as the highest form of praise when I say I don't buy it. I can't believe it. Brian Jones (00:55:44): I hope everybody's okay that we just talked for like an hour on just like listening to somebody at a high school reunion, I think, or something. Is this like me talking about how great I played in that one game? And you're like, "Yeah, that was a great basket". Rob Collie (00:55:54): Yeah. "Man, my jumper was on". the thing that's hard to appreciate, I think, is that you got to come back to the fact that we're talking about the tools that everyone in the world uses every day, that we rely on. And I think being gone from Microsoft for the last 12 years, I'm able to better appreciate that sense of wonder. This isn't just you and I catching up, I don't think. People enjoy, for good reason I think, hearing the stories of how these things came to be. People don't know by default how hard it was to get to a million rows in the file format. If you're like a robot, you're like, "I don't care how I got here. I just care what it is", then you're not listening to this show. We call it data with a human element. Robots can exit stage left. I think you should feel zero guilt. This isn't just self-indulgence. Brian Jones (00:56:55): Well, on the off chance everybody else ... I've listened to a lot of Rob's other podcasts, and they're awesome. So if you're bored with this one, it's okay. Go check out some of the other ones. They'r

Geeks on Screens with Coffee Episode 59 - BiML Queen Look who I bumped into at the water cooler/tea room/corridor? It's only Cathrine Wilhelmsen (@cathrinew ) Bio: Cathrine loves teaching and sharing knowledge. She is based in Norway and works as a Senior Business Intelligence Consultant in Inmeta, focusing on Data Warehousing, Data Integration, Analytics, and Reporting projects. Her core skills are Azure Data Factory, SSIS, Biml and T-SQL development, but she enjoys everything from programming to data visualization. Outside of work she's active in the SQL Server community as a Microsoft Data Platform MVP, BimlHero Certified Expert, author, speaker, blogger, organizer and chronic volunteer. I must dash, the kettle has almost boiled. Love you! Youtube: https://youtu.be/F6fJDOb4xz8

Geeks on Screens with Coffee Episode 60 - Wild SQL Look who I bumped into at the water cooler/tea room/corridor? It's only Gail Shaw (@SQLintheWild ) Bio: Gail is a Database Specialist focusing on database performance tuning and database recovery, with a particular interest in topics such as indexing strategies, execution plans, and writing T-SQL code that performs well and scales gracefully. Gail is a Data Platform MVP, a Pluralsight author, writes articles for both SQLServerCentral.com and Simple-Talk.com, and has spoken at SQL Saturdays locally and internationally, SQL Bits and at the PASS Community Summit. Gail has been responsible for extending the lifespan of many an application by performance-tuning their databases as well as providing technical guidance on all things SQL Server related. I must dash, the kettle has almost boiled. Love you! Youtube: https://youtu.be/k1oNvDaLSZo

We're coming up on the middle of the year, and I wonder if you are improving your career this year? Did you make plans and are you sticking to them? I've been tracking progress on my goals, and so far mediocre progress. It's tough to learn, but even tougher to learn when you don't have a specific focus. Choosing to "get better at T-SQL" isn't a good goal. It's vague and hard to measure. To me, you need to pick some specific areas to focus on and then develop some goals on what you want to accomplish. For example, you might want to learn to work with APPLY and convert some queries from joins to APPLY or pick a series of challenging problems that you can solve with that operator. Read the rest of Learning Intentionally with a List

Abstract of the talk… Are you a seasoned T-SQL developer, used to solve each and every challenge by writing plain old SQL? But, now you need to leverage data coming from semi-structured or unstructured sources? What if I tell you that you can get your mission accomplished by writing your favorite T-SQL syntax? In this session, you will learn what is a Serverless SQL pool within Azure Synapse Analytics, how it works behind the scenes, and how can you preserve your "T-SQL Ninja" status even when dealing with the data coming from CSV and Parquet files, or from NoSQL database. Bio… I'm making music from the data! PowerBI and SQLServer addict, MCT, Pluralsight Author, blogger, speaker...Interested in everything related to data - always eager to extract valuable info from raw data in the most effective way. Multi-year experience working with (predominantly) Microsoft Data Platform (SQL Server, SSAS, SSIS, SSRS, and Power BI). Father of 2 and true football (and Barca) fan!

About Patrick McFadinPatrick McFadin is the VP of Developer Relations at DataStax, where he leads a team devoted to making users of Apache Cassandra successful. He has also worked as Chief Evangelist for Apache Cassandra and consultant for DataStax, where he helped build some of the largest and exciting deployments in production. Previous to DataStax, he was Chief Architect at Hobsons and an Oracle DBA/Developer for over 15 years.Twitter: @PatrickMcFadinLinkedIn: Patrick McFadin DataStax website: datastax.comK8ssandra: k8ssandra.ioStargate: stargate.ioDataStax Astra: Cassandra-as-a-ServiceWatch this episode on YouTube: https://youtu.be/-BcIL3VlrjEThis episode sponsored by CBT Nuggets and Fauna.TranscriptJeremy: Hi everyone, I'm Jeremy Daly and this is Serverless Chats. Today I'm chatting with Patrick McFadin. Hey Patrick, thanks for joining me.Patrick: Hi Jeremy. How are you doing today?Jeremy: I am doing really well. So you are the VP of Developer Relations at DataStax, so I'd love it if you could tell the listeners a little bit about yourself and what DataStax is all about.Patrick: Sure. Well, I mean mostly I'm just a nerd with a cool job. I get to talk about technology a lot and work with technology. So DataStax, we're a company that was founded around Apache Cassandra, just supporting and making it awesome. And that's really where I came to the company. I've been working with Apache Cassandra for about 10 years now. I've been a part of the project as a contributor.But yeah, I mean mostly data infrastructure has been my life for most of my career. I did this in the dotcom era, back when it was really crazy when we had dozens of users. And when that washed out, I'm like, oh, then real scale started and during that period of time I worked a lot in just trying to scale infrastructure. It seems like that's been what I've been doing for like 30 years it seems like, 20 years, 20 years, I'm not that old. Yeah. But yeah, right now, I spend a lot of my time just working with developers on what's next in Kubernetes and I'm part of CNCF now, so yeah. I just can't to seem to stay in one place.Jeremy: Well, so I'm super interested in the work that DataStax is doing because I have had the pleasure/misfortune of managing a Cassandra ring for a start-up that I was at. And it was a very painful process, but once it was set up and it was running, it wasn't too, too bad. I mean, we always had some issues here and there, but this idea of taking a really good database, because Cassandra's great, it's an excellent data store, but managing it is a nightmare and finding people who can manage it is sort of a nightmare, and all that kind of stuff. And so this idea of taking these services and DataStax isn't the only one to do this, but to take these open-source services and turn them into these hosted solutions is pretty fantastic. So can you tell me a little bit more, though? What this shift is about? This moving away from hosting your own databases to using databases as a service?Patrick: Yeah. Well, you touched on something important. You want to take that power, I mean Cassandra was a database that was built in the scale world. It was built to solve a problem, but it was also built by engineers who really loved distributed computing, like myself, and it's funny you say like, "Oh, once I got it running, it was great," well, that's kind of the experience with most distributed databases, is it's hard to reason around having, "Oh, I have 100 mouths to feed now. And if one of them goes nuts, then I have to figure it out."But it's the power, that power, it's like stealing fire from the gods, right? It's like, "Oh, we could take the technology that Netflix and Apple and Facebook use and use it in our own stuff." But you got to pay the price, the gods demand their payment. And that's something that we've been really trying to tackle at DataStax for a couple of years now, actually three, which is how ... Because the era of running your own database is coming to an end. You should not run your own database. And my philosophy as a technologist is that proper, really important technology like your data layer should just fade into the background and it's just something you use, it's not something you have to reason through very much.There's lots of technology that's like that today. How many times have you ... When was the last time you managed your own memory in your code?Jeremy: Right. Right. Good point. I know.Patrick: Thank god, huh?Jeremy: Exactly.Patrick: Whew.Jeremy: But I think that you make a really good point, because you do have these larger companies like Facebook or whatever that are using these technologies and you mentioned data layers, which I don't think I've worked for a single company, I don't think I actually ... I founded a start-up one time and we built a data layer as well, because it's like, the complexity of understanding the transaction models and the routing, especially if you're doing things like sharding and all kinds of crazy stuff like that, hiding that complexity from your developers so that they can just say, "I need to get this piece of information," or, "I need to set this piece of information," is really powerful.But then you get stuck with these data layers that are bespoke and they're generally fragile and things like that, so how is that you can take data as a service and maybe get rid of some of that, I don't know, some of that liability I guess?Patrick: Yeah. It's funny because you were talking about sharding and things like that. These are things that we force on developers to reason through, and it's just cognitive load. I have an app to get out, and I have some business desire to get this application online, the last thing I need to worry about is my sharding algorithm. Jeremy, friends don't let friends shard.Jeremy: Right. That's right. That's a good point.Patrick: But yeah, I mean I think we actually have all the parts that we need and it's just about, this is closer than you think. Look at where we've already started going, and that is with APIs, using REST. Now GraphQL, which I think is deserving its hotness, is starting to bring together some things that are really important for this kind of world we want to live in. GraphQL is uni-fettering data and collecting and actual queries, it's a QL, and why they call it Graph, I have no idea. But it gives you this ability to have this more abstract layer.I think GraphQL will, here's a prediction is that it's going to be like the SQL of working with data services on the internet and for cloud-native applications. And so what does that mean? Well, that means I just have to know, well, I need some data and I don't really care what's underneath it. I don't care if I have this field indexed or anything like that. And that's pretty exciting to me because then we're writing apps at that point.Jeremy: Right. Yeah. And actually, that's one of the things I really like about GraphQL too is just this idea that it's almost like a universal data access layer in a sense because it does, you still have to know it, you have to know what you're requesting if you're an end developer, but it makes it easier to request the things that you need and have those mutations set and have some of those other things standardized across the company, but in a common format because isn't that another problem? Where it's like, I'm working with company A and I move to company B maybe and now company B is using a different technology and a different bespoke data layer and some of these other things.So, I think data as a service for one, maybe with GraphQL in front of it is a great way to have this alignment across companies, or I guess, just makes it easier for developers to switch and start developing right away when they move into a new company.Patrick: Yeah, and this is a concept I've been trying to push pretty hard and it's driven by some conversations I've had with some friends that they're engineering leaders and they have this common desire. We want to have a zero day dev, which is the first day that someone starts, they should be producing production code. And I don't think that's crazy talk, we can do this, but there's a lot of things that are in front of it. And the database is one of them. I think that's one of the first things you do when you show up at company X is like, "Okay, what database are you using? What flavor of SQL or GRPC or CQL, Cassandra query language? What's the data model? Quick, where's that big diagram on the wall with my ERD? I got to go look at that for a while."Jeremy: How poorly did you structure your Git repositories? Yeah.Patrick: Yeah, exactly. It's like all these things. And no, I would love to see a world where the most troublesome part of your first day is figuring out where the coffee and the bathroom are, and then the rest of it is just total, "Hey, I can do this. This is what I get paid to do."Jeremy: Right. Yeah. So that idea of zero day developer, I love that idea and I know other companies are trying to do that, but what enables that? Is it getting the idea of having to understand something bespoke? Is it getting that off of the table? Or not having to deal with the low-level database aspect of things? I mean because APIs, I had this conversation with Rob Sutter, actually, a couple weeks ago. And we were talking about the API economy and how everything is moving towards APIs. And even data, it was around data as well.So, is that the interface, you think, of the future that just says, "Look, trying to interface directly with a database or trying to work with some other layer of abstraction just doesn't make sense, let's just go straight from code right to the data, with a very simple API interface?"Patrick: Yeah, I think so. And it's this idea of data services because if you think of if you're doing React, or something like a front-end code, I don't want to have a driver. Drivers are a total impediment. It's like, driver hell can be difficult at large organizations, getting the matching right. Oh, we're using this database so you have to use this driver. And if you don't, you are now rejected at the gate. So it's using HTTP protocols, but it's also things like when you're using React or Angular, View, whatever you're using on the front-end, you have direct access.But most times what you're needing is just a collection or an object. And so just do a get, "I need this thing right now. I'm doing a pick list. I need your collection." I don't need a complicated setup and spend the first three days figuring out which driver I'm using and make sure my Gradle file is just perfect. Yeah. So, I think that's it.Jeremy: Yeah. No, I'd be curious how you feel about ORMs, or O-R-Ms, certainly for relational databases, I know a lot of people love them. I can't stand them. I think it adds a layer of abstraction and just more complexity where I just want access to the database. I want to write the query myself, and as soon as you start adding in all this extra stuff on top of it to try to make it easier, I don't know, it just seems to mess it up for me.Patrick: All right. So yeah, I think we have an accord. I am really not a fan of ORMs at all. And I mean this goes back to Hibernate. Everyone's like, "Oh, Hibernate's going to be the end of databases." No, it's not. Oh yeah, it was the end of the database at the other side because it would create these ridiculous queries. It's like, why is every query a full table scan?Jeremy: Exactly.Patrick: Because that's the way Hibernate wanted it. Yeah. I actually banned Hibernate at one company I was working at. I was Chief Architect there and I just said, "Don't ever put Hibernate in our production." Because I had more meetings about what it was doing wrong than what it was doing right.Jeremy: Right. Right. Yeah. No, that's sounds, yeah.Patrick: Is that a long answer? Like, no.Jeremy: No, I've had the same experience where certain ORMs you're just like, no. Certain things, you can't do this because it's going to one, I think it locks you in in a sense, I mean there's all kind of lock-in in the cloud, and if you're using a data service or an API or you're using something native in AWS, or IBM Cloud, you're still going to be locked in in some way, but I do feel like whenever you start going down that path of building custom things, or forcing developers to get really low level, that just builds up all kinds of tech debt, right? That you eventually are going to have to work down.Patrick: Well, it's organizational inertia. When you start getting into this, when you start using annotations in Hibernate where you're just cutting through all the layers and now you're way down in the weeds, try to move that. There's a couple of companies that I've worked with now that are looking at the true reality of portability in their data stores. Like, "Oh, we want to move from one to a different, from a key value to a document without developers knowing." Well, how do you get to that point?Jeremy: Right. Yeah.Patrick: And it's just, that's not giving access to those things, first of all, but this is that tech debt that's going to get in your way. We're really good, technologists, we're really good at just wracking up the charges on our tech debt credit card, especially whenever we're trying to get things out the door quickly. And I think that's actually one of the problems that we all face. I mean, I don't think I've ever talked to a developer who was ahead of schedule and didn't have somebody breathing down their neck.Jeremy: Very true.Patrick: You take shortcuts. You're like, "We've got to shift this code this week. Skip the annotations and go straight into the database and get the data you need." Or something. You start making trade-offs real fast.Jeremy: What can we hard code that will just get us past.Patrick: Yeah. Is it green? Shift it. Yeah.Jeremy: Yeah, no, I totally, totally agree. All right. So let's talk a little bit more about, I guess, skillsets and things like that. Because there are so many different databases out there. Cassandra is just one and if you're a developer working just at the driver level, I guess, with something like Cassandra, it's not horrible to work with. It's relatively easy once a lot of these things are set up for you.Same is true of MongoBD, or I mean, DynamoDB, or any of these other ones where the interface to it isn't overly difficult, but there's always some sort of something you want to build on top of it to make it a little bit easier. But I'm just curious, in terms of learning these different things and switching between organizations and so forth, there is a cognitive load going from saying, "I'm working on Cassandra," to going to saying, "I'm working on DynamoDB," or something like that. There's going to be a shift in understanding of how the data can be brought back, what the limitations are, just a whole bunch of things that you kind of have to think about. And that's not even including managing the actual thing. That's a whole other thing.So, hiring people, I guess, or hiring developers, how much do we want developers to know? Are you on board with me where it's like, I mean I like understanding how Cassandra works and I like understanding how DynamoDB works, and I like knowing the limits, but I also don't want to think about them when I'm writing code.Patrick: Yeah. Well, it's interesting because Cassandra, one of the things I really loved about Cassandra initially was just how it works. As a computer scientist, I was like, "This is really neat." I mean, my degree field is in distributed computing, so of course, I'm going to nerd out.Jeremy: There you go.Patrick: But that doesn't mean that it doesn't have mass appeal because it's doing the thing that people want. And I think that's going to be the challenge of any properly built service layer. I think I've mentioned to you before we started this, I work on a project called Stargate. And Stargate is a project that is meant to build a data layer on top of databases. And right now it's with Cassandra. And it's abstracting away some of the harder to understand or reason things.For instance, with distributed computing, we're trying to reduce the reliance on coordination. There is a great article about this by Pat Helland about how coordination is the last really expensive thing that we have in development. Memory, CPU, super cheap. I can rent that all day long. Coordination is really, really hard, and I don't expect a new programmer to understand, to reason through coordination problems. "Oh, yeah, the just in time race conditions," and things like that.And I think that's where distributed computing, it's super powerful, but then whenever people see what eventual consistency are, they freak out and they're like, "I just want my SQL Lite on my laptop. It's very safe." But that's not going to get you there. That's not a global database, it's not going to be able to take you to a billion users. Come on, don't cut ...Jeremy: Maybe you don't need to be.Patrick: ... your apps short Jeremy. You're going to have a billion users.Jeremy: You should strive for it, at least, is how I feel about it. So that's, I guess, the point I was trying to get to is that if the developers are the ones that you don't want learning some of this stuff, and there's ways to abstract it away again, going like we talked about data as a service and APIs and so forth. And I think that's where I would love to see things shifting. And as you said earlier, that's probably where things are going.But if you did want to run your own database cluster, and you wanted to do this on your own, I mean you have to hire people that know how to do this stuff. And the more I see the market heating up for this type of person, there is very, very few specialists out there that are probably available. So how would you even hire somebody to run your Cassandra ring? They probably all work at DataStax.Patrick: No, not all of them. There's a few that work at Target and FedEx, Apple, the biggest Cassandra users in the world. Huawei. We just found out lately that Huawei now has the biggest cluster on the planet. Yeah. They just showed up at ApacheCon and said, "Oh yeah, hold my beer." But I mean, you're right, it's a specialized skillset and one of the things we're doing at DataStax, we feel, yeah, you should just rent that. And so we have Astra, which is our database as a service.It's fully compatible with open-source Cassandra. If you don't like it, you can just take it over and use open-source. But we agree and we actually can run Cassandra cheaper than you can, and it's just because we can do it at scale. And right now Astra, the way we run it is truly serverless, you only pay for what you need, and that's something that we're bringing to the open-source side of Cassandra as well, but we're getting Cassandra closer to Kubernetes internally.So if you don't want to think about Kubernetes, if you don't want to think about all that stuff, you can just rent it from us, or you could just go use it in open-source, either way. But you're right. I mean, it should not be a 2020s skillset is, "Get better at running Cassandra." I think those days should be, leave it to, if you want to go work at DataStax and run Cassandra, great, we're hiring right now, you will love it. You don't have to. Yeah.Jeremy: So the idea of it being open-source, so again, I'm not a huge fan of this idea of vendor lock-in. I think if you want to run on AWS Lambda, yeah, most of what you can do can only run on AWS Lambda, but changing the compute, switching that over to Azure or switching that over to GCP or something like that, the compute itself is probably not that hard to move, right? I think especially depending on what you're doing, setting up an entire Kubernetes cluster just to run a few functions is probably not worth it. I mean, obviously, if you've got a much bigger implementation, that's a little different.But with data, data is just locked in. No matter where you go, it is very hard to move a lot of data. So even with the open-source flair that you have there, do you still see a worry about lock in from a data side?Patrick: Yeah. And it's becoming more of a concern with larger companies too, because options, #options. There was a pretty famous story a few years ago where the CEO of Target said, "I am not paying Amazon any more money," and they just picked up shop and moved from AWS to Google Cloud. And the CEO made a technical decision. It was like everybody downstream had to deal with that. And I think that luckily Target's a huge Cassandra shop and they were just like, "Okay, we'll just move it over there."But the thing is that you're right, I mean, and I love talking about this because back when cloud was first starting and I was talking about it and thinking about it, just what do the clouds promise you? Oh, you get commodity scale of CPU and network and storage. And that's what they want to sell you because that what they're building. Those big buildings in north Virginia, they are full of compute network and storage, but the thing they know they need to hook you in and the way that they're hooking you in, there's some services that are really handy, they're great, but really the hook is the data.Once you get into the database, the bespoke database for the cloud, one of the features of that database is it will not connect to any other database outside of that cloud, and they know that. I mean, and this is why I really strongly am starting to advocate this idea of this move towards data on Kubernetes is a way where open-source gets to take back the cloud. Because now we're deploying these virtual data centers and using open-source technology to create this portability. So we can use the compute network and storage, a Google, Amazon, Azure, OnPrem wherever, doesn't matter.But you need to think of like, "All right. How is that going to work?" And that's why we're like, "If you rent your Cassandra from DataStax with Astra, you can also use the open-source Cassandra as well." And if we aren't keeping you happy, you should feel totally fine with moving it to an open-source workload. And we're good with that. One way or the other, we would love for you to use a database that works for you.Jeremy: Right. And so this Stargate project that you're working on, is that the one that allows you to basically route to multiple databases?Patrick: That's the dream. Right now it just does Cassandra, but there's been some really interesting ... There's some folks coming out of the woodwork that really want to bring their database technology to Stargate. And that's what I'm encouraged by. It's an open-source project, Stargate.io, and you can contribute any of the connectors for underlying data store, but if we're using GraphQL, if you're using GRPC, if you're using REST, the underlying data store is really somewhat irrelevant in that case. You're just doing gets and puts, or gets and sets. Gets and puts, yeah, that's right. Gets, sets, puts, it's a lot of words.Jeremy: Whatever words. Yeah. Exactly.Patrick: That's what I love about standard, Jeremy, there's so many to pick from.Jeremy: Right, because there are ... Exactly, which standard do you choose? Yeah. So, because that's an interesting thing for me too, is just this idea of, I mean, it would be great to live in a perfect little cloud where you could say like, "Oh, well AWS has all the services I need. And I can just keep all my stuff there, whatever." But best of breed services, or again, the cost of hosting something in AWS maybe if you're hosting a Cassandra cluster there, versus maybe hosting it in GCP or maybe hosting it with you, you said you could host it cheaper than those could, or that we could host it ourselves.And so I do think that there is ... and again, we've had this conversation about multi-cloud and things like that where it's not about agnostic, it's not about being cloud agnostic, it's about using the best of breed for any service that you want to use. And APIs seem to be the way to get you there. So I love this idea of the Stargate project because it just seems like that's the way where it could be that standard across all these different clouds and onto all these different databases, well I mean, right now Cassandra, but eventually these other ones. I don't know, that seems like a pretty powerful project to me.Patrick: Well, the time has come. It's cloud native ... I work a lot with CNCF and cloud-native data is a kind of emerging topic. It's so emerging that I'm actually in the middle of writing a book, an O'Reilly book on it. So, yeah. Surprise. I just dropped it. This just in.Yeah, because I can see that this is going to be the future, but when we build cloud-native, cloud applications, cloud-native applications, we want scale, we want elasticity, and we want self-healing. Those are the three cloud-native things that we want. And that doesn't give us a whole lot ... So if I want to crank out a quick REACT app, that's what I'm going to use. And Netlify's a great example, or Vercel, they're creating this abstraction layer. But Netlify and Vercel are both working, they've been partnering with us on the Stargate project, because they're seeing like, "Okay, we want to have that very light touch, developers just come in and use it," in building cloud-native applications.And whenever you're building your application, you're just paying for what you use. And I think that's really key, not spinning up a bunch of infrastructure that you get a monthly bill for. And that bill can be expensive.Jeremy: It seems crazy. Doesn't it seem crazy nowadays? Actually provisioning an EC2 instance and paying for it to run even if it does nothing. That seems crazy to me.Patrick: There are start-ups around the idea of finding the instance that's running that's causing you money that you're not using.Jeremy: Which is crazy, isn't it? It's crazy. All right. So let's go a little bit more into standards, because you mentioned standards. So there are standards now for a lot of things, and again, GraphQL being a great example, I think. But also from a database perspective, looking at things like TSQL and developers come into an organization and they're familiar with MySQL, or they're familiar with PostgreSQL, whatever it is. Or maybe they're familiar with Cassandra or something like that, but I think most people, at least from what I've seen, have been very, very comfortable with the TSQL approach to getting data. So, how do you bring developers in and start teaching them or getting them to understand more of that NoSQL feel?Patrick: I think it's already happened, it's just the translation hasn't happened in a lot of minds. When you go to build an application, you're designing your application around the workflows your application's going to have. You're always thinking about like, "I click on this. I go there." I mean, this is where we wireframe out the application. At that point, your database is now involved and I don't think a lot of folks know that.It's like, at every point you need to put data or get data. And I think this is where we've taught could be anybody building applications, which makes it really difficult to be like, "No, no, no, start with your data domain first and build out all those models. And then you write your application to go against those models." And I'll tell you, I've been involved in a few of these application boot camps, like JavaScript boot camps and things, they don't go into data modeling. It's just not a part of it.Jeremy: Really?Patrick: And I think this is that thing where we have to acknowledge like, "Yeah, we don't really need that anymore as much, because we're just building applications." If I build a React app, and I have a form and I'm managing the authentication and I click a button and then I get a profile information, I just described every database interaction that I need and the objects that I need. And I'm going to put my user profile at some point, I'm going to click my ID and get that profile back as an object. Those are the interactions that I need. At no point did I say, "And then I'm going to write select from where." No, I just need to get that data.Jeremy: And I love thinking about data as objects anyways. It makes more sense, rather than rows of spreadsheets essentially that you join together, describing an object even if it's got nested data, like a document form or things like that, I think makes a ton of sense. But is SQL, is it still relevant do you think? I mean, in the world we're moving into? Should I be teaching my daughters how to write TSQL? Or would I be wasting my time?Patrick: Yeah. Well, yes and no. Depends on what your kid's doing. I think that SQL will go to where it originally started and where it will eventually end, which is in data engineering and data science. And I mean, I still use SQL every once in a while, Bigtable, that sort of thing, for exploring my data. I mean for an analytics career or reporting data and things like that, SQL is very expressive. I don't see any reason to change that. But this is a guy who's been writing SQL for a million years.But I mean, that world is still really moving. I mean, like a Presto and Snowflake and all these, Redshift, they all use Bigtable, they all use SQL to express the reporting capabilities. But ... And I think this is how you and I got sucked into this is like, well that was the database that we had, so we started using reporting languages to build applications. And how'd that work out?Jeremy: Yeah. Well, it certainly didn't scale very well, I can tell you that, going back to sharding, because that is always something that was very hard to do. So I guess, I get the point that essentially if you're going to be in the data sciences and you actually need to analyze that data and maybe you do need to do joins, or maybe you need to work with big data in a way, that's a specialized aspect of it and I think people could dabble in that if they were just regular developers and they didn't want to go too deep.But it sounds like the bigger, or the end goal here, maybe altruistic, is to just give people access to data. So even if they don't know SQL or they don't know something complex, just make it so that whatever data is there that anybody, with whatever level is, they can consume it.Patrick: Yeah. And move fast with the thing that you're building. Actually, I use a Facebook term, but Facebook does do this. Internally there's a system called Occhio that provides gets and puts for your data, but it abstracts things like geographics and things like that. But the companies that are trying to move quickly, they understood this a long time ago. If you have to reason through, "Am I doing a full table scan? Is that an efficient interjoin?" If you have to reason through that, you're not moving fast anymore.Jeremy: Right. Right. All right. Cool. All right, so let's talk about Astra a little bit more and this whole idea of, because Astra is the serverless version, the hosted version, the serverless version of Cassandra, right? Through DataStax?Patrick: Right. And ...Jeremy: Did I get that right?Patrick: You got it right. And so it gives you full access. You could do Port 9042 if you still want to use a driver, but it gives you access via GraphQL, REST, and there's also a document API. So if you just want to persist your JavaScript API or JavaScript and then pull it back out your JSON, it does full documents. So it emulates what a MongoDB or DocumenDB does. But the important thing, and this is the somewhat revolutionary side of this, and again, this is something that we're looking to put into open-source, is the serverless nature of it.You only pay for what you use. And when you want to create a Cassandra database, we don't even call it a Cassandra database on the Astra panel anymore. We just create a database. You give it a name. You click. And it's ready. And it will scale infinitely. As long as we can find some compute and network for you to use somewhere, it'll just keep scaling and that's kind of that true portion of serverless that we're really trying to make happen. And for me, that's exciting because finally, all that power that I feel like I've been hoarding for a long time is now available for so many more people.And then if you do a million writes per second for 10 minutes and then you turn it off, you only pay for that little short amount of time. And it scales back. You're not paying a persistent charge forever.Jeremy: I'm just curious from a technical implementation, because I'm thinking about PTSD or nightmares back of my days running Cassandra, and so I'm just trying to think how this works. Is it a shared tenancy model? Or is there a way to do single tenancy if you wanted that as a service?Patrick: Under the covers, yes, it is multi-tenant, but the way that we are created ... so we had to do some really interesting engineering inside. So my RCO's going to kill me if I talk about this, but hey, you know what, Jeremy? We're friends, we can do this. He's like, "Don't talk about the underlying architecture." I'm talking about the underlying architecture. The thing that we did was we took Cassandra and we decomposed it into microservices mostly. That's probably, it's still Cassandra, it's just how we run it makes it way more amenable to doing multi-tenant and scale in that fashion where the queries are separated from the storage and things that are running in the background, like if you're familiar with Cassandra because it's a log structure storage, you ask to do compactions and things like that, all that's just kind of on the side. It doesn't impact your query.But it gives us the ability to, if you create a database and all of a sudden you just hammer it with a million writes per second, there's enough infrastructure in total to cover it. And then we'll spin up more in the back to cover everything else. And then whenever you're done, we retract it back. That's how we keep our costs down. But then the storage side is separated and away from the compute side, and the storage side can scale its own way as well.And so whenever you need to store a petabyte of Cassandra data, you're just storing, you're just charged for the petabyte of storage on disk, not the thousandth of a cluster that you just created. Yeah.Jeremy: No. I love that. Thank you for explaining that though, because that is, every time I talk to somebody who's building a database or running some complex thing for a database, there's always magic. Somebody has to build some magic to make it actually work the way everyone hopes it would work. And so if anybody is listening to this and is like, "Ah, I'm just getting ready to spin up our own Cassandra ring," just think about these things because these are the really hard problems that are great to have a team of people working on that can solve this specific problem for you and abstract all of that crap away.Patrick: Yeah. Well, I mean it goes back to the Dynamo paper, and how distributed databases work, but it requires that they have a certain baseline. And they're all working together in some way. And Cassandra is a share-nothing architecture. I mean you don't have a leader note or anything like that. But like I said, because that data is spread out, you could have these little intermittent problems that you don't want to have to think about. Just leave that to somebody else. Somebody else has got a Grafana dashboard that's freaking out. Let them deal with it. But you can route around those problems really easily.Jeremy: Yeah. No, that's amazing. All right. So a couple more technical questions, because I'm always curious how some of these things work. So if somebody signs up and they set up this database and they want to connect to it, you mentioned you could use the driver, you mentioned you can use GraphQL or the REST API, or the Document API. What's the authentication method look like for that?Patrick: Yeah. So, it's a pretty standard thing with tokens. You create your access tokens, so when you create the database, you define the way that you access it with the token, and then whenever you connect to it, if you're using JavaScript, there's a couple of collection libraries that just have that as one of the environment variables.And so it's pretty standard for connecting the cloud databases now where you have your authentication token. And you can revoke that token at any time. So for instance, if you mistakenly commit that into your Git ...Jeremy: Say GitHub. We've never done that before.Patrick: No judging. You can revoke it immediately. But it also gives you our back, the controls over it's a read or write or admin, if you need to create new tables and that sort of thing. You can give that level of access to whatever that token is. So, very simple model, but then at that point, you're just interacting through a REST call or using any of the HTTP protocols or SQL protocol.Jeremy: And now, can you create multiple tokens with different levels of permission or is it all just token gives you full access?Patrick: No, it's multiple levels of protection and actually that's probably the best way to do it, for instance, if your CI/CD system, has the ability to, it should be able to create databases and tear them down, right? That would be a good use for that, but if you have, for instance, a very basic application, you just want it to be able to read and write. You don't want to change any of the underlying data structures.Jeremy: Right. Right.Patrick: That's a good layer of control, and so you can have all these layers going on one single database. But you can even have read-only access too, for ... I think that's something that's becoming more and more common now that there's reporting systems that are on the side.Jeremy: Right. Right. Good.Patrick: No, you can only read from the database.Jeremy: And what about data backups or exporting data or anything like that?Patrick: Yeah, we have a pretty rudimentary backup now, and we will probably, we're working on some more sophisticated versions of it. Data backup in Cassandra is pretty simple because it's all based on snapshots because if you know Cassandra the database, the data you write is immutable and that's a great way to start when you come to backup data. But yeah, we have a rudimentary backup system now where you have to, if you need to restore your data, you need to put in a ticket to have it restored at a certain point.I don't personally like that as much. I like the self-service model, and that's what we're working towards. And with more granularity, because with snapshots you can do things like snapshot, this is one of the things that we're working on, is doing like a snapshot of your production database and restoring it into a QA cluster. So, works for my house, oh, try it again. Yeah.Jeremy: That's awesome. No, so this is amazing. And I love this idea of just taking that pain of managing a database away from you. I love the idea of just make it simple to access the data. Don't create these complex things where people have to build more, and if people want to build a data access layer, the data access layer should maybe just be enforcing a model or something like that, and not having to figure out if you're on this shard, we route you to this particular port, or whatever. All that stuff is just insane, so yeah, I mean maybe go back to kind of the idea of this whole episode here, which is just, stop using databases. Start using these data services because they're so much easier to use. I mean, I'm sure there's concerns for some people, especially when you get to larger companies and you have all the compliance and things like that. I'm sure Astra and DataStax has all the compliance things and things like that. But yeah, just any final words, advice to people who might still be thinking databases are a good idea?Patrick: Well, I have an old 6502 on a breadboard, which I love to play with. It doesn't make it relevant. I'm sorry. That was a little catty, wasn't it?Jeremy: A little bit, but point well taken. I totally get what you're saying.Patrick: I mean, I think that it's, what do we do with the next generation? And this is one of the things, this will be the thought that I leave us with is, it's incumbent on a generation of engineers and programmers to make the next generation's job easier, right? We should always make it easier. So this is our chance. If you're currently working with database technology, this is your chance to not put that pain on the next generation, the people that will go past where you are. And so, this is how we move forward as a group.Jeremy: Yeah. Love it. Okay. Well Patrick, thank you so much for sharing all this and telling us about DataStax and Astra. So if people want to find out more about you or they want to find out more about Astra and DataStax, how do they do that?Patrick: All right. Well, plenty of ways at www.datastax.com and astra.datastax.com if you just want the good stuff. Cut the marketing, go to the good stuff, astra.datastax.com. You can find me on LinkedIn, Patrick McFadin. And I'm everywhere. If you want to connect with me on LinkedIn or on Twitter, I love connecting with folks and finding out what you're working on, so please feel free. I get more messages now on LinkedIn than anything, and it's great.Jeremy: Yeah. It's been picking up a lot. I know. It's kind of crazy. Linked in has really picked up. It's ...Patrick: I'm good with it. Yeah.Jeremy: Yeah. It's ...Patrick: I'm really good with it.Jeremy: It's a little bit better format maybe. So you also have, we mentioned the Stargate project, so that's just Stargate.io. We didn't talk about the K8ssandra project. Is that how you say that?Patrick: Yeah, the K8ssandra project.Jeremy: K8ssandra? Is that how you say it?Patrick: K8ssandra. Isn't that a cute name?Jeremy: It's K-8-S-S-A-N-D-R-A.io.Patrick: Right.Jeremy: What's that again? That's the idea of moving Cassandra onto Kubernetes, right?Patrick: Yeah. It's not Cassandra on Kubernetes, it's Cassandra in Kubernetes.Jeremy: In Kubernetes. Oh.Patrick: So it's like in concert and working with how Kubernetes works. Yes. So it's using Cassandra as your default data store for Kubernetes. It's a very, actually it's another one of the projects that's just taking off. KubeCon was last week from where we're recording now, or two weeks ago, and it was just a huge hit because again, it's like, "Kubernetes makes my infrastructure to run easier, and Cassandra is hard, put those together. Hey, I like this idea."Jeremy: Awesome.Patrick: So, yeah.Jeremy: Cool. All right. Well, if anybody wants to find out about that stuff, I will put all of these links in the show notes. Thanks again, Patrick. Really appreciate it.Patrick: Great. Thanks, Jeremy.

I've been working on various skills over the last few years, trying to solve some simple problems in Python and PowerShell, in addition to T-SQL, to continue to improve my skills. It's an interesting challenge at times, especially when I need to use new features or functions to which I haven't had exposure in the past. I also work with R lightly, as I need to build some Questions of the Day for SQLServerCentral and I try to alternate Python and R questions every week. This has caused me to dig in and try to learn more about the language and how to manipulate data. Read the rest of Should You Learn R or Python?

About Rob SutterRob Sutter, a Principal Developer Advocate at Fauna, has woven application development into his entire career, from time in the U.S. Army and U.S. Government to stints with the Big Four and Amazon Web Services. He has started his own company – twice – once providing consulting services and most recently with WorkFone, a software as a service startup that provided virtual digital identities to government clients. Rob loves to build in public with cloud architectures, Node.js or Go, and all things serverless!Twitter: @rts_robPersonal email: rob@fauna.com Personal website: robsutter.comFauna Homepage Learn more about Fauna Supported Languages and Frameworks Try Fauna for FreeThe Calvin PaperThis episode sponsored by CBT Nuggets: https://www.cbtnuggets.com/Watch this video on YouTube: https://youtu.be/CUx1KMJCbvkTranscriptJeremy: Hi, everyone. I'm Jeremy Daly, and this is Serverless Chats. Today, I'm joined by Rob Sutter. Hey, Rob. Thanks for joining me.Rob: Hey, Jeremy. Thanks for having me.Jeremy: So you are now the or a Principal Developer Advocate at Fauna. So I'd love it if you could tell the listeners a little bit about your background and what Fauna is all about.Rob: Right. So as you've said, I'm a DA at Fauna. I've been a serverless user in production since 2017, started the Serverless User Group in Dubai. So that's how I got into serverless in general. Previously, I was a DA on the Serverless Team at AWS, and I've been a SaaS startup co-founder, a government employee, an IT auditor, and like a lot of serverless people I find, I have a lot of Ops in my background, which is why I don't want to do it anymore. There's a lot of us that end up here that way, I think. Fauna is the data API for modern applications. So it's a database that you access as an API just as you would access Stripe for payments, Twilio for messaging. You just put your data into Fauna and access it that way. It's flexible, serverless. It's transactional. So it's a distributed database with asset transactions, and again, it's as simple as accessing any other API that you're already accessing as a developer so that you can simplify your code and ship faster.Jeremy: Awesome. All right. Well, so I want to talk more about Fauna, but I want to talk about it actually in a broader ... I think in the broader ecosystem of what's happening in the cloud right now, and we hear this term "multicloud" all the time. By the way, I'm super excited to have you on. I wanted to have you on for the longest time, and then just schedules, and it's like ...Rob: Yeah.Jeremy: You know how it is, but anyways.Rob: Thank you.Jeremy: No. But seriously, I'm super excited because your tweets, and everything that you've written, and the things that you were doing at AWS and things like that I think just all reinforced this idea that we are living in this multicloud world, right, and that when people think of multicloud ... and this is something I try to be very clear on. Multicloud is not cloud-agnostic, right?Rob: Right.Jeremy: It's a very different thing, right? We're not talking about running the same work load in parallel on multiple service providers or whatever.Rob: Right.Jeremy: We're talking about this idea of using the best services that are available to you across the spectrum of providers, whether those are cloud service providers, whether those are SaaS companies, or even to some degree, some open-source projects that are out there that make up this strategy. So let's start there right from the beginning. Just give me your thoughts on this idea of what multicloud is.Rob: Right. Well, it's sort of a dirty word today, and people like to rail against it. I think rightly so because it's that multicloud 1.0, the idea of, as you said, cloud-agnostic that "write once, run everywhere." All that is, is a race to the bottom, right? It's the lowest common denominator. It's, "What do I have available on every cloud service provider? And then let me write for that as a risk management strategy." That's a cost center when you want to put it in business terms.Jeremy: Right.Rob: Right? You're not generating any value there. You're managing risk by investing against that. In contrast, what you and I are talking about today is this idea of, "Let me use best in class everywhere," and that's a value generation strategy. This cloud service provider offers something that this team understands, and wants to build with, and creates value for the customer more quickly. So they're going to write on that cloud service provider. This team over here has different needs, different customers. Let them write over there. Quite frankly, a lot of this is already happening today at medium businesses and enterprises. It's just not called multicloud, right?Jeremy: Right.Rob: So it's this bottom-up approach that individual teams are consuming according to their needs to create the greatest value for customers, and that's what I like to see, and that's what I like to promote.Jeremy: Yeah, yeah. I love that idea of bottom-up because I think that is absolutely true, and I don't think you've seen this as aggressively as you have in the last probably five years as more SaaS companies have become or SaaS has become a household name. I mean, probably 10 years ago, I think Salesforce was around, and some of these other things were around, right?Rob: Yeah.Jeremy: But they just weren't ... They weren't the household names that they are now. Now, you watch any sport, any professional sport, and you see advertisements for all these SaaS companies now because that seems to be the modern economy. But the idea of the bottom-up approach, that is something where you basically give a developer or maybe you don't give them, but the developer takes the liberties, I would say, to maybe try and experiment with something new without having to do years of research, go through procurement, get approval to use some platform. Even companies trying to move to AWS, or on to Azure, or something like that, they have to go through hoops. Basically, jump through hoops in order to get them there. So this idea of the bottom-up approach, the developers are the ones who are experimenting. Very low-risk experiments, by the way, with some of these other services. That approach, that seems like the right marketing approach for companies that are building these services, right?Rob: Yeah. It seems like a powerful approach for them. Maybe not necessarily the only one, but it is a good one. I mean, there's a historical lesson here as well, right? I want to come back to your point about the developers after, but I think of this as shadow cloud. Right? You saw this with the early days of SaaS where people would go out and sign up for accounts for their business and use them. They weren't necessarily regulated, but we saw even before that with shadow IT, right, where people were bringing their own software in?Jeremy: Right.Rob: So for enterprises that are afraid of this that are heavily risk-focused or control-focused top-down, I would say don't be so afraid because there's an entire set of lessons you can learn about this as you bring it, as you come forward to it. Then, with the developers, I think it's even more powerful than the way you put it because a lot of times, it's not an experiment. I mean, you've seen the same things on Twitter I've seen, the great tech turnover of 2021, right? That's normal for tech. There's such a turnover that a lot of times, people are coming in already having the skills that they know will enhance delivery and add customer value more quickly. So it's not even an experiment. They already have the evidence, and they're able to get their team skilled up and building quickly. If you hire someone who's coming from an AWS shop, you hire someone who's coming from an Azure shop on to two different teams, they're likely going to evolve that excellence or that capability independently, and I don't necessarily think there's a reason to stop that as long as you have the right controls around it.Jeremy: Right. I mean, and you mentioned controls, and I think that if I'm the CTO of some company or whatever, or I'm the CIO because we're dealing in a super enterprise-y world, that you have developers that are starting to use tools ... Maybe not Stripe, but maybe like a Twilio or maybe they're using, I don't know, ChaosSearch or something, something where data that is from within their corporate walls are going out somewhere or being stored somewhere else, like the security risk around that. I mean, there's something there though, right?Rob: Yeah, there absolutely is. I think it's incumbent on the organizations to understand what's going on and adapt. I think it's also imcu,bent on the cloud service providers to understand those organizational concerns and evolve their product to address them, right?Jeremy: Right.Rob: This is one thing. My classic example of this is data exfiltration in a Lambda function. Some companies get ... I want to be able to inspect every packet that leaves, and they have that hard requirement for reasons, right?Jeremy: Right.Rob: You can't argue with them that they're right or wrong. They made that decision as a company. But then, they have to understand the impact of that is, "Okay. Well, every single Lambda function that you ever create is going to run inside of VPC or is going to run connected to a VPC." Now, you have the added complexity of managing a VPC, managing your firewall rules, your NACLs, your security groups. All of this stuff that ... Maybe you still have to do it. Maybe it really is a requirement. But if you examine your requirements from a business perspective and say, "Okay. There's another way we can address this with tightly-scoped IAM permissions that only allow me to read certain records or from certain tables, or access certain keys, or whatever." Then, we assume all that traffic goes out and that's okay. Then, you get to throw all of that complexity away and get back to delivering value more quickly. So they have to meet together, right? They have to meet.Jeremy: Right.Rob: This led to a lot of the work that AWS did with VPC networking for Lambda functions or removing the cold start because a lot ... Those enterprises weren't ready to let go of that requirement, and AWS can't tell them, "You're wrong." It's their business. It's their requirement. So AWS built that for them to reduce the cold start so that Lambda became a viable platform for them to build against.Jeremy: Right, and so if you're a developer and you're looking at some solution because ... By the way, I mean, like you said, choosing the best of breed. There are just a lot of good services out there. There are thousands and thousands of SaaS companies, and I think ... I don't know if we made this point, but I certainly consider SaaS companies themselves to be part of the cloud. I would think you would probably agree with that, right?Rob: Yeah.Jeremy: It might as well be cloud providers themselves. Most of them run on top of the cloud providers anyways, but they found ...Rob: But they don't have to, and that's interesting to me and another truth that you could be consuming services from somebody else's data center and that's still multicloud.Jeremy: Right, right. So, anyway. So my thought here or I guess the question I have is if you're a developer and you're trying to choose something best in breed, right? Whatever that is. Let's say I'm trying to send text messages, and I just think Twilio is ... It's got all the features that I need. I want to go with Twilio. If you're a developer, what are the things that you need to look for in some of these companies that maybe don't have ... I mean, I would say Twilio does, but like don't necessarily have the trust or the years of experience or I guess years under their belts of providing these services, and keeping data secure, and things like that. What's the advice that you give to developers looking to choose something like that to be aware of?Rob: To developers in particular I think is a different answer ...Jeremy: Well, I mean, yeah. Well, answer it both ways then.Rob: Yeah, because there's the builder and the buyer.Jeremy: Right.Rob: Right?Jeremy: Right.Rob: Whoever the buyer is, and a lot of times, that could just be the software development manager who's the buyer, and they still would approach it different ways. I think the developer is first going to be concerned with, "Does it solve my problem?" Right? "Overall, does it allow me to ship faster?" The next thing has to be stability. You have to expect that this company will be around, which means there is a certain level of evidence that you need of, "Okay. This company has been around and has serviced," and that's a bit of a chicken and an egg problem.Jeremy: Right.Rob: I think the developer is going to be a lot less concerned with that and more concerned with the immediacy of the problem that they're facing. The buyer, whether that's a manager, or CIO, or anywhere in between, is going to need to be concerned with other things according to their size, right? You even get the weird multicloud corner cases of, "Well, we're a major supplier of Walmart, and this tool only runs on a certain cloud service provider that they don't want us to use. So we're not going to use it." Again, that's a business decision, like would I build my software that way? No, but I'm not subject to that constraint. So that means nothing in that equation.Jeremy: Right. So you mentioned a little bit earlier this idea of bringing people in from different organizations like somebody comes in and they can pick up where somebody else left off. One of the things that I've noticed quite a bit in some of the companies that I've worked with is that they like to build custom tools. They build custom tools to solve a job, right? That's great. But as soon as Fred or Sarah leave, right, then all of a sudden, it's like, "Well, who takes over this project?" That's one of the things where I mentioned ... I said "experiments," and I said "low-risk." I think something that's probably more low-risk than building your own thing is choosing an API or a service that solves your problem because then, there's likely someone else who knows that API or that service that can come in, and can replace it, and then can have that seamless transition.And as you said, with all the turnover that's been happening lately, it's probably a good thing if you have some backup, and even if you don't necessarily have that person, if you have a custom system built in-house, there is no one that can support that. But if you have a custom ... or if you have a system you've used, you're interfacing with Twilio, or Stripe, or whatever it is, you can find a lot of developers who could come in even as consultants and continue to maintain and solve your problems for you.Rob: Yeah, and it's not just those external providers. It's the internal tooling as well.Jeremy: Right.Rob: Right?Jeremy: Right.Rob: We're guilty of this in my company. We wrote a lot of stuff. Everybody is, right, like you like to do it?Jeremy: Right.Rob: It's a problem that you recognized. It feels good to solve it. It's a quick win, and it's almost always the wrong answer. But when you get into things like ... a lot of cases it doesn't matter what specific tool you use. 10 years ago, if you had chosen Puppet, or Chef, or Ansible, it wouldn't be as important which one as the fact that you chose one of those so that you could then go out and find someone who knew it. Today, of course, we've got Pulumi, Terraform, and all these other things that you could choose from, and it's just better than writing a bunch of Bash scripts to tile the stuff together. I believe Bash should more or less be banned in the cloud, but that's another ... That's my hot take for another time. Come at me on Twitter if you don't like that one.Jeremy: So, yeah. So I think just bringing up this idea of tooling is important because the other thing that you potentially run into is with the variety of tooling that's out there, and you mentioned the original IAC. I guess they would... Right? We call those like Ansible and those sort of things, right?Rob: Right.Jeremy: All of those things, the Chefs and the Puppets. Those were great because you could have repeatable deployments and things like that. I mean, there's still work to be done there, but that was great because you made the choice to not building something yourself, right?Rob: Right.Jeremy: Something that somebody else could jump in on. But now with Terraform and with ... You mentioned Pulumi. We've got CloudFormation and even Microsoft has their own ... I think it's called ARM or something like that that is infrastructure as code. We've got the Serverless Framework. We've got SAM. We've got Begin. You've got ... or Architect, right? You've got all of these choices, and I think what happens too is that if teams don't necessarily ... If they don't rally around a tool, then you end up with a bunch of people using a bunch of different tools. Maybe not all these tools are going to be compatible, but I've seen really interesting mixes of people using Terraform, and CloudFormation, and SAM, and Serverless Framework, like binding it all together, and I think that just becomes ... I think that becomes a huge mess.Rob: It does, and I get back to my favorite quote about complexity, right? "Simplicity before complexity is worthless. Simplicity beyond complexity is priceless." I find it hard to get to one tool that's like artificial, premature optimization, fake simplicity.Jeremy: Yeah.Rob: If you force yourself into one tool all the time, then you're going to find it doing what it wasn't built to do. A good example of this, you talked about Terraform and the Serverless Framework. My opinion, they weren't great together, but your Terraform comes for your persistent infrastructure, and your Serverless Framework comes in and consumes the outputs of those Terraform stacks, but then builds the constantly churning infrastructure pieces of it. Right? There's a blast radius issue there as well where you can't take down your database, or S3 bucket, or all of this from a bad deploy when all of that is done in Terraform either by your team, or by another team, or by another process. Right? So there's a certain irreducible complexity that we get to, but you don't want to have duplication of effort with multiple tools, right?Jeremy: Right.Rob: You don't want to use CloudFormation to manage your persistent data over here and Terraform to manage your persistent data over here because then you're not ... That's like that agnostic model where you're not benefiting from the excellent features in each. You're only using whatever is common between them.Jeremy: Right, right, and I totally agree with you. I do like the idea of consuming. I mean, I have been using AWS for a very, very long time like 2007, 2008.Rob: Yeah, same. Oh, yeah.Jeremy: Right when EC2 instances were a thing. I guess 2008. But the biggest thing for me looking at using Terraform or something like that, I always felt like keeping it in, keeping it in the family. That's the wrong way to say it, but like using CloudFormation made a lot of sense, right, because I knew that CloudFormation ... or I thought I knew that CloudFormation would always support the services that needed to be built, and that was one of my big complaints about it. It was like you had this delay between ... They would release some service, and you had to either do it through the CLI or through the console. But then, CloudFormation support came months later. The problem that you have with some of that was then again other tools that were generating CloudFormation, like a Serverless Framework, that they would have to wait to get CloudFormation support before they could support it, and that would be another delay or they'd have to build something custom, which is not always the cleanest way to do it.Rob: Right.Jeremy: So anyways, I've always felt like the CloudFormation route was great if you could get to that CloudFormation, but things have happened with CDK. We didn't even mention CDK, but CDK, and Pulumi, and Terraform, and all of these other things, they've all provided these different ways to do things. But the thing that I always thought was funny was, and this is ... and maybe you have some insight into this if you can share it, but with SAM, for example, SAM wasn't extensible, right? You would just run into issues where you're like, "Oh, I can't do that with SAM." Whereas the Serverless Framework had this really great third-party plug-in system that allowed you to do some of these other things. Now, granted not all third-party plug-ins were super stable and were the best way to do something, right, because they'd either interact with APIs directly or whatever, but at least it gave you ... It unblocked you. Whereas I felt like with SAM and even CloudFormation when it didn't support something would block you.Rob: Yeah. Yeah, and those are just two different implementation philosophies from two different companies at two different stages of their existence, right? Like AWS ... and let's separate the reality from the theory here. The theory is that a large company can exert control over release cycles and limit what it delivers, but deliver it with a bar of excellence. A small company can open things up, and it depends on its community members for contributions to solve problems. It's very much like this is the cathedral and the Bazaar of cloud tooling, right?AWS has that CloudFormation architecture that they're working around with its own goals and approach, the one way to do it. Serverless Framework is, "Look, you need to ... You want to set up a stall here and insert IAM policies per function. Set up a stall. It will be great. Maybe people come and maybe they don't," and the system inherently sorts or bubbles up the value, right? So you see things like the Step Functions plug-in for Serverless Framework. It was one of the early ones that became very popular very quickly, whereas Step Functions supporting SAM trailed, but eventually came in. I think that team, by the way, deserves a lot of credit for really being focused on developers, but that's not the point of the difference between the two.A small young company like Serverless Framework that is moving very quickly can't have that cathedral approach to it, and both are valid, right? They're both just different strategies and good for the marketplace, quite frankly. I have my preferred approach, which is not about AWS or SAM vs Serverless Framework. It's the extensibility of plug-in frameworks to me are a key component of tooling that adapts as quickly as the clouds change, and you see this. Like Terraform was the first place that I really learned about plug-ins, and their plug-in framework is fantastic, the way they do providers. Serverless Framework as well is another good example, but you can't know how developers are going to build with your services. You just can't.You do customer development. You talk to them ahead of time. You get all this research. You talk to a thousand customers, and then you release it to 14 million customers, right? You're never going to guess, so let them. Let them build it, and if people ... They put the work in. People find there's value in it. Sometimes you can bring it in. Sometimes you leave it up to the community to maintain, but you just ... You have to be willing to accept that customers are going to use your product in different ways than you envisioned, and that's a good thing because it means customers are using your product.Jeremy: Right, right. Yeah. So I mean, from your perspective though ... because let's talk about SAM for a minute because I was excited when SAM came out. I was thinking to myself. I'm like, "All right. A simplified tooling that is focused on serverless. Right? Like gives me all the things that I think I'm going to need." And then I did ... from a developer experience standpoint, and let's call out the elephant in the room. AWS and developer experience are not always the same. They don't always give you that developer experience that you would want. They give you tons of tools, right, but they don't always give you that ...Rob: Funny enough, you can spell "developer experience" without AWS.Jeremy: Right. So I mean, that's my ... I was disappointed when I started using SAM, and I immediately reverted back to the Serverless Framework. Not because I thought that it wasn't good or that it wasn't well-thought-out. Like you said, there was a level of excellence there, which certainly you cannot diminish, It just didn't do the things I needed it to do, and I'm just curious if that was a consistent feedback that you got as being someone on the dev advocate team there. Was that something that you felt as well?Rob: I need to give two answers to this to be fair, to be honest. That was something that I felt as well. I never got as comfortable with SAM as I am with the Serverless Framework, but there's another side to this coin, and that's that enterprise uptake of SAM CLI has been really strong.Jeremy: Right.Rob: Enterprise it, it does what they need it to do, and it addresses their concerns, and they liked getting tooling from AWS. It just goes back to there being a place for both, right?Jeremy: Right. Rob: Enterprises are much more likely to build cathedrals. They want that top-down, "Okay, everybody. This is how you define something. In fact, we've created a module for you. Consume it here. Thou shalt not write new S3 to web server configuration in your SAM templates. Thou shalt consume this." That's not wrong, and the usage numbers don't lie with SAM. It's got a lot of fans, and it's got a lot of uptake, but that's an entirely different answer from how I feel about it. I think it also goes back to I'm not running an enterprise. I've never run an enterprise. The biggest I've got in terms of responsibility is at best a small company, right? So I think it's natural for me to feel that way when I try to use a tool that has such popularity amongst enterprise. Now, of course, you have the switch, right? You have enterprises using Serverless Framework, and you have small builders using SAM. But in general, I think the success there was with the enterprise, and it's a validation of their strategy.Jeremy: Right, right. So let's talk about enterprises for a second because this is where we look at tools like the CDK and SAM, Serverless Framework, things like that. You look at all those different tools, and like you said, there's adoption across some of those. But at the end of the day, most of those tools are compiling down to a CloudFormation or they're compiling down to ... What's it called? The Azure Resource Manager Language or whatever the heck it is, right?Rob: ARM templates.Jeremy: ARM templates. What's the value now in CloudFormation and those sort of things that the final product that you get to ... I mean, certainly, it's so much easier to build those using these frameworks, but do we need CloudFormation in those things anymore? Do we need to know those? Does an individual developer need to be able to understand those, or can they just basically take a step back and say, "Look, CDK does it for me," or, "Pulumi does it for me. So why do I need to know what's baked into those templates?"Rob: Yeah. So let's set Terraform aside and talk about it after because it's different. I think the choice of JSON and YAML as implementation languages for most of this tooling, most of this tooling is very ... It was a very effective choice because you don't necessarily have to know CloudFormation to look at a template and define what it's doing.Jeremy: Right.Rob: Right? You don't have to understand transforms. You don't have to understand parameter replacement and all of this stuff to look at the final transformed template in CloudFormation in the console and get a very quick reasoning about what's happening. That's good. Do I think there's value in learning to create multi-thousand-line CloudFormation templates by hand? I don't. It's the assembly language of the cloud, right?Jeremy: Right.Rob: It's there when you need it, and just like with procedural languages, you might want to look underneath at the instructions, how it unrolled certain loops, how it decided not to unroll others so that you can make changes at the next level. But I think that's rare, and that's optimization. In terms of getting things done and getting things shipped and delivered, to start, I wouldn't start with plain CloudFormation for any of these, especially of anything for any meaningful production size. That's not a criticism of CloudFormation. It's just like you said. It's all these other tooling is there to help us generate what we want consistently.The other benefit of it is once you have that underlying lingua franca of the cloud, you can build visualization, and debugging, and monitoring, and like ... I mean, all of these other evaluatory forensic. "Evaluatory?" Is that a word? It's a word now. You heard it here first on this podcast. Like forensic, cloud forensic type tooling that lets you see what's going on because it is a universal language among all of the tools.Jeremy: Yeah, and I want to get back to Terraform because I know you mentioned that, but I also want to be clear. I don't suggest you write CloudFormation. I think it is horribly verbose, but probably needs to be, right?Rob: Yeah.Jeremy: It probably needs to have that level of fidelity there or that just has all that descriptive information. Yeah. I would not suggest. I'm with you, don't suggest that people choose that as their way to go. I'm just wondering if it's one of those things where we don't need to be able to look at ones and zeroes anymore, and understand what they do, right?Rob: Right.Jeremy: We've got higher-level constructs that do that for us. I wouldn't quite put ... I get the assembly language comparison. I think that's a good comparison, but it's just that if you are an enterprise, right, is that ... Do you trust? Do you trust something like CDK to do everything you need it to do and make sure that it's covering all its bases because, again, you're writing layers of abstraction on top of a layer of abstraction that then abstracts it even more. So yeah, I'm just wondering. You had mentioned forensic tools. I think there's value there in being able to understand exactly what gets put into the cloud.Rob: Yeah, and I'd agree with that. It takes 15 seconds to run into the limit of my knowledge on CDK, by the way. But that knowledge includes the fact that CDK Synth is there, which generates the CloudFormation template for you, and that's actually the thing, which is uploaded to the CloudFormation service and executed. You'd have to bring in somebody like Richard Boyd or someone to talk about the guard rails that are there around it. I know they exist. I don't know what they are. It's wildly popular, and adoption is through the roof. So again, whether I philosophically think it's a good idea or not is irrelevant. Developers want it and want to build with it, right?Jeremy: Yeah.Rob: It's a Bazaar-type tool where they give you some basic constructs, and you can write your own constructs around it and get whatever you need. But ultimately, that comes back to CloudFormation, which is then subject to all the controls that your organization puts around CloudFormation, so it is ... There's value there. It can't be denied.Jeremy: Right. No, and the thing that I like about the CDK is the idea of being able to create those constructs because I think, especially from a ... What's the right word? Compliance standpoint or something like that, that you can write in these constructs that you say, "You need to use these constructs when you deploy a microservice or you deploy this," or whatever it is, and then you have those guard rails as you mentioned or whatever, but those ... All of those checkboxes are ticked because you can put that all into one construct. So I totally think that's great. All right. So let's talk about Terraform.Rob: Yeah, so there's ... First, it's a completely different model, right?Jeremy: Right.Rob: This is an interesting discussion to have because it's API calls. You write your provider, whatever your infrastructure is, and anything that can be an API call can now be a Terraform declarative resource. So it's that mapping between declarative and imperative that I find fascinating. Well, also building the dependency graph for you. So it handles all of those aspects of it, which is a really powerful tool. The thing that they did so well ... Terraform is equally verbose as CloudFormation. You've got to configure all the same options. You get the same defaults, et cetera. It can be terribly verbose, but it's modular. Every Terraform file that you have in one directory is concatenated, and that is a huge distinction between how CloudFormation wants everything in one template, or well, you can refer to something in an S3 bucket, but that's not actually useful to me as a developer.Jeremy: Right, right.Rob: I can't mount an S3 bucket as a drive on my workstation and compose all of these independent files at once and do them that way. Sidebar here. Maybe I can. Maybe it supports that, and I haven't been able to discover it, right? Whereas Terraform by default, out of the box put everything in its own file according to function. It's very easy to look in your databases.tf and understand what's in there, look in your VPC.tf and understand what's in there, and not have to go through thousands of lines of code at once. Yes, we have find and replace. Yes, we have search, and you ... Anybody who's ever built any of this stuff knows that's not the same thing. It's not the same thing as being able to open a hundred lines in your text editor, and look at everything all at once, and gain an understanding of it, and then dive into the next level of detail a hundred lines at a time and understand that.Jeremy: Right. But now, just a question here, without... because the API thing, I love that idea, and actually, Serverless Components used an API thing to do it and bypass CloudFormation. Actually, I believe Architect originally was using APIs, and then switched to CloudFormation, but the question I have about that is, if you don't have a CloudFormation template, if you don't have that assembly language of the web, and that's not sitting in your CloudFormation tool built into the dashboard, you don't get the drift protection, right, or the detection, and you don't get ... You don't have that resource map necessarily up there, right?Rob: First, I don't think CloudFormation is the assembly language of the web. I think it's the assembly language of AWS.Jeremy: I'm sorry. Right. Yes. Yeah. Yeah.Rob: That leads into my point here, which is, "Okay. AWS gives you the CloudFormation dashboard, but what if you're now consuming things from Datadog, or from Fauna, or from other places that don't map this the same way?Jeremy: Right.Rob: Terraform actually does manage that. You can do a plan against your existing file, and it will go out and check the actual existing state of all of your resources, and compare them to what you've asked for declaratively, and show you what the changeset will be. If it's zero, there's no drift. If there is something, then there's either drift or you've added new functionality. Now, with Terraform Cloud, which I've only used at a basic level, I'm not sure how automatic that is or whether it provides that for you. If you're from HashiCorp and listening to this, I would love to learn more. Get in touch with me. Please tell me. But the tooling is there to do that, but it's there to do it across anything that can be treated as an API that has ... really just create and retrieve. You don't even necessarily need the update and delete functionality there.Jeremy: Right, right. Yeah, and I certainly ... I am a fan of Terraform and all of these services that make it easier for you to build clouds more easily, but let's talk about APIs because you mentioned anything with an API. Well, everything has APIs now, right? I mean, essentially, we are living in a ... I mentioned SaaS, but now we're sort of ... This whole idea of the API economy. So just overall, what are your thoughts on this idea of basically anything you want to do is available now via an API?Rob: It's not anything you want to do. It's everything you want to do short of fulfilling your customer's needs, short of solving your customer's specific problem is available as an API. That means that you get to choose best-in-class for everything, right? Your customer's need isn't, "I want to spend $25 on my credit card." Your customer's need is, "I need a book."Jeremy: Right.Rob: So it's not, "I want to store information about books in a database." It's, "I need a book." So everything and every step of the way there can now be consumed from an API. Again, it's like serverless in general, right? It allows you to focus purely or as close to purely as we can right now on generating customer value and addressing customer problems so that you can ship faster, and so that you have it as a competitive advantage. I can write a payment processing program. I know I can because I've done it back in 2004, and it was horrible, and it was awful. It wasn't a very good one, and it worked. It took your money, but this was like pre-PCIDSS.If I had to comply with all of those things, why would I do that? I'm not a credit card payment processor. Stripe is, and they have specialists in all of the areas related to the problem of, "I need to take and process payments." That's the customer problem that they're solving. The specialization of labor that comes along with the API economy is fantastic. Ops never went away. All the ops people work at the cloud service providers now.Jeremy: Right.Rob: Right? Audit never went away. All the auditors have disappeared from view and gone into internal roles in payments companies. All of this continues to happen where the specialists are taking their deep, deep knowledge and bringing it inside companies that specialize in that domain.Jeremy: Right, and I think the domain expertise value that you get from whatever it is, whether it's running a database company or whether it's running a payment company, the number of people that you would need to hire to have a level of specialization for what you're paying for two cents per transaction or whatever, $50 a month for some service, you couldn't even begin. The total cost of ownership on those things are ... It's not even a conversation you would want to have, but I also built a payment processing system, and I did have to pass PCI, which we did pass, but it was ...Rob: Oh, good for you.Jeremy: Let's put it this way. It was for a customer, and we lost money on that customer because we had to go through PCI compliance, but it was good. It was a good experience to have, and it's a good experience to have because now I know I never want to do it again.Rob: Yeah, yeah. Back to my earlier point on Ops and serverless.Jeremy: Right. Exactly.Rob: These things are hard, right?Jeremy: Right, right.Rob: Sorry.Jeremy: No, no. Go ahead.Rob: Not to interrupt, but these are all really hard problems that people with graduate degrees and post-graduate research who have ... They're 30 years old when they start working on the problem are solving. There's a supply question there as well, right? There's just not enough people, and so you and I can like ... Well, I'm not going to project this on to you. I can stumble through an implementation and check off the requirements just like I worked in an optical microscopy lab in college, and I could create computer programs that modeled those concepts, but I was not an optical microscopist. I was not a PhD-level generating understandings of these things, and all of these, they're just so hard. Why would you do that when customer problems are equally hard and this set is solved?Jeremy: Right, right.Rob: This set of problems over here is solved, and you can't differentiate yourself by solving it better, and you're not likely to solve it better either. But even if you did, it wouldn't matter. This set of problems are completely unsolved. Why not just assemble the pieces from best-in-class so that you can attack those problems yourself?Jeremy: Again, I think that makes a ton of sense. So speaking about expertise, let's talk about what you might have to pay say a database administrator. If you were to hire a database administrator to maintain all the databases for you and keep all that uptime, and maybe you have to hire six database administrators in order for them to ... Well, I'm thinking multi-region and all that kind of stuff. Maybe you have to hire a hundred, depending on what it is. I mean, I'm getting a little ahead of myself, but ... So if I can buy a service like Fauna, so tell me a little bit more about just how that works.Rob: Right. Well, I mean, six database engineers in the US, you're over a million dollars a year easily, right?Jeremy: Right.Rob: I don't know what the exact number is, but when you consider benefits, and total cost, and all of that, it's a million dollars a year for six database engineers. Then, there are some very difficult problems in especially distributed databases and database scaling that Fauna solves. A number of other products or services solve some of them. I'm biased, of course, but I happen to think Fauna solves all of them in a way that no other product does, but you're looking ... You mentioned distributed transactions. Fauna is built atop the Calvin paper, which came out of Yale. It's a very brief, but dense academic research paper. It's a PHC research paper, and it talks about a model for distributed transactions and databases. It's a layer, a serialization layer, that sits atop your database.So let's say you wanted to replicate something like Fauna. So not only do you need to get six database engineers who understand the underlying database, but you need to find engineers that understand this paper, understand the limitations of the theory in the paper, and how to overcome them in operations. In reality, what happens when you actually start running regions around the world, replicating transactions between those regions? Quite frankly, there's a level of sophistication there that most of the set of people who satisfy that criteria already work at Fauna. So there's not much of a supply there. Now, there are other database competitors that solve this problem in different ways, and most of the specialists already work at those companies as well, right? I'm not saying that they aren't equally competent database engineers. I'm just saying there's not a lot of them.Jeremy: Right.Rob: So if you're thing is to sell books at a certain scale like Amazon, then that makes sense to have them because you are a database creator as well. But if your thing is to sell books at some level below that, why would you compete for that talent rather than just consuming it?Jeremy: Right. Yeah, and I would say unless you're a horse with a horn on your head, it's probably not worth maintaining your own database and things like that. So let's talk a little bit more, though, about that. I guess just this idea of maybe a shortage of people, like if you're ... You're right. There's a limited number of resources, right? I'm sure there's brilliant database engineers all around the world, and they have the experience where, right, they could come in and they could probably really help you maintain your database. Even if you could afford six of them and you wanted to do that, I think the problem is it's got to be the interestingness of the problem. I don't think "interestingness" is a word either, but like if I'm a database engineer, wouldn't I want to be working on something like Fauna that I could help millions and millions of people as opposed to helping some trucking company maintain their internal database or something like that?Rob: Yeah, and I would hope so. I hope it's okay that I mention we're hiring. So come to Fauna.com and look at our roles database engineers.Jeremy: You just read that Calvin paper first. Go ahead.Rob: But read the Calvin paper first. I think it's only like 12 pages, and even just the first page is enough. I'm happy to talk about that at any length because I find it fascinating and it's public. It is an interesting problem and the ... It's the reification or the implementation of theory. It's bringing that theory to the real world and ... Okay. First off, the theory is brilliant. This is not to take away from it, but the theory is conceived inside someone's mind. They do some tests, they prove it, and there's a world of difference between that point, which is foundational, and deploying it to production where people are trusting their workloads on it around the world. You're actually replicating across multiple cloud providers, and you're actually replicating across multiple regions, and you're actually delivering on the promise of the paper.What's described in the paper is not what we run at Fauna other than as a kernel, as a nugget, right, as the starting point or the first principle. That I think is wildly interesting for that class of talent like you talked about, the really world-class engineers who want to do something that can't be done anywhere else. I think one thing that Fauna did smartly early was be a remote-first company, which means that they can take advantage of those world-class engineers and their thirst for innovation regardless of wherever Fauna finds them. So that's a big deal, right? Would you rather work on a world-class or global problem or would you rather work on a local problem? Now, look, we need people working on local problems too. It's not to disparage that, but if this is your wheelhouse, if innovation is the thing that you want to do, if you want to be doing things with databases that nobody else is doing, this is where you want to be. So I think there's a strong argument there for coming to work in a place like Fauna.Jeremy: Yeah, and I want to make sure I apologize to any database engineer working at a trucking company because I'm sure there are actually probably really interesting problems with logistics and things like that that they are solving. So maybe not the best example. Maybe. I don't know. I can't think of another example. I don't want to offend anybody who's chosen a more local problem because you're right. I mean, there are local problems that need to be solved, but I do think that there are people ... I mean, even someone like me. I want to work on a bigger problem. You know what I mean? I owned a web development company for 12 years, and I was solving other people's problems by building them a website or whatever, and it just got to a point where I'm like, "I'm not making enough of an impact here." You're not solving a big enough problem. You want to work on something more interesting.Rob: Yeah. Humans crave challenge, right? Challenge is a necessary precondition for growth, and at least most of us, we want to grow. We want to be better at whatever it is we're doing or just however we think of ourselves next year that we aren't today, and you can't do that with challenge. If you build other people's websites for 12 years, eventually, you get to a point where maybe you're too good at it. Maybe that's great from a business perspective, but it's not so great from a personal fulfillment perspective.Jeremy: Right.Rob: If it's, "Oh, look, another brochure website. Okay. Here you go. Oh, you need a contact form?" Again, it's not to disparage this. It's the fact that if you do anything for 12 years, sometimes mastery is stasis. Not always.Jeremy: Right, and I have nightmares of contact forms, of building contact forms, by the way, but...Rob: It makes sense. Yeah. You know what you should do is just put all of those directly into Fauna and don't worry about it.Jeremy: Easy enough. Easy enough.Rob: Yeah, but it's not necessarily stasis, but I think about craftsmen and people who actually make things with their hands, physical builders, and I think a lot of that ... Like if you're making furniture, you're a cabinet maker. I think a lot of that is every time, it's just a little bit wrong, right? Not wrong, but just a little bit off from your optimum no matter how long you do it, and so everything has a chance to evolve. That's there with software to a certain extent, but the problem is never changing.Jeremy: Right.Rob: So, yeah. I can see both sides of it, but for me, I ... You can see it when I was on serverless four years ago and now that I'm on a serverless database now. I like to be out at the edge, pushing that edge out for everyone who's coming behind. It can be challenging because sometimes there's just no way forward, and sometimes everybody is not ready to come with you. In a lot of ways, being early is the same as being wrong.Jeremy: Right. Well, I've been ...Rob: Not an original statement, but ...Jeremy: No, but I've been early on many things as well where like five years after we tried to do something, like then, all of a sudden, it was like this magical thing where everybody is doing it, but you mentioned the edge. That would be something ... or you said on the edge. I know you mean this way, but the edge in terms of the actual edge. That's going to be an interesting data problem to solve.Rob: Oh, that's a fascinating data problem, especially for us at Fauna. Yeah, compute, and Andy Jassy, when he was at AWS, talked about how compute was bifurcating, right? It's either moving all the way out to the edge or it's moving all the way into the cloud, and that's true. But I think at Fauna, we take that a step further, right? The edge part is true, and a lot of the work that we've done recently, announcements with CloudFlare workers. We're ready for that. We believe in it, and we like pushing that out as close to the user as possible. One thing that we do uniquely is we have this concept of user-defined functions, and anybody who's written T-SQL back in the day, who wrote store procedures is going to be familiar with this, but it's ... You bring that business logic and that code to your data. Not near your data, to your data.Jeremy: Right.Rob: So you bring the compute not just to the cloud where it still needs to pass through top-of-rack and all of this. You bring it literally on to the same instance as your data where these functions execute against them there. So you get not just the database, but you get a compute layer in there, and this helps for things like filtering for things like the equivalent of joins, stuff that just ... If you've got to load gigabytes of data and move it somewhere, compute against it, reduce it to something, and then store that back, the speed of light still matters. Even if it's the speed of light across a couple switches, it still matters, and so there are some really interesting things that you can begin to do as you pull more and more of that logic into your data layer, and you also protect that logic from other components of your application.So I like that because things like GraphQL that endpoints already speak and already understand, just send it over, and again, they don't care about the architectural, quite frankly, genius--I can say that because I didn't create it--the genius behind all of this stuff. They just care that, "Look, I send this request over and I get it back," and entire workflows, and complex processes, and everything are executing behind the scenes just so that the endpoints can send and retrieve what they need more effectively and more quickly. The edge is fascinating. The thing I regret the most about the edge is I have no hardware skills, right? So I can't make fun things to do fun things in my house. I have to buy them, but you can't do everything.Jeremy: Yeah. Well, no. I think you make a good point though about bringing the compute to the data side, and other people have said there's no ... Ben Kehoe has been talking about this for a while too where like it just makes sense. Run the compute where the data is, and then send that data somewhere else, right, because there's more things that can be done with data after that initial bit of compute. But certainly, like you said, filtering it down or getting the bits that are relevant and moving a small amount of data as opposed to a large amount of data I think is hugely important.Now, the other thing I just want to mention before I let you go or I want to talk about quickly is this idea of going back to the API economy aspect of things and buying versus building. If you think about what you've had to do at Fauna, and I know you're relatively recent there, but you know what they've done and the work that had to go in in order to build this distributed system. I mean, I think about most systems now, and I think like anything I'm going to build now, I got to think about scale, right?I don't necessarily have to build to scale right away, especially if I'm doing an MVP or something like that. But if I was going to build a service that did something, I need to think about multi-region, and I need to think about failover, and I need to think about potentially providing it at the edge, and all these other things. So you come down to this thing, and I will just use the database example. But even if you were say using like MySQL, or Postgres, or something like that, that's going to scale. That's going to scale pretty well to get to a certain point, and then you're going to have to start sharding, right? When data gets hard, it's time to shard, right? You just have to start sharding everything.Rob: Yeah.Jeremy: Essentially, what you end up doing is rebuilding DynamoDB, or trying to rebuild Fauna, or something like that. So just thinking about that, anything you're building ... Maybe you have some advice for developers who ... I know we've talked about this a little bit, but I just go back to this idea of like, if you think about how complex some of these SaaS companies and these services that are being built out right now, why would you ever want to take that complexity on yourself?Rob: Pride. Hubris. I mean, the correct answer is you wouldn't. You shouldn't.Jeremy: People do.Rob: Yeah, they do. I would beg and plead with them like, "Look, we did take a lot of that on. Fauna scales. You don't need to plan for sharding. You don't need to plan for global replication. All of these things are happening." I raise that as an example of understanding the customer's problem. The customer didn't want to think about, "Okay, past a thousand TPS, I need to create a new read replica. Past a million TPS, I need to have another region with active-active." The customer wanted to store some data and get that data, knowing that they had the ASA guarantees around it, right, and that's what the customer has.So get that good understanding of what your customer really wants. If you can buy that, then you don't have a product yet. This is even out of software development and into product ideation at startups, right? If you can go ... Your customer's problem isn't they can't send text messages programmatically. They can do that through Twilio. They can do that through Amazon. They can do that through a number of different services, right? Your customer's problem is something else. So really get a good understanding of it. This is where knowing a little ... Like Joe Emison loves to rage against senior developers for knowing not quite enough. This is where knowing like, "Oh, yeah, Postgres. You can just shard it." "Just," the worst word in computer science, right?Jeremy: Right.Rob: "You can just shard it." Okay. Now, we're back to those database engineers that you talked about, and your customer doesn't want to shard a database. Your customer wants to store and retrieve data.Jeremy: Right.Rob: So any time that you can really focus in, and I guess I really got this one, this customer obsession beaten into me from my time at AWS. Really focus in on what the customer is asking you to do or what the customer needs even if they don't know how to express it, and build for that.Jeremy: Right, right. Like the saying. I forgot who said it. Somebody from Harvard Business Review, but, "Your customers don't want a quarter-inch drill. They want a quarter-inch hole."Rob: Right, right.Jeremy: That's basically true. I mean, the complexity that goes behind the scenes are something that I think a vast majority of customers don't necessarily want, and you're right. If you focus on that product ideation thing, I think that's a big piece of that. All right. Well, anyway. So I have one more question for you just to ...Rob: Please.Jeremy: We've been talking for a while here. Hopefully, we haven't been boring people to death with our talk about APIs and stuff like that, but I would like to get a little bit academic here and go into that Calvin paper just a tiny bit because I think most people probably will not want to read it. Not because they don't want to, but because people are busy, right, and so they're listening to the podcast.Rob: Yeah.Jeremy: Just give us a quick summary of this because I think this is actually really fascinating and has benefits beyond just I think solving data problems.Rob: Yeah. So what I would say first. I actually have this paper on my desk at all times. I would say read Section 1. It's one page, front and back. So if you're interested in it, you don't have to read the whole paper. Read that, and then listeners to this podcast will probably understand when I say this. Previously, for distributed databases and distributed transactions, you had what was called a two-phase commit. The first was you'd go out to all of your replicas, and you say, "Hey, I need lock." When everybody comes back, and acknowledges, and says, "Okay. You have the lock," then you do your transaction, and then you replicate it, and then you say, "Hey, everybody. I'm done. Release the lock." So it's a two-phase commit. If something went wrong, you rolled it all the way back and you said, "Hey, everybody. Forget it."Calvin is event-sourcing for databases. So if I could distill the entire paper down into one concept, that's it. Right? Instead of saying, "Hey, everybody. Give me a lock, I'm going to do something," you say, "Hey, everybody. Here's what we're going to do." It's a deterministic application of the transaction so that you can ... You both create the lock and execute the transaction at the same time. So rather than having this outbound round trip, and then doing the thing in an outbound round trip, you have the outbound round trip, and you're done.They all apply it independently, and then it gets into how you structure the guarantees around that, which again is very similar to event-sourcing in that you use snapshotting or checkpointing. "So, hey. At this point, we all agree. So we can forget all of our old transactions, and we roll forward from here." If somebody leaves the cluster, they come back in at that checkpoint. They reapply all of the events that occurred prior to that. Those events are transactions. They'd get up to working speed, and then off they go. The paper I think uses Paxos. That's an implementation detail, but the really interesting thing about it is you're not having a double round trip.Jeremy: Yeah.Rob: Again, I love the idea of event-sourcing. I think Amazon EventBridge is the best service that they've released in the past couple years.Jeremy: Totally.Rob: If you understand all of that and are already building serverless applications that way, well, that's what we're doing, just event-sourcing for database. That's it. That's it.Jeremy: Just event-sourcing. It's easy. Simple. All right. All the words you never want to hear. Simple, easy, just. Right. Yeah. Perfect.Rob: Yeah, but we do the hard work, so you don't have to. You don't care about all of that. You want to write your data somewhere, and you want to retrieve your data from an API, and that's what Fauna gives you.Jeremy: Which I think is the main point here. So awesome. All right. Well, Rob, listen. This was great, and I'm super happy that I finally got you on the show. Congratulations for the new role at Fauna and on what's happening over there because it is pretty exciting.Rob: Thank you.Jeremy: I love companies that are innovating, right? It's not just another hosted database. You're actually building something here that is innovative, which is pretty amazing. So if people want to find out more about you, follow you on Twitter, or find out more about Fauna, how do they do that?Rob: Right. Twitter, rts_rob. Probably the easiest way, go to my website, robsutter.com, and you will link to me from there. From there, of course, you'll get to fauna.com and all of our resources there. Always open to answer questions on Twitter. Yeah. Oh, rob@fauna.com. If you're old-school like me and you prefer the email, there you go.Jeremy: All right. Awesome. Well, I will get all that into the show notes. Thanks again, Rob.Rob: Thank you, Jeremy. Thanks for having me.

Check out the episode where we start movements to change the way everyone writes T-SQL code, and/or deliver data platform meetups! And we also talk about a bunch of other stuff as well. Relevant Links: Join the SQL Server Community Slack and our new channel #SQLServerRadio ! Ask Us Anything: Database Performance Tuning and Optimization | Data Platform Meetup Israel Redgate Software takes over from PASS and commits to the future of the Summit, SQLSaturday and the community Educational Series (Red-Gate University) Database DevOps Considerations for SQL Server Availability Groups GenerateDeployReport method of DacServices | Microsoft Docs Database DevOps Implementation | Madeira Data Solutions I learned to love WHILE (true) and you should too | Eitan Blumin We’re taking Azure certifications for data and AI to the next level | Microsoft Tech Community Azure Data Fundamentals: Explore Core Data Concepts (Part 1) | Microsoft Reactor Tel-Aviv Meetup Watching out for Deferred Name Resolution  

היום אנחנו מתמקדים בכל מה שקשור ל-Service Broker... פיצ'ר שצובר הרבה אבק בארגז הכלים של SQL Server. לינקים רלוונטיים: טריגרים אסינכרונים בעזרת סרוויס ברוקר מימוש מקביליות בעזרת סרוויס ברוקר האק שפותר בעיות ביצועים נפוצות בסרוויס ברוקר פתרונות לוגיקה אסינכרוניים באז'ור תחליף TSQL במקום סרוויס ברוקר

פרק מספר 400(!!!) של רברס עם פלטפורמה - וזהו באמפרס מספר 71: רן, דותן ואלון מקליטים מעל גלי האתר (?) - שלום, מה שלומכם?בוקר טוב לכולם, והתאריך היום (מועד ההקלטה) הוא ה-6 בינואר 2021 - הנה, עברנו שנה, מזל לטוב לנו שסיימנו את 2020 - הקורונה עדיין פה אבל את השנה סיימנו . . .ואנחנו בבאמפרס - סדרה של קצרצרים שבה אנחנו מספרים על דברים מעניינים שקראנו, עשינו או ראינו בחודש האחרון - ואני (רן) מיד מתחיל:רן -נתקלתי ב-GitHub Repo שנקרא xg2xg - זה בעצם ex-Googlers to ex-Googlers: איזשהו מיפוי של טכנולוגיות וכלים שהיו קיימות (או שעדיין קיימות) בתוך Google - והמקבילות שלהן מבחוץ למשל - אם מדובר על Borg הפנימי - אז המקבילה החיצונית שלו היא לא אחרת מאשר Kubernetesאו GSLB שזה Cloud Load Balancing ו-Protocol Buffer שזה Protobuf ו-Stubby שזה gRPC וכו’אז למי שעבד בגוגל תקופה מסויימת, ועכשיו יצא החוצה - יכול להשתמש בטבלת התרגום הזאת כדי למצוא את המקבילות שנמצאות שם בחוץבין אם זה פשוט כדי לשפשף את הז’רגון שלו או שבאמת מחפשים כדי כלי לעשות משהו, וזה יכול לעזורכמובן שלא הכל נמצא שם - לא כל הטכנולוגיות הפנימיות נמצאות בחוץוכמובן שגם הפוך - יש הרבה טכנולוגיות שלא קיימות בגוגל וכן נמצאות בחוץ, אבל המיפוי הזה יכול לעזורויש משהו דומה, ממש על אותו הקונספט, ליוצאי פייסבוק - זה נקרא xfb2xfb, וגם פה יש מיפוי של טכנולוגיות עם השמות הפנימיים שלהן בתוך פייסבוק לשמות החיצוניים.למשל - Buck (שזה Buck, Pants) או BunnyLOL (ל-Jack Bunny, Gopherlol) או Codemod (ל - Codemod) או Configerator (שזה Protoconf) וכו’.(אלון) יש כאן דברים נחמדים, כמו “Money”, שזה מתורגם ל-”User” ודברים כאלה . . .(רן) שמור את זה למצחיקולים בסוף . . .אז כן, אני חושב שגם למי שלא עבד באותן החברות זה יכול להיות מעניין, כדי לדעת פחות או יותר איך נראה הנוף מבפנים, להבין אילו כלים קיימים בפנים ומה הם הם עושים - זה יכול להוות השראה.אם מדי פעם הייתם שואלים את עצמכם “מה גוגל הייתה עושה?” או “מה פייסבוק הייתה עושה?” (ההערה של אלון …), אז זה לפחות נותן תשובה חלקית על השאלה הזאת, ואני חושב שזה מעניין.(אלון) יש כאן דברים מוזרים קצת . . . נגיד על Hive רשום שזה BigQuery - אבל Hive זה Hive . . . כאילו, זה לא משהו פנימי.(רן) נכון . . . Hive פותח בפייסבוק, ויש לו גרסא חיצונית - אבל אני חושב שהגרסא החיצונית היא מאוד שונה מהגרסא הפנימית, וככל הנראה אולי “כוונת המשורר” הייתה שהגרסא של BigQuery כנראה הרבה יותר דומה ל-Hive הפנימי שיש בפייסבוק, אבל זה רק ניחוש, אני לא אומר את זה מתוך ידיעה.אבל כן - זה נכון ש-Hive פותח בפייסבוק.(דותן) הגישה היא מאוד שונה, לפחות ממה שאני מכיר - גוגל “מפתחים במחשכים”, כזה פנימי, ופייסבוק מאוד מאוד פומבי . . . לפחות השמות הם לא שונים - כמעט כל מה שיש פה הוא די מוכר לי מפייסבוק . . .(רן) כן, אני חושב שזה גם נכון להגיד - אתה צודק שהגישה שונה, אני חושב שגוגל מפתחים יותר במחשכים, לא לגמרי במחשכים - וכשהם משחררים, הם בדר”כ, לפחות הסטורית, הם לא היו משחררים את ה-Source Code אלא משחררים איזשהו white paper או בלוגים או דברים כאלהזאת אומרת שהם כן היו מספרים על מה שהם עושים, אבל לא משחררים את הקוד.היום זה כבר פחות נכון, היום משחררים הרבה יותר קודאבל פייסבוק, היסטורית, תמיד שחררו יותר קוד, זאת אומרת - פייסבוק, היסטורית, תמיד שחררו יותר כלים, אז בקטע הזה אני חושב שזה לגמרי נכון.הנושא הבא שעליו רציתי לדבר הוא בלוג-פוסט חדש של OpenAI, שמדבר על מחקר שלהם שנקרא DALL·Eאז למי שלא מכיר - OpenAI זו חברת Research מאוד מתקדמת בתחום של AIהשם “Open” אולי כבר פחות רלוונטי, כי הם כבר עושים דברים שהם הרבה יותר סגורים, אבל בכל אופן זו חברת Research מאוד מתקדמתהם ידועים בעיקר במודלים שלהם - מודלי ה-GPT שלהם, מודלי שפההאחרון שבהם - GPT3 - עורר הרבה מאוד הדים, אני מניח שהרבה מהמאזינים שמעו עליו (נגיד בבאמפרס 397…).אז הבלוג-פוסט הזה מדבר על מודל חדש שנקרא DALL·E, שהוא למעשה עושה סינתזה בין שפה לבין Vision . . .למעשה, התוצר הסופי שלו הוא שאתה בעצם אומר משפט, מתאר איזשהו ציור - והמודל מצייר לך אותו . . . אתה אומר לו “צייר לי כורסא בצורת אבוקדו” (כבשה! צייר לי כבשה!) - והמודל מצייר לך כורסא בצורת אבוקדו (או כבשה) - וזה עובד די יפה, אני חייב להגיד.יש פה הרבה מאוד דוגמאות - למשל, “צייר לי חתול” - והוא מצייר חתול; “צייר לי חתול עם פרווה שחורה” - והוא מוסיף לו פרווה שחורה; “צייר לי שתי קוביות, אחת על השנייה, אחת אדומה ואחת ירוקה” - והוא עושה את זה.זאת אומרת - התוצאות הן מאוד מרשימות, מאוד נחמד - גם דברים ריאליסטיים כמו . . . גם החתול שדיברנו עליו הוא חתול ריאליסטי - הוא יכול לעשות גם סקיצה של חתולים אבל גם חתולים ריאליסטיים.הוא אפילו מכיר מקומות בעולם, נגיד - “צייר לי איזושהי שכונה בסאן-פרנסיסקו” - והוא מצייר אותה, או שהוא מייצר צילום שלה.או “צייר לי אוכל סיני טיפוסי” - והוא מראה אוכל . . . כופתאות כאלה, עם בצק, סיניות . . .(דותן) שמע, זה ישר מעורר לי כזה . . . זה מדהים לסיפורי ילדים - אתה מספר איזה סיפור מטורף כזה, ואז בום, יש לך את כל האילוסטרציות ואתה פשוט כותב ספר (ויזואלי) לילדים ככה.(רן) כן, זה מדהים.עכשיו - זה מחקר מאוד יפה, זאת אומרת - המחקר עצמו לא פורסם, אבל התוצאות מאוד מרשימות.הם כן אומרים שיש פה ושם Caveats, זה לא עובד באופן מושלם - אם אתם פה ושם משנים את ה-Titles או את ה-Description אז התמונה או הציור יכולים לצאת שונים, אבל בכל אופן היכולות מאוד מרשימותהיכולת לקבל כ-Input משפט ולהוציא כ-Output תמונה - והדוגמאות שהם מראים פה הן בהחלט מאוד יפות - אני חושב שזו יכולת שהיא מאוד מאוד מרשימה.וגם אם אתם מסתכלים על ה. . . זאת אומרת - הבלוג-פוסט הזה הוא מאוד אינטראקטיבי, אתם יכולים ללחוץ על הדוגמאות ולראות את תהליך הייצור של התמונה, וזה נחמד.בקיצור, אני חושב שזו עבודה מאוד מעניינת.עוד נושא שהוא קצת מעניין בתחום הזה, מעבר לתוצאות המרשימות, זה שהרשת הזו היא “רזה” הרבה יותר מ-GPT3ה-GPT3 היייתה ידועה בזה שזו רשת ענקית עם 175 מיליארד פרמטרים, ולוקח משהו כמו . . .אני לא יודע כמה זמן, אבל סדר גודל שהם פרסמו זה משהו כמו 12 מיליון דולר כדי לאמן את הרשת . . .אז הרשת הזו, החדשה - DALL·E - יש בה הרבה פחות פרמטרים, לא 175 מיליארד, אני חושב שהם פרסמו משהו כמו 12 מיליארד פרמטרים.זה עדיין לא קטן, זה לא פצפון, עדיין 12 מיליארד - וזה עדיין בסדר גודל יותר נמוך מה-175 מיליארד של GPT3 ובכל אופן התוצאות הן מאוד מאוד יפות.זהו - אני מאוד מליץ, גם למי שלא נמצא בתחום, לבוא ולראות מה ה-State of the Art שם.(אלון) זה מאוד מרשים . . . זה באמת מגניב, השאלה רק מה יקחו מזה הלאה? כי השאלה היא מה אפשר לעשות עם זה, שהוא באמת שימושי ופרקטי חוץ ממשחק מגניב . . .(רן) אני דווקא יכול לחשוב על הרבה דברים פרקטיים - למשל אם אתה עשים Mock-ים, ואתה אומר “אוקיי, פה אני רוצה שיהיה משתמש שמח, או איש שיושב ליד לפטופ ושמח” או מה שדותן הציע קודם על סיפורי ילדיםאני חושב על הרבה דוגמאות שכן הייתי רוצה . . . זה כאילו שילוב של Google Image Search יחד עם היכולת לייצר, אם לא מצאת תמונה שאתה אוהב - היכולת לייצר סינתזה בין תמונות או לייצר תמונה שאתה כן אוהב.איזשהו צייר, איזשהו Designer שיכול להקשיב לך ועושה את מה שאתה רוצה.זה נחמד, נחמד מאוד. אני רק מחכה שישחררו את זה לשימוש של כולם ויהיה אפשר להתחיל להרביץ שם תיאורים ולקבל תמונות בחזרה..נושא אחר, פחות או יותר באותו איזור - המלצה שקיבלנו ממאזין, והאמת שמאזין מאוד פעיל - עפר, שעושה את כל התקצירים לפודקאסט, עפר פורר (זה מאוד מוזר לכתוב רפרנס לעצמך, רק מציין) עשה קורס ב-Coursera והמליץ עליו - לפחות אחד מהמודולים שם - זה קורס שבא ללמד אנשי מוצר מה זה Data Science ומהם הכלים שמשתמשים בהםאז יש פה קישור לקורס - זה נקרא (הקורס כולו) Agile Analytics (זה החלק הספציפי בקורס, זו כל הסדרה) ועפר מאוד ממליץאני אקח את ההמלצה שלך, עפר, למי שככה רוצה טעימה מאיך נראה היום-יום של Data Scientistלא משהו כבד, משהו יחסית קליל ויכול להנגיש את הנושא.תודה רבה עפר על ההמלצה (לכבוד הוא לי).(אלון) עפר יקבל תקליט, אני מקווה . . . כל הפודקאסטים, האוסף הראשון, העונה הראשונה בתקליט . . .(אמרנו אופנוע וכובע גרב, לא?)אלון - יש איזה ספר מפורסם של The Effective Manager באמאזון, וכנראה לא רק באמאזון, של Mark Horstmanבקיצור, מישהו עשה על זה TL;DR (הפנייה מעגלית עם ה”בקיצור”, לא?) של 30 שניות בערךפשוט - ארבעה דברים שאתה צריך לעשות כדי לתת פידבק - לשאול - “האם אני יכול לתת פידבק?”מה - (“…State the behavior (“When you Xההשפעה - (“…State the impact (“…the result is Yמה הלאה - Encourage effective future behavior (“Keep it up!” for positive feedback or “Can you change that?” for negative feedback)וזהו . . . לא יודע אם מישהו קרא את הספר, אבל זה ממש . . חסכתי לכם שעות של האזנה ל-Audible . . .אני יכול לעשות את הקטע הזה ולתרגם אותו בעוד שפות נוספות, אם יהיה ביקוש.זה היה פשוט נחמד, האמת שהייתה סביב זה די הרבה התהלבות על זה שהצליחו לתמצת ספר בארבעה משפטים, אבל אלו טיפים לא רעים.הדבר הבא - AWS הולכים ראש בראש עם מיקרוסופט והם עשו Microsoft SQL Server ל - Auroraזה Convertorהם אומרים בעצם שכל מי שיש לו Microsoft SQL “ותקוע איתו” - “הנה, עשינו לכם כלי נחמד ובואו תעברו ל-Aurora, ואז תוכלו לעבור ל-Cloud המגניב שלנו”.מעניין אותי איך מיקרוסופט יגיבו לזה, כי אני חושב שאחת הבעיות בלצאת ממיקרוסופט זה שאתה די “שבוי” אליהם, ל-Cloud שלהם באופן כללי - וזה יהיה מעניין.(דותן) ה-Babelfish זה ה-Translator בעצם? (כמובן, לפי המדריך - “Probably the oddest thing in the Universe.”)(אלון) כן . . . הוא פשוט בונה לך את זה, מתרגם את ה-Queries, למיטב זכרוני.לאותם Queries, כן? כי יש דברים שהם Syntax-specific, אז הוא קולט את ה-Syntax ועושה אותו על PostgreSQL(דותן) גישה מעניינת, האמת . . . אני חושב שזו הפעם הראשונה שאני נתקל בדבר כזה - Database שאומר - חוץ מ-DocumentDB, או שזה לא אותו הדבר? - אבל זה סוג של Translator של Queries ועולם חישובי לתוך Database אחר - ואז אתה יכול לעבוד עם כל Database, ובכלל לא צריך לדבר על ORM ולא צריך לדבר על כל . . . לא יודע, סתם זורק פה.(אלון) תראה, יכול להיות שתיהיה לזה איזושהי בעיה של Performance, אבל זה מה שהם אומרים - “translation layer for SQL Server’s proprietary SQL dialect (T-SQL) and communications protocol so that businesses can switch to AWS’ Aurora”בתיאוריה, אתה אומר “יש לי Query כזה ועכשיו להתחיל סדר . . .”, או אומר “SQL זה SQL, אבל פה ב-Query יש לי משהו…” ובפועל זה לא ממש יכול לעבור כי ב-Databases גדולים בכל Query אתה כבר צריך לעבור ולתקן משהו ולבדוק.אז אם הם פתרו את זה אז זה מגניב.כמובן שיש בעיות של Performance, אולי - אני לא יודע, כי הרבה פעמים אתה כותב את ה-Query בצורה מאוד מסויימת בשביל שתיהיה איזושהי אופטימיזציה עם ה-Database, ובעצם יכול להיות שזה ילך לך לאיבוד, לא יודע.(דותן) מעניין, כל ה-Stored Procedures . . . עולים לי כל מיני זכרונות מהעבר . . . אז זה מתרגם את הכל? לא זוכר אם אתה יכול להריץ Net. בתוך SQL Server, אבל . . . לא יודע.(אלון) נכון, יש Custom Filters . . . וואו, לא יודע - אני לא יודע עד כמה רחוק זה יכול ללכת, אבל זה קונספט מאוד מעניין, כי בעצם אומרים “הכל עובד - אתה לא צריך לשנות כלום”.מעניין - אם זה טוב, זו גישה מאוד מאוד מעניינת.אם מישהו מכיר את זה ועבד עם זה - האמת שמאוד מעניין אותי לשמוע האם זה שווה משהו או האם זה עובד בכלל, כל הדבר הזה.הדבר הבא - A Complete Guide to LLVM ליוצרי שפות תוכנות , for Programming Language Creators . . .אז LLVM זה מה שבעצם כל השפות ה- Low-level מיתרגמות איתו - אליו - והוא בעצם מתורגם למעבד, x86 או ARM וכו’.אז C ו-++C או Rust כו’ואז הם אומרים שאם אתה רוצה אתה יכול לראות איך זה עובד, ובתיאוריה אתה גם יכול לכתוב שפה משלך שמתורגמת לזה, ואז לעשות שם את האופטימיזציות שמתאימות לך.למי שמעניין אותו קצת LLVM-Under-the-Hood ואיך שהדברים האלה עובדים ואוהב להיכנס לדברים האלה - יש פה משהו מאוד ציורי ומאוד User-friendly, בעיקר עם המון דוגמאותהמון-המון-המון דוגמאותלמי שאוהב את העולם הזה - זה אחלה חומר קריאה.(דותן) אני חייב להוסיף שאם אתה עובד עם LLVM אז אתה ברמה של לבנות “שפה אמיתית” ולא “שפה צעצוע”, כי זה מנוע מאוד חזק - אז למי שאוהב את זה שווה להשקיע ולהבין איך LLVM עובד, ואז לכתוב שפה שהיא ברמה מאוד גבוהה ולא Translator כזה או משהו שהוא “צעצוע”.(אלון) כן, אתה יכול לעשות . . . ככה כותבים שפות טובות.(דותן) בדיוק . . .(אלון) . . . משהו to-last. אפשר, בעיקרון, אם למישהו יש משהו מאוד מאוד ספציפי שהוא צריך אופטימיזציה אליו, אז יש מקרים מאוד נדירים ששווה לך אולי לכתוב את זה עם משהו מעל LLVM כדי שיעשה את האופטימיזציות שאתה צריך, ולא יעבור דרך Rust או C, שהם עושים את האופטימיזציות שטובות להם ל80/20 או 90/10, אבל לא בהכרח בדיוק מה שאתה צריך לפעמים ל-High performance או משהו כזה - מעניין מאוד.באופן כללי זה חומר קריאה מאוד מעניין כי הוא מאוד נגיש, ממש נגיש.אז לדותן . . .(רן) רגע, חווה! . . .(אלון) אה! סליחה, דילגתי עליה - ויש הסבר, כי זה בעברית, וזה זז לי יימינה אז לא ראיתי אותה (עפר מזדהה) - יש את חווה סקריפטזה אתר שיושב תחת github.io בשם חווה סקריפט -” שמי חווה, וזו שפת התכנות שלי, חווה סקריפט, שפת תסריט בעברית בלבד, אשר מומרת בזמן אמת ל-JavaScript על ידי תרגום מילות המפתח ושמות המשתנים מעברית ללועזית” . . . וכן הלאה.אפשר לכתוב פה את התוכנית ולהריץ אותה תוך כדי.אז יש פה יכולת לברך(מבורך) { בקרה.תעד(״היוש, ״ + מבורך);}לברך(״עולם״);נורא קשה לנסות לתרגם את זה בלי לראות, אבל מה שזה עושה זה JavaScript Function שמקבל משהו ועושה Const וכו’ממש נחמד, בכנות - זה נשמע לי סיוט, אבל לא יודע, אולי יש ילדים בעולם שזה יהיה להם נוח, אני לא יודע . . . יודע שזה חצי בדיחה אבל זה קונספט מעניין.(דותן) כן(רן) כן, זה נראה לי בדיחה, אבל מצד שני זה נחמד, גם בדיחות זה נחמד . . . דרך אגב, הסתכלתי על ה-Contributors והייתי בטוח שה-Contributor הראשית זו אישה בשם חווה, אבל מסתבר שלא וזה קצת מאכזב, אבל לא נורא.בכל אופן נשמע (נראה) שכולם ישראלים - יש לנו כאן את אביב ודורי ועידו, יותם, עידן ועוד מישהו שלא ברור איך קוראים לו בגלל ה-username שלו ב-GitHub, אבל בכל אופן - כן, נשמע כאילו כולם ישראלים.צריך עכשיו רק לגייס חווה מאיפשהו - אז בבקשה, חבר’ה.(אלון) “דרושה חווה לפרויקט”, כן . . . זה משעשע, וחבר’ה - גם אם יש לכם ביקורת SOX וכאלה - אז לא יבינו אם אתם עושים שטויות בקוד, מומלץ בחוםנא לא לראות את זה כייעוץ משפטי, אבל . . . סתם.ועכשיו באמת לדותן.דותן - מעולה - אז נחבר את זה רק עם ה-LLVM - ה-חווה סקריפט זה Transpiler, זאת אומרת - הוא לוקח קוד ועושה תרגום, בסופו של דבר, ל-JavaScript - שזה גם תחום מעניין: אם רוצים ליצור שפה ממש ממש מהר ובקלות זה מאוד מומלץ.ראיתי ככה קצת ממה שהם השתמשו בו וזה כן בחירות נכונות, אז אם מישהו אי פעם חלם לבנות שפה, אפשר לקחת את חווה סקריפט ולבנות שפה שעושה משהו אחר, אולי בשפה אחרת, אולי ביידיש - הרמתי פה להנחתה?(רן) “יאדה-סקריפט”.(דותן) אז נתחיל - סתם רקע לכל האייטמים האלה, שכולם באים מאותו קו, כמעט כולם כתובים Rust: הייתי צריך לייעץ לכמה אנשים איך ללמוד Rust ואיפה להתחיל וכו’, ואחד הדברים שאני תמיד אומר זה “להתחיל בלבנות כלי מגניב” - איזשהו כלי Command Line - לא ללכת דווקא לבנות עכשיו איזשהו Service מטורף עם כל מה ש-Rust יודע לעשות,צריך לבנות לעצמך איזשהו “סולם מאמצים” יותר נכון, כדי לא להתייאשכל הכלים האלה, באופן מקרי, הם מאוד שימושיים ליום-יוםאחד מהם (מהכלים) ראינו באחד הפרקים הקודמים, כלי שאני משתמש בו גם ביום-יום, שעושה delta ב-Command line של Git - כתוב ב-Rust, והוא צובע יפה את כל ה-Diff-ים, וזה מאוד נחמד לי ועובד פגז.אז הכלי הראשון נקרא hors - אני מניח שזה קיצור של סוס, Horse, רק מאויית לא נכון - וזה פשוט Port או כתיבה מחדש ב-Rust לכלי שנקרא howdoi, שהיה, ועדיין, כלי שב-Terminal אתה כותב “howdoi עושה משהו?” והוא הולך ל-Stack overflow, מושך כמה דוגמאות קוד וזורק לך אותן ב-Terminal - כשחלק מהדברים שקורים כאן זה גם לצבוע לך אותו וכל מיני דברים כאלה.אז זה מעניין, למי שרוצה עכשיו להבין איך אני בונה איזשהו כלי שיוצא לרשת ולוקח ועושה דבר שהוא יותר מטריויאלי, עושה Syntax highlighting ודברים כאלה.וזה גם כלי שימושי למי שרוצה לפקס את כל העבודה שלו מסביב ה-Terminal, ועובד עם Stack overflow ורוצה לדעת בשנייה איך עושים משהו, ורוצה לקבל את הכל ב-Terminal, ולא רוצה לצאת ל-Browser.אז זה דבר אחד.הדבר השני הוא כלי שנקרא mdcat - כלומר Cat ל-Markdown - כמו Cat רגיל, רק ברגע שאתה עושה Cat ל-Markdown, או Variant של Cat ל-Markdown, אתה מקבל את זה צבועבעצם כמו ש Markdown אמור להיות, ולא השפת-מקור של Markdown.כותרת נראית כמו כותרת, Bold נראה כמו Bold, ושוב - הכל ב-Terminal.זה הכל מניפולציות מוזרות של ANSI וצבעים מעבר ל-16 צבעים שמותר וכל מיני דברים כאלה מגניבים.וכשיש תמונה, ויש Terminals שתומכים זה, אז אתה רואה פשוט תמונה . . . שימושי למי שממש, אבל ממש, אוהב לעבוד מה-Terminal, לעשות Cat ככה וקבל איזשהו קובץ Markdown ולשחק עם זה ככה.עוד פעם - מבחינת ללמוד Rust ולבנות כלי זה אחלה - אין שם יותר מכמה קבצים בודדיםזה קורא קובץ, עושה משהו, מוציא איזשהו תחכום ל-Terminal - וזה אחלה בסיס להתחיל.האייטם הבא נקרא amp.rs - זה Editor שהוא דמוי VIM (לא אלה), וחופף ל-VIM - כתוב כולו ב-Rustסופר-סופר מהיר, מן הסתם.עובד מדהים, ניסיתי אותו קצת - לא יודע אם אני אחליף את ה-VIM שלי בכזאת קלות אבל הוא מדהים.זה לאנשים שאוהבים טיפה יותר אלגוריתמיקה - ב-Editors יש המון אלגוריתמיקה - ואם אתם אוהבים אלגוריתמיקה ורוצים לעשות Manipulations, לראות איך עושים את זה ב-Rust, לראות איך זה קורה, איך זה “מריח”, איך זה שונה מלעשות עיבוד שפה בשפה אחרת.סתם לדוגמא - היום אני בונה קוד שמעבד שפה, ואיך שעושים את זה ב-Rust זה יוצא הרבה יותר קומפקטי מ-Python, וגם יותר Safe - זה אומר שאני עובד עם Float 32 או 64, אז חייב לנקוב ב-Types שלי - והקוד שיוצא הוא פשוט יותר Safe ויש בו פחות טעויות שמגלים הרבה אחר כך, אחרי שמאמנים מודל.האייטם הבא נקרא zola - זה בעצם תחליף-HUGO(אלון) תחליף זול . . .(דותן) תחליף זול ל- HUGO . . . האמת שאני לא יודע, מנסה לראות אם יש פה מישהו שיודע עברית . . . לא נראה לי, לא נראה שזה זול - “זולה” . . . (אולי לזה).אבל יש פה תחליף ל-HUGO, כשאנחנו יודעים ש-HUGO זה תחליף ל-Jekyll - ו-Jekyll זה המקור לכל ה . . . בעצם הסבא-רבא של כל הדבר הזה.במקור Jekyll נכתב ב-Rubyמה שמעניין פה זה שזה לא סתם צעצוע ולא סתם פרויקט שאפשר ללמד ממנו - כשמסתכלים על מה שיש לו ומה שלאחרים אין - יש פה טבלה מאוד פשוטה - כמעט הכל כאן מסומן ב-V עם הרבה יכולות, כשיש דברים שאין ל-HUGO ויש לו - מאוד מעניין.וכמובן, בגלל שזה Rust אז זה כנראה הכי מהיר והכי יעיל מבחינת Performance שאתה יכול לקבל.זהו - מעניין.(אלון) “הפרק בחסות Rust!” . .. (דותן) אמרתי שכל האייטמים פה הם מוכווני-Rust, כן? . . .אני רק לא יודע אם Rust יכולה לממן משהו, כי ה-Community effort כבר לא ב-Mozilla ,לצערנו . . .האייטם הבא נקרא syntect_server - זה Server שעושה Syntax highlightingיושב תחת Sourcegraphזה קשור לדברים שאני עושה ביום-יום - להתעסק עם קוד, לפרסר (Parsing) קוד, לנתח אותו - וזה אחלה פרויקט לקחת שנייה וללמוד איך אני יכול לבנות עכשיו איזשהו server מאוד מאוד קליל ופשוט כדי להגיש סוג של תוכן, או “לבנות סוג של Microservice שעושה משהו אחד”.(רן) לא נעים להגיד, אבל אני חושב ש-Sourcegraph זו חברה שהייתה מאוד ממוקדת ב-Go, מבחינה טכנולוגית, אז אני לא יודע כמה מה-Services שלהם הם העבירו ל-Rustאני רואה שיש להם כרגע, אם אני מסתכל על ה GitHub שלהם, יש להם עדיין הרבה מאוד דברים ב-Go, וגם ב-TypeScript וגם ב-Python, אפילו גם קצת ב-Rubyאני לא רואה הרבה דברים ב-Rust, בינתיים.(דותן) אני חושב שהפרויקט הזה מסמל קצת את מה שהם עושים - הם לקחו חבילה שנקראית syntect, שזה syntax highlighter שכתוב ב-Rust, באמת אחד הטובים שיש - בכלל, בכל שפה שהיא.למה זה טוב? זה אומר שהוא מכבד את חבילות ה-Syntax של Text Mate, לימים Sublime Text, והמנוע הזה ש-Text Mate ייצר ו- Sublime Text אימץ הוא אחד המנועים הטובים לעשות Syntax highlighting.ואז כתבו את syntect, שהוא מנוע שמציית לחוקים, זאת אומרת - לקחו tmbundle, שזה ה-Resource שבאמצעותו אתה מתאר את השפה, מפרסר (Parsing) אותו ועושה highlighting של Source code במהירות מסחררת.נראה לי שמה שהם רוצים לעשות זה להתשמש באותה חבילה, לעטוף אותה ב-service - כי אין להם יכולת לכתוב משהו מ-Rust או להמיר ל-Rust - ואז פשוט כדי לעשות את ה-Syntax highlighting הם פונים לתוך Service שמדבר איתם ב-API מאוד ברור.(רן) כן, זה בעצם הפרויקט היחיד שיש להם ב-Rust, חיפשתי . . .(דותן) כן, נראה הגיוני.יש פה עוד פרויקט, שלא קשור בהכרח ל-Rust, לא בדקתי - זה נקרא Depix . . . סוג של מפחיד(אלון) אני חייב לציין שאני הרבה זמן אמרתי שבטוח יש משהו כזה, כי זה פשוט גיאוני מדי . . .(דותן) אז אני אסביר רגע, כי יש פה משהו ויזואלי:השורה הראשונה היא טקסט שהוא Pixelized, זה אומרת שמישהו רצה להחביא טקסט - סודי, Classified, מה שזה לא יהיה - והפיקסליזציה היא . . (רן) רק אני אגיד רגע לפני זה - הרבה פעמים נהוג שאם אתם לוקחים איזשהו Screenshot, ואתם רוצים להחביא פרטים אישיים, אז בהרבה כלים יש יכולת לעשות פיקסליזציה.(דותן) לא הכרתי את זה . . . משהו שמזהה טקסט ואז עושה מזה פיקסלים אוטומטית?(רן) אני לא יודע אם יש אוטומטי, אבל יש כלי שאתה יכול לבוא ידנית ולסמן איזורים שזה יהפוך אותם ל-Pixels, כדי שלא יזהו - זה שימושי, נגיד, אם אתה רוצה להוציא איזשהו Screenshot . . .(אלון) ל-URL-ים לפעמים עושים את זה, כשאתה רוצה להראות עמוד, נגיד אם נכסנתי לאיזה בנק, אז אני עושה לחלק למעלה פיקסליזציה של ה-URL או כאלה, יש מלא(רן) כן . . . אז מה הכלי הזה עושה?(דותן) אז אילו שני דברים שתמיד הייתי פרנואיד לגביהם - תמיד הייתי שם בלוק שחור, ותמיד הייתי מוודא שזה לא איזה SVG או איזה PDF, ששמר לי את הבלוק השחור אבל אז מתחת יש את הטקסט . . .אז בכל אופן - הדבר הזה מקבל תמונה, עם פיקסליזציה של טקסט, ומנחש מהו הטקסט שמאחוריה . . . זאת אומרת כמו בסרטים, כשהוא אומר “תחדד את זה! תעשה פי שתיים! תחדד” - אז כזה.ורואים פה שזה די מצליח.כשאני רואה פה מול העיניים, כנראה שלאותיות מסויימות יש כזה . . נגיד L זו אות שחוצה קו . . .(רן) אני מניח שמה שיכול לעזור לו זה להסתכל על הפונט בשאר הדף, ואז הוא יודע באיזה פונט זה היה לפני זה, או פשוט, מן הסתם הוא מסתכל על המשבצות ורואה מהי דרגת הכהות או הבהירות ויכול לנחש.הוא גם יכול להכיר את האלגוריתם שבו נוצרה הפיקסליזציה, נגיד לקחת את הממוצע של חמשת הפיקסלים הקרובים ולהפוך את זה לצבע אחד או משהו כזה.ועם כל הדברים האלה - נשמע סביר לגמרי שתוכל לעשות Reverse Engineering . . .(דותן) כן, אז אתם מוזמנים לקחת ולהתחיל לגלות את מה שמחביאים מכם, לכל מי שאוהב תיאוריות קונספירציה.אז זה היה Depix.(רן) מעניין, דרך אגב - עכשיו, כשאני מגדיל את התמונה, אני רואה שגם בטקסט המקורי יש לא מעט Anti-aliasing, זאת אומרת שאם תעשה Zoom-in, אתה תראה ששזה לא רק שחור אלא שחור עם גבול של אדום ותכלתואני תוהה האם ה-Anti-aliasing הזה גם משפיע על כל הסיפור הזה, או שזה סתם איזשהו משהו בתצוגה פה, של . . .(דותן) מחשבה חכמה, אני חושב שיש לך ראש של פושע . . .האייטם הבא נקרא ImHex - וזה Hex Editor, בכותרת כתוב שזה ל-Reverse Engineers, אבל אני אוהב Hex Editors באופן כללי, אז לאו דווקא עבור Reverse Engineers.מתי אתה אוהב את זה? כשיש טקסט מוזר והכל נראה לך נורמאלי אבל משהו לא מתקמפל (Compile), ואז אתה מגלה שיש איזה Unicode או איזושהי אות בסינית שהבאת מאיפשהו או Emoji . . .(רן) הגרשיים . . . תמיד כשאתה עושה Copy-Paste מתארים, הגרשיים מתחרבנות(מזדהה גם כאן”)(דותן) בדיוק, ואז, אם אתה רוצה להתחיל לסבול, אתה יכול לנחש, אבל אתה גם יכול לפתוח Hex Editor ולראות איך דברים נראים מאחורה.ופה יש Hex Editor שהוא מאוד מאוד מרשים - מזכיר לי את ה-Editor-ים הישנים והטובים של לפני 20 שנה, עם כל מיני זיהויים של מילים ו-Patternsיש כאן אפילו כזה Distribution משוגע.דבר אחרון - למה זה נקרא ImHex? בגלל שהוא משתמש בספרייה שהזכרנו פה בעבר (?), שבונה GUI - שנקראית ImGUI, כתובה ב-Cמשתמשים בה לרוב במשחקים - כשיש לך משחק ואתה רוצה תפריט בתוך או משחק, או איזשהו Developers Console בתוך המשחקהספרייה הזו לגמרי מצויירת ידנית, לא עובדת עם מערכת ההפעלה, כל ה-Widgets כולם ממש מצויירים מאפס, ולכן אין לה שום תלות במערכת ההפעלה.גם לא מפתיע אותי שהדבר הזה הוא Cross-platform, מה שנקרא “בהליכה”.זהו, זה Open-source וחינם, אז שווה לתרום ולהוריד, או רק להוריד - מה שבא לכם.בשביל הבאזז של ה-Crypto, שמתי פה פרויקט שנקרא freqtradeזה קריפטו-בוט שעושה Trading - מה שנקרא: “מסיר אחריות”יש כאן גם Disclaimer ענק בתוך הפרויקט“נסו ותרוויחו”, מה שנקרא.(אלון) מה אתה רומז פה? (דותן) כן, אז היום שוב פעם הביטקוין חצה רף (היסטורי כלשהו), ומה שמעניין זה שכאילו זה שובר שיאים, אבל אנחנו לא רואים את הפטריות שאחרי הגשם, וכל האלגוריתמי-קסם שפותרים את ה-CAP Theorem - זה הפסיק לקרות, לא ראיתי אנשים שאומרים שפתרו את ה-CAP Theorem.(אלון) כן, אבל מה שכן מעניין בקשר לזה, זה שדווקא אתמול Goldman Sachs הוציאו הערכה שהביטקוין יגיע ל- $145,000, כי הוא ניהיה כמו זהב, וכדי שהוא יהיה כמו זהב . . . לא משנה, יש שם איזשהו הסבר מפורט למה, אבל הוא בכלל לא כסף, הוא סתם סחורה . . .(דותן) אז לכל אלה שפתרו את ה-CAP Theorem ב-Cycle הקודם, אפשר לראות האם הם יחזרו שוב פעם ויגידו שפתרו את ה-CAP Theorem עוד פעם . . .(אלון) לא, הם לא צריכים - הם יושבים על איזה אי מלא ב-Bitcoin-ים . . . לא יפתרו את זה כבר, זהו, נהיו עצלנים . . .(רן) אני רואה, דרך אגב, שיש לו ממשק ל-Telegram שנראה נחמד, ל-Bot הזה - אתה יכול להתחבר אליו ולשלוח פקודות ולקבל סטטוסים.(דותן) אז זהו - תגידו לי, כשאתם פותחים את ה-Bot אז איפה הוא נמצא ב-Telegram, ואני אדבר איתו . . .הפרויקט הבא נקרא photoprism - לאנשים שלא אוהבים את כל ה . . . נקרא לזה Hosted Photo Services, או Hosted-כל-דבר, אנשים שאוהבים פרטיותאז זה כמו Google Photos, רק ב-Hosted, כתוב ב-Go, עובד עם TensorFlow, ועוד כל מיני Buzzwords כאלה ואחרים.אני אישית בסדר עם זה שכולם גונבים לי את כל הדאטה ועושים מזה כסף, אז . . .(רן) ואתה גם משלם להם על זה, רק נזכיר . . . עכשיו Google כבר מתחילים לקחת כסף אם אתה עובר את המכסה, מה שקודם הם לא היו עושים.(דותן) אז זהו, אני בסדר עם זה, כי ה-Overhead של לנהל את זה לבד בבית - אני כבר לא . . . אני זקן מדי בשביל זה.פרויקט נוסף שנקרא TabFS - מאוד נוגע לי בנקודה כואבת אבל אישית לא ממש הצלחתי לעבוד עם זה, אולי אחרים יצליחו - זה בעצם לוקח את כל ה-Tab-ים ב-Chrome והופך אותם ל-File System, ועכשיו אפשר להתייחס ל-Tabs כמו אל File System.זהו, מן דבר משוגע כזה. אני ממש צריך משהו כזה, אבל לא יודע, לא הצלחתי לעבוד עם זה כל כך.(אלון) אני אגיד לך גם למה לא, אני נתקלתי בזה בעבר . . . בוא אני אסביר לך, בוא תקשיב(דותן) יאללה(אלון) ה-File System של ה-Mac זה אחד הדברים עם ה-UX הכי פחות נוחים בעולם הזה . . . קח משהו עם UX סבבה, או בינוני, לא משנה - ולקחת אותו, שינמכת לו את ה-UI ל-Usability כמעט בלתי אפשרית, אז איך זה יהיה נוח?(רן) אבל הוא דווקא מראה פה על שליטה מה-Command Line, אולי בשביל יכולת לעשות Copy מתוך הדפים, או משהו כזה.למרות שאני עדיין לא ממש רואה את ה-Use case פה, אבל לא יודע - אולי Search . . .(אלון) פרויקט סוף שנה של סטודנט . . .(דותן) לא חפרתי פה מספיק . . .לא חפרתי בזה כל כך לעומק, אבל אתה חייב משהו כדי לנהל את כל ה-Tab-ים, ואני לא יודע איך Chrome לא עושים משהו עם זה(רן) אולי אוטומציה? אני רואה פה יכולת לעשות כאילו קצת כמו . . . איך זה נקרא? זה ששולט ב . . . הכלי בדיקה הזה?(דותן) Selenium?(רן) כן, קצת כמו Selenium, שאתה רק צריך לפתוח את ה-Tab מחדש, ומאותו הרגע הוא מנוהל לך באופן אוטומטי, אז אולי זה גם Use case.(דותן) הייתי שמח לאיזה כפתור ב-Chrome שאני לוחץ עליו ואז הוא אוסף את כל הטאבים של Github ושם לי אותם ברשימה אחת ב-Note - וזהו.אני לא באמת צריך אותם פתוחים . . . זה נחמד שזה שמור בטאביםאני כמו כולם עם המחלה הזו של לפחד לסגור טאבבקיצור - הנה רעיון למי שמקשיב . . .(רן) אתם משתמשים ב-Tab Groups ב-Chrome, אגב? נושא אחר . . . יש ב-Chrome פיצ’ר כזה שנקרא Tab Groups והוא מאוד נחמד(דותן) מתי זה נולד, הפיצ’ר הזה?(רן) לא יודע . . .(מאי 2020?)(אלון) האמת - אני השתמשתי בו, נורא התלהבתי - ושכחתי ממנו אחרי איזה יום וזהו . . . ועכשיו הזכרת לי.זה מסוג הדברים שאתה עושה “וואו! איך זה משנה את חיי!” ואז . . .(רן) אני דווקא חייב להגיד שאני משתמש בזה הרבה, וגם הילדות שלי מתות על זה - כל פעם שאני מגיע ורואה את ה-Chrome שלהם, אז הכל זה בצבעים ועם שמותהן מאוד מתלהבות ונהנות לסדר את הטאבים שלהןאפילו עם זה טאב אחד בקבוצה, עדיין יש לזה צבע ושם.(דותן) כן, זה פותר לי משהו! נחמד, רן!(רן) יאללה, בכיף.(אלון) אפשר לסיים את הפרק, חברים, זהו! הגענו לשיא, תודה רבה.(דותן) כן . . . (טוב, אז אייטם אחרון באמת - יש פה פרויקט שנקרא redocלמי שעובד עם Swagger ורוצה להרוויח ככה משהו בחינם - זה פרויקט שלוקח את -Swagger-ים שלכם ומייצר Documentation “כמו של הגדולים”, מה שנקרא.וזהו, בעצם.רן, אוקיי - אז כמעט לפני סיום, יש לנו עוד את הקטע האחרון שלך המצחיקולים - אייטם ראשון בעניין הזה הוא OKify - זה כלי שדיברנו עליו קצת בעבר (עבר רחוק, פרק 399), של מיכאל שעובד איתי ב-Appsflyer והוציא כלי שגורם לך להרגיש טוב, שיהיה לך מצב רוח טוב.לא משנה אם ה-Build נכשל, לא משנה אם ה-Server למטה - הוא תמיד ייתן לך איזושהי מילה טובה.רק תפנה אליו והוא יתן לך איזושהי מחמאהאז עכשיו יש גם וידאו! וידאו קצרצר, של מיכאל, שמדגים את הכלי הזהאז לכו וצפו - חמש דקות של כיף והנאה וצחוקים.בעניין אחר - בוודאי שמעתם ש-Apple הכריזו על שהם הולכים עכשיו להוציא מכוניות, ולא מזמן התפרסם ב-Twitter תמונה של אחת המכוניות שאותן הם הולכים להוציאאז למי שיש לו עכבר של Apple כנראה יוכל להזדהות - אני אנסה לתאר מה אנחנו רואים פהמכונית הפוכה על הגב, ושקע הטעינה תקוע לה מלמטה . . .אז כל מי שהיה לו אי פעם עכבר של Apple כנראה יבין את המצוקה הזו, ובכל אופן אתם מוזמנים לבוא לבלוג שלנו (אתם כבר כאן אם אתם קוראים את זה…) וגם לראות את התמונה הזו.והאייטם הבא - זה בעצם עמוד פייסבוק חדש, שהכינוי שלו הוא @mad.programm3r (עם 3 במקום e בסוף)ויש שם כמה Video-ים של מפתח עצבני, שבא ורוטן על מספר תופעות שאני בטוח שכולן יהיו מוכרות לכםהחל מ-Build ב-Jenkins שנכשל ועד קונפיגורציה שלא הותקנה והוא עושה את זה בצורה מקסימה וחיננית - לכו ותראו, כל וידאו הוא קצרצר ומצחיק.(אלון) אני אמשיך - למי שעקב בפרקים הקודמים (ספציפית הקודם, 399), היה מישהו משעשע שהוציא וידאו של Stories בתוך VSCodeאז הבחור הזה הקשיב לנו, כנראה, לאחד הפרקים של 1 באפריל - הוא הוציא VSCode Tinder . . . שזה בעצם נותן לך לחפש מפתחים לפי כל מיני כישורי Skills שלהם, וזה מעיין Tinder למפתחים . . .אתה שם דוגמאות של הקוד, ואז אתה רואה אם אתה אוהב את ה-Sample Code הזה - ואם אתה רוצה לדבר עם המפתח וכן הלאה . . .אז שוב - הוא עשה וידאו היתולי ומשעשע, אתם מוזמנים לראות.(רן) וכמובן שאנחנו מודים לאותו מאזין, שאומנם לא דובר עברית אבל ברור הוא הקשיב לאחד מפרקי ה-1 באפריל שלנו, ומשם הוא גנב את הרעיון המהמם הזה.(חבר של ארני?)ואיך שלא יהיה - גם אם הרעיון נשמע לכם פחות סביר - לכו תראו את הוידאו. זה נחמד, יש שם הרבה בדיחות גיקיות, שווה צפייה.(אלון) כן, זה ממש חמוד והומוריסטי ומגניב - שווה ממש.דבר אחרון - מישהו עשה וידאו קצר ומהיר על איך שהוא עושה את השחקן The Rock - ב-CSS בלבד . . .זה די פסיכי . . .אבל זה מגניב ממש - הוא מצליח בסוף להציג אותו ממש יפה.זה חמוד לכולם, גם למי שלא אוהב CSS(רן) זה בעצם סשן של קידוד, שהוא מריץ אותו במהירות, נגיד x10 או משהו כזה, אבל מה הוא יוצר בסוף, מה זה The Rock?(אלון) זה השחקן קולנוע המופרסם (Dwayne Johnson), המתאבק . . . בסוף הוא יוצא, אם תראה עד הסוף אתה תזהה.(רן) הדמות ממשחקי הכס?(אלון) לא! The Rock, בחור קירח כזה, 1.90מ’, שיחק ב-Jumanji(דוגמא בעייתית, זה לנצח יהיה רובין וויליאמס)(רן) אהה . . .(אלון) אוי איזה עולם יש לי פה . . . גרמת לי פה לחשוף . . .(רן) מההתחלה ידעתי, רק רציתי שתגיד . . .(אלון) כן, עכשיו רק צריך להוריד את זה בעריכה, לעשות “ביפ!”(לעולם לא)(רן) מעולה, תודה, לא סיימתי לראות את הוידאו אבל אני כבר מתחיל להבין את הדמות, יפה.תודה, אז זהו - כאן אנחנו מסיימים - תודה רבה לכם, מקווים שנהנתם, נתראה בפעם הבאה.הקובץ נמצא כאן, האזנה נעימה ותודה רבה לעופר פורר על התמלול

In this episode of Data Exposed, Kate Smith shows us how to run and monitor Elastic Jobs using both PowerShell and T-SQL. [01:01] Demos More episodes in the Elastic Jobs for Azure SQL Series:Elastic Jobs in Azure SQL Database - Part 1Elastic Jobs in Azure SQL Database - Part 2Elastic Jobs in Azure SQL Database - Part 3Elastic Jobs in Azure SQL Database - Part 4Elastic Jobs in Azure SQL Database - Part 5 Resources:Elastic Jobs in Azure SQL Database – What and WhyFundamental Concepts for Elastic Jobs in Azure SQL DatabaseCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database in PowerShellCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database using T-SQLRunning, Scheduling and Monitoring Elastic Jobs in Azure SQL DatabaseTroubleshooting Common issues with Elastic Jobs in Azure SQL Database

In this episode of Data Exposed, Kate Smith shows us how to run and monitor Elastic Jobs using both PowerShell and T-SQL. [01:33] PowerShell Demo[09:43] T-SQL Demo More episodes in the Elastic Jobs for Azure SQL Series:Elastic Jobs in Azure SQL Database - Part 1Elastic Jobs in Azure SQL Database - Part 2Elastic Jobs in Azure SQL Database - Part 3Elastic Jobs in Azure SQL Database - Part 4 Resources:Elastic Jobs in Azure SQL Database – What and WhyFundamental Concepts for Elastic Jobs in Azure SQL DatabaseCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database in PowerShellCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database using T-SQLRunning, Scheduling and Monitoring Elastic Jobs in Azure SQL DatabaseTroubleshooting Common issues with Elastic Jobs in Azure SQL Database

In this episode of Data Exposed, Kate Smith walks us through the process of how to create Elastic Jobs for Azure SQL using T-SQL.[01:00] Demo[07:35] Where to create jobs[08:47] Advantages of using T-SQLMore episodes in the Elastic Jobs for Azure SQL Series:Elastic Jobs in Azure SQL Database - Part 1Elastic Jobs in Azure SQL Database - Part 2Elastic Jobs in Azure SQL Database - Part 3 Resources:Elastic Jobs in Azure SQL Database – What and WhyFundamental Concepts for Elastic Jobs in Azure SQL DatabaseCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database in PowerShellCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database using T-SQLRunning, Scheduling and Monitoring Elastic Jobs in Azure SQL DatabaseTroubleshooting Common issues with Elastic Jobs in Azure SQL Database

In this episode of Data Exposed, Kate Smith walks us through the process of how to create Elastic Jobs for Azure SQL using T-SQL.[01:00] Demo[07:35] Where to create jobs[08:47] Advantages of using T-SQLMore episodes in the Elastic Jobs for Azure SQL Series:Elastic Jobs in Azure SQL Database - Part 1Elastic Jobs in Azure SQL Database - Part 2Elastic Jobs in Azure SQL Database - Part 3 Resources:Elastic Jobs in Azure SQL Database – What and WhyFundamental Concepts for Elastic Jobs in Azure SQL DatabaseCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database in PowerShellCreating an Elastic Jobs Agent, Credentials, and Jobs for Azure SQL Database using T-SQLRunning, Scheduling and Monitoring Elastic Jobs in Azure SQL DatabaseTroubleshooting Common issues with Elastic Jobs in Azure SQL Database

Geeks on Screens with Coffee Episode 37 - Not for landlubbers Look who I bumped into at the water cooler/tea room/corridor? It's only Christina Leo ( @christinaleo ) I think this episode will mostly be about saling .... Christina Leo is a Cloud Solution Architect at Microsoft specialising in the Data and AI Platform. Prior to joining Microsoft in 2016, she worked as an independent consultant focusing on T-SQL and BI development, as well as database performance tuning in the finance and investment banking sector. When she isn't sharing her 18 years of technical experience with the data community, you can find her racing around the Solent on a 40' sailboat or often in the Thames on a little Laser. I must dash, the kettle has almost boiled. Love you! https://geeksonscreens.azurewebsites.net/ Check out the podcast version on https://anchor.fm/geeksonscreenswithcoffee (Or find it in your favorite podcast app) This content is mostly my own, but I have to honor the podfather Richard K Herring, it's inspired by and dedicated to his memory.... (Okay he's not dead, but we will be dead one day). Check out Richard's work, become a monthly badger. https://www.comedy.co.uk/podcasts/richard_herring_lst_podcast/

Join Ben Weissman on a tour through the possibilities of Big Data Clusters (BDC). He will give a brief overview about the general architecture and components of a BDC, followed by a demo where he will integrate data from external and internal sources using the same connection for python and T-SQL, and lastly come up with a real-world analytics scenario that will show you how easy it is to use them. You may not even need petabytes of data to leverage what they have to offer![01:30] It's not your Grandpa's SQL Server[02:00] Big Data Clusters Architecture[04:30] Big Data Clusters Real-World Example[06:30] Demo in Azure Data StudioAdditional Resources:Sample codeAbout Ben Weissman:Ben Weissman has been working with SQL Server since SQL Server 6.5, mainly in the BI/Datawarehousing field. He is a Data Platform MVP, MCSE Data Management and Analytics, and a Certified Data Vault Data Modeler. He is also the first BimlHero Certified Expert in Germany and a co-author of "SQL Server Big Data Clusters" and "The Biml Book". Ben has been involved in more than 150 BI Projects and is always looking for ways to become more productive and make SQL Server even more fun! Together with his team at Solisyon, Ben provides training, implementation, and consultancy for SQL/BI developers and data analysts in upper-mid-market companies around the globe.About MVPs:Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. They are always on the "bleeding edge" and have an unstoppable urge to get their hands on new, exciting technologies. They have very deep knowledge of Microsoft products and services, while also being able to bring together diverse platforms, products, and solutions, to solve real-world problems. MVPs make up a global community of over 4,000 technical experts and community leaders across 90 countries/regions and are driven by their passion, community spirit, and the quest for knowledge. Above all and in addition to their amazing technical abilities, MVPs are always willing to help others - that's what sets them apart. Learn more: https://aka.ms/mvpprogram

Join Ben Weissman on a tour through the possibilities of Big Data Clusters (BDC). He will give a brief overview about the general architecture and components of a BDC, followed by a demo where he will integrate data from external and internal sources using the same connection for python and T-SQL, and lastly come up with a real-world analytics scenario that will show you how easy it is to use them. You may not even need petabytes of data to leverage what they have to offer![01:30] It's not your Grandpa's SQL Server[02:00] Big Data Clusters Architecture[04:30] Big Data Clusters Real-World Example[06:30] Demo in Azure Data StudioAdditional Resources:Sample codeAbout Ben Weissman:Ben Weissman has been working with SQL Server since SQL Server 6.5, mainly in the BI/Datawarehousing field. He is a Data Platform MVP, MCSE Data Management and Analytics, and a Certified Data Vault Data Modeler. He is also the first BimlHero Certified Expert in Germany and a co-author of "SQL Server Big Data Clusters" and "The Biml Book". Ben has been involved in more than 150 BI Projects and is always looking for ways to become more productive and make SQL Server even more fun! Together with his team at Solisyon, Ben provides training, implementation, and consultancy for SQL/BI developers and data analysts in upper-mid-market companies around the globe.About MVPs:Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. They are always on the "bleeding edge" and have an unstoppable urge to get their hands on new, exciting technologies. They have very deep knowledge of Microsoft products and services, while also being able to bring together diverse platforms, products, and solutions, to solve real-world problems. MVPs make up a global community of over 4,000 technical experts and community leaders across 90 countries/regions and are driven by their passion, community spirit, and the quest for knowledge. Above all and in addition to their amazing technical abilities, MVPs are always willing to help others - that's what sets them apart. Learn more: https://aka.ms/mvpprogram

Tomaž Kastrun is BI & DEV developer, data analyst & data science.With more than 15 years of experiences in the field of databases, business warehouses and development, with a focus on T-SQL programming and query optimization. His focus is also data mining, statistics and research. He has been working with Microsoft SQL Server since version 2000.He is Microsoft MVP, Microsoft Certified Professional and Microsoft Trainer.Tomaž is a blogger, author of many articles, speaker at various community and Microsoft events and an avid coffee drinker.This talk has taken place after SQL Saturday 2019 in Ljubljana, Slovenia on 14 December 2019 (Saturday).Interviewer: Kamil Nowinski (T).

Tomaž Kastrun is BI & DEV developer, data analyst & data science.With more than 15 years of experiences in the field of databases, business warehouses and development, with a focus on T-SQL programming and query optimization. His focus is also data mining, statistics and research. He has been working with Microsoft SQL Server since version 2000.He is Microsoft MVP, Microsoft Certified Professional and Microsoft Trainer.Tomaž is a blogger, author of many articles, speaker at various community and Microsoft events and an avid coffee drinker.This talk has taken place after SQL Saturday 2019 in Ljubljana, Slovenia on 14 December 2019 (Saturday).Interviewer: Kamil Nowinski (T).

Greg Stephen graduated with Part-time Web Development Cohort Evening 9. Experience in fast-paced work environments over the years has provided ample practice with critical thinking, multi-tasking and working with others, all in a high stress atmosphere. Skilled in JavaScript, React.js, HTML, CSS, C# and T-SQL and Customer Service. Strong operations professional graduating from Nashville Software School.

It's 2020 - how are your T-SQL skills? Richard chats with Pam Lahoud and Pedro Lopes about the state of SQL Server today and how T-SQL continues to evolve. The conversation starts out talking about SQL Server 2019 and its shared query engine with SQL Azure - and how moving to the latest versions of SQL Server can provide benefits to your existing queries. Pedro and Pam have collaborated on a book on querying that focuses on the good patterns and anti-patterns of query writing - and how that affects your database over time!

You're probably familiar with Collations in SQL Server, but how do Collations change in Azure SQL Database Managed Instance?In this video, Anna Thomas and Colin Murphy cover a brief overview of collations (https://docs.microsoft.com/en-us/sql/t-sql/statements/collations) and how they're different in Azure SQL Database Managed Instance.[00:00] - Introduction[00:29] - What is a collation?[00:55] - Checking current collation in T-SQL and Azure Portal[02:13] - How many collations are there and listing the available collations in T-SQL[02:41] - Specifying a collation[03:24] - Usecase for collations[03:46] - Wrap-up

Python and SQL Server together? Julie Smith says yes! Richard chats with Julie about the strengths of Python in various tasks around databases including ETL. Julie talks about the tooling available for doing data extraction, transformation and loading - you can find existing Python implementations on most tasks you need to do! And as of SQL Server 2017, Python is directly supported, so you can call Python code from within T-SQL. And that's just the beginning - wait til you see Python in data analytics and visualization!

https://podcast.wikiazure.com/episode-01-msignite-experience-and-takeaways/This episode was recorded during Microsoft Ignite 2019, Microsoft is giving away Microsoft Surface Earbuds headphones to our listeners. To participate, visit https://aka.ms/PodcastSweepstakes before December 15, 2019.On this episode we talk about the overall Microsoft Ignite experience and takeaways with 3 special guests:Special GuestsRonen Ariely: Senior consultant and architect, in the fields of data platform and application development. He has more than 15 years’ experience in variety programming languages and technologies, leading development teams, and software companies, SQL & BI enterprise level solutions. Ronen is active in communities (aka pituach) in the field of Programming, SQL Server, and T-SQL.Raphi Solarsh: Acquisitions Editor at A Cloud Guru where he focuses on developing awesome Azure training content with passionate Azure professionals. His Ignite highlights were getting his first Azure certification and meeting lots of great people. If you love Azure and sharing your knowledge with the community feel free to contact him on LinkedIn to find out more about making content with ACG!Amit Malik: IT enthusiast and technology evangelist focussed on cloud and emerging technologies.He is currently associated with Spektra Systems as Director of Technology where he helps Microsoft partners grow their cloud business by using effective tools and strategies. He specializes in cloud, DevOps, software defined infrastructure, application modernization, data platform and emerging technologies around AI.Line Of Fire by Audionautix is licensed under a Creative Commons Attribution license (https://creativecommons.org/licenses/by/4.0/) Artist: http://audionautix.com/

Rob Farley is a Microsoft Certified Master, Microsoft Certified Trainer and is a recipient of the Microsoft MVP Award for SQL Server since 2006. Rob provides consulting and training courses around the world in SQL and BI topics. His community involvement keeps him quite busy -  as well as running the Adelaide [adelejde] SQL Server User Group, he is a former Director of PASS and helps co-ordinate many events around the world.He has also assisted Microsoft Learning in developing many certification exams, writes a blog and co-writes "Professional SQL Server 2012 Internals and Troubleshooting".On his blog, you can find him writing about various SQL topics, particularly T-SQL and the Query Optimizer.This talk has taken place during SQLDay Summit 2019 in Wroclaw, Poland on 13 May 2019 (Monday).Interviewers: Kamil Nowinski, Michal Sadowski.

Rob Farley is a Microsoft Certified Master, Microsoft Certified Trainer and is a recipient of the Microsoft MVP Award for SQL Server since 2006. Rob provides consulting and training courses around the world in SQL and BI topics. His community involvement keeps him quite busy -  as well as running the Adelaide [adelejde] SQL Server User Group, he is a former Director of PASS and helps co-ordinate many events around the world.He has also assisted Microsoft Learning in developing many certification exams, writes a blog and co-writes "Professional SQL Server 2012 Internals and Troubleshooting".On his blog, you can find him writing about various SQL topics, particularly T-SQL and the Query Optimizer.This talk has taken place during SQLDay Summit 2019 in Wroclaw, Poland on 13 May 2019 (Monday).Interviewers: Kamil Nowinski, Michal Sadowski.

Itzik Ben-Gan is a Mentor and Co-Founder of SolidQ. A Microsoft Data Platform MVP (Most Valuable Professional) since 1999, Itzik has delivered numerous training events around the world focused on T-SQL Querying, Query Tuning and Programming. Itzik is the author of several books including Microsoft SQL Server High-Performance T-SQL Using Window Functions, T-SQL Fundamentals and T-SQL Querying. He has written articles for SQL Server Pro, SolidQ Journal and MSDN. Itzik's speaking activities include SQLPASS, SQLBits, SQLTeach and various user groups around the world. Itzik is the author of SolidQ's Advanced T-SQL Querying, Programming and Tuning and T-SQL Fundamentals courses along with being a primary resource within the company for their T-SQL related activities.This talk has taken place during SQLDay conference in Wroclaw (Poland), on 16th May 2018 (Wednesday).Interviewers: Kamil Nowinski & Damian Widera.Do you know what Itzik does like doing the most? What kind of thing is important when writing a book? Why worth to be at conferences and also when he has started working with Microsoft SQL Server databases? Find out more in this talk.

Itzik Ben-Gan is a Mentor and Co-Founder of SolidQ. A Microsoft Data Platform MVP (Most Valuable Professional) since 1999, Itzik has delivered numerous training events around the world focused on T-SQL Querying, Query Tuning and Programming. Itzik is the author of several books including Microsoft SQL Server High-Performance T-SQL Using Window Functions, T-SQL Fundamentals and T-SQL Querying. He has written articles for SQL Server Pro, SolidQ Journal and MSDN. Itzik's speaking activities include SQLPASS, SQLBits, SQLTeach and various user groups around the world. Itzik is the author of SolidQ's Advanced T-SQL Querying, Programming and Tuning and T-SQL Fundamentals courses along with being a primary resource within the company for their T-SQL related activities.This talk has taken place during SQLDay conference in Wroclaw (Poland), on 16th May 2018 (Wednesday).Interviewers: Kamil Nowinski & Damian Widera.Do you know what Itzik does like doing the most? What kind of thing is important when writing a book? Why worth to be at conferences and also when he has started working with Microsoft SQL Server databases? Find out more in this talk.

Topics in this episode: Introduction; Installing SQL Server and Visual Studio

Cathrine loves teaching and sharing knowledge. She works as a consultant, technical architect and developer, focusing on Data Warehouse and Business Intelligence projects. Her core skills are ETL, SSIS, Biml and T-SQL development, but she enjoys everything from programming to data visualization. Outside of work she's active in the SQL Server and PASS communities as a Microsoft Data Platform MVP, BimlHero Certified Expert, author, speaker, blogger, organizer and chronic volunteer.This talk has taken place during PASS Summit in Seattle, WA, on 2nd November 2017 (Thursday).Do you want to know how Cathrine has started her adventure in IT? How difficult is to be a woman in IT and how we can help in struggling with prejudices? Who is one of her personal guru? Finally, how is a correct Norwegian pronunciation of her surname? Listen to this podcast to get to know how Cathrine became her journey to IT and how tough can be for women in the IT.

Cathrine loves teaching and sharing knowledge. She works as a consultant, technical architect and developer, focusing on Data Warehouse and Business Intelligence projects. Her core skills are ETL, SSIS, Biml and T-SQL development, but she enjoys everything from programming to data visualization. Outside of work she's active in the SQL Server and PASS communities as a Microsoft Data Platform MVP, BimlHero Certified Expert, author, speaker, blogger, organizer and chronic volunteer.This talk has taken place during PASS Summit in Seattle, WA, on 2nd November 2017 (Thursday).Do you want to know how Cathrine has started her adventure in IT? How difficult is to be a woman in IT and how we can help in struggling with prejudices? Who is one of her personal guru? Finally, how is a correct Norwegian pronunciation of her surname? Listen to this podcast to get to know how Cathrine became her journey to IT and how tough can be for women in the IT.

Constantine Kokkinos currently works as a DevOps DBA with a deep focus on performance tuning TSQL; nonetheless his first love will always be the *nix ecosystem. His primary side hustle is the dbatools project where he works on bug fixing, git support, testing, and helping newcomers jump start their contributions, wherever they need help. Interviewer: Rajib Bahar, Shabnam Khan Agenda: - You're presenting this year at PASS precon. PASS stands for "Professional Association of SQL Server". It must be really exciting to present there. Is this the first time you're presenting there or have you done it before? Tell us more about the topic you're presenting, the venue, and culture over there... - PowerShell reminds me of the unix Bash Shell, Korn shell during the years in university. In the early years, open source and commercial platforms would be diametrically opposed to each other. However, the current trend appears to be that commercial platforms are adopting more open source projects and vice versa. Is that fair to say? What has been your experience with this regard. - Can you break down the evolution of various iteration of PowerShell in SQL Server? Which releases brought about the most significant changes? - To those of us who are unaware of the DBA role, DBA is some one who administers one or more Databases or Servers or DataCenters. Let's say I am a DBA who is afraid of command prompt or powershell. Teach me some basics... - You are involved with community driven open source project called DBATools? What can you do with it? How did you get involved? Any new updates coming out from that team? - Where do you blog or tweet? How can our listeners connect with you? Music: www.freesfx.co.uk

If you need to add, remove, or replace hints from ad-hoc queries and you can't change the code, plan guides can help. See a demo of removing hints from parameterized TSQL run from an application, and get tips on how to make your plan guides work in SQL Server. Get code samples from this episode here: https://sqlworkbooks.com/2017/05/removing-query-hints-with-plan-guides-dear-sql-dba-episode-43/

Itzik Ben-Gan has many T-SQL tricks. In this show, we talk about a few of them: How to get Batch Mode Processing with regular tables How to share Identity between different tables How to use Cross Apply for Unpivot In addition, we talk about what SOS_Scheduler_Yield means and answer the question - is it always the DBA's fault? If you would like your question to be featured on the show, just head over to SQLServerRadio.com, drop us a line, and we'll talk about it in one of the upcoming shows. Items mentioned in the show: Itzik Ben-Gan - What You Need to Know About Batch Mode Window Aggregate Operator in SQL Server 2016

Tobias och Fredrik snackar med Dag König från Microsoft om att podda. Och om .NET cores nyliga släpp för Linux och Mac. Vad det innebär, vad det är som släpps och hur det visar på Microsofts anda nu och framåt. En teaser: när Kodsnack fyller 200 avsnitt vill vi hitta på något trevligt. Håll några luckor i kalendern framåt våren! Ett stort tack till Cloudnet som sponsrar vår VPS! Har du kommentarer, frågor eller tips? Vi är @kodsnack, @tobiashieta, @isallmaroon och @bjoreman på Twitter, har en sida på Facebook och epostas på info@kodsnack.se om du vill skriva längre. Vi läser allt som skickas. Gillar du Kodsnack får du hemskt gärna recensera oss i iTunes! Länkar Kristoffer var på konferens i Tyskland Dag König Xenix Bash Bash kommer till Windows Dev.cast - Dags podcast buzzfrog.se Azure - Microsofts molnplattform Kodsnack 59 - Lära sig nya saker eller att bara skeppa - ett av många avsnitt där vi pratat om att ständigt lära sig Safari books online T-SQL .NET core Red hat devnation - konferensen under vilken Dotnet core släpps ASP.net ASP.net MVC Express - utbrett ramverk för att skapa webbapplikationer med Node Roslyn - Microsofts öppna kompilator IIS Apache Kestrel Jetty Docker Containerteknologi i Windows - nanoserver och virtualiseringsteknologier, man får lättviktscontainrar Xboxen och virtualisering Mono Visual studio code Yeoman Sublime Devsum (kan man se videon av presentationen) innehöll en bra presentation om Visual studio code Plattformen Stack overflow körs på Tech empower mäter prestanda på webbramverk Microsofts SQL server kommer till Linux Typescript Node på Chakra - Microsofts javascriptmotor Titlar Microsoftplattformen När man slutar är det fara å färde Jag är en radiokille Den nya bilden av Microsoft Vi började med source open En read-only-version av koden Allting är inför öppen ridå En nystart Bara det som är nödvändigt Allting som är nödvändigt Allting är avstängt Snabba och små Affärsmodellen att sälja allt Produktutveckling i 40 år Jag har inte varit bekväm på windowsplattformen

Two weeks have passed, and another show is here! We start by talking about scary DBA sentences We continue with PASS Summit 2015 and about the sessions we’ll deliver at the summit. We then talk about: T-SQL code patterns interesting scenarios with small but very active SQL Server tables jobs that will be replaced by machines Tempdb Contention and improvements in SQL Server 2016 The power of SQL for other data solutions like Couchbase and Azure Stream Analytics And more.. Items mentioned in the show: Kendra Little – Contest: Write the Scariest DBA Sentence PASS Summit 2015 Matan Yungman – The Data Loading Performance Presentation (from 24 Hours of PASS) Dwain Camps – High Performance T-SQL using Code Patterns Matan Yungman – Can a SQL Server Table be Too Small? Will Your Job Be Done By a Machine? The Azure Stream Analytics Query Language Couchbase N1QL

Did you know that SQL was initially called “Sequel” (Structured English Query Language)? On this show, we go back to basics and talk about our favorite language – Transact-SQL (T-SQL). We cover a lot of topics in the fields of development, performance and maintenance. Among them: The logical phases of query processing Our favorite and least favorite T-SQL features T-SQL development best practices How to avoid writing long Stored Procedures How to avoid writing the “mother of all queries” The Axis of Evil T-SQL featrues Why you should (almost) never use table variables Items mentioned in the show: Matan Yungman – The Axis of Evil Itzik Ben-Gan – Creative Uses of the APPLY Operator (Part1) Itzik Ben-Gan – Creative Uses of the APPLY Operator (Part2)

It’s the monthly all-over-the-place version of the show, and this time we have a mix of various topics like the cloud, T-SQL development, performance, SQL Server 2014, events and presentation skills. We also have some more mystery around the Non-Blocking Sort issue from earlier shows. More specifically, we talk about: Is the cloud a threat or opportunity for DBAs? The ways to pass a list of values to a Stored Procedure The distribution of installations between SQL Server versions Non-Blocking Sort – Is there a hidden Iterator that helps it? Various presentations worth watching The new SQL Server 2014 Cardinality Estimator And more..   Items mentioned in the show: Guy Glantser – DBA in the Cloud: Threat or Opportunity? Adam Machanic – SQLCLR String Splitting Part 2: Even Faster, Even More Scalable Erland Sommarskog – Arrays and Lists in SQL Server 24 Hours of Pass Summit Preview Edition Julie Koesmarno – “I Want It NOW!” Data Visualization with Power View Kalen Delaney – In-Memory OLTP Internals: How is a 30x Performance Boost Possible? 24 Hours of Pass Portuguese Edition 24 Hours of Pass SQL Server 2014 Edition Sunil Agarwal – SQL Server 2014: In-Memory OLTP for Database Administrators Sunil Agarwal – Columnstore Technical Deep Dive Jimmy May – Columnstore Indexes in SQL Server 2014: Flipping the DW Faster Bit SQLSaturday Israel Aaron Bertrand – Tricks to play on your DBA Madeira Course – What’s New in SQL Server 2014?

Summary Grant Fritchey and I discuss database dev op and how it can anything to anyone. Details what he does; origin of scary DBA nickname; what is dev ops, day to day dev ops tasks; DBA and developer interactions, communications, DBA's favorite word is “no”; dev ops and source control, putting a DB in source control, integration with dev, auditing; moving DB from production to source control, ssdt, red gate sql source control, DBA resistance to source control, changing methodologies and mindsets, teething pains; tooling; keeping DB source in same place as software source, merges; benefits of source control, auditing, legislative requirements, tight coupling with dev, versioning, commenting, labeling a version; shared dev DB server vs individual dev DB server; comparing production to source control; continuous integration and automated deployment, complete replace of DB vs incremental builds, breaking changes; maturity of tools for CI, automated testing, app code vs TSQL for testing, testing before check in; replication and automated deployment; Entity Framework Migrations, breaking changes, EF Migrations vs SSDT and Red Gate SQL Source Control, up and down migrations*; ORMs, dbas don't like ORMs, performance, Glimpse to assess executed SQL; book choice – The Phoenix Projec, a parable on dev ops and making teams work together; Grant is presenting at the PASS summit full day seminar on query tuning, Grant's book SQL Server Query Performance Tuning coming in Sept, wearing rainbow fuzzies for Argenis Without Borders.

Richard flies solo at TechDays Toronto talking to Tom Moreau about some of the new T-SQL features in the next version of SQL Server, code named Denali. Tom waxed poetic about sequences and their advantages (and disadvantages) compared to identity columns. Other features Tom digs into include Lead and Lag and the new Throw feature that further extends the Try Catch capabilities in T-SQL.

Richard and Greg talk to Itzik Ben-Gan about all things T-SQL in SQL Server 2008. Itzik doesn't just talk about what's new, he talks about what's missing (in his opinion). This is a must-listen for serious SQL geeks! Check out Itzik's web site at http://www.sql.co.il/

SQL Server MVP Itzik Ben-Gan discusses techniques for using the T-SQL language to solve common problems, shows how considering a negative point of view might help and discusses the upcoming enhancemen

Bob Beauchemin talks about the new features in SQL Server 2005, and it's a lot more than just the .NET Framework! Bob gets into some great details on the dos and don'ts of framework inside SQL Server, talks about the relative strengths and weaknesses of T-SQL, and addresses various issues and new features.Support this podcast at — https://redcircle.com/net-rocks/donations