Podcasts about hdinsight

In this week's episode, we look at recent Microsoft Tech updates. By popular request, we're expanding the scope beyond just Azure to include Microsoft 365, Power Platform, and similar Microsoft platforms and capabilities. What's new? What's interesting? What's retiring? Also, Jussi asks Tobi an unexpected question.(00:00) - Intro and catching up.(04:54) - Show content starts.Show links- JavaScript challenge on Azure WAF integrated with Azure Front Door- Enforce passwordless with Azure Cache for Redis- Retirement: Azure Monitor experience (preview) in HDInsight is retiring by February 1, 2025 (microsoft.com)- Retirement: Migrating to TLS 1.2+ with the Deprecation of Outdated Security Protocols (microsoft.com)- Retirement: Azure Logic Apps Integration Service Environment is retired as of August 31, 2024 (microsoft.com)- Retirement: Azure HDInsight on AKS will retire on January 31, 2025 (microsoft.com)- New Copilot+ PCs with Intel and AMD CPUs- Cloud.microsoft details- Public Preview: Customer managed planned failover for Azure Storage (microsoft.com)- Monitoring traffic flows in Azure Firewall using VNet Flow Logs- Generally Available: Enable multifactor authentication for your tenant by 15 October 2024 (microsoft.com)- Give us feedback!

Balaji Sankaran, General Manager of Azure HDInsight, joins us to share the latest announcement from HDInsight - an updated, managed HDInsight service that runs on AKS, giving a better experience for hosting opensource analytics workloads in the cloud.   Media file: https://azpodcast.blob.core.windows.net/episodes/Episode476.mp3 YouTube: https://youtu.be/RiQzpkZeT9c Resources: HD Insight on AKS: Manage your big data needs with HDInsight on AKS Enterprise scale open-source analytics on containers (Video) Get started with HDInsights on AKS Get in touch - Ask HDInsight Other news: Announcing Microsoft Playwright Testing: Scalable end-to-end testing for modern web apps The Microsoft Azure Incubations Team launches Radius, a new open application platform for the cloud

We speak to Julie Koesmarno about Jupyter Notebooks on Azure generally, and specifically about using them to help with Incident Response. We also cover security news about .NET 6.0, Azure Monitor, HDInsight, Azure Static Web Apps, Azure Key Vault, Kubernetes, Firewall, Sentinel, Ransomware, IoT Solutions and more!

On The Cloud Pod this week, the team's collective brain power got a boost from guest hosts Rob Martin of the FinOps Foundation and Ben Garrison of JumpCloud. Also, AWS releases Data Exchange, Google automates Cloud DLP, and Azure Synapse Analytics is available for pre-purchase.  A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. JumpCloud, which offers a complete platform for identity, access, and device management — no matter where your users and devices are located.  This week's highlights

Ingesting and migrating data from a source system or legacy Hadoop stack to Azure platforms such as Synapse, ADLS and HDInsight may seem easy. But optimizing the data for a given Azure platform requires real data engineering. And operationalizing the data pipelines to run non-stop requires smart data pipelines that embed DataOps. Learn how StreamSets, a Microsoft startup program member, has helped customers speed their adoption of Azure cloud with its DataOps platform for smart data pipelines. Ready to learn more? https://streamsets.com and https://aka.ms/AzureMarketplaceStreamSets

Michael Howard, Sarah Young and Mark Simos with guest Nick Fadziewicz who is a Principal Consultant at Microsoft working in the Azure and AI team.In this episode we cover security news about Azure Datalake Storage Gen 2 ACLs, HDInsight and Azure Batch now support Private Link in preview. TLS protocol version support on Storage accounts, Azure Security Center vulnerability assessments and improved Kubernetes support, Azure Firewall DNS updates and more. Of note is a free e-book "Azure for Architects 3rd Ed" is now available.

Are you looking to gain more control of the costs of operating your growing Azure HDInsight clusters? Are you interested in driving higher utilization of your Azure HDInsight clusters? If yes, then you should definitely watch this video to learn about how to leverage Azure HDInsight Autoscale feature to help you achieve higher cost efficiency. Found our more at the following links:- Drive higher utilization of Azure HDInsight clusters with autoscale: https://azure.microsoft.com/en-us/blog/drive-higher-utilization-of-azure-hdinsight-clusters-with-autoscale/?WT.mc_id=dataexposed-c9-niner- Automatically scale Azure HDInsight clusters: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-autoscale-clusters?WT.mc_id=dataexposed-c9-niner[00:33] Introduction[01:00] Customer challenge[01:55] Static clusters[03:40] Solution is HDInsightAutoscale[06:00] Demo of deployment[09:05] Demo of load-balanced scaling[10:00] Demo of scheduled scaling[11:20] Validating cluster size

Are you looking to gain more control of the costs of operating your growing Azure HDInsight clusters? Are you interested in driving higher utilization of your Azure HDInsight clusters? If yes, then you should definitely watch this video to learn about how to leverage Azure HDInsight Autoscale feature to help you achieve higher cost efficiency. Found our more at the following links:- Drive higher utilization of Azure HDInsight clusters with autoscale: https://azure.microsoft.com/en-us/blog/drive-higher-utilization-of-azure-hdinsight-clusters-with-autoscale/?WT.mc_id=dataexposed-c9-niner- Automatically scale Azure HDInsight clusters: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-autoscale-clusters?WT.mc_id=dataexposed-c9-niner[00:33] Introduction[01:00] Customer challenge[01:55] Static clusters[03:40] Solution is HDInsightAutoscale[06:00] Demo of deployment[09:05] Demo of load-balanced scaling[10:00] Demo of scheduled scaling[11:20] Validating cluster size

Maxim Lukiyanov and Scott Hanselman discuss intricate ways in which Apache Spark jobs can fail in production and how new diagnostics tools, now available in Azure HDInsight, visualize these problems in a new intuitive way and help discover and understand them from the first glance.Jump To: [04:13] Demo Start For more information:Spark Debugging and Diagnostics Toolset for Azure HDInsight (blog post)Apache Spark for Azure HDInsight overviewCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @Azure

Maxim Lukiyanov and Scott Hanselman discuss intricate ways in which Apache Spark jobs can fail in production and how new diagnostics tools, now available in Azure HDInsight, visualize these problems in a new intuitive way and help discover and understand them from the first glance.Jump To: [04:13] Demo Start For more information:Spark Debugging and Diagnostics Toolset for Azure HDInsight (blog post)Apache Spark for Azure HDInsight overviewCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @Azure

Dhruv Goel and Scott Hanselman discuss why enterprise customers trust Apache Kafka on Azure HDInsight with their streaming ingestion needs. Integrate Kafka with Azure Active Directory for authentication and set up fine-grained access control with Apache Ranger to let multiple users access Kafka easily and securely. With Azure HDInsight, you get the best of open source on a managed platform.Jump To: [04:45] Demo Start For more information, see:Tutorial: Configure Kafka policies in HDInsight with Enterprise Security Package (Preview)Azure HDInsight - Hadoop, Spark, and Kafka ServiceAzure HDInsight pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @AzureHDInsight

Dhruv Goel and Scott Hanselman discuss why enterprise customers trust Apache Kafka on Azure HDInsight with their streaming ingestion needs. Get even more control over the security of your data at rest with Bring-Your-Own-Key encryption for Kafka. With Azure HDInsight, you get the best of open source and the security and reliability of a managed platform.Jump To: [06:57] Demo Start For more information:Bring your own key for Apache Kafka on Azure HDInsight (Preview)Azure HDInsight - Hadoop, Spark, and Kafka ServiceAzure HDInsight pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @AzureHDInsight

Dhruv Goel and Scott Hanselman discuss why enterprise customers trust Apache Kafka on Azure HDInsight with their streaming ingestion needs. Integrate Kafka with Azure Active Directory for authentication and set up fine-grained access control with Apache Ranger to let multiple users access Kafka easily and securely. With Azure HDInsight, you get the best of open source on a managed platform.Jump To: [04:45] Demo Start For more information, see:Tutorial: Configure Kafka policies in HDInsight with Enterprise Security Package (Preview)Azure HDInsight - Hadoop, Spark, and Kafka ServiceAzure HDInsight pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @AzureHDInsight

Dhruv Goel and Scott Hanselman discuss why enterprise customers trust Apache Kafka on Azure HDInsight with their streaming ingestion needs. Get even more control over the security of your data at rest with Bring-Your-Own-Key encryption for Kafka. With Azure HDInsight, you get the best of open source and the security and reliability of a managed platform.Jump To: [06:57] Demo Start For more information:Bring your own key for Apache Kafka on Azure HDInsight (Preview)Azure HDInsight - Hadoop, Spark, and Kafka ServiceAzure HDInsight pricingCreate a free account (Azure)Follow @SHanselman Follow @AzureFriday Follow @AzureHDInsight

Thomas Alex joins Lara Rubbelke to discuss how Microsoft uses Apache Kafka for HDInsight to power Siphon, a data ingestion service for internal use. Apache Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service. Microsoft created Siphon as a highly available and reliable service to ingest massive amounts of data for processing in near real time. Siphon handles ingestion of over a trillion events per day across multiple business-critical scenarios at Microsoft. In this episode, learn how Siphon uses Apache Kafka for HDInsight as its scalable pub/sub message queue.Jump To: [01:50] Demo Start For more information:Quickstart: Create a Kafka on HDInsight clusterTutorial: Use Spark Structured Streaming with Kafka on HDInsightApache Kafka for HDInsight overviewAzure HDInsight pricingSiphon: Streaming data ingestion with Apache KafkaCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

Thomas Alex joins Lara Rubbelke to discuss how Microsoft uses Apache Kafka for HDInsight to power Siphon, a data ingestion service for internal use. Apache Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service. Microsoft created Siphon as a highly available and reliable service to ingest massive amounts of data for processing in near real time. Siphon handles ingestion of over a trillion events per day across multiple business-critical scenarios at Microsoft. In this episode, learn how Siphon uses Apache Kafka for HDInsight as its scalable pub/sub message queue.Jump To: [01:50] Demo Start For more information:Quickstart: Create a Kafka on HDInsight clusterTutorial: Use Spark Structured Streaming with Kafka on HDInsightApache Kafka for HDInsight overviewAzure HDInsight pricingSiphon: Streaming data ingestion with Apache KafkaCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

Murali Krishnaprasad joins Lara Rubbelke to discuss Interactive Query (also called Hive LLAP, or Low Latency Analytical Processing, or Live Long and Process), which is an Azure HDInsight cluster type. Interactive Query supports in-memory caching, which makes Hive queries super-fast and interactive. See how to use HDInsight Interactive Query to analyze extremely large datasets (~100TB) in common file formats such as ORC and CSV using common BI/SQL tools including Zeppelin notebooks and VS Code.Jump To: [04:12] Demo Start For more information, see:Use Interactive Query with HDInsight (docs)HDInsight overviewHDInsight pricing detailsCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

Murali Krishnaprasad joins Lara Rubbelke to discuss Interactive Query (also called Hive LLAP, or Low Latency Analytical Processing, or Live Long and Process), which is an Azure HDInsight cluster type. Interactive Query supports in-memory caching, which makes Hive queries super-fast and interactive. See how to use HDInsight Interactive Query to analyze extremely large datasets (~100TB) in common file formats such as ORC and CSV using common BI/SQL tools including Zeppelin notebooks and VS Code.Jump To: [04:12] Demo Start For more information, see:Use Interactive Query with HDInsight (docs)HDInsight overviewHDInsight pricing detailsCreate a free account (Azure)Follow @sqlgal Follow @AzureFriday Follow @AzureHDInsight

@ROCKYRECRUITER -- LINKEDIN:  I specialize in full-cycle technical recruiting for: - Broadcast Media: Software and Infrastructure talent experienced in TV Everywhere (TVE), Video-On-Demand (VOD), Over-The-Top (OTT), Online Video Platforms (OVP), and Media Asset Management (MAM).  - Broadcast Media: Ad Sales, Digital Analytics and Consumer Insights, Distribution.  - Cloud: Amazon Web Services (AWS) Global Infrastructure: EC2, S3, Lambda, CloudFront, CloudWatch, DynamoDB, Aurora, etc.  - Senior and Director-Level Leaders for Projects and Products, and Processes -  Attracting top 10% of engineering talent in Java, JavaScript, Go, Python and Ruby through committed repositories.  -  Big Data: Apache Hadoop, Hive, Spark, HDInsight, NoSQL DBs - COTS: Workday HCM implementations (all modules). 

@RockVanNiel -- LINKEDIN:  I specialize in full-cycle technical recruiting for: - Broadcast Media: Software and Infrastructure talent experienced in TV Everywhere (TVE), Video-On-Demand (VOD), Over-The-Top (OTT), Online Video Platforms (OVP), and Media Asset Management (MAM).  - Broadcast Media: Ad Sales, Digital Analytics and Consumer Insights, Distribution.  - Cloud: Amazon Web Services (AWS) Global Infrastructure: EC2, S3, Lambda, CloudFront, CloudWatch, DynamoDB, Aurora, etc.  - Senior and Director-Level Leaders for Projects and Products, and Processes -  Attracting top 10% of engineering talent in Java, JavaScript, Go, Python and Ruby through committed repositories.  -  Big Data: Apache Hadoop, Hive, Spark, HDInsight, NoSQL DBs - COTS: Workday HCM implementations (all modules). 

Raghav Mohan joins Scott Hanselman to talk about Apache Kafka on HDInsight, which added the open-source distributed streaming platform last year to complete a scalable, big data streaming scenario on Azure. The service has come a long way since - processing millions of events/sec, petabytes of data/day to power scenarios like Toyota's connected car, Office 365's clickstream analytics, fraud detection for large banks, etc. Deploy managed, cost-effective Kafka clusters on Azure HDInsight with a 99.9% SLA with just 4 clicks or pre-created ARM templates.For more information, see:Announcing Apache Kafka for Azure Hdinsight General AvailabilityApache Kafka for HDInsightStart with Apache Kafka on HDInsight (docs)Use Apache Kafka with Storm on HDInsight (docs)Apache Spark streaming (DStream) example with Kafka on HDInsight (docs)Analyze logs for Apache Kafka on HDInsight (docs)Create a Free Account (Azure)Follow @SHanselman Follow @AzureFriday Follow @theycallmeraj

Raghav Mohan joins Scott Hanselman to talk about Apache Kafka on HDInsight, which added the open-source distributed streaming platform last year to complete a scalable, big data streaming scenario on Azure. The service has come a long way since - processing millions of events/sec, petabytes of data/day to power scenarios like Toyota's connected car, Office 365's clickstream analytics, fraud detection for large banks, etc. Deploy managed, cost-effective Kafka clusters on Azure HDInsight with a 99.9% SLA with just 4 clicks or pre-created ARM templates.For more information, see:Announcing Apache Kafka for Azure Hdinsight General AvailabilityApache Kafka for HDInsightStart with Apache Kafka on HDInsight (docs)Use Apache Kafka with Storm on HDInsight (docs)Apache Spark streaming (DStream) example with Kafka on HDInsight (docs)Analyze logs for Apache Kafka on HDInsight (docs)Create a Free Account (Azure)Follow @SHanselman Follow @AzureFriday Follow @theycallmeraj

Mitä on data-analytiikka pilvessä ja miten big dataa murskataan Azurella? Bigdatapumpin Oskari Heikkinen kertoo, millaista data-analyysi on, mitä data scientistin pitää osata ja millaisia työkaluja Azure tarjoaa. Tutuksi tulevat niin SQL Data Warehouse, HDinsight kuin Azure Data Lakekin.

We talk to Janet Yeilding about Azure Stream Analytics. How StackExchange handles 560 million pageview each month. And unit tests are finally getting smarter.

Im zweiten Teil dieser Video-Podcast Serie dreht sich alles um die Installation von HDInsight. Dabei zeige ich einerseits wie man den lokalen Microsoft HDInsight Emulator mittels Web Platform Installer installiert. Andererseits stelle ich vor wie der Windows Azure HDInsight Dienst via Management Portal, PowerShell Skript oder Windows Azure CLI Tool erstellt werden kann.

Im ersten Teil dieser Podcast Serie beschäftigen wir uns mit den Fragen "Was ist Big Data?" und "Was ist Hadoop bzw. Microsoft HDInsight?"

Carl and Richard talk to Andrew Brust about Big Data. Andrew starts off connecting together the definitions of business intelligence, data analytics, OLAP, data warehousing and big data. They're all related, even though they've come at the problem of understanding data from different directions. The conversation digs deeply into Hadoop, the Linux-centric MapReduce technology that has come to define the idea of Big Data, as well as Microsoft's implementation once called Project Isotope and now known as HDInsight. How big is Big Data? That's up to you!Support this podcast at — https://redcircle.com/net-rocks/donations

Carl and Richard talk to Andrew Brust about Big Data. Andrew starts off connecting together the definitions of business intelligence, data analytics, OLAP, data warehousing and big data. They're all related, even though they've come at the problem of understanding data from different directions. The conversation digs deeply into Hadoop, the Linux-centric MapReduce technology that has come to define the idea of Big Data, as well as Microsoft's implementation once called Project Isotope and now known as HDInsight. How big is Big Data? That's up to you!Support this podcast at — https://redcircle.com/net-rocks/donations