Monthly Archives: November 2020

Publish PySpark Streaming Query Metrics to Azure Log Analytics using the Data Collector REST API.

Posted on November 27, 2020 by jbernec

Overview. At the time of this writing, there doesn’t seem to be built-in support for writing PySpark Structured Streaming query metrics from Azure Databricks to Azure Log Analytics. After some research, I found a work around that enables capturing the … Continue reading →

Posted in PySpark Streaming Logs | Tagged , , , , , , , | 1 Comment

Write Data from Azure Databricks to Azure Dedicated SQL Pool(formerly SQL DW) using ADLS Gen 2.

Posted on November 13, 2020 by jbernec

In this post, I will attempt to capture the steps taken to load data from Azure Databricks deployed with VNET Injection (Network Isolation) into an instance of Azure Synapse DataWarehouse deployed within a custom VNET and configured with a private … Continue reading →

Posted in Apache Spark, Azure Synapse DW | Tagged , , , , , , , , , , | 2 Comments