Monthly Archives: November 2020

Publish PySpark Streaming Query Metrics to Azure Log Analytics using the Data Collector REST API.

Overview. At the time of this writing, there doesn’t seem to be built-in support for writing PySpark Structured Streaming query metrics from Azure Databricks to Azure Log Analytics. After some research, I found a work around that enables capturing the … Continue reading

Posted in PySpark Streaming Logs | Tagged , , , , , , , | Leave a comment

Write Data from Azure Databricks to Azure Dedicated SQL Pool(formerly SQL DW) using ADLS Gen 2.

In this post, I will attempt to capture the steps taken to load data from Azure Databricks deployed with VNET Injection (Network Isolation) into an instance of Azure Synapse DataWarehouse deployed within a custom VNET and configured with a private … Continue reading

Posted in Apache Spark, Azure Synapse DW | Tagged , , , , , , , , , , | Leave a comment