Tag Archives: Azure Databricks Cluster

Publish PySpark Streaming Query Metrics to Azure Log Analytics using the Data Collector REST API.

Posted on November 27, 2020 by jbernec

Overview. At the time of this writing, there doesn’t seem to be built-in support for writing PySpark Structured Streaming query metrics from Azure Databricks to Azure Log Analytics. After some research, I found a work around that enables capturing the … Continue reading →

Posted in PySpark Streaming Logs | Tagged , , , , , , , | 1 Comment

Build a Jar file for the Apache Spark SQL and Azure SQL Server Connector Using SBT.

Posted on June 29, 2020 by jbernec

The Apache Spark Azure SQL Connector is a huge upgrade to the built-in JDBC Spark connector. It is more than 15x faster than generic JDBC connector for writing to SQL Server. In this short post, I articulate the steps required … Continue reading →

Posted in Unified Analytics | Tagged , , , , , , , | 5 Comments