Category: PySpark Streaming Logs

• Publish PySpark Streaming Query Metrics to Azure Log Analytics using the Data Collector REST API.

  Overview. At the time of this writing, there doesn’t seem to be built-in support for writing PySpark Structured Streaming query metrics from Azure Databricks to Azure Log Analytics. After some research, I found a work around that enables capturing the Streaming query metrics as a Python dictionary object from within a notebook session and publishing…

  jbernec

  November 27, 2020

  PySpark Streaming Logs

  Azure Databricks Cluster, Azure Log Analytics, Azure Monitor, HTTP Data Collector API, PySpark Application Logs, PySpark Streaming Logs, Python Wheel Package, setup.py