Tag Archives: Unified Analytics

Incrementally Process Data Lake Files Using Azure Databricks Autoloader and Spark Structured Streaming API.

Posted on September 30, 2020 by jbernec

Use Case. In this post, I will share my experience evaluating an Azure Databricks feature that hugely simplified a batch-based Data ingestion and processing ETL pipeline. Implementing an ETL pipeline to incrementally process only new files as they land in … Continue reading →

Posted in Azure Databricks | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , | Leave a comment

Build a Jar file for the Apache Spark SQL and Azure SQL Server Connector Using SBT.

Posted on June 29, 2020 by jbernec

The Apache Spark Azure SQL Connector is a huge upgrade to the built-in JDBC Spark connector. It is more than 15x faster than generic JDBC connector for writing to SQL Server. In this short post, I articulate the steps required … Continue reading →

Posted in Unified Analytics | Tagged , , , , , , , | 5 Comments