Tag: Unified Analytics

• Incrementally Process Data Lake Files Using Azure Databricks Autoloader and Spark Structured Streaming API.

  Use Case. In this post, I will share my experience evaluating an Azure Databricks feature that hugely simplified a batch-based Data ingestion and processing ETL pipeline. Implementing an ETL pipeline to incrementally process only new files as they land in a Data Lake in near real time (periodically, every few minutes/hours) can be complicated. Since…

  jbernec

  September 30, 2020

  Azure Databricks

  Analytics, Apache Spark, Apache Spark Connector, Apache Spark JDBC Connector, Autoloader, Azure Data Factory, Azure Data Lake Gen 2, Azure Databricks, Azure Event Grid, Azure SQL DB, Big Data, cloudFiles, CSV, Data, ETL, Ingestion, JSON, Pipeline, PySpark, Python, Queue Service, schema, StructType, Structured Streaming API, udf, Unified Analytics

• Build a Jar file for the Apache Spark SQL and Azure SQL Server Connector Using SBT.

  The Apache Spark Azure SQL Connector is a huge upgrade to the built-in JDBC Spark connector. It is more than 15x faster than generic JDBC connector for writing to SQL Server. In this short post, I articulate the steps required to build a JAR file from the Apache Spark connector for Azure SQL that can…

  jbernec

  June 29, 2020

  Unified Analytics

  Apache Spark, Azure Databricks, Azure Databricks Cluster, Microsoft, sbt, Spark, sql-spark-connector, Unified Analytics