Incrementally Process Data Lake Files Using Azure Databricks Autoloader and Spark Structured Streaming API.
Use Case. In this post, I will share my experience evaluating an Azure Databricks feature that hugely simplified a batch-based Data ingestion and processing ETL pipeline. Implementing an ETL pipeli…
