Category: Cluster Init Scripts

• Configure a Databricks Cluster-scoped Init Script in Visual Studio Code.

  Databricks is a distributed data analytics and processing platform designed to run in the Cloud. This platform is built on Apache Spark which is currently at version 2.4.4. In this post, I will demonstrate the deployment and installation of custom R based machine learning packages into Azure Databricks Clusters using Cluster Init Scripts. So, what…

  jbernec

  March 2, 2020

  Apache Spark, Bash, Cluster Init Scripts, Databricks Notebooks, Install.packages(), Logs, R, Shell

  Apache Spark, Azure Databricks, Bash, Cluster Init Scripts, Databricks CLI, Databricks Notebooks, Install.packages(), Logs, R

• Automate Azure Databricks Job Execution using Custom Python Functions.

  Introduction Thanks to a recent Azure Databricks project, I’ve gained insight into some of the configuration components, issues and key elements of the platform. Let’s take a look at this project to give you some insight into successfully developing, testing, and deploying artifacts and executing models. One note: This post is not meant to be…

  jbernec

  March 23, 2019

  Apache Spark, Azure Databricks, Cluster Init Scripts, Databricks Notebooks, Python

  Azure Data Factory, Databricks, Databricks CLI, Git, Jobs API, Jobs REST API, Logging module, MLFlow, Python, Subprocess module, Version Control