graadi / azure-databricks-jar-upload-jenkins-pipeline

Jenkins pipeline to automate the deployment (upload) of a JAR executable file into Azure DataBricks, to be used on DataBricks pipelines processing.

Azure Databricks JAR executable artifact deployment pipeline

The pipeline automates the upload of an executable Java artifact into Azure DataBricks.

Azure Databricks comes with a CLI tool that provides a way to interface with resources in Azure Databricks. It’s built on top of the Databricks REST API and can be used with the Workspace, DBFS, Jobs, Clusters, Libraries and Secrets API

To install the CLI, you’ll need Python version 2.7.9 and above if you’re using Python 2 or Python 3.6 and above if you’re using Python 3.

# Create a virtual environment in which you can install the Databricks CLI.
virtualenv -p /usr/bin/python2.7 databrickscli

# switch to the virtual environment you created.
source databrickscli/bin/activate

# install the Databricks CLI.
pip install databricks-cli

Pipeline stages diagram as follows: