dbx by Databricks Labs

DataBricks CLI eXtensions - aka dbx is a CLI tool for advanced Databricks jobs management.

local

Concept

dbx simplifies jobs launch and deployment process across multiple environments. It also helps to package your project and deliver it to your Databricks environment in a versioned fashion. Designed in a CLI-first manner, it is built to be actively used both inside CI/CD pipelines and as a part of local tooling for fast prototyping.

Requirements

Python Version > 3.6
pip or conda

Installation

with pip:

pip install dbx

Quickstart

Please refer to the Quickstart section.

Documentation

Please refer to the docs page.

Differences from other tools

Tool	Comment
databricks-cli	dbx is NOT a replacement for databricks-cli. Quite the opposite - dbx is heavily dependent on databricks-cli and uses most of the APIs exactly from databricks-cli SDK.
mlflow cli	dbx is NOT a replacement for mlflow cli. dbx uses some of the MLflow APIs under the hood to store serialized job objects, but doesn't use mlflow CLI directly.
Databricks Terraform Provider	While dbx is primarily oriented on versioned job management, Databricks Terraform Provider provides much wider set of infrastructure settings. In comparison, dbx doesn't provide infrastructure management capabilities, but brings more flexible deployment and launch options.
Databricks Stack CLI	Databricks Stack CLI is a great component for managing a stack of objects. dbx concentrates on the versioning and packaging jobs together, not treating files and notebooks as a separate component.

Limitations

Development:
- dbx currently doesn't provide interactive debugging capabilities.
  If you want to use interactive debugging, you can use Databricks Connect + dbx for deployment operations.
- dbx execute only supports Python-based projects which use spark_python_task (Notebooks or Repos are not supported in dbx execute).
- dbx execute can only be used on clusters with Databricks ML Runtime 7.X or higher.
General:
- dbx doesn't support Delta Live Tables at the moment.

Versioning

For CLI interfaces, we support SemVer approach. However, for API components we don't use SemVer as of now. This may lead to instability when using dbx API methods directly.

Legal Information

This software is provided as-is and is not officially supported by Databricks through customer technical support channels. Support, questions, and feature requests can be communicated through the Issues page of this repo. Please see the legal agreement and understand that issues with the use of this code will not be answered or investigated by Databricks Support.

Feedback

Issues with dbx? Found a bug? Have a great idea for an addition? Feel free to file an issue.

Contributing

Please find more details about contributing to dbx in the contributing doc.

copdips / dbx