Token which works for databricks cli does not work for dbx commands
kenanzh opened this issue · comments
Expected Behavior
dbx commands should not raise invalid access token errors when they are valid with databricks cli.
Current Behavior
% dbx deploy -e dev --assets-only
[dbx][2023-08-30 11:07:53.117] 🔎 Deployment file is not provided, searching in the conf directory
[dbx][2023-08-30 11:07:53.118] 💡 Auto-discovery found deployment file conf/deployment.yml
[dbx][2023-08-30 11:07:53.119] 🆗 Deployment file conf/deployment.yml exists and will be used for deployment
[dbx][2023-08-30 11:07:53.119] Starting new deployment for environment dev
[dbx][2023-08-30 11:07:53.120] Using profile provided from the project file
[dbx][2023-08-30 11:07:53.120] Found auth config from provider DbxEnvironmentVariableConfigProvider, verifying it
[dbx][2023-08-30 11:07:53.121] Found auth config from provider DbxEnvironmentVariableConfigProvider, verification successful
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /.../site-packages/databricks_cli/sdk/api_client.py:166 in perform_query │
│ │
│ 163 │ │ │ │ │ resp = self.session.request(method, self.get_url(path, version=versi │
│ 164 │ │ │ │ │ │ │ │ │ │ │ │ verify = self.verify, headers = headers) │
│ 165 │ │ try: │
│ ❱ 166 │ │ │ resp.raise_for_status() │
│ 167 │ │ except requests.exceptions.HTTPError as e: │
│ 168 │ │ │ message = e.args[0] │
│ 169 │ │ │ try: │
│ │
│ /Users/kenan/Library/Caches/pypoetry/virtualenvs/datascience-databricks-WE8lb4aI-py3.10/lib/pyth │
│ on3.10/site-packages/requests/models.py:1021 in raise_for_status │
│ │
│ 1018 │ │ │ ) │
│ 1019 │ │ │
│ 1020 │ │ if http_error_msg: │
│ ❱ 1021 │ │ │ raise HTTPError(http_error_msg, response=self) │
│ 1022 │ │
│ 1023 │ def close(self): │
│ 1024 │ │ """Releases the connection back to the pool. Once this method has been │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HTTPError: 403 Client Error: Forbidden for url: https://dbc-dafad129-c9ae.cloud.databricks.com/api/2.0/workspace/mkdirs
During handling of the above exception, another exception occurred:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /.../site-packages/dbx/commands/deploy.py:104 in deploy │
│ │
│ 101 │ else: │
│ 102 │ │ headers = None │
│ 103 │ │
│ ❱ 104 │ api_client = prepare_environment(environment_name, headers) │
│ 105 │ additional_tags = parse_multiple(tags) │
│ 106 │ no_rebuild = no_rebuild or no_package │
│ 107 │
│ │
│ /.../site-packages/dbx/utils/common.py:31 in prepare_environment │
│ │
│ 28 def prepare_environment(env_name: str, headers: Optional[Dict[str, str]] = None) -> ApiC │
│ 29 │ info = ProjectConfigurationManager().get(env_name) │
│ 30 │ transfer_profile_name(info) │
│ ❱ 31 │ MlflowStorageConfigurationManager.prepare(info) │
│ 32 │ return DatabricksClientProvider.get_v2_client(headers) │
│ 33 │
│ 34 │
│ │
│ /.../site-packages/dbx/api/storage/mlflow_based.py:31 in prepare │
│ │
│ 28 │ @classmethod │
│ 29 │ def prepare(cls, properties: EnvironmentInfo): │
│ 30 │ │ cls._setup_tracking_uri() │
│ ❱ 31 │ │ cls._prepare_workspace_dir(properties) │
│ 32 │ │ cls._setup_experiment(properties) │
│ 33 │ │
│ 34 │ @classmethod │
│ │
│ /.../site-packages/dbx/api/storage/mlflow_based.py:39 in _prepare_workspace_dir │
│ │
│ 36 │ │ api_client = DatabricksClientProvider().get_v2_client() │
│ 37 │ │ p = str(PurePosixPath(env.properties.workspace_directory).parent) │
│ 38 │ │ service = WorkspaceService(api_client) │
│ ❱ 39 │ │ service.mkdirs(p) │
│ 40 │ │
│ 41 │ @staticmethod │
│ 42 │ def _get_experiment_safe(name: str) -> Optional[Experiment]: │
│ │
│ /.../site-packages/databricks_cli/sdk/service.py:1007 in mkdirs │
│ │
│ 1004 │ │ _data = {} │
│ 1005 │ │ if path is not None: │
│ 1006 │ │ │ _data['path'] = path │
│ ❱ 1007 │ │ return self.client.perform_query('POST', '/workspace/mkdirs', data=_data, header │
│ 1008 │ │
│ 1009 │ def list(self, path, headers=None): │
│ 1010 │ │ _data = {} │
│ │
│ /.../site-packages/databricks_cli/sdk/api_client.py:174 in perform_query │
│ │
│ 171 │ │ │ │ message += '\n Response from server: \n {}'.format(reason) │
│ 172 │ │ │ except ValueError: │
│ 173 │ │ │ │ pass │
│ ❱ 174 │ │ │ raise requests.exceptions.HTTPError(message, response=e.response) │
│ 175 │ │ return resp.json() │
│ 176 │
│ 177 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HTTPError: 403 Client Error: Forbidden for url: <my-host-redacted>/api/2.0/workspace/mkdirs
Response from server:
{'error_code': '403', 'message': 'Invalid access token.'}
But I can list for example the available cluster spark versions
% databricks clusters spark-versions --profile dev
{
"versions": [
{
"key": "12.2.x-scala2.12",
"name": "12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)"
},
{
"key": "11.3.x-photon-scala2.12",
"name": "11.3 LTS Photon (includes Apache Spark 3.3.0, Scala 2.12)"
},
...
Steps to Reproduce (for bugs)
Generate a new token using databricks ui
initialize new project using dbx init
configure a profile using token
execute dbx command
Context
I am trying to use dbx commands to deploy a job using --from-assets as part of our development workflow. This worked for me until my PAT expired. I generated a new one following this databricks documentation. I've replaced the token in my databrickscfg profile, and confirmed I can run databricks commands like listing spark versions and the workspace filesystem. However when trying to run dbx commands such as deploy or repo sync, I get {'error_code': '403', 'message': 'Invalid access token.'}.
I don't think I've missed anything, but hope its user error!
Your Environment
- dbx version used: 0.8.15
- Databricks Runtime version: 13.2
Hello @kenanzh,
Have you tried to reconfigure your dbx? It might be caused by a new update of the databricks-cli.
Please give it a try and let us know.
I have a smiliar problem, but slightly different:
For one of my workspace I get the following error message when I call dbx deploy workflowname -e PROD:
HTTPError: 401 Client Error: Unauthorized for url: https://adb-xxxxx.10.azuredatabricks.net/api/2.0/workspace/mkdirs
This puzzles me as I can use the same token if I directly call
https://adb-xxxxx.10.azuredatabricks.net/api/2.0/workspace/mkdirs
via Postman as post request. And it seems to work via dbx for another workspace. How is that possible? Could it be caused by some workspace settings? Or do I have to give some permissions somewhere to the service principals that I use to run my workflows?
Thanks for any hints.
Ok, my problem was a protected variable in gitlab which can used by pipelines on protected branches or tags. And it makes sense as this precaution or versioning does not take place in the test environment.
@AnastasiaProkaieva Sorry for the slow response here.
I have tried going through the dbx configure process again, but still the same issue. It's weird, as I can successfully run dbx sync repo ... commands but I cannot run dbx deploy .... I can share my project.json file with some redactions if it helps!