(Unauthorized (invalid credentials) (failed to locate provided credentials)) when running local docker server

Question

(Unauthorized (invalid credentials) (failed to locate provided credentials)) when running local docker server

Inquisitive-ME opened this issue a year ago · comments

Describe the bug

I am unable to use ClearML with a local docker server.

I have setup a local server as described here:
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac/

I run
clearml-init and it seems successful:

Port 8080 is the web port. Replacing 8080 with 8008 for API server

ClearML Hosts configuration:
Web App: http://localhost:8080
API: http://localhost:8008
File Store: http://localhost:8081

Verifying credentials ...
Credentials verified!

New configuration stored in /home/richard/clearml.conf
ClearML setup completed successfully.

But then when I try to track an experiment with:

from clearml import Task

task = Task.init(project_name='great project', task_name='best experiment')

I get the following error:

2023-09-18 08:42:51,038 - clearml.Task - ERROR - Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project$, only_fields=['id'], shallow_search=False, search_hidden=True)
Traceback (most recent call last):
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 89, in get_or_create_project
    return _get_or_create_project(session, project_name, description=description, system_tags=system_tags, project_id=project_id)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 103, in _get_or_create_project
    res = session.send(
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 109, in send
    return self._send(session=self.session, req=req, ignore_errors=ignore_errors, raise_on_errors=raise_on_errors,
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 103, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project$, only_fields=['id'], shallow_search=False, search_hidden=True)

To reproduce

Follow Linux docker setup guide to create new clearML server

run clearml-init

run

from clearml import Task

task = Task.init(project_name='great project', task_name='best experiment')

Expected behavior

I would expect that the project is created and tracking begins working

Environment

Server type self hosted
ClearML SDK Version: '1.12.2'
ClearML Server Version WebApp: 1.12.1-397 • Server: 1.12.1-397 • API: 2.26
OS Linux

Jake Henning · Answer 1 · Tue Sep 19 2023 17:50:34 GMT+0800 (China Standard Time)

Hi @Inquisitive-ME , can you share the clearml.conf file you have in your home folder?

Inquisitive-ME · Answer 2 · Tue Sep 19 2023 21:22:22 GMT+0800 (China Standard Time)

Sure it is attached below. I tried a server running locally and a server running on another computer. This is from trying to connect to a server running on another computer.

clearml.txt

Inquisitive-ME · Answer 3 · Tue Sep 19 2023 21:23:16 GMT+0800 (China Standard Time)

from this issue I tried changing the default request method to POST but that didn't seem to solve the issue
#517

Jake Henning · Answer 4 · Tue Sep 19 2023 21:53:16 GMT+0800 (China Standard Time)

This shouldn't matter unless you have some cloud ingress in the way (such as Google Cloud LBs).
Can you do curl -u <key>:<secret> <api-server-address> from the machine trying to connect to the server?

Jake Henning · Answer 5 · Tue Sep 19 2023 21:54:13 GMT+0800 (China Standard Time)

Also, can you share the entire experiment log?

Inquisitive-ME · Answer 6 · Tue Sep 19 2023 23:01:02 GMT+0800 (China Standard Time)

I am running

from clearml import Task
task = Task.init(project_name='great project 2', task_name='best experiment')

and the result is:

2023-09-19 08:12:01,065 - clearml.Task - ERROR - Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)
2023-09-19 08:12:01,131 - clearml.Task - ERROR - Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)
Traceback (most recent call last):
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 89, in get_or_create_project
    return _get_or_create_project(session, project_name, description=description, system_tags=system_tags, project_id=project_id)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 103, in _get_or_create_project
    res = session.send(
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 109, in send
    return self._send(session=self.session, req=req, ignore_errors=ignore_errors, raise_on_errors=raise_on_errors,
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 103, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-5d0eabc5465f>", line 2, in <module>
    task = Task.init(project_name='great project 2', task_name='best experiment')
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/task.py", line 563, in init
    task = cls._create_dev_task(
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/task.py", line 3524, in _create_dev_task
    task = cls(
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/task.py", line 206, in __init__
    super(Task, self).__init__(**kwargs)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 180, in __init__
    self.id = self._auto_generate(project_name=project_name, task_name=task_name, task_type=task_type)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 356, in _auto_generate
    project_id = get_or_create_project(self, project_name)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 96, in get_or_create_project
    return _get_or_create_project(session, project_name, description=description, system_tags=system_tags, project_id=project_id)
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/util.py", line 103, in _get_or_create_project
    res = session.send(
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 109, in send
    return self._send(session=self.session, req=req, ignore_errors=ignore_errors, raise_on_errors=raise_on_errors,
  File "/home/richard/.virtualenvs/deep_learning/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 103, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)

Jake Henning · Answer 7 · Tue Sep 19 2023 23:36:22 GMT+0800 (China Standard Time)

So what do you get if you do this?
curl -u <key>:<secret> <api-server-address>/auth.login

Inquisitive-ME · Answer 8 · Tue Sep 19 2023 23:50:36 GMT+0800 (China Standard Time)

I get

{"meta":{"id":"6ec98ec424e94b04958bf0171a57b913","trx":"6ec98ec424e94b04958bf0171a57b913","endpoint":{"name":"auth.login","requested_version":"2.26","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"token":"Long Token"

Jake Henning · Answer 9 · Wed Sep 20 2023 00:36:25 GMT+0800 (China Standard Time)

OK, this basically means the credentials are correct and you are communicating with the server successfully. The question is why will your SDK session start to use incorrect credentials in the middle of execution (or. alternatively, start using a different server)

Jake Henning · Answer 10 · Wed Sep 20 2023 00:37:19 GMT+0800 (China Standard Time)

Do you have any environment variables set when running the script? (those related to clearml)

Inquisitive-ME · Answer 11 · Wed Sep 20 2023 00:52:23 GMT+0800 (China Standard Time)

I don't believe so.

From what I can tell by trying to debug in backend_api/session/session.py it appears that the username is the key from above and the password is the secret sometimes when _send_request is called but sometimes _send_request gets called with an auth object that does not have a username or password.

Jake Henning · Answer 12 · Wed Sep 20 2023 00:58:20 GMT+0800 (China Standard Time)

The first step is always using the key/secret to generate a token using the auth.login endpoint (which we've tried). After that, the token will be used to send requests. Also, the error you get can only be issued by the server if you're using key/secret and not when using a token, so the fact projects.get_all endpoint fails on this is very unusual since the SDK will never send it with explicit key/secret credentials

Jake Henning · Answer 13 · Wed Sep 20 2023 01:02:13 GMT+0800 (China Standard Time)

Can you try setting the CLEARML_API_VERBOSE=1 env var and see if you get more in the task log?

Inquisitive-ME · Answer 14 · Wed Sep 20 2023 01:07:20 GMT+0800 (China Standard Time)

Below is the additional logs I get

task = Task.init(project_name='great project 2', task_name='best experiment')
2023-09-19 12:06:25,073 - clearml.session - INFO - Refreshing token from http://192.168.99.24:8008 (access_key=87AQDMJ74FBK39Q5AIDY, exp=None)
2023-09-19 12:06:25,116 - clearml.session - INFO - Received new token
2023-09-19 12:06:25,200 - clearml.session - INFO - Refreshing token from http://192.168.99.24:8008 (access_key=87AQDMJ74FBK39Q5AIDY, exp=None)
2023-09-19 12:06:25,248 - clearml.session - INFO - Received new token
2023-09-19 12:06:25,270 - clearml.Task - ERROR - Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)
2023-09-19 12:06:25,291 - clearml.session - INFO - Refreshing token from http://192.168.99.24:8008 (access_key=87AQDMJ74FBK39Q5AIDY, exp=None)
2023-09-19 12:06:25,322 - clearml.session - INFO - Received new token
2023-09-19 12:06:25,340 - clearml.Task - ERROR - Action failed <401/22: projects.get_all/v1.0 (Unauthorized (invalid credentials) (failed to locate provided credentials))> (name=^great\ project\ 2$, only_fields=['id'], shallow_search=False, search_hidden=True)
Traceback (most recent call last):

Jake Henning · Answer 15 · Wed Sep 20 2023 23:11:46 GMT+0800 (China Standard Time)

Perhaps attach the apiserver logs? Maybe there's a hint there?