SelfHosted ClearML ApiServer Losses Fixed Users after restart
ramberg opened this issue · comments
Describe the bug
Since my latest upgrade I'm unable to login into the WebUI with the users stated in apiserver.conf. In order to drill down on the issue I analyzed it using a fresh / clean ClearML instance (see reproduction, please).
To reproduce
For reproduction I setup a completely fresh local ClearML instance using docker compose, with a fresh data directory and an "empty" config while only specifying the apiserver.conf as follows:
auth {
fixed_users {
enabled: true
pass_hashed: true
users: [
{
username: "jane"
password: "JDJiJDEyJFhYVUxFV3pnWnlNZFlGMnc1anJYTS5pUW5lOFlGRDhjTkRkZ21haHlJSkY2TDJZMlMxNFcy"
name: "Jane Doe"
}
]
}
}
I then started ClearML using docker-compose up
docker-compose -f docker-compose.yml up
and was able to login using the user jane. The apiserver logged the expected following lines during startup:
[2024-05-02 12:25:24,609] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Finished mongodb migrations
[2024-05-02 12:25:24,615] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating company: clearml
[2024-05-02 12:25:24,635] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: apiserver
[2024-05-02 12:25:24,639] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: webserver
[2024-05-02 12:25:24,641] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: services_agent
[2024-05-02 12:25:24,643] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: tests
[2024-05-02 12:25:24,649] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled
[2024-05-02 12:25:24,650] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: Jane Doe
[2024-05-02 12:25:24,653] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/examples-1.9.1.zip
[2024-05-02 12:25:28,172] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/nvidia.zip
[2024-05-02 12:25:28,854] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/services.zip
[2024-05-02 12:25:28,891] [9] [WARNING] [clearml.apiserver.mongo.initialize] Invalid pre-populate entry /opt/clearml/db-pre-populate, skipping
[2024-05-02 12:25:28,892] [9] [INFO] [clearml.service_repo] Loading services from /opt/clearml/apiserver/services
I then restarted the apiserver using docker itself (but that does not matter, it also happens using compose down / up)
docker restart clearml-apiserver
From now on I will receive the message " Invalid User/Password combination"; and that is because during the second startup (first restart) the apiserver deleted the user Jane from the auth database, and logged:
clearml-apiserver | [2024-05-02 12:26:37,888] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Finished mongodb migrations
clearml-apiserver | [2024-05-02 12:26:37,893] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: webserver
clearml-apiserver | [2024-05-02 12:26:37,898] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: tests
clearml-apiserver | [2024-05-02 12:26:37,899] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled
clearml-apiserver | [2024-05-02 12:26:37,900] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: Jane Doe
clearml-apiserver | [2024-05-02 12:26:37,902] [9] [INFO] [clearml.apiserver.mongo.initialize] Removing user that is no longer in configuration: 3cd2831cdf07aede577ec69d4671c92c 3cd2831cdf07aede577ec69d4671c92c@example.com Jane Doe
clearml-apiserver | [2024-05-02 12:26:37,903] [9] [INFO] [clearml.service_repo] Loading services from /opt/clearml/apiserver/services
Expected behaviour
I would like the users stated in the apiserver.conf to be persistent over restarts.
Environment
- Self-Hosted under Linux
- redis:6.2
- mongo:4.4.29
- docker.elastic.co/elasticsearch/elasticsearch:7.17.18
- allegroai/clearml:1.15.0-472
- allegroai/clearml-agent-services:services-1.3.0-77
Related Discussion
None so far.
Workaround
Adding delete_missing_autocreated_users
to the apiserver.conf circumvents the issue for now.
Thanks for reporting @ramberg. A v1.15.1 release should become available soon with a fix.
Hello, I'm having the same issue
- is this the correct way to add the workaround? :
auth {
fixed_users {
delete_missing_autocreated_users: true
enabled: true
pass_hashed: false
users: [.... ]
}
}
- Is there a place where I can see the available options for apiserver.conf?
@volcanihpc look to the default file.
You'll note you misplaced the entry, as it should not be inside fixed_users
.
Hey @ramberg! clearml-server 1.16.1 is now out and this issue has been resolved. Let us know if there are any issues :)