SelfHosted ClearML ApiServer Losses Fixed Users after restart

[Ronny Amberg]

Describe the bug

Since my latest upgrade I'm unable to login into the WebUI with the users stated in apiserver.conf. In order to drill down on the issue I analyzed it using a fresh / clean ClearML instance (see reproduction, please).

To reproduce

For reproduction I setup a completely fresh local ClearML instance using docker compose, with a fresh data directory and an "empty" config while only specifying the apiserver.conf as follows:

auth {
    fixed_users {
        enabled: true
        pass_hashed: true
        users: [
             {
                 username: "jane"
                 password: "JDJiJDEyJFhYVUxFV3pnWnlNZFlGMnc1anJYTS5pUW5lOFlGRDhjTkRkZ21haHlJSkY2TDJZMlMxNFcy"
                 name: "Jane Doe"
             }
        ]
    }
}

I then started ClearML using docker-compose up

docker-compose -f docker-compose.yml up

and was able to login using the user jane. The apiserver logged the expected following lines during startup:

[2024-05-02 12:25:24,609] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Finished mongodb migrations
[2024-05-02 12:25:24,615] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating company: clearml
[2024-05-02 12:25:24,635] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: apiserver
[2024-05-02 12:25:24,639] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: webserver
[2024-05-02 12:25:24,641] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: services_agent
[2024-05-02 12:25:24,643] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: tests
[2024-05-02 12:25:24,649] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled
[2024-05-02 12:25:24,650] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: Jane Doe
[2024-05-02 12:25:24,653] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/examples-1.9.1.zip
[2024-05-02 12:25:28,172] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/nvidia.zip
[2024-05-02 12:25:28,854] [9] [INFO] [clearml.apiserver.mongo.initialize] Pre-populating using /opt/clearml/db-pre-populate/services.zip
[2024-05-02 12:25:28,891] [9] [WARNING] [clearml.apiserver.mongo.initialize] Invalid pre-populate entry /opt/clearml/db-pre-populate, skipping
[2024-05-02 12:25:28,892] [9] [INFO] [clearml.service_repo] Loading services from /opt/clearml/apiserver/services

I then restarted the apiserver using docker itself (but that does not matter, it also happens using compose down / up)

docker restart clearml-apiserver

From now on I will receive the message " Invalid User/Password combination"; and that is because during the second startup (first restart) the apiserver deleted the user Jane from the auth database, and logged:

clearml-apiserver | [2024-05-02 12:26:37,888] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Finished mongodb migrations
clearml-apiserver | [2024-05-02 12:26:37,893] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: webserver
clearml-apiserver | [2024-05-02 12:26:37,898] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: tests
clearml-apiserver | [2024-05-02 12:26:37,899] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled
clearml-apiserver | [2024-05-02 12:26:37,900] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: Jane Doe
clearml-apiserver | [2024-05-02 12:26:37,902] [9] [INFO] [clearml.apiserver.mongo.initialize] Removing user that is no longer in configuration: 3cd2831cdf07aede577ec69d4671c92c        [e-mail]    Jane Doe
clearml-apiserver | [2024-05-02 12:26:37,903] [9] [INFO] [clearml.service_repo] Loading services from /opt/clearml/apiserver/services

Expected behaviour

I would like the users stated in the apiserver.conf to be persistent over restarts.

Environment

• Self-Hosted under Linux
  • redis:6.2
  • mongo:4.4.29
  • docker.elastic.co/elasticsearch/elasticsearch:7.17.18
  • allegroai/clearml:1.15.0-472
  • allegroai/clearml-agent-services:services-1.3.0-77

Related Discussion

None so far.

Workaround

Adding delete_missing_autocreated_users to the apiserver.conf circumvents the issue for now.

[Noam Wasersprung]

Thanks for reporting @ramberg. A v1.15.1 release should become available soon with a fix.

[volcanihpc]

Hello, I'm having the same issue

1. is this the correct way to add the workaround? :

 auth {
   fixed_users {
       delete_missing_autocreated_users: true
       enabled: true
       pass_hashed: false
       users: [.... ]
}
}

2. Is there a place where I can see the available options for apiserver.conf?

[Noam Wasersprung]

@volcanihpc look to the default file.

You'll note you misplaced the entry, as it should not be inside fixed_users.