Cloudbeaver fails to load management database after restart
LifetimeMistake opened this issue · comments
Describe the bug
I'm attempting to run Cloudbeaver with Docker Compose. Upon starting the container it initializes successfully and I can go through the setup phase in the web UI. After entering a license key everything seems to be working (I can add users, connections, etc.). However, after restarting the container, the server fails to reload the database, leading to a crash.
Curiously, this occurs regardless of the driver being used to store the mangement DB (I tried H2, mysql and postgres, giving me the same result) or whether I complete the setup phase or not.
To Reproduce
Steps to reproduce the behavior:
- Start a fresh instance of Cloudbeaver EE in Docker.
- Wait for the database migration to complete and the web server to start
- Restart the container
- Observe the crash with an error related to database schema migration
Screenshots
N/A
Additional context
This error occurs regardless of the driver being used to store the internal database (
I tried running Cloudbeaver from the 23, 24, ea, and latest branches all resulting in the same crash.
Another weird quirk occurs when I don't bind mount /opt/cloudbeaver/workspace to a persistent location and configure Cloudbeaver to store the management and metadata databases in an external provider (such as MySQL or Postgres). Performing the above steps results in a successful start and no database migration (sounds fair?). However, all the data is gone and I'm forced to go through the setup again. Connecting to the MySQL/Postgres db I can see that there is definitely some data being stored in both the management and metadata databases so I'm unsure why clearing the workspace (which only seems to contain the H2 embedded databases?) would cause this.
Here's the error I get when restarting a fresh Cloudbeaver instance:
20-03-2024 22:35:11.673 [main] DEBUG com.dbeaver.ee.qm.db.impl.QMDBModel - Initialize QMDB...
20-03-2024 22:35:11.700 [main] DEBUG com.dbeaver.ee.qm.db.impl.QMDBModel - Initiate connection pool with management database (PostgreSQL; jdbc:postgresql://<database host redacted>/cloudbeaver-metadata?ssl-mode=verify-full&sslrootcert=/run/secrets/ca_cert)
20-03-2024 22:35:11.792 [main] DEBUG com.dbeaver.ee.qm.db.impl.QMDBModel - QMDB connected
20-03-2024 22:35:12.198 [main] DEBUG com.dbeaver.ee.qm.db.impl.QMDBModel - Check QMDB deployment
20-03-2024 22:35:12.255 [main] DEBUG com.dbeaver.ee.qm.db.impl.QMDBModel - QMDB server started (PostgreSQL 14.11 (Debian 14.11-1.pgdg120+2))
20-03-2024 22:35:12.263 [main] DEBUG c.d.m.l.e.LicenseServiceEmbedded - Initialize license manager for cloudbeaver-ee
20-03-2024 22:35:12.264 [DBeaver: CloudBeaver license verifier] INFO i.c.server.ee.LicenseCheckJob - Check license status
20-03-2024 22:35:12.267 [DBeaver: CloudBeaver license verifier] ERROR c.d.m.l.e.LicenseServiceEmbedded - No license found for CloudBeaver EE Server
20-03-2024 22:35:12.273 [main] DEBUG i.c.s.auth.ReverseProxyConfigurator - Reverse proxy provider disabled, migration not needed
20-03-2024 22:35:12.274 [main] DEBUG i.c.s.a.WebServiceBindingAwsConfigurator - IAM provider disabled, migration not needed
20-03-2024 22:35:12.294 [main] DEBUG i.c.service.security.db.CBDatabase - Initiate management database
20-03-2024 22:35:12.300 [main] DEBUG i.c.service.security.db.CBDatabase - Initiate connection pool with management database (PostgreSQL; /opt/cloudbeaver/workspace)
20-03-2024 22:35:12.302 [main] ERROR io.cloudbeaver.server.CBApplication - Error initializing database
org.jkiss.dbeaver.DBException: Error updating management database schema
at io.cloudbeaver.service.security.db.CBDatabase.initialize(CBDatabase.java:226)
at io.cloudbeaver.service.security.EmbeddedSecurityControllerFactory.createSecurityService(EmbeddedSecurityControllerFactory.java:65)
at io.cloudbeaver.server.ee.CBApplicationEE.createGlobalSecurityController(CBApplicationEE.java:63)
at io.cloudbeaver.server.CBApplication.initializeSecurityController(CBApplication.java:529)
at io.cloudbeaver.server.CBApplication.startServer(CBApplication.java:350)
at io.cloudbeaver.model.app.BaseWebApplication.start(BaseWebApplication.java:240)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:208)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:651)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:588)
at org.eclipse.equinox.launcher.Main.run(Main.java:1459)
at org.eclipse.equinox.launcher.Main.main(Main.java:1432)
Caused by: java.lang.IllegalStateException: Connection factory returned null from createConnection
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at io.cloudbeaver.service.security.db.CBDatabase.initialize(CBDatabase.java:190)
... 18 common frames omitted
A weird thing that I noticed is that QMDB seems to connect successfully, printing the database URI to the logs:
Initiate connection pool with management database (PostgreSQL; jdbc:postgresql://<database host redacted>/cloudbeaver-metadata?ssl-mode=verify-full&sslrootcert=/run/secrets/ca_cert)
Meanwhile the management database prints /opt/cloudbeaver/workspace instead, while still listing the correct driver:
Initiate connection pool with management database (PostgreSQL; /opt/cloudbeaver/workspace)
I'd be happy to provide any relevant snippets of the cloudbeaver configuration file, my compose deployment or the environment variables I'm setting. Hope to hear from you guys soon
Hello @LifetimeMistake
Did you change the database URL from https://github.com/dbeaver/cloudbeaver/wiki/Server-configuration#database-configuration (server.database.url) or just QM(qm.url)?
Could you please provide the log files? And your config? (workspace/.data/.cloudbeaver.runtime.conf)
If you have any sensitive information inside, you can always write to our Tech Support (it is available for Enterprise versions): https://dbeaver.com/tickets
Hey there!
I set my database URLs using an environment file only accessible by the root user. I have modified the default config like so:
database: {
driver: "${CLOUDBEAVER_MAN_DB_DRIVER:h2_embedded_v2}",
url: "${CLOUDBEAVER_MAN_DB_URL:jdbc:h2:${workspace}/.data/cb.h2v2.dat}",
user: "${CLOUDBEAVER_MAN_DB_USER:''}",
password: "${CLOUDBEAVER_MAN_DB_PASSWORD:''}",
createDatabase: false,
initialDataConfiguration: "conf/initial-data.conf",
pool: {
minIdleConnections: 4,
maxIdleConnections: 10,
maxConnections: 100,
validationQuery: "SELECT 1"
}
}
...
qm: {
driver: "${CLOUDBEAVER_QM_DB_DRIVER:h2_embedded_v2}",
url: "${CLOUDBEAVER_QM_DB_URL:jdbc:h2:${workspace}/.metadata/qmdb/qmdb}",
user: "${CLOUDBEAVER_QM_DB_USER:''}",
password: "${CLOUDBEAVER_QM_DB_PASSWORD:''}",
schema: "${CLOUDBEAVER_QM_DB_SCHEMA:''}",
pool: {
minIdleConnections: "${CLOUDBEAVER_QM_DB_MIN_IDLE_CONNECTIONS:4}",
maxIdleConnections: "${CLOUDBEAVER_QM_DB_MAX_IDLE_CONNECTIONS:10}",
maxConnections: "${CLOUDBEAVER_QM_DB_MAX_CONNECTIONS:100}",
validationQuery: "${CLOUDBEAVER_QM_DB_VALIDATION_QUERY:SELECT 1}"
}
In my env file:
CLOUDBEAVER_QM_DB_DRIVER: postgresql
CLOUDBEAVER_QM_DB_URL: jdbc:postgresql://.../cloudbeaver-metadata?ssl-mode=verify-full&sslrootcert=/run/secrets/ca_cert
CLOUDBEAVER_QM_DB_USER: ...
CLOUDBEAVER_QM_DB_PASSWORD: ...
CLOUDBEAVER_MAN_DB_DRIVER: postgresql
CLOUDBEAVER_MAN_DB_URL: jdbc:postgresql://.../cloudbeaver-management?ssl-mode=verify-full&sslrootcert=/run/secrets/ca_cert
CLOUDBEAVER_MAN_DB_USER: ...
CLOUDBEAVER_MAN_DB_PASSWORD: ...
CB_SERVER_NAME: ...
CB_SERVER_URL: ...
CB_ADMIN_NAME: admin
CB_ADMIN_PASSWORD: admin
I'm planning to use postgres in production but as you can see the server should fall back to the H2 driver if the vars are not set. Please note that the server fails to start regardless of the driver I choose. If I use H2 as the backend, the server fails with the same message, but this time listing the embedded H2 driver and the H2 db path instead of the postgres connection string.
I also use this initial-data.conf file to allow me to create the "admin" user.
{
teams: [
{
subjectId: "admins",
teamName: "Admin",
description: "Administrative access. Has all permissions.",
permissions: [ "admin" ]
},
{
subjectId: "user",
teamName: "User",
description: "All users, including anonymous.",
permissions: [ ]
}
]
}
Not sure if this change is related in any way, but thought it might help.
The other parts of my runtime config don't differ from the defaults shipped with cloudbeaver in any way. The log snippet I provided seems to be the only interesting part of the log but I can provide the full file if you need it.
@LifetimeMistake Can you check what value you have in .cloudbeaver.runtime.conf? This config has a higher priority, and if you started the server before changing the root configuration, there will be no env variable there
and you need to change driver (CLOUDBEAVER_QM_DB_DRIVER/CLOUDBEAVER_MAN_DB_DRIVER) env values to postgres-jdbc
@LifetimeMistake I found the problem, since your default configuration uses 2 variables at the same time, only the last one is written to the runtime config. To solve this problem, you can change the template from "${CLOUDBEAVER_MAN_DB_URL:jdbc:h2:${workspace}/.data/cb.h2v2.dat}", to "${CLOUDBEAVER_MAN_DB_URL:jdbc:h2:/opt/cloudbeaver/workpspace/.data/cb.h2v2.dat}". After changing the template, it's better to remove .cloudbeaver.runtime.conf file and complete the configuration again.
Hi @alexander-skoblikov, you're right that using 2 variables at the same time does not work. I don't know how I missed that. Combining that with changing the driver to postgres-jdbc and setting the correct env vars I am able to persist the database in postgres without the server breaking after reboots.
One thing is left unclear, however. When the env vars are left unset the server should fall back to H2. Even after hardcoding the workspace path. the management database still fails to connect after a reboot (qm is just fine though?), now with a different exception - org.h2.jdbc.JdbcSQLInvalidAuthorizationSpecException: Wrong user name or password [28000-214].
I assume this is because my defaults for the user and password are empty strings.
"user": "${CLOUDBEAVER_MAN_DB_USER:}",
"password": "${CLOUDBEAVER_MAN_DB_PASSWORD:}",
I got a working setup with postgres so this question is really just me trying to satisfy my own curiosity, but would it be possible to unset these variables somehow? Anything other than removing these fields from the config completely doesn't seem to fix the issue.
Closing as the original problem was solved. For now dropping the embedded DB credential issues, will ask again when I actually need that to work. Thanks for all your support.
Thanks for the update @LifetimeMistake