Giters

unmade / shelf-showcase

An example of Shelf Cloud ready-to-run docker-compose

Home Page:https://getshelf.cloud

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Error while starting EdgeDB

kekersdev opened this issue · comments

commented

Tried setting up Shelf using shelf-showcase docker-compose.yml but encountered some issues. If i understand correctly everything boils down to postgres crashing during server bootstrap:

INFO 40 2023-12-15T21:15:13.209 edb.pgcluster: Initializing database cluster in /var/lib/edgedb/data
WARNING 40 2023-12-15T21:15:27.672 edb.server: server bootstrap did not complete successfully, removing the data directory

Full edgedb container log:

shelf-showcase-edgedb-1        | Bootstrapping EdgeDB instance on the local volume...
shelf-showcase-edgedb-1        | INFO 40 2023-12-15T21:15:09.477 edb.server: starting EdgeDB server 3.3+00041f3, built on 2023-09-06T20:36Z from revision daca65ac2 (2023-09-06T01:08Z) for x86_64-unknown-linux-gnu
shelf-showcase-edgedb-1        | INFO 40 2023-12-15T21:15:09.477 edb.server: instance name: '_unknown'
shelf-showcase-edgedb-1        | INFO 40 2023-12-15T21:15:13.083 edb.server: Using 2577 max backend connections based on total memory.
shelf-showcase-edgedb-1        | INFO 40 2023-12-15T21:15:13.209 edb.pgcluster: Initializing database cluster in /var/lib/edgedb/data
shelf-showcase-edgedb-1        | WARNING 40 2023-12-15T21:15:27.672 edb.server: server bootstrap did not complete successfully, removing the data directory
shelf-showcase-edgedb-1        | ---- Exception occurred: [Errno 16] Device or resource busy: '/var/lib/edgedb/data' ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | 3. edb.server.pgcluster.PostgresPidFileNotReadyError: 
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | ---- Traceback ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 665, in _test_connection
shelf-showcase-edgedb-1        |         > conn_addr = self._get_connection_addr()
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 602, in _get_connection_addr
shelf-showcase-edgedb-1        |         > self._connection_addr = self._connection_addr_from_pidfile()
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 640, in _connection_addr_from_pidfile
shelf-showcase-edgedb-1        |         > raise PostgresPidFileNotReadyError
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | edb.server.pgcluster.PostgresPidFileNotReadyError: 
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | ---- During handling of the above exception, another exception occurred ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | 2. edb.server.pgcluster.ClusterError: The backend exited with -9
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | ---- Traceback ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/main.py, line 550, in run_server
shelf-showcase-edgedb-1        |         > await cluster.start()
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 494, in start
shelf-showcase-edgedb-1        |         > await self._test_connection(timeout=wait)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 678, in _test_connection
shelf-showcase-edgedb-1        |         > raise ClusterError(f"The backend exited with {code}")
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | edb.server.pgcluster.ClusterError: The backend exited with -9
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | ---- During handling of the above exception, another exception occurred ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | 1. OSError: [Errno 16] Device or resource busy: '/var/lib/edgedb/data'
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | ---- Traceback ----
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/bin/edgedb-server.py, line 7, in <module>
shelf-showcase-edgedb-1        |         > sys.exit(main())
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/core.py, line 829, in __call__
shelf-showcase-edgedb-1        |         > return self.main(*args, **kwargs)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/core.py, line 782, in main
shelf-showcase-edgedb-1        |         > rv = self.invoke(ctx)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/core.py, line 1236, in invoke
shelf-showcase-edgedb-1        |         > return Command.invoke(self, ctx)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/core.py, line 1066, in invoke
shelf-showcase-edgedb-1        |         > return ctx.invoke(self.callback, **ctx.params)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/core.py, line 610, in invoke
shelf-showcase-edgedb-1        |         > return callback(*args, **kwargs)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/click/decorators.py, line 21, in new_func
shelf-showcase-edgedb-1        |         > return f(get_current_context(), *args, **kwargs)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/main.py, line 728, in main
shelf-showcase-edgedb-1        |         > server_main(**kwargs)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/main.py, line 708, in server_main
shelf-showcase-edgedb-1        |         > asyncio.run(run_server(server_args))
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/asyncio/runners.py, line 190, in run
shelf-showcase-edgedb-1        |         > return runner.run(main)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/asyncio/runners.py, line 118, in run
shelf-showcase-edgedb-1        |         > return self._loop.run_until_complete(task)
shelf-showcase-edgedb-1        |     uvloop/loop.pyx, line 1517, in uvloop.loop.Loop.run_until_complete
shelf-showcase-edgedb-1        |         
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/main.py, line 653, in run_server
shelf-showcase-edgedb-1        |         > cluster.destroy()
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/site-packages/edb/server/pgcluster.py, line 529, in destroy
shelf-showcase-edgedb-1        |         > shutil.rmtree(self._data_dir)
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/shutil.py, line 738, in rmtree
shelf-showcase-edgedb-1        |         > onerror(os.rmdir, path, sys.exc_info())
shelf-showcase-edgedb-1        |     /usr/lib/x86_64-linux-gnu/edgedb-server-3/lib/python3.11/shutil.py, line 736, in rmtree
shelf-showcase-edgedb-1        |         > os.rmdir(path, dir_fd=dir_fd)
shelf-showcase-edgedb-1        | 
shelf-showcase-edgedb-1        | OSError: [Errno 16] Device or resource busy: '/var/lib/edgedb/data'
shelf-showcase-edgedb-1        | ERROR: Could not complete instance bootstrap
shelf-showcase-edgedb-1 exited with code 1

Could not find any info on such behavior. Is there any way of finding out what could be the reason behind this problem?

commented

Using Debian 12 btw

commented

Looks like I figured out the [Errno 16] Device or resource busy: '/var/lib/edgedb/data'
Now its just ---- Exception occurred: The backend exited with -9 ----

commented

Increasing available memory and swap to 4GB solved the -9 issue.
However now it freezes on loading TLS certificates consuming as much CPU power as possible indefinetely

commented

Tried EDGEDB_SERVER_SECURITY=insecure_dev_mode. Still no luck: generates certificate successfully and hangs on loading TLS certificates again

commented

Hey @kekersdev!

Apologies for replying so slow.

The issue is indeed seems to be somewhere in the EdgeDB. Can you please provide more details, specifically the command you're trying to run and what platform you're running on (x84_64, arm64, rasberry pi, etc...)?

Also, I'm planning to upgrade to EdgeDB v4.4 so maybe it will solve the issue

commented

I've updated EdgeDB to v4.4. Please checkout the latest master. Note that you might want to delete any volumes and start from scratch

commented

Thanks for the reply!

I'm running on x84_64.
Tried a fresh installation:

  • On the first try all containers shut down because apparently it took edgedb container to much time to start
  • On the second try it showed some huge postgres error, consumed around 3GB of memory, hanged docker and made the entire system highly unresponsive (it even caused the whole pve node to reboot when I tried to reboot the shelf vm)
  • Third try - same behaviour as the second one: consumed ~3 gigs of memory and hanged the system (thankfully no full node reboot happened this time)
  • Postgress error mentioned something about concurrent db access so I decided to try one more time from scratch. Sadly nothing changed: still getting could not serialize access due to concurrent update error.

Full docker log from the last attempt: docker.log

So the initial problem is gone but there is another one...