SQLite file growing

Question

SQLite file growing

SystemKeeper opened this issue 9 months ago · comments

I am wondering if we would need some method to shrink the SQLite database used by garm? (I think that's VACUUM on Sqlite?).

When I checked our database file on 5. October it was 937M in size, today it is at 1.15GB.

Gabriel · Answer 1 · Tue Oct 31 2023 16:01:45 GMT+0800 (China Standard Time)

I think you're seeing this due to the fact that we had a few relations missing to instance status messages and jobs in version 0.1.3. This (I hope) is fixed in the main branch, but I still want to track down a few issues before I make a new release.

Quick way to test is to connect to the sqlite database:

sqlite3 /path/to/file.db

and just count the status messages:

SELECT COUNT(*) FROM instance_status_updates;

Chances are you have a lot of them. Feel free to remove them all, as they are only informational. Then you can VACUUM. In the next few weeks I plan to cut a new release which includes a ton of updates from 0.1.3.

Gabriel · Answer 2 · Tue Oct 31 2023 16:15:12 GMT+0800 (China Standard Time)

The main branch already has a relation between instances and their status updates, so when instances get removed, so will the status updates, maintaining the database at a decent size.

Marcel Müller · Answer 3 · Thu Nov 02 2023 01:40:10 GMT+0800 (China Standard Time)

sqlite> SELECT COUNT(*) FROM instance_status_updates;
2545815

but I noticed that we missed quite some updates on that machine, we're still at v0.1.0-38-g67b8714 of garm. I'll try to schedule some time to upgrade to the latest version!

Gabriel · Answer 4 · Thu Nov 02 2023 04:34:49 GMT+0800 (China Standard Time)

Make sure to properly back up the database before you do. While I try to be careful with database upgrades, there is still a chance things may go wrong. A backup is always a good idea.

Marcel Müller · Answer 5 · Sun Nov 26 2023 22:33:37 GMT+0800 (China Standard Time)

Updated to 0.1.3, removed all entries from instance_status_updates and executed VACUUM. Now the DB is at 268K. I'll monitor it the next couple of days and see how it works out :)

Gabriel · Answer 6 · Mon Nov 27 2023 02:39:43 GMT+0800 (China Standard Time)

Version 0.1.3 might not have the proper DB relations set between the instances and status messages. I'll try to allocate some time to finish up a couple of things in current main and make another release. The next release will be a big one, with tons of updates, fixes and optimizations.

Gabriel · Answer 7 · Tue Dec 19 2023 03:39:10 GMT+0800 (China Standard Time)

The latest RC1 release of GARM now has a proper relation set between the instance and status messages. When instances get removed, so will their status messages.

Take great care when updating (if you decide to do it). Make sure you back everything up. You may still need to remove old status messages yourself and execute a VACUUM.

Marcel Müller · Answer 8 · Mon Jan 15 2024 02:47:17 GMT+0800 (China Standard Time)

Just a quick update, we are still on 0.1.3 at the moment and the DB has a size of 1.1 MB today, which is a really nice improvement to the size increase we were seeing before 👍

Gabriel · Answer 9 · Mon Jan 15 2024 05:46:41 GMT+0800 (China Standard Time)

That is awesome! Thanks for the update!