Database backed games cache to improve performance
kimzeevaarders opened this issue · comments
Is your feature request related to a problem? Please describe.
Been using Hakchi for 2 years now. Works perfectly. Over the years my collection has grown humongously and this has a heavy impact on performance. Not only does starting hakchi take over 5 minutes (loading games...) but also every action you take like scraping games or setting cores always triggers the "saving selected games/saving selected configuration" which takes up to a minute to complete. I analyzed this a little bit and appearantly hakchi saves the selected games in the config.json file. And i've also noticed it does this per console (if NOT using seperate game storage). In my opinion the performance hit occurs when persisting to this file but Im not sure.
Describe the solution you'd like
In my opinion, persisting the selected games/config to the json file using plain .NET FileWriters could be very inefficient. Wouldn't you guys please consider migrating the persistence of this data to some high performance (file) cache. In Java, we use infinispan or reddis, which would take (i assume) only a fraction of the performance thah hakchi is using. Cahces like these should also be readily available for .NET i would think. Also, would it be possible implement some kind of "dirty" config flag so that it knows (more intelligently) when to trigger the save selected games /save selected configuration action, instead of just always triggering it (may be this is already implemented, am not sure)?
Describe alternatives you've considered
I noticed that not using seperate games storage has less impact on performance, since only 1 list of selected games is persisted.
Additional context
I love your work and I admire that you have time to work on this next to your regular job. All things stated here are my personal opinion and only based on assumptions (so they might be false. In that case i'm sorry). As a user, this performance problem makes working with hakchi very unpleasant and time consuming. If there would be some fix for this problem, I think it would make the application a lot more pleasant to work with :)
Regards from the Netherlands,
Kim
There's absolutely a lot of room for improvement, and using individual files for metadata is because that's how the nes/snes classic handles the games on the system.
Implementing a cache system backed by something like SQLite wouldn't be too difficult I don't think, but the game directories would still have to be scanned in some way to ensure the cached data doesn't get out of sync. Perhaps storing the data along with a timestamp that the .desktop file was last modified?
You'd still be going out to the filesystem though, but I don't think there'd be nearly as much of a hit compared to opening the each .desktop file and then parsing them