Question about process safety
tuhe opened this issue · comments
Hi!
I need a simple key/value store to track the output of unittests. In the application, multiple tests will run at once and write to the same key/value store. The tests will be run from different python processes (i.e. python -m unittest ...), i.e. two different processes may try to set/update the same key at the same time. In some of the processes, multiple threads will read from the database as well.
Performance is not an issue, but I am worried about corrupting the database, and I got burned once by a 'process safe' key/value database that ended up with an unreadable database. My question is how to approach that? Should I wrap every read/write operation with a Lock, or is this unnecessary, as long as the order of updates is irrelevant to me?
Most every API method is atomic so there's no need for locks. The tests/benchmarks specifically read+write the same set of keys from multiple processes without corruption.
There are a set of "caveats" documented at https://grantjenks.com/docs/diskcache/tutorial.html#caveats but none of them regard process safety.