NotADBError: file is not a database returned instead of BusyError
rogerbinns opened this issue · comments
This creates two connections to the same file. The first connection begins a transaction. The second connection then tries to do an insert. If you comment out the lines setting the keys then you get the expected
apsw.BusyError: BusyError: database is locked
But setting the encryption keys gives
apsw.NotADBError: NotADBError: file is not a database
Reproducer
import apsw
con1 = apsw.Connection("testdb")
con1.pragma("key", "hello world")
con2 = apsw.Connection("testdb")
con2.pragma("key", "hello world")
con1.execute("create table test(x,y)")
con1.execute("begin")
con1.execute("insert into test values(123,'abc')")
con2.execute("insert into test values(456, 'def')")
This creates two connections to the same file.
This is not exactly correct. AFAIK your tests clean up before starting. That is, no database file exists. Therefore both connections will try to create a new database file. But you can't create the same database file twice.
Well, it seems to work, because SQLite creates an empty database file on opening connection 1, so that on opening connection 2 it "sees" a directory entry and does not create a file, but simply opens it.
However, with encryption things work a little bit different. The implementation of SQLite3 Multiple Ciphers uses the given passphrase and some random data to generate the key material - for security reasons, so that the same passphrase will not create the same key material. The random data is stored in the header of the database file.
The PRAGMA key statement only sets up the key material, but does not yet write content the database file. This only happens when the first execute method is invoked. However, the PRAGMA key statement looks at the header of the opened file. If the file is a valid database file, the random data will be read. If the file is empty, random data will be generated.
The problem in your sample is that both connections see an empty database file on executing the PRAGMA key statement. And as a consequence new (and different) random data are generated, leading to differing key material for both connections. When connection 2 tries to insert data, the key material doesn't match and hence the error message not a db.
In your sample connection 2 takes for granted that the database file already exists and was initialized (table test created). If connection 2 would be opened in a separate thread or process, the insert statement would fail, if connection 1 has not yet created table test.
That is, your application has to make sure that a non-empty database file actually already exists. Move establishing connection 2 after executing the create table statement, and you should get the expected result.
Either modify the order of statements (as explained above) or use an existing database with an already created table test and clean up by removing all rows from table test before performing the test case.
It isn't an application but rather from the test suite for an issue that existed before 2007 around busy handling, and is the least amount of code to reproduce that and verify fixed behaviour.
Connection 2 does not expect the table to exist - it expects to get an error on that line starting an automatic transaction, and the test fails unless that line gives busyerror. ie it would be a test failure if the SQL suceeeds.
It isn't an application but rather from the test suite for an issue that existed before 2007 around busy handling, and is the least amount of code to reproduce that and verify fixed behaviour.
Whether it is an application or part of a test suite really doesn't matter. The code is written for a plain database, in which case it works as expected.
For an encrypted database it is a bit different: you have to make sure that the database schema is initialized, otherwise the connections produce different key material, and you get the not a db error.
Connection 2 does not expect the table to exist
Implicitly it does, because you issue an insert statement for table test which is not created by connection 2, but by connection 1. What error do you get, when you try to insert something from connection 2 into a not existing table, say test2? Note that you still have to make sure that the database file is properly initialized. In my test I got the error no such table: test2.
- it expects to get an error on that line starting an automatic transaction, and the test fails unless that line gives busyerror. ie it would be a test failure if the SQL suceeeds.
As said in my prior response you should get the expected behaviour, if you make sure that the database file is properly initialized resp that the table test exists in the database file.
Seriously, connection 2 does not expect the table to exist! The insert in connection 2 must fail and it must fail because the database is locked. This is testing locking. The only reason connection 2 does an insert is because it forces SQLite to try to acquire a database lock.
Replace the INSERT in connection 2 with BEGIN EXCLUSIVE for clarity. This is all about locking!
my test I got the error no such table: test2.
If I remove the two key pragmas I get:
$ rm testdb* ; python issue.py
Traceback (most recent call last):
File "/space/mc/issue.py", line 13, in <module>
con2.execute("begin exclusive")
File "src/cursor.c", line 169, in resetcursor
AddTraceBackHere(__FILE__, __LINE__, "resetcursor", "{s: i}", "res", res);
apsw.BusyError: BusyError: database is locked
ie connection 2 correctly sees a locked database.
With the two key pragmas present:
$ rm testdb* ; python issue.py
Traceback (most recent call last):
File "/space/mc/issue.py", line 13, in <module>
con2.execute("begin exclusive")
File "src/cursor.c", line 169, in resetcursor
AddTraceBackHere(__FILE__, __LINE__, "resetcursor", "{s: i}", "res", res);
apsw.NotADBError: NotADBError: file is not a database
Seriously, connection 2 does not expect the table to exist! The insert in connection 2 must fail and it must fail because the database is locked. This is testing locking. The only reason connection 2 does an insert is because it forces SQLite to try to acquire a database lock.
Of course, I know what the intended test is. All I want to make clear is that the database must be initialized, before connection 2 is established. It is enough to execute the create statement before opening connection 2. Then connection 2 sees an existing database file, creates the same key material as connection 1, and the test will show the expected behaviour.
Replace the
INSERTin connection 2 withBEGIN EXCLUSIVEfor clarity. This is all about locking!
sigh I know.
The problem is that you expect that PRAGMA key really creates the database. No, it does not. And again: this is documented behaviour.
Move the create statement of connection 1 in front of opening connection 2, and all will work as expected.
If I remove the two key pragmas I get:
$ rm testdb* ; python issue.py Traceback (most recent call last): File "/space/mc/issue.py", line 13, in <module> con2.execute("begin exclusive") File "src/cursor.c", line 169, in resetcursor AddTraceBackHere(__FILE__, __LINE__, "resetcursor", "{s: i}", "res", res); apsw.BusyError: BusyError: database is lockedie connection 2 correctly sees a locked database.
Yes. For plain unencrypted databases it works that way. However, I'm pretty sure that SQLite reads the database header of the database file, when the processing of the insert statement of connection 2 starts. And in that moment the file is no longer empty, because connection 1 created a table.
With the two key pragmas present:
$ rm testdb* ; python issue.py Traceback (most recent call last): File "/space/mc/issue.py", line 13, in <module> con2.execute("begin exclusive") File "src/cursor.c", line 169, in resetcursor AddTraceBackHere(__FILE__, __LINE__, "resetcursor", "{s: i}", "res", res); apsw.NotADBError: NotADBError: file is not a database
The PRAGMA key statement checks internally whether the database file is completely empty (file length 0) or not. If it is empty, it uses freshly generated random data and the user's passphrase to derive the key material.
Because the PRAGMA key statements of both connections see an empty database file, different key material is derived. Nothing else happens at that moment.
However, when SQLite accesses the database file header on starting to process the insert statement of connection 2, the read will fail, because the key material is different from that of connection 1. Hence, you get the not a db error, because SQLite not even tries to establish a lock.
You may argue that PRAGMA key should access the database file, so that it gets created. But what kind of access should be done? You can execute as many PRAGMA statements as you want - that will not initialize the database file, it would still be empty. You must execute a DDL statement to actually create a database object, otherwise SQLite will not touch the database file.
The expectation is that sqlite3mc does not change locking behaviour!
The problem is that you expect that PRAGMA key really creates the database
Nope. The only expectation is that when a database is accessed it uses the provided encryption key.
BTW I have no problem with you closing this is as wontfix - it is an example of differing sqlite3mc versus sqlite behaviour.
Perhaps a code solution is to not generate keying material until a database access. Your wording implies the keying material is generated at the time of the pragma. If it was delayed until first access then it looks like this test should work.
But also if the keying material is generated at pragma time that implies it could also be tested against the database and report an error on a wrong key which was a different issue.
You can execute as many PRAGMA statements as you want - that will not initialize the database file
You can cause a database initialisation by running PRAGMA user_version=0 which is the default initial value anyway, so it has no semantic side effects. Perhaps it is worth considering reading or writing the user_version in the key pragma as a way of getting the database file in a good state, and also allows key checking?
For example doing con1.pragma("user_version", 0) in the issue code makes busy error happen as expected. And if I set the wrong key for con2 and read user_version, I get an immediate file is not a database error.
Oh you won't be able to do pragmas from a VFS so I added a section to the readme "Best Practice" that says to do so manually.
The apply_key best practise in the README makes locking behave as expected. So the fix for this issue is to use that best practise.
The expectation is that sqlite3mc does not change locking behaviour!
sqlite3mc does not change locking behaviour. It is the handling of the encryption key in conjunction with the sample code in its current form causing a not a db error before SQLite even tries to lock anything.
Whether the database file exists or not at the beginning of the test case, has nothing to do with the goal of testing locking behaviour. So, why do you refuse to change the sample code to guarantee that the database exists?
There are 2 ways to accomplish this:
- Move the
createstatement in front of opening the connection 2 - Add a few statements at the beginning of the test
con0 = apsw.Connection("testdb")
con0.pragma("key", "hello world")
con0.pragma("user_version", con0.pragma("user_version"))
con0.close()
An alternative would be to apply one additional PRAGMA to each connection, namely
conX.pragma("cipher", "aes128cbc")
# or
conX.pragma("cipher", "aes256cbc")
This changes the cipher to be used for encryption. The default cipher chacha20 uses additional random data for the key material, the ciphers aes128cbc and aes256cbc do not do that, and therefore they generate the same key material for a passphrase.
BTW I have no problem with you closing this is as wontfix - it is an example of differing sqlite3mc versus sqlite behaviour.
Yes, it is differing behaviour, but only in the special case that 2 connections start with a non-existing resp empty database and using a cipher which uses additional random data for the key material for increased security.
The problem can easily be avoided, if the database exists and is not empty.
Perhaps a code solution is to not generate keying material until a database access. Your wording implies the keying material is generated at the time of the pragma. If it was delayed until first access then it looks like this test should work.
In theory this seems to be a simple and clean solution, but in practise it is extremely difficult to implement, if at all possible.
The recipe to avoid problems is really simple: Separate the initial creation of the database from the use of the database. That is, let a single connection create, initialize, and close the database. Thereafter as many connections as needed can access the (now existing) database.
But also if the keying material is generated at pragma time that implies it could also be tested against the database and report an error on a wrong key which was a different issue.
Sorry, for the use case of 2 connections accessing an empty database file, there is nothing against which could be tested.
You can cause a database initialisation by running
PRAGMA user_version=0which is the default initial value anyway, so it has no semantic side effects. Perhaps it is worth considering reading or writing theuser_versionin the key pragma as a way of getting the database file in a good state, and also allows key checking?
Yes, in principal, this could be done, but it would differ from documented "official" behaviour.
The
apply_keybest practise in the README makes locking behave as expected. So the fix for this issue is to use that best practise.
apply_key forces SQLite to write something to disk. There are several ways to do so. In principal, the problem exists only, if a new database needs to be initialized. This task should usually be the responsibility of a single connection. Later on one can have many concurrent connections, of course.
sqlite3mc does not change locking behaviour
The goal here is for the developer to be successful. If they had code that worked perfectly without sqlite3mc, and the only change they made was to add the pragma key, then that code no longer works perfectly.
The C interface doc does mention the whole empty file caveats, but the pragma doc does not.
Until today the apsw-sqlite3mc doc only had the pragma key. That isn't sufficient around new files as this issue shows you can get different behaviour.
A developer doing the apply_key will ensure previously working code will continue to work with encryption, and as a bonus also ensures the key is correct. I've also updated it to make sure the database is not in a transaction when the pragma is run, as per the doc.
Future runs of the apsw test suite will use apply_key
sqlite3mc does not change locking behaviour
The goal here is for the developer to be successful. If they had code that worked perfectly without sqlite3mc, and the only change they made was to add the pragma key, then that code no longer works perfectly.
Sorry, I disagree. Under the circumstances the behaviour is absolutely correct (and simply reveals a flaw in the pre-conditions of the test). Adding apply_key just enforces the pre-condition that the database file exists and is initialized with a proper database header. Fulfilling the pre-condition can be achieved in various ways - using apply_key is certainly one of them.
One could call SQLite itself flawed, because opening a database connection for a non-existing file and then closing the connection without performing any other action on the database leaves an empty file behind. This file can't be identified as a SQLite database file, because it has no SQLite header. Actually, it could be anything. Opening the file again, but this time executing some DDL SQL statements would let SQLite happily create a database file from the existing empty file, although this may not be what a user expects or wants. Creating a new database file, if there wasn't a file with the given name, is ok, but changing an existing empty file into a database file may not be ok.
Unfortunately, the SQLite documentation of sqlite3_open_v2 does not mention anything about how it handles empty, zero-length files.
The C interface doc does mention the whole empty file caveats, but the pragma doc does not.
Until today the apsw-sqlite3mc doc only had the pragma key. That isn't sufficient around new files as this issue shows you can get different behaviour.
Sure. Documentation can always be improved. I will copy the note about new, empty files over to the pragma page.
A developer doing the
apply_keywill ensure previously working code will continue to work with encryption, and as a bonus also ensures the key is correct. I've also updated it to make sure the database is not in a transaction when the pragma is run, as per the doc.
AFAICR there is no public function to check whether the database is currently within a transaction or not. If there is, the code could be changed to emit an error message.
The goal here is for the developer to be successful
Sorry, I disagree.
We have a very different view of the world.
Developers using Python, SQLite, APSW etc are not subject matter experts. They are trying to get their work done. They do not read all the documentation, all the source, have extensive test suites, or devote weeks learning 100% of the material. What they want to do is read the least amount of material, copy and paste in whatever the getting started doc says, and then work on the other things that are the value to them. They don't want to be later surprised, have to dedicate time to debugging, deal with race conditions etc.
You keep explaining why sqlite3mc works the way it does. I have no disagreement or comment on that. Quite simply it does not matter.
What does matter is - copy and paste this code to make your database encrypted and all will be good.
AFAICR there is no public function to check whether the database is currently within a transaction or not
- https://rogerbinns.github.io/apsw/connection.html#apsw.Connection.in_transaction
- https://sqlite.org/c3ref/get_autocommit.html
If there is, the code could be changed to emit an error message.
Already did.
The goal here is for the developer to be successful
Sorry, I disagree.We have a very different view of the world.
That is a bit of a misunderstanding. I didn't mean to disagree with the goal for a developer being succesful. Of course, I'd like to make developers using my components happy and successful.
I disagree with the statement that the code no longer works perfectly. It does what I expect that will happen. There is nothing wrong. Due to the changed circumstances - encryption was added - a problem occurs before the locking test is performed.
Unfortunately, we don't have DWIM tools at hand (DWIM = Do What I Mean). And I fear even ChatGPT will not change that.
Developers using Python, SQLite, APSW etc are not subject matter experts. They are trying to get their work done. They do not read all the documentation,
Who does?
all the source, have extensive test suites, or devote weeks learning 100% of the material. What they want to do is read the least amount of material, copy and paste in whatever the getting started doc says, and then work on the other things that are the value to them. They don't want to be later surprised, have to dedicate time to debugging, deal with race conditions etc.
Well, the test code was changed - by adding the pragma. So, different things can happen - such is life. The behaviour can be explained and can be fixed easily.
AFAICR there is no public function to check whether the database is currently within a transaction or not
Ouch, I overlooked this function.
Time to address this issue:
- There are no code changes that should be made to sqlite3mc
- It is completely reasonable to expect that setting a key is the only thing you do and get no differences in behaviour. sqlite3mc doesn't actually document how to use it, but SEE and sqleet both just show using a pragma set the key and all is good
- One example where you could get a difference in behaviour is this issue, which would happen if worker processes get started from a service and whoever got there first initialised the database
The issue has been addressed in apsw-sqlite3mc by having an apply_encryption function that expresses in code all the notes, gotchas, foot guns etc that I could find, and ensures developer success.
Specifically it does the following:
- Checks that configurations are applied by checking returns from pragmas which catches errors in the pragma name or values
- Deals with the pragma names being case insensitive
- Refuses to run when in a transaction
- Ensures pragmas are run in the correct order: cipher, legacy, *legacy*, the rest except keys, keys
- Does a read from the database which catches incorrect keys or cipher configuration
- Does a nochange write to the database which ensures the header is populated on empty database files
Correction to previous comment: sqlite3mc does document usage on the overview page. It also basically says to set the key and all is good.
- There are no code changes that should be made to sqlite3mc
Agreed.
- It is completely reasonable to expect that setting a key is the only thing you do and get no differences in behaviour. sqlite3mc doesn't actually document how to use it, but SEE and sqleet both just show using a pragma set the key and all is good
AFAIK none of the popular encryption extensions handles it differently.
- One example where you could get a difference in behaviour is this issue, which would happen if worker processes get started from a service and whoever got there first initialised the database
IMHO it would be good practice to initialize the database in exclusive access mode. Thereafter concurrent access should not impose problems.
The issue has been addressed in apsw-sqlite3mc by having an
apply_encryptionfunction that expresses in code all the notes, gotchas, foot guns etc that I could find, and ensures developer success.Specifically it does the following:
- Checks that configurations are applied by checking returns from pragmas which catches errors in the pragma name or values
SQLite3MC does not check for errors in pragma names, only values for known values will be checked. If a value is out of range for the requested configuration parameter, the current value of the parameter will be returned.
Pragmas with unrecognized names will be forwarded to SQLite. If SQLite doesn't know the pragma name, it silently ignores it. Nothing will be returned.
- Deals with the pragma names being case insensitive
SQL is case insensitive, not only pragma names.
- Refuses to run when in a transaction
Not sure whether this is the right thing to do. SQLite itself allows to execute pragmas within a transaction and doesn't revert their effect if the transaction was rolled back, unless otherwise documented.
- Ensures pragmas are run in the correct order: cipher, legacy, legacy, the rest except keys, keys
How do you do that?
- Does a read from the database which catches incorrect keys or cipher configuration
- Does a nochange write to the database which ensures the header is populated on empty database files
Ok. The wrapper can do whatever seems to be appropriate.
AFAIK none of the popular encryption extensions handles it differently
That isn't relevant. What is relevant is what a developer would expect after reading the overview of sqlite3mc, or sqleet, or SEE. But that is also why apsw-sqlite3mc makes it very clear what the developer should do, and is as foolproof as I can make it.
IMHO it would be good practice to initialize the database in exclusive access mode. Thereafter concurrent access should not impose problems.
The apsw-sqlite3mc method works just fine and means a developer doesn't have to come up with coordination schemes. If their code worked without encryption, it will work with encryption.
SQLite3MC does not check for errors in pragma names
I'm not expecting it to. The apsw-sqlite3mc method detects if the developer wrote cypher instead of cipher or legecy instead of legacy or any other number of possible mistakes.
SQL is case insensitive, not only pragma names
I meant that the apsw-sqlite3mc correctly handles someone saying CIPHER or Cipher or anything else. ie it catches and prevents mistakes. This is especially relevant for key because it will refused having both KEY and reKey or any other combinations. ie there are no names whatever the case where it won't correctly check the pragma was understood, and will insist on exactly one key-like name.
Not sure whether [Refuses to run when in a transaction] is the right thing to do
The doc:
It is strongly recommended to avoid executing PRAGMA statements for the configuration of the encryption extension within a transaction. The effect of these PRAGMA statements can’t be rolled back. In some cases execution will even fail ....
ie refusing is the right thing for apsw-sqlite3mc. Note that developers who know better can ignore the change_encryption method provided and go their own way. But those looking at the one page of documentation and using change_encryption are far less like to have mistakes. If they do make any mistakes an exception is raised.
How do you do [run in the correct order]?
By sorting the pragmas first. See the code.
The wrapper can do whatever seems to be appropriate
It isn't even the wrapper - it is the readme that shows up in the github repo and will show up in the pypi listing.
It captures in code and comments the right way to do things, without having to comb all the rest of the doc.
The PRAGMA key statement only sets up the key material, but does not yet write content the database file. This only happens when the first execute method is invoked. However, the PRAGMA key statement looks at the header of the opened file. If the file is a valid database file, the random data will be read. If the file is empty, random data will be generated.
@utelle It would be very be beneficial to defer reading the header of the file until the key is first used. This would allow "connection 2" to create the database after "connection 1" as called PRAGMA key and still have everything work properly. It is definitely true that doing it this way is significantly more complicated, however, it will save a significant amount of headache in the long term.
In the specific test case that @rogerbinns has raised it may be technically possible for an application to re-order connection operations to get the right behavior, or add statements like PRAGMA user_version to force database creation. However, there are other more complicated cases where that will be very difficult, if not impossible, for a developer. One very common example is when an application is using a data access API that features connection pooling. It is a common practice for a pool to spin up multiple connections at one time, and then hand them out as needed. In this cases every underlying connection would generate different random data, and it would result in every connection generating a different key.
Connection pooling is a common feature of many APIs that sqlite3mc already is, or eventually will, target including sqlite-net. Microsoft.Data.Sqlite.Core, Microsoft.EntityFrameworkCore.Sqlite.Core, SQLite Android Bindings, etc. If sqlite3mc doesn't defer the header check and salt generation until first use, it will eventually lead to widespread confusion, difficult to fix behavior, and undesirable workarounds for application developers.
It would be very be beneficial to defer reading the header of the file until the key is first used.
At first glance it is indeed tempting to do so. However, I decided against this approach for 2 reasons:
- If key derivation is deferred until first use, you have to keep the plain passphrase in memory - at least until it was used for key derivation. From a security standpoint of view this is not good.
- Key derivation may take quite some time - depending on the actual cipher configuration. Opening the database connection is often done on initializing an application, where a certain delay is usually acceptable. When accessing the database for example in a real-time application such a delay may be highly undesirable.
This would allow "connection 2" to create the database after "connection 1" as called
PRAGMA keyand still have everything work properly. It is definitely true that doing it this way is significantly more complicated, however, it will save a significant amount of headache in the long term.
Of course it is somewhat more complicated, but in principal managable.
I know you are involved in the development of SQLCipher. I just took a quick look at its source code ... and it seems SQLCipher tries to do it in the way you described. However, I haven't actually tested it. Maybe you did.
In the specific test case that @rogerbinns has raised it may be technically possible for an application to re-order connection operations to get the right behavior, or add statements like
PRAGMA user_versionto force database creation. However, there are other more complicated cases where that will be very difficult, if not impossible, for a developer. One very common example is when an application is using a data access API that features connection pooling. It is a common practice for a pool to spin up multiple connections at one time, and then hand them out as needed. In this cases every underlying connection would generate different random data, and it would result in every connection generating a different key.
Problems can occur only if a properly initialized database doesn't exist yet, and 2 or more connections try to create it. IMHO it is good practice to initialize new databases in exclusive mode, before accessing them with 2 or more connections in parallel.
Connection pooling is a common feature of many APIs that sqlite3mc already is, or eventually will, target including
sqlite-net.Microsoft.Data.Sqlite.Core,Microsoft.EntityFrameworkCore.Sqlite.Core,SQLite Android Bindings, etc. If sqlite3mc doesn't defer the header check and salt generation until first use, it will eventually lead to widespread confusion, difficult to fix behavior, and undesirable workarounds for application developers.
I doubt that creating a new database from more than 1 database connection is really a common use case. But even if it is, fixing the problem requires just to reopen the database. That's it.
I'm not convinced that this issue justifies to partially sacrifice security, but I'm open to discuss this further.
You can cause a database initialisation by running
PRAGMA user_version=0which is the default initial value anyway, so it has no semantic side effects.
Another option to force SQLite to write page 1 (with the database header) to disk is to run
BEGIN IMMEDIATE; COMMIT;Changes to the current behaviour in SQLite3 Multiple Ciphers are currently not planned. Closing ...