cursor.lastrowid sometimes returning zero instead of the correct value

[Paul Moore]

Description

I'm seeing somewhat unreliable behaviour with the lastrowid property of a cursor. I'm doing INSERT ... ON CONFLICT DO UPDATE statements on multiple rows:

    for rel in releases:
        for file in releases[rel]:
            async with db.execute(PROJECT_FILES_SQL, project_files_args(file)) as cursor:
                file_id = cursor.lastrowid
                print(file_id, name, file.get("filename"))

What I'm seeing is many rows with a file_id of zero, but when I query the database directly, the file_id is not zero, and has a perfectly valid value.

Unfortunately, I cannot reproduce this in a smaller test case. I suspect that this could be a race condition, with the lastrowid value being "lost" because another statement is executed on the database thread before lastrowid is called, but I don't have enough understanding of asyncio to be sure if that's possible.

Details

• OS: Windows 10
• Python version: 3.8.5
• aiosqlite version: 0.15.0
• Can you repro on 'main' branch? Not tried
• Can you repro in a clean virtualenv? Not tried

[Caleb Hattingh]

@pfmoore There is definitely a race in how set_result() and set_exception() are called within the inner thread's run() method. Based on the report in #80, I submitted #89. I don't know whether this will resolve your specific issue, but I thought it was worth mentioning.

Please disregard, I don't think this has anything to do with #80 after all.

[Amethyst Reese]

I'm not certain, since I wasn't able to reproduce the issue, but I don't think this would be an issue with aiosqlite, as aiosqlite.Cursor.lastrowid is just a proxy to the underlying sqlite3.Cursor.lastrowid. Unless you are aware of sqlite3 reusing existing/open cursors, I'm not sure how this would happen.

[Eric Tröbs]

This is probably not a problem with this wrapper as the others stated already. I noticed this behaviour with the default sqlite3 module if an ON CONFLICT clause applies. In this case I used the unique keys that caused the conflict to retrieve the row after insert.

A small example, that should work with the default sqlite3 module and which may be useful to reproduce the issue with this wrapper:

cursor.execute('''
    CREATE TABLE IF NOT EXISTS foo (
        id       INTEGER PRIMARY KEY AUTOINCREMENT,
        project  INTEGER,
        name     TEXT,
        UNIQUE(project)
    )
''')
cursor.execute('''
    INSERT INTO foo (project, name)
    VALUES (?, ?)
    ON CONFLICT (project) DO
    UPDATE SET name = ?
''', (project_id, name, name))

# lastrowid is 0 if on conflict clause applies
if cursor.lastrowid > 0:
    row_id = cursor.lastrowid
else:
    cursor.execute('SELECT id FROM foo WHERE project = ?',
                   (project_id,))
    row_id = cursor.fetchone()[0]