cursor.close() does not close when raise exceptions

[fridary]

• OS: Ubuntu 20.04
• Psycopg version: 2.9.7
• Python version: 3.9.17
• PostgreSQL version: 12.16

I have a lot of pains with psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block error and don't know how to wrap it correct. The main problem is my app hangs after getting this error, although it is in while loop and I wrapped code in try {} exception.

Users send tasks to mytable, script get task like in queue, mark them executed=1 and save back calculation results. The same db and table use 3 python scripts (for better speed performance, because of internal calculations).

conn = psycopg2.connect(**db_config_psql)

if __name__ == "__main__":

    while 1:
        try:
            cursor = conn.cursor()

            cursor.execute(f"""WITH cte AS (
                    SELECT id, body
                    FROM \"mytable\"
                    WHERE executed=0
                    ORDER BY id DESC
                    LIMIT 10
                )
                UPDATE \"mytable\" t
                SET executed=1
                FROM cte
                WHERE cte.id = t.id
                RETURNING t.id, t.body""")
            tasks = cursor.fetchall()
            conn.commit()


            if not tasks:
                continue


            # marking fetched rows as executed, because other processes (python scripts) get take tasks while calculating
            cursor.execute("UPDATE \"mytable\" SET executed=1, body='' WHERE \"id\">{}-1 AND \"id\"<{}+1;".format(tasks[-1][0], tasks[0][0]))
            conn.commit()

            # making calculations here

            cursor.execute(f"""update \"user\" set score=score-data_table.value
            from (select unnest(array['1,2,3,4,5']) as id, 
                    unnest(array['10,20,30,40,50']) as value) as data_table
            where \"user\".id = data_table.id;""")
            conn.commit()


            cursor.close()



        except Exception as e:
            error_logger.error('~~~~~~~ error in main while loop ~~~~~~~')
            error_logger.error(traceback.format_exc())
            error_logger.error('{}: {}'.format(type(e), e))
            try:
                if cursor:
                    error_logger.error('in cursor')
                    cursor.close()
            except NameError:
                error_logger.error('exception NameError')
                pass
            error_logger.error('~~~~~~~ finished ~~~~~~~')
    

After this I'm getting psycopg2.errors.DeadlockDetected: deadlock detected:

2023-09-28 17:21:22,164 ERROR (754) ~~~~~~~ error in main while loop ~~~~~~~
2023-09-28 17:21:22,165 ERROR (755) Traceback (most recent call last):
  File "/task.py", line 557, in <module>
    cursor.execute("UPDATE \"mytable\" SET executed=1, body='' WHERE \"id\">{}-1 AND \"id\"<{}+1;".format(tasks[-1][0], tasks[0][0]))
psycopg2.errors.DeadlockDetected: deadlock detected
DETAIL:  Process 481841 waits for ShareLock on transaction 27176187; blocked by process 475489.
Process 475489 waits for ShareLock on transaction 27176197; blocked by process 481841.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (7638,3) in relation "mytable"


2023-09-28 17:21:22,165 ERROR (756) <class 'psycopg2.errors.DeadlockDetected'>: deadlock detected
DETAIL:  Process 481841 waits for ShareLock on transaction 27176187; blocked by process 475489.
Process 475489 waits for ShareLock on transaction 27176197; blocked by process 481841.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (7638,3) in relation "mytable"

2023-09-28 17:21:22,165 ERROR (760) in cursor
2023-09-28 17:21:22,165 ERROR (765) ~~~~~~~ finished ~~~~~~~
2023-09-28 17:21:27,174 ERROR (754) ~~~~~~~ error in main while loop ~~~~~~~
2023-09-28 17:21:27,174 ERROR (755) Traceback (most recent call last):
  File "/task.py", line 519, in <module>
    cursor.execute(f"""WITH cte AS (
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block


2023-09-28 17:21:27,174 ERROR (756) <class 'psycopg2.errors.InFailedSqlTransaction'>: current transaction is aborted, commands ignored until end of transaction block

2023-09-28 17:21:27,174 ERROR (760) in cursor
2023-09-28 17:21:27,175 ERROR (765) ~~~~~~~ finished ~~~~~~~
2023-09-28 17:21:32,183 ERROR (754) ~~~~~~~ error in main while loop ~~~~~~~
2023-09-28 17:21:32,184 ERROR (755) Traceback (most recent call last):
  File "/task.py", line 519, in <module>
    cursor.execute(f"""WITH cte AS (
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block


2023-09-28 17:21:32,184 ERROR (756) <class 'psycopg2.errors.InFailedSqlTransaction'>: current transaction is aborted, commands ignored until end of transaction block

2023-09-28 17:21:32,184 ERROR (760) in cursor
2023-09-28 17:21:32,184 ERROR (765) ~~~~~~~ finished ~~~~~~~


...

Okey, DeadlockDetected I can understand somehow why it happens (perhaps 3 python scripts use database). But Why I get current transaction is aborted, commands ignored until end of transaction block? Because of this error script hangs, while looping sends this error each time.

I suppose until end of transaction block is because DeadlockDetected, but in try {} I set cursor.close(), meaning don't wait ending of that failed transaction and I want start again in loop. Why it happens? How can I fix that? And if I stop and restart script, then it will start looping again without errors (until next current transaction is aborted)

[Daniele Varrazzo]

I don't see any rollback. It means that, even if you handle the exception in Python, the postgres connection is left in a failed transaction status.

Your error is assuming that cursor.close() terminate the transaction or close the connection: it doesn't. A cursor is only an object to send a command and handle a result, but the transaction control belongs to the connection object. You can terminate the transaction using conn.rollback().

You can also use with connection to terminate a transaction automatically, with a commit in case of success, or with a rollback in case of exceptions:

while 1:
    try:
        with conn:
            cursor = conn.cursor()
            ...
    except Exception:
        ...

In this code you shouldn't use conn.commit(), because it would be executed automatically on with context exit.

All the details are in the docs.

Please note that with conn will only close the transaction, not the connection, in psycopg2, so you can enter as many with blocks you need (but you can't nest them). In Psycopg 3 with conn will close the connection at the end of the block, but you can use with conn.transaction() for the same commit-or-rollback pattern.