`BaseCursor#close(..)` can exhaust the system memory
briandamaged opened this issue · comments
Describe the bug
The BaseCursor#close(..)
method invokes BaseCursor#nextset(..)
, which will invoke self.fetchall()
under certain circumstances:
mysqlclient/MySQLdb/cursors.py
Line 83 in 8f0cbac
mysqlclient/MySQLdb/cursors.py
Line 134 in 8f0cbac
Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.
To Reproduce
- Create a new table
- Populate it with hundreds of millions of records
- Create a cursor that will yield all of the records from this table. (Ex:
SELECT * FROM HugeTable
) - Fetch a handful of records from the cursor. (ie:
cursor.fetchmany(100)
) - Close the cursor
Environment
This bug is present regardless of environment; however, here's the environment that I was using:
MySQL Server
- AWS RDS. (Presumably, the use Amazon Linux)
- Server Version 8.0.23
MySQL Client
- OS: Ubuntu Jammy
- Python >= 3.7
- mysqlclient 2.0.3 and mysqlclient 2.1.1
Additional Context
I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to self.fetchall()
probably needs to be replaced with a call to self.fetchmany(..)
. However, this might be complicated by structural details that I'm not aware of.