PyMySQL / mysqlclient

MySQL database connector for Python (with Python 3 support)

Home Page:https://mysqlclient.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`BaseCursor#close(..)` can exhaust the system memory

briandamaged opened this issue · comments

Describe the bug

The BaseCursor#close(..) method invokes BaseCursor#nextset(..), which will invoke self.fetchall() under certain circumstances:

Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.

To Reproduce

  1. Create a new table
  2. Populate it with hundreds of millions of records
  3. Create a cursor that will yield all of the records from this table. (Ex: SELECT * FROM HugeTable)
  4. Fetch a handful of records from the cursor. (ie: cursor.fetchmany(100) )
  5. Close the cursor

Environment

This bug is present regardless of environment; however, here's the environment that I was using:

MySQL Server

  • AWS RDS. (Presumably, the use Amazon Linux)
  • Server Version 8.0.23

MySQL Client

  • OS: Ubuntu Jammy
  • Python >= 3.7
  • mysqlclient 2.0.3 and mysqlclient 2.1.1

Additional Context

I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to self.fetchall() probably needs to be replaced with a call to self.fetchmany(..). However, this might be complicated by structural details that I'm not aware of.