Doesn't support upload of large files, due to buffering
RoganDawes opened this issue · comments
When trying to upload a file larger than available memory, the process gets killed by the OOM handler (Linux).
This appears to be because the file is read entirely into memory before being written out to disk. If at all possible (given the constraints of the now-deprecated CGI module in use), the file should be streamed to disk as it is being received, in order to minimise memory usage.
Someone sent a PR a while ago to do that, but it introduced a lot of dependencies and I'd like to keep this to just built-in Python modules. If you can make a patch that allows file streaming without adding dependencies or update burden I'd be happy to look at it.
My python is not good enough to figure this one out, I think. Sorry.
I'll close this for now then. I'm fairly sure there are some threading issues that would crop up with trying to stream large files, so this may just require a more full-featured server.
I don't think that there should be any threading issues. It's just a question of using an HTTP/cgi api that allows streaming access to the parts of the message. But I lack the familiarity with python to easily identify those apis, so happy to close this.
The RAM use issue on large files has been fixed by #23 , so I'm changing this to "completed".
.