Memory leak when using Granian in Django
divaltor opened this issue · comments
Hello, thanks for your tool. It's really great, but there is problem with (possible) memory leak
Tested it with WSGI protocol on Django version 5.2, Granian 1.4.0 (and 1.4.3). In my case I upload a big video file to the server via multipart/form-data request and after request is done memory stay the same. I investigate it a lot and in the end found that the leak is related to Granian, because I've tried to do the same with standard manage.py runserver tool from Django and gunicorn
Granian:
granian --interface wsgi --host 0.0.0.0 --port 8000 server.wsgi:application
Gunicorn:
gunicorn server.wsgi
I also attach a link with both .bin files from Memray profiling for Granian and Gunicorn.
My guess that the problem is located here, because eventually Django read body from WSGI request and execute underline read method from this struct
Related Django code - https://github.com/django/django/blob/4ee68bb4f50dbe842461f8f340bb1633fddd4a53/django/core/handlers/wsgi.py#L78
Hi @divaltor, thank you for reporting this. Can you please also attach a MRE and the test method used?
Hi @divaltor, thank you for reporting this. Can you please also attach a MRE and the test method used?
@gi0baro of course. Take a look - https://github.com/divaltor/granian-mre-leak
@divaltor this should be fixed with upcoming 1.4.4
@divaltor this should be fixed with upcoming 1.4.4
Thank you very much, it become a lot better, but I guess it's not fixed completely
I've tried to load a big file multiple times in a row (~750 MB) and there is still small memory leak. It's not that big as it was in 1.4.3, but it's still there
See heap usage chart in memray window
@divaltor from what I can tell from local debugging, the leak is due to the fact wsgi.input in environ never gets garbage collected.
I'm not sure why this happens only in Granian (at least from your investigation), but there's not so much Granian can do itself to prevent this. Also, it is not clear to me why this happens only with Django, probably some references to input are kept alive forever, and thus never GCs.
I think the best thing to do here is to open up an issue on Django repo referencing this in order to get some support from them, as of today I can't really tell why this is happening and what's the best way for Granian to prevent Django to behave like this.
@divaltor from what I can tell from local debugging, the leak is due to the fact
wsgi.inputin environ never gets garbage collected.I'm not sure why this happens only in Granian (at least from your investigation), but there's not so much Granian can do itself to prevent this. Also, it is not clear to me why this happens only with Django, probably some references to input are kept alive forever, and thus never GCs.
I think the best thing to do here is to open up an issue on Django repo referencing this in order to get some support from them, as of today I can't really tell why this is happening and what's the best way for Granian to prevent Django to behave like this.
First time I debugged that issue with memory leaking I found that every ref is collected by GCs in Django. May be I missed something, I'll try to debug it few more times to find why it's happening.
Appreciate a lot of your contribution on fix ❤️