Memory usage spike caused by piexif. Incoming requests should be logged by thumbor before image processing
bondido opened this issue · comments
Thumbor request URL
/unsafe/fit-in/320x240/filters:no_upscale():format(webp)/host/piexif_crash_picture.jpg
Expected behaviour
Thumbor should process all images readable by other applications, or generate error in log instead of exploding with memory usage
Actual behaviour
Processing some (really really rare in our case) images is causing huge memory usage spikes. Thumbor run on kubernetes with memory usage limits is OOMKilled and image not served. Our regular limit is under 1Gi, and the attached sample image crashes thumbor even at 8Gi memory limit. No sign of problems in log, even at debug level.
The problems appeared after we've upgraded from thumbor 6 (6.7.1) to 7 (7.7.1). Sample image, and the others that we've found were processed successfully with thumbor 6.7.1
Sample image:
zipped image in case the github manipulates the image file content while upoading:
piexif_crash_picture.zip
As we were able to dig in, the problem is caused by piexif library.
We've found the issue #1204 which seems similar to ours. The issue status is closed but as I've followed the discussion, it seems like the changes are in still pending pull request.
I've also found that the piexif library replaced pyexiv2 with thumbor 7.3.1 release (https://github.com/thumbor/thumbor/releases/tag/7.3.1).
That's probably the cause that the problematic images are processed succesfully by thumbor 6.7.1.
Another problem, which made it really difficult to point exact corrupted images within all the traffic running through our thumbors, is that there is no single information logged - even if the log level is set to debug. I've observed, that after receiving the request and while trying to allocate the memory, logs seem to be suspended/buffered (?) and when the thumbor is killed with SIGKILL by memory manager, obviously nothing is dump.
I think, at least at some higher detail log levels (info, debug ?), the request received should be logged BEFORE any further processing get kicked.
Operating system
Ubuntu
This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 15 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.
Still an issue
This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 15 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.
This issue was closed because it has been stale for 15 days with no activity.