Crash for response larger than 4GB
uranusjr opened this issue · comments
Originally reported in pypa/pip#9549.
CacheControl crashes with ValueError: bytes is too large when trying to downloading a file larger than 4GB. Since this is carried over from msgpack’s limitation, it may be reasonable to skip trying to cache such a response (maybe with a warning), or provide e.g. a callback for the user code to rescure when this happens.
An alternative could be a cc=5 format which stores the request/response content lengths as integer in the msgpack message and puts the data contents into the data after message
This could also enable using optimization for retrieval of responses data to actual files as it could be passed from a file descriptor
Doing some triage: I believe this has been fixed upstream via pypa/pip#11143, which uses SeparateBodyFileCache to work around msgpack's size limits.