upstream hijacking
pschrammel opened this issue · comments
Is your feature request related to a problem? Please describe.
Uploading big files (> 100 GB) is a pain as it's first buffered on the filesystem.
Describe the solution you'd like
On having received the headers the stream could be passed and routed to the application. The application then could handle each block of the body.
Describe alternatives you've considered
I tried nodejs as they have a proper streaming api/io but it's a pain. I used goliath server in the past but it's unmaintained.
Additional context
The rough context is a fileserver (webdav, ....). Where ppl can upload files but the real storage is handled by s3. The parsing of the body or pass through should be somehow selective (some paths' body should be parsed, some not ....).
Isn't this possible? At least since #2740
I think the question is about a way to stream the request body, not the response body. I haven't had time to think about it.
It is somewhat similar to a partial hijack, in that we'd send the env to the app with the client socket included. Might get a bit tricky to have the app then 'return' the socket along with the app 'reply', containing the response. to the request...
Do you have a reverse proxy (i.e. Nginx) in front of Puma, at least? Might not solve the problem but it will probably help.