[BUG] [Girder-Upload] Upload doesn't tolerate very large numbers of files.
subdavis opened this issue · comments
I tried to upload 2K at once. Upload threw a resource exceeded exception. I think we need to limit the number of outstanding POST requests.
Even tallying the files to provide pre-upload summary takes too long.
Upload threw a resource exceeded exception
Is this a server side or client side exception? Could you provide the error message?
I intend to use a pool-based approach moving forward, so this wouldn't be an issue anymore.
I don't know if this is related, but in another project I tried to fetch a huge number of small files at once and I got a similar error. Chrome fails in odd ways (using both HTTP 1.1 and HTTP 2.0), and not always at the same point in the process. Bluebird has a nice method to limit concurrent requests, and I ended using that with a concurrency limit that was ~24. In HTTP 1.1 most browsers will limit concurrency to 6, but there is overhead to start and stop. I ended up with the best throughput by using roughly between 2 and 5 times that number.
Is this a server side or client side exception? Could you provide the error message?
Client error. I believe too many outstanding requests are allocated at once. This isn't about concurrent connections to girder, but rather whatever memory Chrome allocates when an XMLHTTPRequest is constructed (probably)
I intend to use a pool-based approach moving forward, so this wouldn't be an issue anymore.
Yeah, we should do something similar here.