Giters

datopian / r2-bucket-uploader

Cloudflare R2 bucket File Uploader

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multipart upload fails for large files

sorin89 opened this issue · comments

commented

Tested with 5GB and 8GB files.
Works fine for a 2GB file.

commented

I realized what the problem is. When the file is too large, it gets stuck on creating the arrayBuffer and fileHash.

commented

Interesting @sorin89 - computation of file hash would happen locally so I wouldn't expect that being a problem but good to know. I believe we should make it optional.

commented

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

commented

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

Solved, it was related to new update of Nextjs issue

commented

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

commented

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

It’s true that for large file , generating the hash string breaks the app. So I tried with a fixed hash string and was able to upload a 10gb file.

So the solution will be to find other alternatives to generate hash string

commented

@steveoni I would say we don't need to generate file hash at all by default. Hashing is useful if you would like to have:

  1. Versioning so that md5 hash can be used as a signature of the file content.
  2. Save storage, i.e., the same file content wouldn't be stored multiple times.

I think the main purpose of this library is to provide a basic app to be able to upload files from local disk to a bucket.

Could you please open a pull request removing hashing function? We can use file name as a prefix in the bucket.