Cannot upload packages bigger than 2GB
Schroedingers-Cat opened this issue · comments
Describe the bug
Uploading a nuget package bigger than 2GB (happens easily when the nuget package is a chocolatey package with embedded installer) causes the following error on the server (Linux, Docker, latest):
nuget-server | Exception thrown during package upload
nuget-server | System.IO.InvalidDataException: Multipart body length limit 2147483647 exceeded.
nuget-server | at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.UpdatePosition(Int32 read)
nuget-server | at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
nuget-server | at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
nuget-server | at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
nuget-server | at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
nuget-server | at Microsoft.AspNetCore.Http.Features.FormFeature.ReadForm()
nuget-server | at BaGetter.Web.HttpRequestExtensions.GetUploadStreamOrNullAsync(HttpRequest request, CancellationToken cancellationToken) in /src/BaGetter.Web/Extensions/HttpRequestExtensions.cs:line 20
nuget-server | at BaGetter.Web.PackagePublishController.Upload(CancellationToken cancellationToken) in /src/BaGetter.Web/Controllers/PackagePublishController.cs:line 49
The problem already existed in the original baget and was resolved by increasing the MultipartBodyLengthLimit to 2GB.
To Reproduce
Steps to reproduce the behavior:
- Install like described here
- Create a package with more than 2GB of content (put a video file inside or a big installer)
- Upload it via
nuget push packagename serverurl
- Observe bagetter's output
Expected behavior
I'd expect the upload to succeed.
Additional context
@Regenhardt made a test build with the MultipartBodyLengthLimit increased to 8GB which fixed the problem for nuget packages smaller than 8GB. We observed that for local file storage, there's no need to have 8GB of free RAM when uploading a package of 8GB because writing to that storage type happens via file streams. Therefore, it would make sense to make the limit configurable by the user. This is not the case for cloud storages like AWS so that configuration should only apply for the local file storage.
About the AWS limit (RAM usage):
I actually had a PR open in the original Baget repo; it was never merged so I ended up forking Baget to get that fix.
https://github.com/bagetter/BaGetter/blob/main/src/BaGetter.Aws/S3StorageService.cs#L76
Compare to the version we use in "production" (in-house):
https://github.com/coveooss/BaGet/blob/master/src/BaGet.Aws/S3StorageService.cs#L72
For some reason I can't explain, the original code copies the whole stream to a MemoryStream; this is completely unnecessary as the AWS SDK is perfectly happy uploading from any stream.
If you have a PR open for your 8GB limit, let me know I'll chip in and fix the AWS part too.
Absolutely, go ahead and make a PR for the stream fix, thanks!