Incompatible with Gitea
wizpresso-steve-cy-fan opened this issue · comments
Containerize(0,0): Error CONTAINER001: Failed to push to output registry: System.AggregateException: One or more errors occurred. (Failed to upload blob to https://<redacted>/blobs/uploads/blrlsok1smjzi7ywxnwdqm1ri; received BadRequest with detail {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"Stream uploads after first write are not allowed"}]}
This corresponds to here:
https://github.com/go-gitea/gitea/blob/1195d66c15ad6ead1f464b5d7ae30308e1278780/routers/api/packages/container/container.go#L324-L325C9
What version of the tooling are you using? Do you perhaps have a binlog we could look at to investigate what calls are happening? Or a network trace?
@baronfel I'm using the default options that comes with v0.3.2. Not sure what binlog you need
That version is quite old - can you try with any of the more recent stable versions? The most recent is 7.0.305, and it should contain performance improvements and protocol compatibility fixes for some registries.
We haven't verified compatibility with Gitea, however - there may be some registry-specific quirks that we'd need to adjust for, like there are with AWS.
@baronfel Well I can't even use 7.0.305 correctly, in Visual Studio when I publish the container using that version, it just terminates with something like 0xFFFFFFFE (I forgot the exact exit code). That said I was using .net7.0 and I'm not sure if I need .net8.0 for that
Health-Check-Server dotnet publish -c Release -p PublishProfile="Properties\PublishProfiles\git.wizpresso.com.pubxml"
MSBuild version 17.5.0+6f08c67f3 for .NET
Determining projects to restore...
Restored C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj (in 181 ms).
C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Program.cs(27,52): warning CS8604: Possible null reference argument for parameter 'taskId' in 'bool TaskItemRepository.UpdateHeartbeat(string taskId, string message)'. [C:\Users\SteveFan\git\git.wizpresso.com\Wiz
presso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Program.cs(27,75): warning CS8604: Possible null reference argument for parameter 'message' in 'bool TaskItemRepository.UpdateHeartbeat(string taskId, string message)'. [C:\Users\SteveFan\git\git.wizpresso.com\Wi
zpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Program.cs(45,53): warning CS8604: Possible null reference argument for parameter 'key' in 'bool Dictionary<string, TaskItemStatus>.ContainsKey(string key)'. [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Hea
lth-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Program.cs(53,51): warning CS8604: Possible null reference argument for parameter 's' in 'int int.Parse(string s)'. [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Program.cs(55,36): warning CS8601: Possible null reference assignment. [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\HealthCheckServer.dll
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\publish\
Building image 'wizpresso/heartbeat-monitor' with tags 20230705043714 on top of base image mcr.microsoft.com/dotnet/runtime-deps:7.0
Uploading layer sha256:9d21b12d5fab9ab82969054d72411ce627c209257df64b6057016c981e163c30 to git.wizpresso.com
Uploading layer sha256:eca6fd7d457aa616420d3c4779ab63315c88f2bf2b5eb5d84de8323a534e526f to git.wizpresso.com
Uploading layer sha256:da7db853689b61bdd84bdf450dc8794d109666da5ccbc2adb8a15b737f5620af to git.wizpresso.com
Layer sha256:9d21b12d5fab9ab82969054d72411ce627c209257df64b6057016c981e163c30 already existed
Layer sha256:eca6fd7d457aa616420d3c4779ab63315c88f2bf2b5eb5d84de8323a534e526f already existed
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : Failed to push to the output registry: System.AggregateException: One or more errors occurred. (Failed to upload blob to https://git.wizpresso.com/v2/
wizpresso/heartbeat-monitor/blobs/uploads/z7rscpu6npftkbi8kclt12e4c; received BadRequest with detail {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"Stream uploads after first write are not allowed"}]} [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\Healt
hCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : ) [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : ---> System.ApplicationException: Failed to upload blob to https://git.wizpresso.com/v2/wizpresso/heartbeat-monitor/blobs/uploads/z7rscpu6npftkbi8kcl
t12e4c; received BadRequest with detail {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"Stream uploads after first write are not allowed"}]} [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.UploadBlobChunked(String name, String digest, Stream contents, HttpClient client, UriBuilder uploadUri)
in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 286 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.UploadBlobContents(String name, String digest, Stream contents, HttpClient client, UriBuilder uploadUri)
in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 340 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.UploadBlob(String name, String digest, Stream contents) in D:\a\_work\1\s\Microsoft.NET.Build.Containers
\Registry.cs:line 373 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.Push(Layer layer, String name, Action`1 logProgressMessage) in D:\a\_work\1\s\Microsoft.NET.Build.Contai
ners\Registry.cs:line 251 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass40_0.<<Push>b__0>d.MoveNext() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry
.cs:line 449 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : --- End of stack trace from previous location --- [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Registry.Push(Image x, String name, String tag, String baseName, Action`1 logProgressMessage) in D:\a\_work\1\s\M
icrosoft.NET.Build.Containers\Registry.cs:line 460 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : --- End of inner exception stack trace --- [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-
Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) [C:\Users\SteveFan\git\git.wizpresso.com\Wizpre
sso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at System.Threading.Tasks.Task.Wait() [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\CreateNewImage.cs:line 149 [C:\Us
ers\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
Here's some log when uploading image to our internal Git server
Nope, 7.0.300 still doesn't work.
@wizpresso-steve-cy-fan Thanks for the additional logs! Can you try again with the 7.0.305 version of the tooling, and potentially also generate a binlog by adding the /bl flag to your publish command? There are a couple ways to try the 7.0.305 version of the tooling:
- update to the 7.0.305 SDK and delete any PackageReferences for the Mcrosoft.NET.Build.Containers package
- use whatever SDK version you want to, and add a PackageReference for to Microsoft.NET.Build.Containers at version 7.0.305
Separately, I see that there is a public demo instance of gitea available at https://try.gitea.io/, so it's very possible we could use that to test that our registry communications work as expected in recent versions.
I did a test on the absolute latest version of the tooling, pointing at a test repo I made on try.gitea.io, and I also was able to use our latest tooling. I'll try and capture this in a demo/sample on github.com/baronfel/sdk-container-demo later today as an example.
The package I made can be seen here: https://try.gitea.io/chethusk/-/packages/container/dotnet-sdk-container-demo/latest
Health-Check-Server dotnet publish -c Release -p PublishProfile="Properties\PublishProfiles\git.wizpresso.com.pubxml" /bl
MSBuild version 17.5.0+6f08c67f3 for .NET
Determining projects to restore...
C:\Program Files\dotnet\sdk\7.0.202\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): warning NETSDK1023: A PackageReference for 'Microsoft.NET.Build.Containers' was included in your p
roject. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [C:\Users\SteveFan\git\git.wizpress
o.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
All projects are up-to-date for restore.
C:\Program Files\dotnet\sdk\7.0.202\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.DefaultItems.Shared.targets(152,5): warning NETSDK1023: A PackageReference for 'Microsoft.NET.Build.Containers' was included in your p
roject. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [C:\Users\SteveFan\git\git.wizpress
o.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\HealthCheckServer.dll
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\publish\
Building image 'wizpresso/heartbeat-monitor' with tags 20230706015711 on top of base image mcr.microsoft.com:443/dotnet/runtime-deps:7.0
Uploading layer sha256:9d21b12d5fab9ab82969054d72411ce627c209257df64b6057016c981e163c30 to git.wizpresso.com
Uploading layer sha256:eca6fd7d457aa616420d3c4779ab63315c88f2bf2b5eb5d84de8323a534e526f to git.wizpresso.com
Uploading layer sha256:de6d1a9009f4674c6fd7b6a510850823ee3b99f076722f7362d089941ef6b37e to git.wizpresso.com
Layer sha256:eca6fd7d457aa616420d3c4779ab63315c88f2bf2b5eb5d84de8323a534e526f already existed
Layer sha256:9d21b12d5fab9ab82969054d72411ce627c209257df64b6057016c981e163c30 already existed
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Failed to push to the output registry: System.ApplicationException: CONT
AINER1001: Failed to upload blob to Chunked PATCH https://git.wizpresso.com/v2/wizpresso/heartbeat-monitor/blobs/uploads/mbwx5mx7d3tphge7rvjsctahy; received BadRequest with detail Docker-Distribution-Api-Version: regist
ry/2.0 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: X-Powered-By: ARR/3.0 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso
\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: Date: Thu, 06 Jul 2023 01:56:14 GMT [C:\Users\SteveFan\git\git.wizpresso
.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\
er\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: at Microsoft.NET.Build.Containers.Registry.PushAsync(Layer layer, Str
ing repository, Action`1 logProgressMessage, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Registry.cs:line 332 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-
Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass51_0.<<Pu
shAsync>b__0>d.MoveNext() in /_/src/Containers/Microsoft.NET.Build.Containers/Registry.cs:line 564 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: --- End of stack trace from previous location --- [C:\Users\SteveFan\git
\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: at Microsoft.NET.Build.Containers.Registry.PushAsync(BuiltImage built
Image, ImageReference source, ImageReference destination, Action`1 logProgressMessage, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Registry.cs:line 575 [C:\Users\SteveFan\git
\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Users\SteveFan\.nuget\packages\microsoft.net.build.containers\7.0.305\build\Microsoft.NET.Build.Containers.targets(195,5): error CONTAINER1013: at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.ExecuteAsync(C
ancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Tasks/CreateNewImage.cs:line 127 [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<EnableSdkContainerSupport>true</EnableSdkContainerSupport>
<SdkContainerSupportPackageVersion>7.0.305</SdkContainerSupportPackageVersion>
</PropertyGroup>
<ItemGroup>
<ContainerPort Include="9091" Type="tcp" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Build.Containers" Version="7.0.305" />
</ItemGroup>
</Project>
The test I did was using the very latest bits from main, so that error doesn't surprise me - I believe that next month's servicing release should have the fix though. Once we can confirm that we'll check in the test and update our docs for supported registries in this repo.
@baronfel Is it possible to do this without pointing a different nuget source?
Technically you could try using the 7.0.400 nightly builds of the SDK from the downloads page on dotnet/installer, or the 7.0.3xx nightlies as well. You could also clone the SDK, run ./build.cmd, and use the package from the package output directory there. But that's still another nuget source, just a local file folder source :)
Just touching base here - 7.0.400 and a few more 7.0.300 builds have happened since the last conversation on that thread - have you been able to successfully push to gitea @wizpresso-steve-cy-fan?
@baronfel I checked it again and surprisingly yes! Although there is a little twist in between:
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\HealthCheckServer.dll
HealthCheckServer -> C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\bin\Release\net7.0\linux-x64\publish\
Building image 'wizpresso/heartbeat-monitor' with tags 20230810030741 on top of base image mcr.microsoft.com/dotnet/runtime-deps:7.0
Uploading layer sha256:1d5252f66ea9b661aceca1027b3d7ca259a50608261a25b51148119ecf086932 to git.wizpresso.com
Uploading layer sha256:bcf64537485a6c5c9551e7024931a75fbcc706246561d3890baae68643021c62 to git.wizpresso.com
Uploading layer sha256:12024b286120c352cd56946ff30e5768fecdce7af30df645ae7c82402f64d8fe to git.wizpresso.com
Finished uploading layer sha256:bcf64537485a6c5c9551e7024931a75fbcc706246561d3890baae68643021c62 to git.wizpresso.com
C:\Program Files\dotnet\sdk\8.0.100-preview.5.23303.2\Containers\build\Microsoft.NET.Build.Containers.targets(196,5): error CONTAINER1013: Failed to push to the output registry: CONTAINER
1001: Failed to upload blob to Whole PATCH https://git.wizpresso.com:443/v2/wizpresso/heartbeat-monitor/blobs/uploads/fszribqr3taea3waofn4rsj0h; received RequestEntityTooLarge with detail
Date: Thu, 10 Aug 2023 03:06:52 GMT [C:\Users\SteveFan\git\git.wizpresso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Program Files\dotnet\sdk\8.0.100-preview.5.23303.2\Containers\build\Microsoft.NET.Build.Containers.targets(196,5): error CONTAINER1013: Connection: close [C:\Users\SteveFan\git\git.wiz
presso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
C:\Program Files\dotnet\sdk\8.0.100-preview.5.23303.2\Containers\build\Microsoft.NET.Build.Containers.targets(196,5): error CONTAINER1013: [C:\Users\SteveFan\git\git.wizpresso.com\Wizpre
sso\Health-Check-Server\HealthCheckServer.csproj]
C:\Program Files\dotnet\sdk\8.0.100-preview.5.23303.2\Containers\build\Microsoft.NET.Build.Containers.targets(196,5): error CONTAINER1013: ??????,???????. [C:\Users\SteveFan\git\git.wizpr
esso.com\Wizpresso\Health-Check-Server\HealthCheckServer.csproj]
So I have to grow the proxy body buffer size in the front proxy maybe, this issue is now technically solved.
Nice - looks like you're using preview 5 - I expect an improvement in previews 6 and 7 here as well based on some refactoring that occurred, so even that problem you see might be mitigated already.
I'm going to close this now that we've seen success here.