Cannot upload input stream directly to Azure Storage without loading stream in memory
fperreaultnv opened this issue · comments
Cannot upload input stream directly to Azure Storage without loading stream in memory
Issue Type : Bug
Actual
The Stream instance provided in ITusStore.AppendDataAsync cannot be uploaded directly to Azure Storage. The Azure Storage SDK provides a few APIs that allow to pass an input Stream, but a NotSupportedException is thrown when supplying the input Stream directly, specifically when the SDK calls Stream.Length property.
Expected
The provided Stream should be permissive enough so that it can be redirected to another service without having to manipulate, chunk, or buffer it.
Workaround
On the client app, we can limit the chunkSize to 50 or 100MB.
When we receive the chunk, we can copy it to a MemoryStream, and use that new Stream in the Azure Storage SDK.
Hi,
The stream passed to AppendDataAsync is the HttpRequestStream provided by ASP.NET Core wrapped with support for max size and client disconnect detection. The wrapping streams just forward the Length call to the HttpRequestStream. HttpRequestStream does not support reading the length as there is no way to determine the length of a remote stream without consuming it, i.e buffering.
The provided Stream should be permissive enough so that it can be redirected to another service without having to manipulate, chunk, or buffer it.
No this is the responsibility of the store. There is no way for tusdotnet to determine the optimal buffer size for all storage solutions used by stores. There's a huge difference in the buffering needed for e.g. writing to disk or writing to a remote backend.
On the client app, we can limit the chunkSize to 50 or 100MB.
When we receive the chunk, we can copy it to a MemoryStream, and use that new Stream in the Azure Storage SDK.
This is the right idea but I wouldn't recommend using the chunkSize on the client to determine your buffer size as you would open yourself up to a denial of service attack and possible memory issues. Looking at e.g. Xtensible.TusDotNet.Azure you can see that it uses the append blob block size as the buffer.
Closing due to inactivity