Exception could be improved for request rate exceeded on a cosmosdb backed table
johlrich opened this issue · comments
If you hit your provisioned RU/s rate limit you'll currently get the following message:
An unhandled exception of type 'System.AggregateException' occurred in mscorlib.dll: 'One or more errors occurred.'
Inner exceptions found, see $exception in variables window for more details.
Innermost exception Microsoft.WindowsAzure.Storage.StorageException : Element 0 in the batch returned an unexpected response code.
The exception has HttpStatusCode set to 429 it could be used to instead throw a nicer exception about rate limits + suggesting a retry.
The only bummer is that I don't see a way using the StorageException to get at the http response header x-ms-retry-after-ms which contains how long you must wait unless I'm missing something. Could still be better to have something explicit even with that?
I'll have a look on a weekend. Should be doable.
The only bad thing is that I have absolutely no idea how to write a failing test for this scenario. Perhaps, we can try to create an explicit test which could be run manually against real CosmosDb (not emulator) .
@jkonecki Oh, cool! Didn't know. I don't even have it installed on my machine ))
@johlrich How do you connect to table api? Connection string format? Have you tried running against local cosmosdb emulator?
Normal connection string format for Storage Tables, just different url worked against an account provisioned in the portal: DefaultEndpointsProtocol=https;AccountName=youraccountname;AccountKey=...;TableEndpoint=https://youraccountname.table.cosmosdb.azure.com:443/;
Unfortunately, I don't believe the emulator can be used at this time since it doesn't support the table endpoints.
@yevhen For manual testing, folks without a current azure subscription should still be able to run the test with one of the time limited instances they started offering. https://azure.microsoft.com/en-us/try/cosmosdb/
As an aside, something I'm finding interesting about this approach is that you can still connect to the documentdb api through its endpoint (youraccountname.documents.azure.com) with the same key. When the API needs a Database/Collection name you supply the literal "TablesDB" for database and the tablename for collection. So far this has worked both with the DocumentDB sdk to dynamically tune provisioned RU/s throughput as well as subscribe to the table with the ChangeFeed support.
I've checked it and I'm afraid that Azure SDK doesn't correctly handle the throttling scenario. What's interesting is that I've then inspected (with Fiddler) the api response and it also doesn't contain this information. Moreover, the response status code is still 202 (Accepted) while 429 is mentioned in the body 😞
Looks like CosmosDb Table api support is half-baked at this moment. @johlrich you need to open an issue against Cosmos, since SS doesn't do anything destructive to error information returned by the service.
You can obtain HTTP status code of the throttled requests:
Please note Console.WriteLine(ex.RequestInformation.HttpStatusCode);
[Test]
public async Task When_hitting_request_limit_during_writes()
{
const int streamsToWrite = 1000; // 800RU the default limit for CosmosDb table
await Task.WhenAll(Enumerable.Range(1, streamsToWrite).Select(async streamIndex =>
{
var partition = new Partition(table, $"test-RU-limit-on-writes-stream-{streamIndex}");
var stream = new Stream(partition);
var events = Enumerable.Range(1, 10)
.Select(CreateEvent)
.ToArray();
try
{
await Stream.WriteAsync(stream, events);
}
catch (StorageException ex)
{
Console.WriteLine(ex.RequestInformation.HttpStatusCode);
Console.WriteLine(ex);
if (Debugger.IsAttached)
Environment.Exit(-1);
}
}));
}