microsoft / ghcrawler

Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

deadletter oddities

jeffmcaffer opened this issue · comments

This morning the deadletter graph in the dashboard looked as pictured below. Since I was not taking any action, (and I assume you were not) its is hard to see how the deadletters could ever go down.
This may be dashboard related but I suspect something is amiss with the server side computation?
deadletter

Just now I used the command line to get the count and it said 260 while the dashboard itself is reporting 180 and storage explorer was saying 173 (though it was acting a little off so I restarted it and it reported 262). Ask three mechanisms and get four answers... may be some funny business on the Azure side?

It looks like it's been happening about every minute just when the value is not cached anymore. Those requests take more than 5 sec to complete when we are polling every 5 sec...
screen shot 2017-03-15 at 9 16 02 am

Interesting. We do cache the deadletter count for 1 minute which might explain why every minute it is longer. I would not expect it to take longer than 5 sec though. And i would expect it to return the right value when it does return...

anyway, lets monitor for a couple days and see if the problem resurfaces.

behavior seems to be back to normal. closing