Nitter reports throttling on bad token before attempting to refill the pool and serve the request
bbappserver opened this issue · comments
@PrivacyDevel
If the token pool is stale nitter will report that it is throttled instead of first attempting to discard all stale tokens and refill the pool as much as it is able, and then try to serve the request.
This seems to happen intermittently, so I'm not sure if it's when a single token or several are stale resulting in a request failure.
It definitely happens when restarting nitter after a while as the tokens still live in redis but are all stale.
The correct algorithm would be something like
def onRequest(url):
while poolNotEmpty:
tok=getOldestTokenFromPool()
r=requestToTwitter(url,tok)
if r.failed():
discardToken(tok)
else:
return r
#pool was completely stale
refillPool() #refill some or all of tokens until you get service unavailable or reach the count i nthe ocnfig
tok=getOldestTokenFromPool()
r=requestToTwitter(url,tok)
if r.failed():
return ThrottledError() #Ok we're out of tokens and we also aren't getting any new ones
else:
return r #Managed to get a working token even though the whole pool was stale before.
I strongly assume that with the use of guest accounts this should no longer be an issue for now. Please reopen if I am wrong.