Add restart-at-key option, allow number of retries to be specified
castle-lemongrab opened this issue · comments
As a follow-up to #38 and as suggested via Twitter: add a CLI option to control the number of retries for server-side errors, and add a new client and CLI option to force Parlance to start at a specific time-series key.
The former will allow the user to increase runtime resilience to server-side errors/flakiness; the latter will provide users with a way to start or restart paged queries at any point in time – whether to recover from a previous failure, or to skip over more recent results.
Thanx mate, really important.
Upon encountering an HTTP 5XX error, we should also – at a minimum – apply the existing randomized delay mechanism unconditionally (instead of treating it like a normal HTTP 2XX request return). Ideally, we'd also increase the randomized delay mechanism for 5XX errors and provide exponential backoff.
These measures are aimed at avoiding theoretical service disruption when Parler API bugs are encountered. Since we're blinded to any and all Parler code, and can't rule this out, it'd be better to err (ayy) on the safe side.
Please also SUBSTANTIALLY increase the delay between retries, 1-2-3, to say 5, 10, 30 seconds, before bailing out. This could account for unstable wifi / vpn issues and of course parler's own un-reliability ;-)
I've merged PR #44, which should take care of everything in this issue except user-specified retry limits with exponential backoff on request errors.
The new option is -k/--start-key. You can use verbose mode (-v) to capture ISO 8601 "extended" timestamps (which include an appended serial number). The -k option should also tolerant of URL-encoded timestamps. Timestamps should be of the form 2021-01-20T17:00:00.000Z_0000.
Warning: Parler does not appear robust in the face of malformed timestamps, and will happily throw an HTTP 500 or issue a non-monotonic key for the next page of data. Existing checks should be sufficient to detect these conditions and abort, but please use caution until the ratelimiter's exponential backoff patch lands.
Any and all testing/feedback is appreciated; I'll try to get the rest in by EOD tomorrow at the latest.
SUPER COOL. BRAVO! Hurra!!! /// Suggestion: Please make last timestamp for -k recovery visible REGARDLESS of -v being specified in command, FOR ABNORMAL Terminations and 500s...
Further to my previous comment, I suggest a) you update the usage/help with the new syntax, maybe under hidden-options
b) upon failure exit due to parler error, 500s, etc, you TELL the user how to restart, maybe with the full command line ready for copy-paste, example:
FATAL ERROR from parler ......
TO RECOVER/Continue from the past page, do
"
parlance $* --start-key 2021-01-20T17:00:00.000Z_0000
"
where $* are whatever input parameters you got and the sample start-key is replaced by the actual start-key to resume where left.
If you really want to make it idiot_proof, you may even suggest using "tee -a" instead of plain "tee" for capturing output ;-)
We now should support both start and end keys; thanks to @milesmcc for the PR. We're also now validating/parsing both the start and end key CLI options (now -S and -E respectively). I'll add exponential backoff, close this, then get to work on initial unit and E2E tests.
I've added exponential backoff on error and increased the retry limit. I'm going to hold off on user-specified retry counts; the exponential backoff should be sufficient for now. I'll close this and move on to unit and E2E testing, which is really the critical path for making this reliable and resilient to regressions (N.B. TFW you don't write unit tests because you don't think the API is going to survive).
Please feel free to reopen refile if you find a problem (quite possible).