Changing Page Limits Does Not Work
bryceandress opened this issue · comments
When trying to change the page limit rate I get the following message
❯ /usr/local/Cellar/node/15.2.1/bin/parlance followers -u --no-delay -p 100 -g 100
[warn] You are responsible for deciding if this is allowed
[warn] The authors bear no responsibility for your actions
[fatal] You have been warned; refusing to continue as-isjjjk
Using the latest version from git HEAD, I'm seeing the following:
[warn] Use --confirm-no-delay if you wish to disable delays
[warn] You are responsible for deciding if this is allowed
[warn] The authors bear no responsibility for your actions
[fatal] You have been warned; refusing to continue as-is
If the first line is missing from your output, this would pretty clearly be a bug.
If not, and --confirm-no-delay / -x works, this behavior is intentional (at least for now). The original authors sought to add a safety mechanism to ensure that a large number of concurrent users couldn't accidentally overwhelm Parler's servers unintentionally.
I'm open to suggestions here. Parler is very clearly struggling to maintain adequate performance, and the authors were genuinely worried about disrupting service in violation of the TOS. On the other hand, options to remove these default limits could probably be better-documented.
Do you think an option like --help-hidden (N.B. I'll try to think of a better name) make this situation clearer?
It's probably also worth mentioning that Parler doesn't always listen to the page size / limit options. Some API endpoints will likely ignore these options and return fewer results per page. Not great, not terrible, but ultimately outside of the authors' control.
❯ /usr/local/Cellar/node/15.2.1/bin/parlance --version
1.9.2
❯ /usr/local/Cellar/node/15.2.1/bin/parlance followers -u TuckerCarlson -p 100 -g 100
[warn] You are responsible for deciding if this is allowed
[warn] The authors bear no responsibility for your actions
[fatal] You have been warned; refusing to continue as-is
❯ /usr/local/Cellar/node/15.2.1/bin/parlance followers -u TuckerCarlson -p 100 -g 100 --no-delay -x
[warn] You are responsible for deciding if this is allowed
[warn] The authors bear no responsibility for your actions
[fatal] You have been warned; refusing to continue as-is
From what I am seeing it is not honoring the page limit option, the delay seems to be working when I tell it to be off with or without page limits. I agree with not fully opening the floodgates on Parler. In my experience barrier to entry would eliminate a lot of potential spam. Even just supplying build flags to allow the building of a version of parlance without the limits may be enough. I am not very familiar with node so I can't say for sure that this is possible but I imagine it has to be.
Sorry, I've removed my previous comment, as it was based on me completely misreading your comment. I see the bugs. The argument parsing code appears to be:
(i) Failing to properly handle cases where both the --confirm-no-delay/-x option and the --no-delay/-n options are supplied; and
(ii) Failing to change the limit parameter in the URL (visible from --verbose output) when a page size is specified, regardless of whether the "guard" option is present or not.
This is likely a combination of an original argument-parsing bug and one or more regressions.
I've just added a ratelimit log message for client-initiated randomized delays (visible in --verbose mode, and written to stderr), so those delays can be differentiated from delays and/or poor performance occurring on Parler's side.
As a temporary workaround, ./bin/parlance followers -u username -vx should provide you usable data with no randomized delay from the client. From experience in past testing, Parler does ignore the limit option more often than not, but the client should at least make an attempt.
Apologies again for the misread and rather-irrelevant response; I'll fix the above as soon as possible and get a couple unit tests added to guard against future regressions.
Ouch; looks like there's a third issue here that's causing the long-form options to be treated differently than the single-letter options:
$ ./bin/parlance followers -u TuckerCarlson -vn
[warn] Use --confirm-no-delay if you wish to disable delays
[warn] You are responsible for deciding if this is allowed
[warn] The authors bear no responsibility for your actions
[fatal] You have been warned; refusing to continue as-is
$ ./bin/parlance followers -u TuckerCarlson -v --no-delay
[network] Fetching v1/profile?username=TuckerCarlson
[ratelimit] Waiting for randomized delay to expire
(Edit: this portion should now be resolved in a73b832)
I'll go ahead and review everything in src/arguments.js and src/cli.js to figure out what's going on. This may take a few days due to a competing deadline.
Thanks for the report; this is definitely a major usability issue and barrier to use.
I think I've fixed this. This should now issue HTTPS requests with the proper limit parameter (visible in -v output on stderr), although most Parler API endpoints appear to ignore the option. Additionally, specifying conflicting options (e.g. -p and -g) should now produce an error as expected.
Although this probably won't let you pull thousand-result pages in practice, it did point out a bunch of argument parsing bugs. I'll mark this as resolved for now but feel free to reopen if you run into any additional weird behavior; I'll do the same.