Add throttling to video downloads to mitigate risk of account suspension
rbrtrs opened this issue · comments
Hi folks,
I would just like to inform you that Blink has suspended my account after using blinkpy-0.17.0 once to download my videos. I have downloaded my videos before using my own script based on the API calls used in blinkpy. Never had an issue (except API changes). Now Blink decides to lock out paying customers that need automation other than Alexa or IFTTT.
Thank you for your excellent work @fronzbot, it was really appreciated. But I think I am done with Blink now.
BR,
rbrtrs
Yeah this tends to happen if too many excessive calls are logged. If I may ask, do you have a copy of your script you could share? Maybe there's something in there I can use to artificially limit API call frequency to prevent this in the future.
I was using my own (shell) script last time one week ago or so. There was the trouble with client verification. Changing it slightly made the verification step work again, but on the next run it again needed verification. So I disabled the job due to lack of time to take care.
Then today I noticed your new release with the new login endpoint and decided to give it a try (with blinkpy itself, not my own shell script). Worked like a charm. Half an hour later got the suspension e-mail.
I can share my script tomorrow, but it is very basic compared to blinkpy. I am indeed throttling the downloads a lot using sleeps.
My account is still suspended.
What I was doing in my script is this:
- login (using v4 endpoint)
- verify client (if required)
- sleep 5 seconds
- retrieve list of media using midnight of the latest downloaded file as 'since' date/time (however 'since' never worked reliably for me)
- sleep 5 seconds
- download thumbnail file
- sleep 5 seconds
- download video file
- repeat from 5 until page finished
- repeat from 3 in case there are more pages
Existing local files and deleted files have been skipped. Average number of files downloaded was about 30 (depending on the frequency of running the script of course). So not what I would call excessive usage.
I'm specifically curious in the methods and/or api endpoints you were using when you used the blinkpy library. The download videos method I have is basically a replica of this script which has been in use for awhile without issues (afaik)
OK, got you...
Basically, I did this:
blink = Blink()
if os.path.exists("authfile"):
auth = Auth(json_load("authfile"))
blink.auth = auth
blink.start()
if not os.path.exists("authfile"):
blink.save("authfile")
blink.download_videos("backup", "2021/01/01")
Cool thanks!
A few follow-up questions:
- I assume you had already sent the 2FA pin via the library prior to attempting the video download, is that correct?
- Did the library successfully download any videos or were you instantly suspended? If it did download videos, how many did it grab before you were suspended?
- How many cameras do you have in total?
As an anecdote, I've run the same code since 0.17.0 and haven't had issues, but I'm usually only grabbing videos from the past 24hrs across two or three cameras.
You're welcome!
- I have entered my credentials, waited for the 2FA pin, then entered the pin. All of that has worked very well.
- The library was successfully downloading my videos. Not sure if it was all of them. But in this case I must admit it has been a lot of files. I have 6 sync modules in my account, so I have 6 times the amount in cloud storage. But they have limited storage time to 30 days recently.
- I have a total of 11 cameras, but only 3 active at the moment.
I believe the problem was really the amount of files (and transferred data) in short time in that case. However, I did that before, but with my script and the sleep periods. Using blinkpy it was quite fast. I think you have implemented also some throttling. Did I use the library wrong?
My account has been unlocked now BTW.
Did I use the library wrong?
No, you did not use the library wrong, you used it exactly as intended. I think you're right that the quantity of items downloaded is what flagged your account.
Using blinkpy it was quite fast. I think you have implemented also some throttling.
I do NOT have any throttling built in between calls to download. Based on your use case, I think adding in a programmable delay between API calls would make sense, so I am going to track this issue as a feature request for built in video download throttling
It's really a pity. All I want is to be able to backup my files, ideally nightly, and that had worked for a couple of months perfectly. I like the product, I like the features, but I dislike that it is not open for at least some integration into home automation and a possibility to backup your data. Downloading individual files in the App is not an option here. That's what I also told them by the way.
Anyways, thanks for listening and considering the delay as a feature!
Amen to that. Even more annoying is I hopped on to the Blink train during the Kickstarter campaign in large part because they promised:
- Local storage
- An open API
We got neither. Hence this library haha.
And this library is a blessing!
I was so frustrated this morning that I ordered a TP-Link Tapo C200 camera with same day delivery. Very nice product for 35 €. Downside: you need a power supply, not weather proof. Upside: local storage, pan and tilt, RTSP protocol, yet also no open API (at least I think so)...