Getting 403 forbidden from itunes.apple.com, requesting too fast?

Question

Getting 403 forbidden from itunes.apple.com, requesting too fast?

codeswhite opened this issue 3 years ago · comments

Max Grinberg commented 3 years ago

I just might implement proxy usage...

Edd Barrett · Answer 1 · Tue Nov 30 2021 00:13:02 GMT+0800 (China Standard Time)

I get this too when running on a large collection.

Rego Sen · Answer 2 · Wed Dec 01 2021 13:37:55 GMT+0800 (China Standard Time)

Have you tried the --throttle option? It specifies the number of seconds to wait between each query (i.e. each album). It defaults to 0, but if you find a good number that works, I could update the default value.

Rego Sen · Answer 3 · Fri Dec 03 2021 06:57:52 GMT+0800 (China Standard Time)

I’m also considering responding to a 403 by sleeping for a long time before trying again (and informing the user of course). Does Apple indicate how long it locks you out for?

…

On Nov 29, 2021, at 10:13 AM, Edd Barrett ***@***.***> wrote: I get this too when running on a large collection. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Rego Sen · Answer 4 · Sun Dec 05 2021 06:15:47 GMT+0800 (China Standard Time)

Ok, I just pushed a new version (1.5.0) that will automatically sleep for 30 minutes and retry whenever it encounters a 403 or 429.

From what I'm reading, Apple enforces a limit per hour, but I'm not sure if it's on absolute hour intervals or relative to the first request, so 30 minutes should be a good duration for this. (It will sleep again if the retry hits another 403 or 429)

Max Grinberg · Answer 5 · Thu Dec 09 2021 12:06:41 GMT+0800 (China Standard Time)

I found out that the 403 is returned from the server if I have mady more than X request in the last Y seconds.
It's enough to wait just few seconds after 403.

A good throttle value i found to be around 1-2 seconds, maybe a little more on faster computers.

Anyway I have forked your code, refactored it and rebranded it to my needs...
I've stripped local cover finding and prioritizing as I don't have those, only embedded.
Added multiple download sources besides Apple, image optimization (size minification) after download, about 20 command line options, integration with youtube_dl, multithreading and more..

I will probably publish it on my profile, maybe we can merge projects

Edd Barrett · Answer 6 · Thu Dec 09 2021 21:59:21 GMT+0800 (China Standard Time)

Might be worth making the wait time configurable and lowering the default: seems you can re-request much quicker than 30 minutes when you get a 403 (10 seconds?).

Rego Sen · Answer 7 · Fri Dec 10 2021 09:21:26 GMT+0800 (China Standard Time)

Good call, I just reintroduced --throttle which will default to 3 but not kick in until a 403 or 429 are encountered.

@codeswhite good call on stripping out all the "local cover finding" options, that made the instructions really complicated and I'm not sure how useful they are. (They were introduced by another user, so I know one person used them at least.)