Overcoming "API rate limit exceeded"
andersjohansson opened this issue · comments
Trying to export all my 315 playlists I quickly run into the message:
{
"error": {
"status": 429,
"message": "API rate limit exceeded"
}
}
Reading up on the limits I see:
Rate limiting is applied on an application basis (based on client id), regardless of how many users are using it.
https://developer.spotify.com/web-api/user-guide/#rate-limiting
So I guess one easy way of overcoming the limit would be to fork (and host it at some other address I guess) and register it as my own application with spotify.
Do you think that is all there is to it? Looking at the code I see that exportify is doing it in batches of 20 lists at a time (I think). Would some waiting between each request help?
Another thing would probably be to implement some "wait and retry"-scheme for when these responses are recieved.
This is just a few loose ideas. I am adding the issue here to start the discussion since I saw some other user having the same problem somewhere.
Exportify seems really great otherwise! Nice that you have taken the time to code this up.
Yes I think you're right that the easiest way to resolve this for now would be to use your own app client_id. To help with this, I've made a change which allows you to use your own client_id in the URL, so you won't need to fork and modify the code.
To set up the app you need to:
- Visit https://developer.spotify.com/my-applications/.
- Click 'Create an App' and enter some details (anything will do).
- Add this URL to 'Redirect URIs':
https://rawgit.com/watsonbox/exportify/master/exportify.html.
Then you can use the client_id from that app as follows:
https://rawgit.com/watsonbox/exportify/master/exportify.html?app_client_id=[your client_id]
Update 30/11/20
Please replace https://rawgit.com/watsonbox/exportify/master/exportify.html with https://watsonbox.github.io/exportify/ in the above instructions since the app is now hosted on Github Pages.
That's brilliant!
Will test it later.
Den 2015-06-10 17:20, Howard Wilson skrev:
Yes I think you're right that the easiest way to resolve this for now
would be to use your own app |client_id|. To help with this, I've made
a change which allows you to use your own |client_id| in the URL, so
you won't need to fork and modify the code.To set up the app you need to:
- Visit https://developer.spotify.com/my-applications/.
- Click 'Create an App' and enter some details (anything will do).
- Add this URL to 'Redirect URIs':
|https://rawgit.com/watsonbox/exportify/master/exportify.html|.Then you can use the |client_id| from that app as follows:
|https://rawgit.com/watsonbox/exportify/master/exportify.html?app_client_id=[your client_id]
|—
Reply to this email directly or view it on GitHub
#6 (comment).
With a bit over a 100 playlists I hit the rate limit even with a client id for an own app.
Yes, there will always be some upper playlist limit on the "bulk export" facility unfortunately. Ideas/PRs welcome.
Maybe a multiple selection option is the easiest solution with the smallest amount of manual effort. Like a checkbox for each playlist which allows you to only export several at once. I can make that these days when I can spare some time.
The Redirect URI isn't working... thoughts?
https://rawgit.com/watsonbox/exportify/master/exportify.html.
I get - INVALID_CLIENT: Invalid redirect URI
@jshebetich You're right, I introduced a change which broke this. Fixed now plus a test to verify behavior.
Dirty workaround adding a 500ms delay between API requests : https://gist.github.com/Alex131089/01611e017d874af51cb52ceefecc2d5d
Using a dirty sleep function found on internet & may hang the browser (but still working & fetching actually), as I don't know how to do this properly in JS / JQuery (async/event function call makes it a bit hard to introduce delay when you're not used to).
Managed to get my 192 playlists with that.
@watsonbox Exportify just won't work for me. Even when i just try individual playlist i never get an option to save is as CSV file, it just shows the whole text within the web page, and i can't find anyway to save it which means i can't use it. Any suggestions?
Just used the "create an app" method, which worked perfectly with 141 playlists. Thanks! :)
@Alex131089 care to writeup a quick blurb on exactly how you did it? Been trying to use your 'Dirty workaround' but haven't been able to figure it out.
- Are we supposed to setup the local server and access http://localhost:8000/exportify.html ?
- What about the Redirect URI? Does that change since we're working locally?
@BrianRoach I set up a local server and used a "create an app" clientid (which wasn't enough by itself). Redirect URI should be correctly handled in latest exportify. I added my client URL 127.0.0.1 (http://127.0.0.1/, http://127.0.0.1/expotify.html) as allowed, got a token and started the export. Maybe you can use rawgit url directly (allowing the url).
What have you tried ?
when you say you added your 'client URL 127.0.0.1 (http://127.0.0.1/, http://127.0.0.1/expotify.html) as allowed' what do you mean? Still learning here so please bare with me.
@BrianRoach : I meant the redirect URI, as in #6 (comment), the URL Spotify will allow to get the token (the location of your exportify).
Ah, I see. I had tried to enter http://localhost:8000/exportify.html. I'll try 127.0.0.1 and see if it works.
Thank you watsonbox for this done-well app.
I saw this error when exporting a single playlist with 8000 tracks. (This is my superset of everything I've tagged in spotify). When I used spotify to copy all these tracks to a new playlist, skipping duplicates, the number dropped to 6000 and I was able to export as expected and saw no error.
Thanks Watsonbox for this! I was about to start writing something similar at my frustration over the Spotify UI and lack of flexibility and getting my artists into Songkick.
I am also encountering a rate limit issue, I have 626 playlists with 10-100 tracks each.
I have tried Alex130189's code with 250/500/750 ms values but I am still hitting the rate limit.
I will continue to try incrementally higher values I suppose! :)
Delay didn't help me but restricting the code to exporting 100-200 playlists at a time did
I was unable to export my 127 playlists (1268 total tracks) even with my own developer token on the rawgit.com site, but running it locally let me download them. Hard to tell for sure if that was the important factor, though.
Was able to successfully export individual playlists via the normal URL just fine. The 'export all' function however gave: "503 Service Temporarily Unavailable [insert page break] nginx", so I attempted your suggestion in comment #2.
Properly inserted my client_id created via the user-made Spotify app into the URL (with AND without the brackets during multiple attempts), e.g.
To which the aforementioned error: "INVALID_CLIENT: Invalid Client" arises. If all else fails, I'm going to export each of my 84 playlists individually for my sake, but getting the 'export all' function to work would be greatly appreciated.
I moved to https://github.com/Alex131089/spotify-backup personally (forked from https://github.com/marpala/spotify-backup with more data retreived).
@jshebetich You're right, I introduced a change which broke this. Fixed now plus a test to verify behavior. 7db76bf
-@watsonbox
I'm still getting INVALID_CLIENT: Invalid redirect URI
The @Alex131089 fork doesn't appear to work anymore, but https://github.com/jnylen/spotify-backup did work for me and it supports a new xspf output format as well.
@Iristyle : any clues how to get this working? I get this error in Terminal, whether i double click it or run from command line.
Traceback (most recent call last):
File "[path]/spotify-backup.py", line 5, in
import http.client
ImportError: No module named http.client
I have tweaked exportify code to allow exporting in batches. I added two input boxes near the 'export all' button. Input the min and max playlist numbers to export, and when you hit 'export all', this range is ONLY what will get exported. A range of about 100 works for me most of the time, you can experiment with higher and lower ranges as you desire.
It's a fairly simple tweak in two places in the exportify.js file. To add the min/max boxes, find this code:
<th style={{width: "100px"}} className="text-right"><button className="btn btn-default btn-xs" type="submit" onClick={this.exportPlaylists}><span className="fa fa-file-archive-o"></span> Export All</button></th>
And replace with:
<th style={{width: "100px"}} className="text-right"><button className="btn btn-default btn-xs" type="submit" onClick={this.exportPlaylists}><span className="fa fa-file-archive-o"></span> Export All</button><input id="plMin" placeholder="min" /><input id="plMax" placeholder="max" /></th>
Then for the range checks, look for this code:
$(playlists).each(function(i, playlist) { playlistFileNames.push(PlaylistExporter.fileName(playlist)); playlistExports.push(PlaylistExporter.csvData(access_token, playlist)); });
And replace with:
$(playlists).each(function(i, playlist) { var minVal = document.getElementById("plMin").value; var maxVal = document.getElementById("plMax").value; if ((i >= minVal) && (i <= maxVal)) // CSP - test limiting batch saving. 100 seems to be a good number. { playlistFileNames.push(PlaylistExporter.fileName(playlist)); playlistExports.push(PlaylistExporter.csvData(access_token, playlist)); } });
That's it! I was able to get my 272 playlists backed up in just 3 batch exports. Hope it works as well for others.
@cpinheir1103 PR? :)
I've made a version that uses Promises to delay server queries. My apiCall function takes a delay parameter, to which I pass some multiple of the iterator index in query-heavy loops/maps. It's sufficient to avoid errors in most cases, even with large playlists, but it can still fail when exporting multiple playlists, because I haven't yet implemented waiting for one csv at a time in my ZipExporter.
@pavelkomarov – Hi, just tried this out and I'm seeing quite a few bugs. In Chrome, the "Get Started" button doesn't respond when clicked. In Firefox, after loading my playlists, the arrows to go to the next page are not responding, and if I click "Export All" on the first page, I get a pop-up window that shows the number "400" w/ nothing else (I have 470 playlists). If I hit OK, it pops up again a couple times, and at the bottom of the page is the usual API Rate Limiting error.
@cpinheir1103 – Is there a web link for this variation of Exportify? Otherwise, how do you edit the exportify.js file and run the app?
@shape55 Open some issues https://github.com/pavelkomarov/exportify/issues, and we'll try to get this sorted out. I've tested exclusively in Firefox, so it's not entirely surprising it's not working perfectly in Chrome, but I just downloaded Chrome and tried it and am not having a problem. I have like 12 playlists of my own, so I didn't notice I broke the next/previous page functionality. 400+ is really pushing it, so I'll need to fix issue #1 to help you there.
@pavelkomarov – Ok, gotcha – I just opened 3 new issues there. Thanks
@Alex131089's approach is the right one. What causes the rate limit error is huge short bursts of traffic, not several bursts over minutes. It's unlikely using your own clientid would really help, because it's unlikely someone else is using exportify at exactly the same time as you are.
I've implemented waiting in my Export All functionality, so now each export is rate-limited, and multiple exports are accomplished in series. I've also fixed the paginator.
Thanks so much everyone for the continued feedback and apologies for my lack of reactivity. Let's use this issue for advancing on rate liming.