404 When Trying to Download galleries

[Michael Riley]

Hi,

I'm currently trying to run the downloader on around 12 years worth of galleries. I ended up running the python script in the repository for both and used the generated token.

However it is giving a 404 Error. Is this still just Smugmug's servers describing it as a 404, when its a 401 like in a previous issue?

time="2021-02-12T15:53:42-06:00" level=info msg="Getting albums for user ideasinmotionmedia...\n"
time="2021-02-12T15:53:45-06:00" level=error msg="404 Not Found"
time="2021-02-12T15:53:45-06:00" level=error msg="404 Not Found"
time="2021-02-12T15:53:45-06:00" level=error msg="404 Not Found"
time="2021-02-12T15:53:45-06:00" level=fatal msg="Error getting user albums: Error getting albums from /api/v2/user/ideasinmotionmedia!albums?start=201&count=50. Error: Too many errors"

Any help on troubleshooting this would be appreciated.

[Michael Riley]

Something else to note:

When I tried running the same request via Postman, I ended up getting a result that wasn't a 404 when I omitted start and count. I am also on Windows 10

[Tony]

Attempting the auto-generated API call from the script: /api/v2/user/<USERNAME>!albums?start=201&count=50 with successfully authenticated tokens using the https://smugmug-api-authenticator.herokuapp.com/ gives this JSON response:

{
    "Response": {
        "Uri": "/api/v2/user/ideasinmotionmedia!albums?start=201&count=50&_pretty=&oauth_consumer_key=<KEY>&oauth_nonce=<NONCE>&oauth_signature_method=HMAC-SHA1&oauth_timestamp=<TIMESTAMP>&oauth_token=<TOKEN>&oauth_version=1.0&oauth_signature=<SIGNATURE>",
        "Locator": "Album",
        "LocatorType": "Objects",
        "UriDescription": "All of user's albums",
        "EndpointType": "UserAlbums"
    },
    "Code": 404,
    "Message": "Not Found"
}

It seems like the start point is being calculated a bit off for some reason, starting on what I assume is around the 10,000th gallery which doesn't exist.

But manually replacing the start param in the endpoint with 1: /api/v2/user/<USERNAME>!albums?start=1&count=50 successfully returns our album list beginning on the most recent album.

[Tony]

Huh, so this might be a weird SmugMug issue. I was wrong about that 10,000th gallery assumption.

So, using their web interactive API, we've got working results starting on gallery 151 until 200. Then the next page just doesn't exist on their servers for some reason?? Very weird.

Galleries 151-200

Galleries 201-250

We'll reach out to their API team and see what kinda funky stuff is going on behind the SmugMug veil.

I guess this does affect the downloader as well though since we can't trust that SmugMug's albums endpoint is iterably stable. Is there another way to query through albums from most recent backwards besides hitting that endpoint and scanning the pages? This could be a SmugMug fluke only affecting us. Will keep you updated on what they have to say

[Tommaso Visconti]

Thanks @tonyketcham @ErriteEpticRikez keep me updated.
As per the credentials, I found out they're also available in the smugmug interface, check the readme at https://github.com/tommyblue/smugmug-backup#obtain-tokens

[Michael Riley]

Hey @tommyblue,
@tonyketcham and I ended up figuring out the issue. Whenever you delete a album, for some reason it was still being detected in the Smugmug API. But when you try to hit the deleted gallery it ends up getting a 404. Smugmug's API team has yet to reply to the issue.

I ended up modifying the source code to ignore albums that threw any http error (which I know is less than ideal). Would you be opposed to adding a parameter to skip albums with 404 errors? I think I could partially contribute, but I wanted to know ahead of time that was something you would be accepting if I were to file a merge request.

[github-actions]

Stale issue message