Video in "Processing" causes download to stop

[Vitaly E.]

Thanks for the great tool! It works extremely well, the documentation is clear and useful.

Unfortunately, it exits when trying to download a video being processed. I have a video from more than a year ago that is stuck in "Processing", probably because of a glitch in SmugMug. The response for that video comes as follows, translated from the Go struct:

{
    "FileName": "00003.MTS",
    "ArchivedSize": "317319168",
    "ArchivedUri": "https://photos.smugmug.com/aaa/bbb/ccc/ddd/eee/0/D/00003-D.jpg",
    "IsVideo": true,
    "Uris": {
        "LargestVideo": {
            "Uri": ""
        }
    }
}

As you may guess, the empty URI concatenated with baseAPIURL becomes https://api.smugmug.com and returns an HTML page, which obviously cannot be parsed as JSON. As a result, I get

2019/06/28 23:04:57 https://api.smugmug.com: decoding response
status code: 200
[ERR] invalid character '<' looking for beginning of value
2019/06/28 23:04:57 #1 : Get : unsupported protocol scheme ""
2019/06/28 23:04:59 #2 : Get : unsupported protocol scheme ""
2019/06/28 23:05:01 #3 : Get : unsupported protocol scheme ""
2019/06/28 23:05:01 Get : unsupported protocol scheme ""
2019/06/28 23:05:01 Get : unsupported protocol scheme ""
2019/06/28 23:05:01 Get : unsupported protocol scheme ""
2019/06/28 23:05:01 Too many errors

Printing of the target URL and status code was added by me, in case json.NewDecoder(resp.Body).Decode(&obj) returned an error. For now, I worked around the problem by deleting the corrupted video. I would be grateful though if you could fix the code.

[casperlk]

I'm running into what appears to be the same issue. Could you share the code that you used to find out which video was causing issues?

[Vitaly E.]

@casperlk have a look at my fork of this repo, I put the modifications there. Keep in mind that it will generate a lot of output, printing every single response until a failure. You might want to redirect it to a file, like this $GOPATH/bin/smugmug-backup -user username -destination path > output.log.

When the program fails, it will print the last URL it was trying to fetch. Here's an example:

2019/09/15 11:54:14 https://api.smugmug.com: decoding response
status code: 200
[ERR] invalid character '<' looking for beginning of value

It will also dump the page contents into a file named smugmug-response-<timestamp>.txt in the destination path.
Here, the program tried to access the API home page (HTML). You need to analyze the generated output in order to understand why it had been redirected there. In my case, it was because of

"LargestVideo": {
        "Uri": ""
}

Good luck!

[casperlk]

Thanks @empovit! That's helpful. The end of the log is:

IsVideo:false Uris:{LargestVideo:{Uri:}}}] Pages:{NextPage:}} {NextPage:} 2019/09/15 13:01:37 https://api.smugmug.com: decoding response status code: 200 [ERR] invalid character '<' looking for beginning of value Getting 2019/09/15 13:01:37 #1 : Get : unsupported protocol scheme "" 2019/09/15 13:01:39 #2 : Get : unsupported protocol scheme "" 2019/09/15 13:01:41 #3 : Get : unsupported protocol scheme "" 2019/09/15 13:01:41 Get : unsupported protocol scheme "" 2019/09/15 13:01:41 Get : unsupported protocol scheme "" 2019/09/15 13:01:41 Get : unsupported protocol scheme "" 2019/09/15 13:01:41 Too many errors

Correct me if I'm wrong, it seems like the script is making a blank request for the next page of images:
Pages:{NextPage:}} {NextPage:}

I'm puzzled about why it would make a blank request, it should be checking
for nextPage != ""
before sending the requests. Any idea what could be happening here?

[Tommaso Visconti]

@casperlk @empovit I'm really sorry, I completely missed the notifs for this repo and so I didn't see new issues. I'm happy to fix this bug.
Reading your messages the fix could be to check the value of LargestVideo > Uri and skip if empty, but I'm also wondering if there's a way to find out from the api response that the video is in Processing status and skip it

[Tommaso Visconti]

Fixed in master. The AlbumImage object has a handy boolean Processing attribute. I'm now skipping files under processing:
841e434